FreeU

[3Diva]

I don't know EXACTLY what it is (I read a little bit about it here: https://github.com/ChenyangSi/FreeU/tree/main ), but I thought I'd go ahead and test it out and document want it looks like when applied. For this test I used the same seed and prompt and disabled the the styles options to assure that nothing else is effecting the generated image. I'm using the checkpoint model CrystalClear XL with no refiner and no lora.

I'm making BIG number value changes - using large number increments so that I can better see how each setting effects the image.

Base image with FreeU Not enabled:

FreeU enabled, with default settings of B1 = 1.02, B2 = 1.02, S1 = 0.99, S2 = 0.95:

FreeU with B1 = 2 (other settings default):

FreeU with B2 = 2 (other settings default):

FreeU with S1 = 1.5 (other settings default):

FreeU with S1 = 2 (other settings default):

FreeU with S1 = 2.5 (other settings default):

FreeU with S1 = 3 (other settings default):

FreeU with S1 = 3.5 (other setting default):

FreeU with S1 = 4 (other settings default):

FreeU with S2 = 1.5 (other settings default):

FreeU with S2 = 2 (other settings default):

FreeU with S2 = 2.5 (other settings default):

FreeU with S2 = 3 (other settings default):

FreeU with S2 = 3.5 (other settings default):

FreeU with S2 = 4 (other settings default):

I REALLY like the results with the default FreeU settings. I think that FreeU improves the images quite nicely!

[3Diva]

Second Round of Testing. This time we're going BELOW the base settings. As above I'll be only changing one setting at a time and leaving the rest on their default settings.

B1 = 0.5:

B1 = 0.25:

B2 = 0.5:

B2 = 0.25:

S1 = 0.5:

S1 = 0.25:

S2 = 0.5:

S2 = 0.25:

S2 = 0:

[3Diva]

Here are some settings combos I ended up really liking for this image.

I'll have to do further testing to see how the settings look with different images.

[AkoZ]

About FreeU : https://github.com/ChenyangSi/FreeU which is already (yesterday ?) included in FoOocus ! ;)
Copying here some youtube FreeU comments :

• the larger blobs are B values control the style and image quality and values around 1 for both give good results.
• tiny details and textures are the S-values control the quality/number (e.g. number of legs of an animal) of the objects on the image
  ** best results with values around and higher for S1 and with low values between 0-0.3 for S2.
  ** When the distance between S1 and S2 becomes smaller (S1~= S2), more and more objects are generated.
• Normally the S and B interplay in a way that tends to find balance the high (S) and low (B) frequencies of the image.

[3Diva]

Thank you for the great info! Could you clarify what this means: "best results with values around and higher for S1"?

[AkoZ]

• just that S1 is > S2 but very near ;) as 0.5 if S2=0.3 but that stays as theory as not so viewable.

• S textured lines are mitigating B (smoother colored)

• • seen in article, well the B=S=1 seems really significant in total balanced scene - then affect more about (B) flat mass and (S) textures, then after those factors are applied to amplify the details S or the semantic whole B.

• here is the paper: https://arxiv.org/pdf/2309.11497.pdf and many here are citations from it ! better understanding about low and high frequencies is described as some "High" seems like (controlnet canny ones)/ (** S = sketches** as lines (but named skip feature scaling factors) and then the flat or well to say the fullness is said about B= Blobs ( named backbone feature factors inside the lines is the colors and bold...)

• this is essential: "the diffusion models are comprised of the diffusion process and the denoising process. During the diffusion process, Gaussian noise is gradually added to the input data and eventually corrupts it into approximately pure Gaussian
  noise. During the denoising process, the original input data is recovered from its noise state through a learned sequence
  of inverse diffusion operations. Usually, a U-Net is trained to iteratively predict the noise to be removed at each denoising step."
  
  -- a " progressive denoising process, showcasing the generated images across successive iteration"
  __ B: a " gradual modulation of low frequency components, exhibiting a subdued rate of change,
  __ B: "global layouts and smooth color. These components encapsulate the foundational global elements that constitute the image’s essence and representation"
  __ B: " the backbone feature factors, aims to amplify the feature maps of the main backbone, thereby bolstering the
  denoising process. However, we find that while the inclusion of backbone feature scaling factors yields significant
  improvements, it can occasionally lead to an undesirable oversmoothing of textures.

• To mitigate this issue is introduced the second factor, skip feature scaling factors, aiming
  to alleviate the problem of texture oversmoothing
  __ S: "..while their high-frequency components display more pronounced dynamics throughout the denoising process"
  __ S: ".. rapid changes in the images, such as edges and textures. These finer details are markedly sensitive to noise, often
  manifesting as random high-frequency information whennoise is introduced to an image. Consequently, denoising processes need to expunge noise while upholding indispensable intricate details

• " FreeU, which substantially enhances the sample quality of diffusion models without incurring any additional computational costs. Motivated by the fundamental role played by both skip connections and backbone features in U-Net architectures, we conduct
  an in-depth analysis of their effects in diffusion U-Net.
  Our investigation reveals that the primary backbone (B-factor) primarily contributes to denoising, while the skip connections (S-factor) predominantly introduce high-frequency features into the decoder, potentially leading to a neglect of essential backbone semantics.
  To address this, we strategically re-weight the contributions originating from the U-Net’s skip connections and backbone feature maps. This re-weighting process capitalizes on the unique strengths of both U-Net components,
  resulting in a substantial improvement in sample quality across a wide range of text prompts and random seeds.

• well it seems that when they pass other "filters" as the controlnet ones they will find other "moderating" factors then.. as here is the only semantic (B flat bold Blobs) and the S which is not precise as a controlnet HED or MLSD .. precisions will be found early now.

[lllyasviel]

freeu has some unknow numerial precision problems with AMD or MAC devices so it cannot be set as default.
also it has same problem as negative adm that it sometimes mess small objects like human eyes ( especially when eyes are small, eg full body images
this can be solved by early stopping of FreeU (and, negative adm already has early stopping now so it will not mess small objects and eyes, in theory)
but FreeU does not have early stopping and I am going to write it if the numerial precision problems with AMD or MAC can be solved and when it is safe to use it

[AkoZ]

@lllyasviel thank YOU so much for your great participâtion to the stable diffusion tools ! Great one you are !

[Abracadd]

And where to upload an image to use the freeU mode ?