Fooocus 2.5.0: Enhance

[mashb1t]

Fooocus 2.5.0: Enhance

This release includes a feature requested multiple times by the community (e.g. in #3113, #3089, #3039, and a few more, also see #3122). Implementation by @mashb1t in v2.5.0 (fork) / mashb1t#42 and now available in main.

What does this feature do?

Enhance allows you to automatically upscale and/or improve parts of the picture based on either a prompt or an input image.
It is comparable to ADetailer (repository), but offers better and more flexible object detection and replacement with detection and replacement prompts instead of static detection models, each having ~140MB.

How do i use it?

Disclaimer

It is highly recommended to use performance Speed or Quality (no performance which loads a LoRA) as any inpaint engine uses a LoRA, which may not produce the best results when combined with other performance LoRAs.
Using inpaint mode Improve Detail (face, hand, eyes, etc.) does not set an inpaint engine, making it compatible with all performances. The documentation of inpaint modes can be found here: #414
All of this is also the case for normal inpainting without enhancements.

ControlNets (ImagePrompt, PyraCanny, CPDS, FaceSwap) are currently not supported for enhance steps but can be used for image generation used as basis for enhance.

With image generation

1. generate any image with the settings you'd like to use
2. if you're satisfied, disable random seed and open the enhance tab
   2.1. (optional) Enable and define order upscaling or variation (default disabled)
   2.2. Enable and configure any amount of other improvement steps.
   2.3. input detection prompt (what you want to detect in the image)
   2.4. input positive / negative prompt (what you want to replace the detected masks with, defaults to your normal prompts if not set)
3. generate image

Based on an existing image

Simply open Image Input and upload an image to the Enhance tab. Prompt processing will be skipped and only enhancement steps are processed. Follow steps 2+ above.
You may set --enable-auto-describe-image to automatically generate the prompt after image upload.

Examples

UI

Original (generated)	Enhance

Original (generated)	Upscale (before)	#1 Yellow Sundress	#2 Hands replacement

Upscaling or Variation

Before First Enhancement

• always use original prompts for upscaling or variation
• improved details when enhancing due to larger canvas size
• less context of image composition due to inpainting with 1024x1024
• recommended for enhancing smaller areas (details, faces, eyes)

After Last Enhancement

• may apply changes to the image which were already fixed by enhance
• recommended for enhancing large areas (backgrounds, clothes, whole persons)
• choose which prompt to use for upscaling or variation, use either original prompts or last filled enhancement prompts with non-empty value (increases subject recognition, prevents subject from being changed back to original prompt after enhancement. Use this when substantially changing the subject described in the original prompts)

Original (generated)	Upscale (before)	Enhance

Original (generated)	Enhance	Upscale (after)

Models

By default it uses the SAM (website, repository) masking model, backed by GroundingDINO (paper, diffusers docs), but offers support for all additional models currently supported by RemBG (repository). GroundingDINO + SAM do not use RemBG as handler, but have natively been implemented into focus for even better results and increased level control.

Currently supported models:

• SAM + GroundingDINO (default): A model combination which allows detection prompts and segmentation of objects in detected boxes in 3 variants (base (default), large, huge)
• u2net (download, source): A pre-trained model for general use cases.
• u2netp (download, source): A lightweight version of u2net model.
• u2net_human_seg (download, source): A pre-trained model for human segmentation.
• u2net_cloth_seg (download, source): A pre-trained model for Cloths Parsing from human portrait. Here clothes are parsed into 3 category: Upper body, Lower body and Full body.
• silueta (download, Model size reduced to 43Mb! and new Webapp xuebinqin/U-2-Net#295): Same as u2net but the size is reduced to 43Mb.
• isnet-general-use (download, source): A new pre-trained model for general use cases.
• isnet-anime (download, source): A high-accuracy segmentation for anime character.

Tech Debt / Code Improvements

While implementing the enhance feature, multiple methods have been introduced to make code reusable and allow for iterations.
The whole async_worker.py has been restructured and is now much more clear to read and easier to use.

Debugging

Please find debugging options in Developer Debug Mode > Inpaint:

• Debug Enhance Masks: outputs generated masks, default disabled
• Debug GroundingDINO: early return in the detection process, uses GroundingDINO boxes instead of refined SAM detections inside those boxes. This option can sometimes improve results when SAM isn't able to correctly detect inside GroundingDINO boxes. FYI: GroundingDINO supports detection prompts, SAM auto-detects objects without further promps inside of the boxes.
• GroundingDINO Box Erode or Dilate: increase/decrease the detected box area by GroundingDINO. This option can sometimes improve SAM detection when GroundingDINO accidentally cuts off elements (very rare)

GroundingDINO	GroundingDINO + SAM

[ErmMonzon]

Appreciate the updates and your hard work on maintaining Fooocus! Enhance works really well. Can you verify that for Pony models you still have to prompt the score? E.g score_9, score_8_up, score_7_up, score_6_up. From my tests so far it looks like you still have to. Just want to verify.

[DavidDragonsage]

The Pony preset loads the Pony style. The Pony style adds "score_9, score_8_up, score_7_up, " to the positive prompt and "score_6, score_5, score_4" to the negative prompt.

Like all other styles, you do not see these additional words on the Fooocus Settings screen. However, the "Full raw prompt" entries in the log shows that they have been added.

[ErmMonzon]

aaaahhhh! Should've looked at the log! hahaha! Appreciate the confirmation

[LuisFernando79]

May God bless you in his holy glory, thank you for your updates to my favorite tool.

[Vsatyam013]

Can i use fooocus 2.5.0 enhance in google colab, ? if yes, then how.

[mashb1t]

@Vsatyam013 You can find the link to Colab in the readme and the docs linked in Fooocus when clicking the enhance checkbox.

[alexatnv]

This is great! Thank you!

I played around and it's very cool. One thing I am wondering - if someone is wearing a t-shirt, how can I say make them wear a suit and tie? I tried a couple things but it seems confined to the bounds of the original clothing. Is there a way to segment this so that it uses more area beyond the bounds of the clothing? I tried "Below the neck and body" "Body' etc.

[mashb1t]

You can use the dilate mask option to expand the segmented mask in all directions, but it's currently not possible to just automatically generate a maks by prompt without any base inage and you're better off using normal inpainting for that.

[mzisa]

Hi, many thanks for the new release and the hard work. I am currently experimenting Enhance on an upload image (in the related section), it works very well with the face but fails while trying to enhance the body, or the torso, I mean it inpaint/enhance also the face when I don't request it explicitly in the enhance prompt. Is there a way to avoid this behavior? Am i doing any wrong steps? Thanks for your feedback

[mzisa]

Additional note: It seems working with "torso" as enhance prompt, fails with body/breasts for the body upper part

[mashb1t]

@mzisa try to experiment with different keywords as well as check if the clothing detection model (upper body) works better in this case.

[Yogesh-DevHub]

Thank you so much for the work. I see 'translate prompt' option as well but didn't find after git pull. was that discarded?

[mashb1t]

@Yogesh-DevHub translate prompt feature is available in my fork and uses google translator, which many of Fooocus' users can't use due to not having access to it (Chinese firewall, blocked by network, offline usage, etc.) => not in upstream, only in my fork.

[linplug]

Thank you for your hard work! Great product!
But after the update, you can clearly see the border of the mask. I do not know with what it is connected, but if I do for example a mask of a person's face and I want the face to be left, and the body or clothes re-rendered can clearly see the boundary of the transition between the mask and the new image. This was not the case before the update. Can you tell me what to do to make the mask was not noticeable?
Thank you!

[mashb1t]

@linplug the generation of images using a mask hasn't changed kn 2.5.0, but new methods of creating masks have been added.
You may dilate the mask to enlarge it (masking the face => use the dilate or erode slider for mask adjustment). You can find the slider directly in each of the enhance tabs or in advanced > advanced > developer debug mode for generating images without enhance.

Also try to experiment with different inpaint modes.

[linplug]

Bro, thanks for the quick reply! You're the best! I'll keep trying and figuring it out! =)

1. Can you please tell me if it is possible to do color correction of the image?
2. If we take a mask of a person's face and change only the environment, is it possible to make the mask perceive the lighting of the environment and color correction of the new environment. Just often the new image is different in color from the mask.
   If not, I think this functionality would just be fanastic.

Bro, thank you again! Hugged you tight!

[mashb1t]

@linplug You're better off using Photoshop or any image software for this purpose, not GenAI, as GenAI tools such as Fooocus do not only change colors in an area. Color corrections without changing the image content are not easily possible in Fooocus or SDXL only. With very low denoising strength you'll not get any effect, with large denoising strength you'll basically get new faces, so you'd have to find the sweet spot.

There are some forks having Photopea included, which can provide this functionality on the same website (currently don't know the fork anymore, but saw some recently with less than 5 stars).

[EvilBT]

I also encountered this bug. Occasionally, there will be an obvious square redraw area, but the mask is the face itself, not a square mask. In theory, the redraw enhancement should not have a square redraw area with obvious square boundaries.

[builder-main]

Thank you it does work well.

I observe a certain behaviour, when Inpainting then enhancing with Upscale or Variant, it seems that the same seed is used for all the enhance Upscaling/Variant. Meaning that when generating 2 inpaints, the 2 Upscale/Variant have the same features out of the masked area (which undergone heavy changes). I'd expected the unmasked area to get a differnt variant from the upscale factor, but they get the very same result.

Is this expected?

Inpaint + Upscale Mask area

Inpaint + Upscale unMasked area

[builder-main]

I believe it is beacause the Mask is also applied to the enhancement. When using Vary Strong you can see that the outside area does not vary at all. Any way to allow enhance to apply on the whole image instead of the mask only ?

[nazimbeltran-ai]

This is amazing. Just finished a course that pointed me towards Fooocus. Keep up the great work!