Tell us what is missing in ControlNet Integrated

[lllyasviel]

The integrated ControlNet is not updated for a while, and we are going to make it a bit more up-to-date.

However that will happen after some other newer experiments. Old features are relatively lower priority - so that we can experiment with more new ideas first.

I personally know some models like uni-controlnet (or called promax by some non-research people) and some preprocessors like DSINE/DepthAnything/etc. Those are very easy to add – after I get other things done, those only need me 1 or 2 hours to add them all.

However I will not add the below things:

1. webui's IPA multi-inputs implementation. The reason is that I find them misleading and technically wrong. One can just use multiple ControlNet Units to have similar logic. The “multi-inputs” implies some feature mixture techniques when it is not. So it is wrong and I cannot add it. (but will consider some technically correct multiple IP mixture later if flexible
2. Automatic model filtering and ipadapter preprocessor recognizing. I find those things very annoying since model filtering is somewhat bind to model load and is very laggy. Also I cannot not know from UI what are the underlying rules and how those automatic recognizing works (and if they worked after all) when I click generate.
3. All things that I find hard to understand – if I cannot quickly understand what it is then most people will not understand it.

By the way, several experiments will be implemented in a “Forge Space” (that I will add later, similar to some local HuggingFace Spaces) rather than that ControlNet extension. So many functionalities will happen elsewhere.

This post is only for controlnets. Experiments with other newer diffusion models will happen before controlnets.

Please tell us what is missing in ControlNet Integrated. Please do not talk about things that are already mentioned in this post – I will take a look several days later when I finish some other things.

---

Update Sep 1:

The rewrite of ControlNet Intergrated will start at about Sep 29. The estimated finish date is about Oct 7. When this note is announced, the main targets include some diffusers formatted Flux ControlNets and some community implementation of Union ControlNets. However, this may be extended if stronger models come out after this note.

---

About IPAdapter:

Note that the CLIP Vision names of sd-webui-controlnet are non-standard. Forge uses correct names:

SD v.	IPadapter	Img encoder
v1.5	ip-adapter_sd15	ViT-H
v1.5	ip-adapter_sd15_light	ViT-H
v1.5	ip-adapter-plus_sd15	ViT-H
v1.5	ip-adapter-plus-face_sd15	ViT-H
v1.5	ip-adapter-full-face_sd15	ViT-H
v1.5	ip-adapter_sd15_vit-G	ViT-bigG
SDXL	ip-adapter_sdxl	ViT-bigG
SDXL	ip-adapter_sdxl_vit-h	ViT-H
SDXL	ip-adapter-plus_sdxl_vit-h	ViT-H
SDXL	ip-adapter-plus-face_sdxl_vit-h	ViT-H
v1.5	FaceID	InsightFace+CLIP-H
v1.5	FaceID Plus	InsightFace+CLIP-H
v1.5	FaceID Plus v2	InsightFace+CLIP-H
v1.5	FaceID Portrait	InsightFace+CLIP-H
SDXL	FaceID	InsightFace+CLIP-H
SDXL	FaceID Plus v2	ViT-H

[sdbds]

There are two new controlnet-based architectures to try:

AnyControl

One input with multiple preprocessor inputs

ControlNeXt

Modified variants of some blocks that seem to perform well on SVD and SDXL.

[BNP1111]

instant id+inpaint PLZ

[LeDXIII]

First of all I want to say - thanks for the Great tool. That the previous version was very good, that the new version looks very promising (judging by commits).

Regarding controlnet.

1. Already realized implementations. I'll just walk through them briefly to avoid forgetting anything.

• Anyline.
• A set of controlnets from Xinsir. In addition to Union (which has problems in realizing openpose in Automatic), the author has Tile, which also works in the Automatic controlnet not quite as expected.
• Ip-adapter compositions.
• I would also like to see SAM (2?), lama cleaner in controlnet in preprocessors.

2. From the new stuff.

• Photomaker 2. In the demo it shows very good results on face repeatability.
• CustomNet. Not sure how good it is.
• Controlnet of SD3 and other new models.
  Thank you.

[IPv6]

A feature request: segmentation tagging control net (not yet existing)

Inputs: character in default frontal pose (photo), a segment map where segment color defines clothing region in frontal pose, a segment map with same colors but in target pose.

Desired output: ControlNet should map clothing features from default pose into target pose using color-coded segments as strict guidance. That means - without adding features not present in segment in default pose, without symmetry breaks and so on. Probably specific color channel should hint about SYMMETRY for robustness. Target pose should not be restricted to "usual poses", theoretically upside-down poses, partially occluded poses, self occlusions can be resolved just fine by such 1-1 segmentation mapping

[ali0une]

The outpaint of xinsir union-controlnet-promax ... if i could outpaint with SDXL it would be wonderful.

Kepp up the good works ...

[dongxiat]

I hopefully update controlnet will work with 2 major extension :
segment-anything
replacer
its help my job alot...and want to use with in webui-forge

btw ur works is awsome

[andupotorac]

I would like to add to this. Don't include just SAM2, but also pair it with Florence as it makes for detecting areas much better.

1. https://huggingface.co/spaces/SkalskiP/florence-sam

[jordanjay29]

The biggest need here might simply be a document outlining how Forge's built-in CN differs in options and behavior. It's not much, but the will-not-adds and the workarounds to some of those would be useful for newcomers and converts from A1111.

[AxelFar94]

First of all, Thank you for your work lllyasviel.
I would like to point out that the tile_colorfix and tile_colorfix+sharp are making the images bleed their colors in a checkered pattern, and this happens since Forge launch, here some comparisons with Auto 1111 results using low ControlNet strength:

The checkered pattern is more noticeble in very low strength:

Would be great if you could fix it! It's a great way to make images strictly follow input image colors.
Thanks!

[GeminiSquishGames]

Well, I can't get it to work in the Photoshop Plugin/Extention... that's missing for me, but probably not something that can be helped since it's pretty much not getting updates on their end. I'd probably rather have it work faster in the UI than have a poor implementation in photoshop anyway at this point. Looking forward to seeing "The Chosen One" for character consistency soon as a CN model or extension of some kind.
https://omriavrahami.com/the-chosen-one/

[Crousere]

Fooocus sdxl inpaint. Inpaint_v26.fooocus.patch never worked as well on Forge for me as it did on Foocus.

[newxhy]

Inpaint_v26.fooocus.patch 在comfyui 上的效果也不如在 Foocus 上。
The effect of Inpaint-v26.foocus.patch on comfyui is not as good as on Foocus.

[andupotorac]

1. For Depth, I think it's best to look at https://depthfm.github.io/. Seems better than even Marigold.

2. Hand and feet fixing. For hands there are several approaches, and I believe the same algos could work on feet also. I'm not too technical to know, but I posted some of those data here: Can this be used for feet too, with datasets like FOOT3D and FIND? agnJason/XHand#2.

For hands you also have:
https://github.com/wenquanlu/HandRefiner

And this is looking interesting as well:
https://github.com/adventurer-w/ClickDiff

I've seen people add manual images to OpenPose, but ClickDiff does it automatically, and the other do too. Would be great if we can finally fix hands and feet with ControlNet.

3. There are some options you could add on the models to make them more robust:

• https://shariqfarooq123.github.io/loose-control/ (more flexibility for shapes)
• https://github.com/liuxiaoyu1104/SmartControl (takes context into account for better generations)
• https://github.com/open-mmlab/AnyControl (like multicontrol net, but better quality)

4. Would be nice to integrate LivePortrait (for humans and animals too) so we can have a ControlNet for emotions. :)

[softtaco1]

1.) SDXL / Pony inpainting would be amazing.

2.) A custom ControlNet which utilizes either photos or prompts to transfer outfits/clothing onto a generation. This could maybe be enhanced by utilizing layerdiffuse to overlay it correctly upon the subject through multiple passes. (Idea, not sure if could be implemented)

3.) I don't know if this is possible either, but instead of something like regional prompter or Forge Couple, creating a controlnet that instead recognizes separate individual characters interacting. I know there are things similar to this that exist, but they aren't quite what I'm looking for to generate multiple characters interacting easily while maintaining a level of versatility and not being constrained to one specific pose via OpenPose. If this idea is at all possible, a controlnet model could utilize a reference image to apply an interaction/action between multiple characters. Or instead of a reference image, it could utilize a syntax similar to Forge Couple's "NEWLINE" , but instead of generating in a new space of the 2d plane, applies character adherence to subject(s) in the scene. Then, an action could be specified, detailing how these characters are interacting. Basically, a prompt adherence helper for models that struggle with prompt adherence, especially as it relates to characteristics of specific individuals in a scene.

Example:
A dark cave
NEWCHAR A frightened middle-aged man, yellow poncho, soaking wet, holding a lantern
NEWCHAR An eldritch horror monster, devilish grin, glowing eyes.
ACTION Fighting

Again, not sure if this is even possible without a multimodal model, but I think it would be more useful to have something that helps ensure characters look a certain way than it is to make sure they are in a specific place/pose (most of the time.)

[pflky]

For me, what's missing is

1. Union ProMax model support
2. Dsine preprocessor support in Forge (most of the time it's better than all the other preprocessors, including depth and canny)

2a. Dsine on independent controlnet for A1111 has a bug that if you use a different resolution/ratio than the image, the preprocessor will generate zoomed into the image rather than crop or resize. Its been a bug since the release of Dsine as a preprocessor. This happens regardless of settings. You essentially have to format images to be the same exact resolution and ratio as the preprocessor settings, otherwise it zooms in, which can be a pain if you're working with images of mixed resolutions.

2b. Propose a feature based on the last problem: A new feature that allows people to use a draggable and resizable selector box over the controlnet image to determine the focus area of the preprocessor. I'd imagine this would also be useful for other controlnet preprocessors too, as you can then take any part of any image and make it the focus of the preprocessor on the fly. So for example, you drag a 1:1 ratio box overlay on the image, have it be resizable, and then it'll use the inputted resolution, even if the image itself is smaller; So you could zoom into a face on a 512x512 image and make just the face area preprocess at 1024x1024. Simple and versatile way of handling controlnet inputs of varying resolutions and aspect ratios and fills the gap that "just resize" and "crop and resize" doesn't cover.

3. Photomaker V2 support (faceid model)
4. DepthAnything V2 support (this outperforms marigold)
5. With the advent of Union models, the overhead for using multiple controlnet preprocessors should be smaller, because you should only need to load one model for multiple control units. I don't know if this is the current behavior, but if not, then if the same model is selected across different units, it should only be loaded into memory once.
6. There is a newer lineart preprocessor called "Anyline", which seems to create better preprocessing for lineart models. https://github.com/TheMistoAI/ComfyUI-Anyline
7. The ability to use extensions that rely on A1111 non-built-in controlnet (if possible). Some of these support things that Forge controlnet does not, like DepthAnything V2 for example. So, some kind of scheme that allows extensions that call on A1111 controlnet, to play nicely with Forge controlnet. It's really one of the only reasons I keep going back to main A1111. Forge has the memory management enhancements, but a lot of stuff gets made for A1111 that doesn't play well with Forge's built-in controlnet. So if it's possible to make things that call on A1111 controlnet to convert to use the built-in controlnet in Forge, that could solve a lot of problems and overhead, including having to go through the trouble of supporting certain new models.
8. "Contrast" feature for post-processing preprocessor results. For example, I often find things can get too blended with depth preprocessing images, where the white parts lose detail because of the depth estimation. The detail is technically still there, just in a very limited visual dynamic range. If you could essentially add contrast on the fly, you could change the preprocessor image to be more contrasty, allowing those details to be more visible, along with being more visible to the model itself. So for example, a simple contrast slider in controlnet that can apply an adjustment to the preprocessor image before it's plugged into the controlnet model. It can be just a little vertical bar beside the image that has like 5 stops on either end, so 0 as default and +5/-5. I think this could be useful for a variety of controlnet preprocessors, This should not apply to the base image, but the preprocessed image, so it should probably be in the preprocessor preview window, and maybe only visible if someone has the "allow preview" option activated. Users can then look at the preview to determine if they want to add or reduce contrast, and whatever setting is active when generation is started, is processed with that contrast on the fly and then sent off to the controlnet model.

[Satochiju]

Illyas, I'm very new to this and probably the most ignorant of all those involved here, but that doesn't stop me from telling you that I admire not only your great work in Forge, which by the way has been very useful for me as a graphic designer, but, above all, your selflessness. I'm sure that if there were more people like you in this world, things would be different, as we say in Chile. Thank you very much!! A big hug.

[Satochiju]

Ah! y lo nuevo de Forge funcionando muy bien en mi 3060ti, algo lento aún pero con muy buena calidad, Flux y SDXL

[V65165188f]

Hello, I'm new switch to ForgeUI but it seems the integrated controlnet batch function doen not work properly, for the batch option it created multiple masks but only generate 1 image like the picture shows.

I really like to use batch controlnet, truly appreciated if this issue can be addressed.

[westNeighbor]

Yeah, this is what I need all the time, but Forge just doesn't work for this simple function. But I rely on Forge cause it's much faster than A1111.

[reaperhammer]

+1

[jt-michels]

Came here to post this but here it is+1

[ivaxsirc]

The integrated ControlNet is not updated for a while, and we are going to make it a bit more up-to-date.

However that will happen after some other newer experiments. Old features are relatively lower priority - so that we can experiment with more new ideas first.

I personally know some models like uni-controlnet (or called promax by some non-research people) and some preprocessors like DSINE/DepthAnything/etc. Those are very easy to add – after I get other things done, those only need me 1 or 2 hours to add them all.

However I will not add the below things:

1. webui's IPA multi-inputs implementation. The reason is that I find them misleading and technically wrong. One can just use multiple ControlNet Units to have similar logic. The “multi-inputs” implies some feature mixture techniques when it is not. So it is wrong and I cannot add it. (but will consider some technically correct multiple IP mixture later if flexible
2. Automatic model filtering and ipadapter preprocessor recognizing. I find those things very annoying since model filtering is somewhat bind to model load and is very laggy. Also I cannot not know from UI what are the underlying rules and how those automatic recognizing works (and if they worked after all) when I click generate.
3. All things that I find hard to understand – if I cannot quickly understand what it is then most people will not understand it.

By the way, several experiments will be implemented in a “Forge Space” (that I will add later, similar to some local HuggingFace Spaces) rather than that ControlNet extension. So many functionalities will happen elsewhere.

This post is only for controlnets. Experiments with other newer diffusion models will happen before controlnets.

Please tell us what is missing in ControlNet Integrated. Please do not talk about things that are already mentioned in this post – I will take a look several days later when I finish some other things.

For me, SUPIR as controlnet or extension option

[Satochiju]

Hi illya, just to thank you and tell you that Spaces is great, but what you've done with Diffusion in Low Bits is amazing, my 8G VRAM card no longer suffers hahaha

[ryderrrr123]

When i try to use controlnet for flux, it doesn't work at all. when there will be fix for it or it is a controlnet model problem?

[stripealipe]

It doesn't yet work with Flux when used in this version of Forge/ControlNet. Hoping the guys manage to get it working soon though! So much hard work going into Forge over the last few days...

[softtaco1]

I’d love to see your take on an upscaler utilizing multiple ControlNets. I know you can kind of do this in Img2Img, but something that needs less tinkering. Basically, something that utilizes multiple ControlNets like IP-Adapter, Tile etc to make an upscaled image. The ControlNets are already preselected and processed on the backend based on the base model (SD1.5,SDXL,Flux).

[gabecastello]

Appreciate this repo a lot! It would be great if the mask had a copy of the input image so that you can see clearly what you are masking.

[SunGreen777]

I've been using ForgeUi recently. It's good to make presets settings.

[alebeard]

I think you've been doing amazing so far I would really love it if you could get Regional prompter extension to work again, But I figure that you are getting stuff squared away at your own pace.

…

On Fri, Aug 23, 2024 at 3:15 PM SunGreen777 ***@***.***> wrote: I've been using ForgeUi recently. It's good to make *presets settings.* — Reply to this email directly, view it on GitHub <#932 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHN7U7YEHNNT4OO5FTNP5ODZS6YBPAVCNFSM6AAAAABMAFM6XCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBTGQ4TGNQ> . You are receiving this because you are subscribed to this thread.Message ID: <lllyasviel/stable-diffusion-webui-forge/repo-discussions/932/comments/10434936 @github.com>

[sangoi-exe]

deep anything v2 😬😁

[westNeighbor]

Really need this option.

[fgtm2023]

i mention some suggestions here: #1564 , but mainly for controlnets i hope you add ability to use photomaker2 with other controlnets like openpose....etc, and i hope we have a refine for sdxl outpainting, something kinda close to photoshop outpainting will be amazing and outpaint even without describing the outpointed area details, i know we can use inpainting to outpaint but the way we can do this with, and the results are so bad, and thanks a lot for your really amazing work and brilliant ideas like the new spaces idea which i hope we have more new spaces on it.

[pitardven]

I would really like to see it Pix2Pix

[Gushousekai195]

Update Sep 1:

The rewrite of ControlNet Intergrated will start at about Sep 29. The estimated finish date is about Oct 7. When this note is announced, the >main targets include some diffusers formatted Flux ControlNets and some community implementation of Union ControlNets. However, this >may be extended if stronger models come out after this note.

It's gonna be a looong month.... -_-

[DHG-Dav]

Hello, and thanks for your awesome work !
i'm missing only one thing in controlnet here, the ability to XYZ plot its parameters. Usually i do img2img batches, and use XYZ plot to change parameters on the way (model/controlnet enable/disable/weight), sadly this is not an option with your tool, everything is perfect but i miss this option that i used a lot in vanilla A1111webui.
thanks again !

[DvST8x]

PuLID + EVA-CLIP support would be cool.

PuLID
https://github.com/ToTheBeginning/PuLID
Mikubill/sd-webui-controlnet#2838

EVA-CLIP
https://github.com/huchenlei/sd-webui-controlnet-evaclip