[Feature Request]: SDXL refiner support

[Shaistrong]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Implement the refiner not as img2img, like done on ComfyUI.

Proposed workflow

Add a dropdown menu for the refiner next to the checkpoint dropdown menu.

Additional information

No response

[daxijiu]

maybe it can works like hi-res fix：
hi-res fix function rename to 2pass
when ckpt select sd1/sd2 it works as orignal hi-res fix.
when ckpt select sdxl it has a option to select refiner model and works as refiner

[alfonslm]

Seems like a good idea but I would prefer if we could still access the original image to re run it with different seeds since the high res fix can and do cause some bad artifacts or good base images.

[Shaistrong]

Seems like a good idea but I would prefer if we could still access the original image to re run it with different seeds since the high res fix can and do cause some bad artifacts or good base images.

I don't think what you said is a good idea. The best workflow should be base>refiner>hires. With the idea I proposed you can just select NONE on the refiner. Both problems solve with this.

[alfonslm]

So you are against having a tick box in settings for cashing images that enable an non destructive workflow and I'm not sure if that workflow will be the best either since the upscaling can change things in weird ways or are you talking about the upscaler under extra?

[Shaistrong]

So you are against having a tick box in settings for cashing images that enable an non destructive workflow and I'm not sure if that workflow will be the best either since the upscaling can change things in weird ways

Idk what kind of upscaling you use. I never experienced what you mentioned. And no, I'm not against having a tickbox that can enable and disable the refiner.

[alfonslm]

So here is an example

If I just ran a batch of eg 12 images while I took a break and comes back and had all looking like the right image as a result with out a cash the computing time would have been wasted or if I had a cash for the last batch I could rerun the cash and try to fix the settings. In this case I actually decreased the denoise of the refiner.

[alfonslm]

BTW. The cash is actually really helpful to find the issue also and is a debug tool in many situations.

[svenkevs]

Personally I would prefer it as an img2img Script. I prefer sometimes to try and make several variations of the base image first before I go on to upscaling. I would like to have the same option available for SDXL refining as well. But I don't see why both options wouldn't be possible.

[dhwz]

@svenkevs you can already use it in img2img, but it's basically a misunderstanding the refiner model is not intended to be used to upscale/improve an image.
It is used when creating the original image, inference is starting with base model for x steps then it's switching to refiner model and continuing inference based on latents from base model.
Your "base" image is created after these steps.

[bghira]

that's one way to use the refiner, but the default design workflow is indeed img2img (SEdit)

1.0 has some new workflow, evidently.

[Aonodensetsu]

+1 for merging sdxl refiner with the base model - my own suggestion is to add it as a Script, which will allow unwilling people to ignore it and later img2img to get the refinement, and others will get it as part of the base, as SDXL predicts

[Kadah]

maybe it can works like hi-res fix： hi-res fix function rename to 2pass when ckpt select sd1/sd2 it works as orignal hi-res fix. when ckpt select sdxl it has a option to select refiner model and works as refiner

I think like the idea of the refiner and the selection of the refiner model being a part of HRF, or at least in the same UI space as HRF and as a new additional optional process.

[MoonRide303]

After gathering some more knowledge about SDXL and ComfyUI, and experimenting few days with both I've ended up with this basic (no upscaling) 2-stage (base + refiner) workflow:

It works pretty well for me - I change dimensions, prompts, and samplers parameters, but the flow itself stays as it is. It might come handy as reference.

UPDATE: after running some more experiments with various flows I've also managed to setup simple, yet nicely working (it doesn't sharpen areas that should stay blurred, for example) hires fix based on SDXL refiner - it looks like this:

Would be nice to have something like that in SD web UI, too.

[catboxanon]

This extension implements the refiner as intended by SAI. https://github.com/wcde/sd-webui-refiner