About Flux LoRA

[lllyasviel]

How to report LoRAs that do not work

Recently many Flux training codebases are very dynamic and lora formats maybe somewhat complicated. If you find that some LoRAs that do not work. Please give BOTH:

1. link to the page of that lora describing what it is
2. the direct download link of that lora

in this post.

How to Skip "Patching LoRAs" & How to make LoRAs more precise on low-bit models

If you use:

Then it means you will bake (precompute) LoRAs to same precision to your diffusion model. If your model is NF4, then your LoRAs will be merged to that model, and everything are still NF4. Patching this may take some time. But diffusion speed will not change after the patching.

If you use:

Then it means your LoRA will always use higher precision, no matter what precision your base model is. This will need to compute LoRAs on-the-fly in every diffusion iteration. If you use one single LoRA, diffusion will only be a bit slower. If you use multiple LoRAs, diffusion will be largely slower. However, you can use this option to skip "Patching LoRAs" since you do not need precomputed patches now.

How to only load LoRA one time rather than each generation

If you do not change LoRA weights, it will only load one time. If not, report a BUG in issue please.

[ali0une]

Hi there.

i'll start with the Impressionist Landscape LoRA for Flux, direct link

i've tried with the flux1-dev-fp8.safetensors and nothing happened, exact same picture with and without LoRA, same seed and parameters obviously.

prompt was :

By Vincent Van Gogh, an impressionist landscape of a lavender field, the sky is a dreamy swirl of pale blues and gentle yellows, blending seamlessly with the land below. <lora:flux_impressionism_v1.0:1>

[lllyasviel]

test and see if it works

[ali0une]

it works!

without LoRA :

with LoRA :

Many thanks 🥇

[ostap667inbox]

Any base XLabs-AI lora's not working.
Only versions with '_comfy_converted' prefix is Ok.

Link:
https://huggingface.co/XLabs-AI/flux-lora-collection/tree/main

For example, anime_lora_comfy_converted.safetensors is Ok.

But anime_lora.safetensors - this error:
[LORA] LoRA version mismatch for KModel: F:\NEURAL\LORA\FLUX\flux-anime_lora.safetensors 20/20 [01:30<00:00, 4.56s/it] Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored. To load target model JointTextEncoder Begin to load 1 model [Memory Management] Current Free GPU Memory: 10937.62 MB [Memory Management] Required Model Memory: 5227.11 MB [Memory Management] Required Inference Memory: 2123.00 MB [Memory Management] Estimated Remaining GPU Memory: 3587.52 MB Moving model(s) has taken 1.28 seconds Distilled CFG Scale: 3.5 To load target model KModel Begin to load 1 model [Memory Management] Current Free GPU Memory: 8344.14 MB [Memory Management] Required Model Memory: 11350.07 MB [Memory Management] Required Inference Memory: 2123.00 MB [Memory Management] Estimated Remaining GPU Memory: -5128.92 MB [Memory Management] Loaded to CPU Swap: 6562.61 MB (asynchronous method) [Memory Management] Loaded to GPU: 4787.44 MB Moving model(s) has taken 2.44 seconds 0%|.........

[VGF64]

@ali0une Wait, am I reading that right? NF4 works with loras now? Are you on ComfyUI?

[ali0une]

NF4 works with loras now?

Yes Sir! No Sir ... only in fp8 and fp16 see my updated comment below.

Are you on ComfyUI?

No Sir ... look at the url above https://github.com/lllyasviel/stable-diffusion-webui-forge/

[VGF64]

I see... I'll make a quick hop over to Forge. I'm very eager to try out the loras.

[ostap667inbox]

@Desm0nt , @ostap667inbox maybe the realism LoRA hasn't much to add to the picture, please try again with the art_lora.safetensors like this :

woman taking a selfie lora:flux1_art_v1.0:1 art

Look at the error in the above code. This line:
[LORA] LoRA version mismatch for KModel: F:\NEURAL\LORA\FLUX\flux-anime_lora.safetensors
This bug is for all base LoRa from XLabs-AI. Only 'comfy_converted' LoRa now work in Forge

Works perfectly for me with nf4/fp8/fp16 Flux1-dev versions.

You are getting something confused. The nf4 version currently does not support any LoRa.

[ali0une]

Well was just trying to give a hand ... but you have all details in my response above and the nf4 works with the unconverted LoRA.in my git clone of this Forge repo.

Edit :

You are totaly right when you say only the _comfy_converted ones work and none work with nf4..

i've tested further and it's the trigger word "art" or "anime" that made my generated images different from the ones without the LoRA.
i've tried the _comfy_converted ones from XLabs-AI/flux-lora-collection and indeed they work not just because of the trigger words, the result are clearly influenced by the LoRA.
Both in fp8 and fp16 but not with nf4.

From left to right :

a boy holding a heart shaped balloon
a boy holding a heart shaped balloon anime
a boy holding a heart shaped balloon anime <lora:flux1_anime_v1.0:1>
a boy holding a heart shaped balloon anime <lora:flux1_anime_comfy_converted_v1.0:1>

Thanks for pointing me in the right direction!

[mistermcluvin]

Tested on my Lora I created yesterday using ostris AI Toolkit (local PC 24GB VRAM). Generated with mklanFluxDevV1FP8_mklanFluxv1 model, works great! Thanks!

[queenofinvidia]

so all regular Flux loras are now working on nf4?

that's amazing if so, you're a genius Illya~

[SaertiS]

Yep, can confirm on XLab LoRas, see an effect on the nf4 model rn
no lora

lora 1.2

lora 1.5

lora 1.8

[VGF64]

So to clarify, Flux Loras made in FP16/FP8 also work on NF4? How's the quality in comparison?

[PierrunoYT]

It's not working for me

[marklance333]

Patching LoRAs: 80% then it crashes every time.

Lora is 164mb. using dev-nf4

(a 30mb lora works fine)

[chubbypillow93]

Same🙁30M lora okay, my own 160M lora big nope...

[jordanjay29]

I can confirm, larger (>25mb as of my testing, my next size up is 80mb) loras crash the patching process, it maxes out my 12GB 3060 card. Regular gen on nf4 (and sometimes fp8) works fine.

[AvaCadava]

I trained a LoRa on Civitai (using Kohya Method and Flux Dev, 18mb LoRa size). I am trying to use this LoRa in Forge, but Forge gives me some errors during LoRa Patching. It gets stuck at step 271/304 and freezes my browser. (System: RTX3060 12GB, 32GB ram)

LOG -->

Startup time: 241.2s (prepare environment: 49.4s, launcher: 39.3s, import torch: 62.4s, setup paths: 0.2s, initialize shared: 3.5s, other imports: 26.0s, setup gfpgan: 0.5s, list extensions: 0.3s, list SD models: 2.7s, load scripts: 34.5s, load upscalers: 0.1s, initialize extra networks: 9.6s, cleanup temp dir: 6.9s, create ui: 6.1s, gradio launch: 2.0s).
Environment vars changed: {'stream': False, 'inference_memory': 1024.0, 'pin_shared_memory': False}
Loading Model: {'checkpoint_info': {'filename': 'F:\AI\StableDiffusion\Shared\CKPT\SSD\FLUX\FLUX_1DevBNBNF4_v2.safetensors', 'hash': 'f0770152'}, 'additional_modules': [], 'unet_storage_dtype': None}
StateDict Keys: {'transformer': 1722, 'vae': 244, 'text_encoder': 198, 'text_encoder_2': 220, 'ignore': 0}
Using Detected T5 Data Type: torch.float8_e4m3fn
Using Detected UNet Type: nf4
Using pre-quant state dict!
Working with z of shape (1, 16, 32, 32) = 16384 dimensions.
K-Model Created: {'storage_dtype': 'nf4', 'computation_dtype': torch.bfloat16}
TI_Boots.civitai.png: no embedded information found.
TI_Dress_Pantyhose.civitai.png: no embedded information found.
Model loaded in 5.9s (unload existing model: 0.2s, forge model load: 5.7s).
[LORA] Loaded F:\AI\StableDiffusion\Shared\LORA\FLUX\AvaCadava-000009.safetensors for KModel-UNet with 304 keys at weight 1.0 (skipped 0 keys)
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
To load target model JointTextEncoder
Begin to load 1 model
[Memory Management] Current Free GPU Memory: 11219.56 MB
[Memory Management] Required Model Memory: 5154.62 MB
[Memory Management] Required Inference Memory: 1024.00 MB
[Memory Management] Estimated Remaining GPU Memory: 5040.94 MB
Moving model(s) has taken 14.93 seconds
Distilled CFG Scale: 3.5
To load target model KModel
Begin to load 1 model
[Memory Management] Current Free GPU Memory: 11176.22 MB
[Memory Management] Required Model Memory: 6246.84 MB
[Memory Management] Required Inference Memory: 1024.00 MB
[Memory Management] Estimated Remaining GPU Memory: 3905.38 MB
Patching LoRAs: 87%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 265/304 [00:27<00:05, 6.69it/s]ERROR lora diffusion_model.single_blocks.25.linear1.weight Allocation on device
Patching LoRAs: 88%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 268/304 [00:28<00:09, 3.92it/s]ERROR lora diffusion_model.single_blocks.26.linear1.weight Allocation on device
Patching LoRAs: 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 271/304 [00:29<00:03, 9.24it/s]

[nadav-lin]

Also getting the same error from a lora I trained on Civitai, loras that I didn't train on it work fine.

ERROR lora diffusion_model.single_blocks.19.linear1.weight Allocation on device ERROR lora diffusion_model.single_blocks.20.linear1.weight Allocation on device ERROR lora diffusion_model.single_blocks.20.linear2.weight Allocation on device

[AvaCadava]

I received a comment on Civitai about this issue from epinikion:
"It works for nf4 but you need A LOT free space on the drive to not interrupt the patching process"

Can someone confirm this solution and explain in more detail how much free space is needed exactly where?

[SaertiS]

yep, looks like nf4 and gguf models LoRa capability were broken lastly

[metzo007]

I downloaded these two nsfw loras ( for testing ;-)) and non of them seems to have any effect on the picture. The resuld is 100% the same like without the lora

https://civitai.com/models/642272/flux-topless?modelVersionId=718406
https://civitai.com/models/91780/breasts-out-fashion-clothing-or?modelVersionId=716799

I did use the
flux1-dev-bnb-nf4-v2.safetensors
flux1-dev-fp8.safetensors

And I get a
[LORA] LoRA version mismatch for KModel:
Error

[jokero3answer]

Mine too.
@lllyasviel

[markdee3]

Can you add CPU swap for the LoRa Patching? still crashing for me and i think that's the issue

[ALBIHANY]

i don't think LoRAs working i tried many flux loRAs and most of them doesn't work with the NF4 (despite the note of one and only native support), take this for example: https://civitai.com/models/633553

[ALBIHANY]

1020437158

Using the same seed same steps same Flux model, and LoRA, still can't get the same results you got (WHAT AM I DOING WRONG!) do you think it's the F16 clip do I need to download it?

PS: Confirm LoRA's actually work

[ostap667inbox]

Yes, try with t5xxl_fp16.safetensors
And second trick: try quantized GGUF Q4 or Q8 model (there you have to set the LoRA weights lower, they have a stronger effect on the result).
P.S. And look at Civitai Helper settings: while it's scanning, it records garbage information from the models page into 'description' field (trash tags and info on LoRA tiles in Extra Network panel on your screenshot). Set OFF the checkbox 'Also add data for WebUI metadata editor' for Civitai Helper in Settings and delete all *.json files in LoRA folders before next scanning for clean the trash.

[ALBIHANY]

and this is with FT16

[ALBIHANY]

Yes, try with t5xxl_fp16.safetensors And second trick: try quantized GGUF Q4 or Q8 model (there you have to set the LoRA weights lower, they have a stronger effect on the result). P.S. And look at Civitai Helper settings: while it's scanning, it records garbage information from the models page into 'description' field (trash tags and info on LoRA tiles in Extra Network panel on your screenshot). Set OFF the checkbox 'Also add data for WebUI metadata editor' for Civitai Helper in Settings and delete all *.json files in LoRA folders before next scanning for clean the trash.

Just downloaded the 9GB+ FP16 and this is the results what am i doing wrong i don't even know, but one thing for sure mine isn't as Balish as your results :

[shojtsy]

Maybe you have different checkpoint? The names differ at least. v2 vs v2_3.

[barnaclejive]

Maybe you already know this, but I created a Lora using standard settings with ai-toolkit (https://github.com/ostris/ai-toolkit)
The Lora works great with your fp8 model (like 15 sec with a 4090) but when using your nf4 it takes like 10 minutes.

[Myoko]

loras work with flux dev nf4 v2?

[BaseMe2]

Yes, but not all and not reliable. I often have OOM cuda errors, after patching a lora or changing options. Something hooks the ram. The larger the lora, the more likely it OOM. Lora that work eat a lot of vram/ram, multiple gigabytes of, even for small loras.

[markdee3]

for me only 34mb loras work, and 18mb loras crash when patching

[markdee3]

[Memory Management] Required Model Memory: 5154.62 MB
[Memory Management] Required Inference Memory: 1024.00 MB
[Memory Management] Estimated Remaining GPU Memory: -2820.25 MB
[Memory Management] Loaded to GPU for backward capability: 73.14 MB
[Memory Management] Loaded to CPU Swap: 3446.00 MB (blocked method)
[Memory Management] Loaded to GPU: 1780.99 MB
Moving model(s) has taken 6.86 seconds
Distilled CFG Scale: 3.5
To load target model KModel
Begin to load 1 model
[Unload] Trying to free 9144.89 MB for cuda:0 with 0 models keep loaded ...
[Unload] Current free memory is 986.94 MB ...
[Unload] Unload model JointTextEncoder
[Memory Management] Current Free GPU Memory: 3325.14 MB
[Memory Management] Required Model Memory: 6246.84 MB
[Memory Management] Required Inference Memory: 1024.00 MB
[Memory Management] Estimated Remaining GPU Memory: -3945.70 MB
Patching LoRAs for KModel:  76%|████████████████████████████████████████▍            | 232/304 [01:55<00:54,  1.33it/s]
Press any key to continue . . . ```

see how the first model uses CPU swap to stop OOM, can this be done for the lora patching? using NF4, 4GB vram

[Iory1998]

2 days ago, I generated some images using the X-Labs' default flux_realism_lora.safetensors. I used it with the fp16 Flux, the GGUF-Q8 and the flux1-dev-bnb-nf4-v2.safetensors. It worked like a charm and the it enhances the quality of the images greatly.
Today, I tried to regenerate the same images and lo and behold the images are so much different. I regenerate the images with and without the LoRA and I get the same image, which means that the LoRA is not working.

In console, I can clearly see that the LoRA is patching. The number of patches perhaps increased?

[markdee3]

why do some lora's have 76 steps and some have 304. the 304 steps always crash for me

[Jonseed]

The LoRA patching takes quite a while for each generation. Is there a way to cache the patch once it has done it once for a particular model, or is that not possible?

[danilomaiaweb]

I tested several lora models and none of them effect. I created several images with the same seed, with different model times and nothing changed. Where is the error?

Thank in advance

[chromesun]

I created a Flux LoRA using CivitAI's generator. The model they use is "Flux.1 D", i.e. the DEV one. It took me a while to find a way to use it in Forge, so I'm posting this in case it's of help to anyone trying something similar.

Using this version of Forge at the moment:
version: f2.0.1v1.10.1-previous-370-g0252ad86  •  python: 3.10.11  •  torch: 2.3.1+cu121  •  xformers: N/A

The only non-crashing reasonable-time way of using the LoRA on my system is with these settings:

My PC: Win10, Ryzen 7-2700, 64GB system ram, Nvidia 3060/12GB.

Forge console shows this about patching the LoRA:
<<<
Patching LoRAs for KModel: 0%| | 0/304 [00:00<?, ?it/s]G:\FORGE\webui\backend\operations_gguf.py:68: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data = torch.tensor(tensor.data)
Patching LoRAs for KModel: 100%|███████████████████████████████████████████████████████████| 304/304 [00:06<00:00, 50.10it/s]
LoRA patching has taken 6.08 seconds

I've made a With-Without comparison image to show that the LoRA is having an obvious effect (although I had to increase the weight to 1.4).
The comparison grid (two images, 896x1152, Euler Simple, 20 steps, D.cfg 3.5) took 3m 26.5s.

The Flux checkpoint I'm using is Flux.1-Dev GGUF Q4.0, that I got on CivitAI here:
https://civitai.com/models/647237?modelVersionId=724099

The With-Without image is in my CivitAI post here (SFW):
https://civitai.com/posts/5664697

[chromesun]

Extra items appeared in "Diffusion in Low Bits" when I used update.bat yesterday, so perhaps you need to do the same?

[Dudekpob]

Extra items appeared in "Diffusion in Low Bits" when I used update.bat yesterday, so perhaps you need to do the same?

Yes solved

[chromesun]

I could not get it to work with nf4-v2 using my LoRA. Only worked with the gguf + ae/clip_l/t5xxl_fp16 as in image above. Any other combination caused a Python crash.

[chromesun]

Now working with nf4-v2 using @ostap667inbox settings (above, in this thread). 😀

Had to lower GPU Weights from default 11263 to 8192 - just a guess, and the value will vary depending on your ram/vram. With these settings I can use my CivitAI-generated LoRA with nft-v2 and no need for anything in "VAE/Text Encoder". Speed is about 4.5s/it, approx 1m40s for 896x1152 image.

Hiresfix works if I don't go too high. Tested *1.25 (896x1152 ---> 1120x1440) and it works OK. Approx 3m06s per image.
I'm using Forge version: f2.0.1v1.10.1-previous-392-g31bed671, and here is screensnip of settings:

[wideload1971]

Thanks, your setup is what worked for me when using loras.

[wzwow]

does lora for flux works now?

[ostap667inbox]

Yes. Look at my comment above, there are examples and a screenshot of the settings.

[eurotaku]

is it neccessary that the patching has to be done before every inference? can loras be prepatched in an nf4/fp8 compatible format? i am asking cause on my setup with a 3060 12gb it takes 150s to prepatch the lora and the whole system freezes from time to time during that patching process @lllyasviel

[markdee3]

change 'Diffusion in low Bits' to 'Automatic (LoRa in fp16)' at the top in forge

[alexxer13]

If no one has said it yet I wanted to share that I solved it by installing the flux1-dev-fp8.safetensors model from lllyasviel flux found in the following link https://huggingface.co/lllyasviel/flux1_dev/tree/main
Only then could I make it work with any Lora. I haven't tried all the ones out there. But a lot of it works as it should. It was just a matter of using a compatible flux model. Also check that your forge installation is up to date in case it doesn't work for you like it did for me.

[ALBIHANY]

Anyone got this loRA to work, it takes forever to process any prompt when i usse this one : https://civitai.com/models/430687/detailifier-fluxsdxlponysd15?modelVersionId=739154

[ostap667inbox]

No LoRA:

With this LoRA:

Keep the console window and system resource monitor open during generation.
It is possible that during LoRA patching the system RAM was exhausted and the system started using a very slow swap disk. All this will be visible in the resource monitor.
Also, sometimes the process hangs on some event or message that is not displayed in the console and can be resumed by pressing Esc in the console.

[chromesun]

This worked for me:

and this was the console text if you want to compare:
txt2img: art styles 2s, magnificent character_mf_1 landscapes_5, (chromesun_colours_2-tinted:0.5), very detailed, intricate, chromesun_outdoor_conditions_3, white marbled ebru design lora:detailer_flux_v1:0.5
[Unload] Trying to free all memory for cuda:0 with 0 models keep loaded ...
[Unload] Current free memory is 5028.32 MB ...
[Unload] Unload model KModel
[Unload] Current free memory is 9872.82 MB ...
[Unload] Unload model IntegratedAutoencoderKL
[LORA] Loaded G:\stable-diffusion-webui-master\webui\models\Lora\FLUX\detailer_flux_v1.safetensors for KModel-UNet with 304 keys at weight 0.5 (skipped 0 keys)
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
To load target model JointTextEncoder
Begin to load 1 model
[Unload] Trying to free 10890.24 MB for cuda:0 with 0 models keep loaded ...
[Memory Management] Current Free GPU Memory: 10037.27 MB
[Memory Management] Required Model Memory: 5227.11 MB
[Memory Management] Required Inference Memory: 4095.00 MB
[Memory Management] Estimated Remaining GPU Memory: 715.17 MB
Moving model(s) has taken 1.45 seconds
Distilled CFG Scale: 3.5
To load target model KModel
Begin to load 1 model
[Unload] Trying to free 12215.89 MB for cuda:0 with 0 models keep loaded ...
[Unload] Current free memory is 4799.38 MB ...
[Unload] Unload model JointTextEncoder
[Memory Management] Current Free GPU Memory: 10032.39 MB
[Memory Management] Required Model Memory: 6246.84 MB
[Memory Management] Required Inference Memory: 4095.00 MB
[Memory Management] Estimated Remaining GPU Memory: -309.45 MB
Patching LoRA in on-the-fly.
Patching LoRAs for KModel: 100%|████████████████████████████████████████████████████████| 304/304 [00:00<00:00, 25310.02it/s]
[Memory Management] Loaded to CPU Swap: 1595.27 MB (blocked method)
[Memory Management] Loaded to GPU: 4651.50 MB
Moving model(s) has taken 3.56 seconds
100%|████████████████████████████████████████████████████████████████████████████████████████| 20/20 [01:24<00:00, 4.22s/it]
To load target model IntegratedAutoencoderKL█████████████████████████████████████████████████| 20/20 [01:20<00:00, 4.21s/it]
Begin to load 1 model
[Unload] Trying to free 4302.84 MB for cuda:0 with 0 models keep loaded ...
[Unload] Current free memory is 5199.18 MB ...
[Memory Management] Current Free GPU Memory: 5199.18 MB
[Memory Management] Required Model Memory: 159.87 MB
[Memory Management] Required Inference Memory: 4095.00 MB
[Memory Management] Estimated Remaining GPU Memory: 944.31 MB
Moving model(s) has taken 0.38 seconds
Total progress: 100%|████████████████████████████████████████████████████████████████████████| 20/20 [01:21<00:00, 4.06s/it]
Total progress: 100%|████████████████████████████████████████████████████████████████████████| 20/20 [01:21<00:00, 4.21s/it]

With the settings I used, Detailifier was way too strong at weight=1. Even at 0.5 it was almost too much.

[ALBIHANY]

No LoRA:

With this LoRA:

Keep the console window and system resource monitor open during generation. It is possible that during LoRA patching the system RAM was exhausted and the system started using a very slow swap disk. All this will be visible in the resource monitor. Also, sometimes the process hangs on some event or message that is not displayed in the console and can be resumed by pressing Esc in the console.

Patching almost take no time for me, now i admit i have low vram only 6gb so i'm use to 8 minutes to 12 max with every prompt but this one as you can see from the console take away longer time, i used this same lora on comfyui and it the time is normal.

[lllyasviel]

@ALBIHANY You selected fp4, forge is now wasting 80 second to convert your nf4 model to fp4. 99% performance problems are just user mistakes. If you still have problems, completely fresh install forge without changing any option and just use "Automatic" or "Automatic (fp16 LoRAs)"

[ALBIHANY]

@ALBIHANY You selected fp4, forge is now wasting 80 second to convert your nf4 model to fp4. 99% performance problems are just user mistakes. If you still have problems, completely fresh install forge without changing any option and just use "Automatic" or "Automatic (fp16 LoRAs)"

Thank you for the responded, and you're absolutely right on that screenshot i mistakenly choose nf4 instead of fp4, and using automatic make the rendering speed back to normal, However not with Automatic FP16, i wonder what's the different between the two of them and if Automatic FP16 has an effect on the quality of the final result?, i don't think i need to reinstall forge that will be a disaster for me as i installed almost all the spaces and i keep up with updates and git pull (almost daily) with every new commit you submit, thank you for your outstanding efforts.