Someone else with a low vram card (<6GB) whose generation times are much worse since today's updates?

[realmalikov]

Since I made the updates today (Aug 15th) I have noticed a massive deterioration in performance (from +-25 s/it to +-37 s/it) and would like to hear your experiences to know if it is due to my hardware / other external circumstances or due to the latest updates.

[realmalikov]

One thing I noticed is that the program now only uses 93.4% of the VRAM instead of 100%
Edit: Before

Now (latest 2f0555f)

[Br00xx]

There is a slider for GPU weights. Maybe this needs to be set higher to fit an entire model in your vram

[JimikoSK]

There is a slider for GPU weights. Maybe this needs to be set higher to fit an entire model in your vram

Entire model fits in my 24GB and it's still way slower.

[Br00xx]

There is a slider for GPU weights. Maybe this needs to be set higher to fit an entire model in your vram

Entire model fits in my 24GB and it's still way slower.

Im using 3070ti and havent noticed any difference in speed

[JimikoSK]

There is a slider for GPU weights. Maybe this needs to be set higher to fit an entire model in your vram

Entire model fits in my 24GB and it's still way slower.

Im using 3070ti and havent noticed any difference in speed

Maybe it has to do with something in older architecture. I'm on Pascal.

[navarisun1982]

same here

[JimikoSK]

Same here, Tesla P40 went from 7.14 it/s to like 16-32.

[ishadowx27]

What does that card compare to for Generations in terms of RTX cards? Is it comparable to the 2080 or 3090 in terms of speed (for Stable Diffusion at least)?

[lllyasviel]

If someone here knows how to git checkout, put full console logs before after some commit with different speed.

Sometimes when I find things slower, it often turns out to be caused by different model - note that nf4, nf4v2, q4 are expected to have speed difference. these days people change models a lot and one may forget they changed models

[JimikoSK]

@lllyasviel here is the latest commit default model weight, everything else standard to my previous tests.

Loading Model: {'checkpoint_info': {'filename': '/mnt/data/SDFiles/models/Stable-Diffusion/flux1-dev-fp8.safetensors', 'hash': 'be9881f4'}, 'additional_modules': [], 'unet_storage_dtype': None}
[Unload] Trying to free 953674316406250018963456.00 MB for cuda:0 with 0 models keep loaded ...
StateDict Keys: {'transformer': 780, 'vae': 244, 'text_encoder': 198, 'text_encoder_2': 220, 'ignore': 0}
Using Detected T5 Data Type: torch.float8_e4m3fn
Using Detected UNet Type: torch.float8_e4m3fn
Working with z of shape (1, 16, 32, 32) = 16384 dimensions.
K-Model Created: {'storage_dtype': torch.float8_e4m3fn, 'computation_dtype': torch.float16}
Model loaded in 1.5s (unload existing model: 0.3s, forge model load: 1.1s).
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
To load target model JointTextEncoder
Begin to load 1 model
[Unload] Trying to free 7725.00 MB for cuda:0 with 0 models keep loaded ...
[Memory Management] Current Free GPU Memory: 24277.62 MB
[Memory Management] Required Model Memory: 5154.62 MB
[Memory Management] Required Inference Memory: 1024.00 MB
[Memory Management] Estimated Remaining GPU Memory: 18099.01 MB
Moving model(s) has taken 1.30 seconds
Distilled CFG Scale: 3.5
To load target model KModel
Begin to load 1 model
[Unload] Trying to free 15779.09 MB for cuda:0 with 0 models keep loaded ...
[Unload] Current free memory is 19007.91 MB ...
[Memory Management] Current Free GPU Memory: 19007.91 MB
[Memory Management] Required Model Memory: 11350.07 MB
[Memory Management] Required Inference Memory: 1024.00 MB
[Memory Management] Estimated Remaining GPU Memory: 6633.85 MB
Moving model(s) has taken 3.57 seconds
 57%|███████████████████████████████████████████████▍                                   | 16/28 [05:12<03:54, 19.54s/it]
To load target model IntegratedAutoencoderKL███████████████▎                            | 16/28 [04:45<03:53, 19.48s/it]
Begin to load 1 model
[Unload] Trying to free 2593.67 MB for cuda:0 with 0 models keep loaded ...
[Unload] Current free memory is 7641.43 MB ...
[Memory Management] Current Free GPU Memory: 7641.43 MB
[Memory Management] Required Model Memory: 319.75 MB
[Memory Management] Required Inference Memory: 1024.00 MB
[Memory Management] Estimated Remaining GPU Memory: 6297.69 MB
Moving model(s) has taken 0.11 seconds
Total progress:  57%|██████████████████████████████████████▎                            | 16/28 [04:53<03:40, 18.37s/it]
Total progress:  57%|██████████████████████████████████████▎                            | 16/28 [04:53<03:53, 19.48s/it]

[lllyasviel]

@JimikoSK try --all-in-fp32 and what will happen?

[navarisun1982]

@JimikoSK try --all-in-fp32 and what will happen?

for me that doesn't change anything on the latest update
Cimmit hash: 8a04293

[JimikoSK]

@lllyasviel

@JimikoSK try --all-in-fp32 and what will happen?

Sorry for delay, I will try this.

[JimikoSK]

@JimikoSK try --all-in-fp32 and what will happen?

@lllyasviel Ok I think this is the solution. It brings generation back to the 7.xx s/it speed with the latest commit. 👍😎

Model selected: {'checkpoint_info': {'filename': '/mnt/data/SDFiles/models/Stable-Diffusion/flux1-dev-bnb-nf4-v2.safetensors', 'hash': 'f0770152'}, 'additional_modules': ['/mnt/data/webui-forge_recheck/models/text_encoder/t5xxl_fp8_e4m3fn.safetensors', '/mnt/data/webui-forge_recheck/models/text_encoder/clip_l.safetensors'], 'unet_storage_dtype': None}
Loading Model: {'checkpoint_info': {'filename': '/mnt/data/SDFiles/models/Stable-Diffusion/flux1-dev-bnb-nf4-v2.safetensors', 'hash': 'f0770152'}, 'additional_modules': ['/mnt/data/webui-forge_recheck/models/text_encoder/t5xxl_fp8_e4m3fn.safetensors', '/mnt/data/webui-forge_recheck/models/text_encoder/clip_l.safetensors'], 'unet_storage_dtype': None}
[Unload] Trying to free 953674316406250018963456.00 MB for cuda:0 with 0 models keep loaded ...
[Unload] Current free memory is 12547.02 MB ...
[Unload] Current free memory is 12547.02 MB ...
[Unload] Current free memory is 12547.02 MB ...
[Unload] Unload model KModel
[Unload] Current free memory is 23897.13 MB ...
[Unload] Unload model IntegratedAutoencoderKL
StateDict Keys: {'transformer': 1722, 'vae': 244, 'text_encoder': 196, 'text_encoder_2': 220, 'ignore': 0}
Using Detected T5 Data Type: torch.float8_e4m3fn
Using Detected UNet Type: nf4
Using pre-quant state dict!
Working with z of shape (1, 16, 32, 32) = 16384 dimensions.
K-Model Created: {'storage_dtype': 'nf4', 'computation_dtype': torch.float32}
Model loaded in 17.5s (unload existing model: 14.7s, forge model load: 2.8s).
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
To load target model JointTextEncoder
Begin to load 1 model
Moving model(s) has taken 0.02 seconds
Distilled CFG Scale: 3.5
To load target model KModel
Begin to load 1 model
[Unload] Trying to free 9152.50 MB for cuda:0 with 0 models keep loaded ...
[Unload] Current free memory is 24217.12 MB ...
[Memory Management] Current Free GPU Memory: 24217.12 MB
[Memory Management] Required Model Memory: 6252.69 MB
[Memory Management] Required Inference Memory: 1024.00 MB
[Memory Management] Estimated Remaining GPU Memory: 16940.44 MB
Moving model(s) has taken 13.80 seconds
100%|███████████████████████████████████████████████████████████████████████████████████| 28/28 [03:28<00:00,  7.46s/it]
To load target model IntegratedAutoencoderKL████████████████████████████████████████████| 28/28 [03:24<00:00,  7.57s/it]

[navarisun1982]

version: f2.0.1v1.10.1-previous-356-g8eeeace7  
•  python: 3.10.10  
•  torch: 2.3.1+cu121  
•  xformers: 0.0.27  
•  gradio: 4.40.0  
•  checkpoint: c161224931

here is my info, it starts with 5s/it which is great improvement over the past commits, but it keep accumulating until it reach 40s/it at the end, i use --all-in-fp32 --xformers on Tesla p40 24gb card

[atbgams]

Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: f2.0.1v1.10.1-previous-354-g9bc2d04c
Commit hash: 9bc2d04
Total VRAM 4096 MB, total RAM 16244 MB
pytorch version: 2.3.1+cu121
xformers version: 0.0.27
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce GTX 1050 Ti : native

It previously set the vram status to LOW-RAM, which I think is more suitable for my poor video card,^_^
I am currently manually setting the GPU weights to 2500M and running at sd1.5 speed.

[testonedev]

UPDATE:

Based on suggestions from #1370 I was able to use the latest version Commit hash: 49435de at old top speeds, where it was before ~ Aug 5th

• pre Aug 5: 13.5s
• after Aug 5 with various settings: 95s
• with Commandline Args (--all-in-fp32 --disable-model-loading-ram-optimization): 17s
• I kept regenerating, it got faster with each generation, now it's back to old times: 13.5s

---

For Pascal, issues seem to start even earlier, at least for me. Switched back and forth between commits to find where my speed issues started, used absolutereality_v181.safetensors as an example model. For me it is a difference of ~13s vs 95s, same settings, seeds etc. yield same images, so things seem to be ok on the settings side.

-> ~before Aug. 5th performance is great
-> somewhere afterwards, issues start, where webUI has loading errors, generation still works, performance is bad. Sorry, I did not investigate on a per-commit basis to exactly pinpoint

Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: f0.0.17v1.8.0rc-v1.7.0-1494-gb34f2c3b
Commit hash: bfee03d
Launching Web UI with arguments: --xformers
Total VRAM 8192 MB, total RAM 16300 MB
xformers version: 0.0.27
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce GTX 1070 Ti : native

(plz refer to the commit hash above - in screenshot it is a local commit, where I stored browser startup settings in the webui-user.bat, no relevance for results)

Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: f1.0.2v1.10.1-v1.7.0-1645-g71eaa5ca
Commit hash: 71eaa5c
Launching Web UI with arguments: --xformers
Total VRAM 8192 MB, total RAM 16300 MB
pytorch version: 2.3.1+cu121
xformers version: 0.0.27
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce GTX 1070 Ti : native

[lllyasviel]

Hey everyone, I merged many of your feedbacks to main, please update, and remove all setting/cmd flags, change back to default settings, and test the performance again.

it can be helpful if you can put test statistics here (after updates)

see also 31bed67

[navarisun1982]

Hey everyone, I merged many of your feedbacks to main, please update, and remove all setting/cmd flags, change back to default settings, and test the performance again.

it can be helpful if you can put test statistics here (after updates)

see also 31bed67

updated to latest version checkpoint: [c161224931]
on my p40 24gb

Time taken: 8 min. 10.8 sec.
for 20 steps on flux1-dev-bnb-nf4.safetensors // Automatic (fp16 LoRA) // Async // CPU // GPU Weights (MB) 7500
A: 7.21 GB, R: 9.88 GB, Sys: 11.8/24 GB (49.0%)

Time taken: 8 min. 15.9 sec.
for 20 steps on flux1-dev-bnb-nf4.safetensors // Automatic (fp16 LoRA) // Async // CPU // GPU Weights (MB) 10879
A: 8.61 GB, R: 10.92 GB, Sys: 12.8/24 GB (53.4%)

[JimikoSK]

Hey everyone, I merged many of your feedbacks to main, please update, and remove all setting/cmd flags, change back to default settings, and test the performance again.

it can be helpful if you can put test statistics here (after updates)

see also 31bed67

Loading Model: {'checkpoint_info': {'filename': '/mnt/data/SDFiles/models/Stable-Diffusion/flux1-dev-fp8.safetensors', 'hash': 'be9881f4'}, 'additional_modules': ['/mnt/data/webui-forge_recheck/models/text_encoder/t5xxl_fp8_e4m3fn.safetensors', '/mnt/data/webui-forge_recheck/models/text_encoder/clip_l.safetensors'], 'unet_storage_dtype': None}
[Unload] Trying to free 953674316406250018963456.00 MB for cuda:0 with 0 models keep loaded ...
StateDict Keys: {'transformer': 780, 'vae': 244, 'text_encoder': 196, 'text_encoder_2': 220, 'ignore': 0}
Using Detected T5 Data Type: torch.float8_e4m3fn
Using Detected UNet Type: torch.float8_e4m3fn
Working with z of shape (1, 16, 32, 32) = 16384 dimensions.
K-Model Created: {'storage_dtype': torch.float8_e4m3fn, 'computation_dtype': torch.float32}
Model loaded in 1.5s (unload existing model: 0.4s, forge model load: 1.2s).
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
To load target model JointTextEncoder
Begin to load 1 model
Moving model(s) has taken 0.02 seconds
Distilled CFG Scale: 3.5
To load target model KModel
Begin to load 1 model
[Unload] Trying to free 15779.09 MB for cuda:0 with 0 models keep loaded ...
[Unload] Current free memory is 24261.37 MB ...
[Memory Management] Current Free GPU Memory: 24261.37 MB
[Memory Management] Required Model Memory: 11350.07 MB
[Memory Management] Required Inference Memory: 1024.00 MB
[Memory Management] Estimated Remaining GPU Memory: 11887.31 MB
Moving model(s) has taken 3.40 seconds
100%|███████████████████████████████████████████████████████████████████████████████████| 28/28 [03:26<00:00,  7.36s/it]
To load target model IntegratedAutoencoderKL████████████████████████████████████████████| 28/28 [03:18<00:00,  7.44s/it]

Looks good. Latest commit, no --all-in-fp32 command line.

[realmalikov]

Performance has improved massively thanks to your updates and is now back to the level it was at before the degradation. Thank you!

[navarisun1982]

Performance is better now but I have one notice, the s/t is accumulating.
it starts with 5s/it and near the end it reach 40s/it