[Bug]: The Modify Content feature under inpainting causes crashes.

[audrito]

Checklist

• The issue has not been resolved by following the troubleshooting guide
• The issue exists on a clean installation of Fooocus
• The issue exists in the current version of Fooocus
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

I tried to use fooocus to generate some new backgrounds for some of the photos I took. For that task, I used the option Modify Content which is an option in Inpainting. It starts generating, everything works out until the last step. at the 29/30 step, it crashes with Connection Errored Out. This doesn't happen with any other generation option, only with Modify Content right now. I really need this fixed.

Steps to reproduce the problem

1. Go to inpainting, prepare everything
2. Select Modify Content
3. Click Generate and wait until the end

What should have happened?

It should complete the generation and provide the generated image.

What browsers do you use to access Fooocus?

Brave

Where are you running Fooocus?

Cloud (Google Colab)

What operating system are you using?

Windows 11

Console logs

Requirement already satisfied: pygit2==1.12.2 in /usr/local/lib/python3.10/dist-packages (1.12.2)
Requirement already satisfied: cffi>=1.9.1 in /usr/local/lib/python3.10/dist-packages (from pygit2==1.12.2) (1.16.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.9.1->pygit2==1.12.2) (2.21)
/content
fatal: destination path 'Fooocus' already exists and is not an empty directory.
/content/Fooocus
Already up-to-date
Update succeeded.
[System ARGV] ['entry_with_update.py', '--share']
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
Fooocus version: 2.2.1
Total VRAM 15102 MB, total RAM 12979 MB
Set vram state to: NORMAL_VRAM
Always offload VRAM
Device: cuda:0 Tesla T4 : native
VAE dtype: torch.float32
Using pytorch cross attention
2024-03-18 09:24:29.363209: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-18 09:24:29.363319: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-18 09:24:29.469713: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-18 09:24:31.987879: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Refiner unloaded.
Running on local URL:  http://127.0.0.1:7865
Running on public URL: https://63d15610b3047ab978.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)
model_type EPS
UNet ADM Dimension 2816
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids', 'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'}
Base model loaded: /content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors
Request to load LoRAs [['sd_xl_offset_example-lora_1.0.safetensors', 0.1], ['None', 1.0], ['None', 1.0], ['None', 1.0], ['None', 1.0]] for model [/content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors].
Loaded LoRA [/content/Fooocus/models/loras/sd_xl_offset_example-lora_1.0.safetensors] for UNet [/content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors] with 788 keys at weight 0.1.
Fooocus V2 Expansion: Vocab with 642 words.
Fooocus Expansion engine loaded for cuda:0, use_fp16 = True.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.70 seconds
Started worker with PID 4084
App started successful. Use the app with http://127.0.0.1:7865/ or 127.0.0.1:7865 or https://63d15610b3047ab978.gradio.live
[Parameters] Adaptive CFG = 7
[Parameters] Sharpness = 2
[Parameters] ControlNet Softness = 0.25
[Parameters] ADM Scale = 1.5 : 0.8 : 0.3
[Parameters] CFG = 4.0
[Parameters] Seed = 6844788250396917905
[Fooocus] Downloading upscale models ...
[Inpaint] Parameterized inpaint is disabled.
[Parameters] Sampler = dpmpp_2m_sde_gpu - karras
[Parameters] Steps = 30 - 15
[Fooocus] Initializing ...
[Fooocus] Loading models ...
Refiner unloaded.
[Fooocus] Processing prompts ...
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] beautiful eyes, cute, perfect pretty, intricate, elegant, highly detailed, extremely sharp focus, shining light, divine background, dramatic professional, cinematic, artistic, real, true colors, radiant, beautiful, creative, positive, pure, attractive, calm, unique, loving, generous, handsome, thought, best, inspirational, glowing, bright, charming, amazing, inspiring
[Fooocus] Preparing Fooocus text #2 ...
[Prompt Expansion] beautiful eyes, cute, elegant, highly detailed, dramatic light, intricate, atmosphere, new, great composition, cinematic, modern, brave, color, shiny, glowing, rich, romantic, inspired, attractive, delicate, pretty, inspiring, deep background, professional, best, winning, novel, epic, artistic, beautiful, surreal, clear, perfect, symmetry, illuminated
[Fooocus] Encoding positive #1 ...
[Fooocus Model Management] Moving model(s) has taken 0.12 seconds
[Fooocus] Encoding positive #2 ...
[Fooocus] Encoding negative #1 ...
[Fooocus] Encoding negative #2 ...
[Fooocus] Image processing ...
[Fooocus] VAE Inpaint encoding ...
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 1.31 seconds
[Fooocus] VAE encoding ...
Final resolution is (4624, 3468), latent is (1024, 1024).
[Parameters] Denoising Strength = 0.5
[Parameters] Initial Latent shape: torch.Size([1, 4, 128, 128])
Preparation time: 9.69 seconds
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 1.2123857736587524
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 1.88 seconds
100% 30/30 [00:25<00:00,  1.16it/s]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.32 seconds
Image generated with private log at: /content/Fooocus/outputs/2024-03-18/log.html
Generating and saving time: 31.86 seconds
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 1.2123857736587524
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 1.84 seconds
100% 30/30 [00:26<00:00,  1.12it/s]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.37 seconds
Image generated with private log at: /content/Fooocus/outputs/2024-03-18/log.html
Generating and saving time: 32.95 seconds
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
Total time: 74.55 seconds
[Fooocus Model Management] Moving model(s) has taken 0.82 seconds
[Parameters] Adaptive CFG = 7
[Parameters] Sharpness = 2
[Parameters] ControlNet Softness = 0.25
[Parameters] ADM Scale = 1.5 : 0.8 : 0.3
[Parameters] CFG = 4.0
[Parameters] Seed = 2381092195298278337
[Fooocus] Downloading upscale models ...
[Fooocus] Downloading inpainter ...
[Inpaint] Current inpaint model is /content/Fooocus/models/inpaint/inpaint_v26.fooocus.patch
[Parameters] Sampler = dpmpp_2m_sde_gpu - karras
[Parameters] Steps = 30 - 15
[Fooocus] Initializing ...
[Fooocus] Loading models ...
Synthetic Refiner Activated
Synthetic Refiner Activated
Request to load LoRAs [['sd_xl_offset_example-lora_1.0.safetensors', 0.1], ['None', 1.0], ['None', 1.0], ['None', 1.0], ['None', 1.0], ('/content/Fooocus/models/inpaint/inpaint_v26.fooocus.patch', 1.0)] for model [/content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors].
Loaded LoRA [/content/Fooocus/models/loras/sd_xl_offset_example-lora_1.0.safetensors] for UNet [/content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors] with 788 keys at weight 0.1.
Loaded LoRA [/content/Fooocus/models/inpaint/inpaint_v26.fooocus.patch] for UNet [/content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors] with 960 keys at weight 1.0.
Request to load LoRAs [['sd_xl_offset_example-lora_1.0.safetensors', 0.1], ['None', 1.0], ['None', 1.0], ['None', 1.0], ['None', 1.0]] for model [/content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors].
Loaded LoRA [/content/Fooocus/models/loras/sd_xl_offset_example-lora_1.0.safetensors] for UNet [/content/Fooocus/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors] with 788 keys at weight 0.1.
Requested to load SDXLClipModel
Loading 1 new model
unload clone 1
[Fooocus Model Management] Moving model(s) has taken 1.85 seconds
[Fooocus] Processing prompts ...
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] painting, cinematic, highly detailed, very sharp, detail, beautiful, light, fine, surreal, vibrant colors, glowing rich vivid color, winning new epic novel, silent, majestic, full coherent, symmetry, great composition, clear, perfect, artistic, complex, elegant, intricate, wonderful, luxury, dramatic, stunning, awesome, creative, ambient, thought, fancy
[Fooocus] Preparing Fooocus text #2 ...
[Prompt Expansion] painting, cinematic, extremely detailed, beautiful, stunning, intricate, epic composition, cool color, winning contemporary fine classic very strong colors, pretty vivid rich deep complex dramatic great grand ambient spectacular light, sharp focus, inspirational, elegant, highly saturated, clear, aesthetic, excellent, pure brilliant, perfect, expressive, shiny, attractive, bright, colorful, fascinating, symmetry
[Fooocus] Encoding positive #1 ...
[Fooocus Model Management] Moving model(s) has taken 0.11 seconds
[Fooocus] Encoding positive #2 ...
[Fooocus] Encoding negative #1 ...
[Fooocus] Encoding negative #2 ...
[Fooocus] Image processing ...
[Fooocus] VAE Inpaint encoding ...
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 1.60 seconds
[Fooocus] VAE encoding ...
Final resolution is (4624, 3468), latent is (896, 1080).
[Parameters] Denoising Strength = 1
[Parameters] Initial Latent shape: Image Space (896, 1080)
Preparation time: 20.22 seconds
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 5.46 seconds
 50% 15/30 [00:13<00:13,  1.15it/s]Requested to load SDXL
Loading 1 new model
unload clone 0
[Fooocus Model Management] Moving model(s) has taken 1.88 seconds
Refiner Swapped
100% 30/30 [00:29<00:00,  1.02it/s]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.35 seconds
^C

Additional information

No response

[mashb1t]

solved in #1710, part of todays release 2.3.0