Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not work in google colab. #6

Open
Rondalter opened this issue Feb 2, 2024 · 27 comments
Open

Not work in google colab. #6

Rondalter opened this issue Feb 2, 2024 · 27 comments
Assignees
Labels
bug Something isn't working

Comments

@Rondalter
Copy link

I am running it in colab, and since the modification that added the photomaker it does not work.
You click on generate and it stays blocked.

@machineminded
Copy link
Owner

I will need to make a separate configuration script for unix-like environments. Shouldn't be too much work. I am not familiar with Google Colab but can make an effort.

@machineminded machineminded self-assigned this Feb 2, 2024
@machineminded machineminded added the bug Something isn't working label Feb 2, 2024
@machineminded
Copy link
Owner

@Rondalter I did go in and update the notebook. I am trying to use it but I don't think it will work because it is saying I have zero compute resources available. I'm not familiar with how Colab works but if you could test it out that would be great.

@Rondalter
Copy link
Author

equirement already satisfied: pygit2==1.12.2 in /usr/local/lib/python3.10/dist-packages (1.12.2)
Requirement already satisfied: cffi>=1.9.1 in /usr/local/lib/python3.10/dist-packages (from pygit2==1.12.2) (1.16.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.9.1->pygit2==1.12.2) (2.21)
/content/Fooocus-inswapper
[System ARGV] ['launch.py', '--share']
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
Fooocus version: 2.1.864
Version mismatch for tb-nightly: Installed version 2.16.0a20240202 does not meet requirement tb-nightly
Installing requirements
Total VRAM 15102 MB, total RAM 12979 MB
Set vram state to: NORMAL_VRAM
Always offload VRAM
Device: cuda:0 Tesla T4 : native
VAE dtype: torch.float32
Using pytorch cross attention
Exception in thread Thread-2 (worker):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/content/Fooocus-inswapper/modules/async_worker.py", line 25, in worker
import modules.default_pipeline as pipeline
File "/content/Fooocus-inswapper/modules/default_pipeline.py", line 1, in
import modules.core as core
File "/content/Fooocus-inswapper/modules/core.py", line 1, in
from modules.patch import patch_all
File "/content/Fooocus-inswapper/modules/patch.py", line 11, in
import ldm_patched.modules.sd1_clip
File "/content/Fooocus-inswapper/ldm_patched/modules/sd1_clip.py", line 3, in
from transformers import CLIPTokenizer
ImportError: cannot import name 'CLIPTokenizer' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/init.py)
2024-02-02 18:25:31.015317: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-02 18:25:31.015376: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-02 18:25:31.021640: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-02 18:25:32.883777: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Refiner unloaded.
model_type EPS
UNet ADM Dimension 2816
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
Base model loaded: /content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors
Request to load LoRAs [['sd_xl_offset_example-lora_1.0.safetensors', 0.1], ['None', 1.0], ['None', 1.0], ['None', 1.0], ['None', 1.0]] for model [/content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors].
Loaded LoRA [/content/Fooocus-inswapper/models/loras/sd_xl_offset_example-lora_1.0.safetensors] for UNet [/content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors] with 788 keys at weight 0.1.
Fooocus V2 Expansion: Vocab with 642 words.
/usr/local/lib/python3.10/dist-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
Fooocus Expansion engine loaded for cuda:0, use_fp16 = True.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.60 seconds
Controls: [textbox, textbox, checkboxgroup, radio, radio, slider, textbox, slider, slider, dropdown, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, checkbox, textbox, radio, image, checkboxgroup, image, textbox, image, image, slider, slider, radio, image, slider, slider, radio, image, slider, slider, radio, image, slider, slider, radio, checkbox, image, number, checkbox, image, image, image, image]
Running on local URL: http://127.0.0.1:7865/
Running on public URL: https://9e35fd3f3edf6da406.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)

@Rondalter
Copy link
Author

Rondalter commented Feb 2, 2024

it stays at that point.
I have once managed to generate a basic image.
But if you try any option, an error appears.

@machineminded
Copy link
Owner

On the free tier of Google Colab, how much VRAM is allocated? I think it would need at least 16GB, but even that is probably not enough for use with Fooocus. Maybe 18GB or so. On my local with a 3090, Fooocus is using just over 16GB during generation.

image

@machineminded
Copy link
Owner

When I run it, I am only allocated 15GB of VRAM on the free tier. If you are using the free T4 then this could be the issue.

image

@machineminded
Copy link
Owner

I should have asked if this was for inswapper or PhotoMaker.

@zachysaur
Copy link

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
Fooocus version: 2.1.864
Version mismatch for tb-nightly: Installed version 2.16.0a20240202 does not meet requirement tb-nightly
Installing requirements
Total VRAM 15102 MB, total RAM 12979 MB
Set vram state to: NORMAL_VRAM
Always offload VRAM
Device: cuda:0 Tesla T4 : native
VAE dtype: torch.float32
Using pytorch cross attention
Exception in thread Thread-2 (worker):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/content/Fooocus-inswapper/modules/async_worker.py", line 25, in worker
import modules.default_pipeline as pipeline
File "/content/Fooocus-inswapper/modules/default_pipeline.py", line 1, in
import modules.core as core
File "/content/Fooocus-inswapper/modules/core.py", line 1, in
from modules.patch import patch_all
File "/content/Fooocus-inswapper/modules/patch.py", line 11, in
import ldm_patched.modules.sd1_clip
File "/content/Fooocus-inswapper/ldm_patched/modules/sd1_clip.py", line 3, in
from transformers import CLIPTokenizer
ImportError: cannot import name 'CLIPTokenizer' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/init.py)
2024-02-02 18:25:31.015317: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-02 18:25:31.015376: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-02 18:25:31.021640: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-02 18:25:32.883777: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Refiner unloaded.
model_type EPS
UNet ADM Dimension 2816
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])
Base model loaded: /content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors
Request to load LoRAs [['sd_xl_offset_example-lora_1.0.safetensors', 0.1], ['None', 1.0], ['None', 1.0], ['None', 1.0], ['None', 1.0]] for model [/content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors].
Loaded LoRA [/content/Fooocus-inswapper/models/loras/sd_xl_offset_example-lora_1.0.safetensors] for UNet [/content/Fooocus-inswapper/models/checkpoints/juggernautXL_v8Rundiffusion.safetensors] with 788 keys at weight 0.1.
Fooocus V2 Expansion: Vocab with 642 words.
/usr/local/lib/python3.10/dist-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
Fooocus Expansion engine loaded for cuda:0, use_fp16 = True.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.60 seconds
Controls: [textbox, textbox, checkboxgroup, radio, radio, slider, textbox, slider, slider, dropdown, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, dropdown, slider, checkbox, textbox, radio, image, checkboxgroup, image, textbox, image, image, slider, slider, radio, image, slider, slider, radio, image, slider, slider, radio, image, slider, slider, radio, checkbox, image, number, checkbox, image, image, image, image]
Running on local URL: http://127.0.0.1:7865/
Running on public URL: https://9e35fd3f3edf6da406.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)

@EntonyHop
Copy link

EntonyHop commented Feb 13, 2024

the same issue, infinite generation of even a normal image. The style button doesn't load either. The fooocus itself doesn't work, I think it's the problem.

@machineminded
Copy link
Owner

I am still working thru this. It will take some time but hopefully soon.

@EntonyHop
Copy link

EntonyHop commented Feb 19, 2024

Let us know when you fix the problem, all work has stopped. Thx

@machineminded
Copy link
Owner

@EntonyHop Understood - I will try to get this working.

@machineminded
Copy link
Owner

I am am able to see normal Fooocus features working now in another branch. I found the issue with the inswapper integration on linux and will resolve that next.

@machineminded
Copy link
Owner

This has been difficult for me to test. But I am making progress. We need a better way to manage the queue:

lllyasviel#2271

Additionally, I believe they have some code in place that detects if you are running Stable Diffusion on the free tier, so the VM/container gets shut down. I don't have a paid subscription otherwise this would be easier for me to test. Will consider paying in order to get this issue closed up.

@zachysaur
Copy link

try to kaggle

@machineminded
Copy link
Owner

try to kaggle

Thanks for the suggestion. I'll check it out!

@mashb1t
Copy link

mashb1t commented Feb 28, 2024

This has been difficult for me to test. But I am making progress. We need a better way to manage the queue:

lllyasviel#2271

Additionally, I believe they have some code in place that detects if you are running Stable Diffusion on the free tier, so the VM/container gets shut down. I don't have a paid subscription otherwise this would be easier for me to test. Will consider paying in order to get this issue closed up.

no additional code, but args for better resource allocation: lllyasviel#1710

@dmitryalexander
Copy link

does in-swapper work for colab now?

@mashb1t
Copy link

mashb1t commented May 28, 2024

@dmitryalexander Fooocus (this repository) does not have an inswapper implementation. Please check out https://github.com/machineminded/Fooocus-inswapper maintained by @machineminded.

omg, nvm, wrong repository. I'm so sorry 😬

@dmitryalexander
Copy link

@dmitryalexander Fooocus (this repository) does not have an inswapper implementation. Please check out https://github.com/machineminded/Fooocus-inswapper maintained by @machineminded.

omg, nvm, wrong repository. I'm so sorry 😬

yeah i got the inswapper working, but i couldn't get the instantID or the photo maker to work

@machineminded
Copy link
Owner

@dmitryalexander Is it just not working in colab? What is the error? I don't have a good way to support colab because I'd have to pay for it to use SD.

@dmitryalexander
Copy link

it's only 10$ a month for colab. but yeah, the instant id, it just doesn't activate... it's like it's invisible and the photomaker, it says something like trigger word activated more than once... and it doesn't work from there, i guess that problem happens on some CUDA cards. but apparently the main branch fixed that issue, but i wasn't able to find out how.

@machineminded
Copy link
Owner

Can you show me the actual log output or a screenshot if you get a chance?

@dmitryalexander
Copy link

Traceback (most recent call last): File "/content/Fooocus-inswapper/modules/async_worker.py", line 1047, in worker handler(task) File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/content/Fooocus-inswapper/modules/async_worker.py", line 915, in handler imgs = generate_photomaker(photomaker_prompt, photomaker_source_images, photomaker_negative_prompt, steps, task['task_seed'], width, height, guidance_scale, loras, sampler_name, scheduler_name, async_task) File "/content/Fooocus-inswapper/modules/pm.py", line 119, in generate_photomaker images = pipe( File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/content/Fooocus-inswapper/photomaker/pipeline.py", line 344, in call ) = self.encode_prompt_with_trigger_word( File "/content/Fooocus-inswapper/photomaker/pipeline.py", line 176, in encode_prompt_with_trigger_word raise ValueError( ValueError: PhotoMaker currently does not support multiple trigger words in a single prompt. Trigger word: img, Prompt: completely normal prompt nothing to see here. Total time: 0.47 seconds

@dmitryalexander
Copy link

apparently the problem was that there's some kind of space before the trigger word... or something like that?

@machineminded
Copy link
Owner

machineminded commented May 29, 2024

Oh, yeah. This is how PhotoMaker works: you need to put "img" in the prompt after "man" or "woman". So your prompt with PhotoMaker enabled might be something like:

"woman img, smile, long hair, standing in a library, incredible lighting"

Check out some of the documentation here:

https://github.com/TencentARC/PhotoMaker?tab=readme-ov-file#use-like-diffusers

"img" is the default trigger word.

@dmitryalexander
Copy link

interesting....i'll try it out :thanks:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

6 participants