Fix safety checker for some models #248
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…M and LCM-LoRA
Recent diffusers changes seem to have broken the safety checker for
diffusers models without a safety_checker/model.safetensors file: for instance,
fluently/Fluently-v4-LCM simply do not apply the check by default anymore.
Also, for single file models, load_safety_checker=False actually _causes_
CompVis/stable-diffusion-safety-checker to be loaded: see
diffusers/loaders/single_file.py:
if load_safety_checker is not None:
deprecation_message = (
"Please pass instances of `StableDiffusionSafetyChecker` and `AutoImageProcessor`"
"using the `safety_checker` and `feature_extractor` arguments in `from_single_file`"
)
deprecate("load_safety_checker", "1.0.0", deprecation_message)
safety_checker_components = _legacy_load_safety_checker(local_files_only, torch_dtype)
This is more noticeable when running offline, since the safety model may still
not be present in the cache.
So move the check config to the init phase, to align the safety model and
pipeline lifetimes, and test for the safety_checker and feature_extractor
attributes right after loading, making sure they match the expected behavior.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The first changeset fixes safety checker behavior for models which do not incorporate the safety model internally (safetensors and some diffusers models).
The other two are somewhat-related minor memory optimizations.
Unfortunately, I couldn't make the check work on OpenVINO at all, regardless of this change (but I can't test it very well, it's too memory-hungry for my PC). If the filter is enabled on OpenVINO, and not working, this change may cause a crash instead; but it shouldn't interfere on a correctly working checker.