Real multi-inputs ControlNet unit #2539
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jan 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previous multi-inputs support #2525 essentially unfolds the multi-input unit into multiple units, and do processing, which is not very efficient, and require user to calculate a fraction weight based on number of input images.
This PR implements the real multi-inputs ControlNet unit, which allow multiple input images to be passed together to FaceID preprocessor in one pass. You can now use weight = 1.0 instead of calculating a fraction weight.
In API, a new syntax is introduced to support this. The simple list syntax has been taken (first item image, second item mask), so we have to do extra workaround.
{"image": [{"image": img} for img in imgs]}Result comparison
faceid portrait multiple units
faceid portrait single unit multiple inputs
There are some slight differences. This can be attributed to floating point calculation loss. I think the performance is equivalent. Please file an issue if you have observed any abnormal results after we switch to real multi-inputs arch.