I’m glad to hear that you found my message exciting and appreciated my willingness to contribute to the project. I understand that implementing ControlNet support is a priority for you, and I’m happy to work closely with you on this.
I’ve gone through the diffusers release notes on GitHub as you suggested, and it seems like a valuable resource for understanding ControlNet. I’ll read them from the bottom up to get a comprehensive understanding of the feature’s evolution. I’ll also check out the ControlNet section in the Diffusers blog post.
While you consider the optimal approach to implementing this feature, I’d like to share the initial steps I plan to take in the meantime. These steps will be added to a fork of the project:
- Given the specific requirements of ControlNet, particularly with MultiControlnets, it appears that custom pipelines specific to different applications will be necessary. Therefore, I intend to incorporate support for these custom pipelines.
- I will introduce a flag that enables the ControlNet feature. This flag will allow users to easily activate or deactivate ControlNet functionality as needed.
- Additionally, I will enhance the capability to download multiple ControlNet features based on user input, taking into account both runtime and build time considerations. This enhancement will ensure that users have access to the ControlNet features they require, tailored to their specific needs.
By implementing these steps, we can lay a foundation for the integration of ControlNet into the project. Please take your time to consider these approaches, and I’m open to any further suggestions or adjustments you may have.
Is there any specific reason you locked the diffusers package to a specific commit? I am thinking of using the latest version in my fork.
I guess you already answered the above question in Is there a plan to update to v0.15.1 for diffusers?