Lpw_stable_diffusion pipeline (longer prompts, prompt weights!)

docker-diffusers-api
1

Warning: this is an experimental feature under active development. This has been running successfully on kiri.art since December, 2022.

This is the first implementation of the new ALL YOUR PIPELINES (are belong to us) - docker-diffusers-api - Kiri.Art Forums support (in dev), starting with the lpw_stable_diffusion pipeline. Quoting their intro:

Long Prompt Weighting Stable Diffusion

Features of this custom pipeline:

  • Input a prompt without the 77 token length limit.
  • Includes tx2img, img2img. and inpainting pipelines.
  • Emphasize/weigh part of your prompt with parentheses as so: a baby deer with (big eyes)
  • De-emphasize part of your prompt as so: a [baby] deer with big eyes
  • Precisely weigh part of your prompt as so: a baby deer with (big eyes:1.3)

Prompt weighting equivalents:

  • a baby deer with == (a baby deer with:1.0)
  • (big eyes) == (big eyes:1.1)
  • ((big eyes)) == (big eyes:1.21)
  • [big eyes] == (big eyes:0.91)

If you see Token indices sequence length is longer than the specified maximum sequence length for this model ( *** > 77 ) . Running this sequence through the model will result in indexing errors . Do not worry, it is normal.

Massive props to SkyTNT for contributing this awesome pipeline to diffusers! :pray:

This is super new and not well tested… please send feedback!
Make sure you’re running the latest commits from **dev branch.

No special build args are required! :tada:

Test like this:

python test.py txt2img \
  --call-arg PIPELINE="lpw_stable_diffusion"
  --call-arg custom_pipeline_method="text2img"  # See notes below.
  --model-arg width=768  # If using stabilityai/stable-diffusion-2
  --model-arg height=768 # If using stabilityai/stable-diffusion-2
  --model-arg seed=1     # Useful to experiment with weight changes
  --model-arg prompt="a baby deer with (big eyes)"

Further explanations:

  • PIPELINE="lpw_stable_diffusion"

    Specifies this community pipeline. Currently this is the only one supported, but we
    may well support more or all out of the box if this is a success.

  • custom_pipeline_method="text2img"

    lpw_stable_diffusion supports text2img (“text”, not “txt”), img2img and inpaint, so you get to enjoy this awesomeness for all these use-cases!

  • width,height

    Make sure to match (or be close) to the trained image sizes for best results. stabilityai/stable-diffusion-2 is native 768x768. stabilityai/stable-diffusion-2-base (and all the older models) are native 512x512. Asking for a 512x512 image from a 768x768 gives incredibly poor results!

  • seed

    Setting this to a constant value (e.g. 1, or any number) can be useful to understand how
    changing prompt weights (and only the weights) affects the “same” image.

  • prompt

    See the examples above from the pipeline author for a good explanation.

Further information is available in the lpw_stable_diffusion.py source, particularly the doc comments for text2img(), img2img and inpaint towards the end of the file (but also some interesting comments higher up on how the prompt weightings work, etc).