#
Multi-modal instruction to image

Multimodal instructions mean you can input arbitrarily interleaved text and image inputs as conditions to guide image generation. You can input multiple images and use prompts to describe the desired output. This approach is more flexible than using only text or images.

##
Examples

Take `OmniGenPipeline` as an example: the input can be a text-image sequence to create new images, he input can be a text-image sequence, with images inserted into the text prompt via special placeholder `<img><|image_i|></img>`.

    import torch
    from diffusers import OmniGenPipeline
    from diffusers.utils import load_image

    pipe = OmniGenPipeline.from_pretrained(
        "Shitao/OmniGen-v1-diffusers",
        torch_dtype=torch.bfloat16
    )
    pipe.to("cuda")

    prompt="A man and a woman are sitting at a classroom desk. The man is the man with yellow hair in <img><|image_1|></img>. The woman is the woman on the left of <img><|image_2|></img>"
    input_image_1 = load_image("https://raw.githubusercontent.com/VectorSpaceLab/OmniGen/main/imgs/docs_img/3.jpg")
    input_image_2 = load_image("https://raw.githubusercontent.com/VectorSpaceLab/OmniGen/main/imgs/docs_img/4.jpg")
    input_images=[input_image_1, input_image_2]
    image = pipe(
        prompt=prompt,
        input_images=input_images,
        height=1024,
        width=1024,
        guidance_scale=2.5,
        img_guidance_scale=1.6,
        generator=torch.Generator(device="cpu").manual_seed(666)).images[0]
    image

input_image_1
input_image_2
generated image

    import torch
    from diffusers import OmniGenPipeline
    from diffusers.utils import load_image

    pipe = OmniGenPipeline.from_pretrained(
        "Shitao/OmniGen-v1-diffusers",
        torch_dtype=torch.bfloat16
    )
    pipe.to("cuda")

    prompt="A woman is walking down the street, wearing a white long-sleeve blouse with lace details on the sleeves, paired with a blue pleated skirt. The woman is <img><|image_1|></img>. The long-sleeve blouse and a pleated skirt are <img><|image_2|></img>."
    input_image_1 = load_image("/share/junjie/code/VISTA2/produce_data/laion_net/diffgpt/OmniGen/docs_img/emma.jpeg")
    input_image_2 = load_image("/share/junjie/code/VISTA2/produce_data/laion_net/diffgpt/OmniGen/docs_img/dress.jpg")
    input_images=[input_image_1, input_image_2]
    image = pipe(
        prompt=prompt,
        input_images=input_images,
        height=1024,
        width=1024,
        guidance_scale=2.5,
        img_guidance_scale=1.6,
        generator=torch.Generator(device="cpu").manual_seed(666)).images[0]

person image
clothe image
generated image

The output image is a `PIL.Image` object that can be saved:

    image.save("generated_image.png")

< > Update on GitHub