🏞️ Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition

By Chun-Hsiao Yeh*, Ta-Ying Cheng*, He-Yen Hsieh*, Chuan-En Lin, Yi Ma, Andrew Markham, Niki Trigoni, H.T. Kung, Yubei Chen ( * equal contribution)

UC Berkeley, University of Oxford, Harvard University, CMU, HKU, UC Davis

tags: stable diffusion personalized text-to-image generation llm

This repo is the official implementation of "Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition".

TL;DR: We introduce a dataset creation pipeline, Gen4Gen, to compose personal concept into realistic scenes with complex compositions, accompanied by detailed text descriptions.

📝 Updates

• [Feb 26 2024] [⚡️NEWS⚡️] Added 🏞️ Gen4Gen: dataset creation pipeline code!"
• [Feb 15 2024] Added demo video 🔥 for "How Gen4Gen Works"

🔎 Overview of This Repository

• gen4gen: Toolkits for Gen4Gen pipeline.

🎁 Prepare Personal Assets

Please prepare your personal images and put them under data/s0_source_images. Our personal images are from Unsplash. The structure of 🗂 data/s0_source_images
looks like this:

🗂 Structure of data/s0_source_images

../data/s0_source_images
└── cat_dog_houseplant_3objs
    ├── cat
    │   └── sergey-semin-agQhOHQipoE-unsplash.jpg
    ├── dog
    │   ├── Copy of 5.jpeg
    │   └── Copy of 6.jpeg
    └── houseplant
        ├── Copy of 1.png
        ├── Copy of 2.png
        ├── Copy of 3.png
        └── Copy of 5.png
└── [folder_of_other_scenes]
    ├── [object_name_1]
    │   ├── [image_name_1.jpg]
    │   ├── ...
    │   └── [image_name_n.jpeg]
    ...
    └── [oject_name_n]

🗺️ Environments

Library versions

• Framework and environment
  • pytorch: 1.13.1
  • cuda: 11.7
  • torchvision: 0.14.1
• 🧨 Go-to library for diffusion models
  • diffusers: 0.21.4
• OpenAI API for LLM-Guide Object Composition
  • openai: 0.28.1
• Programming language
  • python: 3.8.5
• Graphics card
  • [Gen4Gen pipeline] GPU: NVIDIA A100-SXM4-40GB x 1 or NVIDIA GeForce RTX 4090-24GB x 1

    • For Step1, Object Association and Foreground Segmentation, requiring around 2.2GB memory footprint
    • For Step3, Background Repainting, requring around 17GB memory footprint

👍 Acknowledgement

Our codebase is built based on DIS, LLM-grounded Diffusion, SD-XL Inpainting, and custom-diffusion. We really appreciate the authors for the nicely organized code and fantastic works!

📬 How to Get Support?

If you have any general questions or need support, please feel free to contact: Chun-Hsiao Yeh, Ta-Ying Cheng and He-Yen Hsieh. Also, we encourage you to open an issue in the GitHub repository. By doing so, you not only receive support but also contribute to the collective knowledge base for others who may have similar inquiries.

❤️ Citation

If you find the codebase and MyCanvas dataset valuable and utilize it in your work, we kindly request that you consider giving our GitHub repository a ⭐ and citing our paper.

@misc{yeh2024gen4gen,
  author        = {Chun-Hsiao Yeh and
                   Ta-Ying Cheng and
                   He-Yen Hsieh and
                   David Chuan-En Lin and
                   Yi Ma and
                   Andrew Markham and
                   Niki Trigoni and
                   H.T. Kung and
                   Yubei Chen},
  title         = {Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition},
  year          = {2024},
  eprint        = {2402.15504},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}