However, many of the ongoing research considers solutions that are quite often quite specific and require large computational resources for training.
🔰 DiffusionFastForward offers a general template for diffusion models for images that can be a starting point for understanding and researching diffusion-based generative models.
- ⚡ PyTorch Lightning to enable easy training!
- 💸 You can run all experiments online on Google colab - no need for own GPU machine!
- 🔎 Examples for both low-resolution and high-resolution data!
- ⛺ Examples of latent diffusion!
- 🎨 Examples of image translation with diffusion!
The code structure is simple, so that you can easily customize it to your own applications.
🚧 Disclaimer: This repository does not provide any weights to the models. The purpose of this software is to be able to train new weights on a previously unexplored type of data.
There are three elements integrated into this project:
- 💻 Code
- 💡 Notes (in
notesdirectory) - 📺 Video Course (to be released on YouTube)
💻 Code
This repository offers a starting point for training diffusion models on new types of data. It can serve as a baseline that can hopefully be developed into more robust solutions based on the specific features of the performed generative task.
It includes notebooks that can be run stand-alone:
01-Diffusion-Sandbox - visualizations of the diffusion process
Assuming torch and torchvision is installed:
pip install pytorch-lightning==1.9.3 diffusers einops💡 Notes
Short summary notes are released as part of this repository and they overlap semantically with the notebooks!
- 01-Diffusion-Theory - visualizations of the diffusion process
- 02-Pixel-Diffusion - basic diffusion suitable for low-resolution data
- 03-Conditional-Pixel-Diffusion - image translation with diffusion for low-resolution data
- 04-Latent-Diffusion - latent diffusion suitable for high-resolution data
- 05-Conditional-Latent-Diffusion - image translation with latent diffusion
📺 Video Course (released on YouTube)
The course is released on YouTube and provides an extension to this repository. Some additional topics are covered, such as seminal papers and on-going research work.
The current plan for the video course (links added upon publishing):
- 📺 #00 Introduction
- 📺 #01 Basics: Denoising Diffusion Process
- 📺 #02 Basics: Denoising Diffusion of Images
- 📺 #03 Practical: Unconditional Diffusion in Low-Resolution
- 🔜 #04 Extra: Summary of Seminal Works
- 📺 #05 Basics: Conditional Diffusion
- 📺 #06 Practical: Condition Diffusion in Low-Resolution
- 📺 #07 Basics: High-Resolution Diffusion
- 📺 #08 Practical: High-Resolution Diffusion
- 🔜 #09 Extra: Diffusion Applications
- 🔜 #10 Extra: Further Insight into Diffusion
Most examples are one of two types of models, trainable within a day:
PixelDiffusion (Good for small images 👶) Appropriate for LR data. Direct diffusion in pixel space.
| Image Resolution | 64x64 |
|---|---|
| Training Time | ~10 hrs |
| Memory Usage | ~4 GB |
LatentDiffusion (Good for large images 🐋) Useful for HR data. Latent diffusion in compressed space.
| Image Resolution | 256x256 |
|---|---|
| Training Time | ~20 hrs |
| Memory Usage | ~5 GB |
There are many great projects focused on diffusion generative models. However, most of them involve somewhat complex frameworks that are not always suitable for learning and preliminary experimentation.
- 🤗 diffusers
- lucidrains PyTorch DDPM
- OpenAI guided-diffusion
- OpenAI improved-diffusion
- CompVis latent-diffusion
- Meta DiT
- MONAI GenerativeModels for Medical Imaging
Some excellent materials have already been published on the topic! Huge respect to all of the creators 🙏 - check them out if their work has helped you!
☕ Blog Posts
- Score-based Perspective by Yang Song
- What are Diffusion Models? by Lilian Weng
- Annotated Diffusion Model by Niels Rogge and Kashif Rasul
- Diffusion as a kind of VAE by Angus Turner
🔮 Explanation Videos
- Diffusion Model Math Explained by Outlier
- What are Diffusion Models? by Ari Seff
- Diffusion Models Beat GANs on Image Synthesis the research paper explained by Yannic Kilcher
🔧 Implementation Videos
- Diffusion Models PyTorch Implementation by Outlier
- High-Resolution Image Synthesis with LDMs | ML Coding Series by Aleksa Gordić
🎓 Video Lectures/Tutorials
- Diffusion Probabilistic Models - MIT 6.S192 lecture by Jascha Sohl-Dickstein
- Generative art using diffusion - MIT 6.S192 lecture by Prafulla Dhariwal
- Learning to Generate Data by Estimating Gradients of the Data Distribution by Yang Song
- Denoising Diffusion-based Generative Modeling: Foundations and Applications tutorial presented at CVPR2022 by Karsten Kreis, Ruiqi Gao and Arash Vahdat
- Generative Modeling by Estimating Gradients of the Data Distribution by Stefano Ermon
- Variational autoencoders and Diffusion Models by Tim Salimans