What diffusion model?

[festivus37]

Hi, thanks for all your work on this. So what diffusion model does it use on the backend? Is this something where we can swap in an SDXL model for the image rendering part of it?

[redrockvp]

It seems to be using RealVisXL_V4.0, but the folder structure doesn't look like anything I'm familiar with. So also hoping for some clarity here :) amazing tool, but would be even more amazing with an easy way to switch out image models.

[lludlow]

If the model is on hugging face and in the correct format yes

I made a few changes to get Juggernaut to work, they have teh models up on HF however they are not in the format the app is expecting.

in gradio_app.py I made the following changes

#sdxl_name = 'SG161222/RealVisXL_V4.0'
# sdxl_name = 'stabilityai/stable-diffusion-xl-base-1.0'
sdxl_name = 'RunDiffusion/Juggernaut-X-v10'

I also changed the way the files are downloaded to exclude requiring fp16

tokenizer = CLIPTokenizer.from_pretrained(sdxl_name, subfolder="tokenizer")
tokenizer_2 = CLIPTokenizer.from_pretrained(sdxl_name, subfolder="tokenizer_2")
text_encoder = CLIPTextModel.from_pretrained(sdxl_name, subfolder="text_encoder")
text_encoder_2 = CLIPTextModel.from_pretrained(sdxl_name, subfolder="text_encoder_2")
vae = AutoencoderKL.from_pretrained(sdxl_name, subfolder="vae")  # Default dtype is fp32
unet = UNet2DConditionModel.from_pretrained(sdxl_name, subfolder="unet")

#tokenizer = CLIPTokenizer.from_pretrained(
#    sdxl_name, subfolder="tokenizer")
#tokenizer_2 = CLIPTokenizer.from_pretrained(
#    sdxl_name, subfolder="tokenizer_2")
#text_encoder = CLIPTextModel.from_pretrained(
#    sdxl_name, subfolder="text_encoder", torch_dtype=torch.float16, variant="fp16")
#text_encoder_2 = CLIPTextModel.from_pretrained(
#    sdxl_name, subfolder="text_encoder_2", torch_dtype=torch.float16, variant="fp16")
#vae = AutoencoderKL.from_pretrained(
#    sdxl_name, subfolder="vae", torch_dtype=torch.bfloat16, variant="fp16")  # bfloat16 vae
#unet = UNet2DConditionModel.from_pretrained(
#    sdxl_name, subfolder="unet", torch_dtype=torch.float16, variant="fp16")

I submitted a PR that is a little cleaner than this #13

[xhoxye]

FP32 causes 8GB vram gpu out of memory error

[lludlow]

Take a look at #27

[xhoxye]

It didn't solve my problem "FP32 causes 8GB vram gpu out of memory error". It requires more vram.

[sild928]

Moving to juggernaut absolutely crushes my iteration speed. Granted I don't have a beefy card, just has a lot of memory, but my iteration times on upscaling go from 1-2 seconds to 1-2 minutes per step. The base generation has a slight increase but nothing like upscaling. Thanks for figuring this out though, hope there are other models out there that will work and are fp16 so other changes aren't necessary. BTW if I just change the model back to RealVis and keep the other changes, I see the same behavior.

[lludlow]

Makes sense, if you see the note in the code change, Default dtype is fp32.

I'll take a look at this function more later. Or maybe @lllyasviel can update

[msalman264]

Diffusion models, such as the Bass diffusion model or the Logistic diffusion model, help analyze the spread of innovations or new products in a market, aiding businesses in understanding adoption patterns and forecasting future trends.

[QijiTec]

我在本地修改了模型文件夹，指向到自己的本地模型（转换了Diffusers格式），可以正常使用。

模型地址 https://huggingface.co/GuangyuanSD/AiARTiSTACG6

不过由于内建hyper加速器，所以采样器最好是使用 Euler A，采样步数为10，CFG 1
我是重度FORGE使用者和传播者，不知道有没有可能把OMOST集成进FORGE的可能？

也希望能有机会的话，可不可以让omost内置ACG6这个加速模型，谢谢开发团队！

I modified the model folder locally to point to my local model (converted to Diffusers format), and it can be used normally.

Model address https://huggingface.co/GuangyuanSD/AiARTiSTACG6

But due to the built-in super accelerator, the sampler is best to use Euler A, the number of sampling steps is 10, and CFG 1
I am a heavy user and evangelist of FORGE. I wonder if it is possible for OMOST to be integrated into FORGE?

If you want the chance, you can use most of the built-in ACG6 acceleration models, thanks to the development team!

[Nurburgring-Zhang]

That's great