Stable Diffusion (GLIGEN) Download Error

[isouf]

Hello and thank you for your great work 🙇

I am trying to use AIT to speed up GLIGEN (stable diffusion). Here is my fork where you can see the main changes I made to the codebase. In short, I updated Docker and adapted examples/05_stable_diffusion to allow me to download, compile and run StableDiffusionGLIGENTextImagePipeline.

The issue I am facing is that I cannot run the download_pipeline.py script successfully. Here is the print out:

root@59db038baadf:/AITemplate/examples/05_stable_diffusion# python3 scripts/download_pipeline.py --model-name "anhnct/Gligen_Inpainting_Text_Image"     
image_project/diffusion_pytorch_model.safetensors not found
Loading pipeline components...: 100%|████████████████████████████████████████████████████████| 8/8 [00:03<00:00,  2.19it/s]
Traceback (most recent call last):
  File "scripts/download_pipeline.py", line 48, in <module>
    download_pipeline_files()
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "scripts/download_pipeline.py", line 39, in download_pipeline_files
    StableDiffusionGLIGENTextImagePipeline.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/diffusers/pipelines/pipeline_utils.py", line 659, in save_pretrained
    save_method(os.path.join(save_directory, pipeline_component_name), **save_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 2112, in save_pretrained
    safe_save_file(shard, os.path.join(save_directory, shard_file), metadata={"format": "pt"})
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 281, in save_file
    serialize_file(_flatten(tensors), filename, metadata=metadata)
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 475, in _flatten
    return {
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 479, in <dictcomp>
    "data": _tobytes(v, k),
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 396, in _tobytes
    raise ValueError(
ValueError: You are trying to save a non contiguous tensor: `visual_projection.weight` which is not allowed. It either means you are trying to save tensors which are reference of each other in which case it's recommended to save only the full tensors, and reslice at load time, or simply call `.contiguous()` on your tensor to pack it before saving.

Steps to reproduce:

• Clone https://github.com/isouf/AITemplate
• Build and run docker: ./docker/build.sh cuda && docker run --gpus all -it ait:latest
• Go to examples/05_stable_diffusion and run: python3 scripts/download_pipeline.py --model-name "anhnct/Gligen_Inpainting_Text_Image"

[ipiszy]

Hi @isouf , the supported SD version is a bit old, please refer to https://github.com/facebookincubator/AITemplate/tree/main/examples/05_stable_diffusion to check the supported version.

[isouf]

Hi @ipiszy, many thanks - may I confirm if I could use the Alternative Pipeline for older SD versions?

[ipiszy]

I think it should work.

[isouf]

@ipiszy when I try to run the example of the Alternative Pipeline I get the following error. Any ideas why is this happening?

root@f020e85e0199:/AITemplate/examples/05_stable_diffusion# python3 scripts/demo_alt.py --hf-hub-or-path runwayml/stable-diffusion-v1-5 --ckpt v1-5-pruned-emaonly.ckpt
INFO:aitemplate.backend.build_cache_base:Build cache disabled
2023-11-17 11:28:10,752 INFO <aitemplate.testing.detect_target> Set target to CUDA
Traceback (most recent call last):
  File "scripts/demo_alt.py", line 64, in <module>
    run()
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "scripts/demo_alt.py", line 44, in run
    pipe = StableDiffusionAITPipeline(
  File "/AITemplate/examples/05_stable_diffusion/src/pipeline_stable_diffusion_ait_alt.py", line 690, in __init__
    self.clip_ait_exe = self.init_ait_module(
  File "/AITemplate/examples/05_stable_diffusion/src/pipeline_stable_diffusion_ait_alt.py", line 778, in init_ait_module
    mod = Model(os.path.join(workdir, model_name, "test.so"))
  File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 228, in __init__
    self.DLL = self._DLLWrapper(lib_path)
  File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 179, in __init__
    self.DLL = ctypes.cdll.LoadLibrary(lib_path)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 451, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: tmp/CLIPTextModel/test.so: cannot open shared object file: No such file or directory