Blew up on me after first 500 steps

[boundlessliving]

Problem just started for me today. After the first 500 steps, the process blows up with an error that STARTS with this:

File "/workspace/Dreambooth-Stable-Diffusion/main.py", line 878, in
trainer.fit(model, data)
File "/opt/conda/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
self._call_and_handle_interrupt(
File "/opt/conda/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 723, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 811, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)

....

and then finally ends with:

File "/workspace/Dreambooth-Stable-Diffusion/main.py", line 880, in
melk()
File "/workspace/Dreambooth-Stable-Diffusion/main.py", line 857, in melk
trainer.save_checkpoint(ckpt_path)
File "/opt/conda/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 2467, in save_checkpoint
self._checkpoint_connector.save_checkpoint(filepath, weights_only=weights_only, storage_options=storage_options)
File "/opt/conda/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 444, in save_checkpoint
_checkpoint = self.dump_checkpoint(weights_only)
File "/opt/conda/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 422, in dump_checkpoint
self.trainer._call_callbacks_on_save_checkpoint(checkpoint)
File "/opt/conda/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1687, in _call_callbacks_on_save_checkpoint
state = callback.on_save_checkpoint(self, self.lightning_module, checkpoint)
File "/workspace/Dreambooth-Stable-Diffusion/ldm/modules/callbacks/captions.py", line 13, in on_save_checkpoint
for image_path in data.image_paths
AttributeError: 'ConcatDataset' object has no attribute 'image_paths'

Anyone else experiencing this and/or have a workaround?

Thanks!

Bbo

[caniyabanci76]

yes. I had to comment a few lines out in main.py to get it going.

#121 (comment)

[boundlessliving]

Ok, great! Glad to know I'm not alone. Willing to share which lines?

[caniyabanci76]

I commented out captions_callback (lines 774,775,776) in main.py. They were some the most recent changes made in the last day.

[boundlessliving]

Thanks so much! That did the trick!