SeamlessM4TForTextToSpeech.generate not working if generation_config is passed

[ydshieh]

For SeamlessM4TForTextToSpeech with checkpoint "facebook/hf-seamless-m4t-medium",

model.generate(**model_inputs, tgt_lang="eng", generation_config=model.generation_config)

will fail with index error.

If not passing generation_config argument, it could run without error.

Reproduction

from transformers import AutoProcessor, SeamlessM4TForTextToSpeech

ckpt = "facebook/hf-seamless-m4t-medium"

processor = AutoProcessor.from_pretrained(ckpt)
model = SeamlessM4TForTextToSpeech.from_pretrained(ckpt)

model_inputs = processor(["This is a test"], return_tensors="pt")

# This works
outputs = model.generate(**model_inputs, tgt_lang="eng")
print(outputs)

# Index error
outputs2 = model.generate(**model_inputs, tgt_lang="eng", generation_config=model.generation_config)
print(outputs2)

Expected behavior

Should not fail if generation_config is passed.

[ydshieh]

cc @ylacombe

[ydshieh]

The traceback:

Traceback (most recent call last):
  File "/transformers/temp.py", line 16, in <module>
    outputs2 = model.generate(**model_inputs, tgt_lang="eng", generation_config=model.generation_config)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/transformers/src/transformers/models/seamless_m4t/modeling_seamless_m4t.py", line 3485, in generate
    waveform, waveform_lengths = self.vocoder(input_ids=unit_ids, spkr_id=spkr_id, lang_id=vocoder_tgt_lang_id)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/transformers/src/transformers/models/seamless_m4t/modeling_seamless_m4t.py", line 2574, in forward
    hidden_states = self.unit_embedding(input_ids).transpose(1, 2)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/sparse.py", line 190, in forward
    return F.embedding(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 2551, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

[ArthurZucker]

cc @eustlb as well!