Stutter-Solver: End-to-end Cross-lingual Dysfluency Detection

Architecture

Datasets

We open sourced our three simulated datasets, VCTK-Art, VCTK-Pro and AISHELL3-Pro.

Download link will be attached soon.

Audio samples

check Audio samples

Pre-requisites

Please refer environment.yml

If you have Miniconda/Anaconda installed, you can directly use the command: conda env create -f environment.yml

Stutter-Solver Inference

We opensourced our inference code and checkpoints, here are the steps to perform inference:

0. Clone this repository

1. Download VITS pretrained model, here we use pretrained_ljs.pth.

2. Download Stutter-Solver-checkpoints, create a folder under stutter-solver, named saved_models, and put all downloaded models into it.

3. We also provide testing datasets for quick inference, you can download it here.

4. Build Monotonic Alignment Search

cd stutter-solver/monotonic_align
python setup.py build_ext --inplace

5. Run stutter-solver/etc/inference.ipynb to perform inference step by step.

TTS-based Dysfluency Simulation

English

We use VITS as our TTS model.

0. Clone this repository

1. Download VITS pretrained models, here we need pretrained_vctk.pth to achieve multi-speaker.

   1. create a folder dysfluency_simulation/path/to, and put the downloaded model into it.

2. Build Monotonoic Alignment Search

cd dysfluency_simulation/monotonic_align
python setup.py build_ext --inplace

3. Generate simulated speech

# Phoneme level
python generate_phn.py

# Word level
python generate_word.py

Chinese Mandarin

• Switch to vits-chinese branch

We use vits_chinese as our TTS model for Chinese simulation. Download checkpoints according to its README and place it in the specified path.

0. Build Monotonoic Alignment Search

cd monotonic_align
python setup.py build_ext --inplace

1. Generate simulated speech

python generate_dysfluency.py

Citataion

If you find our paper helpful, please cite it by:

@misc{zhou2024stuttersolverendtoendmultilingualdysfluency,
      title={Stutter-Solver: End-to-end Multi-lingual Dysfluency Detection}, 
      author={Xuanru Zhou and Cheol Jun Cho and Ayati Sharma and Brittany Morin and David Baquirin and Jet Vonk and Zoe Ezzes and Zachary Miller and Boon Lead Tee and Maria Luisa Gorno Tempini and Jiachen Lian and Gopala Anumanchipalli},
      year={2024},
      eprint={2409.09621},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2409.09621}, 
}