Speech-to-Speech Translation with Fairseq/ESPnet-ST

Hello! Welcome back.

In our last lesson, we took a deep dive into the architecture of Meta's SeamlessM4T, understanding how its interconnected components—the speech encoder, text decoder, Text-to-Unit model, and vocoder—work together to perform direct Speech-to-Speech Translation (S2ST). We saw how this design, while complex, overcomes many limitations of older cascaded systems.

Today, we transition from architectural theory to practical application. Your goal for this lesson is to run a speech-to-speech translation task using a pre-built recipe from a toolkit like Fairseq or ESPnet-ST. We'll explore what a "recipe" entails in these major research toolkits and get our hands dirty by running a state-of-the-art S2ST model.

1. Running SeamlessM4T: From Theory to a Live Demo

SeamlessM4T is built upon Fairseq, Meta's sequence modeling toolkit. While we could dive straight into the command line, the easiest way to experience the model we just studied is through a pre-packaged environment. This gives you a tangible sense of the model's capabilities before we dissect the underlying scripts.

We'll use a Google Colab notebook that wraps the SeamlessM4T model in a user-friendly Gradio web interface.

Now, it's your turn.

Practical Exercise: Your First S2ST Run

Please follow the steps from the video to run your own S2ST task.

1. Open the Google Colab notebook linked in the video's description (or you can typically find it by searching for "SeamlessM4T Gradio Colab" on GitHub).
2. In Colab, go to Runtime > Change runtime type and ensure T4 GPU is selected as the hardware accelerator.
3. Run the setup cell(s) as shown in the video. This will install all dependencies and download the SeamlessM4T model. This may take several minutes.
4. Once the setup is complete and a public Gradio URL is generated, open it in a new tab.
5. In the Gradio interface:
   • Select the task: S2ST: Speech to Speech translation.
   • For the input, use the microphone to record yourself saying a simple English sentence, like "Hello, this is a test of the translation system."
   • Choose a target language you are interested in (e.g., Spanish, French, Hindi).
   • Click "Translate" and listen to the generated audio output.

This simple exercise successfully fulfills our primary goal: you've just run a state-of-the-art, direct S2ST system. The user-friendly UI hides the complexity, but underneath, it's executing a pre-built inference recipe: loading the model, preprocessing the input audio, running the two-pass decoding we discussed (speech -> text -> units), and invoking the vocoder to generate the final waveform.

2. Anatomy of a Recipe: The Fairseq Command-Line Approach

The Gradio interface is an application layer. As a researcher and developer, you need to understand the underlying scripts—the "recipe"—that power it. Let's examine the structure of a typical S2ST recipe in Fairseq, using the documentation for S2UT (Speech-to-Unit Translation), a model that laid the groundwork for systems like SeamlessM4T.

Let's break down the key stages of this recipe, which you'll find are common across most speech processing toolkits.

Stage 1: Data Preparation

Before any training can occur, the data must be formatted correctly. The Fairseq recipe uses Python scripts for this:

• prep_s2ut_data.py: This script is responsible for creating manifest files that list the paths to source audio and their corresponding target units.
• Key Concept: A critical step for S2UT is generating the target units. This involves taking the target language audio, passing it through a pre-trained model like HuBERT to get its hidden representations, and then quantizing these representations into a sequence of discrete integers (the "units"). This is exactly the concept of "discrete acoustic units" we covered in the last lesson.

Stage 2: Training

The core of the recipe is the fairseq-train command. This is a powerful, generic trainer that is configured for a specific task via command-line arguments.

Look at the example training command for S2UT:

fairseq-train $DATA_ROOT \
--config-yaml config.yaml \
--task speech_to_speech --target-is-code --target-code-size 100 \
--criterion speech_to_unit --label-smoothing 0.2 \
--arch s2ut_transformer_fisher ...

Even without knowing every flag, you can infer the key configurations:

• --task speech_to_speech: Tells Fairseq we are doing an S2S task.
• --target-is-code: A crucial flag indicating the target is not text, but discrete codes (units).
• --criterion speech_to_unit: Specifies the loss function designed for this task.
• --arch s2ut_transformer_fisher: Defines the model architecture to be a specific variant of the Transformer.

Stage 3: Inference (Decoding)

Inference in this recipe is a two-step process that perfectly mirrors the model's architecture:

1. Generate Units: The fairseq-generate command is used first. It takes the trained model and the source audio from the test set and outputs a sequence of predicted target units.

   fairseq-generate $DATA_ROOT \
   --task speech_to_speech --target-is-code ... \
   --path $MODEL_DIR/checkpoint_best.pt ...
   

2. Synthesize Waveform: The generated unit file is then fed into a second script, generate_waveform_from_code.py. This script loads a pre-trained vocoder (like the HiFi-GAN we discussed) and uses it to convert the sequence of discrete units into an audible waveform.

   python examples/speech_to_speech/generate_waveform_from_code.py \
   --in-code-file .../generate-test.unit \
   --vocoder $VOCODER_CKPT --vocoder-cfg $VOCODER_CFG ...
   

This command-line recipe exposes the full end-to-end pipeline that was abstracted away by the Gradio UI, giving you a clear picture of how these models are run in a development or research context.

3. An Alternative Paradigm: ESPnet Recipes

Another major toolkit you requested to learn is ESPnet. It handles recipes in a slightly different but conceptually similar way. ESPnet recipes are typically self-contained within a single directory and driven by a master shell script, run.sh.

The general structure of an ESPnet2 recipe is: egs2/<dataset_name>/<task_name>/
For example, a speech translation experiment on the CVSS dataset would live in egs2/cvss/st1/.

Inside this directory, you will always find a run.sh script. This script is the single entry point for the entire experiment and is divided into numbered stages.

While the provided video tutorial for ESPnet (LINK) focuses on an ASR task, the principles it demonstrates are universal across all ESPnet recipes, including S2ST (which ESPnet refers to as ST).

A typical run.sh for an ST task would look like this:

#!/usr/bin/env bash
# Set bash to 'debug' mode, it will exit on :
# -e 'error', -u 'undefined variable', -o ... 'error in pipeline', -x 'print commands',
set -e
set -u
set -o pipefail




# ... (argument parsing for stages, configs, etc.) ...

if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
    echo "stage 1: Data preparation"



    # Scripts to download data and create Kaldi-style manifests
fi

if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
    echo "stage 2: Speed perturbation"



    # Data augmentation
fi




# ... (other prep stages) ...

if [ ${stage} -le 10 ] && [ ${stop_stage} -ge 10 ]; then
    echo "stage 10: ST model training"



    # Calls st_train.py with a specified YAML config file
fi

if [ ${stage} -le 11 ] && [ ${stop_stage} -ge 11 ]; then
    echo "stage 11: Decoding"



    # Calls st_recog.py to run inference on the test set
fi

if [ ${stage} -le 12 ] && [ ${stop_stage} -ge 12 ]; then
    echo "stage 12: Scoring"



    # Computes BLEU score on the decoded output
fi

This staged approach is very powerful for research and development. You can run the entire pipeline from scratch or execute a single stage, for example, re-running only the decoding (stage 11) with different parameters without having to re-prepare data or re-train the model.

Conclusion

In this lesson, you bridged the gap between theory and practice. You saw that complex models like SeamlessM4T can be run through simple user interfaces, but that underneath these UIs are structured "recipes" that manage the entire experimental lifecycle.

Key Takeaways:

• Recipes are standardized workflows for data preparation, training, and evaluation, essential for reproducibility in research.
• Fairseq recipes often consist of a series of distinct Python scripts and fairseq-train/fairseq-generate commands configured with extensive command-line flags.
• ESPnet recipes are typically consolidated into a single, staged run.sh script, providing a modular way to control the experimental flow.
• The core steps are consistent across toolkits: prepare data, train model, run inference, and evaluate results.
• You successfully ran a direct S2ST task using a pre-built recipe, translating your own speech from one language to another.

Preview of the Next Lesson:

We've now explored models that translate between speech and text modalities. The next logical step is to look at models that can process and reason about both simultaneously in a more integrated fashion. In the next lesson, we will describe the architecture of multimodal audio-text LLMs like AudioPaLM, investigating how language models are being extended to understand and generate audio directly.