Swahili ASR + LLM Error Correction

Build a Swahili speech recognition pipeline with LLM-based error correction on Modal.

Introduction

30 Second Summary

Speech assistants work remarkably well for English, but try using one in Swahili and you will notice far more errors. Languages with fewer digital resources still lag behind in recognition accuracy, and simply running a bigger model does not solve the problem.

In this project, you will build a pipeline that transcribes Swahili audio using Whisper, generates multiple candidate transcriptions, and uses an African-language-optimized LLM to correct errors. The entire system deploys on Modal with T4 GPUs.

What You'll Build

A complete Swahili speech recognition pipeline combining OpenAI Whisper large-v3-turbo for initial transcription with AfriqueLlama-8B for generative error correction, deployed on Modal's serverless GPU infrastructure.

By the end of this project, you'll have:

  • A Modal app with separate ASR and LLM inference functions running on T4 GPUs.
  • An N-best hypothesis generation module that produces diverse transcription candidates via temperature sampling.
  • A Generative Error Correction module using Task-Activating Prompting to fix Swahili transcription errors with measurable WER improvement.
  • Secret Mission: Implement RAG-enhanced error correction using a TF-IDF retrieval knowledge base for few-shot prompting.

Are there any prerequisites?

This project assumes Python proficiency and familiarity with ML/NLP fundamentals. You should be comfortable with the Hugging Face ecosystem (loading models, tokenizers, and datasets).

You will need a Modal account and the Modal CLI set up on your machine. Step 1 walks you through this.

Before We Start

This project builds an end-to-end speech recognition pipeline for Swahili. The pipeline combines Whisper for initial transcription with an African-language-optimized LLM for error correction, all running on serverless GPUs.

Before writing any pipeline code, you need the right infrastructure. In this step, you will install the Modal CLI for serverless GPU deployment, authenticate your account, and scaffold the project directory with placeholder modules.

In this step, get ready to:
  • Install and authenticate the Modal CLI.
  • Create the project directory with placeholder modules.
  • Install local Python dependencies for data handling and evaluation.
Install and authenticate the Modal CLI

Modal is a serverless GPU platform that lets you run Python functions on cloud GPUs without managing any infrastructure. You define your compute requirements in code, and Modal handles provisioning, scaling, and teardown.

  • Install the Modal package by running this command:
pip install modal
  • Authenticate the Modal CLI by running:
modal setup

What happens when you run this?

Running modal setup opens your browser for GitHub OAuth authentication. You will sign in with your GitHub account to create a Modal workspace.

Modal only requests read access to your email and organization memberships. It does not access any code repositories.

  • Verify the installation worked by running:
modal run --help

You should see usage information listing options like --name, --quiet, --detach, and --help.

Command not found?

If modal is not recognized, your Python scripts directory may not be on your PATH. Try running python -m modal run --help instead, or add your Python bin directory to PATH.

If the browser did not open during modal setup, check that you have a default browser configured on your system.

help me troubleshoot the Modal CLI installation

Create the project directory and install dependencies

The pipeline has four distinct concerns: ASR transcription, LLM correction, orchestration, and evaluation. Each gets its own module file so the codebase stays organized as complexity grows.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create the project folder, move into it, and create the placeholder module files by running these commands:
mkdir swahili-asr-pipeline
cd swahili-asr-pipeline
touch asr_module.py llm_correction.py pipeline.py evaluation.py
  • Confirm the files exist by running:
ls

You should see asr_module.py, evaluation.py, llm_correction.py, and pipeline.py in the output.

What are these files for?

  • asr_module.py will hold the Whisper-based transcription logic on Modal GPUs.
  • llm_correction.py will hold the LLM-based generative error correction module.
  • pipeline.py will orchestrate the end-to-end flow from audio input to corrected transcription.
  • evaluation.py will compute Word Error Rate metrics to measure improvement.
  • Install the local Python packages for data loading and evaluation by running:
pip install datasets jiwer

What do these packages do?

datasets is the Hugging Face library for streaming audio datasets directly from the hub without manual downloads.

jiwer computes Word Error Rate (WER), the standard metric for evaluating speech recognition quality.

You should see a success message once both packages finish installing.

Installation errors?

If you see permission errors, try running the command with --user appended, or activate a virtual environment first.

help me fix pip install errors for datasets and jiwer

Your environment is ready. Next up, you will build the ASR module that deploys Whisper on a Modal T4 GPU and generates multiple transcription hypotheses via temperature sampling.

Set Up Modal Infrastructure and Secrets

Your ASR and LLM modules will need to download large model weights from Hugging Face. Whisper large-v3-turbo is around 3GB, and AfriqueLlama-8B is around 16GB.

To avoid re-downloading these weights every time a container spins up, you will create persistent Modal Volumes for caching. You will also store your Hugging Face API token as a Modal Secret so your containers can authenticate with the Hugging Face Hub.

In this step, get ready to:
  • Create a Hugging Face API token and store it as a Modal Secret.
  • Create Modal Volumes for caching Whisper and vLLM model weights.
  • Verify the infrastructure is ready for Modal app definitions.
Create a Hugging Face access token

Both Whisper and AfriqueLlama-8B are hosted on Hugging Face. To download their weights inside Modal containers, you need an access token stored as a Modal Secret.

  • Open https://huggingface.co/settings/tokens in your browser.
  • Click Create new token.
  • Give it a name (e.g. modal-asr-pipeline) and select Read access.
  • Click Create token and copy the token value.

Why a read-only token?

You only need to download model weights from Hugging Face. A read-only token keeps your account secure by preventing any writes or changes to your repositories.

Now store this token as a Modal Secret so your containers can access it at runtime.

  • Create the Modal Secret by running this command:
modal secret create huggingface-secret HF_TOKEN=your-hf-token-here

What does this command do?

This creates an encrypted secret named huggingface-secret in your Modal workspace. The key HF_TOKEN will be injected as an environment variable into any Modal container that requests this secret.

  • Verify the secret was created by running:
modal secret list

You should see huggingface-secret in the output list with a recent timestamp.

Don't see your secret?

Make sure you replaced your-hf-token-here with the actual token you copied from Hugging Face. If you accidentally ran the command with the placeholder text, delete it and recreate it:

modal secret delete huggingface-secret
modal secret create huggingface-secret HF_TOKEN=your-actual-token

Why isn't my Modal secret showing up?

Create Modal Volumes for model weight caching

Modal Volumes are persistent storage that survives container restarts. Without them, every container cold start would re-download gigabytes of model weights.

  • Create the volume for Whisper model weights by running this command:
modal volume create huggingface-cache

You should see output confirming the volume was created: Created volume 'huggingface-cache' in environment 'main'.

  • Create a second volume for vLLM model weights by running this command:
modal volume create vllm-cache

You should see: Created volume 'vllm-cache' in environment 'main'.

Why two separate volumes?

The ASR module (Whisper) and the LLM correction module (AfriqueLlama via vLLM) run in separate containers with different GPU configurations. Separate volumes let each container mount only the weights it needs, keeping startup fast.

  • Confirm both volumes exist by running:
modal volume list

You should see both huggingface-cache and vllm-cache in the output.

Seeing a volume already exists error?

If you ran the command before and the volume already exists, the error is harmless. Run modal volume list to confirm the volume is present and continue.

Why is modal volume create failing?

Verify the project structure is ready

Your project directory already has the placeholder modules from the previous step. Confirm everything is in place before you start writing Modal app definitions.

  • Run this command from inside your project directory:
ls

You should see asr_module.py, llm_correction.py, pipeline.py, and evaluation.py listed in the output.

Quick infrastructure recap

You now have three pieces of Modal infrastructure ready:

  • A secret (huggingface-secret) for authenticating Hugging Face downloads.
  • A volume (huggingface-cache) for caching Whisper weights.
  • A volume (vllm-cache) for caching AfriqueLlama-8B weights.

Your Modal infrastructure is ready to go. Next up, you will build the ASR module that loads Whisper onto a T4 GPU and generates multiple transcription hypotheses from Swahili audio.

Build the ASR Module with Whisper

Your pipeline needs a way to convert Swahili audio into text. Whisper large-v3-turbo is OpenAI's multilingual speech recognition model, and it handles Swahili reasonably well out of the box.

The challenge is that a single transcription pass often produces errors, especially for low-resource languages. To give the LLM correction module useful material to work with, you will generate multiple candidate transcriptions (N-best hypotheses) by running Whisper at different temperatures.

In this step, get ready to:
  • Define a Modal image with Whisper dependencies and load the model on a T4 GPU.
  • Implement N-best hypothesis generation using temperature-based sampling.
  • Test the ASR module with a live Swahili audio sample.
Define the Modal image and load Whisper on a T4 GPU

Modal containers start from a base image that you configure with your Python dependencies. You will define a Debian slim image with the libraries Whisper needs, then create a class that loads the model once per container startup using the @modal.enter() lifecycle hook.

  • Open asr_module.py in your editor and add the module-level setup by pasting the following code:
import modal
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
import numpy as np

# Define the Modal app
app = modal.App('swahili-asr-pipeline')

# Build a container image with ASR dependencies
asr_image = (
    modal.Image.debian_slim(python_version='3.12')
    .pip_install('transformers', 'torch', 'librosa', 'soundfile', 'accelerate')
)

# Reference the persistent volumes for caching model weights
huggingface_cache = modal.Volume.from_name('huggingface-cache')
vllm_cache = modal.Volume.from_name('vllm-cache')

What does this code do?

This sets up the foundation for your Modal app. modal.App('swahili-asr-pipeline') creates a named app that Modal uses to track your deployed functions.

The asr_image builds a container environment with Python 3.12 and the packages Whisper needs: transformers for the model interface, torch for GPU inference, and librosa for audio processing.

The two Volume.from_name calls connect to the persistent volumes you created in Step 1. These cache downloaded model weights so subsequent runs skip the download.

  • Below the module setup, add the WhisperASR class with its model loading logic:
@app.cls(
    image=asr_image,
    gpu='T4',
    volumes={'/cache': huggingface_cache},
    secrets=[modal.Secret.from_name('huggingface-secret')],
)
class WhisperASR:
    @modal.enter()
    def load_model(self):
        import os
        model_id = 'openai/whisper-large-v3-turbo'
        self.device = 'cuda' if torch.cuda.is_available() else 'cpu'
        self.torch_dtype = torch.float16
        # Load the Whisper model and cache weights to the volume
        self.model = AutoModelForSpeechSeq2Seq.from_pretrained(
            model_id,
            torch_dtype=self.torch_dtype,
            cache_dir='/cache',
            token=os.environ['HF_TOKEN'],
        ).to(self.device)
        # Load the processor for tokenization and feature extraction
        self.processor = AutoProcessor.from_pretrained(
            model_id,
            cache_dir='/cache',
            token=os.environ['HF_TOKEN'],
        )

What does this code do?

The @app.cls decorator registers this class with Modal. It specifies a single T4 GPU (16 GB VRAM), mounts your cache volume at /cache, and injects your Hugging Face token as an environment variable.

The @modal.enter() method runs exactly once when a container spins up. It loads openai/whisper-large-v3-turbo (~3 GB in FP16) onto the GPU and saves the weights to /cache so future containers skip the download entirely.

Why use @modal.cls instead of @app.function?

Loading a ~3 GB model is expensive. With a plain function, Modal would reload the model on every invocation. The class pattern with @modal.enter() loads it once per container and reuses it across multiple calls. This is the standard Modal pattern for GPU inference workloads.

  • Save asr_module.py.

Seeing import errors in your editor?

The imports at the top (torch, transformers) are only needed inside the Modal container, not on your local machine. Your editor may show unresolved import warnings. These are safe to ignore because asr_image installs them in the remote container.

Help me handle import warnings in my Modal file.

Implement N-best hypothesis generation with temperature sampling

Beam search is the typical decoding strategy for ASR, but Hugging Face Transformers has a known issue where num_return_sequences does not produce distinct hypotheses for Whisper. The reliable workaround is to run separate decode passes at different temperatures.

  • Add the transcribe_nbest method inside the WhisperASR class, below the load_model method:
    @modal.method()
    def transcribe_nbest(self, audio_array: list, sampling_rate: int = 16000):
        temperatures = [0.0, 0.2, 0.4, 0.6, 0.8]
        hypotheses = []
        # Convert the audio list back to a numpy array
        audio_np = np.array(audio_array, dtype=np.float32)
        # Preprocess the audio into model input features
        inputs = self.processor(audio_np, sampling_rate=sampling_rate, return_tensors='pt')
        input_features = inputs.input_features.to(self.device, dtype=self.torch_dtype)
        # Run one decode pass per temperature for diverse hypotheses
        for temp in temperatures:
            with torch.no_grad():
                output = self.model.generate(
                    input_features,
                    language='sw',
                    task='transcribe',
                    temperature=temp if temp > 0 else None,
                    do_sample=temp > 0,
                    return_dict_in_generate=True,
                    output_scores=True,
                )
            # Decode token IDs back to text
            text = self.processor.batch_decode(output.sequences, skip_special_tokens=True)[0].strip()
            # Compute average log probability as a confidence signal
            scores = output.scores
            avg_logprob = float(torch.stack([s.max(dim=-1).values for s in scores]).mean()) if scores else 0.0
            hypotheses.append({'text': text, 'avg_logprob': avg_logprob, 'temperature': temp})
        return hypotheses

What does this code do?

For each audio input, this method runs Whisper 5 times at temperatures [0.0, 0.2, 0.4, 0.6, 0.8]. Temperature 0.0 gives the greedy (most likely) transcription. Higher temperatures introduce randomness, producing alternative decodings that may capture words the greedy pass missed.

The language='sw' parameter forces Whisper to decode as Swahili. The avg_logprob score gives the LLM correction module a confidence signal for each hypothesis.

  • Save asr_module.py.

Indentation error?

Make sure the transcribe_nbest method is indented one level inside the WhisperASR class (4 spaces from the class body). It should be at the same indentation level as load_model.

Help me fix my class indentation.

Test the ASR module with a Swahili audio sample

Before wiring the full pipeline, you should confirm that Whisper produces distinct hypotheses on real Swahili audio. You will stream a single test utterance from the cdli/common_voice_swahili_small dataset and send it to the remote WhisperASR class.

  • Add a temporary test entrypoint at the bottom of asr_module.py, below the WhisperASR class:
@app.local_entrypoint()
def test_asr():
    from datasets import load_dataset
    # Stream a single test utterance from the Swahili dataset
    ds = load_dataset('cdli/common_voice_swahili_small', split='test', streaming=True)
    sample = next(iter(ds))
    audio = sample['audio']['array'].tolist()
    sr = sample['audio']['sampling_rate']
    # Call the remote Whisper class method
    whisper = WhisperASR()
    results = whisper.transcribe_nbest.remote(audio, sr)
    for h in results:
        print(f"[temp={h['temperature']}] (logprob={h['avg_logprob']:.3f}) {h['text']}")

What does this code do?

The @app.local_entrypoint() decorator marks a function that runs on your local machine. It streams one audio sample from the dataset (no full download needed), converts it to a list, then calls transcribe_nbest.remote() which executes on Modal's T4 GPU.

  • Save asr_module.py and run the test by executing this command:
modal run asr_module.py

What should I see?

Modal will build the container image on first run (this takes a few minutes as it installs PyTorch). After the container starts, you will see 5 lines of output, one per temperature.

The temperature 0.0 line shows the greedy transcription. Higher temperatures should show slight variations in the text. Example output (your text will differ based on the audio sample):

[temp=0.0] (logprob=-0.142) Habari za asubuhi
[temp=0.2] (logprob=-0.198) Habari za asubuhi
[temp=0.4] (logprob=-0.267) Habari za subuhi
[temp=0.6] (logprob=-0.345) Habari za asubuhi hii
[temp=0.8] (logprob=-0.512) Habari za asubuhi hi

Seeing a timeout or image build failure?

The first run builds the container image, which installs PyTorch (~2 GB). This can take 3-5 minutes. If it times out, run the command again. Modal caches the built image so subsequent runs start immediately.

If you see an authentication error related to HF_TOKEN, confirm your Modal Secret named huggingface-secret has a key called HF_TOKEN set to a valid Hugging Face access token.

Help me troubleshoot my Modal run failure.

  • Once you see 5 hypotheses printed, remove the test_asr function from the bottom of asr_module.py. You will not need it going forward because the pipeline orchestrator will call WhisperASR directly.
  • Save asr_module.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved asr_module.py with the test function removed.

ⓧ I'd like to double check the full code

Here is the complete asr_module.py after removing the test entrypoint:

import modal
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
import numpy as np

app = modal.App('swahili-asr-pipeline')

asr_image = (
    modal.Image.debian_slim(python_version='3.12')
    .pip_install('transformers', 'torch', 'librosa', 'soundfile', 'accelerate')
)

huggingface_cache = modal.Volume.from_name('huggingface-cache')
vllm_cache = modal.Volume.from_name('vllm-cache')

@app.cls(
    image=asr_image,
    gpu='T4',
    volumes={'/cache': huggingface_cache},
    secrets=[modal.Secret.from_name('huggingface-secret')],
)
class WhisperASR:
    @modal.enter()
    def load_model(self):
        import os
        model_id = 'openai/whisper-large-v3-turbo'
        self.device = 'cuda' if torch.cuda.is_available() else 'cpu'
        self.torch_dtype = torch.float16
        self.model = AutoModelForSpeechSeq2Seq.from_pretrained(
            model_id,
            torch_dtype=self.torch_dtype,
            cache_dir='/cache',
            token=os.environ['HF_TOKEN'],
        ).to(self.device)
        self.processor = AutoProcessor.from_pretrained(
            model_id,
            cache_dir='/cache',
            token=os.environ['HF_TOKEN'],
        )

    @modal.method()
    def transcribe_nbest(self, audio_array: list, sampling_rate: int = 16000):
        temperatures = [0.0, 0.2, 0.4, 0.6, 0.8]
        hypotheses = []
        audio_np = np.array(audio_array, dtype=np.float32)
        inputs = self.processor(audio_np, sampling_rate=sampling_rate, return_tensors='pt')
        input_features = inputs.input_features.to(self.device, dtype=self.torch_dtype)
        for temp in temperatures:
            with torch.no_grad():
                output = self.model.generate(
                    input_features,
                    language='sw',
                    task='transcribe',
                    temperature=temp if temp > 0 else None,
                    do_sample=temp > 0,
                    return_dict_in_generate=True,
                    output_scores=True,
                )
            text = self.processor.batch_decode(output.sequences, skip_special_tokens=True)[0].strip()
            scores = output.scores
            avg_logprob = float(torch.stack([s.max(dim=-1).values for s in scores]).mean()) if scores else 0.0
            hypotheses.append({'text': text, 'avg_logprob': avg_logprob, 'temperature': temp})
        return hypotheses

Your ASR module is working. It takes raw Swahili audio and produces 5 candidate transcriptions ranked by confidence. Next up, you will build the LLM correction module that reads these hypotheses and produces a single, more accurate transcription.

Build the LLM Generative Error Correction Module

Your ASR module now generates multiple transcription candidates for each Swahili utterance. Raw ASR output inevitably contains errors, especially for low-resource languages where training data is limited.

In this step, you will deploy an African-language-optimized LLM that reads those candidate transcriptions and produces a corrected version. This is the core of the Generative Error Correction (GER) paradigm.

In this step, get ready to:
  • Deploy AfriqueLlama-8B on 2x T4 GPUs with tensor parallelism via vLLM.
  • Implement a Task-Activating Prompt that presents N-best hypotheses to the LLM.
  • Test the correction module with manually-crafted erroneous Swahili hypotheses.
Deploy AfriqueLlama-8B with vLLM on 2x T4 GPUs

AfriqueLlama-8B is an 8-billion parameter model (~16GB in FP16) built on Llama 3.1-8B and further trained on 26B tokens covering 20 African languages including Swahili. A single T4 only has 16GB of VRAM, so you need two T4s working together via tensor parallelism to fit the model in memory.

Why AfriqueLlama-8B?

AfriqueLlama-8B is specifically adapted for African languages with demonstrated improvements on Swahili tasks. Alternatives like Qwen2.5 list Swahili as supported but lack Swahili-specific fine-tuning. The CC BY 4.0 license also places no restrictions on usage.

  • Open llm_correction.py in your editor. This file is currently empty.
  • Add the imports, app reference, image definition, volume setup, and the AfriqueLlamaGER class with its model-loading method by pasting the following code:
import modal
import os

app = modal.App('swahili-asr-pipeline')

# Define a container image with vLLM for LLM inference
llm_image = (
    modal.Image.debian_slim(python_version='3.12')
    .pip_install('vllm', 'torch')
)

# Reference the volume for caching AfriqueLlama-8B weights
vllm_cache = modal.Volume.from_name('vllm-cache')

@app.cls(
    image=llm_image,
    gpu=modal.gpu.T4(count=2),
    volumes={'/vllm-cache': vllm_cache},
    secrets=[modal.Secret.from_name('huggingface-secret')],
    timeout=600,
)
class AfriqueLlamaGER:
    @modal.enter()
    def load_model(self):
        from vllm import LLM, SamplingParams
        # Load AfriqueLlama-8B with tensor parallelism across 2 GPUs
        self.llm = LLM(
            model='McGill-NLP/AfriqueLlama-8B',
            tensor_parallel_size=2,
            download_dir='/vllm-cache',
            dtype='float16',
            token=os.environ['HF_TOKEN'],
        )
        # Low temperature for deterministic corrections
        self.sampling_params = SamplingParams(temperature=0.1, max_tokens=256)

What does this code do?

The llm_image defines a container with vLLM installed. vLLM is a high-throughput inference engine that handles tensor parallelism automatically.

The @app.cls decorator requests 2x T4 GPUs via modal.gpu.T4(count=2). This gives the container 32GB of combined VRAM. The tensor_parallel_size=2 parameter tells vLLM to split the model's weight matrices across both GPUs.

The download_dir='/vllm-cache' ensures model weights persist in the Modal Volume between container restarts, so you only download the ~16GB model once.

  • Save llm_correction.py.

You will not see a change yet. The next chunk adds the prompt construction logic.

Seeing an import error for modal?

Make sure you installed Modal in Step 1 with pip install modal. Also confirm your virtual environment is active if you are using one.

help me debug a Modal import error in my llm_correction.py

Implement the Task-Activating Prompt for GER

Task-Activating Prompting (TAP) is a technique that provides the LLM with structured context about the task, the input data, and the expected output format. For GER, the prompt presents the N-best hypotheses along with their confidence scores and instructs the model to reason about Swahili phonetics and morphology.

  • In llm_correction.py, add the _build_tap_prompt method inside the AfriqueLlamaGER class, below the load_model method:
    def _build_tap_prompt(self, hypotheses: list) -> str:
        # Format each hypothesis with its index and confidence score
        hyp_lines = '\n'.join(
            [f'  Hypothesis {i+1} (confidence: {h["avg_logprob"]:.3f}): {h["text"]}'
             for i, h in enumerate(hypotheses)]
        )
        # Construct the Task-Activating Prompt
        prompt = (
            'You are an expert in Swahili language and speech recognition error correction.\n'
            'Below are N-best transcription hypotheses from an automatic speech recognition system for a Swahili utterance.\n'
            'The hypotheses may contain phonetic confusions, morphological errors, or named entity mistakes common in Swahili ASR.\n'
            'Using your knowledge of Swahili phonology, morphology, and context, produce the single most accurate Swahili transcription.\n'
            'Output only the corrected transcription with no explanation.\n\n'
            f'N-best hypotheses:\n{hyp_lines}\n\n'
            'Corrected Swahili transcription:'
        )
        return prompt

What does this prompt do?

The prompt activates the LLM's language knowledge by explicitly naming the task (Swahili ASR error correction) and the error types to watch for (phonetic confusions, morphological errors, named entities).

Each hypothesis is presented with its confidence score (avg_logprob) so the model can weigh higher-confidence candidates more heavily. The final instruction constrains the output to just the corrected text with no explanations.

  • Save llm_correction.py.

You will not see a change yet. The next chunk adds the method that calls the LLM and runs the test.

Indentation errors?

The _build_tap_prompt method must be indented at the same level as load_model (one level inside the class). Python uses indentation to determine class membership.

help me fix indentation in my AfriqueLlamaGER class

Add the correction method and test with sample hypotheses

The final piece connects the prompt builder to the vLLM inference engine. You will also add a temporary test entrypoint to verify the module produces corrections.

  • In llm_correction.py, add the correct_transcription method below _build_tap_prompt:
    @modal.method()
    def correct_transcription(self, hypotheses: list) -> str:
        # Build the TAP prompt and run LLM inference
        prompt = self._build_tap_prompt(hypotheses)
        outputs = self.llm.generate([prompt], self.sampling_params)
        # Extract the generated text
        corrected = outputs[0].outputs[0].text.strip()
        return corrected

What does this method do?

The @modal.method() decorator exposes correct_transcription as a remotely callable function. It builds the TAP prompt from the hypotheses list, passes it to vLLM's generate method, and returns the cleaned output string.

  • Now add a temporary test entrypoint at the bottom of the file (outside the class) to verify the module works:
@app.local_entrypoint()
def test_correction():
    # Manually-crafted erroneous Swahili hypotheses simulating ASR mistakes
    test_hypotheses = [
        {'text': 'Habari yako leo ni nzuri sana', 'avg_logprob': -0.15, 'temperature': 0.0},
        {'text': 'Habari yako leo ni nzuli sana', 'avg_logprob': -0.32, 'temperature': 0.2},
        {'text': 'Habali yako leo ni nzuri sana', 'avg_logprob': -0.41, 'temperature': 0.4},
        {'text': 'Habari yako leo ni nzuli thana', 'avg_logprob': -0.55, 'temperature': 0.6},
        {'text': 'Habali yako leo ni nzuli sana', 'avg_logprob': -0.62, 'temperature': 0.8},
    ]
    ger = AfriqueLlamaGER()
    result = ger.correct_transcription.remote(test_hypotheses)
    print(f'Input hypotheses (best): {test_hypotheses[0]["text"]}')
    print(f'LLM corrected output: {result}')

What are these test hypotheses?

The test data simulates common Swahili ASR errors. "nzuli" is a phonetic confusion of "nzuri" (good). "Habali" confuses the /r/ and /l/ phonemes in "Habari" (news/hello). "thana" is a phonetic distortion of "sana" (very). These are typical error patterns in speech recognition for Bantu languages.

  • Save llm_correction.py.
  • Run the test by executing this command:
modal run llm_correction.py

This will take a few minutes on first run

The first execution downloads AfriqueLlama-8B (~16GB) to the Modal Volume. Subsequent runs reuse the cached weights and start much faster. You will see vLLM initialization logs followed by the corrected transcription output.

You should see output showing the corrected Swahili transcription. The LLM should produce something close to "Habari yako leo ni nzuri sana" by resolving the phonetic confusions in the lower-confidence hypotheses.

Seeing a timeout or OOM error?

The 600-second timeout in the class decorator gives the container time to download and load the model. If you see a timeout, the download may be slow. Run the command again, as the partial download is cached in the volume.

If you see an out-of-memory error, confirm that your code uses modal.gpu.T4(count=2) and tensor_parallel_size=2. Both T4s must be active for the model to fit.

help me debug an OOM or timeout error when running AfriqueLlama-8B on Modal

  • Once you confirm the test produces a corrected transcription, remove the test_correction function and the @app.local_entrypoint() decorator from the bottom of llm_correction.py. The pipeline orchestrator in the next step will call this module directly.
  • Save llm_correction.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved llm_correction.py with the test entrypoint removed.

ⓧ I'd like to double check the full code

Here is the complete llm_correction.py file after removing the test entrypoint:

import modal
import os

app = modal.App('swahili-asr-pipeline')

llm_image = (
    modal.Image.debian_slim(python_version='3.12')
    .pip_install('vllm', 'torch')
)

vllm_cache = modal.Volume.from_name('vllm-cache')

@app.cls(
    image=llm_image,
    gpu=modal.gpu.T4(count=2),
    volumes={'/vllm-cache': vllm_cache},
    secrets=[modal.Secret.from_name('huggingface-secret')],
    timeout=600,
)
class AfriqueLlamaGER:
    @modal.enter()
    def load_model(self):
        from vllm import LLM, SamplingParams
        self.llm = LLM(
            model='McGill-NLP/AfriqueLlama-8B',
            tensor_parallel_size=2,
            download_dir='/vllm-cache',
            dtype='float16',
            token=os.environ['HF_TOKEN'],
        )
        self.sampling_params = SamplingParams(temperature=0.1, max_tokens=256)

    def _build_tap_prompt(self, hypotheses: list) -> str:
        hyp_lines = '\n'.join(
            [f'  Hypothesis {i+1} (confidence: {h["avg_logprob"]:.3f}): {h["text"]}'
             for i, h in enumerate(hypotheses)]
        )
        prompt = (
            'You are an expert in Swahili language and speech recognition error correction.\n'
            'Below are N-best transcription hypotheses from an automatic speech recognition system for a Swahili utterance.\n'
            'The hypotheses may contain phonetic confusions, morphological errors, or named entity mistakes common in Swahili ASR.\n'
            'Using your knowledge of Swahili phonology, morphology, and context, produce the single most accurate Swahili transcription.\n'
            'Output only the corrected transcription with no explanation.\n\n'
            f'N-best hypotheses:\n{hyp_lines}\n\n'
            'Corrected Swahili transcription:'
        )
        return prompt

    @modal.method()
    def correct_transcription(self, hypotheses: list) -> str:
        prompt = self._build_tap_prompt(hypotheses)
        outputs = self.llm.generate([prompt], self.sampling_params)
        corrected = outputs[0].outputs[0].text.strip()
        return corrected

Your LLM correction module is working. Next up, you will wire both modules together into a full pipeline and run batch evaluation on 397 Swahili test utterances.

Wire the Pipeline and Run Batch Evaluation

You have a working ASR module and an LLM correction module. Now it is time to connect them into a single pipeline that processes real Swahili audio end-to-end.

In this step, you will build the orchestrator that streams the Common Voice Swahili test split through both modules, then measure the Word Error Rate across three conditions to quantify the improvement.

In this step, get ready to:
  • Build a pipeline orchestrator that feeds audio through ASR then LLM correction.
  • Implement WER evaluation comparing greedy, best-of-5, and LLM-corrected transcriptions.
  • Run the full batch evaluation and produce an error analysis.
Build the pipeline orchestrator

The pipeline orchestrator is a single Modal function that loads audio data, calls your two GPU-backed modules in sequence, and collects results for evaluation.

  • Open pipeline.py in your editor and replace its contents with the following imports and image definition:
import modal
from datasets import load_dataset
from asr_module import WhisperASR
from llm_correction import AfriqueLlamaGER
from evaluation import compute_wer

app = modal.App('swahili-asr-pipeline')

# Image for the orchestrator - needs dataset loading and evaluation libraries
pipeline_image = (
    modal.Image.debian_slim(python_version='3.12')
    .pip_install('datasets', 'jiwer', 'soundfile')
)

What does this code do?

The orchestrator imports your two GPU modules (WhisperASR and AfriqueLlamaGER) plus the compute_wer function you will write next.

The pipeline_image only needs lightweight packages since it delegates GPU work to the ASR and LLM containers via .remote() calls.

  • Below the image definition in pipeline.py, add the run_pipeline function:
@app.function(image=pipeline_image, timeout=3600)
def run_pipeline():
    # Load the Common Voice Swahili test split (397 utterances)
    dataset = load_dataset('cdli/common_voice_swahili_small', split='test', streaming=False)
    asr = WhisperASR()
    ger = AfriqueLlamaGER()
    results = []
    for sample in dataset:
        audio = sample['audio']
        reference = sample['sentence']
        # Generate 5 hypotheses via temperature sampling
        hypotheses = asr.transcribe_nbest.remote(audio['array'].tolist(), audio['sampling_rate'])
        # Greedy = temperature 0.0, which is always the first hypothesis
        greedy = hypotheses[0]['text']
        # Best-of-5 = hypothesis with lowest avg_logprob (highest confidence)
        best_of_5 = min(hypotheses, key=lambda h: h['avg_logprob'])['text']
        # LLM-corrected = GER module output
        corrected = ger.correct_transcription.remote(hypotheses)
        results.append({
            'reference': reference,
            'greedy': greedy,
            'best_of_5': best_of_5,
            'corrected': corrected,
            'hypotheses': hypotheses,
        })
    # Compute WER across all conditions
    wer_results = compute_wer(results)
    print(wer_results)
    return results, wer_results

What does this code do?

  • load_dataset('cdli/common_voice_swahili_small', split='test') streams the 397-utterance test set with audio arrays and reference sentences.
  • For each sample, asr.transcribe_nbest.remote() sends the audio to a T4 GPU container running Whisper.
  • ger.correct_transcription.remote() sends the N-best hypotheses to the 2x T4 container running AfriqueLlama-8B.
  • The three conditions (greedy, best-of-5, corrected) let you isolate how much improvement comes from multi-hypothesis selection versus LLM correction.
  • Save pipeline.py.

Import errors when saving?

You will see an import error for evaluation if you try to run the pipeline before creating evaluation.py. That is expected. You will write the evaluation module next.

help me fix import errors in my pipeline

Implement WER evaluation with jiwer

WER (Word Error Rate) is the standard metric for ASR quality. It measures the minimum edit distance between a reference transcription and a hypothesis, normalized by the reference length.

  • Open evaluation.py and replace its contents with the following:
from jiwer import wer

def compute_wer(results: list) -> dict:
    # Extract all references and hypotheses into parallel lists
    references = [r['reference'] for r in results]
    greedy_hyps = [r['greedy'] for r in results]
    best5_hyps = [r['best_of_5'] for r in results]
    corrected_hyps = [r['corrected'] for r in results]
    # Compute aggregate WER for each condition
    wer_greedy = wer(references, greedy_hyps)
    wer_best5 = wer(references, best5_hyps)
    wer_corrected = wer(references, corrected_hyps)
    # Compute per-utterance WER for error analysis
    per_utterance = []
    for r in results:
        per_utterance.append({
            'reference': r['reference'],
            'wer_greedy': wer([r['reference']], [r['greedy']]),
            'wer_best5': wer([r['reference']], [r['best_of_5']]),
            'wer_corrected': wer([r['reference']], [r['corrected']]),
        })
    return {
        'aggregate': {
            'wer_greedy': wer_greedy,
            'wer_best5': wer_best5,
            'wer_corrected': wer_corrected,
        },
        'per_utterance': per_utterance,
    }

What does this code do?

  • jiwer's wer(reference, hypothesis) accepts either single strings or lists of strings. Passing lists computes a corpus-level WER across all utterances.
  • wer_greedy measures the baseline: Whisper at temperature 0.0 (deterministic decoding).
  • wer_best5 measures the oracle best among 5 temperature-sampled hypotheses. This shows how much headroom exists from diversity.
  • wer_corrected measures the LLM's output after seeing all 5 hypotheses. This is the GER result.
  • Per-utterance scores let you identify which individual utterances improved or degraded under LLM correction.
  • Save evaluation.py.

ModuleNotFoundError for jiwer?

Make sure you installed jiwer locally in your virtual environment. Run pip install jiwer in your project terminal. The pipeline_image already includes jiwer for the remote execution.

help me resolve a jiwer import error

Run the pipeline and analyze errors

With both files complete, you can now execute the full 397-utterance evaluation on Modal's GPU infrastructure.

  • Launch the batch evaluation by running this command:
modal run pipeline.py::run_pipeline

What does this command do?

modal run pipeline.py::run_pipeline creates an ephemeral Modal app, spins up the orchestrator container, which then calls out to the ASR and LLM containers via .remote(). The timeout=3600 gives the pipeline up to one hour to process all 397 utterances.

Expect a long run time

Processing 397 utterances through Whisper (5 passes each) plus AfriqueLlama correction takes 20-40 minutes depending on GPU cold-start times. Watch the Modal dashboard for real-time container logs.

  • Once the run completes, examine the printed aggregate WER results in your terminal output.
  • Compare the three values: wer_greedy (baseline), wer_best5 (oracle multi-hypothesis), and wer_corrected (LLM GER output).

You should see wer_corrected lower than wer_greedy, confirming the LLM improves transcription accuracy.

How do I interpret the error analysis?

Look through the per_utterance results for patterns. Successful LLM corrections typically fall into three categories:

  • Named entities: Whisper misspells proper nouns that the LLM recognizes from training data.
  • Morphological errors: Swahili agglutinative prefixes/suffixes that Whisper splits incorrectly.
  • Phonetic confusions: Similar-sounding phonemes that Whisper conflates but the LLM disambiguates from context.

Failure modes occur when the LLM introduces hallucinations (adding words not in the audio) or overcorrects (replacing a correct word with a more common alternative).

Pipeline times out or containers fail to start?

If you see timeout errors, check that your Modal Volumes still have cached model weights. Cold-starting both Whisper and AfriqueLlama from scratch within the orchestrator timeout can fail. Run each module individually first to warm the caches.

help me debug Modal pipeline timeout errors

✔️ Awesome, I've got everything!

Great. Double check you have saved both pipeline.py and evaluation.py.

ⓧ I'd like to double check the full code

Here is the complete pipeline.py:

import modal
from datasets import load_dataset
from asr_module import WhisperASR
from llm_correction import AfriqueLlamaGER
from evaluation import compute_wer

app = modal.App('swahili-asr-pipeline')

pipeline_image = (
    modal.Image.debian_slim(python_version='3.12')
    .pip_install('datasets', 'jiwer', 'soundfile')
)

@app.function(image=pipeline_image, timeout=3600)
def run_pipeline():
    dataset = load_dataset('cdli/common_voice_swahili_small', split='test', streaming=False)
    asr = WhisperASR()
    ger = AfriqueLlamaGER()
    results = []
    for sample in dataset:
        audio = sample['audio']
        reference = sample['sentence']
        hypotheses = asr.transcribe_nbest.remote(audio['array'].tolist(), audio['sampling_rate'])
        greedy = hypotheses[0]['text']
        best_of_5 = min(hypotheses, key=lambda h: h['avg_logprob'])['text']
        corrected = ger.correct_transcription.remote(hypotheses)
        results.append({
            'reference': reference,
            'greedy': greedy,
            'best_of_5': best_of_5,
            'corrected': corrected,
            'hypotheses': hypotheses,
        })
    wer_results = compute_wer(results)
    print(wer_results)
    return results, wer_results

Here is the complete evaluation.py:

from jiwer import wer

def compute_wer(results: list) -> dict:
    references = [r['reference'] for r in results]
    greedy_hyps = [r['greedy'] for r in results]
    best5_hyps = [r['best_of_5'] for r in results]
    corrected_hyps = [r['corrected'] for r in results]
    wer_greedy = wer(references, greedy_hyps)
    wer_best5 = wer(references, best5_hyps)
    wer_corrected = wer(references, corrected_hyps)
    per_utterance = []
    for r in results:
        per_utterance.append({
            'reference': r['reference'],
            'wer_greedy': wer([r['reference']], [r['greedy']]),
            'wer_best5': wer([r['reference']], [r['best_of_5']]),
            'wer_corrected': wer([r['reference']], [r['corrected']]),
        })
    return {
        'aggregate': {
            'wer_greedy': wer_greedy,
            'wer_best5': wer_best5,
            'wer_corrected': wer_corrected,
        },
        'per_utterance': per_utterance,
    }

You now have quantitative proof that cascading an African-language LLM after Whisper measurably reduces word error rates on Swahili audio. Next, take on the Secret Mission to push accuracy even further with retrieval-augmented correction.

Secret mission

RAG-Enhanced Generative Error Correction

Your standard GER pipeline corrects errors using only the LLM's internal knowledge. What if you could show the LLM real examples of ASR mistakes and their corrections before asking it to fix a new utterance? In this mission, you'll build a retrieval-augmented GER system that finds similar error patterns from a knowledge base and feeds them as few-shot examples to the LLM.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Modal scales to zero automatically when not receiving requests, so there are no ongoing costs while your functions are idle.

Resources you used:

  • Modal App swahili-asr-pipeline (ASR and LLM functions deployed on T4 GPUs).
  • Modal Volume huggingface-cache (cached Whisper large-v3-turbo weights).
  • Modal Volume vllm-cache (cached AfriqueLlama-8B weights).
  • Local project folder on your Desktop containing asr_module.py, llm_correction.py, pipeline.py, evaluation.py, and rag_ger.py.

Keep everything running

No action needed. Choose this if you are still actively building on this pipeline or plan to run more experiments soon.

  • Modal functions scale to zero automatically. You are not charged while containers are idle.
  • Modal Volumes persist your cached model weights so future cold starts are faster.
  • Your local project files remain on your Desktop for editing.

Pause - I'll come back to this later

Modal auto-scales to zero when functions are not receiving requests. There is nothing extra to pause.

  • Your deployed app, volumes, and local files all remain in place at no cost.
  • When you are ready to resume, run modal deploy again from your project folder to redeploy.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to fully clean up.

  • Stop the deployed Modal App by running:
modal app stop swahili-asr-pipeline
  • Delete the Whisper model cache volume by running:
modal volume rm huggingface-cache
  • Delete the vLLM model cache volume by running:
modal volume rm vllm-cache
  • Remove your local project folder on macOS or Linux by running:
rm -rf ~/Desktop/project
  • On Windows, remove the folder by running:
rmdir /s /q %USERPROFILE%\Desktop\project

That's a Wrap!

That's a Wrap!

Nice work. You just built a production-grade speech recognition pipeline that measurably improves Swahili transcription accuracy by cascading Whisper with an African-language-optimized LLM for generative error correction.

You've learned how to:

  • Deployed a Whisper large-v3-turbo ASR module on Modal T4 GPUs that generates N-best hypotheses via temperature-based sampling.
  • Built an LLM correction module using AfriqueLlama-8B with tensor parallelism across 2x T4 GPUs served via vLLM, applying Task-Activating Prompting to correct Swahili-specific ASR errors.
  • Measured WER improvement across 397 Swahili test utterances comparing baseline Whisper greedy, best-of-5, and LLM-corrected transcriptions with per-utterance error analysis.
  • Secret Mission: Added RAG-enhanced GER with a TF-IDF retrieval knowledge base for few-shot correction examples.

Ready to quiz yourself?