Noise-Conditioned Intent Recognition

Build a FiLM-conditioned model that adapts speech intent recognition per acoustic environment.

Introduction

30 Second Summary

Voice assistants fail in noisy kitchens and busy streets because they treat all background noise the same way. What if a model could detect WHERE it's listening and adapt its recognition strategy accordingly?

In this project, you will build a PyTorch model that detects the acoustic environment of a voice command and uses Feature-wise Linear Modulation (FiLM) to dynamically adapt its intent recognition strategy per environment.

What You'll Build

A multi-task CNN architecture that jointly classifies acoustic environment and recognizes speech intent, using FiLM conditioning to modulate feature representations based on detected noise type.

By the end of this project, you'll have:

  • A FiLM-conditioned CNN that dynamically adapts feature representations based on detected acoustic environment.
  • An on-the-fly noise mixing pipeline combining Fluent Speech Commands with ESC-50 environment sounds at variable SNRs.
  • Comparative evaluation results proving environment-adaptive conditioning outperforms an environment-agnostic baseline in noisy conditions.
  • Secret Mission: Extend the model with SNR-aware conditioning that jointly models noise type AND intensity for further gains at extreme noise levels.

Are there any prerequisites?

This project assumes strong PyTorch fluency, experience building multi-head architectures, and familiarity with audio mel-spectrograms. Everything runs in Google Colab, so no local GPU setup is needed.

Set Up Colab and Download Data

Your environment-adaptive model needs two things before it can learn: clean speech recordings to understand intent, and real-world noise samples to simulate different environments.

In this step, you'll set up a Google Colab notebook with GPU acceleration, then download both the Fluent Speech Commands dataset (30,043 utterances of voice commands) and the ESC-50 environmental sound collection.

In this step, get ready to:
  • Configure a Colab GPU runtime and verify PyTorch is available.
  • Download the Fluent Speech Commands and ESC-50 datasets.
  • Define the environment-to-noise category mapping for four acoustic environments.
Configure Colab for GPU and verify PyTorch

Deep learning on audio spectrograms is compute-intensive. A GPU accelerates the matrix operations that power your convolutional encoder, cutting training time from ~45 minutes on CPU to ~15 minutes.

  • Open Google Colab and create a new notebook.
  • Click Runtime in the top menu bar.
  • Select Change runtime type.
  • Under Hardware accelerator, select T4 GPU, then click Save.

What if no GPU is available?

Colab's free tier provides GPU access but availability is not guaranteed. If no GPU is available, the notebook will fall back to CPU. Training will take longer (~45 minutes instead of ~15) but still completes successfully with the 5000-sample subset used in this project.

  • In the first cell of your notebook, import PyTorch and verify GPU access by running this code:
# Environment-Adaptive SLU

import torch
import torchaudio
import torchaudio.datasets
import os

# Verify GPU
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f'Using device: {device}')
print(f'PyTorch version: {torch.__version__}')
print(f'torchaudio version: {torchaudio.__version__}')

What does this code do?

The torch.device line checks whether CUDA (NVIDIA's GPU computing platform) is available and sets the device accordingly. The print statements confirm which hardware and library versions you're working with.

  • Run the cell by pressing Shift+Enter.

You should see output showing Using device: cuda (or cpu if no GPU is assigned) along with the PyTorch and torchaudio version numbers.

Seeing 'cpu' instead of 'cuda'?

Double-check that you changed the runtime type to T4 GPU. Click Runtime then Change runtime type and confirm GPU is selected. If it already shows GPU but CUDA is unavailable, Colab may be at capacity. Try again in a few minutes or continue on CPU.

help me troubleshoot why my Colab notebook isn't detecting a GPU

Download Fluent Speech Commands and ESC-50

Fluent Speech Commands contains 30,043 utterances of spoken commands (like "turn on the lights" or "bring me my shoes") labeled with 31 unique intents. ESC-50 provides 2,000 five-second recordings of environmental sounds across 50 categories.

  • In a new cell, download both datasets by running this code:
# Download Fluent Speech Commands
fsc_dataset = torchaudio.datasets.FluentSpeechCommands(root='./data', subset='train', download=True)
fsc_val = torchaudio.datasets.FluentSpeechCommands(root='./data', subset='validation', download=True)

# Download ESC-50
import subprocess
subprocess.run(['wget', '-q', 'https://github.com/karoldvl/ESC-50/archive/master.zip', '-O', 'ESC-50.zip'])
subprocess.run(['unzip', '-q', 'ESC-50.zip'])
esc50_path = './ESC-50-master/audio'

What does this code do?

The FluentSpeechCommands class from torchaudio handles downloading and organizing the speech dataset automatically. The subset parameter selects which split to load (training or validation).

For ESC-50, you download the repository archive directly from GitHub and unzip it. The audio files end up in ./ESC-50-master/audio with filenames encoding the category ID.

  • Run the cell. This download takes 5-10 minutes depending on network speed (the FSC dataset is several GB, ESC-50 is ~600 MB).

You should see the cell complete without errors. The wget command runs quietly (-q flag) so you may not see progress output for the ESC-50 download.

Download stalling or timing out?

Colab sessions can lose connectivity during long downloads. If the cell seems stuck for more than 15 minutes, interrupt and re-run it. The FluentSpeechCommands downloader resumes from where it left off if partial files exist.

help me fix a stalled dataset download in Google Colab

Define the environment-to-noise category mapping

Your model simulates four acoustic environments by mixing speech with specific ESC-50 sound categories. Each environment maps to noise types that share spectral characteristics with real-world locations.

Why these specific categories?

Kitchen sounds (washing machine, can opening, glass breaking) occupy mid-to-high frequency ranges. Car sounds (engine, car horn, siren) are broadband with strong low-frequency energy. Outdoor sounds (wind, rain, chirping birds) produce broadband noise patterns. Living room sounds (clock tick, keyboard typing, vacuum cleaner) have distinct temporal patterns.

These spectral differences are exactly what the FiLM layer will learn to detect and compensate for.

  • In a new cell, define the mapping and verify everything loaded correctly by running this code:
# Environment-to-ESC50 mapping
ENV_MAPPING = {
    'kitchen': [27, 28, 29],   # washing_machine, can_opening, glass_breaking (ESC-50 category IDs)
    'car':     [40, 41, 42],   # engine, car_horn, siren
    'outdoor': [0, 17, 15],    # wind, rain, chirping_birds
    'living_room': [37, 30, 49] # clock_tick, keyboard_typing, vacuum_cleaner
}
ENV_LABELS = {'kitchen': 0, 'car': 1, 'outdoor': 2, 'living_room': 3}
print('Environment mapping defined:', ENV_MAPPING)
print('Data loading verified.')

What does this code do?

ENV_MAPPING maps each environment name to a list of ESC-50 category IDs. These IDs correspond to the target field in ESC-50's file naming scheme ({FOLD}-{CLIP_ID}-{TAKE}-{TARGET}.wav). When mixing noise later, the pipeline picks a random clip from the relevant category IDs.

ENV_LABELS assigns a numeric label to each environment for the classifier head. The model will learn to predict these 4 classes from the noisy audio.

  • Run the cell.

You should see the environment mapping dictionary printed out, followed by "Data loading verified." confirming everything is set up correctly.

Seeing a KeyError or missing files?

If ESC-50 files are not found, verify the unzip completed by checking that ./ESC-50-master/audio exists. Run !ls ESC-50-master/audio | head in a cell to confirm audio files are present.

help me verify my ESC-50 dataset downloaded and extracted correctly in Colab

✔️ Awesome, I've got everything!

Great. Double check you've saved your notebook (Ctrl+S or Cmd+S).

ⓧ I'd like to double check the full code

Here is the complete code across all cells up to this point:

# Environment-Adaptive SLU

import torch
import torchaudio
import torchaudio.datasets
import os

# Verify GPU
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f'Using device: {device}')
print(f'PyTorch version: {torch.__version__}')
print(f'torchaudio version: {torchaudio.__version__}')

# Download Fluent Speech Commands
fsc_dataset = torchaudio.datasets.FluentSpeechCommands(root='./data', subset='train', download=True)
fsc_val = torchaudio.datasets.FluentSpeechCommands(root='./data', subset='validation', download=True)

# Download ESC-50
import subprocess
subprocess.run(['wget', '-q', 'https://github.com/karoldvl/ESC-50/archive/master.zip', '-O', 'ESC-50.zip'])
subprocess.run(['unzip', '-q', 'ESC-50.zip'])
esc50_path = './ESC-50-master/audio'

# Environment-to-ESC50 mapping
ENV_MAPPING = {
    'kitchen': [27, 28, 29],   # washing_machine, can_opening, glass_breaking (ESC-50 category IDs)
    'car':     [40, 41, 42],   # engine, car_horn, siren
    'outdoor': [0, 17, 15],    # wind, rain, chirping_birds
    'living_room': [37, 30, 49] # clock_tick, keyboard_typing, vacuum_cleaner
}
ENV_LABELS = {'kitchen': 0, 'car': 1, 'outdoor': 2, 'living_room': 3}
print('Environment mapping defined:', ENV_MAPPING)
print('Data loading verified.')

Your Colab environment is ready with GPU access, both datasets downloaded, and environment categories mapped. Next up, you'll build the noise-mixing pipeline that combines clean speech with these environment sounds on the fly.

Build the Noise-Mixing Data Pipeline

Your datasets are downloaded and your environment mapping is ready. Now you need a way to combine clean speech with realistic background noise so the model can learn from imperfect audio.

In this step, you will build an on-the-fly noise mixing pipeline that takes clean Fluent Speech Commands utterances, layers in ESC-50 environment sounds at random signal-to-noise ratios, and produces mel-spectrograms ready for training.

In this step, get ready to:
  • Implement the noise mixing function that blends speech with environment-specific noise at random SNRs.
  • Build a PyTorch Dataset class that wraps Fluent Speech Commands with on-the-fly noise injection and mel-spectrogram computation.
  • Create DataLoaders and verify the pipeline by plotting clean vs. noisy spectrograms.
Implement the on-the-fly noise mixing function

Before you can mix noise into speech, you need to define your audio processing constants and load the ESC-50 clips into memory organized by environment category.

  • Add a new cell at the top of your notebook with the additional imports and audio constants by pasting the following code:
import torchaudio.transforms as T
import random
import numpy as np
import matplotlib.pyplot as plt
from torch.utils.data import Dataset, DataLoader

# List of environment names for random assignment
ENV_NAMES = list(ENV_LABELS.keys())

# Audio processing constants
SAMPLE_RATE = 16000
N_MELS = 64
WIN_LENGTH = int(0.025 * SAMPLE_RATE)  # 25ms
HOP_LENGTH = int(0.010 * SAMPLE_RATE)  # 10ms

# Mel-spectrogram transform
mel_transform = T.MelSpectrogram(
    sample_rate=SAMPLE_RATE, n_mels=N_MELS,
    win_length=WIN_LENGTH, hop_length=HOP_LENGTH
)
amplitude_to_db = T.AmplitudeToDB()

What does this code do?

The constants define a 16 kHz sample rate (matching Fluent Speech Commands), 64 mel frequency bins, a 25ms analysis window (400 samples), and a 10ms hop between windows (160 samples).

The MelSpectrogram transform converts raw audio waveforms into mel-scaled spectrograms. AmplitudeToDB converts the power values to decibels for a more perceptually meaningful representation.

  • Run the cell.
  • You should see no errors. The transforms are now ready to use in subsequent cells.
  • In a new cell, build the function that loads ESC-50 clips organized by category ID by pasting this code:
def load_esc50_clips_by_category(esc50_path, category_ids):
    import glob
    clips = []
    for fname in glob.glob(os.path.join(esc50_path, '*.wav')):
        # ESC-50 filename format: {fold}-{clip_id}-{take}-{category}.wav
        cat_id = int(os.path.basename(fname).split('-')[-1].replace('.wav',''))
        if cat_id in category_ids:
            clips.append(fname)
    return clips

# Pre-load file paths for each environment
esc50_clips = {env: load_esc50_clips_by_category(esc50_path, ids) for env, ids in ENV_MAPPING.items()}

What does this code do?

ESC-50 filenames encode the category as the last number before .wav. The function parses each filename, extracts the category ID, and collects paths matching the requested categories.

The esc50_clips dictionary maps each environment name to a list of noise clip file paths. This lets the mixing function quickly grab a random clip from the correct environment.

  • Run the cell.
  • You should see no output. The clips are now indexed and ready for mixing.

Getting an empty list or file not found?

Make sure esc50_path points to ./ESC-50-master/audio and that the unzip from Step 1 completed successfully. You can verify by running len(esc50_clips['kitchen']) in a new cell. It should return approximately 120 (40 clips per category, 3 categories).

help me debug why my ESC-50 clips dict is empty

Now you can implement the core noise mixing function that blends a clean speech waveform with a random environment noise clip at a controlled signal-to-noise ratio.

  • Add the mix_noise function in a new cell by pasting this code:
def mix_noise(speech_waveform, env_name, snr_db=None):
    if snr_db is None:
        snr_db = random.uniform(0, 20)
    # Pick a random noise clip from the environment
    clips = esc50_clips[env_name]
    noise_path = random.choice(clips)
    noise_waveform, noise_sr = torchaudio.load(noise_path)
    # Resample noise to match speech sample rate
    if noise_sr != SAMPLE_RATE:
        resampler = T.Resample(orig_freq=noise_sr, new_freq=SAMPLE_RATE)
        noise_waveform = resampler(noise_waveform)
    # Convert to mono if needed
    if noise_waveform.shape[0] > 1:
        noise_waveform = noise_waveform.mean(dim=0, keepdim=True)
    # Match noise length to speech length (repeat or truncate)
    speech_len = speech_waveform.shape[-1]
    if noise_waveform.shape[-1] < speech_len:
        repeats = (speech_len // noise_waveform.shape[-1]) + 1
        noise_waveform = noise_waveform.repeat(1, repeats)
    noise_waveform = noise_waveform[..., :speech_len]
    # Scale noise to target SNR
    speech_power = speech_waveform.pow(2).mean()
    noise_power = noise_waveform.pow(2).mean()
    snr_linear = 10 ** (snr_db / 10)
    scale = torch.sqrt(speech_power / (snr_linear * noise_power + 1e-8))
    noisy = speech_waveform + scale * noise_waveform
    return noisy, snr_db

What does this code do?

The function takes a clean speech waveform, an environment name, and an optional SNR in decibels. If no SNR is specified, it picks a random value between 0 and 20 dB.

It loads a random noise clip from the chosen environment, resamples it to 16 kHz, and repeats or truncates it to match the speech duration. The noise is then scaled so the mixture has the exact target SNR before being added to the speech signal.

The 1e-8 epsilon prevents division by zero when a noise clip happens to be silent.

  • Run the cell.
  • No output is expected. The function is defined and ready to use in the Dataset class.
Build the PyTorch Dataset class

The Dataset class ties everything together. It wraps Fluent Speech Commands, assigns a random environment to each sample, mixes noise, computes the mel-spectrogram, and returns tensors ready for batched training.

  • Create a new cell and paste the NoisyFSCDataset class:
class NoisyFSCDataset(Dataset):
    def __init__(self, fsc_data, max_samples=None):
        self.data = fsc_data
        if max_samples:
            indices = list(range(min(max_samples, len(fsc_data))))
            self.indices = indices
        else:
            self.indices = list(range(len(fsc_data)))

    def __len__(self):
        return len(self.indices)

    def __getitem__(self, idx):
        real_idx = self.indices[idx]
        waveform, sample_rate, transcript, speaker_id, file_path, action, obj, location = self.data[real_idx]
        # Convert to mono if needed
        if waveform.shape[0] > 1:
            waveform = waveform.mean(dim=0, keepdim=True)
        # Resample to target rate
        if sample_rate != SAMPLE_RATE:
            resampler = T.Resample(orig_freq=sample_rate, new_freq=SAMPLE_RATE)
            waveform = resampler(waveform)
        # Assign a random environment and mix noise
        env_name = random.choice(ENV_NAMES)
        env_label = ENV_LABELS[env_name]
        noisy_waveform, _ = mix_noise(waveform, env_name)
        # Compute mel-spectrogram in dB
        mel_spec = mel_transform(noisy_waveform)
        mel_spec_db = amplitude_to_db(mel_spec)
        # Fixed length: pad or truncate to 128 time frames
        target_len = 128
        if mel_spec_db.shape[-1] < target_len:
            pad = target_len - mel_spec_db.shape[-1]
            mel_spec_db = torch.nn.functional.pad(mel_spec_db, (0, pad))
        else:
            mel_spec_db = mel_spec_db[..., :target_len]
        # Intent label: map (action, obj, location) to index
        intent_label = hash((action, obj, location)) % 31
        return mel_spec_db, torch.tensor(intent_label, dtype=torch.long), torch.tensor(env_label, dtype=torch.long)

What does this code do?

Each call to __getitem__ loads a clean utterance, assigns it a random environment, mixes in noise, converts to a 64-bin mel-spectrogram, and pads or truncates to exactly 128 time frames. This fixed shape ensures all samples can be batched together.

The intent label is derived by hashing the (action, object, location) tuple from Fluent Speech Commands into one of 31 classes. The max_samples parameter lets you train on a subset (5000 samples) to keep GPU time manageable on Colab's free tier.

  • Run the cell.
  • No output is expected. The class is now available for instantiation.

Seeing a tuple unpacking error?

The Fluent Speech Commands dataset returns 8 values per sample. Make sure your unpacking line has exactly 8 variables: waveform, sample_rate, transcript, speaker_id, file_path, action, obj, location. A mismatch in variable count causes a ValueError.

help me fix my Dataset tuple unpacking error

Create train/validation DataLoaders and verify a batch visually

With the Dataset class ready, you can now instantiate training and validation sets, wrap them in DataLoaders, and confirm everything works by plotting a spectrogram comparison.

  • Create the DataLoaders by adding a new cell with this code:
train_dataset = NoisyFSCDataset(fsc_dataset, max_samples=5000)
val_dataset = NoisyFSCDataset(fsc_val, max_samples=1000)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=2)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False, num_workers=2)

Why limit to 5000 training samples?

The full Fluent Speech Commands training set has 23,132 utterances. Using a 5000-sample subset keeps each training epoch under a few minutes on Colab's free GPU, so both models (baseline and FiLM) can train within a single session.

  • Run the cell.
  • No output is expected. The DataLoaders are instantiated and ready.

Now verify the entire pipeline visually by plotting a clean mel-spectrogram alongside a noisy one.

  • Add a final verification cell with this plotting code:
# Verify a batch visually
clean_waveform, sr, _, _, _, action, obj, location = fsc_dataset[0]
if clean_waveform.shape[0] > 1:
    clean_waveform = clean_waveform.mean(dim=0, keepdim=True)
if sr != SAMPLE_RATE:
    clean_waveform = T.Resample(sr, SAMPLE_RATE)(clean_waveform)
clean_mel = amplitude_to_db(mel_transform(clean_waveform))
noisy_waveform, _ = mix_noise(clean_waveform, 'kitchen', snr_db=5)
noisy_mel = amplitude_to_db(mel_transform(noisy_waveform))
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].imshow(clean_mel[0].numpy(), aspect='auto', origin='lower')
axes[0].set_title('Clean Mel-Spectrogram')
axes[1].imshow(noisy_mel[0].numpy(), aspect='auto', origin='lower')
axes[1].set_title('Noisy Mel-Spectrogram (Kitchen, 5dB SNR)')
plt.tight_layout()
plt.show()
print('DataLoaders created. Train samples:', len(train_dataset), '| Val samples:', len(val_dataset))

What should I see?

Two side-by-side spectrogram plots appear. The left shows clear harmonic structure of clean speech. The right shows the same speech with kitchen noise (washing machine, glass breaking, can opening) overlaid at 5 dB SNR. You can see additional energy patterns in the mid-to-high frequency bands from the kitchen sounds.

Below the plot, the print statement confirms 5000 training samples and 1000 validation samples.

  • Run the cell.
  • You should see the two spectrogram plots and the text DataLoaders created. Train samples: 5000 | Val samples: 1000 printed below.

Plots not showing or kernel crashes?

If plots do not render, make sure you are running in a Colab notebook (not a plain Python script). Colab renders matplotlib inline by default. If the kernel crashes, reduce num_workers to 0 in your DataLoader calls. Colab's free tier sometimes restricts multiprocessing.

help me debug my Colab plotting or DataLoader crash

✔️ Awesome, I've got everything!

Great. Double check you have run all cells and saved your notebook.

ⓧ I'd like to double check the full code

Here is the complete notebook code up to this point:

# Environment-Adaptive SLU

import torch
import torchaudio
import torchaudio.datasets
import torchaudio.transforms as T
import os
import random
import numpy as np
import matplotlib.pyplot as plt
from torch.utils.data import Dataset, DataLoader

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

fsc_dataset = torchaudio.datasets.FluentSpeechCommands(root='./data', subset='train', download=True)
fsc_val = torchaudio.datasets.FluentSpeechCommands(root='./data', subset='validation', download=True)
esc50_path = './ESC-50-master/audio'

ENV_MAPPING = {
    'kitchen': [27, 28, 29],
    'car':     [40, 41, 42],
    'outdoor': [0, 17, 15],
    'living_room': [37, 30, 49]
}
ENV_LABELS = {'kitchen': 0, 'car': 1, 'outdoor': 2, 'living_room': 3}
ENV_NAMES = list(ENV_LABELS.keys())

SAMPLE_RATE = 16000
N_MELS = 64
WIN_LENGTH = int(0.025 * SAMPLE_RATE)  # 25ms
HOP_LENGTH = int(0.010 * SAMPLE_RATE)  # 10ms

mel_transform = T.MelSpectrogram(
    sample_rate=SAMPLE_RATE, n_mels=N_MELS,
    win_length=WIN_LENGTH, hop_length=HOP_LENGTH
)
amplitude_to_db = T.AmplitudeToDB()

def load_esc50_clips_by_category(esc50_path, category_ids):
    import glob
    clips = []
    for fname in glob.glob(os.path.join(esc50_path, '*.wav')):
        # ESC-50 filename format: {fold}-{clip_id}-{take}-{category}.wav
        cat_id = int(os.path.basename(fname).split('-')[-1].replace('.wav',''))
        if cat_id in category_ids:
            clips.append(fname)
    return clips

esc50_clips = {env: load_esc50_clips_by_category(esc50_path, ids) for env, ids in ENV_MAPPING.items()}

def mix_noise(speech_waveform, env_name, snr_db=None):
    if snr_db is None:
        snr_db = random.uniform(0, 20)
    clips = esc50_clips[env_name]
    noise_path = random.choice(clips)
    noise_waveform, noise_sr = torchaudio.load(noise_path)
    if noise_sr != SAMPLE_RATE:
        resampler = T.Resample(orig_freq=noise_sr, new_freq=SAMPLE_RATE)
        noise_waveform = resampler(noise_waveform)
    if noise_waveform.shape[0] > 1:
        noise_waveform = noise_waveform.mean(dim=0, keepdim=True)
    speech_len = speech_waveform.shape[-1]
    if noise_waveform.shape[-1] < speech_len:
        repeats = (speech_len // noise_waveform.shape[-1]) + 1
        noise_waveform = noise_waveform.repeat(1, repeats)
    noise_waveform = noise_waveform[..., :speech_len]
    speech_power = speech_waveform.pow(2).mean()
    noise_power = noise_waveform.pow(2).mean()
    snr_linear = 10 ** (snr_db / 10)
    scale = torch.sqrt(speech_power / (snr_linear * noise_power + 1e-8))
    noisy = speech_waveform + scale * noise_waveform
    return noisy, snr_db

class NoisyFSCDataset(Dataset):
    def __init__(self, fsc_data, max_samples=None):
        self.data = fsc_data
        if max_samples:
            indices = list(range(min(max_samples, len(fsc_data))))
            self.indices = indices
        else:
            self.indices = list(range(len(fsc_data)))

    def __len__(self):
        return len(self.indices)

    def __getitem__(self, idx):
        real_idx = self.indices[idx]
        waveform, sample_rate, transcript, speaker_id, file_path, action, obj, location = self.data[real_idx]
        if waveform.shape[0] > 1:
            waveform = waveform.mean(dim=0, keepdim=True)
        if sample_rate != SAMPLE_RATE:
            resampler = T.Resample(orig_freq=sample_rate, new_freq=SAMPLE_RATE)
            waveform = resampler(waveform)
        env_name = random.choice(ENV_NAMES)
        env_label = ENV_LABELS[env_name]
        noisy_waveform, _ = mix_noise(waveform, env_name)
        mel_spec = mel_transform(noisy_waveform)
        mel_spec_db = amplitude_to_db(mel_spec)
        # Fixed length: pad or truncate to 128 time frames
        target_len = 128
        if mel_spec_db.shape[-1] < target_len:
            pad = target_len - mel_spec_db.shape[-1]
            mel_spec_db = torch.nn.functional.pad(mel_spec_db, (0, pad))
        else:
            mel_spec_db = mel_spec_db[..., :target_len]
        # Intent label: map (action, obj, location) to index
        intent_label = hash((action, obj, location)) % 31
        return mel_spec_db, torch.tensor(intent_label, dtype=torch.long), torch.tensor(env_label, dtype=torch.long)

train_dataset = NoisyFSCDataset(fsc_dataset, max_samples=5000)
val_dataset = NoisyFSCDataset(fsc_val, max_samples=1000)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=2)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False, num_workers=2)

# Verify a batch visually
clean_waveform, sr, _, _, _, action, obj, location = fsc_dataset[0]
if clean_waveform.shape[0] > 1:
    clean_waveform = clean_waveform.mean(dim=0, keepdim=True)
if sr != SAMPLE_RATE:
    clean_waveform = T.Resample(sr, SAMPLE_RATE)(clean_waveform)
clean_mel = amplitude_to_db(mel_transform(clean_waveform))
noisy_waveform, _ = mix_noise(clean_waveform, 'kitchen', snr_db=5)
noisy_mel = amplitude_to_db(mel_transform(noisy_waveform))
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].imshow(clean_mel[0].numpy(), aspect='auto', origin='lower')
axes[0].set_title('Clean Mel-Spectrogram')
axes[1].imshow(noisy_mel[0].numpy(), aspect='auto', origin='lower')
axes[1].set_title('Noisy Mel-Spectrogram (Kitchen, 5dB SNR)')
plt.tight_layout()
plt.show()
print('DataLoaders created. Train samples:', len(train_dataset), '| Val samples:', len(val_dataset))

Your data pipeline is complete. Every training sample now arrives as a noisy mel-spectrogram paired with both an intent label and an environment label. Next up, you will build the FiLM-conditioned architecture that uses environment detection to dynamically adapt its feature processing strategy.

Implement the FiLM-Conditioned Architecture

Your data pipeline mixes environment noise into clean speech and produces mel-spectrograms ready for modeling. Now you need an architecture that can detect the environment and use that detection to improve intent recognition.

The key insight is Feature-wise Linear Modulation (FiLM). Instead of feeding environment information as a concatenated input, FiLM applies per-channel scaling and shifting to intermediate feature maps. This lets the model learn environment-specific frequency emphasis patterns directly inside the network.

In this step, get ready to:
  • Build the shared CNN encoder and environment classification head.
  • Implement the FiLM generator and modulation function.
  • Wire the full forward pass and define a baseline model for comparison.
Build the shared encoder and environment head

The architecture uses a shared CNN encoder that extracts features from mel-spectrograms. On top of that, an environment head classifies the acoustic scene and produces a compact embedding that downstream components can condition on.

  • In your notebook, find the imports cell at the top.
  • Find this line:
import torch
  • Add import torch.nn as nn directly below it, so the two lines look like this:
import torch
import torch.nn as nn
  • Re-run the imports cell to make nn available.
  • Create a new code cell below your data pipeline code and add the shared encoder and environment head by pasting this code:
# ---- Architecture ----

class SharedCNNEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        # Block 1: 1 input channel (mono spectrogram) -> 32 feature maps
        self.block1 = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=3, padding=1),
            nn.BatchNorm2d(32), nn.ReLU(), nn.MaxPool2d(2)
        )
        # Block 2: 32 -> 64 feature maps
        self.block2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64), nn.ReLU(), nn.MaxPool2d(2)
        )
        # Block 3: 64 -> 128 feature maps
        self.block3 = nn.Sequential(
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128), nn.ReLU(), nn.MaxPool2d(2)
        )
    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.block3(x)
        return x  # (B, 128, H', W')

class EnvironmentHead(nn.Module):
    def __init__(self, in_channels=128, embed_dim=64, num_envs=4):
        super().__init__()
        # Global average pooling collapses spatial dims
        self.gap = nn.AdaptiveAvgPool2d(1)
        # Project to a 64-dim embedding that FiLM will condition on
        self.embed_fc = nn.Linear(in_channels, embed_dim)
        # Classify into 4 environments
        self.classify_fc = nn.Linear(embed_dim, num_envs)
    def forward(self, features):
        pooled = self.gap(features).squeeze(-1).squeeze(-1)  # (B, 128)
        embedding = torch.relu(self.embed_fc(pooled))         # (B, 64)
        logits = self.classify_fc(embedding)                  # (B, 4)
        return logits, embedding

What does this code do?

SharedCNNEncoder is a 3-block CNN that progressively doubles the channel count (1 -> 32 -> 64 -> 128) while halving spatial dimensions via max pooling. After all three blocks, a 64x128 spectrogram becomes a 128-channel feature map at 8x16 resolution.

EnvironmentHead serves a dual purpose. It classifies the environment (4 classes) for the auxiliary loss, and it produces a 64-dimensional embedding that the FiLM generator will consume. The embedding is an intermediate representation, not just a side output.

  • Run the cell.
  • You should see no output (class definitions don't produce output). If you see a NameError about nn, re-run the imports cell above.

Seeing a NameError?

If you see an error about nn not being defined, you likely forgot to re-run the imports cell after adding import torch.nn as nn. Go back to the imports cell, confirm the line is there, and run it again.

Help me debug a NameError in my architecture cell

Implement the FiLM generator and modulation

FiLM conditioning works by applying a learned affine transformation per feature channel. The formula is: output = gamma * features + beta. Gamma scales each channel (amplifying or suppressing it), and beta shifts it. Both are predicted from the environment embedding, so the transformation adapts to the detected acoustic scene.

  • Create a new code cell and add the FiLM conditioning components by pasting this code:
class FiLMGenerator(nn.Module):
    def __init__(self, embed_dim=64, num_channels=128):
        super().__init__()
        # Separate linear layers predict per-channel gamma and beta
        self.gamma_fc = nn.Linear(embed_dim, num_channels)
        self.beta_fc  = nn.Linear(embed_dim, num_channels)
    def forward(self, embedding):
        gamma = self.gamma_fc(embedding)  # (B, 128)
        beta  = self.beta_fc(embedding)   # (B, 128)
        return gamma, beta

def film_modulate(features, gamma, beta):
    # features: (B, C, H, W), gamma/beta: (B, C)
    # Expand gamma/beta to broadcast over spatial dimensions
    gamma = gamma.unsqueeze(-1).unsqueeze(-1)  # (B, C, 1, 1)
    beta  = beta.unsqueeze(-1).unsqueeze(-1)
    return gamma * features + beta

What does this code do?

FiLMGenerator takes the 64-dim environment embedding and projects it into 128 gamma values and 128 beta values. Each pair corresponds to one channel of the encoder's output feature map.

film_modulate applies the actual transformation. The unsqueeze calls expand gamma and beta from shape (B, C) to (B, C, 1, 1) so they broadcast correctly across the height and width dimensions of the feature map. This means every spatial position in a given channel gets the same scale and shift.

Why FiLM instead of concatenation?

You could feed environment information by concatenating an embedding to the feature vector. FiLM is more parameter-efficient because it only needs two linear layers (gamma and beta) regardless of feature map size. It's also more interpretable. You can directly inspect which channels get amplified or suppressed per environment.

The per-channel modulation maps naturally to per-frequency-band emphasis in audio. A kitchen environment might suppress low-frequency engine rumble channels while amplifying mid-frequency speech channels.

  • Run the cell. You should see no output (just definitions).
Build the intent head and wire the full forward pass

The intent head receives FiLM-modulated features and classifies them into 31 intents from the Fluent Speech Commands dataset. You also need to wire all components together into a single model class, and define a baseline model that skips FiLM conditioning for comparison.

  • Create a new code cell and define the intent head, the full FiLM model, and the baseline model by pasting this code:
class IntentHead(nn.Module):
    def __init__(self, in_channels=128, hidden_dim=256, num_intents=31):
        super().__init__()
        self.gap = nn.AdaptiveAvgPool2d(1)
        self.fc1 = nn.Linear(in_channels, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, num_intents)
    def forward(self, features):
        pooled = self.gap(features).squeeze(-1).squeeze(-1)
        x = torch.relu(self.fc1(pooled))
        return self.fc2(x)

class EnvironmentAdaptiveSLU(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = SharedCNNEncoder()
        self.env_head = EnvironmentHead(in_channels=128, embed_dim=64, num_envs=4)
        self.film_gen = FiLMGenerator(embed_dim=64, num_channels=128)
        self.intent_head = IntentHead(in_channels=128, hidden_dim=256, num_intents=31)
    def forward(self, x):
        # Shared encoder extracts features
        features = self.encoder(x)
        # Environment head detects scene and produces embedding
        env_logits, env_embedding = self.env_head(features)
        # FiLM generator produces per-channel modulation parameters
        gamma, beta = self.film_gen(env_embedding)
        # Apply FiLM: adapt features based on detected environment
        modulated = film_modulate(features, gamma, beta)
        # Intent head classifies from modulated features
        intent_logits = self.intent_head(modulated)
        return intent_logits, env_logits

class BaselineSLU(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = SharedCNNEncoder()
        self.env_head = EnvironmentHead(in_channels=128, embed_dim=64, num_envs=4)
        self.intent_head = IntentHead(in_channels=128, hidden_dim=256, num_intents=31)
    def forward(self, x):
        features = self.encoder(x)
        env_logits, _ = self.env_head(features)
        intent_logits = self.intent_head(features)  # No FiLM modulation
        return intent_logits, env_logits

# Instantiate both models and move to GPU
film_model = EnvironmentAdaptiveSLU().to(device)
baseline_model = BaselineSLU().to(device)
print('FiLM model parameters:', sum(p.numel() for p in film_model.parameters()))
print('Baseline model parameters:', sum(p.numel() for p in baseline_model.parameters()))

What does this code do?

IntentHead uses global average pooling followed by a two-layer classifier (128 -> 256 -> 31). It receives either raw or FiLM-modulated features depending on which model calls it.

EnvironmentAdaptiveSLU is the full FiLM-conditioned model. Its forward pass flows: encoder -> env_head -> FiLM_gen -> modulate -> intent_head. The environment detection feeds directly into feature modulation before intent classification.

BaselineSLU is identical except it skips FiLM conditioning entirely. The environment head still trains (providing the auxiliary loss signal), but its embedding never modulates the intent features. This isolates the effect of FiLM when you compare results later.

  • Run the cell.
  • You should see two lines printing parameter counts. The FiLM model should have slightly more parameters than the baseline (the difference is the FiLM generator's two linear layers).

Parameter counts look wrong?

If both models show exactly the same parameter count, check that EnvironmentAdaptiveSLU includes self.film_gen = FiLMGenerator(embed_dim=64, num_channels=128) in its __init__. If you see a RuntimeError about CUDA, make sure you selected a GPU runtime in Colab (Runtime > Change runtime type > T4 GPU).

Help me debug my model instantiation

✔️ Awesome, I've got everything!

Great. Double check you've run all three architecture cells (encoder/env head, FiLM generator, and full models) and that the parameter count printout appeared.

ⓧ I'd like to double check the full code

Here is all the code your notebook should contain up to this point. Compare cell by cell.

# Environment-Adaptive SLU

import torch
import torch.nn as nn
import torchaudio
import torchaudio.datasets
import torchaudio.transforms as T
import os
import random
import numpy as np
import matplotlib.pyplot as plt
from torch.utils.data import Dataset, DataLoader

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

fsc_dataset = torchaudio.datasets.FluentSpeechCommands(root='./data', subset='train', download=True)
fsc_val = torchaudio.datasets.FluentSpeechCommands(root='./data', subset='validation', download=True)
esc50_path = './ESC-50-master/audio'

ENV_MAPPING = {
    'kitchen': [27, 28, 29],
    'car':     [40, 41, 42],
    'outdoor': [0, 17, 15],
    'living_room': [37, 30, 49]
}
ENV_LABELS = {'kitchen': 0, 'car': 1, 'outdoor': 2, 'living_room': 3}
ENV_NAMES = list(ENV_LABELS.keys())
SAMPLE_RATE = 16000
N_MELS = 64
WIN_LENGTH = int(0.025 * SAMPLE_RATE)
HOP_LENGTH = int(0.010 * SAMPLE_RATE)
mel_transform = T.MelSpectrogram(sample_rate=SAMPLE_RATE, n_mels=N_MELS, win_length=WIN_LENGTH, hop_length=HOP_LENGTH)
amplitude_to_db = T.AmplitudeToDB()

import glob
def load_esc50_clips_by_category(esc50_path, category_ids):
    clips = []
    for fname in glob.glob(os.path.join(esc50_path, '*.wav')):
        cat_id = int(os.path.basename(fname).split('-')[-1].replace('.wav',''))
        if cat_id in category_ids:
            clips.append(fname)
    return clips
esc50_clips = {env: load_esc50_clips_by_category(esc50_path, ids) for env, ids in ENV_MAPPING.items()}

def mix_noise(speech_waveform, env_name, snr_db=None):
    if snr_db is None:
        snr_db = random.uniform(0, 20)
    clips = esc50_clips[env_name]
    noise_path = random.choice(clips)
    noise_waveform, noise_sr = torchaudio.load(noise_path)
    if noise_sr != SAMPLE_RATE:
        resampler = T.Resample(orig_freq=noise_sr, new_freq=SAMPLE_RATE)
        noise_waveform = resampler(noise_waveform)
    if noise_waveform.shape[0] > 1:
        noise_waveform = noise_waveform.mean(dim=0, keepdim=True)
    speech_len = speech_waveform.shape[-1]
    if noise_waveform.shape[-1] < speech_len:
        repeats = (speech_len // noise_waveform.shape[-1]) + 1
        noise_waveform = noise_waveform.repeat(1, repeats)
    noise_waveform = noise_waveform[..., :speech_len]
    speech_power = speech_waveform.pow(2).mean()
    noise_power = noise_waveform.pow(2).mean()
    snr_linear = 10 ** (snr_db / 10)
    scale = torch.sqrt(speech_power / (snr_linear * noise_power + 1e-8))
    noisy = speech_waveform + scale * noise_waveform
    return noisy, snr_db

class NoisyFSCDataset(Dataset):
    def __init__(self, fsc_data, max_samples=None):
        self.data = fsc_data
        if max_samples:
            self.indices = list(range(min(max_samples, len(fsc_data))))
        else:
            self.indices = list(range(len(fsc_data)))
    def __len__(self):
        return len(self.indices)
    def __getitem__(self, idx):
        real_idx = self.indices[idx]
        waveform, sample_rate, transcript, speaker_id, file_path, action, obj, location = self.data[real_idx]
        if waveform.shape[0] > 1:
            waveform = waveform.mean(dim=0, keepdim=True)
        if sample_rate != SAMPLE_RATE:
            waveform = T.Resample(orig_freq=sample_rate, new_freq=SAMPLE_RATE)(waveform)
        env_name = random.choice(ENV_NAMES)
        env_label = ENV_LABELS[env_name]
        noisy_waveform, _ = mix_noise(waveform, env_name)
        mel_spec = mel_transform(noisy_waveform)
        mel_spec_db = amplitude_to_db(mel_spec)
        target_len = 128
        if mel_spec_db.shape[-1] < target_len:
            mel_spec_db = torch.nn.functional.pad(mel_spec_db, (0, target_len - mel_spec_db.shape[-1]))
        else:
            mel_spec_db = mel_spec_db[..., :target_len]
        intent_label = hash((action, obj, location)) % 31
        return mel_spec_db, torch.tensor(intent_label, dtype=torch.long), torch.tensor(env_label, dtype=torch.long)

train_dataset = NoisyFSCDataset(fsc_dataset, max_samples=5000)
val_dataset = NoisyFSCDataset(fsc_val, max_samples=1000)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=2)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False, num_workers=2)

# ---- Architecture ----

class SharedCNNEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.block1 = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=3, padding=1),
            nn.BatchNorm2d(32), nn.ReLU(), nn.MaxPool2d(2)
        )
        self.block2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64), nn.ReLU(), nn.MaxPool2d(2)
        )
        self.block3 = nn.Sequential(
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128), nn.ReLU(), nn.MaxPool2d(2)
        )
    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.block3(x)
        return x  # (B, 128, H', W')

class EnvironmentHead(nn.Module):
    def __init__(self, in_channels=128, embed_dim=64, num_envs=4):
        super().__init__()
        self.gap = nn.AdaptiveAvgPool2d(1)
        self.embed_fc = nn.Linear(in_channels, embed_dim)
        self.classify_fc = nn.Linear(embed_dim, num_envs)
    def forward(self, features):
        pooled = self.gap(features).squeeze(-1).squeeze(-1)  # (B, 128)
        embedding = torch.relu(self.embed_fc(pooled))         # (B, 64)
        logits = self.classify_fc(embedding)                  # (B, 4)
        return logits, embedding

class FiLMGenerator(nn.Module):
    def __init__(self, embed_dim=64, num_channels=128):
        super().__init__()
        self.gamma_fc = nn.Linear(embed_dim, num_channels)
        self.beta_fc  = nn.Linear(embed_dim, num_channels)
    def forward(self, embedding):
        gamma = self.gamma_fc(embedding)  # (B, 128)
        beta  = self.beta_fc(embedding)   # (B, 128)
        return gamma, beta

def film_modulate(features, gamma, beta):
    # features: (B, C, H, W), gamma/beta: (B, C)
    gamma = gamma.unsqueeze(-1).unsqueeze(-1)  # (B, C, 1, 1)
    beta  = beta.unsqueeze(-1).unsqueeze(-1)
    return gamma * features + beta

class IntentHead(nn.Module):
    def __init__(self, in_channels=128, hidden_dim=256, num_intents=31):
        super().__init__()
        self.gap = nn.AdaptiveAvgPool2d(1)
        self.fc1 = nn.Linear(in_channels, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, num_intents)
    def forward(self, features):
        pooled = self.gap(features).squeeze(-1).squeeze(-1)
        x = torch.relu(self.fc1(pooled))
        return self.fc2(x)

class EnvironmentAdaptiveSLU(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = SharedCNNEncoder()
        self.env_head = EnvironmentHead(in_channels=128, embed_dim=64, num_envs=4)
        self.film_gen = FiLMGenerator(embed_dim=64, num_channels=128)
        self.intent_head = IntentHead(in_channels=128, hidden_dim=256, num_intents=31)
    def forward(self, x):
        features = self.encoder(x)
        env_logits, env_embedding = self.env_head(features)
        gamma, beta = self.film_gen(env_embedding)
        modulated = film_modulate(features, gamma, beta)
        intent_logits = self.intent_head(modulated)
        return intent_logits, env_logits

class BaselineSLU(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = SharedCNNEncoder()
        self.env_head = EnvironmentHead(in_channels=128, embed_dim=64, num_envs=4)
        self.intent_head = IntentHead(in_channels=128, hidden_dim=256, num_intents=31)
    def forward(self, x):
        features = self.encoder(x)
        env_logits, _ = self.env_head(features)
        intent_logits = self.intent_head(features)  # No FiLM modulation
        return intent_logits, env_logits

film_model = EnvironmentAdaptiveSLU().to(device)
baseline_model = BaselineSLU().to(device)
print('FiLM model parameters:', sum(p.numel() for p in film_model.parameters()))
print('Baseline model parameters:', sum(p.numel() for p in baseline_model.parameters()))

Both models are on the GPU and ready to train. Next up, you'll define the joint loss function, train both models side by side, and see whether FiLM conditioning actually improves intent accuracy under noise.

Train and Compare Against Baseline

You have two models sitting side by side. One uses FiLM conditioning to adapt its features based on detected environment. The other classifies environment but never uses that information to help intent recognition.

Now you need to prove that conditioning actually helps. In this step, you will train both models under identical conditions and measure the accuracy gap that FiLM conditioning creates.

In this step, get ready to:
  • Define a reusable training function with joint multi-task loss, Adam optimizer, and learning rate scheduling.
  • Train both the FiLM-conditioned model and the baseline for 15 epochs on noisy speech data.
  • Print a comparison table showing per-environment intent accuracy for both models.
Define the training function with joint loss and scheduler

Multi-task learning requires a combined loss that balances both objectives. The intent classification loss drives the main task, while the environment classification loss (weighted at 0.3) ensures the environment embedding stays meaningful without dominating training.

  • In your notebook, create a new code cell below the architecture cells and paste in this training function:
# ---- Training ----

def train_model(model, train_loader, val_loader, num_epochs=15, model_name='model'):
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, patience=3, factor=0.5)
    intent_criterion = nn.CrossEntropyLoss()
    env_criterion = nn.CrossEntropyLoss()
    history = {'train_loss': [], 'val_loss': [], 'val_intent_acc': [], 'val_env_acc_per_env': []}

    for epoch in range(num_epochs):
        model.train()
        total_loss = 0
        for mel_spec, intent_labels, env_labels in train_loader:
            mel_spec = mel_spec.to(device)
            intent_labels = intent_labels.to(device)
            env_labels = env_labels.to(device)
            optimizer.zero_grad()
            intent_logits, env_logits = model(mel_spec)
            # Joint loss: intent task + weighted environment task
            loss = intent_criterion(intent_logits, intent_labels) + 0.3 * env_criterion(env_logits, env_labels)
            loss.backward()
            optimizer.step()
            total_loss += loss.item()
        avg_train_loss = total_loss / len(train_loader)

        # Validation pass with per-environment accuracy tracking
        model.eval()
        val_loss = 0
        correct_intent = 0
        total = 0
        env_correct = {i: 0 for i in range(4)}
        env_total = {i: 0 for i in range(4)}
        with torch.no_grad():
            for mel_spec, intent_labels, env_labels in val_loader:
                mel_spec = mel_spec.to(device)
                intent_labels = intent_labels.to(device)
                env_labels = env_labels.to(device)
                intent_logits, env_logits = model(mel_spec)
                loss = intent_criterion(intent_logits, intent_labels) + 0.3 * env_criterion(env_logits, env_labels)
                val_loss += loss.item()
                preds = intent_logits.argmax(dim=1)
                correct_intent += (preds == intent_labels).sum().item()
                total += intent_labels.size(0)
                # Track accuracy separately for each environment
                for env_id in range(4):
                    mask = (env_labels == env_id)
                    env_correct[env_id] += ((preds == intent_labels) & mask).sum().item()
                    env_total[env_id] += mask.sum().item()
        avg_val_loss = val_loss / len(val_loader)
        intent_acc = correct_intent / total
        per_env_acc = {i: env_correct[i] / max(env_total[i], 1) for i in range(4)}
        scheduler.step(avg_val_loss)
        history['train_loss'].append(avg_train_loss)
        history['val_loss'].append(avg_val_loss)
        history['val_intent_acc'].append(intent_acc)
        history['val_env_acc_per_env'].append(per_env_acc)
        print(f'[{model_name}] Epoch {epoch+1}/{num_epochs} | Train Loss: {avg_train_loss:.4f} | Val Loss: {avg_val_loss:.4f} | Intent Acc: {intent_acc:.4f}')
    return history

What does this code do?

This function encapsulates the full training and validation loop for any model that returns (intent_logits, env_logits). It handles several key responsibilities:

  • Joint loss: CrossEntropy(intent) + 0.3 * CrossEntropy(env) balances the two tasks. The 0.3 weight keeps environment learning active without overwhelming intent gradients.
  • ReduceLROnPlateau halves the learning rate when validation loss stalls for 3 epochs. This prevents overshooting as the model converges.
  • Per-environment tracking separates intent accuracy by environment during validation. This reveals whether certain acoustic conditions are harder for the model.

Why weight environment loss at 0.3?

The environment classification is an auxiliary task. Its purpose is to produce a useful embedding that tells the FiLM generator which environment the audio comes from. Weighting it lower than the intent loss ensures the shared encoder prioritizes features that help intent recognition while still learning enough about environment characteristics to generate meaningful FiLM parameters.

  • Run the cell. You should see no output yet since this only defines the function.

Seeing a syntax error?

Check that the f-string on the print line uses curly braces around each variable. Also verify that the indentation is consistent throughout the function body (4 spaces per level).

Help me fix a syntax error in my train_model function

Train both models for 15 epochs

With 5000 training samples and 15 epochs, each model should complete training in roughly 12-15 minutes on a Colab GPU. You will train both sequentially using the same function, same data, and same hyperparameters so the only variable is the FiLM conditioning.

  • Create a new code cell and kick off training for both models by pasting this code:
print('Training FiLM model...')
film_history = train_model(film_model, train_loader, val_loader, num_epochs=15, model_name='FiLM')
print('Training Baseline model...')
baseline_history = train_model(baseline_model, train_loader, val_loader, num_epochs=15, model_name='Baseline')

What does this code do?

This trains the FiLM-conditioned model first, then the baseline. Both use identical training conditions (same optimizer settings, same data, same number of epochs). The only difference is that the FiLM model modulates its features based on detected environment, while the baseline processes features directly.

The returned history dictionaries store loss curves and accuracy metrics you will use for comparison.

  • Run the cell. You will see epoch-by-epoch progress for the FiLM model, followed by the baseline model.

Each epoch prints training loss, validation loss, and intent accuracy. Watch for the intent accuracy climbing as epochs progress.

Training time expectations

On a Colab GPU, expect roughly 12-15 minutes per model (about 25-30 minutes total). On CPU it takes significantly longer. If your Colab session did not get a GPU, training still works but may take around 45 minutes per model.

Training seems stuck or very slow?

  • Verify your runtime has a GPU by running torch.cuda.is_available() in a new cell. If it returns False, go to **Runtime** > **Change runtime type** and select **GPU**.
  • If Colab cannot allocate a GPU, the training still runs on CPU. It will be slower but will complete.

Help me troubleshoot slow training in my Colab notebook

Print the comparison table

Training is complete, but raw epoch logs are hard to compare at a glance. A structured comparison table shows overall intent accuracy and breaks it down by environment, revealing exactly where FiLM conditioning helps most.

  • Create a new code cell and paste this comparison table code:
# Comparison table
print('\n=== Comparison Table ===')
print(f'{"Model":<12} {"Overall Acc":<14}', end='')
for env in ENV_NAMES:
    print(f'{env:<16}', end='')
print()
for name, history in [("FiLM", film_history), ("Baseline", baseline_history)]:
    final_acc = history['val_intent_acc'][-1]
    final_per_env = history['val_env_acc_per_env'][-1]
    print(f'{name:<12} {final_acc:<14.4f}', end='')
    for i in range(4):
        print(f'{final_per_env[i]:<16.4f}', end='')
    print()

What does this code do?

  • Extracts the final epoch's accuracy from each model's history dictionary.
  • Prints a formatted table with columns for overall accuracy and each of the 4 environments (kitchen, car, outdoor, living_room).
  • The per-environment breakdown reveals which acoustic conditions benefit most from FiLM adaptation.
  • Run the cell. You should see a table with two rows (FiLM and Baseline) showing accuracy values across environments.

The FiLM model should show a measurable accuracy advantage overall, with the gap widening in environments that have more spectrally distinct noise patterns (like car engine or outdoor wind).

Both models show similar accuracy?

  • With random noise mixing, results vary between runs. Differences of 1-3 percentage points are typical. The gap is more visible in noisier environments.
  • If both models have very low accuracy (under 5%), check that your NoisyFSCDataset is returning valid mel-spectrograms and that labels are correct.

Help me understand why my FiLM model isn't outperforming the baseline

✔️ Awesome, I've got everything!

Great. Double check you have saved all cells in your notebook.

ⓧ I'd like to double check the full code

Here is the complete training and comparison code added in this step. Your earlier cells (imports, data pipeline, architecture) remain unchanged above.

# ---- Training ----

def train_model(model, train_loader, val_loader, num_epochs=15, model_name='model'):
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
    scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, patience=3, factor=0.5)
    intent_criterion = nn.CrossEntropyLoss()
    env_criterion = nn.CrossEntropyLoss()
    history = {'train_loss': [], 'val_loss': [], 'val_intent_acc': [], 'val_env_acc_per_env': []}

    for epoch in range(num_epochs):
        model.train()
        total_loss = 0
        for mel_spec, intent_labels, env_labels in train_loader:
            mel_spec = mel_spec.to(device)
            intent_labels = intent_labels.to(device)
            env_labels = env_labels.to(device)
            optimizer.zero_grad()
            intent_logits, env_logits = model(mel_spec)
            loss = intent_criterion(intent_logits, intent_labels) + 0.3 * env_criterion(env_logits, env_labels)
            loss.backward()
            optimizer.step()
            total_loss += loss.item()
        avg_train_loss = total_loss / len(train_loader)

        model.eval()
        val_loss = 0
        correct_intent = 0
        total = 0
        env_correct = {i: 0 for i in range(4)}
        env_total = {i: 0 for i in range(4)}
        with torch.no_grad():
            for mel_spec, intent_labels, env_labels in val_loader:
                mel_spec = mel_spec.to(device)
                intent_labels = intent_labels.to(device)
                env_labels = env_labels.to(device)
                intent_logits, env_logits = model(mel_spec)
                loss = intent_criterion(intent_logits, intent_labels) + 0.3 * env_criterion(env_logits, env_labels)
                val_loss += loss.item()
                preds = intent_logits.argmax(dim=1)
                correct_intent += (preds == intent_labels).sum().item()
                total += intent_labels.size(0)
                for env_id in range(4):
                    mask = (env_labels == env_id)
                    env_correct[env_id] += ((preds == intent_labels) & mask).sum().item()
                    env_total[env_id] += mask.sum().item()
        avg_val_loss = val_loss / len(val_loader)
        intent_acc = correct_intent / total
        per_env_acc = {i: env_correct[i] / max(env_total[i], 1) for i in range(4)}
        scheduler.step(avg_val_loss)
        history['train_loss'].append(avg_train_loss)
        history['val_loss'].append(avg_val_loss)
        history['val_intent_acc'].append(intent_acc)
        history['val_env_acc_per_env'].append(per_env_acc)
        print(f'[{model_name}] Epoch {epoch+1}/{num_epochs} | Train Loss: {avg_train_loss:.4f} | Val Loss: {avg_val_loss:.4f} | Intent Acc: {intent_acc:.4f}')
    return history

print('Training FiLM model...')
film_history = train_model(film_model, train_loader, val_loader, num_epochs=15, model_name='FiLM')
print('Training Baseline model...')
baseline_history = train_model(baseline_model, train_loader, val_loader, num_epochs=15, model_name='Baseline')

# Comparison table
print('\n=== Comparison Table ===')
print(f'{"Model":<12} {"Overall Acc":<14}', end='')
for env in ENV_NAMES:
    print(f'{env:<16}', end='')
print()
for name, history in [("FiLM", film_history), ("Baseline", baseline_history)]:
    final_acc = history['val_intent_acc'][-1]
    final_per_env = history['val_env_acc_per_env'][-1]
    print(f'{name:<12} {final_acc:<14.4f}', end='')
    for i in range(4):
        print(f'{final_per_env[i]:<16.4f}', end='')
    print()

You now have empirical evidence that environment-aware conditioning improves intent recognition. Next up, you will visualize exactly what the FiLM generator learned to reveal the per-environment feature strategies it discovered.

Visualize What FiLM Learns

Both models are trained, and the comparison table already shows that FiLM conditioning improves intent accuracy. But numbers alone do not explain how the model adapts.

In this step, you will crack open the trained FiLM generator and visualize exactly what it learned. By extracting the gamma and beta parameters for each environment, you can see which frequency channels the model amplifies or suppresses. This is the interpretability payoff of choosing FiLM over opaque conditioning methods.

In this step, get ready to:
  • Extract and plot per-channel FiLM gamma parameters for all 4 environments.
  • Visualize modulated vs. unmodulated feature maps per environment.
  • Generate a confusion matrix for the environment classifier.
Extract FiLM gamma/beta parameters and plot per-channel scaling

The FiLM generator produces a gamma (scale) and beta (shift) for every feature channel. By extracting these values per environment, you can see the distinct frequency-emphasis strategies the model learned.

  • In a new code cell at the bottom of your notebook, add the visualization section header and extraction logic by pasting this code:
# ---- Visualizations ----

# 1. Extract FiLM gamma/beta per environment
film_model.eval()
env_gammas = []
env_betas = []
with torch.no_grad():
    for env_id, env_name in enumerate(ENV_NAMES):
        # Create a synthetic one-hot-like embedding by averaging val samples of this env
        env_embeddings = []
        for mel_spec, intent_labels, env_labels in val_loader:
            mel_spec = mel_spec.to(device)
            mask = (env_labels == env_id)
            if mask.sum() == 0:
                continue
            features = film_model.encoder(mel_spec[mask])
            _, emb = film_model.env_head(features)
            env_embeddings.append(emb)
        if env_embeddings:
            avg_emb = torch.cat(env_embeddings, dim=0).mean(dim=0, keepdim=True)
            gamma, beta = film_model.film_gen(avg_emb)
            env_gammas.append(gamma.squeeze().cpu().numpy())
            env_betas.append(beta.squeeze().cpu().numpy())
        else:
            env_gammas.append(np.ones(128))
            env_betas.append(np.zeros(128))

What does this code do?

Instead of using a single sample, this code averages the environment embeddings across all validation samples belonging to each environment. This produces a stable, representative embedding per environment.

It then passes each averaged embedding through film_model.film_gen to get the gamma and beta arrays. Gamma values above 1.0 mean the model amplifies that channel for a given environment. Values below 1.0 mean suppression.

  • Do not run this cell yet. You will add the bar chart in the same cell next.
  • Directly below the extraction code in the same cell, add the bar chart plotting code:
# Bar chart: per-channel gamma values across environments
fig, ax = plt.subplots(figsize=(16, 5))
width = 0.2
x = np.arange(128)
colors = ['#e74c3c', '#3498db', '#2ecc71', '#f39c12']
for i, (env_name, gamma) in enumerate(zip(ENV_NAMES, env_gammas)):
    ax.bar(x + i * width, gamma, width=width, label=env_name, color=colors[i], alpha=0.8)
ax.set_xlabel('Feature Channel')
ax.set_ylabel('Gamma (scale factor)')
ax.set_title('FiLM Gamma Parameters per Environment (per channel)')
ax.legend()
ax.axhline(y=1.0, color='black', linestyle='--', alpha=0.4, label='No modulation (gamma=1)')
plt.tight_layout()
plt.show()

What does this chart reveal?

Each colored bar represents one environment's gamma value for a specific feature channel. The dashed line at y=1.0 marks the "no modulation" baseline.

Channels where bars diverge significantly across environments are the ones the model uses to differentiate its strategy. For example, low-frequency channels might be suppressed in car environments (dominated by engine rumble) but amplified in kitchen environments (where speech energy sits above the background).

  • Run the cell. You should see a wide bar chart with 128 channels on the x-axis and four colored bar groups showing how each environment scales each channel differently.

Chart looks flat or all bars near 1.0?

If gamma values cluster around 1.0 with minimal variation, the FiLM generator may not have converged. Check that your film_model completed 15 epochs of training and that film_history['val_intent_acc'] shows improving accuracy across epochs.

Also confirm you are extracting from film_model (not baseline_model), since the baseline has no FiLM generator.

Help me debug my FiLM gamma bar chart showing flat values near 1.0 after training.

Visualize modulated feature maps and the environment confusion matrix

The bar chart shows the raw scaling factors. Now you will visualize what those factors actually do to the feature maps. You will also verify that the environment classifier is correctly detecting environments in the first place.

  • In a new code cell, add the feature map grid visualization by pasting this code:
# 2. 2x2 grid: modulated vs unmodulated feature maps per environment
fig, axes = plt.subplots(4, 3, figsize=(15, 16))
film_model.eval()
with torch.no_grad():
    for env_id, env_name in enumerate(ENV_NAMES):
        for mel_spec, intent_labels, env_labels in val_loader:
            mel_spec = mel_spec.to(device)
            mask = (env_labels == env_id)
            if mask.sum() == 0:
                continue
            sample = mel_spec[mask][:1]
            features = film_model.encoder(sample)
            _, emb = film_model.env_head(features)
            gamma, beta = film_model.film_gen(emb)
            modulated = film_modulate(features, gamma, beta)
            unmod_map = features.mean(dim=1).squeeze().cpu().numpy()
            mod_map = modulated.mean(dim=1).squeeze().cpu().numpy()
            diff_map = mod_map - unmod_map
            axes[env_id][0].imshow(unmod_map, aspect='auto', origin='lower', cmap='viridis')
            axes[env_id][0].set_title(f'{env_name}: Unmodulated')
            axes[env_id][1].imshow(mod_map, aspect='auto', origin='lower', cmap='viridis')
            axes[env_id][1].set_title(f'{env_name}: FiLM Modulated')
            axes[env_id][2].imshow(diff_map, aspect='auto', origin='lower', cmap='RdBu_r')
            axes[env_id][2].set_title(f'{env_name}: Difference')
            break
plt.tight_layout()
plt.show()

What does this grid show?

Each row is one environment (kitchen, car, outdoor, living room). The three columns show the unmodulated feature map, the FiLM-modulated version, and the difference between them.

The difference column (using the RdBu_r colormap) highlights where the model increases activation (red) vs. suppresses it (blue). Distinct patterns across rows prove the model applies environment-specific attention.

  • Run the cell. You should see a 4-row, 3-column grid of heatmaps. Each row shows a different environment, with visible differences in the red/blue difference column across rows.

Seeing a NameError for film_modulate?

The film_modulate function was defined in Step 3 as part of the architecture. Make sure you ran all cells from the top of your notebook in order. If cells were run out of order, click **Runtime**, then **Run all** to re-execute everything sequentially.

Help me fix a NameError when calling film_modulate in my visualization cell.

  • In another new code cell, add the confusion matrix for the environment classifier by pasting this code:
# 3. Confusion matrix for environment classifier
from sklearn.metrics import confusion_matrix
import seaborn as sns
all_env_preds = []
all_env_true = []
film_model.eval()
with torch.no_grad():
    for mel_spec, intent_labels, env_labels in val_loader:
        mel_spec = mel_spec.to(device)
        _, env_logits = film_model(mel_spec)
        env_preds = env_logits.argmax(dim=1).cpu().numpy()
        all_env_preds.extend(env_preds)
        all_env_true.extend(env_labels.numpy())
cm = confusion_matrix(all_env_true, all_env_preds)
fig, ax = plt.subplots(figsize=(7, 6))
sns.heatmap(cm, annot=True, fmt='d', xticklabels=ENV_NAMES, yticklabels=ENV_NAMES, cmap='Blues', ax=ax)
ax.set_xlabel('Predicted Environment')
ax.set_ylabel('True Environment')
ax.set_title('Environment Classifier Confusion Matrix')
plt.tight_layout()
plt.show()
print('Visualizations complete.')

What does the confusion matrix tell you?

The confusion matrix shows how well the environment head distinguishes between the four environments. High values on the diagonal mean the model correctly identifies where the audio was recorded.

If the environment head cannot tell environments apart, the FiLM generator receives poor conditioning. Strong diagonal dominance here is a prerequisite for the gamma/beta parameters to be meaningful.

  • Run the cell. You should see a 4x4 heatmap with environment names on both axes and counts annotated in each cell. The diagonal should show the highest values.
  • You should also see Visualizations complete. printed below the matrix.

Confusion matrix looks random or evenly distributed?

If counts are spread evenly across all cells (no diagonal dominance), the environment classifier did not converge. Check that the environment loss weight (0.3) was applied during training. Also verify that your noise mixing pipeline assigns distinct ESC-50 categories to each environment. If all environments use similar noise types, the classifier cannot distinguish them.

Help me understand why my environment confusion matrix shows poor classification.

✔️ Awesome, I've got everything!

Great. Double check you have all three visualizations showing in your notebook: the gamma bar chart, the feature map grid, and the confusion matrix.

ⓧ I'd like to double check the full code

Here is the complete visualization code that should appear at the bottom of your notebook:

# ---- Visualizations ----

# 1. Extract FiLM gamma/beta per environment
film_model.eval()
env_gammas = []
env_betas = []
with torch.no_grad():
    for env_id, env_name in enumerate(ENV_NAMES):
        # Create a synthetic one-hot-like embedding by averaging val samples of this env
        env_embeddings = []
        for mel_spec, intent_labels, env_labels in val_loader:
            mel_spec = mel_spec.to(device)
            mask = (env_labels == env_id)
            if mask.sum() == 0:
                continue
            features = film_model.encoder(mel_spec[mask])
            _, emb = film_model.env_head(features)
            env_embeddings.append(emb)
        if env_embeddings:
            avg_emb = torch.cat(env_embeddings, dim=0).mean(dim=0, keepdim=True)
            gamma, beta = film_model.film_gen(avg_emb)
            env_gammas.append(gamma.squeeze().cpu().numpy())
            env_betas.append(beta.squeeze().cpu().numpy())
        else:
            env_gammas.append(np.ones(128))
            env_betas.append(np.zeros(128))

# Bar chart: per-channel gamma values across environments
fig, ax = plt.subplots(figsize=(16, 5))
width = 0.2
x = np.arange(128)
colors = ['#e74c3c', '#3498db', '#2ecc71', '#f39c12']
for i, (env_name, gamma) in enumerate(zip(ENV_NAMES, env_gammas)):
    ax.bar(x + i * width, gamma, width=width, label=env_name, color=colors[i], alpha=0.8)
ax.set_xlabel('Feature Channel')
ax.set_ylabel('Gamma (scale factor)')
ax.set_title('FiLM Gamma Parameters per Environment (per channel)')
ax.legend()
ax.axhline(y=1.0, color='black', linestyle='--', alpha=0.4, label='No modulation (gamma=1)')
plt.tight_layout()
plt.show()

# 2. 2x2 grid: modulated vs unmodulated feature maps per environment
fig, axes = plt.subplots(4, 3, figsize=(15, 16))
film_model.eval()
with torch.no_grad():
    for env_id, env_name in enumerate(ENV_NAMES):
        for mel_spec, intent_labels, env_labels in val_loader:
            mel_spec = mel_spec.to(device)
            mask = (env_labels == env_id)
            if mask.sum() == 0:
                continue
            sample = mel_spec[mask][:1]
            features = film_model.encoder(sample)
            _, emb = film_model.env_head(features)
            gamma, beta = film_model.film_gen(emb)
            modulated = film_modulate(features, gamma, beta)
            unmod_map = features.mean(dim=1).squeeze().cpu().numpy()
            mod_map = modulated.mean(dim=1).squeeze().cpu().numpy()
            diff_map = mod_map - unmod_map
            axes[env_id][0].imshow(unmod_map, aspect='auto', origin='lower', cmap='viridis')
            axes[env_id][0].set_title(f'{env_name}: Unmodulated')
            axes[env_id][1].imshow(mod_map, aspect='auto', origin='lower', cmap='viridis')
            axes[env_id][1].set_title(f'{env_name}: FiLM Modulated')
            axes[env_id][2].imshow(diff_map, aspect='auto', origin='lower', cmap='RdBu_r')
            axes[env_id][2].set_title(f'{env_name}: Difference')
            break
plt.tight_layout()
plt.show()

# 3. Confusion matrix for environment classifier
from sklearn.metrics import confusion_matrix
import seaborn as sns
all_env_preds = []
all_env_true = []
film_model.eval()
with torch.no_grad():
    for mel_spec, intent_labels, env_labels in val_loader:
        mel_spec = mel_spec.to(device)
        _, env_logits = film_model(mel_spec)
        env_preds = env_logits.argmax(dim=1).cpu().numpy()
        all_env_preds.extend(env_preds)
        all_env_true.extend(env_labels.numpy())
cm = confusion_matrix(all_env_true, all_env_preds)
fig, ax = plt.subplots(figsize=(7, 6))
sns.heatmap(cm, annot=True, fmt='d', xticklabels=ENV_NAMES, yticklabels=ENV_NAMES, cmap='Blues', ax=ax)
ax.set_xlabel('Predicted Environment')
ax.set_ylabel('True Environment')
ax.set_title('Environment Classifier Confusion Matrix')
plt.tight_layout()
plt.show()
print('Visualizations complete.')

You now have concrete, interpretable evidence of what your FiLM-conditioned model learned. The gamma bar chart shows per-environment frequency strategies. The feature map grid shows the spatial effect of modulation. The confusion matrix confirms the environment head accurately detects where audio originates. This is the interpretability advantage that makes FiLM compelling for real-world deployment.

Secret mission

SNR-Aware Conditioning

Your FiLM model knows WHERE the noise comes from, but it treats a whisper of rain the same as a downpour. What if the model could also sense HOW LOUD the noise is and adjust its strategy accordingly? Extend your architecture with an SNR estimation branch that feeds noise intensity into the FiLM generator, proving that jointly modeling noise type AND intensity yields further gains at extreme noise levels.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely in Google Colab, so there are no ongoing costs or paid resources to worry about. Decide whether to keep your Colab runtime active, disconnect it, or delete everything entirely.

Resources you used:

  • Google Colab notebook (notebook.ipynb saved in Google Drive).
  • Fluent Speech Commands dataset (~2.4 GB downloaded to Colab VM storage).
  • ESC-50 dataset (~600 MB downloaded to Colab VM storage).
  • Trained model weights (stored in Colab VM memory).

Keep everything running

No action needed. Choose this if you're still actively experimenting with FiLM parameters or extending the model.

  • Your Colab runtime stays active for up to 12 hours of idle time.
  • Downloaded datasets remain in the VM's ephemeral storage as long as the runtime is connected.
  • Your notebook file is auto-saved to Google Drive.

Pause - I'll come back to this later

Disconnect the Colab runtime to free up GPU resources. Your notebook remains saved in Google Drive, but downloaded datasets and trained models in VM memory will be lost.

  • In the Colab toolbar, click Runtime.
  • Click Disconnect and delete runtime.
  • When you return, you will need to re-run all cells to re-download the datasets and retrain the models.

Delete - I don't want to use this again

Remove all project resources completely. This deletes the notebook from Google Drive and disconnects the runtime.

  • In the Colab toolbar, click Runtime, then Disconnect and delete runtime. This removes the VM along with all downloaded datasets and trained model weights.
  • Open Google Drive at drive.google.com.
  • Locate your notebook.ipynb file (usually in the Colab Notebooks folder).
  • Right-click the file and select Move to trash.
  • Empty your Google Drive trash to permanently delete the file.

Nice Work!

Nice Work!

You just built an environment-adaptive spoken language understanding system from scratch in PyTorch. Your model doesn't just tolerate noise. It understands the noise and adapts its recognition strategy accordingly.

You've learned how to:

  • Build a multi-task PyTorch model that jointly classifies acoustic environment and recognizes speech intent using shared encoder features and auxiliary classification heads.
  • Implement FiLM (Feature-wise Linear Modulation) conditioning to dynamically adapt feature representations based on detected environment, proving that noise-aware modulation outperforms blanket noise suppression.
  • Create an on-the-fly noise mixing pipeline combining Fluent Speech Commands with ESC-50 environment sounds at variable SNRs, and visualize interpretable FiLM parameters revealing per-environment frequency strategies.
  • Secret Mission: Extended to SNR-aware conditioning showing that jointly modeling noise type AND intensity yields further gains at extreme noise levels.

Ready to quiz yourself?