Multimodal Student Distress Detector

Fuse facial expressions and text sentiment to detect masked student distress.

Introduction

30 Second Summary

When a student types "I'm fine" in a chat, their face might tell a completely different story. Detecting this kind of emotional masking is one of the hardest challenges in student wellbeing systems.

In this project, you will build a multimodal distress detection pipeline that combines facial expression analysis with text sentiment analysis. A Gated Multimodal Unit learns to dynamically weight each signal, catching masked emotions that either modality alone would miss.

What You'll Build

You feed in a student's video clip and text message, and the pipeline tells you whether they are genuinely fine or quietly struggling.

By the end of this project, you'll have:

  • A working extraction pipeline that processes your video dataset with MediaPipe and text messages with VADER, producing 52 facial blendshape scores and 4 sentiment scores you can inspect for each sample.
  • A trained GMU classifier in PyTorch that catches masked emotions by dynamically weighing face data against text data for each student, even when face data is completely missing.
  • A side-by-side accuracy comparison with gate analysis showing the model automatically shifting toward text when face data is missing and toward face when detecting masked emotions.
  • Secret Mission: Add stratified k-fold cross-validation to produce robust accuracy estimates with standard deviations for your university report.

Do I need a labeled dataset?

This project assumes you have 80-100 labeled video and text samples prepared in a CSV format with separate video and text distress labels. You also need Python and a code editor (like VS Code) installed on your machine.

Before We Start

Before we write any code, let's lock in what you're about to build and why it matters.

Set Up Your Environment and Download the Model

Before your pipeline can extract facial features or train any models, it needs the right foundation. This step sets up an isolated Python environment with every package your multimodal detector requires.

You will install MediaPipe for facial expression analysis, VADER for text sentiment scoring, PyTorch for building the neural network, and scikit-learn for evaluation. You will also download the pre-trained face landmark model and organize your labeled dataset.

In this step, get ready to:
  • Install MediaPipe, VADER, PyTorch, scikit-learn, and matplotlib in a virtual environment.
  • Download the MediaPipe FaceLandmarker model file.
  • Organize your labeled video and text dataset into the expected folder structure.
Create a virtual environment and install packages

A virtual environment keeps this project's packages separate from your system Python. This prevents version conflicts with other projects on your machine.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Move to your Desktop and create your project folder by running these commands:
cd ~/Desktop
mkdir distress-detector
cd distress-detector
  • Create your virtual environment by running:
python -m venv venv
  • Confirm the virtual environment was created by running:
ls

You should see a venv folder listed in the output.

Seeing command not found for python?

  • On macOS, the command is often python3 instead of python. Try running:
python3 -m venv venv

help me fix a python command not found error

  • Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

Your terminal prompt now starts with (venv), confirming the virtual environment is active.

  • Install all required packages by running:
pip install mediapipe==1.0.1 vaderSentiment==3.3.2 torch scikit-learn matplotlib

What do these packages do?

  • mediapipe extracts 52 facial blendshape scores from video frames.
  • vaderSentiment scores text sentiment (negative, neutral, positive, compound) without needing any training data.
  • torch provides the neural network framework you will use to build and train the GMU classifier.
  • scikit-learn handles train/test splitting and evaluation metrics like accuracy, precision, and recall.
  • matplotlib generates the training loss plot you will save at the end of the project.

Seeing an OpenCV conflict error?

mediapipe depends on opencv-contrib-python, which conflicts with opencv-python if both are installed. Uninstall the conflicting package first, then retry the install:

pip uninstall opencv-python -y
pip install mediapipe==1.0.1

help me fix an OpenCV conflict when installing mediapipe

Before you run the next command, what do you expect to see if all four packages installed correctly?

  • Verify all packages imported successfully by running:
python -c "import mediapipe; import torch; from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer; print('mediapipe', mediapipe.__version__); print('torch', torch.__version__); print('All imports OK')"

You should see version numbers for mediapipe and torch, followed by All imports OK with no errors.

Seeing a ModuleNotFoundError?

  • Make sure your terminal prompt shows (venv). If it does not, you need to activate the virtual environment again.
  • PyTorch requires Python 3.10 or later. Run python --version to check. If your version is older, install Python 3.10+ before continuing.

help me troubleshoot a ModuleNotFoundError after pip install

Download the MediaPipe FaceLandmarker model bundle

The FaceLandmarker model detects faces in video frames and outputs 52 blendshape scores per face. You need to download this model file separately from the MediaPipe package.

  • Download the face_landmarker.task model file into your project root by running:

macOS

curl -o face_landmarker.task https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/latest/face_landmarker.task

Windows

curl.exe -o face_landmarker.task https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/latest/face_landmarker.task

Why curl.exe on Windows?

Windows PowerShell 5.1 maps curl to a different command called Invoke-WebRequest. Using curl.exe calls the real curl binary directly.

  • Confirm the model file was downloaded by running:
ls face_landmarker.task

You should see face_landmarker.task listed. This file contains the face detection, face mesh, and blendshape models bundled together.

File not found?

  • If the download failed, open the model download page directly in your browser. Your browser will download the file automatically.
  • Move the downloaded file from your Downloads folder into the distress-detector folder on your Desktop.

help me download the face_landmarker.task model file

Prepare your data folder structure

Your feature extraction script expects video files and a CSV of labels in a specific folder layout. This substep sets that up using the labeled samples you have already prepared.

  • Create the data/ folder with a videos/ subfolder by running:

macOS

mkdir -p data/videos

Windows

mkdir data\videos
  • Copy your video files into the data/videos/ folder.
  • Place your labels.csv file inside the data/ folder so its path is data/labels.csv.
  • Confirm the folder structure by running:
ls data/

You should see labels.csv and a videos folder listed.

What should labels.csv look like?

Your CSV needs four columns: video_file, label, text, text_label.

  • video_file is the filename of each video in the data/videos/ folder.
  • label is the facial/video distress label (0 for non-distress, 1 for distress).
  • text is the student's text message for that sample.
  • text_label is the text distress label (0 or 1). This can differ from label when a student is masking their emotions.

Before you run the next check, what do you expect to see if your CSV is correctly formatted with all four columns?

  • Verify your CSV structure by running:
python -c "import csv; rows = list(csv.DictReader(open('data/labels.csv'))); print(f'{len(rows)} samples loaded'); print('Columns:', list(rows[0].keys()))"

You should see your sample count printed along with the four column names: video_file, label, text, text_label.

Seeing a FileNotFoundError?

  • Check that labels.csv is directly inside the data/ folder, not nested inside data/videos/.
  • Make sure your terminal is in the distress-detector folder. Run pwd to confirm you see distress-detector at the end of the path.
  • If your columns do not match, open labels.csv in your code editor and check that the first line reads video_file,label,text,text_label with no extra spaces.

help me fix my labels.csv file format

Your environment is locked in, the model is downloaded, and your data is organized. Next up, you will write the feature extraction script that turns raw videos and text into the numerical vectors your neural network needs.

Extract Facial and Text Features from Your Dataset

A neural network cannot read a raw video file or parse a sentence of text directly. It needs fixed-size numeric vectors as input.

In this step, you will convert each video into a 52-dimensional facial blendshape vector using MediaPipe FaceLandmarker, and each text message into a 4-dimensional sentiment vector using VADER. When no face is detected in a video (for example, a student wearing a mask), the facial vector becomes all zeros so the model can learn to rely on text instead.

In this step, get ready to:
  • Extract facial blendshape features from video files using MediaPipe FaceLandmarker.
  • Extract text sentiment features using VADER.
  • Save all features as NumPy arrays for model training.
Write the feature extraction functions

Each video needs to become a single row of numbers that captures the student's facial expression. MediaPipe FaceLandmarker outputs 52 blendshape scores per frame (values like browDownLeft, eyeBlinkRight, mouthFrownLeft) that directly encode facial expressions.

  • Create a new file called extract_features.py in your project root directory.
  • Add the following imports and constant to the top of the file:
import os
import csv
import numpy as np
import cv2
import mediapipe as mp
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

# Number of facial blendshape scores per frame
NUM_BLENDSHAPES = 52

What are these imports?

  • cv2 (OpenCV) reads video files frame by frame.
  • mediapipe provides the FaceLandmarker that detects facial expressions as blendshape scores.
  • SentimentIntensityAnalyzer from VADER gives you four sentiment scores for any piece of text.
  • NUM_BLENDSHAPES = 52 is the number of blendshape categories MediaPipe returns per face.
  • Below NUM_BLENDSHAPES = 52, add the extract_facial_features function:
def extract_facial_features(video_path, landmarker):
    # Open the video and get the frame rate
    cap = cv2.VideoCapture(video_path)
    fps = cap.get(cv2.CAP_PROP_FPS)
    if fps == 0:
        fps = 30.0
    all_blendshapes = []
    frame_idx = 0
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        # Convert BGR to RGB and wrap as a MediaPipe Image
        rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb_frame)
        # Timestamps must increase monotonically for VIDEO mode
        timestamp_ms = int(frame_idx * 1000 / fps)
        result = landmarker.detect_for_video(mp_image, timestamp_ms)
        # Collect the 52 blendshape scores when a face is found
        if result.face_blendshapes and len(result.face_blendshapes) > 0:
            scores = [bs.score for bs in result.face_blendshapes[0]]
            all_blendshapes.append(scores)
        frame_idx += 1
    cap.release()
    # Return mean vector across all frames, or None if no face detected
    if len(all_blendshapes) == 0:
        return None
    return np.mean(all_blendshapes, axis=0)

What does this function do?

It loops through every frame of a video, converts each frame to RGB, and passes it to the MediaPipe FaceLandmarker. For each frame where a face is detected, it collects the 52 blendshape scores.

At the end, it averages all the per-frame scores into a single 52-dimensional vector representing the student's overall expression. If no face was detected in any frame, it returns None so the caller knows to use a zero vector instead.

  • Save extract_features.py.
  • Confirm the module loads by running:
python -c "from extract_features import extract_facial_features; print('Facial extraction function loaded')"

You should see Facial extraction function loaded with no errors.

Seeing an ImportError?

  • Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt.
  • Check that extract_features.py is saved in your project root directory, not inside data/ or another subfolder.
  • help me debug an ImportError when loading extract_features

Now you need a companion function for the text side. VADER's polarity_scores method returns four scores: negative, neutral, positive, and a compound score that combines all three.

  • Add the extract_text_features function below extract_facial_features:
def extract_text_features(text, analyzer):
    # Get the four VADER sentiment scores
    scores = analyzer.polarity_scores(text)
    return [scores['neg'], scores['neu'], scores['pos'], scores['compound']]
  • Save extract_features.py.
  • Test the VADER function on a sample sentence by running:
python -c "from extract_features import extract_text_features; from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer; a = SentimentIntensityAnalyzer(); print(extract_text_features('I am feeling great today', a))"

You should see a list of four float values like [0.0, 0.367, 0.633, 0.6588]. The exact numbers may vary slightly, but you should see four values between -1 and 1.

Not seeing four values?

  • Make sure you placed extract_text_features below the extract_facial_features function, not inside it. Check the indentation is at the module level (no leading spaces on the def line).
  • help me debug my extract_text_features function
Build the main processing loop

The two extraction functions are ready. Now you need a main() function that initializes both tools, reads your labels.csv, and runs each sample through both extraction functions.

  • Below extract_text_features, add the start of the main() function with the MediaPipe and VADER initialization:
def main():
    # Set up MediaPipe FaceLandmarker for VIDEO mode
    BaseOptions = mp.tasks.BaseOptions
    FaceLandmarker = mp.tasks.vision.FaceLandmarker
    FaceLandmarkerOptions = mp.tasks.vision.FaceLandmarkerOptions
    VisionRunningMode = mp.tasks.vision.RunningMode

    options = FaceLandmarkerOptions(
        base_options=BaseOptions(model_asset_path='face_landmarker.task'),
        running_mode=VisionRunningMode.VIDEO,
        output_face_blendshapes=True,
        num_faces=1,
    )
    landmarker = FaceLandmarker.create_from_options(options)

    # Initialize VADER sentiment analyzer
    analyzer = SentimentIntensityAnalyzer()

    # Collect features and labels for every sample
    facial_features_list = []
    text_features_list = []
    combined_labels_list = []
    face_available_list = []

What do these options mean?

  • running_mode=VisionRunningMode.VIDEO tells MediaPipe to expect sequential frames with increasing timestamps.
  • output_face_blendshapes=True enables the 52 blendshape scores you need for expression analysis.
  • num_faces=1 limits detection to one face per frame, which is all you need for a single-student video.

Next, you need the loop that reads each row from labels.csv and runs both extraction functions. For each sample, the combined label is computed as max(video_label, text_label) so distress is flagged if either modality indicates it.

Why use max() for the combined label?

Your dataset has separate distress labels for video and text because students can mask their emotions. A student might type "I'm fine" (text label 0) while their face shows distress (video label 1).

Taking the maximum means: if either channel signals distress, the student IS in distress. This is exactly the kind of masked emotion the GMU needs to learn to detect.

  • Add the following CSV processing loop inside main(), directly below the face_available_list = [] line:
    with open('data/labels.csv', 'r') as f:
        reader = csv.DictReader(f)
        for row in reader:
            video_path = os.path.join('data', 'videos', row['video_file'])
            text = row['text']
            video_label = int(row['label'])
            text_label = int(row['text_label'])

            # Flag distress if EITHER modality indicates it
            combined_label = max(video_label, text_label)

            print(f"Processing {row['video_file']}...")
            facial = extract_facial_features(video_path, landmarker)

            # Use zero vector when no face is detected
            if facial is None:
                print(f"  No face detected, using zero vector")
                facial = np.zeros(NUM_BLENDSHAPES)
                face_available_list.append(False)
            else:
                face_available_list.append(True)

            text_feat = extract_text_features(text, analyzer)

            facial_features_list.append(facial)
            text_features_list.append(text_feat)
            combined_labels_list.append(combined_label)

    landmarker.close()

What happens when no face is detected?

When extract_facial_features returns None, the code substitutes a zero vector of 52 dimensions. This is not a throwaway placeholder.

The GMU will learn that when all 52 facial values are zero, the face modality is uninformative. The gate will shift its weight toward the text modality for these samples.

  • Save extract_features.py.
  • Confirm the full function loads without syntax errors by running:
python -c "from extract_features import main; print('main() ready')"

You should see main() ready printed with no errors.

Seeing a SyntaxError or IndentationError?

  • The CSV loop must be indented inside main(). Every line should start with at least 4 spaces.
  • Make sure the loop code starts directly below face_available_list = [] with no blank def or return statement between them.
  • help me fix a SyntaxError in my extract_features.py main function
Save features and run the full extraction

The loop processes every sample, but the results are still in memory. You need to save the feature arrays to disk as .npy files so the training scripts in later steps can load them.

  • Add the following code below landmarker.close(), still inside main(). Then add the if __name__ block at the bottom of the file (not indented):
    # Save all feature arrays to disk
    np.save('data/facial_features.npy', np.array(facial_features_list))
    np.save('data/text_features.npy', np.array(text_features_list))
    np.save('data/combined_labels.npy', np.array(combined_labels_list))
    np.save('data/face_available.npy', np.array(face_available_list))

    # Print extraction summary
    num_face_available = sum(face_available_list)
    print(f"\nExtracted features for {len(combined_labels_list)} samples")
    print(f"Facial features shape: {np.array(facial_features_list).shape}")
    print(f"Text features shape: {np.array(text_features_list).shape}")
    print(
        f"Face available: {num_face_available}/{len(face_available_list)}"
    )
    print(
        f"Distress: {sum(combined_labels_list)}, "
        f"Non-distress: {len(combined_labels_list) - sum(combined_labels_list)}"
    )


if __name__ == '__main__':
    main()

What does this code do?

  • np.save() writes each list as a NumPy array to a .npy file, which the training scripts will load directly.
  • The summary prints let you verify the shapes, how many samples had visible faces, and how your distress vs. non-distress labels are distributed.
  • if __name__ == '__main__' ensures main() only runs when you execute the script directly, not when importing functions from it.
  • Save extract_features.py.

Before you run this, think about your dataset. How many of your samples do you expect to have a visible face? Do any of your videos show a student wearing a mask or facing away from the camera?

  • Run the feature extraction script:
python extract_features.py

You should see a "Processing..." line for each video in your dataset, followed by a summary that looks something like this:

Extracted features for 85 samples
Facial features shape: (85, 52)
Text features shape: (85, 4)
Face available: 78/85
Distress: 42, Non-distress: 43

What should I see?

Your exact numbers will differ based on your dataset size. The key things to check:

  • The sample count matches the number of rows in your labels.csv.
  • Facial features shape is (N, 52) where N is your sample count.
  • Text features shape is (N, 4) (the four VADER scores).
  • Face available count is less than or equal to total samples. Any videos where no face was detected show "No face detected, using zero vector" during processing.
  • Confirm the four .npy files were saved by running:
ls data/*.npy

You should see combined_labels.npy, face_available.npy, facial_features.npy, and text_features.npy listed.

Script failing with a file not found error?

  • Check that face_landmarker.task is in your project root directory (the same folder as extract_features.py), not inside data/.
  • Verify that data/labels.csv exists and that the video_file column values match the actual filenames in data/videos/.
  • If you see an OpenCV error about a codec or unreadable frame, make sure your video files are in a standard format like .mp4 or .avi.
  • help me debug my extract_features.py script

✔️ Awesome, I've got everything!

Great. Double check you have saved extract_features.py and that all four .npy files appear in your data/ folder.

ⓧ I'd like to double check the full code

Your extract_features.py should contain these sections in order. Scroll up and confirm each one is present:

  • Six imports (os, csv, numpy, cv2, mediapipe, SentimentIntensityAnalyzer) and the NUM_BLENDSHAPES = 52 constant.
  • extract_facial_features(video_path, landmarker) function that reads video frames, collects blendshape scores, and returns the mean vector or None.
  • extract_text_features(text, analyzer) function that returns the four VADER scores as a list.
  • main() function that initializes FaceLandmarker and VADER, loops through labels.csv, computes combined labels with max(), handles missing faces with zero vectors, and saves four .npy files.
  • if __name__ == '__main__': main() at the bottom of the file.

Your raw videos and text messages are now numeric feature vectors ready for training. Next, you will train a text-only baseline classifier to see how far sentiment analysis alone can get before adding the facial modality.

Train a Text-Only Baseline Classifier

You have your feature files extracted and saved. Now it is time to answer a critical question: how well can text sentiment alone detect student distress?

Before building anything multimodal, you need a baseline to compare against. In this step, you will train a simple classifier that uses only the four VADER sentiment scores. The results will reveal a real limitation: text sentiment alone misses masked emotions, where a student types something neutral but their face tells a different story.

In this step, get ready to:
  • Define a TextOnlyClassifier neural network in model.py.
  • Write a training script that loads your extracted features, splits the data, and trains the classifier.
  • Evaluate the text-only accuracy and see where single-modality detection falls short.
Define the text-only classifier in model.py

The classifier is a small feedforward neural network built with PyTorch. It takes in four VADER sentiment scores and outputs a single probability between 0 (no distress) and 1 (distress).

  • Create a new file called model.py in your project root and add the following code:
import torch
import torch.nn as nn


# Simple feedforward network for text-only classification
class TextOnlyClassifier(nn.Module):
    def __init__(self, input_dim=4, hidden_dim=32):
        super().__init__()
        # 4 VADER scores in, 32 hidden units
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.dropout = nn.Dropout(0.3)
        # Single output for binary classification
        self.fc2 = nn.Linear(hidden_dim, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout(x)
        # Sigmoid squashes output to a probability between 0 and 1
        x = torch.sigmoid(self.fc2(x))
        return x

What does this code do?

This defines a PyTorch neural network with one hidden layer. The architecture is intentionally small to avoid overfitting on your 80-100 samples.

  • fc1 takes 4 inputs (the VADER neg, neu, pos, compound scores) and maps them to 32 hidden units.
  • Dropout(0.3) randomly zeroes out 30% of neurons during training to prevent overfitting.
  • fc2 outputs a single value, and sigmoid squashes it to a 0-1 probability for binary classification.
  • Save model.py.
  • Verify the class imports correctly by running this command:
python -c "from model import TextOnlyClassifier; print('TextOnlyClassifier imported successfully')"

You should see TextOnlyClassifier imported successfully printed with no errors.

Seeing a ModuleNotFoundError?

  • Make sure you saved the file as model.py (not models.py or Model.py) in your project root directory.
  • Check that your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.

Still stuck? Help me debug why my TextOnlyClassifier import is failing in model.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved model.py.

ⓧ I'd like to double check the full code

Here is the complete model.py so far. Make sure yours matches:

import torch
import torch.nn as nn


class TextOnlyClassifier(nn.Module):
    def __init__(self, input_dim=4, hidden_dim=32):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.dropout = nn.Dropout(0.3)
        self.fc2 = nn.Linear(hidden_dim, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout(x)
        x = torch.sigmoid(self.fc2(x))
        return x
Set up the training script with data loading and model initialization

The training script loads your extracted .npy feature files, splits them into training and test sets, and prepares the PyTorch data pipeline.

  • Create a new file called train_baseline.py in your project root and add the following code:
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from model import TextOnlyClassifier


def main():
    # Load text features and labels from the extraction step
    text_features = np.load('data/text_features.npy')
    labels = np.load('data/combined_labels.npy')

    # 80/20 stratified split preserves class distribution
    X_train, X_test, y_train, y_test = train_test_split(
        text_features, labels, test_size=0.2, random_state=42, stratify=labels
    )

    # Convert numpy arrays to PyTorch tensors
    X_train_t = torch.FloatTensor(X_train)
    y_train_t = torch.FloatTensor(y_train).unsqueeze(1)
    X_test_t = torch.FloatTensor(X_test)

    train_dataset = TensorDataset(X_train_t, y_train_t)
    train_loader = DataLoader(train_dataset, batch_size=16, shuffle=True)

    # Initialize model, loss function, and optimizer
    model = TextOnlyClassifier(input_dim=4, hidden_dim=32)
    criterion = nn.BCELoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

What does this code do?

  • train_test_split with stratify=labels ensures the train and test sets have the same ratio of distress to non-distress samples.
  • random_state=42 locks the split so you get the same train/test sets every time. This is critical for a fair comparison with the multimodal model later.
  • BCELoss (Binary Cross-Entropy Loss) is the standard loss for binary classification with sigmoid outputs.
  • Still in train_baseline.py, add the training loop directly below the optimizer line. Make sure this code stays inside the main() function (indented one level):
    # Train for 100 epochs, logging loss every 20
    print("Training text-only baseline...")
    for epoch in range(100):
        model.train()
        epoch_loss = 0
        for batch_X, batch_y in train_loader:
            optimizer.zero_grad()
            outputs = model(batch_X)
            loss = criterion(outputs, batch_y)
            loss.backward()
            optimizer.step()
            epoch_loss += loss.item()

        if (epoch + 1) % 20 == 0:
            avg_loss = epoch_loss / len(train_loader)
            print(f"Epoch {epoch + 1}/100, Loss: {avg_loss:.4f}")

What does the training loop do?

  • Each epoch passes every training sample through the network in batches of 16.
  • optimizer.zero_grad() clears the previous gradients, loss.backward() computes new ones, and optimizer.step() updates the weights.
  • The loss is printed every 20 epochs so you can confirm it is decreasing over time.
  • Save train_baseline.py.
  • Confirm the syntax is valid by running this command:
python -c "import train_baseline; print('Syntax OK')"

You should see Syntax OK printed. This confirms the imports resolve and the function body is well-formed so far.

Seeing a SyntaxError or IndentationError?

  • Make sure the training loop code is indented inside def main():. Every line should be indented by 4 spaces from the function definition.
  • Check that the training loop is placed directly after the optimizer = torch.optim.Adam(...) line, not outside the function.

Need more help? Help me fix a syntax or indentation error in my train_baseline.py training script.

Add evaluation and run the baseline

After training, the script needs to evaluate how well the model performs on the held-out test set. This is where you will see the text-only accuracy number that the multimodal model must beat.

  • In train_baseline.py, add the following evaluation code directly below the training loop (still inside main()), followed by the if __name__ guard at the bottom of the file:
    # Evaluate on the held-out test set
    model.eval()
    with torch.no_grad():
        predictions = model(X_test_t)
        predicted_labels = (predictions >= 0.5).int().numpy().flatten()

    true_labels = y_test.astype(int)
    accuracy = accuracy_score(true_labels, predicted_labels)

    print(f"\nText-Only Baseline Accuracy: {accuracy:.4f}")
    print("\nClassification Report:")
    print(
        classification_report(
            true_labels, predicted_labels,
            target_names=['Non-distress', 'Distress'],
        )
    )
    print("Confusion Matrix:")
    print(confusion_matrix(true_labels, predicted_labels))


if __name__ == '__main__':
    main()

What does the evaluation code do?

  • model.eval() switches the model out of training mode, disabling dropout so predictions are deterministic.
  • torch.no_grad() tells PyTorch not to track gradients during evaluation, saving memory.
  • classification_report prints precision, recall, and F1 for both the Non-distress and Distress classes, giving you a complete picture beyond just accuracy.
  • Save train_baseline.py.

Before you run this, think about it: do you expect text sentiment alone to catch every case of student distress? What about a student who types "I'm fine" but is visibly upset on camera?

  • Run the baseline training script:
python train_baseline.py

You should see training loss printed every 20 epochs, followed by the text-only baseline accuracy, a classification report with precision, recall, and F1 for both classes, and a confusion matrix.

Why is the accuracy limited?

The text-only classifier can only see VADER sentiment scores. It has no way to detect masked emotions, where a student writes neutral or positive text but their facial expression signals distress.

Look at your confusion matrix. Any distressed samples misclassified as non-distress are likely masked emotion cases that text alone cannot catch. This is exactly the gap the multimodal GMU will close in the next steps.

Seeing a FileNotFoundError for the .npy files?

  • Make sure you are running the script from your project root directory, not from inside the data/ folder. The script expects data/text_features.npy as a relative path.
  • Confirm the feature files exist by running ls data/. You should see text_features.npy and combined_labels.npy listed. If they are missing, re-run python extract_features.py from the previous step.

Still stuck? Help me debug why train_baseline.py cannot find my .npy feature files.

✔️ Awesome, I've got everything!

Great. Double check you have saved both model.py and train_baseline.py.

ⓧ I'd like to double check the full code

Your model.py should contain the TextOnlyClassifier class. Scroll up to the model.py chunk and confirm it matches.

For train_baseline.py, confirm these three sections appear in order inside your main() function:

  • Data loading and splitting (imports, np.load, train_test_split, tensor conversion, DataLoader, model/criterion/optimizer setup).
  • Training loop (100 epochs with loss printing every 20).
  • Evaluation (model.eval(), predictions, accuracy, classification report, confusion matrix), followed by if __name__ == '__main__': main() at the bottom of the file.

You have your baseline number. The text-only classifier does a reasonable job on straightforward cases, but it is blind to what the face reveals. Next up, you will build the Gated Multimodal Unit that fuses both modalities and learns to trust the right signal for each student.

Build the Gated Multimodal Unit

Your text-only baseline is trained, and the classification report reveals the gap. Text sentiment alone cannot catch a student who types "I'm fine" while their face shows distress.

To close that gap, you need a model that takes both modalities as input and learns which one to trust for each sample. A Gated Multimodal Unit (GMU) does exactly this. It uses a learned gate to dynamically weight facial features versus text features. In this step, you'll implement the GMU from scratch in PyTorch and wrap it in a classifier.

In this step, get ready to:
  • Add a GatedMultimodalUnit class that learns to dynamically weight facial and text features.
  • Add a MultimodalDistressClassifier class that wraps the GMU with a classification head.
  • Verify the full model's forward pass with random test tensors.
Add the GatedMultimodalUnit class

The GMU fuses two modalities using three learnable transformations and a gate. Each modality is projected into a shared hidden space, and a sigmoid gate decides how much to rely on each one.

  • In model.py, add the GatedMultimodalUnit class below the TextOnlyClassifier class by copying this code:
class GatedMultimodalUnit(nn.Module):
    def __init__(self, visual_dim, text_dim, hidden_dim):
        super().__init__()
        # Project each modality into a shared hidden space
        self.visual_transform = nn.Linear(visual_dim, hidden_dim)
        self.text_transform = nn.Linear(text_dim, hidden_dim)
        # Gate takes concatenated raw inputs to decide modality weighting
        self.gate = nn.Linear(visual_dim + text_dim, hidden_dim)

    def forward(self, x_visual, x_text):
        h_v = torch.tanh(self.visual_transform(x_visual))
        h_t = torch.tanh(self.text_transform(x_text))
        # z near 1 = trust visual, z near 0 = trust text
        z = torch.sigmoid(self.gate(torch.cat([x_visual, x_text], dim=1)))
        h = z * h_v + (1 - z) * h_t
        return h

What does this code do?

The GMU implements four equations from the Arevalo et al. (2017) paper:

  • h_v = tanh(W_v * x_v) transforms the 52-dimensional facial blendshape vector into the hidden space.
  • h_t = tanh(W_t * x_t) transforms the 4-dimensional VADER sentiment vector into the same hidden space.
  • z = sigmoid(W_z * [x_v; x_t]) is the gate. It looks at both raw inputs and outputs a value between 0 and 1 for each hidden dimension.
  • h = z * h_v + (1-z) * h_t blends the two modalities. When z is close to 1, the model trusts the face. When z is close to 0, it trusts the text.

Why a gate instead of simple concatenation?

Simple concatenation treats both modalities equally for every sample. The GMU's sigmoid gate adapts per sample. When facial features are all zeros (face unavailable), the gate learns to shift weight toward text automatically.

This is critical for your use case. A student wearing a mask has no facial data, so the model must rely entirely on text sentiment for that sample while still trusting the face for other students.

  • Save model.py.
  • Verify the GMU class works on its own by running this command:
python -c "import torch; from model import GatedMultimodalUnit; gmu = GatedMultimodalUnit(52, 4, 32); out = gmu(torch.randn(1,52), torch.randn(1,4)); print('GMU output shape:', out.shape)"

You should see GMU output shape: torch.Size([1, 32]) printed. The 52-dimensional visual input and 4-dimensional text input have been fused into a single 32-dimensional hidden vector.

Seeing an ImportError?

  • Make sure you saved model.py before running the command.
  • Check that the GatedMultimodalUnit class is placed after the TextOnlyClassifier class, with a blank line separating them.
  • Confirm your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.

Why can't Python find my GatedMultimodalUnit class?

Add the MultimodalDistressClassifier class

The GMU produces a fused hidden vector, but you still need a classification head to turn that vector into a distress probability. The MultimodalDistressClassifier wraps the GMU and adds a small feedforward network on top.

  • Still in model.py, add the MultimodalDistressClassifier class below the GatedMultimodalUnit class:
class MultimodalDistressClassifier(nn.Module):
    def __init__(self, visual_dim=52, text_dim=4, hidden_dim=32):
        super().__init__()
        # Compose the GMU for modality fusion
        self.gmu = GatedMultimodalUnit(visual_dim, text_dim, hidden_dim)
        # Classification head: fused vector -> distress probability
        self.classifier = nn.Sequential(
            nn.Linear(hidden_dim, 16),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(16, 1),
            nn.Sigmoid(),
        )

    def forward(self, x_visual, x_text):
        fused = self.gmu(x_visual, x_text)
        return self.classifier(fused)

What does this code do?

  • self.gmu creates an instance of the GatedMultimodalUnit you just wrote. It handles the fusion of facial and text features.
  • self.classifier is a small feedforward network. It takes the 32-dimensional fused vector, compresses it to 16 dimensions, applies dropout for regularization, and outputs a single probability between 0 and 1.
  • The forward() method takes separate visual and text tensors, fuses them through the GMU, and passes the result through the classifier head.
  • Save model.py.
Verify the full model with random tensors

Before you run this, think about what the output shape should be. The model takes a 52-dimensional visual vector and a 4-dimensional text vector for one sample. What shape should the output be, and what range of values would you expect?

  • Test the full MultimodalDistressClassifier by running this command:
python -c "import torch; from model import MultimodalDistressClassifier; m = MultimodalDistressClassifier(); out = m(torch.randn(1,52), torch.randn(1,4)); print('Output shape:', out.shape, 'Value:', out.item())"

You should see Output shape: torch.Size([1, 1]) followed by a Value: between 0 and 1. That single probability is the model's distress prediction for one student. The exact value will differ each run because the weights are random.

Seeing a shape mismatch error?

  • Check that the gate layer in GatedMultimodalUnit uses visual_dim + text_dim as its input dimension (56 total), not just one of them.
  • Verify the torch.cat call uses dim=1 to concatenate along the feature dimension, not dim=0.
  • Make sure MultimodalDistressClassifier defaults match the feature sizes: visual_dim=52 and text_dim=4.

Why am I getting a RuntimeError about tensor sizes in my GMU forward pass?

✔️ Awesome, I've got everything!

Great. Make sure you've saved model.py with all three classes.

ⓧ I'd like to double check the full code

Your model.py is now over 30 lines, so confirm these three classes appear in order by scrolling through the file:

  • The import torch and import torch.nn as nn lines at the top.
  • TextOnlyClassifier (from Step 3): __init__ with fc1, dropout, fc2, and a forward() method.
  • GatedMultimodalUnit (this step): visual_transform, text_transform, gate, and a forward() method with h_v, h_t, z, and h.
  • MultimodalDistressClassifier (this step): self.gmu and self.classifier with a nn.Sequential block ending in Sigmoid().

Your multimodal architecture is built and verified. Next up, you'll train this classifier on your real dataset and see whether fusing both modalities through the GMU actually outperforms the text-only baseline.

Train and Evaluate the Multimodal Classifier

In Step 4, you built the GatedMultimodalUnit and MultimodalDistressClassifier classes and verified they produce the right output shape. The architecture is ready, but the real question remains: does fusing facial expressions with text sentiment actually catch distress that text alone misses?

This step is the payoff. You will train the multimodal classifier on the same data split as your text-only baseline, compare the two models side by side, and then inspect the GMU gate values to see whether the model learned to shift toward text when face data is unavailable.

In this step, get ready to:
  • Train the MultimodalDistressClassifier on both facial and text features.
  • Compare multimodal accuracy against the text-only baseline.
  • Analyze GMU gate values to see how the model weights each modality.
Set up the training script

The multimodal training script needs to load all four .npy feature files, split them using the exact same seed as the baseline, and feed both modalities through the GMU during training.

  • Create a new file called train_multimodal.py in your project root and add the following imports and data loading code:
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from model import MultimodalDistressClassifier


def main():
    # Load all four feature files from Step 2
    facial_features = np.load('data/facial_features.npy')
    text_features = np.load('data/text_features.npy')
    labels = np.load('data/combined_labels.npy')
    face_available = np.load('data/face_available.npy')

    # Use index-based split with same seed as baseline
    indices = np.arange(len(labels))
    train_idx, test_idx = train_test_split(
        indices, test_size=0.2, random_state=42, stratify=labels
    )

    # Convert training data to PyTorch tensors
    X_facial_train = torch.FloatTensor(facial_features[train_idx])
    X_text_train = torch.FloatTensor(text_features[train_idx])
    y_train = torch.FloatTensor(labels[train_idx]).unsqueeze(1)

What does this code do?

  • Loads all four .npy files you created in Step 2: facial features, text features, combined labels, and face availability flags.
  • Splits by index rather than by raw arrays. This keeps facial features, text features, and labels aligned to the same train/test samples.
  • Uses random_state=42 and stratify=labels to produce the exact same split as your baseline, ensuring a fair comparison.
  • matplotlib.use('Agg') sets a non-interactive backend so the plot saves to a file without requiring a display server.
  • Save train_multimodal.py. You will run it after adding the training loop and evaluation code.
  • Still inside the main() function, add the data preparation and training loop directly below the code you just wrote:
    # Prepare test tensors and DataLoader
    X_facial_test = torch.FloatTensor(facial_features[test_idx])
    X_text_test = torch.FloatTensor(text_features[test_idx])
    y_test_np = labels[test_idx].astype(int)
    face_available_test = face_available[test_idx]
    train_dataset = TensorDataset(X_facial_train, X_text_train, y_train)
    train_loader = DataLoader(train_dataset, batch_size=16, shuffle=True)
    # Initialize model with same hyperparameters as baseline
    model = MultimodalDistressClassifier(
        visual_dim=52, text_dim=4, hidden_dim=32
    )
    criterion = nn.BCELoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    # Train for 100 epochs, tracking loss per epoch
    train_losses = []
    print("Training multimodal GMU classifier...")
    for epoch in range(100):
        model.train()
        epoch_loss = 0
        for batch_facial, batch_text, batch_y in train_loader:
            optimizer.zero_grad()
            outputs = model(batch_facial, batch_text)
            loss = criterion(outputs, batch_y)
            loss.backward()
            optimizer.step()
            epoch_loss += loss.item()
        avg_loss = epoch_loss / len(train_loader)
        train_losses.append(avg_loss)
        if (epoch + 1) % 20 == 0:
            print(f"Epoch {epoch + 1}/100, Loss: {avg_loss:.4f}")

What does this code do?

  • Converts the test split into tensors and wraps training data in a DataLoader that batches facial, text, and label tensors together (batch size 16).
  • Initializes the MultimodalDistressClassifier with BCELoss and Adam optimizer at learning rate 0.001, matching the baseline setup.
  • The training loop feeds both modality tensors through the GMU for 100 epochs, printing loss every 20 epochs so you can watch convergence.
  • train_losses records each epoch's average loss for plotting later.
  • Save train_multimodal.py. The evaluation code comes next.
Evaluate the model and compare with the baseline

With the training loop in place, you need evaluation code that prints accuracy, a classification report, and a confusion matrix. You also need the training loss plot and the if __name__ entry point to make the script runnable.

  • Below the training loop in train_multimodal.py, add the following code to complete the file:
    # Evaluate on test set and print metrics
    model.eval()
    with torch.no_grad():
        predictions = model(X_facial_test, X_text_test)
        predicted_labels = (predictions >= 0.5).int().numpy().flatten()

    accuracy = accuracy_score(y_test_np, predicted_labels)

    print(f"\nMultimodal GMU Accuracy: {accuracy:.4f}")
    print("\nClassification Report:")
    print(
        classification_report(
            y_test_np, predicted_labels,
            target_names=['Non-distress', 'Distress'],
        )
    )
    print("\nConfusion Matrix:")
    print(confusion_matrix(y_test_np, predicted_labels))

    plt.figure(figsize=(10, 5))
    plt.plot(train_losses)
    plt.title('Training Loss')
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.savefig('training_loss.png')
    print("\nTraining loss plot saved to training_loss.png")


if __name__ == '__main__':
    main()

What does this code do?

  • model.eval() disables dropout so predictions are deterministic, then runs the test data through the trained model.
  • Predictions above 0.5 become class 1 (distress). The script prints accuracy, per-class precision/recall/F1, and a confusion matrix.
  • Saves the training loss curve as training_loss.png so you can check whether the model converged smoothly.

Before you run the script, predict: do you think the multimodal GMU will outperform the text-only baseline you saw in Step 3, or will adding facial features not help?

  • Save train_multimodal.py and run your training script:
python train_multimodal.py

You will see training progress printed every 20 epochs, followed by the multimodal GMU accuracy, a classification report, and a confusion matrix. Compare the accuracy to the text-only baseline accuracy you recorded in Step 3.

Why should the multimodal model improve?

The text-only baseline relies entirely on VADER sentiment scores. It misses cases where a student types neutral or positive text but their face shows distress (emotion masking).

The GMU adds 52 facial blendshape features and learns when to trust each modality. For masked emotions, the gate shifts weight toward the face. The improvement may be modest on a small dataset, but the confusion matrix should show fewer missed distress cases.

Seeing an error when running the script?

  • If you see ModuleNotFoundError: No module named 'model', make sure you are running the script from your project root folder where model.py lives.
  • If you see a shape mismatch error, confirm your facial_features.npy has shape (N, 52) and text_features.npy has shape (N, 4) by re-running extract_features.py.
  • If accuracy is very low (below 0.5), check that your combined_labels.npy has a reasonable balance of 0s and 1s. A heavily imbalanced dataset can cause the model to predict all one class.
  • Still stuck? Help me debug my multimodal training script. I'm seeing this error when I run python train_multimodal.py.
Analyze the GMU gate values

Accuracy tells you the model improved, but the gate values tell you why. By extracting the gate tensor z from the GMU, you can see whether the model learned to favor text when facial data is missing.

  • In train_multimodal.py, find this line near the bottom of the file:
if __name__ == '__main__':
  • Add the following gate analysis code above that line, right after the print("\nTraining loss plot saved to training_loss.png") line:
    # Extract gate values to see which modality the GMU favors
    with torch.no_grad():
        z = torch.sigmoid(
            model.gmu.gate(
                torch.cat([X_facial_test, X_text_test], dim=1)
            )
        )

        print("\n--- GMU Gate Analysis ---")
        print("Gate > 0.5: model favors visual (facial) modality")
        print("Gate < 0.5: model favors text (sentiment) modality\n")

        avg_gate_all = z.mean().item()
        print(f"Average gate (all test samples): {avg_gate_all:.4f}")

        if face_available_test.any():
            face_mask = torch.BoolTensor(face_available_test)
            avg_gate_face = z[face_mask].mean().item()
            print(
                f"Average gate (face available):   {avg_gate_face:.4f}"
            )

        if not face_available_test.all():
            no_face_mask = torch.BoolTensor(~face_available_test)
            avg_gate_no_face = z[no_face_mask].mean().item()
            print(
                f"Average gate (face unavailable): {avg_gate_no_face:.4f}"
            )

What does this code do?

  • Passes the concatenated test features through the GMU's gate layer to extract the raw gate tensor z.
  • Gate values above 0.5 mean the model favors the visual (facial) modality. Values below 0.5 mean it favors text sentiment.
  • Computes the average gate separately for face-available and face-unavailable samples. This reveals whether the GMU learned to shift toward text when facial data is a zero vector.

Before you re-run the script, predict: will the average gate value be higher or lower for samples where no face was detected?

  • Save train_multimodal.py and re-run the script:
python train_multimodal.py

After the accuracy and confusion matrix, you will now see the GMU Gate Analysis section. The average gate for face-available samples should be higher than for face-unavailable samples, showing the model learned to lean on visual features when they exist and shift to text when they do not.

  • Confirm that training_loss.png was saved in your project root by running:
ls training_loss.png

You should see training_loss.png listed. Open it in your file browser or image viewer to check that the loss curve trends downward over 100 epochs.

Gate values all around 0.5?

If the gate values for face-available and face-unavailable samples are nearly identical, your dataset may have too few face-unavailable samples for the gate to learn a meaningful split. Check your face_available.npy to see how many samples have no face detected. With only 2-3 face-unavailable samples, the gate may not differentiate.

Help me understand my GMU gate analysis results. The face-available and face-unavailable gate values are almost the same.

What is the GMU gate telling you?

The gate analysis is the key academic contribution you can report. It proves the GMU is not just averaging the two modalities blindly. It dynamically adjusts per sample based on what information is actually available.

When a student wears a mask and facial features are zeros, the gate drops toward text. When a student types "I'm fine" but their face shows distress, the gate shifts toward the visual modality. This adaptive behavior is exactly what makes multimodal fusion more robust than either modality alone.

✔️ Awesome, I've got everything!

Double check that you have saved train_multimodal.py and that training_loss.png exists in your project root.

ⓧ I'd like to double check the full code

train_multimodal.py is too long to show in one block. Scroll up and confirm that each of these sections is present in order:

  • Imports (numpy, torch, sklearn, matplotlib, MultimodalDistressClassifier).
  • def main() with data loading (np.load for all four .npy files) and train_test_split.
  • Tensor conversion for training and test data, plus TensorDataset and DataLoader.
  • Model initialization (MultimodalDistressClassifier), BCELoss, Adam optimizer, and the 100-epoch training loop.
  • Evaluation block (model.eval(), accuracy, classification report, confusion matrix).
  • Matplotlib plot saved to training_loss.png.
  • GMU gate analysis with average gate values for face-available and face-unavailable samples.
  • if __name__ == '__main__': main() at the very bottom.

Secret mission

Add Stratified K-Fold Cross-Validation

With only 80-100 samples, a single 80/20 split may produce unreliable accuracy estimates. Can you restructure the evaluation to train and test across multiple splits, producing a mean accuracy with a standard deviation for your university report?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your resources for your university submission, or delete them to free disk space.

Resources you used:

  • Virtual environment (venv/) with mediapipe, vaderSentiment, torch, scikit-learn, and matplotlib installed.
  • Python scripts: extract_features.py, model.py, train_baseline.py, train_multimodal.py, train_kfold.py.
  • MediaPipe model file (face_landmarker.task).
  • Extracted feature files in data/ (facial_features.npy, text_features.npy, combined_labels.npy, face_available.npy).
  • Training loss plot (training_loss.png).
  • Your dataset: data/labels.csv and data/videos/.

Keep everything running

No action needed. Choose this if you plan to continue refining your multimodal distress detector or need the results for your university report.

  • Your trained models, extracted .npy feature files, and evaluation results are all saved locally.
  • The training_loss.png plot is ready for your report.
  • To re-enter the virtual environment later, activate it again from your project folder.

Pause - I'll come back to this later

Deactivate the virtual environment to free up your terminal, but keep all files on disk so you can pick up where you left off.

  • Deactivate the virtual environment by running:
deactivate
  • Optionally, delete the .npy feature files to save disk space. You can regenerate them by running extract_features.py again later.
rm data/facial_features.npy data/text_features.npy data/combined_labels.npy data/face_available.npy

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes the virtual environment, scripts, model file, feature files, and output plot.

  • Deactivate the virtual environment first (if it is still active):
deactivate
  • Delete the virtual environment on macOS/Linux:
rm -rf venv
  • On Windows, run this instead:
rmdir /s /q venv
  • Delete the extracted feature files and training loss plot:
rm data/facial_features.npy data/text_features.npy data/combined_labels.npy data/face_available.npy
rm training_loss.png
  • Delete the Python scripts and model file:
rm extract_features.py model.py train_baseline.py train_multimodal.py train_kfold.py
rm face_landmarker.task
  • Your original dataset (data/labels.csv and data/videos/) is your own data. Keep it if you might use it for another project, or delete the entire data/ folder to remove everything.
rm -rf data

Nice Work!

Nice Work!

You just built a multimodal distress detection pipeline that fuses facial expressions and text sentiment to catch what neither modality can detect alone.

You've learned how to:

  • Built a feature extraction pipeline that processes video files with MediaPipe FaceLandmarker into 52 facial blendshape scores and text messages with VADER into 4 sentiment scores, with a zero-vector fallback for samples where no face is detected.
  • Implemented a Gated Multimodal Unit (GMU) in PyTorch that uses learned gates to dynamically weight how much to trust the face versus the text for each individual sample.
  • Trained and evaluated a multimodal distress classifier against a text-only baseline, demonstrating improved accuracy and inspecting the GMU gate values to confirm the model shifts toward text when face data is unavailable and toward face when detecting masked emotions.
  • Secret Mission: Completed an optional challenge to push your multimodal classification skills further.

Ready to quiz yourself?