Benchmark V-JEPA 2.1 vs Qwen2.5-VL
Compare a world model against a multimodal LLM on physical video reasoning.
Introduction
30 Second Summary
When you watch a video of someone pushing a cup across a table, you instantly understand the physics involved. You know the cup will stop if the hand stops pushing, and you know it will fall if it reaches the edge. Teaching AI to understand video at this level is one of the hardest open problems in the field, and two fundamentally different approaches are racing to solve it.
In this project, you will build a benchmarking pipeline that pits Meta's V-JEPA 2.1 world model against Alibaba's Qwen2.5-VL-7B multimodal LLM on temporal video understanding tasks. You will discover where embedding-space prediction outperforms next-token prediction for physical reasoning, and vice versa.
What You'll Build
A research-grade benchmarking pipeline that evaluates a world model and a multimodal LLM on the same 500 videos from the Something-Something V2 dataset, producing quantitative per-category accuracy comparisons and visualizations.
By the end of this project, you'll have:
- A working evaluation pipeline that runs V-JEPA 2.1 (world model) and Qwen2.5-VL-7B (multimodal LLM) on the same set of video understanding tasks.
- Quantitative per-category accuracy comparisons showing where embedding-space prediction outperforms next-token prediction and vice versa.
- A set of visualizations and analysis scripts that reveal the strengths and weaknesses of each paradigm for physical reasoning from video.
- Secret Mission: A PCA visualization of V-JEPA 2.1's dense features to see what the world model "sees" versus what the LLM "says."
Are there any prerequisites?
This project assumes familiarity with PyTorch and deep learning concepts. You should have experience running models on GPU clusters and a basic understanding of vision transformers.
You will need access to a machine with A100 GPUs (or equivalent), a Hugging Face account, and roughly 20 GB of disk space for the SSv2 dataset and model checkpoints.
Set Up the Research Environment
Comparing two AI paradigms on the same task requires a carefully prepared environment where both models can process the same data. You need V-JEPA 2.1 (Meta's world model) and Qwen2.5-VL-7B (a multimodal LLM) installed side by side, along with the Something-Something V2 dataset that tests temporal video understanding.
In this step, you will create a Conda environment, install both model frameworks, download the pretrained checkpoints, and grab the dataset.
In this step, get ready to:
- Create a conda environment and install the V-JEPA 2 codebase.
- Install Qwen2.5-VL dependencies and analysis libraries.
- Download model checkpoints and the SSv2 dataset.
Create the conda environment and install V-JEPA 2.1
V-JEPA 2.1 ships as a pip-installable package inside the vjepa2 GitHub repository. Installing it pulls in PyTorch, timm, einops, decord, and all other dependencies the model needs.
- Create a fresh conda environment with Python 3.12 and activate it by running these commands:
conda create -n vjepa-bench python=3.12 -y
conda activate vjepa-benchWhat do these commands do?
conda create -n vjepa-bench python=3.12 -y creates an isolated Python 3.12 environment named vjepa-bench.
conda activate vjepa-bench switches your shell into that environment so every package you install next stays isolated from the rest of your system.
- You should see (vjepa-bench) at the start of your terminal prompt after activation.
Conda command not found?
Your supercomputer may require a module load before conda is available. Try running module load anaconda or module load miniconda first, then retry. Check your cluster's documentation for the exact module name.
help me load conda on my supercomputer
- Clone the V-JEPA 2 repository and install it by running these commands:
git clone https://github.com/facebookresearch/vjepa2.git
cd vjepa2
pip install .
cd ..What does this install?
git clone downloads the full V-JEPA 2 codebase from Meta's GitHub repository.
pip install . installs the package and all its dependencies, including PyTorch with CUDA support, timm (PyTorch image models), einops (tensor operations), and decord (video decoding). The final cd .. moves you back to the parent directory.
- Confirm the install succeeded by running:
python -c "import torch; print(torch.__version__); print('CUDA:', torch.cuda.is_available())"- You should see a PyTorch version number and CUDA: True in the output.
Seeing CUDA: False?
Your supercomputer may need a CUDA module loaded before PyTorch can detect GPUs. Run module load cuda and try again. If you are on a login node, GPU access may only be available inside a job allocation.
help me get CUDA working on my supercomputer
Install Qwen2.5-VL dependencies
Qwen2.5-VL loads through the Hugging Face transformers library. You also need the Qwen VL utilities for video processing and a few analysis libraries.
- Install the remaining dependencies by running:
pip install transformers accelerate
pip install qwen-vl-utils[decord]
pip install scikit-learn matplotlib tqdmWhat are these packages for?
- transformers and accelerate provide the model loading and multi-GPU sharding for Qwen2.5-VL.
- qwen-vl-utils[decord] handles video frame extraction for Qwen's vision-language pipeline.
- scikit-learn provides the logistic regression classifier for the V-JEPA linear probe.
- matplotlib generates the comparison charts. tqdm provides progress bars during long evaluations.
- Confirm the Qwen dependencies installed correctly by running:
python -c "from transformers import Qwen2_5_VLForConditionalGeneration; print('Qwen2.5-VL ready')"- You should see Qwen2.5-VL ready printed to the terminal.
Import error for Qwen2_5_VLForConditionalGeneration?
This class requires transformers version 4.40 or higher. Upgrade with pip install --upgrade transformers and retry.
help me fix my transformers import
Download model checkpoints and the SSv2 dataset
You need three large downloads: the V-JEPA 2.1 ViT-g checkpoint (1B parameters, 384 resolution), the Qwen2.5-VL-7B-Instruct weights, and the Something-Something V2 video dataset.
- Create the checkpoints directory and download the V-JEPA 2.1 ViT-g checkpoint by running:
mkdir -p checkpoints
wget https://dl.fbaipublicfiles.com/vjepa2/vjepa2_1_vitg_384.pt -P checkpoints/What is this checkpoint?
This is the frozen encoder from V-JEPA 2.1's ViT-g model, trained via self-supervised learning on large-scale video data. It contains 1 billion parameters at 384-pixel resolution.
Your evaluation script will use torch.hub.load to load the model, which auto-downloads the checkpoint to PyTorch's cache. This manual download gives you a local backup and lets you verify the file.
- Confirm the checkpoint downloaded by running:
ls -lh checkpoints/- You should see vjepa2_1_vitg_384.pt listed (approximately 3-4 GB in size).
Download timing out or failing?
Large file downloads on shared clusters can be slow. If wget times out, add the -c flag to resume: wget -c https://dl.fbaipublicfiles.com/vjepa2/vjepa2_1_vitg_384.pt -P checkpoints/.
help me download large model checkpoints on my cluster
- Download the Qwen2.5-VL-7B-Instruct weights from Hugging Face by running:
huggingface-cli download Qwen/Qwen2.5-VL-7B-Instruct --local-dir checkpoints/qwen2.5-vl-7bWhy download the weights ahead of time?
Pre-downloading ensures the model is available when you run the evaluation script, even if compute nodes do not have internet access. The --local-dir flag saves the weights to a specific folder you can point to later.
- Confirm the Qwen weights downloaded by running:
ls checkpoints/qwen2.5-vl-7b/- You should see model shard files (.safetensors), config.json, tokenizer.json, and other model files.
Authentication error from Hugging Face?
Make sure you are logged in to Hugging Face CLI. Run huggingface-cli login and paste your access token when prompted. You can generate a token at huggingface.co/settings/tokens.
help me authenticate with Hugging Face on my cluster
The last download is the SSv2 dataset itself. This requires a free registration on the official website.
- Register for the Something-Something V2 dataset at 20bn.com/datasets/something-something/v2.
- After registration, download the video archive (20 parts, approximately 19.4 GB total) and the annotation JSON files.
- Create the dataset directory structure and extract the videos by running:
mkdir -p data/sthv2/annotations
# Move your downloaded video parts and annotation files into data/sthv2/
# Then extract the videos:
cd data/sthv2
cat 20bn-something-something-v2-?? | tar zx
cd ../..What does the extraction command do?
The SSv2 videos are split across 20 compressed archive parts. The cat command concatenates all the parts together, and tar zx decompresses them into a folder called 20bn-something-something-v2 containing thousands of .webm video files.
- Copy the annotation files (something-something-v2-labels.json and something-something-v2-validation.json) into data/sthv2/annotations/.
- Verify your directory structure by running:
ls data/sthv2/annotations/
ls data/sthv2/20bn-something-something-v2/ | head -5- The first command should show something-something-v2-labels.json and something-something-v2-validation.json.
- The second command should show .webm video files (like 100.webm, 1000.webm, etc.).
Dataset download or extraction issues?
If the SSv2 download page is unavailable, the dataset is also mirrored on Hugging Face Datasets. Search for something-something-v2 on Hugging Face as an alternative source.
If extraction fails partway, delete the partial 20bn-something-something-v2 folder and re-run the cat | tar command.
help me get the Something-Something V2 dataset onto my cluster
Your research environment is fully assembled. Next up, you will curate a focused evaluation subset from the SSv2 validation data, selecting videos that require genuine temporal reasoning.
Curate the Evaluation Subset
Your research environment is ready with V-JEPA 2.1, Qwen2.5-VL-7B, and the full Something-Something V2 dataset downloaded. Now you need a focused evaluation subset that tests something specific: temporal reasoning.
SSv2 has 174 template categories and ~24,800 validation videos, but not all categories require genuine motion understanding. Some actions are identifiable from a single frame. In this step, you will curate a balanced subset of 500 videos across 20 categories that demand real temporal comprehension.
In this step, get ready to:
- Explore how SSv2 annotations are structured.
- Write and run a curation script that selects 25 videos from each of 20 temporal action categories.
- Verify the resulting evaluation subset is balanced and complete.
Explore the SSv2 annotation structure
Before writing any curation code, take a look at how SSv2 organizes its annotations. Each validation entry is a JSON object with three key fields: id (the video filename), label (the specific action description like "Pushing a book from left to right"), and template (the generalized category like "Pushing [something] from left to right").
- Inspect a sample annotation entry by running this command:
python -c "
import json
data = json.load(open('data/sthv2/annotations/something-something-v2-validation.json'))
print(json.dumps(data[0], indent=2))
print(f'\nTotal validation videos: {len(data)}')
"You should see a single entry with id, label, and template fields, plus a count of roughly 24,800 total validation videos.
Why the template field matters
The template field generalizes specific labels into categories. "Pushing a book from left to right" and "Pushing a cup from left to right" both share the template "Pushing [something] from left to right." This lets you group videos by the type of motion, regardless of what object is involved.
SSv2 has 174 template categories total. Your curation script will select 20 of these that specifically require temporal understanding to classify correctly.
Write and run prepare_subset.py
The curation script picks 20 temporal action categories that are impossible to identify from a single frame. Actions like "Pushing [something] from left to right" versus "Pushing [something] from right to left" look identical in a still image. Only a model that truly understands motion over time can distinguish them.
- Create a new file called prepare_subset.py in your project root (the same directory that contains the data/ and checkpoints/ folders).
- Add the imports, configuration constants, and the list of 20 temporal categories:
"""Curate a balanced evaluation subset of Something-Something V2 for benchmarking."""
import json
import os
import random
from collections import Counter
# Path to SSv2 dataset root (override with SSV2_ROOT env var if needed)
SSV2_ROOT = os.environ.get("SSV2_ROOT", "data/sthv2")
SUBSET_DIR = os.path.join(SSV2_ROOT, "eval_subset")
NUM_VIDEOS_PER_CATEGORY = 25
SEED = 42
# 20 categories that require genuine temporal understanding
# (not solvable from a single frame)
TEMPORAL_CATEGORIES = [
"Pushing [something] from left to right",
"Pushing [something] from right to left",
"Moving [something] up",
"Moving [something] down",
"Picking [something] up",
"Putting [something] onto [something]",
"Throwing [something]",
"Dropping [something]",
"Rolling [something] on a flat surface",
"Turning [something] upside down",
"Pouring [something] into [something]",
"Folding [something]",
"Tearing [something] into two pieces",
"Covering [something] with [something]",
"Opening [something]",
"Closing [something]",
"Spinning [something] that quickly stops spinning",
"Pulling [something] from left to right",
"Pulling [something] from right to left",
"Lifting [something] up completely without letting it drop down",
]
# Map each category name to a numeric index for classification
CATEGORY_TO_IDX = {cat: idx for idx, cat in enumerate(TEMPORAL_CATEGORIES)}What does this code do?
SSV2_ROOT points to the dataset directory. It reads from the SSV2_ROOT environment variable if set, falling back to data/sthv2 by default.
The 20 categories in TEMPORAL_CATEGORIES are hand-picked because they test motion direction, object interaction, and physical dynamics. Notice the directional pairs like "Pushing left to right" vs. "Pushing right to left." These are the hardest categories for any model that does not genuinely track motion over time.
CATEGORY_TO_IDX maps each category string to a numeric label (0 through 19) for use in classification later.
- Now add the main() function below the CATEGORY_TO_IDX line. This function loads the annotations, samples videos, and saves the subset:
def main():
random.seed(SEED)
# Paths to SSv2 annotation files
labels_path = os.path.join(SSV2_ROOT, "annotations", "something-something-v2-labels.json")
val_path = os.path.join(SSV2_ROOT, "annotations", "something-something-v2-validation.json")
with open(labels_path) as f:
all_labels = json.load(f)
with open(val_path) as f:
val_data = json.load(f)
# Group validation videos by their template category
category_videos = {cat: [] for cat in TEMPORAL_CATEGORIES}
for item in val_data:
template = item.get("template", "")
if template in category_videos:
category_videos[template].append(item)
# Sample NUM_VIDEOS_PER_CATEGORY videos from each category
subset = []
for cat in TEMPORAL_CATEGORIES:
videos = category_videos[cat]
if len(videos) == 0:
print(f"WARNING: No videos found for category: {cat}")
continue
sampled = random.sample(videos, min(NUM_VIDEOS_PER_CATEGORY, len(videos)))
for v in sampled:
subset.append({
"id": v["id"],
"label": v.get("label", ""),
"template": cat,
"category_idx": CATEGORY_TO_IDX[cat],
})
# Save the curated subset to disk
os.makedirs(SUBSET_DIR, exist_ok=True)
metadata_path = os.path.join(SUBSET_DIR, "subset_metadata.json")
with open(metadata_path, "w") as f:
json.dump(subset, f, indent=2)
# Print a summary of the subset
counts = Counter(item["template"] for item in subset)
print(f"Created evaluation subset: {len(subset)} videos from {len(counts)} categories")
print("\nVideos per category:")
for cat, count in sorted(counts.items()):
print(f" {cat}: {count}")
print(f"\nMetadata saved to: {metadata_path}")
if __name__ == "__main__":
main()What does this function do?
- random.seed(SEED) locks the random sampling to seed 42 so results are reproducible.
- The function loads both annotation files, then loops through every validation video and groups it by template field. Only videos matching one of the 20 chosen categories are kept.
- random.sample() picks exactly 25 videos per category, giving a balanced 500-video subset.
- Each entry in the output JSON includes the video id, the specific label, the generalized template, and a numeric category_idx for classification.
- Save prepare_subset.py.
- Run the curation script by executing this command:
python prepare_subset.pyYou should see: "Created evaluation subset: 500 videos from 20 categories" followed by a per-category count of 25 each.
Seeing a FileNotFoundError?
- Check that your annotation files are in data/sthv2/annotations/. The script expects something-something-v2-labels.json and something-something-v2-validation.json in that directory.
- Make sure you are running the command from your project root directory (the one containing the data/ folder), not from inside the vjepa2/ directory.
- If your dataset is in a different location, set the SSV2_ROOT environment variable to your actual path before running the script.
Still stuck? Help me debug why prepare_subset.py cannot find my SSv2 annotation files.
✔️ Awesome, I've got everything!
Great. Double check you have saved prepare_subset.py and confirmed the 500-video output.
ⓧ I'd like to double check the full code
"""Curate a balanced evaluation subset of Something-Something V2 for benchmarking."""
import json
import os
import random
from collections import Counter
SSV2_ROOT = os.environ.get("SSV2_ROOT", "data/sthv2")
SUBSET_DIR = os.path.join(SSV2_ROOT, "eval_subset")
NUM_VIDEOS_PER_CATEGORY = 25
SEED = 42
TEMPORAL_CATEGORIES = [
"Pushing [something] from left to right",
"Pushing [something] from right to left",
"Moving [something] up",
"Moving [something] down",
"Picking [something] up",
"Putting [something] onto [something]",
"Throwing [something]",
"Dropping [something]",
"Rolling [something] on a flat surface",
"Turning [something] upside down",
"Pouring [something] into [something]",
"Folding [something]",
"Tearing [something] into two pieces",
"Covering [something] with [something]",
"Opening [something]",
"Closing [something]",
"Spinning [something] that quickly stops spinning",
"Pulling [something] from left to right",
"Pulling [something] from right to left",
"Lifting [something] up completely without letting it drop down",
]
CATEGORY_TO_IDX = {cat: idx for idx, cat in enumerate(TEMPORAL_CATEGORIES)}
def main():
random.seed(SEED)
labels_path = os.path.join(SSV2_ROOT, "annotations", "something-something-v2-labels.json")
val_path = os.path.join(SSV2_ROOT, "annotations", "something-something-v2-validation.json")
with open(labels_path) as f:
all_labels = json.load(f)
with open(val_path) as f:
val_data = json.load(f)
category_videos = {cat: [] for cat in TEMPORAL_CATEGORIES}
for item in val_data:
template = item.get("template", "")
if template in category_videos:
category_videos[template].append(item)
subset = []
for cat in TEMPORAL_CATEGORIES:
videos = category_videos[cat]
if len(videos) == 0:
print(f"WARNING: No videos found for category: {cat}")
continue
sampled = random.sample(videos, min(NUM_VIDEOS_PER_CATEGORY, len(videos)))
for v in sampled:
subset.append({
"id": v["id"],
"label": v.get("label", ""),
"template": cat,
"category_idx": CATEGORY_TO_IDX[cat],
})
os.makedirs(SUBSET_DIR, exist_ok=True)
metadata_path = os.path.join(SUBSET_DIR, "subset_metadata.json")
with open(metadata_path, "w") as f:
json.dump(subset, f, indent=2)
counts = Counter(item["template"] for item in subset)
print(f"Created evaluation subset: {len(subset)} videos from {len(counts)} categories")
print("\nVideos per category:")
for cat, count in sorted(counts.items()):
print(f" {cat}: {count}")
print(f"\nMetadata saved to: {metadata_path}")
if __name__ == "__main__":
main()Verify the evaluation subset
The script created the metadata file, but you should confirm that the referenced video files actually exist on disk. A metadata entry is useless if its video file is missing.
- Spot-check a few video paths from the subset by running this command:
python -c "
import json, os
subset = json.load(open('data/sthv2/eval_subset/subset_metadata.json'))
missing = 0
for item in subset:
path = os.path.join('data/sthv2/20bn-something-something-v2', item['id'] + '.webm')
if not os.path.exists(path):
missing += 1
print(f'Total videos in subset: {len(subset)}')
print(f'Missing video files: {missing}')
print(f'Sample entry: {json.dumps(subset[0], indent=2)}')
"You should see 500 total videos and 0 missing files. The sample entry shows the four fields: id, label, template, and category_idx.
Seeing missing video files?
- Confirm that you extracted all 20 parts of the SSv2 video archive. If some parts were not downloaded, certain video IDs will be missing.
- Verify the video files are in data/sthv2/20bn-something-something-v2/ and use the .webm extension. SSv2 videos are VP9-encoded WebM files.
Need help? Help me figure out why some SSv2 video files are missing from my extracted dataset.
Why 500 videos instead of the full 24,800?
Running Qwen2.5-VL-7B on every validation video would take hours of autoregressive generation. A 500-video subset (25 per category) gives statistically meaningful per-category accuracy while keeping total evaluation time practical.
The fixed seed of 42 ensures that anyone running this pipeline gets the exact same subset for reproducibility.
Your evaluation subset is locked and loaded. Next up, you will run V-JEPA 2.1 on these 500 videos and see how well a world model's frozen features can classify temporal actions.
Evaluate V-JEPA 2.1 on Video Understanding
Your curated subset of 500 videos is ready. Now it is time to see what a world model actually learns from watching video.
V-JEPA 2.1 learns by predicting missing patches of video in an abstract embedding space, not by generating pixels or predicting text tokens. The question is whether those learned representations are rich enough to distinguish 20 different temporal actions. You will find out by extracting features from the model and training a lightweight linear probe classifier on top.
In this step, get ready to:
- Write the V-JEPA 2.1 evaluation script with model loading and feature extraction.
- Train a logistic regression linear probe on the extracted features.
- Run the full pipeline and review per-category accuracy results.
Load V-JEPA 2.1 and extract features
V-JEPA 2.1 does not output class labels directly. It outputs dense token-level embeddings that encode what it has learned about the video's spatiotemporal structure. To evaluate those embeddings, you first need to extract them, then hand them to a simple classifier.
- Create a new file called evaluate_vjepa.py in your project root (the same directory as prepare_subset.py).
- Add the imports, constants, and helper functions by pasting this code:
"""Evaluate V-JEPA 2.1 on SSv2 subset using feature extraction + linear probe."""
import json
import os
import numpy as np
import torch
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
from tqdm import tqdm
# Paths matching the subset preparation script
SSV2_ROOT = os.environ.get("SSV2_ROOT", "data/sthv2")
SUBSET_META = os.path.join(SSV2_ROOT, "eval_subset", "subset_metadata.json")
VIDEO_DIR = os.path.join(SSV2_ROOT, "20bn-something-something-v2")
RESULTS_DIR = "results"
SEED = 42
def load_subset():
"""Load the curated evaluation subset metadata."""
with open(SUBSET_META) as f:
return json.load(f)
def get_video_path(video_id):
"""Construct the full path for a given video ID."""
return os.path.join(VIDEO_DIR, f"{video_id}.webm")What does this code do?
The imports pull in PyTorch for model inference, scikit-learn for the linear probe classifier, and tqdm for progress bars during feature extraction.
The constants reuse the same paths and seed from prepare_subset.py so both scripts point at the same data. The two helpers keep the main function clean by abstracting metadata loading and video path construction.
You will not see any output yet. The next chunk builds the core logic.
The heart of this script is the main() function. It loads the V-JEPA 2.1 model via torch.hub, iterates over all 500 videos, and average-pools the token embeddings into one feature vector per video.
- Below your helper functions in evaluate_vjepa.py, add the main() function with model loading and feature extraction:
def main():
os.makedirs(RESULTS_DIR, exist_ok=True)
# Load V-JEPA 2.1 ViT-g (1B params) via torch.hub
print("Loading V-JEPA 2.1 ViT-g model...")
processor = torch.hub.load("facebookresearch/vjepa2", "vjepa2_preprocessor")
model = torch.hub.load("facebookresearch/vjepa2", "vjepa2_1_vit_giant_384")
model.eval().cuda()
subset = load_subset()
print(f"Loaded {len(subset)} videos from subset metadata.")
features_list = []
labels_list = []
video_ids = []
skipped = 0
# Extract one feature vector per video by average-pooling token embeddings
print("Extracting features...")
for item in tqdm(subset):
video_path = get_video_path(item["id"])
if not os.path.exists(video_path):
skipped += 1
continue
try:
# Preprocess the video into a tensor
video_tensor = processor(video_path)
if video_tensor.dim() == 4:
video_tensor = video_tensor.unsqueeze(0)
video_tensor = video_tensor.cuda()
with torch.no_grad():
# Forward pass through the encoder
output = model(video_tensor)
if hasattr(output, "last_hidden_state"):
tokens = output.last_hidden_state
elif isinstance(output, torch.Tensor):
tokens = output
else:
tokens = output[0]
# Average-pool all tokens into a single feature vector
pooled = tokens.mean(dim=1).squeeze(0).cpu().numpy()
features_list.append(pooled)
labels_list.append(item["category_idx"])
video_ids.append(item["id"])
except Exception as e:
print(f"Error processing {item['id']}: {e}")
skipped += 1
if skipped > 0:
print(f"Skipped {skipped} videos due to errors or missing files.")
features = np.array(features_list)
labels = np.array(labels_list)
print(f"Extracted features shape: {features.shape}")What does this code do?
torch.hub.load("facebookresearch/vjepa2", "vjepa2_preprocessor") downloads the official video preprocessor, which handles frame sampling and tensor conversion. torch.hub.load("facebookresearch/vjepa2", "vjepa2_1_vit_giant_384") loads the 1B-parameter ViT-g encoder at 384 resolution.
The model's output is a set of token-level embeddings. Each token represents a spatiotemporal patch of the video. Average-pooling across all tokens collapses them into a single fixed-size vector that captures the entire video's temporal dynamics.
The output-handling logic (checking for last_hidden_state, raw tensors, or tuples) makes the script robust to minor differences in how torch.hub returns results.
Why use torch.hub instead of HuggingFace?
V-JEPA 2.1 is not yet supported via HuggingFace from_pretrained(). A HuggingFace integration is in progress, but for now torch.hub is the official loading path from Meta's repository.
- Save evaluate_vjepa.py.
You will not see output yet. The function is defined but not called, and still needs the linear probe section.
Train a linear probe and evaluate
A linear probe tests how useful frozen features are. If a simple logistic regression can classify videos accurately from V-JEPA's embeddings alone, that means the model has genuinely learned to represent temporal action differences.
Why a linear probe instead of fine-tuning?
Fine-tuning adjusts the model's own weights to fit the downstream task, which makes it hard to tell what the model learned during pre-training versus what it learned from your labels. A linear probe freezes the model entirely and only trains a single classification layer on top.
This is the standard evaluation protocol for self-supervised learning research. It isolates the quality of the learned representations.
- In evaluate_vjepa.py, add the following code inside main() directly below the print(f"Extracted features shape: {features.shape}") line:
# Split 80% train / 20% test, stratified so every category is represented
X_train, X_test, y_train, y_test, ids_train, ids_test = train_test_split(
features, labels, video_ids, test_size=0.2, random_state=SEED, stratify=labels
)
# Train a logistic regression classifier on the frozen features
print("Training linear probe...")
clf = LogisticRegression(max_iter=1000, random_state=SEED, C=1.0)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
overall_acc = accuracy_score(y_test, y_pred)
print(f"\nV-JEPA 2.1 Overall Accuracy: {overall_acc:.4f}")
# Build a mapping from category index back to human-readable names
with open(SUBSET_META) as f:
all_items = json.load(f)
idx_to_cat = {}
for item in all_items:
idx_to_cat[item["category_idx"]] = item["template"]
category_names = [idx_to_cat.get(i, f"cat_{i}") for i in sorted(idx_to_cat.keys())]
# Compute per-category accuracy on the test split
per_category = {}
for true, pred in zip(y_test, y_pred):
cat_name = idx_to_cat.get(true, f"cat_{true}")
if cat_name not in per_category:
per_category[cat_name] = {"correct": 0, "total": 0}
per_category[cat_name]["total"] += 1
if true == pred:
per_category[cat_name]["correct"] += 1
for cat in per_category:
t = per_category[cat]["total"]
c = per_category[cat]["correct"]
per_category[cat]["accuracy"] = c / t if t > 0 else 0.0
# Package everything into a results dict
results = {
"model": "V-JEPA 2.1 ViT-g (1B)",
"method": "linear_probe",
"overall_accuracy": overall_acc,
"num_train": len(X_train),
"num_test": len(X_test),
"per_category": per_category,
"predictions": [
{"id": vid, "true": int(t), "pred": int(p)}
for vid, t, p in zip(ids_test, y_test, y_pred)
],
}
results_path = os.path.join(RESULTS_DIR, "vjepa_results.json")
with open(results_path, "w") as f:
json.dump(results, f, indent=2)
print(f"\nResults saved to {results_path}")
print("\nPer-category accuracy:")
for cat, data in sorted(per_category.items(), key=lambda x: x[1]["accuracy"], reverse=True):
print(f" {cat}: {data['accuracy']:.2%} ({data['correct']}/{data['total']})")
if __name__ == "__main__":
main()What does this code do?
- train_test_split with stratify=labels ensures every action category appears in both the training and test sets, preventing the classifier from never seeing a category during training.
- LogisticRegression(max_iter=1000, C=1.0) trains a simple linear classifier. max_iter=1000 gives the solver enough iterations to converge on high-dimensional features.
- The per-category accuracy loop tallies correct predictions for each of the 20 action templates separately, so you can see which temporal actions V-JEPA distinguishes well and which it struggles with.
- The final results dict captures everything needed for the comparison step: overall accuracy, per-category breakdowns, and individual predictions keyed by video ID.
- Save evaluate_vjepa.py.
✔️ Awesome, I've got everything!
Double check you have saved the file before running it in the next substep.
ⓧ I'd like to double check the full code
Here is the complete evaluate_vjepa.py file. Make sure yours matches exactly.
"""Evaluate V-JEPA 2.1 on SSv2 subset using feature extraction + linear probe."""
import json
import os
import numpy as np
import torch
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
from tqdm import tqdm
SSV2_ROOT = os.environ.get("SSV2_ROOT", "data/sthv2")
SUBSET_META = os.path.join(SSV2_ROOT, "eval_subset", "subset_metadata.json")
VIDEO_DIR = os.path.join(SSV2_ROOT, "20bn-something-something-v2")
RESULTS_DIR = "results"
SEED = 42
def load_subset():
with open(SUBSET_META) as f:
return json.load(f)
def get_video_path(video_id):
return os.path.join(VIDEO_DIR, f"{video_id}.webm")
def main():
os.makedirs(RESULTS_DIR, exist_ok=True)
print("Loading V-JEPA 2.1 ViT-g model...")
processor = torch.hub.load("facebookresearch/vjepa2", "vjepa2_preprocessor")
model = torch.hub.load("facebookresearch/vjepa2", "vjepa2_1_vit_giant_384")
model.eval().cuda()
subset = load_subset()
print(f"Loaded {len(subset)} videos from subset metadata.")
features_list = []
labels_list = []
video_ids = []
skipped = 0
print("Extracting features...")
for item in tqdm(subset):
video_path = get_video_path(item["id"])
if not os.path.exists(video_path):
skipped += 1
continue
try:
video_tensor = processor(video_path)
if video_tensor.dim() == 4:
video_tensor = video_tensor.unsqueeze(0)
video_tensor = video_tensor.cuda()
with torch.no_grad():
output = model(video_tensor)
if hasattr(output, "last_hidden_state"):
tokens = output.last_hidden_state
elif isinstance(output, torch.Tensor):
tokens = output
else:
tokens = output[0]
pooled = tokens.mean(dim=1).squeeze(0).cpu().numpy()
features_list.append(pooled)
labels_list.append(item["category_idx"])
video_ids.append(item["id"])
except Exception as e:
print(f"Error processing {item['id']}: {e}")
skipped += 1
if skipped > 0:
print(f"Skipped {skipped} videos due to errors or missing files.")
features = np.array(features_list)
labels = np.array(labels_list)
print(f"Extracted features shape: {features.shape}")
X_train, X_test, y_train, y_test, ids_train, ids_test = train_test_split(
features, labels, video_ids, test_size=0.2, random_state=SEED, stratify=labels
)
print("Training linear probe...")
clf = LogisticRegression(max_iter=1000, random_state=SEED, C=1.0)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
overall_acc = accuracy_score(y_test, y_pred)
print(f"\nV-JEPA 2.1 Overall Accuracy: {overall_acc:.4f}")
with open(SUBSET_META) as f:
all_items = json.load(f)
idx_to_cat = {}
for item in all_items:
idx_to_cat[item["category_idx"]] = item["template"]
category_names = [idx_to_cat.get(i, f"cat_{i}") for i in sorted(idx_to_cat.keys())]
per_category = {}
for true, pred in zip(y_test, y_pred):
cat_name = idx_to_cat.get(true, f"cat_{true}")
if cat_name not in per_category:
per_category[cat_name] = {"correct": 0, "total": 0}
per_category[cat_name]["total"] += 1
if true == pred:
per_category[cat_name]["correct"] += 1
for cat in per_category:
t = per_category[cat]["total"]
c = per_category[cat]["correct"]
per_category[cat]["accuracy"] = c / t if t > 0 else 0.0
results = {
"model": "V-JEPA 2.1 ViT-g (1B)",
"method": "linear_probe",
"overall_accuracy": overall_acc,
"num_train": len(X_train),
"num_test": len(X_test),
"per_category": per_category,
"predictions": [
{"id": vid, "true": int(t), "pred": int(p)}
for vid, t, p in zip(ids_test, y_test, y_pred)
],
}
results_path = os.path.join(RESULTS_DIR, "vjepa_results.json")
with open(results_path, "w") as f:
json.dump(results, f, indent=2)
print(f"\nResults saved to {results_path}")
print("\nPer-category accuracy:")
for cat, data in sorted(per_category.items(), key=lambda x: x[1]["accuracy"], reverse=True):
print(f" {cat}: {data['accuracy']:.2%} ({data['correct']}/{data['total']})")
if __name__ == "__main__":
main()Run the evaluation
The script is complete. Time to run the full pipeline: load the model, extract features from all 500 videos, train the linear probe, and produce per-category accuracy results.
- Make sure your vjepa-bench conda environment is active.
- Run the evaluation script by executing this command:
python evaluate_vjepa.pyWhat should I see?
The script prints status messages as it progresses. First, you will see the model loading from torch.hub (this may download weights on the first run). Then a progress bar appears as features are extracted from each video.
After feature extraction completes, the linear probe trains in seconds. The script finishes by printing overall accuracy and a per-category breakdown sorted from highest to lowest accuracy.
- Verify that results/vjepa_results.json was created by running:
ls results/You should see vjepa_results.json in the output.
Seeing a CUDA out of memory error?
The ViT-g model has 1B parameters. Make sure you are running on a GPU with enough VRAM (an A100 handles this comfortably). If you are sharing the GPU, check whether another process is using memory with nvidia-smi.
If many videos are being skipped, confirm that the video files in data/sthv2/20bn-something-something-v2/ were fully extracted and that the filenames match the IDs in subset_metadata.json (they should be <id>.webm).
Help me debug a CUDA out of memory or missing video error when running evaluate_vjepa.py
Feature extraction is slow
Processing 500 videos through a 1B-parameter encoder takes time, even on an A100. Expect the extraction loop to run for several minutes. The tqdm progress bar shows you exactly how far along it is.
You now have quantitative results for how well a world model's self-supervised representations handle temporal action classification. Next up, you will run Qwen2.5-VL-7B on the exact same videos to see how a multimodal LLM approaches the same task.
Evaluate Qwen2.5-VL-7B on the Same Videos
You now have V-JEPA 2.1's results sitting in results/vjepa_results.json. Time to put the other paradigm to the test.
In this step, you will evaluate Qwen2.5-VL-7B on the exact same 500 videos using prompted multiple-choice classification. Instead of extracting features and training a linear probe, this approach feeds each video directly to the LLM and asks it to pick the correct action from a set of five options.
In this step, get ready to:
- Write evaluate_qwen.py with a multiple-choice prompt and answer parser.
- Run prompted video classification on all 500 subset videos.
- Save per-category and overall accuracy results to results/qwen_results.json.
Load Qwen2.5-VL-7B and design the prompt
V-JEPA 2.1 learned video representations through self-supervised prediction in embedding space. Qwen2.5-VL takes a completely different approach. It processes video frames as visual tokens and generates text responses through autoregressive generation.
To make the comparison fair and parseable, you will frame each video as a multiple-choice question with five options. One option is the correct action label, and four are randomly sampled distractors from the other 19 categories.
- Create a new file called evaluate_qwen.py in your project root (the same folder as evaluate_vjepa.py).
- Add the imports, constants, and utility functions by pasting this code:
"""Evaluate Qwen2.5-VL-7B-Instruct on SSv2 subset using prompted video classification."""
import json
import os
import random
import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from tqdm import tqdm
# Paths matching the V-JEPA evaluation script
SSV2_ROOT = os.environ.get("SSV2_ROOT", "data/sthv2")
SUBSET_META = os.path.join(SSV2_ROOT, "eval_subset", "subset_metadata.json")
VIDEO_DIR = os.path.join(SSV2_ROOT, "20bn-something-something-v2")
RESULTS_DIR = "results"
SEED = 42
def load_subset():
with open(SUBSET_META) as f:
return json.load(f)
def get_video_path(video_id):
return os.path.join(VIDEO_DIR, f"{video_id}.webm")
def build_choices(correct_template, all_templates, num_distractors=4):
"""Build a multiple-choice list with the correct answer and random distractors."""
distractors = [t for t in all_templates if t != correct_template]
chosen = random.sample(distractors, min(num_distractors, len(distractors)))
choices = [correct_template] + chosen
random.shuffle(choices)
return choices
def parse_answer(response_text, choices):
"""Parse the model response to find which choice it selected."""
response_lower = response_text.strip().lower()
# First pass: check if the response starts with a letter like "A" or "(A)"
for letter_idx, choice in enumerate(choices):
letter = chr(ord("A") + letter_idx)
if response_lower.startswith(letter.lower()) or response_lower.startswith(f"({letter.lower()})"):
return letter_idx
# Second pass: check if any choice text appears in the response
for idx, choice in enumerate(choices):
if choice.lower() in response_lower:
return idx
# No match found
return -1What does this code do?
The constants (SSV2_ROOT, SUBSET_META, etc.) match the paths in evaluate_vjepa.py so both scripts read the exact same videos.
- build_choices() constructs a 5-option multiple-choice list for each video. It takes the correct action template, samples 4 random distractors from the other 19 categories, shuffles them together, and returns the list.
- parse_answer() extracts the model's predicted choice from its text response. It first checks for a leading letter ("A", "(B)"), then falls back to matching the full choice text in the response. If neither works, it returns -1 (unparseable).
Why multiple-choice instead of open-ended?
Open-ended classification ("What is happening in this video?") produces free-form text that is hard to parse reliably into one of 20 categories. The model might rephrase an action or describe it at a different level of detail.
Multiple-choice with 5 options gives a clean 20% random baseline and produces parseable answers. This is consistent with benchmarks like MVPBench that evaluate multimodal LLMs on video tasks.
You won't see a change yet. The next chunk adds the main() function that wires everything together.
Write the classification loop and results saving
The main() function loads the Qwen model, iterates over every video in the subset, constructs a chat message with the video and the multiple-choice question, and records each prediction.
- In evaluate_qwen.py, add the main() function below parse_answer() by pasting this code:
def main():
random.seed(SEED)
os.makedirs(RESULTS_DIR, exist_ok=True)
# Load the Qwen2.5-VL-7B model in bfloat16 with automatic GPU sharding
print("Loading Qwen2.5-VL-7B-Instruct...")
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2.5-VL-7B-Instruct",
torch_dtype=torch.bfloat16,
device_map="auto",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
# Load the same 500-video subset and extract all unique category templates
subset = load_subset()
all_templates = sorted(set(item["template"] for item in subset))
print(f"Loaded {len(subset)} videos, {len(all_templates)} categories.")
predictions = []
correct_count = 0
total_count = 0
per_category = {}
skipped = 0
print("Running prompted classification...")
for item in tqdm(subset):
video_path = get_video_path(item["id"])
if not os.path.exists(video_path):
skipped += 1
continue
# Build a 5-option multiple-choice question for this video
choices = build_choices(item["template"], all_templates)
correct_idx = choices.index(item["template"])
choices_text = "\n".join(
f"({chr(ord('A') + i)}) {choice}" for i, choice in enumerate(choices)
)
question = (
f"Watch this video carefully. What action is being performed?\n\n"
f"{choices_text}\n\n"
f"Reply with only the letter of the correct answer."
)
# Construct the chat message with the video and the question
messages = [
{
"role": "user",
"content": [
{"type": "video", "path": video_path},
{"type": "text", "text": question},
],
}
]
try:
# Process video at 1 fps and tokenize the chat
inputs = processor.apply_chat_template(
messages,
fps=1,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
# Generate the model's answer
generated_ids = model.generate(**inputs, max_new_tokens=32)
generated_ids_trimmed = [
out_ids[len(in_ids):]
for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
response = processor.batch_decode(
generated_ids_trimmed,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
# Parse and score the prediction
pred_idx = parse_answer(response, choices)
is_correct = pred_idx == correct_idx
cat = item["template"]
if cat not in per_category:
per_category[cat] = {"correct": 0, "total": 0}
per_category[cat]["total"] += 1
if is_correct:
per_category[cat]["correct"] += 1
correct_count += 1
total_count += 1
predictions.append({
"id": item["id"],
"true_category": cat,
"true_idx": item["category_idx"],
"pred_idx": pred_idx,
"correct": is_correct,
"response": response.strip(),
})
except Exception as e:
print(f"Error processing {item['id']}: {e}")
skipped += 1
if skipped > 0:
print(f"Skipped {skipped} videos.")
# Compute overall and per-category accuracy
overall_acc = correct_count / total_count if total_count > 0 else 0.0
print(f"\nQwen2.5-VL-7B Overall Accuracy: {overall_acc:.4f}")
for cat in per_category:
t = per_category[cat]["total"]
c = per_category[cat]["correct"]
per_category[cat]["accuracy"] = c / t if t > 0 else 0.0
# Save results in the same format as vjepa_results.json
results = {
"model": "Qwen2.5-VL-7B-Instruct",
"method": "prompted_multiple_choice",
"overall_accuracy": overall_acc,
"num_evaluated": total_count,
"per_category": per_category,
"predictions": predictions,
}
results_path = os.path.join(RESULTS_DIR, "qwen_results.json")
with open(results_path, "w") as f:
json.dump(results, f, indent=2)
print(f"Results saved to {results_path}")
print("\nPer-category accuracy:")
for cat, data in sorted(per_category.items(), key=lambda x: x[1]["accuracy"], reverse=True):
print(f" {cat}: {data['accuracy']:.2%} ({data['correct']}/{data['total']})")
if __name__ == "__main__":
main()What does this code do?
This function drives the entire Qwen evaluation pipeline:
- Qwen2_5_VLForConditionalGeneration.from_pretrained() loads the 7B-parameter model in bfloat16 precision. The device_map="auto" setting automatically shards the model across your available GPUs.
- processor.apply_chat_template() converts each video and question into tokens the model can process. The fps=1 setting samples one frame per second from each video, keeping input length manageable.
- model.generate(**inputs, max_new_tokens=32) runs autoregressive generation. The response is trimmed to remove the input tokens, then decoded back to text.
- The results dictionary mirrors the structure of vjepa_results.json so the comparison script in the next step can align them by video ID.
How is this evaluation different from V-JEPA's?
The V-JEPA evaluation extracted frozen features and trained a logistic regression classifier on labeled examples. That is a supervised evaluation of the representation quality.
The Qwen evaluation is zero-shot. It sees no training examples at all. It relies entirely on the knowledge baked into its weights during pre-training. This asymmetry is intentional. The research question is about paradigm differences, not about which model scores higher on a leaderboard.
- Save evaluate_qwen.py.
Double check your complete file matches before running it.
✔️ Awesome, I've got everything!
Great. Make sure you have saved evaluate_qwen.py before moving on.
ⓧ I'd like to double check the full code
"""Evaluate Qwen2.5-VL-7B-Instruct on SSv2 subset using prompted video classification."""
import json
import os
import random
import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from tqdm import tqdm
SSV2_ROOT = os.environ.get("SSV2_ROOT", "data/sthv2")
SUBSET_META = os.path.join(SSV2_ROOT, "eval_subset", "subset_metadata.json")
VIDEO_DIR = os.path.join(SSV2_ROOT, "20bn-something-something-v2")
RESULTS_DIR = "results"
SEED = 42
def load_subset():
with open(SUBSET_META) as f:
return json.load(f)
def get_video_path(video_id):
return os.path.join(VIDEO_DIR, f"{video_id}.webm")
def build_choices(correct_template, all_templates, num_distractors=4):
"""Build a multiple-choice list with the correct answer and random distractors."""
distractors = [t for t in all_templates if t != correct_template]
chosen = random.sample(distractors, min(num_distractors, len(distractors)))
choices = [correct_template] + chosen
random.shuffle(choices)
return choices
def parse_answer(response_text, choices):
"""Parse the model response to find which choice it selected."""
response_lower = response_text.strip().lower()
for letter_idx, choice in enumerate(choices):
letter = chr(ord("A") + letter_idx)
if response_lower.startswith(letter.lower()) or response_lower.startswith(f"({letter.lower()})"):
return letter_idx
for idx, choice in enumerate(choices):
if choice.lower() in response_lower:
return idx
return -1
def main():
random.seed(SEED)
os.makedirs(RESULTS_DIR, exist_ok=True)
print("Loading Qwen2.5-VL-7B-Instruct...")
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2.5-VL-7B-Instruct",
torch_dtype=torch.bfloat16,
device_map="auto",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
subset = load_subset()
all_templates = sorted(set(item["template"] for item in subset))
print(f"Loaded {len(subset)} videos, {len(all_templates)} categories.")
predictions = []
correct_count = 0
total_count = 0
per_category = {}
skipped = 0
print("Running prompted classification...")
for item in tqdm(subset):
video_path = get_video_path(item["id"])
if not os.path.exists(video_path):
skipped += 1
continue
choices = build_choices(item["template"], all_templates)
correct_idx = choices.index(item["template"])
choices_text = "\n".join(
f"({chr(ord('A') + i)}) {choice}" for i, choice in enumerate(choices)
)
question = (
f"Watch this video carefully. What action is being performed?\n\n"
f"{choices_text}\n\n"
f"Reply with only the letter of the correct answer."
)
messages = [
{
"role": "user",
"content": [
{"type": "video", "path": video_path},
{"type": "text", "text": question},
],
}
]
try:
inputs = processor.apply_chat_template(
messages,
fps=1,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=32)
generated_ids_trimmed = [
out_ids[len(in_ids):]
for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
response = processor.batch_decode(
generated_ids_trimmed,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
pred_idx = parse_answer(response, choices)
is_correct = pred_idx == correct_idx
cat = item["template"]
if cat not in per_category:
per_category[cat] = {"correct": 0, "total": 0}
per_category[cat]["total"] += 1
if is_correct:
per_category[cat]["correct"] += 1
correct_count += 1
total_count += 1
predictions.append({
"id": item["id"],
"true_category": cat,
"true_idx": item["category_idx"],
"pred_idx": pred_idx,
"correct": is_correct,
"response": response.strip(),
})
except Exception as e:
print(f"Error processing {item['id']}: {e}")
skipped += 1
if skipped > 0:
print(f"Skipped {skipped} videos.")
overall_acc = correct_count / total_count if total_count > 0 else 0.0
print(f"\nQwen2.5-VL-7B Overall Accuracy: {overall_acc:.4f}")
for cat in per_category:
t = per_category[cat]["total"]
c = per_category[cat]["correct"]
per_category[cat]["accuracy"] = c / t if t > 0 else 0.0
results = {
"model": "Qwen2.5-VL-7B-Instruct",
"method": "prompted_multiple_choice",
"overall_accuracy": overall_acc,
"num_evaluated": total_count,
"per_category": per_category,
"predictions": predictions,
}
results_path = os.path.join(RESULTS_DIR, "qwen_results.json")
with open(results_path, "w") as f:
json.dump(results, f, indent=2)
print(f"Results saved to {results_path}")
print("\nPer-category accuracy:")
for cat, data in sorted(per_category.items(), key=lambda x: x[1]["accuracy"], reverse=True):
print(f" {cat}: {data['accuracy']:.2%} ({data['correct']}/{data['total']})")
if __name__ == "__main__":
main()Run the evaluation
With the script complete, run it on the full 500-video subset. This step takes significantly longer than the V-JEPA evaluation because Qwen generates text autoregressively for each video instead of running a single forward pass.
- Make sure your vjepa-bench conda environment is active.
- Run the Qwen evaluation by executing this command:
python evaluate_qwen.py- You should see a progress bar from tqdm as the script processes each video.
- When it finishes, the terminal prints the overall accuracy and a per-category breakdown.
- Confirm the results file exists by running:
ls results/You should see both vjepa_results.json and qwen_results.json listed in the output.
Script crashing or running out of memory?
- If you see an OutOfMemoryError, the model may be loading onto a single GPU. Check that accelerate is installed. The device_map="auto" setting requires it to shard the model across GPUs.
- If you see FileNotFoundError for video files, double check that your SSV2_ROOT environment variable points to the correct location. The script expects videos at data/sthv2/20bn-something-something-v2/<id>.webm.
- If the model weights are not cached locally, the first run downloads ~14 GB from HuggingFace. If your supercomputer restricts outbound network access on compute nodes, download on a login node first using huggingface-cli download (you did this in Step 1).
Still stuck? Help me debug why my evaluate_qwen.py script is failing. I'm running Qwen2.5-VL-7B-Instruct on A100 GPUs with device_map auto.
Expect a long wait
Processing 500 videos through autoregressive generation is much slower than a single-pass feature extraction. On an A100, expect the Qwen evaluation to take several times longer than the V-JEPA run. Consider starting it and checking back later.
You now have results from both paradigms. Next up, you will write compare_results.py to put them side by side and see where each model shines.
Compare World Model vs. LLM Performance
You have individual evaluation results for both V-JEPA 2.1 and Qwen2.5-VL-7B sitting in your results/ directory. Raw accuracy numbers alone only tell half the story.
In this step, you will bring both models' predictions together into a single comparison script. You will compute per-category accuracy breakdowns to see where embedding-space prediction (the world model) outperforms next-token prediction (the LLM) and vice versa. Then you will produce a grouped bar chart and a written analysis summary.
In this step, get ready to:
- Load and align both models' result files for a fair comparison.
- Compute per-category accuracy and identify where each model wins or loses.
- Generate a grouped bar chart and a written analysis of world model vs. LLM performance.
Load and align results from both models
Both vjepa_results.json and qwen_results.json contain a per_category dictionary mapping category names to accuracy scores. The comparison script needs to merge these into aligned lists so each category has a score from both models side by side.
- Create a new file called compare_results.py in your project root (the same directory as your other scripts) and paste the following code:
"""Compare V-JEPA 2.1 and Qwen2.5-VL-7B results and generate visualizations."""
import json
import os
import matplotlib.pyplot as plt
import numpy as np
RESULTS_DIR = "results"
def load_results(filename):
path = os.path.join(RESULTS_DIR, filename)
with open(path) as f:
return json.load(f)
def main():
vjepa = load_results("vjepa_results.json")
qwen = load_results("qwen_results.json")
print("=" * 70)
print("WORLD MODEL vs. MULTIMODAL LLM COMPARISON")
print("=" * 70)
print(f"\nV-JEPA 2.1 ({vjepa['method']}): {vjepa['overall_accuracy']:.2%}")
print(f"Qwen2.5-VL-7B ({qwen['method']}): {qwen['overall_accuracy']:.2%}")
# Align categories from both result files
all_cats = sorted(
set(list(vjepa["per_category"].keys()) + list(qwen["per_category"].keys()))
)
vjepa_accs = []
qwen_accs = []
labels = []
for cat in all_cats:
v_acc = vjepa["per_category"].get(cat, {}).get("accuracy", 0.0)
q_acc = qwen["per_category"].get(cat, {}).get("accuracy", 0.0)
# Shorten labels for chart readability
short_label = cat.replace("[something]", "[s]").replace("something", "s")
if len(short_label) > 35:
short_label = short_label[:32] + "..."
vjepa_accs.append(v_acc)
qwen_accs.append(q_acc)
labels.append(short_label)What does this code do?
load_results() is a small helper that reads a JSON file from the results/ directory. The main() function loads both result files and prints the overall accuracy for each model.
The alignment loop takes the union of all category names from both files, then builds parallel lists of accuracies. This ensures you compare both models on exactly the same set of categories, even if one file happens to be missing a category.
You won't run this script yet. The next chunk adds the comparison table and win/loss classification.
Compute per-category accuracy breakdown
With aligned accuracy lists ready, you can now classify each category as a win for V-JEPA 2.1, a win for Qwen2.5-VL-7B, or a tie. This breakdown reveals the pattern behind the overall numbers.
- In compare_results.py, find the last line of the for cat in all_cats: loop (the line labels.append(short_label)). Add the following code below it, inside main() but outside the loop:
vjepa_wins = []
qwen_wins = []
ties = []
print("\n" + "-" * 70)
print(f"{'Category':<45} {'V-JEPA':>8} {'Qwen':>8} {'Winner':>10}")
print("-" * 70)
for cat, v, q in zip(all_cats, vjepa_accs, qwen_accs):
short = cat.replace("[something]", "[s]")
if len(short) > 42:
short = short[:39] + "..."
if v > q:
winner = "V-JEPA"
vjepa_wins.append(cat)
elif q > v:
winner = "Qwen"
qwen_wins.append(cat)
else:
winner = "Tie"
ties.append(cat)
print(f" {short:<43} {v:>7.0%} {q:>7.0%} {winner:>10}")
print("-" * 70)
print(f"\nV-JEPA wins: {len(vjepa_wins)} categories")
print(f"Qwen wins: {len(qwen_wins)} categories")
print(f"Ties: {len(ties)} categories")What does this code do?
This loop iterates over every category and compares the two models' accuracies head to head. Each category is classified into one of three lists: vjepa_wins, qwen_wins, or ties.
The formatted table prints each category's accuracy for both models alongside a winner label. This gives you an at-a-glance view of which paradigm dominates which action types.
You won't run the script yet. The next chunk adds the chart and summary file, which complete the script.
Generate visualizations and save the analysis
A comparison table in the terminal is useful, but a grouped bar chart makes the differences between V-JEPA 2.1 and Qwen2.5-VL-7B immediately visible. You will also write a structured analysis summary to a text file for future reference.
- In compare_results.py, find the line print(f"Ties: {len(ties)} categories"). Add the following code below it, still inside main():
# Create grouped bar chart
fig, ax = plt.subplots(figsize=(14, 8))
x = np.arange(len(labels))
width = 0.35
bars1 = ax.bar(x - width / 2, vjepa_accs, width, label="V-JEPA 2.1 (World Model)", color="#2196F3")
bars2 = ax.bar(x + width / 2, qwen_accs, width, label="Qwen2.5-VL-7B (LLM)", color="#FF9800")
ax.set_ylabel("Accuracy", fontsize=12)
ax.set_title("World Model vs. Multimodal LLM: Per-Category Accuracy on SSv2", fontsize=14)
ax.set_xticks(x)
ax.set_xticklabels(labels, rotation=45, ha="right", fontsize=8)
ax.legend(fontsize=11)
ax.set_ylim(0, 1.05)
ax.axhline(y=0.2, color="gray", linestyle="--", alpha=0.5, label="Random (1/5)")
plt.tight_layout()
chart_path = os.path.join(RESULTS_DIR, "comparison_chart.png")
plt.savefig(chart_path, dpi=150)
print(f"\nChart saved to {chart_path}")What does the chart code do?
matplotlib draws a grouped bar chart with blue bars for V-JEPA 2.1 and orange bars for Qwen2.5-VL-7B, placed side by side for each category.
The dashed gray line at y=0.2 represents random chance (1 out of 5 choices). Any bar below this line means the model is performing worse than guessing.
- Now add the summary text file writer and the __name__ guard directly below the chart code you just pasted, to complete the file:
# Write structured analysis summary
summary_lines = [
"WORLD MODEL vs. MULTIMODAL LLM: ANALYSIS SUMMARY",
"=" * 50,
"",
f"V-JEPA 2.1 ViT-g (1B params) - Linear Probe",
f" Overall Accuracy: {vjepa['overall_accuracy']:.2%}",
f" Evaluation method: Feature extraction + logistic regression",
"",
f"Qwen2.5-VL-7B-Instruct",
f" Overall Accuracy: {qwen['overall_accuracy']:.2%}",
f" Evaluation method: Prompted multiple-choice classification",
"",
f"V-JEPA wins {len(vjepa_wins)} categories, Qwen wins {len(qwen_wins)}, Ties: {len(ties)}",
"",
"KEY INSIGHT:",
"V-JEPA 2.1 predicts in embedding space, learning temporal dynamics",
"from self-supervised video observation. Qwen2.5-VL predicts next tokens,",
"relying on language grounding to describe what it sees.",
"",
"Categories where V-JEPA excels likely involve fine-grained motion",
"distinctions (e.g., pushing left vs. right) that require genuine",
"temporal understanding rather than scene-level description.",
"",
"Categories where Qwen excels likely involve actions that benefit",
"from semantic/language priors (e.g., pouring, folding) where the",
"LLM's training data provides strong action-label associations.",
]
summary_path = os.path.join(RESULTS_DIR, "analysis_summary.txt")
with open(summary_path, "w") as f:
f.write("\n".join(summary_lines))
print(f"Summary saved to {summary_path}")
if __name__ == "__main__":
main()What does the summary writer do?
The script writes a plain text file that records both models' overall accuracy, the win/loss/tie breakdown, and a key insight paragraph. This file serves as a portable record of your experiment that you can reference or share without re-running the script.
- Save compare_results.py.
✔️ Awesome, I've got everything!
Great. Double check you have saved the file before running it.
ⓧ I'd like to double check the full code
"""Compare V-JEPA 2.1 and Qwen2.5-VL-7B results and generate visualizations."""
import json
import os
import matplotlib.pyplot as plt
import numpy as np
RESULTS_DIR = "results"
def load_results(filename):
path = os.path.join(RESULTS_DIR, filename)
with open(path) as f:
return json.load(f)
def main():
vjepa = load_results("vjepa_results.json")
qwen = load_results("qwen_results.json")
print("=" * 70)
print("WORLD MODEL vs. MULTIMODAL LLM COMPARISON")
print("=" * 70)
print(f"\nV-JEPA 2.1 ({vjepa['method']}): {vjepa['overall_accuracy']:.2%}")
print(f"Qwen2.5-VL-7B ({qwen['method']}): {qwen['overall_accuracy']:.2%}")
all_cats = sorted(
set(list(vjepa["per_category"].keys()) + list(qwen["per_category"].keys()))
)
vjepa_accs = []
qwen_accs = []
labels = []
for cat in all_cats:
v_acc = vjepa["per_category"].get(cat, {}).get("accuracy", 0.0)
q_acc = qwen["per_category"].get(cat, {}).get("accuracy", 0.0)
short_label = cat.replace("[something]", "[s]").replace("something", "s")
if len(short_label) > 35:
short_label = short_label[:32] + "..."
vjepa_accs.append(v_acc)
qwen_accs.append(q_acc)
labels.append(short_label)
vjepa_wins = []
qwen_wins = []
ties = []
print("\n" + "-" * 70)
print(f"{'Category':<45} {'V-JEPA':>8} {'Qwen':>8} {'Winner':>10}")
print("-" * 70)
for cat, v, q in zip(all_cats, vjepa_accs, qwen_accs):
short = cat.replace("[something]", "[s]")
if len(short) > 42:
short = short[:39] + "..."
if v > q:
winner = "V-JEPA"
vjepa_wins.append(cat)
elif q > v:
winner = "Qwen"
qwen_wins.append(cat)
else:
winner = "Tie"
ties.append(cat)
print(f" {short:<43} {v:>7.0%} {q:>7.0%} {winner:>10}")
print("-" * 70)
print(f"\nV-JEPA wins: {len(vjepa_wins)} categories")
print(f"Qwen wins: {len(qwen_wins)} categories")
print(f"Ties: {len(ties)} categories")
fig, ax = plt.subplots(figsize=(14, 8))
x = np.arange(len(labels))
width = 0.35
bars1 = ax.bar(x - width / 2, vjepa_accs, width, label="V-JEPA 2.1 (World Model)", color="#2196F3")
bars2 = ax.bar(x + width / 2, qwen_accs, width, label="Qwen2.5-VL-7B (LLM)", color="#FF9800")
ax.set_ylabel("Accuracy", fontsize=12)
ax.set_title("World Model vs. Multimodal LLM: Per-Category Accuracy on SSv2", fontsize=14)
ax.set_xticks(x)
ax.set_xticklabels(labels, rotation=45, ha="right", fontsize=8)
ax.legend(fontsize=11)
ax.set_ylim(0, 1.05)
ax.axhline(y=0.2, color="gray", linestyle="--", alpha=0.5, label="Random (1/5)")
plt.tight_layout()
chart_path = os.path.join(RESULTS_DIR, "comparison_chart.png")
plt.savefig(chart_path, dpi=150)
print(f"\nChart saved to {chart_path}")
summary_lines = [
"WORLD MODEL vs. MULTIMODAL LLM: ANALYSIS SUMMARY",
"=" * 50,
"",
f"V-JEPA 2.1 ViT-g (1B params) - Linear Probe",
f" Overall Accuracy: {vjepa['overall_accuracy']:.2%}",
f" Evaluation method: Feature extraction + logistic regression",
"",
f"Qwen2.5-VL-7B-Instruct",
f" Overall Accuracy: {qwen['overall_accuracy']:.2%}",
f" Evaluation method: Prompted multiple-choice classification",
"",
f"V-JEPA wins {len(vjepa_wins)} categories, Qwen wins {len(qwen_wins)}, Ties: {len(ties)}",
"",
"KEY INSIGHT:",
"V-JEPA 2.1 predicts in embedding space, learning temporal dynamics",
"from self-supervised video observation. Qwen2.5-VL predicts next tokens,",
"relying on language grounding to describe what it sees.",
"",
"Categories where V-JEPA excels likely involve fine-grained motion",
"distinctions (e.g., pushing left vs. right) that require genuine",
"temporal understanding rather than scene-level description.",
"",
"Categories where Qwen excels likely involve actions that benefit",
"from semantic/language priors (e.g., pouring, folding) where the",
"LLM's training data provides strong action-label associations.",
]
summary_path = os.path.join(RESULTS_DIR, "analysis_summary.txt")
with open(summary_path, "w") as f:
f.write("\n".join(summary_lines))
print(f"Summary saved to {summary_path}")
if __name__ == "__main__":
main()- Run the comparison script by executing this command:
python compare_results.pyYou should see the overall accuracy for both models, a full comparison table with a winner for each of the 20 categories, and confirmation messages that the chart and summary were saved.
What should I see?
The terminal output starts with a header showing overall accuracy percentages for both models. Below that is a table listing each of the 20 temporal action categories with accuracy columns for V-JEPA and Qwen, plus a "Winner" column.
At the bottom, you see a summary line like "V-JEPA wins: X categories" and "Qwen wins: Y categories". Then two file-saved confirmation messages for comparison_chart.png and analysis_summary.txt.
- Verify that the output files were created by running:
ls results/You should see comparison_chart.png and analysis_summary.txt alongside your existing vjepa_results.json and qwen_results.json.
Seeing a FileNotFoundError?
This means one of your result JSON files is missing or named differently. Make sure both results/vjepa_results.json and results/qwen_results.json exist from the previous steps.
If you see a KeyError on per_category, the JSON structure may not match what the script expects. Re-run evaluate_vjepa.py or evaluate_qwen.py to regenerate the results.
Why is my comparison script failing to load results?
What do the results reveal about world models vs. LLMs?
V-JEPA 2.1 predicts in embedding space, learning temporal dynamics from self-supervised video observation. Qwen2.5-VL-7B predicts next tokens, relying on language grounding to describe what it sees.
Categories where V-JEPA excels likely involve fine-grained motion distinctions (pushing left vs. right) that require genuine temporal understanding. Categories where Qwen excels likely involve actions that benefit from semantic priors (pouring, folding) where the LLM's training data provides strong action-label associations.
Why are these evaluation methods different?
The linear probe requires labeled training data, while the prompted approach is zero-shot. This is intentional. The research question is about paradigm differences, not which model is "better" in absolute terms.
A linear probe measures the quality of the frozen features a model has already learned. Prompted classification measures how well a model can reason about video content through language. Both are standard evaluation protocols for their respective paradigms.
Secret mission
Visualize What V-JEPA 2.1 Sees Internally
You have the numbers. Now see what the world model actually sees. Extract V-JEPA 2.1's dense token-level features and visualize them with PCA to compare how embedding-space prediction captures motion differently from language-based reasoning.
Before you go
Clean Up Your Resources
This project runs entirely locally on your supercomputer, so there are no ongoing cloud costs. The main consideration is disk space, since the model checkpoints and dataset take up significant storage.
Decide whether to keep your resources for further experimentation, pause by removing the heaviest files, or delete everything entirely.
Resources you used:
- Conda environment vjepa-bench with Python 3.12, PyTorch, and all dependencies.
- V-JEPA 2.1 cloned repository (vjepa2/).
- V-JEPA 2.1 ViT-g checkpoint in checkpoints/vitg-384.pt.
- Qwen2.5-VL-7B-Instruct weights in checkpoints/qwen2.5-vl-7b/.
- Something-Something V2 dataset (~19.4 GB) in data/sthv2/.
- Project scripts (prepare_subset.py, evaluate_vjepa.py, evaluate_qwen.py, compare_results.py).
- Results directory with JSON outputs and visualizations (results/).
Keep everything running
No action needed. Choose this if you plan to continue experimenting with the benchmarking pipeline, add more evaluation categories, or extend the analysis.
- All files, checkpoints, and the conda environment stay in place.
- Be aware that the dataset and model checkpoints together use over 30 GB of disk space on your supercomputer.
Pause - I'll come back to this later
Free up disk space by removing the largest files while keeping your scripts and results intact. You can re-download checkpoints and data later.
- Delete the V-JEPA checkpoint to reclaim space:
rm checkpoints/vitg-384.pt- Delete the Qwen model weights:
rm -rf checkpoints/qwen2.5-vl-7b- Optionally remove the SSv2 video data (the largest portion at ~19.4 GB) while keeping your curated subset metadata:
rm -rf data/sthv2/20bn-something-something-v2- Keep your scripts, results, annotations, and conda environment so you can pick up where you left off.
Delete - I don't want to use this again
Remove all project resources and start fresh. This deletes everything including scripts, data, checkpoints, and the conda environment.
- Deactivate and remove the conda environment:
conda deactivate
conda env remove -n vjepa-bench- Delete the entire project directory (run this from the parent folder):
rm -rf vjepa2
rm -rf checkpoints
rm -rf data
rm -rf results
rm -f prepare_subset.py evaluate_vjepa.py evaluate_qwen.py compare_results.py- Optionally clear the HuggingFace cache if you downloaded Qwen weights there:
rm -rf ~/.cache/huggingface/hub/models--Qwen--Qwen2.5-VL-7B-InstructMission accomplished
That's a Wrap!
Nice work! You just built a research-grade benchmarking pipeline comparing two fundamentally different approaches to video AI.
You've learned how to:
- Set up V-JEPA 2.1 and Qwen2.5-VL-7B in a shared evaluation environment, and curate a focused Something-Something V2 subset targeting temporal reasoning.
- Evaluate a world model using linear probing on frozen self-supervised features, and a multimodal LLM using prompted multiple-choice classification.
- Produce quantitative per-category accuracy breakdowns and visualizations revealing where embedding-space prediction outperforms next-token prediction and vice versa.
- Secret Mission: Visualize V-JEPA 2.1's dense token-level features using PCA to see how a world model represents physical dynamics internally.
Ready to quiz yourself?