LLM Inference Optimization with vLLM

Deploy, benchmark, and optimize LLM inference using vLLM and AWQ quantization.

Introduction

30 Second Summary

Every time you send a prompt to an LLM, a GPU somewhere is doing billions of matrix operations to generate your response token by token. Making that process faster and cheaper is one of the highest-value engineering problems in AI today.

In this project, you will deploy an LLM with vLLM on your own GPU, benchmark its performance, apply AWQ quantization to cut memory usage, and write a custom GPU kernel in OpenAI Triton to prove kernel-level programming ability.

What You'll Build

A professional inference benchmarking repository that demonstrates LLM deployment, systematic quantization comparison, and custom GPU kernel development on VRAM-constrained consumer hardware.

By the end of this project, you'll have:

  • A working vLLM inference server on WSL2 serving Qwen2.5-1.5B-Instruct with benchmarked baseline metrics (TTFT, TPOT, throughput).
  • Quantitative benchmark results comparing FP16 baseline against INT4 AWQ quantization, showing memory, throughput, and latency tradeoffs.
  • A custom fused softmax Triton kernel with benchmarks proving speedup over PyTorch, packaged in a GitHub-ready repository.
  • Secret Mission: Extend your Triton kernel to implement fused softmax-matmul, the core primitive behind FlashAttention.

Are there any prerequisites?

This project assumes you have Windows 11 with an NVIDIA GPU (RTX 4060 or similar with 8GB+ VRAM) and CUDA already installed. You should be comfortable with Python, familiar with basic CUDA concepts, and have a general understanding of transformer architecture.

Before We Start

Before We Start...

Set Up WSL2 and vLLM Environment

vLLM is the industry-standard serving framework for LLM inference. It powers production deployments at scale and is explicitly listed in NVIDIA job postings for inference engineering roles.

There is one constraint: vLLM does not run on Windows natively. In this step, you will set up WSL2 (Windows Subsystem for Linux) with Ubuntu 22.04, install Python 3.12, and get vLLM ready to serve models from your GPU.

In this step, get ready to:
  • Install WSL2 with Ubuntu 22.04 and verify GPU passthrough.
  • Set up Python 3.12 and a virtual environment inside WSL2.
  • Install vLLM and all benchmark dependencies.
Install and configure WSL2 with Ubuntu

vLLM requires a Linux environment with direct GPU access. WSL2 provides this on Windows 11 through GPU passthrough, letting your NVIDIA card work inside a Linux instance without dual-booting.

  • Press the Windows key to open your search bar.
  • Type PowerShell, right-click Windows PowerShell, and select Run as administrator.
  • Install WSL2 with Ubuntu 22.04 by running this command:
wsl --install -d Ubuntu-22.04

What does this command do?

This enables the WSL2 feature on your machine and downloads the Ubuntu 22.04 distribution. WSL2 uses a real Linux kernel, which is what allows GPU passthrough to work.

If WSL is already installed, you may see help text instead. In that case, run wsl --list --online to verify Ubuntu-22.04 is available, then re-run the install command.

  • The installer will prompt you to restart your machine. Restart now.
  • After restarting, Ubuntu 22.04 will launch automatically (or search for Ubuntu 22.04 in your Start menu).
  • When prompted, create a UNIX username and password. This is your Linux account inside WSL2.

Remember your UNIX password

You will need this password for any sudo commands. The terminal does not show characters as you type the password.

Verify your installation is running WSL2 by returning to PowerShell and running:

  • Confirm WSL2 is active by running this in PowerShell:
wsl -l -v

You should see Ubuntu-22.04 listed with VERSION 2.

Seeing VERSION 1 instead of 2?

If your distribution shows VERSION 1, upgrade it by running wsl --set-version Ubuntu-22.04 2 in PowerShell. This converts the instance to use the WSL2 kernel which supports GPU passthrough.

Help me fix my WSL2 version

Verify GPU passthrough and install Python 3.12

Your NVIDIA driver on Windows automatically enables GPU access inside WSL2. You need to confirm this works before installing vLLM.

  • Open your Ubuntu 22.04 terminal (search for Ubuntu 22.04 in your Start menu if it is not already open).
  • Verify your GPU is visible inside WSL2 by running:
nvidia-smi

You should see your RTX 4060 listed with driver version and CUDA version.

nvidia-smi not found?

This means the NVIDIA driver on your Windows host is not recent enough or was not installed with WSL support. Update to the latest NVIDIA Game Ready or Studio driver from nvidia.com, then restart WSL with wsl --shutdown in PowerShell and reopen Ubuntu.

Help me get nvidia-smi working in WSL2

Ubuntu 22.04 ships with Python 3.10 by default. vLLM requires Python 3.12, so you need to install it from the deadsnakes PPA.

  • Update your system packages and install the PPA helper by running:
sudo apt update && sudo apt install -y software-properties-common
  • Add the deadsnakes PPA and install Python 3.12 by running:
sudo add-apt-repository ppa:deadsnakes/ppa -y
sudo apt update
sudo apt install -y python3.12 python3.12-venv python3.12-dev

What is the deadsnakes PPA?

A PPA (Personal Package Archive) is a community-maintained repository for Ubuntu. The deadsnakes PPA provides newer Python versions that are not available in Ubuntu's default repositories. It is the standard way to get Python 3.12 on Ubuntu 22.04.

  • Confirm Python 3.12 is installed by running:
python3.12 --version

You should see Python 3.12 followed by a patch number.

Package not found error?

Make sure you ran sudo apt update after adding the PPA. If the error persists, verify the PPA was added by running apt-cache policy python3.12 and checking that it shows a deadsnakes source.

Help me install Python 3.12 on Ubuntu 22.04

Install vLLM and benchmark dependencies

With Python 3.12 ready, you can create an isolated virtual environment for the project and install vLLM along with all benchmarking tools.

  • Create your project directory and virtual environment by running:
mkdir ~/inference-bench && cd ~/inference-bench
python3.12 -m venv venv
source venv/bin/activate

Why a virtual environment?

A virtual environment isolates your project's packages from the system Python. This prevents version conflicts and keeps your WSL2 instance clean. The source venv/bin/activate command activates it for your current terminal session.

  • Install vLLM with benchmark support by running:
pip install vllm[bench] --extra-index-url https://download.pytorch.org/whl/cu129

What does this command do?

This installs vLLM along with its benchmarking CLI tools. The --extra-index-url flag points pip to PyTorch's CUDA 12.9 wheel repository, ensuring you get GPU-accelerated versions of PyTorch and Triton.

This installation takes several minutes due to the size of PyTorch and CUDA dependencies.

  • Install additional benchmarking and analysis dependencies by running:
pip install numpy pandas matplotlib
  • Verify vLLM is installed correctly by running:
python -c "import vllm; print(vllm.__version__)"

You should see a version number printed without errors.

Import error or CUDA not found?

Make sure your virtual environment is activated (you should see (venv) at the start of your prompt). If you see a CUDA-related error, confirm nvidia-smi still works. Sometimes WSL2 needs a restart after heavy installs. Run wsl --shutdown from PowerShell, reopen Ubuntu, reactivate the venv, and try again.

Help me fix vLLM import errors in WSL2

Finally, connect VS Code to your WSL2 environment so you can edit files comfortably.

  • Open VS Code on Windows.
  • Press Ctrl+Shift+X to open the Extensions panel.
  • Search for WSL and install the extension by Microsoft (ms-vscode-remote.remote-wsl).
  • Back in your Ubuntu terminal (inside the ~/inference-bench folder), open the project in VS Code by running:
code .

VS Code opens with a green WSL: Ubuntu-22.04 indicator in the bottom-left corner. This confirms your editor is connected to the Linux filesystem.

First time running code . in WSL?

The first time you run code . from WSL2, VS Code downloads a small server component into your Linux instance. This takes a minute on first launch. Subsequent opens are instant.

Your WSL2 environment is fully configured with GPU access, Python 3.12, and vLLM ready to serve models. Next up, you will deploy your first model and establish baseline performance metrics.

Deploy and Benchmark Baseline Model

With your environment ready, it is time to deploy an actual LLM and measure how it performs. Performance benchmarking is how inference engineers prove that their optimizations work.

In this step, you will serve Qwen2.5-1.5B-Instruct using vLLM and capture baseline metrics. These numbers become your "before" snapshot that you will compare against after applying AWQ quantization in the next step.

In this step, get ready to:
  • Serve Qwen/Qwen2.5-1.5B-Instruct with vLLM using memory-constrained settings.
  • Run a latency benchmark using vLLM's built-in benchmark CLI.
  • Record baseline metrics: TTFT, TPOT, and throughput.
Serve the baseline model with vLLM

Your RTX 4060 has 8GB of VRAM. The Qwen2.5-1.5B-Instruct model in FP16 takes roughly 3GB, leaving about 5GB for the KV cache and runtime overhead. You need to constrain vLLM's memory usage to avoid out-of-memory errors.

  • Switch to your project directory and activate your virtual environment by running these commands:
cd ~/inference-bench
source venv/bin/activate
  • Start the vLLM inference server with memory-constrained settings by running this command:
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --gpu-memory-utilization 0.85 \
  --max-model-len 2048 \
  --enforce-eager

What do these flags do?

  • --gpu-memory-utilization 0.85 tells vLLM to use at most 85% of your GPU's VRAM. This leaves a safety margin so your system does not crash.
  • --max-model-len 2048 caps the maximum sequence length. Shorter contexts use less KV cache memory.
  • --enforce-eager disables CUDA graph capture. On Ada Lovelace GPUs inside WSL2, CUDA graphs can sometimes cause issues.

The first time you run this, vLLM downloads the model weights from HuggingFace. This may take a few minutes depending on your internet speed.

You should see the server start with output showing it is listening on http://0.0.0.0:8000.

Server fails to start or runs out of memory?

  • If you see an OOM error, try lowering --gpu-memory-utilization to 0.80.
  • Make sure no other processes are using your GPU. Run nvidia-smi in another terminal to check.
  • If you see CUDA graph errors, confirm --enforce-eager is present in your command.

Help me troubleshoot my vLLM server startup in WSL2 on an RTX 4060 with 8GB VRAM.

Create the benchmark script

Rather than typing the benchmark command every time, you will save it as a reusable script. This makes your results reproducible and shareable in your portfolio repository.

  • Open a second WSL2 terminal. Keep the vLLM server running in the first terminal.
  • Create the directories for your scripts and results by running these commands:
cd ~/inference-bench
source venv/bin/activate
mkdir -p scripts results
  • Confirm the directories exist by running:
ls

You should see scripts, results, and venv listed in the output.

  • Create the benchmark script by running this command:
cat << 'EOF' > scripts/benchmark_baseline.sh
#!/bin/bash
# Benchmark script for Qwen2.5-1.5B-Instruct baseline (FP16)
# Requires vLLM server running on port 8000

MODEL="Qwen/Qwen2.5-1.5B-Instruct"
RESULT_DIR="./results"
RESULT_FILE="baseline_results.json"

echo "=== Baseline Benchmark: $MODEL (FP16) ==="
echo "Waiting for server to be ready..."
sleep 5

# Run the serving benchmark with random prompts
vllm bench serve \
  --model $MODEL \
  --backend vllm \
  --endpoint /v1/completions \
  --dataset-name random \
  --random-input-len 128 \
  --random-output-len 128 \
  --num-prompts 20 \
  --save-result \
  --result-dir $RESULT_DIR \
  --result-filename $RESULT_FILE

echo "=== Benchmark complete ==="
echo "Results saved to $RESULT_DIR/$RESULT_FILE"

# Record VRAM usage
echo ""
echo "=== GPU Memory Usage ==="
nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader
EOF

What does this script do?

The script calls vllm bench serve which sends requests to your running vLLM server and measures the response performance.

  • --dataset-name random generates synthetic prompts so you do not need to download a dataset.
  • --random-input-len 128 and --random-output-len 128 set the token lengths for each request.
  • --num-prompts 20 keeps the benchmark short while still giving meaningful averages.
  • --save-result writes the metrics to a JSON file for later comparison.
  • Make the script executable by running:
chmod +x scripts/benchmark_baseline.sh
Run the benchmark and record baseline metrics

Now you will execute the benchmark against your running server. The results give you three key metrics that NVIDIA interviewers care about: TTFT (Time To First Token), TPOT (Time Per Output Token), and throughput in tokens per second.

  • Run the benchmark script (make sure the vLLM server is still running in your first terminal):
bash scripts/benchmark_baseline.sh

The benchmark sends 20 requests to the server and reports timing statistics. You should see a progress bar followed by a summary table with TTFT, TPOT, and throughput numbers.

What should I see?

The output prints percentile metrics for TTFT (how long until the first token arrives), TPOT (how long between each output token), and overall request throughput. Exact numbers depend on your hardware, but on an RTX 4060 you can expect TTFT under 200ms and throughput around 50-100 tokens/sec for this model size.

The nvidia-smi line at the end shows how much VRAM the FP16 model is consuming. You will compare this against the AWQ model in the next step.

Benchmark errors out or times out?

  • Confirm the server is still running in your first terminal. If it crashed, restart it with the same vllm serve command.
  • Check that port 8000 is accessible. The default host for vllm bench serve is 127.0.0.1 on port 8000.
  • If you see a connection refused error, the model may still be loading. Wait 30 seconds and try again.

Help me troubleshoot why my vLLM benchmark is failing to connect to the server.

  • Verify your results file was created by running:
ls results/

You should see baseline_results.json in the output.

  • View the key metrics from your baseline results by running:
cat results/baseline_results.json | python3 -m json.tool | head -50

Note down your TTFT (ms), TPOT (ms), and throughput (tokens/sec) values. You will compare these to the quantized model's results in the next step.

Record your VRAM usage too

Run nvidia-smi while the server is still running to capture the baseline VRAM consumption. The FP16 model typically uses 4-5GB (model weights plus KV cache). This number will drop significantly with AWQ quantization.

✔️ Awesome, I've got everything!

Great. Make sure you have saved your benchmark script and have your results JSON file ready.

ⓧ I'd like to double check the full code

Here is the full benchmark script at ~/inference-bench/scripts/benchmark_baseline.sh:

#!/bin/bash
# Benchmark script for Qwen2.5-1.5B-Instruct baseline (FP16)
# Requires vLLM server running on port 8000

MODEL="Qwen/Qwen2.5-1.5B-Instruct"
RESULT_DIR="./results"
RESULT_FILE="baseline_results.json"

echo "=== Baseline Benchmark: $MODEL (FP16) ==="
echo "Waiting for server to be ready..."
sleep 5

# Run the serving benchmark with random prompts
vllm bench serve \
  --model $MODEL \
  --backend vllm \
  --endpoint /v1/completions \
  --dataset-name random \
  --random-input-len 128 \
  --random-output-len 128 \
  --num-prompts 20 \
  --save-result \
  --result-dir $RESULT_DIR \
  --result-filename $RESULT_FILE

echo "=== Benchmark complete ==="
echo "Results saved to $RESULT_DIR/$RESULT_FILE"

# Record VRAM usage
echo ""
echo "=== GPU Memory Usage ==="
nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader

You now have quantitative proof of how your model performs in FP16 on VRAM-constrained hardware. Next up, you will deploy the AWQ-quantized version of the same model and see exactly how much performance changes when you cut precision from 16-bit to 4-bit.

Deploy AWQ-Quantized Model and Compare

You have baseline performance numbers for Qwen2.5-1.5B-Instruct running at full FP16 precision. The next question is: how much performance can you gain by reducing the model's weight precision from 16-bit to 4-bit?

AWQ (Activation-aware Weight Quantization) compresses model weights to INT4 while preserving the most important channels for accuracy. On a VRAM-constrained GPU like your RTX 4060, this frees memory for larger KV caches and reduces memory bandwidth pressure. In this step, you will serve the AWQ variant of the same model, run an identical benchmark, and compare the results side by side.

In this step, get ready to:
  • Serve the AWQ-quantized model with vLLM using the same memory settings.
  • Run the same benchmark against the quantized model to collect comparable metrics.
  • Generate a side-by-side comparison table showing FP16 vs AWQ tradeoffs.
Serve the AWQ-quantized model

Qwen provides an official INT4 AWQ-quantized variant of the same model on HuggingFace. By using Qwen/Qwen2.5-1.5B-Instruct-AWQ with the --quantization awq flag, vLLM loads the pre-quantized weights directly.

  • Create the AWQ benchmark script by running this command:
cat > ~/inference-bench/scripts/benchmark_awq.sh << 'EOF'
#!/bin/bash
# Benchmark script for AWQ-quantized Qwen2.5-1.5B-Instruct

MODEL="Qwen/Qwen2.5-1.5B-Instruct-AWQ"
HOST="localhost"
PORT=8000
RESULTS_DIR="$HOME/inference-bench/results"

# Activate virtual environment
source ~/inference-bench/venv/bin/activate

# Start vLLM server in background with AWQ quantization
vllm serve $MODEL \
  --quantization awq \
  --gpu-memory-utilization 0.85 \
  --max-model-len 2048 \
  --port $PORT &

SERVER_PID=$!

# Wait for server to be ready
echo "Waiting for vLLM AWQ server to start..."
sleep 90

# Record VRAM usage before benchmark
VRAM_USAGE=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -1)
echo "VRAM usage: ${VRAM_USAGE} MiB"

# Run benchmark with same settings as baseline
vllm bench serve \
  --model $MODEL \
  --host $HOST \
  --port $PORT \
  --dataset-name random \
  --random-input-len 128 \
  --random-output-len 128 \
  --num-prompts 20 \
  --save-result \
  --result-dir $RESULTS_DIR \
  --result-filename awq_results.json

# Stop server
kill $SERVER_PID
wait $SERVER_PID 2>/dev/null

echo "AWQ benchmark complete. Results saved to $RESULTS_DIR/awq_results.json"
EOF
chmod +x ~/inference-bench/scripts/benchmark_awq.sh

What does this script do?

This script mirrors your baseline benchmark with one key difference: the --quantization awq flag tells vLLM to load INT4 quantized weights.

The server settings (--gpu-memory-utilization 0.85 and --max-model-len 2048) stay identical so the comparison is fair. The benchmark parameters (input length, output length, number of prompts) also match the baseline exactly.

The nvidia-smi call captures VRAM usage while the model is loaded, giving you a direct memory comparison against FP16.

What is AWQ quantization?

AWQ identifies which weight channels matter most for preserving activation distributions, then quantizes the remaining channels more aggressively. This gives better accuracy than naive INT4 quantization at the same compression ratio.

The AWQ model weights drop from ~3GB (FP16) to roughly 1GB (INT4). That freed VRAM goes to KV cache, allowing longer sequences or higher batch sizes.

Run the AWQ benchmark

With the script ready, you can now execute it to collect AWQ performance metrics. The benchmark uses the same random dataset and prompt configuration as your baseline run.

  • Run the AWQ benchmark script:
cd ~/inference-bench && bash scripts/benchmark_awq.sh

What should I see?

The server takes about 60-90 seconds to download and load the AWQ model on first run. After loading, you will see the benchmark output with the same format as your baseline run.

Look for the ============ Serving Benchmark Result ============ header in the terminal output. You should see TTFT, TPOT, and throughput metrics reported in milliseconds and tokens per second.

  • Confirm the results file was saved by running:
ls ~/inference-bench/results/awq_results.json

You should see the file listed in the output.

Server not starting or OOM error?

If you see an out-of-memory error, make sure no other vLLM server is still running from your baseline benchmark. Kill any lingering processes with pkill -f "vllm serve" and try again.

If the server hangs during model download, check your internet connection inside WSL2. The AWQ model is about 1GB and downloads from HuggingFace.

help me debug my vLLM AWQ server startup issue

Compare FP16 and AWQ results

With both benchmark runs saved as JSON, you can now programmatically compare the results. A Python script will read both files and generate a side-by-side table showing exactly where AWQ wins and where it trades off.

  • Create the comparison script by running:
cat > ~/inference-bench/scripts/compare_results.py << 'EOF'
import json
import pandas as pd
import os

# Paths to benchmark results
results_dir = os.path.expanduser("~/inference-bench/results")
baseline_path = os.path.join(results_dir, "baseline_results.json")
awq_path = os.path.join(results_dir, "awq_results.json")

# Load results from both benchmark runs
with open(baseline_path, "r") as f:
    baseline = json.load(f)

with open(awq_path, "r") as f:
    awq = json.load(f)

# Extract key metrics from vLLM bench output
# vLLM saves median_ttft_ms, mean_tpot_ms, output_throughput in the JSON
baseline_metrics = {
    "VRAM (GB)": baseline.get("vram_usage_gb", baseline.get("gpu_memory_used_mb", 0) / 1024),
    "TTFT (ms)": baseline.get("median_ttft_ms", 0),
    "TPOT (ms)": baseline.get("mean_tpot_ms", 0),
    "Throughput (tok/s)": baseline.get("output_throughput", baseline.get("output_token_throughput", 0)),
}

awq_metrics = {
    "VRAM (GB)": awq.get("vram_usage_gb", awq.get("gpu_memory_used_mb", 0) / 1024),
    "TTFT (ms)": awq.get("median_ttft_ms", 0),
    "TPOT (ms)": awq.get("mean_tpot_ms", 0),
    "Throughput (tok/s)": awq.get("output_throughput", awq.get("output_token_throughput", 0)),
}

# Build comparison DataFrame
df = pd.DataFrame({
    "Metric": list(baseline_metrics.keys()),
    "FP16 Baseline": list(baseline_metrics.values()),
    "AWQ INT4": list(awq_metrics.values()),
})

# Calculate percentage change
df["Delta (%)"] = ((df["AWQ INT4"] - df["FP16 Baseline"]) / df["FP16 Baseline"] * 100).round(1)

# Print formatted comparison table
print("\n" + "=" * 55)
print("  FP16 vs AWQ INT4 Comparison")
print("=" * 55)
print(df.to_string(index=False))
print("=" * 55)

# Save to CSV for the repository
output_path = os.path.join(results_dir, "comparison_table.csv")
df.to_csv(output_path, index=False)
print(f"\nComparison saved to {output_path}")
EOF

What does this script do?

The script loads both JSON result files and extracts the four key metrics: VRAM usage, TTFT (time to first token), TPOT (time per output token), and throughput.

It uses pandas to build a comparison table with a percentage-change column so you can immediately see where AWQ improves (negative delta for latency means faster) and where it regresses. The output is saved as a CSV for your repository.

  • Run the comparison script:
cd ~/inference-bench && python scripts/compare_results.py

You should see a formatted table in your terminal showing FP16 Baseline, AWQ INT4, and the percentage delta for each metric.

  • Confirm the CSV was generated:
cat ~/inference-bench/results/comparison_table.csv

You should see four rows of data: VRAM, TTFT, TPOT, and Throughput with values for both model variants.

What results should I expect?

On an 8GB RTX 4060 Laptop, you should see AWQ using significantly less VRAM (roughly 1-2GB less than FP16). Throughput typically improves because less data moves across the memory bus per inference step.

TTFT and TPOT may improve slightly or stay similar. The biggest win is the VRAM reduction, which on constrained hardware means you can serve longer contexts or higher batch sizes.

Seeing KeyError when running the comparison?

The JSON keys depend on your vLLM version. The script tries multiple key names for compatibility. If you still get errors, open awq_results.json and baseline_results.json to inspect the actual key names, then update the script's .get() calls to match.

help me fix my comparison script to match my vLLM benchmark JSON output format

✔️ Awesome, I've got everything!

Great. Make sure you have saved all three files: benchmark_awq.sh, compare_results.py, and comparison_table.csv.

ⓧ I'd like to double check the full code

Here is the complete AWQ benchmark script:

#!/bin/bash
# Benchmark script for AWQ-quantized Qwen2.5-1.5B-Instruct

MODEL="Qwen/Qwen2.5-1.5B-Instruct-AWQ"
HOST="localhost"
PORT=8000
RESULTS_DIR="$HOME/inference-bench/results"

# Activate virtual environment
source ~/inference-bench/venv/bin/activate

# Start vLLM server in background with AWQ quantization
vllm serve $MODEL \
  --quantization awq \
  --gpu-memory-utilization 0.85 \
  --max-model-len 2048 \
  --port $PORT &

SERVER_PID=$!

# Wait for server to be ready
echo "Waiting for vLLM AWQ server to start..."
sleep 90

# Record VRAM usage before benchmark
VRAM_USAGE=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -1)
echo "VRAM usage: ${VRAM_USAGE} MiB"

# Run benchmark with same settings as baseline
vllm bench serve \
  --model $MODEL \
  --host $HOST \
  --port $PORT \
  --dataset-name random \
  --random-input-len 128 \
  --random-output-len 128 \
  --num-prompts 20 \
  --save-result \
  --result-dir $RESULTS_DIR \
  --result-filename awq_results.json

# Stop server
kill $SERVER_PID
wait $SERVER_PID 2>/dev/null

echo "AWQ benchmark complete. Results saved to $RESULTS_DIR/awq_results.json"

And the full comparison script:

import json
import pandas as pd
import os

# Paths to benchmark results
results_dir = os.path.expanduser("~/inference-bench/results")
baseline_path = os.path.join(results_dir, "baseline_results.json")
awq_path = os.path.join(results_dir, "awq_results.json")

# Load results from both benchmark runs
with open(baseline_path, "r") as f:
    baseline = json.load(f)

with open(awq_path, "r") as f:
    awq = json.load(f)

# Extract key metrics from vLLM bench output
# vLLM saves median_ttft_ms, mean_tpot_ms, output_throughput in the JSON
baseline_metrics = {
    "VRAM (GB)": baseline.get("vram_usage_gb", baseline.get("gpu_memory_used_mb", 0) / 1024),
    "TTFT (ms)": baseline.get("median_ttft_ms", 0),
    "TPOT (ms)": baseline.get("mean_tpot_ms", 0),
    "Throughput (tok/s)": baseline.get("output_throughput", baseline.get("output_token_throughput", 0)),
}

awq_metrics = {
    "VRAM (GB)": awq.get("vram_usage_gb", awq.get("gpu_memory_used_mb", 0) / 1024),
    "TTFT (ms)": awq.get("median_ttft_ms", 0),
    "TPOT (ms)": awq.get("mean_tpot_ms", 0),
    "Throughput (tok/s)": awq.get("output_throughput", awq.get("output_token_throughput", 0)),
}

# Build comparison DataFrame
df = pd.DataFrame({
    "Metric": list(baseline_metrics.keys()),
    "FP16 Baseline": list(baseline_metrics.values()),
    "AWQ INT4": list(awq_metrics.values()),
})

# Calculate percentage change
df["Delta (%)"] = ((df["AWQ INT4"] - df["FP16 Baseline"]) / df["FP16 Baseline"] * 100).round(1)

# Print formatted comparison table
print("\n" + "=" * 55)
print("  FP16 vs AWQ INT4 Comparison")
print("=" * 55)
print(df.to_string(index=False))
print("=" * 55)

# Save to CSV for the repository
output_path = os.path.join(results_dir, "comparison_table.csv")
df.to_csv(output_path, index=False)
print(f"\nComparison saved to {output_path}")

You now have quantitative evidence of what AWQ quantization does on VRAM-constrained hardware. Next up, you will move beyond model-level optimizations and write a custom GPU kernel using OpenAI Triton to demonstrate kernel-level programming.

Write a Custom Fused Softmax Triton Kernel

You have benchmarked vLLM inference at both FP16 and INT4 precision, proving you can deploy and measure model serving on constrained hardware.

Now you will go deeper and write a GPU kernel from scratch. OpenAI Triton lets you write custom GPU operations in Python that compile down to highly efficient machine code. You will implement a fused softmax kernel that computes the entire softmax in a single GPU pass, avoiding multiple round-trips to global memory.

In this step, get ready to:
  • Implement a numerically-stable fused softmax kernel in OpenAI Triton.
  • Benchmark your kernel against torch.softmax across varying matrix sizes.
  • Validate correctness against PyTorch reference implementation.
Implement the fused softmax Triton kernel

A standard softmax requires three passes over the data: one to find the max, one to compute exponentials, and one to normalize. A fused kernel does all three in a single pass per row, keeping data in fast on-chip SRAM instead of reading from slow global memory multiple times.

  • Create the kernels directory inside your project by running this command:
mkdir -p ~/inference-bench/kernels
  • Create and open the kernel file by running:
code ~/inference-bench/kernels/fused_softmax.py
  • Add the imports and kernel function by copying this code into fused_softmax.py:
import torch
import triton
import triton.language as tl


@triton.jit
def softmax_kernel(
    output_ptr,
    input_ptr,
    input_row_stride,
    output_row_stride,
    n_rows,
    n_cols,
    BLOCK_SIZE: tl.constexpr,
    num_stages: tl.constexpr,
):
    # Each program instance processes one row
    row_idx = tl.program_id(0)

    # Compute pointer to the start of this row
    row_start_ptr = input_ptr + row_idx * input_row_stride

    # Create column offsets for loading a BLOCK_SIZE chunk
    col_offsets = tl.arange(0, BLOCK_SIZE)

    # Mask for valid columns (handles non-power-of-two widths)
    mask = col_offsets < n_cols

    # Load the row, using -inf for masked positions
    row = tl.load(row_start_ptr + col_offsets, mask=mask, other=-float('inf'))

    # Subtract per-row max for numerical stability
    row_max = tl.max(row, axis=0)
    numerator = tl.exp(row - row_max)

    # Sum of exponentials for normalization
    denominator = tl.sum(numerator, axis=0)

    # Compute softmax output
    softmax_output = numerator / denominator

    # Write result back to global memory
    output_row_start_ptr = output_ptr + row_idx * output_row_stride
    tl.store(output_row_start_ptr + col_offsets, softmax_output, mask=mask)

What does this kernel do?

Each program instance (one per row) loads an entire row into on-chip registers, computes the max, exponentiates, sums, and normalizes without ever writing intermediate results back to global memory.

The mask handles matrices whose column count is not a power of two. Masked positions are loaded as -inf so they contribute zero after exponentiation.

  • Now add the Python wrapper function that launches the kernel. Copy this code below the softmax_kernel function in the same file:
def fused_softmax(x):
    """Compute softmax along the last dimension using a fused Triton kernel."""
    n_rows, n_cols = x.shape

    # Allocate output tensor with same shape and dtype
    output = torch.empty_like(x)

    # Block size must be a power of 2 >= n_cols, capped at 4096
    BLOCK_SIZE = triton.next_power_of_2(n_cols)
    BLOCK_SIZE = min(BLOCK_SIZE, 4096)

    # Use more warps for larger blocks to maximize parallelism
    num_warps = 4 if BLOCK_SIZE <= 2048 else 8

    # Launch one program per row
    softmax_kernel[(n_rows,)](
        output,
        x,
        x.stride(0),
        output.stride(0),
        n_rows,
        n_cols,
        BLOCK_SIZE=BLOCK_SIZE,
        num_stages=4,
        num_warps=num_warps,
    )
    return output

Why cap BLOCK_SIZE at 4096?

Each thread block has a limited number of registers. If BLOCK_SIZE is too large, the kernel spills registers to slow local memory and performance drops. 4096 is a safe upper bound for the RTX 4060's register file.

  • Save fused_softmax.py.
  • Confirm the file exists by running:
ls ~/inference-bench/kernels/fused_softmax.py

You should see the file path printed with no errors.

File not found?

Make sure you created the kernels directory with mkdir -p before creating the file. Double-check that you saved the file inside ~/inference-bench/kernels/ and not in a different directory.

Help me troubleshoot my Triton kernel file setup.

✔️ Awesome, I've got everything!

Great. Double check you have saved fused_softmax.py and move on to the benchmark script.

ⓧ I'd like to double check the full code

import torch
import triton
import triton.language as tl


@triton.jit
def softmax_kernel(
    output_ptr,
    input_ptr,
    input_row_stride,
    output_row_stride,
    n_rows,
    n_cols,
    BLOCK_SIZE: tl.constexpr,
    num_stages: tl.constexpr,
):
    # Each program instance processes one row
    row_idx = tl.program_id(0)

    # Compute pointer to the start of this row
    row_start_ptr = input_ptr + row_idx * input_row_stride

    # Create column offsets for loading a BLOCK_SIZE chunk
    col_offsets = tl.arange(0, BLOCK_SIZE)

    # Mask for valid columns (handles non-power-of-two widths)
    mask = col_offsets < n_cols

    # Load the row, using -inf for masked positions
    row = tl.load(row_start_ptr + col_offsets, mask=mask, other=-float('inf'))

    # Subtract per-row max for numerical stability
    row_max = tl.max(row, axis=0)
    numerator = tl.exp(row - row_max)

    # Sum of exponentials for normalization
    denominator = tl.sum(numerator, axis=0)

    # Compute softmax output
    softmax_output = numerator / denominator

    # Write result back to global memory
    output_row_start_ptr = output_ptr + row_idx * output_row_stride
    tl.store(output_row_start_ptr + col_offsets, softmax_output, mask=mask)


def fused_softmax(x):
    """Compute softmax along the last dimension using a fused Triton kernel."""
    n_rows, n_cols = x.shape

    # Allocate output tensor with same shape and dtype
    output = torch.empty_like(x)

    # Block size must be a power of 2 >= n_cols, capped at 4096
    BLOCK_SIZE = triton.next_power_of_2(n_cols)
    BLOCK_SIZE = min(BLOCK_SIZE, 4096)

    # Use more warps for larger blocks to maximize parallelism
    num_warps = 4 if BLOCK_SIZE <= 2048 else 8

    # Launch one program per row
    softmax_kernel[(n_rows,)](
        output,
        x,
        x.stride(0),
        output.stride(0),
        n_rows,
        n_cols,
        BLOCK_SIZE=BLOCK_SIZE,
        num_stages=4,
        num_warps=num_warps,
    )
    return output
Build the benchmark and validation script

You need to prove two things: that your kernel produces correct results, and that it is faster than the PyTorch built-in. The benchmark script will test both across multiple matrix sizes.

  • Create the benchmark script by running:
code ~/inference-bench/kernels/benchmark_softmax.py
  • Add the imports, configuration, and correctness validation function by copying this code into benchmark_softmax.py:
import torch
import pandas as pd
import triton
from fused_softmax import fused_softmax

# Matrix sizes to benchmark (number of columns)
SIZES = [512, 1024, 2048, 4096]
# Number of rows for each test matrix
N_ROWS = 4096


def validate_correctness(n_cols):
    """Check that Triton kernel matches PyTorch reference."""
    x = torch.randn(N_ROWS, n_cols, device='cuda', dtype=torch.float32)
    triton_out = fused_softmax(x)
    torch_out = torch.softmax(x, dim=-1)
    # Compute maximum absolute difference
    max_error = (triton_out - torch_out).abs().max().item()
    return max_error

What does correctness validation check?

It generates a random matrix, runs both your Triton kernel and torch.softmax on it, then compares element-by-element. A max absolute error below 1e-5 confirms that your kernel is numerically equivalent to the reference.

  • Now add the benchmarking function and main block below the validation function in the same file:
def benchmark_softmax(n_cols):
    """Measure latency and throughput for both implementations."""
    x = torch.randn(N_ROWS, n_cols, device='cuda', dtype=torch.float32)

    # Total bytes: read input + write output (both float32)
    nbytes = 2 * x.numel() * x.element_size()

    # Benchmark Triton kernel (returns mean time in ms)
    triton_ms = triton.testing.do_bench(lambda: fused_softmax(x))
    triton_gbps = nbytes / (triton_ms * 1e-3) / 1e9

    # Benchmark PyTorch softmax
    torch_ms = triton.testing.do_bench(lambda: torch.softmax(x, dim=-1))
    torch_gbps = nbytes / (torch_ms * 1e-3) / 1e9

    return triton_ms, torch_ms, triton_gbps, torch_gbps


if __name__ == "__main__":
    print("=" * 60)
    print("Fused Softmax Benchmark: Triton vs PyTorch")
    print("=" * 60)

    # Validate correctness first
    print("\n--- Correctness Validation ---")
    for size in SIZES:
        max_err = validate_correctness(size)
        status = "PASS" if max_err < 1e-5 else "FAIL"
        print(f"  n_cols={size:>5}: max_abs_error={max_err:.2e} [{status}]")

    # Run benchmarks
    print("\n--- Performance Benchmark ---")
    print(f"  Matrix shape: ({N_ROWS}, n_cols), dtype=float32")
    print(f"  {'Size':<8} {'Triton(ms)':<12} {'Torch(ms)':<12} {'Triton(GB/s)':<14} {'Torch(GB/s)':<14} {'Speedup':<8}")
    print(f"  {'-'*68}")

    results = []
    for size in SIZES:
        triton_ms, torch_ms, triton_gbps, torch_gbps = benchmark_softmax(size)
        speedup = torch_ms / triton_ms
        print(f"  {size:<8} {triton_ms:<12.3f} {torch_ms:<12.3f} {triton_gbps:<14.1f} {torch_gbps:<14.1f} {speedup:<8.2f}x")
        results.append({
            "matrix_size": size,
            "triton_latency_ms": round(triton_ms, 3),
            "torch_latency_ms": round(torch_ms, 3),
            "triton_gbps": round(triton_gbps, 1),
            "torch_gbps": round(torch_gbps, 1),
            "speedup": round(speedup, 2),
        })

    # Save results to CSV
    df = pd.DataFrame(results)
    df.to_csv("../results/softmax_benchmark.csv", index=False)
    print(f"\nResults saved to ../results/softmax_benchmark.csv")

How does the throughput calculation work?

Memory throughput (GB/s) measures how fast the kernel moves data. It counts total bytes transferred (input read + output written) and divides by the kernel's runtime. Higher GB/s means the kernel is more efficient at utilizing GPU memory bandwidth.

  • Save benchmark_softmax.py.

Import error when you run the script later?

The benchmark imports from fused_softmax import fused_softmax which only works when you run the script from inside the kernels/ directory. You will cd into that directory in the next substep.

Help me fix my import path for the Triton benchmark script.

✔️ Awesome, I've got everything!

Great. Make sure both files are saved and move on to running the benchmark.

ⓧ I'd like to double check the full code

import torch
import pandas as pd
import triton
from fused_softmax import fused_softmax

# Matrix sizes to benchmark (number of columns)
SIZES = [512, 1024, 2048, 4096]
# Number of rows for each test matrix
N_ROWS = 4096


def validate_correctness(n_cols):
    """Check that Triton kernel matches PyTorch reference."""
    x = torch.randn(N_ROWS, n_cols, device='cuda', dtype=torch.float32)
    triton_out = fused_softmax(x)
    torch_out = torch.softmax(x, dim=-1)
    # Compute maximum absolute difference
    max_error = (triton_out - torch_out).abs().max().item()
    return max_error


def benchmark_softmax(n_cols):
    """Measure latency and throughput for both implementations."""
    x = torch.randn(N_ROWS, n_cols, device='cuda', dtype=torch.float32)

    # Total bytes: read input + write output (both float32)
    nbytes = 2 * x.numel() * x.element_size()

    # Benchmark Triton kernel (returns mean time in ms)
    triton_ms = triton.testing.do_bench(lambda: fused_softmax(x))
    triton_gbps = nbytes / (triton_ms * 1e-3) / 1e9

    # Benchmark PyTorch softmax
    torch_ms = triton.testing.do_bench(lambda: torch.softmax(x, dim=-1))
    torch_gbps = nbytes / (torch_ms * 1e-3) / 1e9

    return triton_ms, torch_ms, triton_gbps, torch_gbps


if __name__ == "__main__":
    print("=" * 60)
    print("Fused Softmax Benchmark: Triton vs PyTorch")
    print("=" * 60)

    # Validate correctness first
    print("\n--- Correctness Validation ---")
    for size in SIZES:
        max_err = validate_correctness(size)
        status = "PASS" if max_err < 1e-5 else "FAIL"
        print(f"  n_cols={size:>5}: max_abs_error={max_err:.2e} [{status}]")

    # Run benchmarks
    print("\n--- Performance Benchmark ---")
    print(f"  Matrix shape: ({N_ROWS}, n_cols), dtype=float32")
    print(f"  {'Size':<8} {'Triton(ms)':<12} {'Torch(ms)':<12} {'Triton(GB/s)':<14} {'Torch(GB/s)':<14} {'Speedup':<8}")
    print(f"  {'-'*68}")

    results = []
    for size in SIZES:
        triton_ms, torch_ms, triton_gbps, torch_gbps = benchmark_softmax(size)
        speedup = torch_ms / triton_ms
        print(f"  {size:<8} {triton_ms:<12.3f} {torch_ms:<12.3f} {triton_gbps:<14.1f} {torch_gbps:<14.1f} {speedup:<8.2f}x")
        results.append({
            "matrix_size": size,
            "triton_latency_ms": round(triton_ms, 3),
            "torch_latency_ms": round(torch_ms, 3),
            "triton_gbps": round(triton_gbps, 1),
            "torch_gbps": round(torch_gbps, 1),
            "speedup": round(speedup, 2),
        })

    # Save results to CSV
    df = pd.DataFrame(results)
    df.to_csv("../results/softmax_benchmark.csv", index=False)
    print(f"\nResults saved to ../results/softmax_benchmark.csv")
Run the benchmark and analyze results

Time to compile your kernel and see how it performs against PyTorch's built-in softmax. The first run will be slower because Triton compiles the kernel to PTX (GPU assembly) on demand.

  • Activate your virtual environment and navigate to the kernels directory by running:
source ~/inference-bench/venv/bin/activate
cd ~/inference-bench/kernels
  • Run the benchmark script:
python benchmark_softmax.py

What should I see?

The script prints two sections. First, a correctness validation showing PASS for each matrix size with errors around 1e-7 to 1e-6. Second, a performance table with latency (ms), throughput (GB/s), and speedup for each size.

On an RTX 4060, expect your Triton kernel to show a speedup between 1.5x and 4x over torch.softmax for larger sizes. The speedup increases with matrix width because the fused approach avoids more memory round-trips on larger data.

  • Verify the results CSV was saved by running:
cat ~/inference-bench/results/softmax_benchmark.csv

You should see a CSV with columns: matrix_size, triton_latency_ms, torch_latency_ms, triton_gbps, torch_gbps, speedup.

Seeing a CUDA error or kernel compilation failure?

If you see a triton.compiler error, make sure your virtual environment is activated and that nvidia-smi shows your GPU from within WSL2. If you see an out-of-memory error, close any other GPU processes (like a vLLM server from the previous step) before running the benchmark.

Help me debug my Triton kernel compilation error.

You now have a working custom GPU kernel that proves you can write, benchmark, and validate performance-critical GPU code. Next up, you will package everything into a clean GitHub repository with a professional README.

Package Results into a GitHub Repository

You have benchmarked both the FP16 baseline and AWQ-quantized models, and your custom Triton kernel proves you can write GPU code.

Now it is time to package everything into a clean GitHub repository that an interviewer can clone and reproduce. A well-structured repo with a clear README turns raw benchmark numbers into a portfolio piece.

In this step, get ready to:
  • Create a .gitignore and requirements.txt to keep the repo clean and reproducible.
  • Write a README.md with hardware specs, performance tables, and analysis.
  • Initialize git, commit all files, and push to GitHub.
Structure the repo with .gitignore and requirements.txt

Before committing, you need two housekeeping files. A .gitignore prevents large or generated files from bloating the repo. A requirements.txt lets anyone recreate your environment with a single command.

  • In your WSL2 terminal, make sure you are inside the project directory by running:
cd ~/inference-bench
  • Create the .gitignore file by running:
cat > .gitignore << 'EOF'
# Virtual environment
venv/

# Python bytecode
__pycache__/
*.pyc
*.pyo

# Cache directories
.cache/

# HuggingFace model cache
~/.cache/huggingface/
models--*/
EOF

What does this file do?

The .gitignore tells Git to skip files that do not belong in a repository. Your venv/ folder alone can be hundreds of megabytes. Model cache files can be gigabytes.

Anyone who clones the repo will recreate the virtual environment from requirements.txt instead of downloading yours.

  • Create the requirements.txt file listing your Python dependencies by running:
cat > requirements.txt << 'EOF'
vllm
torch
triton
numpy
pandas
matplotlib
EOF

Why list packages without version pins?

For a portfolio benchmarking project, unpinned versions let reviewers install the latest compatible stack without fighting dependency conflicts. In production, you would pin exact versions.

  • Verify both files exist by running:
ls .gitignore requirements.txt

You should see both filenames printed in the output.

Don't see a file listed?

Make sure you ran the cat > commands from inside ~/inference-bench/. Run pwd to confirm your current directory.

Help me figure out why my file wasn't created in the right location.

Write a README.md with performance tables and analysis

The README is what an interviewer reads first. It needs to communicate what you built, what hardware you used, how to reproduce it, and what the results show.

  • Create README.md by running:
cat > README.md << 'EOF'
# LLM Inference Optimization: Benchmarking Quantization with vLLM

A benchmarking and optimization suite for LLM inference on VRAM-constrained consumer hardware. Demonstrates vLLM deployment, AWQ quantization tradeoffs, and custom GPU kernel development with OpenAI Triton.

## Hardware

| Component | Spec |
|-----------|------|
| GPU | NVIDIA RTX 4060 Laptop (8GB VRAM) |
| Architecture | Ada Lovelace (sm_89) |
| RAM | 32GB DDR5 |
| OS | Windows 11 + WSL2 (Ubuntu 22.04) |
| CUDA | 12.x |

## Project Structure

```
inference-bench/
├── kernels/
│   ├── fused_softmax.py       # Triton fused softmax kernel
│   └── benchmark_softmax.py   # Kernel benchmark script
├── scripts/
│   ├── benchmark_baseline.sh  # FP16 vLLM benchmark
│   ├── benchmark_awq.sh       # AWQ vLLM benchmark
│   └── compare_results.py     # Results comparison
├── results/
│   ├── baseline_results.json  # FP16 benchmark output
│   ├── awq_results.json       # AWQ benchmark output
│   ├── comparison_table.csv   # Side-by-side metrics
│   └── softmax_benchmark.csv  # Triton vs torch results
├── requirements.txt
└── README.md
```

## Reproduction Instructions

### Prerequisites

- Windows 11 with WSL2 enabled
- NVIDIA GPU with compute capability 7.5+ and CUDA installed
- Python 3.12

### Setup

```bash
# Clone the repository
git clone https://github.com/<your-username>/inference-bench.git
cd inference-bench

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt
```

### Run Benchmarks

```bash
# FP16 baseline benchmark
bash scripts/benchmark_baseline.sh

# AWQ quantized benchmark
bash scripts/benchmark_awq.sh

# Compare results
python scripts/compare_results.py

# Triton kernel benchmark
python kernels/benchmark_softmax.py
```

## Results: FP16 vs AWQ Quantization

| Metric | FP16 (Baseline) | AWQ (INT4) | Change |
|--------|-----------------|------------|--------|
| VRAM Usage | ~3 GB | ~1 GB | -67% |
| TTFT (ms) | *your value* | *your value* | |
| TPOT (ms) | *your value* | *your value* | |
| Throughput (tok/s) | *your value* | *your value* | |

> Replace *your value* placeholders with actual numbers from results/comparison_table.csv.

## Results: Triton Fused Softmax vs torch.softmax

| Matrix Size | Triton (ms) | PyTorch (ms) | Speedup |
|-------------|-------------|--------------|--------|
| 512x512 | *your value* | *your value* | |
| 1024x1024 | *your value* | *your value* | |
| 2048x2048 | *your value* | *your value* | |
| 4096x4096 | *your value* | *your value* | |

> Replace with actual numbers from results/softmax_benchmark.csv.

## Analysis

### Quantization Tradeoffs

AWQ (Activation-aware Weight Quantization) reduces model weights from FP16 to INT4, cutting VRAM usage by approximately 67%. On a VRAM-constrained 8GB GPU, this frees memory for larger KV caches and longer context windows. The throughput change depends on whether the workload is compute-bound or memory-bandwidth-bound.

### Triton Kernel Performance

The fused softmax kernel eliminates intermediate memory round-trips by computing max, subtract, exp, sum, and normalize in a single GPU pass. On larger matrices, the memory bandwidth savings dominate and the kernel achieves significant speedup over the PyTorch implementation.

### Key Takeaways

- VRAM-constrained deployment benefits most from quantization (memory savings unlock deployment that is otherwise impossible)
- Custom Triton kernels shine when the operation is memory-bandwidth-bound
- vLLM's PagedAttention and continuous batching provide production-ready serving even on consumer hardware

## License

MIT
EOF

What does each README section accomplish?

  • Hardware section immediately tells the reader the constraints you worked under. NVIDIA interviewers want to see you understand hardware limitations.
  • Reproduction Instructions let anyone clone and re-run your benchmarks. Reproducibility is critical for engineering credibility.
  • The Analysis section demonstrates you understand WHY the numbers look the way they do, not just that you ran a script.
  • Now fill in your actual benchmark numbers. Open results/comparison_table.csv to get the FP16 vs AWQ values:
cat results/comparison_table.csv
  • Open results/softmax_benchmark.csv to get the Triton kernel values:
cat results/softmax_benchmark.csv
  • Edit README.md to replace the *your value* placeholders with your actual numbers. You can use nano README.md or open the file in VS Code.

Open the folder in VS Code from WSL2

Run code . from inside ~/inference-bench/ to open the project in VS Code with the WSL extension. This gives you a graphical editor for the README while staying connected to your WSL2 filesystem.

  • Save README.md after inserting your numbers.
  • Verify the README renders correctly by checking its first few lines:
head -20 README.md

You should see the title, description, and the start of the Hardware table.

README looks garbled or has extra characters?

Heredoc copy-paste can sometimes introduce Windows line endings (\r\n). Run sed -i 's/\r$//' README.md to strip carriage returns.

Help me fix formatting issues in my README.md file.

Initialize git and push to GitHub

With all files in place, you can initialize the repository and push it to GitHub. This makes your work shareable and demonstrates version control competence.

  • Initialize the git repository by running:
git init
  • Stage all project files by running:
git add .
  • Verify that venv/ is excluded by checking what is staged:
git status

You should see your scripts, kernels, results, README, requirements.txt, and .gitignore staged. You should NOT see venv/ or any __pycache__/ directories listed.

Seeing venv/ in the staged files?

Your .gitignore might not have been created correctly. Run cat .gitignore to confirm it contains venv/ on its own line. If it does, run git rm -r --cached venv/ to unstage it.

Help me fix my .gitignore so venv is excluded.

  • Commit all staged files by running:
git commit -m "Initial commit: vLLM inference benchmarks and Triton kernel"

What if git asks for my name and email?

If this is your first time using git in WSL2, it will ask you to configure your identity. Run these two commands with your own details:

git config --global user.email "your-email@example.com"
git config --global user.name "Your Name"

Then run the commit command again.

  • Now create the remote repository on GitHub. Open your browser and go to github.com/new.
  • Enter inference-bench as the repository name.
  • Leave the repository as Public.
  • Do NOT check any of the initialization boxes (no README, no .gitignore, no license). You already have these files locally.
  • Click Create repository.
  • GitHub will show setup instructions. Connect your local repo to the remote and push by running:
git branch -M main
git remote add origin https://github.com/[[GITHUB_USERNAME="your-github-username"]]/inference-bench.git
git push -u origin main

What do these commands do?

  • git branch -M main renames your default branch to main (GitHub's default).
  • git remote add origin links your local repository to the GitHub remote.
  • git push -u origin main uploads all committed files and sets main as the default tracking branch.

Authentication failed when pushing?

GitHub no longer accepts password authentication over HTTPS. You need a Personal Access Token (PAT) or SSH key. The fastest fix is to create a PAT at github.com/settings/tokens and paste it when prompted for a password.

Alternatively, switch to SSH by changing your remote URL:

git remote set-url origin git@github.com:<your-username>/inference-bench.git

Help me authenticate git push to GitHub from WSL2.

  • Verify the push succeeded by refreshing your GitHub repository page in the browser.

You should see all your folders (kernels/, scripts/, results/), your README.md rendered with tables, and requirements.txt listed.

  • Open the project in VS Code by running:
code .

VS Code opens with the full project structure visible in the Explorer sidebar.

Secret mission

Implement the Core FlashAttention Primitive

FlashAttention revolutionized transformer inference by fusing softmax and matrix multiplication into a single GPU pass. The key insight: you never need to store the full attention matrix in memory. In this secret mission, you will implement a fused softmax-matmul Triton kernel that computes softmax(X) @ V without materializing the intermediate softmax output. This is the exact primitive that makes long-context inference possible on VRAM-constrained hardware.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally (plus a GitHub repository), so there are no ongoing cloud costs.

Resources you used:

  • WSL2 Ubuntu distribution (with CUDA passthrough configured).
  • Python virtual environment at ~/inference-bench/venv.
  • Project directory at ~/inference-bench/ (inside WSL2).
  • HuggingFace model cache at ~/.cache/huggingface/hub (downloaded Qwen models).
  • GitHub repository (inference-bench).

Keep everything running

No action needed. Choose this if you plan to continue experimenting with vLLM, Triton kernels, or adding new benchmarks to your repository.

  • Your WSL2 environment, virtual environment, and model cache remain on disk.
  • The HuggingFace model cache (~/.cache/huggingface/hub) may use several GB of storage. You can check its size with du -sh ~/.cache/huggingface/hub.
  • WSL2 uses memory only while running. Shut it down with wsl --shutdown from PowerShell when not in use.

Pause - I'll come back to this later

Free up disk space from model downloads while keeping your code and WSL2 environment intact.

  • Delete the HuggingFace model cache to reclaim disk space. Open your WSL2 terminal and run:
rm -rf ~/.cache/huggingface/hub
  • Shut down WSL2 from PowerShell to free memory:
wsl --shutdown
  • Your project files at ~/inference-bench/ and your virtual environment remain untouched.
  • When you come back, reactivate your venv and the models will re-download automatically on next use.

Delete - I don't want to use this again

Remove all project resources completely. Your GitHub repository preserves the code if you ever want to revisit it.

  • Delete the project directory and model cache. Open your WSL2 terminal and run:
rm -rf ~/inference-bench
rm -rf ~/.cache/huggingface/hub
  • If you want to remove WSL2 entirely, open PowerShell on Windows and run:
wsl --unregister Ubuntu

What does unregister do?

This removes the entire Ubuntu distribution and all files stored inside it. Only do this if you have no other projects running in WSL2.

  • To delete the GitHub repository, go to your inference-bench repository on GitHub.
  • Click Settings.
  • Scroll to the bottom to the Danger Zone section.
  • Click Delete this repository and follow the confirmation prompts.

Nice Work!

Nice Work!

You just built a professional inference optimization portfolio piece that demonstrates the exact skills NVIDIA hires for in their ML Inference and AI Inference Performance Engineering roles.

You've learned how to:

  • Deploy and serve an LLM with vLLM on VRAM-constrained hardware, and benchmark it using industry-standard metrics (TTFT, TPOT, throughput).
  • Apply AWQ quantization to reduce model memory footprint and systematically compare performance tradeoffs against an FP16 baseline.
  • Write a custom fused softmax Triton kernel that outperforms PyTorch, demonstrating GPU kernel-level programming ability.
  • Secret Mission: implement a fused softmax-matmul kernel that computes the core FlashAttention primitive without materializing the full attention matrix.

Ready to quiz yourself?