Build ComputeCache in vLLM

Introduction

30 Second Summary

When many people send similar instructions to an AI service, repeated work should make later responses faster. A burst of one-off requests can crowd out that useful work before it is reused.

In this project, you will build a production-aware ComputeCache release candidate inside a pinned vLLM v0.32.0 fork. The candidate protects reusable prefixes in the GPU KV cache while preserving a fast LRU rollback path.

What You'll Build

Your real vLLM server will keep a warmed prefix available through one-use traffic while Prometheus and Grafana show the cache behavior and time-to-first-token evidence behind the result.

By the end of this project, you'll have:

• A real GPU pressure test where warmed-prefix cached tokens fall under LRU pressure before active ComputeCache protects more of them.
• A deterministic safety suite you can run to prove bounded work, protected-block handling, quick demotion, and fail-safe LRU ordering.
• An observable canary and rollback drill where production metrics expose cache pressure and response latency before one setting returns the server to LRU without a rebuild.
• Secret Mission: Test scan limits 16, 32, and 64 to find the smallest window that protects the warmed prefix without breaking the p95 time-to-first-token budget.

Are there any prerequisites?

You need SSH access from Windows with WSL2 to a remote Linux server where you can run Docker on a supported NVIDIA GPU. NVIDIA Container Toolkit setup is included if container GPU access is missing.

Before We Start

Before any hands-on work begins, define the production behavior that ComputeCache must protect. Your target is a localhost-only single-instance vLLM canary that preserves a repeated production-incident system prefix under one-use tenant pressure. Immediate rollback follows request failures, no improvement in hot-prefix cached tokens, or a clear TTFT regression.

Pin the Source and Prove GPU Access

Your vLLM canary needs evidence from the real remote GPU. It also needs a pinned Git revision so every later test runs against the same source.

In this step, you will connect through SSH from WSL2. You will prove Docker can reach the GPU through NVIDIA Container Toolkit version 1.20.1. Then you will pin the source to v0.32.0 before preparing the project files.

In this step, get ready to:

• Connect to the remote Linux GPU server over SSH.
• Prove that the host GPU is available inside a Docker container.
• Pin the vLLM source before preparing the project structure.

Connect to the remote server

All builds run on the remote Linux server because that machine owns the physical NVIDIA GPU. Your WSL2 terminal acts as the secure control point.

• Press the Windows key to open Windows search.
• Type Windows Terminal and press Enter.
• Select your WSL2 profile from the terminal tab menu.
• Connect to the remote Linux GPU server by running this command:

ssh [[REMOTE_USER="your remote Linux username"]]@[[REMOTE_HOST="your remote GPU server hostname or IP"]]

What does this command do?

The ssh client creates an encrypted session with your remote server. The username selects your remote account.

The hostname or IP selects the server that owns the GPU. Every command after login runs on that remote machine.

Your prompt changes to the remote account when the connection succeeds. That confirms the rest of this step is running on the GPU server.

Unable to connect over SSH?

Confirm that the remote server is running. Check that your provider allows SSH access from your current network.

Check the remote username. Confirm that the hostname or IP matches the server details from your provider.

Ask for help with the connection details without sharing a private key or password: help me troubleshoot my SSH connection to a remote Linux GPU server

Docker Compose starts the serving stack later in the project. Check that its plugin is available before you depend on it.

• Check Docker Compose on the remote server by running this command:

docker compose version

What does this command check?

This asks the Docker CLI to load its Compose plugin. A version response confirms the plugin is available for the multi-service stack.

✔️ I see a version number

Docker Compose is available. You can use the Compose commands required by the canary.

ⓧ Command not found

The Docker Compose plugin is missing. The following installation path applies to Ubuntu or Debian hosts.

• Install the Docker Compose plugin by running these commands:

sudo apt-get update
sudo apt-get install docker-compose-plugin
docker compose version

What do these commands do?

The first command refreshes the package index. The second installs the Docker Compose plugin.

The final command checks the installation. You should see a version response.

If your server uses another Linux distribution, follow the official Docker Compose installation guide for that distribution.

Prove host and container GPU access

The host driver proves Linux can see the physical GPU. Container passthrough proves the vLLM service can use that same device.

• Check the host GPU by running this command:

nvidia-smi

What does this command prove?

The command asks the installed NVIDIA driver to report its visible devices. A GPU table confirms the remote Linux host can communicate with the hardware.

You should see the remote NVIDIA GPU listed with driver information. This establishes the host-side half of the GPU path.

Host GPU missing?

Confirm that you connected to the intended GPU server. A CPU-only server cannot complete this project.

Check the GPU instance status with your provider. Start the instance if it is stopped.

Ask for help without sharing server credentials: help me diagnose why nvidia-smi cannot see my remote NVIDIA GPU

The container runtime needs a compatible NVIDIA integration. Check the installed toolkit version before configuring Docker.

• Check the NVIDIA Container Toolkit version by running this command:

nvidia-ctk --version

What does this command check?

The nvidia-ctk utility configures supported container runtimes for NVIDIA GPUs. Its version output tells you whether the pinned toolkit is present.

✔️ I see version 1.20.1 or higher

The toolkit is ready. Continue with Docker runtime configuration below.

ⓧ I see an older version

Package setup can be fiddly because each Linux family uses a different repository. The commands below apply to Ubuntu or Debian hosts.

• Upgrade the NVIDIA Container Toolkit packages to version 1.20.1 by running these commands:

sudo apt-get update && sudo apt-get install -y --no-install-recommends \
   ca-certificates \
   curl \
   gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
    nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

What do these commands install?

The first group installs the repository prerequisites. The second group adds NVIDIA's signed production repository.

The final group installs the four toolkit packages at the same pinned release. Keeping the package versions aligned avoids a mixed runtime installation.

If your server uses another Linux distribution, use the official NVIDIA Container Toolkit installation guide to select its package manager.

ⓧ Command not found

The toolkit is not installed. The commands below install version 1.20.1 on Ubuntu or Debian hosts.

• Install the pinned NVIDIA Container Toolkit packages by running these commands:

sudo apt-get update && sudo apt-get install -y --no-install-recommends \
   ca-certificates \
   curl \
   gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
    nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

What do these commands install?

The commands add NVIDIA's signed package source. They install every toolkit component at version 1.20.1.

The runtime configuration happens next in the main path. That keeps installation separate from Docker setup.

If your server uses another Linux distribution, use the official NVIDIA Container Toolkit installation guide to select its package manager.

Docker still needs its runtime configuration updated after the toolkit is available. Restarting Docker applies that configuration to new containers.

• Configure Docker for NVIDIA GPU requests by running these commands:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

What do these commands change?

The first command updates Docker's daemon configuration for the NVIDIA runtime. The second restarts Docker so the runtime change takes effect.

The final proof runs nvidia-smi inside a CUDA container. The first run can take a few minutes while Docker downloads the image.

Before you run the container, do you expect it to report the same physical GPU as the host command?

• Validate container GPU passthrough by running this command:

docker run --rm --gpus all nvidia/cuda:12.9.0-base-ubuntu22.04 nvidia-smi

What does this command prove?

Docker starts a temporary CUDA container with access to every requested GPU. The container runs nvidia-smi before removing itself.

A matching GPU report proves passthrough from the remote host into a container. This is physical GPU access from Linux.

You should see the remote NVIDIA GPU listed again. That completes the GPU path your vLLM container needs.

Container cannot request the GPU?

Run the runtime configuration commands again if the toolkit was installed during this step. Confirm that Docker restarted successfully.

Check that the host command still reports the GPU. A host driver problem must be resolved before container passthrough can work.

Ask for help with the container output: help me troubleshoot NVIDIA GPU passthrough in Docker

Pin the source and prepare files

A production experiment needs an exact source revision. The tag fixes the upstream release while the custom branch gives ComputeCache a separate history.

• Move to your Desktop on the remote server by running this command:

cd ~/Desktop

Why use this location?

The Desktop gives the checkout a predictable location on the remote server. Later commands can enter the same computecache-vllm directory without guessing where it was created.

• Clone the pinned vLLM release by running this command:

git clone --branch v0.32.0 --depth 1 https://github.com/vllm-project/vllm.git computecache-vllm

What does this clone command do?

The branch option checks out the v0.32.0 release tag. The shallow depth downloads only the history needed for that revision.

The final argument names the local checkout computecache-vllm. This keeps the upstream source and ComputeCache work together.

The shallow clone can take a moment on a slower connection. You will return to the prompt when the source download finishes.

• Enter the checkout by running this command:

cd computecache-vllm

What changes here?

Your shell moves into the cloned repository. Every file and Git command that follows now targets the pinned vLLM source.

Before checking the source, what exact tag do you expect Git to report?

• Verify the exact checked-out tag by running this command:

git describe --tags --exact-match

What does this verification prove?

Git compares the current commit with the repository's tags. An exact match confirms the checkout has not drifted to another revision.

You should see v0.32.0. This is the source boundary used by every guarded edit later in the project.

• Create the ComputeCache branch by running this command:

git switch -c computecache/v0.32.0

Why create this branch?

The new branch starts at the pinned release commit. Your policy changes remain separate from the untouched release tag.

• Print the full source commit SHA by running this command:

git rev-parse HEAD

Why record the commit SHA?

A tag is a readable release label. The full commit SHA identifies the exact source object used for builds and canary results.

• Record the printed value here: your source commit SHA.

The checkout is now reproducible. The same tag and commit can be attached to every later test result.

• Create the required project directories by running this command:

mkdir -p tools scripts monitoring/grafana/provisioning/datasources tests/v1/core

What does this command create?

The parent option creates missing folders along each path. Existing folders remain intact.

The directories separate installation tooling from workloads. They also reserve locations for monitoring configuration and policy tests.

• Leave docker/Dockerfile unchanged.
• Copy the environment template by running this command:

cp .env.example .env

Why copy the environment template?

The copied .env file becomes the local runtime configuration. The upstream template remains available as a clean reference.

The next edit touches credentials. Keep both values private and exclude them from screenshots.

• Open .env in the terminal editor by running this command:

nano .env

How does this editor work?

GNU nano opens .env inside the SSH terminal. The file starts with the values copied from .env.example.

• Replace the VLLM_API_KEY placeholder with a long random secret from your password manager.
• Replace the GRAFANA_ADMIN_PASSWORD placeholder with a different long random secret.
• Press Ctrl+O to write the file.
• Press Enter to confirm the filename.
• Press Ctrl+X to exit nano.

Use the tabs below to compare the non-secret settings. Keep your two real secret values in .env.

✔️ Awesome, I've got everything!

Your environment file has two distinct secrets. Its ComputeCache settings still match the initial LRU configuration.

ⓧ I'd like to double check the full code

Compare the non-secret lines in your .env file with this template. Your first two values must contain your own distinct secrets.

VLLM_API_KEY=replace-with-a-long-random-secret
GRAFANA_ADMIN_PASSWORD=replace-with-a-different-long-random-secret
COMPUTECACHE_MODE=lru
COMPUTECACHE_SCAN_LIMIT=64
COMPUTECACHE_REUSE_WEIGHT=64
COMPUTECACHE_DEPTH_WEIGHT=1
COMPUTECACHE_AGE_WEIGHT=1
COMPUTECACHE_QUICK_DEMOTE_PENALTY=4096
COMPUTECACHE_LOG_EVERY=256

What should match?

The seven ComputeCache lines should match character for character. They establish untouched LRU behavior with the initial scan and scoring values.

The first two lines are placeholders in the reference only. Your .env file must contain two different non-placeholder secrets.

Before the final check, what tag and branch do you expect Git to report?

• Verify the pinned source and prepared paths by running these commands:

git describe --tags --exact-match
git rev-parse --abbrev-ref HEAD
git rev-parse HEAD
ls .env tools scripts monitoring/grafana/provisioning/datasources tests/v1/core

What does this checkpoint verify?

The first three commands report the exact release tag. They also report the active branch and full source commit.

The final command checks the environment file and required directories. It does not print either secret.

You should see v0.32.0 followed by computecache/v0.32.0 and the recorded commit SHA. The path check should complete without reporting a missing file or directory.

Checkpoint does not match?

Return to the clone command if the tag differs. The guarded integration depends on the exact release layout.

Run the branch command again if Git reports another branch. Recreate any missing directory with the directory command above.

Ask for help without sharing your environment file: help me fix my pinned vLLM checkout and project directory checkpoint

Your source and GPU path are pinned to real infrastructure. Next, you will run untouched LRU against a constrained KV cache and capture the scan-pollution baseline.

Reproduce LRU Cache Pollution

Your version-pinned vLLM checkout can already reach the remote GPU. You now need a real serving baseline that shows how the untouched cache behaves under pressure.

This step deliberately floods LRU cache replacement with one-use prompts after warming a shared prefix. The resulting JSON captures whether the valuable prefix survives.

In this step, get ready to:

• Configure the localhost-only model server and monitoring stack.
• Build a workload that measures real prefix-cache hit deltas.
• Run the LRU baseline until the warmed prefix loses cached tokens.

Configure the localhost serving stack

Docker Compose keeps the model server and its monitoring services on one private network. Loopback port bindings keep every host-facing endpoint limited to the remote server.

Prometheus collects the cache metrics. Grafana connects to that stored data for later canary inspection.

• Create compose.yaml beside .env in the computecache-vllm folder by using the same remote file editor you used earlier.
• Add the vLLM build settings and policy environment by pasting this first section:

services:
  vllm:
    build:
      context: .
      dockerfile: docker/Dockerfile
      target: vllm-openai
      args:
        VLLM_USE_PRECOMPILED: "1"
    image: computecache-vllm:0.32.0
    restart: unless-stopped
    ipc: host
    environment:
      COMPUTECACHE_MODE: ${COMPUTECACHE_MODE}
      COMPUTECACHE_SCAN_LIMIT: ${COMPUTECACHE_SCAN_LIMIT}
      COMPUTECACHE_REUSE_WEIGHT: ${COMPUTECACHE_REUSE_WEIGHT}
      COMPUTECACHE_DEPTH_WEIGHT: ${COMPUTECACHE_DEPTH_WEIGHT}
      COMPUTECACHE_AGE_WEIGHT: ${COMPUTECACHE_AGE_WEIGHT}
      COMPUTECACHE_QUICK_DEMOTE_PENALTY: ${COMPUTECACHE_QUICK_DEMOTE_PENALTY}
      COMPUTECACHE_LOG_EVERY: ${COMPUTECACHE_LOG_EVERY}

What does this configuration do?

• The build uses the upstream docker/Dockerfile with its vllm-openai target.
• The image name pins this project candidate to computecache-vllm:0.32.0.
• The environment entries pass your existing cache-policy settings into the server process.

• Continue the vllm service below its environment block by pasting:

    command:
      - --model
      - Qwen/Qwen3-0.6B
      - --max-model-len
      - "1024"
      - --num-gpu-blocks-override
      - "128"
      - --enable-prefix-caching
      - --kv-cache-metrics
      - --api-key
      - ${VLLM_API_KEY}
    ports:
      - 127.0.0.1:8000:8000
    volumes:
      - hf-cache:/root/.cache/huggingface
      - vllm-cache:/root/.cache/vllm
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

How does the canary create pressure?

• The server loads Qwen/Qwen3-0.6B with a 1024 token model limit.
• The 128 block override creates a constrained real KV cache.
• Prefix caching and KV metrics expose the behavior that the workload measures.
• The API port binds to 127.0.0.1:8000 on the remote host.

• Save compose.yaml.
• Validate the vLLM service by running:

docker compose config

What does this check prove?

Docker Compose resolves the values from .env and renders the vLLM service. A rendered configuration with no validation error confirms that the first service is valid.

Does the configuration fail to render?

Check the indentation beneath services and vllm. YAML uses spaces to represent nesting.

Confirm that .env remains beside compose.yaml.

Help me diagnose my Compose validation output.

• Add the Prometheus service below the vLLM service by pasting:

  prometheus:
    image: prom/prometheus:v3.15.0
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - 127.0.0.1:9090:9090
    depends_on: [vllm]

What does the Prometheus service do?

Prometheus 3.15.0 reads its scrape configuration from the project. Its data persists in the prometheus-data volume.

The dashboard endpoint binds to 127.0.0.1:9090. This keeps it off the server's public network interfaces.

• Add the Grafana service and named volumes below the Prometheus service by pasting:

  grafana:
    image: grafana/grafana:13.2.4
    restart: unless-stopped
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
    volumes:
      - ./monitoring/grafana/provisioning:/etc/grafana/provisioning:ro
      - grafana-data:/var/lib/grafana
    ports:
      - 127.0.0.1:3000:3000
    depends_on: [prometheus]

volumes:
  hf-cache:
  vllm-cache:
  prometheus-data:
  grafana-data:

What does the Grafana service do?

Grafana 13.2.4 loads provisioned settings from the monitoring/grafana/provisioning directory. Its local data persists in grafana-data.

The Grafana endpoint binds to 127.0.0.1:3000. Your existing password enters the container through its environment.

• Save compose.yaml.
• Validate all three service definitions by running:

docker compose config

What should the rendered stack contain?

The rendered output should contain vllm, prometheus, and grafana services. Each published port should begin with 127.0.0.1.

• Create monitoring/prometheus.yml inside computecache-vllm by using your remote file editor.
• Add the scrape configuration by pasting:

global:
  scrape_interval: 5s
  evaluation_interval: 5s
scrape_configs:
  - job_name: vllm-computecache
    metrics_path: /metrics
    static_configs:
      - targets: ["vllm:8000"]

How does metric collection work?

Prometheus requests /metrics from vllm:8000 every five seconds. The private Compose network resolves the vllm service name.

• Save monitoring/prometheus.yml.
• Create monitoring/grafana/provisioning/datasources/prometheus.yml by using your remote file editor.
• Add the provisioned data source by pasting:

apiVersion: 1
prune: true
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

Why provision the data source?

Provisioning connects Grafana to Prometheus whenever the container starts. The data source becomes the default source without a manual dashboard setup.

• Save monitoring/grafana/provisioning/datasources/prometheus.yml.
• Create .env.example beside .env by using your remote file editor.
• Add the safe configuration template by pasting:

VLLM_API_KEY=replace-with-a-long-random-secret
GRAFANA_ADMIN_PASSWORD=replace-with-a-different-long-random-secret
COMPUTECACHE_MODE=lru
COMPUTECACHE_SCAN_LIMIT=64
COMPUTECACHE_REUSE_WEIGHT=64
COMPUTECACHE_DEPTH_WEIGHT=1
COMPUTECACHE_AGE_WEIGHT=1
COMPUTECACHE_QUICK_DEMOTE_PENALTY=4096
COMPUTECACHE_LOG_EVERY=256

Why keep an environment template?

The template records every required setting with non-secret placeholders. Your real credentials remain only in .env.

• Save .env.example.

Do the monitoring files look misplaced?

Confirm that monitoring/prometheus.yml sits directly inside monitoring.

Confirm that the Grafana file sits inside monitoring/grafana/provisioning/datasources.

Help me check my monitoring file paths.

✔️ Awesome, I've got everything!

Great. Save all four configuration files before you continue.

ⓧ I'd like to double check the full code

services:
  vllm:
    build:
      context: .
      dockerfile: docker/Dockerfile
      target: vllm-openai
      args:
        VLLM_USE_PRECOMPILED: "1"
    image: computecache-vllm:0.32.0
    restart: unless-stopped
    ipc: host
    environment:
      COMPUTECACHE_MODE: ${COMPUTECACHE_MODE}
      COMPUTECACHE_SCAN_LIMIT: ${COMPUTECACHE_SCAN_LIMIT}
      COMPUTECACHE_REUSE_WEIGHT: ${COMPUTECACHE_REUSE_WEIGHT}
      COMPUTECACHE_DEPTH_WEIGHT: ${COMPUTECACHE_DEPTH_WEIGHT}
      COMPUTECACHE_AGE_WEIGHT: ${COMPUTECACHE_AGE_WEIGHT}
      COMPUTECACHE_QUICK_DEMOTE_PENALTY: ${COMPUTECACHE_QUICK_DEMOTE_PENALTY}
      COMPUTECACHE_LOG_EVERY: ${COMPUTECACHE_LOG_EVERY}
    command:
      - --model
      - Qwen/Qwen3-0.6B
      - --max-model-len
      - "1024"
      - --num-gpu-blocks-override
      - "128"
      - --enable-prefix-caching
      - --kv-cache-metrics
      - --api-key
      - ${VLLM_API_KEY}
    ports:
      - 127.0.0.1:8000:8000
    volumes:
      - hf-cache:/root/.cache/huggingface
      - vllm-cache:/root/.cache/vllm
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  prometheus:
    image: prom/prometheus:v3.15.0
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - 127.0.0.1:9090:9090
    depends_on: [vllm]

  grafana:
    image: grafana/grafana:13.2.4
    restart: unless-stopped
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
    volumes:
      - ./monitoring/grafana/provisioning:/etc/grafana/provisioning:ro
      - grafana-data:/var/lib/grafana
    ports:
      - 127.0.0.1:3000:3000
    depends_on: [prometheus]

volumes:
  hf-cache:
  vllm-cache:
  prometheus-data:
  grafana-data:

VLLM_API_KEY=replace-with-a-long-random-secret
GRAFANA_ADMIN_PASSWORD=replace-with-a-different-long-random-secret
COMPUTECACHE_MODE=lru
COMPUTECACHE_SCAN_LIMIT=64
COMPUTECACHE_REUSE_WEIGHT=64
COMPUTECACHE_DEPTH_WEIGHT=1
COMPUTECACHE_AGE_WEIGHT=1
COMPUTECACHE_QUICK_DEMOTE_PENALTY=4096
COMPUTECACHE_LOG_EVERY=256

global:
  scrape_interval: 5s
  evaluation_interval: 5s
scrape_configs:
  - job_name: vllm-computecache
    metrics_path: /metrics
    static_configs:
      - targets: ["vllm:8000"]

apiVersion: 1
prune: true
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

Build the prefix-pressure workload

The workload measures cache-hit counter changes around individual replay requests. It uses the real model endpoint and the real metrics endpoint.

• Create scripts/prefix_pressure.py inside the existing scripts directory by using your remote file editor.
• Add the imports and project configuration by pasting:

#!/usr/bin/env python3
from __future__ import annotations

import json
import os
import time
import urllib.error
import urllib.request
from pathlib import Path


def load_project_env() -> None:
    path = Path(__file__).resolve().parents[1] / ".env"
    if not path.exists():
        return
    for raw in path.read_text(encoding="utf-8").splitlines():
        line = raw.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        os.environ.setdefault(key, value)


load_project_env()
BASE_URL = os.getenv("VLLM_BASE_URL", "http://127.0.0.1:8000")
API_KEY = os.environ["VLLM_API_KEY"]
MODEL = "Qwen/Qwen3-0.6B"
HIT_METRIC = "vllm:prefix_cache_hits"

What does this setup do?

• The load_project_env() function reads settings from the project-level .env file.
• The request base defaults to the loopback-bound vLLM server.
• The workload measures changes in vllm:prefix_cache_hits.

• Continue scripts/prefix_pressure.py with the authenticated request helper by pasting:

def request(path, payload=None):
    data = None if payload is None else json.dumps(payload).encode("utf-8")
    headers = {"Authorization": f"Bearer {API_KEY}"}
    if data is not None:
        headers["Content-Type"] = "application/json"
    req = urllib.request.Request(
        f"{BASE_URL}{path}",
        data=data,
        headers=headers,
        method="POST" if data else "GET",
    )
    try:
        with urllib.request.urlopen(req, timeout=120) as response:
            return response.read().decode("utf-8")
    except urllib.error.HTTPError as exc:
        body = exc.read().decode("utf-8", errors="replace")
        raise RuntimeError(f"HTTP {exc.code}: {body}") from exc

How are requests authenticated?

The request() helper adds your bearer credential to every request. It also turns Python payloads into JSON for chat requests.

HTTP failures include the response body in the raised error. This makes server-side request problems easier to diagnose.

• Add the metrics counter reader below request() by pasting:

def counter_value(metric):
    accepted, total = {metric, f"{metric}_total"}, 0.0
    for line in request("/metrics").splitlines():
        if not line or line.startswith("#"):
            continue
        sample, value, *_ = line.split()
        if sample.split("{", 1)[0] in accepted:
            total += float(value)
    return total

How is the hit counter read?

The counter_value() function accepts either form of the metric name. It adds matching samples across every exposed label set.

• Add the chat request helper below counter_value() by pasting:

def chat(system_prompt, user_prompt):
    request(
        "/v1/chat/completions",
        {
            "model": MODEL,
            "messages": [
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_prompt},
            ],
            "max_completion_tokens": 1,
            "temperature": 0.0,
        },
    )

Why keep each response short?

The chat() helper sends the repeated prefix through /v1/chat/completions. A one-token completion keeps the experiment focused on prefix reuse.

• Add the measured replay helper below chat() by pasting:

def measured_chat(system_prompt, user_prompt):
    before = counter_value(HIT_METRIC)
    chat(system_prompt, user_prompt)
    time.sleep(0.2)
    return max(0, round(counter_value(HIT_METRIC) - before))

How is one replay measured?

The measured_chat() function samples the cache-hit counter before the request. It returns the non-negative counter increase after the request.

• Start the pressure scenario below measured_chat() by pasting:

def main():
    hot = (
        "You are the production incident assistant. Preserve this shared policy, "
        "cite observations before conclusions, and answer in one sentence. "
    ) * 80
    chat(hot, "Warm zero")
    warm_hit = measured_chat(hot, "Warm one")
    post_flood_hit, flood_requests = warm_hit, 0
    for batch in range(1, 9):
        for index in range(10):
            tenant = batch * 10 + index
            unique = (
                f"Tenant {tenant} has a unique one-use incident policy {tenant}. "
            ) * 90
            chat(unique, f"Summarize incident {tenant}")
            flood_requests += 1
        post_flood_hit = measured_chat(hot, f"Replay after {flood_requests}")
        if post_flood_hit < warm_hit:
            break

How does the workload create pollution?

The first two requests establish a reusable incident-assistant prefix. Each tenant request then inserts a different one-use prefix into the same constrained cache.

The workload replays the hot prefix after each group of ten flood requests. It stops once the replay receives fewer cached tokens.

• Finish main() with the result report and script entry point by pasting:

    print(
        json.dumps(
            {
                "mode": os.getenv("COMPUTECACHE_MODE", "unknown"),
                "warm_prefix_cached_tokens": warm_hit,
                "post_flood_cached_tokens": post_flood_hit,
                "flood_requests": flood_requests,
                "hot_prefix_preserved": post_flood_hit >= warm_hit and warm_hit > 0,
            },
            indent=2,
            sort_keys=True,
        )
    )


if __name__ == "__main__":
    main()

What does the final JSON capture?

The report records the active mode and both cache-hit measurements. It also records how many one-use requests were needed.

The hot_prefix_preserved field turns the comparison into a clear canary result.

• Save scripts/prefix_pressure.py.

Does the script show an indentation problem?

Confirm that the final print() call remains inside main() with four leading spaces.

Confirm that the last main() call remains below the entry-point condition.

Help me compare my workload script with the complete version.

✔️ Awesome, I've got everything!

The pressure workload is ready. Make sure scripts/prefix_pressure.py is saved.

ⓧ I'd like to double check the full code

#!/usr/bin/env python3
from __future__ import annotations

import json
import os
import time
import urllib.error
import urllib.request
from pathlib import Path


def load_project_env() -> None:
    path = Path(__file__).resolve().parents[1] / ".env"
    if not path.exists():
        return
    for raw in path.read_text(encoding="utf-8").splitlines():
        line = raw.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        os.environ.setdefault(key, value)


load_project_env()
BASE_URL = os.getenv("VLLM_BASE_URL", "http://127.0.0.1:8000")
API_KEY = os.environ["VLLM_API_KEY"]
MODEL = "Qwen/Qwen3-0.6B"
HIT_METRIC = "vllm:prefix_cache_hits"


def request(path, payload=None):
    data = None if payload is None else json.dumps(payload).encode("utf-8")
    headers = {"Authorization": f"Bearer {API_KEY}"}
    if data is not None:
        headers["Content-Type"] = "application/json"
    req = urllib.request.Request(
        f"{BASE_URL}{path}",
        data=data,
        headers=headers,
        method="POST" if data else "GET",
    )
    try:
        with urllib.request.urlopen(req, timeout=120) as response:
            return response.read().decode("utf-8")
    except urllib.error.HTTPError as exc:
        body = exc.read().decode("utf-8", errors="replace")
        raise RuntimeError(f"HTTP {exc.code}: {body}") from exc


def counter_value(metric):
    accepted, total = {metric, f"{metric}_total"}, 0.0
    for line in request("/metrics").splitlines():
        if not line or line.startswith("#"):
            continue
        sample, value, *_ = line.split()
        if sample.split("{", 1)[0] in accepted:
            total += float(value)
    return total


def chat(system_prompt, user_prompt):
    request(
        "/v1/chat/completions",
        {
            "model": MODEL,
            "messages": [
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_prompt},
            ],
            "max_completion_tokens": 1,
            "temperature": 0.0,
        },
    )


def measured_chat(system_prompt, user_prompt):
    before = counter_value(HIT_METRIC)
    chat(system_prompt, user_prompt)
    time.sleep(0.2)
    return max(0, round(counter_value(HIT_METRIC) - before))


def main():
    hot = (
        "You are the production incident assistant. Preserve this shared policy, "
        "cite observations before conclusions, and answer in one sentence. "
    ) * 80
    chat(hot, "Warm zero")
    warm_hit = measured_chat(hot, "Warm one")
    post_flood_hit, flood_requests = warm_hit, 0
    for batch in range(1, 9):
        for index in range(10):
            tenant = batch * 10 + index
            unique = (
                f"Tenant {tenant} has a unique one-use incident policy {tenant}. "
            ) * 90
            chat(unique, f"Summarize incident {tenant}")
            flood_requests += 1
        post_flood_hit = measured_chat(hot, f"Replay after {flood_requests}")
        if post_flood_hit < warm_hit:
            break
    print(
        json.dumps(
            {
                "mode": os.getenv("COMPUTECACHE_MODE", "unknown"),
                "warm_prefix_cached_tokens": warm_hit,
                "post_flood_cached_tokens": post_flood_hit,
                "flood_requests": flood_requests,
                "hot_prefix_preserved": post_flood_hit >= warm_hit and warm_hit > 0,
            },
            indent=2,
            sort_keys=True,
        )
    )


if __name__ == "__main__":
    main()

The first image build is the slowest part of this step. Docker compiles the custom vLLM image before it can start the model server.

• Build the image and start all three services by running:

docker compose up -d --build

What does this command start?

Docker Compose builds computecache-vllm:0.32.0 from the pinned source checkout. It then starts vLLM, Prometheus, and Grafana on their private network.

The named volumes preserve model downloads and monitoring data across container restarts. The three host ports remain bound to loopback.

Does the stack fail to start?

Check the build output for the first failed stage. A source build failure appears before the services start.

Check that the GPU reservation still uses one NVIDIA device with the gpu capability.

Help me diagnose my Compose build or startup failure.

Expose LRU cache pollution

The server must be ready before the workload sends chat requests. The authenticated model-list response provides a concrete readiness signal.

• Export the existing .env values into your current SSH session by running:

set -a; . ./.env; set +a

Why export the environment?

This loads VLLM_API_KEY and the cache settings into the current shell. The health request and workload can now use the same configuration as the container.

• Check model readiness by running this command until the served model appears:

curl -H "Authorization: Bearer $VLLM_API_KEY" http://127.0.0.1:8000/v1/models

What does readiness look like?

The response should list Qwen/Qwen3-0.6B as an available model. This proves the authenticated API is accepting requests.

The model server is live on the remote GPU. You can now put untouched LRU through the workload it is expected to mishandle.

Before you run the workload, do you think the replay will retain the same cached-token count after dozens of one-use prefixes?

• Run the fixed prefix-pressure workload by using:

python3 scripts/prefix_pressure.py

What does the result reveal?

The JSON should report lru mode with a positive warm_prefix_cached_tokens value. That positive value proves the shared prefix warmed successfully.

Under sufficient pressure, post_flood_cached_tokens becomes lower than the warm measurement. That drop is the intended LRU scan-pollution result.

• Copy the complete JSON object from the terminal.
• Save the copied object in your canary notes under LRU baseline.

Did the hot prefix survive the flood?

Confirm that COMPUTECACHE_MODE remains set to lru in .env.

Confirm that compose.yaml uses the 128 GPU-block override. A larger cache can absorb the flood without evicting the warmed prefix.

Help me understand why my LRU baseline preserved the hot prefix.

You have captured the failure that the replacement policy must solve. Next, you will add bounded victim selection while leaving vLLM's allocation ownership intact.

Install Bounded ComputeCache

Your real vLLM baseline exposed the weakness in LRU. One-use prompts pushed the warmed prefix out of the constrained GPU cache.

This step adds ComputeCache as a bounded ordering policy. It changes which already-free blocks reach eviction first while leaving allocation ownership inside BlockPool.

In this step, get ready to:

• Add fixed-size policy state for reuse frequency and access age.
• Apply six guarded integration edits to the pinned vLLM source.
• Review the ownership boundary and verify the source diff.

Add bounded policy state

The policy needs a narrow view of each block and its free queue. Protocols define that boundary without taking ownership of vLLM's block objects.

• Create vllm/v1/core/computecache_policy.py inside the existing computecache-vllm checkout using your remote code editor.
• Add the imports and queue contracts by copying this first chunk into the file:

# SPDX-License-Identifier: Apache-2.0
"""Bounded production-aware ordering for already-free KV blocks."""
from __future__ import annotations

import logging
import os
from collections.abc import Iterable, Iterator
from typing import Protocol

logger = logging.getLogger(__name__)


class BlockLike(Protocol):
    block_id: int
    ref_cnt: int
    block_hash: object | None
    block_hash_num_tokens: int | None
    is_null: bool


class FreeQueueLike(Protocol):
    def iter_blocks_after(self, cursor: BlockLike | None) -> Iterator[BlockLike]: ...
    def remove(self, block: BlockLike) -> None: ...
    def prepend_n(self, blocks: list[BlockLike]) -> None: ...

What Does This Code Do?

• The BlockLike protocol lists the block fields used for ranking and safety checks.
• The FreeQueueLike protocol exposes iteration and reordering operations that already exist on vLLM's free queue.
• These contracts keep the policy focused on victim ordering.

• Save vllm/v1/core/computecache_policy.py.
• Confirm your editor now lists BlockLike and FreeQueueLike in the file outline.

Missing the protocol classes?

Check that the file sits at vllm/v1/core/computecache_policy.py inside the checkout. Confirm the class lines start at the left edge.

Ask for help if the editor flags the protocol definitions: help me check the imports and indentation in my ComputeCache protocol definitions

Environment values need defensive parsing because a malformed setting must never leave eviction behavior partially configured. Invalid numeric values fall back to known defaults.

• Add the integer configuration reader below FreeQueueLike by copying this chunk:

def _read_int(name: str, default: int, minimum: int) -> int:
    raw = os.getenv(name)
    if raw is None:
        return default
    try:
        value = int(raw)
    except ValueError:
        logger.warning("Invalid %s=%r; using %d", name, raw, default)
        return default
    if value < minimum:
        logger.warning("%s must be at least %d; using %d", name, minimum, default)
        return default
    return value

How Does Fallback Work?

• The reader accepts an environment-variable name with a default and minimum value.
• A missing value uses the default immediately.
• A malformed or undersized value produces a warning before the default is used.

• Save vllm/v1/core/computecache_policy.py.
• Confirm the editor recognizes _read_int as a top-level function.

Is _read_int nested?

Make sure def _read_int starts at the left edge. Indenting it beneath FreeQueueLike turns it into part of that protocol.

Use this prompt for an indentation check: help me place _read_int at module scope without changing its code

Policy metadata stays proportional to the configured block count. Two fixed-size arrays store reuse frequency and the last access clock for each block ID.

• Add the policy constructor below _read_int by copying this chunk:

class ComputeCachePolicy:
    def __init__(
        self,
        num_blocks: int,
        hash_block_size: int,
        mode: str,
        scan_limit: int,
        reuse_weight: int,
        depth_weight: int,
        age_weight: int,
        quick_demote_penalty: int,
        log_every: int,
    ) -> None:
        self.num_blocks = num_blocks
        self.hash_block_size = hash_block_size
        self.mode = mode
        self.scan_limit = scan_limit
        self.reuse_weight = reuse_weight
        self.depth_weight = depth_weight
        self.age_weight = age_weight
        self.quick_demote_penalty = quick_demote_penalty
        self.log_every = log_every
        self._clock = 0
        self._reuse_count = [0] * num_blocks
        self._last_access = [0] * num_blocks
        self.decisions = 0
        self.reordered_decisions = 0

Why Fixed-Size Arrays?

• Each block ID maps directly to one position in _reuse_count and _last_access.
• The arrays avoid per-decision metadata allocation.
• The decision counters provide low-volume evidence that active mode is running.

• Save vllm/v1/core/computecache_policy.py.
• Confirm the editor outline now shows ComputeCachePolicy with an __init__ method.

Is the class missing methods?

Check that class ComputeCachePolicy starts at the left edge. Every constructor line beneath it must use consistent indentation.

Ask for a structural review here: help me check the ComputeCachePolicy constructor indentation

The environment factory turns the existing .env settings into one policy instance. Only lru and active are accepted modes.

• Continue inside ComputeCachePolicy by adding the environment parsing portion of from_environ below the constructor:

    @classmethod
    def from_environ(
        cls, num_blocks: int, hash_block_size: int
    ) -> "ComputeCachePolicy":
        mode = os.getenv("COMPUTECACHE_MODE", "lru").lower()
        if mode not in {"lru", "active"}:
            logger.warning("Invalid COMPUTECACHE_MODE=%r; using lru", mode)
            mode = "lru"
        policy = cls(
            num_blocks,
            hash_block_size,
            mode,
            _read_int("COMPUTECACHE_SCAN_LIMIT", 64, 1),
            _read_int("COMPUTECACHE_REUSE_WEIGHT", 64, 0),
            _read_int("COMPUTECACHE_DEPTH_WEIGHT", 1, 0),
            _read_int("COMPUTECACHE_AGE_WEIGHT", 1, 0),
            _read_int("COMPUTECACHE_QUICK_DEMOTE_PENALTY", 4096, 0),
            _read_int("COMPUTECACHE_LOG_EVERY", 256, 1),
        )

What Gets Loaded?

• The mode controls whether the policy leaves LRU ordering untouched or activates candidate reordering.
• The scan limit caps how many free blocks one decision can inspect.
• The remaining settings control reuse value and recompute-depth value plus age cost and quick demotion.

• Save vllm/v1/core/computecache_policy.py.
• Confirm the method contains all seven COMPUTECACHE_ settings from your existing .env file.

Seeing an incomplete method?

Keep this chunk indented beneath ComputeCachePolicy. The next chunk completes the same method.

Ask for help comparing the environment names: help me compare my ComputeCache environment settings with from_environ

• Complete from_environ by adding its startup log and return statement immediately after the policy construction:

        logger.warning(
            "ComputeCache mode=%s scan_limit=%d reuse_weight=%d "
            "depth_weight=%d age_weight=%d quick_demote_penalty=%d",
            policy.mode,
            policy.scan_limit,
            policy.reuse_weight,
            policy.depth_weight,
            policy.age_weight,
            policy.quick_demote_penalty,
        )
        return policy

Why Log the Settings?

The startup log exposes the effective mode and ranking weights. Operators can verify active behavior without reading the container's environment.

• Save vllm/v1/core/computecache_policy.py.
• Confirm return policy aligns with the startup logging call inside from_environ.

Is the return misplaced?

Keep return policy inside from_environ. It must align with logger.warning.

Use this prompt for a whitespace check: help me check the indentation at the end of from_environ

Access and eviction hooks keep the arrays synchronized with real cache activity. A full cache reset also clears the policy clock and counters.

• Add the metadata lifecycle methods below from_environ by copying this chunk:

    def _tick(self) -> int:
        self._clock += 1
        return self._clock

    def on_access(self, block: BlockLike) -> None:
        if block.is_null:
            return
        now = self._tick()
        block_id = block.block_id
        self._reuse_count[block_id] = min(self._reuse_count[block_id] + 1, 255)
        self._last_access[block_id] = now

    def on_evicted(self, block: BlockLike) -> None:
        self._reuse_count[block.block_id] = 0
        self._last_access[block.block_id] = 0

    def reset(self) -> None:
        self._clock = 0
        self._reuse_count[:] = [0] * self.num_blocks
        self._last_access[:] = [0] * self.num_blocks
        self.decisions = 0
        self.reordered_decisions = 0

How Does Metadata Stay Safe?

• Each real cache hit increments a capped reuse count and records a logical access time.
• A true eviction clears metadata for the recycled block ID.
• A prefix-cache reset clears every policy value in place.

• Save vllm/v1/core/computecache_policy.py.
• Confirm your editor outline lists _tick, on_access, on_evicted, and reset.

Are lifecycle methods missing?

Check that all four methods align with from_environ inside the class. A method nested inside from_environ does not appear in the class outline.

Ask for help with the method boundaries: help me check the lifecycle method indentation in ComputeCachePolicy

Ranking estimates the value of retaining a cached block. Reuse frequency and prefix depth increase keep value while age lowers it.

• Add the ranking method below reset by copying this chunk:

    def _rank(self, block: BlockLike, position: int) -> tuple[int, int, int, int]:
        if block.block_hash is None:
            return (0, 0, position, block.block_id)
        block_id = block.block_id
        hits = self._reuse_count[block_id]
        last_access = self._last_access[block_id]
        age = self._clock - last_access if last_access else self._clock
        depth = max(
            1,
            (block.block_hash_num_tokens or self.hash_block_size)
            // self.hash_block_size,
        )
        keep_value = (
            hits * self.reuse_weight
            + min(depth, 4096) * self.depth_weight
            - min(age, 4096) * self.age_weight
        )
        if hits == 0:
            keep_value -= self.quick_demote_penalty
        return (1, keep_value, position, block_id)

How Are Victims Ranked?

• Unhashed free blocks receive the lowest rank because they hold no reusable prefix.
• Repeated access and deeper cached prefixes raise a block's keep value.
• Never-reused blocks receive the quick-demotion penalty.
• Queue position and block ID provide deterministic tie breaking.

• Save vllm/v1/core/computecache_policy.py.
• Confirm the editor recognizes _rank as a method returning a four-value tuple.

Is the rank method flagged?

Check the parentheses around keep_value. Confirm the integer division line remains inside the max call.

Use this prompt for a focused syntax review: help me compare my _rank method with the provided ComputeCache code

Victim selection must have constant configured work even when the total cache grows. The method first copies at most scan_limit entries from the front of the existing free queue.

• Add the bounded candidate collection portion of reorder_free_queue below _rank by copying this chunk:

    def reorder_free_queue(
        self,
        queue: FreeQueueLike,
        num_blocks: int,
        protected_block_ids: Iterable[int],
    ) -> None:
        if self.mode != "active" or num_blocks <= 0:
            return
        protected = set(protected_block_ids)
        window: list[BlockLike] = []
        for block in queue.iter_blocks_after(None):
            if len(window) >= self.scan_limit:
                break
            window.append(block)
        candidates: list[tuple[int, BlockLike]] = []
        for position, block in enumerate(window):
            if block.is_null or block.block_id in protected:
                continue
            if block.ref_cnt != 0:
                raise RuntimeError(
                    f"ComputeCache received pinned block {block.block_id}"
                )
            candidates.append((position, block))
        target = min(num_blocks, len(candidates))
        if target == 0:
            return

Where Is the Safety Boundary?

• LRU mode returns before the queue is inspected.
• The loop stops as soon as the configured candidate limit is reached.
• Null blocks and active reuse-watcher IDs stay outside candidate selection.
• A nonzero reference count triggers an immediate failure because the policy received a pinned block.

• Save vllm/v1/core/computecache_policy.py.
• Confirm the candidate loop checks self.scan_limit before appending another block.

Is the scan unbounded?

Make sure the break statement remains inside the scan-limit condition. The append line must stay outside that condition.

Ask for a bounded-loop review here: help me verify that reorder_free_queue inspects at most scan_limit blocks

The final selection sorts only the bounded candidate set. It moves selected victims to the front while preserving vLLM's existing eviction path.

• Complete reorder_free_queue by adding the selection and logging portion immediately after the zero-target check:

        selected = [
            block
            for position, block in sorted(
                candidates, key=lambda item: self._rank(item[1], item[0])
            )[:target]
        ]
        self.decisions += 1
        if [b.block_id for b in selected] != [b.block_id for b in window[:target]]:
            self.reordered_decisions += 1
        for block in selected:
            queue.remove(block)
        queue.prepend_n(selected)
        if self.decisions % self.log_every == 0:
            logger.warning(
                "ComputeCache decisions=%d reordered=%d scan_limit=%d",
                self.decisions,
                self.reordered_decisions,
                self.scan_limit,
            )

What Changes in the Queue?

• Sorting places the lowest keep-value candidates first.
• The selected blocks are removed from their old positions before being prepended.
• BlockPool still performs hash removal and reference-count changes after it pops those blocks.
• Periodic logs reveal how often ComputeCache changed the original ordering.

• Save vllm/v1/core/computecache_policy.py.
• Confirm the file outline shows reorder_free_queue as the final method of ComputeCachePolicy.

Is queue reordering incomplete?

Confirm queue.remove runs once for each selected block. Confirm queue.prepend_n runs after that loop.

Use this prompt for a final method comparison: help me compare my reorder_free_queue method with the complete version

Use the reference below to catch any missing line before the installer imports the policy.

✔️ Awesome, I've got everything!

Great. Confirm vllm/v1/core/computecache_policy.py is saved before continuing.

ⓧ I'd like to double check the full code

# SPDX-License-Identifier: Apache-2.0
"""Bounded production-aware ordering for already-free KV blocks."""
from __future__ import annotations

import logging
import os
from collections.abc import Iterable, Iterator
from typing import Protocol

logger = logging.getLogger(__name__)


class BlockLike(Protocol):
    block_id: int
    ref_cnt: int
    block_hash: object | None
    block_hash_num_tokens: int | None
    is_null: bool


class FreeQueueLike(Protocol):
    def iter_blocks_after(self, cursor: BlockLike | None) -> Iterator[BlockLike]: ...
    def remove(self, block: BlockLike) -> None: ...
    def prepend_n(self, blocks: list[BlockLike]) -> None: ...


def _read_int(name: str, default: int, minimum: int) -> int:
    raw = os.getenv(name)
    if raw is None:
        return default
    try:
        value = int(raw)
    except ValueError:
        logger.warning("Invalid %s=%r; using %d", name, raw, default)
        return default
    if value < minimum:
        logger.warning("%s must be at least %d; using %d", name, minimum, default)
        return default
    return value


class ComputeCachePolicy:
    def __init__(
        self,
        num_blocks: int,
        hash_block_size: int,
        mode: str,
        scan_limit: int,
        reuse_weight: int,
        depth_weight: int,
        age_weight: int,
        quick_demote_penalty: int,
        log_every: int,
    ) -> None:
        self.num_blocks = num_blocks
        self.hash_block_size = hash_block_size
        self.mode = mode
        self.scan_limit = scan_limit
        self.reuse_weight = reuse_weight
        self.depth_weight = depth_weight
        self.age_weight = age_weight
        self.quick_demote_penalty = quick_demote_penalty
        self.log_every = log_every
        self._clock = 0
        self._reuse_count = [0] * num_blocks
        self._last_access = [0] * num_blocks
        self.decisions = 0
        self.reordered_decisions = 0

    @classmethod
    def from_environ(
        cls, num_blocks: int, hash_block_size: int
    ) -> "ComputeCachePolicy":
        mode = os.getenv("COMPUTECACHE_MODE", "lru").lower()
        if mode not in {"lru", "active"}:
            logger.warning("Invalid COMPUTECACHE_MODE=%r; using lru", mode)
            mode = "lru"
        policy = cls(
            num_blocks,
            hash_block_size,
            mode,
            _read_int("COMPUTECACHE_SCAN_LIMIT", 64, 1),
            _read_int("COMPUTECACHE_REUSE_WEIGHT", 64, 0),
            _read_int("COMPUTECACHE_DEPTH_WEIGHT", 1, 0),
            _read_int("COMPUTECACHE_AGE_WEIGHT", 1, 0),
            _read_int("COMPUTECACHE_QUICK_DEMOTE_PENALTY", 4096, 0),
            _read_int("COMPUTECACHE_LOG_EVERY", 256, 1),
        )
        logger.warning(
            "ComputeCache mode=%s scan_limit=%d reuse_weight=%d "
            "depth_weight=%d age_weight=%d quick_demote_penalty=%d",
            policy.mode,
            policy.scan_limit,
            policy.reuse_weight,
            policy.depth_weight,
            policy.age_weight,
            policy.quick_demote_penalty,
        )
        return policy

    def _tick(self) -> int:
        self._clock += 1
        return self._clock

    def on_access(self, block: BlockLike) -> None:
        if block.is_null:
            return
        now = self._tick()
        block_id = block.block_id
        self._reuse_count[block_id] = min(self._reuse_count[block_id] + 1, 255)
        self._last_access[block_id] = now

    def on_evicted(self, block: BlockLike) -> None:
        self._reuse_count[block.block_id] = 0
        self._last_access[block.block_id] = 0

    def reset(self) -> None:
        self._clock = 0
        self._reuse_count[:] = [0] * self.num_blocks
        self._last_access[:] = [0] * self.num_blocks
        self.decisions = 0
        self.reordered_decisions = 0

    def _rank(self, block: BlockLike, position: int) -> tuple[int, int, int, int]:
        if block.block_hash is None:
            return (0, 0, position, block.block_id)
        block_id = block.block_id
        hits = self._reuse_count[block_id]
        last_access = self._last_access[block_id]
        age = self._clock - last_access if last_access else self._clock
        depth = max(
            1,
            (block.block_hash_num_tokens or self.hash_block_size)
            // self.hash_block_size,
        )
        keep_value = (
            hits * self.reuse_weight
            + min(depth, 4096) * self.depth_weight
            - min(age, 4096) * self.age_weight
        )
        if hits == 0:
            keep_value -= self.quick_demote_penalty
        return (1, keep_value, position, block_id)

    def reorder_free_queue(
        self,
        queue: FreeQueueLike,
        num_blocks: int,
        protected_block_ids: Iterable[int],
    ) -> None:
        if self.mode != "active" or num_blocks <= 0:
            return
        protected = set(protected_block_ids)
        window: list[BlockLike] = []
        for block in queue.iter_blocks_after(None):
            if len(window) >= self.scan_limit:
                break
            window.append(block)
        candidates: list[tuple[int, BlockLike]] = []
        for position, block in enumerate(window):
            if block.is_null or block.block_id in protected:
                continue
            if block.ref_cnt != 0:
                raise RuntimeError(
                    f"ComputeCache received pinned block {block.block_id}"
                )
            candidates.append((position, block))
        target = min(num_blocks, len(candidates))
        if target == 0:
            return
        selected = [
            block
            for position, block in sorted(
                candidates, key=lambda item: self._rank(item[1], item[0])
            )[:target]
        ]
        self.decisions += 1
        if [b.block_id for b in selected] != [b.block_id for b in window[:target]]:
            self.reordered_decisions += 1
        for block in selected:
            queue.remove(block)
        queue.prepend_n(selected)
        if self.decisions % self.log_every == 0:
            logger.warning(
                "ComputeCache decisions=%d reordered=%d scan_limit=%d",
                self.decisions,
                self.reordered_decisions,
                self.scan_limit,
            )

This reference is the complete bounded policy file. Compare spacing and identifiers exactly.

Apply guarded integration

The pinned source needs six precise hooks. A guarded installer verifies the exact tag and each source anchor before touching vllm/v1/core/block_pool.py.

• Create tools/install_computecache.py inside the existing computecache-vllm checkout using your remote code editor.
• Add the installer setup and single-anchor replacement helper by copying this chunk:

#!/usr/bin/env python3
"""Apply guarded ComputeCache hooks to vLLM v0.32.0."""
from __future__ import annotations

import subprocess
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
TARGET = ROOT / "vllm/v1/core/block_pool.py"
EXPECTED_TAG = "v0.32.0"


def replace_once(text: str, old: str, new: str, label: str) -> str:
    count = text.count(old)
    if count != 1:
        raise RuntimeError(
            f"Refusing {label}: expected one verified anchor, found {count}"
        )
    return text.replace(old, new, 1)

Why Guard Every Anchor?

• The installer targets the source file relative to its own location.
• The expected tag pins the integration to v0.32.0.
• The replacement helper refuses missing or duplicated anchors instead of guessing where code belongs.

• Save tools/install_computecache.py.
• Confirm your editor outline lists replace_once as a top-level function.

Is the installer path wrong?

Confirm the file is inside the existing tools directory. The ROOT calculation depends on that location.

Ask for help checking the path layout: help me verify the installer location and ROOT calculation

The installer checks the repository tag before reading the target file. A second run exits cleanly when all hooks are already present.

• Add the initial tag and idempotency checks below replace_once by copying this chunk:

def main() -> None:
    tag = subprocess.run(
        ["git", "describe", "--tags", "--exact-match"],
        cwd=ROOT,
        check=True,
        capture_output=True,
        text=True,
    ).stdout.strip()
    if tag != EXPECTED_TAG:
        raise RuntimeError(f"Expected {EXPECTED_TAG}, found {tag}")
    text = TARGET.read_text(encoding="utf-8")
    if "from vllm.v1.core.computecache_policy import ComputeCachePolicy" in text:
        print("ComputeCache hooks are already installed for v0.32.0")
        return

What Gets Verified First?

• The repository must resolve to the exact pinned tag.
• The target source is read only after the tag passes.
• An existing policy import proves the guarded integration has already run.

• Save tools/install_computecache.py.
• Confirm main appears in the editor outline beneath replace_once.

Is main nested incorrectly?

Make sure def main starts at the left edge. It must not sit inside replace_once.

Use this prompt for a function-boundary check: help me check whether main is defined at module scope

The first two edits import the policy and construct it from the existing environment. Construction receives the real GPU block count and hash-block size.

• Continue inside main by adding the import and construction replacements after the idempotency check:

    text = replace_once(
        text,
        "from vllm.v1.core.kv_cache_metrics import KVCacheMetricsCollector\n",
        "from vllm.v1.core.computecache_policy import ComputeCachePolicy\n"
        "from vllm.v1.core.kv_cache_metrics import KVCacheMetricsCollector\n",
        "policy import",
    )
    text = replace_once(
        text,
        "        self.metrics_collector = metrics_collector\n"
        "        # Callbacks for blocks released with ``unpin_blocks`` whose contents\n",
        "        self.metrics_collector = metrics_collector\n"
        "        self.compute_cache_policy = ComputeCachePolicy.from_environ(\n"
        "            num_blocks=num_gpu_blocks, hash_block_size=hash_block_size\n"
        "        )\n"
        "        # Callbacks for blocks released with ``unpin_blocks`` whose contents\n",
        "policy construction",
    )

What Do These Edits Add?

• The import makes ComputeCachePolicy available to BlockPool.
• The constructor creates one policy for the real GPU block pool.
• Each replacement preserves the verified upstream lines around the new code.

• Save tools/install_computecache.py.
• Confirm this chunk contains the labels policy import and policy construction.

Are replacement strings broken?

Keep each quoted line ending with \n exactly as shown. Python joins adjacent quoted strings inside the call.

Ask for help checking only the string boundaries: help me compare the first two replace_once calls with the reference

The next hooks change victim ordering before BlockPool pops free blocks. Metadata resets only after a block loses its cached hash through the real eviction path.

• Add the victim-reordering and true-eviction replacements after the construction replacement:

    text = replace_once(
        text,
        "        ret: list[KVCacheBlock] = self.free_block_queue.popleft_n(num_blocks)\n",
        "        self.compute_cache_policy.reorder_free_queue(\n"
        "            self.free_block_queue, num_blocks, self._reuse_watchers\n"
        "        )\n"
        "        ret: list[KVCacheBlock] = self.free_block_queue.popleft_n(num_blocks)\n",
        "victim reordering",
    )
    text = replace_once(
        text,
        "        if not evicted_hashes:\n"
        "            # The block doesn't have hash, eviction is not needed\n"
        "            return False\n\n"
        "        self._emit_block_removed_events(evicted_hashes)\n",
        "        if not evicted_hashes:\n"
        "            # The block doesn't have hash, eviction is not needed\n"
        "            return False\n\n"
        "        self.compute_cache_policy.on_evicted(block)\n"
        "        self._emit_block_removed_events(evicted_hashes)\n",
        "eviction reset",
    )

Why Place the Hooks Here?

• Reordering runs immediately before the existing free-queue pop.
• BlockPool still decides whether cached hashes require eviction.
• Policy metadata resets only when an actual cached hash is removed.

• Save tools/install_computecache.py.
• Confirm this chunk contains the labels victim reordering and eviction reset.

Is the eviction hook too early?

Confirm on_evicted appears after the no-eviction return. Placing it before that return would clear metadata for a block that kept its hash.

Use this prompt for placement help: help me verify the on_evicted hook follows true hash eviction

The final runtime hooks record real prefix hits and clear policy state with the existing cache reset. Both remain adjacent to vLLM's own metrics lifecycle.

• Add the access and cache-reset replacements after the eviction replacement:

    text = replace_once(
        text,
        "            if self.metrics_collector:\n"
        "                self.metrics_collector.on_block_accessed(block)\n",
        "            if self.metrics_collector:\n"
        "                self.metrics_collector.on_block_accessed(block)\n"
        "            self.compute_cache_policy.on_access(block)\n",
        "reuse access",
    )
    text = replace_once(
        text,
        "        if self.metrics_collector:\n"
        "            self.metrics_collector.reset()\n\n"
        "        logger.info(\"Successfully reset prefix cache\")\n",
        "        if self.metrics_collector:\n"
        "            self.metrics_collector.reset()\n"
        "        self.compute_cache_policy.reset()\n\n"
        "        logger.info(\"Successfully reset prefix cache\")\n",
        "cache reset",
    )

How Is Policy State Synchronized?

• Every real prefix-cache access reaches on_access after vLLM records its own metric.
• A full prefix-cache reset also resets the policy metadata.
• The installer does not replace either upstream lifecycle operation.

• Save tools/install_computecache.py.
• Confirm this chunk contains the labels reuse access and cache reset.

Are the lifecycle hooks outside main?

Keep both replace_once calls indented inside main. They must align with the earlier replacement calls.

Ask for help checking the installer structure: help me check the final two guarded replacements in main

The installer writes the transformed source only after every guarded replacement succeeds. This prevents a known anchor failure from leaving a partially edited file.

• Complete the installer by adding the write step and script entry point after the cache-reset replacement:

    TARGET.write_text(text, encoding="utf-8")
    print(f"Applied six guarded ComputeCache edits to {TARGET.relative_to(ROOT)}")


if __name__ == "__main__":
    main()

When Does the File Change?

The target write occurs after all six anchors pass. The final message confirms the number of edits and the modified source path.

• Save tools/install_computecache.py.
• Confirm the final line calls main() beneath the script-entry condition.

Does the installer do nothing?

Confirm the if __name__ condition starts at the left edge. Confirm main() is indented beneath it.

Use this prompt for a final structure check: help me verify the installer's script entry point

Compare the completed installer before allowing it to edit the pinned upstream source.

✔️ Awesome, I've got everything!

Excellent. Confirm tools/install_computecache.py is saved before running it.

ⓧ I'd like to double check the full code

#!/usr/bin/env python3
"""Apply guarded ComputeCache hooks to vLLM v0.32.0."""
from __future__ import annotations

import subprocess
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
TARGET = ROOT / "vllm/v1/core/block_pool.py"
EXPECTED_TAG = "v0.32.0"


def replace_once(text: str, old: str, new: str, label: str) -> str:
    count = text.count(old)
    if count != 1:
        raise RuntimeError(
            f"Refusing {label}: expected one verified anchor, found {count}"
        )
    return text.replace(old, new, 1)


def main() -> None:
    tag = subprocess.run(
        ["git", "describe", "--tags", "--exact-match"],
        cwd=ROOT,
        check=True,
        capture_output=True,
        text=True,
    ).stdout.strip()
    if tag != EXPECTED_TAG:
        raise RuntimeError(f"Expected {EXPECTED_TAG}, found {tag}")
    text = TARGET.read_text(encoding="utf-8")
    if "from vllm.v1.core.computecache_policy import ComputeCachePolicy" in text:
        print("ComputeCache hooks are already installed for v0.32.0")
        return
    text = replace_once(
        text,
        "from vllm.v1.core.kv_cache_metrics import KVCacheMetricsCollector\n",
        "from vllm.v1.core.computecache_policy import ComputeCachePolicy\n"
        "from vllm.v1.core.kv_cache_metrics import KVCacheMetricsCollector\n",
        "policy import",
    )
    text = replace_once(
        text,
        "        self.metrics_collector = metrics_collector\n"
        "        # Callbacks for blocks released with ``unpin_blocks`` whose contents\n",
        "        self.metrics_collector = metrics_collector\n"
        "        self.compute_cache_policy = ComputeCachePolicy.from_environ(\n"
        "            num_blocks=num_gpu_blocks, hash_block_size=hash_block_size\n"
        "        )\n"
        "        # Callbacks for blocks released with ``unpin_blocks`` whose contents\n",
        "policy construction",
    )
    text = replace_once(
        text,
        "        ret: list[KVCacheBlock] = self.free_block_queue.popleft_n(num_blocks)\n",
        "        self.compute_cache_policy.reorder_free_queue(\n"
        "            self.free_block_queue, num_blocks, self._reuse_watchers\n"
        "        )\n"
        "        ret: list[KVCacheBlock] = self.free_block_queue.popleft_n(num_blocks)\n",
        "victim reordering",
    )
    text = replace_once(
        text,
        "        if not evicted_hashes:\n"
        "            # The block doesn't have hash, eviction is not needed\n"
        "            return False\n\n"
        "        self._emit_block_removed_events(evicted_hashes)\n",
        "        if not evicted_hashes:\n"
        "            # The block doesn't have hash, eviction is not needed\n"
        "            return False\n\n"
        "        self.compute_cache_policy.on_evicted(block)\n"
        "        self._emit_block_removed_events(evicted_hashes)\n",
        "eviction reset",
    )
    text = replace_once(
        text,
        "            if self.metrics_collector:\n"
        "                self.metrics_collector.on_block_accessed(block)\n",
        "            if self.metrics_collector:\n"
        "                self.metrics_collector.on_block_accessed(block)\n"
        "            self.compute_cache_policy.on_access(block)\n",
        "reuse access",
    )
    text = replace_once(
        text,
        "        if self.metrics_collector:\n"
        "            self.metrics_collector.reset()\n\n"
        "        logger.info(\"Successfully reset prefix cache\")\n",
        "        if self.metrics_collector:\n"
        "            self.metrics_collector.reset()\n"
        "        self.compute_cache_policy.reset()\n\n"
        "        logger.info(\"Successfully reset prefix cache\")\n",
        "cache reset",
    )
    TARGET.write_text(text, encoding="utf-8")
    print(f"Applied six guarded ComputeCache edits to {TARGET.relative_to(ROOT)}")


if __name__ == "__main__":
    main()

This reference is the complete guarded installer. Every anchor and replacement must match exactly.

The installer is ready to test its six anchors against the pinned checkout. Before you run it, do you expect the guarded edit to complete or refuse the source layout?

• Apply the six guarded BlockPool edits by running:

python3 tools/install_computecache.py

What Does This Command Prove?

The script verifies the exact Git tag and all six source anchors before writing. A successful run proves this checkout matches the integration layout.

You should see Applied six guarded ComputeCache edits to vllm/v1/core/block_pool.py. The currently running LRU containers remain unchanged because you have not rebuilt or restarted them.

Did the guarded installer refuse?

A refusal protects the source from a guessed edit. Confirm the checkout still resolves to v0.32.0 and compare both new files with the full references above.

Ask for help with the reported anchor label: help me diagnose which guarded ComputeCache anchor did not match

Review the ownership boundary

The source diff is the final safety boundary. Git should show one new policy file and only six guarded additions inside BlockPool.

• Use your remote editor's version-control diff view to inspect vllm/v1/core/computecache_policy.py.
• Confirm reorder_free_queue collects candidates only from queue.iter_blocks_after(None).
• Confirm protected reuse-watcher block IDs are excluded before ranking.
• Use the same diff view to inspect vllm/v1/core/block_pool.py.
• Confirm the diff adds policy import and construction plus victim reordering and three lifecycle hooks.
• Confirm the diff leaves ref_cnt mutations and cached hash maps plus KV events and request block tables under BlockPool ownership.

Why Keep Ownership in BlockPool?

BlockPool already coordinates allocation and reference counts with cache hashes and KV events. ComputeCache only changes the order of free eviction candidates.

That narrow boundary preserves vLLM's existing correctness path while improving victim choice.

Before the final check, do you expect Git to report whitespace problems in either modified source file?

• Verify the source diff by running:

git diff --check

You should see no output and an exit code of 0. That confirms the policy file and six BlockPool edits contain no whitespace errors.

Did the diff check report a line?

Open the reported file and line in your remote editor. Remove only the flagged trailing whitespace or spacing error before rerunning the check.

Ask for help interpreting the result without changing policy logic: help me fix the whitespace issue reported by git diff --check

The bounded policy is now installed without disturbing the running LRU baseline. Next, you will prove its fallback and victim-selection safety before rebuilding the GPU server.

Prove Policy Safety

Your guarded vLLM integration now changes only the order of already-free cache blocks. The next risk sits below cache performance because an unsafe victim choice could disrupt serving state.

Before rebuilding the GPU server, deterministic tests must prove each safety boundary without requiring GPU access.

In this step, get ready to:

• Add deterministic policy tests with fake blocks and a fake free queue.
• Prove LRU fallback behavior plus bounded active-mode selection.
• Run static checks before committing the release candidate.

Add deterministic policy tests

Python's unittest framework can exercise the replacement policy with small in-memory objects. Direct module loading avoids importing the full vLLM package.

• In your remote editor's file explorer for the computecache-vllm checkout, create tests/v1/core/test_computecache_policy.py.
• Add the imports, direct module loader, and fake block model by pasting this first chunk:

#!/usr/bin/env python3
from __future__ import annotations

import importlib.util
import os
import sys
import unittest
from dataclasses import dataclass
from pathlib import Path
from unittest.mock import patch

MODULE_PATH = Path(__file__).resolve().parents[3] / "vllm/v1/core/computecache_policy.py"
SPEC = importlib.util.spec_from_file_location("computecache_policy_tested", MODULE_PATH)
assert SPEC and SPEC.loader
MODULE = importlib.util.module_from_spec(SPEC)
sys.modules[SPEC.name] = MODULE
SPEC.loader.exec_module(MODULE)
ComputeCachePolicy = MODULE.ComputeCachePolicy


@dataclass
class FakeBlock:
    block_id: int
    block_hash: bytes | None = None
    block_hash_num_tokens: int | None = None
    ref_cnt: int = 0
    is_null: bool = False

What does this test setup do?

• The direct loader imports only computecache_policy.py. This keeps the safety suite independent from a full vLLM runtime.
• The FakeBlock fields match the block attributes used by ComputeCachePolicy.
• The patch helper lets one test supply an invalid mode through the environment.

• Save tests/v1/core/test_computecache_policy.py.
• Confirm the file ends with the is_null field inside FakeBlock.

Does the module loader look broken?

Check that MODULE_PATH uses parents[3]. Confirm that the path ends with vllm/v1/core/computecache_policy.py.

Ask for help if the editor flags the direct import before the file is complete. Help me verify the direct module loader in this policy test file.

• Add the fake queue plus the first policy test below FakeBlock by pasting this chunk:

class FakeQueue:
    def __init__(self, blocks):
        self.blocks = list(blocks)

    def iter_blocks_after(self, cursor):
        start = 0 if cursor is None else self.blocks.index(cursor) + 1
        yield from self.blocks[start:]

    def remove(self, block):
        self.blocks.remove(block)

    def prepend_n(self, blocks):
        self.blocks[0:0] = blocks


class PolicyTest(unittest.TestCase):
    def policy(self, mode="active", scan_limit=64):
        return ComputeCachePolicy(16, 16, mode, scan_limit, 64, 1, 1, 4096, 256)

    def test_lru_noop(self):
        queue = FakeQueue([FakeBlock(i, b"cached", 16) for i in range(4)])
        self.policy("lru").reorder_free_queue(queue, 2, ())
        self.assertEqual([b.block_id for b in queue.blocks], [0, 1, 2, 3])

What does the first test prove?

• The FakeQueue implements the three queue operations used by the policy.
• The policy() helper creates a small fixed-size policy for every test.
• The test_lru_noop assertion proves that LRU mode preserves the original queue order.

• Save tests/v1/core/test_computecache_policy.py.
• Run the current test file through the unittest command-line runner:

python3 -m unittest tests/v1/core/test_computecache_policy.py

What should this first run prove?

You should see one passing test. This confirms that direct loading works without importing the full serving engine.

Did the first test fail?

Run the command from inside the computecache-vllm checkout. Check the spelling of computecache_policy.py if the module cannot load.

If the queue assertion fails, compare the four expected block IDs with the code above. Help me diagnose the first ComputeCache policy test.

• Add the quick-demotion and protected-block tests below test_lru_noop by pasting this chunk:

    def test_quick_demotion(self):
        hot, cold = FakeBlock(0, b"hot", 512), FakeBlock(1, b"cold", 512)
        queue, policy = FakeQueue([hot, cold]), self.policy()
        for _ in range(3):
            policy.on_access(hot)
        policy.reorder_free_queue(queue, 1, ())
        self.assertEqual(queue.blocks[0].block_id, cold.block_id)

    def test_protected_block(self):
        protected, cold = FakeBlock(0, b"protected", 16), FakeBlock(1, b"cold", 16)
        queue = FakeQueue([protected, cold])
        self.policy().reorder_free_queue(queue, 1, {protected.block_id})
        self.assertEqual(queue.blocks[0].block_id, cold.block_id)

What do these tests protect?

• The quick-demotion test records three accesses for the hot block. It confirms that the never-reused cold block moves to the victim position.
• The protected-block test passes the protected block ID to the policy. It confirms that the unprotected cold block becomes the victim.

• Save tests/v1/core/test_computecache_policy.py.
• Run the expanded suite through the unittest command-line runner:

python3 -m unittest tests/v1/core/test_computecache_policy.py

What should this run prove?

You should see three passing tests. The suite now covers fallback ordering plus two active-mode victim-selection rules.

Is quick demotion selecting the wrong block?

Check that policy.on_access(hot) remains inside the three-iteration loop. Confirm that the final assertion compares the first queue block with cold.block_id.

If the protected-block test fails, confirm that the protected ID is passed inside a set. Help me debug the quick-demotion or protected-block test.

• Complete the suite below test_protected_block by pasting the bounded-scan test, fallback test, and test runner:

    def test_scan_bound(self):
        first, second = FakeBlock(0, b"hot", 16), FakeBlock(1, b"cold", 16)
        queue, policy = FakeQueue([first, second]), self.policy(scan_limit=1)
        policy.on_access(first)
        policy.reorder_free_queue(queue, 1, ())
        self.assertEqual(queue.blocks[0].block_id, first.block_id)

    def test_invalid_mode_fails_safe(self):
        with patch.dict(os.environ, {"COMPUTECACHE_MODE": "invalid"}, clear=False):
            policy = ComputeCachePolicy.from_environ(16, 16)
        self.assertEqual(policy.mode, "lru")


if __name__ == "__main__":
    unittest.main(verbosity=2)

What do the final tests prove?

• The scan-bound test sets scan_limit=1. The colder second block stays outside the inspected window.
• The invalid-mode test supplies an unsupported value. The policy falls back to lru.
• The final two lines make the file directly executable with verbose test output.

• Save tests/v1/core/test_computecache_policy.py.

Before you run the complete suite, do you expect every safety boundary to pass without a GPU?

• Run all five deterministic tests directly:

python3 tests/v1/core/test_computecache_policy.py

What does the complete suite prove?

You should see five passing tests with exit code 0. The policy preserves LRU mode while enforcing bounded active-mode decisions.

• Confirm the output lists five successful tests.
• Confirm the terminal returns to its prompt without a failure summary.

Is one of the five tests failing?

Use the failing test name to locate the matching method. Compare that method with the relevant chunk above before changing the policy itself.

Pay close attention to indentation inside PolicyTest. Help me diagnose a failing ComputeCache safety test.

✔️ Awesome, I've got everything!

Great. Save tests/v1/core/test_computecache_policy.py before moving to the static checks.

ⓧ I'd like to double check the full code

#!/usr/bin/env python3
from __future__ import annotations

import importlib.util
import os
import sys
import unittest
from dataclasses import dataclass
from pathlib import Path
from unittest.mock import patch

MODULE_PATH = Path(__file__).resolve().parents[3] / "vllm/v1/core/computecache_policy.py"
SPEC = importlib.util.spec_from_file_location("computecache_policy_tested", MODULE_PATH)
assert SPEC and SPEC.loader
MODULE = importlib.util.module_from_spec(SPEC)
sys.modules[SPEC.name] = MODULE
SPEC.loader.exec_module(MODULE)
ComputeCachePolicy = MODULE.ComputeCachePolicy


@dataclass
class FakeBlock:
    block_id: int
    block_hash: bytes | None = None
    block_hash_num_tokens: int | None = None
    ref_cnt: int = 0
    is_null: bool = False


class FakeQueue:
    def __init__(self, blocks):
        self.blocks = list(blocks)

    def iter_blocks_after(self, cursor):
        start = 0 if cursor is None else self.blocks.index(cursor) + 1
        yield from self.blocks[start:]

    def remove(self, block):
        self.blocks.remove(block)

    def prepend_n(self, blocks):
        self.blocks[0:0] = blocks


class PolicyTest(unittest.TestCase):
    def policy(self, mode="active", scan_limit=64):
        return ComputeCachePolicy(16, 16, mode, scan_limit, 64, 1, 1, 4096, 256)

    def test_lru_noop(self):
        queue = FakeQueue([FakeBlock(i, b"cached", 16) for i in range(4)])
        self.policy("lru").reorder_free_queue(queue, 2, ())
        self.assertEqual([b.block_id for b in queue.blocks], [0, 1, 2, 3])

    def test_quick_demotion(self):
        hot, cold = FakeBlock(0, b"hot", 512), FakeBlock(1, b"cold", 512)
        queue, policy = FakeQueue([hot, cold]), self.policy()
        for _ in range(3):
            policy.on_access(hot)
        policy.reorder_free_queue(queue, 1, ())
        self.assertEqual(queue.blocks[0].block_id, cold.block_id)

    def test_protected_block(self):
        protected, cold = FakeBlock(0, b"protected", 16), FakeBlock(1, b"cold", 16)
        queue = FakeQueue([protected, cold])
        self.policy().reorder_free_queue(queue, 1, {protected.block_id})
        self.assertEqual(queue.blocks[0].block_id, cold.block_id)

    def test_scan_bound(self):
        first, second = FakeBlock(0, b"hot", 16), FakeBlock(1, b"cold", 16)
        queue, policy = FakeQueue([first, second]), self.policy(scan_limit=1)
        policy.on_access(first)
        policy.reorder_free_queue(queue, 1, ())
        self.assertEqual(queue.blocks[0].block_id, first.block_id)

    def test_invalid_mode_fails_safe(self):
        with patch.dict(os.environ, {"COMPUTECACHE_MODE": "invalid"}, clear=False):
            policy = ComputeCachePolicy.from_environ(16, 16)
        self.assertEqual(policy.mode, "lru")


if __name__ == "__main__":
    unittest.main(verbosity=2)

Run static safety checks

The behavioral tests cover policy decisions. Static checks catch syntax failures plus whitespace errors across the release candidate.

• Compile the policy, installer, and pressure workload by running these checks:

python3 -m py_compile vllm/v1/core/computecache_policy.py tools/install_computecache.py scripts/prefix_pressure.py
git diff --check

What do these checks cover?

• The first command asks Python to compile the three executable files. A successful result confirms that each file parses.
• The second command checks the pending Git diff for whitespace errors.

Both commands should return you to the prompt without a traceback or whitespace warning.

• Confirm that test_scan_bound uses scan_limit=1.
• Confirm that reorder_free_queue stops inspecting candidates at self.scan_limit in vllm/v1/core/computecache_policy.py.

Did a static check report a problem?

A compile failure identifies the affected file plus its line number. Compare that location with the corresponding Complete Code artifact.

A diff warning usually points to trailing whitespace or an extra blank line. Help me interpret a failed policy static check.

Commit the release candidate

A release candidate needs one commit that ties the policy to its integration, tests, workload, and monitoring configuration. The commit SHA becomes the source identifier for the next GPU build.

• Return to .env from earlier.
• Confirm the mode remains COMPUTECACHE_MODE=lru.
• Confirm the search bound remains COMPUTECACHE_SCAN_LIMIT=64.
• Confirm the reuse weight remains COMPUTECACHE_REUSE_WEIGHT=64.
• Confirm the depth weight remains COMPUTECACHE_DEPTH_WEIGHT=1.
• Confirm the age weight remains COMPUTECACHE_AGE_WEIGHT=1.
• Confirm the quick-demotion penalty remains COMPUTECACHE_QUICK_DEMOTE_PENALTY=4096.
• Confirm the logging interval remains COMPUTECACHE_LOG_EVERY=256.

• Stage the release-candidate files and create one commit by running:

git add vllm/v1/core/computecache_policy.py vllm/v1/core/block_pool.py tools/install_computecache.py tests/v1/core/test_computecache_policy.py scripts/prefix_pressure.py compose.yaml .env.example monitoring/prometheus.yml monitoring/grafana/provisioning/datasources/prometheus.yml
git commit -m "Add bounded ComputeCache release candidate"

What does this commit capture?

The staged paths connect policy behavior with the guarded BlockPool hooks. They also capture the deterministic tests plus the production canary configuration.

The private .env file stays outside the command. Its API key and Grafana password do not enter the commit.

You should see one new commit reported on the computecache/v0.32.0 branch.

• Print the release candidate's full source commit SHA by running:

git rev-parse HEAD

Why record the full commit SHA?

The SHA identifies the exact source snapshot used for the custom image. It lets later canary results point back to one reproducible policy revision.

• Record the printed value here: your source commit SHA.
• Record the custom image tag as computecache-vllm:0.32.0.

Before the final check, do you expect the committed candidate to preserve all five safety guarantees?

• Run the deterministic suite one final time:

python3 tests/v1/core/test_computecache_policy.py

What does the final check prove?

All five tests should pass with exit code 0. The committed candidate now has reproducible evidence for fallback, demotion, protection, bounded work, and fail-safe mode handling.

• Confirm the final output reports five passing tests.
• Confirm the command returns exit code 0.

Did the final run change after the commit?

A commit does not alter the file contents that the tests execute. Compare the failing test name with the earlier successful run to find any unsaved editor change.

Confirm that your terminal remains inside the computecache-vllm checkout. Help me diagnose why the committed safety suite now fails.

Strong work. Your release candidate now has five passing safety tests plus clean static checks before it touches the real GPU server.

Your bounded policy is committed with reproducible safety evidence. Next up, you will rebuild the custom image and run ComputeCache against real GPU KV blocks.

Run the Canary and Rollback

Your five deterministic tests proved the bounded policy's safety before it touched the remote GPU. Now the vLLM release candidate must prove itself against the real cache.

Production readiness requires active eviction on real GPU blocks. It also requires observable results plus a rehearsed LRU recovery path.

In this step, get ready to:

• Rebuild the custom image in active mode and run the pressure workload.
• Compare prefix retention plus TTFT against the saved LRU baseline.
• Rehearse the LRU rollback and choose the accepted final mode.

Activate the ComputeCache canary

Active mode applies ComputeCache ordering to already-free eviction candidates. The existing Docker Compose stack keeps allocation ownership inside BlockPool.

• In .env, replace COMPUTECACHE_MODE=lru with COMPUTECACHE_MODE=active.
• Save .env.

The image rebuild can take a while because it packages the modified vLLM source. A quiet terminal during part of the build does not mean the process has stalled.

• Rebuild the custom vLLM image and replace the running service by running:

docker compose up -d --build vllm

What does this command do?

Docker Compose rebuilds computecache-vllm:0.32.0 from the checked-out source. It replaces only the vLLM service while Prometheus plus Grafana remain available.

The rebuilt container now includes the guarded BlockPool integration. Its startup log provides the first proof that active mode reached the real server.

Before you inspect the logs, which mode plus scan limit do you expect the policy to report?

• Inspect the vLLM startup log by running:

docker compose logs vllm

What should the log confirm?

The policy logs its selected mode during construction. It also reports the bounded value loaded from COMPUTECACHE_SCAN_LIMIT.

You should see active mode with a scan limit of 64. This proves the rebuilt server loaded the canary configuration.

Still seeing LRU mode?

Confirm .env contains COMPUTECACHE_MODE=active. Save the file before rebuilding the service again.

Check that the log belongs to the newly replaced vLLM container. Old log entries can still appear earlier in the output.

help me diagnose why my rebuilt vLLM container still reports LRU mode

• Export the values from .env into your current shell with the same command used for the LRU baseline.

Before you rerun the pressure workload, do you expect active mode to retain more of the warmed prefix after the flood?

• Run the real GPU pressure workload by running:

python3 scripts/prefix_pressure.py

What does the result measure?

• The warm_prefix_cached_tokens value measures cached tokens before one-use prompt pressure.
• The post_flood_cached_tokens value measures retained tokens after the flood.
• The flood_requests value records how much pressure the run generated.
• The hot_prefix_preserved value summarizes whether the warmed prefix survived.

You should receive a JSON result with mode set to active. Save this result beside the LRU baseline for the acceptance comparison.

Did the active workload fail?

Confirm the vLLM service finished starting before you retry the workload. The model must be available before the first chat request can succeed.

Confirm your current shell received the same VLLM_API_KEY value used by the container. Do not paste the key into your troubleshooting notes.

help me diagnose why scripts/prefix_pressure.py fails against my active vLLM container

Inspect the canary evidence

Prometheus stores the cache plus latency measurements emitted by the real server. Grafana Explore lets you place those signals beside the workload result.

• Return to Grafana Explore through the loopback SSH forwarding route used for the LRU baseline.
• Select the Prometheus data source.
• Add a query row for vllm:prefix_cache_hits.
• Add a query row for vllm:prefix_cache_queries.
• Add a query row for vllm:kv_cache_usage_perc.
• Add a query row for vllm:time_to_first_token_seconds.

You should see cache activity during the pressure run plus TTFT samples from the same window. These measurements connect the retention result to serving performance.

A canary earns promotion through evidence from the same workload. Prefix retention only counts as a win when requests remain healthy plus TTFT stays within the canary budget.

• Compare the active JSON result with your saved LRU JSON result.
• Compare cache hits plus cache queries across the matching workload windows.
• Compare KV usage across the matching workload windows.
• Compare TTFT across the matching workload windows.
• Reject active mode if any workload request failed.
• Reject active mode if it preserved no more hot-prefix cached tokens than LRU.
• Reject active mode if TTFT clearly regressed under the same workload.
• Accept active mode only when all three acceptance checks pass.

• Confirm the existing host bindings in compose.yaml remain set to 127.0.0.1.

Why keep the canary on loopback?

The vLLM API key does not protect every endpoint. Loopback bindings keep the canary behind the existing SSH forwarding route instead of exposing it directly to the Internet.

Rehearse the LRU rollback

Rollback changes one setting before restarting the existing image. It does not revert code or rebuild the release candidate.

• In .env, replace COMPUTECACHE_MODE=active with COMPUTECACHE_MODE=lru.
• Save .env.

• Restart vLLM without rebuilding the image by running:

docker compose up -d vllm

What does this rollback do?

Docker Compose recreates the vLLM service with LRU mode from .env. The custom image plus all guarded integration code remain unchanged.

• Export the updated values from .env into your current shell with the same command used earlier.

Before you verify the rollback, do you expect the same image to serve requests successfully with ComputeCache ordering disabled?

• Repeat the authenticated /v1/models health check from your LRU baseline.

You should receive a successful response listing Qwen/Qwen3-0.6B. This proves the server recovered without an image rebuild.

• Verify authenticated chat completions in rollback mode by running:

python3 scripts/prefix_pressure.py

What does this final run prove?

The workload sends authenticated requests to POST /v1/chat/completions through the restarted server. A completed JSON result proves the LRU path still serves real GPU requests.

You should receive another JSON result with mode set to lru. That is the rollback path exercised from configuration change to successful request.

Did rollback fail to serve requests?

Check the vLLM logs for the selected mode. The startup entry should report LRU before you test either endpoint.

Confirm the current shell was refreshed from .env. A stale shell can make the workload report the previous mode.

help me diagnose why my LRU rollback restarted but authenticated vLLM requests still fail

The rollback drill is complete. Your acceptance decision now determines which mode should remain running.

✔️ Active mode passed acceptance

• In .env, replace COMPUTECACHE_MODE=lru with COMPUTECACHE_MODE=active.
• Save .env.
• Restore the accepted active canary by running:

docker compose up -d vllm

Why is no rebuild needed?

The existing custom image contains both active plus LRU behavior. Restart-time configuration selects the policy mode.

• Export the updated .env values with the same shell command used earlier.
• Repeat the authenticated /v1/models health check.

You should see Qwen/Qwen3-0.6B in the response. The accepted active canary is now serving from the loopback-only stack.

ⓧ Active mode failed acceptance

• Keep COMPUTECACHE_MODE=lru in .env.
• Repeat the authenticated /v1/models health check.

You should see Qwen/Qwen3-0.6B in the response. LRU remains the safe final mode while you investigate the rejected canary evidence.

That recovery path is now proven on the real server. Finish by capturing the exact release candidate that remains deployed.

• Record the model as Qwen/Qwen3-0.6B in the checkpoint below.
• Record the GPU identity from your earlier passthrough validation in the checkpoint below.
• Record the image as computecache-vllm:0.32.0 in the checkpoint below.
• Record the source commit from the release-candidate commit in the checkpoint below.
• Record the accepted final mode in the checkpoint below.
• Record the scan limit of 64 in the checkpoint below.
• Record the baseline-versus-active comparison in the checkpoint below.

Your release candidate has now survived an active canary plus a real rollback drill. You have evidence for its cache behavior and a working recovery path.

Secret mission

Tune the Scan Budget Without Breaking the SLO

Find the smallest victim-search window that protects the warmed prefix within your p95 TTFT budget. Compare clean real-GPU runs at scan limits 16, 32, and 64 before repeating the LRU rollback drill.

Clean Up Your Resources

Clean Up Your Resources

Choose whether to keep the canary running, pause it, or delete its resources. A metered remote GPU still incurs its provider's normal running cost.

Cost warning

The project software is free.

Stopping Compose releases the GPU process. Provider billing may continue while the remote server runs.

• Stop or terminate the remote GPU instance through your provider's instance controls when you no longer need it.

Resources you used:

• The computecache-vllm source checkout containing the custom vLLM policy, tests, workload, monitoring configuration, Git history, and tuning evidence.
• The computecache-vllm:0.32.0 custom image stored by Docker Engine.
• The Docker Compose services named vllm, prometheus, and grafana, plus their private network.
• The hf-cache, vllm-cache, prometheus-data, and grafana-data volumes containing model data, compile data, Prometheus history, and Grafana state.

Keep everything running

No action is needed. Choose this if you want to continue observing the accepted canary or extend the release candidate.

• Keep the vllm, prometheus, and grafana services bound to 127.0.0.1.
• Retain .env with the accepted COMPUTECACHE_MODE and selected COMPUTECACHE_SCAN_LIMIT.
• Preserve the LRU baseline, active canary result, scan-budget record, and rollback evidence.
• Monitor your provider usage while the remote GPU instance remains running.

Pause - I'll come back to this later

Shut down the Compose processes to release the GPU workload. The custom image, named volumes, and source checkout remain available.

• Return to the computecache-vllm checkout in the remote Linux terminal from earlier.
• Stop the three Compose services by running this command:

docker compose stop

What does this command do?

This command stops the three service containers without removing them or their volumes. The source checkout and custom image also remain on the remote server.

• Confirm the terminal reports that all three services stopped.

Compose cannot stop the services?

• Confirm that your remote terminal is inside the computecache-vllm checkout containing compose.yaml.
• Help me diagnose why my Compose services will not stop on the remote Linux server.

Delete - I don't want to use this again

Deleting the Compose volumes erases the model cache and monitoring history. Deleting the checkout permanently removes any canary evidence stored inside that folder.

• Return to the computecache-vllm checkout in the remote Linux terminal from earlier.
• Remove the service containers, private network, named volumes, and orphaned containers by running this command:

docker compose down -v --remove-orphans

What does this command remove?

This command removes the three service containers, the private Compose network, and all four named volumes. It preserves the source checkout and the custom image.

• Confirm Docker reports that the containers, network, and volumes were removed.

Some Compose resources remain?

• Confirm that you ran the command from the computecache-vllm checkout containing compose.yaml.
• Help me find why my Compose containers or volumes remain after cleanup.

• Delete the local image named computecache-vllm:0.32.0 through Docker Engine's documented image-removal workflow.
• Confirm the custom image no longer appears in the local image list.

Removing the checkout is permanent. The recorded Git commit remains recoverable only from another repository that contains it.

• Move to the checkout's parent directory and delete computecache-vllm by running these commands:

cd ..
rm -rf computecache-vllm

What did these commands remove?

The first command moves your terminal to the checkout's parent directory. The second command permanently deletes the computecache-vllm folder.

• Confirm that computecache-vllm no longer appears in the current directory.

Checkout still present?

• Confirm that your terminal is in the directory that previously contained computecache-vllm.
• Help me safely remove the remaining ComputeCache checkout from my remote Linux server.

• Check that no other workload needs the remote GPU instance.
• Stop or terminate the metered GPU instance through your provider's instance controls.
• Confirm your provider marks the instance as stopped or terminated.

Nice Work!

Nice Work!

Strong finish! You built a production-aware release candidate on real GPU-backed vLLM. ComputeCache now applies bounded replacement to real KV blocks with a rehearsed recovery path.

You've learned how to:

• Build a version-pinned vLLM fork that reorders a bounded window of already-free GPU KV blocks while preserving BlockPool ownership.
• Reproduce LRU scan pollution with a real pressure workload. Prove safe fallback plus bounded work with five deterministic tests.
• Observe production cache behavior through Prometheus plus Grafana. Rehearse an LRU rollback without rebuilding the custom image.
• Secret Mission: Tune the victim-search window at 16, 32, plus 64. Select the smallest limit that protects the hot prefix within the p95 TTFT budget.

Ready to quiz yourself?