Measure What Concise Drops

Introduction

30 Second Summary

When someone gives you a shorter answer, the first thing you wonder is whether they left something important out. The same instinct kicks in when a tool promises to cut the fluff.

In this project, you will run six prompts under two output styles in Claude Code and build a scorecard that shows whether the shorter style dropped anything you needed.

What You'll Build

You open your scorecard and see twelve runs scored against the same fixed checklist. One glance at the totals tells you whether the shorter style cost you anything or whether the savings were free.

By the end of this project, you'll have:

• A completed scorecard showing line counts and required-items hits for six prompts run under both Default and Concise output styles.
• A data-backed answer to whether shorter answers cost you required information or whether the savings were free.
• A reusable test set of six prompts with locked required items you can point at any future model or setting change.
• Secret Mission: Write a custom output style that keeps what Concise drops while still cutting preamble, and re-test it against both Default and Concise.

Are there any prerequisites?

You need Claude Code v2.1.237 or later, installed and authenticated on a paid plan (Pro, Max, Team, Enterprise, or Console). No Python runtime is needed.

Before We Start

Before you run any prompts or record any numbers, you need to name exactly what you are measuring and what result would change your mind.

Writing this down first keeps the rest of the project honest.

Write the Prompts and Lock the Required Items

You've pinned down what you're measuring and what evidence would change your mind. Now it's time to set up the test inputs: a codebase for Claude Code to read, and a scorecard that defines what counts as a correct answer.

The scorecard is the critical piece. Writing down what a correct answer must contain before you see any output is what separates measurement from preference. If you decide what matters after reading the response, you compare to your impression, not to a standard.

In this step, get ready to:

• Set up a project folder with code for Claude Code to read.
• Write a scorecard with six prompts and three required items per prompt, locked before any runs.

Confirm your Claude Code version

The Concise output style does not exist before v2.1.237. Before going further, confirm your version is current.

• Check your Claude Code version by running this command in your terminal:

claude --version

You should see a version number of 2.1.237 or higher.

Seeing an older version?

Update Claude Code by running:

claude update

Then run claude --version again to confirm.

Help me update Claude Code to the latest version.

Create the test codebase and open Claude Code

Claude Code needs files to read so your prompts have something to target. You will create three small Python files that contain real bugs, missing validation, and design issues.

Already have a project open?

If you already have Claude Code open in a project you're working on, skip ahead to the scorecard substep. Write your own six prompts targeting your own codebase instead of using the provided ones. The method is the same.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create a project folder and move into it by running these commands:

mkdir concise-test
cd concise-test

Now create three Python files inside this folder. Open concise-test in your preferred text editor and create each file with the code below. Claude Code reads these files as text. No Python runtime is needed.

• Create a file called inventory.py and paste the following code:

_stock = {}


def add_product(sku, name, quantity):
    _stock[sku] = {"name": name, "qty": quantity}


def adjust_stock(sku, quantity):
    if sku not in _stock:
        raise KeyError(f"Unknown SKU: {sku}")
    _stock[sku]["qty"] += quantity


def remove_product(sku):
    del _stock[sku]


def get_low_stock(threshold=10):
    return [item for item in _stock.values() if item["qty"] < threshold]


def snapshot():
    return dict(_stock)

What does inventory.py do?

This file manages product stock using a module-level dictionary. It has functions to add products, adjust quantities, remove items, and find low-stock products. Several of these have deliberate gaps your prompts will probe.

• Create a file called api_client.py and paste the following code:

import urllib.request
import json
import time

_BASE = "https://api.pricing-service.internal/v2"


def fetch_prices(skus):
    out = {}
    for sku in skus:
        url = f"{_BASE}/prices/{sku}"
        for attempt in range(3):
            try:
                resp = urllib.request.urlopen(url, timeout=30)
                data = json.loads(resp.read())
                out[sku] = data["unit_price"]
                break
            except Exception:
                time.sleep(2 ** attempt)
    return out


def post_order(payload):
    req = urllib.request.Request(
        f"{_BASE}/orders",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
    )
    try:
        resp = urllib.request.urlopen(req)
        return json.loads(resp.read())
    except Exception:
        return None

What does api_client.py do?

This file wraps HTTP calls to an internal pricing API. The fetch_prices function retries on failure, and post_order sends an order payload. Both have error-handling issues your prompts will target.

• Create a file called report.py and paste the following code:

from inventory import snapshot, get_low_stock
from api_client import fetch_prices


def _xf(val, rate=1.0):
    if val is None:
        return "N/A"
    return f"${val * rate:,.2f}"


def generate_report(include_prices=True):
    items = snapshot()
    low = get_low_stock()
    skus = list(items.keys())

    prices = {}
    if include_prices:
        prices = fetch_prices(skus)

    lines = []
    lines.append("INVENTORY REPORT")
    lines.append("=" * 40)

    for sku, info in items.items():
        is_low = any(l["name"] == info["name"] for l in low)
        price = prices.get(sku)
        line = f"{sku}: {info['name']} | qty={info['qty']} | price={_xf(price)}"
        if is_low:
            line += " [LOW]"
        lines.append(line)

    lines.append(f"\nTotal SKUs: {len(items)}")
    lines.append(f"Low stock items: {len(low)}")
    return "\n".join(lines)

What does report.py do?

This file pulls data from both other modules to generate an inventory report. It has a cryptically named helper function and a performance issue hiding inside its main loop. Two of your six prompts will target this file specifically.

• Confirm all three files exist by running:

ls

You should see api_client.py, inventory.py, and report.py listed in the output.

Missing a file?

Make sure you saved each file inside the concise-test folder, not its parent directory. Check that each file name ends in .py with no extra extension like .py.txt.

Help me check why my Python files are not showing up in my project folder.

• Start Claude Code inside this folder by running:

claude

Claude Code starts and reads all three files in the folder. You should see the Claude Code prompt ready for input.

Does Claude Code run the Python files?

No. Claude Code reads the files as text to understand the codebase. It does not execute them. You do not need Python installed on your machine for this project.

Write the scorecard and lock the required items

A required item is something a correct answer must contain. A file path. A number. A caveat. A specific line of code. A refusal where the right answer is that something cannot be done safely.

You write these down now, before you run anything. You do not change them after this step. Writing them afterwards means you decide in favour of whichever answer you enjoyed reading.

• Create a file called scorecard.md in your concise-test folder and paste this first section, which contains the six prompts and their required items:

# Concise vs Default Scorecard

## Prompts and Required Items

Decide what a correct answer MUST contain before you run anything.
Deciding afterwards is not measuring. It is preferring.

| #  | Prompt (paste into Claude Code)                                                                  | Required A                                           | Required B                                          | Required C                                             |
|----|--------------------------------------------------------------------------------------------------|------------------------------------------------------|-----------------------------------------------------|--------------------------------------------------------|
| 1  | the retry logic in fetch_prices retries on every failure - what should it not retry on           | HTTP 4xx errors are not transient                    | the bare except Exception catches everything        | after 3 failures the function silently moves on        |
| 2  | what happens if adjust_stock is called with quantity=-100 on a product that has qty=5            | qty becomes -5                                       | no validation prevents this                         | the line _stock[sku]["qty"] += quantity                 |
| 3  | what does _xf do and what should it be called                                                    | formats a number as USD currency                     | a better name like format_price or format_currency  | handles None by returning N/A                          |
| 4  | trace what happens when the pricing API is unreachable and generate_report runs with prices=True | fetch_prices silently returns partial or empty dict  | _xf shows N/A for missing prices                    | no error is raised or logged anywhere                  |
| 5  | can two threads call adjust_stock at the same time without corrupting the inventory              | yes there is a race condition                        | _stock is module-level mutable state                | needs a lock or other synchronisation                  |
| 6  | is there a performance problem in generate_report                                                | the any() inside the for loop is O(n*m)              | should pre-compute a set of low-stock SKUs          | the specific line with the nested check                |

Why these six prompts?

Each prompt targets a different kind of question: error handling, missing validation, naming, cross-file tracing, concurrency, and performance. Phrasing them as natural work questions rather than clean test cases is deliberate. Clean prompts compress identically under both styles, so the comparison would show nothing.

If you are using your own codebase, write six prompts that cover a similar spread of question types. The variety is what makes the test useful.

• Below the prompts table in scorecard.md, paste the results template:

## Results

| #  | Default lines | Default items (of 3) | Concise lines | Concise items (of 3) |
|----|---------------|----------------------|---------------|----------------------|
| 1  |               |                      |               |                      |
| 2  |               |                      |               |                      |
| 3  |               |                      |               |                      |
| 4  |               |                      |               |                      |
| 5  |               |                      |               |                      |
| 6  |               |                      |               |                      |

## Totals

|                    | Default | Concise |
|--------------------|---------|--------|
| Total lines        |         |         |
| Total items hit    |         |         |
| Total items missed |         |         |

## Findings

Write what the numbers say. Not what you thought would happen.

- Did Concise drop required items? Which ones?
- Did Default also miss items? Which ones?
- If both missed the same items, the style was not the cause.

What is the results template for?

The Results table has two columns per style: line count and required-items count. You will fill these in during the next two steps. The Totals section will give you the numbers you need to compare the two styles. The Findings section is where you write what the data shows.

• Save scorecard.md.
• Confirm the file exists alongside your Python files by running this in a separate terminal window (outside Claude Code):

ls

You should see four files: api_client.py, inventory.py, report.py, and scorecard.md.

✔️ Awesome, I've got everything!

Great. Double check you've saved scorecard.md and confirm it has six prompt rows in the Prompts and Required Items table with three required items each. The Results and Totals tables should still be empty.

ⓧ I'd like to double check the full code

Your scorecard.md was built across two chunks above. Scroll up and confirm both sections are present in order:

• Chunk 1: The heading, the two instruction lines, and the Prompts and Required Items table with six rows.
• Chunk 2: The Results table (six empty rows), the Totals table (three empty rows), and the Findings section with three bullet prompts.

Why lock the required items now?

The required-items list is the weakest joint in the whole measurement. If you read the Default answer first and then decide what matters, you compare to your impression of that answer. If both styles miss the same item, you would never notice because you never expected it.

Pre-committing makes the comparison honest. A hit is a hit. A miss is a miss. The scorecard does not know which style it is grading.

Take one last look at your scorecard. It has six rows. Each row has a prompt and three required items. The results columns are empty.

Your test is loaded. The prompts are ready and the required items are locked. Next up, you'll run all six prompts under the Default output style and fill in the first column of results.

Run All Six Under Default

Your scorecard has six prompts with required items locked in place. Default is the output style Claude Code ships with, and it is your baseline.

You will run all six prompts under Default and record two numbers for each answer: how many lines, and how many of your three required items it contains. These baseline numbers are what you compare against when you switch styles.

In this step, get ready to:

• Confirm Output style is set to Default.
• Run all six prompts and record line counts and required-items hits.
• Total the Default columns.

Confirm the output style is Default

Output style is part of the system prompt. Claude Code reads the system prompt once when a session starts, so changing a style mid-session does not take effect until you restart.

• In Claude Code, check your current output style by running:

/config

• Look at the Output style row.

✔️ It already says Default

You are on the right style. Move on to the next substep.

ⓧ It shows something else

• Select Output style and pick Default.
• Restart the session so the new style loads by running:

/clear

Claude Code restarts the session. Your files are untouched, but the conversation history resets and the new output style takes effect.

Why /clear after changing the style?

Output style is baked into the system prompt, which Claude Code reads once at session start. Changing the setting in /config saves your preference, but the running session keeps using the old prompt until you /clear or start a new session.

Run each prompt and record the results

Each prompt gets two numbers. A line count measures how long the answer is. A required-items count measures whether the answer contains what you said it must.

• Switch to your editor and open scorecard.md so you can fill in the Results table as you go.
• Copy the first prompt from the Prompts and Required Items table in your scorecard.
• Switch back to Claude Code and paste the prompt.
• When Claude Code finishes answering, count the number of lines in the response.
• Write this number in the Default lines column for row 1.
• Check each of your three required items against the actual text of the answer. A required item counts as present only if the answer states that information directly.
• Write how many are present (0, 1, 2, or 3) in the Default items (of 3) column.

Counting lines quickly

Select Claude Code's entire answer and paste it into a scratch file in your editor. Your editor's line count at the bottom of the window tells you the number without counting manually.

• Repeat this process for prompts 2 through 6. Copy each prompt from the scorecard, paste it into Claude Code, count lines, check required items, and fill in the row.

What counts as a hit?

A required item is present if the answer states the same information, even in different words. It is absent if the answer skips it entirely or only vaguely gestures at it.

For example, if your required item is "no validation prevents negative quantity" and the answer says "there is no guard against qty going below zero", that is a hit. If the answer only says "the function modifies qty" without mentioning the missing guard, that is a miss.

Getting different answers on a second try?

Claude Code's answers vary between runs. If you accidentally re-ran a prompt, use the most recent answer. Small variation between runs is normal. This is exactly why a small gap in the final comparison is noise rather than a finding.

Help me understand why my Claude Code answers are different each time.

Total the Default columns

• Add up the six numbers in the Default lines column. Write the sum in the Totals row.
• Add up the six numbers in the Default items (of 3) column. Write that sum in the Totals row.
• Save scorecard.md.

The maximum possible items total is 18 (six prompts times three items each). Before you look at your total, predict: did Default hit all 18, or did it miss some?

Check your scorecard. The Default column should now have twelve numbers filled in: six line counts and six item counts. The Totals row should have a sum for each column.

What do these totals tell you?

The total items count is your baseline. If Default hit all 18, every answer contained everything you said it must. If Default hit fewer, some required items were missing before you ever changed styles.

That matters in the next step. Any item Default itself misses cannot be blamed on Concise.

Your Default baseline is recorded. Next, you will switch to Concise and run the exact same six prompts to find out what, if anything, the shorter style drops.

Switch to Concise and Run the Same Six

Your Default baseline is complete. Six prompts, twelve numbers, and column totals that show what full-length answers look like.

Now you run the same six prompts under Concise. Each answer is measured against the required items you locked in Step 1, the same fixed target you used for Default.

In this step, get ready to:

• Switch Claude Code's output style from Default to Concise.
• Run all six prompts and record line counts and required-items hits.
• Total the Concise columns to complete the scorecard.

Switch to the Concise output style

• Type /config in your Claude Code session.
• Select Output style from the menu.
• Pick Concise from the list of styles.
• Type /clear to restart the session with the new style applied.

Why do I need to run /clear?

Output style is part of the system prompt. Claude Code loads the system prompt once when a session starts. Changing the style mid-session does not take effect until you run /clear or start a new session.

• Run /config again to confirm the switch.
• You should see Output style set to Concise.

Output style did not change?

• Make sure you selected Concise in the Output style menu, not another style.
• Run /clear after changing the setting. The new style only loads after the session restarts.
• My Claude Code output style still shows Default after running /config and /clear.

Run the six prompts under Concise

You already know the process from the Default run. Same prompts, same order, same two numbers per answer.

• Run each of the six prompts one at a time, in the same order as your Default run.
• After each answer, count the number of lines in the response.
• Check each of the three required items against the actual text of the answer. Mark each item as a hit or a miss.
• Record both numbers in the Concise column of scorecard.md.

What if the answer looks completely different?

Two runs of the same prompt can produce different text. That is expected. You are checking whether each required item appears in the answer, not whether the wording matches the Default answer.

You have all six Concise answers scored. Before you add up the totals, predict: did Concise hit the same total number of required items as Default, or fewer?

Total the Concise columns

• Add up the six line counts in the Concise column and write the total in the Total lines row.
• Add up the six required-items counts. Write the total in Total items hit, and record 18 minus that number in Total items missed.
• Save scorecard.md.

Your scorecard now has two complete columns. Each column has six line counts, six required-items counts, and a row of totals.

Numbers look off?

• Double-check that each required-items count is out of 3, not out of 6. Each prompt has exactly three required items.
• Make sure you filled in the Concise column, not overwritten the Default column. Each column should have its own set of numbers.
• My scorecard totals do not add up correctly across the two columns.

Both columns are filled in. Next, you will read what the numbers say about whether Concise drops required information or only preamble.

Read the Scorecard

You've run all six prompts under both Default and Concise. Your scorecard has two complete columns of data.

Numbers beat impressions. Without the scorecard, you'd pick whichever style "felt" better. With it, you can see exactly which required items each style hit and which it missed.

In this step, get ready to:

• Compare total line counts between Default and Concise.
• Compare total required-items counts to check if shorter meant incomplete.
• Write your findings in scorecard.md.

Compare the total line counts

The first question is whether Concise actually produced shorter answers. The line count totals answer this directly.

• Look at the Totals row at the bottom of your scorecard.
• Compare the Default total lines number to the Concise total lines number.

What do the line counts tell you?

If the Concise total is lower, the style compressed the answers as described. This is the expected outcome.

If the totals are close or Concise is higher, your prompts may have been too specific to compress. Short questions with short correct answers leave little preamble to cut.

Compare the required-items counts

Shorter answers are only better if they keep the information you need. The required-items counts tell you whether shorter was free or whether it cost you detail.

Before you look at the numbers, take a guess: do you think Concise missed any of your required items?

• Compare the Default total items hit to the Concise total items hit.
• If the totals are equal, Concise cut only preamble. Shorter was free.
• If Concise hit fewer items, scroll through each prompt row and find which prompts lost items.
• For each prompt that lost items, identify which specific required item is missing from the Concise answer.

Why identify the specific items?

The specific items that went missing tell you what categories of information the style trades away. File paths? Caveats? Edge cases? That pattern is more useful than a total count alone.

Check whether Default missed items and write your findings

Before you draw conclusions about Concise, check whether Default itself was perfect. If Default also missed items, the comparison shifts.

• Check whether the Default total items hit is below 18 (6 prompts × 3 items each).
• If Default missed items, check whether they are the same items Concise missed on the same prompts.

What if both styles missed the same items?

If Default and Concise both missed the same required items on the same prompts, the output style was not the cause. You found a gap in the model, not the setting.

That finding is more useful than a style comparison. It tells you what the model itself struggles with, regardless of how it formats the answer.

Now write what the numbers say. Your scorecard template has three questions in the Findings section waiting for your answers.

• Switch back to scorecard.md in your editor and scroll to the Findings section at the bottom.
• Answer the first question: did Concise drop required items, and if so, which ones?
• Answer the second question: did Default also miss items, and if so, which ones?
• Answer the third question: if both missed the same items, name that the style was not the cause.
• Save scorecard.md.

Write what the numbers say, not what you hoped

The Findings section is for observations, not preferences. If Concise missed nothing, say that. If it missed items, name them. If Default was also imperfect, that is the most useful finding of all.

Your scorecard should now have totals for both columns and a Findings section with specific answers. You can name exactly which required items each style hit or missed.

Not sure if an item was hit or missed?

Go back to the original answer in your Claude Code session and search for the specific required item. If the information is there in any form, it counts as a hit. If the concept is completely absent, it counts as a miss.

Help me decide whether a required item was present in the answer.

Secret mission

Write a Custom Style That Keeps What Concise Drops

Your scorecard shows exactly what Concise cuts. Custom output styles let you write your own instructions for how Claude Code responds. Create one that targets what Concise drops while still cutting preamble, then test it against two of your prompts.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally using your existing Claude Code subscription, so there are no ongoing costs. Decide whether to keep your project files for future testing, or delete them entirely.

Resources you used:

• Project files: inventory.py, api_client.py, report.py, and scorecard.md.
• Custom output style: .claude/output-styles/concise-complete.md (Secret Mission).
• Output style setting saved in .claude/settings.local.json inside your project folder.

Keep everything running

No action needed. Choose this if you plan to reuse the scorecard or add more prompts to it later.

• Your project folder stays on disk with all Python files, the scorecard, and the custom output style.
• Your Claude Code output style is currently set to your custom style for this project. It only applies when you open Claude Code inside this folder.
• To switch back to the built-in Default style at any time, run /config inside Claude Code, select Output style, and pick Default.

Pause - I'll come back to this later

Exit your Claude Code session to free up the terminal. Your files stay on disk for when you return.

• Type /exit in Claude Code to end the session.
• When you come back, open a terminal in your project folder and run claude to start a new session. Your files and custom output style will still be there.

Delete - I don't want to use this again

Remove all project files and reset your output style. This clears everything you created in this project.

• In Claude Code, run /config, select Output style, and pick Default. This resets the output style for this project folder.
• Type /exit to end the Claude Code session.
• If you created a dedicated folder for this project, delete it. On macOS or Linux, run the command below from your Desktop (replace the folder name with yours):

cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]

• On Windows, run:

cd ~\Desktop
rmdir /s /q [[PROJECT_FOLDER="your project folder name"]]

• If you used an existing project folder and only want to remove the files from this project, delete them individually instead:

rm inventory.py api_client.py report.py scorecard.md
rm -rf .claude/output-styles

What about the .claude folder?

The .claude/ folder inside your project stores project-level Claude Code settings, including your output style choice and the custom style file. Deleting the project folder removes it automatically. If you kept the project folder but removed .claude/output-styles with the command above, Claude Code falls back to the built-in styles on the next session.

Nice Work!

Nice Work!

You just built a measurement system for something most people evaluate by gut feel. You have a scorecard with real numbers that tells you whether Concise drops required information or only preamble.

You've learned how to:

• Pre-commit required items before running a single prompt, turning evaluation from preference into measurement.
• Run a controlled side-by-side comparison of two output styles by recording line counts and required-items hits for every prompt.
• Read a completed scorecard and state, from the numbers, exactly which categories of information each style keeps or drops.
• Secret Mission: Wrote a custom output style targeting the specific categories the scorecard revealed, and tested it against both Default and Concise.

Ready to quiz yourself?