Build a Claude AI Evaluation Report

Test Claude's capabilities and build a professional AI evaluation report.

Introduction

30 Second Summary

Everyone talks about using AI at work, but few people have actually tested where it fails. Knowing the difference between "Claude can help with this" and "Claude will confidently make this up" is the skill that separates effective AI users from everyone else.

In this project, you will build a professional Claude Evaluation Report by deliberately testing Claude's strengths and weaknesses yourself. You'll document prompt engineering techniques, catch real hallucinations, design a responsible-use workflow, and create a model selection guide that maps directly to the CCAO-F exam domains.

What You'll Build

Imagine walking into a team meeting with a polished document that shows exactly where Claude excels, where it breaks, and how your company should use it safely.

By the end of this project, you'll have:

A prompt engineering test suite you can present to your team, showing 3 business tasks where structured prompts dramatically outperformed vague ones.
A documented hallucination hunt where you caught Claude making things up, complete with your analysis of why it failed and a red flags checklist for spotting unreliable outputs.
A responsible-use workflow and model selection matrix you can share with colleagues, recommending exactly when to use Claude and when to escalate to a human.
Secret Mission: Build a complete Claude Onboarding Guide for a new team member at your company, covering dos, don'ts, and escalation triggers.

Are there any prerequisites?

None. You need a free Claude.ai account and VS Code installed on your computer. This project works entirely on the free plan with no paid tools or cloud resources required.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters. You're creating a Claude Evaluation Report that serves double duty: a professional document for your workplace AND hands-on practice for every CCAO-F exam domain.

Think about a specific team or use case at your company where Claude could help. You'll use that as your test scenario throughout the project.

Write Your First Structured Prompt

You can't evaluate AI output quality if you haven't seen the full spectrum. A vague prompt and a structured prompt produce dramatically different responses from Claude.

In this step, you'll run both extremes back-to-back in Claude.ai, then create your evaluation report file to document what you find. This first comparison becomes the foundation for all the critical analysis you'll do in later steps.

In this step, get ready to:

Send a vague prompt to Claude and observe the generic response.
Write a structured prompt using XML tags and compare the quality difference.
Create claude-evaluation-report.md in VS Code and document your first test case.

Send a vague prompt to Claude

Before you can judge prompt quality, you need a baseline. A vague request shows you Claude's default behavior when it has almost nothing to work with.

Open Claude.ai in your browser.
Start a new conversation by clicking the pencil icon at the top of the left sidebar.
Type this vague prompt into the message box and press Enter:

help me with emails

Read through Claude's response.
Notice how it gives broad, generic advice because it doesn't know who you are, what kind of emails you need help with, or what format you want.

What should I see?

Claude will likely respond with a long list of general email tips. It might cover subject lines, tone, formatting, and sign-offs all at once. That scattered response is exactly the baseline you need for comparison.

Write a structured prompt with XML tags

Structured prompting uses XML-style tags to give Claude a clear role, context, task, and output format. This is one of the key techniques the CCAO-F exam tests in Domain 1 (Prompting and Task Execution).

In the same Claude conversation, type this structured version of the same email request:

<role>You are a professional business communication specialist.</role>
<context>I work in [[TEAM="your team or department"]] at [[COMPANY="your company name"]].</context>
<task>Draft a follow-up email to a client who missed our last meeting, keeping a polite but firm tone.</task>
<format>Subject line + 3-paragraph email body, under 150 words total.</format>

Press Enter to send.
Compare this response to the first one. Notice how the output is focused, correctly formatted, and immediately usable.

Why does this work so much better?

Each XML tag solves a specific problem. <role> tells Claude what expertise to draw on. <context> gives it your specific situation.

<task> defines exactly what to produce. <format> constrains the shape and length of the output. Without these, Claude has to guess what you want at every level.

Keep both responses visible

Scroll up in the same conversation to see both responses. You'll paste both into your report in the next substep.

Create your evaluation report file

Now you'll capture this experiment in a structured Markdown file. This becomes the living document you'll build throughout the project.

Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
Type VS Code and press Enter to open it.
Create a new file by pressing Cmd+N (macOS) or Ctrl+N (Windows).
Save it immediately with Cmd+S (macOS) or Ctrl+S (Windows). Name it claude-evaluation-report.md and save it to your Desktop.
Paste this report header into the file:

# Claude AI Evaluation Report

## Executive Summary

*(To be completed)*

---

What does this structure do?

The title and Executive Summary section give your report a professional structure. You'll fill in the summary at the very end once all your test results are complete. The horizontal rule (---) creates a visual separator before your test sections.

Below the horizontal rule, paste this Section 1 template:

## Section 1: Prompt Engineering Tests

### Test Case 1: Basic vs. Structured Prompt

**Bad Prompt:**
```
help me with emails
```

**Bad Prompt Response:**
*(paste Claude's generic response here)*

**Good Prompt (Structured with XML tags):**
```xml
<role>You are a professional business communication specialist.</role>
<context>I work in [[TEAM="your team or department"]] at [[COMPANY="your company name"]].</context>
<task>Draft a follow-up email to a client who missed our last meeting, keeping a polite but firm tone.</task>
<format>Subject line + 3-paragraph email body, under 150 words total.</format>
```

**Good Prompt Response:**
*(paste Claude's structured response here)*

**Analysis:** *(notes on what improved and why)*

What does this template do?

This template captures a complete before/after comparison. The "Bad Prompt" section records what happens when Claude gets a vague input. The "Good Prompt" section shows the structured version that dramatically improves the output.

The Analysis line at the end is where you write your own observations about what changed and why. This mirrors how the CCAO-F exam asks you to evaluate prompt effectiveness.

Replace *(paste Claude's generic response here)* with the actual vague-prompt response you got from Claude earlier.
Replace *(paste Claude's structured response here)* with the structured-prompt response.
Replace *(notes on what improved and why)* with 2-3 sentences describing the quality difference you observed.
Save the file with Cmd+S (macOS) or Ctrl+S (Windows).

File not saving as Markdown?

If VS Code saved your file without the .md extension, click the language mode indicator in the bottom-right corner of VS Code (it might say "Plain Text") and select Markdown from the list. Then re-save with the correct filename.

help me fix my VS Code file format

✔️ Awesome, I've got everything!

Great. Double check you've saved claude-evaluation-report.md and that it contains both prompts, both responses, and your analysis notes.

ⓧ I'd like to double check the full code

Your claude-evaluation-report.md file should contain these two sections in order:

The report header (title, Executive Summary placeholder, and horizontal rule).
Section 1: Prompt Engineering Tests with Test Case 1 containing your bad prompt, bad prompt response, good prompt with XML tags, good prompt response, and your analysis notes.

Scroll through your file and confirm each section is present and filled in with your actual Claude responses.

Before you move on, what do you think would happen if you ran your structured prompt again but removed the <format> tag entirely?

You now have your first documented test case showing the power of structured prompting. Your report file has a professional structure that you'll fill with more experiments in the coming steps.

You've seen firsthand how structure transforms Claude's output. Next up, you'll push this further by designing three different business prompts using advanced techniques like task decomposition, multi-shot examples, and reusable templates.

Run the Prompt Engineering Lab

Your first structured prompt showed how XML tags transform a vague request into something precise. The CCAO-F exam goes deeper than that single technique. It tests whether you can pick the right prompting technique for different types of tasks.

In this step, you'll run three experiments using techniques the exam specifically covers: task decomposition, multi-shot prompting, and structured templates. For each one, you'll deliberately write a bad prompt first, see how Claude fumbles, then apply the technique and watch the output quality jump.

In this step, get ready to:

Design 3 business-relevant prompts using task decomposition, multi-shot prompting, and reusable templates.
Run bad-vs-good comparisons for each technique and document the quality differences.
Summarize which technique works best for which task type.

Design three prompts using exam techniques

The CCAO-F exam covers three prompting techniques beyond basic XML structuring. Each one solves a different kind of problem.

What are these three techniques?

Task decomposition breaks a complex request into numbered steps so Claude follows a logical sequence instead of dumping everything at once.

Multi-shot prompting provides 2-3 examples of desired input/output before your actual request. Claude mimics the pattern you demonstrate.

Structured templates create a reusable prompt where you swap in different inputs each time. This is for tasks you repeat often.

You'll design one prompt for each technique, customized to your work. Here are example prompts you can use directly or adapt.

Switch to your Claude.ai browser tab.
Start a new conversation.
Paste the following task decomposition prompt (Prompt A), adapting the topic to something relevant to your work:

I need to prepare a quarterly performance review for my team.
Break this into numbered steps and complete each one:

Step 1: List the 3 most important metrics to evaluate for a [your role] team.
Step 2: For each metric, write one sentence describing what "exceeds expectations" looks like.
Step 3: Draft a 2-sentence opening paragraph for the review document.
Step 4: Suggest 2 specific improvement areas with actionable next steps.

What makes this task decomposition?

Instead of asking Claude to "write a performance review" in one shot, you've broken the complex task into four distinct steps. Claude completes each step before moving to the next, producing more thorough and organized output.

Read Claude's response. Notice how it follows your numbered structure and gives focused output for each step.
Start another new conversation in Claude.ai.
Paste this multi-shot prompt (Prompt B), adapting the examples to your industry:

I need you to write internal team updates in a specific style.
Here are examples of the format I want:

Example 1:
Input: We shipped the new checkout flow.
Output: Checkout v2 is live. Conversion testing begins Monday. No action needed from backend.

Example 2:
Input: The database migration broke staging.
Output: Staging is down (DB migration issue). Fix ETA: 4 hours. Workaround: use production read replica for testing.

Now write an update in this same style for:
Input: We hired two new engineers starting next month.

What makes this multi-shot?

You gave Claude 2 concrete examples of input and desired output before asking your actual question. Claude learns the pattern (concise, action-oriented, includes next steps) and applies it to your new input without you needing to describe the style in words.

Read Claude's response. Notice how it matches the style and structure of your examples.
Start one more new conversation in Claude.ai.
Paste this structured template prompt (Prompt C), filling in your own variable values:

You are a [ROLE] specialist.

Context: I work at [COMPANY] in the [DEPARTMENT] team.

Task: Write a [DELIVERABLE_TYPE] about [TOPIC] for [AUDIENCE].

Constraints:
- Length: [WORD_COUNT] words
- Tone: [TONE]
- Must include: [REQUIRED_ELEMENTS]

Format: [OUTPUT_FORMAT]

What makes this a reusable template?

The bracketed placeholders make this prompt reusable across dozens of different tasks. You swap in new values each time without redesigning the prompt structure. This is ideal for tasks you repeat often, like drafting different types of documents for different audiences.

Read Claude's response. Notice how the constraints kept the output focused and on-target.

You now have three strong responses from Claude using three different techniques. Next, you'll write the bad versions to see exactly how much these techniques improve output quality.

Run bad-vs-good comparisons and document results

Seeing good output is useful. Seeing the gap between bad and good output is what builds exam instincts. You'll now write the worst possible version of each prompt, run it, and document the difference.

Start a new conversation in Claude.ai.
Type a vague, unstructured version of your Prompt A task. For example, if your good prompt was about a performance review, your bad version might be:

help me with a performance review

Read Claude's response. Notice how generic and unfocused it is compared to the decomposed version.
Repeat this process for Prompt B and Prompt C. Write a vague, example-free version of each and run it in a new conversation.

Bad prompt ideas for B and C

For Prompt B (multi-shot), try: "write a team update about new hires". For Prompt C (template), try: "write me a document for work". The vaguer, the better. You want to see Claude guess at everything you didn't specify.

Now it's time to document everything in your report.

Switch back to VS Code with claude-evaluation-report.md open.
Below your existing Test Case 1, add the following structure for Test Case 2. Paste your actual prompts and responses where indicated:

---

### Test Case 2: Task Decomposition (Prompt A)

**Bad Prompt:**
```
(paste your vague prompt here)
```
**Bad Prompt Response:** (paste Claude's generic response)

**Good Prompt (Task Decomposition):**
```
(paste your numbered-steps prompt here)
```
**Good Prompt Response:** (paste Claude's structured response)

**Analysis:** (note what changed - was it more organized? more thorough? more actionable?)

What does this document?

Each test case captures the full experiment: the bad prompt, its weak response, the improved prompt using a specific technique, the better response, and your analysis of why it improved. This is both your study material and a professional deliverable.

Save the file.
Now add Test Case 3 below Test Case 2. Paste your multi-shot prompts and responses:

---

### Test Case 3: Multi-Shot Prompting (Prompt B)

**Bad Prompt:** (paste your no-examples prompt)
**Bad Prompt Response:** (paste Claude's response)

**Good Prompt (Multi-Shot with 2-3 examples):**
```
Example 1: Input: ... Output: ...
Example 2: Input: ... Output: ...
Actual request: ...
```
**Good Prompt Response:** (paste Claude's response)

**Analysis:** (note what changed - did it match your desired style? was the format consistent?)

Save the file.
Add Test Case 4 below Test Case 3. Paste your template prompts and responses:

---

### Test Case 4: Structured Template with Variables (Prompt C)

**Bad Prompt:** (paste your unstructured one-off request)
**Bad Prompt Response:** (paste Claude's response)

**Good Prompt (Reusable Template):**
```
(paste your template with [VARIABLE] placeholders)
```
**Good Prompt Response:** (paste Claude's response)

**Analysis:** (note what changed - was it more targeted? could you reuse this prompt for other tasks?)

Save claude-evaluation-report.md.
Scroll through your file. You should now see four test cases (the original XML-tag test from Step 1 plus your three new technique tests), each with bad and good prompt pairs.

Responses look too similar between bad and good?

Make your bad prompts worse. If Claude still produces decent output from a vague prompt, it means your "bad" version is still giving too much context. Strip it down to 5 words or fewer with no formatting.

Help me make my bad prompts vague enough to show a clear quality difference.

Write your Lessons Learned summary

The CCAO-F exam's Domain 1 objective is "adapt prompts by task type." Your Lessons Learned paragraph is where you crystallize that skill into words.

At the bottom of Section 1 in your report, add a Lessons Learned heading:

---

### Lessons Learned

(Write 3-5 sentences here summarizing which technique worked best for which type of task. Consider: Which technique gave the biggest quality jump? Which tasks need decomposition vs. examples vs. templates? When would you combine techniques?)

What should I write?

Think about what you observed across your three experiments. Task decomposition tends to shine for complex multi-part tasks. Multi-shot works best when you need a specific style or format. Templates excel for repeatable tasks where the structure stays constant but inputs change.

Write what YOU actually saw. Your specific observations will stick in your memory better than generic statements when exam questions ask about choosing techniques.

Write your Lessons Learned paragraph based on your actual experimental results.
Save claude-evaluation-report.md.

Before you check your work, think about this: which of the three techniques produced the biggest quality jump for your specific use case?

Scroll through Section 1 from top to bottom. You should see Test Cases 1 through 4, each with a bad prompt, bad response, good prompt, good response, and analysis. At the end, your Lessons Learned paragraph ties it all together.

✔️ Awesome, I've got everything!

Confirm you've saved your file. Section 1 should now have 4 test cases and a Lessons Learned paragraph.

ⓧ I'd like to double check the full code

Scroll through claude-evaluation-report.md and confirm these sections are present, in this order:

Title: "Claude AI Evaluation Report".
Executive Summary (still blank, that's fine).
Section 1: Prompt Engineering Tests heading.
Test Case 1: Basic vs. Structured Prompt (from Step 1).
Test Case 2: Task Decomposition (Prompt A) with bad/good pair and analysis.
Test Case 3: Multi-Shot Prompting (Prompt B) with bad/good pair and analysis.
Test Case 4: Structured Template with Variables (Prompt C) with bad/good pair and analysis.
Lessons Learned paragraph at the end of Section 1.

Your prompt engineering lab is complete. You've built real instincts for choosing the right technique. Next up, you'll push Claude to its limits and hunt for hallucinations.

Hunt for Hallucinations

You've seen how structured prompts improve Claude's responses. But here's the harder question: how do you know when Claude's output is wrong?

Output Evaluation is the CCAO-F exam's heaviest domain at 21%. Most people have never deliberately tested where Claude fails. In this step, you'll intentionally try to break Claude, catch it making things up, and build a personal checklist of warning signs you can use on the exam and at work.

In this step, get ready to:

Test Claude with prompts designed to trigger hallucinations.
Document each failure with your analysis and verification method.
Build a Red Flags Checklist of unreliable output patterns.

Deliberately trigger hallucinations in Claude

A hallucination is when an AI model generates something that sounds confident and plausible but is factually wrong. Claude doesn't "know" it's making things up. Certain types of questions are more likely to trigger this behaviour.

Switch back to Claude.ai in your browser and start a new conversation.
Try at least 3 of these hallucination triggers, sending each one as a separate message:

Recent event: Ask about something that happened in the past week. For example, "What were the key announcements at [specific conference] last Tuesday?"
Specific citation: Ask for a statistic with a source URL. For example, "What percentage of companies adopted AI in 2025? Provide the exact URL of the study."
Obscure niche topic: Ask about something very specific and hard to verify. Mix in a fake-sounding detail to see if Claude corrects you or plays along.
Named person in a role: Ask Claude to name the person in a specific role at a real company. For example, "Who is the VP of Engineering at [a mid-size company you know]?"

Pick questions you can actually verify

The best hallucination tests are ones where YOU already know the answer or can quickly Google it. That way you can confirm whether Claude got it right, hedged, or made something up.

After each response, take note of whether Claude:

Hallucinated: gave a confident, specific answer that turns out to be wrong.
Hedged: used cautious language like "I believe" or "as of my last update" to signal uncertainty.
Correctly refused: said it didn't know or couldn't verify the information.

You should now have at least 3 Claude responses where you can identify the type of behaviour. For each one, do a quick check (a Google search, a company website visit, or your own knowledge) to verify whether Claude was right or wrong.

Why does Claude hallucinate?

Claude generates text by predicting what comes next based on patterns in its training data. It has no real-time internet access and no way to "look things up." When it encounters a gap in its knowledge, it sometimes fills the gap with plausible-sounding text rather than admitting uncertainty.

Hedging language is a designed safety behaviour. When you see phrases like "I believe" or "approximately," Claude is signalling lower confidence. On the exam, recognising these signals is key to Domain 2.

Document your hallucination tests in the report

Each test needs four pieces of information: the prompt you used, Claude's response, your assessment of the behaviour type, and how you verified the answer. This structured format mirrors how the exam expects you to evaluate AI output.

Switch back to claude-evaluation-report.md in VS Code.
Scroll to the bottom of the file, after the Section 1 content.
Add the Section 2 heading and your test documentation by pasting the following template:

## Section 2: Output Evaluation and Hallucination Analysis

### Hallucination Test 1

**Prompt:** *(paste your prompt here)*
**Claude's Response:** *(paste Claude's response)*
**Assessment:** *(Did it hallucinate, hedge, or correctly refuse?)*
**Verification Method:** *(How did you check?)*

---

### Hallucination Test 2

**Prompt:** *(paste your prompt here)*
**Claude's Response:** *(paste Claude's response)*
**Assessment:** *(hallucinate / hedge / refuse)*
**Verification Method:** *(How did you check?)*

---

### Hallucination Test 3

**Prompt:** *(paste your prompt here)*
**Claude's Response:** *(paste Claude's response)*
**Assessment:** *(hallucinate / hedge / refuse)*
**Verification Method:** *(How did you check?)*

What does each field capture?

Prompt: The exact text you sent to Claude. Copy it verbatim so you can reproduce the test.
Claude's Response: The key part of what Claude said back. You can summarise long responses but keep the critical claims.
Assessment: Your judgment call. Did it hallucinate (confidently wrong), hedge (cautious language), or correctly refuse (said it didn't know)?
Verification Method: How you confirmed the answer was wrong. A Google search, a company website, your own expertise.

Fill in each test with your actual prompts and Claude's responses from the previous substep.
Save claude-evaluation-report.md.

Your report now has 3 documented hallucination tests. Each one shows a specific way Claude can fail and how you caught it.

Can't get Claude to hallucinate?

Claude has gotten better at hedging and refusing over time. If you're getting refusals on everything, try asking for something very specific and obscure. For example, ask it to name the third employee hired at a specific startup, or to quote a specific page number from an uncommon book.

If Claude hedges instead of hallucinating, that's still a useful finding. Document it as a "hedge" assessment. The exam tests whether you can recognise hedging as a signal.

Build your Red Flags Checklist

Now that you've seen Claude fail in specific ways, capture those patterns as a reusable checklist. This becomes your personal reference for evaluating any Claude output, and it maps directly to what the exam tests in Domain 2.

At the bottom of Section 2 in claude-evaluation-report.md, add the following Red Flags Checklist:

---

### Red Flags Checklist

- [ ] Hedging language: "I believe", "approximately", "as of my last update"
- [ ] Citation provided without a verifiable URL or source
- [ ] Confident tone on topics that are obscure or time-sensitive
- [ ] Specific statistics or numbers without a named source
- [ ] Named individuals in specific roles without verification
- [ ] Failure to refuse when the answer is unknowable

What does this checklist do?

Each item is a pattern you observed during your hallucination tests. When you're evaluating Claude's output in the future (or on the exam), you can run through this list mentally. If you spot any of these flags, the output needs verification before you trust it.

Feel free to add more items based on patterns you noticed in your own tests. The checklist in the template covers the most common ones, but your specific experiments may have revealed others.

Save claude-evaluation-report.md.
Scroll through Section 2 in VS Code. You should see 3 test entries (each with prompt, response, assessment, and verification) followed by your Red Flags Checklist.

Before you move on, think about this: across your 3 tests, did you find at least one case where Claude was confidently wrong (not just hedging)?

If the answer is yes, you've experienced firsthand why Output Evaluation matters. If all your tests resulted in hedges or refusals, that's still valuable data. It shows Claude's safety training at work. Either way, you now have documented evidence of how Claude handles the boundaries of its knowledge.

Section 2 looking incomplete?

Make sure each of your 3 test entries has all four fields filled in (Prompt, Claude's Response, Assessment, Verification Method). The Red Flags Checklist should appear below all three tests, separated by a horizontal rule.

If you're unsure whether your assessments are correct, try verifying one more time with a quick search.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

Scroll through claude-evaluation-report.md from top to bottom. Confirm these sections are present, in order:

Title and Executive Summary placeholder at the top.
Section 1: Prompt Engineering Tests with your 4 test cases and Lessons Learned (from the previous step).
Section 2: Output Evaluation and Hallucination Analysis with 3 completed test entries.
Red Flags Checklist with 6 checkbox items at the bottom of Section 2.

You've just completed the most heavily weighted exam domain through real experimentation. You didn't just read about hallucinations. You hunted them, caught them, and documented the evidence. Next up, you'll take your worst failure and fix it with better prompting techniques.

Fix Problems and Build a Model Selection Matrix

In the last step, you deliberately broke Claude and documented where it fails. You now have real evidence of hallucinations, hedging, and confident fabrications.

The CCAO-F exam does not just test whether you can spot problems. It tests whether you can fix them. In this step, you will apply techniques to prevent hallucinations and build a model selection matrix that maps the right Claude model to each business task.

In this step, get ready to:

Fix your worst hallucination from Section 2 using improved prompting techniques.
Build a Model Selection Matrix recommending Haiku, Sonnet, or Opus for different tasks.
Document scenarios where Claude should NOT be used at all.

Fix your worst hallucination with better prompting

Finding failures is only half the battle. The exam's Troubleshooting domain (10% weight) tests whether you can revise a prompt to prevent the failure from recurring.

Switch to your claude-evaluation-report.md file in VS Code.
Look at your Section 2 hallucination tests and pick the worst one. This is the example where Claude was most confidently wrong or gave the most misleading response.

You are going to rewrite that same prompt using techniques that force Claude to be more careful. Your goal is to get a better response to the same question.

Switch back to Claude.ai in your browser and start a new conversation.
Write a revised version of your worst prompt. Add one or more of these techniques:
Send the revised prompt and compare the response to the original hallucinated answer.

What should I see?

Claude's response should be noticeably more cautious. Instead of confident fabrication, you might see hedging language, explicit uncertainty statements, or an outright refusal to answer. That caution is the improvement you were aiming for.

Now document this fix in your report.

In claude-evaluation-report.md, add the following markdown after your Section 2 content:

---

## Section 3: Troubleshooting and Optimization

### Fix Example: Preventing Hallucination

**Original Problematic Prompt:** *(worst hallucination prompt from Section 2)*
**Original Response:** *(the hallucinated response)*

**Improved Prompt:**
```
*(revised prompt with instructions like: if you are uncertain, say "I don't know"; provide a confidence level; reason step-by-step before answering)*
```
**Improved Response:** *(Claude's better response)*
**Analysis:** *(what changed and why it works)*

What does this structure do?

This before/after format makes the improvement visible at a glance. The "Analysis" field is where you explain WHY the techniques worked. Did chain-of-thought force Claude to realize it was guessing? Did the explicit "say I don't know" instruction trigger a refusal?

This maps directly to Domain 7 of the exam, which covers troubleshooting Claude outputs and applying corrective strategies.

Replace the placeholder text with your actual prompts, responses, and analysis from the experiment you just ran.
Save claude-evaluation-report.md.

You should now see Section 3 in your report with a complete before/after comparison showing how your revised prompt produced a more careful response.

Claude still hallucinated even with your fix?

If the improved prompt still produced fabricated content, try combining multiple techniques in one prompt. For example, add both "say I don't know if uncertain" AND "reason step-by-step before answering" to the same prompt. Sometimes a single guardrail is not enough for tricky topics.

If nothing works, that is also a valid finding. Document it in your Analysis as a case where prompt engineering alone cannot fix the problem, and note that human verification is required.

Help me figure out why my improved prompt still produces hallucinations.

Build your Model Selection Matrix

The Product and Model Selection domain accounts for 12% of the exam. It tests whether you can recommend the right Claude model for a given task based on complexity, speed, and cost.

Claude offers three model tiers: Haiku (fastest, cheapest, best for simple tasks), Sonnet (balanced quality and speed), and Opus (strongest reasoning, highest cost). Your job is to map these to real business tasks.

In claude-evaluation-report.md, add the following markdown after your Section 3 content:

---

## Section 4: Product and Model Selection

### Model Selection Matrix

| Task Type | Recommended Model | Reasoning | Cost Consideration |
|-----------|------------------|-----------|-------------------|
| Quick email drafting | Haiku | Fast, low-latency for simple tasks | Lowest cost per token |
| Complex data analysis | Opus | Strongest reasoning for nuanced tasks | Highest cost, use selectively |
| General writing/summarization | Sonnet | Balanced quality and speed | Mid-range cost |
| Customer FAQ responses | Haiku | Repetitive, structured, low complexity | Cost-efficient at scale |
| Legal/contract review | Opus | Requires deep reasoning and accuracy | Cost justified by risk reduction |

What does this table cover?

The Task Type column names a specific business use case.
The Recommended Model column picks one of Haiku, Sonnet, or Opus.
The Reasoning column explains why that model fits.
The Cost Consideration column acknowledges the trade-off. On the exam, you will need to justify model choices based on both capability and cost.

Customize the table rows with tasks that match your company's actual work. The five rows above are starting examples. Add or replace rows with tasks your team performs daily.
Make sure you have at least 5 rows total.
Save claude-evaluation-report.md.

You should see a clean markdown table in your report with at least 5 rows covering a range of tasks from simple (Haiku) to complex (Opus).

Table not rendering correctly in VS Code?

Markdown tables require the pipe characters to align with the header separator row. Make sure each row has the same number of pipe characters as the header. You can preview the table by pressing Cmd+Shift+V (macOS) or Ctrl+Shift+V (Windows) to open Markdown Preview in VS Code.

Help me fix my markdown table formatting.

Document when NOT to use Claude

The exam specifically tests whether you know when to stop using Claude and involve a human expert. This is part of the "determine when human review is necessary" objective.

In claude-evaluation-report.md, add the following markdown directly after your Model Selection Matrix table:

### When NOT to Use Claude

- Tasks requiring real-time or live data (e.g., current stock prices, breaking news)
- Safety-critical decisions (e.g., medical diagnosis, legal advice without review)
- Tasks requiring guaranteed factual accuracy with no human verification step
- Processing PII or sensitive credentials
- Any output that goes directly to customers without human review

Why does this matter for the exam?

The CCAO-F tests your judgment about Claude's boundaries, not just its capabilities. A common exam question pattern gives you a scenario and asks whether Claude is the right tool. Your "When NOT to Use Claude" list trains exactly that instinct.

Think about your own hallucination results from Section 2. Did any of your test failures map to these categories? The connection between your experiments and these rules is what makes this evaluation report genuine.

Add any additional "don't use" scenarios that you discovered during your own experiments. Your hallucination hunt in Section 2 likely revealed specific types of questions that Claude cannot handle reliably.
Save claude-evaluation-report.md.

Before you review your work, think about this: what should Sections 3 and 4 together prove about your ability to work responsibly with Claude?

Open Markdown Preview (Cmd+Shift+V on macOS or Ctrl+Shift+V on Windows) and scroll through your report. You should see Section 3 with a clear before/after fix example, and Section 4 with a formatted table of 5+ rows plus your "When NOT to Use Claude" list.

✔️ Awesome, I've got everything!

Confirm you have saved claude-evaluation-report.md with both new sections.

ⓧ I'd like to double check the full code

Scroll through your claude-evaluation-report.md file and confirm you have each of these sections in order:

Executive Summary (still blank, to be completed later).
Section 1: Prompt Engineering Tests (from Step 2).
Section 2: Output Evaluation and Hallucination Analysis (from Step 3).
Section 3: Troubleshooting and Optimization with your fix example.
Section 4: Product and Model Selection with a table of 5+ rows AND a "When NOT to Use Claude" list.

Your report now covers troubleshooting, model selection, and responsible boundaries. In the next step, you will design a complete workflow that ties all of these skills together into something your team can actually use.

Design a Responsible-Use Workflow

You've proven you can spot Claude's failures and fix them with better prompts. You've also built a model selection matrix for choosing the right tool for each task. But knowing what Claude can do isn't enough.

The exam tests whether you can design safe workflows around Claude, with guardrails that protect your team from mistakes. Governance (15%) and Workflow Integration (16%) together make up 31% of the exam. In this step, you'll think like the exam does: not just "can I use Claude?" but "should I, and with what safeguards?"

In this step, get ready to:

Map out a 4-step responsible-use workflow for a real business scenario at your company.
Write governance guardrails covering data safety, human review, and escalation triggers.
Spec out a Claude Project configuration for team-wide knowledge management.

Map out a 4-step responsible-use workflow

A responsible-use workflow isn't just "ask Claude and use the answer." It breaks every interaction into four stages: what goes in, what Claude does, how a human validates the output, and what finally gets delivered. Each stage carries a specific risk that needs a specific mitigation.

Think of a real workflow at your company where Claude could help. Examples: drafting customer communications, summarizing meeting notes, researching competitors, or writing internal documentation.
Switch back to claude-evaluation-report.md in VS Code.
After the Section 4 content, add this Section 5 template and customize the table cells for your specific use case:

## Section 5: Responsible-Use Workflow

### Workflow: [[USECASE="your chosen business use case, e.g. Drafting Customer Communications"]]

| Step | Description | Potential Risk | Mitigation |
|------|-------------|---------------|------------|
| 1. Input | What data/request goes to Claude | Sensitive data included accidentally | Pre-input checklist; strip PII |
| 2. Processing | What Claude does with the input | Hallucination or off-brand tone | Use structured prompt with constraints |
| 3. Validation | Human reviews Claude's output | Reviewer skips check under time pressure | Mandatory review gate before delivery |
| 4. Output | Final deliverable sent to recipient | Unreviewed content reaches customer | Only reviewed outputs leave the workflow |

What does this workflow do?

Each row represents a stage in the pipeline from raw request to final deliverable. The "Potential Risk" column names what can go wrong at that stage, and "Mitigation" names the specific control that prevents it.

This 4-step structure maps directly to Domain 4 of the exam (Workflow Integration and Solution Design). The exam asks you to identify where human checkpoints belong in an AI-assisted process.

Replace the example descriptions with specifics from your chosen workflow. For example, if your use case is summarizing meeting notes, the Input row might say "Raw meeting transcript pasted into Claude" and the risk might be "Transcript contains client names or deal values."
Save claude-evaluation-report.md.

You should see the new Section 5 heading and a formatted table with 4 rows in your Markdown preview.

Table not rendering properly?

Markdown tables require the separator row (the |------| line) between the header and data rows. Make sure every row has the same number of pipe characters.

Help me fix my Markdown table formatting in my evaluation report.

Add governance guardrails

The exam tests four specific categories of governance knowledge: what data is off-limits, what outputs need human review, when to stop using Claude entirely, and how your company's acceptable-use policy applies. This subsection covers all four.

Below the workflow table in Section 5, add this Governance Guardrails subsection:

### Governance Guardrails

**Data that should NEVER be pasted into Claude:**
- Personally Identifiable Information (PII): names, emails, phone numbers, addresses
- Login credentials, API keys, passwords
- Proprietary source code or trade secrets
- Confidential financial data

**Outputs that require human review before use:**
- Anything customer-facing
- Any content containing statistics, numbers, or factual claims
- Legal, compliance, or HR-related content

Why these specific categories?

Domain 6 of the exam (Governance, Risk, and Responsible Use) weighs 15%. It specifically tests whether you can identify data categories that pose privacy or security risks when shared with an AI system.

The "human review" list targets a common exam scenario: distinguishing outputs Claude can generate autonomously from outputs that need a human checkpoint before they leave your organization.

Now add the escalation triggers and acceptable-use section directly below:

**Escalation Triggers - stop using Claude and involve a specialist when:**
- The task involves a legal or regulatory decision
- Claude expresses uncertainty on a safety-critical topic
- Output will be published externally without further editing
- A data breach or privacy concern arises

**Acceptable Use Considerations:**
- Use Claude as a drafting and ideation tool, not a final authority
- Always disclose AI involvement per company policy
- Follow Anthropic's usage policies

What are escalation triggers?

Escalation triggers are the red-line moments when you stop relying on Claude and bring in a human expert. The exam tests whether you can identify these boundaries. Think of them as the "fire alarm" for your AI workflow.

Customize the guardrails for your company. Add any industry-specific risks (healthcare data, financial regulations, student records) that apply to your team.
Save claude-evaluation-report.md.

Scroll through Section 5. You should see the workflow table followed by four bold-headed guardrail categories, each with a bullet list of specific items.

Guardrails section looks unformatted?

Make sure each bold heading uses double asterisks (**text**) and that there is a blank line between the heading and the first bullet item.

Help me fix the Markdown formatting in my Governance Guardrails section.

Spec out a Claude Project configuration

Claude Projects let teams set persistent system instructions and upload knowledge sources that Claude references in every conversation. This feature requires a Pro plan, so you won't build one today. But understanding how you would configure Projects is what the exam tests in Domain 5 (Configuration and Knowledge Management, 12%).

After Section 5 (with a --- divider), add this Section 6 template and fill it in based on your team's needs:

---

## Section 6: Configuration and Knowledge Management

### Claude Project Configuration Spec

**System Instructions I would set:**
*(e.g., tone guidelines, output format defaults, topics to avoid, role definition)*

**Knowledge Sources I would upload:**
*(e.g., brand style guide, product FAQs, approved response templates)*

**Multi-Project Organization:**
*(e.g., Project 1: Customer Support, Project 2: Internal Comms, Project 3: Research - each with tailored instructions and knowledge bases)*

What does this section cover?

System instructions are persistent rules Claude follows in every conversation within a Project. Think of them as the "personality and boundaries" you set once so you don't have to repeat them in every prompt.

Knowledge sources are documents (style guides, FAQs, templates) you upload so Claude can reference them without you pasting content every time. Multi-Project organization means different teams or workflows get their own tailored setup.

Replace the example text in parentheses with specifics for your team. Think about what tone your company uses, what documents you reference most often, and how many distinct workflows could each benefit from their own Project.
Save claude-evaluation-report.md.

You should now see Section 6 at the bottom of your report with three bold subheadings and your personalized configuration details under each.

Not sure what to write for system instructions?

Think about the prompts you wrote in Step 2. The role definition and output constraints you used there are exactly the kind of instructions you'd set as persistent system instructions in a Project. Pull from your best prompt's structure.

Help me brainstorm system instructions for a Claude Project at my company.

✔️ Awesome, I've got everything!

Great. Double check that you've saved your file and that both Section 5 and Section 6 appear in your report.

ⓧ I'd like to double check the full code

Scroll through your file from top to bottom. Confirm these sections appear in order, each separated by a --- divider:

Section 1: Prompt Engineering Tests (from Step 2).
Section 2: Output Evaluation and Hallucination Analysis (from Step 3).
Section 3: Troubleshooting and Optimization (from Step 4).
Section 4: Product and Model Selection (from Step 4).
Section 5: Responsible-Use Workflow with a 4-row workflow table and Governance Guardrails subsection (from this step).
Section 6: Configuration and Knowledge Management with system instructions, knowledge sources, and multi-project organization (from this step).

Before you wrap up, ask yourself: could you explain to a colleague why each guardrail exists? The exam will present scenarios and ask which control applies.

Your report now has Sections 5 and 6 complete. You have a full workflow diagram with risks and mitigations, governance guardrails covering all four categories the exam tests, and a Project configuration spec. You can explain why each guardrail exists and when to escalate beyond Claude.

Secret mission

Build a Claude Onboarding Guide

The ultimate test of your knowledge is teaching it to someone else. Can you synthesize everything you've learned across all 7 CCAO-F exam domains into a single, clear onboarding guide that a non-technical colleague could follow? This stretches your understanding from "I can use Claude" to "I can help others use Claude responsibly."

Clean Up Your Resources

This project runs entirely locally and uses the Claude.ai free plan, so there are no ongoing costs or paid resources to worry about.

Decide whether to keep your resources as-is, or delete them entirely.

Resources you used:

claude-evaluation-report.md file in VS Code (local).
Claude.ai conversation history (free plan, no cost).

Keep everything running

No action needed. Choose this if you're still actively building on your report or studying for the exam.

Your claude-evaluation-report.md file stays on your machine ready to extend.
Your Claude.ai conversations remain in your chat history for reference.
Consider sharing your report with your team as a genuine Claude evaluation document.
Keep adding test cases as you work through the CCAO-F study plan.

Pause - I'll come back to this later

Since everything is local and free, there is nothing to pause or shut down. Your file stays where it is.

Your claude-evaluation-report.md file remains in VS Code whenever you reopen it.
Your Claude.ai conversations persist in your account until you delete them.
When you're ready to continue, open the file in VS Code and pick up where you left off.

Delete - I don't want to use this again

Remove all project resources and start fresh.

Delete the local report file by running this command in your terminal:

macOS

rm ~/Desktop/claude-evaluation-report.md

Windows

Remove-Item ~\Desktop\claude-evaluation-report.md

Delete your Claude.ai conversations from this project:
In Claude.ai, click Chats on the left sidebar.
Hover over each conversation you created during this project and check the selection box that appears.
Click Delete Selected to remove them.

That's a Wrap!

Nice work. You just built a professional Claude Evaluation Report that covers all 7 CCAO-F exam domains through hands-on experimentation.

You've learned how to:

Engineer structured prompts using XML tags, multi-shot examples, and task decomposition to dramatically improve Claude's output quality.
Hunt and document real hallucinations, build a Red Flags Checklist for spotting unreliable output, and apply fixes that prevent failures.
Design a responsible-use workflow with governance guardrails, a model selection matrix for business tasks, and a Project configuration spec for knowledge management.
Secret Mission: Created a complete Claude Onboarding Guide that synthesizes all 7 exam domains into clear teaching material for a new team member.

Ready to quiz yourself?