AI Product Strategy for a GenAI Copilot

Build a portfolio-ready AI product strategy workspace in Notion.

Introduction

30 Second Summary

Every company wants to add AI to their product. The difference between AI features that users love and ones they abandon comes down to strategy, not technology.

In this project, you will build a complete AI product strategy workspace in Notion for a GenAI-powered copilot. You'll create the exact artifacts AI PM hiring managers test for: problem discovery, prioritization scorecards, evaluation frameworks, and a confidence-based roadmap.

What You'll Build

A portfolio-ready AI product strategy workspace for "Arcadia AI Copilot," a GenAI assistant that helps customer success managers at a mid-size B2B SaaS company draft responses, summarize ticket histories, and surface knowledge base articles.

By the end of this project, you'll have:

A JTBD problem discovery document with job statements, four forces analysis, desired outcomes, and trust UX requirements for an AI copilot.
An AI feature prioritization database with a 6-layer scoring framework including privacy and security gates that filter out risky features automatically.
A tri-track AI product roadmap with confidence-based milestones, an evaluation framework with launch criteria, and stakeholder-specific views for engineering, executives, and customers.
Secret Mission: Build an executive strategy one-pager and a layered AI metrics tree that connects model performance to business outcomes.

Are there any prerequisites?

None. This project covers everything from account setup to finished strategy artifacts. You do not need prior PM experience or technical skills to complete it.

You will use Notion's free plan, which supports unlimited pages and blocks for single-user workspaces. No paid tools are required.

Set Up Your Notion Workspace

Every great product starts with a structured workspace where strategy lives in one place. Before you can build a product roadmap, prioritize features, or design an evaluation framework, you need a home for all of it.

In this step, you will create a Notion workspace, set up the page structure for your AI copilot strategy, and build your first database for feature prioritization.

In this step, get ready to:

Create a Notion account and workspace on the free plan.
Build a 5-page project structure for your AI copilot strategy.
Set up a feature prioritization database with custom properties.

Create your Notion account and workspace

Notion is a free workspace tool that product managers use to build strategy documents, databases, and roadmaps. You will use it to create every artifact in this project.

Open your browser and go to notion.com.
Click Sign up in the top right corner.
Enter your email address (or choose to continue with Google or Apple).
Follow the prompts to verify your email and create your account.

Why keep it single-user?

Notion's Free plan gives you unlimited pages and blocks when your workspace has only one member. If you add a second member, the workspace is capped at 1,000 blocks. Keep it single-user for this project so you have no limits.

When Notion asks you to set up a workspace, give it any name you like (your name or "Product Strategy" both work).
Skip any prompts to invite teammates.
If Notion offers a template, dismiss it. You will build your own structure from scratch.

You should now see an empty Notion workspace with a left sidebar and a blank page area.

Don't see a blank workspace?

If Notion loaded a pre-filled template or onboarding content, click the workspace name at the top-left of the sidebar to confirm you are in the right workspace. You can delete any template pages by right-clicking them in the sidebar and selecting Delete.

Help me set up my Notion workspace from scratch.

Build the project structure

Your AI copilot strategy needs five distinct workstreams: problem discovery, feature prioritization, evaluation design, roadmapping, and stakeholder communication. Each gets its own page.

In the left sidebar, click the + icon (or press Cmd+N on macOS or Ctrl+N on Windows) to create a new page.
Title this page Arcadia AI Copilot - Product Strategy.
Click into the body of this page and type the following context block:

Arcadia is a customer success platform with ~500 enterprise clients. Leadership wants to add an AI-powered copilot that helps CSMs draft responses, summarize ticket histories, and surface KB articles.

What is this context block for?

This paragraph sets the scenario for everything you build in this workspace. It acts as a quick reference so you never lose sight of the product context while working through later steps.

Hover over Arcadia AI Copilot - Product Strategy in the left sidebar.
Click the + icon that appears to create a nested sub-page.
Title this sub-page Problem Discovery & JTBD.
Repeat the hover-and-click process to create four more sub-pages, each titled exactly as follows:

Feature Prioritization
Evaluation Framework
Product Roadmap
Stakeholder Views

Expand the parent page in the sidebar by clicking the toggle arrow next to it.
Verify that all five sub-pages are nested underneath Arcadia AI Copilot - Product Strategy.

Your sidebar should now show one parent page with five nested sub-pages listed below it.

Sub-pages not nested correctly?

If a page appears at the top level instead of nested, drag it in the sidebar and drop it onto the parent page name. The page will indent to show it is nested.

Help me nest pages correctly in Notion.

Set up your first database

Databases in Notion are structured tables with custom properties that let you score, filter, and sort items. You will build a prioritization database to evaluate AI features in the next step.

In the left sidebar, click on Feature Prioritization to open that sub-page.
Click into the body of the page and type /database to open the slash command menu.
Select Database - Inline (or Table) from the dropdown.

What is a Notion database?

A Notion database is an advanced page type that structures information into rows and columns. Each column is a "property" with a specific type (text, number, select, formula, etc.) that defines what kind of data it holds.

Think of it like a spreadsheet, but each row is also its own page where you can add detailed notes.

Now add the custom properties that will form your prioritization scorecard. The first column (titled "Name" by default) will become your Feature Name.

Click the default Name column header and rename it to Feature Name.
Click the + button to the right of the last column header to add a new property.
Name it Problem-AI Fit and set the type to Select.
Click Add an option and create two options: Pass and Fail.
Add another property named Data Readiness and set the type to Number.
Add another property named UX Trust and set the type to Number.
Add another property named Privacy Risk and set the type to Select.
Create four options for Privacy Risk: Low, Medium, High, and Blocker.
Add another property named Competitive Moat and set the type to Number.
Add a final property named Priority Score and set the type to Formula.

Leave the formula blank for now

You will write the actual formula in Step 3 after you understand how each scoring layer works. For now, just confirm the property exists and is set to the Formula type.

Verify your database displays in table view with all 7 columns visible: Feature Name, Problem-AI Fit, Data Readiness, UX Trust, Privacy Risk, Competitive Moat, and Priority Score.

Your database should show an empty table with the seven property columns. No rows are needed yet.

Missing a column or wrong property type?

Click any column header and select Edit property to rename it or click Change type to switch between Number, Select, and Formula. Property names are case-sensitive in formulas, so double-check the spelling now.

Help me fix my database properties in Notion.

Your workspace is ready. Next up, you will fill the Problem Discovery page with a JTBD analysis that grounds your entire product strategy in real user needs.

Build Problem Discovery & User-Centered Design

Your workspace is set up and ready for content. Now it is time to fill in the most important strategic artifact an AI PM produces: a structured problem discovery document.

Before building any AI feature, a PM needs to deeply understand the user's job and the forces influencing their decision to adopt something new. The Jobs-to-be-Done framework gives you that structure. It also forces you to think about trust, because users will only adopt an AI copilot if they feel confident in its outputs.

In this step, get ready to:

Write a JTBD problem framing with job statement and Four Forces analysis.
Define measurable desired outcomes and trust UX requirements.
Map user-centered design principles for the AI copilot.

Write the JTBD problem framing

The JTBD framework asks: what job is the user hiring this product to do? For Arcadia's AI copilot, the user is a Customer Success Manager drowning in complex tickets. Your job is to define their situation precisely enough that the solution becomes obvious.

Switch to your Problem Discovery & JTBD page in Notion.
Type /h1 and press Enter to create a Heading 1 block.
Title it Job Performer Definition.
Below the heading, type the following definition as a paragraph:

"Customer Success Managers (CSMs) at enterprise B2B SaaS companies managing 20-40 accounts, handling 50+ tickets/week."

Why start with the job performer?

AI products fail when they solve a problem for a vague "user." By naming the exact person (CSMs with 20-40 accounts and 50+ tickets/week), you anchor every feature decision to a real workload. This specificity is what separates strategic PM work from generic feature brainstorming.

Create another Heading 1 below your definition by typing /h1 and pressing Enter.
Title it Core Job Statement.
Below it, write the job statement using the JTBD format: "When [situation], I want to [motivation], so I can [desired outcome]."
Use this example statement:

"When I receive a complex customer escalation with a long ticket history, I want to quickly understand the full context and draft an accurate, empathetic response, so I can resolve the issue efficiently without missing critical details or damaging the relationship."

What makes a good JTBD statement?

The format is: "When [situation], I want to [motivation], so I can [desired outcome]." Notice there is no mention of AI, a copilot, or any specific technology. The job statement is deliberately solution-agnostic. It describes the user's goal, not the tool. This ensures your product decisions stay grounded in real user needs rather than technology enthusiasm.

Create a Heading 1 titled Four Forces of Progress.
Below it, create four sub-sections using Heading 2 blocks (type /h2 and press Enter for each):

Push (Current Pains) - Write 3-4 bullet points about what frustrates CSMs today. Examples: "Spending 15+ minutes re-reading ticket history before responding," "Missing context from previous interactions leads to embarrassing repeat questions," "Copy-pasting from knowledge base articles feels robotic and slow."
Pull (Attractions of AI Copilot) - Write 3-4 bullet points about what excites CSMs about the copilot. Examples: "Instant context summary means I can respond in minutes instead of 15+," "Suggested drafts give me a starting point so I'm not staring at a blank screen."
Anxiety (Fears About AI) - Write 3-4 bullet points about fears. Examples: "What if the AI sends something wrong to my biggest client?" "Will my manager think I'm lazy if I use AI drafts?" "Will this eventually replace my job?"
Habit (Inertia of Current Workflows) - Write 3-4 bullet points about existing routines that resist change. Examples: "I already have my own templates saved," "My current process works, even if it's slow," "I'd have to learn a new tool."

What are the Four Forces of Progress?

The Four Forces model explains why people switch (or don't switch) to new solutions. Push and Pull drive adoption. Anxiety and Habit resist it. For AI products, the Anxiety force is especially powerful because users fear unpredictable outputs reaching their customers.

Understanding all four forces helps you design features that reduce anxiety (like confidence indicators) and overcome habit (like seamless integration into existing workflows).

Use Notion's bulleted list shortcut

Type - followed by Space to quickly create bulleted list items for each force. This keeps your content scannable and mirrors how you would present it in a real strategy document.

Define desired outcomes and trust UX requirements

Desired outcomes translate fuzzy user goals into measurable targets. For AI products, you also need to define what trust looks like in the interface. This is because users cannot inspect an AI's reasoning the way they inspect a spreadsheet formula.

Create a Heading 1 titled Desired Outcomes.
Write 5 measurable outcomes using the format: [Minimize/Maximize] + [Metric] + [Context].
Add these as a numbered list below the heading:

"Minimize the time to understand full ticket context from 15 minutes to under 2 minutes."
"Minimize the risk of sending an inaccurate response to an enterprise client."
"Maximize the percentage of responses that require only minor edits before sending."
"Minimize the time between receiving a ticket and sending a first response."
"Maximize CSM confidence that AI-surfaced KB articles are relevant to the specific ticket."

Why Minimize/Maximize format?

This outcome format comes from Tony Ulwick's Outcome-Driven Innovation methodology. By starting each outcome with a direction (minimize or maximize), you create outcomes that are inherently measurable. You can later survey users: "How satisfied are you with the time it takes to understand ticket context?" This directly feeds your evaluation framework in Step 4.

Create a Heading 1 titled Trust UX Requirements.
Below it, create four Heading 2 sub-sections and fill each with a description paragraph and 2-3 bullet points of specific requirements:

Confidence Indicators - Visual signals showing AI confidence level. Examples: "Color-coded confidence bar on each draft (green/yellow/red)," "Explicit 'I'm less certain about this section' callout when the AI lacks source material."
Source Citations - Linking AI outputs back to knowledge base articles and ticket history. Examples: "Every sentence in the draft links to the specific KB article or past ticket it drew from," "Hovering a citation shows a preview of the source."
Override Mechanisms - Easy edit, reject, and regenerate controls. Examples: "One-click 'regenerate' button," "Inline editing without leaving the copilot pane," "'Ignore this suggestion' with optional feedback."
Failure Mode UX - What happens when the AI produces an irrelevant or wrong draft. Examples: "The copilot says 'I don't have enough context for this ticket' rather than guessing," "If confidence is below threshold, show a manual template instead."

Why is trust UX critical for AI products?

Traditional software is deterministic. If you click a button, you know what will happen. AI outputs are probabilistic. The same input can produce different outputs, and some will be wrong. Trust UX is the design layer that helps users navigate this uncertainty. Without it, users either over-trust (sending bad drafts to clients) or under-trust (never adopting the tool).

Create a Heading 1 titled Emotional & Social Jobs.
Write a paragraph explaining the emotional jobs: "CSMs want to feel confident (not anxious about AI mistakes reaching customers) and be perceived as knowledgeable by their clients (not as someone who blindly sends AI-generated text)."
Add 2-3 bullets exploring the social dimension: How will colleagues perceive AI usage? What does "being good at your job" mean if an AI does the drafting?

What are emotional and social jobs?

JTBD goes beyond functional tasks. The emotional job is how the user wants to feel (confident, in control). The social job is how they want to be perceived (knowledgeable, thoughtful). AI products often fail the social job because users worry that relying on AI makes them look lazy or unskilled. Your product design must address this by positioning the AI as a tool that amplifies expertise rather than replacing it.

Map user-centered design principles for AI

Design principles act as a decision filter. When your team debates a UX choice later, these principles resolve the argument without you in the room.

Create a Heading 1 titled Trust Design Principles.
Write 4-5 principles as numbered items. Each principle should be a single declarative sentence followed by a brief explanation of what it means in practice.
Use these principles:

"The CSM always reviews before sending (human-in-the-loop)." This means the copilot never auto-sends. Every draft requires explicit CSM approval.
"Confidence signals are visible on every AI output." No AI-generated text appears without an accompanying confidence indicator.
"The copilot explains its reasoning (source transparency)." Users can always trace an output back to a specific source document.
"Failure modes are graceful (the copilot says 'I'm not sure' rather than hallucinating)." When confidence is low, the system communicates uncertainty rather than guessing.
"AI augments expertise, it does not replace judgment." The copilot is positioned as a research assistant, not an autonomous agent.

Create a Heading 1 titled Progressive Disclosure Design.
Below it, write a paragraph explaining the concept: progressive disclosure means showing only the most essential information first, then revealing more detail on demand.
Create a table or structured list describing three layers of information:

Immediately visible - The draft text and confidence indicator. This is what the CSM sees the moment they open the copilot.
On hover - Source citations and links to the specific KB articles or ticket history the draft drew from.
On expand - Full reasoning chain. Why did the copilot choose this approach? What alternative responses were considered? What was excluded and why?

Why progressive disclosure matters for AI

AI outputs carry more context than traditional UI elements. If you show everything at once (the draft, the sources, the confidence score, the reasoning, the alternatives), you overwhelm the user. Progressive disclosure lets power users dig deep while casual users stay in flow. It is the same principle behind how search engines show a snippet first and the full page on click.

Review your completed page from top to bottom. You should see six major sections: Job Performer Definition, Core Job Statement, Four Forces of Progress, Desired Outcomes, Trust UX Requirements, Emotional & Social Jobs, Trust Design Principles, and Progressive Disclosure Design.

Page looking cluttered?

If your page feels long, you can convert any Heading 1 into a toggle heading. Hover over the heading, click the ⋮⋮ icon in the left margin, select Turn into, then choose Toggle heading 1. This collapses sections so you can focus on one at a time.

If sections are missing, check that you created each Heading 1 using /h1 rather than just bolding text. Bold text looks similar but does not create navigable structure.

Help me organize my Notion page structure for this problem discovery document.

Your problem discovery document is complete. You now have the strategic foundation that every feature decision will trace back to. Next up, you will use this discovery work to score and prioritize specific AI features in a structured database.

Build the AI Feature Prioritization Scorecard

Your problem discovery work identified what CSMs need and why they need it. Now you need to decide which AI features to build first.

Standard prioritization frameworks like RICE don't account for AI-specific variables. Data readiness, model uncertainty, and privacy constraints all change which features are viable. In this step, you'll score candidate features using a 6-layer AI prioritization framework that includes privacy and security as first-class gates.

In this step, get ready to:

Score 6-8 candidate features on the AI prioritization framework.
Apply privacy and security gates to identify blockers.
Calculate priority scores and create a ranked backlog with rationale.

Why a 6-layer framework instead of RICE?

RICE (Reach, Impact, Confidence, Effort) treats all features equally. AI features are different. A feature might score high on impact but be impossible to build because the training data doesn't exist, or illegal to ship because it requires sending PII to a third-party model.

The 6-layer framework adds AI-specific gates (Problem-AI Fit, Data Readiness, Privacy Risk) so you can eliminate non-viable features early. This is how experienced AI PMs avoid wasting engineering cycles on features that will never pass a privacy review.

Score features using the 6-layer AI prioritization framework

Your database has the properties ready. Now you'll populate it with candidate features and score each one across the framework's layers.

Switch back to your Feature Prioritization sub-page in Notion.
In the database you created earlier, click into the first empty row to add a new entry.
Add the following 6 features as rows, typing each into the Feature Name column:

Draft response to customer ticket
Summarize ticket history
Surface relevant KB articles
Auto-categorize ticket urgency
Detect customer sentiment shift
Generate internal handoff notes

Add more if you'd like

You can add 1-2 more features of your own. Think about what else a CSM might need from an AI assistant. The framework works best with 6-8 candidates to compare.

For each feature, score the Problem-AI Fit column. Set it to Pass if the feature genuinely requires AI (language understanding, pattern recognition, generation). Set it to Fail if a rule-based system, filter, or simple workflow could handle it.
Score Data Readiness (1-5) for each feature. Consider: does Arcadia have enough labeled ticket data? Is the knowledge base structured enough for retrieval? A score of 5 means data is clean and plentiful. A score of 1 means the data barely exists.
Score UX Trust (1-5) for each feature. Consider: how confident will a CSM feel using this AI output with an enterprise customer? A draft email (high stakes) scores lower than an internal summary (low stakes).
Score Competitive Moat (1-5) for each feature. Consider: does this feature create lasting competitive advantage using Arcadia's proprietary data, or could any competitor replicate it with a generic API call?

What does each score represent?

Problem-AI Fit is a hard gate. If a feature fails, it should not be built with AI regardless of how cool it sounds. Auto-categorize ticket urgency, for example, might fail this gate if simple keyword rules already achieve 95% accuracy.

The numeric scores (1-5) are judgment calls. Use your problem discovery work to inform them. Features that align closely with the desired outcomes you documented will naturally score higher on UX Trust because you've already mapped what CSMs need to feel confident.

Your database should now show all 6+ features with scores filled in for Problem-AI Fit, Data Readiness, UX Trust, and Competitive Moat.

Apply the privacy and security gate

Privacy is not a nice-to-have for enterprise AI. A single PII leak to a customer can end a contract. This gate determines which features are even legal and safe to ship.

Open each feature row in your database by clicking on it.
Inside each feature's page, add a new text section titled Data Flow Assessment.
For each feature, document what customer data touches the LLM. Consider: ticket text, customer names, account details, contract values, internal employee notes.

Here's how to think about each feature's data flow:

Draft response to customer ticket: ticket text (contains customer names, account details), KB articles, previous response history.
Summarize ticket history: full ticket thread (names, emails, issue details, potentially contract references).
Surface relevant KB articles: search query from ticket context, KB article content (internal documentation).
Auto-categorize ticket urgency: ticket subject and body text (may contain names and account identifiers).
Detect customer sentiment shift: historical ticket threads, tone patterns, possibly financial data tied to churn signals.
Generate internal handoff notes: full account context, internal notes, possibly contract values and financial details.

Based on your data flow assessment, score the Privacy Risk select property for each feature:
Low: No PII in prompts sent to the model.
Medium: PII is present but can be masked before sending to the model.
High: PII is required for the feature to work correctly.
Blocker: Regulatory prohibition (e.g., sending financial data to a third-party model).

Below the Data Flow Assessment in each feature page, add a Security Considerations section covering three risks:

Prompt injection: Could malicious content in a customer ticket manipulate the copilot's output?
Data residency: Do EU customers require that their data stays in EU-based infrastructure?
Model provider data policies: Does the LLM provider train on your customer data?

What makes a feature a Blocker?

Any feature scored Blocker on Privacy Risk is automatically deprioritized regardless of how well it scores on other dimensions. For example, if "Detect customer sentiment shift" requires sending financial churn data to a third-party model, it is a Blocker even if its Competitive Moat score is 5.

This mirrors how real AI product teams work. Legal and privacy reviews have veto power over the backlog.

Calculate priority scores and create the ranked backlog

With all your scores in place, you'll now configure the formula that calculates each feature's overall priority. Then you'll sort your database and write your strategic rationale.

Click on the Priority Score property header in your database.
Select Edit property, then set the property type to Formula if it is not already.
Paste the following formula into the formula editor:

if(prop("Privacy Risk") == "Blocker", 0, if(prop("Problem-AI Fit") == "Pass", prop("Data Readiness") + prop("UX Trust") + prop("Competitive Moat"), 0))

What does this formula do?

The formula applies two gates in sequence. First, if Privacy Risk is Blocker, the score is automatically 0. Second, if Problem-AI Fit is Pass, it sums Data Readiness + UX Trust + Competitive Moat (max possible score: 15). If Problem-AI Fit is Fail, the score is 0.

This means a feature must clear both the privacy gate AND the AI-fit gate before its numeric scores even matter. Features that fail either gate are immediately ranked at the bottom.

Verify your formula is working by checking that features with Fail on Problem-AI Fit or Blocker on Privacy Risk show a Priority Score of 0.
Sort your database by Priority Score in descending order. Click the Sort button above your database, select Priority Score, and choose Descending.

Seeing errors in your formula?

Make sure your property names match exactly. The formula references Privacy Risk, Problem-AI Fit, Data Readiness, UX Trust, and Competitive Moat. If you named them differently in Step 1, update the formula to match your exact property names.

Also check that Privacy Risk and Problem-AI Fit are Select type properties (not text), and that Data Readiness, UX Trust, and Competitive Moat are Number type properties.

Help me fix my Notion Priority Score formula.

Below your sorted database, add a new text block titled Top 3 Rationale.
Write 2-3 sentences for each of your top 3 features explaining why they rank highest and what makes them ship-ready versus what still needs investigation.

Your rationale should address both the scoring logic and your strategic judgment. For example, a feature might score highest because it has strong data readiness AND low privacy risk, making it the safest first bet.

What makes a strong rationale?

A strong rationale connects scores back to your JTBD discovery. Your top feature should directly address the core job statement you wrote earlier. Name the specific data sources that make it ready, the trust UX patterns that make it safe, and any remaining unknowns that need validation before engineering starts.

This rationale is what you'd present to an engineering lead to justify why Feature A gets resources before Feature B. It demonstrates AI PM judgment beyond just running a formula.

Your prioritization scorecard is complete. You now have a ranked backlog that accounts for AI-specific constraints that standard frameworks miss. Next up, you'll design the evaluation framework that defines what "good enough to ship" actually means for your top feature.

Design the Evaluation Framework & Launch Criteria

Your prioritization database now ranks every candidate feature by its readiness to ship. But a high priority score does not mean a feature is ready to go live.

AI products are probabilistic. A response-drafting copilot might produce brilliant outputs 85% of the time and hallucinate dangerously the other 15%. In this step, you will define exactly what "good enough to ship" means for your top feature, design the tests that prove it, and set the governance rules that keep customer data safe.

In this step, get ready to:

Define quality, safety, and latency thresholds for your top-priority feature.
Document data governance policies and the precision/recall trade-off decision.
Create a ship/no-ship launch criteria checklist with a post-launch monitoring plan.

Design the evaluation framework

An evaluation framework answers one question: how do you know the AI output is good enough? Without explicit thresholds, teams ship features that feel impressive in demos but fail under real-world pressure.

Switch back to your Notion workspace and open the Evaluation Framework sub-page.
Create a heading called Launch Thresholds.
Below the heading, define what "good enough to ship" means for your top-priority feature (draft response to customer ticket). Structure it as three thresholds:

Quality threshold: 90% of drafts rated "usable with minor edits" by CSMs in blind evaluation.
Safety threshold: 0% of drafts contain hallucinated policy information. Less than 2% contain PII leakage.
Latency threshold: Draft appears within 3 seconds of request.

Why three separate thresholds?

Quality tells you the feature is useful. Safety tells you it will not harm the customer relationship. Latency tells you it fits into the CSM's real-time workflow.

A single "accuracy" metric hides dangerous trade-offs. Separating them forces your team to meet all three bars independently.

Below the thresholds, add a section called Golden Test Set.
Describe a set of 50-100 representative tickets across four categories: billing disputes, technical issues, feature requests, and escalations.
For each category, note that the test set includes expected "good" responses that will serve as ground-truth comparisons during evaluation.

What is a golden test set?

A golden test set is a curated collection of inputs with known-good outputs. You run the AI against these inputs before every release. If scores drop, the model has regressed and should not ship.

Spreading tickets across four categories ensures the copilot is tested against the full range of real-world scenarios, not just easy ones.

Add a final section under the framework called Evaluation Methods.
Define three complementary methods:

Automated metrics: BLEU/ROUGE scores against reference responses in the golden test set.
LLM-as-judge: A second model evaluates quality, tone, and accuracy of the copilot's outputs.
Human evaluation sampling: CSMs rate 5% of outputs weekly to catch issues automated metrics miss.

Why three evaluation methods?

Automated metrics are fast and cheap but miss nuance. LLM-as-judge catches subtler quality issues at scale. Human evaluation is expensive but catches edge cases that neither automated approach detects.

Layering all three gives you speed for daily checks, breadth for weekly reviews, and depth for strategic decisions.

Define data governance and safety requirements

Your copilot will process sensitive customer data every time it drafts a response. Without explicit governance rules, teams accidentally send PII to model providers or retain data longer than regulations allow.

Create a new heading on the same page called Data Governance Policy.
Document what data enters the model: ticket text and KB articles.
Document what data is excluded: financial details, contract values, and internal employee notes.
Define the PII handling rule: customer names and emails are masked before sending to the LLM.
Set the data retention policy: prompts and responses are logged for evaluation but purged after 30 days.

Why 30-day retention?

You need enough history to detect model drift and investigate incidents. But retaining customer data indefinitely creates liability under data protection regulations.

30 days gives your team four weekly evaluation cycles to spot regressions before the data is purged.

Below the governance policy, add a section called Model Provider Assessment.
Document three assessment questions:

Does the provider train on your data? (If yes, what opt-out mechanisms exist?)
What is their data residency? (EU customers may require data stays in EU.)
What happens during an outage? (Graceful degradation: copilot feature disappears, CSMs work manually.)

Add a final section here called Precision/Recall Trade-off Decision.
Write the core question: For this copilot, is it worse to produce a bad draft (false positive, damages customer trust) or to refuse to draft (false negative, CSM writes manually)?
Document the answer: False positives are more costly. The system should have high precision even at the cost of lower recall.
Add the reasoning: It is better for the copilot to say "I can't help with this one" than to produce a risky draft that damages an enterprise relationship.

What is the precision/recall trade-off?

Precision measures how often the AI's outputs are correct when it does produce one. Recall measures how often the AI produces an output when it could have.

High precision, lower recall means the copilot drafts less often but rarely produces something bad. High recall, lower precision means the copilot always tries to help but sometimes produces harmful outputs. For customer-facing text, precision wins.

Specify launch criteria as ship/no-ship gates

Thresholds only matter if someone enforces them. A ship/no-ship checklist turns your evaluation framework into an actionable gate that prevents premature launches.

Create a new heading on the same page called Ship/No-Ship Launch Criteria.
Add a checklist with these five gates:

Quality gate passed: Golden test set scores above the 90% usable-draft threshold.
Safety gate passed: 0 PII leakage incidents in a 1000-sample test.
Privacy review complete: Legal sign-off on the documented data flow.
User acceptance test: 10 CSMs use the copilot for 2 weeks. 80% report measurable time savings.
Rollback plan documented: Feature can be disabled in under 5 minutes if a critical issue surfaces.

Why include a rollback plan as a gate?

AI models can degrade unpredictably after deployment. A model that passed all pre-launch tests might encounter input patterns in production that trigger new failure modes.

Having a documented kill switch means your team can respond in minutes rather than scrambling during an incident.

Below the checklist, add a section called Post-Launch Monitoring Plan.
Define the weekly review cadence with these leading indicators:

Weekly quality sampling against the golden test set.
User override rate tracking. If CSMs reject or heavily edit more than 40% of drafts, trigger an investigation.
Cost-per-inference tracking to catch unexpected spend increases.
Latency percentile monitoring (p50 and p95) to detect performance degradation.

Why 40% as the override threshold?

If CSMs are rewriting more than 4 out of every 10 drafts, the copilot is creating work rather than saving it. The feature becomes a hindrance rather than a tool.

This threshold acts as an early warning system. It triggers investigation before user trust erodes enough to cause abandonment.

Connect monitoring to your metrics tree

If you complete the Secret Mission later, your monitoring indicators here will map directly to Layer 3 and Layer 4 of the AI metrics tree. Designing them now gives you a head start.

Your evaluation framework now defines exactly what "ready to ship" means. No ambiguity, no hand-waving, no shipping on vibes. Next up, you will build a tri-track roadmap that sequences when each capability, experience, and infrastructure milestone lands.

Build the Tri-Track AI Product Roadmap

Your evaluation framework defines what "good enough" looks like for shipping AI features. Now you need to sequence when things ship and how each track of work depends on the others.

Traditional roadmaps sequence features against calendar dates. AI roadmaps are different because model capability timelines are inherently uncertain. You cannot promise "response drafting ships March 15" when the model's quality score might plateau at 85% instead of 90%. The tri-track roadmap structure solves this by separating deterministic work (UX, infrastructure) from probabilistic work (model performance) and using confidence levels instead of fixed dates for AI capabilities.

In this step, get ready to:

Build a tri-track roadmap structure with three horizons per track.
Populate the roadmap with confidence-based capability milestones and a data governance lane.
Create stakeholder-specific views for engineering and customer-facing teams.

Build the tri-track roadmap structure

A tri-track AI roadmap separates your work into three parallel streams. Each stream has its own pace and level of predictability.

Switch back to your Notion workspace and open the "Product Roadmap" sub-page.
Create three sections (or a database with a "Track" property): Track 1 - Model & AI Capabilities, Track 2 - Product Experience (UX, guardrails, trust features), and Track 3 - Infrastructure & Evaluation (data pipelines, monitoring, safety).
Within each track, create three sub-sections: Horizon 1 (0-3 months, ship-ready), Horizon 2 (3-6 months, prototyping), and Horizon 3 (6-12 months, strategic bets).
Add a note at the top of Track 1 explaining the key difference from traditional roadmaps: Track 1 uses confidence ranges instead of fixed dates (e.g. "70% likely to ship Q3, 90% by Q4") because model performance is probabilistic.

Why three tracks instead of one?

Track 2 (Product Experience) and Track 3 (Infrastructure) are largely deterministic. You can assign dates to them because building a UI or deploying a monitoring pipeline follows predictable engineering timelines.

Track 1 (Model & AI) is probabilistic. You cannot guarantee when the model will hit a 90% quality threshold. Separating the tracks prevents false-precision dates from infecting your entire roadmap.

Populate the roadmap with capability milestones

Each horizon item describes a capability outcome, not a feature name. The earlier horizons are specific and measurable. The later horizons are framed as bets with explicit confidence levels.

Populate Track 1 (Model & AI Capabilities) with these milestones:

H1: "Achieve 90% usable-draft rate on ticket response generation using retrieval-augmented generation with the company's KB."
H2: "Expand to multi-turn conversation summarization (confidence: Medium)."
H3: "Proactive anomaly detection - surface accounts showing early churn signals (confidence: Low, capability bet)."

Populate Track 2 (Product Experience) with these milestones:

H1: "Ship response drafting with confidence indicators, source citations, and one-click edit/send UX."
H2: "Add ticket history summarization with progressive disclosure."
H3: "Build AI coaching mode that suggests response improvements rather than full drafts."

Populate Track 3 (Infrastructure & Evaluation) with these milestones:

H1: "Deploy evaluation pipeline (golden test set + weekly human review), PII masking layer, and cost monitoring."
H2: "Build automated drift detection and model regression alerts."
H3: "Implement A/B testing infrastructure for model upgrades."

Add a Data Governance Lane within Track 3 with its own set of horizons:

H1: "Legal review of data flow, PII masking pipeline, model provider DPA signed."
H2: "EU data residency solution for European customers."
H3: "SOC 2 audit covering AI features."

Add a confidence tag to each roadmap item: High (validated, in development), Medium (prototyped, research positive), or Low (hypothesis only).

Why confidence tags matter for stakeholders

When a stakeholder sees "Low confidence" on a Horizon 3 item, they know not to make downstream plans against it. This prevents the common failure mode where sales promises a feature that engineering is still treating as a research hypothesis.

Create stakeholder-specific views

Different audiences need different slices of the same roadmap. Engineers need dependencies and technical requirements. Customer-facing teams need value framing and rough timing without internal dates.

Open the "Stakeholder Views" sub-page in your workspace.
Create a section titled Engineering View.
In the Engineering View, document Horizon 1 technical requirements and dependencies. Include: KB needs vector embeddings, ticket data needs PII masking pipeline, and evaluation infrastructure is a hard dependency before any model feature ships.
Add a capacity allocation note describing team size and time needed for the H1 deliverables.
Create a second section titled Customer-Facing View.
In the Customer-Facing View, use value-and-timing language without specific dates. Example: "Coming soon: AI-assisted response drafting to help your team respond faster."
Do not include Horizon 3 items in customer-facing communication. Only Horizon 1 and Horizon 2 items with high or medium confidence belong here.

Why exclude Horizon 3 from customers?

Horizon 3 items are capability bets, not commitments. If you share them externally, customers plan around them. When a Low-confidence bet gets killed (as it should be when evidence says stop), you damage trust.

The Engineering View includes all horizons because engineers need full context. The Customer-Facing View stays conservative because external trust is harder to rebuild than internal alignment.

Use Notion database views for stakeholder filtering

If you built your roadmap as a Notion database with a "Track" property, you can create filtered views that show only Engineering-relevant items or Customer-relevant items. This keeps one source of truth with multiple presentation layers.

Review your completed roadmap. You should see three tracks, each with three horizons, confidence tags on every item, a data governance lane within Track 3, and two stakeholder-specific views on the Stakeholder Views page.

Your roadmap now sequences AI capabilities, product experience, and infrastructure as three coordinated streams with honest confidence levels. Next up, you will tackle the Secret Mission: building the executive strategy one-pager and a metrics tree that connects model performance all the way up to business outcomes.

Secret mission

Executive Strategy View & AI Metrics Tree

You've built the strategy, prioritization, evaluation, and roadmap. Now synthesize everything into the artifact that separates senior AI PMs from the rest: an executive one-pager with strategic bets, kill conditions, and a 4-layer metrics tree that traces model performance all the way up to business outcomes.

Clean Up Your Resources

This project runs entirely on Notion's free plan, so there are no ongoing costs or running processes to worry about. Decide whether to keep your workspace as-is, tidy it up, or remove it entirely.

Resources you used:

Notion workspace pages: "Arcadia AI Copilot - Product Strategy" parent page and all sub-pages (Problem Discovery & JTBD, Feature Prioritization, Evaluation Framework, Product Roadmap, Stakeholder Views, Executive Strategy: Arcadia AI Copilot).
Notion account (free plan, no charges).

Keep everything running

No action needed. Choose this if you plan to continue refining your AI product strategy workspace or want to reference it during interviews.

Your Notion workspace stays as-is with all pages and databases intact.
The free plan has no expiration, so your content remains accessible indefinitely.
Your Notion account remains active. You can use it for other projects or as a portfolio workspace.

Pause - I'll come back to this later

There are no running processes or costs to pause. If you want a tidier workspace sidebar while you work on other things, you can move the parent page out of your top-level navigation.

In Notion, hover over the "Arcadia AI Copilot - Product Strategy" page in your sidebar.
Click the three-dot menu icon that appears, then select a section lower in your sidebar to move it out of your primary view.
Your pages and databases remain fully intact and searchable whenever you want to return.

Delete - I don't want to use this again

Remove all project pages from your Notion workspace. This moves them to Trash, where they remain recoverable for 30 days before permanent deletion.

In Notion, hover over the "Arcadia AI Copilot - Product Strategy" page in your sidebar.
Click the three-dot menu icon, then select Move to Trash. This deletes the parent page and all sub-pages (Problem Discovery & JTBD, Feature Prioritization, Evaluation Framework, Product Roadmap, Stakeholder Views).
If you also created the "Executive Strategy: Arcadia AI Copilot" page as a separate top-level page, repeat the same process for that page.
To confirm deletion, click Trash in your sidebar and verify your project pages appear there.

Nice Work!

You just built a complete AI product strategy workspace from the ground up. This is the exact artifact set that AI PM hiring managers evaluate in 2026 interviews.

Here's what you accomplished:

Built a JTBD problem discovery document with a job statement, four forces analysis, measurable desired outcomes, and trust UX requirements for an AI copilot.
Created an AI feature prioritization database with a 6-layer scoring framework and privacy/security gates that automatically deprioritize features with regulatory risk.
Built a tri-track AI product roadmap with confidence-based milestones, a data governance lane, and stakeholder-specific views for engineering and customer-facing teams.
Secret Mission: Created an executive strategy one-pager with strategic bets and kill conditions, plus a 4-layer AI metrics tree connecting model performance to business outcomes.

Ready to quiz yourself?