Scaffold an AI Portfolio with LLM Spans

Build a versioned LLM span schema with OTel GenAI conventions and pytest validation.

Introduction

30 Second Summary

Every time you ask an AI model a question, dozens of data points are generated: which model responded, how many tokens it consumed, how long it took, and whether it finished cleanly. Without a schema to structure that data, building cost dashboards, SLO boards, or evaluation gates on top of it becomes guesswork.

In this project, you will scaffold a production-grade portfolio repository and design a versioned LLM span schema grounded in OpenTelemetry GenAI semantic conventions. You will write an RFC, create the JSON Schema, and validate it against three LLM providers with pytest before a single line of infrastructure code exists.

What You'll Build

A public GitHub repository that serves as the foundation for a five-system AI cloud engineering portfolio, starting with a complete RFC, a JSON Schema (Draft 2020-12) for LLM span data, and a test suite that validates provider-specific example spans.

By the end of this project, you'll have:

  • A versioned LLM span schema (v0.1.0) mapped to OTel GenAI semantic conventions, with deliberate divergences documented in an RFC.
  • A pytest suite validating sample spans for Anthropic, OpenAI, and AWS Bedrock against the JSON Schema, proving the schema works across providers before any infrastructure lands.
  • A production-grade repository with ADRs, architecture docs, build logs, operational stubs, and a data-handling declaration that establishes the pattern for all future systems.
  • A negative-test fixture that asserts the schema correctly rejects malformed spans with missing fields, wrong types, and invalid provider names.

Are there any prerequisites?

This project assumes fluency with AWS fundamentals, Terraform, Kubernetes basics, OpenTelemetry at the spec level, Python 3.12+, and GitHub repo conventions. It does not teach any of those topics. You should also have git installed and a GitHub account ready to go.

Before We Start

You are scaffolding a portfolio repository and writing the foundational LLM span schema that every downstream system in this portfolio will consume. The cost dashboard, SLO board, eval gate, and drift detector all depend on this schema as their shared contract.

This schema is a one-way-door decision. Once downstream systems depend on its mandatory fields, changing those fields is a breaking change that cascades across the entire portfolio. That is why you are writing an RFC and validating with tests before any infrastructure exists.

The schema is grounded in OpenTelemetry GenAI semantic conventions but serves a different purpose. OTel semconv tells instrumentation libraries which attributes to set on a span. Your schema defines a portable JSON envelope that analytical consumers can validate and parse predictably after export.

Initialize the Repository and Install Dependencies

Every system in this portfolio will depend on a single repository with a consistent structure. Before you write a single line of schema or RFC, you need the repo itself, the directory layout that future systems will slot into, and the Python libraries that power your test suite.

In This Step, Get Ready To:
  • Create and clone a public GitHub repository with license, .gitignore, and top-level files.
  • Build the full directory structure covering ADRs, architecture docs, build logs, and the first system.
  • Install jsonschema and pytest as Python dependencies and verify the installation.
Create the GitHub Repository

This repo is the single source of truth for all five portfolio systems. Making it public from day one means every commit, RFC, and test result is visible to anyone reviewing your work.

  • Go to github.com/new in your browser.
  • Set the repository name to ai-cloud-systems-engineering.
  • Set visibility to **Public**.
  • Check **Add a README file**.
  • Select **MIT License** from the license dropdown.
  • Click **Create repository**.

Now clone the repo to your local machine.

  • Open your terminal and run the clone command.
git clone https://github.com/[[GITHUB_USERNAME="your-github-username"]]/ai-cloud-systems-engineering.git
cd ai-cloud-systems-engineering

Next, create the .gitignore file. This keeps secrets, Terraform state, and Python cache files out of version control.

  • Create a file called .gitignore in the repo root with the following contents.
# Python
__pycache__/
*.pyc
.pytest_cache/
venv/
.venv/

# Terraform
.terraform/
*.tfstate
*.tfstate.backup

# Secrets and credentials
.env
secrets/
*.pem
*.key

Why such an aggressive .gitignore?

Future systems in this portfolio will use Terraform and environment variables with secrets. Excluding .terraform/, *.tfstate, .env, and *.pem now prevents accidental credential leaks before any infrastructure exists.

Finally, create the cert-proof.md placeholder file. You will populate this fully in the next step.

  • Create an empty file called cert-proof.md in the repo root.
touch cert-proof.md
Create the Full Directory Structure

A portfolio repo is not a single project. It is a system of systems. The directory layout you create now defines where every future artifact lives: ADRs, architecture diagrams, RFCs, schemas, tests, scripts, and build logs. Getting this right means every future system slots in without restructuring.

  • Run the following command to create all directories in one shot.
mkdir -p adrs \
  architecture \
  systems/01-ai-observability-platform/rfcs \
  systems/01-ai-observability-platform/tests \
  systems/01-ai-observability-platform/schemas \
  systems/01-ai-observability-platform/scripts \
  systems/01-ai-observability-platform/measurements \
  archive \
  publications \
  retrospectives \
  build-log/2026/Q2

What does this command do?

The mkdir -p flag creates parent directories as needed. A single command builds the entire tree. The systems/01-ai-observability-platform/ path is the first of five numbered systems. Each future system (02 through 05) will follow the same subdirectory pattern: rfcs/, tests/, schemas/, scripts/, measurements/.

Git does not track empty directories. You need placeholder files so the structure survives a commit.

  • Create README stubs in each top-level directory.
echo "# Architecture Decision Records" > adrs/README.md
echo "# Architecture Overview" > architecture/overview.md
echo "# Publications" > publications/README.md
echo "# Retrospectives" > retrospectives/README.md
  • Verify the directory tree looks correct.
find . -type d | sort

You should see all the directories you just created, including the nested paths under systems/01-ai-observability-platform/.

Install Python Dependencies

The test suite you will write in Step 5 depends on two libraries. jsonschema validates JSON data against a JSON Schema definition. pytest runs the tests themselves. Pinning these in a requirements.txt makes the environment reproducible for anyone who clones the repo.

  • Create a requirements.txt file in the repo root with the following contents.
jsonschema>=4.26.0
pytest>=9.0.0

Why jsonschema instead of Pydantic?

The LLM span schema you are building is a language-agnostic contract. Future systems may consume it from Go, Rust, or TypeScript. A standalone .json schema file can be validated by any language with a JSON Schema library. Pydantic models are Python-only. The jsonschema library is used here as one consumer of that schema, not as the schema definition itself.

  • Install the dependencies.
pip install -r requirements.txt
  • Verify that jsonschema installed correctly.
python -c "import jsonschema; print(jsonschema.__version__)"

You should see version 4.26.0 or higher printed to the terminal.

I see a version number

You are all set. Both libraries are installed and ready for the test suite you will write in Step 5.

I see an error

Having trouble with the install?

If you see ModuleNotFoundError, make sure you are running the same Python interpreter that pip installed to. Try python3 -c "import jsonschema; print(jsonschema.__version__)" instead. The jsonschema library requires Python 3.10 or higher.

If you are using a virtual environment, make sure it is activated before running pip install. Ask NextWork for help debugging your Python environment.

  • Also verify pytest is available.
pytest --version

You should see pytest 9.0.0 or higher.

Your repo is initialized, the directory structure is in place, and your Python dependencies are installed. Next up, you will write the README, build log, and cert proof that give the repo its narrative.

Write the README, Build Log, and Cert Proof

Your repo structure is in place. Now you need to fill in the documents that make this portfolio legible to anyone who lands on it. A bare directory tree tells a reviewer nothing about your intent, your trajectory, or what differentiates you from every other GitHub profile with a terraform-aws-* repo.

In this step, you will write the repo-level README with a clear differentiation hook, create a certification proof document, and start the build log that tracks every decision you make across the portfolio.

In This Step, Get Ready To:
  • Write a repo-level README.md with a 20-second sell and placeholder sections for all five portfolio systems.
  • Create cert-proof.md listing six target certifications with status tracking.
  • Write the first build-log entry recording today's scaffold work and decisions.
Write the Repo-Level README

A portfolio README has a different job than a library README. It is not documenting an API. It is making a case for you as an engineer in under 20 seconds. The reader is a hiring manager or a peer reviewer scanning your GitHub profile. They will decide whether to keep reading based on the first paragraph.

  • Open README.md at the repo root in your editor.
  • Replace any placeholder content with the following structure.
# AI Cloud Systems Engineering

Five production-grade systems demonstrating AI infrastructure at scale.
I'll show you the cost dashboard, the SLO board, and the eval gate. Not just a GitHub link.

## Who This Is For

Hiring managers and engineering leads evaluating hands-on AI infrastructure skills.
Every system in this repo has an RFC, a schema, passing tests, and measurable outcomes.

## Featured Systems

| # | System | Status | Headline Metric |
|---|--------|--------|-----------------|
| 01 | AI Observability Platform | In Progress | Reduced LLM cost anomaly detection MTTR from 4h to 15min |
| 02 | Cost Intelligence Dashboard | Planned | TBD |
| 03 | SLO Board | Planned | TBD |
| 04 | Eval Gate | Planned | TBD |
| 05 | Drift Detector | Planned | TBD |

## Repository Structure

```
adrs/                  # Architecture Decision Records
architecture/          # Cross-system architecture docs
build-log/             # Daily engineering journal
publications/          # Blog posts and writeups
retrospectives/        # Post-iteration retrospectives
systems/               # Each system lives here
  01-ai-observability-platform/
    rfcs/
    schemas/
    tests/
    scripts/
    measurements/
```

## Certifications

See [cert-proof.md](cert-proof.md) for current certification status.

## Build Log

See [build-log/](build-log/) for the daily engineering journal.

Why lead with a differentiation hook?

The line "I'll show you the cost dashboard, the SLO board, and the eval gate" is not decoration. It signals that this portfolio produces observable, measurable systems. Most portfolio repos are static code dumps. This one promises live artifacts a reviewer can inspect.

Why a featured systems table?

The table gives the reader a map of the entire portfolio at a glance. System 01 is the only row with real content right now. As you build each system in future iterations, you will update the status and fill in the headline metric. The "Planned" rows show scope and ambition without overpromising.

Write the Certification Proof Document

Certification claims without proof are noise. The cert-proof.md file creates a single place to track certification status, link to verification URLs, and record completion dates. This makes the claim auditable.

  • Open cert-proof.md at the repo root.
  • Add the following content.
# Certification Proof

All certifications listed below are either completed or actively in progress.
Verification links will be added as each certification is earned.

| Certification | Provider | Status | Target Date | Verification |
|---------------|----------|--------|-------------|--------------|
| Solutions Architect Professional (SAP) | AWS | In Progress | TBD | - |
| AI Engineer Associate (AI-102) | Azure | In Progress | TBD | - |
| Professional Machine Learning Engineer | Google Cloud | In Progress | TBD | - |
| Generative AI Developer - Professional | AWS | In Progress | TBD | - |
| Terraform Associate | HashiCorp | In Progress | TBD | - |
| Certified Kubernetes Administrator (CKA) | CNCF | In Progress | TBD | - |

Pro tip: update this as you go

Each time you pass a certification, update the status to "Completed", fill in the date, and add the verification URL. A reviewer seeing "Completed" with a live verification link trusts it instantly. A reviewer seeing "AWS SAP" in a LinkedIn headline with no proof does not.

Write the First Build-Log Entry

The build log is your engineering journal. Each entry records what you shipped, what decisions you made, and what comes next. Over time, this log becomes evidence of sustained, deliberate work. It also forces you to articulate decisions while they are fresh.

  • Open the file at build-log/2026/Q2/2026-04-20.md that you created in Step 1.
  • Add the following content, adjusting the date if yours differs.
# Build Log: 2026-04-20

**Duration:** ~60 min

## What Shipped

- Repository scaffold with full directory structure
- RFC-001: LLM Span Schema (Draft)
- JSON Schema v0.1.0 for LLM span envelope
- pytest fixture validating Anthropic, OpenAI, and AWS Bedrock spans
- Operational stubs: spinup.sh, teardown.sh, cost-check.sh
- Data-handling declaration (Class 0: synthetic data only)

## Decisions Made

- **Schema over Pydantic**: Used JSON Schema (Draft 2020-12) so non-Python consumers can validate spans. Pydantic is Python-only. The schema is the contract, Python is just one consumer.
- **One-way-door framing**: Mandatory schema fields are a one-way door. Once downstream systems depend on them, changes are breaking. RFC forces the decision through review before it becomes load-bearing.
- **gen_ai.provider.name not gen_ai.system**: OTel renamed this attribute in June 2025. Using the current name, not the deprecated one.
- **Flat dotted keys**: OTel attributes use dots in names (gen_ai.provider.name). These are flat JSON keys, not nested objects. Matches OTel export format.

## Next Action

- Iteration 02: Lay the Terraform AWS foundation. Provision the infrastructure this schema will flow through.

What makes a good build-log entry?

Notice the "Decisions Made" section records the reasoning, not just the outcome. Six months from now, you will not remember why you chose JSON Schema over Pydantic. The build log preserves that context. It also gives a reviewer direct evidence that you think about trade-offs, not just implementations.

Why record gen_ai.provider.name vs gen_ai.system?

The OTel GenAI semantic conventions renamed gen_ai.system to gen_ai.provider.name in June 2025. Many tutorials and blog posts still reference the old name. Recording this decision in your build log shows you are working from the current spec, not stale documentation.

Your portfolio now has a clear front door, a certification tracking system, and an engineering journal. Next up, you will write RFC-001, the document that defines the LLM span schema every downstream system will depend on.

Write RFC-001 LLM Span Schema

Your repo structure is in place, your README tells visitors what this portfolio is about, and your build log is tracking progress. Now you need the document that makes every downstream system possible: the RFC that defines your LLM span schema.

An RFC forces you to articulate design decisions before they become load-bearing. Once systems 02 through 05 depend on your schema's mandatory fields, changing those fields is a breaking change that cascades across the entire portfolio. Writing the RFC now means you catch bad decisions while they are still cheap to fix.

In This Step, Get Ready To:
  • Write the RFC header and Motivation section explaining why a custom schema exists alongside OTel GenAI semantic conventions.
  • Define mandatory and optional attributes with their OTel mappings and three complete provider example spans.
  • Document non-goals and open questions to scope the schema tightly.
Create the RFC File and Write the Motivation

The RFC lives inside the system it governs. Its header captures the metadata reviewers need at a glance: status, decision type, version. The Motivation section answers the most important question a reviewer will ask: why does this schema need to exist when OpenTelemetry already defines GenAI semantic conventions?

  • Create the file systems/01-ai-observability-platform/rfcs/RFC-001-llm-span-schema.md.
  • Add the following RFC content.
# RFC-001: LLM Span Schema

| Field            | Value                              |
|------------------|------------------------------------|
| **Title**        | LLM Span Schema                    |
| **Status**       | Draft                              |
| **Decision Type**| One-Way Door (ADR-001)             |
| **Author**       | [[YOURNAME="your name"]]          |
| **Date**         | [[TODAY="YYYY-MM-DD"]]            |
| **Schema Version** | v0.1.0                           |

## Motivation

OTel GenAI semantic conventions (`gen_ai.*` namespace) define span-level
attributes for instrumentation libraries. They tell an instrumentation
library what attributes to set on a span when it wraps an LLM call.

This schema solves a different problem. It defines a **portable JSON
envelope** for downstream analytical consumers (cost dashboards, SLO
boards, eval gates, drift detectors) that ingest span data after export.
The schema pins a **versioned contract** those consumers can depend on.

Why not rely on OTel semconv directly?

1. OTel GenAI semantic conventions are in **Development** status and
   subject to breaking changes. This schema absorbs those changes at the
   boundary and presents a stable interface to downstream systems.
2. OTel semconv does not prescribe which attributes are mandatory for
   analytical use cases. For example, `gen_ai.usage.input_tokens` is
   Recommended in OTel semconv, but this schema makes it **mandatory**
   because cost calculation requires it.
3. The schema adds envelope fields (`span_id`, `trace_id`, `timestamp`,
   `duration_ms`) that are part of the span record but not part of OTel
   semconv attributes.

What Is a One-Way Door Decision?

A one-way door decision is one that is difficult or impossible to reverse once downstream dependencies exist. Once systems 02 through 05 consume this schema's mandatory fields, renaming or removing a field is a breaking change that cascades across the entire portfolio. Two-way door decisions (like adding an optional field) are safe to make quickly. The RFC process exists specifically to slow down one-way door decisions and force explicit reasoning.

Why gen_ai.provider.name and Not gen_ai.system?

As of the convention update merged in June 2025 (PR #2046), gen_ai.system has been renamed to gen_ai.provider.name. The old attribute is deprecated. Many tutorials and blog posts still reference the pre-June-2025 name. This schema uses the current name.

Write the Schema Design Section

The Schema Design section is the core of the RFC. It lists every attribute, maps it to its OTel semantic convention source, and explains why each mandatory field is mandatory. It also includes three complete example spans so reviewers can see exactly what the schema produces for each provider.

  • Append the following Schema Design section to your RFC file.
## Schema Design

### Mandatory Attributes

| Attribute                    | Type    | OTel Semconv Source         | Rationale                                      |
|------------------------------|---------|-----------------------------|-------------------------------------------------|
| `span_id`                    | string  | Span envelope               | Unique identifier for deduplication             |
| `trace_id`                   | string  | Span envelope               | Correlates spans within a single request chain  |
| `timestamp`                  | string  | Span envelope (ISO 8601)    | Required for time-series analysis               |
| `duration_ms`                | number  | Span envelope               | SLO latency calculations require duration       |
| `gen_ai.operation.name`      | string  | Required in OTel semconv    | Identifies the operation type (chat, embeddings)|
| `gen_ai.provider.name`       | string  | Required in OTel semconv    | Identifies the provider for cost-rate lookup    |
| `gen_ai.request.model`       | string  | Conditionally Required      | Every downstream consumer needs the model name  |
| `gen_ai.usage.input_tokens`  | integer | Recommended in OTel semconv | Cost calculation requires token counts          |
| `gen_ai.usage.output_tokens` | integer | Recommended in OTel semconv | Cost calculation requires token counts          |

### Optional Attributes

| Attribute                                | Type            | Notes                                         |
|------------------------------------------|-----------------|-----------------------------------------------|
| `gen_ai.response.model`                  | string          | Actual model that responded (may differ)      |
| `gen_ai.response.id`                     | string          | Provider's completion ID                      |
| `gen_ai.response.finish_reasons`         | array of strings| Why the model stopped generating              |
| `gen_ai.request.temperature`             | number          | Sampling temperature                          |
| `gen_ai.request.max_tokens`              | integer         | Max tokens requested                          |
| `gen_ai.request.top_p`                   | number          | Nucleus sampling parameter                    |
| `gen_ai.usage.cache_read.input_tokens`   | integer         | Tokens served from cache                      |
| `gen_ai.usage.cache_creation.input_tokens`| integer        | Tokens written to cache                       |
| `gen_ai.conversation.id`                 | string          | Multi-turn conversation identifier            |
| `error.type`                             | string          | Error classification if the call failed       |
| `aws.bedrock.guardrail.id`              | string          | Bedrock-specific. OTel marks Required for Bedrock, but not all Bedrock calls use guardrails. Deliberate divergence. |
| `server.address`                         | string          | Hostname of the provider endpoint             |

Why Are Recommended OTel Attributes Mandatory Here?

OTel semconv marks gen_ai.usage.input_tokens and gen_ai.usage.output_tokens as Recommended, not Required. This schema promotes them to mandatory because every downstream consumer in this portfolio needs token counts. The cost dashboard cannot calculate spend without them. The SLO board cannot measure throughput without them.

Making them optional would force every consumer to handle the missing-data case. That adds complexity for no benefit when the instrumentation library always has the data.

Now add the three provider example spans. These examples will become the basis for your test fixtures in Step 5.

  • Append the example spans section to your RFC file.
### Example Spans

#### Anthropic

```json
{
  "span_id": "abc123def456",
  "trace_id": "trace-7890-abcd-ef01",
  "timestamp": "2026-04-20T14:30:00Z",
  "duration_ms": 1250,
  "gen_ai.operation.name": "chat",
  "gen_ai.provider.name": "anthropic",
  "gen_ai.request.model": "claude-sonnet-4-20250514",
  "gen_ai.usage.input_tokens": 150,
  "gen_ai.usage.output_tokens": 350,
  "gen_ai.response.model": "claude-sonnet-4-20250514",
  "gen_ai.response.finish_reasons": ["end_turn"],
  "gen_ai.request.temperature": 0.7,
  "gen_ai.request.max_tokens": 1024
}
```

#### OpenAI

```json
{
  "span_id": "openai-span-001",
  "trace_id": "trace-openai-5678",
  "timestamp": "2026-04-20T14:31:00Z",
  "duration_ms": 980,
  "gen_ai.operation.name": "chat",
  "gen_ai.provider.name": "openai",
  "gen_ai.request.model": "gpt-4",
  "gen_ai.usage.input_tokens": 200,
  "gen_ai.usage.output_tokens": 500,
  "gen_ai.response.model": "gpt-4-0613",
  "gen_ai.response.id": "chatcmpl-abc123",
  "gen_ai.response.finish_reasons": ["stop"]
}
```

#### AWS Bedrock

```json
{
  "span_id": "bedrock-span-001",
  "trace_id": "trace-bedrock-9012",
  "timestamp": "2026-04-20T14:32:00Z",
  "duration_ms": 2100,
  "gen_ai.operation.name": "chat",
  "gen_ai.provider.name": "aws.bedrock",
  "gen_ai.request.model": "anthropic.claude-3-sonnet-20240229-v1:0",
  "gen_ai.usage.input_tokens": 175,
  "gen_ai.usage.output_tokens": 420,
  "gen_ai.response.model": "anthropic.claude-3-sonnet-20240229-v1:0",
  "gen_ai.response.finish_reasons": ["end_turn"],
  "aws.bedrock.guardrail.id": "guardrail-abc123"
}
```

Why Dots in JSON Keys?

OTel semantic convention attribute names use dots (e.g., gen_ai.provider.name). In JSON, these are flat string keys. They do not imply nested objects. The schema uses {"gen_ai.provider.name": "anthropic"}, not {"gen_ai": {"provider": {"name": "anthropic"}}}. This matches how OTel exports span attributes.

Finally, add the downstream consumer mapping so the RFC connects the schema to the systems that will depend on it.

  • Append the downstream consumers section to your RFC file.
### Downstream Consumers

| System | Consumes | Purpose |
|--------|----------|---------|
| 02 - Cost Dashboard | `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | Per-model cost calculation and anomaly detection |
| 03 - SLO Board | `duration_ms`, `gen_ai.operation.name`, `error.type` | Latency percentile tracking and error budget burn |
| 04 - Eval Gate | `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.response.finish_reasons` | Pre-deploy model quality validation |
| 05 - Drift Detector | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.request.model`, `timestamp` | Token distribution shift detection over time |
Write the Non-Goals and Open Questions

A good RFC is as clear about what it does not do as what it does. Non-goals prevent scope creep. Open questions flag decisions that are not yet resolved, giving future-you (or a reviewer) explicit permission to revisit them.

  • Append the Non-Goals and Open Questions sections to your RFC file.
## Non-Goals

1. **Span collection.** This schema does not define how spans are
   collected or instrumented. That is OTel's job. The schema only
   governs the shape of data after export.
2. **Storage backend.** This schema does not prescribe whether spans
   land in Prometheus, ClickHouse, S3, or any other store. The JSON
   envelope is storage-agnostic.
3. **Streaming chunks.** This schema does not handle streaming
   responses where token counts arrive incrementally. Streaming
   support is deferred to a future RFC.

## Open Questions

1. Should `gen_ai.usage.cache_read.input_tokens` and
   `gen_ai.usage.cache_creation.input_tokens` be mandatory or
   optional? Cache-aware pricing is becoming standard (Anthropic
   prompt caching, OpenAI cached context). If the cost dashboard
   needs cache-granular pricing, these fields may need to be
   mandatory.
2. Should the schema require `gen_ai.request.model` or accept spans
   where the model is only known from `gen_ai.response.model`? Some
   routing layers (e.g., OpenRouter) do not set the request model
   until the response arrives.

Pro tip: open questions are features

In a portfolio context, open questions signal to a hiring manager that you think about edge cases and future evolution. They show you know the difference between "I haven't decided yet" and "I decided not to decide yet." Both are valid, but only one is intentional.

Your RFC is complete. It captures the motivation, the full attribute design with OTel mappings, three provider examples, the downstream consumer map, and the boundaries of what this schema does and does not cover. Next, you will turn this design into a machine-readable JSON Schema and declare your data handling policy.

Create the JSON Schema and Data Handling Declaration

Your RFC defines the schema in human-readable prose. Now you need a machine-readable version that downstream consumers can validate against programmatically. JSON Schema Draft 2020-12 is the contract format. Every cost dashboard, SLO board, and eval gate in your portfolio will validate incoming spans against this single file.

In This Step, Get Ready To:
  • Create the llm-span.schema.json file with mandatory and optional attributes mapped to OTel GenAI semantic conventions.
  • Write a data-handling declaration classifying all test data as public synthetic prompts.
  • Record the schema version in a measurements file for traceability.
Write the JSON Schema File

OTel semantic convention attribute names use dots in their keys (e.g., gen_ai.provider.name). In JSON Schema, these are flat string property keys. They do not imply nested objects. Your schema uses {"gen_ai.provider.name": "anthropic"}, not a deeply nested structure.

  • Create the file systems/01-ai-observability-platform/schemas/llm-span.schema.json.
  • Paste the following schema definition:
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://github.com/YOUR_USERNAME/ai-cloud-systems-engineering/schemas/llm-span.schema.json",
  "title": "LLM Span Schema v0.1.0",
  "description": "Portable JSON envelope for LLM span data, grounded in OTel GenAI semantic conventions. Consumed by cost dashboards, SLO boards, eval gates, and drift detectors.",
  "type": "object",
  "required": [
    "span_id",
    "trace_id",
    "timestamp",
    "duration_ms",
    "gen_ai.operation.name",
    "gen_ai.provider.name",
    "gen_ai.request.model",
    "gen_ai.usage.input_tokens",
    "gen_ai.usage.output_tokens"
  ],
  "properties": {
    "span_id": {
      "type": "string",
      "description": "Unique span identifier from the span envelope."
    },
    "trace_id": {
      "type": "string",
      "description": "Trace identifier from the span envelope."
    },
    "timestamp": {
      "type": "string",
      "format": "date-time",
      "description": "ISO 8601 timestamp of when the span started."
    },
    "duration_ms": {
      "type": "number",
      "description": "Span duration in milliseconds."
    },
    "gen_ai.operation.name": {
      "type": "string",
      "enum": [
        "chat",
        "create_agent",
        "embeddings",
        "execute_tool",
        "generate_content",
        "invoke_agent",
        "retrieval",
        "text_completion"
      ],
      "description": "OTel Required. The operation type performed."
    },
    "gen_ai.provider.name": {
      "type": "string",
      "enum": [
        "anthropic",
        "openai",
        "aws.bedrock",
        "azure.ai.inference",
        "azure.ai.openai",
        "cohere",
        "deepseek",
        "gcp.gemini",
        "gcp.gen_ai",
        "gcp.vertex_ai",
        "groq",
        "ibm.watsonx.ai",
        "mistral_ai",
        "perplexity",
        "x_ai"
      ],
      "description": "OTel Required. The LLM provider name. Formerly gen_ai.system, renamed June 2025."
    },
    "gen_ai.request.model": {
      "type": "string",
      "description": "OTel Conditionally Required. Made mandatory in this schema because downstream consumers need it for cost and SLO calculations."
    },
    "gen_ai.usage.input_tokens": {
      "type": "integer",
      "description": "OTel Recommended, made mandatory. Total input tokens including cached tokens."
    },
    "gen_ai.usage.output_tokens": {
      "type": "integer",
      "description": "OTel Recommended, made mandatory. Output/completion tokens."
    },
    "gen_ai.response.model": {
      "type": "string",
      "description": "Actual model that responded. May differ from requested model."
    },
    "gen_ai.response.id": {
      "type": "string",
      "description": "Provider completion ID."
    },
    "gen_ai.response.finish_reasons": {
      "type": "array",
      "items": { "type": "string" },
      "description": "Why the model stopped generating."
    },
    "gen_ai.request.temperature": {
      "type": "number",
      "description": "Sampling temperature."
    },
    "gen_ai.request.max_tokens": {
      "type": "integer",
      "description": "Maximum tokens to generate."
    },
    "gen_ai.request.top_p": {
      "type": "number",
      "description": "Nucleus sampling parameter."
    },
    "gen_ai.usage.cache_read.input_tokens": {
      "type": "integer",
      "description": "Tokens served from cache. Value SHOULD be included in gen_ai.usage.input_tokens."
    },
    "gen_ai.usage.cache_creation.input_tokens": {
      "type": "integer",
      "description": "Tokens written to cache. Value SHOULD be included in gen_ai.usage.input_tokens."
    },
    "gen_ai.conversation.id": {
      "type": "string",
      "description": "Conversation or session identifier."
    },
    "error.type": {
      "type": "string",
      "description": "Error type if the span represents a failed operation."
    },
    "aws.bedrock.guardrail.id": {
      "type": "string",
      "description": "Bedrock-specific. OTel Required for Bedrock, optional in this schema since not all Bedrock calls use guardrails."
    },
    "server.address": {
      "type": "string",
      "description": "Server address of the LLM provider endpoint."
    }
  },
  "additionalProperties": true
}

What does this schema do?

The required array lists the nine fields that every valid LLM span must include. These are the fields your cost dashboard and SLO board cannot function without.

The enum constraints on gen_ai.provider.name and gen_ai.operation.name restrict values to the well-known list from OTel semantic conventions. This catches typos and unsupported providers at validation time.

Setting "additionalProperties": true means the schema accepts extra fields beyond what is explicitly defined. This is intentional. Provider-specific attributes (like aws.bedrock.guardrail.id) and future optional fields can appear without breaking validation.

Why JSON Schema instead of Pydantic?

Pydantic models are Python-only. A standalone .json schema file can be consumed by Go, Rust, TypeScript, or any language with a JSON Schema validator. Since later systems in this portfolio may use different languages, the schema stays language-agnostic. Python is just one consumer.

Why gen_ai.provider.name and not gen_ai.system?

The OTel GenAI semantic conventions renamed gen_ai.system to gen_ai.provider.name in June 2025 (PR #2046). The old attribute is deprecated. Many tutorials and blog posts still reference the pre-June-2025 name. Your schema uses the current name.

Create the Data Handling Declaration

Your test suite will contain example prompts and completions. Before writing any test data, you need to declare the data classification for this system. This declaration governs what example data can appear in tests, schemas, and documentation.

  • Create the file systems/01-ai-observability-platform/data-handling.md.
  • Paste the following content:
# Data Handling Declaration

## System: 01 - AI Observability Platform

## Classification: Class 0 - Public Synthetic Data Only

All prompts, completions, and span data in this system are **public synthetic data**.

- No customer data.
- No PII (Personally Identifiable Information).
- No proprietary content.
- No production API keys, tokens, or credentials.

All example spans in tests and schemas use fabricated prompts
(e.g., "Explain quantum computing in simple terms") that carry
no sensitivity.

## Scope

This declaration covers:

- Test fixtures in `tests/`
- Example spans in `rfcs/`
- Schema descriptions in `schemas/`

## Referenced By

- RFC-001: LLM Span Schema (v0.1.0)

Why declare data handling for synthetic data?

In a portfolio that will eventually handle real LLM traffic, establishing the data classification boundary early prevents accidental leaks. When you add production instrumentation in later iterations, this file becomes the checkpoint. If the classification changes from Class 0 to something higher, you know exactly what needs to change.

Record the Schema Version

Schema versioning is how you track breaking changes over time. When a downstream consumer breaks, the first question is always: which schema version was it built against? This file answers that question.

  • Create the file systems/01-ai-observability-platform/measurements/schema-version.md.
  • Paste the following content, updating the date to today's date:
# Schema Version Tracker

## Current Version: v0.1.0

| Version | Date       | RFC                                          | Schema File                          |
|---------|------------|----------------------------------------------|--------------------------------------|
| v0.1.0  | 2026-04-20 | [RFC-001](../rfcs/RFC-001-llm-span-schema.md) | [llm-span.schema.json](../schemas/llm-span.schema.json) |

## Version History

### v0.1.0 - Initial Release

- Nine mandatory attributes defined.
- Provider enum constrained to OTel well-known values.
- additionalProperties set to true for extensibility.
- Three provider examples validated: Anthropic, OpenAI, AWS Bedrock.

Pro tip: version tracking

Every time you change a mandatory field in a future iteration, add a new row to this table. The version history becomes your changelog for the schema contract. Adding optional fields is a minor bump. Changing or removing required fields is a major bump.

Your schema is defined, your data classification is declared, and your version is tracked. Next up, you will write the test fixture that proves this schema actually works against real provider span examples.

Write the Test Fixture and Operational Stubs

Your schema and RFC define the contract. Now you need proof that the contract actually works. In this step, you will write a pytest suite that validates real provider spans against your JSON Schema, create the operational script stubs that every future system will follow, and write the system-level README that ties everything together.

In This Step, Get Ready To:
  • Build a pytest fixture that validates Anthropic, OpenAI, and AWS Bedrock example spans against your JSON Schema.
  • Create operational script stubs (spinup, teardown, cost-check) that establish the pattern for all five portfolio systems.
  • Write the system-level README with a measurable headline and links to the RFC, schema, and tests.
Create the Test Fixture File

The test file needs to do three things: load the JSON Schema from disk, define sample span dictionaries that match the RFC examples, and validate each span against the schema. Each provider gets its own test function so failures pinpoint exactly which provider's span is broken.

The jsonschema library provides Draft202012Validator to explicitly pin validation to Draft 2020-12. You will use this instead of the generic validate() function so the draft selection is visible and intentional in the test code.

  • Create the file systems/01-ai-observability-platform/tests/test_schema_fixture.py.
  • Paste the following code into the file.
import json
import pathlib

import pytest
from jsonschema import Draft202012Validator

# ---------------------------------------------------------------------------
# Schema loader
# ---------------------------------------------------------------------------

SCHEMA_PATH = pathlib.Path(__file__).resolve().parent.parent / "schemas" / "llm-span.schema.json"

@pytest.fixture(scope="module")
def schema():
    """Load the LLM span JSON Schema once per test module."""
    with open(SCHEMA_PATH) as f:
        return json.load(f)

@pytest.fixture(scope="module")
def validator(schema):
    """Create a Draft 2020-12 validator pinned to the correct draft."""
    Draft202012Validator.check_schema(schema)  # fail fast if schema itself is invalid
    return Draft202012Validator(schema)

# ---------------------------------------------------------------------------
# Sample spans matching RFC-001 examples
# ---------------------------------------------------------------------------

ANTHROPIC_SPAN = {
    "span_id": "abc123def456",
    "trace_id": "trace-001-anthropic",
    "timestamp": "2026-04-20T14:30:00Z",
    "duration_ms": 1250,
    "gen_ai.operation.name": "chat",
    "gen_ai.provider.name": "anthropic",
    "gen_ai.request.model": "claude-sonnet-4-20250514",
    "gen_ai.usage.input_tokens": 150,
    "gen_ai.usage.output_tokens": 320,
}

OPENAI_SPAN = {
    "span_id": "openai-span-789",
    "trace_id": "trace-002-openai",
    "timestamp": "2026-04-20T14:35:00Z",
    "duration_ms": 980,
    "gen_ai.operation.name": "chat",
    "gen_ai.provider.name": "openai",
    "gen_ai.request.model": "gpt-4",
    "gen_ai.usage.input_tokens": 200,
    "gen_ai.usage.output_tokens": 450,
}

BEDROCK_SPAN = {
    "span_id": "bedrock-span-456",
    "trace_id": "trace-003-bedrock",
    "timestamp": "2026-04-20T14:40:00Z",
    "duration_ms": 1500,
    "gen_ai.operation.name": "chat",
    "gen_ai.provider.name": "aws.bedrock",
    "gen_ai.request.model": "anthropic.claude-3-sonnet-20240229-v1:0",
    "gen_ai.usage.input_tokens": 175,
    "gen_ai.usage.output_tokens": 290,
}

# ---------------------------------------------------------------------------
# Positive validation tests
# ---------------------------------------------------------------------------

def test_anthropic_span_valid(validator):
    """Anthropic span with all required fields passes validation."""
    validator.validate(ANTHROPIC_SPAN)

def test_openai_span_valid(validator):
    """OpenAI span with all required fields passes validation."""
    validator.validate(OPENAI_SPAN)

def test_bedrock_span_valid(validator):
    """AWS Bedrock span with all required fields passes validation."""
    validator.validate(BEDROCK_SPAN)

What Does This Code Do?

The schema() fixture loads your JSON Schema file from disk once per test module. The validator() fixture creates a Draft202012Validator instance and also calls check_schema() to verify the schema itself is valid before any tests run.

Each sample span dictionary uses flat dotted keys like gen_ai.provider.name. These are plain JSON string keys, not nested objects. This matches how OTel exports span attributes.

Why Draft202012Validator Instead of validate()?

The generic jsonschema.validate() function auto-detects the draft from the schema's $schema field. That works, but it hides the draft selection. Using Draft202012Validator explicitly makes the pinning visible in the test code. If someone accidentally changes the schema's $schema field, the test still validates against Draft 2020-12.

Create the Operational Script Stubs

Every system in this portfolio will follow the same operational pattern: a script to spin up infrastructure, a script to tear it down, and a script to check costs. These stubs establish that pattern now so future systems inherit a consistent interface.

  • Create systems/01-ai-observability-platform/scripts/spinup.sh with the following content.
#!/usr/bin/env bash
# spinup.sh - Provisions infrastructure for the AI Observability Platform.
# Future iterations will call Terraform apply, configure OTel collectors,
# and deploy Grafana dashboards.

set -euo pipefail

echo "[STUB] spinup.sh - No infrastructure to provision yet. Schema and RFC are the deliverables for iteration 01."
  • Create systems/01-ai-observability-platform/scripts/teardown.sh with the following content.
#!/usr/bin/env bash
# teardown.sh - Destroys infrastructure for the AI Observability Platform.
# Future iterations will call Terraform destroy and clean up remote state.

set -euo pipefail

echo "[STUB] teardown.sh - No infrastructure to tear down yet."
  • Create systems/01-ai-observability-platform/scripts/cost-check.sh with the following content.
#!/usr/bin/env bash
# cost-check.sh - Reports current cloud spend for the AI Observability Platform.
# Future iterations will query AWS Cost Explorer and output a summary.

set -euo pipefail

echo "[STUB] cost-check.sh - No cloud resources provisioned. Current cost: \$0.00."
  • Make all three scripts executable.
chmod +x systems/01-ai-observability-platform/scripts/spinup.sh
chmod +x systems/01-ai-observability-platform/scripts/teardown.sh
chmod +x systems/01-ai-observability-platform/scripts/cost-check.sh

Why Stubs Instead of Real Scripts?

The stubs exist to establish the interface contract. Every system in the portfolio will have the same three scripts with the same names. When iteration 02 adds Terraform, spinup.sh gets real commands. Anyone reviewing the repo sees the operational pattern immediately, even before infrastructure exists.

Write the System-Level README

The system-level README is different from the repo-level README you wrote earlier. This one lives inside the 01-ai-observability-platform directory and describes what this specific system will become, its measurable headline, and where to find the key artifacts.

  • Create systems/01-ai-observability-platform/README.md with the following content.
# System 01: AI Observability Platform

## Measurable Headline

Reduced LLM cost anomaly detection MTTR from 4 hours to 15 minutes using custom OTel GenAI span instrumentation and automated SLO alerting.

## Overview

An AI observability platform built on OpenTelemetry, Prometheus, Grafana, and custom LLM span instrumentation. This system collects, validates, and visualizes LLM usage data across Anthropic, OpenAI, and AWS Bedrock providers.

## Current Iteration: 01 (Schema Foundation)

- [RFC-001: LLM Span Schema](rfcs/RFC-001-llm-span-schema.md)
- [JSON Schema v0.1.0](schemas/llm-span.schema.json)
- [Test Fixture](tests/test_schema_fixture.py)
- [Data Handling Declaration](data-handling.md)
- [Schema Version Tracking](measurements/schema-version.md)

## Operational Scripts

| Script | Purpose | Status |
|--------|---------|--------|
| `scripts/spinup.sh` | Provision infrastructure | Stub |
| `scripts/teardown.sh` | Destroy infrastructure | Stub |
| `scripts/cost-check.sh` | Report cloud spend | Stub |

## Downstream Consumers

Systems 02-05 depend on the LLM span schema defined here:

1. **System 02**: Cost Dashboard
2. **System 03**: SLO Board
3. **System 04**: Eval Gate
4. **System 05**: Drift Detector

Pro tip: make the headline specific

The measurable headline is what a hiring manager reads first. "Reduced MTTR from 4 hours to 15 minutes" is specific and verifiable. Avoid vague claims like "improved observability." You will update this headline with real numbers once the platform is running.

Your test fixture, operational stubs, and system README are all in place. Next up, you will run the test suite and push everything to GitHub.

Run Tests and Commit

You have a schema, an RFC, and a test suite. None of it matters until the tests pass. In this step, you will run the full pytest suite to prove the schema validates all three provider spans, then commit everything to your public repository.

In This Step, Get Ready To:
  • Run the pytest suite and confirm all three provider span tests pass.
  • Debug any schema validation failures by comparing sample spans to required properties.
  • Commit and push the entire scaffold to your public GitHub repository.
Run the Test Suite

The test file you created in Step 5 uses Draft202012Validator to validate each sample span against your JSON Schema. Running the suite from the repo root confirms that all three provider examples (Anthropic, OpenAI, AWS Bedrock) conform to the schema's required properties and types.

  • Open your terminal and navigate to the root of your ai-cloud-systems-engineering repository.
  • Run the test suite with verbose output.
pytest systems/01-ai-observability-platform/tests/ -v

What Should I See?

You should see three test functions collected and all three passing:

tests/test_schema_fixture.py::test_anthropic_span_valid PASSED
tests/test_schema_fixture.py::test_openai_span_valid PASSED
tests/test_schema_fixture.py::test_bedrock_span_valid PASSED

========================= 3 passed =========================

Three green passes mean your schema accepts all three provider spans. The schema contract is proven before a single line of infrastructure code exists.

Seeing a ValidationError?

If a test fails with a ValidationError, the sample span dict does not match the schema's required properties. Check these common causes:

  • A required field is missing from the sample span. Compare the span dict keys against the required array in llm-span.schema.json.
  • A field has the wrong type. For example, gen_ai.usage.input_tokens must be an integer, not a string.
  • The gen_ai.provider.name value does not match one of the enum values in your schema. Verify you are using "anthropic", "openai", or "aws.bedrock".
  • The file path to the schema in your test is wrong. Make sure the open() call resolves relative to where you run pytest from.

Read the error message carefully. jsonschema tells you exactly which property failed and why. Help me debug my failing schema validation test.

Commit and Push to GitHub

With all tests green, your schema contract is validated. Everything you have built across this project (the repo scaffold, RFC-001, the JSON Schema, the test suite, operational stubs, and documentation) is ready to go public.

  • Stage all files in the repository.
git add -A
  • Commit with a conventional commit message that describes what shipped.
git commit -m "feat: scaffold repo and ship RFC-001 LLM span schema v0.1.0"

Why This Commit Message Format?

The feat: prefix follows the Conventional Commits specification. It signals that this commit introduces new functionality. This convention makes commit history scannable and enables automated changelog generation in future iterations.

  • Push to the main branch.
git push origin main
  • Open your repository in a browser to verify everything is live.

What Should I See?

Your public GitHub repository should show the full directory structure: adrs/, architecture/, systems/01-ai-observability-platform/ with its rfcs/, schemas/, tests/, and scripts/ subdirectories, plus build-log/, publications/, and retrospectives/. The repo README should render with your 20-second sell and the featured systems list.

Secret mission

Add Negative Test Coverage

Your positive tests prove the schema accepts valid spans. But how do you know it rejects invalid ones? Add negative test fixtures that assert your schema catches missing fields, wrong types, and invalid provider names.

Before you go

Clean Up Your Resources

This project runs entirely locally with no cloud provisioning, so there are no ongoing costs. Decide whether to keep your resources as they are, pause them to come back later, or delete them entirely.

Resources you used:

  • Local Git repository: ai-cloud-systems-engineering.
  • GitHub remote repository: ai-cloud-systems-engineering (public).
  • Python packages installed via pip: jsonschema and pytest.
  • Local project files: RFC, JSON Schema, test fixtures, operational script stubs, build log entries.

Keep everything running

No action needed. Choose this if you are continuing to the next iteration (Iteration 02: Terraform AWS foundation) or plan to keep building on this repo soon.

  • Your local repo and GitHub remote stay as they are.
  • Your installed Python packages remain available in your environment.
  • There are no ongoing costs since everything is local and the GitHub repo is free.

Pause - I'll come back to this later

Free up local disk space while keeping your work safe on GitHub. You can re-clone and reinstall dependencies when you return.

  • Confirm all changes are committed and pushed to GitHub by running git status.
  • Optionally delete the local clone of the repository. You can re-clone it from GitHub later.
  • If you used a virtual environment, deactivate it with deactivate and optionally delete the venv/ directory.
  • When you return, re-clone the repo and run pip install -r requirements.txt to restore your dependencies.

Delete - I don't want to use this again

Remove all project resources and start fresh. This is permanent.

  • Delete the local repository folder from your machine.
  • Delete the GitHub repository. Go to your repo on GitHub, open Settings, scroll to the bottom, and select Delete this repository.
  • If you used a virtual environment, delete the venv/ directory.
  • Uninstall the Python packages if you installed them globally by running pip uninstall jsonschema pytest.

Mission accomplished

That's a Wrap!

Nice work! You shipped a production-grade foundation before writing a single line of infrastructure code. Your portfolio now has a versioned schema contract, an RFC that documents every design decision, and a test suite that proves the schema works across three major LLM providers.

Here's what you learned:

  • Designed a JSON Schema (Draft 2020-12) for LLM spans grounded in OTel GenAI semantic conventions, with mandatory attributes mapped to the gen_ai.* namespace and deliberate divergences documented in RFC-001.
  • Applied one-way-door decision framing to identify that mandatory schema fields are a breaking-change risk, and used an RFC to force that decision through review before downstream systems depend on it.
  • Built a pytest suite validating Anthropic, OpenAI, and AWS Bedrock example spans against the schema using Draft202012Validator, proving the schema works across providers before any infrastructure exists.
  • Added negative test coverage proving the schema correctly rejects malformed spans with missing required fields, wrong types, and invalid provider names.

Ready to quiz yourself?