Build a CI/CD Pipeline Runner

Build a GitHub Actions clone that runs Docker containers and streams logs live.

Introduction

30 Second Summary

Every time you push code and see green checkmarks appear, a CI/CD system is spinning up containers behind the scenes to run your tests and report results. Most developers use these systems every day without understanding how they actually work.

In this project, you will build your own CI/CD pipeline runner from scratch. You will use Python, FastAPI, Docker, and WebSocket streaming, then deploy it to EC2 with a public URL anyone can use.

What You'll Build

A self-hosted CI/CD pipeline runner with a live web dashboard where users write YAML workflows, pick from pre-loaded templates, and watch Docker containers execute their commands in real-time.

By the end of this project, you'll have:

A FastAPI backend with REST API endpoints and real-time WebSocket log streaming.
A Docker execution engine that spins up isolated containers for each pipeline step and streams output line by line.
A live web dashboard with a YAML editor, template picker, and terminal-style log viewer deployed to EC2 with a public URL.
Secret Mission: add parallel step execution and webhook triggers so external services can fire your pipelines.

Are there any prerequisites?

No prior experience with Docker, FastAPI, or CI/CD systems is required. This project includes full setup instructions for every tool you need.

Install Python, Docker, and Set Up the Project

Your pipeline runner needs two core tools to work. Python powers the backend logic that manages pipelines, and Docker provides the isolated containers where each pipeline step actually executes.

In this step, you'll install both tools, create your project folder, and set up the file structure that holds your entire application.

In this step, get ready to:

Install Python 3.10+ and Docker.
Create the project directory with a virtual environment and dependencies.
Set up the project file structure.

Install Python and Docker

FastAPI requires Python 3.10 or higher. Let's confirm you have the right version installed.

Check your Python version by running this command:

python3 --version

✔️ I see version 3.10 or higher

Python is ready to go. Skip ahead to the Docker check below.

ⓧ I see an older version

Your system has an older Python that won't work with FastAPI. macOS often ships with Python 3.9 by default, which gets used even after you install a newer version.

Download Python 3.10 or newer from python.org/downloads.
Run the installer and follow the prompts.
After installation, close and reopen your terminal.
Run the version check again.

macOS users with Python 3.9

If python3 --version still shows the old version after installing, try running python3.12 --version (or whichever version you downloaded) to confirm the new install landed. You can also use a version manager like pyenv to switch your default.

ⓧ Command not found

Python is not installed on your machine.

Download Python 3.10 or newer from python.org/downloads.
Run the installer and follow the prompts.
Close and reopen your terminal after installation.
Run the version check again to confirm it works.

Now let's verify Docker is available. Docker is the engine that spins up containers for each pipeline step.

Check if Docker is installed by running:

docker --version

✔️ I see a version number

Docker is installed. Let's verify it can actually run containers.

ⓧ Command not found

Docker is not installed yet. Install Docker Desktop, which includes the Docker Engine and CLI tools.

Download Docker Desktop from docker.com/get-docker.
Run the installer and follow the setup wizard.
Launch Docker Desktop after installation.
Close and reopen your terminal.
Run the version check again to confirm Docker is available.

Verify Docker can run containers by running:

docker run hello-world

What does this command do?

This pulls a tiny test image from Docker Hub and runs it. If Docker is working correctly, you'll see a message that says "Hello from Docker!" along with some explanation text.

Seeing a permission error?

On Linux, you may need to prefix the command with sudo or add your user to the docker group. On macOS or Windows, make sure Docker Desktop is running (look for the whale icon in your menu bar or system tray).

Help me fix my Docker permissions

Create the project directory and install dependencies

With Python and Docker confirmed, you'll create a dedicated folder for your pipeline runner and set up an isolated Python environment inside it.

Move to your Desktop by running this command:

cd ~/Desktop

Create your project folder and move into it by running these commands:

mkdir pipeline-runner
cd pipeline-runner

Initialize a Git repository by running:

git init

Why start with Git?

Initializing Git now means every file you create is tracked from the very beginning. You'll push this repo to GitHub later when you deploy to EC2.

A virtual environment keeps your project's packages separate from your system Python. This prevents version conflicts with other projects on your machine.

Create a virtual environment by running:

python3 -m venv venv

Activate the virtual environment:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

Your terminal prompt should now show (venv) at the beginning, confirming the virtual environment is active.

Install the project dependencies by running:

pip install "fastapi[standard]==0.136.1" sqlalchemy==2.0.49 docker==7.1.0 "pyyaml>=6.0"

What are these packages?

fastapi[standard] is the web framework for your REST API and WebSocket endpoints.
sqlalchemy handles database models and queries for tracking pipelines and runs.
docker is the Python SDK that lets you create and manage containers programmatically.
pyyaml parses the YAML pipeline definitions that users write.

You should see pip downloading and installing each package along with its dependencies.

Seeing installation errors?

Make sure your virtual environment is active (you should see (venv) in your prompt). If you see a permission error, you may have forgotten to activate the venv and are trying to install globally.

Help me debug my pip install error

Create the project file structure

Your pipeline runner has several files that each handle one responsibility. Let's create them all now so the project structure is clear from the start.

Create the project files and templates directory by running these commands:

touch main.py database.py models.py schemas.py executor.py
mkdir templates

Confirm the files exist by running:

ls

You should see database.py, executor.py, main.py, models.py, schemas.py, templates/, and venv/ in the output.

What is each file for?

main.py is your FastAPI application with all the API endpoints.
database.py handles the SQLite database connection.
models.py defines the database tables (pipelines, runs, step logs).
schemas.py defines the API request and response shapes.
executor.py is the Docker execution engine that runs pipeline steps.
templates/ holds the HTML dashboard file.

Now let's record your dependencies in a requirements.txt file so anyone (including your EC2 server later) can install the exact same versions.

Open requirements.txt in VS Code and add these pinned dependencies:

fastapi[standard]==0.136.1
sqlalchemy==2.0.49
docker==7.1.0
pyyaml>=6.0

Why pin versions?

Pinning versions (like ==0.136.1) ensures your project works identically on every machine. Without pins, a new package release could introduce breaking changes and make your app stop working.

Save requirements.txt.

✔️ Awesome, I've got everything!

Great. Double check you've saved your file.

ⓧ I'd like to double check the full code

Here's what your requirements.txt should look like:

fastapi[standard]==0.136.1
sqlalchemy==2.0.49
docker==7.1.0
pyyaml>=6.0

Your project skeleton is ready. Next up, you'll define the database models that track pipelines, runs, and step logs.

Build the Database Models and Pipeline Schema

Every CI/CD system needs to remember what happened. Which pipelines exist, which runs are in progress, and what output each step produced.

In this step, you'll build the data layer that tracks all of this. You'll define database models for pipelines, runs, and step logs, then create the Pydantic schemas that validate data flowing through your API.

In this step, get ready to:

Create the database connection and three ORM models.
Define Pydantic schemas for API request and response validation.
Write an example pipeline YAML file for testing.

Create the database connection

Your pipeline runner needs a database to persist pipelines and their execution history. SQLAlchemy is Python's most popular database toolkit. It lets you define tables as Python classes and query them with Python code instead of raw SQL.

Open database.py in VS Code and paste the following code:

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, DeclarativeBase

# Path to the SQLite database file (created automatically)
SQLALCHEMY_DATABASE_URL = "sqlite:///./pipeline_runner.db"

# Create the database engine with SQLite-specific threading config
engine = create_engine(
    SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False}
)
# Session factory for database connections
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)


# Base class that all ORM models will inherit from
class Base(DeclarativeBase):
    pass


# Dependency function that provides a database session to each request
def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

What does this code do?

This module sets up the connection between your Python app and a SQLite database file called pipeline_runner.db. SQLite stores everything in a single file, so there is no separate database server to install.

engine is the core connection to the database. The check_same_thread argument allows SQLite to work across multiple threads (required for FastAPI).
SessionLocal is a factory that creates new database sessions (one per API request).
Base is the parent class for all your table models.
get_db() is a dependency function that FastAPI calls to open a session, give it to the endpoint, and close it when done.

Save database.py.
Verify the module loads by running this in your terminal:

python3 -c "from database import Base; print('Database module loaded')"

You should see Database module loaded printed with no errors.

Seeing a ModuleNotFoundError?

Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
Confirm you are running the command from inside the pipeline-runner/ directory where database.py lives.

help me fix a ModuleNotFoundError when importing my database module

✔️ Awesome, I've got everything!

Great. Make sure you have saved database.py before moving on.

ⓧ I'd like to double check the full code

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, DeclarativeBase

SQLALCHEMY_DATABASE_URL = "sqlite:///./pipeline_runner.db"

engine = create_engine(
    SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False}
)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)


class Base(DeclarativeBase):
    pass


def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

Define the ORM models

Your pipeline runner tracks three things: pipeline definitions, individual runs of those pipelines, and the log output from each step. Each of these becomes a database table represented by a Python class.

Open models.py and paste the following code:

from datetime import datetime

from sqlalchemy import Column, Integer, String, Text, DateTime, ForeignKey
from sqlalchemy.orm import relationship

from database import Base


# Stores a named pipeline with its YAML workflow definition
class Pipeline(Base):
    __tablename__ = "pipelines"

    id = Column(Integer, primary_key=True, index=True)
    name = Column(String, nullable=False)
    yaml_content = Column(Text, nullable=False)
    created_at = Column(DateTime, default=datetime.utcnow)

    runs = relationship("PipelineRun", back_populates="pipeline")


# Tracks each execution of a pipeline (status: queued > running > passed/failed)
class PipelineRun(Base):
    __tablename__ = "pipeline_runs"

    id = Column(Integer, primary_key=True, index=True)
    pipeline_id = Column(Integer, ForeignKey("pipelines.id"), nullable=True)
    status = Column(String, default="queued")
    started_at = Column(DateTime, nullable=True)
    finished_at = Column(DateTime, nullable=True)
    yaml_content = Column(Text, nullable=True)

    pipeline = relationship("Pipeline", back_populates="runs")
    steps = relationship("StepLog", back_populates="run", order_by="StepLog.order")


# Records the output of a single step within a run
class StepLog(Base):
    __tablename__ = "step_logs"

    id = Column(Integer, primary_key=True, index=True)
    run_id = Column(Integer, ForeignKey("pipeline_runs.id"), nullable=False)
    step_name = Column(String, nullable=False)
    command = Column(String, nullable=False)
    status = Column(String, default="pending")
    output = Column(Text, default="")
    order = Column(Integer, nullable=False)
    started_at = Column(DateTime, nullable=True)
    finished_at = Column(DateTime, nullable=True)

    run = relationship("PipelineRun", back_populates="steps")

What does this code do?

These three classes map directly to three database tables. SQLAlchemy uses relationship() to link them together so you can navigate between related records in Python.

Pipeline stores a named workflow definition (the YAML content). One pipeline can have many runs.
PipelineRun tracks a single execution. Its status field acts as a state machine: queued to running to passed or failed.
StepLog records what happened in each individual step (the command that ran, its output, and whether it passed or failed).

Save models.py.
Verify the models load correctly by running:

python3 -c "from models import Pipeline, PipelineRun, StepLog; print('Models loaded')"

You should see Models loaded with no errors.

Seeing an ImportError?

Check that database.py is saved in the same folder as models.py. The import from database import Base needs both files in the same directory.
Make sure you did not accidentally name the file model.py (singular). The filename must be models.py.

help me debug an ImportError in my models.py file

✔️ Awesome, I've got everything!

Great. Make sure you have saved models.py before moving on.

ⓧ I'd like to double check the full code

from datetime import datetime

from sqlalchemy import Column, Integer, String, Text, DateTime, ForeignKey
from sqlalchemy.orm import relationship

from database import Base


class Pipeline(Base):
    __tablename__ = "pipelines"

    id = Column(Integer, primary_key=True, index=True)
    name = Column(String, nullable=False)
    yaml_content = Column(Text, nullable=False)
    created_at = Column(DateTime, default=datetime.utcnow)

    runs = relationship("PipelineRun", back_populates="pipeline")


class PipelineRun(Base):
    __tablename__ = "pipeline_runs"

    id = Column(Integer, primary_key=True, index=True)
    pipeline_id = Column(Integer, ForeignKey("pipelines.id"), nullable=True)
    status = Column(String, default="queued")
    started_at = Column(DateTime, nullable=True)
    finished_at = Column(DateTime, nullable=True)
    yaml_content = Column(Text, nullable=True)

    pipeline = relationship("Pipeline", back_populates="runs")
    steps = relationship("StepLog", back_populates="run", order_by="StepLog.order")


class StepLog(Base):
    __tablename__ = "step_logs"

    id = Column(Integer, primary_key=True, index=True)
    run_id = Column(Integer, ForeignKey("pipeline_runs.id"), nullable=False)
    step_name = Column(String, nullable=False)
    command = Column(String, nullable=False)
    status = Column(String, default="pending")
    output = Column(Text, default="")
    order = Column(Integer, nullable=False)
    started_at = Column(DateTime, nullable=True)
    finished_at = Column(DateTime, nullable=True)

    run = relationship("PipelineRun", back_populates="steps")

Define the API request and response schemas

Your API needs to validate incoming data and shape outgoing responses. Pydantic models handle both. They define exactly which fields a request must include and what format the response takes.

Open schemas.py and paste the following code:

from datetime import datetime

from pydantic import BaseModel


# Request body for creating a named pipeline
class PipelineCreate(BaseModel):
    name: str
    yaml_content: str


# Request body for running raw YAML directly (no saved pipeline needed)
class RunYaml(BaseModel):
    yaml_content: str


# Response shape for a single step's log entry
class StepLogResponse(BaseModel):
    id: int
    step_name: str
    command: str
    status: str
    output: str
    order: int
    started_at: datetime | None
    finished_at: datetime | None

    class Config:
        from_attributes = True


# Response shape for a pipeline run (includes nested step logs)
class RunResponse(BaseModel):
    id: int
    pipeline_id: int | None
    status: str
    started_at: datetime | None
    finished_at: datetime | None
    steps: list[StepLogResponse] = []

    class Config:
        from_attributes = True


# Response shape for a pipeline definition
class PipelineResponse(BaseModel):
    id: int
    name: str
    yaml_content: str
    created_at: datetime

    class Config:
        from_attributes = True

What does this code do?

These schemas separate your API contract from your database models. FastAPI uses them to automatically validate request bodies and serialize responses to JSON.

PipelineCreate and RunYaml define what the client sends when creating a pipeline or running raw YAML.
StepLogResponse and RunResponse define what the API sends back. RunResponse nests a list of step logs so one request returns the full picture.
from_attributes = True tells Pydantic to read data from SQLAlchemy model attributes (instead of expecting plain dictionaries).

Save schemas.py.
Verify the schemas load correctly by running:

python3 -c "from schemas import PipelineCreate, RunResponse, PipelineResponse; print('Schemas loaded')"

You should see Schemas loaded with no errors.

Seeing a syntax error?

The datetime | None syntax requires Python 3.10 or higher. Run python3 --version to confirm your version.
Check for any missing colons after class Config or missing indentation inside the class bodies.

help me fix a syntax error in my Pydantic schemas file

✔️ Awesome, I've got everything!

Great. Make sure you have saved schemas.py before moving on.

ⓧ I'd like to double check the full code

from datetime import datetime

from pydantic import BaseModel


class PipelineCreate(BaseModel):
    name: str
    yaml_content: str


class RunYaml(BaseModel):
    yaml_content: str


class StepLogResponse(BaseModel):
    id: int
    step_name: str
    command: str
    status: str
    output: str
    order: int
    started_at: datetime | None
    finished_at: datetime | None

    class Config:
        from_attributes = True


class RunResponse(BaseModel):
    id: int
    pipeline_id: int | None
    status: str
    started_at: datetime | None
    finished_at: datetime | None
    steps: list[StepLogResponse] = []

    class Config:
        from_attributes = True


class PipelineResponse(BaseModel):
    id: int
    name: str
    yaml_content: str
    created_at: datetime

    class Config:
        from_attributes = True

Create the example pipeline definition

Your runner needs a sample pipeline to test with. Pipeline definitions use YAML to describe a sequence of steps, where each step specifies a Docker image and the commands to run inside it.

Create a new file called example_pipeline.yaml in your pipeline-runner/ folder and paste the following:

name: demo-pipeline
steps:
  - name: setup
    image: alpine:latest
    commands:
      - echo "Installing dependencies..."
      - apk add --no-cache curl > /dev/null 2>&1
      - echo "Dependencies installed!"
  - name: test
    image: alpine:latest
    commands:
      - echo "Running test suite..."
      - sleep 2
      - echo "Test 1 auth module... PASSED"
      - echo "Test 2 database layer... PASSED"
      - echo "Test 3 API endpoints... PASSED"
      - echo "All 3 tests passed!"
  - name: deploy
    image: alpine:latest
    commands:
      - echo "Deploying to production..."
      - sleep 1
      - echo "Build artifacts uploaded"
      - echo "Deployment complete!"

What does this YAML define?

This is a 3-step pipeline that mimics a real CI/CD workflow. Each step gets its own fresh container running Alpine Linux.

setup installs a dependency (curl) to simulate a real install phase.
test prints simulated test results with a sleep so you can watch the real-time streaming in action later.
deploy simulates pushing artifacts to production.

Save example_pipeline.yaml.
Confirm the file exists by running:

ls

You should see example_pipeline.yaml listed alongside your other project files.

Don't see the file?

Make sure you created the file inside pipeline-runner/ (not inside the templates/ subdirectory).
Check the file extension is .yaml and not .yml or .txt.

help me find my missing example_pipeline.yaml file

✔️ Awesome, I've got everything!

Great. Your example pipeline is ready for testing later.

ⓧ I'd like to double check the full code

name: demo-pipeline
steps:
  - name: setup
    image: alpine:latest
    commands:
      - echo "Installing dependencies..."
      - apk add --no-cache curl > /dev/null 2>&1
      - echo "Dependencies installed!"
  - name: test
    image: alpine:latest
    commands:
      - echo "Running test suite..."
      - sleep 2
      - echo "Test 1 auth module... PASSED"
      - echo "Test 2 database layer... PASSED"
      - echo "Test 3 API endpoints... PASSED"
      - echo "All 3 tests passed!"
  - name: deploy
    image: alpine:latest
    commands:
      - echo "Deploying to production..."
      - sleep 1
      - echo "Build artifacts uploaded"
      - echo "Deployment complete!"

Your data layer is solid. You have models that track every pipeline, every run, and every step's output. Next up, you'll build the Docker execution engine that actually spins up containers and runs commands inside them.

Build the Docker Execution Engine

Your database models define what a pipeline run is. Now you need the engine that actually executes it. The execution engine is the core of your CI/CD runner.

In this step, you will build the Docker execution engine. It spins up fresh containers for each pipeline step, runs commands inside them, captures output line by line, and broadcasts every line to connected WebSocket clients in real-time.

In this step, get ready to:

Build a ConnectionManager class to track WebSocket connections per run.
Write the execute_run function that orchestrates Docker containers and streams logs.
Verify Docker SDK connectivity from Python.

Create the ConnectionManager class

When a user triggers a pipeline run, they need to see logs appear in real-time. WebSocket connections allow the server to push log lines to the browser the instant they appear. The ConnectionManager class keeps track of which WebSocket clients are watching which run, so each log line reaches the right browser.

Create a new file called executor.py in your pipeline-runner/ folder.
Add the imports, ConnectionManager class, and module-level manager instance by pasting this code:

import asyncio
from datetime import datetime

import docker
import yaml

from database import SessionLocal
from models import PipelineRun, StepLog


class ConnectionManager:
    def __init__(self):
        self.active_connections: dict[int, list] = {}

    async def connect(self, run_id: int, websocket):
        await websocket.accept()
        if run_id not in self.active_connections:
            self.active_connections[run_id] = []
        self.active_connections[run_id].append(websocket)

    def disconnect(self, run_id: int, websocket):
        if run_id in self.active_connections:
            self.active_connections[run_id].remove(websocket)
            if not self.active_connections[run_id]:
                del self.active_connections[run_id]

    async def broadcast(self, run_id: int, message: dict):
        if run_id in self.active_connections:
            for connection in self.active_connections[run_id]:
                try:
                    await connection.send_json(message)
                except Exception:
                    pass


manager = ConnectionManager()

What does this code do?

The ConnectionManager class stores a dictionary mapping each run_id to a list of active WebSocket connections. When a user opens the dashboard and starts watching a run, their WebSocket gets added to that run's list.

connect() accepts a WebSocket and registers it under a specific run ID.
disconnect() removes a WebSocket when the client disconnects, and cleans up empty entries.
broadcast() sends a JSON message to every WebSocket watching a given run. This is how log lines reach the browser instantly.

Save executor.py.

You won't see a visible change yet. The next chunk adds the function that uses this manager to stream Docker output.

Seeing an import error?

Make sure your virtual environment is activated. You should see (venv) at the start of your terminal prompt.
Confirm you installed docker and pyyaml in your virtual environment from Step 1.
Help me figure out which package is missing

Build the execute_run function

The execute_run function is the heart of your pipeline runner. It parses YAML, iterates through each step, spins up a fresh Docker container, runs commands inside it, and broadcasts every line of output to connected WebSocket clients.

It also handles failures. If any command errors out or a container crashes, the function marks the step as failed, stops execution of further steps, and cleans up the container. This mirrors how real CI/CD systems like GitHub Actions work.

In executor.py, add the execute_run function below the manager = ConnectionManager() line by pasting this code:

async def execute_run(run_id: int, yaml_content: str):
    db = SessionLocal()
    client = docker.from_env()

    try:
        run = db.query(PipelineRun).filter(PipelineRun.id == run_id).first()
        run.status = "running"
        run.started_at = datetime.utcnow()
        db.commit()

        await manager.broadcast(run_id, {"type": "status", "status": "running"})

        pipeline_def = yaml.safe_load(yaml_content)
        steps = pipeline_def.get("steps", [])

        for i, step in enumerate(steps):
            step_name = step.get("name", f"step-{i+1}")
            image = step.get("image", "alpine:latest")
            commands = step.get("commands", [])

            step_log = StepLog(
                run_id=run_id,
                step_name=step_name,
                command=" && ".join(commands),
                status="running",
                order=i,
                started_at=datetime.utcnow(),
            )
            db.add(step_log)
            db.commit()
            db.refresh(step_log)

            await manager.broadcast(run_id, {
                "type": "step_start",
                "step_name": step_name,
                "order": i,
            })

            container = None
            step_output = ""
            step_failed = False

            try:
                container = client.containers.run(
                    image,
                    command="sleep 3600",
                    detach=True,
                    remove=False,
                    mem_limit="128m",
                    nano_cpus=500000000,
                )

                for cmd in commands:
                    await manager.broadcast(run_id, {
                        "type": "log",
                        "step_name": step_name,
                        "line": f"$ {cmd}",
                    })

                    exec_result = container.exec_run(
                        ["/bin/sh", "-c", cmd],
                        stream=True,
                        demux=True,
                    )

                    for stdout_chunk, stderr_chunk in exec_result.output:
                        if stdout_chunk:
                            line = stdout_chunk.decode("utf-8", errors="replace").strip()
                            if line:
                                step_output += line + "\n"
                                await manager.broadcast(run_id, {
                                    "type": "log",
                                    "step_name": step_name,
                                    "line": line,
                                })
                                await asyncio.sleep(0.05)
                        if stderr_chunk:
                            line = stderr_chunk.decode("utf-8", errors="replace").strip()
                            if line:
                                step_output += line + "\n"
                                await manager.broadcast(run_id, {
                                    "type": "log",
                                    "step_name": step_name,
                                    "line": f"[stderr] {line}",
                                })
                                await asyncio.sleep(0.05)

                container.stop(timeout=2)
                container.remove(force=True)

            except Exception as e:
                step_output += f"\nError: {str(e)}"
                step_failed = True
                if container:
                    try:
                        container.stop(timeout=2)
                        container.remove(force=True)
                    except Exception:
                        pass

            step_log.status = "failed" if step_failed else "passed"
            step_log.output = step_output
            step_log.finished_at = datetime.utcnow()
            db.commit()

            await manager.broadcast(run_id, {
                "type": "step_end",
                "step_name": step_name,
                "status": step_log.status,
            })

            if step_failed:
                run.status = "failed"
                run.finished_at = datetime.utcnow()
                db.commit()
                await manager.broadcast(run_id, {"type": "status", "status": "failed"})
                return

        run.status = "passed"
        run.finished_at = datetime.utcnow()
        db.commit()
        await manager.broadcast(run_id, {"type": "status", "status": "passed"})

    except Exception as e:
        run = db.query(PipelineRun).filter(PipelineRun.id == run_id).first()
        if run:
            run.status = "failed"
            run.finished_at = datetime.utcnow()
            db.commit()
        await manager.broadcast(run_id, {"type": "error", "message": str(e)})

    finally:
        db.close()

What does this function do?

This is a large function, so here is the breakdown of its four main sections:

Setup: Opens a database session and connects to the Docker daemon via docker.from_env(). Marks the run as "running" and broadcasts that status to any connected WebSocket clients.
Container execution: For each step, it starts a container with sleep 3600 to keep it alive, then runs each command via container.exec_run() with stream=True and demux=True. This yields stdout and stderr as separate byte chunks, which get decoded and broadcast line by line.
Failure handling: If any command fails or the container crashes, the inner except block marks the step as failed, cleans up the container, and stops executing further steps. The outer except catches unexpected errors and marks the entire run as failed.
Resource cleanup: After each step, container.stop() and container.remove(force=True) ensure no orphaned containers remain. The finally block always closes the database session.

The asyncio.sleep(0.05) between chunks gives the event loop time to send WebSocket frames. Without it, the loop would block and clients would receive logs in bursts instead of line by line.

Why container-per-step instead of one container for the whole pipeline?

Each step gets a fresh container so that steps are completely isolated. A failed npm install in one step cannot leave broken files that affect the next step. This is the same model GitHub Actions uses.

The container starts with sleep 3600 as a keep-alive command, then individual commands are injected via exec_run. This lets you stream output from each command separately rather than getting one combined blob at the end.

Save executor.py.

You cannot run the executor directly yet because it needs to be triggered by the API (which you will build in the next step). You can verify Docker connectivity in the next substep.

Worried about the function length?

This is the most complex function in the project. The key thing to check is that your indentation is correct. The inner try/except block (container execution) is nested inside the for loop, which is inside the outer try block.

If you see indentation errors when Python parses the file, compare your code against the full file in the double-check tab below.

Help me find where my indentation went wrong

✔️ Awesome, I've got everything!

Great. Double check you have saved executor.py.

ⓧ I'd like to double check the full code

import asyncio
from datetime import datetime

import docker
import yaml

from database import SessionLocal
from models import PipelineRun, StepLog


class ConnectionManager:
    def __init__(self):
        self.active_connections: dict[int, list] = {}

    async def connect(self, run_id: int, websocket):
        await websocket.accept()
        if run_id not in self.active_connections:
            self.active_connections[run_id] = []
        self.active_connections[run_id].append(websocket)

    def disconnect(self, run_id: int, websocket):
        if run_id in self.active_connections:
            self.active_connections[run_id].remove(websocket)
            if not self.active_connections[run_id]:
                del self.active_connections[run_id]

    async def broadcast(self, run_id: int, message: dict):
        if run_id in self.active_connections:
            for connection in self.active_connections[run_id]:
                try:
                    await connection.send_json(message)
                except Exception:
                    pass


manager = ConnectionManager()


async def execute_run(run_id: int, yaml_content: str):
    db = SessionLocal()
    client = docker.from_env()

    try:
        run = db.query(PipelineRun).filter(PipelineRun.id == run_id).first()
        run.status = "running"
        run.started_at = datetime.utcnow()
        db.commit()

        await manager.broadcast(run_id, {"type": "status", "status": "running"})

        pipeline_def = yaml.safe_load(yaml_content)
        steps = pipeline_def.get("steps", [])

        for i, step in enumerate(steps):
            step_name = step.get("name", f"step-{i+1}")
            image = step.get("image", "alpine:latest")
            commands = step.get("commands", [])

            step_log = StepLog(
                run_id=run_id,
                step_name=step_name,
                command=" && ".join(commands),
                status="running",
                order=i,
                started_at=datetime.utcnow(),
            )
            db.add(step_log)
            db.commit()
            db.refresh(step_log)

            await manager.broadcast(run_id, {
                "type": "step_start",
                "step_name": step_name,
                "order": i,
            })

            container = None
            step_output = ""
            step_failed = False

            try:
                container = client.containers.run(
                    image,
                    command="sleep 3600",
                    detach=True,
                    remove=False,
                    mem_limit="128m",
                    nano_cpus=500000000,
                )

                for cmd in commands:
                    await manager.broadcast(run_id, {
                        "type": "log",
                        "step_name": step_name,
                        "line": f"$ {cmd}",
                    })

                    exec_result = container.exec_run(
                        ["/bin/sh", "-c", cmd],
                        stream=True,
                        demux=True,
                    )

                    for stdout_chunk, stderr_chunk in exec_result.output:
                        if stdout_chunk:
                            line = stdout_chunk.decode("utf-8", errors="replace").strip()
                            if line:
                                step_output += line + "\n"
                                await manager.broadcast(run_id, {
                                    "type": "log",
                                    "step_name": step_name,
                                    "line": line,
                                })
                                await asyncio.sleep(0.05)
                        if stderr_chunk:
                            line = stderr_chunk.decode("utf-8", errors="replace").strip()
                            if line:
                                step_output += line + "\n"
                                await manager.broadcast(run_id, {
                                    "type": "log",
                                    "step_name": step_name,
                                    "line": f"[stderr] {line}",
                                })
                                await asyncio.sleep(0.05)

                container.stop(timeout=2)
                container.remove(force=True)

            except Exception as e:
                step_output += f"\nError: {str(e)}"
                step_failed = True
                if container:
                    try:
                        container.stop(timeout=2)
                        container.remove(force=True)
                    except Exception:
                        pass

            step_log.status = "failed" if step_failed else "passed"
            step_log.output = step_output
            step_log.finished_at = datetime.utcnow()
            db.commit()

            await manager.broadcast(run_id, {
                "type": "step_end",
                "step_name": step_name,
                "status": step_log.status,
            })

            if step_failed:
                run.status = "failed"
                run.finished_at = datetime.utcnow()
                db.commit()
                await manager.broadcast(run_id, {"type": "status", "status": "failed"})
                return

        run.status = "passed"
        run.finished_at = datetime.utcnow()
        db.commit()
        await manager.broadcast(run_id, {"type": "status", "status": "passed"})

    except Exception as e:
        run = db.query(PipelineRun).filter(PipelineRun.id == run_id).first()
        if run:
            run.status = "failed"
            run.finished_at = datetime.utcnow()
            db.commit()
        await manager.broadcast(run_id, {"type": "error", "message": str(e)})

    finally:
        db.close()

Verify Docker SDK connectivity

Before moving on, confirm that the Docker SDK can actually talk to your Docker daemon. If this fails, the executor will crash at runtime with no helpful error message.

Test Docker SDK connectivity by running this command:

python3 -c "import docker; client = docker.from_env(); print(client.ping())"

You should see True printed to your terminal. This confirms the Docker SDK can reach the Docker daemon.

Seeing a connection error?

Make sure Docker Desktop (or Docker Engine on Linux) is running. The daemon must be active for the SDK to connect.
On Linux, your user may need to be in the docker group. Try running the command with sudo to confirm it is a permissions issue.
Help me troubleshoot the Docker connection

Your execution engine is ready. Next up, you will build the REST API and WebSocket endpoint that triggers this engine and connects it to the browser.

Build the REST API and WebSocket Endpoint

Your execution engine knows how to spin up containers and stream logs. Now you need a way for the outside world to trigger pipelines and receive those logs.

In this step, you will build the FastAPI server that exposes your pipeline runner as a REST API with a WebSocket endpoint for real-time log streaming.

In this step, get ready to:

Create the FastAPI app with pipeline CRUD and run endpoints.
Add a direct YAML execution endpoint for the dashboard.
Wire up a WebSocket endpoint for real-time log streaming.

Create the FastAPI app and pipeline endpoints

The API layer connects your executor to HTTP clients. It handles creating pipelines, triggering runs, and serving the dashboard page.

Create main.py and add the imports, app initialization, and basic pipeline endpoints by copying this code:

import asyncio
from pathlib import Path

from fastapi import FastAPI, Depends, HTTPException, WebSocket, WebSocketDisconnect
from fastapi.responses import FileResponse
from sqlalchemy.orm import Session

from database import engine, get_db
from models import Base, Pipeline, PipelineRun
from schemas import PipelineCreate, PipelineResponse, RunResponse, RunYaml
from executor import execute_run, manager

# Create all database tables on startup
Base.metadata.create_all(bind=engine)

app = FastAPI(
    title="Pipeline Runner",
    description="A self-hosted CI/CD pipeline runner with real-time log streaming",
    version="1.0.0",
)


# Serve the web dashboard at the root URL
@app.get("/", include_in_schema=False)
def serve_dashboard():
    return FileResponse(Path("templates/dashboard.html"))


# Create a named pipeline from YAML content
@app.post("/pipelines", response_model=PipelineResponse)
def create_pipeline(data: PipelineCreate, db: Session = Depends(get_db)):
    pipeline = Pipeline(name=data.name, yaml_content=data.yaml_content)
    db.add(pipeline)
    db.commit()
    db.refresh(pipeline)
    return pipeline


# List all saved pipelines
@app.get("/pipelines", response_model=list[PipelineResponse])
def list_pipelines(db: Session = Depends(get_db)):
    return db.query(Pipeline).all()

What does this code do?

Base.metadata.create_all(bind=engine) creates all the database tables you defined in models.py when the server starts.
The serve_dashboard() function returns the HTML file at the root URL. The dashboard is built in the next step.
POST /pipelines saves a named pipeline. GET /pipelines lists them all. FastAPI automatically validates request bodies against your Pydantic schemas.

You won't see a change yet. The next chunk adds the run-triggering and retrieval endpoints.

Add the run-triggering and retrieval endpoints below the list_pipelines function in main.py:

# Trigger a run for a saved pipeline
@app.post("/pipelines/{pipeline_id}/run", response_model=RunResponse)
def trigger_run(pipeline_id: int, db: Session = Depends(get_db)):
    pipeline = db.query(Pipeline).filter(Pipeline.id == pipeline_id).first()
    if not pipeline:
        raise HTTPException(status_code=404, detail="Pipeline not found")
    run = PipelineRun(pipeline_id=pipeline_id, status="queued", yaml_content=pipeline.yaml_content)
    db.add(run)
    db.commit()
    db.refresh(run)
    asyncio.create_task(execute_run(run.id, pipeline.yaml_content))
    return run


# List the 20 most recent runs
@app.get("/runs", response_model=list[RunResponse])
def list_runs(db: Session = Depends(get_db)):
    return db.query(PipelineRun).order_by(PipelineRun.id.desc()).limit(20).all()


# Get a specific run with its step logs
@app.get("/runs/{run_id}", response_model=RunResponse)
def get_run(run_id: int, db: Session = Depends(get_db)):
    run = db.query(PipelineRun).filter(PipelineRun.id == run_id).first()
    if not run:
        raise HTTPException(status_code=404, detail="Run not found")
    return run

What does this code do?

trigger_run() creates a PipelineRun record, then launches the executor as a background task with asyncio.create_task(). The API returns the run immediately while Docker execution happens in the background.
GET /runs returns the 20 most recent runs (newest first). GET /runs/{run_id} returns a single run with all its step logs for detailed inspection.

Save main.py.
Start the development server by running:

fastapi dev main.py

Open http://127.0.0.1:8000/docs in your browser. You should see the interactive API documentation page listing your endpoints: POST /pipelines, GET /pipelines, POST /pipelines/{pipeline_id}/run, GET /runs, and GET /runs/{run_id}.

What is fastapi dev?

The fastapi dev command starts a development server with auto-reload enabled on 127.0.0.1. Every time you save a file, the server restarts automatically. Leave it running for the rest of this step.

Don't see the docs page?

Check that your virtual environment is activated. If you see a ModuleNotFoundError, your dependencies are not installed in the active environment.
Make sure main.py is in the same folder as database.py and models.py. Import errors mean Python cannot find your modules.
If you see a port-in-use error, another process is using port 8000. Stop it or run fastapi dev main.py --port 8001 instead.

Help me debug why my FastAPI server won't start.

Add the run-yaml endpoint

The pipeline CRUD endpoints require saving a pipeline first, then triggering it. For the dashboard, visitors should be able to paste YAML and run it immediately without saving.

In main.py, add the /run-yaml endpoint below the get_run function:

# Run a pipeline directly from raw YAML (no need to save first)
@app.post("/run-yaml", response_model=RunResponse)
def run_from_yaml(data: RunYaml, db: Session = Depends(get_db)):
    run = PipelineRun(pipeline_id=None, status="queued", yaml_content=data.yaml_content)
    db.add(run)
    db.commit()
    db.refresh(run)
    asyncio.create_task(execute_run(run.id, data.yaml_content))
    return run

Why a separate endpoint?

This endpoint sets pipeline_id=None because the run is anonymous. The YAML content is stored directly on the PipelineRun record so you can replay or inspect it later.

This powers the dashboard's "edit and run" workflow where visitors can execute custom YAML without creating a named pipeline.

Save main.py. The dev server auto-reloads.
Refresh http://127.0.0.1:8000/docs in your browser. You should see POST /run-yaml listed among your endpoints.

Don't see the new endpoint?

Check your terminal for reload errors. A syntax error in the new code prevents the server from restarting.
Make sure the function is placed outside (below) the get_run function, not accidentally indented inside it.

Help me figure out why my new endpoint isn't showing up.

Add the WebSocket endpoint

REST endpoints let clients trigger runs, but they cannot push live updates back. A WebSocket keeps a persistent connection open so the server can broadcast every log line as it happens.

Add the WebSocket endpoint at the bottom of main.py, below the run_from_yaml function:

# WebSocket endpoint for real-time log streaming
@app.websocket("/ws/runs/{run_id}")
async def websocket_logs(websocket: WebSocket, run_id: int):
    await manager.connect(run_id, websocket)
    try:
        while True:
            await websocket.receive_text()
    except WebSocketDisconnect:
        manager.disconnect(run_id, websocket)

How does the WebSocket work with the executor?

manager.connect() registers the client in the ConnectionManager you built in executor.py. When the executor broadcasts a log line, it sends it to every WebSocket registered for that run_id.
The while True: await websocket.receive_text() loop keeps the connection alive. When the client disconnects, FastAPI raises WebSocketDisconnect and we clean up.

Save main.py. The server reloads.
Refresh http://127.0.0.1:8000/docs. You should now see all your endpoints listed, including the WebSocket route at /ws/runs/{run_id}.

Seeing an import error for manager?

Verify that executor.py defines manager = ConnectionManager() at the module level. The import from executor import execute_run, manager at the top of main.py pulls both in.
Make sure there are no circular imports. The import chain is: main.py imports from executor.py, which imports from database.py and models.py. None of those import from main.py.

Help me fix import errors in my FastAPI pipeline runner.

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

Here is the complete main.py file. Compare yours against it:

import asyncio
from pathlib import Path

from fastapi import FastAPI, Depends, HTTPException, WebSocket, WebSocketDisconnect
from fastapi.responses import FileResponse
from sqlalchemy.orm import Session

from database import engine, get_db
from models import Base, Pipeline, PipelineRun
from schemas import PipelineCreate, PipelineResponse, RunResponse, RunYaml
from executor import execute_run, manager

Base.metadata.create_all(bind=engine)

app = FastAPI(
    title="Pipeline Runner",
    description="A self-hosted CI/CD pipeline runner with real-time log streaming",
    version="1.0.0",
)


@app.get("/", include_in_schema=False)
def serve_dashboard():
    return FileResponse(Path("templates/dashboard.html"))


@app.post("/pipelines", response_model=PipelineResponse)
def create_pipeline(data: PipelineCreate, db: Session = Depends(get_db)):
    pipeline = Pipeline(name=data.name, yaml_content=data.yaml_content)
    db.add(pipeline)
    db.commit()
    db.refresh(pipeline)
    return pipeline


@app.get("/pipelines", response_model=list[PipelineResponse])
def list_pipelines(db: Session = Depends(get_db)):
    return db.query(Pipeline).all()


@app.post("/pipelines/{pipeline_id}/run", response_model=RunResponse)
def trigger_run(pipeline_id: int, db: Session = Depends(get_db)):
    pipeline = db.query(Pipeline).filter(Pipeline.id == pipeline_id).first()
    if not pipeline:
        raise HTTPException(status_code=404, detail="Pipeline not found")
    run = PipelineRun(pipeline_id=pipeline_id, status="queued", yaml_content=pipeline.yaml_content)
    db.add(run)
    db.commit()
    db.refresh(run)
    asyncio.create_task(execute_run(run.id, pipeline.yaml_content))
    return run


@app.post("/run-yaml", response_model=RunResponse)
def run_from_yaml(data: RunYaml, db: Session = Depends(get_db)):
    run = PipelineRun(pipeline_id=None, status="queued", yaml_content=data.yaml_content)
    db.add(run)
    db.commit()
    db.refresh(run)
    asyncio.create_task(execute_run(run.id, data.yaml_content))
    return run


@app.get("/runs", response_model=list[RunResponse])
def list_runs(db: Session = Depends(get_db)):
    return db.query(PipelineRun).order_by(PipelineRun.id.desc()).limit(20).all()


@app.get("/runs/{run_id}", response_model=RunResponse)
def get_run(run_id: int, db: Session = Depends(get_db)):
    run = db.query(PipelineRun).filter(PipelineRun.id == run_id).first()
    if not run:
        raise HTTPException(status_code=404, detail="Run not found")
    return run


@app.websocket("/ws/runs/{run_id}")
async def websocket_logs(websocket: WebSocket, run_id: int):
    await manager.connect(run_id, websocket)
    try:
        while True:
            await websocket.receive_text()
    except WebSocketDisconnect:
        manager.disconnect(run_id, websocket)

Your API is live and ready to accept pipeline runs. Next up, you will build the web dashboard that lets anyone write YAML, pick templates, and watch real-time execution.

Build the Live Dashboard with YAML Editor

Your backend is complete. You have a REST API that creates pipelines, triggers runs, and streams logs via WebSocket. The missing piece is a frontend where visitors can actually use your pipeline runner.

In this step, you will build a single-page dashboard with a YAML editor on the left and a terminal-style log viewer on the right. Visitors pick a template, customize it, click Run, and watch Docker containers execute their commands in real-time.

In this step, get ready to:

Create the dashboard layout with split-pane CSS and a dark GitHub-style theme.
Add pre-written pipeline templates that populate the YAML editor.
Wire up the Run button to POST YAML and stream logs over WebSocket.

Create the dashboard layout and styling

The dashboard is a single self-contained HTML file with inline CSS and JavaScript. No build tools, no npm. Everything lives in one file that FastAPI serves at the root URL.

In VS Code, open the templates/ directory inside your pipeline-runner folder.
Create a new file called dashboard.html and paste the following code:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Pipeline Runner</title>
    <style>
        * { margin: 0; padding: 0; box-sizing: border-box; }
        body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: #0d1117; color: #c9d1d9; min-height: 100vh; }
        .header { padding: 16px 24px; border-bottom: 1px solid #21262d; display: flex; align-items: center; justify-content: space-between; }
        .header h1 { font-size: 18px; color: #58a6ff; }
        .badge { padding: 4px 12px; border-radius: 12px; font-size: 12px; font-weight: 600; }
        .badge-queued { background: #362d00; color: #d29922; }
        .badge-running { background: #0c2d6b; color: #58a6ff; }
        .badge-passed { background: #0d2818; color: #3fb950; }
        .badge-failed { background: #3d1114; color: #f85149; }
        .container { display: flex; height: calc(100vh - 57px); }
        .editor-pane { width: 45%; border-right: 1px solid #21262d; display: flex; flex-direction: column; }
        .editor-header { padding: 12px 16px; border-bottom: 1px solid #21262d; display: flex; align-items: center; gap: 12px; }
        .editor-header select { background: #21262d; color: #c9d1d9; border: 1px solid #30363d; border-radius: 6px; padding: 6px 12px; font-size: 13px; }
        .editor-header .btn { padding: 8px 16px; }
        .yaml-editor { flex: 1; padding: 0; }
        .yaml-editor textarea { width: 100%; height: 100%; background: #161b22; color: #c9d1d9; border: none; padding: 16px; font-family: 'Courier New', monospace; font-size: 13px; line-height: 1.5; resize: none; outline: none; }
        .output-pane { flex: 1; display: flex; flex-direction: column; }
        .terminal { flex: 1; background: #161b22; padding: 16px; overflow-y: auto; font-family: 'Courier New', monospace; font-size: 13px; line-height: 1.6; }
        .terminal .line { white-space: pre-wrap; word-break: break-all; }
        .terminal .cmd { color: #79c0ff; }
        .terminal .step-header { color: #3fb950; font-weight: bold; margin-top: 12px; border-bottom: 1px solid #21262d; padding-bottom: 4px; }
        .terminal .step-header.failed { color: #f85149; }
        .terminal .stderr { color: #f85149; }
        .btn { padding: 10px 20px; border: none; border-radius: 6px; cursor: pointer; font-size: 13px; font-weight: 600; }
        .btn-primary { background: #238636; color: white; }
        .btn-primary:hover { background: #2ea043; }
        .btn-primary:disabled { background: #21262d; color: #484f58; cursor: not-allowed; }
        .run-history { padding: 12px 16px; border-top: 1px solid #21262d; max-height: 120px; overflow-y: auto; }
        .run-item { display: flex; align-items: center; justify-content: space-between; padding: 6px 8px; border-radius: 4px; margin-bottom: 4px; background: #161b22; }
        .run-item span { font-size: 11px; }
        .status-bar { padding: 10px 16px; border-top: 1px solid #21262d; font-size: 11px; color: #8b949e; }
    </style>
</head>
<body>
    <div class="header">
        <h1>Pipeline Runner</h1>
        <span id="statusBadge" class="badge badge-queued" style="display:none;">queued</span>
    </div>
    <div class="container">
        <div class="editor-pane">
            <div class="editor-header">
                <select id="templateSelect" onchange="loadTemplate()">
                    <option value="nodejs">Node.js CI</option>
                    <option value="python">Python Lint + Test</option>
                    <option value="deploy">Deploy Pipeline</option>
                    <option value="custom">Custom (blank)</option>
                </select>
                <button class="btn btn-primary" id="runBtn" onclick="runPipeline()">Run Pipeline</button>
            </div>
            <div class="yaml-editor">
                <textarea id="yamlEditor" spellcheck="false"></textarea>
            </div>
            <div class="run-history">
                <div style="font-size:11px; color:#8b949e; margin-bottom:6px;">RECENT RUNS</div>
                <div id="runHistory"></div>
            </div>
        </div>
        <div class="output-pane">
            <div class="terminal" id="terminal">
                <div class="line" style="color:#8b949e;">Select a template or write your own YAML, then click "Run Pipeline"...</div>
            </div>
            <div class="status-bar">Pipeline Runner v1.0 | Docker-powered isolated execution | Each step runs in a fresh container</div>
        </div>
    </div>
    <script>
    </script>
</body>
</html>

What does this code do?

The page uses a flexbox split-pane layout. The left .editor-pane takes 45% width and holds the template dropdown, YAML textarea, and run history. The right .output-pane fills the remaining space with a terminal-style log viewer.

The CSS uses a dark color scheme inspired by GitHub's dark mode. Status badges (.badge-queued, .badge-running, .badge-passed, .badge-failed) use color-coded backgrounds so pipeline status is visible at a glance.

Save templates/dashboard.html.
Start the dev server by running this command in your terminal (make sure your virtual environment is active):

fastapi dev main.py

Open http://localhost:8000 in your browser. You should see a dark page with a "Pipeline Runner" header, a template dropdown on the left, and an empty terminal area on the right.

Seeing a blank page or error?

Make sure dashboard.html is inside the templates/ folder, not in the project root. The serve_dashboard() function in main.py looks for templates/dashboard.html specifically.
Check your terminal for import errors. If you see one, make sure your virtual environment is activated and all dependencies from requirements.txt are installed.

help me debug why my dashboard isn't loading at localhost:8000

Add the pipeline templates

The template dropdown needs pre-written YAML workflows. When a visitor selects a template, the editor fills with a working pipeline they can customize before running.

In templates/dashboard.html, find the empty <script> tags and paste the following code between them:

        const templates = {
            nodejs: `name: nodejs-ci\nsteps:\n  - name: install\n    image: node:20-alpine\n    commands:\n      - echo "Creating package.json..."\n      - echo '{"name":"app","scripts":{"test":"echo All tests passed!"}}' > package.json\n      - npm install\n      - echo "Dependencies installed!"\n  - name: test\n    image: node:20-alpine\n    commands:\n      - echo "Running test suite..."\n      - sleep 1\n      - echo "PASS src/auth.test.js"\n      - echo "PASS src/api.test.js"\n      - echo "PASS src/db.test.js"\n      - echo "Tests: 3 passed, 3 total"\n  - name: build\n    image: node:20-alpine\n    commands:\n      - echo "Building production bundle..."\n      - sleep 1\n      - echo "Bundle size: 142kb (gzipped)"\n      - echo "Build complete!"`,
            python: `name: python-lint-test\nsteps:\n  - name: lint\n    image: alpine:latest\n    commands:\n      - echo "Running Python linter..."\n      - sleep 1\n      - echo "main.py - no issues"\n      - echo "utils.py - no issues"\n      - echo "tests/ - no issues"\n      - echo "All files passed linting!"\n  - name: test\n    image: alpine:latest\n    commands:\n      - echo "Running pytest..."\n      - sleep 2\n      - echo "test_auth.py::test_login PASSED"\n      - echo "test_auth.py::test_register PASSED"\n      - echo "test_api.py::test_get_users PASSED"\n      - echo "test_api.py::test_create_user PASSED"\n      - echo "4 passed in 1.23s"`,
            deploy: `name: deploy-pipeline\nsteps:\n  - name: setup\n    image: alpine:latest\n    commands:\n      - echo "Checking environment..."\n      - echo "Region: ap-south-1"\n      - echo "Cluster: production"\n      - echo "Environment ready!"\n  - name: build\n    image: alpine:latest\n    commands:\n      - echo "Building Docker image..."\n      - sleep 2\n      - echo "Layer 1/5: base image"\n      - echo "Layer 2/5: dependencies"\n      - echo "Layer 3/5: application code"\n      - echo "Layer 4/5: configuration"\n      - echo "Layer 5/5: entrypoint"\n      - echo "Image built: app:v1.2.3"\n  - name: deploy\n    image: alpine:latest\n    commands:\n      - echo "Deploying to production..."\n      - sleep 1\n      - echo "Rolling update started"\n      - echo "Pod 1/3 updated"\n      - echo "Pod 2/3 updated"\n      - echo "Pod 3/3 updated"\n      - echo "Deployment complete!"`,
            custom: `name: my-pipeline\nsteps:\n  - name: step-1\n    image: alpine:latest\n    commands:\n      - echo "Hello from step 1!"\n  - name: step-2\n    image: alpine:latest\n    commands:\n      - echo "Hello from step 2!"`
        };

        function loadTemplate() {
            const sel = document.getElementById('templateSelect').value;
            document.getElementById('yamlEditor').value = templates[sel];
        }

What does this code do?

The templates object stores four pre-written YAML pipelines as template literal strings. Each template defines a realistic CI/CD workflow with named steps, Docker images, and shell commands.

The loadTemplate() function reads the dropdown value and fills the textarea with the matching template. It fires on every dropdown change thanks to the onchange="loadTemplate()" attribute in the HTML.

Save templates/dashboard.html and refresh your browser.
You should see the YAML editor populated with the Node.js CI template. Try switching the dropdown to "Python Lint + Test" or "Deploy Pipeline" and watch the editor content change.

Dropdown not filling the editor?

Open your browser's DevTools console (Cmd+Option+J on macOS, Ctrl+Shift+J on Windows). Look for JavaScript syntax errors. A missing backtick or unescaped quote inside the template strings will break everything.
Make sure your code is inside the <script> tags, not outside them.

help me fix JavaScript errors in my dashboard template strings

Wire up real-time pipeline execution

The Run button needs to POST the editor content to your /run-yaml endpoint, then open a WebSocket connection to stream execution logs as they arrive from Docker containers.

In templates/dashboard.html, paste the following code directly below the loadTemplate() function (still inside the <script> tags):

        async function runPipeline() {
            const yaml = document.getElementById('yamlEditor').value;
            if (!yaml.trim()) return;
            const btn = document.getElementById('runBtn');
            btn.disabled = true;
            btn.textContent = 'Running...';
            const terminal = document.getElementById('terminal');
            terminal.innerHTML = '';
            showBadge('queued');

            const res = await fetch('/run-yaml', {
                method: 'POST',
                headers: {'Content-Type': 'application/json'},
                body: JSON.stringify({yaml_content: yaml})
            });
            const run = await res.json();
            showBadge('running');

            const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
            const ws = new WebSocket(`${protocol}//${window.location.host}/ws/runs/${run.id}`);
            ws.onmessage = (event) => {
                const msg = JSON.parse(event.data);
                if (msg.type === 'step_start') {
                    addLine(`\n--- Step: ${msg.step_name} ---`, 'step-header');
                } else if (msg.type === 'log') {
                    const cls = msg.line.startsWith('$') ? 'cmd' : (msg.line.startsWith('[stderr]') ? 'stderr' : '');
                    addLine(msg.line, cls);
                } else if (msg.type === 'step_end') {
                    const cls = msg.status === 'passed' ? 'step-header' : 'step-header failed';
                    addLine(`--- ${msg.step_name}: ${msg.status.toUpperCase()} ---`, cls);
                } else if (msg.type === 'status') {
                    showBadge(msg.status);
                    if (msg.status === 'passed' || msg.status === 'failed') {
                        btn.disabled = false;
                        btn.textContent = 'Run Pipeline';
                        loadRuns();
                    }
                } else if (msg.type === 'error') {
                    addLine(`ERROR: ${msg.message}`, 'stderr');
                    btn.disabled = false;
                    btn.textContent = 'Run Pipeline';
                }
            };
        }

        function addLine(text, cls) {
            const terminal = document.getElementById('terminal');
            const div = document.createElement('div');
            div.className = `line ${cls}`;
            div.textContent = text;
            terminal.appendChild(div);
            terminal.scrollTop = terminal.scrollHeight;
        }

        function showBadge(status) {
            const badge = document.getElementById('statusBadge');
            badge.style.display = 'inline';
            badge.textContent = status;
            badge.className = `badge badge-${status}`;
        }

        async function loadRuns() {
            const res = await fetch('/runs');
            const runs = await res.json();
            const hist = document.getElementById('runHistory');
            hist.innerHTML = runs.slice(0, 5).map(r => `
                <div class="run-item">
                    <span>Run #${r.id}</span>
                    <span class="badge badge-${r.status}">${r.status}</span>
                </div>
            `).join('');
        }

        loadTemplate();
        loadRuns();

What does this code do?

runPipeline() disables the button, clears the terminal, POSTs the YAML to /run-yaml, then opens a WebSocket to /ws/runs/{run_id}. Each incoming message gets routed by msg.type to render step headers, log lines, or status updates.
addLine() appends a new div to the terminal and scrolls to the bottom automatically. The cls parameter applies CSS classes for color-coding (blue for commands, green for step headers, red for errors).
showBadge() updates the status pill in the header. Each status maps to a different badge color class.
loadRuns() fetches the 5 most recent runs from /runs and renders them in the run history panel below the editor.
The last two lines (loadTemplate() and loadRuns()) run on page load to pre-fill the editor and show any previous runs.

Save templates/dashboard.html.
Refresh your browser at http://localhost:8000. Make sure the "Node.js CI" template is selected and click the green Run Pipeline button.
You should see the status badge change from "queued" to "running", then watch log lines stream into the terminal panel in real-time. After a few seconds, the badge turns green with "passed".

Clicking Run but nothing happens?

Make sure Docker is running. The executor needs a live Docker daemon to create containers. Check with docker ps in your terminal.
If you see a "connection refused" error in the terminal panel, confirm your FastAPI dev server is still running in another terminal tab.
If the Node.js CI template fails because it cannot pull node:20-alpine, try the "Python Lint + Test" template first. It uses alpine:latest which is a smaller image that downloads faster.

help me debug why my pipeline run isn't streaming logs to the browser

✔️ Awesome, I've got everything!

Great. Double check you have saved templates/dashboard.html and that clicking Run Pipeline produces streaming logs.

ⓧ I'd like to double check the full code

Here is the complete templates/dashboard.html file:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Pipeline Runner</title>
    <style>
        * { margin: 0; padding: 0; box-sizing: border-box; }
        body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: #0d1117; color: #c9d1d9; min-height: 100vh; }
        .header { padding: 16px 24px; border-bottom: 1px solid #21262d; display: flex; align-items: center; justify-content: space-between; }
        .header h1 { font-size: 18px; color: #58a6ff; }
        .badge { padding: 4px 12px; border-radius: 12px; font-size: 12px; font-weight: 600; }
        .badge-queued { background: #362d00; color: #d29922; }
        .badge-running { background: #0c2d6b; color: #58a6ff; }
        .badge-passed { background: #0d2818; color: #3fb950; }
        .badge-failed { background: #3d1114; color: #f85149; }
        .container { display: flex; height: calc(100vh - 57px); }
        .editor-pane { width: 45%; border-right: 1px solid #21262d; display: flex; flex-direction: column; }
        .editor-header { padding: 12px 16px; border-bottom: 1px solid #21262d; display: flex; align-items: center; gap: 12px; }
        .editor-header select { background: #21262d; color: #c9d1d9; border: 1px solid #30363d; border-radius: 6px; padding: 6px 12px; font-size: 13px; }
        .editor-header .btn { padding: 8px 16px; }
        .yaml-editor { flex: 1; padding: 0; }
        .yaml-editor textarea { width: 100%; height: 100%; background: #161b22; color: #c9d1d9; border: none; padding: 16px; font-family: 'Courier New', monospace; font-size: 13px; line-height: 1.5; resize: none; outline: none; }
        .output-pane { flex: 1; display: flex; flex-direction: column; }
        .terminal { flex: 1; background: #161b22; padding: 16px; overflow-y: auto; font-family: 'Courier New', monospace; font-size: 13px; line-height: 1.6; }
        .terminal .line { white-space: pre-wrap; word-break: break-all; }
        .terminal .cmd { color: #79c0ff; }
        .terminal .step-header { color: #3fb950; font-weight: bold; margin-top: 12px; border-bottom: 1px solid #21262d; padding-bottom: 4px; }
        .terminal .step-header.failed { color: #f85149; }
        .terminal .stderr { color: #f85149; }
        .btn { padding: 10px 20px; border: none; border-radius: 6px; cursor: pointer; font-size: 13px; font-weight: 600; }
        .btn-primary { background: #238636; color: white; }
        .btn-primary:hover { background: #2ea043; }
        .btn-primary:disabled { background: #21262d; color: #484f58; cursor: not-allowed; }
        .run-history { padding: 12px 16px; border-top: 1px solid #21262d; max-height: 120px; overflow-y: auto; }
        .run-item { display: flex; align-items: center; justify-content: space-between; padding: 6px 8px; border-radius: 4px; margin-bottom: 4px; background: #161b22; }
        .run-item span { font-size: 11px; }
        .status-bar { padding: 10px 16px; border-top: 1px solid #21262d; font-size: 11px; color: #8b949e; }
    </style>
</head>
<body>
    <div class="header">
        <h1>Pipeline Runner</h1>
        <span id="statusBadge" class="badge badge-queued" style="display:none;">queued</span>
    </div>
    <div class="container">
        <div class="editor-pane">
            <div class="editor-header">
                <select id="templateSelect" onchange="loadTemplate()">
                    <option value="nodejs">Node.js CI</option>
                    <option value="python">Python Lint + Test</option>
                    <option value="deploy">Deploy Pipeline</option>
                    <option value="custom">Custom (blank)</option>
                </select>
                <button class="btn btn-primary" id="runBtn" onclick="runPipeline()">Run Pipeline</button>
            </div>
            <div class="yaml-editor">
                <textarea id="yamlEditor" spellcheck="false"></textarea>
            </div>
            <div class="run-history">
                <div style="font-size:11px; color:#8b949e; margin-bottom:6px;">RECENT RUNS</div>
                <div id="runHistory"></div>
            </div>
        </div>
        <div class="output-pane">
            <div class="terminal" id="terminal">
                <div class="line" style="color:#8b949e;">Select a template or write your own YAML, then click "Run Pipeline"...</div>
            </div>
            <div class="status-bar">Pipeline Runner v1.0 | Docker-powered isolated execution | Each step runs in a fresh container</div>
        </div>
    </div>
    <script>
        const templates = {
            nodejs: `name: nodejs-ci\nsteps:\n  - name: install\n    image: node:20-alpine\n    commands:\n      - echo "Creating package.json..."\n      - echo '{"name":"app","scripts":{"test":"echo All tests passed!"}}' > package.json\n      - npm install\n      - echo "Dependencies installed!"\n  - name: test\n    image: node:20-alpine\n    commands:\n      - echo "Running test suite..."\n      - sleep 1\n      - echo "PASS src/auth.test.js"\n      - echo "PASS src/api.test.js"\n      - echo "PASS src/db.test.js"\n      - echo "Tests: 3 passed, 3 total"\n  - name: build\n    image: node:20-alpine\n    commands:\n      - echo "Building production bundle..."\n      - sleep 1\n      - echo "Bundle size: 142kb (gzipped)"\n      - echo "Build complete!"`,
            python: `name: python-lint-test\nsteps:\n  - name: lint\n    image: alpine:latest\n    commands:\n      - echo "Running Python linter..."\n      - sleep 1\n      - echo "main.py - no issues"\n      - echo "utils.py - no issues"\n      - echo "tests/ - no issues"\n      - echo "All files passed linting!"\n  - name: test\n    image: alpine:latest\n    commands:\n      - echo "Running pytest..."\n      - sleep 2\n      - echo "test_auth.py::test_login PASSED"\n      - echo "test_auth.py::test_register PASSED"\n      - echo "test_api.py::test_get_users PASSED"\n      - echo "test_api.py::test_create_user PASSED"\n      - echo "4 passed in 1.23s"`,
            deploy: `name: deploy-pipeline\nsteps:\n  - name: setup\n    image: alpine:latest\n    commands:\n      - echo "Checking environment..."\n      - echo "Region: ap-south-1"\n      - echo "Cluster: production"\n      - echo "Environment ready!"\n  - name: build\n    image: alpine:latest\n    commands:\n      - echo "Building Docker image..."\n      - sleep 2\n      - echo "Layer 1/5: base image"\n      - echo "Layer 2/5: dependencies"\n      - echo "Layer 3/5: application code"\n      - echo "Layer 4/5: configuration"\n      - echo "Layer 5/5: entrypoint"\n      - echo "Image built: app:v1.2.3"\n  - name: deploy\n    image: alpine:latest\n    commands:\n      - echo "Deploying to production..."\n      - sleep 1\n      - echo "Rolling update started"\n      - echo "Pod 1/3 updated"\n      - echo "Pod 2/3 updated"\n      - echo "Pod 3/3 updated"\n      - echo "Deployment complete!"`,
            custom: `name: my-pipeline\nsteps:\n  - name: step-1\n    image: alpine:latest\n    commands:\n      - echo "Hello from step 1!"\n  - name: step-2\n    image: alpine:latest\n    commands:\n      - echo "Hello from step 2!"`
        };

        function loadTemplate() {
            const sel = document.getElementById('templateSelect').value;
            document.getElementById('yamlEditor').value = templates[sel];
        }

        async function runPipeline() {
            const yaml = document.getElementById('yamlEditor').value;
            if (!yaml.trim()) return;
            const btn = document.getElementById('runBtn');
            btn.disabled = true;
            btn.textContent = 'Running...';
            const terminal = document.getElementById('terminal');
            terminal.innerHTML = '';
            showBadge('queued');

            const res = await fetch('/run-yaml', {
                method: 'POST',
                headers: {'Content-Type': 'application/json'},
                body: JSON.stringify({yaml_content: yaml})
            });
            const run = await res.json();
            showBadge('running');

            const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
            const ws = new WebSocket(`${protocol}//${window.location.host}/ws/runs/${run.id}`);
            ws.onmessage = (event) => {
                const msg = JSON.parse(event.data);
                if (msg.type === 'step_start') {
                    addLine(`\n--- Step: ${msg.step_name} ---`, 'step-header');
                } else if (msg.type === 'log') {
                    const cls = msg.line.startsWith('$') ? 'cmd' : (msg.line.startsWith('[stderr]') ? 'stderr' : '');
                    addLine(msg.line, cls);
                } else if (msg.type === 'step_end') {
                    const cls = msg.status === 'passed' ? 'step-header' : 'step-header failed';
                    addLine(`--- ${msg.step_name}: ${msg.status.toUpperCase()} ---`, cls);
                } else if (msg.type === 'status') {
                    showBadge(msg.status);
                    if (msg.status === 'passed' || msg.status === 'failed') {
                        btn.disabled = false;
                        btn.textContent = 'Run Pipeline';
                        loadRuns();
                    }
                } else if (msg.type === 'error') {
                    addLine(`ERROR: ${msg.message}`, 'stderr');
                    btn.disabled = false;
                    btn.textContent = 'Run Pipeline';
                }
            };
        }

        function addLine(text, cls) {
            const terminal = document.getElementById('terminal');
            const div = document.createElement('div');
            div.className = `line ${cls}`;
            div.textContent = text;
            terminal.appendChild(div);
            terminal.scrollTop = terminal.scrollHeight;
        }

        function showBadge(status) {
            const badge = document.getElementById('statusBadge');
            badge.style.display = 'inline';
            badge.textContent = status;
            badge.className = `badge badge-${status}`;
        }

        async function loadRuns() {
            const res = await fetch('/runs');
            const runs = await res.json();
            const hist = document.getElementById('runHistory');
            hist.innerHTML = runs.slice(0, 5).map(r => `
                <div class="run-item">
                    <span>Run #${r.id}</span>
                    <span class="badge badge-${r.status}">${r.status}</span>
                </div>
            `).join('');
        }

        loadTemplate();
        loadRuns();
    </script>
</body>
</html>

Your pipeline runner now has a full interactive frontend. Visitors can pick a template, customize it, run it, and watch Docker containers execute their commands live. Next up, you will deploy this entire system to AWS EC2 so anyone on the internet can access it.

Deploy to AWS EC2

Your pipeline runner works locally. Now it needs a public URL that anyone can open, write their own YAML, and watch Docker containers execute in real-time.

In this step, you will launch an EC2 instance on AWS, install Docker and Python on it, deploy your code, and start the app on port 80. The result is a live public URL you can share on your resume.

In this step, get ready to:

Launch an EC2 instance and configure its security group.
Install Docker and clone your project onto the instance.
Start the app and test it from a public URL.

Launch an EC2 instance with Ubuntu 22.04

Your pipeline runner needs a Linux server with Docker installed. EC2 gives you a virtual machine in the cloud that anyone on the internet can reach.

Open the AWS Management Console at https://console.aws.amazon.com/ec2/.
Click Launch instance from the EC2 dashboard.
Under Name and tags, enter pipeline-runner as the name.
Under Application and OS Images, click Quick Start, then select Ubuntu.
Choose the Ubuntu Server 22.04 LTS AMI (marked Free tier eligible).
Under Instance type, select t3.micro.
Under Key pair (login), click Create new key pair.
Name it pipeline-runner-key, keep the defaults (RSA, .pem), and click Create key pair.
A .pem file downloads automatically. Save this file somewhere safe. You will need it to connect to your instance.

What is a key pair?

A key pair is how you securely log into your EC2 instance via SSH. The .pem file is your private key. Anyone with this file can access your server, so keep it private.

Now configure the security group to allow traffic on the ports your app needs.

Under Network settings, click Edit.
Keep the default SSH rule (port 22) that allows you to connect.
Click Add security group rule.
Set Type to HTTP. This opens port 80 to the public so anyone can access your dashboard.
Leave Source type as Anywhere.
Click Launch instance in the Summary panel.
Wait for the instance state to show Running and all status checks to pass.

What is a security group?

A security group acts as a firewall for your EC2 instance. It controls which network traffic can reach your server. By opening port 22 (SSH) and port 80 (HTTP), you allow yourself to log in and the public to view your dashboard.

Now SSH into your instance. You need to find your instance's public IP address first.

On the Instances page, click your instance's ID to open its details.
Copy the Public IPv4 address shown on the details page.
Open a terminal on your local machine and connect by running this command:

chmod 400 [[PEMPATH="path/to/pipeline-runner-key.pem"]]
ssh -i [[PEMPATH="path/to/pipeline-runner-key.pem"]] ubuntu@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]

What do these commands do?

`chmod 400` restricts the key file so only you can read it. SSH refuses to use a key with open permissions.
`ssh -i` connects to the remote server using your private key. The default username for Ubuntu AMIs is ubuntu.

When prompted to confirm the fingerprint, type yes and press Enter.

You should see a welcome message from Ubuntu and a command prompt ending in $. You are now logged into your EC2 instance.

Connection timed out or refused?

Double check that your security group has an inbound rule for SSH (port 22) with source 0.0.0.0/0.
Make sure you are using the correct public IPv4 address (not the private IP).
Verify the instance state is Running and status checks have passed.

Why can't I connect to my EC2 instance via SSH?

Now install Docker Engine on your EC2 instance. Ubuntu 22.04 already has Python 3.10 pre-installed, so you only need Docker.

Set up Docker's apt repository by running these commands:

# Add Docker's official GPG key
sudo apt update
sudo apt install ca-certificates curl -y
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to apt sources
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

What do these commands do?

These commands add Docker's official package repository to your system. This ensures you install Docker Engine directly from Docker (not the older docker.io package in Ubuntu's default repositories).

Install Docker Engine by running:

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Verify Docker is working by running:

sudo docker run hello-world

You should see a message that says "Hello from Docker!" confirming that Docker can pull images and run containers.

Docker command not found?

Make sure you ran sudo apt update after adding the Docker repository.
Check that no errors appeared during the GPG key download step.

Help me troubleshoot Docker installation on my Ubuntu EC2 instance.

Also install pip and git (you will need both shortly) by running:

sudo apt install python3-pip python3-venv git -y

Push your code to GitHub and clone it onto EC2

Your project code lives on your local machine. To get it onto EC2, you will push it to a GitHub repository and then clone it from there.

On your local machine (not the EC2 terminal), open a new terminal in your pipeline-runner/ folder.
Create a .gitignore file to exclude files that should not be committed by running:

echo -e "venv/\n__pycache__/\n*.db" > .gitignore

Initialize the repo and make your first commit by running these commands:

git init
git add .
git commit -m "Initial commit"

Go to https://github.com/new and create a new repository named pipeline-runner.
Keep it Public (so your EC2 instance can clone without authentication).
Do not initialize with a README.
Push your local code to GitHub by running:

git remote add origin [[YOUR_REPO_URL="https://github.com/YOUR_USERNAME/pipeline-runner.git"]]
git branch -M main
git push -u origin main

Why a public repository?

A public repository lets your EC2 instance clone the code without needing to set up SSH keys or personal access tokens on the server. This also makes it easy for interviewers to browse your source code.

Now switch back to your EC2 SSH session and clone the repository.

In your EC2 terminal, clone your repository by running:

cd ~
git clone [[YOUR_REPO_URL="https://github.com/YOUR_USERNAME/pipeline-runner.git"]]
cd pipeline-runner

Confirm the files are present by running:

ls

You should see main.py, executor.py, requirements.txt, and the other project files listed.

Create a virtual environment and install dependencies by running:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

What does this do?

This creates an isolated Python environment on the EC2 instance and installs FastAPI, SQLAlchemy, the Docker SDK, and PyYAML. The same packages you used locally.

Pull the Docker images your pipeline templates need by running:

sudo docker pull alpine:latest && sudo docker pull node:20-alpine

This downloads the container images so the first pipeline run does not have to wait for a pull.

Permission denied when pulling images?

Make sure you prefix Docker commands with sudo. The default ubuntu user is not in the docker group on a fresh install.
If you want to avoid typing sudo every time, run sudo usermod -aG docker ubuntu then log out and back in.

Help me fix Docker permission errors on my EC2 instance.

Start the app and test your live deployment

Your code is on the server, dependencies are installed, and Docker images are ready. Time to start the FastAPI app on port 80 so the world can reach it.

Start the FastAPI app on port 80 by running:

sudo $(which fastapi) run main.py --port 80

Why sudo and $(which fastapi)?

Port 80 requires root privileges on Linux. However, sudo does not load your virtual environment's PATH by default. The $(which fastapi) trick expands to the full path of the fastapi binary inside your venv, so sudo can find it.

The fastapi run command starts the app in production mode on 0.0.0.0 without auto-reload.

You should see output indicating the server started and is listening on port 80.

Open a new browser tab on your local machine.
Navigate to your instance's public IP address:

http://[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]]

You should see the Pipeline Runner dashboard with the YAML editor on the left and the terminal viewer on the right.
Select a template from the dropdown (try Node.js CI).
Click Run Pipeline.
Watch the real-time log output stream in the terminal pane as Docker containers spin up and execute your commands on the remote server.

The status badge should transition from "queued" to "running" to "passed". Your pipeline runner is live.

Dashboard not loading in the browser?

Check that your security group has an inbound HTTP rule on port 80.
Make sure you are using http:// (not https://). Your instance does not have TLS configured.
Verify the FastAPI process is still running in your SSH session (it should show active log lines).

Why can't I access my FastAPI app from the browser?

Keep the app running after you close the terminal

If you close your SSH session, the app stops. To keep it running in the background, press Ctrl+C to stop it, then restart it with nohup:

nohup sudo $(which fastapi) run main.py --port 80 > app.log 2>&1 &

This runs the server in the background and writes output to app.log.

Your CI/CD pipeline runner is now live on the internet. Anyone with the URL can write their own YAML workflow, pick a template, and watch Docker containers execute in real-time. This is the URL you put on your resume.

Secret mission

Add Parallel Step Execution and Webhook Triggers

Your pipeline runner executes steps one after another. Real CI/CD systems like GitHub Actions support parallel jobs and external triggers. Ready to upgrade your runner with concurrent execution, webhook endpoints, and a timeline API?

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses an AWS EC2 instance that consumes your free credits while running.

Cost warning

Your EC2 t3.micro instance costs approximately $0.01/hour from your AWS credits while running. If you are not actively showing this project to interviewers, stop or terminate the instance to preserve your credits.

Resources you used:

EC2 instance (t3.micro) with attached EBS volume (8 GB).
Security group allowing inbound HTTP (port 80) and SSH (port 22).
GitHub repository for your pipeline-runner code.
Local project files (pipeline-runner/ directory on your Desktop).

Keep everything running

No action needed. Choose this if you are still actively building or want interviewers to access your live demo URL.

Your EC2 instance stays running and accessible at its public IP address.
The instance costs approximately $0.01/hour from your AWS free credits.
Your local project files, virtual environment, and GitHub repository remain untouched.

Pause - I'll come back to this later

Stop your EC2 instance to pause billing for compute while keeping your data intact. You can restart it anytime.

Open the AWS EC2 console and navigate to Instances.
Select the checkbox next to your pipeline-runner instance.
Click Instance state, then select Stop instance.
Confirm the stop action. Your instance will shut down and compute billing pauses. The EBS volume still exists so you can restart later.

Restarting later

To resume, select your stopped instance and choose Instance state > Start instance. Your public IP address will change, so update any links you have shared.

Delete - I don't want to use this again

Remove all project resources permanently. Do this when you no longer need the live demo.

Terminate the EC2 instance: In the EC2 console, select your instance. Click Instance state, then select Terminate (delete) instance. This removes the instance and its attached EBS volume.
Delete the security group: In the EC2 console sidebar, click Security Groups. Select the security group you created for this project, click Actions, then Delete security groups.
Delete the GitHub repository: Open your repository on GitHub. Click Settings. Scroll down to the Danger Zone section and click Delete this repository. Type the repository name to confirm.
Delete local project files (macOS/Linux):

rm -rf ~/Desktop/pipeline-runner

Delete local project files (Windows):

rmdir /s /q %USERPROFILE%\Desktop\pipeline-runner

Nice Work!

You just built a production-grade CI/CD pipeline runner from scratch. This is real infrastructure engineering that proves you can build the same category of tooling that powers every software company.

You've learned how to:

Built a container orchestration engine using the Docker SDK for Python that programmatically spins up isolated containers and executes commands.
Implemented real-time log streaming via WebSocket so users can watch pipeline execution live in the browser.
Designed and deployed a full-stack pipeline runner with a YAML editor, template picker, and terminal-style dashboard on AWS EC2 with a public URL.
Added parallel step execution and webhook triggers to support enterprise-grade pipeline workflows.

Ready to quiz yourself?