Build a Live Market Data Gateway

Introduction

30 Second Summary

A stock price can change before you finish reading it. A screen that shows one number gives you no clue where it came from or whether it is still fresh.

In this project, you will build a Robinhood-inspired market-data gateway using Node.js with the Alpaca Market Data API. The finished system will share normalized trade and quote state across browser tabs from one upstream connection.

What You'll Build

Your final demo shows two browser tabs moving from reconnecting back to subscribed after you cut their one shared provider connection.

By the end of this project, you'll have:

• A live market dashboard that shows each symbol's last trade, bid, ask, midpoint, feed status, event age, and stale state.
• A resilience demo where both browser tabs recover from a forced upstream disconnect. A slow tab shows its coalesced-update count rising without an unbounded queue.
• A scale-up architecture that traces market data through Kafka, Redis, symbol partitions, and regional streaming gateways.
• Secret Mission: Redesign the scaled system so one hot symbol cannot overwhelm its Kafka partition or regional gateway fleet.

Are there any prerequisites?

You need a Mac with VS Code, Safari or Chrome, plus an Alpaca account. Comfort with JavaScript is the only technical prerequisite.

Before We Start

Before any hands-on work, lock in the educational market-data delivery system you are about to build. Trading actions stay outside the project because your goal is to explain the distinct trade, quote, derived midpoint, freshness, and feed-status signals behind a live market view.

Verify the Runtime and Protect Credentials

Your market-data gateway relies on Node.js with a stable built-in WebSocket client. A version mismatch can stop the probe before it reaches the provider.

Your Alpaca Market Data API credentials need their own boundary. A local .env file lets Node.js load them without placing them in browser code.

In this step, get ready to:

• Verify Node.js 24.20.0 in the integrated terminal.
• Create the market-data project files with probe-mode configuration.
• Store your Alpaca credentials outside browser code and shared source.

Verify Node.js 24.20.0

Node.js 24.20.0 provides the runtime used by this project. You will check the installed version from the terminal in VS Code.

• Press Cmd+Space on macOS or the Windows key on Windows to open system search.
• Type VS Code and press Enter to open it.
• Click Terminal in the top menu.
• Select New Terminal.
• Check your installed Node.js version by running this command:

node --version

What Does This Check?

The command reads the active Node.js runtime version. It does not change your installation or project files.

• Choose the tab that matches the output in your terminal.

✔️ I see version 24.20.0

Your terminal is using v24.20.0. Your runtime is ready for the market-data probe.

ⓧ I see an older version

The installed runtime is below the project pin. Update it before creating the project files.

• Open the official Node.js 24.20.0 release page.
• Choose the macOS package that matches your Mac processor.
• Complete the installer using the macOS prompts.
• Return to the integrated terminal in VS Code.
• Check the updated runtime by running this command:

node --version

What Does This Confirm?

The repeated check confirms which Node.js installation the integrated terminal now resolves. The output should read v24.20.0.

ⓧ Command not found

Your terminal cannot find a Node.js installation. Install the pinned release before continuing.

• Open the official Node.js 24.20.0 release page.
• Choose the macOS package that matches your Mac processor.
• Complete the installer using the macOS prompts.
• Return to the integrated terminal in VS Code.
• Check the new installation by running this command:

node --version

What Does This Confirm?

The command confirms that the terminal can now locate Node.js. The output should read v24.20.0.

Still Seeing the Wrong Version?

Close the existing integrated terminal after installation. Create a fresh terminal so VS Code reloads the available runtime paths.

If the old version still appears, help me identify which Node.js installation my VS Code terminal is using.

Create the protected project files

The project separates reusable configuration from private credentials. Its source uses ESM imports while two placeholder files reserve space for the later dashboard and architecture design.

The initial folder setup has several small clicks. Take them one screen at a time.

• Click File in the VS Code top menu.
• Select Open Folder.
• Select your Desktop.
• Click New Folder.
• Enter market-data-gateway as the folder name.
• Press Enter to create the folder.
• Select the market-data-gateway folder.
• Click Open.

Where Are You Now?

VS Code is now focused on the market-data-gateway folder on your Desktop. Every file you create next belongs to this project.

• Create .env.example in market-data-gateway with the New File control in the Explorer sidebar.
• Create .development.env in market-data-gateway with the New File control.
• Create .gitignore in market-data-gateway with the New File control.
• Create .env in market-data-gateway with the New File control.
• Create index.js in market-data-gateway with the New File control.
• Create ARCHITECTURE.md in market-data-gateway with the New File control.
• Create a public folder with the New Folder control.
• Create index.html inside public with the New File control.

The Explorer now shows six files beside the public folder. Inside public, you can see index.html.

• Fill .env.example with the reusable credential template by pasting this code:

APCA_API_KEY_ID=your-api-key-here
APCA_API_SECRET_KEY=your-api-secret-here

Why Keep an Example File?

The example records the required environment variables with dummy values. Teammates can copy the structure without receiving your credentials.

• Save .env.example.
• Confirm the file contains the two Alpaca variable names with placeholder values.

Missing One of the Credential Fields?

Check that .env.example contains one line for APCA_API_KEY_ID plus one line for APCA_API_SECRET_KEY.

If the template still looks different, help me compare my Alpaca environment template with the required two-line structure.

• Set the initial runtime behavior in .development.env by pasting this code:

RUN_MODE=probe
FEED_MODE=test

What Does This Configuration Control?

The RUN_MODE=probe value starts the terminal probe. The FEED_MODE=test value selects Alpaca's always-available test stream.

This combination lets the first data check work outside US equity trading hours.

• Save .development.env.
• Confirm the first line contains RUN_MODE=probe.
• Confirm the second line contains FEED_MODE=test.

Probe or Test Mode Missing?

Check that each setting occupies its own line. Remove any spaces around the equals signs.

If the values still fail to load later, help me inspect the formatting of my development environment file.

• Exclude the local credential file in .gitignore by pasting this code:

.env

Why Ignore the Local Environment File?

The ignore rule tells Git to leave .env out of tracked source. Your reusable example remains available while the real values stay local.

• Save .gitignore.
• Confirm the file contains only .env.

Local Environment File Still Unprotected?

Check that the ignored filename is exactly .env. Keep the leading full stop.

If the filenames are hard to distinguish, help me check whether my local credential file matches the ignore rule.

Credentials are the sensitive part of this setup. You will copy them into a local file that stays hidden from the source files you can safely share.

• Select .env.example in the Explorer sidebar.
• Press Cmd+A on macOS or Ctrl+A on Windows.
• Press Cmd+C on macOS or Ctrl+C on Windows.
• Select .env in the Explorer sidebar.
• Press Cmd+V on macOS or Ctrl+V on Windows.

What Did You Copy?

Your local .env now has the same two variable names as the example. Only this local copy receives the real Alpaca values.

• Sign in to your Alpaca dashboard.
• Select the API Keys area in the right sidebar.
• Select Generate New Keys if your account does not have a usable key pair.
• Return to .env in VS Code.
• Replace your-api-key-here with your Alpaca key ID.
• Replace your-api-secret-here with your Alpaca secret key.
• Save .env.

Keep the Secret Recoverable

Alpaca does not recover a lost secret value. Generate a new key pair if you no longer have the secret.

Never paste either credential into index.js or public/index.html.

• Switch to .gitignore so the credential values are no longer visible.

Add the probe-mode starter

The starter source keeps the provider connection inside Node.js. It also turns Alpaca's compact trade and quote fields into readable terminal records.

The current configuration runs only the probe path. The browser dashboard files remain empty placeholders for a later step.

• Select index.js in the Explorer sidebar.
• Use the tabs below to add or compare the initial probe source.

✔️ Awesome, I've got everything!

Great. Confirm that your index.js source includes the environment setup, message helpers, and runProbe() handling shown in the comparison tab.

ⓧ I'd like to double check the full code

import { createServer } from "node:http";
import { readFile } from "node:fs/promises";

const apiKey = process.env.APCA_API_KEY_ID;
const apiSecret = process.env.APCA_API_SECRET_KEY;
const runMode = process.env.RUN_MODE ?? "gateway";
const feedMode = process.env.FEED_MODE ?? "test";

if (!apiKey || !apiSecret) {
  console.error("Missing APCA_API_KEY_ID or APCA_API_SECRET_KEY.");
  process.exit(1);
}

const isTestFeed = feedMode === "test";
const streamUrl = isTestFeed
  ? "wss://stream.data.alpaca.markets/v2/test"
  : "wss://stream.data.alpaca.markets/v2/iex";
const symbols = isTestFeed ? ["FAKEPACA"] : ["AAPL", "AMD", "SPY"];

function sendAuthentication(socket) {
  socket.send(JSON.stringify({
    action: "auth",
    key: apiKey,
    secret: apiSecret,
  }));
}

function sendSubscriptions(socket) {
  socket.send(JSON.stringify({
    action: "subscribe",
    trades: symbols,
    quotes: symbols,
  }));
}

function normalizedConsoleRecord(message) {
  if (message.T === "t") {
    return {
      type: "trade",
      symbol: message.S,
      price: message.p,
      size: message.s,
      exchange: message.x,
      marketTimestamp: message.t,
    };
  }

  if (message.T === "q") {
    return {
      type: "quote",
      symbol: message.S,
      bidPrice: message.bp,
      bidSizeRoundLots: message.bs,
      askPrice: message.ap,
      askSizeRoundLots: message.as,
      marketTimestamp: message.t,
    };
  }

  return null;
}

function runProbe() {
  const socket = new WebSocket(streamUrl);

  socket.addEventListener("message", (event) => {
    const messages = JSON.parse(String(event.data));

    for (const message of messages) {
      if (message.T === "success" && message.msg === "connected") {
        console.log("connected");
        sendAuthentication(socket);
        continue;
      }

      if (message.T === "success" && message.msg === "authenticated") {
        console.log("authenticated");
        sendSubscriptions(socket);
        continue;
      }

      if (message.T === "subscription") {
        console.log("subscribed", message);
        continue;
      }

      if (message.T === "error") {
        console.error("stream error", message);
        continue;
      }

      const normalized = normalizedConsoleRecord(message);
      if (normalized) console.log(normalized);
    }
  });

  socket.addEventListener("error", () => {
    console.error("WebSocket transport error");
  });

  socket.addEventListener("close", (event) => {
    console.log("closed", event.code, event.reason);
  });
}

if (runMode === "probe") {
  runProbe();
} else {
  await runGateway();
}

How Is the Starter Organized?

• The imports reserve Node.js server and file helpers for the later gateway mode.
• The environment constants read credentials plus runtime choices from files loaded by Node.js.
• The feed selection uses FAKEPACA in test mode. Live mode selects AAPL, AMD, and SPY.
• The authentication and subscription helpers create the provider control messages.
• The normalization helper preserves trades and quotes as different record types.
• The runProbe() function handles connection events plus normalized terminal output.
• The current RUN_MODE=probe setting executes only the probe branch.

• Save index.js.
• Keep public/index.html empty as the dashboard placeholder.
• Keep ARCHITECTURE.md empty as the scale-design placeholder.
• Confirm .development.env still contains RUN_MODE=probe.
• Confirm .development.env still contains FEED_MODE=test.

Starter Source Does Not Match?

Compare the opening imports plus every helper name with the full-code reference. A missing brace or changed identifier can prevent the next step from running the probe.

If you cannot find the mismatch, help me compare my initial probe-mode index.js with the reference source.

Before you rerun the runtime check, what exact version do you expect the terminal to print?

• Return to the integrated terminal from earlier.
• Rerun the required version check with this command:

node --version

What Does the Final Check Prove?

This confirms that the integrated terminal still resolves the pinned runtime after the project setup. It isolates the runtime check from the feed test in the next step.

The terminal prints v24.20.0. Your runtime is ready to execute the probe.

• Expand the public folder in the Explorer sidebar.
• Confirm index.html appears inside public.
• Switch to index.js so your .env values remain hidden.

Final Version Check Changed?

Create a fresh integrated terminal if the final check shows a different runtime. The new terminal should resolve the Node.js installation you verified earlier.

If the output still differs, help me restore Node.js 24.20.0 in my VS Code terminal.

Your runtime is pinned. Your Alpaca credentials now stay behind a local environment boundary. Next, you will connect the probe and decode its first trade and quote messages.

Decode the Feed and Hit the Connection Limit

Your protected credentials can now reach the test feed through Node.js. This step turns that connection into evidence you can inspect.

A browser-per-provider-connection design looks convenient. Running two probes reveals whether the account can support that shape.

In this step, get ready to:

• Run one probe against the always-available test stream.
• Distinguish control messages from trade and quote records.
• Test a second concurrent provider connection.

Run the first probe

The Alpaca Market Data API sends each server message as a JSON array. The test stream uses FAKEPACA.

Your probe opens one WebSocket connection. Each control message unlocks the next stage.

• Use the tabs below to compare index.js with the probe version.

✔️ Awesome, I've got everything!

Your existing probe source matches the version needed for this experiment.

ⓧ I'd like to double check the full code

import { createServer } from "node:http";
import { readFile } from "node:fs/promises";

const apiKey = process.env.APCA_API_KEY_ID;
const apiSecret = process.env.APCA_API_SECRET_KEY;
const runMode = process.env.RUN_MODE ?? "gateway";
const feedMode = process.env.FEED_MODE ?? "test";

if (!apiKey || !apiSecret) {
  console.error("Missing APCA_API_KEY_ID or APCA_API_SECRET_KEY.");
  process.exit(1);
}

const isTestFeed = feedMode === "test";
const streamUrl = isTestFeed
  ? "wss://stream.data.alpaca.markets/v2/test"
  : "wss://stream.data.alpaca.markets/v2/iex";
const symbols = isTestFeed ? ["FAKEPACA"] : ["AAPL", "AMD", "SPY"];

function sendAuthentication(socket) {
  socket.send(JSON.stringify({
    action: "auth",
    key: apiKey,
    secret: apiSecret,
  }));
}

function sendSubscriptions(socket) {
  socket.send(JSON.stringify({
    action: "subscribe",
    trades: symbols,
    quotes: symbols,
  }));
}

function normalizedConsoleRecord(message) {
  if (message.T === "t") {
    return {
      type: "trade",
      symbol: message.S,
      price: message.p,
      size: message.s,
      exchange: message.x,
      marketTimestamp: message.t,
    };
  }

  if (message.T === "q") {
    return {
      type: "quote",
      symbol: message.S,
      bidPrice: message.bp,
      bidSizeRoundLots: message.bs,
      askPrice: message.ap,
      askSizeRoundLots: message.as,
      marketTimestamp: message.t,
    };
  }

  return null;
}

function runProbe() {
  const socket = new WebSocket(streamUrl);

  socket.addEventListener("message", (event) => {
    const messages = JSON.parse(String(event.data));

    for (const message of messages) {
      if (message.T === "success" && message.msg === "connected") {
        console.log("connected");
        sendAuthentication(socket);
        continue;
      }

      if (message.T === "success" && message.msg === "authenticated") {
        console.log("authenticated");
        sendSubscriptions(socket);
        continue;
      }

      if (message.T === "subscription") {
        console.log("subscribed", message);
        continue;
      }

      if (message.T === "error") {
        console.error("stream error", message);
        continue;
      }

      const normalized = normalizedConsoleRecord(message);
      if (normalized) console.log(normalized);
    }
  });

  socket.addEventListener("error", () => {
    console.error("WebSocket transport error");
  });

  socket.addEventListener("close", (event) => {
    console.log("closed", event.code, event.reason);
  });
}

if (runMode === "probe") {
  runProbe();
}

How Is the Probe Wired?

• The runProbe() function creates one provider connection with new WebSocket(streamUrl).
• The message handler parses each incoming array before inspecting its records.
• The authentication and subscription helpers advance the connection through its control-message sequence.
• The normalization helper converts provider fields into readable trade and quote objects.
• The transport handlers expose connection errors and close events in the terminal.

Before you run the probe, predict which control message appears before market records.

• Start the first probe in the existing terminal by running this command:

node --env-file=.env --env-file=.development.env index.js

How Does the Probe Flow Work?

• The first environment file supplies the local credentials.
• The second environment file selects RUN_MODE=probe with FEED_MODE=test.
• A success message with connected triggers sendAuthentication().
• A success message with authenticated triggers sendSubscriptions().
• A subscription message confirms the trade and quote channels for FAKEPACA.
• A t record becomes a normalized trade object.
• A q record becomes a normalized quote object.

Probe Stops Before Market Records?

• Confirm that .development.env still contains RUN_MODE=probe.
• Confirm that .development.env still contains FEED_MODE=test.
• Replace invalid values in the local .env file with the current Alpaca key pair.

Need help tracing the probe? Help me diagnose why my Alpaca probe does not authenticate or print test-feed records.

• Keep the first probe running while you inspect its output.

Compare trades with quotes

The normalized output separates completed transactions from standing market offers. That distinction prevents one displayed number from hiding how the market currently looks.

• Find an object with type set to trade in the first terminal output.
• Read its price value.
• Read its size value.
• Find an object with type set to quote.
• Compare bidPrice with askPrice.
• Compare bidSizeRoundLots with askSizeRoundLots.

Three Different Market Values

A trade captures a completed transaction at price for size shares.

A quote captures the standing bid price and ask price. Its displayed sizes use round lots.

The midpoint sits halfway between the bid and ask. No completed transaction is implied by that derived display value.

Open a second provider connection

The first probe currently owns one upstream stream. A browser-per-connection design requires another process to open the same provider endpoint.

Before you run the second probe, predict whether both processes can keep receiving the same stream.

• Create a second terminal tab with the plus control in the terminal panel.
• Start the second probe by running this command:

node --env-file=.env --env-file=.development.env index.js

Why Does the Second Connection Fail?

The second terminal should print an error record with code 406. Its message says connection limit exceeded.

Alpaca Basic permits one concurrent stream connection. The first probe already occupies that connection.

You found the architecture pressure point. A separate provider connection for every browser would make clients compete for the same account-level slot.

Second Probe Does Not Show Error 406?

• Confirm that the first terminal is still printing FAKEPACA records.
• Confirm that both terminal tabs use the same local .env file.
• Restart the first probe with the same command above if its process has stopped.

Still missing the designed failure? Help me reproduce the Alpaca connection-limit response with two probe processes.

• Stop the second probe by pressing Ctrl+C in its terminal tab.
• Stop the first probe by pressing Ctrl+C in its terminal tab.

Both terminal tabs should return to a shell prompt. No more market records should appear.

You have proved why the provider stream belongs behind a shared gateway. Next, you will centralize ingestion so every browser can use the same upstream connection.

Centralize Ingestion and Cache Latest Prices

The two-probe test exposed the account-level connection limit. A browser-per-provider design cannot grow beyond one client.

A single Node.js gateway now owns one upstream WebSocket. This shared boundary creates one place to normalize messages. It also creates one place to recover the feed.

In this step, get ready to:

• Switch the project from probe mode to gateway mode.
• Normalize trades and quotes into one latest-state record per symbol.
• Add bounded reconnect behavior that authenticates and resubscribes after each connection.

Switch to gateway mode

The run mode controls which entry path starts when Node.js loads index.js. Gateway mode keeps one process alive as the owner of the provider stream.

• Switch back to .development.env in the VS Code editor.
• Replace its contents with this configuration:

RUN_MODE=gateway
FEED_MODE=test

What does this configuration do?

• The RUN_MODE=gateway value selects runGateway() when the application starts.
• The FEED_MODE=test value keeps the project on the always-available Alpaca test stream.
• The test stream produces events for FAKEPACA outside normal market hours.

• Save .development.env.

The first line now reads RUN_MODE=gateway. Your next application run follows the shared-gateway path.

Build the shared ingestion state

Each symbol gets one Map record inside the latest-value cache. A new client can eventually receive that record immediately instead of waiting for another market event.

The record keeps completed trades separate from standing quotes. Its version increases whenever either side changes.

Why preserve two timestamps?

The provider timestamp records when the market event occurred. The local receivedAt value records when your gateway processed it.

Keeping both values supports market-time analysis. It also lets the gateway measure local event age.

The Alpaca Market Data API uses compact provider fields such as p for trade price. The normalizeMessage() function converts those fields into one stable internal model.

• Switch back to index.js from earlier.
• Keep the existing probe helpers in place.
• Add runGateway() beneath runProbe() by comparing your file with the cumulative reference below.
• Update the entry point so every non-probe run awaits runGateway().

✔️ Awesome, I've got everything!

• Save index.js.

Your file now contains the probe path plus the shared gateway path. Continue to the verification below.

ⓧ I'd like to double check the full code

import { createServer } from "node:http";
import { readFile } from "node:fs/promises";

const apiKey = process.env.APCA_API_KEY_ID;
const apiSecret = process.env.APCA_API_SECRET_KEY;
const runMode = process.env.RUN_MODE ?? "gateway";
const feedMode = process.env.FEED_MODE ?? "test";

if (!apiKey || !apiSecret) {
  console.error("Missing APCA_API_KEY_ID or APCA_API_SECRET_KEY.");
  process.exit(1);
}

const isTestFeed = feedMode === "test";
const streamUrl = isTestFeed
  ? "wss://stream.data.alpaca.markets/v2/test"
  : "wss://stream.data.alpaca.markets/v2/iex";
const symbols = isTestFeed ? ["FAKEPACA"] : ["AAPL", "AMD", "SPY"];

function sendAuthentication(socket) {
  socket.send(JSON.stringify({
    action: "auth",
    key: apiKey,
    secret: apiSecret,
  }));
}

function sendSubscriptions(socket) {
  socket.send(JSON.stringify({
    action: "subscribe",
    trades: symbols,
    quotes: symbols,
  }));
}

function normalizedConsoleRecord(message) {
  if (message.T === "t") {
    return {
      type: "trade",
      symbol: message.S,
      price: message.p,
      size: message.s,
      exchange: message.x,
      marketTimestamp: message.t,
    };
  }

  if (message.T === "q") {
    return {
      type: "quote",
      symbol: message.S,
      bidPrice: message.bp,
      bidSizeRoundLots: message.bs,
      askPrice: message.ap,
      askSizeRoundLots: message.as,
      marketTimestamp: message.t,
    };
  }

  return null;
}

function runProbe() {
  const socket = new WebSocket(streamUrl);

  socket.addEventListener("message", (event) => {
    const messages = JSON.parse(String(event.data));

    for (const message of messages) {
      if (message.T === "success" && message.msg === "connected") {
        console.log("connected");
        sendAuthentication(socket);
        continue;
      }

      if (message.T === "success" && message.msg === "authenticated") {
        console.log("authenticated");
        sendSubscriptions(socket);
        continue;
      }

      if (message.T === "subscription") {
        console.log("subscribed", message);
        continue;
      }

      if (message.T === "error") {
        console.error("stream error", message);
        continue;
      }

      const normalized = normalizedConsoleRecord(message);
      if (normalized) console.log(normalized);
    }
  });

  socket.addEventListener("error", () => {
    console.error("WebSocket transport error");
  });

  socket.addEventListener("close", (event) => {
    console.log("closed", event.code, event.reason);
  });
}

async function runGateway() {
  const host = "127.0.0.1";
  const port = 3000;
  const staleAfterMs = 15_000;
  const dashboardHtml = await readFile(
    new URL("./public/index.html", import.meta.url),
    { encoding: "utf8" },
  );

  const latestBySymbol = new Map(
    symbols.map((symbol) => [symbol, {
      symbol,
      version: 0,
      lastReceivedAt: null,
      trade: null,
      quote: null,
    }]),
  );

  const clients = new Set();
  const metrics = {
    upstreamConnectionsOpened: 0,
    reconnectsScheduled: 0,
    coalescedUpdates: 0,
  };
  const feedState = {
    state: "starting",
    detail: null,
    lastFrameAt: null,
    reconnectAttempt: 0,
  };

  let upstream = null;
  let socketGeneration = 0;
  let reconnectTimer = null;

  function snapshot() {
    const now = Date.now();
    return {
      generatedAt: now,
      feed: {
        mode: feedMode,
        coverage: isTestFeed ? "Alpaca always-on test stream" : "Real-time IEX single-exchange feed",
        symbols,
        ...feedState,
      },
      metrics: {
        ...metrics,
        browserClients: clients.size,
      },
      prices: [...latestBySymbol.values()].map((record) => ({
        ...record,
        stale: record.lastReceivedAt === null
          || now - record.lastReceivedAt > staleAfterMs,
        ageMs: record.lastReceivedAt === null
          ? null
          : now - record.lastReceivedAt,
      })),
    };
  }

  function updateFeedState(state, detail = null) {
    feedState.state = state;
    feedState.detail = detail;
  }

  function normalizeMessage(message) {
    if (message.T !== "t" && message.T !== "q") return null;

    const record = latestBySymbol.get(message.S);
    if (!record) return null;

    const receivedAt = Date.now();
    record.version += 1;
    record.lastReceivedAt = receivedAt;

    if (message.T === "t") {
      record.trade = {
        price: message.p,
        size: message.s,
        exchange: message.x,
        conditions: message.c ?? [],
        marketTimestamp: message.t,
        receivedAt,
      };
    } else {
      record.quote = {
        bidPrice: message.bp,
        bidSizeRoundLots: message.bs,
        bidExchange: message.bx,
        askPrice: message.ap,
        askSizeRoundLots: message.as,
        askExchange: message.ax,
        conditions: message.c ?? [],
        marketTimestamp: message.t,
        receivedAt,
      };
    }

    return record;
  }

  function scheduleReconnect() {
    if (reconnectTimer) return;

    feedState.reconnectAttempt += 1;
    metrics.reconnectsScheduled += 1;
    const delayMs = Math.min(
      1_000 * (2 ** (feedState.reconnectAttempt - 1)),
      30_000,
    );
    updateFeedState("reconnecting", `retry in ${delayMs} ms`);

    reconnectTimer = setTimeout(() => {
      reconnectTimer = null;
      connectUpstream();
    }, delayMs);
  }

  function connectUpstream() {
    const generation = ++socketGeneration;
    metrics.upstreamConnectionsOpened += 1;
    updateFeedState("connecting", streamUrl);

    const socket = new WebSocket(streamUrl);
    upstream = socket;

    socket.addEventListener("message", (event) => {
      if (generation !== socketGeneration) return;

      let messages;
      try {
        messages = JSON.parse(String(event.data));
      } catch {
        feedState.detail = "ignored malformed provider frame";
        return;
      }

      feedState.lastFrameAt = Date.now();

      for (const message of messages) {
        if (message.T === "success" && message.msg === "connected") {
          updateFeedState("authenticating");
          sendAuthentication(socket);
          continue;
        }

        if (message.T === "success" && message.msg === "authenticated") {
          console.log("authenticated");
          updateFeedState("subscribing");
          sendSubscriptions(socket);
          continue;
        }

        if (message.T === "subscription") {
          console.log("subscribed", symbols.join(", "));
          feedState.reconnectAttempt = 0;
          updateFeedState("subscribed");
          continue;
        }

        if (message.T === "error") {
          console.error("stream error", message);
          updateFeedState("provider-error", `${message.code}: ${message.msg}`);
          socket.close();
          continue;
        }

        const record = normalizeMessage(message);
        if (record) {
          console.log("cache", record.symbol, "version", record.version);
        }
      }
    });

    socket.addEventListener("error", () => {
      if (generation !== socketGeneration) return;
      feedState.detail = "WebSocket transport error";
    });

    socket.addEventListener("close", (event) => {
      if (generation !== socketGeneration) return;
      upstream = null;
      updateFeedState("disconnected", event.reason || `code ${event.code}`);
      scheduleReconnect();
    });
  }

  const server = createServer((request, response) => {
    if (request.method === "GET" && request.url.startsWith("/")) {
      response.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
      response.end(dashboardHtml);
      return;
    }

    response.writeHead(404, { "Content-Type": "text/plain; charset=utf-8" });
    response.end("Not found");
  });

  server.listen(port, host, () => {
    console.log(`Dashboard listening on http://${host}:${port}`);
    connectUpstream();
  });
}

if (runMode === "probe") {
  runProbe();
} else {
  await runGateway();
}

How does the gateway fit together?

• The latestBySymbol map gives every symbol one current record.
• The snapshot() function calculates event age from the latest local receipt time.
• The normalizeMessage() function preserves separate trade and quote objects.
• The scheduleReconnect() function increases the retry delay from one second up to a thirty-second cap.
• The socketGeneration value prevents callbacks from an old connection from changing the current gateway state.
• The connectUpstream() function authenticates every replacement connection before resubscribing.

• Compare each section with your copy of index.js.
• Save index.js.

Start the shared gateway

The gateway is ready to replace both competing probes. One process should now start the local server before opening one authenticated provider connection.

Before you run it, which sequence do you expect to see from server startup through the first cache update?

• Run the gateway from the existing VS Code terminal with this command:

node --env-file=.env --env-file=.development.env index.js

What does this command do?

The command loads variables from .env first. It then loads .development.env so the development settings select gateway mode.

The process keeps running because the local HTTP server remains available on port 3000.

You will see Dashboard listening on http://127.0.0.1:3000 first. The provider sequence then prints authenticated followed by subscribed.

New FAKEPACA events produce lines containing cache plus an increasing version number. That output proves one upstream stream is updating one canonical symbol record.

Gateway does not reach subscribed?

• Confirm both probe processes from the previous step remain stopped.
• Check that .development.env contains RUN_MODE=gateway.
• Check that your local .env still contains both Alpaca credentials.

Still stuck? Help me diagnose why my Node.js Alpaca gateway does not authenticate or subscribe.

That is the connection limit solved at the ingestion boundary. Next, you will fan this shared state out to browser clients without exposing provider credentials.

Fan Out Safely and Recover from Failure

Your Node.js gateway now normalizes market events through one upstream connection. That shared connection removes the provider limit you exposed earlier.

The gateway still needs a safe path to every browser. This step adds cached delivery through Server-Sent Events. It also makes stale values visible. Slow screens get bounded delivery instead of an ever-growing queue.

In this step, get ready to:

• Stream cached market snapshots to multiple browser tabs.
• Show trade age plus fresh or stale state for every symbol.
• Coalesce updates for slow clients plus recover from an upstream disconnect.

Serve cached snapshots to browser clients

A one-way market dashboard only needs the server to push updates toward the browser. Server-Sent Events provides that flow through one long-lived HTTP response per client.

Why Server-Sent Events?

The browser opens an event stream with EventSource. The gateway sends named snapshot events over the same response.

The upstream provider connection stays inside the gateway. Browser clients never receive your Alpaca credentials.

• Stop the running gateway by pressing Ctrl+C in the VS Code terminal.
• In index.js, replace the import section with this version:

import { createServer } from "node:http";
import { readFile } from "node:fs/promises";

What does this import do?

• The HTTP import creates the local dashboard server.
• The file-system import reads the complete browser page before the server starts.

• Inside runGateway(), add this block directly below the staleAfterMs declaration:

  const dashboardHtml = await readFile(
    new URL("./public/index.html", import.meta.url),
    { encoding: "utf8" },
  );

What does this code do?

The gateway reads public/index.html once during startup. Every dashboard request can then reuse the same HTML string.

• Save index.js.
• Check the new file-loading path by restarting the gateway with this command:

node --env-file=.env --env-file=.development.env index.js

What should you see?

You should see the dashboard address followed by authenticated plus subscribed. Cache-version logs should continue as test events arrive.

• Stop the gateway by pressing Ctrl+C.
• In index.js, replace the existing updateFeedState() function with this version:

  function updateFeedState(state, detail = null) {
    feedState.state = state;
    feedState.detail = detail;
    broadcastSnapshot();
  }

Why broadcast state changes?

Feed transitions matter even when no price changes. Broadcasting here lets every client see states such as reconnecting as soon as they occur.

• Add offerSnapshot() directly below snapshot() by copying this code:

  function offerSnapshot(client, payload) {
    if (client.response.writableEnded) return;

    if (client.blocked || client.pacing) {
      client.pending = payload;
      metrics.coalescedUpdates += 1;
      return;
    }

    const frame = `event: snapshot\ndata: ${JSON.stringify(payload)}\n\n`;
    const accepted = client.response.write(frame);
    if (!accepted) client.blocked = true;

    if (client.delayMs > 0) {
      client.pacing = true;
      client.timer = setTimeout(() => {
        client.pacing = false;
        flushClient(client);
      }, client.delayMs);
    }
  }

What does this code do?

• Each payload becomes a named snapshot event that the browser can handle.
• A writable client receives the frame immediately.
• A blocked or paced client retains only the newest payload.
• Slow mode waits two seconds before offering the next pending snapshot.

• Save index.js.
• Restart the gateway with this command:

node --env-file=.env --env-file=.development.env index.js

What should you see?

You should see the same authenticated gateway logs. This confirms the new delivery function loads without a JavaScript syntax error.

• Stop the gateway by pressing Ctrl+C.
• Add these delivery helpers directly below offerSnapshot() by copying this code:

  function flushClient(client) {
    if (client.blocked || client.pacing || !client.pending) return;
    const pending = client.pending;
    client.pending = null;
    offerSnapshot(client, pending);
  }

  function broadcastSnapshot() {
    const payload = snapshot();
    for (const client of clients) offerSnapshot(client, payload);
  }

How do these helpers work?

• The flush helper sends a pending payload when the client becomes writable again.
• The broadcast helper creates one snapshot for the current moment.
• Every connected client receives an offer for that same snapshot.

• Add addSseClient() directly below connectUpstream() by copying this code:

  function addSseClient(request, response) {
    response.writeHead(200, {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
    });
    response.write("retry: 1000\n\n");

    const client = {
      response,
      blocked: false,
      pacing: false,
      pending: null,
      delayMs: request.url.includes("slow=1") ? 2_000 : 0,
      timer: null,
    };

    clients.add(client);

    response.on("drain", () => {
      client.blocked = false;
      flushClient(client);
    });

    response.on("close", () => {
      clearTimeout(client.timer);
      clients.delete(client);
      broadcastSnapshot();
    });

    broadcastSnapshot();
  }

What does this client record track?

• The response stays open with the event-stream content type.
• The client record tracks blocked delivery plus simulated pacing.
• The drain handler resumes delivery after the response buffer clears.
• A newly connected browser receives the current cached snapshot immediately.

• Save index.js.
• Restart the gateway with this command:

node --env-file=.env --env-file=.development.env index.js

What should you see?

The gateway should authenticate again. Incoming test events should continue increasing the cache version.

Gateway fails after the delivery helpers?

Check that offerSnapshot(), flushClient(), plus broadcastSnapshot() are all inside runGateway().

Need another pair of eyes? Help me check the placement of my Server-Sent Events delivery functions.

Render stale-aware market state

A connected stream can still show an old value for one symbol. The dashboard therefore calculates freshness per symbol from the latest local receipt time.

• Stop the gateway by pressing Ctrl+C.
• Replace the existing snapshot() function in index.js with this stale-aware version:

  function snapshot() {
    const now = Date.now();
    return {
      generatedAt: now,
      feed: {
        mode: feedMode,
        coverage: isTestFeed ? "Alpaca always-on test stream" : "Real-time IEX single-exchange feed",
        symbols,
        ...feedState,
      },
      metrics: {
        ...metrics,
        browserClients: clients.size,
      },
      prices: [...latestBySymbol.values()].map((record) => ({
        ...record,
        stale: record.lastReceivedAt === null
          || now - record.lastReceivedAt > staleAfterMs,
        ageMs: record.lastReceivedAt === null
          ? null
          : now - record.lastReceivedAt,
      })),
    };
  }

How does staleness work?

• The snapshot includes the current feed state plus gateway metrics.
• Each symbol receives an age measured from its most recent provider event.
• A record becomes stale after the project's 15-second learning threshold.
• The last known trade plus quote remain visible while stale.

• Find the successful normalization block inside connectUpstream().
• Replace that block with this version so every cache update reaches browser clients:

        const record = normalizeMessage(message);
        if (record) {
          console.log("cache", record.symbol, "version", record.version);
          broadcastSnapshot();
        }

Why broadcast after normalization?

The cache update is the canonical change. Broadcasting immediately lets every client receive the newest trade or quote state.

• Replace the existing server block near the bottom of runGateway() with this routing block:

  const server = createServer((request, response) => {
    if (request.method === "GET" && request.url.startsWith("/events")) {
      addSseClient(request, response);
      return;
    }

    if (request.method === "POST" && request.url === "/debug/disconnect") {
      upstream?.close(4000, "failure drill");
      response.writeHead(204);
      response.end();
      return;
    }

    if (request.method === "GET" && request.url.startsWith("/")) {
      response.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
      response.end(dashboardHtml);
      return;
    }

    response.writeHead(404, { "Content-Type": "text/plain; charset=utf-8" });
    response.end("Not found");
  });

  setInterval(broadcastSnapshot, 1_000);

  server.listen(port, host, () => {
    console.log(`Dashboard listening on http://${host}:${port}`);
    connectUpstream();
  });

What does the server expose?

• The events route registers normal or slow Server-Sent Events clients.
• The debug route closes the active upstream socket for the failure drill.
• The page route serves the browser dashboard.
• The timer refreshes event age plus stale state every second.

Your gateway logic is complete. Use the next tabs to compare both files against the finished local system.

✔️ Awesome, I've got everything!

Save index.js before moving to the browser dashboard.

ⓧ I'd like to double check the full code

import { createServer } from "node:http";
import { readFile } from "node:fs/promises";

const apiKey = process.env.APCA_API_KEY_ID;
const apiSecret = process.env.APCA_API_SECRET_KEY;
const runMode = process.env.RUN_MODE ?? "gateway";
const feedMode = process.env.FEED_MODE ?? "test";

if (!apiKey || !apiSecret) {
  console.error("Missing APCA_API_KEY_ID or APCA_API_SECRET_KEY.");
  process.exit(1);
}

const isTestFeed = feedMode === "test";
const streamUrl = isTestFeed
  ? "wss://stream.data.alpaca.markets/v2/test"
  : "wss://stream.data.alpaca.markets/v2/iex";
const symbols = isTestFeed ? ["FAKEPACA"] : ["AAPL", "AMD", "SPY"];

function sendAuthentication(socket) {
  socket.send(JSON.stringify({
    action: "auth",
    key: apiKey,
    secret: apiSecret,
  }));
}

function sendSubscriptions(socket) {
  socket.send(JSON.stringify({
    action: "subscribe",
    trades: symbols,
    quotes: symbols,
  }));
}

function normalizedConsoleRecord(message) {
  if (message.T === "t") {
    return {
      type: "trade",
      symbol: message.S,
      price: message.p,
      size: message.s,
      exchange: message.x,
      marketTimestamp: message.t,
    };
  }

  if (message.T === "q") {
    return {
      type: "quote",
      symbol: message.S,
      bidPrice: message.bp,
      bidSizeRoundLots: message.bs,
      askPrice: message.ap,
      askSizeRoundLots: message.as,
      marketTimestamp: message.t,
    };
  }

  return null;
}

function runProbe() {
  const socket = new WebSocket(streamUrl);

  socket.addEventListener("message", (event) => {
    const messages = JSON.parse(String(event.data));

    for (const message of messages) {
      if (message.T === "success" && message.msg === "connected") {
        console.log("connected");
        sendAuthentication(socket);
        continue;
      }

      if (message.T === "success" && message.msg === "authenticated") {
        console.log("authenticated");
        sendSubscriptions(socket);
        continue;
      }

      if (message.T === "subscription") {
        console.log("subscribed", message);
        continue;
      }

      if (message.T === "error") {
        console.error("stream error", message);
        continue;
      }

      const normalized = normalizedConsoleRecord(message);
      if (normalized) console.log(normalized);
    }
  });

  socket.addEventListener("error", () => {
    console.error("WebSocket transport error");
  });

  socket.addEventListener("close", (event) => {
    console.log("closed", event.code, event.reason);
  });
}

async function runGateway() {
  const host = "127.0.0.1";
  const port = 3000;
  const staleAfterMs = 15_000;
  const dashboardHtml = await readFile(
    new URL("./public/index.html", import.meta.url),
    { encoding: "utf8" },
  );

  const latestBySymbol = new Map(
    symbols.map((symbol) => [symbol, {
      symbol,
      version: 0,
      lastReceivedAt: null,
      trade: null,
      quote: null,
    }]),
  );

  const clients = new Set();
  const metrics = {
    upstreamConnectionsOpened: 0,
    reconnectsScheduled: 0,
    coalescedUpdates: 0,
  };
  const feedState = {
    state: "starting",
    detail: null,
    lastFrameAt: null,
    reconnectAttempt: 0,
  };

  let upstream = null;
  let socketGeneration = 0;
  let reconnectTimer = null;

  function snapshot() {
    const now = Date.now();
    return {
      generatedAt: now,
      feed: {
        mode: feedMode,
        coverage: isTestFeed ? "Alpaca always-on test stream" : "Real-time IEX single-exchange feed",
        symbols,
        ...feedState,
      },
      metrics: {
        ...metrics,
        browserClients: clients.size,
      },
      prices: [...latestBySymbol.values()].map((record) => ({
        ...record,
        stale: record.lastReceivedAt === null
          || now - record.lastReceivedAt > staleAfterMs,
        ageMs: record.lastReceivedAt === null
          ? null
          : now - record.lastReceivedAt,
      })),
    };
  }

  function offerSnapshot(client, payload) {
    if (client.response.writableEnded) return;

    if (client.blocked || client.pacing) {
      client.pending = payload;
      metrics.coalescedUpdates += 1;
      return;
    }

    const frame = `event: snapshot\ndata: ${JSON.stringify(payload)}\n\n`;
    const accepted = client.response.write(frame);
    if (!accepted) client.blocked = true;

    if (client.delayMs > 0) {
      client.pacing = true;
      client.timer = setTimeout(() => {
        client.pacing = false;
        flushClient(client);
      }, client.delayMs);
    }
  }

  function flushClient(client) {
    if (client.blocked || client.pacing || !client.pending) return;
    const pending = client.pending;
    client.pending = null;
    offerSnapshot(client, pending);
  }

  function broadcastSnapshot() {
    const payload = snapshot();
    for (const client of clients) offerSnapshot(client, payload);
  }

  function updateFeedState(state, detail = null) {
    feedState.state = state;
    feedState.detail = detail;
    broadcastSnapshot();
  }

  function normalizeMessage(message) {
    if (message.T !== "t" && message.T !== "q") return null;

    const record = latestBySymbol.get(message.S);
    if (!record) return null;

    const receivedAt = Date.now();
    record.version += 1;
    record.lastReceivedAt = receivedAt;

    if (message.T === "t") {
      record.trade = {
        price: message.p,
        size: message.s,
        exchange: message.x,
        conditions: message.c ?? [],
        marketTimestamp: message.t,
        receivedAt,
      };
    } else {
      record.quote = {
        bidPrice: message.bp,
        bidSizeRoundLots: message.bs,
        bidExchange: message.bx,
        askPrice: message.ap,
        askSizeRoundLots: message.as,
        askExchange: message.ax,
        conditions: message.c ?? [],
        marketTimestamp: message.t,
        receivedAt,
      };
    }

    return record;
  }

  function scheduleReconnect() {
    if (reconnectTimer) return;

    feedState.reconnectAttempt += 1;
    metrics.reconnectsScheduled += 1;
    const delayMs = Math.min(
      1_000 * (2 ** (feedState.reconnectAttempt - 1)),
      30_000,
    );
    updateFeedState("reconnecting", `retry in ${delayMs} ms`);

    reconnectTimer = setTimeout(() => {
      reconnectTimer = null;
      connectUpstream();
    }, delayMs);
  }

  function connectUpstream() {
    const generation = ++socketGeneration;
    metrics.upstreamConnectionsOpened += 1;
    updateFeedState("connecting", streamUrl);

    const socket = new WebSocket(streamUrl);
    upstream = socket;

    socket.addEventListener("message", (event) => {
      if (generation !== socketGeneration) return;

      let messages;
      try {
        messages = JSON.parse(String(event.data));
      } catch {
        feedState.detail = "ignored malformed provider frame";
        broadcastSnapshot();
        return;
      }

      feedState.lastFrameAt = Date.now();

      for (const message of messages) {
        if (message.T === "success" && message.msg === "connected") {
          updateFeedState("authenticating");
          sendAuthentication(socket);
          continue;
        }

        if (message.T === "success" && message.msg === "authenticated") {
          console.log("authenticated");
          updateFeedState("subscribing");
          sendSubscriptions(socket);
          continue;
        }

        if (message.T === "subscription") {
          console.log("subscribed", symbols.join(", "));
          feedState.reconnectAttempt = 0;
          updateFeedState("subscribed");
          continue;
        }

        if (message.T === "error") {
          console.error("stream error", message);
          updateFeedState("provider-error", `${message.code}: ${message.msg}`);
          socket.close();
          continue;
        }

        const record = normalizeMessage(message);
        if (record) {
          console.log("cache", record.symbol, "version", record.version);
          broadcastSnapshot();
        }
      }
    });

    socket.addEventListener("error", () => {
      if (generation !== socketGeneration) return;
      feedState.detail = "WebSocket transport error";
      broadcastSnapshot();
    });

    socket.addEventListener("close", (event) => {
      if (generation !== socketGeneration) return;
      upstream = null;
      updateFeedState("disconnected", event.reason || `code ${event.code}`);
      scheduleReconnect();
    });
  }

  function addSseClient(request, response) {
    response.writeHead(200, {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
    });
    response.write("retry: 1000\n\n");

    const client = {
      response,
      blocked: false,
      pacing: false,
      pending: null,
      delayMs: request.url.includes("slow=1") ? 2_000 : 0,
      timer: null,
    };

    clients.add(client);

    response.on("drain", () => {
      client.blocked = false;
      flushClient(client);
    });

    response.on("close", () => {
      clearTimeout(client.timer);
      clients.delete(client);
      broadcastSnapshot();
    });

    broadcastSnapshot();
  }

  const server = createServer((request, response) => {
    if (request.method === "GET" && request.url.startsWith("/events")) {
      addSseClient(request, response);
      return;
    }

    if (request.method === "POST" && request.url === "/debug/disconnect") {
      upstream?.close(4000, "failure drill");
      response.writeHead(204);
      response.end();
      return;
    }

    if (request.method === "GET" && request.url.startsWith("/")) {
      response.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
      response.end(dashboardHtml);
      return;
    }

    response.writeHead(404, { "Content-Type": "text/plain; charset=utf-8" });
    response.end("Not found");
  });

  setInterval(broadcastSnapshot, 1_000);

  server.listen(port, host, () => {
    console.log(`Dashboard listening on http://${host}:${port}`);
    connectUpstream();
  });
}

if (runMode === "probe") {
  runProbe();
} else {
  await runGateway();
}

• In the VS Code Explorer sidebar, select public/index.html.
• Replace its placeholder content with the complete dashboard from the second tab below.

✔️ Awesome, I've got everything!

Save public/index.html after replacing the placeholder.

ⓧ I'd like to double check the full code

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Market Data Gateway</title>
  <style>
    :root {
      color-scheme: dark;
      font-family: Inter, ui-sans-serif, system-ui, sans-serif;
      background: #07110d;
      color: #f4fff8;
    }

    body {
      margin: 0;
      min-height: 100vh;
      background: radial-gradient(circle at top, #123828, #07110d 55%);
    }

    main {
      width: min(1100px, calc(100% - 32px));
      margin: 0 auto;
      padding: 32px 0 64px;
    }

    header, .panel, .card {
      border: 1px solid #28513e;
      background: rgba(8, 26, 18, 0.9);
      border-radius: 16px;
      box-shadow: 0 18px 50px rgba(0, 0, 0, 0.25);
    }

    header, .panel {
      padding: 20px;
      margin-bottom: 18px;
    }

    h1, h2, p {
      margin-top: 0;
    }

    .architecture {
      color: #a9eac6;
      font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
      overflow-x: auto;
    }

    .controls, .metrics, .cards {
      display: flex;
      gap: 12px;
      flex-wrap: wrap;
    }

    button, a {
      border: 1px solid #39d98a;
      background: #123828;
      color: #dffff0;
      border-radius: 10px;
      padding: 10px 14px;
      text-decoration: none;
      cursor: pointer;
    }

    .metric {
      min-width: 150px;
      color: #b8cfc1;
    }

    .metric strong {
      display: block;
      color: #ffffff;
      font-size: 1.1rem;
    }

    .cards {
      align-items: stretch;
    }

    .card {
      flex: 1 1 280px;
      padding: 18px;
    }

    .price {
      font-size: 2rem;
      font-weight: 750;
      margin: 8px 0;
    }

    .quote-grid {
      display: grid;
      grid-template-columns: 1fr 1fr;
      gap: 10px;
    }

    .label, .muted {
      color: #91ac9d;
      font-size: 0.86rem;
    }

    .pill {
      display: inline-block;
      padding: 4px 9px;
      border-radius: 999px;
      background: #174d34;
      color: #bff7d8;
      font-size: 0.8rem;
    }

    .pill.stale {
      background: #5a3612;
      color: #ffd6a8;
    }

    .warning {
      color: #ffd6a8;
    }
  </style>
</head>
<body>
  <main>
    <header>
      <h1>Robinhood-Style Market Data Gateway</h1>
      <p class="architecture">Alpaca WebSocket -> ingestion and normalization -> latest-price cache -> browser fan-out</p>
      <div class="controls">
        <button id="disconnect">Disconnect upstream</button>
        <a href="/?slow=1" target="_blank" rel="noreferrer">Open slow-consumer tab</a>
      </div>
    </header>

    <section class="panel">
      <h2>System state</h2>
      <div class="metrics">
        <div class="metric"><span class="label">Browser stream</span><strong id="browserState">connecting</strong></div>
        <div class="metric"><span class="label">Upstream state</span><strong id="upstreamState">starting</strong></div>
        <div class="metric"><span class="label">Feed coverage</span><strong id="coverage">unknown</strong></div>
        <div class="metric"><span class="label">Browser clients</span><strong id="clients">0</strong></div>
        <div class="metric"><span class="label">Upstream opens</span><strong id="opens">0</strong></div>
        <div class="metric"><span class="label">Coalesced updates</span><strong id="coalesced">0</strong></div>
      </div>
      <p id="detail" class="muted"></p>
    </section>

    <section id="cards" class="cards"></section>

    <section class="panel">
      <h2>What does live price mean?</h2>
      <p>A last trade is a completed transaction. A quote is a standing best bid and ask on the selected feed. The midpoint is derived from the quote and is not itself a completed trade. A fresh transport connection also does not guarantee a fresh symbol value, so each card carries its own age and stale state.</p>
      <p class="warning">The free live feed covers IEX, a single exchange, rather than the consolidated SIP across all US exchanges. This demo is educational and is not investment advice.</p>
    </section>
  </main>

  <script>
    const slowMode = new URLSearchParams(window.location.search).get("slow") === "1";
    const eventPath = slowMode ? "/events?slow=1" : "/events";
    const source = new EventSource(eventPath);

    const browserState = document.getElementById("browserState");
    const upstreamState = document.getElementById("upstreamState");
    const coverage = document.getElementById("coverage");
    const clients = document.getElementById("clients");
    const opens = document.getElementById("opens");
    const coalesced = document.getElementById("coalesced");
    const detail = document.getElementById("detail");
    const cards = document.getElementById("cards");

    source.onopen = () => {
      browserState.textContent = slowMode ? "connected, slow mode" : "connected";
    };

    source.onerror = () => {
      browserState.textContent = "reconnecting";
    };

    source.addEventListener("snapshot", (event) => {
      render(JSON.parse(event.data));
    });

    document.getElementById("disconnect").addEventListener("click", async () => {
      await fetch("/debug/disconnect", { method: "POST" });
    });

    function money(value) {
      return typeof value === "number" ? `$${value.toFixed(2)}` : "waiting";
    }

    function ageLabel(ageMs) {
      return ageMs === null ? "no event received" : `${(ageMs / 1000).toFixed(1)}s old`;
    }

    function render(payload) {
      upstreamState.textContent = payload.feed.state;
      coverage.textContent = payload.feed.coverage;
      clients.textContent = payload.metrics.browserClients;
      opens.textContent = payload.metrics.upstreamConnectionsOpened;
      coalesced.textContent = payload.metrics.coalescedUpdates;
      detail.textContent = payload.feed.detail ?? "No active error";

      cards.innerHTML = payload.prices.map((record) => {
        const trade = record.trade;
        const quote = record.quote;
        const midpoint = quote
          ? (quote.bidPrice + quote.askPrice) / 2
          : null;
        const stateClass = record.stale ? "pill stale" : "pill";
        const stateText = record.stale ? "stale or waiting" : "fresh";

        return `
          <article class="card">
            <span class="${stateClass}">${stateText}</span>
            <h2>${record.symbol}</h2>
            <div class="label">Last completed trade</div>
            <div class="price">${money(trade?.price)}</div>
            <div class="muted">Size: ${trade?.size ?? "waiting"} | Exchange: ${trade?.exchange ?? "waiting"}</div>
            <hr>
            <div class="quote-grid">
              <div>
                <div class="label">Bid</div>
                <strong>${money(quote?.bidPrice)}</strong>
                <div class="muted">${quote?.bidSizeRoundLots ?? "waiting"} round lots</div>
              </div>
              <div>
                <div class="label">Ask</div>
                <strong>${money(quote?.askPrice)}</strong>
                <div class="muted">${quote?.askSizeRoundLots ?? "waiting"} round lots</div>
              </div>
            </div>
            <p><span class="label">Derived midpoint:</span> ${money(midpoint)}</p>
            <p class="muted">Version ${record.version} | ${ageLabel(record.ageMs)}</p>
          </article>
        `;
      }).join("");
    }
  </script>
</body>
</html>

• Save public/index.html.
• Start the completed gateway with this command:

node --env-file=.env --env-file=.development.env index.js

What should you see?

You should see the gateway listen on http://127.0.0.1:3000. The terminal should also show authentication plus subscription success.

• Enter http://127.0.0.1:3000 in the Safari or Chrome address bar.

You should see one FAKEPACA card with a last trade plus quote fields. The system panel should show one browser client plus one upstream open.

That is the shared gateway working. A newly connected browser receives cached state without creating another Alpaca stream.

Test slow delivery and upstream recovery

A slow browser creates backpressure when updates arrive faster than the response can accept them. The gateway protects memory by keeping one newest pending snapshot per slow client.

What gets coalesced?

Coalescing applies to client-facing latest-state snapshots. A market screen needs the newest visible state after it catches up.

Authoritative trade history still belongs in a durable event path. Intermediate trades cannot be discarded when audit or chart reconstruction depends on them.

• Select Open slow-consumer tab in the normal dashboard.

The new tab should show connected, slow mode. Both tabs should report two browser clients while the upstream-open count remains at one.

• Watch the Coalesced updates metric in either browser tab.

The count should increase as the gateway replaces older pending snapshots for the paced client. Memory stays bounded to one pending snapshot for that browser.

Before the failure drill, what sequence do you expect both tabs to show when the active provider socket closes?

• Select Disconnect upstream in the normal dashboard.

Both tabs should move through disconnected or reconnecting states. They should return to subscribed after the gateway reconnects.

The upstream-open count should increase by one. The browser-client count should remain at two.

You have now proved the complete recovery path. The gateway reconnects to Alpaca. It authenticates again. It resubscribes before new cache updates resume.

Dashboard does not recover?

• Keep the terminal process running while you select the disconnect control.
• Confirm .development.env still contains RUN_MODE=gateway plus FEED_MODE=test.
• Refresh both tabs if the browser event stream was interrupted separately from the provider connection.

Still stuck? Help me debug why my market-data dashboard does not reconnect and resubscribe after the failure drill.

Your gateway now serves many screens through one upstream connection. Next, you will map this local path onto durable ingestion, latest-state caching, plus regional streaming fleets.

Design the Scaled Market Data Platform

Your local Node.js gateway now keeps one upstream WebSocket connection to the Alpaca Market Data API. Normal browser tabs receive bounded snapshots through Server-Sent Events.

That local success still concentrates every responsibility inside one process. A process failure can interrupt the complete data path.

A scaled market-data platform separates durable ingestion from latest-state serving. It also gives regional fan-out its own recovery boundary.

In this step, get ready to:

• Map the local gateway onto a scaled market-data architecture.
• Separate the durable event backbone from the latest-state cache.
• Trace recovery across each failure boundary.

Map the local path to scaled components

The local Map holds current state inside the gateway. Server-Sent Events carry that state to browser clients.

At scale, Apache Kafka holds durable events. Redis serves current snapshots.

Regional streaming gateways own client connections. This separation limits the blast radius of each failure.

• Select ARCHITECTURE.md in the VS Code Explorer sidebar.
• Select the existing placeholder sentence.
• Replace the placeholder with the local architecture by pasting the following content:

# Market Data Architecture

This is an educational Robinhood-style design. It is not a claim about Robinhood's private implementation.

## Local system

```text
Alpaca IEX or test WebSocket
  -> one Node.js ingestion connection
  -> provider-message normalizer
  -> in-memory Map keyed by symbol
  -> Server-Sent Events gateway
  -> normal and slow browser clients
```

The local cache stores two different facts per symbol:

- Latest trade: a completed transaction with price, size, exchange, conditions, and market timestamp.
- Latest quote: the current bid and ask on the selected feed, including round-lot sizes and exchanges.

The displayed midpoint is derived from bid and ask. It is not a trade. Freshness is measured from local receipt time, while the provider timestamp remains available for market-time reasoning.

How Does the Local Map Work?

• The diagram follows data from the provider connection to the browser clients.
• The cache preserves trades as completed transactions.
• The cache preserves quotes as standing bid and ask values.
• Local receipt time measures freshness without discarding the provider timestamp.

• Save ARCHITECTURE.md.
• Press Cmd+Shift+V on macOS or Ctrl+Shift+V on Windows to open the Markdown preview.

You should see the local pipeline as a six-stage text diagram. You should also see separate definitions for the latest trade and latest quote.

Local Diagram Missing From the Preview?

• Confirm that the placeholder sentence was removed before you pasted the architecture.
• Check that the opening and closing code fences each contain three backticks.

Still stuck? Help me fix the local diagram in my ARCHITECTURE.md Markdown preview.

• Switch back to the ARCHITECTURE.md editor tab.
• Add the scaled path at the end of ARCHITECTURE.md by pasting the following content:

## Scaled system

```text
Exchanges and market-data providers
  -> redundant regional ingestion workers
  -> normalization and validation
  -> Kafka market-events topic, keyed by symbol
  -> independent consumer groups
       -> latest-price projectors -> Redis price cache
       -> audit and history storage
       -> alerting and analytics
  -> sharded live-update distribution
  -> regional streaming gateway fleets
  -> millions of browser and mobile clients
```

## Component responsibilities

| Layer | Responsibility | Failure and recovery |
| --- | --- | --- |
| Ingestion workers | Maintain provider sessions, decode wire messages, normalize schemas, attach source metadata, and resubscribe after reconnect | Run redundant workers with lease or ownership rules so replacement does not create uncontrolled duplicate sessions |
| Kafka | Keep a durable, replayable event log and decouple ingestion from cache, analytics, and gateway consumers | Replication protects partitions; consumers resume from offsets or replay to rebuild derived state |
| Symbol partitioning | Route the same symbol to the same partition so its events retain partition order | Reassign failed consumers; monitor skew because a hot symbol can dominate one partition |
| Redis price cache | Serve the newest normalized trade and quote state for fast snapshots and reconnect bootstrap | Rebuild keys from Kafka replay or a compacted latest-state topic; treat cache loss as recoverable |
| Live-update distribution | Move fresh changes from cache projectors toward gateways | Redis Pub/Sub is suitable only for transient fan-out because disconnected subscribers miss messages; Kafka or Redis Streams is needed when replay matters |
| Regional gateways | Hold client connections, authorize subscriptions, fetch an initial Redis snapshot, and push incremental updates | Use multiple instances behind regional traffic routing; a reconnecting client lands on another instance and receives a new snapshot |
| Client | Render trade and quote state, track versions and age, and label stale values | Reconnect, request a fresh snapshot, and never assume transport health proves symbol freshness |

What Does the Scaled Path Change?

• A symbol key sends each symbol's events to a consistent Kafka partition.
• Independent consumer groups let cache projectors and analytics process the same durable events.
• Redis latest-price records give reconnecting clients an immediate snapshot.
• Regional gateway fleets keep millions of client connections away from ingestion workers.

• Save ARCHITECTURE.md.
• Return to the Markdown preview.

You should see a scaled pipeline ending in millions of clients. A seven-row table should assign one responsibility and one recovery path to each layer.

Scaled Table Not Rendering Correctly?

• Confirm that the scaled diagram's closing code fence appears before the component table heading.
• Check that every table row starts and ends with a pipe character.

Need a second pair of eyes? Help me repair the scaled architecture table in ARCHITECTURE.md.

Separate durable events from latest state

Durable history must survive consumer restarts. Current screens need a fast snapshot.

Redis Pub/Sub carries transient notifications. Kafka remains the source that consumers can replay.

• Switch back to the ARCHITECTURE.md editor tab.
• Add the storage responsibilities at the end of the file by pasting the following content:

## Why Kafka and Redis have different jobs

Kafka is the durable event backbone. A symbol is the event key, so events for that symbol land in one partition and retain order within that partition. Multiple partitions provide parallelism, and consumer groups let cache projectors, analytics, and alerting scale independently. Retention and replay make cache reconstruction possible.

Redis is the low-latency latest-state layer. A gateway asks Redis for the current record when a client connects or resynchronizes. Redis Pub/Sub can notify active gateways about updates, but it is at-most-once and not a durable source of truth. If a gateway disconnects, it must reconcile from the cache rather than wait for a missed Pub/Sub message.

Why Do Kafka and Redis Have Different Jobs?

Kafka retains events so consumers can restart or rebuild derived state. Symbol keys preserve order within each symbol's partition.

Redis keeps the newest record close to the gateways. A gateway reconciles from that cache after missing a transient Pub/Sub notification.

• Save ARCHITECTURE.md.
• Return to the Markdown preview.

You should see separate explanations for durable event replay and low-latency snapshot reads. The Redis section should identify Pub/Sub as transient delivery.

Storage Responsibilities Look Blurred?

• Confirm that the Kafka paragraph includes retention and replay.
• Confirm that the Redis paragraph sends reconnecting gateways back to the latest-state cache.

Want help checking the boundary? Help me distinguish Kafka replay from Redis latest-state serving in ARCHITECTURE.md.

Client-facing snapshots follow a different delivery contract. Backpressure controls what happens when a browser cannot keep pace.

• Switch back to the ARCHITECTURE.md editor tab.
• Add the slow-consumer policy at the end of the file by pasting the following content:

## Slow consumers and backpressure

The authoritative pipeline should retain normalized events according to its durability policy. A client-facing latest-price screen has a different contract: it needs the newest state, not every intermediate quote. A gateway can therefore keep one pending snapshot per slow client and replace older pending snapshots. This bounds memory while the per-symbol version reveals that intermediate updates were skipped.

Trades intended for audit, chart reconstruction, or order decisions must remain in the durable event path. Coalescing is a presentation-layer optimization, not permission to discard the authoritative stream.

Where Is Coalescing Safe?

A slow screen needs the newest state, so the gateway can replace an older pending snapshot. One pending snapshot bounds memory for that client.

Audit records need every authoritative event. Those records remain in the durable path even when a browser skips intermediate presentation updates.

• Save ARCHITECTURE.md.
• Return to the Markdown preview.

You should see a slow-consumer policy that preserves the newest screen state. You should also see an explicit boundary protecting authoritative events from coalescing.

Coalescing Policy Missing Its Boundary?

• Check that the first paragraph limits each slow client to one pending snapshot.
• Check that the second paragraph keeps audit and history events in the durable path.

Need help reviewing the policy? Help me verify that my slow-consumer design bounds memory without discarding authoritative events.

Trace recovery across failure boundaries

A scaled diagram becomes useful when every stateful layer has a recovery path. Each failure walkthrough identifies what restarts and where state comes from.

• Switch back to the ARCHITECTURE.md editor tab.
• Add the recovery walkthroughs at the end of the file by pasting the following content:

## Failure walkthroughs

1. Provider disconnect: the ingestion owner reconnects, authenticates, resubscribes, and marks data stale until new events arrive.
2. Ingestion worker crash: another worker acquires ownership, restores subscriptions, and resumes publishing. Consumers handle duplicate or resumed events according to source identity and timestamp policy.
3. Kafka consumer crash: the consumer group reassigns partitions and processing resumes from committed offsets.
4. Redis loss: latest-state projectors replay retained Kafka events or a compacted latest-state topic to rebuild the cache.
5. Gateway loss: clients reconnect to another regional gateway, receive Redis snapshots, then resume incremental updates.
6. Redis Pub/Sub gap: the gateway detects a version gap or reconnect and fetches the latest Redis record instead of expecting Pub/Sub replay.
7. Provider silence: transport may remain connected while a symbol receives no data, so freshness is tracked per symbol and surfaced as stale.

How Does Each Failure Recover?

• Provider sessions recover through reconnecting and resubscribing.
• Kafka consumers recover from committed offsets after partition reassignment.
• Redis state recovers through replay from the durable event backbone.
• Clients recover through a replacement gateway and a fresh snapshot.

• Save ARCHITECTURE.md.
• Return to the Markdown preview.

You should see seven numbered failure walkthroughs. Each walkthrough should identify a replacement owner or a source for rebuilding state.

Failure List Not Showing Seven Items?

• Confirm that every walkthrough starts on its own line.
• Check that the list runs continuously from one through seven.

Still missing a recovery path? Help me check the seven failure walkthroughs in ARCHITECTURE.md.

Symbol-based ordering creates a final capacity risk. One hot symbol can concentrate traffic on a single path.

• Switch back to the ARCHITECTURE.md editor tab.
• Add the traffic-skew warning at the end of the file by pasting the following content:

## Hot-symbol warning

A symbol-keyed partition preserves order but can create skew. One very active symbol can overload its Kafka partition, cache update path, Pub/Sub channel, or gateway subscription set. The mitigation must state whether strict total order for that symbol is preserved. Dedicated partitions and consumers preserve order more naturally. Subpartitioning raises throughput but requires a merge or sequencing stage if strict order still matters.

Why Does Traffic Skew Matter?

Symbol keys preserve per-symbol order by concentrating that symbol's events. Extreme activity can therefore overload its partition and downstream delivery path.

Any future mitigation must state its ordering contract. Extra parallelism can require a sequencing stage before clients receive updates.

• Save ARCHITECTURE.md.
• Return to the Markdown preview.

You should see the hot-symbol warning after the failure walkthroughs. It should name the Kafka partition, cache path, Pub/Sub channel, and gateway subscriptions as possible pressure points.

Hot-Symbol Warning Missing a Pressure Point?

• Confirm that the warning appears after the seventh failure walkthrough.
• Check that the warning covers ordering and throughput.

Need help checking the trade-off? Help me review the hot-symbol warning in ARCHITECTURE.md.

✔️ Awesome, I've got everything!

Your complete architecture document is saved. Keep the Markdown preview open for the final trace.

ⓧ I'd like to double check the full code

# Market Data Architecture

This is an educational Robinhood-style design. It is not a claim about Robinhood's private implementation.

## Local system

```text
Alpaca IEX or test WebSocket
  -> one Node.js ingestion connection
  -> provider-message normalizer
  -> in-memory Map keyed by symbol
  -> Server-Sent Events gateway
  -> normal and slow browser clients
```

The local cache stores two different facts per symbol:

- Latest trade: a completed transaction with price, size, exchange, conditions, and market timestamp.
- Latest quote: the current bid and ask on the selected feed, including round-lot sizes and exchanges.

The displayed midpoint is derived from bid and ask. It is not a trade. Freshness is measured from local receipt time, while the provider timestamp remains available for market-time reasoning.

## Scaled system

```text
Exchanges and market-data providers
  -> redundant regional ingestion workers
  -> normalization and validation
  -> Kafka market-events topic, keyed by symbol
  -> independent consumer groups
       -> latest-price projectors -> Redis price cache
       -> audit and history storage
       -> alerting and analytics
  -> sharded live-update distribution
  -> regional streaming gateway fleets
  -> millions of browser and mobile clients
```

## Component responsibilities

| Layer | Responsibility | Failure and recovery |
| --- | --- | --- |
| Ingestion workers | Maintain provider sessions, decode wire messages, normalize schemas, attach source metadata, and resubscribe after reconnect | Run redundant workers with lease or ownership rules so replacement does not create uncontrolled duplicate sessions |
| Kafka | Keep a durable, replayable event log and decouple ingestion from cache, analytics, and gateway consumers | Replication protects partitions; consumers resume from offsets or replay to rebuild derived state |
| Symbol partitioning | Route the same symbol to the same partition so its events retain partition order | Reassign failed consumers; monitor skew because a hot symbol can dominate one partition |
| Redis price cache | Serve the newest normalized trade and quote state for fast snapshots and reconnect bootstrap | Rebuild keys from Kafka replay or a compacted latest-state topic; treat cache loss as recoverable |
| Live-update distribution | Move fresh changes from cache projectors toward gateways | Redis Pub/Sub is suitable only for transient fan-out because disconnected subscribers miss messages; Kafka or Redis Streams is needed when replay matters |
| Regional gateways | Hold client connections, authorize subscriptions, fetch an initial Redis snapshot, and push incremental updates | Use multiple instances behind regional traffic routing; a reconnecting client lands on another instance and receives a new snapshot |
| Client | Render trade and quote state, track versions and age, and label stale values | Reconnect, request a fresh snapshot, and never assume transport health proves symbol freshness |

## Why Kafka and Redis have different jobs

Kafka is the durable event backbone. A symbol is the event key, so events for that symbol land in one partition and retain order within that partition. Multiple partitions provide parallelism, and consumer groups let cache projectors, analytics, and alerting scale independently. Retention and replay make cache reconstruction possible.

Redis is the low-latency latest-state layer. A gateway asks Redis for the current record when a client connects or resynchronizes. Redis Pub/Sub can notify active gateways about updates, but it is at-most-once and not a durable source of truth. If a gateway disconnects, it must reconcile from the cache rather than wait for a missed Pub/Sub message.

## Slow consumers and backpressure

The authoritative pipeline should retain normalized events according to its durability policy. A client-facing latest-price screen has a different contract: it needs the newest state, not every intermediate quote. A gateway can therefore keep one pending snapshot per slow client and replace older pending snapshots. This bounds memory while the per-symbol version reveals that intermediate updates were skipped.

Trades intended for audit, chart reconstruction, or order decisions must remain in the durable event path. Coalescing is a presentation-layer optimization, not permission to discard the authoritative stream.

## Failure walkthroughs

1. Provider disconnect: the ingestion owner reconnects, authenticates, resubscribes, and marks data stale until new events arrive.
2. Ingestion worker crash: another worker acquires ownership, restores subscriptions, and resumes publishing. Consumers handle duplicate or resumed events according to source identity and timestamp policy.
3. Kafka consumer crash: the consumer group reassigns partitions and processing resumes from committed offsets.
4. Redis loss: latest-state projectors replay retained Kafka events or a compacted latest-state topic to rebuild the cache.
5. Gateway loss: clients reconnect to another regional gateway, receive Redis snapshots, then resume incremental updates.
6. Redis Pub/Sub gap: the gateway detects a version gap or reconnect and fetches the latest Redis record instead of expecting Pub/Sub replay.
7. Provider silence: transport may remain connected while a symbol receives no data, so freshness is tracked per symbol and surfaced as stale.

## Hot-symbol warning

A symbol-keyed partition preserves order but can create skew. One very active symbol can overload its Kafka partition, cache update path, Pub/Sub channel, or gateway subscription set. The mitigation must state whether strict total order for that symbol is preserved. Dedicated partitions and consumers preserve order more naturally. Subpartitioning raises throughput but requires a merge or sequencing stage if strict order still matters.

How to Use This Reference

Compare each heading and paragraph with your saved file. The complete document connects local behavior to scaled responsibilities and recovery boundaries.

Before you trace the design, which component do you expect to restore current state after Redis loses its data?

• Trace one AAPL quote from its provider to the market-events topic.
• Continue the quote trace from the topic to a Redis latest-price record.
• Finish the quote trace from Redis to a browser client.
• Trace a gateway crash using the fifth failure walkthrough.
• Trace a Redis cache miss back to Kafka replay using the fourth failure walkthrough.
• Trace a Kafka consumer restart using the third failure walkthrough.
• Trace provider silence using the seventh failure walkthrough.

You should see Kafka remain the replayable event backbone when derived state disappears. Redis returns fast current snapshots after projectors rebuild the cache.

Replacement gateways give reconnecting clients a fresh snapshot. Per-symbol age keeps silent feeds visibly stale.

You've now turned a working local gateway into a scale design with explicit state ownership and recovery boundaries.

Secret mission

Tame a Hot Symbol

One extremely active symbol can overwhelm the ordered path that normally keeps market data predictable. Extend your architecture with dedicated Kafka capacity, a symbol-specific gateway tier, measurable activation signals, and a complete recovery plan.

Clean Up Your Resources

Clean Up Your Resources

Your gateway runs locally using the Alpaca Market Data API's free IEX coverage, so keeping the project files creates no ongoing cost. Decide whether to keep the gateway available, pause its process, or delete the local project files.

Resources you used:

• The local Node.js gateway process started from index.js.
• The local credential copy in .env.
• The credential template in .env.example.
• The runtime configuration in .development.env.
• The source-control exclusion in .gitignore.
• The gateway source in index.js.
• The browser dashboard in public/index.html.
• The scaled system design in ARCHITECTURE.md.

Keep everything running

No action needed. Choose this if you want to keep testing the gateway or refining the scaled architecture.

• Your gateway can keep serving the normal dashboard and slow-consumer mode while its terminal process remains active.
• Your .env file stays on your Mac with the Alpaca key pair.
• Your .gitignore file keeps .env excluded from source control.
• Your ARCHITECTURE.md file retains the scaled design and hot-symbol decision.

Pause - I'll come back to this later

Stop the running gateway to close the local server and upstream connection. Your source files and architecture document remain ready for your return.

• Switch back to the VS Code terminal that is running the gateway.
• Press Ctrl+C to stop the process.
• Keep .env private while the project is paused.

The terminal prompt returns. That's the local gateway safely paused with its files intact.

Delete - I don't want to use this again

Remove the local gateway files when you no longer need the demo or architecture design. This also removes the local copy of your Alpaca credentials.

Deletion is permanent

The cleanup command permanently removes these files without moving them to Trash. Your Alpaca account remains unchanged.

Your dashboard API keys remain active.

• Copy ARCHITECTURE.md elsewhere before continuing if you want to preserve your scaled design.

• Return to the VS Code terminal that ran index.js.
• Press Ctrl+C if the gateway is still running.
• Remove the listed local files by running this command in that same terminal:

rm -rf .env .env.example .development.env .gitignore ARCHITECTURE.md index.js public

What does this command remove?

• .env removes the local credential copy.
• .env.example removes the credential template.
• .development.env removes the gateway mode configuration.
• .gitignore removes the source-control exclusion.
• index.js removes the gateway source.
• public removes the dashboard file and its folder.
• ARCHITECTURE.md removes the scaled architecture design.

• Check the VS Code file sidebar.
• Confirm that none of the resources from the checklist remain.

That clears the local market-data gateway from your Mac. Your Alpaca account remains untouched.

Still see project files?

A remaining file usually means the cleanup command ran from a different folder.

• Return to the VS Code terminal that ran index.js.
• Run the cleanup command again from that terminal.
• Help me remove the remaining local market-data project files.

Nice Work!

Nice Work!

You made it! You built a market-data gateway inspired by Robinhood using the Alpaca Market Data API.

You've learned how to:

• Built a single-upstream gateway in Node.js that normalizes provider messages into separate trade records plus quote records. Kept the midpoint as a derived display value. Cached one latest state per symbol for immediate client snapshots.
• Delivered resilient browser fan-out through Server-Sent Events. Exposed stale values through the dashboard. Proved reconnect plus resubscribe with a forced upstream disconnect. Bounded each slow client's memory to one newest pending snapshot.
• Mapped the local system to a scale architecture with symbol-keyed Kafka partitions plus Redis latest-state records. Placed regional streaming gateways at the client fan-out boundary. Defined recovery paths across provider, ingestion, cache, gateway, plus client failures.
• Secret Mission: Designed a dedicated hot-symbol Kafka partition with its own consumer pool. Added a symbol-specific regional gateway tier to preserve strict per-symbol order. Defined measurable capacity triggers plus bounded slow-client memory. Specified replacement-path failover with Kafka replay, Redis restoration, plus fresh client snapshot resynchronization.

Ready to quiz yourself?