Build a Real-Time Clickstream Pipeline

Stream clickstream events through Kinesis, Lambda, and Firehose to S3.

Introduction

30 Second Summary

Every click, scroll, and button tap on a website generates a data point. Processing millions of these events per second in real time is what powers live dashboards, personalized recommendations, and fraud detection.

In this project, you will build a real-time clickstream pipeline using Amazon Kinesis Data Streams. You'll push simulated click events into a stream and watch two independent consumers process the exact same data simultaneously.

What You'll Build

You'll send click events from AWS CloudShell, watch them appear in real-time Lambda logs within seconds, then open S3 and find the same events delivered as files by Amazon Data Firehose with zero delivery code.

By the end of this project, you'll have:

  • A live streaming pipeline where you send click events and see them decoded and logged in CloudWatch within seconds.
  • An automatic S3 delivery path where the same events arrive as files in your bucket, without you writing any delivery code.
  • First-hand proof that stream replay works. You'll re-read old events that were already processed, something impossible with a traditional queue.
  • Secret Mission: Split your stream into 2 shards and observe how partition keys distribute data across them.

Are there any prerequisites?

You need an AWS account with console access. This project assumes you have completed the SQS and SNS fan-out project first, so you are comfortable navigating the AWS Console and creating resources like Lambda functions and S3 buckets.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters. This project bridges your SQS knowledge with real-time streaming, so nailing the "why Kinesis" question now will anchor everything that follows.

Create Your Kinesis Data Stream

Before you can stream any data, you need a place for that data to land. In this step, you create a Kinesis Data Stream that acts as the durable buffer between your producers and consumers.

You will use Provisioned mode with 1 shard so you can physically see the capacity unit the SAA-C03 exam tests. Each shard gives you exactly 1 MB/s for writes and 2 MB/s for reads.

In this step, get ready to:
  • Create a Kinesis Data Stream named clickstream-demo in Provisioned mode with 1 shard.
  • Explore the Configuration tab to understand retention and capacity settings.
  • Confirm the stream is Active and ready to receive data.
Navigate to Kinesis and create your data stream
  • In the AWS Management Console, use the search bar at the top to search for Kinesis and select it.
  • In the left navigation pane, click Data streams.
  • Click Create data stream.

The Create data stream page opens with fields for naming and configuring your stream.

  • In the Data stream name field, enter clickstream-demo.
  • Under Data stream capacity, select Provisioned.
  • Enter 1 for the number of shards.
  • Click Create data stream at the bottom of the page.

The stream status shows Creating while AWS provisions your shard. Wait until the status changes to Active (this takes about 10-15 seconds).

What is a shard?

A shard is the base throughput unit of a Kinesis data stream. Each shard supports 1 MB/s and 1,000 records/second for writes, and 2 MB/s for reads. When you need more throughput, you add more shards.

By choosing Provisioned mode with 1 shard, you explicitly control capacity. This is the mode the SAA-C03 exam heavily tests because it makes shard math visible.

Why Provisioned mode instead of On-Demand?

On-Demand mode hides shards from you. AWS manages capacity automatically, and you never see a shard count. That is operationally simpler, but it makes it impossible to learn how shards work.

Provisioned mode lets you physically see and control shards. You will add shards in the Secret Mission and observe how partition keys route to them.

Stream stuck on Creating?

If the stream stays in Creating status for more than 30 seconds, refresh the page. If it moves to a failed state, check that your account has not hit its shard limit (default is 500 shards per account in most regions). Help me troubleshoot why my Kinesis stream won't create.

Explore the Configuration tab

Now that your stream is Active, you can inspect its settings to understand the exam-relevant concepts.

  • Click the stream name clickstream-demo to open its details page.
  • Click the Configuration tab.
  • Find the Data retention period section. Confirm it shows 24 hours (the default).
  • Note the Edit capacity mode option. This is how you could switch to On-Demand mode later (up to twice per 24 hours per stream).

What does the retention period mean?

The retention period is how long Kinesis keeps your data in the stream after it arrives. The default is 24 hours, and you can extend it up to 8,760 hours (365 days). During this window, any consumer can read and re-read the data. This is what makes replay possible.

In SQS, a message disappears once a consumer processes and deletes it. In Kinesis, data stays for the entire retention period regardless of how many times it has been read.

Check the Monitoring tab
  • Click the Monitoring tab on the stream details page.
  • Confirm you see empty graphs for metrics like Incoming data (bytes) and Get records (bytes). All values should be zero because no data has flowed through the stream yet.

The flat-zero baseline confirms your stream is provisioned but idle. You will see these graphs change once you start sending records in the next step.

Before you move on, let's confirm the end state of this step.

  • Return to the Data streams list in the left navigation.
  • What do you expect to see in the Status column for clickstream-demo?

You should see Active with 1 open shard listed. This confirms your stream is ready to ingest data.

Your stream is live and waiting for data. Next up, you will open CloudShell and push clickstream events into the stream to see Kinesis in action.

Produce Clickstream Events

Your clickstream-demo stream is active and ready to receive data.

A stream with no data flowing through it is just an empty pipe. In this step, you will push clickstream events into the stream using partition keys, then read them back to see how Kinesis Data Streams preserves ordering and keeps data immutable for replay.

In this step, get ready to:
  • Send 5 clickstream records into your Kinesis stream using partition keys.
  • Retrieve those records and confirm they arrived in order.
  • Prove replay works by re-reading the same data a second time.
Send clickstream records to your stream

Every record you put into Kinesis needs two things: a data payload (the event itself) and a partition key (which determines which shard the record lands on). Records with the same partition key always go to the same shard, which guarantees ordering for that key.

  • Open AWS CloudShell by clicking the CloudShell icon in the bottom-left of the AWS Console navigation bar (it looks like a terminal prompt).
  • Send your first clickstream record by running this command:
aws kinesis put-record \
    --stream-name clickstream-demo \
    --partition-key user-123 \
    --data '{"userId":"user-123","page":"/home","action":"click","timestamp":"2026-08-04T10:00:01Z"}'

What does this command do?

The --stream-name flag targets your clickstream-demo stream.

The --partition-key determines which shard receives this record. Kinesis hashes the key using MD5 and maps the result to a shard. All records with user-123 as their partition key will always land on the same shard, preserving their order.

The --data flag carries your JSON clickstream event payload.

You should see a JSON response containing a ShardId and a SequenceNumber. The sequence number is a unique identifier Kinesis assigns to the record within the shard.

  • Send 4 more records by running these commands one at a time:
aws kinesis put-record \
    --stream-name clickstream-demo \
    --partition-key user-123 \
    --data '{"userId":"user-123","page":"/products","action":"click","timestamp":"2026-08-04T10:00:02Z"}'
aws kinesis put-record \
    --stream-name clickstream-demo \
    --partition-key user-456 \
    --data '{"userId":"user-456","page":"/home","action":"click","timestamp":"2026-08-04T10:00:03Z"}'
aws kinesis put-record \
    --stream-name clickstream-demo \
    --partition-key user-123 \
    --data '{"userId":"user-123","page":"/cart","action":"click","timestamp":"2026-08-04T10:00:04Z"}'
aws kinesis put-record \
    --stream-name clickstream-demo \
    --partition-key user-456 \
    --data '{"userId":"user-456","page":"/products","action":"click","timestamp":"2026-08-04T10:00:05Z"}'

Each command returns its own SequenceNumber. Notice that all 5 records go to shardId-000000000000 because your stream only has 1 shard.

Why two different partition keys?

You are simulating two website users browsing at the same time. With a single shard, both keys land in the same place. In the Secret Mission, you will split into 2 shards and see these users route to different shards based on their partition key hash.

Seeing an error about the stream not being found?

Make sure your CloudShell session is in the same region where you created clickstream-demo. Check the region selector in the top-right corner of the AWS Console.

help me fix a stream not found error in CloudShell

Retrieve your records from the stream

To read data back from a Kinesis shard, you need a shard iterator. A shard iterator is a pointer that tells Kinesis where in the shard to start reading. Using TRIM_HORIZON means "start from the oldest available record."

  • Get a shard iterator and then fetch your records by running these commands:
SHARD_ITERATOR=$(aws kinesis get-shard-iterator \
    --shard-id shardId-000000000000 \
    --shard-iterator-type TRIM_HORIZON \
    --stream-name clickstream-demo \
    --query 'ShardIterator')
aws kinesis get-records --shard-iterator $SHARD_ITERATOR

What do these commands do?

The first command requests a shard iterator starting at TRIM_HORIZON (the oldest record in the shard). The --query 'ShardIterator' flag extracts just the iterator string and stores it in a shell variable.

The second command uses that iterator to fetch the actual data records from the shard.

Before you press Enter on the get-records command, predict: will you see your records in plain text, or will the data field look encoded?

You will see your 5 records returned in the Records array. The Data field is base64-encoded. Each record also shows its PartitionKey and SequenceNumber. Records for user-123 appear in the exact order you sent them: /home, then /products, then /cart.

  • Decode one of the base64 data values to confirm it contains your JSON by running:
echo 'eyJ1c2VySWQiOiJ1c2VyLTEyMyIsInBhZ2UiOiIvaG9tZSIsImFjdGlvbiI6ImNsaWNrIiwidGltZXN0YW1wIjoiMjAyNi0wOC0wNFQxMDowMDowMVoifQ==' | base64 --decode

Grab a Data value from your output

The base64 string above is an example. Copy an actual Data value from your get-records output and paste it after echo to decode your own records.

You should see the original JSON payload you sent, confirming the data arrived intact.

Getting an empty Records array?

Shard iterators expire after 300 seconds. If you waited too long between getting the iterator and calling get-records, re-run both commands together.

help me troubleshoot empty records from Kinesis get-records

Prove replay by reading the same data again

Here is the key difference between a stream and a queue. In SQS, once a message is processed and deleted, it is gone forever. In Kinesis, data is immutable and stays in the shard for the entire retention period (24 hours by default).

  • Before you re-run, predict: will Kinesis return the same 5 records, or will it return nothing because you already read them?
  • Get a fresh shard iterator and read the records again by running:
SHARD_ITERATOR=$(aws kinesis get-shard-iterator \
    --shard-id shardId-000000000000 \
    --shard-iterator-type TRIM_HORIZON \
    --stream-name clickstream-demo \
    --query 'ShardIterator')

aws kinesis get-records --shard-iterator $SHARD_ITERATOR

You get the same 5 records back. The data has not been deleted or consumed. This is replay. You can re-read the same data as many times as you want within the retention window.

Why does replay matter?

Replay means you can reprocess data if something goes wrong, add new consumers that read historical data, or debug production issues by re-reading the exact events that caused a problem.

On the SAA-C03 exam, if a question mentions "replay", "reprocess old data", or "multiple consumers reading the same data", the answer is Kinesis Data Streams, not SQS.

You now have live data flowing through your stream, and you have proven that Kinesis keeps records immutable and replayable. Next up, you will attach a Lambda consumer that automatically processes these records in real time.

Add a Lambda Consumer

In the previous step, you manually fetched records from your stream using get-records. You proved that Kinesis keeps data immutable and replayable.

Manually polling a stream is not how production systems work. In this step, you'll connect a Lambda function as a consumer that automatically polls your stream and processes records in real time. This is the pattern the SAA-C03 exam tests under "serverless stream processing."

In this step, get ready to:
  • Create a Lambda function that decodes Kinesis records from base64 and logs them.
  • Add a Kinesis trigger so Lambda automatically polls your stream.
  • Verify that Lambda replays all records from the beginning of the stream.
Create the Lambda function

You need a function that can receive batches of Kinesis records, decode each one from base64, and log the result. The Lambda console lets you create the function and edit the code inline.

  • In the AWS Console search bar, type Lambda and select the Lambda service.
  • Click Create function.
  • Select Author from scratch.
  • Enter clickstream-processor in the Function name field.
  • Set Runtime to Python 3.12.
  • Leave the permissions as default (Lambda will create a new execution role) and click Create function.

You should see a success banner confirming the function was created. Now you need to grant this function permission to read from your Kinesis stream.

  • Click the Configuration tab.
  • Select Permissions in the left sidebar.
  • In the Execution role pane, click the link under Role name. This opens the IAM console in a new tab.
  • In the Permissions policies pane, click Add permissions, then select Attach policies.
  • Search for AWSLambdaKinesisExecutionRole.
  • Select the checkbox next to it and click Add permissions.

What does this policy grant?

The AWSLambdaKinesisExecutionRole managed policy gives Lambda the permissions it needs to poll your Kinesis stream. This includes kinesis:GetRecords, kinesis:GetShardIterator, kinesis:DescribeStream, and CloudWatch Logs permissions for writing logs.

You should see the policy now listed under your role's permissions. Close the IAM tab and return to the Lambda function tab.

  • Click the Code tab to open the inline editor.
  • Select all the default code in lambda_function.py and delete it.
  • Paste the following function code:
import base64
import json

def lambda_handler(event, context):
    print(f"Received {len(event['Records'])} records")

    for record in event['Records']:
        # Kinesis data is base64 encoded
        payload = base64.b64decode(record['kinesis']['data']).decode('utf-8')
        partition_key = record['kinesis']['partitionKey']
        sequence_number = record['kinesis']['sequenceNumber']

        print(f"Partition Key: {partition_key}")
        print(f"Sequence Number: {sequence_number}")
        print(f"Data: {payload}")

        # Parse the JSON payload
        try:
            click_event = json.loads(payload)
            print(f"User: {click_event.get('userId')}, Page: {click_event.get('page')}, Action: {click_event.get('action')}")
        except json.JSONDecodeError:
            print(f"Could not parse payload as JSON: {payload}")

    return {
        'statusCode': 200,
        'body': f"Processed {len(event['Records'])} records"
    }

What does this code do?

  • Kinesis delivers records with the data payload encoded in base64. The function decodes each record back to a UTF-8 string using base64.b64decode().
  • It extracts the partitionKey and sequenceNumber from each record's metadata and logs them. These are the two fields you'll use to prove ordering and replay.
  • It parses the decoded string as JSON and logs the individual fields from your clickstream events.
  • Click Deploy to save the function code.

You should see a "Successfully updated the function" banner at the top of the page.

Don't see the Deploy button?

Make sure you're on the Code tab (not Configuration or Test). The Deploy button appears in the code editor section.

If you see a syntax error after deploying, double check that you copied the entire code block including the import statements at the top. Help me fix a syntax error in my Lambda function code.

✔️ Awesome, I've got everything!

Great. Double check you saved by clicking Deploy and seeing the success banner.

ⓧ I'd like to double check the full code

Your lambda_function.py should contain exactly this code:

import base64
import json

def lambda_handler(event, context):
    print(f"Received {len(event['Records'])} records")

    for record in event['Records']:
        # Kinesis data is base64 encoded
        payload = base64.b64decode(record['kinesis']['data']).decode('utf-8')
        partition_key = record['kinesis']['partitionKey']
        sequence_number = record['kinesis']['sequenceNumber']

        print(f"Partition Key: {partition_key}")
        print(f"Sequence Number: {sequence_number}")
        print(f"Data: {payload}")

        # Parse the JSON payload
        try:
            click_event = json.loads(payload)
            print(f"User: {click_event.get('userId')}, Page: {click_event.get('page')}, Action: {click_event.get('action')}")
        except json.JSONDecodeError:
            print(f"Could not parse payload as JSON: {payload}")

    return {
        'statusCode': 200,
        'body': f"Processed {len(event['Records'])} records"
    }
Add a Kinesis trigger

Your function exists but it does not know to read from your stream yet. You need to create an event source mapping that tells Lambda which stream to poll and where to start reading.

  • On your clickstream-processor function page, scroll up to the Function overview pane.
  • Click Add trigger.
  • In the Trigger configuration dropdown, select Kinesis.
  • In the Kinesis stream field, select your clickstream-demo stream.
  • Set Starting position to TRIM_HORIZON.
  • Confirm Batch size is set to 100 (this is the default).
  • Click Add.

Why TRIM_HORIZON?

TRIM_HORIZON tells Lambda to start reading from the oldest record available in the stream. This means Lambda will process ALL existing records, not just new ones. You sent 5 records in the previous step, and Lambda will replay them all.

The alternative, LATEST, would skip old records and only process records that arrive after the trigger is created. For this project, TRIM_HORIZON demonstrates replay.

After clicking Add, you should see the trigger appear in the Function overview diagram with a status of Enabled. Lambda is now automatically polling your stream once per second.

Trigger shows an error?

If you see an error about permissions, go back to the Configuration > Permissions tab and confirm that AWSLambdaKinesisExecutionRole is listed under your execution role's policies. The trigger cannot read from the stream without this policy.

If the dropdown didn't show your stream, your role may also need kinesis:ListStreams permission. You can type the stream ARN directly instead of selecting from the list. Help me fix my Lambda Kinesis trigger permissions.

Send new records and verify replay in CloudWatch

Your Lambda function is now polling the stream. Before you check CloudWatch Logs, send a few more records so you can confirm Lambda processes both old and new data.

  • Switch back to AWS CloudShell.
  • Send 3 new clickstream records by running these commands:
aws kinesis put-record \
  --stream-name clickstream-demo \
  --partition-key user-123 \
  --data '{"userId":"user-123","page":"/products","action":"click","timestamp":"2026-08-04T10:05:01Z"}' \
  --cli-binary-format raw-in-base64-out

aws kinesis put-record \
  --stream-name clickstream-demo \
  --partition-key user-456 \
  --data '{"userId":"user-456","page":"/checkout","action":"purchase","timestamp":"2026-08-04T10:05:02Z"}' \
  --cli-binary-format raw-in-base64-out

aws kinesis put-record \
  --stream-name clickstream-demo \
  --partition-key user-123 \
  --data '{"userId":"user-123","page":"/confirmation","action":"view","timestamp":"2026-08-04T10:05:03Z"}' \
  --cli-binary-format raw-in-base64-out

Each command should return a response with a ShardId and SequenceNumber, confirming the record was written.

What does --cli-binary-format do?

AWS CloudShell uses AWS CLI version 2, which defaults to expecting base64-encoded data. The --cli-binary-format raw-in-base64-out flag tells the CLI to accept your raw JSON string as-is and handle the base64 encoding for you.

Now think about this: you set the trigger's Starting position to TRIM_HORIZON. Do you think Lambda will only process these 3 new records, or will it process all 8 records (5 old + 3 new)?

  • Wait 10-15 seconds for Lambda to poll and process the records.
  • In the AWS Console search bar, type CloudWatch and open the CloudWatch service.
  • In the left sidebar, click Logs, then Log groups.
  • Click on the log group named /aws/lambda/clickstream-processor.
  • Click on the most recent log stream (the one at the top).

You should see ALL 8 records logged, including the 5 you sent in the previous step. Lambda replayed the entire stream from the beginning because you chose TRIM_HORIZON.

Look at the log entries. You should see partition keys (user-123, user-456), sequence numbers, and decoded JSON payloads for every record.

Why does this matter for the exam?

You just proved that Kinesis supports replay. Lambda re-read records that were already in the stream before the trigger existed. In SQS, once a message is processed and deleted, it is gone forever. In Kinesis, data is immutable for the entire retention period (24 hours by default, up to 365 days).

When an exam question says "reprocess old events" or "replay data from the start of the stream," the answer is Kinesis Data Streams.

Don't see the log group?

Lambda can take up to 30 seconds to poll and process its first batch. Wait a moment and refresh the Log groups page. If the log group still doesn't appear after a minute, check that your trigger status shows Enabled (not Creating or Disabling).

If you see records missing, remember that get-records and Lambda may batch records differently. Lambda processes in batches of up to 100 records and may split them across multiple invocations. Check all log streams in the log group. Help me find my Lambda CloudWatch logs for the Kinesis trigger.

Your first consumer is live. Lambda is automatically polling the stream, decoding records, and logging them in real time. Next up, you'll add a second consumer that delivers the same data to S3 for long-term storage, with zero code.

Add Firehose for S3 Delivery

Your Lambda function processes clickstream events in real time and logs them to CloudWatch. However, CloudWatch Logs is not a long-term analytics store.

You need the raw data in S3 for analytics and archiving. Amazon Data Firehose does this with zero code. It reads from your stream, buffers records, and delivers them as files to S3 automatically.

In this step, get ready to:
  • Create an S3 bucket for long-term storage of clickstream data.
  • Create an Amazon Data Firehose stream connected to your Kinesis stream.
  • Verify that both Lambda and Firehose consume the same data independently.
Create an S3 bucket and open the Firehose console

Before you can deliver streaming data to S3, you need a destination bucket. You also need to know where to find the Firehose console, which now lives under its own service name.

  • In the AWS Console search bar at the top, type S3 and select S3.
  • Click Create bucket.
  • For Bucket name, enter clickstream-firehose- followed by your 12-digit AWS account ID (e.g. clickstream-firehose-123456789012). This ensures a globally unique name.
  • Leave all other settings at their defaults.
  • Scroll down and click Create bucket.

You should see a green success banner confirming your bucket was created.

What is Amazon Data Firehose?

Amazon Data Firehose (formerly called Amazon Kinesis Data Firehose) is a fully managed delivery service. It reads records from a source, buffers them, and delivers them to a destination like S3 with zero custom code.

The key exam fact is that a Firehose stream has ONE destination per stream. It is not pub/sub. It buffers data up to 5 MB or for 5 minutes by default, whichever comes first, then delivers it as a single S3 object.

  • Return to the AWS Console search bar, type Firehose, and select Amazon Data Firehose.
  • Click Create Firehose stream.
  • Under Source, select Amazon Kinesis Data Streams.
  • In the Source settings section, choose your clickstream-demo stream from the dropdown.
  • Under Destination, select Amazon S3.

Don't see your stream in the dropdown?

Make sure you are in the same AWS region where you created clickstream-demo. Check the region selector in the top-right corner of the console.

help me find my Kinesis stream in the Firehose source dropdown

Configure buffering and create the Firehose stream

Firehose buffers incoming data before delivering it to S3. The defaults are 5 MB or 5 minutes. For faster testing, you will reduce these to the minimum so results appear in about 60 seconds.

  • Under Destination settings, in the S3 bucket field, select the bucket you just created (clickstream-firehose- followed by your account ID).
  • For New line delimiter, choose Enabled. This places each record on its own line, making the files readable.
  • Scroll down to Buffer hints.
  • Set Buffer size to 1 MB.
  • Set Buffer interval to 60 seconds.
  • Leave all other settings at their defaults. Under Advanced settings, Firehose will create a new IAM role automatically with the required permissions.
  • Click Create Firehose stream.

Wait for the Firehose stream status to change to Active. This takes about 30 seconds.

Why reduce the buffer interval?

Firehose delivers data to S3 when either the buffer size (1 MB) or the buffer interval (60 seconds) is reached, whichever comes first. With small test payloads of a few hundred bytes, the 60-second timer will trigger delivery long before you hit 1 MB.

Seeing a permissions error?

If you get an error about IAM permissions when creating the stream, ensure you allowed Firehose to create a service role automatically. The auto-created role grants Firehose read access to your Kinesis stream and write access to your S3 bucket.

help me fix a permissions error when creating my Firehose stream

Send records and verify S3 delivery

Now you have two independent consumers reading from the same stream. Lambda processes records in real time. Firehose buffers and delivers to S3. Time to prove they both work simultaneously.

  • Switch back to AWS CloudShell.
  • Send 5 new clickstream records by running each of these commands:
aws kinesis put-record \
  --stream-name clickstream-demo \
  --partition-key user-123 \
  --data '{"userId":"user-123","page":"/products","action":"click","timestamp":"2026-08-04T11:00:01Z"}'
aws kinesis put-record \
  --stream-name clickstream-demo \
  --partition-key user-456 \
  --data '{"userId":"user-456","page":"/checkout","action":"click","timestamp":"2026-08-04T11:00:02Z"}'
aws kinesis put-record \
  --stream-name clickstream-demo \
  --partition-key user-123 \
  --data '{"userId":"user-123","page":"/cart","action":"view","timestamp":"2026-08-04T11:00:03Z"}'
aws kinesis put-record \
  --stream-name clickstream-demo \
  --partition-key user-456 \
  --data '{"userId":"user-456","page":"/home","action":"view","timestamp":"2026-08-04T11:00:04Z"}'
aws kinesis put-record \
  --stream-name clickstream-demo \
  --partition-key user-123 \
  --data '{"userId":"user-123","page":"/products/item-42","action":"click","timestamp":"2026-08-04T11:00:05Z"}'

What does each command do?

Each put-record call sends one JSON event to the clickstream-demo stream. The --partition-key determines which shard receives the record. The --data flag accepts raw text in CloudShell, and the CLI base64-encodes it automatically before sending to the API.

Each command returns a JSON response with a ShardId and SequenceNumber. This confirms the record was accepted into the stream.

  • Now wait 60 to 90 seconds. This gives the Firehose buffer interval time to trigger delivery.

Before you check S3, think about this: do you expect to find one file or multiple files in the bucket? What format will the records be in?

  • Navigate back to the S3 console.
  • Open your clickstream-firehose- bucket.
  • You should see a folder structure with a date-partitioned prefix (e.g. 2026/08/04/11/). Click through the folders until you find a file.
  • Click the file name, then click Download or use the Object actions menu to Query with S3 Select to preview its contents.

You should see your clickstream JSON records, one per line, delivered by Firehose.

Why is this powerful?

Both Lambda AND Firehose consumed the exact same records from the stream. Neither consumer "stole" data from the other. This is the fundamental difference between a stream and a queue.

In SQS, each message goes to exactly one consumer. In Kinesis, every consumer reads the full stream independently. Two consumers, same stream, zero data loss.

Don't see any files in S3?

  • Confirm at least 60 seconds have passed since you sent records. Firehose waits for the full buffer interval before delivering.
  • Check that your Firehose stream status is Active in the Firehose console.
  • Verify the Firehose stream's source is clickstream-demo and the destination bucket matches the one you created.

help me troubleshoot why Firehose is not delivering records to my S3 bucket

Your pipeline now has two independent consumers processing the same data. Lambda gives you real-time processing. Firehose gives you zero-code archival to S3. Next up, you will prove the fundamental differences between Kinesis and SQS by replaying data and testing ordering.

Prove the Stream vs Queue Difference

Your pipeline is fully operational. Lambda processes events in real time, and Firehose delivers the same events to S3. Both consumers read from the same stream independently.

The SAA-C03 exam repeatedly asks "when should you use Kinesis vs SQS?" This step makes you physically experience the three differences: replay (data stays after processing), multiple consumers (both Lambda and Firehose read the same data), and ordering (same partition key = same sequence). After this step, you will never forget the answer.

In this step, get ready to:
  • Prove replay by forcing Lambda to re-process all old records from the beginning.
  • Prove multiple consumers by confirming Lambda and Firehose received the same data independently.
  • Prove ordering by verifying sequence numbers arrive in order per partition key.
Replay test: re-read data that was already processed

In SQS, once a consumer processes and deletes a message, that message is gone forever. In Kinesis, data is immutable. It stays in the stream for the entire retention period regardless of how many times consumers have read it. You are about to prove this by forcing Lambda to re-read every record from the very beginning.

To replay from the start, you need to delete the existing Lambda trigger and add a fresh one with Starting position set to TRIM_HORIZON. This forces Lambda to begin reading from the oldest available record.

  • Return to the Lambda console and open your clickstream-processor function.
  • Scroll down to the Configuration tab.
  • Select Triggers in the left sidebar.
  • Select the Kinesis trigger (clickstream-demo) and click Delete.
  • Confirm the deletion.

The trigger is now removed. Lambda has stopped polling the stream.

  • In the Function overview pane at the top of the page, click Add trigger.
  • Select Kinesis as the source.
  • Select your clickstream-demo stream.
  • Set Starting position to TRIM_HORIZON.
  • Set Batch size to 100.
  • Click Add.

What does TRIM_HORIZON do here?

TRIM_HORIZON tells Lambda to start reading from the oldest available record in the stream. Since your records are still within the 24-hour retention period, Lambda will re-process every single record you sent earlier. This is replay in action.

Before you check CloudWatch, predict: how many records do you expect Lambda to process this time? Remember, TRIM_HORIZON reads from the very beginning of the stream.

  • Wait 15-20 seconds for the new trigger to start polling.
  • Open CloudWatch Logs in a new tab.
  • Open the log group for /aws/lambda/clickstream-processor.
  • Click on the most recent log stream.

You should see ALL your old records appearing in the new log stream. Lambda re-processed every record from the beginning. In SQS, once a message is processed and deleted, it is gone forever. In Kinesis, data is immutable for the entire retention period.

Not seeing old records in the new log stream?

The new trigger can take up to 60 seconds to begin polling. Wait a moment and refresh the CloudWatch Logs page. If you still see no logs, confirm the trigger shows Enabled in the Lambda Configuration > Triggers view.

Help me troubleshoot why my Lambda trigger isn't processing records after I re-added it with TRIM_HORIZON.

Multiple consumers test: same data, two readers, zero interference

With SQS, each message is delivered to only one consumer. Once one consumer processes it, no other consumer can see it. Kinesis works differently. A stream supports multiple independent consumers reading the exact same data simultaneously.

You already have two consumers: Lambda (real-time processing to CloudWatch Logs) and Firehose (batch delivery to S3). Time to prove neither one "stole" records from the other.

  • In the CloudWatch Logs stream you just opened, note the partition keys and data payloads Lambda logged. Look for records with partition keys user-123 and user-456.
  • Switch to the S3 console and open your clickstream-firehose- bucket.
  • Navigate into the date-partitioned folders until you find the delivered files.
  • Click on one of the files and choose Download, or use the Select from query option to preview the contents.

Compare what you see. The S3 files contain the same clickstream records that Lambda logged. Both consumers received the full dataset. Neither consumer took records away from the other.

Why is this impossible with SQS?

In SQS, a message becomes invisible to other consumers once one consumer picks it up. After successful processing, the consumer deletes the message permanently. If you need two systems to process the same message, you would need SNS fan-out to duplicate the message into two separate queues.

With Kinesis, multiple consumers simply read from the same stream independently. No duplication, no fan-out infrastructure needed. The data just sits there for anyone to read within the retention window.

Ordering test: same partition key, guaranteed sequence

The final exam differentiator is ordering. Kinesis guarantees that records with the same partition key are delivered to consumers in the exact order they were written. Records with different partition keys may interleave.

  • Return to your CloudWatch Logs stream for clickstream-processor.
  • Look for lines showing Partition Key: user-123 and note the sequence numbers printed on the line above each one.
  • Confirm that all records for user-123 have increasing sequence numbers.
  • Now look for records with Partition Key: user-456 and confirm those also have increasing sequence numbers within their own group.

Notice that records for user-123 and user-456 may interleave with each other. That is expected. Ordering is guaranteed per partition key within a shard, not across different partition keys.

Why does this matter for the exam?

The SAA-C03 exam tests this concept directly. When a question mentions "ordered processing" or "events for the same user must arrive in sequence," the answer is Kinesis with the user ID as the partition key. SQS standard queues offer best-effort ordering only. SQS FIFO queues guarantee ordering but with a throughput limit of 300 messages per second per group.

What about Enhanced Fan-Out?

Without Enhanced Fan-Out, all consumers share the shard's 2 MB/s read capacity. Enhanced Fan-Out gives each registered consumer a dedicated 2 MB/s per shard. You did not configure it in this project, but the exam tests the concept. If a question mentions "dedicated throughput per consumer" or "consumer falling behind," Enhanced Fan-Out is the answer.

Before we wrap up, predict: if you ran get-records in CloudShell right now, would the old records still be there even though Lambda just re-processed them?

  • Confirm your answer by switching back to CloudShell and running a shard iterator followed by get-records:
aws kinesis get-shard-iterator --stream-name clickstream-demo --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --query 'ShardIterator' --output text
  • Copy the shard iterator output and run:
aws kinesis get-records --shard-iterator YOUR_SHARD_ITERATOR_HERE --query 'Records[].{PartitionKey:PartitionKey,SequenceNumber:SequenceNumber}' --output table

The records are still there. Lambda processed them, Firehose delivered them, you manually read them, and they are still available for anyone else to read. This is the immutable, replayable nature of a stream. The data does not disappear until the retention period expires.

Getting an expired iterator error?

Shard iterators expire after 5 minutes. If you see an expired iterator error, simply re-run the get-shard-iterator command to get a fresh one, then immediately use it with get-records.

Help me fix an expired shard iterator error when running get-records on my Kinesis stream.

You just physically experienced every Kinesis vs SQS differentiator that appears on the SAA-C03 exam. Replay, multiple independent consumers, and per-partition-key ordering. Next up, take on the Secret Mission and split your stream into multiple shards to see how partition keys distribute across them.

Secret mission

Split Into 2 Shards and Observe Partition Key Distribution

You proved Kinesis can replay data and support multiple consumers. But so far, you have been working with a single shard. The SAA-C03 exam tests what happens when you scale out. In this mission, you will split your stream into 2 shards and physically see how partition keys route records to specific shards.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. Kinesis Data Streams is not included in the AWS Free Tier, and provisioned shards charge continuously at $0.015 per shard per hour even with no traffic.

Cost warning

Your stream currently has 2 provisioned shards. That costs approximately $0.72 per day ($0.03/hour) while the stream exists. Delete your resources promptly when you are done to avoid unnecessary charges.

Resources you used:

  • Kinesis Data Stream (clickstream-demo) — 2 provisioned shards at $0.015/shard/hour.
  • Amazon Data Firehose delivery stream delivering to S3.
  • S3 bucket (clickstream-firehose-[account-id]) containing delivered clickstream files.
  • Lambda function (clickstream-processor) with Kinesis trigger.
  • IAM execution role for the Lambda function with AWSLambdaKinesisExecutionRole policy attached.

Keep everything running

No action needed. Choose this if you are still actively building or plan to extend this pipeline immediately.

  • Your 2 provisioned shards will continue costing approximately $0.72 per day.
  • Firehose charges $0.029 per GB delivered, which is negligible for test data volumes.
  • Lambda and S3 storage costs are negligible at this scale.

Pause - I'll come back to this later

Delete the cost-incurring resources (Kinesis stream and Firehose) while keeping your Lambda function and S3 data intact for later.

  • Delete the Firehose delivery stream using the steps in the Delete tab below.
  • Delete the Kinesis Data Stream using the steps in the Delete tab below.
  • Keep your Lambda function and S3 bucket. Both are effectively free when idle.
  • When you return, recreate the Kinesis stream and Firehose stream, and re-add the Lambda trigger.

Delete - I don't want to use this again

Remove all project resources completely. Follow these steps in order from CloudShell and the AWS Console.

Step 1: Delete the Firehose delivery stream.

  • In the AWS Console, navigate to the Amazon Data Firehose service.
  • Select your delivery stream from the list.
  • Click Delete and confirm when prompted.
  • Wait for the stream to enter the Deleting state. This may take a few minutes.

Step 2: Remove the Lambda Kinesis trigger.

  • Open the Lambda console and select the clickstream-processor function.
  • Click the Configuration tab, then click Triggers.
  • Select the Kinesis trigger and click Delete. Confirm when prompted.

Step 3: Delete the Kinesis Data Stream.

  • In CloudShell, run this command:
aws kinesis delete-stream --stream-name clickstream-demo

Step 4: Empty and delete the S3 bucket.

  • Empty the bucket by running:
aws s3 rm s3://[[BUCKET_NAME="clickstream-firehose-your-account-id"]] --recursive
  • Delete the empty bucket by running:
aws s3 rb s3://[[BUCKET_NAME="clickstream-firehose-your-account-id"]]

Step 5: Delete the Lambda function.

  • Run this command in CloudShell:
aws lambda delete-function --function-name clickstream-processor

Step 6: Delete the IAM execution role.

  • Open the IAM console and click Roles in the left navigation.
  • Search for roles containing clickstream-processor.
  • Select the Lambda execution role and click Delete.
  • Also search for any role created by Firehose (it may contain firehose in its name) and delete it.
  • Confirm each deletion when prompted.

That's a Wrap!

That's a Wrap!

Nice work. You just built a complete real-time streaming data pipeline from scratch using Amazon Kinesis Data Streams.

You've learned how to:

  • Built a real-time streaming pipeline end-to-end: producing clickstream events from CloudShell, streaming them through Kinesis Data Streams, processing them with Lambda, and delivering them to S3 via Firehose.
  • Experienced stream replay first-hand by re-reading data that was already processed. This is impossible with SQS, where messages disappear after consumption.
  • Proved that multiple independent consumers can read the same data simultaneously without interfering with each other, and observed partition key ordering within a shard.
  • Secret Mission: Split your stream into 2 shards and proved that partition keys deterministically route to specific shards.

Ready to quiz yourself?