Secure Financial Data Pipeline

Introduction

30 Second Summary

Every time you tap your credit card, sensitive transaction data flows through a pipeline that must be encrypted, validated, and stored securely. Financial institutions spend billions protecting this data from breaches.

In this project, you will build a secure, serverless data pipeline on AWS that simulates credit card transactions, encrypts them with a customer-managed key, transforms them with a serverless function, and stores them for SQL-based analytics.

What You'll Build

An end-to-end financial data pipeline on AWS that ingests, encrypts, transforms, and queries simulated transaction data using production-grade security patterns.

By the end of this project, you'll have:

• A Python data generator that streams simulated financial transactions to Amazon Data Firehose.
• A customer-managed KMS key encrypting all data at rest in Amazon S3, enforced by a bucket policy.
• A Lambda transformation function that validates, enriches, and scores each transaction with a risk level.
• An Athena analytics layer for querying encrypted financial data with SQL.
• Secret Mission: Add a real-time fraud alert system that flags high-value transactions and writes alerts to a separate prefix.

Are there any prerequisites?

You need an AWS account (Free Tier or paid), Python 3 installed on your machine, and the AWS CLI installed and configured. Step 1 walks you through verifying each of these.

Before We Start

You are about to build a secure financial data pipeline on AWS. The pipeline will ingest simulated credit card transactions, encrypt them with a customer-managed key, transform them with a serverless function, and land them in a data lake for analytics.

Before you start building, you need to confirm your tools are connected and your environment is ready. In this step, you will verify your AWS CLI connection, confirm Python 3 is available, install the AWS SDK, and set up a project folder.

In this step, get ready to:

• Verify your AWS CLI is configured and connected to your AWS account.
• Verify Python 3 and install the boto3 AWS SDK.
• Set your default AWS region and create a project folder.

Verify your AWS CLI connection

The AWS CLI is your command-line interface to AWS services. To confirm it is installed and properly connected to your account, you will run a command that asks AWS to identify who is calling.

• Open your terminal by pressing Cmd+Space (macOS) or the Windows key (Windows), typing Terminal, and pressing Enter.
• Confirm your AWS CLI is connected by running this command:

aws sts get-caller-identity

✔️ I see a JSON response with my Account and Arn

You should see output similar to this:

{
    "UserId": "AIDASAMPLEUSERID",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/YourUsername"
}

Your AWS CLI is connected and ready. Move on to the next substep.

ⓧ I see a credentials error

This means the AWS CLI is installed but has not been configured with your access keys yet.

• Run the following command to configure your credentials:

aws configure

The CLI will prompt you for four values. Enter your AWS Access Key ID, Secret Access Key, default region (us-east-1), and default output format (json).

After configuring, run aws sts get-caller-identity again to confirm it works.

ⓧ Command not found

This means the AWS CLI is not installed on your machine.

• Install the AWS CLI by running this command:

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg" && sudo installer -pkg AWSCLIV2.pkg -target /

• After installation completes, close and reopen your terminal.
• Run aws configure to set up your credentials, then run aws sts get-caller-identity to confirm.

What does this command do?

The aws sts get-caller-identity command asks AWS Security Token Service to identify who is making the request. It returns your Account ID, User ID, and ARN (Amazon Resource Name). This confirms your credentials are valid and your CLI can reach AWS.

Verify Python 3 and install boto3

Your data generator script will use Python and the boto3 library to send transactions to AWS. You need to confirm Python 3 is available and then install the SDK.

• Verify Python 3 is installed by running:

python3 --version

✔️ I see a version number

You should see something like Python 3.x.x printed in your terminal. Python is ready.

ⓧ Command not found

Python 3 is not installed on your machine. Visit the official Python downloads page at python.org/downloads and install the latest Python 3 release for your operating system.

After installing, close and reopen your terminal, then run python3 --version again to confirm.

• Install the boto3 library (the AWS SDK for Python) by running:

pip3 install boto3

What is boto3?

The boto3 library is the official AWS SDK for Python. It lets your Python scripts interact with AWS services like Amazon Data Firehose, S3, and Lambda.

You will use it later to send simulated transactions into your pipeline.

Seeing a permissions error with pip3?

If you see a permissions error, try running pip3 install boto3 --user instead. This installs the package to your user directory without needing administrator privileges.

Help me fix this permissions error

Set your AWS region and create a project folder

All your AWS resources need to live in the same region so they can communicate. You will set us-east-1 as your default region, which has the broadest free tier coverage.

• Set your default AWS region by running:

aws configure set region us-east-1

Why us-east-1?

The us-east-1 region (N. Virginia) has the broadest service availability and free tier coverage. All the services in this project (KMS, Firehose, Lambda, S3, Athena) are available here.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create your project folder and move into it by running these commands:

mkdir financial-pipeline
cd financial-pipeline

• Confirm the folder was created by running:

ls

You should see an empty directory. This is where you will save your Python scripts throughout the project.

Seeing 'No such file or directory' for Desktop?

If your system uses a different path for the Desktop (common on some Linux distros), replace ~/Desktop with any folder you can easily find later. The important thing is that you have a dedicated financial-pipeline folder to work in.

Help me find where to create my project folder

Your environment is ready. Next up, you will create a customer-managed encryption key in AWS KMS to protect your financial data.

Set Up Your AWS Environment

Before building the pipeline, you need to confirm your local environment can communicate with AWS. The three tools you need are the AWS CLI, Python 3, and the boto3 library.

In this step, you will verify each tool is working and lock in your default AWS region. This ensures every command you run later targets the correct environment.

In this step, get ready to:

• Confirm your AWS CLI connection.
• Verify Python 3 and boto3.
• Set your default AWS region to us-east-1.

Confirm your AWS CLI connection

The AWS CLI is how you will create and manage every resource in this project from your terminal. A quick identity check confirms your credentials are valid and your account is reachable.

• Verify your AWS CLI can reach your account by running this command:

aws sts get-caller-identity

You should see JSON output containing your UserId, Account, and Arn fields. This proves your CLI is authenticated and connected.

Seeing an error instead of JSON?

If you see a credentials error, your AWS CLI may not be configured yet. Run aws configure and enter your Access Key ID, Secret Access Key, and default region when prompted.

If the command is not found, you may need to install the AWS CLI. On macOS, run the following:

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg" && sudo installer -pkg AWSCLIV2.pkg -target /

Help me troubleshoot my AWS CLI connection.

Verify Python 3 and boto3

The data generator script you will write later uses Python 3 and the boto3 library to send records to AWS. Confirming both are available now saves debugging later.

• Check that Python 3 is installed by running:

python3 --version

You should see a version number like Python 3.x.x printed in your terminal.

• Ensure the boto3 library is installed by running:

pip3 install boto3

If boto3 is already installed, you will see a message confirming the requirement is satisfied. If it was missing, pip will download and install it.

What is boto3?

boto3 is the official AWS SDK for Python. It lets your Python scripts create, configure, and manage AWS services programmatically. In this project, you will use it to send transaction records to Amazon Data Firehose.

Command not found for python3 or pip3?

If python3 is not found, you need to install Python 3 from python.org before continuing.

If pip3 is not found, try python3 -m pip install boto3 as an alternative.

Help me get Python 3 and pip working on my machine.

Set your default AWS region

Every AWS resource lives in a specific region. Setting a default means you will not need to pass a region flag on every command for the rest of this project.

• Set your default region to us-east-1 by running this command:

aws configure set region us-east-1

Why us-east-1?

The US East (N. Virginia) region has the broadest free tier coverage and is the first region to receive new AWS features. It is the standard choice for learning projects.

• Confirm the region is set correctly by running:

aws configure get region

You should see us-east-1 printed in your terminal.

Seeing a different region or no output?

If you see a different region, run the aws configure set region us-east-1 command again. Make sure there are no typos.

If you see no output at all, your AWS CLI configuration file may not have been created yet. Run aws configure to walk through the full setup.

Help me fix my AWS region configuration.

Your environment is ready to go. Next up, you will create a KMS encryption key to protect your financial data at rest.

Create a KMS Customer-Managed Key

Your environment is ready. Now it's time to create the foundation of your pipeline's security: an encryption key.

In production financial systems, encryption keys protect sensitive data like credit card numbers and transaction amounts. You'll create a customer-managed AWS KMS key that gives you full control over who can encrypt and decrypt your data.

In this step, get ready to:

• Create a symmetric KMS key for encrypting financial data.
• Assign a human-readable alias to your key.
• Verify the key is active and ready for use.

Create a symmetric KMS key

A symmetric encryption key uses the same key for both encrypting and decrypting data. This is the standard key type for protecting data at rest in AWS services like S3.

• Create your KMS key by running this command:

aws kms create-key

What does this command do?

The aws kms create-key command with no parameters creates a symmetric encryption key using the default settings: SYMMETRIC_DEFAULT key spec, ENCRYPT_DECRYPT usage, and AWS-generated key material.

The key material never leaves AWS KMS unencrypted. This is the foundational security property that makes KMS suitable for financial data.

You should see a JSON response containing a KeyMetadata object. The output includes fields like KeyId, Arn, KeyState, and KeyUsage.

• Copy the KeyId value from the output. It looks like a UUID (e.g. 1234abcd-12ab-34cd-56ef-1234567890ab). You will need this in the next commands.

Seeing an access denied error?

Your IAM user or role needs the kms:CreateKey permission. If you're using an admin user or a user with the AWSKeyManagementServicePowerUser managed policy, this should work. Check that your CLI credentials are configured correctly.

help me troubleshoot a KMS access denied error when running aws kms create-key

Create an alias for easy reference

Key IDs are long UUIDs that are hard to remember. An alias gives your key a friendly name you can use in other AWS commands and services.

• Create an alias for your key by running this command, replacing the key ID with your own:

aws kms create-alias \
    --alias-name alias/financial-pipeline-key \
    --target-key-id [[KMS_KEY_ID="your KeyId from the previous command"]]

What does this command do?

The --alias-name flag sets the friendly name. Alias names must start with alias/ and cannot start with alias/aws/ (that prefix is reserved for AWS-managed keys).

The --target-key-id flag links the alias to your specific key. Once created, you can use alias/financial-pipeline-key anywhere you would use the full key ID.

This command produces no output on success. Silence means it worked.

Seeing an AlreadyExistsException?

This means an alias with the name alias/financial-pipeline-key already exists in your account and region. If you're re-running this project, you can delete the old alias first with aws kms delete-alias --alias-name alias/financial-pipeline-key and try again.

help me fix a KMS AlreadyExistsException when creating an alias

Verify the key is active

Before using the key in your pipeline, confirm it is in the Enabled state.

• Describe your key using the alias by running:

aws kms describe-key \
    --key-id alias/financial-pipeline-key

What should I see?

The output shows a KeyMetadata object. Look for these fields:

• "KeyState": "Enabled" confirms the key is ready for cryptographic operations.
• "KeyUsage": "ENCRYPT_DECRYPT" confirms it can encrypt and decrypt data.
• "KeyManager": "CUSTOMER" confirms this is your customer-managed key, not an AWS-managed key.

Why a customer-managed key instead of an AWS-managed key?

AWS-managed keys (like aws/s3) are free but give you no control over key policies, rotation schedules, or audit trails. Financial compliance requires that you control who can access encryption keys and when they rotate.

Customer-managed keys cost $1/month but provide full key policy control, automatic CloudTrail logging of every use, and configurable rotation. This is the pattern used in real financial services environments.

Your encryption key is created, aliased, and verified. Next up, you'll create the S3 bucket that uses this key to encrypt every piece of financial data that lands in your data lake.

Create the Encrypted S3 Data Lake

Your KMS key is ready to protect data. Now you need somewhere to store that data.

In this step, you will create an Amazon S3 bucket that serves as your encrypted data lake. Every object written to this bucket will be encrypted with your customer-managed KMS key, enforced by a bucket policy that rejects any unencrypted uploads.

In this step, get ready to:

• Create an S3 bucket with default SSE-KMS encryption.
• Set up the folder prefix structure for your pipeline.
• Apply a bucket policy that enforces KMS encryption on all uploads.

Create the S3 bucket with KMS encryption

Your data lake needs a globally unique bucket name. A common pattern is to append your AWS account ID to ensure uniqueness.

• First, capture your AWS account ID into a variable by running this command:

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
echo $ACCOUNT_ID

What does this command do?

`aws sts get-caller-identity` returns details about your current AWS credentials. The `--query Account --output text` flags extract just the 12-digit account ID and store it in the `ACCOUNT_ID` variable.

• Create the S3 bucket by running this command:

aws s3api create-bucket \
  --bucket financial-pipeline-data-$ACCOUNT_ID \
  --region us-east-1

Why no LocationConstraint?

When creating a bucket in `us-east-1`, you do not need to specify `--create-bucket-configuration`. That parameter is only required for regions other than `us-east-1`.

• You should see a JSON response with a `Location` field showing your new bucket name.
• Now configure default SSE-KMS encryption on the bucket by running this command:

aws s3api put-bucket-encryption \
  --bucket financial-pipeline-data-$ACCOUNT_ID \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "alias/financial-pipeline-key"
        },
        "BucketKeyEnabled": true
      }
    ]
  }'

What does this command do?

`put-bucket-encryption` sets the default encryption for all new objects uploaded to the bucket. The `SSEAlgorithm` of `aws:kms` tells S3 to use KMS encryption. `KMSMasterKeyID` points to the alias you created in the previous step.

`BucketKeyEnabled: true` activates S3 Bucket Keys, which reduce the number of calls from S3 to KMS and lower encryption costs.

• Verify the encryption is configured by running:

aws s3api get-bucket-encryption \
  --bucket financial-pipeline-data-$ACCOUNT_ID

• You should see a JSON response showing `SSEAlgorithm` as `aws:kms` and your `KMSMasterKeyID` matching `alias/financial-pipeline-key`.

Seeing an AccessDenied error?

Make sure the IAM user or role you are using has `s3:PutEncryptionConfiguration` permission on the bucket. The bucket owner has this by default.

Also confirm the KMS key alias `alias/financial-pipeline-key` exists and is active by running `aws kms describe-key --key-id alias/financial-pipeline-key`.

Help me troubleshoot my S3 bucket encryption setup

Create the bucket prefix structure

S3 uses prefixes (similar to folders) to organize objects. Your pipeline needs three prefixes to separate raw input, transformed output, and Athena query results.

• Create the three prefixes by running these commands:

aws s3api put-object --bucket financial-pipeline-data-$ACCOUNT_ID --key raw/
aws s3api put-object --bucket financial-pipeline-data-$ACCOUNT_ID --key transformed/
aws s3api put-object --bucket financial-pipeline-data-$ACCOUNT_ID --key athena-results/

What do these prefixes do?

• `raw/` will hold the original transaction data from Firehose before transformation.
• `transformed/` will store the validated and enriched records after Lambda processes them.
• `athena-results/` will store query output when you run SQL against your data later.

• Verify the prefixes exist by listing the bucket contents:

aws s3 ls s3://financial-pipeline-data-$ACCOUNT_ID/

• You should see three entries listed: `PRE athena-results/`, `PRE raw/`, and `PRE transformed/`.

Apply a bucket policy to enforce encryption

Default encryption handles new objects, but it does not prevent someone from explicitly uploading with a different encryption type. A bucket policy adds a hard enforcement layer that denies any upload not using SSE-KMS.

• Create a file called `bucket-policy.json` by running this command:

cat > bucket-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnencryptedObjectUploads",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::financial-pipeline-data-ACCOUNT_ID_PLACEHOLDER/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "aws:kms"
        }
      }
    }
  ]
}
EOF

What does this policy do?

This policy uses an explicit `Deny` effect. Any `PutObject` request that does not include the `x-amz-server-side-encryption: aws:kms` header will be rejected.

The `Principal: "*"` means this applies to everyone, including your own account. Combined with default encryption, this creates a belt-and-suspenders approach that ensures no data lands in your bucket without KMS protection.

• Before applying the policy, update the placeholder with your actual account ID by running:

sed -i '' "s/ACCOUNT_ID_PLACEHOLDER/$ACCOUNT_ID/g" bucket-policy.json

On Linux, drop the empty quotes

If you are on Linux instead of macOS, run `sed -i "s/ACCOUNT_ID_PLACEHOLDER/$ACCOUNT_ID/g" bucket-policy.json` without the empty quotes after `-i`.

• Apply the bucket policy by running:

aws s3api put-bucket-policy \
  --bucket financial-pipeline-data-$ACCOUNT_ID \
  --policy file://bucket-policy.json

• Verify the policy is in place by running:

aws s3api get-bucket-policy \
  --bucket financial-pipeline-data-$ACCOUNT_ID \
  --output text

• You should see your policy JSON with the `DenyUnencryptedObjectUploads` statement displayed.

Getting a MalformedPolicy error?

Check that the `Resource` field in `bucket-policy.json` matches your actual bucket name exactly. Run `cat bucket-policy.json` to inspect the file and confirm the account ID was substituted correctly.

Also make sure the JSON is valid. A missing comma or extra bracket is the most common cause.

Help me fix my S3 bucket policy

Your encrypted data lake is ready. Every object stored here will be protected by your customer-managed KMS key, with a bucket policy guaranteeing no unencrypted data can sneak in. Next up, you will build the Lambda function that validates and enriches financial transactions before they land in this bucket.

Build the Lambda Transformation Function

Your encrypted S3 data lake is ready to receive financial transactions. Before data lands there, it needs to be validated and enriched.

In this step, you will build an AWS Lambda function that acts as a transformation layer. It validates incoming transaction records, drops invalid ones, and enriches valid records with a risk score based on the transaction amount.

In this step, get ready to:

• Write a Python Lambda function that validates and enriches financial transactions.
• Create an IAM role with least-privilege permissions for the Lambda function.
• Deploy the Lambda function to AWS using the CLI.

Write the Lambda transformation function

Amazon Data Firehose can invoke a Lambda function on every batch of records it receives. Your function will decode each record, check it for required fields and valid values, assign a risk score, and return the result to Firehose in the expected format.

• Create a new file called lambda_transform.py on your Desktop (or in the same working directory you have been using for this project).
• Paste the following code into lambda_transform.py:

import json
import base64


def lambda_handler(event, context):
    output_records = []

    for record in event["records"]:
        # Decode the base64-encoded record from Firehose
        payload = base64.b64decode(record["data"]).decode("utf-8")

        try:
            transaction = json.loads(payload)
        except json.JSONDecodeError:
            # Return failed records back to Firehose for retry
            output_records.append({
                "recordId": record["recordId"],
                "result": "ProcessingFailed",
                "data": record["data"]
            })
            continue

        # Validate required fields exist
        required_fields = ["transaction_id", "timestamp", "card_number",
                           "merchant", "amount", "currency", "location"]
        if not all(field in transaction for field in required_fields):
            output_records.append({
                "recordId": record["recordId"],
                "result": "Dropped",
                "data": record["data"]
            })
            continue

        # Validate amount is a number
        if not isinstance(transaction["amount"], (int, float)):
            output_records.append({
                "recordId": record["recordId"],
                "result": "Dropped",
                "data": record["data"]
            })
            continue

        # Validate amount is not negative
        if transaction["amount"] < 0:
            output_records.append({
                "recordId": record["recordId"],
                "result": "Dropped",
                "data": record["data"]
            })
            continue

        # Enrich with risk score based on amount thresholds
        amount = transaction["amount"]
        if amount > 5000:
            risk_score = "HIGH"
        elif amount > 1000:
            risk_score = "MEDIUM"
        else:
            risk_score = "LOW"

        transaction["risk_score"] = risk_score

        # Encode the enriched record back to base64 for Firehose
        output_payload = json.dumps(transaction) + "\n"
        encoded_data = base64.b64encode(output_payload.encode("utf-8")).decode("utf-8")

        output_records.append({
            "recordId": record["recordId"],
            "result": "Ok",
            "data": encoded_data
        })

    return {"records": output_records}

What does this code do?

This function processes a batch of records that Firehose sends to it. For each record, it:

• Decodes the base64 payload that Firehose sends.
• Validates the JSON structure and checks for all seven required fields (transaction_id, timestamp, card_number, merchant, amount, currency, location).
• Drops records with missing fields, non-numeric amounts, or negative amounts.
• Assigns a risk score: HIGH for amounts over $5,000, MEDIUM for amounts over $1,000, and LOW for everything else.
• Returns each record with a result status (Ok, Dropped, or ProcessingFailed) so Firehose knows what to do with it.

• Save lambda_transform.py.

Not sure where to save the file?

Save it in your current working directory on your Desktop. You will zip and upload this file in substep 4.3. Help me find the right place to save my Lambda function file.

Create an IAM role for the Lambda function

Every Lambda function needs an IAM role that defines what AWS services it can access. Following the principle of least privilege, this role will only grant permission to write logs to CloudWatch.

• Create the IAM role by running this command:

aws iam create-role \
  --role-name lambda-firehose-transform-role \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

What does this command do?

This creates a role called lambda-firehose-transform-role with a trust policy that allows the Lambda service to assume it. The trust policy says "Lambda is allowed to use this role."

The role has no permissions yet. You will add a permissions policy next.

• You should see JSON output showing the role details, including an Arn field. Copy and save the ARN value. You will need it when deploying the Lambda function.
• Attach a CloudWatch Logs permissions policy to the role by running:

aws iam put-role-policy \
  --role-name lambda-firehose-transform-role \
  --policy-name LambdaCloudWatchLogs \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],"Resource":"arn:aws:logs:*:*:*"}]}'

Why only CloudWatch Logs?

This Lambda function only needs to process data in memory and return it. It does not read from S3 or call any other AWS service directly. The only external action it performs is writing execution logs to CloudWatch for debugging.

This is the least-privilege approach. You give a role only the permissions it absolutely needs. If the function needed S3 access later, you would add those permissions separately.

• Verify the role has the correct policy attached by running:

aws iam get-role-policy \
  --role-name lambda-firehose-transform-role \
  --policy-name LambdaCloudWatchLogs

• You should see JSON output showing the policy document with logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents actions.

Getting an access denied error?

Make sure your AWS CLI user has IAM permissions to create roles and put role policies. If you are using an admin user, this should work without issues. Help me troubleshoot IAM permission errors when creating roles.

Deploy the Lambda function

With the code written and the IAM role ready, you can now package and deploy the function to AWS.

• Package the Lambda function into a zip file by running:

zip lambda_transform.zip lambda_transform.py

• You should see output confirming the file was added to the archive.
• Deploy the Lambda function by running the following command. Replace the --role value with the ARN you copied earlier:

aws lambda create-function \
  --function-name firehose-transform-function \
  --runtime python3.12 \
  --handler lambda_transform.lambda_handler \
  --role [[LAMBDA_ROLE_ARN="arn:aws:iam::123456789012:role/lambda-firehose-transform-role"]] \
  --zip-file fileb://lambda_transform.zip

What does this command do?

• --function-name sets the function's name in AWS to firehose-transform-function.
• --runtime python3.12 tells Lambda to use the Python 3.12 runtime environment.
• --handler lambda_transform.lambda_handler tells Lambda which file and function to invoke. The format is filename.function_name.
• --role attaches the IAM role you just created, giving the function its permissions.
• --zip-file fileb:// uploads the deployment package from your local machine. The fileb:// prefix tells the CLI to read the file as binary.

• You should see JSON output with details about your new function, including "State": "Active" or "State": "Pending". If the state is Pending, wait a few seconds and it will become Active.
• Verify the function is active by running:

aws lambda get-function --function-name firehose-transform-function --query 'Configuration.State'

• You should see "Active" in the output.

Seeing an InvalidParameterValueException?

This usually means one of two things:

• The role ARN is incorrect. Double check you copied the full ARN from the create-role output.
• The role was just created and IAM has not finished propagating it. Wait 10 seconds and try the command again.

Help me fix an InvalidParameterValueException when creating my Lambda function.

Your Lambda transformation function is deployed and ready to process records. Next up, you will create the Firehose delivery stream that connects everything together.

Create the Firehose Delivery Stream

Your Lambda function is deployed and ready to transform records. Now you need the central piece of the pipeline: the delivery stream that receives raw transactions, passes them through your Lambda for validation and enrichment, then delivers the processed data to S3.

In this step, you will create an IAM role for Amazon Data Firehose, configure the delivery stream with your Lambda transformation and encrypted S3 destination, and verify it reaches an active state.

In this step, get ready to:

• Create an IAM role granting Firehose permissions to invoke Lambda, write to S3, and use your KMS key.
• Create the Amazon Data Firehose delivery stream with Lambda transformation, S3 destination, and SSE-KMS encryption.
• Verify the delivery stream status is Active.

Create the Firehose IAM role

Firehose needs its own IAM role to access your Lambda function, S3 bucket, and KMS key. This follows the same least-privilege pattern you used for the Lambda role, but scoped to what Firehose specifically needs.

• Create the trust policy file that allows Firehose to assume the role by running this command:

cat > firehose-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

• Create the IAM role by running:

aws iam create-role \
  --role-name firehose-delivery-role \
  --assume-role-policy-document file://firehose-trust-policy.json

What does this command do?

The --assume-role-policy-document flag attaches a trust policy that tells AWS which service can assume this role. Here, only firehose.amazonaws.com can use it.

• Now create the permissions policy file that grants Firehose access to your Lambda, S3, and KMS resources by running:

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
KEY_ARN=$(aws kms describe-key --key-id alias/financial-pipeline-key --query KeyMetadata.Arn --output text)

cat > firehose-permissions-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "lambda:InvokeFunction",
        "lambda:GetFunctionConfiguration"
      ],
      "Resource": "arn:aws:lambda:us-east-1:${ACCOUNT_ID}:function:firehose-transform-function"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::financial-pipeline-data-${ACCOUNT_ID}",
        "arn:aws:s3:::financial-pipeline-data-${ACCOUNT_ID}/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": "${KEY_ARN}"
    }
  ]
}
EOF

What does this policy do?

This policy has three statements, each granting the minimum permissions for one job.

• The Lambda statement allows Firehose to invoke your transformation function and read its configuration.
• The S3 statement allows Firehose to write objects to your bucket and manage multipart uploads.
• The KMS statement allows Firehose to encrypt data before writing it to S3.

• Attach the permissions policy to your role by running:

aws iam put-role-policy \
  --role-name firehose-delivery-role \
  --policy-name FirehoseDeliveryPermissions \
  --policy-document file://firehose-permissions-policy.json

You should see no output, which means the command succeeded.

Seeing an access denied error?

Make sure your AWS CLI user has iam:CreateRole and iam:PutRolePolicy permissions. If you are using an IAM user with limited permissions, you may need to add these actions to your user's policy.

Help me troubleshoot IAM permission errors when creating the Firehose delivery role.

Create the delivery stream

With the IAM role in place, you can now create the Firehose delivery stream. This stream ties together your Lambda transformation, S3 destination, and KMS encryption in a single configuration.

• Wait about 10 seconds for the IAM role to propagate, then create the delivery stream by running:

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
ROLE_ARN="arn:aws:iam::${ACCOUNT_ID}:role/firehose-delivery-role"
LAMBDA_ARN="arn:aws:lambda:us-east-1:${ACCOUNT_ID}:function:firehose-transform-function"
KEY_ARN=$(aws kms describe-key --key-id alias/financial-pipeline-key --query KeyMetadata.Arn --output text)

aws firehose create-delivery-stream \
  --delivery-stream-name financial-transactions-stream \
  --delivery-stream-type DirectPut \
  --extended-s3-destination-configuration "{
    \"RoleARN\": \"${ROLE_ARN}\",
    \"BucketARN\": \"arn:aws:s3:::financial-pipeline-data-${ACCOUNT_ID}\",
    \"Prefix\": \"transformed/\",
    \"BufferingHints\": {
      \"SizeInMBs\": 5,
      \"IntervalInSeconds\": 60
    },
    \"CompressionFormat\": \"UNCOMPRESSED\",
    \"EncryptionConfiguration\": {
      \"KMSEncryptionConfig\": {
        \"AWSKMSKeyARN\": \"${KEY_ARN}\"
      }
    },
    \"ProcessingConfiguration\": {
      \"Enabled\": true,
      \"Processors\": [
        {
          \"Type\": \"Lambda\",
          \"Parameters\": [
            {
              \"ParameterName\": \"LambdaArn\",
              \"ParameterValue\": \"${LAMBDA_ARN}\"
            },
            {
              \"ParameterName\": \"NumberOfRetries\",
              \"ParameterValue\": \"3\"
            },
            {
              \"ParameterName\": \"BufferIntervalInSeconds\",
              \"ParameterValue\": \"60\"
            }
          ]
        }
      ]
    }
  }"

What is this command configuring?

This single command creates the entire delivery stream with all its components wired together.

• The --delivery-stream-type DirectPut means your Python script will send records directly to Firehose (no Kinesis Data Stream as a source).
• The Prefix of transformed/ tells Firehose to write all processed records under that S3 prefix.
• The BufferingHints with IntervalInSeconds: 60 means Firehose batches records for up to 60 seconds (or 5 MB, whichever comes first) before writing to S3.
• The EncryptionConfiguration with your KMS key ARN ensures all data written to S3 is encrypted with your customer-managed key.
• The ProcessingConfiguration enables your Lambda function as a transformation step. Firehose sends each batch of records to Lambda before writing to S3.

• You should see a JSON response containing a DeliveryStreamARN value. This confirms the stream creation has started.

Seeing an InvalidArgumentException?

This usually means the IAM role has not finished propagating. Wait 15-20 seconds and try the command again. If the error mentions the KMS key, verify that your key alias is correct by running aws kms describe-key --key-id alias/financial-pipeline-key.

Help me debug why my Firehose delivery stream creation is failing with an InvalidArgumentException.

Verify the stream is active

The delivery stream creation is asynchronous. It starts in a CREATING state and transitions to ACTIVE once it is ready to receive data.

• Wait about 30 seconds, then check the stream status by running:

aws firehose describe-delivery-stream \
  --delivery-stream-name financial-transactions-stream \
  --query DeliveryStreamDescription.DeliveryStreamStatus

• You should see "ACTIVE" in the output.

What if the status is still CREATING?

Wait another 30 seconds and run the describe command again. Stream creation typically takes under a minute. If the status shows CREATING_FAILED, you will need to delete the stream and recreate it after fixing the configuration issue.

Your delivery stream is live and ready to receive data. Next up, you will write a Python script to generate simulated financial transactions and stream them through this pipeline.

Generate and Stream Financial Data

Your pipeline infrastructure is fully wired. The Firehose delivery stream is active, the Lambda transformation function is deployed, and your encrypted S3 bucket is waiting to receive data.

Now it's time to see the pipeline in action. In this step, you'll write a Python script that simulates realistic credit card transactions and streams them directly into Firehose using the put_record_batch API.

In this step, get ready to:

• Write a Python data generator script that creates simulated financial transactions.
• Stream 50 test transactions to your Firehose delivery stream.
• Verify encrypted, transformed data lands in S3.

Write the data generator script

You need a script that creates realistic-looking transaction records and sends them to Amazon Data Firehose in batches. Each transaction will include a unique ID, timestamp, masked card number, merchant name, amount, currency, and location.

• Create a new file called generate_transactions.py in the same directory where you saved lambda_transform.py earlier.
• Add the imports, Firehose client, constants, and the transaction generator function by pasting this code:

import boto3
import json
import random
import uuid
from datetime import datetime, timedelta
import time
import base64

# Create a Firehose client targeting us-east-1
firehose_client = boto3.client("firehose", region_name="us-east-1")

# The delivery stream name must match what you created in Step 5
STREAM_NAME = "financial-transactions-stream"

# Realistic merchant names for simulated transactions
MERCHANTS = [
    "Amazon", "Walmart", "Target", "Starbucks", "Shell Gas",
    "Netflix", "Uber", "DoorDash", "Apple Store", "Home Depot"
]

# Weighted toward USD with some international currencies
CURRENCIES = ["USD", "USD", "USD", "EUR", "GBP"]

# Major US cities for transaction locations
LOCATIONS = [
    "New York, NY", "Los Angeles, CA", "Chicago, IL",
    "Houston, TX", "Miami, FL", "Seattle, WA",
    "Denver, CO", "Atlanta, GA", "Boston, MA", "Phoenix, AZ"
]


def generate_transaction(include_high_value=False):
    # High-value transactions will trigger HIGH risk scores in Lambda
    if include_high_value:
        amount = round(random.uniform(5000, 15000), 2)
    else:
        amount = round(random.uniform(5.00, 2500.00), 2)

    # Mask the card number for security (only show last 4 digits)
    card_number = f"****-****-****-{random.randint(1000, 9999)}"
    transaction = {
        "transaction_id": str(uuid.uuid4()),
        "timestamp": datetime.utcnow().isoformat() + "Z",
        "card_number": card_number,
        "merchant": random.choice(MERCHANTS),
        "amount": amount,
        "currency": random.choice(CURRENCIES),
        "location": random.choice(LOCATIONS)
    }
    return transaction

What does this code do?

The top section sets up the boto3 Firehose client and defines lookup arrays of merchants, currencies, and locations that the generator picks from randomly.

The generate_transaction() function creates one simulated record. When include_high_value is True, it generates amounts above $5,000. These will receive a HIGH risk score from your Lambda transformation function later.

You won't see output yet. The next chunk adds the functions that actually send data to Firehose and orchestrate the run.

• Below the generate_transaction function, add the batch sending and main orchestration logic:

def send_batch(transactions):
    # Format each transaction as a Firehose record (bytes)
    records = []
    for txn in transactions:
        records.append({
            "Data": json.dumps(txn).encode("utf-8")
        })

    # Send up to 500 records per batch via put_record_batch
    response = firehose_client.put_record_batch(
        DeliveryStreamName=STREAM_NAME,
        Records=records
    )
    failed = response.get("FailedPutCount", 0)
    print(f"Sent {len(records)} records, {failed} failed.")
    return response


def main():
    print(f"Generating transactions for stream: {STREAM_NAME}")
    all_transactions = []

    # Generate 45 normal transactions (amounts $5 - $2,500)
    for i in range(45):
        txn = generate_transaction(include_high_value=False)
        all_transactions.append(txn)

    # Generate 5 high-value transactions (amounts $5,000 - $15,000)
    for i in range(5):
        txn = generate_transaction(include_high_value=True)
        all_transactions.append(txn)

    # Shuffle so high-value transactions appear at random positions
    random.shuffle(all_transactions)

    # Send in batches of 10 with a 1-second pause between batches
    batch_size = 10
    for i in range(0, len(all_transactions), batch_size):
        batch = all_transactions[i:i + batch_size]
        send_batch(batch)
        time.sleep(1)

    print(f"\nDone! Sent {len(all_transactions)} transactions total.")
    print("Wait 60-90 seconds for Firehose buffer to flush to S3.")


if __name__ == "__main__":
    main()

What does this code do?

The send_batch() function takes a list of transactions, encodes each as JSON bytes, and sends them to Firehose using put_record_batch. This API accepts up to 500 records per call.

The main() function generates 45 normal transactions and 5 high-value ones (above $5,000), shuffles them randomly, then streams them in batches of 10. The 1-second pause between batches simulates a realistic data stream rather than a single burst.

• Save generate_transactions.py.

Seeing a syntax error when you save?

Make sure the send_batch and main functions are at the same indentation level as generate_transaction above them. Python is sensitive to indentation.

Also check that the if __name__ == "__main__": line is at the file's top level (no indentation).

help me fix a syntax error in my generate_transactions.py file

✔️ Awesome, I've got everything!

Great. Make sure you've saved generate_transactions.py before moving on.

ⓧ I'd like to double check the full code

import boto3
import json
import random
import uuid
from datetime import datetime, timedelta
import time
import base64

firehose_client = boto3.client("firehose", region_name="us-east-1")

STREAM_NAME = "financial-transactions-stream"

MERCHANTS = [
    "Amazon", "Walmart", "Target", "Starbucks", "Shell Gas",
    "Netflix", "Uber", "DoorDash", "Apple Store", "Home Depot"
]

CURRENCIES = ["USD", "USD", "USD", "EUR", "GBP"]

LOCATIONS = [
    "New York, NY", "Los Angeles, CA", "Chicago, IL",
    "Houston, TX", "Miami, FL", "Seattle, WA",
    "Denver, CO", "Atlanta, GA", "Boston, MA", "Phoenix, AZ"
]


def generate_transaction(include_high_value=False):
    if include_high_value:
        amount = round(random.uniform(5000, 15000), 2)
    else:
        amount = round(random.uniform(5.00, 2500.00), 2)

    card_number = f"****-****-****-{random.randint(1000, 9999)}"
    transaction = {
        "transaction_id": str(uuid.uuid4()),
        "timestamp": datetime.utcnow().isoformat() + "Z",
        "card_number": card_number,
        "merchant": random.choice(MERCHANTS),
        "amount": amount,
        "currency": random.choice(CURRENCIES),
        "location": random.choice(LOCATIONS)
    }
    return transaction


def send_batch(transactions):
    records = []
    for txn in transactions:
        records.append({
            "Data": json.dumps(txn).encode("utf-8")
        })

    response = firehose_client.put_record_batch(
        DeliveryStreamName=STREAM_NAME,
        Records=records
    )
    failed = response.get("FailedPutCount", 0)
    print(f"Sent {len(records)} records, {failed} failed.")
    return response


def main():
    print(f"Generating transactions for stream: {STREAM_NAME}")
    all_transactions = []

    for i in range(45):
        txn = generate_transaction(include_high_value=False)
        all_transactions.append(txn)

    for i in range(5):
        txn = generate_transaction(include_high_value=True)
        all_transactions.append(txn)

    random.shuffle(all_transactions)

    batch_size = 10
    for i in range(0, len(all_transactions), batch_size):
        batch = all_transactions[i:i + batch_size]
        send_batch(batch)
        time.sleep(1)

    print(f"\nDone! Sent {len(all_transactions)} transactions total.")
    print("Wait 60-90 seconds for Firehose buffer to flush to S3.")


if __name__ == "__main__":
    main()

Run the script to stream transactions

With the script saved, you can now push simulated financial data through your entire pipeline. The script sends 50 transactions (including 5 high-value ones above $5,000) in batches of 10.

• Run the data generator by executing this command in your terminal:

python3 generate_transactions.py

What should I see?

You should see output like this, printed once per second as each batch of 10 records is sent:

Generating transactions for stream: financial-transactions-stream
Sent 10 records, 0 failed.
Sent 10 records, 0 failed.
Sent 10 records, 0 failed.
Sent 10 records, 0 failed.
Sent 10 records, 0 failed.

Done! Sent 50 transactions total.
Wait 60-90 seconds for Firehose buffer to flush to S3.

If any records show as failed, the script will still continue. A small number of failures can happen due to throttling, but 0 failed is the expected result for 50 records.

Getting a credentials or access denied error?

Your AWS CLI credentials need permission to call firehose:PutRecordBatch. Verify your CLI is configured correctly by running aws sts get-caller-identity and checking that it returns your expected account.

If you see a ResourceNotFoundException, confirm that the stream name in your script matches financial-transactions-stream exactly and that your region is set to us-east-1.

help me fix an access denied or resource not found error when running my Firehose data generator script

Verify data has landed in S3

Firehose buffers records before delivering them to S3. Your delivery stream is configured with a 60-second buffer interval, so you need to wait before checking for files.

• Wait 60 to 90 seconds after the script finishes. This gives Firehose time to flush the buffer to S3.
• List the objects in your transformed/ prefix by running this command:

aws s3 ls s3://[[BUCKET_NAME="financial-pipeline-data-YOUR-ACCOUNT-ID"]]/transformed/ --recursive

What should I see?

You should see one or more objects listed with a date, size, and key path. The output looks something like:

2026-05-22 14:32:10       4521 transformed/2026/05/22/14/financial-transactions-stream-1-2026-05-22...

The exact filenames and timestamps will differ. The important thing is that files exist under the transformed/ prefix. These files contain your transactions after Lambda validated and enriched them with risk scores, encrypted at rest by your KMS key.

Not seeing any objects?

If the listing is empty, wait another 30 seconds and try again. The buffer interval is 60 seconds but delivery can take slightly longer.

If objects still don't appear, check the Firehose delivery stream status by running aws firehose describe-delivery-stream --delivery-stream-name financial-transactions-stream and confirm the DeliveryStreamStatus is ACTIVE.

help me troubleshoot why my Firehose delivery stream is not delivering records to S3

Your data is now flowing through the full pipeline. Transactions are generated, streamed to Firehose, validated and enriched by Lambda, encrypted with your KMS key, and stored in S3. Next up, you'll query this encrypted data using SQL with Amazon Athena.

Query Financial Data with Athena

Your pipeline has ingested, transformed, and encrypted financial transactions into S3. Now you need a way to actually analyze that data without downloading files or writing custom scripts.

In this step, you will set up Amazon Athena to run SQL queries directly against your encrypted S3 data. Athena uses a schema-on-read approach, meaning you define the table structure at query time rather than when the data is written.

In this step, get ready to:

• Create an Athena workgroup with query results stored in your encrypted bucket.
• Define a Glue database and table so Athena knows the schema of your transaction data.
• Run SQL analytics queries on your encrypted financial data.

Create an Athena workgroup

Athena workgroups let you isolate query execution and control where results are stored. You will create a workgroup that sends query results to the athena-results/ prefix in your encrypted bucket.

• Create the Athena workgroup by running this command:

aws athena create-work-group \
    --name financial-pipeline-workgroup \
    --configuration ResultConfiguration={OutputLocation="s3://[[BUCKET_NAME="financial-pipeline-data-YOUR-ACCOUNT-ID"]]/athena-results/"},EnforceWorkGroupConfiguration="true"

What does this command do?

• --name sets the workgroup name to financial-pipeline-workgroup.
• ResultConfiguration={OutputLocation=...} tells Athena where to store query results in S3.
• EnforceWorkGroupConfiguration="true" ensures all queries in this workgroup use this output location, overriding any client-side settings.

Why a separate workgroup?

The default Athena workgroup requires you to specify a results location for every query. By creating a dedicated workgroup with a fixed output location, you guarantee all query results land in your KMS-encrypted bucket automatically.

• Verify the workgroup was created by running:

aws athena get-work-group --work-group financial-pipeline-workgroup

• You should see a JSON response showing the workgroup details, including the OutputLocation pointing to your athena-results/ prefix.

Getting an access denied error?

Make sure your IAM user has the athena:CreateWorkGroup permission. Also confirm the S3 bucket and athena-results/ prefix already exist from Step 3.

Help me troubleshoot my Athena workgroup creation error.

Create a Glue database and table

Athena uses the AWS Glue Data Catalog to understand the structure of your data. You need to create a database and an external table definition that maps to your transformed JSON files in S3.

• Create the Glue database by running:

aws glue create-database \
    --database-input '{"Name":"financial_pipeline_db"}'

What does this command do?

This creates a new database called financial_pipeline_db in the Glue Data Catalog. The database acts as a container for your table definitions. Athena will reference this database when you run SQL queries.

• Create the external table that maps to your transformed transaction data by running:

aws glue create-table \
    --database-name financial_pipeline_db \
    --table-input '{
        "Name": "transactions",
        "TableType": "EXTERNAL_TABLE",
        "StorageDescriptor": {
            "Columns": [
                {"Name": "transaction_id", "Type": "string"},
                {"Name": "timestamp", "Type": "string"},
                {"Name": "card_number", "Type": "string"},
                {"Name": "merchant", "Type": "string"},
                {"Name": "amount", "Type": "double"},
                {"Name": "currency", "Type": "string"},
                {"Name": "location", "Type": "string"},
                {"Name": "risk_score", "Type": "string"}
            ],
            "Location": "s3://[[BUCKET_NAME="financial-pipeline-data-YOUR-ACCOUNT-ID"]]/transformed/",
            "InputFormat": "org.apache.hadoop.mapred.TextInputFormat",
            "OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
            "SerdeInfo": {
                "SerializationLibrary": "org.openx.data.jsonserde.JsonSerDe"
            }
        },
        "Parameters": {
            "classification": "json"
        }
    }'

What does this command do?

• Creates an external table named transactions in your financial_pipeline_db database.
• The Columns array defines the schema that matches your Lambda-transformed JSON output, including the risk_score field added during transformation.
• Location points Athena at the transformed/ prefix where Firehose delivered your processed records.
• SerializationLibrary uses the JSON SerDe so Athena knows how to parse each line of JSON.

How does Athena read encrypted data?

Athena reads SSE-KMS encrypted data transparently. You do not need to add any special table properties or encryption configuration to the table definition. As long as your IAM user has permission to use the KMS key, Athena handles decryption automatically when scanning the S3 objects.

• Verify the table was created by running:

aws glue get-table \
    --database-name financial_pipeline_db \
    --name transactions

• You should see a JSON response showing the table schema with all 8 columns and the S3 location.

Table creation failed?

Double-check that financial_pipeline_db was created successfully in the previous command. Also ensure the JSON in your --table-input flag is properly escaped with matching quotes and braces.

Help me debug my Glue table creation error.

Run analytical SQL queries

Now for the payoff. You will run SQL queries against your encrypted financial data to produce real analytics.

• Run a query to see total spend by merchant by running:

aws athena start-query-execution \
    --query-string "SELECT merchant, SUM(amount) as total_spend, COUNT(*) as transaction_count FROM transactions GROUP BY merchant ORDER BY total_spend DESC" \
    --work-group "financial-pipeline-workgroup" \
    --query-execution-context Database=financial_pipeline_db

What does this command do?

This submits a SQL query to Athena using your financial-pipeline-workgroup workgroup. The query groups transactions by merchant and calculates total spend and transaction count for each. Athena returns a QueryExecutionId that you will use to retrieve results.

• Copy the QueryExecutionId from the output.
• Wait a few seconds for the query to complete, then retrieve the results by running:

aws athena get-query-results \
    --query-execution-id [[QUERY_ID="your-query-execution-id"]]

• You should see a JSON response containing rows of merchant names with their total spend and transaction counts.

• Run a second query to count high-risk transactions by running:

aws athena start-query-execution \
    --query-string "SELECT risk_score, COUNT(*) as count, AVG(amount) as avg_amount FROM transactions GROUP BY risk_score" \
    --work-group "financial-pipeline-workgroup" \
    --query-execution-context Database=financial_pipeline_db

• Copy the new QueryExecutionId and retrieve the results:

aws athena get-query-results \
    --query-execution-id [[QUERY_ID_2="your-second-query-execution-id"]]

• You should see results showing HIGH, MEDIUM, and LOW risk categories with their counts and average amounts.

• Run a third query for the average transaction amount by running:

aws athena start-query-execution \
    --query-string "SELECT COUNT(*) as total_transactions, AVG(amount) as avg_amount, MIN(amount) as min_amount, MAX(amount) as max_amount FROM transactions" \
    --work-group "financial-pipeline-workgroup" \
    --query-execution-context Database=financial_pipeline_db

• Copy the QueryExecutionId and retrieve the results:

aws athena get-query-results \
    --query-execution-id [[QUERY_ID_3="your-third-query-execution-id"]]

• You should see the total number of transactions alongside the average, minimum, and maximum amounts.

What just happened?

Athena just queried SSE-KMS encrypted data in S3 without any special decryption step. This is the power of schema-on-read analytics combined with AWS-native encryption. Your data stays encrypted at rest while being fully queryable through standard SQL.

In a production environment, these queries could power dashboards, compliance reports, or automated fraud detection systems.

Query returned zero results?

Make sure you waited at least 60-90 seconds after running generate_transactions.py in the previous step for the Firehose buffer to flush. Verify data exists in the transformed/ prefix by running:

aws s3 ls s3://[[BUCKET_NAME="financial-pipeline-data-YOUR-ACCOUNT-ID"]]/transformed/ --recursive

If you see files listed but Athena returns no results, check that the table Location exactly matches the S3 path where your files are stored (including the trailing slash).

Help me troubleshoot why my Athena query returns no results.

Your secure data pipeline is complete. You have built an end-to-end system that ingests, encrypts, transforms, stores, and queries financial transactions, all serverless and all encrypted. Next up, the Secret Mission challenges you to add real-time fraud alerting on top of this foundation.

Secret mission

Real-Time Fraud Alerts

Your pipeline detects high-risk transactions, but what if you could catch them the moment they land in S3? Build a second Lambda function that triggers on new objects in the transformed/ prefix, scans for transactions above $5,000, and writes JSON alerts to a dedicated alerts/ prefix. Then query those alerts with Athena.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The KMS key costs $1/month (prorated hourly), but all other resources incur negligible charges with no active data flow.

Cost warning

Your KMS customer-managed key costs $1/month even when idle. If you leave all resources running for several days, you will see a small charge on your AWS bill. If you are done experimenting, choose the Delete option below to avoid ongoing costs.

Resources you used:

• KMS customer-managed key (alias/financial-pipeline-key).
• S3 bucket (financial-pipeline-data-{account-id}) with encrypted transaction data.
• Amazon Data Firehose delivery stream (financial-transactions-stream).
• Lambda functions (firehose-transform-function and fraud-alert-function).
• IAM roles (lambda-firehose-transform-role, firehose-delivery-role, fraud-alert-lambda-role).
• Athena workgroup (financial-pipeline-workgroup).
• Glue database (financial_pipeline_db) with tables transactions and fraud_alerts.
• Local Python files (generate_transactions.py, lambda_transform.py, fraud_alert.py).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with your pipeline.

• The KMS key will cost $1/month (prorated hourly).
• S3 storage for your test data is less than 1 MB, costing fractions of a cent.
• Lambda functions and Athena queries remain within free tier at this scale.
• Firehose incurs no charge unless you send additional data.

Pause - I'll come back to this later

Shut down the active data flow to stop ingestion charges. The KMS key still incurs $1/month, but no other charges apply.

• Delete the Firehose delivery stream to stop any accidental ingestion by running:
• Your S3 data, Lambda functions, Athena workgroup, and Glue database remain intact for when you return.

aws firehose delete-delivery-stream --delivery-stream-name financial-transactions-stream

When you return, you can recreate the delivery stream and resume sending data.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow these steps in order.

• Delete the Firehose delivery stream by running:

aws firehose delete-delivery-stream --delivery-stream-name financial-transactions-stream

• Delete both Lambda functions by running:

aws lambda delete-function --function-name firehose-transform-function
aws lambda delete-function --function-name fraud-alert-function

• Delete the inline policies and IAM roles. For each role, delete the inline policy first, then delete the role:

aws iam delete-role-policy --role-name lambda-firehose-transform-role --policy-name lambda-firehose-transform-policy
aws iam delete-role --role-name lambda-firehose-transform-role

aws iam delete-role-policy --role-name firehose-delivery-role --policy-name firehose-delivery-policy
aws iam delete-role --role-name firehose-delivery-role

aws iam delete-role-policy --role-name fraud-alert-lambda-role --policy-name fraud-alert-lambda-policy
aws iam delete-role --role-name fraud-alert-lambda-role

• Empty and delete the S3 bucket by running:

aws s3 rm s3://financial-pipeline-data-[[AWS_ACCOUNT_ID="your-account-id"]]/  --recursive
aws s3 rb s3://financial-pipeline-data-[[AWS_ACCOUNT_ID="your-account-id"]]

• Delete the Glue tables and database by running:

aws glue delete-table --database-name financial_pipeline_db --name transactions
aws glue delete-table --database-name financial_pipeline_db --name fraud_alerts
aws glue delete-database --name financial_pipeline_db

• Delete the Athena workgroup by running:

aws athena delete-work-group --work-group financial-pipeline-workgroup --recursive-delete-option

• Schedule the KMS key for deletion by running:

aws kms schedule-key-deletion --key-id alias/financial-pipeline-key --pending-window-in-days 7

Why does KMS deletion take 7 days?

AWS enforces a minimum 7-day waiting period before permanently deleting a KMS key. This safety net lets you cancel the deletion if you realize data still depends on the key. After the waiting period, the key and all data encrypted with it become permanently unrecoverable.

• Delete your local project files by running:

rm generate_transactions.py lambda_transform.py fraud_alert.py

Nice Work!

Nice Work!

Nice work! You just built a production-grade secure data pipeline that demonstrates how financial institutions protect sensitive transaction data on AWS.

You've learned how to:

• Built a serverless streaming pipeline with Amazon Data Firehose that ingests financial transactions in real time, transforms them with a Lambda function that validates and enriches records with risk scores, and lands them encrypted in S3.
• Implemented encryption at rest using a customer-managed AWS KMS key, enforced via S3 bucket policies that deny any unencrypted writes.
• Queried encrypted financial data transparently with Amazon Athena using schema-on-read analytics, applying IAM least-privilege principles across all pipeline components.
• Built a real-time fraud detection alert system triggered by S3 events that flags high-value transactions and writes alerts to a separate prefix for querying.

Ready to quiz yourself?