Automated Document Intelligence on AWS

Build a serverless RAG pipeline that lets you query PDFs with AI.

Introduction

30 Second Summary

Think about how much time you spend digging through long PDF documents to find one specific answer. Now imagine dropping a file into a folder and instantly being able to ask questions about it in plain English.

In this project, you will build a serverless document intelligence pipeline on AWS that automatically extracts text from uploaded PDFs and makes them searchable using AI-powered natural language Q&A.

What You'll Build

An event-driven pipeline that connects Amazon S3 uploads to Amazon Textract for document extraction, then feeds processed text into an Amazon Bedrock Knowledge Base for Retrieval-Augmented Generation (RAG) queries.

By the end of this project, you'll have:

  • An event-driven pipeline where uploading a PDF to S3 automatically triggers a Lambda function to extract text using Textract.
  • A Bedrock Knowledge Base backed by S3 Vectors that generates vector embeddings from your documents for semantic search.
  • A working RAG query flow where you ask natural language questions and receive grounded, cited answers powered by Claude.
  • Secret Mission: Build a Python CLI tool that lets you upload a document and chat with your corpus from the terminal.

Are there any prerequisites?

You'll need an AWS account with billing enabled. Familiarity with Python and a basic understanding of AWS services (S3, Lambda, IAM) will help you move through the steps confidently.

Set Up Your AWS Environment

Before you can build the document intelligence pipeline, you need a working connection between your local machine and AWS services.

In this step, you will install the AWS CLI, enable the AI models your pipeline will use, and create an IAM identity with the right permissions to orchestrate everything.

In this step, get ready to:
  • Install and configure the AWS CLI v2 on your machine.
  • Enable model access in Amazon Bedrock for Titan Text Embeddings V2 and Anthropic Claude.
  • Create an IAM user with permissions for S3, Lambda, Textract, and Bedrock.
Install and configure the AWS CLI v2

The AWS CLI is your command-line gateway to every AWS service. You will use it throughout this project to create resources, upload files, and query your knowledge base.

  • Download the AWS CLI v2 MSI installer for Windows by visiting this link in your browser:
  • https://awscli.amazonaws.com/AWSCLIV2.msi
  • Run the downloaded AWSCLIV2.msi installer and follow the on-screen prompts. Accept all the defaults.
  • Once the installer completes, open a new Command Prompt or PowerShell window. You need a fresh window so that your system PATH includes the new CLI.
  • Verify the installation by running this command:
aws --version

✔️ I see a version number starting with aws-cli/2

The AWS CLI v2 is installed and ready. You should see output similar to:

aws-cli/2.27.41 Python/3.11.6 Windows/10 exe/AMD64 prompt/off

Your exact version numbers may differ. Any output starting with aws-cli/2 confirms you have version 2 installed.

ⓧ I see aws-cli/1 or an older version

You have an older version of the AWS CLI installed. Download and run the latest MSI installer from the link above. The installer will overwrite the previous version.

After reinstalling, close and reopen your terminal window, then run aws --version again.

ⓧ Command not found

Windows cannot find the aws command. This usually means you need to close and reopen your terminal after installation so the PATH updates.

If that does not help, re-download the MSI installer from the link above and run it again. Make sure you have admin rights on your machine.

Now configure the CLI with your AWS credentials. You will set these up properly in substep 3 after creating your IAM user. For now, run this command to prepare the configuration:

aws configure

What does aws configure do?

This command prompts you for four values: your Access Key ID, Secret Access Key, default region, and output format. It saves them to a local credentials file so every future aws command authenticates automatically.

You will come back to this command after creating your IAM user and generating access keys in substep 3.

  • When prompted, enter the following values (you will replace the key values after substep 3):
  • For Default region name: enter us-east-1.
  • For Default output format: enter json.
  • For the Access Key ID and Secret Access Key prompts, press Enter to skip for now. You will fill these in after creating your IAM user.
Enable model access in Amazon Bedrock

Your pipeline uses two AI models: Amazon Titan Text Embeddings V2 to convert text into vector embeddings, and Anthropic Claude to generate natural-language answers from retrieved documents.

Before you can call these models, you need to enable access in the Amazon Bedrock console.

  • Open the AWS Management Console in your browser and sign in to your AWS account.
  • In the top-right corner, confirm your region is set to US East (N. Virginia) us-east-1. If it is not, click the region name and select it.
  • In the search bar at the top, type Bedrock and select Amazon Bedrock from the results.
  • In the left navigation pane, under Bedrock configurations, choose Model access.
  • Click Modify model access.
  • Select the checkbox next to Amazon Titan Text Embeddings V2.
  • Select the checkbox next to Anthropic Claude (select all Claude models in the Anthropic group).
  • Click Next.

Why does Anthropic require a use case form?

Anthropic requires first-time customers to submit use case details before invoking a Claude model. A short form will appear asking about your intended use. You can describe a learning project or personal portfolio.

Access is typically granted immediately after submitting the form.

  • If prompted to submit use case details for Anthropic, click Submit use case details, fill in the form describing your learning project, and click Submit form.
  • Review the terms and click Submit.
  • Wait for the Access status to change to Access granted for both models. This can take a few minutes.

Still seeing Access status as pending?

Model access changes can take several minutes to reflect in the console. Refresh the page after 2-3 minutes. If you see an error about AWS Marketplace permissions, make sure your account has a valid payment method configured.

Ask for help: Why can't I enable Bedrock model access?

Create an IAM user with the necessary permissions

Your pipeline needs permission to interact with multiple AWS services. You will create a dedicated IAM user with access to S3, Lambda, Textract, Bedrock, and CloudWatch Logs. Using a dedicated user follows the principle of least privilege and keeps your root account secure.

  • In the AWS Console search bar, type IAM and select IAM from the results.
  • In the left navigation pane, click Users.
  • Click Create user.
  • For User name, enter doc-intel-user.
  • Click Next.
  • For Permission options, choose Attach policies directly.
  • In the search box, search for and select each of the following managed policies (search for each one individually):
  • AmazonS3FullAccess
  • AWSLambda_FullAccess
  • AmazonTextractFullAccess
  • AmazonBedrockFullAccess
  • CloudWatchLogsFullAccess
  • Click Next, review the summary, and click Create user.

Why so many policies?

Each policy grants access to a specific AWS service your pipeline needs. S3 stores your documents. Lambda runs your processing code. Textract extracts text from PDFs. Bedrock powers the RAG knowledge base. CloudWatch Logs captures Lambda execution logs for debugging.

In production, you would create a custom policy with narrower permissions. For this learning project, managed policies keep setup simple.

Now create access keys so the AWS CLI can authenticate as this user.

  • Click on your new doc-intel-user in the Users list.
  • Select the Security credentials tab.
  • Under Access keys, click Create access key.
  • For the use case, select Command Line Interface (CLI).
  • Acknowledge the recommendation at the bottom and click Next.
  • Click Create access key.
  • Copy your Access key ID and Secret access key to a safe location (such as a text file). You will not be able to see the secret key again after leaving this page.

Save your secret access key now

The secret access key is shown only once during creation. If you lose it, you must delete the access key and create a new one. Copy it immediately before closing the page.

Now configure the AWS CLI with your new credentials.

  • Return to your terminal and run aws configure again:
aws configure
  • Enter your Access Key ID when prompted (paste the value you just copied).
  • Enter your Secret Access Key when prompted.
  • For Default region name, enter us-east-1.
  • For Default output format, enter json.
  • Verify that your CLI is properly authenticated by running:
aws sts get-caller-identity

What should I see?

You should see JSON output containing your Account ID, User ARN (ending in doc-intel-user), and UserId. This confirms your CLI is authenticated with the correct identity.

✔️ I see my account ID and user ARN

Your AWS CLI is properly configured and authenticated. You are ready to start building the pipeline.

ⓧ I see an error about credentials

This means the access key was not entered correctly. Run aws configure again and carefully paste your Access Key ID and Secret Access Key. Make sure there are no extra spaces.

If you lost your secret key, go back to the IAM console, delete the old access key, and create a new one.

Seeing an InvalidClientTokenId error?

This error means the Access Key ID is incorrect or does not exist. Double check you copied the full key ID from the IAM console. If you accidentally closed the page without copying, delete the access key from the Security credentials tab and create a new one.

Ask for help: Why is my aws sts get-caller-identity failing with InvalidClientTokenId?

Your environment is ready. Next up, you will create the S3 buckets and Lambda function that form the backbone of your document processing pipeline.

Create the Document Processing Infrastructure

Your AWS environment is configured and ready to go. Now you need the core infrastructure that will power the document processing pipeline.

In this step, you will create S3 buckets for storing raw and processed documents, set up an IAM role that grants your Lambda function the permissions it needs, and deploy the Python function that extracts text from PDFs using Amazon Textract.

In this step, get ready to:
  • Create two S3 buckets for raw uploads and processed text output.
  • Create a Lambda execution role with permissions for S3, Textract, and CloudWatch Logs.
  • Deploy a Python Lambda function that extracts text from uploaded PDFs.
Create the S3 buckets

The pipeline uses two buckets to separate concerns. One bucket receives raw PDF uploads and the other stores the extracted text output.

  • First, find your AWS account ID by running this command:
aws sts get-caller-identity --query Account --output text
  • Copy the 12-digit account ID that appears. You will use it in the bucket names to ensure they are globally unique.
  • Create both S3 buckets by running these commands (replace your 12-digit AWS account ID with the account ID you just copied):
aws s3 mb s3://doc-intel-raw-[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]
aws s3 mb s3://doc-intel-processed-[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]

What do these commands do?

Each aws s3 mb command creates a new S3 bucket. The bucket names include your account ID to make them globally unique across all AWS accounts.

The doc-intel-raw bucket will receive PDF uploads. The doc-intel-processed bucket will store the extracted text files after Textract processes each PDF.

  • Verify both buckets were created by running:
aws s3 ls | grep doc-intel
  • You should see both doc-intel-raw- and doc-intel-processed- buckets listed with today's date.

Seeing a BucketAlreadyExists error?

S3 bucket names must be globally unique. Make sure you replaced the placeholder with your actual 12-digit account ID. Double-check for typos or extra spaces in the bucket name.

help me fix my S3 bucket creation error

Create the Lambda execution role

Your Lambda function needs an IAM role that grants it permission to read from S3, call Textract, write results back to S3, and send logs to CloudWatch.

  • Create a file named trust-policy.json with the following content:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

What is a trust policy?

A trust policy defines which AWS service is allowed to assume (use) this role. By specifying lambda.amazonaws.com as the principal, you are telling AWS that only the Lambda service can use this role's permissions.

  • Create the IAM role by running this command:
aws iam create-role \
    --role-name doc-intel-lambda-role \
    --assume-role-policy-document file://trust-policy.json
  • You should see JSON output containing the role's ARN. Copy this ARN (it looks like arn:aws:iam::123456789012:role/doc-intel-lambda-role). You will need it when creating the Lambda function.
  • Attach the AmazonTextractFullAccess managed policy to the role by running:
aws iam attach-role-policy \
    --role-name doc-intel-lambda-role \
    --policy-arn arn:aws:iam::aws:policy/AmazonTextractFullAccess
  • Now create a file named lambda-s3-logs-policy.json with permissions for S3 and CloudWatch Logs:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::doc-intel-raw-*/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::doc-intel-processed-*/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}
  • Attach this inline policy to the role by running:
aws iam put-role-policy \
    --role-name doc-intel-lambda-role \
    --policy-name doc-intel-s3-logs-policy \
    --policy-document file://lambda-s3-logs-policy.json

What does this policy do?

This inline policy grants three specific permissions. s3:GetObject on the raw bucket lets the function read uploaded PDFs. s3:PutObject on the processed bucket lets it write extracted text files.

The CloudWatch Logs permissions (logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents) allow Lambda to write execution logs for debugging.

Getting an access denied error?

Make sure your IAM user has permission to create roles and attach policies. The IAM identity you configured in Step 1 must include iam:CreateRole, iam:AttachRolePolicy, and iam:PutRolePolicy permissions.

help me fix my IAM role creation error

Create the Python Lambda function

The Lambda function is the core of the pipeline. It receives an S3 event notification when a PDF is uploaded, calls Textract to extract text, and writes the result to the processed bucket.

  • Create a file named lambda_function.py with the following code:
import boto3
import time
import urllib.parse

s3_client = boto3.client('s3')
textract_client = boto3.client('textract')

def lambda_handler(event, context):
    # Get the bucket name and object key from the S3 event notification
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])

    # Start async Textract text detection on the uploaded PDF
    response = textract_client.start_document_text_detection(
        DocumentLocation={
            'S3Object': {
                'Bucket': bucket,
                'Name': key
            }
        }
    )
    job_id = response['JobId']

    # Poll for Textract job completion
    status = 'IN_PROGRESS'
    while status == 'IN_PROGRESS':
        time.sleep(5)
        result = textract_client.get_document_text_detection(JobId=job_id)
        status = result['JobStatus']

    if status != 'SUCCEEDED':
        raise Exception(f'Textract job failed with status: {status}')

    # Collect all text blocks across paginated results
    pages = [result]
    next_token = result.get('NextToken')
    while next_token:
        result = textract_client.get_document_text_detection(
            JobId=job_id,
            NextToken=next_token
        )
        pages.append(result)
        next_token = result.get('NextToken')

    # Concatenate all LINE-type blocks into the output text
    extracted_text = ''
    for page in pages:
        for block in page['Blocks']:
            if block['BlockType'] == 'LINE':
                extracted_text += block['Text'] + '\n'

    # Write the extracted text to the processed bucket
    output_bucket = bucket.replace('raw', 'processed')
    output_key = key.rsplit('.', 1)[0] + '.txt'

    s3_client.put_object(
        Bucket=output_bucket,
        Key=output_key,
        Body=extracted_text.encode('utf-8')
    )

    return {
        'statusCode': 200,
        'body': f'Successfully processed {key}'
    }

What does this code do?

  • Extracts the bucket name and object key from the S3 event that triggered the function.
  • Calls start_document_text_detection to begin an async Textract job on the uploaded PDF. The async API supports multi-page PDFs up to 3,000 pages.
  • Polls get_document_text_detection every 5 seconds until the job completes.
  • Collects all LINE-type text blocks from the paginated results and concatenates them into a single string.
  • Writes the extracted text as a .txt file to the processed bucket, using the same key prefix as the original PDF.
  • Package the function into a zip file by running:

macOS

zip function.zip lambda_function.py

Windows

Compress-Archive -Path lambda_function.py -DestinationPath function.zip
  • Deploy the Lambda function by running this command (replace your 12-digit AWS account ID with your account ID):
aws lambda create-function \
    --function-name doc-intel-processor \
    --runtime python3.14 \
    --role arn:aws:iam::[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]:role/doc-intel-lambda-role \
    --handler lambda_function.lambda_handler \
    --zip-file fileb://function.zip \
    --timeout 300

What do these flags mean?

  • --runtime python3.14 sets the Lambda runtime to the latest Python version available.
  • --handler lambda_function.lambda_handler tells Lambda to call the lambda_handler function inside lambda_function.py.
  • --timeout 300 gives the function up to 5 minutes to complete. Textract polling can take 10-30 seconds for small documents.
  • --zip-file fileb://function.zip uploads your packaged code. The fileb:// prefix tells the CLI to send the file as binary data.
  • You should see JSON output with "State": "Active" or "State": "Pending" confirming the function was created. It will become active within a few seconds.

Seeing an InvalidParameterValueException?

This usually means the role ARN is incorrect. Double-check that the role ARN matches what was output when you created the role. It should follow the format arn:aws:iam::ACCOUNT_ID:role/doc-intel-lambda-role.

If you see a message about the role not being assumable, wait 10 seconds and try again. IAM role propagation can take a few moments.

help me fix my Lambda function creation error

✔️ Awesome, I've got everything!

Great. Make sure you have noted the role ARN and your account ID. You will need both in the next step.

ⓧ I'd like to double check the full code

Here is the full lambda_function.py for comparison:

import boto3
import time
import urllib.parse

s3_client = boto3.client('s3')
textract_client = boto3.client('textract')

def lambda_handler(event, context):
    # Get the bucket name and object key from the S3 event notification
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])

    # Start async Textract text detection on the uploaded PDF
    response = textract_client.start_document_text_detection(
        DocumentLocation={
            'S3Object': {
                'Bucket': bucket,
                'Name': key
            }
        }
    )
    job_id = response['JobId']

    # Poll for Textract job completion
    status = 'IN_PROGRESS'
    while status == 'IN_PROGRESS':
        time.sleep(5)
        result = textract_client.get_document_text_detection(JobId=job_id)
        status = result['JobStatus']

    if status != 'SUCCEEDED':
        raise Exception(f'Textract job failed with status: {status}')

    # Collect all text blocks across paginated results
    pages = [result]
    next_token = result.get('NextToken')
    while next_token:
        result = textract_client.get_document_text_detection(
            JobId=job_id,
            NextToken=next_token
        )
        pages.append(result)
        next_token = result.get('NextToken')

    # Concatenate all LINE-type blocks into the output text
    extracted_text = ''
    for page in pages:
        for block in page['Blocks']:
            if block['BlockType'] == 'LINE':
                extracted_text += block['Text'] + '\n'

    # Write the extracted text to the processed bucket
    output_bucket = bucket.replace('raw', 'processed')
    output_key = key.rsplit('.', 1)[0] + '.txt'

    s3_client.put_object(
        Bucket=output_bucket,
        Key=output_key,
        Body=extracted_text.encode('utf-8')
    )

    return {
        'statusCode': 200,
        'body': f'Successfully processed {key}'
    }

Your processing infrastructure is in place. Next up, you will wire the S3 bucket to automatically trigger the Lambda function whenever a PDF is uploaded.

Wire Up the Event-Driven Pipeline

Your Lambda function and S3 buckets are ready, but they do not know about each other yet.

In this step, you will connect the raw uploads bucket to the Lambda function using an S3 event notification. When a PDF lands in the bucket, S3 automatically invokes your function to kick off text extraction.

In this step, get ready to:
  • Configure an S3 event notification to trigger your Lambda function on PDF uploads.
  • Upload a sample PDF and verify the Lambda executes successfully.
  • Confirm the extracted text file appears in your processed bucket.
Configure the S3 event notification

S3 event notifications let a bucket send a message to a destination whenever specific events occur. You will configure your raw bucket to notify your Lambda function every time a .pdf file is created.

Before S3 can invoke the function, you need a resource-based policy granting it permission.

  • Grant S3 permission to invoke your Lambda function by running this command:
aws lambda add-permission \
  --function-name doc-intel-processor \
  --action lambda:InvokeFunction \
  --statement-id s3-trigger \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::doc-intel-raw-[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]

What does this command do?

  • --principal s3.amazonaws.com tells Lambda that the S3 service is allowed to invoke your function.
  • --source-arn restricts the permission so only your specific raw bucket can trigger the function.
  • Without this policy, S3 would receive a permissions error when trying to call the function.
  • You should see a JSON response containing a Statement field confirming the permission was added.

Now configure the bucket to send notifications to your Lambda function whenever a .pdf file is uploaded.

  • Create a file called notification.json with the following content:
{
  "LambdaFunctionConfigurations": [
    {
      "LambdaFunctionArn": "arn:aws:lambda:us-east-1:REPLACE_WITH_ACCOUNT_ID:function:doc-intel-processor",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "suffix",
              "Value": ".pdf"
            }
          ]
        }
      }
    }
  ]
}
  • Replace REPLACE_WITH_ACCOUNT_ID with your 12-digit AWS account ID (the same one in your bucket names).

What does this configuration do?

  • LambdaFunctionArn points to the exact function S3 should invoke.
  • s3:ObjectCreated:* means any method of adding an object (PUT, POST, COPY, multipart upload) fires the notification.
  • The suffix filter .pdf ensures only PDF files trigger the function. Uploading a .txt or .png will not invoke Lambda.
  • Apply the event notification configuration to your bucket by running this command:
aws s3api put-bucket-notification-configuration \
  --bucket doc-intel-raw-[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]] \
  --notification-configuration file://notification.json
  • If the command succeeds, you will see no output. This is normal for S3 API calls that return HTTP 200 without a response body.

Seeing an error about the notification destination?

If you get an error stating S3 is unable to validate the notification destination, the add-permission command may not have run successfully. Re-run it and verify you see the JSON confirmation before retrying.

Also check that the --source-arn in the permission matches your actual bucket name exactly, including the account ID suffix.

Help me fix an S3-to-Lambda permissions error for my event notification.

Upload a sample PDF and verify the Lambda executes

With the event trigger wired up, uploading a PDF to the raw bucket should automatically invoke your Lambda function and start the Textract extraction.

  • If you do not already have a sample PDF, create a simple one-page test document. Any PDF with selectable text from your computer will work.
  • Upload the sample PDF to your raw bucket by running this command:
aws s3 cp [[SAMPLE_PDF_PATH="path/to/your/sample.pdf"]] s3://doc-intel-raw-[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]/sample.pdf

What happens after the upload?

S3 detects the new .pdf object and sends an event notification to your Lambda function. The function starts a Textract job, polls until it completes, then writes the extracted text to your processed bucket.

This entire chain runs automatically. You do not need to invoke the function manually.

  • Wait about 30 to 60 seconds for the Textract job to complete, then verify the Lambda executed by checking CloudWatch Logs:
aws logs filter-log-events \
  --log-group-name /aws/lambda/doc-intel-processor \
  --no-paginate
  • You should see log entries showing the function received the S3 event, started the Textract job, and wrote the output file.

Not seeing any log events?

If the log group does not exist, your Lambda function may not have been invoked. Check that the bucket name in notification.json matches your raw bucket name exactly (including the account ID).

If you see log entries containing errors, check that your Lambda execution role has Textract and S3 permissions and that the processed bucket name in your function code is correct.

Help me debug why my Lambda function is not being triggered by my S3 upload.

Confirm the extracted text file appears in the processed bucket

The final proof that your pipeline works end-to-end is seeing the extracted text in your processed bucket.

  • List the contents of your processed bucket by running this command:
aws s3 ls s3://doc-intel-processed-[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]/
  • You should see a .txt file corresponding to your uploaded PDF (for example, sample.txt).
  • Download and display the extracted text by running this command:
aws s3 cp s3://doc-intel-processed-[[AWS_ACCOUNT_ID="your 12-digit AWS account ID"]]/sample.txt -

What does the dash at the end do?

The - at the end of the aws s3 cp command tells the CLI to print the file contents directly to your terminal instead of saving to disk. This is a quick way to inspect small text files.

  • You should see the extracted text from your PDF printed in the terminal. The content should match the readable text in your original document.

Text file is empty or missing?

If the file exists but is empty, your PDF may be a scanned image without embedded text. Try uploading a PDF that contains selectable text (not a photo scan).

If the file is missing entirely, go back to CloudWatch Logs and look for error messages. Common issues include the Lambda timing out (Textract can take time for large documents) or a permissions error when writing to the processed bucket.

Help me figure out why my extracted text file is missing from the processed bucket.

Your event-driven pipeline is fully operational. Next up, you will create a Bedrock Knowledge Base that ingests these processed text files and makes them queryable using natural language.

Create the Bedrock Knowledge Base

Your document processing pipeline is working. PDFs go in, clean text comes out. Now you need a way to actually search that text using natural language questions.

This is where Retrieval-Augmented Generation (RAG) comes in. A Bedrock Knowledge Base takes your processed text, converts it into vector embeddings, and stores those vectors so you can query them semantically. In this step, you will create a Knowledge Base backed by Amazon S3 Vectors and sync your documents into it.

In this step, get ready to:
  • Create an Amazon Bedrock Knowledge Base with S3 as the data source.
  • Configure S3 Vectors as the vector store with automatic setup.
  • Sync your processed documents into the knowledge base.
Create a new Knowledge Base in the Bedrock console

A Knowledge Base connects your data source to an embedding model and a vector store. Bedrock handles the chunking, embedding, and indexing automatically.

  • Open the Amazon Bedrock console at https://console.aws.amazon.com/bedrock/.
  • Make sure your region is set to US East (N. Virginia) (us-east-1) in the top right corner.
  • In the left navigation pane, choose Knowledge Bases.
  • Choose Create.
  • In the dropdown list, choose Knowledge Base with vector store.

You are now on the Knowledge Base details page.

  • Enter a name for your Knowledge Base, such as doc-intel-kb.
  • Under IAM permissions, select Create and use a new service role. Bedrock will create a role with the permissions it needs to access your S3 data and embedding model.
  • For Choose a data source type, select Amazon S3.
  • Choose Next.

Why does Bedrock need its own IAM role?

The Knowledge Base service role is separate from the Lambda execution role you created earlier. Bedrock needs its own permissions to read objects from your S3 bucket, invoke the embedding model, and write vectors to the vector store. Letting Bedrock create this role automatically ensures the correct trust policy and permissions are in place.

You are now on the Configure data source page.

  • Keep the Data source name at its default or enter a descriptive name like processed-docs-source.
  • In the S3 URI field, choose Browse S3 and select your doc-intel-processed- bucket (the one containing your extracted text files).
  • Leave Parsing strategy and Chunking strategy at their default settings. The default chunking splits your text into appropriately sized segments for embedding.
  • Choose Next.

Why point to the processed bucket, not the raw bucket?

Your raw bucket contains PDF files. The Knowledge Base needs plain text to chunk and embed. Your Lambda function already extracted the text from those PDFs and stored it in the processed bucket. Pointing Bedrock at the processed bucket lets it work with clean, extracted text directly.

Select the embedding model and vector store

Vector embeddings are numerical representations of your text. Similar concepts end up close together in this numerical space, which is what makes semantic search possible.

  • In the Embeddings model section, choose Select model.
  • Select Titan Text Embeddings v2.
  • Expand Additional configurations and confirm Vector dimensions is set to 1024 (the default for this model).

What is Amazon Titan Text Embeddings V2?

Amazon Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0) converts text into 1024-dimensional vectors. Each dimension captures a different aspect of meaning. When you later ask a question, Bedrock converts your question into the same vector space and finds the document chunks closest to it.

The model supports up to 8,192 tokens per input and costs $0.02 per million tokens. For a typical 10-page PDF, the embedding cost is fractions of a cent.

  • In the Vector store section, select Quick create a new vector store.
  • Choose Amazon S3 Vectors as the vector store type.
  • Leave encryption at the default (SSE-S3).
  • Choose Next.

Why S3 Vectors instead of OpenSearch?

Amazon OpenSearch Serverless is the traditional vector database choice for Bedrock Knowledge Bases. However, its minimum cost is approximately $174 per month even in dev mode. S3 Vectors offers up to 90% lower costs with pay-per-query pricing, making it dramatically more affordable for learning and small workloads.

S3 Vectors delivers sub-second cold query latency and approximately 100ms warm latency. For a document Q&A project, this is more than sufficient.

You are now on the review page.

  • Review the configuration. Confirm the data source points to your processed bucket and the embedding model is Titan Text Embeddings v2 with 1024 dimensions.
  • Choose Create Knowledge Base.

The knowledge base creation takes a few minutes. Bedrock is provisioning a new S3 vector bucket and index behind the scenes.

  • Wait until the status shows as Available (or Ready).
  • Note down the Knowledge Base ID displayed at the top of the page. It looks like XXXXXXXXXX (a 10-character alphanumeric string). You will need this ID in the next step.

Knowledge Base creation failed?

If creation fails with a permissions error, confirm that you have Bedrock model access enabled for Titan Text Embeddings V2 in us-east-1. You enabled this in Step 1, but double-check by navigating to Model access in the Bedrock console left sidebar and verifying the model shows Access granted.

If you see an error about S3 bucket access, verify your processed bucket is in the same region (us-east-1) as the Knowledge Base.

help me troubleshoot my Bedrock Knowledge Base creation error

Sync the data source to ingest your documents

Creating the Knowledge Base set up the infrastructure. Now you need to sync the data source so Bedrock actually reads your text files, chunks them, generates embeddings, and stores those embeddings in the vector index.

  • On the Knowledge Base details page, scroll down to the Data source section.
  • Select the checkbox next to your data source.
  • Choose Sync.

The sync processes each text file in your processed bucket. Bedrock splits each file into chunks, sends each chunk through Titan Embeddings V2 to generate a 1024-dimension vector, and stores the result in the S3 Vectors index.

  • Wait for the sync status to show a green success banner. This may take a minute or two depending on the size and number of your documents.
  • Choose the data source name to view its Sync history. Confirm the status shows Completed and the number of files ingested matches what you expect.

What happens during a sync?

Bedrock scans your S3 data source, reads each document, and checks whether it has already been indexed. For new documents, it applies the chunking strategy (default: fixed-size chunks), generates vector embeddings for each chunk using Titan Text Embeddings V2, and writes those vectors to the S3 Vectors index. Future syncs are incremental and only process new or modified files.

Sync shows warnings or zero files ingested?

If zero files were ingested, confirm that your processed bucket (doc-intel-processed-) actually contains .txt files from your Lambda processing. You can verify this by running:

aws s3 ls s3://doc-intel-processed-[[ACCOUNTID="your AWS account ID"]]/

If you see your text files listed but the sync still shows zero files, check that the S3 URI in the data source configuration matches your bucket name exactly.

help me figure out why my Bedrock Knowledge Base sync did not ingest any files

Your document corpus is now embedded and indexed. In the next step, you will query the Knowledge Base and see RAG in action.

Query Your Document Corpus with RAG

Your knowledge base is synced and ready to answer questions. The documents you uploaded have been chunked, embedded, and indexed in S3 Vectors.

Now you will put the full pipeline to the test by asking natural-language questions and verifying that the answers come grounded in your actual documents. You will query through the console first, then programmatically via the AWS CLI, and finally upload new documents to confirm the pipeline handles fresh content end to end.

In this step, get ready to:
  • Test the knowledge base in the Bedrock console test panel.
  • Query the knowledge base programmatically using the AWS CLI.
  • Upload additional documents and verify new content is retrievable.
Test in the Bedrock console

The Bedrock console includes a built-in test panel that lets you chat with your knowledge base without writing any code. This is the fastest way to confirm that RAG retrieval is working correctly.

  • Return to the Amazon Bedrock console at https://console.aws.amazon.com/bedrock.
  • In the left navigation pane, click Knowledge bases.
  • Click the name of your knowledge base to open its details page.
  • A test window expands from the right side of the page.
  • Turn on Generate responses so the knowledge base generates full answers (not just raw chunks).
  • Click Select model, choose an Anthropic Claude model, then click Apply.
  • Type a question about the content of your uploaded PDF in the text box. For example, ask about a specific topic or fact you know is in the document.
  • Click Run.

You should see a generated answer with footnote citations. Each footnote links back to the source chunk from your processed text file in S3.

What are citations in RAG responses?

When Bedrock generates an answer, it includes footnotes that point to the exact text chunks it used. Click a footnote to see the excerpt from your source document.

This grounding mechanism is what makes RAG trustworthy. You can always trace an answer back to its source material.

  • Click a footnote number in the response to verify it references text from your uploaded document.
  • Try 2-3 different questions to confirm the knowledge base retrieves relevant content consistently.

Not seeing any results?

Make sure Generate responses is toggled on. Without it, the panel only shows raw retrieved chunks without a generated answer.

If the response says it cannot find relevant information, verify that the data source sync completed successfully in the Data source section of your knowledge base page.

Why isn't my knowledge base returning answers about my document?

Query programmatically with the AWS CLI

The console test panel is useful for quick checks. For automation and integration, you query the knowledge base using the bedrock-agent-runtime CLI commands.

  • Query your knowledge base programmatically by running this command:
aws bedrock-agent-runtime retrieve-and-generate \
  --region us-east-1 \
  --input '{"text": "What are the main topics covered in the document?"}' \
  --retrieve-and-generate-configuration '{"type": "KNOWLEDGE_BASE", "knowledgeBaseConfiguration": {"knowledgeBaseId": "[[KB_ID=\"your-knowledge-base-id\"]]", "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0"}}'

What does this command do?

The retrieve-and-generate command sends your question to the knowledge base. Bedrock retrieves the most relevant text chunks from your vector store, then passes them to Claude as context to generate a grounded answer.

The --input flag contains your question. The --retrieve-and-generate-configuration flag tells Bedrock which knowledge base to search and which model to use for generation.

You should see a JSON response containing an output field with the generated answer and a citations array showing which source chunks were used.

  • Try a second question to confirm consistent retrieval by running:
aws bedrock-agent-runtime retrieve-and-generate \
  --region us-east-1 \
  --input '{"text": "Summarize the key findings in the document."}' \
  --retrieve-and-generate-configuration '{"type": "KNOWLEDGE_BASE", "knowledgeBaseConfiguration": {"knowledgeBaseId": "[[KB_ID=\"your-knowledge-base-id\"]]", "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0"}}'

You should see another JSON response with a generated summary and citations pointing to relevant chunks from your document.

Getting an access denied error?

Your IAM user or role needs the bedrock:RetrieveAndGenerate permission. If you used an admin-level IAM user during setup, this should already be in place.

Also confirm that model access is enabled for the Claude model you specified in the modelArn field. You enabled this in Step 1.

Help me fix an access denied error when calling retrieve-and-generate

Upload additional documents and verify retrieval

A real pipeline needs to handle new documents continuously. You will upload another PDF, let the Lambda process it, re-sync the knowledge base, and confirm the new content appears in query results.

  • Upload a second PDF document to your raw bucket by running:
aws s3 cp [[SECOND_PDF="path/to/your/second-document.pdf"]] s3://[[RAW_BUCKET="doc-intel-raw-your-account-id"]]/second-document.pdf

What happens when you upload?

The S3 event notification fires, triggering your Lambda function. Lambda calls Textract to extract text from the new PDF and writes the result to your processed bucket. This is the same automated pipeline you verified in Step 3.

  • Wait 1-2 minutes for the Lambda function to finish processing.
  • Confirm the extracted text appeared in the processed bucket by running:
aws s3 ls s3://[[PROCESSED_BUCKET="doc-intel-processed-your-account-id"]]/

You should see a new .txt file corresponding to your second PDF.

  • Re-sync the knowledge base data source so Bedrock indexes the new document by running:
aws bedrock-agent start-ingestion-job \
  --knowledge-base-id [[KB_ID="your-knowledge-base-id"]] \
  --data-source-id [[DS_ID="your-data-source-id"]]

What does syncing do?

Syncing is incremental. Bedrock only processes documents that were added, modified, or deleted since the last sync. It parses the new text, generates vector embeddings with Titan Text Embeddings V2, and indexes them in your S3 Vectors store.

  • Wait for the ingestion job to complete. The status field in the response will show STARTING initially. Check the status by running:
aws bedrock-agent list-ingestion-jobs \
  --knowledge-base-id [[KB_ID="your-knowledge-base-id"]] \
  --data-source-id [[DS_ID="your-data-source-id"]]

When status shows COMPLETE, the new document has been indexed and is queryable.

  • Query the knowledge base about content from your second document by running:
aws bedrock-agent-runtime retrieve-and-generate \
  --region us-east-1 \
  --input '{"text": "What information is in the second document I uploaded?"}' \
  --retrieve-and-generate-configuration '{"type": "KNOWLEDGE_BASE", "knowledgeBaseConfiguration": {"knowledgeBaseId": "[[KB_ID=\"your-knowledge-base-id\"]]", "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0"}}'

You should see a response that references content from your newly uploaded document, proving the full end-to-end pipeline works.

New document not showing up in results?

Confirm the ingestion job status is COMPLETE before querying. If it still shows IN_PROGRESS, wait another minute and check again.

If the job completed but your query returns nothing from the new document, check CloudWatch Logs for the Lambda function to confirm the text extraction succeeded for the second PDF.

My new document isn't appearing in knowledge base query results after syncing

Your document intelligence pipeline is fully validated. Documents uploaded to S3 are automatically processed by Textract, and the knowledge base serves grounded answers powered by Claude. Next up, you will extend this with a terminal chat interface in the Secret Mission.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses serverless AWS services that incur minimal costs when idle, but Bedrock inference and S3 storage will generate small ongoing charges.

Cost warning

Bedrock model inference (embeddings and Claude) charges per token each time you query the knowledge base. S3 Vectors charges per query and for storage. If you leave the pipeline active and continue querying, expect costs of a few cents per query.

If you are done experimenting, choose Pause or Delete below to avoid unexpected charges.

Resources you used:

  • S3 bucket doc-intel-raw-{accountid} (raw PDF uploads).
  • S3 bucket doc-intel-processed-{accountid} (extracted text output).
  • S3 Vectors bucket (auto-created by Bedrock for vector embeddings).
  • Lambda function (Textract document processor).
  • IAM role (Lambda execution role with S3, Textract, and CloudWatch permissions).
  • Amazon Bedrock Knowledge Base (with S3 data source and S3 Vectors store).
  • Local file doc_chat.py (Secret Mission CLI tool).

Keep everything running

No action needed. Choose this if you are still actively experimenting with the pipeline or plan to upload more documents soon.

  • The Lambda function and S3 event trigger remain dormant until you upload a new PDF. Lambda does not charge when idle.
  • S3 storage costs are negligible for a small number of documents (fractions of a cent per month).
  • S3 Vectors charges only when you run queries. No queries means no vector-store cost.
  • Total idle cost for a small dataset is less than $1 per month.
  • AWS CLI remains installed on your machine.

Pause - I'll come back to this later

Shut down running processes to free up resources, but keep your files and configuration intact for later.

  • No action is required for serverless resources. Lambda, S3, and Bedrock Knowledge Bases do not charge when idle (other than minimal S3 storage).
  • You can resume at any time by uploading a new PDF or querying the knowledge base.
  • Your doc_chat.py script will work immediately when you return.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow these steps in order.

  • Delete the Bedrock Knowledge Base by running:
aws bedrock-agent delete-knowledge-base --knowledge-base-id [[KB_ID="your knowledge base ID"]]
  • Empty and delete the raw uploads bucket by running:
aws s3 rb s3://[[RAW_BUCKET="doc-intel-raw-youraccountid"]] --force
  • Empty and delete the processed text bucket by running:
aws s3 rb s3://[[PROCESSED_BUCKET="doc-intel-processed-youraccountid"]] --force
  • Empty and delete the S3 Vectors bucket (auto-created by Bedrock). Find its name in the S3 console, then run:
aws s3 rb s3://[[VECTORS_BUCKET="your S3 Vectors bucket name"]] --force
  • Delete the Lambda function by running:
aws lambda delete-function --function-name [[FUNCTION_NAME="your Lambda function name"]]
  • Detach all managed policies from the Lambda execution role, then delete the role. Replace the role name and policy ARN with your values:
aws iam detach-role-policy --role-name [[ROLE_NAME="your Lambda execution role name"]] --policy-arn [[POLICY_ARN="arn:aws:iam::aws:policy/PolicyName"]]
aws iam delete-role --role-name [[ROLE_NAME="your Lambda execution role name"]]

What if the role has multiple policies?

Run the detach-role-policy command once for each managed policy attached to the role. You may have attached policies for S3 access, Textract, and CloudWatch Logs. Repeat the detach command with each policy ARN before running delete-role.

  • Delete the local CLI script by running:
rm doc_chat.py

That's a Wrap!

That's a Wrap!

Nice work! You've just built a production-pattern document intelligence pipeline on AWS that automatically processes PDFs and makes them queryable through natural language.

You've learned how to:

  • Build an event-driven pipeline that chains S3 upload notifications to a Lambda function calling Amazon Textract for automated text extraction.
  • Create a Bedrock Knowledge Base backed by Amazon S3 Vectors for cost-effective vector storage and semantic search over your document corpus.
  • Query your documents using Retrieval-Augmented Generation to get grounded, cited answers powered by Claude on Bedrock.
  • Built a terminal chat tool that uploads documents and lets you have a conversational Q&A session with your corpus from the command line.

Ready to quiz yourself?