Fetch Data with AWS Lambda

Learn how to retrieve DynamoDB table data using AWS Lambda function!

Introduction

⚡️ 30 second Summary

Welcome to this project on Fetching Data with AWS Lambda! 🌟

Imagine building applications that can grow effortlessly to handle thousands or millions of users without managing a single server. That's the power of serverless computing!

This project is your introduction to serverless and you will learn how to use a serverless AWS Lambda function to retrieve data from a DynamoDB table.

This is perfect for roles such as Backend Developers, Cloud Engineers, and Cloud Architects-where it is a key responsibility to create scalable cloud solutions that handle large amounts of data efficiently.

In this project, get ready to...
  • 🗄️ Create a database table to store user data.
  • 🐑 Create a serverless function to retrieve user data.
  • 📝 Write tests to validate if your function can fetch data from DynamoDB.
  • 🔑 Secure your serverless function with proper permissions.
  • 💎 Secure your database with an inline policy
Three-Tier Architecture

This is also the THIRD project of an awesome three-tier architecture series. You can do this project standalone, or as a part of this series.

Three-tier architecture splits applications into three essential layers: presentation, logic, and data.

In this project, you're diving into the data tier, which is all about how you store and manage the data relevant to your application.

We'll talk more about this architecture in later projects - for now, let's focus on Lambda and DynamoDB. Make sure to follow the rest of this series to build up a fully functional three-tier solution! 🚀

Want a complete demo of how to do this project, from start to finish? Check out our 🎬 walkthrough with Natasha 🎬

If you're up for a bit of a challenge, quiz yourself on the key concepts up ahead in this project.

This project is part of a series:

  1. Part 1: Website Delivery with CloudFront
  2. Part 2: APIs with Lambda + API Gateway
  3. Part 3: You are here!

Before we start Step #1...

It's always good to understand exactly what you're here to do.

Set up DynamoDB

In this step, we'll create a DynamoDB table to store some user data.

In this step, get ready to:

  • Create a DynamoDB table.

Head to the DynamoDB console

  • Log in to the AWS Management Console as your IAM Admin user.
  • Make sure you're in the AWS region closest to you.
  • Head to the DynamoDB console.

What is DynamoDB?

DynamoDB is one of AWS's database services. It stands out as a fast and flexible way to store data, which makes it a great choice for apps that need quick access to large volumes of data e.g. games.

If you enjoy using DynamoDB in this project, make sure to check out the Load Data into DynamoDB project!

Create the DynamoDB table

  • Select Create table.
  • For Table name, enter UserData.
  • For Partition key, enter userId.

What is a partition key?

A partition key is the heart of how DynamoDB organizes data. Think of it as a label that you can use to group similar items. Under the hood, the partition key is how DynamoDB spreads out your data across different servers for quick access and efficient querying.

Every item in your table must have a unique partition key.

  • Select String as the data type for the partition key.
  • Leave the default settings for the rest of the options.
  • Select Create table.

Great stuff! You've taken the first big step in this project and set up your DynamoDB table.

Add a Table Item

Now that we have our DynamoDB table set up, let's add some sample data so we can see our Lambda function in action later.

In this step, get ready to:

  • Add a sample user to your UserData table.

Add a table item

  • Once the table status changes to Active, select your UserData table.
  • Select Explore table items.
  • At the Items returned panel, select Create item.
  • Select Switch to JSON view.

Why is there a JSON view in DynamoDB?

Under the hood, DynamoDB stores your data in JSON! By switching to JSON view, you can edit your data in a code format instead of filling out a form.

This is also a great way to save time - if you have data with lots of attributes, you wouldn't want to fill each of them out one by one for every item.

  • Toggle off View DynamoDB JSON.

What is DynamoDB JSON?

DynamoDB JSON is a special flavor of JSON used by DynamoDB that's a bit more specific. It tells your database exactly what type of data each piece is (e.g. a string or a number) on top of storing the data's values.

We won't be writing in DynamoDB JSON so we can stick with the simpler JSON format, so we can turn this off.

  • Paste the following JSON into the editor:
{
  "userId": "1",
  "name": "Test User",
  "email": "test@example.com"
}

What does this JSON code say?

This JSON code defines a new item for our UserData table.

This item represents a user with...

  • A userId of 1
  • A name of Test User, and
  • An email of test@example.com

Notice how we didn't have to add name or email as new attributes (i.e. table fields) beforehand!

DynamoDB is schemaless, meaning you can add attributes as you need, and every item in your database can have a different set of attributes. This flexibility is one of the key benefits of using a NoSQL database like DynamoDB.

  • Select Create item.
  • Verify that the item was created successfully. You should see it listed in the Items returned tab.

Phew! That's a piece of data in our DyanmoDB table now. Awesome to have some sample data we can use for the rest of this project.

Create the Lambda Function

Time for Lambda to enter the picture!

Now that our DynamoDB table and data are set up, let's set up a function that can retrieve table items on demand.

In this step, get ready to:

  • Set up a Lambda function.

Create the function

  • Navigate to the Lambda service in the AWS Management Console.

What is AWS Lambda?

AWS Lambda is a service that lets you run code without needing to manage any computers/servers - Lambda will manage them for you.

Lambda runs your code only when you need it to (so you're not paying for any idle time).

It also scales automatically, from a few requests per day to thousands per second - all you need to do is supply your code in one of the languages that Lambda supports.

  • Select Create function.

What is a Lambda function?

A Lambda function is a piece of code that you run in AWS Lambda.

You write Lambda functions to do things like process data, respond to events, or run automated tasks.

  • Select Author from scratch.

There are other ways to create a Lambda function?

Yup! We'll create ours from scratch (i.e. we're writing the function code ourselves), but Lambda functions can be created from blueprints, cloned from existing functions in your account, imported from a ZIP file, or deployed through a repository to jumpstart development.

  • For Function name, enter RetrieveUserData.
  • For Runtime, choose a Node.js runtime.

What does runtime mean?

The runtime is like the environment where your code lives and runs.

While you write code in a programming language, the runtime provides the actual environment where that code runs. For example, Python code needs a Python runtime environment.

Selecting the right runtime for your AWS Lambda function makes sure it has everything (i.e. dependencies) needed to run efficiently.

💡 Extra for Experts: What is Node.js?

Node.js is a popular JavaScript runtime environment.

Traditionally, JavaScript is used for scripts that run right in your web browser, handling tasks like animations and form validations. This browser-based setup means all the JavaScript action happens on your device after the web page loads.

But with Node.js, you can take JavaScript to the server side. Now, you're running scripts on a server before anything even reaches the browser.

This is a big deal because it lets developers use JavaScript for both frontend and backend tasks, which makes development much more efficient. You can use Node.js to build things like server-side apps, command-line tools, or even chat with databases using JavaScript.

In our case, we're using Node.js to create the backend for a web app.

  • Expand the Change default execution role arrow.
  • Keep Create a new role with basic Lambda permissions.

What's an execution role?

An execution role is an IAM role for your Lambda function. It defines what the function is allowed to do, e.g. accessing other AWS services like DynamoDB.

By default, AWS creates a role for Lambda with basic permissions for writing logs to CloudWatch. That's why you could see an error message when you tested your Lambda function!

💡 Why do we need an execution role?

Security! We don't want our Lambda function to have unlimited access to everything in our AWS account.

The execution role makes sure the function only has the permissions it needs to do its job, minimizing the potential for security vulnerabilities.

  • Note the message at the bottom that says "Lambda will create an execution role named RetrieveUserData-role-9w66kle5".
  • Select Create function.

Easy as! Is your Lambda setup all done?

Well, although we've created a function, it's not going to do anything until we write its code. Let's do that next.

Implement the Lambda Function Logic

Let's get coding! We'll write the code that retrieves user data from our DynamoDB table.

In this step, get ready to:

  • Add code to your Lambda function to retrieve data from DynamoDB.
  • Deploy the function.

Add the code

  • In the Lambda code editor, replace the existing code with the following:
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, GetCommand } from "@aws-sdk/lib-dynamodb";

const ddbClient = new DynamoDBClient({ region: '[[YOURREGION="enter your region"]]' }); // Make sure to replace this with your actual AWS region
const ddb = DynamoDBDocumentClient.from(ddbClient);

async function handler(event) {
const userId = String(event.userId);  // Make sure to extract userId from the event
const params = {
        TableName: 'UserData',
        Key: { userId }
    };

    try {
        const command = new GetCommand(params);
        const data = await ddb.send(command);  // Log the raw response from DynamoDB
        console.log("DynamoDB Response:", JSON.stringify(data));
        const { Item } = data;
        if (Item) {
            console.log("User data retrieved:", Item);
            return Item;
        } else {
            console.log("No user data found for userId:", userId);
            return null;
        }
    } catch (err) {
        console.error("Unable to retrieve data:", err);
        return err;
    }
}

export { handler };

What's this code doing?

The first half of the code uses the AWS SDK for JavaScript to interact with DynamoDB.

It takes a userId as input, grabs the corresponding data from the UserData table, and returns it to you.

The second half of the code (i.e. beginning with try {) handles potential errors during the database operation, so you get a tailored error message that tells you what went wrong.

💡 What is AWS SDK?

The AWS Software Development Kit (SDK) is a set of tools that let developers build apps that interact with AWS.

Think of it like a library of pre-written code that lets you use AWS services without having to write all the code yourself. It gives you functions and classes that simplify common tasks, like creating an S3 bucket or launching an EC2 instance.

There's a different SDK for different programming languages or platforms, and JavaScript is one of them! Your code uses the AWS SDK to use pre-written functions for communicating with DynamoDB and getting data from a table. Without the SDK, you'd have to manually write the code to interact with AWS, which would be much more complex and error-prone.

  • Make sure you've replaced YOUR-REGION with your actual AWS region code (e.g., us-east-1).
  • Select Deploy.
  • We'll wait until the deployment is successful...
  • Check: Is your deployment a success?

Great! You've just written and deployed the Lambda function's code. It should be working now, let's give it a test.

Write a Lambda Function Test

Let's test our Lambda function to make sure it's working. Lucky for us, AWS Lambda comes with a handy testing feature we can use.

In this step, get ready to:

  • Create a test event for your Lambda function.
  • Run the test and observe the results.

Create and run the test

  • Still in your Lambda function, select the Test tab.

What does the Test tab do?

The Test tab is your playground for running your Lambda function.

In a function test, you can send fake events to your function to see how it would react.

In our case, we're writing tests that lets us see how our function would react if we made a data request.

  • In the Event JSON panel, enter the following JSON:
{
  "userId": "1"
}

What will this test do?

In this test, we're asking our Lambda function to search for an item in our DynamDB table. The item needs to have a userID of 1

💡 Why are we entering the test in JSON?

By using JSON to input test data, we ensure it's in a format that’s easy for Lambda to understand and work with.

  • Select Test.
  • Yay! Looks like we've got a successful test?
  • Expand the Details arrow for your successful test.
  • Nevermind, we have an error!

I don't see the same error

Did you get an ENOTFOUND error instead?

Check whether you've correctly replaced YOUR-REGION with your actual AWS region (e.g., us-east-1) in the function code.

  • Expand the full error message.
  • Can you tell why we've run into an error?

Why did you get this error?

Even though we created an execution role for our Lambda function, we haven't given it explicit permission to access our DynamoDB table. This means DynamoDB is currently blocking off our Lambda function from reading the table's items!

Because Lambda can't read any items, it has no choice but to tell us that its access was denied.

💡 Why did the test still show as a success, when your access was actually denied?

The "success" message simply means the function itself could run (there are no errors with the code), it doesn't mean the function achieved what you want it to do!

Grant DynamoDB Access to Lambda

Let's fix that permissions issue! We'll give our Lambda function the necessary permissions to read data from our DynamoDB table.

In this step, get ready to:

  • Grant your Lambda function read access to DynamoDB.

Grant DynamoDB access to Lambda

  • Switch to the Configuration tab in your Lambda function.
  • Select Permissions.

What is in the Permissions tab?

The Permissions tab shows the execution role linked with your Lambda function and the permissions given to that role. It's where you control what your function is allowed to access.

  • Select the execution role name (it will look something like RetrieveUserData-role-xxxxxxxx).
  • This shortcut will take you to the IAM console, with your Lambda function readily open.
  • Select Add permissions.
  • Select Attach policies.
  • Type DynamoDB in the search bar.

Ooo, we have four different permission policies that grants access to DynamoDB.

  • To figure out which permission policy to apply, first it's best to head back to the error message.
  • What was the specific action that was denied?
  • Aha, it was GetItem!

Why are we reviewing the error message again?

Reviewing the error message tells us exactly which permissions the Lambda function lacks.

To solve the access denied error, we can add a permission policy that give us the permissions we're lacking!

  • Head back to the policy set up page.
  • Expand the bottom two permission policies - can you spot the GetItem permission?

Nope... What do "AWSLambdaDynamoDBExecutionRole" and "AWSLambdaInvocation-DynamoDB" do?

  • AWSLambdaDynamoDBExecutionRole gives your Lambda function the permissions to see a DynamoDB stream i.e. a live news feed of changes to your tables (like new, updated, or deleted items) in the last 24 hours.
  • AWSLambdaInvocation-DynamoDB is used to automatically trigger your Lambda functions in response to events captured in DynamoDB stream. You'd use this in apps where you need immediate action based on data changes - for example, if a user should get a notification when they add a new product to their in-app shopping cart.
  • Now expand AmazonDynamoDBReadOnlyAccess.
  • Can you spot GetItem in the policy?
  • Yess, we can!
  • Select AmazonDynamoDBReadOnlyAccess as the permission policy we'll use.

What about "AmazonDynamoDBFullAccess"?

AmazonDynamoDBFullAccess lets you do everything with DynamoDB, like creating, deleting, and managing tables. It also lets you read and write data.

But, if your Lambda function only needs to read data, you don't need this level of access. Granting this permission when you don't need it can make your resources less secure. For example, if someone gets unauthorized access to your function, they could manipulate or delete crucial data by editing your function, which could disrupt your service or lead to data loss.

  • Select Add permissions.

Yay! Permissions added. This means your Lambda function should be able to read DynamoDB table items.

Re-test Your Lambda Function

Let's re-run our test and make sure our Lambda function now has the necessary permissions to access DynamoDB.

In this step, get ready to:

  • Re-run the Lambda function test.
  • Validate that your Lambda function now works!

Re-run the test

  • Head back to the Test tab in your Lambda function.
  • Select Test.

You should now see a successful test execution without the AccessDeniedException error.

Yay! The response should show the user data we added in our DynamoDB table.

What are some uses cases of what I've built today?

Lambda can be a serverless backend for web apps, grabbing data from DynamoDB when a user logs in or interacts with your app. For example...

  1. Lambda can help customers find products, get product information or see their order history by fetching data from DynamoDB.
  2. Lambda can help social media apps fetch user profiles or automatically retrieve all content (e.g. videos or images) linked with a profile.
  3. Lambda can help news sites or blogs fetch articles based on user queries.

Secret mission

Welcome to your 🤫 exclusive 🤫 secret mission!

Your mission, should you choose to accept it, is to update your Lambda function's permissions to better secure your database.

💎 In this secret mission, get ready to:

  • Update your Lambda function's permission policies.
  • Validate that your Lambda function still works.
  • Showcase your secret mission in your project documentation.

Tighten DynamoDB Security

Delete your resources

Delete your resources

Time to clean up our AWS resources to avoid unnecessary costs.

Resources to delete

  • Your Lambda function.
  • Your DynamoDB table.

🚨 Steps below

  • Delete the Lambda function
  • Head to the Lambda service.
  • Select the RetrieveUserData function.
  • Select the Actions dropdown.
  • Select Delete.
  • Type confirm to confirm the deletion.
  • Select Delete.
  • Delete the DynamoDB table
  • Head to the DynamoDB service.
  • From the left hand navigation bar, select Tables.
  • Select the UserData table.
  • Select Delete.

What are these warnings about?

  • Delete all CloudWatch alarms: Keep this checked ✅ Deleting any CloudWatch alarms associated with this table will save you from unnecessary charges.
  • Create an on-demand backup: Keep this unchecked ❌ You can create a backup of your table to save for the long term, so you restore your data to its exact state before deleting it. Additional charges apply for on-demand backup and restore, so we won't use this option.
  • Type confirm to confirm the deletion.
  • Select Delete.

That's a wrap!

That's a wrap!

High five! You've built a Lambda function that retrieves data from a database – no servers required (how cool).

You've just learned how to:

  • 🗄️ Set up a DynamoDB database.
  • 🐑 Create and configure an AWS Lambda function.
  • 💻 Write code to interact with DynamoDB using the AWS SDK.
  • 🧪 Test your Lambda function.
  • 💎 Tighten permission settings for your Lambda function.

Ready to quiz yourself? You got this! 💪

Are you ready to tie everything together in the last project of this series?!

See you there: Build a Three-Tier Web App

p.s. Does it say "Still tasks to complete!" at the bottom of the screen?

This means you still have screenshots left to upload, or questions left to answer!

  1. Press Ctrl+F (Windows) or Command+F (Mac) on your keyboard.
  2. Search for the text Return to later.
  3. Jump straight to your incomplete tasks!
  4. 🙋‍♀️ Still stuck? Ask the community!