Automate S3 Storage with Lifecycle Policies
Build S3 lifecycle rules that automatically tier and expire objects via CLI.
Introduction
30 Second Summary
Every organization stores data that becomes less useful over time. Last month's log files and outdated reports still cost the same to store as the files you access every day.
In this project, you will use the AWS CLI to create an S3 bucket, upload objects to different storage classes, and build a lifecycle policy that automatically moves data to cheaper storage over time.
What You'll Build
You will build a multi-tier S3 lifecycle configuration that transitions objects through progressively cheaper storage classes and expires them automatically.
By the end of this project, you'll have:
- An S3 bucket with objects stored in multiple storage classes, uploaded and verified entirely through CLI commands.
- A JSON lifecycle configuration with transition and expiration rules that automate storage cost optimization.
- A clear mental model of S3 storage class constraints that appear on the AWS Solutions Architect Associate exam.
- Secret Mission: Design a multi-prefix lifecycle strategy for different departments with varying retention needs.
Are there any prerequisites?
You need an AWS account with an IAM user already created. No prior CLI experience is required.
Install and Configure the AWS CLI
To manage S3 lifecycle policies from the command line, you need the AWS CLI installed and connected to your AWS account.
The AWS CLI lets you run commands that create buckets, upload objects, and configure lifecycle rules without touching the console. In this step, you will install the CLI, create an access key, and connect everything together.
In this step, get ready to:
- Install AWS CLI v2 on Windows.
- Create an access key for your IAM user.
- Configure the AWS CLI with your credentials and verify it works.
Install AWS CLI v2 on Windows
The AWS CLI v2 is a command-line tool that lets you interact with AWS services directly from your terminal. You need version 2 for this project.
- Check whether you already have the AWS CLI installed by pressing the Windows key on your keyboard to open the search bar.
- Type Command Prompt and press Enter to open it.
- Run the following version check command:
aws --version
Check your output and pick the tab that matches what you see.
✔️ I see aws-cli/2.x
Your output starts with aws-cli/2. followed by a version number. You already have AWS CLI v2 installed and ready to go.
Skip ahead to the next substep.
ⓧ I see aws-cli/1.x
Your output starts with aws-cli/1. which means you have an older version installed. You need to uninstall v1 first, then install v2.
- Press the Windows key and type Add or remove programs.
- Press Enter to open the settings page.
- Search for AWS CLI in the app list.
- Click Uninstall and follow the prompts to remove it.
- Close your Command Prompt window and reopen it.
- Download the AWS CLI v2 installer from this link:
https://awscli.amazonaws.com/AWSCLIV2.msi
- Double-click the downloaded AWSCLIV2.msi file and follow the installation wizard. Accept the default options.
- Close your Command Prompt window and open a new one.
- Run aws --version again. You should now see output starting with aws-cli/2. confirming v2 is installed.
ⓧ Command not found
The AWS CLI is not installed yet. Let's install it now.
- Download the AWS CLI v2 installer from this link:
https://awscli.amazonaws.com/AWSCLIV2.msi
- Double-click the downloaded AWSCLIV2.msi file and follow the installation wizard. Accept the default options.
- Close your Command Prompt window and open a new one.
- Run aws --version again. You should now see output starting with aws-cli/2. confirming v2 is installed.
Still seeing command not found?
You may need to close and reopen your Command Prompt window to refresh the system PATH. If it still does not work, try restarting your computer.
help me troubleshoot why aws --version is not recognized after installing the AWS CLI v2 MSI on Windows
Create an access key for your IAM user
The AWS CLI needs credentials to prove who you are when it talks to AWS. You will create an access key in the AWS Management Console, then use it to configure the CLI.
- Open your browser and sign into the AWS Management Console at https://console.aws.amazon.com using your IAM user credentials.
- In the search bar at the top, type IAM and click the IAM service result.
- In the left sidebar, click Users.
- Click your username in the list.
- Click the Security credentials tab.
- Scroll down to the Access keys section and click Create access key.
- Select Command Line Interface (CLI) as the use case.
- Check the confirmation box at the bottom and click Next.
- Click Create access key.
Copy your keys now
You will see your Access Key ID and Secret Access Key on this page. Copy both values and paste them somewhere safe (a text file or password manager). You will not be able to see the Secret Access Key again after you leave this page.
Do not share these keys with anyone. They grant full access to your AWS account.
- Click Done once you have saved both keys.
What is an access key?
An access key is a pair of values (an Access Key ID and a Secret Access Key) that act as your identity when making API calls from outside the console. The CLI sends these with every request so AWS knows which user is asking.
Configure the AWS CLI with your credentials
Now you will connect the CLI to your AWS account by entering the access key you just created.
- Switch back to your Command Prompt window and run the configuration command:
aws configure
What will this command ask me?
The aws configure command prompts you for four values, one at a time. Enter each value and press Enter to move to the next prompt.
- AWS Access Key ID: paste the Access Key ID you copied.
- AWS Secret Access Key: paste the Secret Access Key you copied.
- Default region name: enter us-east-1 (or your preferred region).
- Default output format: enter json.
- Enter your Access Key ID when prompted and press Enter.
- Enter your Secret Access Key when prompted and press Enter.
- Enter us-east-1 for the default region and press Enter.
- Enter json for the output format and press Enter.
- Verify that your CLI is authenticated by running:
aws sts get-caller-identity
You should see a JSON response containing your IAM user ARN, Account ID, and User ID. This confirms the CLI is talking to AWS as your IAM user.
What does this output mean?
The aws sts get-caller-identity command asks AWS "who am I?" and returns three fields:
- UserId: a unique identifier for your IAM user.
- Account: your 12-digit AWS account number.
- Arn: the full Amazon Resource Name of your user, confirming exactly which identity the CLI is using.
Seeing an error instead of JSON?
If you see a message about invalid credentials, double-check that you pasted the correct Access Key ID and Secret Access Key. Run aws configure again to re-enter them.
If you see a network-related error, check your internet connection and try again.
help me fix the error I'm seeing when I run aws sts get-caller-identity after configuring my AWS CLI credentials
Your CLI is authenticated and ready. Next up, you will create an S3 bucket and start uploading objects to different storage classes.
Create a Bucket and Upload Objects
Your CLI is ready to talk to AWS. Now it is time to create your first S3 bucket and fill it with objects stored in different storage classes.
S3 offers multiple storage classes, each designed for a different access pattern. By uploading objects to specific classes, you control how much you pay for storage versus retrieval. This is the foundation of the cost optimization strategy you will automate with lifecycle policies later.
In this step, get ready to:
- Create an S3 bucket using the AWS CLI.
- Upload objects to three different storage classes (Standard, Standard-IA, Glacier Instant Retrieval).
- Verify that each object landed in the correct storage class.
Create an S3 bucket
Every object in S3 lives inside a bucket. Bucket names must be globally unique across all AWS accounts, so you will include something personal in the name to avoid collisions.
- Create your S3 bucket by running this command:
aws s3api create-bucket --bucket [[BUCKET_NAME="your-lifecycle-demo-jd42"]] --region us-east-1
What does this command do?
- aws s3api create-bucket calls the S3 API to provision a new bucket.
- --bucket sets the globally unique name. Replace the placeholder with something unique to you (e.g., your initials and a number like your-lifecycle-demo-jd42).
- --region us-east-1 places the bucket in the US East (N. Virginia) region.
- You should see a JSON response containing a Location field with your bucket name. This confirms the bucket was created successfully.
Using a region other than us-east-1?
For any region besides us-east-1, you must add --create-bucket-configuration LocationConstraint=your-region to the command. The us-east-1 region is the only one that does not require this extra flag.
Seeing a BucketAlreadyExists error?
This means another AWS account already owns a bucket with that name. Bucket names are globally unique, so try adding more characters to make yours distinct (e.g., add today's date or extra digits).
Help me pick a unique bucket name
Upload objects to different storage classes
You will create three small text files locally, then upload each one to a different storage class. This simulates a real-world scenario where hot, warm, and cold data live side by side in the same bucket.
- Create three sample files by running these commands:
echo "This is frequently accessed data" > hot-data.txt
echo "This is infrequently accessed data" > warm-data.txt
echo "This is archive-candidate data" > cold-data.txt
- Confirm the files exist by running:
dir
- You should see hot-data.txt, warm-data.txt, and cold-data.txt listed in the output.
- Upload hot-data.txt to Standard storage (the default) by running:
aws s3api put-object --bucket [[BUCKET_NAME="your-lifecycle-demo-jd42"]] --key hot-data.txt --body hot-data.txt
- Upload warm-data.txt to Standard-IA storage by running:
aws s3api put-object --bucket [[BUCKET_NAME="your-lifecycle-demo-jd42"]] --key warm-data.txt --body warm-data.txt --storage-class STANDARD_IA
- Upload cold-data.txt to Glacier Instant Retrieval storage by running:
aws s3api put-object --bucket [[BUCKET_NAME="your-lifecycle-demo-jd42"]] --key cold-data.txt --body cold-data.txt --storage-class GLACIER_IR
What do these storage classes mean?
- STANDARD is for data accessed frequently. It has the highest storage cost but the lowest retrieval cost.
- STANDARD_IA (Infrequent Access) costs less to store, but charges a per-GB retrieval fee. Good for data accessed less than once a month.
- GLACIER_IR (Glacier Instant Retrieval) has the lowest storage cost of the three, with millisecond retrieval. Ideal for data accessed roughly once per quarter.
- Each command returns a JSON response with an ETag field. This confirms the object was uploaded successfully.
Seeing a NoSuchBucket error?
Double check that the bucket name in your upload command matches exactly what you used in the create-bucket command. Even a single character difference will cause this error.
Help me fix this NoSuchBucket error
Verify storage classes of your objects
A quick verification confirms each object landed in the storage class you specified. This same command is useful in production for auditing storage costs across a bucket.
- List all objects in your bucket with their storage classes by running:
aws s3api list-objects-v2 --bucket [[BUCKET_NAME="your-lifecycle-demo-jd42"]] --query "Contents[].{Key:Key,StorageClass:StorageClass}"
What does this command do?
The --query flag uses JMESPath syntax to filter the API response down to just the object key and storage class. Without it, you would see the full metadata for every object.
- You should see output like this:
[
{
"Key": "cold-data.txt",
"StorageClass": "GLACIER_IR"
},
{
"Key": "hot-data.txt",
"StorageClass": "STANDARD"
},
{
"Key": "warm-data.txt",
"StorageClass": "STANDARD_IA"
}
]
- Confirm that hot-data.txt shows STANDARD, warm-data.txt shows STANDARD_IA, and cold-data.txt shows GLACIER_IR.
Storage class showing as null for hot-data.txt?
This is normal. S3 sometimes displays Standard class objects with a null or missing StorageClass field in API responses, since Standard is the default. Your object is still in Standard storage.
Is my storage class correct?
Your bucket now holds objects across three different storage tiers. Next up, you will build a lifecycle configuration that automates these transitions over time so you never have to move objects manually.
Build and Apply a Lifecycle Configuration
Your bucket now holds objects spread across three different storage classes. You placed each one manually, which works for a few files.
In a real account, you might have thousands of objects whose access patterns change over time. S3 lifecycle configurations let you define rules that automatically transition objects between storage classes and delete them when they expire. In this step, you will build a lifecycle policy as a JSON file and apply it to your bucket using the CLI.
In this step, get ready to:
- Create a JSON lifecycle configuration with transition and expiration rules.
- Apply the lifecycle configuration to your bucket via the CLI.
- Upload an object to a prefix that triggers both rules.
Create the lifecycle configuration JSON file
A lifecycle configuration is a JSON document that tells S3 what to do with objects as they age. You will create a file with two rules: one that transitions objects to cheaper storage classes over time, and one that permanently deletes log files after a year.
- Open Notepad by pressing the Windows key, typing Notepad, and pressing Enter.
- Copy the following JSON and paste it into Notepad:
{
"Rules": [
{
"ID": "TransitionToLowerCostStorage",
"Filter": {
"Prefix": ""
},
"Status": "Enabled",
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 90,
"StorageClass": "GLACIER"
}
]
},
{
"ID": "ExpireLogsAfterOneYear",
"Filter": {
"Prefix": "logs/"
},
"Status": "Enabled",
"Expiration": {
"Days": 365
}
}
]
}
What do these rules do?
Rule 1 (TransitionToLowerCostStorage) applies to every object in the bucket because its prefix filter is empty. It transitions objects from Standard to Standard-IA after 30 days, then to Glacier Flexible Retrieval after 90 days.
Rule 2 (ExpireLogsAfterOneYear) applies only to objects whose keys start with logs/. It permanently deletes those objects after 365 days.
- In Notepad, click File, then Save As.
- Change the Save as type dropdown to All Files (*.*).
- In the File name field, type lifecycle.json.
- Navigate to the same folder your terminal is open in. Run cd in your terminal with no arguments to check the current path.
- Click Save.
- Confirm the file was saved by running this in your terminal:
dir lifecycle.json
You should see lifecycle.json listed with its file size.
File not found?
Make sure you saved the file in the same directory your terminal is open in. Run cd to check the path, then verify the file is there.
Check that the file is not saved as lifecycle.json.txt. If Save as type was set to Text Documents (*.txt), Notepad appends .txt automatically. Rename the file or re-save with All Files (*.*) selected.
help me fix a file not found error when applying my lifecycle configuration
✔️ Awesome, I've got everything!
Great. Double check you have lifecycle.json saved and move on to the next substep.
ⓧ I'd like to double check the full code
{
"Rules": [
{
"ID": "TransitionToLowerCostStorage",
"Filter": {
"Prefix": ""
},
"Status": "Enabled",
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 90,
"StorageClass": "GLACIER"
}
]
},
{
"ID": "ExpireLogsAfterOneYear",
"Filter": {
"Prefix": "logs/"
},
"Status": "Enabled",
"Expiration": {
"Days": 365
}
}
]
}
Apply the lifecycle configuration to your bucket
Now that the JSON file exists locally, you can push it to your S3 bucket. The put-bucket-lifecycle-configuration command reads the file and attaches the rules to the bucket.
- Apply the lifecycle configuration by running:
aws s3api put-bucket-lifecycle-configuration --bucket [[BUCKET_NAME="your-lifecycle-demo-UNIQUE"]] --lifecycle-configuration file://lifecycle.json
What should I see?
This command produces no output on success. If you see your terminal prompt return with no error message, the configuration was applied successfully.
- Verify the rules are now attached to the bucket by running:
aws s3api get-bucket-lifecycle-configuration --bucket [[BUCKET_NAME="your-lifecycle-demo-UNIQUE"]]
You should see your two rules printed back as JSON, with TransitionToLowerCostStorage and ExpireLogsAfterOneYear listed.
Seeing a MalformedXML or JSON parse error?
Open lifecycle.json and check for missing commas, extra trailing commas, or mismatched braces. JSON does not allow trailing commas after the last item in an array or object.
Also confirm the file path is correct. The file:// prefix tells the CLI to read from a local file. If the file is in a different folder, provide the full path.
help me fix a MalformedXML error when applying my S3 lifecycle configuration
Upload objects with the logs/ prefix
Your expiration rule targets objects with the logs/ prefix. Any object uploaded under that prefix will be subject to both Rule 1 (transitions) and Rule 2 (expiration after 365 days).
- Create a sample log file by running:
echo "Application log entry 2024-01-15" > log-entry.txt
- Upload it to the logs/ prefix in your bucket by running:
aws s3api put-object --bucket [[BUCKET_NAME="your-lifecycle-demo-UNIQUE"]] --key logs/app-log-2024-01-15.txt --body log-entry.txt
What happens to this object over time?
This object matches both rules. Rule 1 transitions it to Standard-IA after 30 days and then to Glacier Flexible Retrieval after 90 days. Rule 2 permanently deletes it after 365 days.
S3 evaluates all matching rules and applies both transition and expiration actions. If there is a conflict, expiration always takes precedence over transition.
- Verify the object was uploaded by running:
aws s3api list-objects-v2 --bucket [[BUCKET_NAME="your-lifecycle-demo-UNIQUE"]] --query "Contents[].{Key:Key,StorageClass:StorageClass}"
You should now see four objects listed, including logs/app-log-2024-01-15.txt with storage class STANDARD.
Don't see the logs/ object?
Make sure log-entry.txt exists in your current directory. Run dir log-entry.txt to check. If the file was not created, re-run the echo command.
Double check that the --key value includes the logs/ prefix. Without it, the object lands in the bucket root and will not match the expiration rule.
help me troubleshoot uploading an object to my S3 bucket with a prefix
Your bucket now has an active lifecycle policy that will automatically manage storage costs over time. Next up, you will verify the configuration and explore the key constraints that the SAA exam tests.
Verify and Understand Lifecycle Constraints
In the previous step, you built and applied a lifecycle configuration that automates storage tiering across your bucket.
Now it is time to verify that the configuration was applied correctly and understand the constraints that AWS enforces on lifecycle transitions. These constraints are heavily tested on the SAA exam, so experiencing them firsthand will make them stick.
In this step, get ready to:
- Retrieve and inspect your lifecycle configuration from the bucket.
- Learn the key lifecycle constraints tested on the SAA exam.
- Test constraint enforcement by attempting an invalid configuration.
Retrieve and inspect your lifecycle configuration
After applying a lifecycle configuration, you should always confirm it was stored correctly on the bucket. The get-bucket-lifecycle-configuration command retrieves the active rules.
- Retrieve your bucket's lifecycle configuration by running this command:
aws s3api get-bucket-lifecycle-configuration --bucket [[BUCKET_NAME="your-lifecycle-demo-UNIQUE"]]
What does this command do?
This command queries S3 for the lifecycle configuration currently applied to your bucket. It returns the full JSON structure including all rules, transitions, and expiration settings.
- Verify that the first rule has StorageClass values of STANDARD_IA and GLACIER with Days values of 30 and 90.
- Verify that the second rule has a Prefix of logs/ and Days of 365 for expiration.
Seeing 'NoSuchLifecycleConfiguration'?
This means the lifecycle configuration was not applied in the previous step. Return to Step 3 and run the put-bucket-lifecycle-configuration command again, making sure you include file:// before the filename.
Help me troubleshoot why my lifecycle rules were not applied
Understand the key constraints that appear on the SAA exam
S3 lifecycle transitions are not a free-for-all. AWS enforces specific constraints that prevent misconfiguration and protect you from unexpected costs. These constraints frequently appear as exam questions.
What are the key lifecycle constraints?
- 30-day minimum: Objects must be stored in Standard for at least 30 days before transitioning to Standard-IA or One Zone-IA. You cannot set Days to less than 30 for these classes.
- Minimum storage duration charges: Standard-IA has a 30-day minimum, Glacier Instant Retrieval and Glacier Flexible Retrieval have 90-day minimums, and Deep Archive has a 180-day minimum. Deleting objects before these minimums incurs pro-rated charges.
- 128 KB default: Objects smaller than 128 KB are not transitioned by default. The cost of the transition request would outweigh any storage savings for tiny objects.
- Transition waterfall: You can only move objects "downward" through storage classes. You cannot use lifecycle rules to move objects back to a more-frequently-accessed class.
- Rules apply to existing objects: When you add a lifecycle configuration to a bucket, the rules apply to both existing objects and objects you add later. The age of existing objects is calculated from their original creation date.
Test constraint enforcement by attempting an invalid configuration
The best way to remember a constraint is to see it enforced. You will create a lifecycle configuration that violates the 30-day minimum and watch AWS reject it.
- Create a new file called invalid-lifecycle.json in the same folder as your other files, with the following content:
{
"Rules": [
{
"ID": "InvalidTransitionRule",
"Filter": {
"Prefix": ""
},
"Status": "Enabled",
"Transitions": [
{
"Days": 15,
"StorageClass": "STANDARD_IA"
}
]
}
]
}
What does this configuration attempt?
This rule tries to transition objects to STANDARD_IA after only 15 days. This violates the 30-day minimum constraint that AWS enforces for Standard-IA and One Zone-IA transitions.
- Save the file as invalid-lifecycle.json.
- Attempt to apply the invalid configuration by running this command:
aws s3api put-bucket-lifecycle-configuration --bucket [[BUCKET_NAME="your-lifecycle-demo-UNIQUE"]] --lifecycle-configuration file://invalid-lifecycle.json
AWS rejects this command with an error. The Days value for transitions to STANDARD_IA must be greater than 30. This confirms that the constraint is enforced at the API level.
Exam takeaway
If an SAA exam question asks about a lifecycle rule transitioning to Standard-IA in fewer than 30 days, the answer is that AWS will reject the configuration. The constraint is enforced during rule creation, not at transition time.
Not seeing an error?
If the command succeeded without an error, check the contents of invalid-lifecycle.json. The Days value must be 15 and the StorageClass must be STANDARD_IA. If the file was malformed, recreate it and try again.
Help me figure out why the constraint was not enforced
You now understand the key constraints that govern S3 lifecycle transitions. Next up, you will clean up all the resources you created during this project.
Secret mission
Multi-Department Lifecycle Strategy
In a real organization, different departments have wildly different data retention needs. Finance might need records for 7 years due to regulatory compliance. Engineering build artifacts might only last a few months. Can you design a single lifecycle policy that handles all three at once?
Clean Up Your Resources
Clean Up Your Resources
This project runs within the AWS Free Tier, so there are no ongoing costs as long as your stored objects stay under 5 GB of Standard storage. Decide whether to keep your resources running or delete them entirely.
Resources you used:
- S3 bucket your-lifecycle-demo-UNIQUE with objects and lifecycle configuration applied.
- Local files: lifecycle.json, invalid-lifecycle.json, multi-department-lifecycle.json, and sample text files (hot-data.txt, warm-data.txt, cold-data.txt, log-entry.txt).
- IAM access key (a security credential, no cost but should be managed).
Keep everything running
No action needed. Choose this if you plan to continue experimenting with S3 lifecycle policies.
- Your bucket and lifecycle rules remain active. Any new objects you upload will follow the lifecycle rules automatically.
- Storage costs are negligible for small test files under the Free Tier.
- Your IAM access key remains active. Keep your credentials secure and do not share them.
Pause - I'll come back to this later
S3 has no running compute to pause. Your bucket simply stores objects at rest, so there is no process to stop and restart.
- If you want to reduce any risk of accidental charges, delete the objects but keep the bucket and lifecycle configuration intact for when you return.
- Remove all objects from your bucket by running:
aws s3 rm s3://[[BUCKET_NAME="your-lifecycle-demo-UNIQUE"]] --recursive
- Your lifecycle configuration stays applied to the empty bucket at no cost.
Delete - I don't want to use this again
Remove all project resources and start fresh. Follow these steps to delete everything.
- Delete all objects in the bucket by running:
aws s3 rm s3://[[BUCKET_NAME="your-lifecycle-demo-UNIQUE"]] --recursive
- Delete the lifecycle configuration by running:
aws s3api delete-bucket-lifecycle --bucket [[BUCKET_NAME="your-lifecycle-demo-UNIQUE"]]
- Delete the bucket by running:
aws s3api delete-bucket --bucket [[BUCKET_NAME="your-lifecycle-demo-UNIQUE"]]
- Delete the local files by running:
del hot-data.txt warm-data.txt cold-data.txt log-entry.txt lifecycle.json invalid-lifecycle.json multi-department-lifecycle.json
- Deactivate or delete your IAM access key if you do not plan to use the AWS CLI again soon.
- Sign into the AWS Management Console and open the IAM service.
- Select Users in the navigation pane, then click your username.
- Go to the Security credentials tab.
- In the Access keys section, select Deactivate or Delete from the Actions dropdown next to your key.
That's a Wrap!
That's a Wrap!
Nice work! You just automated S3 storage cost optimization using lifecycle policies and the AWS CLI. This is exactly the kind of hands-on knowledge that separates exam-ready candidates from those who only read about storage classes.
You've learned how to:
- Upload objects to different S3 storage classes (Standard, Standard-IA, Glacier Instant Retrieval) using CLI commands and verify their placement with list-objects-v2.
- Build a JSON lifecycle configuration with transition and expiration rules, then apply and verify it using put-bucket-lifecycle-configuration and get-bucket-lifecycle-configuration.
- Validate lifecycle constraints tested on the SAA exam, including the 30-day minimum for Standard-IA transitions, minimum storage duration charges, and the transition waterfall.
- Secret Mission: Designed a multi-prefix lifecycle strategy simulating real-world departmental retention policies for finance, engineering, and marketing.
Ready to quiz yourself?