Build a Recoverable S3 Data Lake

Build a private S3 data lake with recovery, secure sharing, and lifecycle rules.

Introduction

30 Second Summary

A sales file can look trustworthy until one bad upload replaces a valid revenue figure. Without a recovery copy, that single mistake can mislead the next report.

In this project, you will build a private sales data lake in Amazon S3. You will test recovery after a bad overwrite before sharing the corrected dataset through a time-limited presigned URL.

What You'll Build

You will demonstrate a private sales dataset that recovers from a bad overwrite while staying available through a short-lived sharing link.

By the end of this project, you'll have:

  • A private sales data lake where you can upload a realistic CSV dataset through AWS CloudShell before inspecting its contents from the command line.
  • A visible version recovery demonstration where incorrect revenue replaces the clean value before S3 Versioning helps you restore the correct dataset.
  • A short-lived sharing link that opens the private dataset without an AWS sign-in while Block Public Access remains enabled.
  • Secret Mission: Add an S3 Lifecycle rule that permanently removes noncurrent versions after seven days. You will explain how that retention choice balances recovery with storage cost.

Are there any prerequisites?

You need a laptop with internet access. You also need an email address plus eligibility as a new AWS customer.

AWS CloudShell runs in your browser, so you do not need to install local command-line tools.

Before We Start

Before any hands-on work, lock in why a private, recoverable Amazon S3 sales data lake matters to the business. Your design prevents an accidental overwrite from becoming permanent data loss while keeping shared access controlled.

Set Up a Zero-Cost AWS Workspace

Your private sales data lake will live in Amazon S3. The AWS account behind it determines the cost boundary.

A safe start proves the account plan before any resources exist. It also proves which identity the command line uses.

In this step, get ready to:
  • Create an AWS account on the Free plan.
  • Confirm your plan status in the Cost and Usage widget.
  • Launch an authenticated AWS CloudShell session.
Create and confirm your AWS account

AWS currently uses two signup experiences. You may see Sign up for AWS (new) or the standard signup fields.

Keep the Free Plan Protected

A payment screen can feel risky when your goal is zero cost. AWS may place a temporary USD $1 verification hold for 3-5 days.

No costs accrue while your account remains on the Free plan.

  • Leave Upgrade plan unselected.
  • Avoid activating paid-only services.
  • Do not join AWS Organizations.
  • Do not set up AWS Control Tower.
  • Visit the AWS account signup page.
  • Choose Sign up for AWS (new) if that option is available.
  • Enter your email address in Root user email address if the standard form appears.
  • Enter a recognizable account name in AWS account name.
  • Choose Verify email address.

Both signup paths lead to the same account verification process.

  • Complete the email verification process.
  • Complete the identity verification process.
  • Choose the Free plan when the signup flow asks you to select an account plan.
  • Sign in to the AWS Management Console after the account is ready.

You'll see AWS Console Home after sign-in succeeds. That's the first cost boundary in place: your account has no paid upgrade.

Confirm Free plan protection

The Cost and Usage widget shows your account plan plus its remaining allowance. It gives you a clear stop signal before you create cloud resources.

  • Locate the Cost and Usage widget on AWS Console Home.
  • Confirm that the account is on the Free plan.
  • Record the remaining credit balance here: your remaining AWS credit balance.
  • Record the remaining days here: your remaining Free plan days.
  • Check that the widget does not say Paid plan.

Seeing Paid Plan?

The project's strict zero-charge path applies only while the account remains on the Free plan.

  • Stop before creating any AWS resources.
  • Return with an eligible account on the Free plan.

That's your cost guardrail confirmed: the widget shows the Free plan plus your remaining allowance.

Launch and verify AWS CloudShell

AWS CloudShell is a browser-based shell that receives your console credentials automatically. It includes AWS CLI version 2.

Why CloudShell Instead of a Local CLI?

CloudShell uses your console credentials automatically. This avoids local credential configuration.

Your commands still use AWS CLI. The workflow remains useful beyond this browser session.

  • Choose the CloudShell icon in the AWS Management Console header.
  • Enter CloudShell in the console search field if the icon is unavailable.
  • Choose CloudShell from the search results.
  • Wait until the Bash prompt is ready.

Before you run the identity check, do you expect CloudShell to return your AWS identity or report a credential problem?

  • Verify the identity available to CloudShell by running this command:
aws sts get-caller-identity

What Does This Command Prove?

The command asks AWS which identity CloudShell is currently using. A successful response proves that the session can call AWS without locally configured credentials.

You'll see your AWS account number and ARN in the terminal. That's the workspace ready: CloudShell is authenticated before any S3 data exists.

Did the Identity Check Fail?

  • Wait until the Bash prompt has finished loading.
  • Rerun the identity command above.
  • Check that the console is signed in to the intended AWS account.
  • Help me diagnose my CloudShell identity check.

✔️ My AWS identity returned

Your authenticated CloudShell Bash session is open. Your AWS account is still on the Free plan.

ⓧ I'd like to double check my workspace

  • Confirm that the Cost and Usage widget shows the Free plan.
  • Confirm that your recorded credit balance is your remaining AWS credit balance.
  • Confirm that your recorded days are your remaining Free plan days.
  • Confirm that no Paid plan upgrade was selected.
  • Confirm that the CloudShell Bash prompt is open.
  • Confirm that the identity command returned your account number and ARN.

Your S3 bucket and sales CSV files do not exist yet. You create them in the next step.

Your zero-cost AWS workspace is authenticated and ready. Next, you'll create the private sales data lake and expose the overwrite problem it needs to solve.

Build the Sales Data Lake and Test an Overwrite

Your authenticated AWS CloudShell session can now control Amazon S3 through the AWS CLI. This gives you a command-line path for building the sales data lake.

A working data lake can still lose good data when a later upload uses the same object key. You will build the private bucket before deliberately exposing that overwrite limitation.

In this step, get ready to:
  • Create a private general purpose S3 bucket with its security defaults enabled.
  • Create clean and incorrect sales datasets in CloudShell.
  • Test a destructive overwrite through the AWS CLI.
Create a private S3 bucket

An S3 bucket is an object storage container for files plus their metadata. The default protections give your sales data a private foundation.

Creating the bucket starts your first cloud resource in this project. Your account remains protected from charges while it stays on the Free plan.

  • Switch back to the AWS Management Console from earlier.
  • Use the console search bar to search for S3.
  • Select Amazon S3 from the search results.
  • Choose General purpose buckets.
  • Choose Create bucket.
  • Enter sales-lake-<yourinitials>-<sixdigits> in the Bucket name field.
  • Keep Bucket owner enforced selected.
  • Keep all four Block Public Access settings enabled.

What do these defaults protect?

  • Bucket owner enforced gives the bucket owner control over every object.
  • Block Public Access protects the bucket from accidental public exposure.
  • Default encryption protects each new object with an Amazon S3 managed key.
  • Keep the default SSE-S3 encryption enabled.
  • Leave Bucket Versioning disabled.
  • Choose Create bucket.
  • Confirm that sales-lake-<yourinitials>-<sixdigits> appears in the General purpose buckets list.

That is your storage foundation in place. The bucket starts with public access blocked plus default encryption enabled.

Bucket name unavailable?

  • Replace the final six digits with another six-digit number.
  • Keep the rest of the bucket name lowercase.
  • Try creating the bucket again.
  • Help me resolve an unavailable S3 bucket name.
Create the clean and incorrect sales files

A CSV stores tabular data as plain text. You need one clean dataset plus one incorrect dataset for the overwrite test.

  • Switch back to AWS CloudShell from earlier.
  • Open sales_data.csv in the native editor by running this command:
edit sales_data.csv

What does this command do?

The edit command opens sales_data.csv in the CloudShell editor. Saving the file stores it in your CloudShell home directory.

  • Paste this clean dataset into sales_data.csv:
order_id,region,product,revenue
1001,North,Laptop,1200
1002,South,Monitor,450
1003,West,Keyboard,95

What does this dataset contain?

Each row represents a synthetic sales order. The Monitor row records revenue of 450.

  • Choose the Save icon.
  • Confirm that the Monitor row contains 450.
  • Close the sales_data.csv editor tab.

Clean file missing or unsaved?

  • Run the editor command again to reopen sales_data.csv.
  • Confirm that the filename uses the underscore between sales and data.
  • Choose the Save icon before closing the tab.
  • Help me check my clean CloudShell CSV file.
  • Open bad_sales_data.csv in the native editor by running this command:
edit bad_sales_data.csv

What does this command do?

This command opens a separate file named bad_sales_data.csv. The separate filename preserves both local datasets for the overwrite test.

  • Paste this incorrect dataset into bad_sales_data.csv:
order_id,region,product,revenue
1001,North,Laptop,1200
1002,South,Monitor,-450
1003,West,Keyboard,95

Where is the incorrect value?

The Monitor row records revenue of -450. Every other value matches the clean dataset.

  • Choose the Save icon.
  • Confirm that the Monitor row contains -450.
  • Close the bad_sales_data.csv editor tab.

Incorrect file missing or unsaved?

  • Run the editor command again to reopen bad_sales_data.csv.
  • Confirm that the filename starts with bad_.
  • Check that the Monitor row contains -450.
  • Help me check my incorrect CloudShell CSV file.

✔️ Awesome, I've got everything!

Good work. CloudShell now holds a clean dataset with 450 plus an incorrect dataset with -450.

ⓧ I'd like to double check the full code

Compare each saved file with its complete reference:

order_id,region,product,revenue
1001,North,Laptop,1200
1002,South,Monitor,450
1003,West,Keyboard,95

What should match?

The clean file has four lines. Its Monitor row ends with 450.

order_id,region,product,revenue
1001,North,Laptop,1200
1002,South,Monitor,-450
1003,West,Keyboard,95

What should match here?

The incorrect file also has four lines. Its Monitor row ends with -450.

Test a same-key overwrite

An object key identifies an object inside a bucket. Both uploads target the sales_data.csv key.

  • Replace every YOUR_BUCKET_NAME occurrence in the commands below with sales-lake-<yourinitials>-<sixdigits>.
  • Upload the clean dataset plus list the stored object by running these commands:
aws s3 cp sales_data.csv s3://YOUR_BUCKET_NAME/sales_data.csv
aws s3 ls s3://YOUR_BUCKET_NAME/

What do these commands do?

  • The first command uploads the local clean dataset to the sales_data.csv object key.
  • The second command lists the current objects in your bucket.

You will see an upload confirmation followed by a bucket listing that includes sales_data.csv.

Your clean sales dataset is now stored in Amazon S3. The next upload tests whether that good copy survives at the same key.

Clean object missing from the list?

  • Check that every YOUR_BUCKET_NAME occurrence was replaced with your exact bucket name.
  • Confirm that the upload source is sales_data.csv.
  • Confirm that the destination key ends with sales_data.csv.
  • Help me diagnose the missing clean S3 object.
  • Replace every YOUR_BUCKET_NAME occurrence in the next commands with sales-lake-<yourinitials>-<sixdigits>.

Before you run the overwrite test, which revenue value do you think the current S3 object will show?

  • Upload the incorrect file to the same object key plus stream the current object by running these commands:
aws s3 cp bad_sales_data.csv s3://YOUR_BUCKET_NAME/sales_data.csv
aws s3 cp s3://YOUR_BUCKET_NAME/sales_data.csv -

The overwrite exposed the gap

You will see four CSV lines in the terminal. The Monitor row now contains -450.

The second upload replaced the clean current object because S3 Versioning is disabled. Amazon S3 has no recoverable version of the original clean upload.

Still seeing 450?

  • Confirm that the upload source is bad_sales_data.csv.
  • Check that bad_sales_data.csv contains -450 in the Monitor row.
  • Verify that both commands use your exact bucket name.
  • Help me find why the current S3 object still shows 450.

You have proved the overwrite risk with your own sales dataset. Next, you will add version history so an earlier clean dataset can become current again.

Enable Versioning and Recover the Dataset

Your private Amazon S3 bucket now holds a bad current object. The earlier overwrite proved that the current object alone cannot protect your sales data.

S3 Versioning keeps a distinct version of each new write to the same object key. That history gives you a recovery path after an accidental overwrite.

This step builds that recovery path. You will prove it by making the clean dataset current again.

In this step, get ready to:
  • Enable S3 Versioning before completing the required propagation wait.
  • Create a clean object version followed by an incorrect current version.
  • Delete the incorrect version to restore the clean dataset.
Enable S3 Versioning and wait

Versioning begins preserving new writes after you enable it. AWS recommends a 15-minute pause after the first activation because writes can behave inconsistently while the change propagates.

  • Return to the Amazon S3 console from earlier.
  • Choose General purpose buckets in the left navigation pane.
  • Select sales-lake-<yourinitials>-<sixdigits>.
  • Choose the Properties tab.
  • Choose Edit under Bucket Versioning.
  • Choose Enable.
  • Choose Save changes.

The Bucket Versioning setting now shows that versioning is enabled.

Why does versioning need time to propagate?

The setting affects writes across Amazon S3. Writes made immediately after the first activation can produce intermittent HTTP 404 NoSuchKey behavior.

The required wait gives the change time to propagate before you create the versions used for recovery.

This pause takes 15 minutes, so a quiet screen during that window is expected. Your CloudShell session can remain open while the setting propagates.

  • Wait 15 minutes before uploading either CSV file again.

That is the longest pause in this project complete. Your bucket is ready to preserve the next writes as separate versions.

Create protected object versions

Both CSV files remain in AWS CloudShell from the previous step. Writing each file to the same object key now creates recoverable history.

  • Switch back to the authenticated AWS CloudShell session from earlier.
  • Replace YOUR_BUCKET_NAME in both commands with the name of your bucket.
  • Create the clean version followed by the bad current version by running these commands:
aws s3 cp sales_data.csv s3://YOUR_BUCKET_NAME/sales_data.csv
aws s3 cp bad_sales_data.csv s3://YOUR_BUCKET_NAME/sales_data.csv

What do these commands do?

  • The first command uploads the clean dataset to the sales_data.csv object key.
  • The second command uploads the bad dataset to the same key.
  • S3 Versioning keeps the clean upload as a noncurrent version when the bad upload becomes current.

CloudShell reports an upload for each command. Those two upload messages confirm that both writes reached the versioning-enabled bucket.

Did an upload fail?

  • Confirm that the full 15-minute wait finished before these writes.
  • Check that both commands use your exact bucket name in place of YOUR_BUCKET_NAME.
  • Confirm that sales_data.csv and bad_sales_data.csv still exist in CloudShell.
  • Help me diagnose an S3 upload failure.

Before you inspect the current object, consider which upload should appear at the top of the version history.

  • Stream the current object back to CloudShell by running this command:
aws s3 cp s3://YOUR_BUCKET_NAME/sales_data.csv -

What does this check prove?

The command reads the current sales_data.csv object directly from the bucket. It does not read either local CSV file.

You will see the row for order 1002 with revenue -450. The bad upload is current even though the clean version remains recoverable.

Do you see 450 instead?

  • Confirm that you uploaded bad_sales_data.csv after sales_data.csv.
  • Check that both upload commands targeted the sales_data.csv object key.
  • Help me check why the bad upload is not current.
Delete the bad version and verify recovery

Each version has its own Version ID. The S3 console lets you identify the latest bad version before permanently removing it.

  • Return to the Objects tab in your S3 bucket.
  • Choose the sales_data.csv object.
  • Choose Versions.

You will see multiple Version IDs for sales_data.csv. The latest entry represents the bad upload that currently returns -450.

Permanent deletion is the risky moment in this recovery test. Removing the latest bad version reveals the clean version directly beneath it.

  • Select the current latest Version ID.
  • Choose Delete.
  • Enter permanently delete in the confirmation field.
  • Choose Delete objects.

The preceding clean version becomes current. The original pre-versioning bad object stays in the history as a noncurrent version.

Before you run the final check, consider which revenue value the current object should contain.

  • Verify the recovered current object by running this command in CloudShell:
aws s3 cp s3://YOUR_BUCKET_NAME/sales_data.csv -

What does this recovery check prove?

The command streams the current object after the bad version was deleted. You will see the row for order 1002 with revenue 450.

That result proves the earlier clean version became current without another upload.

You have completed the recovery path. Your bucket now returns the clean sales data while S3 Versioning protects its history.

Does the current object still show -450?

  • Return to the version history for sales_data.csv.
  • Confirm that you deleted the latest version created by the second upload.
  • Check that the verification command uses the same bucket from this project.
  • Help me restore the clean S3 object version.

Your sales data lake can recover from a destructive overwrite. Next, you will share the private dataset without disabling its public access protections.

Share the Dataset Privately

The clean sales dataset is current again. Amazon S3 now protects future overwrites with version history.

The next goal is to share sales_data.csv with someone who has no AWS credentials. A presigned URL provides temporary download permission while Block Public Access remains enabled.

In this step, get ready to:
  • Confirm that the bucket remains private.
  • Create a presigned URL with a 10-minute validity period.
  • Test the temporary download from a private browser window.
Confirm the bucket stays private

The bucket's privacy boundary comes from its public access controls. Temporary sharing leaves that boundary unchanged.

  • Return to the sales-lake-<yourinitials>-<sixdigits> bucket in Amazon S3.
  • Choose the Permissions tab.
  • Find the Block public access (bucket settings) section.
  • Confirm that all public access remains blocked.
  • Confirm that no public bucket policy has been added.
  • Leave all four Block Public Access settings enabled.

You have preserved the bucket's security boundary. The restored dataset remains private while you prepare temporary access for one recipient.

Why use a presigned URL?

A presigned URL carries permission from your signed-in AWS identity. Its validity window limits how long someone can use that permission.

Block Public Access remains enabled for the bucket. The recipient receives access to the selected object through the temporary link.

Generate temporary access

This link deserves the same care as a temporary credential. Its 10-minute window limits exposure. Anyone holding the link can download the object before it expires.

  • Return to the Objects tab in the same bucket.
  • Select sales_data.csv.
  • Open Object actions.
  • Choose Share with a presigned URL.
  • Set the validity period to 10 minutes.
  • Choose Create presigned URL.
  • Keep the copied URL on your clipboard for the next substep.
  • Keep the presigned URL query string out of screenshots.

AWS copies the new presigned URL to your clipboard. The bucket configuration remains private.

That sensitive step is handled. The temporary link is ready for one controlled test.

Can't create the presigned URL?

  • Confirm that sales_data.csv remains selected in Objects.
  • Open Object actions again after selecting the object.
  • Check that your signed-in AWS identity has permission to read the object.
  • Help me create a presigned URL for my private S3 object.
Test the recipient experience

A fresh private browser window separates this test from the signed-in console session. A successful download there proves that access comes from the presigned URL.

Before you test the link, do you think the recipient will need an AWS sign-in?

  • Use your browser's menu to open a private or incognito window.
  • Paste the copied presigned URL into the address bar.
  • Press Enter.

The CSV downloads or opens without an AWS sign-in. The row for order 1002 shows revenue 450.

Does the link fail to open?

  • Generate a new 10-minute presigned URL if the previous link has expired.
  • Paste the full replacement URL into the private window's address bar.
  • Help me test my presigned URL safely.
  • Open the downloaded sales_data.csv in the private window if your browser saved the file.
  • Capture a screenshot of the successful CSV contents with the address bar excluded.
  • Return to the Permissions tab from earlier.
  • Confirm that Block public access (bucket settings) still shows all public access blocked.
  • Capture a separate screenshot of the enabled public access controls.
  • Redact account identifiers from either screenshot before uploading it.
  • Redact the bucket name if you prefer.

You have now proven the full sharing model. The clean CSV reaches a recipient while the bucket stays private.

Secret mission

Control the Cost of Old Versions

Your bucket can now recover from accidental overwrites. Every stored version is a full object. Add a seven-day lifecycle guardrail that limits old-version storage while preserving a short recovery window.

Clean Up Your Resources

Clean Up Your Resources

Your verified AWS Free plan keeps this project at USD $0 while the account remains on that plan. AWS CloudShell has no additional charge.

The choices below cover keeping the architecture, pausing its stored data, or removing every project resource.

Cost warning

Amazon S3 meters storage. It also meters requests. Data transfer is metered too.

Each stored object version is a full object. The tiny synthetic files in this project keep usage small.

Your account stays at USD $0 while it remains on the Free plan. That protection ends after six months or when credits are exhausted.

  • Choose the Pause or Delete option if you no longer need the stored copies.

Resources you used:

  • One Amazon S3 general purpose bucket named sales-lake-<yourinitials>-<sixdigits>.
  • One current sales_data.csv object containing the clean revenue value 450.
  • Noncurrent sales_data.csv object versions.
  • One enabled S3 Lifecycle rule named delete-old-sales-versions.
  • The sales_data.csv file in the AWS CloudShell home directory.
  • The bad_sales_data.csv file in the AWS CloudShell home directory.

The tested presigned URL expires automatically after 10 minutes. It needs no manual cleanup.

Keep everything running

No action needed. Choose this if you want to keep the storage architecture available for more practice.

  • The current clean object remains private behind Block Public Access.
  • S3 Versioning remains enabled for future overwrites.
  • The delete-old-sales-versions rule permanently deletes noncurrent versions after 7 days.
  • The two CSV files remain available in AWS CloudShell.
  • The Free plan ends after six months or when credits are exhausted, whichever comes first.

Pause - I'll come back to this later

Keep the empty bucket configuration for later practice. Remove the stored object versions to reduce metered S3 storage.

Emptying the bucket is irreversible. Your bucket settings remain available after its objects are removed.

  • Save your project screenshots to your laptop.
  • Return to Amazon S3 in the AWS Management Console.
  • Choose General purpose buckets.
  • Select the sales-lake-<yourinitials>-<sixdigits> bucket.
  • Choose Empty.
  • Complete the permanent empty confirmation shown by the console.

You should see an empty bucket with no current objects. Its object versions and delete markers have been permanently removed.

  • Switch back to the authenticated AWS CloudShell session from earlier.
  • Remove both CSV files by running:
rm sales_data.csv bad_sales_data.csv

What does this command do?

The rm command removes both project CSV files from the CloudShell home directory. Your empty S3 bucket remains available.

CloudShell returns to the Bash prompt without an error when both files are removed.

Seeing a file removal error?

A message about a missing file usually means that file was already removed. The other CSV file may still remain.

Ask for help checking which CloudShell file still needs removal.

Delete - I don't want to use this again

Remove the S3 bucket, every stored object version, the lifecycle rule, and both CloudShell files.

Deleting version history is permanent. Your saved screenshots preserve the project result after the AWS resources are gone.

  • Return to Amazon S3 in the AWS Management Console.
  • Choose General purpose buckets.
  • Select the sales-lake-<yourinitials>-<sixdigits> bucket.
  • Choose Empty.
  • Complete the permanent empty confirmation shown by the console.

You should see no objects in the bucket. Every object version and delete marker has been removed.

  • Return to General purpose buckets.
  • Select the empty sales-lake-<yourinitials>-<sixdigits> bucket.
  • Choose Delete.
  • Enter sales-lake-<yourinitials>-<sixdigits> as the bucket name.
  • Choose Delete bucket.
  • Search the General purpose buckets list for sales-lake-<yourinitials>-<sixdigits>.

You should find no matching bucket. Deleting the bucket also removes the delete-old-sales-versions lifecycle configuration.

  • Switch back to the authenticated AWS CloudShell session from earlier.
  • Remove both CSV files by running:
rm sales_data.csv bad_sales_data.csv

What does this command do?

The rm command removes the clean file and the deliberately incorrect file from the CloudShell home directory.

CloudShell returns to the Bash prompt without an error when both local project files are removed.

Seeing a file removal error?

A message about a missing file usually means that file was already removed. The remaining filename identifies what still needs attention.

Ask for help finishing the CloudShell cleanup.

Nice Work!

Nice Work!

Excellent work! You built a private sales data lake on Amazon S3 that can recover clean data after a destructive overwrite. You also proved that a recipient can download the private dataset through temporary access.

You've learned how to:

  • Create a private sales data lake. Keep every object protected with Block Public Access. Apply Bucket owner enforced ownership. Use SSE-S3 encryption by default.
  • Use AWS CloudShell to reproduce a destructive overwrite through the AWS CLI. Restore the clean 450 value with S3 Versioning.
  • Create a 10-minute presigned URL for sales_data.csv. Prove that it works without an AWS sign-in. Keep all public access blocked.
  • Secret Mission: Configure delete-old-sales-versions to permanently remove noncurrent versions after seven days. Explain the resilience versus cost trade-off behind the lifecycle rule.

Ready to quiz yourself?