Multi-Cloud Data Transfer with AWS and GCP

Transfer data between AWS S3 and GCP Cloud Storage in this multi-cloud project!

Introduction

This project has been archived

We're no longer updating this project - we recommend checking out our active projects instead!

If you've already completed this project, don't worry! Your documentation is still available for you to check out and share.

If you have any questions, leave a question in our Discord and we'll get back to you ASAP!

⚡️ 30 second Summary

Welcome to this project on multi-cloud data transfer with AWS and Google Cloud Platform (GCP)! ☁️☁️

More and more engineering teams are spreading their resources across different cloud providers. Why? It gives them better redundancy, helps avoid getting locked into one vendor, and often saves money too. In this hands-on project, you'll learn exactly how to move data smoothly between AWS S3 and Google Cloud Storage!

This is a skill that's in high demand right now, especially for roles like Cloud Engineer, DevOps Engineer, and Solutions Architect. Being comfortable working across multiple cloud platforms is becoming essential for building flexible and resilient systems.

In this project, get ready to...

🪣 Set up and configure AWS S3 and GCP Cloud Storage buckets.
🔑 Use AWS IAM roles to connect GCP with AWS.
🚚 Troubleshoot the data transfer.
💎 Implement real-time data transfers using Amazon SQS.

Want a complete demo of how to do this project, from start to finish? Check out our 🎬 walkthrough with Natasha 🎬

If you're up for a bit of a challenge, quiz yourself on the key concepts up ahead in this project.

Before we start Step #1...

In this project, you’ll set up a multi-cloud storage system by backing up your files in S3 to Google Cloud Storage.

We’ll break this project down into three key parts:

Set up AWS and GCP
- In Step #1, we’ll create an S3 bucket in AWS to store our data.
- In Step #2, we’ll set up a Google Cloud Platform (GCP) account.
Connect AWS and GCP
- In Step #3, we’ll use Storage Transfer Service in GCP to set up a system for backing up files from AWS.
- In Step #4, we’ll create an IAM role to grant Storage Transfer Service access to AWS resources.
Back up your data!
- In Step #5, we’ll run a transfer to see our our files in S3 get backed up in GCP too.
- In the 💎 Secret Mission, we’ll set up a selective file transfer - pick exactly which files get transferred using a manifest file!

Can you see the steps in this architecture diagram?

What is multi-cloud?

Multi-cloud means spreading your data and services across multiple cloud providers. Companies like this because it means:

Higher reliability – If AWS has an outage, GCP can keep your apps running and have a copy of your data.
More flexibility – You can make the most of each provider's different strengths. For example, AWS is best known for their compute services, and GCP for their data/AI services.
Cost savings – Since pricing varies across cloud platforms, multi-cloud lets you optimize on costs.

Store Data in an S3 Bucket

Let's get started by creating an S3 bucket in AWS. This bucket will serve as the source for our data transfer to GCP Cloud Storage.

In this step, you're going to:

Create a new S3 bucket.
Upload files to your S3 bucket.

Navigate to S3

First, let's find the S3 service in the AWS Management Console.

Log in to the AWS Management Console as your IAM Admin user.
In the search bar at the top of the console, type S3.
Select S3 from the search results.

What is Amazon S3?

Amazon S3 (Simple Storage Service) is AWS's service for storing objects. Think of it as a highly scalable and reliable place to store files of any type and size in the cloud. We're using S3 as the main storage solution, which we'll back up using GCP in this project.

Make sure you're doing this project in one of the following regions! You'll find later on that the service responsible for transferring our files (GCP's Storage Transfer Service) can only work these S3 regions.

Create an S3 Bucket

Now that we're in the S3 console, let's create our bucket.

Select Create bucket.

In the Create bucket page, under Bucket name, enter a unique bucket name:

nextwork-data-transfer-source-[[YOURNAME="yourname"]]

Make sure you've replaced yourname with your name or initials to make the bucket name unique.

Leave all other settings as default for now. We'll keep things simple for this project.
Scroll to the bottom of the page and select Create bucket.
You should see a success message indicating that your bucket has been created.

Awesome! You've created your first S3 bucket. It's looking great. Now, let's upload some data to it.

Upload Files

Let's get some files into our S3 bucket.

Select your newly created bucket from the list of buckets.

Inside your S3 bucket, select the Upload button.
On the Upload page, select Add files.
In the file selection dialog, choose at least 1-3 random files from your computer to upload.
If you're unsure about the files you'd like to upload, you can download this zip file and upload the PDF documents stored inside!

What kind of files can I upload?

You can upload virtually any type of file to S3, from documents and images to videos and backups. S3 is versatile and can store any data you need in the cloud!

Once you've selected your files, they should be ready to upload.

After selecting your files, select Upload at the bottom of the page.

Let's gooooooo. You've just uploaded data to your S3 bucket.

In the next step, we'll switch gears and set up our Google Cloud Platform account.

Create a Free GCP Account

Now, let's move over to Google Cloud Platform (GCP) and set up an account. We'll need an account to set up another storage location in GCP to backup our data in S3.

In this step, you're going to:

Do you already have a Google Cloud Platform (GCP) account?

Not sure if you have a GCP account?

Try visiting console.cloud.google.com and signing in with your Google account to check. If you can access the Google Cloud Console (without getting taken to a sign up screen), you already have an account!

Yes, I already have an account

Great! If you already have a Google Cloud Platform account, you can skip ahead to Step #3.

Just make sure you can sign in to your existing GCP account before moving on!

What about the tasks and screenshots in this step? Do I still need to complete them?

Nope - we've marked the tasks in this step as optional, so you're free to start the next step now (see you there)!

Nope, I don't have a GCP account

Let's set up your free Google Cloud Platform account from scratch! WOOOOOOOOO 👏

Follow the detailed steps below to create your new GCP account and take advantage of the free offerings.

Visit Google Cloud Platform

Let's start by heading to the Google Cloud website.

Head to cloud.google.com.

What is Google Cloud Platform (GCP)?

Google Cloud Platform (GCP) is Google's suite of cloud computing services and products. Similar to AWS, GCP has a wide range of services including computing power, storage, databases, machine learning, and networking tools. It's one of the major cloud providers in the market alongside AWS and Microsoft Azure!

💡 Why are we creating a GCP account?

We're creating a GCP account in this project to set up your multi-cloud environment. Since the goal is to transfer data between AWS and Google Cloud, we need accounts on both platforms for this to work.

Creating a GCP account gives you:

Access to Google Cloud Storage, which will be the destination storage location for data transferred from AWS S3
Access to Google's Storage Transfer Service, which manages the transfer process between the cloud providers

The GCP free tier and trial credits ($300) that come with a new GCP account will let you to do this project without incurring any charges.

Now, let's sign up for your free GCP account.

Select Get started for free.

Free? What can I get for free at GCP?

GCP has free offerings that lets you use its services without paying. Let's take a look at what it includes:

Free Tier services: Free use of more than 25 services (e.g. Cloud Storage service) up to their monthly limits. This is very similar to AWS's Free Tier!
Free Trial: When you first open a Goosgle Cloud account, you also get $300USD in credits to spend on GCP services that are not covered by the Free Tier. The free credit expires in 90 days from the day you open your account.

💡 What are GCP's Free Tier services?

Some of the Free Tier services in GCP include:

Compute Engine: For virtual machines.
Cloud Storage: For storing data.
BigQuery: For data analytics.
Cloud Run: For deploying containerized applications.

Yes!

Great! Since you already have a Google account, you can use that to sign up for Google Cloud Platform.

I don't have a Google account

No problem! You'll just need to create a Google account first before setting up Google Cloud Platform.

When prompted to sign in, select Create a Google account.
Follow the steps to create your Google account:
- Enter your name, desired email, and a password.
- Verify your phone number.
- Complete Google's account setup process.
Once your Google account is created, you'll be taken back to the GCP setup flow.

Select Agree and continue to accept the Terms of Service.

Fill in your payment profile details. Select Create new payments profile.

Change the Organization type to Individual.

Enter your address and other required information.

Select Create to create your payment profile.

Select Add a payment method.

Select Add credit or debit card.

Why do I need to enter credit card details?

We need to give GCP a credit card for identity verification - that's it! You won't be charged unless you explicitly upgrade to a paid account or go over GCP's free tier limits.

GCP will also automatically stop all your resources at the end of your 90 day free trial, so that you also don't get charged for leaving resources running in your account.

The NextWork community is always here if you have any questions or concerns - join us here!

Select Save card after entering your card details.

Select Start free to continue with the account setup.

Complete the project setup by answering the questions. For Organization needs, select Other and specify Personal

For What brings you to Google Cloud?, select Learn more / explore.

Select your interests in Google Cloud.

For Your role, select Other and specify Student

All done!

Awesome! You've successfully set up your GCP account.

Recap: So what's in GCP's free trial?

Just like AWS, GCP has a Free Tier that includes 25+ services - including their storage solution (i.e. their version of S3) called Cloud Storage.

On top of this Free Tier, GCP also gives you a free trial. In this free trial, you get $300 in credits to spend on services not covered by GCP's Free Tier. The free trial starts from the day you open your account, and only lasts 90 days.

Once the trial is over, GCP automatically stops all your resources so you don't get accidentally charged. If you want to keep using GCP, you'll have to upgrade your account to a Billing (i.e. paying) account where you'll still get access to the Free Tier services. The only difference is that you won't get free credit to help pay for any additional services/resources beyond the Free Tier.

You can learn more about GCP's free trial and Free Tier services here.

Ready to get started with Google Cloud?

Set Up a Storage Transfer from AWS to GCP

Turns out, there's a handy service in GCP that can help us transfer data from AWS to GCP!

Let's set up a transfer in GCP's Storage Transfer Service. This service simply needs a source (AWS S3 bucket) and destination (GCP Cloud Storage bucket) for our data transfer.

In this step, you're going to:

Create a new transfer job in GCP's Storage Transfer Service.

Create a New Transfer Job

In your Google Cloud dashboard, select the menu icon on the top left hand corner.

Woah! Welcome to GCP's services menu. GCP organizes services a little different from AWS - main service categories are already categorized in the menu, and you can hover over each one to see the most popular GCP services under each category.

Select View all products at the bottom of the navigation panel.

Select Storage from the left hand navigation menu.

Select Storage Transfer.

What is Storage Transfer Service?

Storage Transfer Service is a Google Cloud service designed for transferring data in and out of GCP Cloud Storage (which is like GCP's version of Amazon S3).

Instead of transferring directly between S3 and GCS, we need this service because cloud providers don't offer native ways to connect to their competitors.

Storage Transfer Service handles authentication between the two platforms, starts and stops the transfer, and makes sure data transfers correctly. Without it, you'd need to build your own transfer system, which would mean downloading all your data to a computer first, then uploading it again to the other cloud - slow, expensive, and prone to errors!

In the Storage Transfer Service console, select CREATE TRANSFER JOB.

Configure Source and Destination

Now, let's set up the source and destination for our transfer job!

In the Get started section:
- Set Source type to Amazon S3 bucket.
- Set Destination type to Google Cloud Storage.
- Under Scheduling mode, select Batch.

What is Batch scheduling?

Batch scheduling means a one-time or scheduled transfer of data. It's great for migrating large datasets or periodic backups.

The other option, Event-driven transfer, automatically transfers data whenever the source bucket gets a new or updated object. Event-driven transfer is ideal for real-time synchronization but comes with a few extra steps.

Select NEXT STEP.

Super cooooool! You've set up the basic settings for your transfer job.

Specify S3 Bucket Name

Next up, let's tell Storage Transfer Service which S3 bucket to use as the data source.

In the Bucket or folder field, enter your S3 bucket's name:

nextwork-data-transfer-source-[[YOURNAME="yourname"]]

Extra for Experts: What are the CloudFront and Managed private network settings under 'Bucket or folder'?

Google Storage Transfer Service includes these options because it's also designed to handle a wide range of enterprise transfer scenarios, not just simple bucket-to-bucket transfers.

CloudFront domain is a setting for the organizations that use CloudFront to distribute their content globally. Connecting to a CloudFront distribution instead of directly to S3 can actually improve transfer speeds and reduce costs, especially for international transfers with terabytes of data.

Managed private network is available because many enterprises need to transfer sensitive data that must never go through the public internet for security or compliance reasons. This option lets you create a secure private connection between GCP and AWS using tools like VPC peering or Direct Connect.

Configure Credentials

Now that GCP knows which S3 bucket it should connect to, it's high time we give GCP the permission to access it.

Still in your Storage Transfer setup, select AWS IAM Role for Identity Federation under Credentials.

What is identity federation?

Identity federation creates a secure trust relationship between cloud providers. It lets GCP temporarily get access to AWS resources without permanent credentials.

You create an IAM role in AWS with S3 permissions and let GCP Transfer Service to use that role. When GCP presents its identity to AWS, AWS will issue temporary credentials based on this role.

These credentials automatically expire after use - much more secure than sharing permanent access keys!

💡 Extra for Experts: How do IAM roles work with federation?

While IAM roles are permanent, federation access is temporary. AWS grants Storage Transfer Service short-lived credentials (15-60 minutes) for each session. Once those 15-60 minutes are up, Storage Transfer Service is back to having no access to AWS until it makes its next request for data!

Fantastic! You've set up the S3 source for your transfer job.

In the next step, we'll create an IAM role in AWS that GCP can use to access your S3 bucket.

Grant Google Storage Transfer Access to AWS

To securely access your S3 bucket from GCP, we need to create an IAM role in AWS. This role will grant GCP's Storage Transfer Service the permission to read data from your S3 bucket.

In this step, you're going to:

Create a custom IAM role for GCP.

Navigate to IAM

Let's head to the IAM console to create our role.

Head back to the AWS Management Console
In the search bar, type IAM and select IAM.

Create a Custom IAM Role

In the IAM console, select Roles in the left-hand navigation panel.
Select Create role.
Under Select trusted entity, choose Custom trust policy.

Why Custom trust policy?

We're choosing Custom trust policy because it allows us to define exactly who can assume this IAM role. In this case, we want to specify that only GCP's Storage Transfer Service can assume this role.

We need to define a custom trust policy that lets GCP to assume this role!
In the Custom trust policy editor, replace the default policy with the following JSON policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "accounts.google.com:sub": "[[AWS_MULTICLOUD_STORAGE_SUBJECT_ID="SUBJECT_ID"]]"
        }
      }
    }
  ]
}

What is a trust policy?

A trust policy tells AWS that external services or users are allowed to use this IAM role.

In our case, this trust policy tells AWS to trust requests coming from Storage Transfer in your GCP project (GCP calls accounts "projects").

💡 How would I know what to write for the custom trust policy?

You can actually find the policy we're using above in GCP's documentation!

Retrieving the subject ID

Before we can finish creating this policy, we need to replace the placeholder for SUBJECT_ID.

Why do we need a subject ID?

A subject ID is how AWS knows whether a request to transfer data is really coming from your GCP project's Storage Transfer, and not from someone else's project.

💡 Extra for Experts: Ooo, how does a subject ID work?

Within each GCP project, Google automatically creates special service accounts for every GCP service. A service account is what gives a service (like Storage Transfer) the permission to do something in your GCP account.

Every service account in every GCP project has a unique ID, called the subject ID! This means Storage Transfer will have a different service account and subject ID from the other services in your account. The Storage Transfer in your project will also have a different service account and subject ID from Storage Transfer in other people's projects.

To retrieve the subject ID, head to this page in Storage Transfer's official documentation: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/googleServiceAccounts/get.

What is the googleServiceAccounts.get method?

This method is like a profile search for a service account. When you call this method with your project ID, it returns details about the unique service account that handles your GCP account's Storage Transfer Service.

You'll see a panel on the right called Try this method.

What does this panel do?

This panel lets you use the API in a visual interface, so you don't need to learn how to call the API with code!

To use this method, we need your GCP project ID.
Head back to your Storage Transfer tab in GCP.
Select your project (e.g., My First Project).

Copy the ID of your project.

Go back to the Google Service Accounts API page.
Paste your project ID into the projectIds field.

Select the Execute button.

You might be asked to log in to Google again for verification that you have permissions to that project. If so, select the same Google account you used to sign up.

Select Continue.

Click Allow to grant Google APIs Explorer permission to your Google Account.

Why is Google APIs Explorer asking for permission?

Google APIs Explorer is asking for temporary access to your Google account so it can find information about the service account and give you the subject ID for your Google Cloud project.

The permission will automatically expire after 7 days, and you can remove it sooner if you want to. Once we have the subject ID, we won't need to use this tool again for this project.

You'll be taken back to the API page.

Scroll down to the Response section.
Copy the subjectId from the response.

Go back to your IAM role creation tab in AWS.
Replace the placeholder SUBJECT_ID in the trust policy below with the subjectId you just copied.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "accounts.google.com:sub": "[[AWS_MULTICLOUD_STORAGE_SUBJECT_ID="SUBJECT_ID"]]"
        }
      }
    }
  ]
}

Now we're ready!
Paste the modified code into the Custom trust policy editor in IAM.

Select Next.

Attach Permissions Policy

Now, let's attach the AmazonS3ReadOnlyAccess policy to our role.

In the Add permissions page, in the search bar, type S3ReadOnly

Select the policy named AmazonS3ReadOnlyAccess.

What is AmazonS3ReadOnlyAccess?

AmazonS3ReadOnlyAccess is an AWS policy that grants read-only access to all S3 buckets in your AWS account. For this project, read-only access is enough because we're just backing up data from S3 to GCS (and read-only access also includes retrieving objects). We don't need to modify or delete any data in the S3 bucket.

This follows the principle of least privilege in cloud architecture, where we grant only the necessary permissions.

Select Next.

In the Role details page, under Role name, enter nextwork-data-transfer-gcp-role.

In the Description, add the following text:

Role that grants my GCP project's Storage Transfer Service Read Only access to my S3 buckets. Created this IAM role as a part of NextWork's multi-cloud project.

Review the policy summary - make sure it includes AmazonS3ReadOnlyAccess.

Select Create role.

Yay! You should see a success message that your role has been created 👌
Select View role to see the details of the role you just created.

Beautiful! You've created an IAM role for GCP access.

In the next step, we'll use this role's to complete the S3 source configuration in GCP.

Transfer Your Objects from S3 to GCS!

Now that we have our IAM role set up, let's go back to GCP Storage Transfer Service and complete the S3 source configuration.

In this step, you're going to:

Finish setting up your transfer in Storage Transfer Service.
Run the transfer job 🏃‍♀️
See your files get backed up from AWS to GCP!

Continue setting up GCP Storage Transfer Service

Now that we've created the role, what's next?

Let's copy the role's ARN (Amazon Resource Number, which is like its unique ID) to use it in our Storage Transfer setup.

Still in the IAM console, copy the Role ARN.

Go back to your Storage Transfer job in GCP.
Under Credentials, paste the IAM role ARN you copied into the Role ARN field.

Extra for Experts: What are these filtering options for?

These checkboxes let you control exactly which files get transferred from your S3 bucket to GCP Cloud Storage.

The Filter by prefix option lets you transfer only files that start with specific characters or folders (like only transferring files in a "backup/" folder).

The Filter by last modified time option lets you transfer only files that were created or updated within a specific timeframe (like only transferring files modified in the last 7 days).

By leaving both unchecked, you're telling GCP to transfer all files from your S3 bucket - regardless of their location or when they were last updated.

Select NEXT STEP.

Create a GCS Bucket

In the Choose a destination section, select BROWSE to set up a destination bucket in GCS i.e. GCP Cloud Storage.

Select the bucket icon to Create new bucket.

What are GCS buckets?

Just like S3 buckets, GCS buckets are folders for storing your objects in Google Cloud Storage.

In the Create a bucket form, under Name your bucket, give your GCS bucket a name.

nextwork-data-transfer-destination-gcp-[[YOURNAME="yourname"]]

Select CONTINUE.

Now, let's choose where to store your data in GCP.
Under Choose where to store your data, select Region for the location type.

What are the different location types?

Think of these bucket location types like different backup strategies for your data:

Region is the most cost-effective option (data is only stored in one region) and works well when you know exactly where your data needs to be stored, perhaps for compliance or performance reasons.

Multi-region automatically replicates your data across multiple regions, which gives you higher availability and protection against regional outages. It costs more but also gives you better resilience.

Dual-region stores your data in exactly two regions that you choose, which is a great in-between option (between region and multi-region). It's ideal for disaster recovery while making sure data is kept local to the regions you choose.

For our project, we're using Region - this keeps our solution simple and sticks to just one region!

Pick a region that is geographically close to your S3 bucket's region - this can help improve transfer speeds!

Select CONTINUE again.

Choose Storage Class

Let's choose a storage class for your GCS bucket.

Under Choose a storage class, select the Standard class.

What are storage classes?

GCP Cloud Storage offers different storage classes based on how frequently you'll be accessing your data. Think of a storage class as a different pricing plan for your S3 bucket.

Standard is for frequently accessed data with high performance and availability.
Nearline is for data accessed less frequently (e.g., once a month). Lower cost than Standard, with slight retrieval latency (which means it takes a big longer for you to retrieve your data).
Coldline is for infrequently accessed data (e.g., once a quarter). Even lower cost than Nearline, with even higher retrieval latency.
Archive is for long-term archival data with very infrequent access (e.g., less than once a year). Lowest cost, with the highest retrieval latency (hours).

For our project, Standard class is suitable as we'll be accessing the transferred data.

Select CONTINUE.

Configure Access Control and Protection

Next up in your GCP storage bucket setup, let's configure access control and data protection settings.

Under Choose how to control access to objects, leave the default settings and select CONTINUE.

Extra for Experts: What are these access control settings?

These are the security settings that determine who can access your files in your GCS bucket. They're designed to keep your data secure by default.

Prevent public access means your data won't be accessible to random people on the internet. It's turned on by default because most cloud storage shouldn't be public-facing (unlike, say, website hosting).

Enforce public access prevention is like an extra safety lock - it stops you or someone on your team from accidentally making things public through other settings.

Access control (Uniform) is the simpler approach to permissions. Instead of setting different access rules for every single file (which gets messy fast), you just set permissions at the bucket level and everything inside inherits those rules.

The Fine-grained option gives you more flexibility to say "this person can access this file but not that one." It's more work to manage, but useful if you need that level of control.

Under Choose how to protect object data, untick the Soft delete policy.

Why are we unticking this policy?

You should untick "Soft delete policy" to keep things simple when you're ready to delete your resources. While it's a nice safety net in production (it lets you recover deleted files), we don't really need it for a shorter-term project!

Select CREATE.

In the Public access prevention popup, select CONFIRM.

What is this popup?

This is Google Cloud confirming your bucket's security settings. It's telling you that your bucket will be private by default - no public access!

You'd usually only disable this if you're hosting public content like a website. The Enforce public access prevention checkbox adds extra protection against accidental public sharing in the future - even if someone tries to make one of the objects public, the individual object will be private until you disable this checkbox.

You've created your bucket! Now, click SELECT at the bottom of the page.

Select NEXT STEP.

Let's gooooo! You've set up a new GCP Cloud Storage bucket as your transfer's destination.

Almost there now - let's schedule our transfer job 🗓️

Schedule the Transfer Job

Since we're performing a batch transfer for this project, we only need to run the job once to migrate the data from S3 to GCP Cloud Storage.

In the Choose when to run job section, select Run once and Starting now.

Select NEXT STEP.

Now we're ready to name our transfer job.

In the Identify your job section, enter the description:

Data transfer from S3 bucket to GCP. Created this transfer job as a part of NextWork's multi-cloud project.

Review the other settings in this section, such as Manifest file and How to handle metadata. For this project, we'll leave the default settings.

Extra for Experts: What do these settings control?

These settings determine how your files are transferred from AWS to GCP. The defaults are optimized for a standard migration where you're just moving data without changing it.

Instead of moving all files from your S3 bucket, you can create a manifest file that specifically lists only the files you want to transfer. This option is unchecked by default, so the transfer job will copy everything in your source bucket. If you enabled it, you'd provide a file with exact paths to the specific objects you want to transfer.
The Storage class setting tells the transfer to use whatever storage class you set up in your destination bucket.
For Time created, it's set to not preserve the original creation timestamps from your source files. The transferred files will get new timestamps when they arrive in GCP.

You can also keep the default settings for When to overwrite and When to delete.

Extra for Experts: What are these additional options?

These settings control what happens to files during the transfer:

When to overwrite asks whether the transfer should be skipped for some files if there's an existing file with the same name in the GCS bucket:

Never would skip all files that have the same name, even if the content is different.
Always would replace the GCP file with the S3 version, even if the content is identical, which can waste bandwidth and time (but is a great option if you want to be sure that everything in GCP is exactly as it is in the source).
If different offers a good balance - updating files in GCP that have updated in S3.

When to delete is set to Never by default, which keeps all files in both locations. The other options let you automatically delete files from the source after transfer (good for migrations) or delete files from the destination that aren't at the source (good for keeping buckets in sync).

Finally, we can also keep the default settings for Logging and Notification options.

Extra for Experts: What are these additional options?

Logging options are off by default to avoid extra costs. If enabled, Cloud Logging would track detailed information about each file transferred, useful for troubleshooting but adds costs.

Notification options are also disabled by default. Enabling this would send automated updates about your transfer job through Google's Pub/Sub service, which is helpful for triggering other processes when transfers complete.

Select CREATE.

Ayooooo! This is it! You've created your transfer job.

Now, let's monitor the job and verify that the data is transferred successfully.

Monitor the Transfer Job

In the Storage Transfer Service interface, you should see your newly created job.
Keep an eye out for the Operation status of the job. Wait until the Operation status changes to Success.
This might take some time depending on the amount of data you are transferring, and the regions you've picked for your S3 vs GCP Storage buckets! On average, it should take less than a minute.

Verify Data Transfer

Once the job status is Success, let's check that the files have been transferred to your GCP Cloud Storage bucket.

In the GCP console, select the navigation menu.
Hover over Cloud Storage.
Select Buckets from the left-hand menu.

Click into your Cloud Storage bucket.

Select the Refresh button.

You should see the files from your S3 bucket listed in your GCP Cloud Storage bucket!

WOOHOOO! You've just transferred data from AWS S3 to GCP Cloud Storage.

That's AMAZING!

Secret mission

Want to experience the full power of Storage Transfer Service?

Here's your chance to go beyond just batch transferring all your files at once!

Your secret mission, should you choose to accept it, is to create a manifest file to selectively transfer specific files between AWS and GCP. This advanced technique is essential for enterprise migrations where you need control over what data moves between environments.

In this secret mission, you're going to:

Create a custom manifest file that lists exactly which files to transfer
Configure Storage Transfer Service to use your manifest
Run a targeted transfer that moves only specific files
Showcase advanced multi-cloud migration skills in your documentation!

Transfer Select Files with a Manifest

Delete your resources

It's time to clean up the resources we created - let's avoid any unnecessary costs!

Resources to delete:

Delete the Transfer Jobs in GCP Storage Transfer Service.
Delete the GCP Cloud Storage bucket.
Delete the IAM Role in AWS IAM.
Delete the S3 bucket in AWS S3.
Delete the manifest files from your local computer (if you did the secret mission 💎)

Transfer Jobs

Still in your Storage Transfer job, select DELETE JOB.

Copy and paste the automatically assigned name.

Select DELETE.

Back in the Storage Transfer home page, you can also delete any other transfers you've set up by selecting the checkbox next to their name.

Select DELETE.

Type in the job's assigned name, and select DELETE.

GCP Bucket

In the GCP console, open the navigation menu and hover over Cloud Storage.
Select Buckets from the left-hand menu.

Select your GCP Cloud Storage bucket (nextwork-data-transfer-destination-gcp-yourname).
Select DELETE at the top of the page.

Confirm the deletion by typing the bucket name, and selecting DELETE.

IAM Role

Now let's jump into the AWS Management Console!
Head to the IAM console.
Select Roles from the left-hand menu.
Select the role you created (nextwork-data-transfer-gcp-role).
Select Delete.

Confirm the deletion by typing the role name (nextwork-data-transfer-gcp-role).

Select Delete.

S3 Bucket

In the AWS Management Console, navigate to S3.
Select General purpose buckets from the left-hand menu.

Select the button next to your S3 bucket (nextwork-data-transfer-source-yourname).
Select Empty.
Confirm the deletion by typing permanently delete
Select Delete objects.

Once all the items have been removed, select General purpose buckets from the left-hand menu again.
Select the button next to your bucket, and this time select Delete.

Enter the bucket name, and select Delete bucket.

💎 Manifest file

Remove the file titled manifest.csv from your local computer.

Type confirm and select Delete.

Have you deleted everything?

Don't forget to check off everything in your Resources to delete checklist!

🗑️ Jump to your checklist to double check it now.

Get your documentation!

You've just set up a multi-cloud data transfer solution between AWS and GCP!

You've learned how to:

🪣 Set up AWS S3 and GCP Cloud Storage buckets.
🔑 Set up IAM roles to secure cross-cloud authentication.
🚚 Run a batch data transfer between AWS and GCP.
💎 Implement a selective transfer using a manifest file.

p.s. Does it say "Still tasks to complete!" at the bottom of the screen?

This means you still have screenshots left to upload, or questions left to answer!

Press Ctrl+F (Windows) or Command+F (Mac) on your keyboard.
Search for the text Return to later.
Jump straight to your incomplete tasks!
🙋‍♀️ Still stuck? Ask the community!