Multi-Cloud Healthcare Data Pipeline

Move FHIR patient data from AWS HealthLake to GCP BigQuery with clinical NLP.

Introduction

30 Second Summary

When you visit a specialist, your medical records often need to travel between completely different computer systems. Enterprise health organizations solve this by placing different workloads on different cloud platforms, each chosen for its strengths.

In this project, you will build a multi-cloud healthcare data pipeline that ingests synthetic patient data in AWS, enriches it with clinical NLP, and streams it to Google Cloud for real-time analytics.

What You'll Build

A production-pattern multi-cloud pipeline that moves FHIR R4 patient data from AWS HealthLake through Amazon Comprehend Medical for NLP enrichment, across cloud boundaries, and into GCP BigQuery for SQL analytics.

By the end of this project, you'll have:

  • An AWS HealthLake FHIR R4 Data Store with synthetic patient data, enriched by clinical NLP to extract medications, conditions, and procedures from clinical text.
  • A cross-cloud data transfer pipeline using Google Cloud Storage Transfer Service to move healthcare data securely from AWS to GCP.
  • Real-time BigQuery analytics running SQL queries against patient data imported from AWS, demonstrating workload-specific multi-cloud architecture.
  • Secret Mission: provision the entire AWS-side infrastructure reproducibly using Terraform Infrastructure as Code.

Are there any prerequisites?

You need an AWS account and a GCP account, both with billing enabled. You should also be comfortable running commands in a terminal using the AWS CLI and gcloud CLI.

Set Up AWS HealthLake Data Store

Healthcare systems store patient data in many formats across many hospitals, labs, and clinics. FHIR R4 is the modern standard that gives all of this data a common language, making it interoperable across systems and clouds.

In this step, you will create an AWS HealthLake Data Store that speaks FHIR R4 natively. HealthLake will preload synthetic patient data from Synthea, giving you a realistic dataset to work with immediately.

In this step, get ready to:
  • Create an AWS HealthLake FHIR R4 Data Store preloaded with Synthea synthetic data.
  • Monitor the Data Store until it reaches ACTIVE status.
  • Verify the Data Store by querying its Capability Statement and Patient resources.
Create the HealthLake FHIR R4 Data Store

AWS HealthLake is a HIPAA-eligible service purpose-built for storing and querying healthcare data in FHIR format. You will create a Data Store with the Synthea preload option, which automatically populates it with realistic synthetic patient records.

  • Create your HealthLake FHIR R4 Data Store with Synthea preloaded data by running this command:
aws healthlake create-fhir-datastore \
  --datastore-type-version R4 \
  --datastore-name "synthea-fhir-datastore" \
  --preload-data-config PreloadDataType="SYNTHEA"

What does this command do?

This command provisions a brand new FHIR R4 data store in AWS HealthLake.

  • --datastore-type-version R4 sets the FHIR version to R4, the only version HealthLake supports.
  • --datastore-name "synthea-fhir-datastore" gives your Data Store a descriptive name.
  • --preload-data-config PreloadDataType="SYNTHEA" tells HealthLake to populate the store with open source Synthea FHIR R4 health data, so you have patients to work with immediately.

The command returns a JSON response containing your DatastoreId, DatastoreEndpoint, and DatastoreStatus.

  • Copy and save the DatastoreId value from the output. You will need it for every subsequent command.

Save your Data Store ID

Store the DatastoreId in a shell variable for convenience throughout this project by running:

DATASTORE_ID="your-datastore-id-here"

Replace the placeholder with the actual ID from your create command output.

What is FHIR R4?

FHIR (Fast Healthcare Interoperability Resources) R4 is a standard for exchanging healthcare data electronically. It defines resource types like Patient, Observation, Condition, and Medication that represent clinical concepts in a structured, machine-readable format.

The R4 release is the current normative version, meaning it is stable and production-ready. Both AWS and GCP support FHIR R4, which is why your data can flow between clouds without translation.

Cost warning

AWS HealthLake charges $0.27 per Data Store hour while the Data Store exists. Complete this project and clean up your resources within 2 hours to keep costs under $1 for HealthLake alone.

Seeing an access denied error?

Your IAM user or role needs the healthlake:CreateFHIRDatastore permission. If you are using an admin-level IAM user, this should work automatically. Check that your AWS CLI is configured with valid credentials by running aws sts get-caller-identity.

help me fix an access denied error when creating an AWS HealthLake Data Store

Monitor the Data Store creation status

HealthLake takes approximately 5-10 minutes to provision a Data Store and preload the Synthea data. You need to wait until the status changes from CREATING to ACTIVE before you can query it.

  • Check the current status of your Data Store by running this command:
aws healthlake describe-fhir-datastore \
  --datastore-id $DATASTORE_ID \
  --query "DatastoreProperties.DatastoreStatus" \
  --output text

What does this command do?

The describe-fhir-datastore command retrieves all properties of your Data Store. The --query flag filters the output to show only the status field, and --output text gives you a clean single-word result.

Possible status values are CREATING, ACTIVE, DELETING, DELETED, and CREATE_FAILED.

  • Re-run the command every couple of minutes until the output shows ACTIVE.
  • Once it shows ACTIVE, retrieve the full Data Store details (including the endpoint URL) by running:
aws healthlake describe-fhir-datastore \
  --datastore-id $DATASTORE_ID

What should I see?

The output is a JSON object with a DatastoreProperties block. Look for DatastoreStatus showing ACTIVE and DatastoreEndpoint showing a URL like https://healthlake.us-east-1.amazonaws.com/datastore/<your-id>/r4/.

Note the DatastoreEndpoint value. This is the base URL you will use to make FHIR API calls against your Data Store.

  • Save the endpoint URL in a shell variable for convenience:
DATASTORE_ENDPOINT="your-datastore-endpoint-here"

Status shows CREATE_FAILED?

This can happen if your AWS region does not support HealthLake. HealthLake is available in a limited number of regions. Try us-east-1 if your current region fails. You can set it by adding --region us-east-1 to your commands.

help me troubleshoot a CREATE_FAILED status on my AWS HealthLake Data Store

Verify the Data Store is operational

Now that your Data Store is active, you will confirm it works by retrieving its FHIR Capability Statement and querying for Patient resources. The Capability Statement describes which FHIR resource types and interactions the Data Store supports.

  • Retrieve the FHIR Capability Statement by running this command:
curl --request GET \
  "${DATASTORE_ENDPOINT}metadata" \
  --aws-sigv4 "aws:amz:$(aws configure get region):healthlake" \
  --user "$(aws configure get aws_access_key_id):$(aws configure get aws_secret_access_key)" \
  --header 'Accept: application/json'

What does this command do?

This sends a GET request to the /metadata endpoint of your FHIR store. In FHIR, the /metadata path returns the server's Capability Statement, which lists every resource type and interaction the server supports.

The --aws-sigv4 flag tells curl to sign the request using AWS Signature Version 4, which is how HealthLake authenticates FHIR API calls.

You should see a large JSON response with resourceType set to CapabilityStatement and a list of supported resource types including Patient, Observation, Condition, and more.

Session tokens

If you are using temporary credentials (SSO or assumed roles), you will also need to pass the session token. Add --header "x-amz-security-token:$(aws configure get aws_session_token)" to the curl command.

  • Now query for Patient resources to confirm the Synthea data was preloaded by running:
curl --request GET \
  "${DATASTORE_ENDPOINT}Patient?_count=5" \
  --aws-sigv4 "aws:amz:$(aws configure get region):healthlake" \
  --user "$(aws configure get aws_access_key_id):$(aws configure get aws_secret_access_key)" \
  --header 'Accept: application/json'

What does this command do?

This queries the Patient resource type with _count=5 to limit results to 5 patients. The response is a FHIR Bundle containing Patient resources from the preloaded Synthea data.

Each Patient resource includes fields like name, gender, birthDate, and address. These are realistic but entirely synthetic records generated by Synthea.

You should see a JSON response with resourceType set to Bundle and an entry array containing Patient objects with names, birth dates, and addresses.

Getting empty results or an error?

If you see zero entries, the Synthea data may still be loading. Wait a few more minutes and try again. If you get a signature error, double-check that your AWS credentials are valid by running aws sts get-caller-identity.

help me troubleshoot FHIR API calls against my AWS HealthLake Data Store

Your HealthLake Data Store is live and serving FHIR data. Next up, you will use Amazon Comprehend Medical to run clinical NLP on this data, extracting medical entities and mapping conditions to standardized codes.

Run Clinical NLP with Amazon Comprehend Medical

Your HealthLake Data Store is loaded with synthetic patient data. The next layer in a healthcare data pipeline is enrichment. Raw clinical text contains valuable medical information, but it's unstructured.

In this step, you will use Amazon Comprehend Medical to run clinical NLP on a sample clinical note. Comprehend Medical extracts structured medical entities (medications, conditions, procedures) and maps them to standardized medical ontologies like ICD-10-CM.

In this step, get ready to:
  • Run detect-entities-v2 on a clinical text sample to extract medical entities.
  • Run infer-icd10-cm to map conditions to ICD-10-CM codes.
  • Understand how HealthLake's integrated NLP generates FHIR resources automatically.
Extract medical entities from clinical text

Clinical notes written by doctors contain medications, diagnoses, and procedures buried in free text. Comprehend Medical's DetectEntities-V2 operation parses this text and categorizes every medical concept it finds.

  • Run the following command to detect medical entities in a representative clinical note:
aws comprehendmedical detect-entities-v2 \
  --text "Patient is a 65-year-old male with a history of type 2 diabetes mellitus treated with metformin 500mg twice daily. He presents with persistent chest pain and shortness of breath. Chest X-ray shows bilateral infiltrates. Started on amoxicillin 875mg for suspected pneumonia. Blood pressure 142/90, heart rate 88bpm."

What does this command do?

The detect-entities-v2 operation scans the clinical text and returns structured JSON with every medical entity it finds.

Each entity is tagged with a Category (like MEDICATION, MEDICAL_CONDITION, or TEST_TREATMENT_PROCEDURE) and a Type (like GENERIC_NAME or DX_NAME). Attributes like dosage, frequency, and route are linked to their parent medication entity.

  • Review the JSON output in your terminal. You should see an Entities array containing multiple detected entities.

Look for entities categorized as MEDICATION (metformin, amoxicillin), MEDICAL_CONDITION (diabetes mellitus, chest pain, pneumonia), and TEST_TREATMENT_PROCEDURE (Chest X-ray). Each entity includes a confidence Score between 0 and 1.

Seeing an AccessDeniedException?

Your IAM user or role needs permissions to call Comprehend Medical. Ensure your credentials have the comprehendmedical:DetectEntitiesV2 permission attached.

Also confirm your configured region supports Comprehend Medical. Try adding --region us-east-1 to the command if your default region is unsupported.

Help me debug an AccessDeniedException from Amazon Comprehend Medical

Map conditions to ICD-10-CM codes

Detecting entities is useful, but healthcare systems need standardized coding. ICD-10-CM is the international classification system used for billing, research, and population health. Comprehend Medical can automatically link detected conditions to their ICD-10-CM codes.

  • Run the following command to map conditions in the same clinical text to ICD-10-CM codes:
aws comprehendmedical infer-icd10-cm \
  --text "Patient is a 65-year-old male with a history of type 2 diabetes mellitus treated with metformin 500mg twice daily. He presents with persistent chest pain and shortness of breath. Chest X-ray shows bilateral infiltrates. Started on amoxicillin 875mg for suspected pneumonia. Blood pressure 142/90, heart rate 88bpm."

What does this command do?

The infer-icd10-cm operation focuses specifically on medical conditions in the text. For each condition it detects, it returns a ranked list of matching ICD-10-CM concept codes with descriptions and confidence scores.

For example, "type 2 diabetes mellitus" maps to ICD-10-CM code E11.9 (Type 2 diabetes mellitus without complications). "Chest pain" maps to codes in the R07 family. This ontology linking transforms free-text notes into coded, queryable data.

  • Review the output. You should see an Entities array where each entity has a Category of MEDICAL_CONDITION and an ICD10CMConcepts array listing matched codes.
  • Look for entities like "type 2 diabetes mellitus", "chest pain", "shortness of breath", and "pneumonia", each with their corresponding ICD-10-CM Code and Description fields.

Seeing empty Entities in the output?

The infer-icd10-cm operation only detects medical conditions, not medications or procedures. If you modified the sample text, ensure it contains at least one clearly stated medical condition (a diagnosis, symptom, or disease name).

Help me understand why my infer-icd10-cm output is empty

Understand HealthLake's integrated NLP pipeline

You just ran Comprehend Medical manually on a single text sample. Inside HealthLake, this same NLP pipeline runs automatically whenever a DocumentReference resource is ingested.

How does HealthLake's integrated NLP work?

When you store a clinical note as a FHIR DocumentReference in HealthLake, the service automatically runs DetectEntities-V2, InferICD10-CM, and InferRxNorm on the document text.

HealthLake then generates new FHIR resources from the NLP results. Detected conditions become Condition resources. Detected medications become MedicationStatement resources. Test results become Observation resources. These generated resources are tagged with SYSTEM_GENERATED so you can distinguish them from clinician-authored data.

This means your Synthea data in HealthLake already has NLP-enriched resources available. The manual Comprehend Medical commands you just ran demonstrate the exact same analysis that HealthLake performs behind the scenes on every DocumentReference.

You now have hands-on experience with clinical NLP. Next, you will export the FHIR data from HealthLake and transfer it cross-cloud to GCP.

Export FHIR Data and Transfer to GCP

In the previous step, you ran clinical NLP on patient data and saw how Amazon Comprehend Medical extracts structured medical entities from unstructured text.

Now it is time to move your FHIR data from AWS into GCP. You will export the data from HealthLake to an S3 bucket, then use Google Cloud Storage Transfer Service to pull it across cloud boundaries into a GCS bucket. This is the cross-cloud bridge that connects your AWS ingestion layer to your GCP analytics layer.

In this step, get ready to:
  • Export FHIR data from HealthLake to an S3 bucket in NDJSON format.
  • Create a GCS bucket and configure cross-cloud credentials.
  • Transfer the FHIR files from S3 to GCS using Storage Transfer Service.
Export FHIR data from HealthLake to S3

HealthLake's export operation writes all FHIR resources from your Data Store to an S3 bucket in NDJSON (Newline Delimited JSON) format. This is the standard format for bulk FHIR data exchange.

Before you can export, you need an S3 bucket to receive the data and an IAM role that grants HealthLake permission to write to it.

  • Store your AWS account ID in an environment variable by running this command:
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
echo $ACCOUNT_ID
  • Create the S3 export bucket by running this command:
aws s3 mb s3://healthlake-fhir-export-${ACCOUNT_ID}
  • Create an IAM role that allows HealthLake to write to your S3 bucket by running these commands:
# Create the trust policy for the HealthLake service
cat > trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "healthlake.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# Create the IAM role
aws iam create-role \
  --role-name HealthLakeExportRole \
  --assume-role-policy-document file://trust-policy.json
  • Attach a policy granting the role permission to write to your S3 bucket by running this command:
# Create the S3 access policy
cat > s3-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["s3:ListBucket", "s3:GetBucketPublicAccessBlock", "s3:GetEncryptionConfiguration"],
      "Resource": ["arn:aws:s3:::healthlake-fhir-export-${ACCOUNT_ID}"],
      "Effect": "Allow"
    },
    {
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::healthlake-fhir-export-${ACCOUNT_ID}/*"],
      "Effect": "Allow"
    },
    {
      "Action": ["kms:DescribeKey", "kms:GenerateDataKey*"],
      "Resource": ["*"],
      "Effect": "Allow"
    }
  ]
}
EOF

# Attach the policy to the role
aws iam put-role-policy \
  --role-name HealthLakeExportRole \
  --policy-name HealthLakeS3Export \
  --policy-document file://s3-policy.json

What do these permissions do?

HealthLake requires an IAM role with specific S3 permissions to export data. The trust policy allows the healthlake.amazonaws.com service to assume the role.

The S3 policy grants three types of access: listing the bucket (to check it exists), reading the bucket's encryption config (to encrypt exported data), and writing objects (the actual FHIR NDJSON files).

  • Start the HealthLake export job by running this command:
aws healthlake start-fhir-export-job \
  --output-data-config '{"S3Configuration": {"S3Uri": "s3://healthlake-fhir-export-'${ACCOUNT_ID}'/export/", "KmsKeyId": "arn:aws:kms:us-east-1:'${ACCOUNT_ID}':key/aws/s3"}}' \
  --datastore-id $(aws healthlake list-fhir-datastores --query 'DatastorePropertiesList[?DatastoreName==`synthea-fhir-datastore`].DatastoreId' --output text) \
  --data-access-role-arn arn:aws:iam::${ACCOUNT_ID}:role/HealthLakeExportRole

What does this command do?

  • The --output-data-config tells HealthLake where to write the exported NDJSON files and which KMS key to use for encryption.
  • The --datastore-id is fetched dynamically from your synthea-fhir-datastore Data Store.
  • The --data-access-role-arn specifies the IAM role that grants HealthLake permission to write to S3.

The command returns a JSON response with a JobId and JobStatus of SUBMITTED. Save the JobId value for the next step.

  • Monitor the export job status until it reaches COMPLETED by running this command (replace the job ID with your actual value):
aws healthlake describe-fhir-export-job \
  --datastore-id $(aws healthlake list-fhir-datastores --query 'DatastorePropertiesList[?DatastoreName==`synthea-fhir-datastore`].DatastoreId' --output text) \
  --job-id [[EXPORT_JOB_ID="your export job ID from the previous command"]]
  • Wait for the JobStatus to show COMPLETED. This typically takes 2-5 minutes. Re-run the command if it still shows IN_PROGRESS.
  • Verify the exported files exist in S3 by running:
aws s3 ls s3://healthlake-fhir-export-${ACCOUNT_ID}/export/ --recursive

You should see multiple .ndjson files listed, one per FHIR resource type (Patient, Condition, Observation, etc.).

Export job stuck or failed?

  • If the job stays in SUBMITTED for more than 10 minutes, check that your IAM role trust policy specifies healthlake.amazonaws.com as the principal.
  • If the job shows FAILED, verify that your S3 bucket is in the same AWS region as your HealthLake Data Store.
  • Make sure the KMS key ARN is valid for your region. If unsure, use the default AWS-managed S3 key.

Help me debug my HealthLake export job failure.

Create a GCS bucket and configure transfer credentials

With your FHIR data sitting in S3, you now need a destination on the GCP side. You will create a Google Cloud Storage bucket and set up a credentials file that allows Storage Transfer Service to read from your AWS S3 bucket.

  • Store your GCP project ID in an environment variable by running:
export GCP_PROJECT=$(gcloud config get-value project)
echo $GCP_PROJECT
  • Create a GCS bucket to receive the FHIR data by running this command:
gcloud storage buckets create gs://healthlake-fhir-gcs-${GCP_PROJECT} --location=us-central1

You should see a message confirming the bucket was created.

Bucket name already exists?

GCS bucket names are globally unique. If you see a 409 error, append a random suffix to the bucket name and update subsequent commands accordingly.

Help me resolve a GCS bucket naming conflict.

  • Now you need AWS credentials that Storage Transfer Service can use to read from your S3 bucket. Create a credentials file by running:
# Get your AWS access key details
cat > aws-creds.json << EOF
{
  "access_key_id": "$(aws configure get aws_access_key_id)",
  "secret_access_key": "$(aws configure get aws_secret_access_key)"
}
EOF

What is this credentials file?

Storage Transfer Service needs to authenticate with AWS to read from your S3 bucket. The aws-creds.json file provides your AWS access key ID and secret access key in the format that Storage Transfer Service expects.

This file stays on your local machine and is passed to the transfer command. It is not uploaded anywhere.

Keep your credentials safe

The aws-creds.json file contains your AWS secret access key. Do not commit it to version control or share it. You will delete it during cleanup.

Execute the cross-cloud transfer and verify

Storage Transfer Service is Google's production-grade tool for moving data between cloud providers. It handles the cross-cloud transfer without you needing to download and re-upload files manually.

Why Storage Transfer Service instead of manual copy?

Storage Transfer Service is the recommended approach for moving data between clouds. It handles retries, checksums, and parallel transfers automatically.

Transfers from other cloud providers are free on the GCP side. You only pay AWS egress charges, which are negligible for this small dataset (under $0.01).

  • Start the cross-cloud transfer from S3 to GCS by running this command:
gcloud transfer jobs create \
  s3://healthlake-fhir-export-${ACCOUNT_ID}/export/ \
  gs://healthlake-fhir-gcs-${GCP_PROJECT} \
  --source-creds-file="aws-creds.json"

What does this command do?

  • The first argument is the source S3 path containing your exported FHIR NDJSON files.
  • The second argument is the destination GCS bucket.
  • The --source-creds-file flag points to the JSON file with your AWS credentials so Storage Transfer Service can authenticate with S3.

The command outputs a transfer job name and begins the transfer. For the small Synthea dataset, this typically completes within 1-2 minutes.

  • Verify the FHIR NDJSON files arrived in your GCS bucket by running:
gcloud storage ls gs://healthlake-fhir-gcs-${GCP_PROJECT}/

You should see the NDJSON files listed in the output. These are the same FHIR resources that were in your S3 bucket, now sitting in GCS and ready for import into the Cloud Healthcare API.

Transfer job shows no files?

  • Verify the S3 path is correct by re-running aws s3 ls on your export bucket. The path must include the trailing /export/ prefix.
  • Check that aws-creds.json contains valid credentials. Run cat aws-creds.json to confirm both fields have values (not empty strings).
  • If the transfer job was created but shows errors, run gcloud transfer jobs describe with your job name to see detailed error messages.

Help me troubleshoot my Storage Transfer Service job.

Your FHIR data has successfully crossed cloud boundaries. The NDJSON files are now sitting in GCS, ready to be imported into GCP's Cloud Healthcare API with real-time BigQuery streaming. Next up, you will set up the analytics layer.

Set Up GCP Cloud Healthcare API with BigQuery Streaming

Your FHIR data has landed safely in Google Cloud Storage. Now it is time to set up the GCP side of your pipeline.

In this step, you will create a Cloud Healthcare API FHIR store with real-time BigQuery streaming enabled. Every FHIR resource that enters the store will automatically appear as a queryable row in BigQuery within seconds.

In this step, get ready to:
  • Create a GCP Healthcare dataset and FHIR R4 store with BigQuery streaming.
  • Grant the Cloud Healthcare Service Agent BigQuery permissions and import FHIR data.
  • Verify the import and confirm FHIR resources are accessible.
Enable the Cloud Healthcare API and create the dataset and FHIR store

GCP organizes healthcare resources in a hierarchy. A Healthcare dataset is a container that holds one or more data stores (FHIR stores, DICOM stores, etc.).

  • Enable the Cloud Healthcare API for your project by running this command:
gcloud services enable healthcare.googleapis.com
  • Create a Healthcare dataset named fhir-analytics-dataset in us-central1 by running:
gcloud healthcare datasets create fhir-analytics-dataset \
  --location=us-central1

What does this command do?

This creates a Healthcare dataset, which is the top-level container for all your FHIR, DICOM, and HL7v2 stores in a given region. You must create a dataset before creating any data stores inside it.

  • You should see output confirming the dataset was created:
Created dataset [fhir-analytics-dataset].
  • Create a FHIR R4 store named synthea-fhir-store inside the dataset by running:
gcloud healthcare fhir-stores create synthea-fhir-store \
  --dataset=fhir-analytics-dataset \
  --location=us-central1 \
  --version=R4

What does this command do?

This creates a FHIR store that accepts FHIR R4 resources. The store sits inside your fhir-analytics-dataset and will hold the patient data you imported from AWS.

  • Confirm you see output like:
Created fhirStore [synthea-fhir-store].

Next, you need to configure BigQuery streaming so that every FHIR resource written to this store is automatically streamed to BigQuery.

  • Create a BigQuery dataset named fhir_analytics to receive the streamed data by running:
bq mk --dataset --location=us-central1 fhir_analytics
  • Configure BigQuery streaming on the FHIR store using the REST API. The gcloud CLI does not support this action, so you will use curl instead. Run this command (replace PROJECT_ID with your GCP project ID):
curl -X PATCH \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "streamConfigs": [
      {
        "bigqueryDestination": {
          "datasetUri": "bq://[[GCP_PROJECT_ID="your-gcp-project-id"]].fhir_analytics",
          "schemaConfig": {
            "schemaType": "ANALYTICS_V2"
          }
        }
      }
    ]
  }' \
  "https://healthcare.googleapis.com/v1/projects/[[GCP_PROJECT_ID="your-gcp-project-id"]]/locations/us-central1/datasets/fhir-analytics-dataset/fhirStores/synthea-fhir-store?updateMask=streamConfigs"

What does this command do?

This PATCH request updates your FHIR store to enable BigQuery streaming. Every time a FHIR resource is created, updated, or deleted in the store, the change is automatically streamed to the fhir_analytics BigQuery dataset.

The ANALYTICS_V2 schema type creates one BigQuery table per FHIR resource type (Patient, Observation, Condition, etc.) with a flattened, query-friendly schema.

  • Confirm the response JSON includes streamConfigs with your BigQuery destination. The streaming config is now active.

Getting a permission denied error?

Make sure you are authenticated with an account that has the roles/healthcare.fhirStoreAdmin or roles/healthcare.datasetAdmin role on your project. You can verify your active account by running gcloud auth list.

help me troubleshoot a permission denied error when configuring BigQuery streaming on my GCP FHIR store

Grant BigQuery permissions and import FHIR data

Before the Cloud Healthcare API can write to BigQuery, you must grant the Cloud Healthcare Service Agent the BigQuery Data Editor role on your project.

  • Find your project number by running:
gcloud projects describe $(gcloud config get-value project) --format="value(projectNumber)"
  • Grant the roles/bigquery.dataEditor role to the Cloud Healthcare Service Agent. Replace PROJECT_NUMBER with the number from the previous command:
gcloud projects add-iam-policy-binding $(gcloud config get-value project) \
  --member=serviceAccount:service-[[PROJECT_NUMBER="your-project-number"]]@gcp-sa-healthcare.iam.gserviceaccount.com \
  --role=roles/bigquery.dataEditor

What is the Cloud Healthcare Service Agent?

The Cloud Healthcare Service Agent is a Google-managed service account that the Cloud Healthcare API uses to interact with other GCP services like BigQuery, Cloud Storage, and Pub/Sub.

Its email follows the pattern service-PROJECT_NUMBER@gcp-sa-healthcare.iam.gserviceaccount.com. Without the bigquery.dataEditor role, streaming to BigQuery will silently fail.

  • Now import the FHIR NDJSON files from your GCS bucket into the FHIR store by running:
gcloud healthcare fhir-stores import gcs synthea-fhir-store \
  --dataset=fhir-analytics-dataset \
  --location=us-central1 \
  --gcs-uri=gs://[[GCS_BUCKET="healthlake-fhir-gcs-your-project-id"]]/*.ndjson \
  --content-structure=RESOURCE

What does this command do?

This imports all .ndjson files from your GCS bucket into the FHIR store. The --content-structure=RESOURCE flag tells the import that each line in the NDJSON files is a standalone FHIR resource.

Because BigQuery streaming is enabled, each imported resource will also be streamed to BigQuery automatically. There may be a lag of a few dozen seconds before the data appears in BigQuery.

The import is a long-running operation. Wait for the command to complete. You will see an operation ID and then a success message.

Import taking a long time?

FHIR imports can take several minutes depending on dataset size. If the command appears to hang, it is polling the operation status. Do not interrupt it.

If the import fails with a permission error on the GCS bucket, grant the Cloud Healthcare Service Agent the roles/storage.objectViewer role on your project.

help me troubleshoot a failed FHIR import from GCS into my Cloud Healthcare API FHIR store

Verify the import and access FHIR resources

With the import complete, confirm that the FHIR resources are now accessible through the FHIR REST API.

  • Query the FHIR store for Patient resources by running:
curl -s \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://healthcare.googleapis.com/v1/projects/[[GCP_PROJECT_ID="your-gcp-project-id"]]/locations/us-central1/datasets/fhir-analytics-dataset/fhirStores/synthea-fhir-store/fhir/Patient?_count=3" | python3 -m json.tool

What should I see?

You should see a FHIR Bundle JSON response containing Patient resources from your Synthea data. The response includes patient names, birth dates, addresses, and other demographic information.

The _count=3 parameter limits the results to 3 patients so the output stays readable.

  • You can also check the total number of resources by running:
curl -s \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://healthcare.googleapis.com/v1/projects/[[GCP_PROJECT_ID="your-gcp-project-id"]]/locations/us-central1/datasets/fhir-analytics-dataset/fhirStores/synthea-fhir-store/fhir/Patient?_summary=count" | python3 -m json.tool

The total field in the response shows how many Patient resources were imported. This confirms your multi-cloud pipeline successfully moved data from AWS HealthLake to GCP.

Seeing an empty response or zero results?

Confirm the import operation completed successfully. Re-run the import command without any changes. It will show the operation status.

Also check that your GCS bucket path matches the NDJSON files. You can verify with gcloud storage ls gs://your-bucket/ to list files.

help me debug why my FHIR store shows zero Patient resources after importing from GCS

Your GCP infrastructure is fully operational. The FHIR store is receiving data, streaming it to BigQuery in real time, and serving it via the FHIR REST API. Next up, you will run SQL analytics against this data in BigQuery.

Run Analytics in BigQuery

In the previous step, you imported FHIR data into GCP's Cloud Healthcare API and enabled real-time streaming to BigQuery.

Now it's time to unlock the analytical power that motivated this entire multi-cloud architecture. You'll run SQL queries directly against the FHIR data that flowed from AWS HealthLake, through the cross-cloud transfer, and into BigQuery. This is where GCP's killer differentiator shines: FHIR resources become queryable BigQuery rows automatically, with no ETL pipeline required.

In this step, get ready to:
  • Query the BigQuery Patient table to verify streaming data is present and explore the schema.
  • Write analytics queries that aggregate conditions, count medications, and analyze demographics.
  • Document the end-to-end multi-cloud architecture for your portfolio.
Query the Patient table and explore the schema

The Cloud Healthcare API streams FHIR resources into BigQuery as one table per resource type. A Patient table holds patient demographics, a Condition table holds diagnoses, and so on.

  • Count the total Patient resources in your BigQuery dataset by running this command:
bq query --use_legacy_sql=false \
  'SELECT COUNT(*) as patient_count
   FROM `fhir_analytics.Patient`'

What does this command do?

The bq query command runs SQL against your BigQuery dataset directly from the terminal.

The --use_legacy_sql=false flag tells BigQuery to use standard SQL syntax. The table path fhir_analytics.Patient references the Patient table inside your fhir_analytics dataset.

You should see a count of patient records returned from the Synthea dataset.

  • Explore the Patient table schema to see what columns are available by running:
bq show --schema --format=prettyjson fhir_analytics.Patient | head -80

What will you see in the schema?

The schema shows how FHIR Patient resources map to BigQuery columns. You'll see fields like id, gender, birthDate, and nested structures like name (containing family and given arrays).

The Cloud Healthcare API automatically flattened the hierarchical FHIR JSON into BigQuery's columnar format. This is the zero-ETL value proposition of the GCP side of your architecture.

  • Query a sample of patient records to see the actual data by running:
bq query --use_legacy_sql=false \
  'SELECT id, gender, birthDate
   FROM `fhir_analytics.Patient`
   LIMIT 5'

You should see a table with patient IDs, gender values, and birth dates from the Synthea synthetic data.

Not seeing any results?

If the Patient table is empty, the FHIR import or BigQuery streaming may not have completed. Return to Step 4 and confirm the import operation finished successfully.

If you see an error about the table not existing, confirm your dataset name is fhir_analytics and that BigQuery streaming was enabled on your FHIR store.

Help me troubleshoot why my BigQuery Patient table has no data from the FHIR import.

Write analytics queries across the Synthea dataset

Now that you've confirmed patient data is flowing, you'll write queries that demonstrate the analytical power of having FHIR data in BigQuery. These queries join across resource-type tables to answer clinical questions.

  • Aggregate patient conditions to find the most common diagnoses by running:
bq query --use_legacy_sql=false \
  'SELECT
     c.code.coding[OFFSET(0)].display AS condition_name,
     COUNT(*) AS patient_count
   FROM `fhir_analytics.Condition` c
   GROUP BY condition_name
   ORDER BY patient_count DESC
   LIMIT 10'

What does this query do?

This query reaches into the Condition table (one row per diagnosed condition) and extracts the human-readable condition name from the nested FHIR coding structure.

The code.coding[OFFSET(0)].display path navigates the FHIR Condition resource's code field, which contains an array of coding entries. Each entry has a display value like "Hypertension" or "Diabetes". Grouping by this field gives you a frequency count of conditions across all patients.

You should see a ranked list of the most common conditions in the Synthea dataset, such as hypertension, diabetes, and respiratory infections.

  • Count medications by type to understand prescribing patterns by running:
bq query --use_legacy_sql=false \
  'SELECT
     m.medicationCodeableConcept.coding[OFFSET(0)].display AS medication_name,
     COUNT(*) AS prescription_count
   FROM `fhir_analytics.MedicationRequest` m
   GROUP BY medication_name
   ORDER BY prescription_count DESC
   LIMIT 10'

What does this query do?

This query targets the MedicationRequest table, which stores every medication prescription as a FHIR resource.

The medicationCodeableConcept.coding[OFFSET(0)].display path extracts the medication's display name. Grouping and ordering by count shows which medications are prescribed most frequently across the synthetic patient population.

  • Analyze patient demographics (gender distribution and average age) by running:
bq query --use_legacy_sql=false \
  'SELECT
     gender,
     COUNT(*) AS patient_count,
     ROUND(AVG(DATE_DIFF(CURRENT_DATE(), CAST(birthDate AS DATE), YEAR)), 1) AS avg_age
   FROM `fhir_analytics.Patient`
   GROUP BY gender'

What does this query do?

This query groups patients by gender and calculates the average age using BigQuery's DATE_DIFF function. It computes the difference in years between today and each patient's birthDate.

This kind of demographic analysis is a common first step in population health analytics. In a real-world scenario, this data originated in AWS HealthLake and is now instantly queryable via standard SQL in GCP.

You should see a breakdown showing male and female patient counts with their average ages from the Synthea data.

Seeing a column not found error?

The exact column names depend on the BigQuery schema type configured during FHIR store setup. If you see a "field not found" error, list the available tables first with bq ls fhir_analytics and check if the table name matches (e.g., Condition vs ConditionView).

Help me fix a column not found error in my BigQuery FHIR analytics query.

Document the end-to-end architecture

The final piece of this project is articulating what you built and why each cloud handles a specific workload. This architectural documentation is what makes this project portfolio-ready.

Why document the architecture?

Enterprise multi-cloud projects require clear justification for which workloads live where. A well-documented architecture diagram and rationale is exactly what hiring managers look for in senior cloud architect portfolios.

Here's a summary of the architecture you've built and the rationale for each cloud's role:

  • AWS handles FHIR ingestion and clinical NLP. HealthLake provides native Synthea preloading, and Comprehend Medical extracts medical entities and ICD-10-CM codes from clinical text.
  • GCP handles analytics and SQL querying. Cloud Healthcare API's BigQuery streaming eliminates a full ETL tier. FHIR resources become BigQuery rows within seconds of being written.
  • Google Cloud Storage Transfer Service bridges the two clouds. It is free for cloud-to-cloud transfers on the GCP side, with only minimal AWS egress charges for small datasets.

Why this workload-specific architecture?

This architecture follows the enterprise "workload-specific" pattern. Rather than forcing everything into one cloud, you place each workload where the cloud provider has the strongest purpose-built service.

AWS HealthLake leads in healthcare partner ecosystem integration and native clinical NLP. GCP BigQuery leads in serverless analytics with its unique FHIR-to-BigQuery streaming that eliminates ETL. Using both together delivers a stronger solution than either alone.

You've now queried real FHIR data that traveled across cloud boundaries, from AWS HealthLake through to GCP BigQuery. Next up, tackle the Secret Mission to codify this entire infrastructure with Terraform.

Secret mission

Infrastructure as Code with Terraform

You built this entire multi-cloud pipeline by hand. But what if you could provision all of it with a single command? In this secret mission, you will write Terraform configuration using the AWSCC and Google providers to define your HealthLake Data Store and GCP Healthcare resources as reproducible infrastructure.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The AWS HealthLake Data Store costs $0.27 per hour while it exists, so costs accrue even when you are not actively using it.

Cost warning

Your AWS HealthLake Data Store charges $0.27/hour while it is in ACTIVE state. If you do not plan to continue working immediately, delete the Data Store to avoid unexpected charges.

GCP resources (Cloud Healthcare API, BigQuery, GCS) have minimal per-operation or per-query costs but should still be cleaned up when you are finished.

Resources you used:

  • AWS HealthLake Data Store (synthea-fhir-datastore).
  • S3 bucket (healthlake-fhir-export-[account-id]).
  • GCP Healthcare API dataset and FHIR store (fhir-analytics-dataset / synthea-fhir-store).
  • GCS bucket (healthlake-fhir-gcs-[project-id]).
  • BigQuery dataset (fhir_analytics).
  • Local files (aws-creds.json, main.tf, Terraform state files).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with the pipeline.

  • Be aware that the HealthLake Data Store charges $0.27/hour continuously while it remains in ACTIVE state.
  • GCP resources incur minimal charges per query or operation.
  • Your local Terraform files and credentials file remain on disk.

Pause - I'll come back to this later

HealthLake does not support pausing a Data Store. The Data Store charges while it exists. Your only option to stop charges is to delete it and recreate it when you return.

  • Delete the AWS HealthLake Data Store (see the Delete tab for the command). You can recreate it later with the same Synthea preload.
  • Keep your GCP resources. Per-operation and per-query charges are negligible when idle.
  • Keep your local files (main.tf, aws-creds.json) so you can pick up where you left off.

Delete - I don't want to use this again

Remove all project resources from both clouds and clean up local files.

Delete AWS resources
  • Delete the HealthLake Data Store by running:
aws healthlake delete-fhir-datastore --datastore-id [[DATASTORE_ID="your HealthLake Data Store ID"]]
  • Empty and delete the S3 bucket by running:
aws s3 rb s3://[[S3_BUCKET="healthlake-fhir-export-your-account-id"]] --force
Delete GCP resources
  • Delete the GCP Healthcare API dataset (this cascading-deletes the FHIR store inside it) by running:
gcloud healthcare datasets delete fhir-analytics-dataset --location=us-central1
  • When prompted to confirm, type Y and press Enter.
  • Delete the GCS bucket and its contents by running:
gcloud storage rm --recursive gs://[[GCS_BUCKET="healthlake-fhir-gcs-your-project-id"]]
  • Delete the BigQuery dataset by running:
bq rm -r -f fhir_analytics
Delete local files
  • Remove the credentials file and Terraform files by running:
rm -f aws-creds.json main.tf
rm -rf .terraform .terraform.lock.hcl terraform.tfstate terraform.tfstate.backup

That's a Wrap!

That's a Wrap!

Nice work! You just built a multi-cloud healthcare data pipeline that spans AWS and GCP. This is the kind of workload-specific architecture that enterprise health systems rely on every day.

You've learned how to:

  • Created an AWS HealthLake FHIR R4 Data Store with synthetic patient data and ran clinical NLP with Amazon Comprehend Medical to extract medical entities and ICD-10-CM codes.
  • Transferred healthcare data cross-cloud using Google Cloud Storage Transfer Service and imported FHIR data into GCP's Cloud Healthcare API with real-time BigQuery streaming.
  • Ran SQL analytics on patient data in BigQuery, aggregating conditions, counting medications by type, and analyzing demographics across the dataset.
  • Secret Mission: Provisioned the entire AWS-side infrastructure as Infrastructure as Code with Terraform using the AWSCC provider.

Ready to quiz yourself?