Deploy a Legal Compliance LLM on AWS

Introduction

30 Second Summary

Legal teams deal with mountains of regulatory questions every day. Imagine giving them an AI endpoint that returns expert-quality compliance guidance in under a second, available as a simple API call.

In this project, you will deploy a quantized large language model on an AWS GPU instance and serve it through a production-ready API that handles concurrent legal compliance queries with autoscaling infrastructure.

What You'll Build

You send a curl request asking about GDPR data retention, and within a second your own hosted LLM returns precise regulatory guidance through a standard OpenAI-compatible endpoint that any existing client library can talk to.

By the end of this project, you'll have:

• A live inference API where you send legal compliance questions and get back expert regulatory guidance in under a second, proving your model is serving real traffic.
• A Ray Serve deployment with autoscaling that handles multiple concurrent requests without degrading, wrapping vLLM with production health checks and routing.
• A load-tested endpoint proving that AWQ 4-bit quantization fits a powerful 8B parameter model into a fraction of the GPU memory while maintaining sub-2-second latency under concurrent load.
• Secret Mission: A custom FastAPI middleware layer that enforces domain-specific request validation and adds compliance audit metadata to every API response.

Are there any prerequisites?

You need a credit or debit card to create an AWS account. The project costs approximately $1 in compute charges for the GPU instance. You will use Windows PowerShell to SSH into the instance. No prior ML or cloud experience is required.

Before We Start

Before We Start...

Before we dive in, let's lock in what you're about to build and why it matters for production ML systems.

Set Up AWS Account and Launch GPU Instance

To serve a large language model, you need GPU hardware. A regular CPU cannot run inference fast enough for production use.

In this step, you will create an AWS account, request access to GPU instances, and launch a g5.xlarge instance with an NVIDIA A10G GPU. This gives you 24GB of VRAM for about $1/hr.

In this step, get ready to:

• Create an AWS account and request GPU quota approval.
• Launch a g5.xlarge GPU instance with the Deep Learning AMI.
• SSH into the instance and confirm the GPU is accessible.

Create an AWS account and request GPU quota

New AWS accounts default to 0 vCPUs for GPU instance families. You need to request a quota increase before you can launch a GPU instance.

• Open your browser and navigate to aws.amazon.com.
• Click Create an AWS Account in the top right corner.
• Follow the sign-up wizard to enter your email, create a password, and add a payment method (credit or debit card).
• Complete the identity verification and select the Basic (Free) support plan.

Once your account is ready, you will request a GPU quota increase.

• Sign into the AWS Management Console.
• In the search bar at the top, type Service Quotas and select it from the results.
• In the navigation pane, click AWS services.
• Search for Amazon Elastic Compute Cloud (Amazon EC2) and select it.
• In the filter field, type Running On-Demand G and VT to locate the quota.
• Click Running On-Demand G and VT instances to open its details page.
• Click Request increase at account-level.
• For Increase quota value, enter 4 (the g5.xlarge has 4 vCPUs).
• Click Request.

Why 4 vCPUs?

AWS GPU quotas are measured in vCPUs, not instance count. The g5.xlarge has 4 vCPUs, so requesting a quota of 4 gives you exactly enough to launch one instance.

💡 How long does approval take?

Smaller quota increases are often auto-approved within 15-30 minutes for new accounts. You can check the status by clicking Dashboard in the Service Quotas navigation pane.

While you wait, continue reading ahead to understand the next steps. Do not proceed to launch the instance until the quota status shows as approved.

📣 Cost reminder

The g5.xlarge costs approximately $1.006/hr while running. You will terminate this instance at the end of the project. If you forget, charges continue to accumulate.

Launch a g5.xlarge GPU instance

Once your quota request is approved, you can launch the GPU instance. The Deep Learning AMI comes pre-installed with CUDA drivers and Python, saving significant setup time.

• In the AWS console search bar, type EC2 and select EC2 from the results.
• Click Launch instance from the dashboard.
• Under Name and tags, enter llm-inference as the instance name.
• Under Application and OS Images (Amazon Machine Image), click Browse more AMIs.
• In the search bar, type Deep Learning Base OSS Nvidia Driver GPU AMI (Ubuntu 22.04) and press Enter.
• Select the AMI from the results and click Select.

What is this AMI?

An AMI (Amazon Machine Image) is a pre-configured operating system template. This specific AMI includes Ubuntu 22.04 with NVIDIA CUDA drivers already installed. It saves you from manually installing GPU drivers, which can take 30+ minutes.

• Under Instance type, search for g5.xlarge and select it.
• Under Key pair (login), click Create new key pair.
• Enter llm-key as the key pair name.
• Select RSA as the key pair type.
• Select .pem as the private key file format.
• Click Create key pair. The llm-key.pem file downloads to your Downloads folder automatically.

Keep your .pem file safe

This file is your only way to SSH into the instance. If you lose it, you cannot connect. Do not move it from your Downloads folder for now.

• Under Network settings, click Edit.
• Confirm that Allow SSH traffic from is checked.
• In the source dropdown, select My IP to restrict SSH access to your current IP address only.
• In the Summary panel on the right, confirm your settings: g5.xlarge instance type, your AMI, and your key pair.
• Click Launch instance.

AWS begins provisioning your instance. This takes 1-2 minutes.

• Click View all instances to go to the Instances page.
• Wait until the Instance state column shows Running and the Status check column shows 2/2 checks passed.

Instance won't launch?

If you see an error about insufficient capacity or vCPU limits, your quota increase has not been approved yet. Check Service Quotas > Dashboard for the request status. Wait for approval before trying again.

If you see a different region error, make sure you are launching in the same region where you requested the quota increase.

help me troubleshoot my EC2 g5.xlarge launch failure

SSH into the instance and verify GPU access

Your instance is running. Now you will connect to it over SSH from Windows PowerShell and confirm the NVIDIA GPU is available.

• On the EC2 Instances page, click on your llm-inference instance to see its details.
• Copy the value from the Public IPv4 address field. You will need this for the SSH command.

• Press the Windows key to open your search bar.
• Type PowerShell and press Enter to open it.
• Connect to your instance by running this command:

ssh -i "$HOME\Downloads\[[KEYFILE="llm-key.pem"]]" ubuntu@[[EC2_IP="your instance public IPv4 address"]]

What does this command do?

• ssh opens a secure shell connection to a remote server.
• -i specifies the private key file for authentication.
• ubuntu@ is the default username on the Deep Learning AMI.
• The IP address after @ is your instance's public address.

• PowerShell will ask if you want to continue connecting (host key fingerprint verification). Type yes and press Enter.

You should now see a command prompt starting with ubuntu@ followed by an IP address. This confirms you are inside the remote instance.

Before you run the GPU check, what do you think nvidia-smi will show on a g5.xlarge instance?

• Verify the GPU is accessible by running this command:

nvidia-smi

You should see a table displaying the NVIDIA A10G GPU with 24GB (24576 MiB) of memory. This confirms your instance has full GPU access.

Don't see the GPU?

If you see a "command not found" error, you may have selected the wrong AMI. The Deep Learning Base OSS Nvidia Driver GPU AMI includes NVIDIA drivers pre-installed. Terminate this instance and launch a new one with the correct AMI.

If you see a driver/library version mismatch warning, run sudo reboot and reconnect after 30 seconds.

help me troubleshoot nvidia-smi not working on my EC2 g5.xlarge

Your GPU instance is live and ready. Next up, you will deploy the quantized LLM onto this GPU using vLLM.

Deploy the Model with vLLM

Your GPU instance is live and confirmed. Now you need an inference engine that can load the quantized model weights onto the A10G and serve them through an API.

vLLM is a high-performance inference engine built specifically for serving LLMs. It handles AWQ quantized weights natively, uses optimized attention kernels, and exposes an OpenAI-compatible API out of the box. Running it standalone first proves the model loads correctly before you add Ray Serve on top in the next step.

In this step, get ready to:

• Install vLLM with CUDA support and launch the model server.
• Verify the model is loaded by querying the models endpoint.
• Test inference with a legal compliance prompt.

Install vLLM with CUDA support

The Deep Learning Base AMI includes CUDA drivers and Python, but vLLM itself is not pre-installed. You need to install it with the CUDA 12.9 variant so it can access the GPU.

• Install vLLM by running this command in your SSH session:

pip install "vllm[cu129]"

What does this command do?

The vllm[cu129] package installs vLLM along with the CUDA 12.9 compute libraries it needs to run inference on the A10G GPU. This is a large install (several GB) and takes 1-2 minutes.

You should see pip downloading and installing packages, finishing with a success message.

Seeing a dependency conflict or out-of-memory error?

If pip reports a conflict, try running pip install --upgrade pip first, then re-run the install. If you see a disk space error, the NVMe local storage on g5.xlarge has 250 GB, so this is unlikely unless other large packages are present.

Still stuck? Help me resolve a pip install error for vLLM on my EC2 GPU instance.

Launch the vLLM server with the AWQ model

With vLLM installed, you can now start the inference server. The server will download the AWQ-quantized Llama model from Hugging Face and load it onto the GPU.

• Launch the vLLM server in the background by running this command:

vllm serve hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 --dtype half --max-model-len 4096 --gpu-memory-utilization 0.85 &

What does this command do?

• vllm serve starts vLLM's OpenAI-compatible HTTP server on port 8000.
• hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 is the 4-bit AWQ-quantized Meta Llama 3.1 8B Instruct model hosted on Hugging Face.
• --dtype half sets FP16 computation, which is the recommended setting for AWQ quantization.
• --max-model-len 4096 caps the maximum sequence length to 4096 tokens, keeping VRAM usage predictable.
• --gpu-memory-utilization 0.85 tells vLLM to use up to 85% of GPU memory for the model and KV cache.
• The trailing & runs the process in the background so you keep control of the terminal.

📣 This takes 2-3 minutes

vLLM will first download the model weights from Hugging Face (about 4 GB), then load them onto the GPU. You will see a progress bar for the download followed by initialization logs. Wait until you see a line indicating the server is ready before running the next command.

Before you verify, predict: what do you think the /v1/models endpoint will return if the model loaded successfully?

• Confirm the model is loaded by running:

curl http://localhost:8000/v1/models

You should see a JSON response containing a data array with an entry whose id field is hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4. This confirms vLLM has the model loaded and is ready to serve requests.

Getting "Connection refused" or no output?

The model may still be downloading or loading. Wait another minute and try the curl command again. You can check if the vLLM process is running with jobs to see background processes.

If you see no background job listed, the server may have crashed during startup. Run the vLLM serve command again without & to see the full error output in the foreground.

Need more help? Help me troubleshoot vLLM not starting on my g5.xlarge instance.

Test with a legal compliance prompt

The model is loaded. Now you will send it a real legal compliance question to verify it produces useful, domain-specific responses through the OpenAI-compatible chat completions endpoint.

• Send a GDPR compliance query by running this command:

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
    "messages": [
      {"role": "system", "content": "You are a legal compliance expert."},
      {"role": "user", "content": "What are the key requirements for data retention under GDPR Article 5(1)(e)?"}
    ],
    "max_tokens": 256,
    "temperature": 0.1
  }'

What does this request do?

• The model field matches the model ID that /v1/models returned.
• The system message sets the model's persona as a legal compliance expert.
• The user message asks a specific GDPR question about data retention.
• Low temperature (0.1) keeps the response focused and deterministic, which is critical for regulatory guidance.

You should see a JSON response with a choices array containing the model's answer about GDPR data retention principles. The response should reference storage limitation and purpose-based retention.

Seeing a timeout or empty response?

The first request after server startup can take a few extra seconds as vLLM warms up CUDA graphs. If you get a timeout, wait 10 seconds and retry. If the response is empty or garbled, verify you copied the JSON payload exactly as shown.

Still seeing issues? Help me debug my vLLM chat completions request.

Stop the standalone vLLM server

You have confirmed that vLLM loads the model correctly and serves quality responses. In the next step, you will wrap vLLM with Ray Serve to add production features like autoscaling. First, stop the standalone server to free the GPU.

• Stop the background vLLM process by running:

kill %1

What does this command do?

The kill %1 command terminates the first background job in your shell session. This frees the GPU memory so Ray Serve can claim it when you deploy in the next step.

Your model is proven. vLLM downloaded the 4-bit AWQ weights, loaded them onto the A10G, and served a legal compliance response through an OpenAI-compatible API. Next, you will wrap this engine with Ray Serve to add autoscaling and production routing.

Add Ray Serve for Production Autoscaling

In the previous step, you proved the model works by running standalone vLLM and getting a legal compliance response. That was a demo.

A production API needs autoscaling, health checks, and request routing. Standalone vLLM has none of these. Ray Serve wraps vLLM with production infrastructure while keeping the same OpenAI-compatible API. This is the gap between a demo and a deployment.

In this step, get ready to:

• Install Ray with serve and LLM extras.
• Create a deployment script that configures autoscaling.
• Verify the Ray Serve deployment responds to compliance queries.

Install Ray with serve and LLM extras

Ray Serve needs the serve and llm extras to deploy LLMs with its built-in OpenAI-compatible serving layer.

• Install Ray by running this command in your SSH session:

pip install "ray[serve,llm]"

What does this install?

The serve extra adds Ray's HTTP serving framework with autoscaling, health checks, and request routing.

The llm extra adds the LLM deployment utilities that integrate with vLLM under the hood.

🙋‍♀️ Installation taking too long?

Ray has many dependencies. The install can take 2-3 minutes on the EC2 instance. If it appears frozen, wait for it to complete.

If you see a permissions error, prefix the command with sudo.

help me debug my Ray installation error

Create the deployment script

The deployment script configures how Ray Serve loads and scales the model. It uses Ray's LLMConfig to define the model source, autoscaling boundaries, and vLLM engine parameters.

• Create deploy.py by opening a new file in nano:

nano deploy.py

• Paste the following code into the editor:

from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app

llm_config = LLMConfig(
    model_loading_config=dict(
        model_id="legal-compliance-llm",
        model_source="hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
    ),
    deployment_config=dict(
        autoscaling_config=dict(
            min_replicas=1,
            max_replicas=2,
        )
    ),
    accelerator_type="A10G",
    engine_kwargs=dict(
        dtype="half",
        max_model_len=4096,
        gpu_memory_utilization=0.85,
    ),
)

app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)

• Save and exit nano by pressing Ctrl+O, then Enter, then Ctrl+X.

What does this code do?

• model_id sets the name your API clients use to reference the model (legal-compliance-llm).
• model_source points to the HuggingFace model that vLLM loads under the hood.
• autoscaling_config tells Ray to keep at least 1 replica running and scale up to 2 when traffic increases.
• accelerator_type ensures Ray schedules the deployment onto the A10G GPU.
• engine_kwargs passes the same vLLM settings you tested in the previous step (FP16 dtype, 4096 context, 85% GPU memory).
• build_openai_app creates an application that exposes standard OpenAI-compatible endpoints.

• Confirm the file was created by running:

ls deploy.py

You should see deploy.py listed in the output.

File not found?

Make sure you pressed Ctrl+O (the letter O, not zero) to write the file in nano before exiting. Run nano deploy.py again and re-paste the code.

help me create deploy.py on my EC2 instance

✔️ Awesome, I've got everything!

Great. Make sure you've saved deploy.py and confirmed it exists with ls.

ⓧ I'd like to double check the full code

Here is the complete deploy.py file. Compare it character-for-character with what you pasted:

from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app

llm_config = LLMConfig(
    model_loading_config=dict(
        model_id="legal-compliance-llm",
        model_source="hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
    ),
    deployment_config=dict(
        autoscaling_config=dict(
            min_replicas=1,
            max_replicas=2,
        )
    ),
    accelerator_type="A10G",
    engine_kwargs=dict(
        dtype="half",
        max_model_len=4096,
        gpu_memory_utilization=0.85,
    ),
)

app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)

Run the deployment

Ray Serve will start a local cluster, load the model through vLLM internally, and expose the OpenAI-compatible endpoints. This takes 2-3 minutes as the model weights load into GPU memory.

• Start the Ray Serve deployment in the background by running:

python deploy.py &

Why the & at the end?

The & sends the process to the background so you get your terminal prompt back. The server keeps running. You will see Ray startup logs printing to your terminal. Press Enter to get a clean prompt once the logs settle.

• Wait 2-3 minutes for the model to load. You will see log lines about downloading model weights and initializing vLLM.
• Once you see a log line mentioning the application is ready or the HTTP proxy is running, press Enter to get your prompt back.
• Verify the model is accessible by running:

curl http://localhost:8000/v1/models

You should see a JSON response listing legal-compliance-llm as an available model.

Getting connection refused?

The model is still loading. Wait another minute and try the curl command again. Ray Serve only starts accepting requests after the model weights are fully loaded into GPU memory.

If you see no background process running, check for errors with jobs. If the job exited, run python deploy.py without & to see the full error output.

help me debug my Ray Serve deployment that won't start

Test with a compliance query and verify autoscaling

The model is now served through Ray Serve with autoscaling configured. Time to prove it handles a real compliance question and confirm the autoscaling infrastructure is active.

• Send a legal compliance query to the Ray Serve endpoint by running:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "legal-compliance-llm",
    "messages": [
      {"role": "system", "content": "You are a legal compliance expert."},
      {"role": "user", "content": "What are the key requirements for GDPR data breach notification?"}
    ],
    "max_tokens": 128,
    "temperature": 0.1
  }'

You should see a JSON response containing a detailed answer about GDPR breach notification requirements. The response follows the standard OpenAI chat completions format with choices, usage, and model fields.

Same API, production infrastructure underneath

Notice the request format is identical to what you sent to standalone vLLM in the previous step. The difference is invisible to the client: Ray Serve now handles autoscaling, health checks, and request routing behind the same endpoint.

• Check the Ray cluster status by running:

ray status

You should see output showing your node with the A10G GPU resource and the serve replica running.

• Before you run the next command, what do you expect to see listed as actors in the Ray cluster?
• Confirm the autoscaling config is active by running:

ray summary actors

You should see actor classes listed with their state counts showing ALIVE actors. These include the serve controller, HTTP proxy, and your model deployment replica.

Ray commands not found?

Make sure you installed Ray with pip install "ray[serve,llm]" earlier in this step. If you used a virtual environment, confirm it is still activated.

help me troubleshoot ray CLI commands not working

Your LLM is now running behind production infrastructure with autoscaling configured. Next up, you will fire concurrent requests at the endpoint to prove it handles real traffic patterns under load.

Load Test and Validate Production Readiness

Your Ray Serve deployment is live and responding to individual requests. A single successful curl proves the model works, but production traffic does not arrive one request at a time.

In this step, you will fire five compliance queries simultaneously to prove the API handles concurrency without degrading. You will measure per-request latency and compute p50/p99 percentiles to quantify production readiness.

In this step, get ready to:

• Write a concurrent load test script using the OpenAI Python client.
• Fire 5 simultaneous legal compliance queries across different regulatory domains.
• Validate that all requests succeed with p50 latency under 2 seconds.

Write the load test script

The load test uses the openai Python package as its client because Ray Serve exposes standard OpenAI-compatible endpoints. Any tool built for the OpenAI API works here without modification.

• Install the OpenAI Python client by running:

pip install openai

• Confirm the installation succeeded by running:

python -c "from openai import OpenAI; print('openai ready')"

You should see openai ready printed with no errors.

Seeing ModuleNotFoundError?

Make sure you ran pip install openai in the same SSH session where your Python environment is active. The Deep Learning AMI may have multiple Python versions. Run which python to confirm you are using the expected interpreter.

Help me debug why the openai package isn't importing on my EC2 instance.

• Create the load test file by running:

nano load_test.py

• Paste the following imports, client configuration, and prompt list into load_test.py:

import time
import concurrent.futures
from openai import OpenAI

# Point the OpenAI client at the local Ray Serve endpoint
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

# Five legal compliance queries covering different regulatory domains
prompts = [
    {"domain": "GDPR", "question": "What are the key requirements for data retention under GDPR Article 5(1)(e)?"},
    {"domain": "SOX", "question": "What internal controls are required for financial reporting under SOX Section 404?"},
    {"domain": "HIPAA", "question": "What constitutes a breach notification obligation under HIPAA?"},
    {"domain": "AML", "question": "What are the customer due diligence requirements under the Bank Secrecy Act?"},
    {"domain": "PCI-DSS", "question": "What are the requirements for encryption of cardholder data in transit under PCI-DSS?"},
]

What does this code do?

• The OpenAI client is pointed at localhost:8000 instead of the real OpenAI API. Ray Serve speaks the same protocol, so the client works without any code changes.
• The api_key is set to "not-needed" because your local deployment has no authentication layer.
• The five prompts cover GDPR, SOX, HIPAA, AML, and PCI-DSS. Each represents a distinct regulatory domain a compliance team might query.

• Below the prompts list, add the function that sends a single request and measures its latency:

def send_request(prompt):
    start = time.time()
    response = client.chat.completions.create(
        model="legal-compliance-llm",
        messages=[
            {"role": "system", "content": "You are a legal compliance expert. Provide precise, actionable regulatory guidance."},
            {"role": "user", "content": prompt["question"]},
        ],
        max_tokens=256,
        temperature=0.1,
    )
    elapsed = time.time() - start
    # Return domain, latency, a preview of the response, and token count
    return {
        "domain": prompt["domain"],
        "latency": elapsed,
        "response": response.choices[0].message.content[:100],
        "tokens": response.usage.total_tokens,
    }

What does this function do?

• send_request wraps a single chat completion call with timing. It records the wall-clock duration so you can measure real latency from request to response.
• model is set to "legal-compliance-llm". This matches the model_id you configured in deploy.py.
• temperature=0.1 keeps responses deterministic. For legal compliance, you want consistent answers, not creative ones.

• Save the file temporarily (Ctrl+O, Enter, Ctrl+X in nano) and verify the syntax loads cleanly by running:

python -c "import load_test; print(f'Loaded {len(load_test.prompts)} prompts, client targeting {load_test.client.base_url}')"

You should see Loaded 5 prompts, client targeting http://localhost:8000/v1 printed. This confirms your imports, client config, and prompts are all syntactically correct.

Seeing a SyntaxError?

Check for missing commas between items in the prompts list. Each dictionary must end with }, except the last one. Also confirm you have matching quotes around every string.

Help me fix a syntax error in my load_test.py file.

• Reopen the file to add the main execution block by running:

nano load_test.py

• At the bottom of load_test.py, below the send_request function, add the main block that fires all requests concurrently and prints results:

if __name__ == "__main__":
    print("Running load test with 5 concurrent compliance queries...\n")

    # Fire all 5 requests simultaneously using a thread pool
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        futures = [executor.submit(send_request, p) for p in prompts]
        results = [f.result() for f in concurrent.futures.as_completed(futures)]

    # Sort latencies for percentile calculations
    latencies = [r["latency"] for r in results]
    latencies.sort()

    # Print per-request results
    print(f"{'Domain':<10} {'Latency':<10} {'Tokens':<8} {'Response Preview'}")
    print("-" * 70)
    for r in results:
        print(f"{r['domain']:<10} {r['latency']:.2f}s     {r['tokens']:<8} {r['response']}...")

    # Print summary statistics
    print(f"\n--- Results ---")
    print(f"All requests successful: {len(results)}/5")
    print(f"p50 latency: {latencies[2]:.2f}s")
    print(f"p99 latency: {latencies[-1]:.2f}s")

What does this block do?

• ThreadPoolExecutor(max_workers=5) launches all five requests at the same time, simulating concurrent users hitting the API.
• concurrent.futures.as_completed collects results as they finish. Faster responses come back first regardless of submission order.
• p50 is the median latency (the middle value when sorted). p99 is the slowest request. Together they tell you what typical and worst-case performance looks like.

• Save and close the file (Ctrl+O, Enter, Ctrl+X).

✔️ Awesome, I've got everything!

Great. Confirm you have saved load_test.py and you are back at the shell prompt.

ⓧ I'd like to double check the full code

Your load_test.py has three sections you built in order. Scroll up and confirm each is present:

• Imports, client configuration, and the 5-item prompts list.
• The send_request function that wraps a single chat completion call with timing.
• The if __name__ == "__main__" block that fires concurrent requests and prints latency stats.

You can view the file without opening an editor by running cat load_test.py.

Run the load test and validate results

AWQ quantization compresses the model to ~4 GB of VRAM, leaving roughly 18 GB free for the KV cache that holds in-flight request context. That headroom is what lets the model serve multiple requests concurrently without running out of GPU memory.

• Before you run this, predict: will all 5 requests succeed, or will some fail under concurrent load? What latency range do you expect for a quantized 8B model generating 256 tokens?

• Run the load test by executing:

python load_test.py

You should see a table of all 5 domains with their latencies, token counts, and response previews. Below the table, the summary reports All requests successful: 5/5 and a p50 latency under 2 seconds.

What should I see?

Each row shows one regulatory domain (GDPR, SOX, HIPAA, AML, PCI-DSS), the wall-clock time for that request, the total tokens used, and the first 100 characters of the LLM response. The model is generating expert-quality compliance guidance across all five domains simultaneously.

A p50 under 2 seconds means half your requests complete in under 2 seconds. The p99 shows worst-case latency. Both values prove the deployment handles concurrent traffic at production quality.

🙋‍♀️ Seeing connection errors or timeouts?

Make sure deploy.py is still running in another terminal or background process. Run ray status to confirm Ray Serve is active. If the deployment was terminated, restart it with python deploy.py & and wait 2-3 minutes for the model to reload.

Help me debug why my load test can't connect to the Ray Serve endpoint.

• Verify the autoscaling configuration is active by running:

ray summary actors

You should see your serve replica listed in the actor summary. This confirms Ray is managing the deployment and ready to scale if traffic increases.

Your deployment just proved it handles real concurrent traffic with consistent sub-2-second latency across five different regulatory domains. Next up, you will build a polished Streamlit interface so users can upload documents and get compliance analysis in a browser.

Build a Demo UI with Streamlit

Your load test proved the API handles concurrent traffic with consistent latency. But showing a colleague a terminal full of JSON output is not a compelling demo.

In this step, you will build a polished Streamlit web interface where users upload legal documents, select a compliance domain, and get formatted analysis back from your LLM. The result is a demo-ready app you can show stakeholders in a browser.

In this step, get ready to:

• Install Streamlit on your GPU instance.
• Build a document upload interface that sends compliance queries to Ray Serve.
• Launch the app and access it via SSH tunnel for a live demo.

Install Streamlit

• Install Streamlit by running:

pip install streamlit

• Confirm the installation succeeded by running:

streamlit --version

You should see a version number (such as 1.59.2) printed with no errors.

Seeing command not found?

Make sure you ran pip install streamlit in the same SSH session and Python environment where you installed vLLM. Run which streamlit to confirm the binary is on your PATH.

Help me debug why Streamlit isn't found after installing it on my EC2 instance.

Build the document upload interface

The app needs three things: a file uploader for legal documents, a compliance domain selector, and a display area for the LLM's formatted analysis. Streamlit handles all of this in a single Python file with no frontend code.

• Create the Streamlit app file by running:

nano demo_app.py

• Paste the page configuration, client setup, and upload interface into demo_app.py:

import streamlit as st
from openai import OpenAI

# Configure the page layout and title
st.set_page_config(
    page_title="Legal Compliance Analyzer",
    page_icon="\u2696\ufe0f",
    layout="wide",
)

# Point the OpenAI client at the local Ray Serve endpoint
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

# App header
st.title("\u2696\ufe0f Legal Compliance Analyzer")
st.markdown("Upload a legal document and get instant compliance analysis.")

# Sidebar: compliance domain selector
with st.sidebar:
    st.header("Configuration")
    domain = st.selectbox(
        "Compliance Domain",
        ["GDPR", "SOX", "HIPAA", "AML", "PCI-DSS"],
    )
    max_tokens = st.slider("Max response length", 128, 512, 256)

# Main area: file uploader
uploaded_file = st.file_uploader(
    "Upload a legal document (.txt or .pdf)",
    type=["txt", "pdf"],
)

What does this code do?

• st.set_page_config sets the browser tab title and uses a wide layout so the analysis has room to breathe.
• The OpenAI client reuses the same localhost:8000 endpoint your Ray Serve deployment exposes. No new backend needed.
• st.file_uploader renders a drag-and-drop area that accepts .txt and .pdf files. The sidebar lets users pick the regulatory domain before running analysis.

• Below the file uploader block, add the analysis logic that reads the document and queries the LLM:

# Process the uploaded document when the user clicks Analyze
if uploaded_file is not None:
    # Read file content based on type
    if uploaded_file.type == "text/plain":
        doc_text = uploaded_file.read().decode("utf-8")
    else:
        # For PDF, extract raw text (basic approach)
        doc_text = uploaded_file.read().decode("utf-8", errors="ignore")

    # Show a preview of the uploaded document
    with st.expander("Document preview", expanded=False):
        st.text(doc_text[:2000])

    if st.button("Analyze for Compliance", type="primary"):
        with st.spinner("Analyzing document..."):
            response = client.chat.completions.create(
                model="legal-compliance-llm",
                messages=[
                    {
                        "role": "system",
                        "content": f"You are a {domain} compliance expert. Analyze the following document and identify compliance issues, risks, and recommendations.",
                    },
                    {"role": "user", "content": doc_text[:3000]},
                ],
                max_tokens=max_tokens,
                temperature=0.1,
            )

        # Display the formatted analysis
        st.subheader(f"{domain} Compliance Analysis")
        st.markdown(response.choices[0].message.content)

What does this code do?

• When a file is uploaded, the app reads it and shows a collapsible preview. The [:2000] slice keeps the preview manageable for large documents.
• The Analyze for Compliance button triggers the LLM call. st.spinner shows a loading indicator while the model generates its response.
• The document text is truncated to 3000 characters before sending to stay within the model's context window. For a demo, this covers the key sections of most legal documents.

• Save and close the file (Ctrl+O, Enter, Ctrl+X).

✔️ Awesome, I've got everything!

Great. Confirm you have saved demo_app.py and you are back at the shell prompt.

ⓧ I'd like to double check the full code

Your demo_app.py has two sections you built in order. Scroll up and confirm each is present:

• Imports, page config, OpenAI client, header, sidebar with domain selector and token slider, and the file uploader.
• The if uploaded_file is not None block that reads the file, shows a preview, and sends it to the LLM when the button is clicked.

You can view the file without opening an editor by running cat demo_app.py.

Launch the app and access it for a live demo

Streamlit runs a local web server. Because your app is on a remote EC2 instance, you will use an SSH tunnel to access it in your local browser.

• Start the Streamlit app by running:

streamlit run demo_app.py --server.port 8501 --server.headless true &

What does this command do?

• --server.port 8501 runs the app on Streamlit's default port.
• --server.headless true prevents Streamlit from trying to open a browser on the server (which has no display).
• The trailing & runs it in the background so you keep terminal access.

You should see Streamlit report that the app is running on port 8501.

• On your local machine (not the EC2 instance), open a new terminal and create an SSH tunnel by running:

ssh -i [[PEMPATH="path to your .pem file"]] -L 8501:localhost:8501 ubuntu@[[EC2_PUBLIC_IP="your EC2 public IPv4 address"]] -N

What does this command do?

The -L 8501:localhost:8501 flag forwards your local port 8501 to the EC2 instance's port 8501. The -N flag means don't run a remote command, just hold the tunnel open. While this tunnel is active, you can access the Streamlit app in your local browser.

• Open your browser and navigate to:

http://localhost:8501

You should see the Legal Compliance Analyzer interface with the file upload area, sidebar domain selector, and response length slider.

• Test the app by uploading any .txt file (a sample contract, privacy policy, or terms of service work well).
• Select a compliance domain from the sidebar.
• Click Analyze for Compliance.

After a few seconds, the LLM's formatted compliance analysis appears below the button. This is what you show stakeholders.

App not loading in the browser?

• Confirm the SSH tunnel is still running in your local terminal. If it disconnected, re-run the ssh -L command.
• Verify Streamlit is running on the EC2 instance by going back to your SSH session and running curl http://localhost:8501. You should see HTML output.
• If you see a connection refused error locally, make sure nothing else is using port 8501 on your machine.

Help me troubleshoot why my Streamlit app isn't accessible through the SSH tunnel.

Your compliance LLM now has a polished, demo-ready interface. Stakeholders can upload real documents and see analysis in seconds. Next up, the Secret Mission challenges you to add an audit middleware layer that enriches every response with traceability metadata.

Secret mission

Add Compliance Audit Middleware

Real legal compliance APIs need audit trails for regulatory inspection. Can you build a FastAPI middleware layer that validates requests by compliance domain and enriches every response with traceable metadata?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. The g5.xlarge instance charges approximately $1.006/hr while running, so leaving it active will accumulate costs quickly.

Cost warning

Your g5.xlarge instance costs approximately $1.006 per hour. If you walk away without stopping or terminating it, you will be charged around $24 per day. Terminate the instance as soon as you are finished to stop all compute charges.

Resources you used:

• 1x AWS EC2 g5.xlarge instance (approximately $1.006/hr while running).
• 1x EBS volume (attached to instance, deleted by default on termination).
• 1x Security Group (allows SSH on port 22).
• 1x Key Pair (RSA .pem file stored locally in Downloads).

Keep everything running

No action needed. Choose this if you plan to continue experimenting with the deployment or want to try additional compliance prompts.

• Your instance continues to run and charge approximately $1.006/hr.
• Ray Serve and the model remain available at http://localhost:8000 over SSH.
• The .pem key file remains in your Downloads folder for future SSH access.

Pause - I'll come back to this later

Stop the instance to halt compute charges while keeping your files and configuration intact. EBS storage charges continue at approximately $0.10/GB/month, but compute charges stop entirely.

• Return to the EC2 console in your browser.
• In the left navigation pane, select Instances.
• Select your g5.xlarge instance.
• Click Instance state, then Stop instance.
• Click Stop when prompted for confirmation.
• To restart later, select the instance and choose Instance state then Start instance. You will need to SSH in again with the new public IP address shown in the console.

What happens when you stop?

Stopping preserves your EBS volume (all files, the downloaded model, and your scripts). Your instance may move to a different host when restarted, so it will get a new public IP address. The .pem key file still works for SSH access.

Delete - I don't want to use this again

Remove all project resources to bring your AWS charges to $0.

• In the EC2 console, select Instances in the left navigation pane.
• Select your g5.xlarge instance.
• Click Instance state, then Terminate (delete) instance.
• Click Terminate (delete) when prompted for confirmation. The attached EBS volume is deleted automatically with the instance (the default setting for root volumes launched from the console).
• In the left navigation pane, select Security Groups.
• Select the security group you created for this project.
• Click Actions, then Delete security groups. Confirm the deletion.
• In the left navigation pane, select Key Pairs.
• Select the key pair you created for this project.
• Click Actions, then Delete. Type Delete in the confirmation field and confirm.
• Delete the local .pem file from your Windows machine by running this in PowerShell:

Remove-Item "$HOME\Downloads\your-key.pem"

After termination, the instance may remain visible in the console briefly with a terminated state. It will disappear automatically within about an hour. Your AWS charges for this project drop to $0.

That's a Wrap!

That's a Wrap!

Nice work! You just deployed a production-grade LLM inference API on AWS GPU infrastructure, serving legal compliance queries through an autoscaling, OpenAI-compatible endpoint.

You've learned how to:

• Deploy Meta Llama 3.1 8B Instruct AWQ (INT4 quantized) on an AWS g5.xlarge with an NVIDIA A10G GPU, serving it through vLLM with 70% memory reduction vs FP16.
• Wrap inference with Ray Serve for production autoscaling, health checks, and an OpenAI-compatible API that any standard SDK client can connect to.
• Load test with concurrent legal compliance queries across GDPR, SOX, HIPAA, AML, and PCI-DSS domains, proving sub-2-second latency under parallel traffic.
• Secret Mission: Built a compliance audit middleware with FastAPI that adds traceability metadata to every API response, including unique audit IDs, timestamps, and request hashes for regulatory inspection.

Ready to quiz yourself?