Deploy a Knowledge Assistant to AKS
Deploy a containerized AI API to Azure Kubernetes Service with production manifests.
Introduction
30 Second Summary
Running an AI service on your laptop proves it works. Running it on a cloud cluster proves it is ready for production. The gap between those two is where Linux Cloud Engineers live.
In this project, you will deploy your Knowledge Assistant API to a live Azure Kubernetes Service cluster with production-grade manifests, inspect Linux container internals, practice RHEL-family administration, and create a comprehensive operations guide.
What You'll Build
You will curl your Knowledge Assistant's endpoint on a live AKS cluster and get an answer back, proving your container runs identically in the cloud as it does on your machine.
By the end of this project, you'll have:
- A Knowledge Assistant answering questions from a live AKS endpoint, with health probes confirming it is healthy and resource limits protecting the cluster.
- Hands-on experience exec-ing into running containers to inspect Linux internals and practicing Rocky Linux administration skills that enterprise environments demand.
- A professional operations guide covering health checks, observability, troubleshooting runbooks, AI platform architecture, and GPU node pool concepts for your portfolio.
- Secret Mission: Trigger a real pod failure on your AKS cluster and troubleshoot it back to health using kubectl.
Are there any prerequisites?
This project assumes you already have a working knowledge-assistant:latest Docker image built and running locally. You also need Docker Desktop installed and running on your machine.
Before We Start
Before diving into Azure and Kubernetes commands, let's lock in what you're about to deploy and why it matters. This is your moment to connect the dots between what you've already built locally and the production-grade infrastructure you're about to operate.
Set Up Azure CLI, Account, and Container Registry
Your Knowledge Assistant image currently lives only on your local machine. To deploy it on a live AKS cluster in the cloud, that image needs to be accessible from Azure's infrastructure.
In this step, you will install the Azure CLI to control Azure resources from your terminal, create a free Azure account, and set up an Azure Container Registry to host your image where AKS can pull it.
In this step, get ready to:
- Install Azure CLI and sign in to your Azure account.
- Create a resource group and container registry.
- Push your image to ACR and kick off AKS cluster creation.
Install Azure CLI and create your Azure account
The Azure CLI lets you manage Azure resources directly from your terminal. You will install it, create a free account, and sign in.
- Install Azure CLI by running this command in PowerShell:
winget install --exact --id Microsoft.AzureCLIWhat does this command do?
The winget package manager downloads and installs the latest Azure CLI release from Microsoft's official repository.
- Close your PowerShell window completely.
- Open a new PowerShell window so it picks up the updated PATH.
- Verify the installation by running:
az --versionYou should see a version number (such as 2.88.0 or higher) along with a list of installed extensions.
✔️ I see a version number
Azure CLI is ready to go.
ⓧ Command not found
This means your new PowerShell window did not pick up the install.
- Make sure you fully closed the old PowerShell window and opened a brand new one.
- If it still fails, try running the install command again, then restart your computer.
- Confirm winget itself works by running winget --version. If that fails, update your App Installer from the Microsoft Store.
ⓧ I see a version lower than 2.50
You have an older Azure CLI installed. Upgrade it by running:
az upgradeAfter the upgrade completes, run az --version again to confirm you are on a recent version.
Next, you need a free Azure account to create resources.
- Open this link to create a free Azure account: Azure free account sign-up.
- Sign in with your Microsoft or GitHub account.
- Provide your phone number for verification.
- Enter a non-prepaid credit or debit card for identity verification.
Will I be charged?
The free account gives you $200 in Azure credit for 30 days. Your card will not be charged unless you manually upgrade to a paid plan. All resources in this project fit well within the free credit.
- Once your account is ready, sign in from the terminal by running:
az loginWhat happens when you run this?
This command opens your web browser for authentication. After you sign in, the browser confirms success and the terminal displays your subscription details.
If the terminal asks you to select a subscription, press Enter to accept the default.
You should see your subscription name and tenant ID printed in the terminal output.
Browser did not open?
Some environments block browser launch. If the browser does not open automatically, copy the URL shown in the terminal and paste it into your browser manually.
Help me sign in to Azure CLI.
Create a resource group and container registry
Azure organizes resources into groups so you can manage and delete them together. You will create one group to hold everything for this project.
- Create a resource group by running:
az group create --name rg-knowledge-assistant --location eastusYou should see JSON output with "provisioningState": "Succeeded" confirming the group was created in the eastus region.
- Create an Azure Container Registry (Basic tier) inside your resource group by running:
az acr create --resource-group rg-knowledge-assistant --name knowasstacr --sku BasicWhat is a container registry?
A container registry is a cloud-hosted storage location for your Docker images. When AKS needs to run your Knowledge Assistant, it pulls the image from this registry instead of from your local machine.
The Basic tier costs approximately $0.17/day and is included in your free credit.
You should see JSON output with the registry's loginServer value set to knowasstacr.azurecr.io.
ACR names are globally unique
If you see an error saying the name is already taken, choose a different name (e.g. knowasstacr2 or add your initials). Use your chosen name in all later commands wherever you see knowasstacr.
- Log Docker into your new registry by running:
az acr login --name knowasstacrYou should see Login Succeeded in the terminal output.
Seeing a login error?
Make sure Docker Desktop is running. The az acr login command needs Docker's daemon to store credentials.
Help me fix my ACR login.
Push your image to ACR and start AKS cluster creation
With Docker authenticated to your registry, you can now push your Knowledge Assistant image so AKS can access it.
- Tag your local image with the registry address by running:
docker tag knowledge-assistant:latest knowasstacr.azurecr.io/knowledge-assistant:v1- Push the tagged image to ACR by running:
docker push knowasstacr.azurecr.io/knowledge-assistant:v1You should see each image layer being pushed, ending with a digest and size summary.
What did tagging and pushing do?
The docker tag command gave your image a new name that points at your ACR. The docker push command uploaded the image layers to that registry.
Now your image lives in the cloud at knowasstacr.azurecr.io/knowledge-assistant:v1 where any Azure service can pull it.
Push denied or unauthorized?
Run az acr login --name knowasstacr again to refresh your credentials, then retry the push.
Help me push my image to ACR.
Now kick off the AKS cluster creation. This takes 5 to 10 minutes, so you will use that wait time productively in the next steps.
- Create your AKS cluster by running:
az aks create --resource-group rg-knowledge-assistant --name aks-knowledge-assistant --node-count 1 --tier free --attach-acr knowasstacr --generate-ssh-keysWhat does this command do?
This creates a single-node Kubernetes cluster on the Free tier (you only pay for the underlying VM, covered by your credit). The --attach-acr flag grants the cluster permission to pull images from your registry without extra authentication.
The command runs for several minutes. Leave it running and do not close the terminal. You will use this wait time in the next steps.
Do not close your terminal
The az aks create command will keep running in this terminal window for 5 to 10 minutes. Open a second terminal window for the remaining verification and next steps.
Before you move on, confirm your container registry is live and holding your image.
What do you think you will see when you list your Azure container registries?
- In your second terminal, verify your registry exists by running:
az acr list --output tableYou should see a table with knowasstacr listed along with its login server knowasstacr.azurecr.io and the eastus location.
Not seeing your registry?
Make sure you are signed in to Azure in the second terminal as well. Run az login if needed.
Help me find my container registry.
Your image is in the cloud and your AKS cluster is spinning up in the background. Next, you will explore what is actually inside your container's Linux filesystem before sending it to the cluster.
Inspect Your Container's Linux Internals
Your image is safely stored in ACR, and your AKS cluster is spinning up in the background. While you wait for that to finish, this is the perfect time to look inside your container.
As a Linux Cloud Engineer, you need to understand the filesystem layout, running processes, and OS configuration of every container you deploy. You are about to exec into your Knowledge Assistant container and explore its Linux internals before AKS runs it in production.
In this step, get ready to:
- Run your Knowledge Assistant container locally for inspection.
- Explore the container's Linux filesystem, processes, and OS configuration.
- Document key findings about PID 1 and the base image.
Run your container and exec into its shell
Your knowledge-assistant:latest image is the same artifact that AKS will pull from ACR. Running it locally lets you poke around inside before it goes live.
- Start your Knowledge Assistant container in the background by running this command:
docker run -d --name ka-inspect -p 8000:8000 knowledge-assistant:latest- Confirm it is running:
docker psYou should see a row for ka-inspect with status Up and port 8000 mapped.
Seeing a port conflict error?
If you see port is already allocated, another process is using port 8000. Stop whatever is using that port, or change the host port:
docker run -d --name ka-inspect -p 8001:8000 knowledge-assistant:latesthelp me fix a Docker port conflict on port 8000
- Exec into the running container to get a shell by running:
docker exec -it ka-inspect /bin/bashYour terminal prompt changes to something like root@<container-id>:/app#. You are now inside the container's Linux filesystem.
What does docker exec do?
The docker exec command attaches a new process to an already-running container. The -it flags give you an interactive terminal so you can type commands inside.
This is the same technique you would use to troubleshoot a misbehaving container in production.
Getting an error about /bin/bash?
Some ultra-minimal images do not include bash. If you see an error, try /bin/sh instead:
docker exec -it ka-inspect /bin/shhelp me exec into a container that doesn't have bash
Explore the container's Linux internals
Before you run the first inspection command, think about this: what operating system do you think your container is based on? Your Dockerfile uses a python:3.x-slim base image. Is that Debian, Alpine, or something else?
- Inspect the operating system by running:
cat /etc/os-releaseYou should see output identifying a Debian-based distribution. The python:3.x-slim images are built on Debian, which is why you see Debian in the output rather than Alpine or RHEL.
- Check disk usage to see how minimal the container filesystem is:
df -hNotice how small the total filesystem is compared to a full Linux installation. Containers share the host kernel and only include the files they need to run.
- List all running processes inside the container:
ps auxLook at the PID column. Your application server process (uvicorn or gunicorn) is PID 1. In a container, PID 1 is the root process. If it exits, the entire container stops.
- Inspect your application files to see the directory layout:
ls -la /appThis shows your application code and dependencies exactly as the Dockerfile WORKDIR arranged them. This is what AKS will run on the cluster node.
What do these commands reveal?
Together, these four commands give you a complete operational picture of your container. You know the base OS (Debian slim), the filesystem size, which process must stay alive (PID 1), and where the application code lives.
This is the same inspection you would perform when inheriting an unfamiliar container in production. Knowing these details helps you write better health probes, resource limits, and troubleshooting runbooks.
Document your findings and clean up
Before leaving the container, take note of the key facts you discovered. These observations confirm you understand what AKS will actually run.
- Note which base image OS your container uses (Debian-based from python:3.x-slim).
- Note that the filesystem is minimal compared to a full Linux install.
- Note which process is PID 1 (your application server).
- Exit the container shell by running:
exitYour terminal prompt returns to your normal host machine prompt.
- Stop and remove the inspection container by running:
docker stop ka-inspect && docker rm ka-inspect- Confirm the container is gone by running:
docker ps -a --filter name=ka-inspectYou should see no results. The inspection container is fully removed and no longer consuming resources.
Container won't stop?
If docker stop hangs, force-remove the container:
docker rm -f ka-inspecthelp me force-remove a stuck Docker container
You now know exactly what lives inside your container. Next up, you will practice RHEL-family Linux administration skills in a dedicated Rocky Linux environment.
Practice RHEL/Rocky Linux Administration
You explored the internals of your Knowledge Assistant container in the previous step and confirmed it runs a Debian-based OS. Enterprise environments run RHEL-family operating systems.
Your target role as a Linux Cloud Engineer requires RHEL/Rocky Linux skills that your Debian-based container does not exercise. You will practice them now in a dedicated Rocky Linux 9 container while your AKS cluster finishes provisioning in the background.
In this step, get ready to:
- Launch a Rocky Linux 9 container and inspect the OS.
- Practice user management, directory creation, and file permissions.
- Understand why chmod 777 is dangerous in production.
Launch a Rocky Linux 9 container and explore
Rocky Linux is a free, community-driven rebuild of RHEL. It uses the same package manager (dnf), same system conventions, and same directory layout that you will encounter on Azure-hosted Linux VMs.
- Pull and run an interactive Rocky Linux 9 container by running this command:
docker run -it rockylinux/rockylinux:9 /bin/bashWhat does this command do?
The -it flags give you an interactive terminal session inside the container.
Your prompt changes to something like [root@<container-id> /]# which confirms you are now inside the Rocky Linux shell as the root user.
Before you run the next command, what do you expect to see in the NAME field?
- Confirm the operating system by running:
cat /etc/os-releaseYou should see NAME="Rocky Linux" and a VERSION_ID starting with 9. This confirms you are running a RHEL-family distribution.
- Update all installed packages by running:
dnf update -yYou will see a stream of output as packages are downloaded and updated. This may take a minute.
- Install common administration tools by running:
dnf install -y vim procps-ng net-toolsWhat did you just install?
- vim is a text editor for editing config files on servers.
- procps-ng provides ps, top, and other process-monitoring tools.
- net-tools provides ifconfig and netstat for network diagnostics.
You should see Complete! at the end of the install output, confirming all three packages are ready.
Seeing network errors during dnf update?
If dnf cannot reach the package mirrors, make sure Docker Desktop is running and your machine has an internet connection. Exit the container with exit, restart Docker Desktop, then re-run the docker run command.
Help me fix Docker networking in my Rocky Linux container
Practice user management and permissions
In production, applications never run as root. You create dedicated service users with minimal privileges so that a compromised process cannot damage the entire system.
- Create a service user with no interactive login shell by running:
useradd -r -s /sbin/nologin appuser- Verify the user was created by running:
id appuserYou should see output showing a uid, gid, and groups entry for appuser.
What do these flags mean?
- -r creates a system account (low UID, reserved for services, does not get a home directory by default).
- -s /sbin/nologin prevents anyone from logging in interactively as this user. The user can only own processes and files.
- Create an application directory and set ownership by running these commands:
mkdir /opt/myapp
chown appuser:appuser /opt/myapp- Verify the directory and its ownership by running:
ls -la /opt/You should see myapp listed with appuser appuser as the owner and group.
- Set secure permissions on the directory by running:
chmod 750 /opt/myapp- Verify the updated permissions by running:
ls -la /opt/The permissions column for myapp should now read drwxr-x---. This means the owner has full access, the group can read and traverse, and everyone else is locked out.
Why is chmod 777 unsafe?
chmod 777 grants read, write, and execute to every user on the system. In production, any compromised process could modify or delete files.
The principle of least privilege means granting only the minimum permissions needed. 750 restricts access to the owner and group only.
Exit and clean up
- Exit the Rocky Linux container by running:
exitThe container stops automatically when you exit the interactive shell. No separate cleanup command is needed.
Why Rocky Linux instead of your Knowledge Assistant container?
Your Knowledge Assistant uses a Debian-based python:3.x-slim image with apt as its package manager. Enterprise Linux environments use dnf and follow RHEL conventions.
Practicing in a Rocky Linux container gives you hands-on experience with the same tools and patterns you will encounter on Azure-hosted VMs and production Kubernetes nodes.
Your AKS cluster should be ready by now. Next up, you will connect to it and deploy your Knowledge Assistant with production-grade Kubernetes manifests.
Deploy to AKS with Production Kubernetes Manifests
You have explored your container's Linux internals and practiced RHEL administration. Now it is time to put your Knowledge Assistant into the cloud where it belongs.
Your AKS cluster has been provisioning in the background and should be ready now. In this step, you will connect kubectl to your live cluster, write production-grade Kubernetes manifests with namespace isolation, health probes, resource limits, and secrets management, then deploy your Knowledge Assistant and verify it responds.
In this step, get ready to:
- Connect kubectl to your AKS cluster.
- Write a complete Kubernetes manifest with namespace, ConfigMap, Secret, Deployment, and Service.
- Deploy and verify your Knowledge Assistant responds from the live cluster.
Connect kubectl to your AKS cluster
Your AKS cluster is running, but kubectl on your machine does not know about it yet. You need to download the cluster credentials so kubectl can communicate with your cluster's API server.
- Download your AKS cluster credentials by running this command:
az aks get-credentials --resource-group rg-knowledge-assistant --name aks-knowledge-assistantWhat does this command do?
This command downloads your cluster's connection certificate and merges it into your local kubeconfig file. After this, kubectl automatically targets your AKS cluster for all commands.
- Verify the connection by checking your cluster nodes:
kubectl get nodesYou should see one node listed with a Ready status. This confirms kubectl is talking to your live AKS cluster.
Not seeing a Ready node?
- If you see an error about the cluster not being found, your AKS cluster creation from Step 1 may still be in progress. Run az aks show --resource-group rg-knowledge-assistant --name aks-knowledge-assistant --query provisioningState to check its status.
- If the cluster shows Succeeded but kubectl still fails, try closing and reopening your terminal, then re-run the az aks get-credentials command.
Help me troubleshoot connecting kubectl to my AKS cluster.
Create the Kubernetes manifest file
Production deployments separate concerns into distinct Kubernetes resources. You will build a single manifest file containing five resources that work together to run your Knowledge Assistant securely.
- Create a new file called k8s-deploy.yaml in your project folder.
- Add the following Namespace, ConfigMap, and Secret resources to k8s-deploy.yaml:
apiVersion: v1
kind: Namespace
metadata:
name: knowledge-assistant
---
apiVersion: v1
kind: ConfigMap
metadata:
name: knowledge-assistant-config
namespace: knowledge-assistant
data:
APP_ENV: "production"
LOG_LEVEL: "info"
---
apiVersion: v1
kind: Secret
metadata:
name: knowledge-assistant-secrets
namespace: knowledge-assistant
type: Opaque
data:
API_KEY: eW91ci1hcGkta2V5LWhlcmU=What do these three resources do?
- Namespace creates an isolated environment called knowledge-assistant so your resources do not mix with other workloads on the cluster.
- ConfigMap stores non-sensitive configuration (APP_ENV, LOG_LEVEL) as key-value pairs that get injected as environment variables into your container.
- Secret stores sensitive values (here a placeholder API key) base64-encoded. Kubernetes keeps Secrets separate from ConfigMaps so you can restrict access to sensitive data independently.
- Save k8s-deploy.yaml and apply these first three resources to your cluster:
kubectl apply -f k8s-deploy.yamlYou should see three lines confirming creation: namespace/knowledge-assistant created, configmap/knowledge-assistant-config created, and secret/knowledge-assistant-secrets created.
- Confirm the namespace is active:
kubectl get ns knowledge-assistantYou should see the namespace listed with Active status.
- Now add the Deployment resource below the existing content in k8s-deploy.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: knowledge-assistant
namespace: knowledge-assistant
labels:
app: knowledge-assistant
spec:
replicas: 1
selector:
matchLabels:
app: knowledge-assistant
template:
metadata:
labels:
app: knowledge-assistant
spec:
containers:
- name: knowledge-assistant
image: knowasstacr.azurecr.io/knowledge-assistant:v1
ports:
- containerPort: 8000
envFrom:
- configMapRef:
name: knowledge-assistant-config
- secretRef:
name: knowledge-assistant-secrets- Directly below the secretRef block (still inside the container spec), add resource limits and health probes:
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3What do probes and resource limits do?
Resource requests (256Mi memory, 250m CPU) tell the scheduler the minimum your container needs. Limits (512Mi, 500m) cap how much it can consume. If your container exceeds memory limits, Kubernetes kills it.
The livenessProbe checks if your app is still alive by hitting /health every 10 seconds. If it fails 3 times in a row, Kubernetes restarts the container. The readinessProbe checks if the app is ready to serve traffic. Until it passes, the Service does not route requests to that pod.
- Save k8s-deploy.yaml and apply again to create the Deployment:
kubectl apply -f k8s-deploy.yamlYou should see deployment.apps/knowledge-assistant created along with unchanged messages for the Namespace, ConfigMap, and Secret.
- Finally, add the Service resource at the end of k8s-deploy.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: knowledge-assistant
namespace: knowledge-assistant
spec:
selector:
app: knowledge-assistant
ports:
- protocol: TCP
port: 80
targetPort: 8000
type: ClusterIPWhat is a ClusterIP Service?
A ClusterIP Service gives your Deployment a stable internal address inside the cluster. External traffic cannot reach it directly. You will use kubectl port-forward to tunnel traffic from your local machine to the Service for testing.
The Service maps port 80 to your container's port 8000, so cluster-internal clients can reach your API on port 80 without knowing the container's actual port.
- Save k8s-deploy.yaml.
✔️ Awesome, I've got everything!
Great. Double check you have saved k8s-deploy.yaml with all five resources. The file should contain Namespace, ConfigMap, Secret, Deployment, and Service separated by --- lines.
ⓧ I'd like to double check the full code
Scroll up and confirm these sections are present in your k8s-deploy.yaml file, in order:
- Namespace (kind: Namespace, name knowledge-assistant).
- ConfigMap (kind: ConfigMap, keys APP_ENV and LOG_LEVEL).
- Secret (kind: Secret, key API_KEY).
- Deployment (kind: Deployment, image knowasstacr.azurecr.io/knowledge-assistant:v1, resources block, livenessProbe, readinessProbe).
- Service (kind: Service, type ClusterIP, port 80 targeting 8000).
Apply and verify the deployment
Your manifest is complete. Time to apply the final version and confirm your Knowledge Assistant is alive on AKS.
- Apply the complete manifest to create the Service:
kubectl apply -f k8s-deploy.yamlYou should see service/knowledge-assistant created alongside unchanged messages for the other resources.
- Watch your pod reach Running status:
kubectl get pods -n knowledge-assistant -wWait until the STATUS column shows Running and READY shows 1/1. This means both the liveness and readiness probes have passed. Press Ctrl+C to stop watching.
Pod not reaching Running status?
- If you see ImagePullBackOff, the cluster cannot pull your image from ACR. Run kubectl describe pod -n knowledge-assistant -l app=knowledge-assistant and check the Events section. Verify your image was pushed successfully with az acr repository show-tags --name knowasstacr --repository knowledge-assistant.
- If you see CrashLoopBackOff, your container is starting but crashing. Check logs with kubectl logs -n knowledge-assistant -l app=knowledge-assistant.
Help me troubleshoot my pod not reaching Running status on AKS.
- Check the application logs to confirm startup:
kubectl logs -n knowledge-assistant -l app=knowledge-assistantYou should see your application's startup messages confirming it is listening on port 8000.
- Start a port-forward to tunnel traffic from your machine to the Service:
kubectl port-forward -n knowledge-assistant svc/knowledge-assistant 8080:80What does port-forward do?
This creates a tunnel from localhost:8080 on your machine to port 80 on the knowledge-assistant Service inside the cluster. The Service then routes to port 8000 on your container. The terminal stays open while the tunnel is active.
- Open a new terminal window. Before you run the next command, what do you think the /health endpoint will return? Your container is the same image that worked locally, just running on AKS now.
- Test the health endpoint:
curl http://localhost:8080/healthYou should see a JSON response confirming the service is healthy. This is your Knowledge Assistant responding from a live AKS pod.
- Test the ask endpoint to confirm the full API works:
curl -X POST http://localhost:8080/ask -H "Content-Type: application/json" -d "{\"question\": \"What is federal contracting?\"}"You should see a JSON response with an answer, confidence score, and sources. Your Knowledge Assistant is now live on Azure Kubernetes Service, running with production-grade manifests.
Getting connection refused on localhost:8080?
- Make sure your port-forward terminal is still running. If it exited, re-run the kubectl port-forward command.
- Confirm the pod shows 1/1 Ready with kubectl get pods -n knowledge-assistant. If it is not ready, the Service will not route traffic.
Help me fix connection refused when port-forwarding to my AKS service.
Your Knowledge Assistant is running on a real AKS cluster with namespace isolation, externalized config, secrets management, health monitoring, and resource limits. Next up, you will document the full operational lifecycle so you can prove you own this service end-to-end.
Create Your Operations Guide
Your Knowledge Assistant is live on AKS and responding to requests. That proves you can deploy. Now you need to prove you can own the service.
Production ownership means documenting how to monitor, troubleshoot, and maintain what you deployed. In this step, you will create a comprehensive operations guide that covers maintenance runbooks, observability, troubleshooting scenarios, AI platform patterns, and GPU node pool concepts.
In this step, get ready to:
- Write a Linux Maintenance Runbook with health checks, log collection, and service management commands.
- Document observability, troubleshooting scenarios, AI platform operations, and GPU node pool concepts.
- Verify your runbook commands against the live AKS cluster.
Write the Linux Maintenance Runbook
The runbook is the section an on-call engineer opens at 2 AM when something breaks. It must contain exact commands, not vague instructions.
- Create a new file called operations-guide.md in the same folder as your k8s-deploy.yaml by pasting this content:
# Knowledge Assistant Operations Guide
## 1. Linux Maintenance Runbook
### Health Checks
- Pod health: `kubectl get pods -n knowledge-assistant`
- Node health: `kubectl get nodes`
- Detailed pod status: `kubectl describe pod -n knowledge-assistant -l app=knowledge-assistant`
- Application health endpoint: `curl http://<service-ip>/health`
- Metrics endpoint: `curl http://<service-ip>/metrics`
### Log Collection
- Pod logs (current): `kubectl logs -n knowledge-assistant -l app=knowledge-assistant`
- Pod logs (previous crash): `kubectl logs -n knowledge-assistant -l app=knowledge-assistant --previous`
- Stream logs: `kubectl logs -n knowledge-assistant -l app=knowledge-assistant -f`
- On a Linux VM, journal logs: `journalctl -u kubelet --since "1 hour ago"`What does this section cover?
The Health Checks subsection gives you five instant-status commands. An operator can copy any one of them to check pod state, node health, or the app's own endpoints.
Log Collection covers current logs, previous-crash logs (critical for diagnosing a pod that already restarted), streaming, and host-level journal logs for when you SSH into a node.
- Below the Log Collection section, add the remaining runbook subsections:
### Service Management
- Restart deployment: `kubectl rollout restart deployment/knowledge-assistant -n knowledge-assistant`
- Scale replicas: `kubectl scale deployment/knowledge-assistant --replicas=2 -n knowledge-assistant`
- Check rollout status: `kubectl rollout status deployment/knowledge-assistant -n knowledge-assistant`
### Disk Monitoring
- Node disk usage: `kubectl exec -n knowledge-assistant <pod-name> -- df -h`
- Check PersistentVolumeClaims: `kubectl get pvc -n knowledge-assistant`
- Alert threshold: 80% disk usage triggers investigation, 90% triggers immediate action
### Patch Planning
- Review available updates: `az aks get-upgrades --resource-group rg-knowledge-assistant --name aks-knowledge-assistant --output table`
- Schedule maintenance window for off-peak hours
- Test patches in staging namespace first
- On RHEL/Rocky nodes: `dnf check-update`, then `dnf update -y` during maintenance window
### Reboot Validation
- After node reboot, verify: `kubectl get nodes` (all nodes Ready)
- Verify pods rescheduled: `kubectl get pods -n knowledge-assistant` (all Running)
- Verify application responds: test /health and /ask endpoints
- Check events for errors: `kubectl get events -n knowledge-assistant --sort-by=.lastTimestamp`
### Documentation Standards
- All changes logged in operations journal with date, operator, change description, and rollback plan
- Runbook reviewed quarterly or after any incidentWhy include Patch Planning and Reboot Validation?
AKS nodes need periodic upgrades (Kubernetes version bumps, OS patches). Your runbook pre-documents the exact commands to check for available upgrades and the validation steps after a reboot so nothing gets missed during a maintenance window.
Before you continue building the guide, test one of your runbook commands against the live cluster.
- Run the first health check command from your runbook:
kubectl get pods -n knowledge-assistantYou should see your Knowledge Assistant pod listed with status Running and 1/1 ready. Your runbook's first command works against your live cluster.
Pod not showing Running?
If the pod shows a different status, it may have been evicted or restarted since your last step. Run kubectl describe pod -n knowledge-assistant -l app=knowledge-assistant and check the Events section at the bottom for clues.
Why is my Knowledge Assistant pod not running on AKS?
Add Observability and Troubleshooting sections
Observability is how you know something is wrong before a user reports it. Troubleshooting is how you fix it once you know.
- Below the Documentation Standards section, add a horizontal rule and the Observability section:
---
## 2. Observability
### Metrics Endpoint
- Your Knowledge Assistant exposes `/metrics` with request counts, latency histograms, confidence score distributions, and error rates
- Prometheus scrapes this endpoint on a configured interval (typically 15-30s)
### Prometheus Configuration (Conceptual)
```yaml
scrape_configs:
- job_name: 'knowledge-assistant'
kubernetes_sd_configs:
- role: pod
namespaces:
names: ['knowledge-assistant']
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: knowledge-assistant
action: keep
```- Directly below the Prometheus config, add the Grafana, Alerting, and Log Aggregation subsections:
### Grafana Dashboards
- Panel 1: Request rate (requests/second to /ask and /health)
- Panel 2: Response latency (p50, p95, p99)
- Panel 3: Confidence score distribution
- Panel 4: Pod CPU and memory usage (from cAdvisor metrics)
- Panel 5: Pod restart count
### Alerting Rules
- Pod not ready for >2 minutes: PodNotReady alert
- Health check failure rate >10%: HealthCheckDegraded alert
- CPU usage >80% sustained for 5 minutes: HighCPU alert
- Memory usage >85%: HighMemory alert (risk of OOMKill)
- Pod restart count >3 in 10 minutes: CrashLooping alert
### Log Aggregation
- Structured JSON logs from the application
- Collected via Fluentd/Fluent Bit DaemonSet or Azure Monitor Container Insights
- Indexed by namespace, pod, container, and severity levelWhat is the alerting threshold logic?
Each alert rule combines a metric with a duration. CPU at 80% for a single second is a spike. CPU at 80% for 5 sustained minutes means the pod is genuinely overloaded and needs attention. The thresholds here match common production defaults.
Now verify that resource monitoring works on your live cluster. This command comes from the Troubleshooting section you are about to write.
- Check pod resource usage by running:
kubectl top pods -n knowledge-assistantYou should see your pod listed with its current CPU and memory consumption.
Seeing 'Metrics API not available'?
The metrics server may take a few minutes to initialize on a new AKS cluster. Wait 2-3 minutes and try again. If it persists, AKS automatically installs the metrics server addon, but it may need a node restart to activate.
Help me troubleshoot kubectl top pods not working on AKS.
- Next, add the Troubleshooting Scenarios section. Start with the first four scenarios:
---
## 3. Troubleshooting Scenarios
### Container Not Starting
- Symptoms: Pod stuck in ContainerCreating or Error state
- Diagnosis: `kubectl describe pod <name> -n knowledge-assistant`, check Events section
- Common causes: missing image, failed volume mount, resource quota exceeded
- Fix: correct image reference, verify PVC exists, check namespace resource quotas
### Health Check Failure
- Symptoms: Pod Running but 0/1 Ready, endpoints removed from Service
- Diagnosis: `kubectl describe pod` shows Readiness probe failed, check application logs
- Common causes: application crash loop, dependency unavailable, wrong probe path/port
- Fix: verify /health endpoint works locally, check probe configuration matches app
### Pod CrashLoopBackOff
- Symptoms: Pod restarts repeatedly with increasing backoff delay
- Diagnosis: `kubectl logs --previous` to see crash output, `kubectl describe pod` for exit codes
- Common causes: unhandled exception at startup, missing environment variable, OOMKilled
- Fix: check logs for stack trace, verify ConfigMap/Secret mounted, increase memory limits if OOM
### ImagePullBackOff
- Symptoms: Pod stuck in ImagePullBackOff or ErrImagePull
- Diagnosis: `kubectl describe pod` shows "Failed to pull image" in Events
- Common causes: wrong image tag, ACR authentication failed, registry unreachable
- Fix: verify image exists in ACR (`az acr repository show-tags`), check AKS-ACR attachment- Below the ImagePullBackOff scenario, add the remaining five troubleshooting scenarios:
### Missing ConfigMap
- Symptoms: Pod fails to start, CreateContainerConfigError
- Diagnosis: `kubectl describe pod` shows "configmap not found"
- Common causes: ConfigMap not created in correct namespace, typo in name
- Fix: `kubectl get configmap -n knowledge-assistant`, create or fix the ConfigMap
### Service Connectivity Issue
- Symptoms: Cannot reach the application through the Service, connection refused
- Diagnosis: `kubectl get endpoints -n knowledge-assistant` shows no endpoints
- Common causes: label selector mismatch, pod not Ready, wrong targetPort
- Fix: verify Service selector matches pod labels, verify pod is Ready, check port numbers
### High Disk Usage
- Symptoms: Node NotReady or pod eviction
- Diagnosis: `kubectl describe node` shows DiskPressure condition
- Common causes: container logs filling disk, large temp files, unused images
- Fix: clean up logs, implement log rotation, prune unused images on node
### High CPU/Memory
- Symptoms: Throttled responses, OOMKilled pods
- Diagnosis: `kubectl top pods -n knowledge-assistant`, `kubectl top nodes`
- Common causes: resource limits too low, memory leak, unexpected load spike
- Fix: increase resource limits, add HPA for autoscaling, investigate memory leak
### DNS/Network Issue
- Symptoms: Application cannot resolve external hostnames or reach other services
- Diagnosis: `kubectl exec <pod> -- nslookup kubernetes.default`, check CoreDNS pods
- Common causes: CoreDNS pod not running, NetworkPolicy blocking traffic, NSG rules
- Fix: `kubectl get pods -n kube-system -l k8s-app=kube-dns`, verify NetworkPoliciesWhy document nine troubleshooting scenarios?
These nine scenarios cover the most common failure modes you will encounter operating a containerized service on Kubernetes. Each follows the same structure (Symptoms, Diagnosis, Common causes, Fix) so any engineer can follow them under pressure.
- Verify another runbook command against your live cluster by running:
kubectl describe pod -n knowledge-assistant -l app=knowledge-assistantYou should see detailed pod information including the Events section at the bottom. This is the same output your troubleshooting scenarios reference for diagnosis.
Add AI Platform Operations and GPU Node Pool sections
These final sections connect your hands-on deployment to the broader AI platform landscape. They show interviewers you understand how your small-scale service maps to enterprise patterns.
- Below the DNS/Network Issue scenario, add the AI Platform Operations Mapping section:
---
## 4. AI Platform Operations Mapping
### How This RAG Service Maps to Production AI Platforms
Your Knowledge Assistant is a small-scale implementation of patterns used in enterprise AI platforms:
- **Open-source model hosting**: Your TF-IDF/scikit-learn retrieval mirrors how teams host open-source embedding models (e.g., sentence-transformers) behind internal APIs.
- **Private access**: The ClusterIP Service type keeps your API internal to the cluster.
- **Data boundaries**: Your local text documents represent proprietary data that must not leave the organization boundary.
- **Secrets management**: The Kubernetes Secret holding your API_KEY mirrors how production systems store model API keys.
- **Model upgrades**: Updating your knowledge base documents or retrieval logic maps to model version upgrades in production.
- **AKS-based backend services**: Your deployment pattern is the exact pattern used for inference backends in Azure AI platforms.Why does this mapping matter for interviews?
When an interviewer asks about AI platform operations, they want to know you understand the patterns, not just the tools. Each bullet connects something you built (ClusterIP, Secrets, Deployments) to the enterprise-scale version of the same pattern.
- Finally, add the GPU Node Pool Concepts section:
---
## 5. GPU Node Pool Concepts for AKS
### When Do AI Workloads Need GPUs?
- Large language model (LLM) inference with billions of parameters
- Model fine-tuning and training
- Real-time embedding generation at scale
- Computer vision inference
- Your Knowledge Assistant uses TF-IDF (CPU-only) and would NOT require GPU nodes
### AKS GPU Node Pool Architecture
- A GPU node pool is a separate pool of VMs in your AKS cluster equipped with NVIDIA (or AMD) GPUs
- GPU nodes are tainted (e.g., nvidia.com/gpu: NoSchedule) so only GPU workloads schedule on them
### AKS-Managed vs Self-Managed GPU Stack
- AKS-Managed (recommended): AKS installs NVIDIA driver, Kubernetes device plugin, and DCGM metrics exporter automatically
- Self-Managed: You control GPU driver versions; more flexibility, more operational overhead
### Multi-Instance GPU (MIG) for Multi-Tenant Inference
- Available on NVIDIA A100, H100, H200 GPUs
- Partitions one physical GPU into up to 7 isolated instances
### GPU Observability
- NVIDIA DCGM metrics exporter exposes Prometheus metrics on port 19400
- Key metrics: DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_GPU_TEMPWhy document GPU concepts if your app does not use GPUs?
Your Knowledge Assistant uses CPU-only TF-IDF retrieval. But a Linux Cloud Engineer role managing AI workloads on AKS will encounter GPU node pools for LLM inference. Documenting these concepts shows you understand the full spectrum of AI infrastructure, not just your current deployment.
- Save operations-guide.md.
Before you call this done, what do you expect to see when you check recent cluster events?
- Run the final verification command from your Reboot Validation section:
kubectl get events -n knowledge-assistant --sort-by=.lastTimestampYou should see a list of recent events including your deployment's scheduling, image pull, and container start. These events confirm your cluster is healthy and your documented commands produce real, actionable output.
Seeing 'No resources found'?
Kubernetes events expire after about 1 hour by default. If your deployment happened more than an hour ago, the events may have been garbage collected. This is normal. Run kubectl get pods -n knowledge-assistant instead to confirm your pod is still Running.
Help me understand why kubectl get events shows no results.
✔️ Awesome, I've got everything!
Your operations-guide.md is complete. Make sure you have saved the file.
ⓧ I'd like to double check the full code
Your operations-guide.md is too long to show in one block. Scroll up and confirm you have all of these sections present, in order:
- File header: # Knowledge Assistant Operations Guide.
- Section 1: Linux Maintenance Runbook (Health Checks, Log Collection, Service Management, Disk Monitoring, Patch Planning, Reboot Validation, Documentation Standards).
- Section 2: Observability (Metrics Endpoint, Prometheus Configuration, Grafana Dashboards, Alerting Rules, Log Aggregation).
- Section 3: Troubleshooting Scenarios (9 scenarios: Container Not Starting, Health Check Failure, CrashLoopBackOff, ImagePullBackOff, Missing ConfigMap, Service Connectivity, High Disk Usage, High CPU/Memory, DNS/Network Issue).
- Section 4: AI Platform Operations Mapping (6 bullet points mapping your service to enterprise patterns).
- Section 5: GPU Node Pool Concepts for AKS (When GPUs are needed, Architecture, Managed vs Self-Managed, MIG, GPU Observability).
You now have a professional operations guide that proves you can own a containerized AI service end-to-end. Next up, you will intentionally break your deployment and troubleshoot it back to health.
Secret mission
Trigger and Troubleshoot a Live Pod Failure
Reading about troubleshooting is not the same as doing it. In this mission, you will intentionally break your live AKS deployment, diagnose the failure using only kubectl output, and bring it back to health. This proves you can handle a real production incident under pressure.
Clean Up Your Resources
Clean Up Your Resources
Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses Azure resources that incur ongoing charges if left running.
Cost warning
Your AKS cluster node VM costs approximately $0.05/hour (~$1.20/day) and the ACR costs approximately $0.17/day. If you are done experimenting, choose Pause or Delete to stop charges.
Resources you used:
- Azure Resource Group: rg-knowledge-assistant (contains all Azure resources below).
- Azure Container Registry: knowasstacr (Basic tier, ~$0.17/day).
- AKS Cluster: aks-knowledge-assistant (Free tier control plane, 1 node VM ~$0.05/hour).
- Local files: k8s-deploy.yaml and operations-guide.md on your Desktop.
- Azure CLI installed via winget.
Keep everything running
No action needed. Choose this if you are still actively experimenting with your AKS deployment or plan to continue building on it today.
- Your AKS cluster will continue running and incur approximately $1.20/day in VM charges.
- Your ACR will continue at approximately $0.17/day.
- The $200 Azure free credit covers these costs for many days of experimentation.
- Your local files (k8s-deploy.yaml, operations-guide.md) remain on your Desktop.
- Azure CLI remains installed on your machine.
Pause - I'll come back to this later
Stop the AKS cluster to eliminate VM charges while keeping your configuration intact. The ACR will still incur a small charge (~$0.17/day) to preserve your stored image.
- Stop your AKS cluster by running:
- When you return, restart it with:
az aks stop --resource-group rg-knowledge-assistant --name aks-knowledge-assistantThis stops the node VM and eliminates compute charges. To restart later:
az aks start --resource-group rg-knowledge-assistant --name aks-knowledge-assistant- After restarting, reconnect kubectl by running:
az aks get-credentials --resource-group rg-knowledge-assistant --name aks-knowledge-assistant- Your local files and Azure CLI remain unchanged.
Delete - I don't want to use this again
Remove all Azure resources and local project files. This stops all billing immediately.
- Delete the entire resource group (this removes the AKS cluster, ACR, and all associated resources in one command):
az group delete --name rg-knowledge-assistant --yes --no-waitWhat does --no-wait do?
The --no-wait flag returns control to your terminal immediately while Azure deletes resources in the background. Deletion typically completes within a few minutes.
- Delete your local project files. On macOS/Linux:
rm ~/Desktop/k8s-deploy.yaml ~/Desktop/operations-guide.md- On Windows (PowerShell):
Remove-Item ~\Desktop\k8s-deploy.yaml, ~\Desktop\operations-guide.md- Uninstall Azure CLI if you no longer need it:
winget uninstall --id Microsoft.AzureCLI --exact- Your Azure free account remains active. No action is needed to close it unless you choose to.
Nice Work!
Nice Work!
You just deployed a production-ready Knowledge Assistant to a live AKS cluster and proved you can own the full operational lifecycle of a containerized AI service.
You've learned how to:
- Deploy a containerized API to Azure Kubernetes Service with production-grade manifests including namespace isolation, health probes, resource limits, and secrets management.
- Inspect Linux container internals with docker exec and practice RHEL/Rocky Linux administration including dnf package management, user creation, and file permissions.
- Create a comprehensive operations guide covering maintenance runbooks, observability, troubleshooting scenarios, AI platform operations mapping, and GPU node pool concepts.
- Secret Mission: Triggered a real pod failure (ImagePullBackOff) and troubleshot it back to health using kubectl rollout commands.
Ready to quiz yourself?