Build a Medical RAG Chatbot

Introduction

30 Second Summary

When you ask a medical question online, you get a wall of links and conflicting information. Imagine a system that reads thousands of medical documents and gives you a single, grounded answer with citations.

In this project, you will build a full-stack medical RAG chatbot that retrieves relevant medical passages from a vector database, re-ranks them for precision, and generates grounded answers using a fine-tuned LLaMA model.

What You'll Build

A production-grade medical question-answering chatbot with a professional chat interface, persistent conversation history, semantic retrieval with cross-encoder re-ranking, and domain-tuned answer generation.

By the end of this project, you'll have:

• A FastAPI backend with full CRUD chat management and PostgreSQL conversation persistence.
• A two-stage retrieval pipeline using Pinecone vector search and MedCPT cross-encoder re-ranking across ~16,000 medical passages.
• A QLoRA-fine-tuned LLaMA 3.1 8B model serving grounded medical responses with source citations.
• Secret Mission: Deploy a live demo to Streamlit Community Cloud so you can share a working link on LinkedIn.

Are there any prerequisites?

This project assumes intermediate Python knowledge, basic familiarity with REST APIs, and experience with Docker. You also need a local NVIDIA GPU for model inference.

You do not need prior experience with fine-tuning, vector databases, or RAG systems. The project teaches each concept as you build.

Set Up the Development Environment

Building a full-stack RAG chatbot requires several moving parts: a Python backend, a vector database for retrieval, a database for chat persistence, and a fine-tuned LLM for generation.

In this step, you will install all the tools and libraries the project depends on. You will also configure Pinecone and Hugging Face accounts, scaffold the project directory, and spin up a PostgreSQL database in Docker.

In this step, get ready to:

• Install Docker, Python dependencies, and create a virtual environment.
• Configure Pinecone and Hugging Face accounts with API credentials.
• Create the project structure, Docker Compose file, and verify PostgreSQL connectivity.

Install core dependencies and Docker

This project needs Python 3.10+ for its ML libraries (transformers, bitsandbytes, PEFT) and Docker for running PostgreSQL in a container.

• In VS Code, click Terminal in the top menu bar.
• Click New Terminal to open the integrated terminal.
• Check your Python version by running:

python3 --version

✔️ I see version 3.10 or higher

You are good to go. Move on to the Docker check below.

ⓧ I see an older version

You need Python 3.10+ for this project.

• Visit python.org/downloads and download the latest Python 3 installer for your platform.
• Run the installer. On Windows, check Add python.exe to PATH before clicking Install Now.
• Close and reopen your terminal, then run python3 --version again to confirm.

macOS default Python

macOS ships with an older Python. After installing from python.org, you may need to use python3.10 or python3.12 (the specific binary name) instead of python3.

ⓧ Command not found

Python is not installed on your system.

• Visit python.org/downloads and download the latest Python 3 installer for your platform.
• Run the installer. On Windows, check Add python.exe to PATH before clicking Install Now.
• Close and reopen your terminal, then run python3 --version again to confirm you see 3.10 or higher.

• Check if Docker is installed by running:

docker --version

✔️ I see a version number

Docker is installed. Make sure Docker Desktop is running (check for the whale icon in your system tray).

ⓧ Command not found

You need to install Docker Desktop.

• Visit docs.docker.com/get-started/get-docker and download Docker Desktop for your platform.
• Run the installer and follow the prompts. A restart may be required.
• After installation, press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
• Type Docker Desktop and press Enter to launch it.
• Wait for the Docker engine to start (the whale icon stops animating), then re-run docker --version in your terminal.

Now create your project folder and virtual environment.

• Move to your Desktop by running this command:

cd ~/Desktop

• Create the project folder and move into it by running these commands:

mkdir medrag
cd medrag

• Create a Python virtual environment by running:

python3 -m venv venv

• Activate the virtual environment by running:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

You should see (venv) appear at the start of your terminal prompt.

• Install all project dependencies by running:

pip install fastapi uvicorn sqlalchemy streamlit pinecone transformers peft bitsandbytes sentence-transformers datasets psycopg2-binary python-dotenv torch

What are these packages?

• fastapi and uvicorn power the REST API backend.
• sqlalchemy and psycopg2-binary handle PostgreSQL database models and connections.
• streamlit builds the chat frontend.
• pinecone is the SDK for the Pinecone vector database.
• transformers, peft, bitsandbytes, and sentence-transformers handle model loading, fine-tuning, quantization, and embeddings.
• datasets loads the medical QA data from Hugging Face.
• python-dotenv reads environment variables from the .env file.

This install may take several minutes because of the ML libraries.

Install failing with a CUDA or torch error?

If bitsandbytes or torch fails to install, make sure your NVIDIA drivers are up to date. You can verify CUDA availability after install with python3 -c "import torch; print(torch.cuda.is_available())".

Help me troubleshoot a pip install failure for bitsandbytes or torch on my NVIDIA GPU setup.

Configure external services

The project uses Pinecone for vector storage (semantic retrieval) and Hugging Face for model access. Both offer free tiers that cover this project.

• Go to pinecone.io and create a free account (Starter plan).
• Once logged in, click API Keys in the left sidebar.
• Copy your default API key. You will need this shortly for your .env file.

What is the Pinecone Starter plan?

The Starter plan is free. It includes up to 5 serverless indexes, 2GB of storage, and 1M read units per month. The MedQuAD dataset (~16K passages) fits comfortably within these limits.

• Go to huggingface.co and create a free account.
• Click your profile icon in the top right, then click Settings.
• Click Access Tokens in the left sidebar.
• Click New token, give it a name like medrag, set the role to Read, and click Generate.
• Copy the token. You will need this for your .env file.

Finally, you need to accept Meta's license for LLaMA 3.1 so you can download the model weights later.

• Go to the meta-llama/Llama-3.1-8B-Instruct model page on Hugging Face.
• Read and accept the license agreement by clicking Agree and access repository.
• Approval is typically instant.

Why do I need to accept the LLaMA license?

Meta gates access to LLaMA models behind a license agreement. Without accepting it, your Hugging Face token cannot download the model weights. You will use these weights for QLoRA fine-tuning in a later step.

Set up project structure and Docker Compose

A clean project structure makes the codebase navigable and keeps concerns separated. The backend, frontend, and notebooks each get their own directory.

• Create the full directory structure by running these commands:

mkdir -p backend/app/routers backend/app/services frontend notebooks
touch backend/app/__init__.py backend/app/main.py backend/app/models.py backend/app/schemas.py backend/app/database.py
touch backend/requirements.txt frontend/app.py notebooks/finetune_llama.ipynb
touch docker-compose.yml .env .gitignore README.md

What does this structure look like?

Your medrag/ folder now contains:

• backend/app/ for FastAPI code (models, schemas, routers, services).
• frontend/ for the Streamlit chat interface.
• notebooks/ for the QLoRA fine-tuning notebook.
• docker-compose.yml, .env, .gitignore, and README.md at the project root.

• Confirm the files exist by running:

ls

You should see backend, frontend, notebooks, docker-compose.yml, .env, .gitignore, and README.md in the output.

Next, configure Docker Compose to run PostgreSQL in a container.

• Open docker-compose.yml in VS Code (click on it in the Explorer sidebar) and paste the following:

version: '3.8'
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: medrag
      POSTGRES_PASSWORD: medrag_password
      POSTGRES_DB: medrag_db
    ports:
      - '5432:5432'
    volumes:
      - postgres_data:/var/lib/postgresql/data
volumes:
  postgres_data:

What does this Docker Compose file do?

• It pulls the official postgres:15 image and starts a container.
• It creates a database called medrag_db with the user medrag.
• Port 5432 maps to your host so your FastAPI app can connect.
• The postgres_data volume persists data between container restarts.

• Save docker-compose.yml.
• Now open .env in VS Code and paste the following:

PINECONE_API_KEY=your_pinecone_api_key
HUGGINGFACE_TOKEN=your_hf_token
DATABASE_URL=postgresql://medrag:medrag_password@localhost:5432/medrag_db
POSTGRES_USER=medrag
POSTGRES_PASSWORD=medrag_password
POSTGRES_DB=medrag_db

• Replace your_pinecone_api_key with the Pinecone API key you copied earlier.
• Replace your_hf_token with the Hugging Face token you generated.
• Save .env.

Never commit your .env file

The .env file contains credentials. You will add it to .gitignore in Step 8 to keep it out of version control.

Finally, start PostgreSQL and verify connectivity.

• Start the PostgreSQL container by running:

docker compose up -d

What does this command do?

The -d flag runs the container in detached mode (in the background). Docker pulls the postgres:15 image if it is not already cached, then starts the container.

• Verify the container is running:

docker compose ps

You should see a postgres service listed with a status of Up.

• Test database connectivity by running:

docker compose exec postgres psql -U medrag -d medrag_db -c "SELECT 1;"

You should see a result of 1 printed in a table. This confirms PostgreSQL is reachable and the database exists.

Connection refused or container not running?

• Make sure Docker Desktop is running (check the whale icon in your system tray).
• If port 5432 is already in use, another PostgreSQL instance may be running. Stop it or change the port mapping in docker-compose.yml.
• Run docker compose logs postgres to check for startup errors.

Help me troubleshoot my PostgreSQL Docker container.

✔️ Awesome, I've got everything!

Great. Double check you have saved both docker-compose.yml and .env.

ⓧ I'd like to double check the full code

Here is the complete docker-compose.yml:

version: '3.8'
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: medrag
      POSTGRES_PASSWORD: medrag_password
      POSTGRES_DB: medrag_db
    ports:
      - '5432:5432'
    volumes:
      - postgres_data:/var/lib/postgresql/data
volumes:
  postgres_data:

And the complete .env:

PINECONE_API_KEY=your_pinecone_api_key
HUGGINGFACE_TOKEN=your_hf_token
DATABASE_URL=postgresql://medrag:medrag_password@localhost:5432/medrag_db
POSTGRES_USER=medrag
POSTGRES_PASSWORD=medrag_password
POSTGRES_DB=medrag_db

Your development environment is fully configured. Next up, you will design the database schema and build the FastAPI backend that powers the chatbot's conversation management.

Design the Database and FastAPI Backend

Your development environment is ready with Docker running PostgreSQL and all Python packages installed. Now it's time to build the backend that powers your chatbot's conversation management.

Every chat application needs two foundational pieces: a persistent data layer that stores conversations, and a structured API that lets the frontend create, read, update, and delete those conversations. In this step, you'll define SQLAlchemy models for your database tables, build FastAPI endpoints for full chat CRUD, and test everything through Swagger's interactive docs.

In this step, get ready to:

• Define SQLAlchemy models for chat sessions and messages with full relationship mapping.
• Build FastAPI CRUD endpoints for creating, listing, reading, deleting, and renaming chats.
• Test all endpoints via the auto-generated Swagger UI.

Set up the database connection layer

Before you can store anything in PostgreSQL, your application needs a connection layer that knows how to reach the database and manage sessions. The database.py module handles this with three key pieces: an engine (the connection), a session factory (for transactions), and a dependency function that FastAPI can inject into each request.

• Open backend/app/database.py in VS Code and add the following code:

from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
import os
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

# Read the database connection string from environment
DATABASE_URL = os.getenv('DATABASE_URL')

# Create the SQLAlchemy engine (connection to PostgreSQL)
engine = create_engine(DATABASE_URL)

# Create a session factory for database transactions
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)

# Base class for all ORM models
Base = declarative_base()

def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

What does this code do?

load_dotenv() reads your .env file and loads DATABASE_URL into the environment. It automatically searches the script's directory and parent directories, so it finds your project root's .env file.

The get_db() function is a FastAPI dependency. Each API request gets a fresh database session, and the finally block guarantees the session closes even if the request fails.

• Save backend/app/database.py.
• Verify the database connection works by running this from your medrag project directory:

python -c "from backend.app.database import engine; print('Connected:', engine.url)"

You should see output containing your PostgreSQL connection string.

Seeing a ModuleNotFoundError?

Make sure you are running the command from the medrag project root directory (the one containing docker-compose.yml). Also confirm your virtual environment is still activated.

If you see an error about dotenv not being found, install it with pip install python-dotenv.

help me debug my database.py import error

Define the Chat and Message models

With the connection layer in place, you can now define the shape of your data. Your chatbot needs two tables: one for chat sessions (with a title and timestamps) and one for individual messages (linked to a chat, with a role indicating user or assistant).

• Open backend/app/models.py and add the following ORM models:

from sqlalchemy import Column, Integer, String, Text, DateTime, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.sql import func
from .database import Base

class Chat(Base):
    __tablename__ = 'chats'
    id = Column(Integer, primary_key=True, index=True)
    title = Column(String(255), default='New Chat')
    created_at = Column(DateTime(timezone=True), server_default=func.now())
    updated_at = Column(DateTime(timezone=True), onupdate=func.now())
    # One chat has many messages; deleting a chat removes its messages
    messages = relationship('Message', back_populates='chat', cascade='all, delete-orphan')

class Message(Base):
    __tablename__ = 'messages'
    id = Column(Integer, primary_key=True, index=True)
    chat_id = Column(Integer, ForeignKey('chats.id'), nullable=False)
    role = Column(String(50), nullable=False)
    content = Column(Text, nullable=False)
    sources_json = Column(Text, nullable=True)
    created_at = Column(DateTime(timezone=True), server_default=func.now())
    # Back-reference to the parent chat
    chat = relationship('Chat', back_populates='messages')

What does this code do?

The Chat model stores each conversation session. The updated_at column auto-updates whenever you modify the chat (like renaming it).

The Message model stores individual messages. The role field distinguishes between user and assistant messages. The sources_json field will hold citations from the retrieval pipeline later.

The cascade='all, delete-orphan' setting means deleting a chat automatically removes all its messages from the database.

• Save backend/app/models.py.
• Verify the models load correctly and can create tables by running:

python -c "from backend.app.database import engine; from backend.app.models import Base; Base.metadata.create_all(bind=engine); print('Tables created!')"

You should see Tables created! printed with no errors.

Seeing a connection refused error?

Make sure your PostgreSQL container is still running. Run docker compose up -d from your project root to restart it.

help me fix my PostgreSQL connection error

Build FastAPI CRUD endpoints

Your data layer is ready. Now you need an API that the frontend can call to manage conversations. FastAPI will auto-generate interactive documentation from your endpoint definitions, making testing effortless.

• Open backend/app/main.py and replace any placeholder content with the following:

from fastapi import FastAPI, Depends, HTTPException
from sqlalchemy.orm import Session
from . import models, schemas
from .database import engine, get_db
from typing import List

# Auto-create database tables on startup
models.Base.metadata.create_all(bind=engine)

app = FastAPI(title='MedRAG API')

@app.post('/chats', response_model=schemas.ChatResponse)
def create_chat(chat: schemas.ChatCreate, db: Session = Depends(get_db)):
    # Create a new chat session
    db_chat = models.Chat(title=chat.title)
    db.add(db_chat)
    db.commit()
    db.refresh(db_chat)
    return db_chat

@app.get('/chats', response_model=List[schemas.ChatResponse])
def list_chats(db: Session = Depends(get_db)):
    # Return all chats, newest first
    return db.query(models.Chat).order_by(models.Chat.created_at.desc()).all()

@app.get('/chats/{chat_id}', response_model=schemas.ChatWithMessages)
def get_chat(chat_id: int, db: Session = Depends(get_db)):
    # Get a single chat with all its messages
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    return chat

@app.delete('/chats/{chat_id}')
def delete_chat(chat_id: int, db: Session = Depends(get_db)):
    # Delete a chat and its messages (cascade)
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    db.delete(chat)
    db.commit()
    return {'message': 'Chat deleted'}

@app.patch('/chats/{chat_id}', response_model=schemas.ChatResponse)
def rename_chat(chat_id: int, update: schemas.ChatUpdate, db: Session = Depends(get_db)):
    # Rename an existing chat
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    chat.title = update.title
    db.commit()
    db.refresh(chat)
    return chat

@app.post('/chats/{chat_id}/messages', response_model=schemas.MessageResponse)
def send_message(chat_id: int, message: schemas.MessageCreate, db: Session = Depends(get_db)):
    # Save the user's message
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    user_msg = models.Message(chat_id=chat_id, role='user', content=message.content)
    db.add(user_msg)
    db.commit()
    # Stub: AI response to be wired in Step 6
    ai_msg = models.Message(chat_id=chat_id, role='assistant', content='[AI response placeholder]')
    db.add(ai_msg)
    db.commit()
    db.refresh(ai_msg)
    return ai_msg

What does this code do?

models.Base.metadata.create_all(bind=engine) auto-creates the chats and messages tables in PostgreSQL when the server starts. This is safe to call repeatedly because it only creates tables that don't already exist.

Each endpoint uses Depends(get_db) to receive a database session. FastAPI injects the session automatically and closes it after the request finishes.

The send_message endpoint currently returns a placeholder response. You'll wire this to the actual RAG pipeline in Step 6.

• Save backend/app/main.py.
• You won't be able to start the server yet because main.py imports schemas which doesn't exist yet. You'll create that next.

Add Pydantic schemas and test the API

Pydantic schemas define the shape of data flowing in and out of your API. They validate incoming requests and serialize database objects into clean JSON responses.

• Open backend/app/schemas.py and add the following schema definitions:

from pydantic import BaseModel
from typing import Optional, List
from datetime import datetime

class MessageBase(BaseModel):
    role: str
    content: str
    sources_json: Optional[str] = None

class MessageCreate(BaseModel):
    # Only content needed from the client; role is set server-side
    content: str

class MessageResponse(MessageBase):
    id: int
    chat_id: int
    created_at: datetime
    class Config:
        from_attributes = True

class ChatBase(BaseModel):
    title: Optional[str] = 'New Chat'

class ChatCreate(ChatBase):
    pass

class ChatUpdate(BaseModel):
    title: str

class ChatResponse(ChatBase):
    id: int
    created_at: datetime
    updated_at: Optional[datetime]
    class Config:
        from_attributes = True

class ChatWithMessages(ChatResponse):
    # Includes full message history when fetching a single chat
    messages: List[MessageResponse] = []

What does this code do?

Each schema class defines what fields an API request or response contains. ChatCreate accepts an optional title. ChatResponse returns the full chat object including its database-generated id and timestamps.

The from_attributes = True config tells Pydantic to read data from SQLAlchemy model attributes instead of expecting a dictionary. This is what lets you return ORM objects directly from your endpoints.

ChatWithMessages inherits from ChatResponse and adds the full list of messages. This schema is used for the single-chat detail endpoint.

• Save backend/app/schemas.py.
• Start the FastAPI server by running this from your medrag project root:

uvicorn backend.app.main:app --reload

You should see output indicating the server is running at http://127.0.0.1:8000.

• Open your browser and navigate to http://localhost:8000/docs.
• You should see the Swagger UI with all your endpoints listed: POST /chats, GET /chats, GET /chats/{chat_id}, DELETE /chats/{chat_id}, PATCH /chats/{chat_id}, and POST /chats/{chat_id}/messages.
• Click on POST /chats to expand it.
• Click Try it out.
• Click Execute to create a new chat with the default title.
• Confirm you see a response body containing an id, title, and created_at timestamp.
• Test GET /chats to verify your new chat appears in the list.
• Test POST /chats/{chat_id}/messages with chat_id set to 1 and a message body like {"content": "What is diabetes?"}.
• Confirm the response includes the placeholder text [AI response placeholder].

Server won't start?

If you see ModuleNotFoundError: No module named 'backend', make sure you are running the uvicorn command from the medrag project root (the folder containing docker-compose.yml), not from inside backend/.

If you see a database connection error, verify your PostgreSQL container is running with docker compose ps.

help me troubleshoot my FastAPI server startup error

Why does FastAPI auto-generate docs?

FastAPI reads your endpoint decorators, Pydantic schemas, and type hints to produce an OpenAPI specification. The Swagger UI at /docs renders this spec as an interactive testing tool. This is one of the key advantages over Flask for API development.

✔️ Awesome, I've got everything!

Great. Double check you've saved all four files and your server is still running.

ⓧ I'd like to double check the full code

Here is the complete code for all four files in this step.

from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
import os
from dotenv import load_dotenv

load_dotenv()

DATABASE_URL = os.getenv('DATABASE_URL')

engine = create_engine(DATABASE_URL)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
Base = declarative_base()

def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

from sqlalchemy import Column, Integer, String, Text, DateTime, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.sql import func
from .database import Base

class Chat(Base):
    __tablename__ = 'chats'
    id = Column(Integer, primary_key=True, index=True)
    title = Column(String(255), default='New Chat')
    created_at = Column(DateTime(timezone=True), server_default=func.now())
    updated_at = Column(DateTime(timezone=True), onupdate=func.now())
    messages = relationship('Message', back_populates='chat', cascade='all, delete-orphan')

class Message(Base):
    __tablename__ = 'messages'
    id = Column(Integer, primary_key=True, index=True)
    chat_id = Column(Integer, ForeignKey('chats.id'), nullable=False)
    role = Column(String(50), nullable=False)
    content = Column(Text, nullable=False)
    sources_json = Column(Text, nullable=True)
    created_at = Column(DateTime(timezone=True), server_default=func.now())
    chat = relationship('Chat', back_populates='messages')

from pydantic import BaseModel
from typing import Optional, List
from datetime import datetime

class MessageBase(BaseModel):
    role: str
    content: str
    sources_json: Optional[str] = None

class MessageCreate(BaseModel):
    content: str

class MessageResponse(MessageBase):
    id: int
    chat_id: int
    created_at: datetime
    class Config:
        from_attributes = True

class ChatBase(BaseModel):
    title: Optional[str] = 'New Chat'

class ChatCreate(ChatBase):
    pass

class ChatUpdate(BaseModel):
    title: str

class ChatResponse(ChatBase):
    id: int
    created_at: datetime
    updated_at: Optional[datetime]
    class Config:
        from_attributes = True

class ChatWithMessages(ChatResponse):
    messages: List[MessageResponse] = []

from fastapi import FastAPI, Depends, HTTPException
from sqlalchemy.orm import Session
from . import models, schemas
from .database import engine, get_db
from typing import List

models.Base.metadata.create_all(bind=engine)

app = FastAPI(title='MedRAG API')

@app.post('/chats', response_model=schemas.ChatResponse)
def create_chat(chat: schemas.ChatCreate, db: Session = Depends(get_db)):
    db_chat = models.Chat(title=chat.title)
    db.add(db_chat)
    db.commit()
    db.refresh(db_chat)
    return db_chat

@app.get('/chats', response_model=List[schemas.ChatResponse])
def list_chats(db: Session = Depends(get_db)):
    return db.query(models.Chat).order_by(models.Chat.created_at.desc()).all()

@app.get('/chats/{chat_id}', response_model=schemas.ChatWithMessages)
def get_chat(chat_id: int, db: Session = Depends(get_db)):
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    return chat

@app.delete('/chats/{chat_id}')
def delete_chat(chat_id: int, db: Session = Depends(get_db)):
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    db.delete(chat)
    db.commit()
    return {'message': 'Chat deleted'}

@app.patch('/chats/{chat_id}', response_model=schemas.ChatResponse)
def rename_chat(chat_id: int, update: schemas.ChatUpdate, db: Session = Depends(get_db)):
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    chat.title = update.title
    db.commit()
    db.refresh(chat)
    return chat

@app.post('/chats/{chat_id}/messages', response_model=schemas.MessageResponse)
def send_message(chat_id: int, message: schemas.MessageCreate, db: Session = Depends(get_db)):
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    user_msg = models.Message(chat_id=chat_id, role='user', content=message.content)
    db.add(user_msg)
    db.commit()
    # Stub: AI response to be wired in Step 6
    ai_msg = models.Message(chat_id=chat_id, role='assistant', content='[AI response placeholder]')
    db.add(ai_msg)
    db.commit()
    db.refresh(ai_msg)
    return ai_msg

Your backend API is fully operational with persistent storage. Next up, you'll populate Pinecone with medical knowledge that your chatbot can retrieve and cite in its answers.

Ingest Medical Data into Pinecone

Your FastAPI backend can manage chat sessions and messages. But for the AI to produce grounded medical answers, it needs access to a knowledge base of real medical information it can search through.

In this step, you'll load over 16,000 medical question-answer pairs from the MedQuAD dataset, chunk them into retrieval-ready passages, embed them with a sentence-transformer model, and store them in a Pinecone vector index for fast semantic search.

In this step, get ready to:

• Load and chunk the MedQuAD medical dataset into retrieval-ready passages.
• Generate dense vector embeddings and upsert them to Pinecone.
• Verify semantic retrieval with sample medical queries.

Load and preprocess the MedQuAD dataset

The MedQuAD dataset on Hugging Face contains 16,407 medical question-answer pairs covering symptoms, treatments, and diagnoses. To make these answers searchable by meaning, you'll split long answers into shorter passages of roughly 200-300 tokens each.

• Create the services directory inside backend/app/ by running this command:

mkdir -p backend/app/services

• Create a new file called ingest.py inside backend/app/services/.
• Add the imports, configuration constants, and text chunking function by pasting this code:

import os
from datasets import load_dataset
from pinecone import Pinecone, ServerlessSpec
from sentence_transformers import SentenceTransformer
from dotenv import load_dotenv
import textwrap

load_dotenv()

# Configuration
BICODER_MODEL = 'sentence-transformers/all-MiniLM-L6-v2'
INDEX_NAME = 'medrag-index'
BATCH_SIZE = 100

def chunk_text(text, max_tokens=250):
    """Split text into chunks of roughly max_tokens words each."""
    words = text.split()
    chunks = []
    for i in range(0, len(words), max_tokens):
        chunks.append(' '.join(words[i:i+max_tokens]))
    return chunks

What does this code do?

• BICODER_MODEL sets the sentence-transformer model that will encode passages into 384-dimensional vectors. This model is lightweight, fast, and well-suited for semantic search.
• INDEX_NAME is the name of the Pinecone index where all medical vectors will live.
• BATCH_SIZE controls how many vectors get sent to Pinecone in a single API call. Batching avoids hitting rate limits.
• chunk_text() splits long medical answers into smaller passages (~250 words each) so that retrieval returns focused, relevant chunks instead of entire articles.

Generate embeddings and upsert to Pinecone

The main ingestion pipeline connects to Pinecone, creates the index if it does not exist, loads the MedQuAD dataset, encodes every passage, and batch-upserts the vectors with metadata.

• Add the ingest() function below chunk_text in backend/app/services/ingest.py:

def ingest():
    # Initialize Pinecone client and sentence encoder
    pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))
    encoder = SentenceTransformer(BICODER_MODEL)
    dim = encoder.get_sentence_embedding_dimension()

    # Create index if it does not already exist
    if INDEX_NAME not in [idx.name for idx in pc.list_indexes()]:
        pc.create_index(
            name=INDEX_NAME,
            dimension=dim,
            metric='cosine',
            spec=ServerlessSpec(cloud='aws', region='us-east-1')
        )
    index = pc.Index(INDEX_NAME)

    # Load the MedQuAD dataset from Hugging Face
    dataset = load_dataset('keivalya/MedQuad-MedicalQnADataset', split='train')
    vectors = []
    for i, row in enumerate(dataset):
        question = row.get('Question', '')
        answer = row.get('Answer', '')
        # Chunk the answer into retrieval-ready passages
        chunks = chunk_text(answer)
        for j, chunk in enumerate(chunks):
            vec_id = f'doc-{i}-chunk-{j}'
            embedding = encoder.encode(chunk).tolist()
            vectors.append({'id': vec_id, 'values': embedding, 'metadata': {'question': question, 'text': chunk}})
            # Batch upsert when buffer is full
            if len(vectors) >= BATCH_SIZE:
                index.upsert(vectors=vectors)
                vectors = []
    # Upsert any remaining vectors
    if vectors:
        index.upsert(vectors=vectors)
    print('Ingestion complete.')

if __name__ == '__main__':
    ingest()

What does this code do?

• pc.create_index() creates a serverless Pinecone index with cosine similarity on the free Starter plan in us-east-1. The dimension (384) is automatically read from the encoder model.
• The loop iterates over all 16,407 MedQuAD rows, chunks each answer, encodes each chunk into a 384-dimensional vector, and attaches the original question and passage text as metadata.
• Vectors are batched into groups of 100 before upserting. This keeps the script efficient without overwhelming the API.
• The if __name__ == '__main__' block lets you run this file directly as a standalone script.

Why a two-stage retrieval pipeline?

This script handles the first stage: fast bi-encoder retrieval. A bi-encoder encodes the query and passages independently, so you can pre-compute all passage embeddings once and search millions of vectors in milliseconds.

In the next step, you'll add a cross-encoder re-ranker that scores each (query, passage) pair together for higher accuracy. The bi-encoder finds the candidates quickly. The cross-encoder picks the best ones precisely.

• Save backend/app/services/ingest.py.
• Run the ingestion script by executing this command from your project root:

python -m backend.app.services.ingest

This will take several minutes. The script downloads the MedQuAD dataset, loads the sentence-transformer model, encodes all passages, and uploads them to Pinecone in batches.

When it finishes, you should see Ingestion complete. printed in your terminal.

Script failing or timing out?

• If you see an authentication error, confirm your PINECONE_API_KEY is set correctly in your .env file. The key must match what you see in the Pinecone console.
• If the model download hangs, check your internet connection. The all-MiniLM-L6-v2 model is about 90MB.
• If you see a module not found error for datasets or sentence_transformers, make sure your virtual environment is activated and all packages from Step 1 are installed.

Having trouble with the ingestion?

Verify retrieval with sample queries

Now that the vectors are stored, you can query the index to confirm that semantically relevant passages come back for medical questions.

• Open a Python shell or create a temporary test file and run the following verification code:

import os
from pinecone import Pinecone
from sentence_transformers import SentenceTransformer
from dotenv import load_dotenv

load_dotenv()

# Connect to the existing index
pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))
index = pc.Index('medrag-index')

# Encode a sample medical question
encoder = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
query = 'What are the symptoms of diabetes?'
query_embedding = encoder.encode(query).tolist()

# Query Pinecone for the top 5 most relevant passages
results = index.query(vector=query_embedding, top_k=5, include_metadata=True)
for match in results['matches']:
    print(f"Score: {match['score']:.4f}")
    print(f"Text: {match['metadata']['text'][:100]}...")
    print()

What should I see?

You should see 5 results with similarity scores between 0 and 1. Higher scores mean stronger semantic relevance. Each result should contain medical text related to diabetes symptoms.

If scores are above 0.3 and the passages discuss diabetes-related topics (blood sugar, insulin, thirst, fatigue), your index is working correctly.

Getting zero results?

• Check the Pinecone console to confirm your medrag-index shows a non-zero vector count. If it shows 0, the ingestion script may not have completed successfully.
• Make sure you are using the same model (sentence-transformers/all-MiniLM-L6-v2) for both encoding and querying. Mismatched models produce incompatible vectors.

Why am I getting bad results from my Pinecone query?

✔️ Awesome, I've got everything!

Great. Make sure you have saved backend/app/services/ingest.py and confirmed retrieval is working.

ⓧ I'd like to double check the full code

import os
from datasets import load_dataset
from pinecone import Pinecone, ServerlessSpec
from sentence_transformers import SentenceTransformer
from dotenv import load_dotenv
import textwrap

load_dotenv()

BICODER_MODEL = 'sentence-transformers/all-MiniLM-L6-v2'
INDEX_NAME = 'medrag-index'
BATCH_SIZE = 100

def chunk_text(text, max_tokens=250):
    words = text.split()
    chunks = []
    for i in range(0, len(words), max_tokens):
        chunks.append(' '.join(words[i:i+max_tokens]))
    return chunks

def ingest():
    pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))
    encoder = SentenceTransformer(BICODER_MODEL)
    dim = encoder.get_sentence_embedding_dimension()

    if INDEX_NAME not in [idx.name for idx in pc.list_indexes()]:
        pc.create_index(
            name=INDEX_NAME,
            dimension=dim,
            metric='cosine',
            spec=ServerlessSpec(cloud='aws', region='us-east-1')
        )
    index = pc.Index(INDEX_NAME)

    dataset = load_dataset('keivalya/MedQuad-MedicalQnADataset', split='train')
    vectors = []
    for i, row in enumerate(dataset):
        question = row.get('Question', '')
        answer = row.get('Answer', '')
        chunks = chunk_text(answer)
        for j, chunk in enumerate(chunks):
            vec_id = f'doc-{i}-chunk-{j}'
            embedding = encoder.encode(chunk).tolist()
            vectors.append({'id': vec_id, 'values': embedding, 'metadata': {'question': question, 'text': chunk}})
            if len(vectors) >= BATCH_SIZE:
                index.upsert(vectors=vectors)
                vectors = []
    if vectors:
        index.upsert(vectors=vectors)
    print('Ingestion complete.')

if __name__ == '__main__':
    ingest()

Your Pinecone index is loaded with medical knowledge. Next up, you'll build the retrieval service that queries this index and adds a cross-encoder re-ranking layer for precise, high-quality passage selection.

Build the Two-Stage Retrieval and Re-Ranking Service

Your Pinecone index now holds ~16,000 medical passages, but raw vector similarity alone is not enough for accurate medical retrieval. A bi-encoder is fast at finding rough matches across the full index, but it can surface passages that are topically related yet not truly relevant to the user's question.

Professional retrieval systems solve this with a two-stage pipeline: retrieve broadly first, then re-rank precisely. In this step, you will build a RetrievalService that queries Pinecone for candidates, re-ranks them with a biomedical cross-encoder, and then build a ContextManager that assembles those results alongside conversation history into a prompt-ready context string.

In this step, get ready to:

• Build a retrieval service that encodes queries and fetches top-20 candidates from Pinecone.
• Integrate a MedCPT cross-encoder that re-ranks candidates and returns the top-5 most relevant passages.
• Create a context manager that combines retrieved passages with recent conversation history.

Implement the retrieval service

The retrieval service is the bridge between user questions and your medical knowledge base. It encodes a query with the same bi-encoder used during ingestion, searches Pinecone for the closest embeddings, and prepares the raw candidates for re-ranking.

• Create a new file called retrieval.py inside the backend/app/services/ folder by running:

touch backend/app/services/retrieval.py

• Confirm the file exists by running:

ls backend/app/services/

You should see retrieval.py alongside ingest.py and __init__.py in the output.

• Open backend/app/services/retrieval.py in your editor and add the imports and configuration constants:

import os
from pinecone import Pinecone
from sentence_transformers import SentenceTransformer, CrossEncoder
from dotenv import load_dotenv
from typing import List, Dict

load_dotenv()

BIENCODER_MODEL = 'sentence-transformers/all-MiniLM-L6-v2'
CROSS_ENCODER_MODEL = 'ncbi/MedCPT-Cross-Encoder'
INDEX_NAME = 'medrag-index'
TOP_K_RETRIEVE = 20
TOP_K_RERANK = 5

What does this code do?

This sets up the foundation for the retrieval service.

• BIENCODER_MODEL points to the same embedding model used during ingestion so queries land in the same vector space as your indexed passages.
• CROSS_ENCODER_MODEL points to ncbi/MedCPT-Cross-Encoder, a biomedical re-ranker trained on 18M PubMed query-article pairs.
• TOP_K_RETRIEVE = 20 is the broad candidate pool the bi-encoder fetches. TOP_K_RERANK = 5 is the final set after the cross-encoder refines relevance.

Add cross-encoder re-ranking

A bi-encoder encodes the query and each passage independently, so it can miss subtle relevance signals. A cross-encoder passes both the query and a candidate passage through the transformer together, enabling full attention between them for much more accurate scoring.

• Below the constants in retrieval.py, add the RetrievalService class:

class RetrievalService:
    def __init__(self):
        self.biencoder = SentenceTransformer(BIENCODER_MODEL)
        self.cross_encoder = CrossEncoder(CROSS_ENCODER_MODEL)
        pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))
        self.index = pc.Index(INDEX_NAME)

    def retrieve(self, query: str) -> List[Dict]:
        query_embedding = self.biencoder.encode(query).tolist()
        results = self.index.query(vector=query_embedding, top_k=TOP_K_RETRIEVE, include_metadata=True)
        candidates = [{'text': m.metadata.get('text', ''), 'question': m.metadata.get('question', ''), 'score': m.score} for m in results.matches]
        pairs = [[query, c['text']] for c in candidates]
        cross_scores = self.cross_encoder.predict(pairs)
        for i, c in enumerate(candidates):
            c['rerank_score'] = float(cross_scores[i])
        reranked = sorted(candidates, key=lambda x: x['rerank_score'], reverse=True)
        return reranked[:TOP_K_RERANK]

What does this code do?

The RetrievalService class orchestrates the full two-stage pipeline in a single retrieve() call.

• Stage 1 (retrieval): The bi-encoder encodes the query into an embedding, queries Pinecone for the 20 nearest vectors, and extracts passage text and metadata from the results.
• Stage 2 (re-ranking): Each candidate is paired with the original query and scored by the MedCPT cross-encoder. The cross-encoder processes both texts jointly, producing a more accurate relevance score.
• The candidates are sorted by their re-rank score in descending order, and only the top 5 are returned.

Why two stages instead of just one?

A bi-encoder is fast because it pre-computes embeddings. It can scan millions of vectors in milliseconds. But it encodes query and passage independently, so it misses nuanced relevance cues.

A cross-encoder is slow because it runs full transformer attention over each query-passage pair. Scoring 16,000 pairs would take minutes. By using the bi-encoder to narrow to 20 candidates first, you get the accuracy of a cross-encoder at a fraction of the cost.

• Save retrieval.py.

Now test the retrieval pipeline to confirm both stages work together.

• Open a Python shell in your terminal by running:

python -c "
from backend.app.services.retrieval import RetrievalService
service = RetrievalService()
results = service.retrieve('What are the symptoms of diabetes?')
for r in results:
    print(f\"Score: {r['rerank_score']:.4f} | {r['text'][:80]}...\")
"

You should see 5 passages printed, each with a re-rank score and a preview of the passage text. The passages should relate to diabetes symptoms.

Not seeing results or getting an error?

• Check that your .env file contains a valid PINECONE_API_KEY. The service reads it via load_dotenv().
• If you see a model download progress bar, wait for it to finish. The ncbi/MedCPT-Cross-Encoder model downloads on first use (~440MB).
• Make sure you run the command from your project root directory so that backend.app.services.retrieval resolves correctly.

help me debug my RetrievalService

✔️ Awesome, I've got everything!

Great. Make sure retrieval.py is saved.

ⓧ I'd like to double check the full code

import os
from pinecone import Pinecone
from sentence_transformers import SentenceTransformer, CrossEncoder
from dotenv import load_dotenv
from typing import List, Dict

load_dotenv()

BIENCODER_MODEL = 'sentence-transformers/all-MiniLM-L6-v2'
CROSS_ENCODER_MODEL = 'ncbi/MedCPT-Cross-Encoder'
INDEX_NAME = 'medrag-index'
TOP_K_RETRIEVE = 20
TOP_K_RERANK = 5

class RetrievalService:
    def __init__(self):
        self.biencoder = SentenceTransformer(BIENCODER_MODEL)
        self.cross_encoder = CrossEncoder(CROSS_ENCODER_MODEL)
        pc = Pinecone(api_key=os.getenv('PINECONE_API_KEY'))
        self.index = pc.Index(INDEX_NAME)

    def retrieve(self, query: str) -> List[Dict]:
        query_embedding = self.biencoder.encode(query).tolist()
        results = self.index.query(vector=query_embedding, top_k=TOP_K_RETRIEVE, include_metadata=True)
        candidates = [{'text': m.metadata.get('text', ''), 'question': m.metadata.get('question', ''), 'score': m.score} for m in results.matches]
        pairs = [[query, c['text']] for c in candidates]
        cross_scores = self.cross_encoder.predict(pairs)
        for i, c in enumerate(candidates):
            c['rerank_score'] = float(cross_scores[i])
        reranked = sorted(candidates, key=lambda x: x['rerank_score'], reverse=True)
        return reranked[:TOP_K_RERANK]

Integrate context management

The retrieval service returns the most relevant medical passages, but the LLM also needs to know what the user said earlier in the conversation. The ContextManager combines both sources into a single structured string that becomes part of the LLM prompt.

• Create a new file called context_manager.py inside backend/app/services/ by running:

touch backend/app/services/context_manager.py

• Open backend/app/services/context_manager.py and add the following code:

from typing import List, Dict

WINDOW_SIZE = 6

class ContextManager:
    def build_context(self, retrieved_passages: List[Dict], conversation_history: List[Dict]) -> str:
        medical_context = '\n\n'.join(
            [f'[Source {i+1}] {p["text"]}' for i, p in enumerate(retrieved_passages)]
        )
        recent_messages = conversation_history[-WINDOW_SIZE:] if len(conversation_history) > WINDOW_SIZE else conversation_history
        history_str = '\n'.join([f'{m["role"].capitalize()}: {m["content"]}' for m in recent_messages])
        context = f'Medical Knowledge:\n{medical_context}\n\nConversation History:\n{history_str}'
        return context

What does this code do?

The ContextManager assembles a structured prompt context from two sources.

• WINDOW_SIZE = 6 limits conversation history to the last 6 messages. This prevents the prompt from growing unbounded while keeping enough context for multi-turn conversations.
• The medical context section labels each passage with [Source 1], [Source 2], etc. so the LLM can cite its sources when generating answers.
• The conversation history section formats each message as User: ... or Assistant: ... so the LLM understands the dialogue flow.

• Save context_manager.py.
• Test the context manager in your terminal by running:

python -c "
from backend.app.services.context_manager import ContextManager

cm = ContextManager()

passages = [{'text': 'Diabetes symptoms include increased thirst.'}, {'text': 'Type 2 diabetes affects insulin resistance.'}]
history = [{'role': 'user', 'content': 'What is diabetes?'}, {'role': 'assistant', 'content': 'Diabetes is a metabolic condition.'}]

print(cm.build_context(passages, history))
"

You should see a formatted string with a Medical Knowledge: section showing numbered sources, followed by a Conversation History: section with the user and assistant messages.

Seeing a ModuleNotFoundError?

• Make sure you are running the command from your project root directory (the folder that contains the backend/ directory).
• Check that your virtual environment is activated. The prompt should show the environment name.

help me fix my import error for context_manager

✔️ Awesome, I've got everything!

Great. Make sure context_manager.py is saved.

ⓧ I'd like to double check the full code

from typing import List, Dict

WINDOW_SIZE = 6

class ContextManager:
    def build_context(self, retrieved_passages: List[Dict], conversation_history: List[Dict]) -> str:
        medical_context = '\n\n'.join(
            [f'[Source {i+1}] {p["text"]}' for i, p in enumerate(retrieved_passages)]
        )
        recent_messages = conversation_history[-WINDOW_SIZE:] if len(conversation_history) > WINDOW_SIZE else conversation_history
        history_str = '\n'.join([f'{m["role"].capitalize()}: {m["content"]}' for m in recent_messages])
        context = f'Medical Knowledge:\n{medical_context}\n\nConversation History:\n{history_str}'
        return context

Your retrieval and context pipeline is complete. Next up, you will fine-tune LLaMA 3.1 8B with QLoRA so it generates grounded medical answers using the context this pipeline produces.

Fine-Tune LLaMA 3.1 8B with QLoRA

Your retrieval pipeline can find relevant medical passages and re-rank them by relevance. The missing piece is a language model that can read those passages and generate grounded, medically accurate answers.

In this step, you will fine-tune Meta's LLaMA 3.1 8B model on medical question-answer data using QLoRA. QLoRA lets you train a large language model on a single GPU by quantizing the base weights to 4-bit precision and training only small adapter layers. You will run training in Google Colab (free T4 GPU), then deploy the adapter locally for inference.

In this step, get ready to:

• Format MedQuAD data as Llama 3.1 instruction-following conversations.
• Train a QLoRA adapter in Google Colab with 4-bit quantization.
• Deploy the fine-tuned model locally and verify generation quality.

Prepare training data in chat format

LLaMA 3.1 Instruct models expect conversations formatted with special header tokens that separate system, user, and assistant turns. You need to transform the MedQuAD question-answer pairs into this chat template so the model learns to respond as a medical assistant.

• Open Google Colab in your browser.
• Click New notebook to create a fresh notebook.
• Rename the notebook to finetune_llama.ipynb by clicking the title at the top left.

• Set the runtime to GPU by clicking Runtime in the menu bar, then Change runtime type.
• Select T4 GPU from the Hardware accelerator dropdown.
• Click Save.

Why Google Colab?

QLoRA training for an 8B model requires around 12-16GB of VRAM. Colab's free T4 GPU provides 16GB, which is enough to train your adapter without paying for cloud GPU instances.

• Install the training dependencies by pasting this into the first cell and running it:

!pip install transformers peft bitsandbytes datasets trl accelerate

What are these packages?

transformers provides the LLaMA model and tokenizer. peft adds LoRA adapter support. bitsandbytes enables 4-bit quantization.

trl provides the SFTTrainer for supervised fine-tuning. accelerate handles device placement and mixed-precision training.

• Add a new cell and paste the imports:

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
import torch

• Add a new cell to load the MedQuAD dataset and format each sample as a Llama 3.1 chat conversation:

dataset = load_dataset('keivalya/MedQuad-MedicalQnADataset', split='train')

def format_sample(row):
    return {'text': f'<|begin_of_text|><|start_header_id|>system<|end_header_id|>\nYou are a helpful medical assistant. Answer questions accurately based on medical knowledge.<|eot_id|><|start_header_id|>user<|end_header_id|>\n{row["Question"]}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n{row["Answer"]}<|eot_id|>'}

formatted = dataset.map(format_sample)

What does this formatting do?

Llama 3.1 uses special tokens to delimit conversation turns. Each sample becomes a structured conversation with three parts:

• A system header establishing the medical assistant persona.
• A user header containing the medical question from MedQuAD.
• An assistant header containing the reference answer the model should learn to produce.

• Run Cell 3. You should see a progress bar as the 16,407 rows are mapped to the chat format.

Seeing a KeyError?

Check that the column names in your format_sample function match the dataset exactly. The MedQuAD dataset uses Question and Answer with capital letters.

help me debug a KeyError when formatting the MedQuAD dataset for Llama fine-tuning

Configure QLoRA and train in Colab

With the data formatted, you can load the base LLaMA model in 4-bit precision and attach trainable LoRA adapters. Only the adapter weights (a tiny fraction of the full model) get updated during training.

• Before loading the gated model, authenticate with Hugging Face. Add a new cell and run:

from huggingface_hub import login
from google.colab import userdata

# Authenticate using Colab Secrets
login(userdata.get('HF_TOKEN'))

• To set up the secret, click the Key icon (Secrets) in the left sidebar of Colab.
• Click Add new secret.
• Name it HF_TOKEN and paste your Hugging Face token (the same one from your local .env file).
• Toggle Notebook access on, then run the cell.

• Load the base model in 4-bit quantization by adding a new cell:

# Configure 4-bit quantization (NF4 with bfloat16 compute)
bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16)

model_id = 'meta-llama/Llama-3.1-8B-Instruct'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map='auto')

# Prepare for k-bit training (freezes base, enables gradient checkpointing)
model = prepare_model_for_kbit_training(model)

What is 4-bit quantization?

The full LLaMA 8B model needs ~16GB in half precision. 4-bit NF4 quantization compresses the base weights to ~4-5GB, leaving room for gradients and activations during training.

prepare_model_for_kbit_training freezes the quantized base weights and enables gradient checkpointing, so only the LoRA adapter layers receive gradient updates.

• Run Cell 4. This downloads the model weights (~5GB) and loads them in 4-bit. It may take 2-3 minutes depending on your connection.

Seeing an access denied error?

Make sure you accepted the Meta Llama 3.1 license on the meta-llama/Llama-3.1-8B-Instruct model page on Hugging Face. Also confirm your HF_TOKEN secret is set correctly in the Colab sidebar.

help me fix Hugging Face authentication errors when loading a gated Llama model in Colab

• Apply LoRA adapters to the model by adding a new cell:

# Configure LoRA: rank 16, alpha 32, target attention projections
lora_config = LoraConfig(r=16, lora_alpha=32, target_modules=['q_proj', 'v_proj'], lora_dropout=0.05, bias='none', task_type='CAUSAL_LM')

model = get_peft_model(model, lora_config)

What do these LoRA parameters mean?

• r=16 sets the rank of the adapter matrices. Higher rank = more capacity but more memory.
• lora_alpha=32 scales the adapter's contribution. The effective learning rate scales as alpha/r.
• target_modules=['q_proj', 'v_proj'] attaches adapters to the query and value attention projections in every transformer layer.
• Only these adapter weights (~6.5M parameters) get trained, while the 8B base weights stay frozen.

• Run Cell 5. You should see a message showing the number of trainable parameters (a tiny fraction of the total).

• Start training by adding the final cell:

# Training configuration: 1 epoch, batch size 2 with gradient accumulation
training_args = TrainingArguments(output_dir='./medrag-adapter', num_train_epochs=1, per_device_train_batch_size=2, gradient_accumulation_steps=4, learning_rate=2e-4, fp16=False, bf16=True, logging_steps=50, save_steps=500)

trainer = SFTTrainer(model=model, train_dataset=formatted, dataset_text_field='text', tokenizer=tokenizer, args=training_args, max_seq_length=512)
trainer.train()

# Save the adapter weights
trainer.model.save_pretrained('./medrag-adapter')

What happens during training?

The SFTTrainer processes the formatted MedQuAD conversations, updating only the LoRA adapter weights. With a batch size of 2 and 4 gradient accumulation steps, the effective batch size is 8.

Training one epoch over 16,407 samples takes approximately 30-45 minutes on a T4 GPU. You will see loss values logged every 50 steps.

• Run Cell 6. Training will take approximately 30-45 minutes. Wait for it to complete before proceeding.
• When training finishes, you should see a final training loss value and the adapter files saved to ./medrag-adapter.

Runtime disconnected or out of memory?

If Colab disconnects mid-training, reconnect and check if a checkpoint was saved in ./medrag-adapter (the save_steps=500 setting saves periodically). If you hit an out-of-memory error, try reducing per_device_train_batch_size to 1.

help me fix memory or disconnection issues while training a QLoRA model in Google Colab

Save adapter and deploy locally

The trained adapter is just a few hundred megabytes of weights. You need to download it from Colab and load it alongside the base model on your local machine for inference.

• Download the adapter from Colab by adding a new cell and running:

import shutil
from google.colab import files

# Zip the adapter directory for download
shutil.make_archive('medrag-adapter', 'zip', './medrag-adapter')
files.download('medrag-adapter.zip')

What does this do?

The adapter directory contains multiple files (config, weights, tokenizer info). Zipping it into a single archive makes the browser download reliable. The files.download() call triggers your browser's download dialog.

• Run the cell. Your browser will download medrag-adapter.zip.
• Back on your local machine, unzip the adapter into your project directory by running these commands in your terminal:

mkdir -p adapters/medrag-llama-adapter
unzip ~/Downloads/medrag-adapter.zip -d adapters/medrag-llama-adapter

• Confirm the adapter files exist by running:

ls adapters/medrag-llama-adapter

• You should see files including adapter_config.json and adapter_model.safetensors in the output.

• Create the generation service that loads the base model with your fine-tuned adapter. Create the file backend/app/services/generation.py and paste the following code:

import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
from dotenv import load_dotenv

load_dotenv()

BASE_MODEL = 'meta-llama/Llama-3.1-8B-Instruct'
ADAPTER_PATH = './adapters/medrag-llama-adapter'

class GenerationService:
    def __init__(self):
        # Load base model in 4-bit quantization for efficient inference
        bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16)
        self.tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, token=os.getenv('HUGGINGFACE_TOKEN'))
        base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, quantization_config=bnb_config, device_map='auto', token=os.getenv('HUGGINGFACE_TOKEN'))
        # Attach the fine-tuned LoRA adapter
        self.model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
        self.model.eval()

    def generate(self, prompt: str, max_new_tokens: int = 512) -> str:
        # Tokenize and generate response
        inputs = self.tokenizer(prompt, return_tensors='pt').to(self.model.device)
        with torch.no_grad():
            outputs = self.model.generate(**inputs, max_new_tokens=max_new_tokens, temperature=0.7, do_sample=True)
        # Decode only the new tokens (skip the input prompt)
        return self.tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)

What does this service do?

• BitsAndBytesConfig loads the 8B base model in 4-bit NF4 precision, using only ~5-6GB of VRAM for inference.
• PeftModel.from_pretrained layers your trained LoRA adapter on top of the frozen base weights.
• The generate method tokenizes the prompt, runs inference with sampling (temperature 0.7), and decodes only the newly generated tokens.

• Save backend/app/services/generation.py.
• Verify the model loads and generates by running this quick test in your terminal:

python -c "from backend.app.services.generation import GenerationService; svc = GenerationService(); print(svc.generate('What are the symptoms of diabetes?'))"

• The first run will take 1-2 minutes as it downloads and quantizes the base model. After loading, you should see a medical answer about diabetes symptoms printed to the terminal.

Model loading fails or gives CUDA errors?

Confirm your local NVIDIA GPU has at least 6GB of free VRAM. Close other GPU-intensive applications. Also check that your HUGGINGFACE_TOKEN is set correctly in your .env file and that the adapter path ./adapters/medrag-llama-adapter contains the downloaded weights.

help me fix CUDA or model loading errors when running the GenerationService locally

✔️ Awesome, I've got everything!

Great. Double check you have saved backend/app/services/generation.py and that the adapter files exist at adapters/medrag-llama-adapter.

ⓧ I'd like to double check the full code

Here is the complete Colab notebook code (all cells combined):

# Cell 1: Install deps
!pip install transformers peft bitsandbytes datasets trl accelerate

# Cell 2: Imports
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
import torch

# Cell 3: Load and format dataset
dataset = load_dataset('keivalya/MedQuad-MedicalQnADataset', split='train')

def format_sample(row):
    return {'text': f'<|begin_of_text|><|start_header_id|>system<|end_header_id|>\nYou are a helpful medical assistant. Answer questions accurately based on medical knowledge.<|eot_id|><|start_header_id|>user<|end_header_id|>\n{row["Question"]}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n{row["Answer"]}<|eot_id|>'}

formatted = dataset.map(format_sample)

# Cell 4: Load model in 4-bit
bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16)
model_id = 'meta-llama/Llama-3.1-8B-Instruct'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map='auto')
model = prepare_model_for_kbit_training(model)

# Cell 5: Apply LoRA
lora_config = LoraConfig(r=16, lora_alpha=32, target_modules=['q_proj', 'v_proj'], lora_dropout=0.05, bias='none', task_type='CAUSAL_LM')
model = get_peft_model(model, lora_config)

# Cell 6: Train
training_args = TrainingArguments(output_dir='./medrag-adapter', num_train_epochs=1, per_device_train_batch_size=2, gradient_accumulation_steps=4, learning_rate=2e-4, fp16=False, bf16=True, logging_steps=50, save_steps=500)
trainer = SFTTrainer(model=model, train_dataset=formatted, dataset_text_field='text', tokenizer=tokenizer, args=training_args, max_seq_length=512)
trainer.train()
trainer.model.save_pretrained('./medrag-adapter')

And here is the local generation service:

import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
from dotenv import load_dotenv

load_dotenv()

BASE_MODEL = 'meta-llama/Llama-3.1-8B-Instruct'
ADAPTER_PATH = './adapters/medrag-llama-adapter'

class GenerationService:
    def __init__(self):
        bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16)
        self.tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, token=os.getenv('HUGGINGFACE_TOKEN'))
        base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, quantization_config=bnb_config, device_map='auto', token=os.getenv('HUGGINGFACE_TOKEN'))
        self.model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
        self.model.eval()

    def generate(self, prompt: str, max_new_tokens: int = 512) -> str:
        inputs = self.tokenizer(prompt, return_tensors='pt').to(self.model.device)
        with torch.no_grad():
            outputs = self.model.generate(**inputs, max_new_tokens=max_new_tokens, temperature=0.7, do_sample=True)
        return self.tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)

Your fine-tuned medical model is now running locally. Next up, you will wire this generation service into the FastAPI message endpoint so the chatbot can answer questions end-to-end.

Wire the Generation Pipeline into FastAPI

Your retrieval pipeline pulls the most relevant medical passages from Pinecone. Your GenerationService loads the fine-tuned LLaMA model. Now it is time to connect all the pieces so a single API call triggers the full RAG pipeline.

In this step, you will upgrade the GenerationService to construct structured RAG prompts, wire the message endpoint to orchestrate the complete pipeline, and test end-to-end with real medical questions.

In this step, get ready to:

• Build a RAG-aware generation method with system prompt, context, and source citations.
• Connect the message endpoint to the full retrieval, context, and generation pipeline.
• Test the pipeline end-to-end with medical questions.

Build the generation service with RAG prompts

The current GenerationService accepts a raw prompt string and generates text. For a proper RAG system, the model needs structured input: a system prompt establishing the medical assistant persona, retrieved medical context, conversation history, and the user question.

• In backend/app/services/generation.py, find the imports at the top of the file:

import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
from dotenv import load_dotenv

• Replace the entire imports section with the following, which adds json for serializing sources and typing imports for the new method signatures:

import os
import json
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
from dotenv import load_dotenv
from typing import List, Dict, Tuple

• Next, find the ADAPTER_PATH line below load_dotenv():

ADAPTER_PATH = './adapters/medrag-llama-adapter'

• Add the SYSTEM_PROMPT constant directly below ADAPTER_PATH:

SYSTEM_PROMPT = 'You are a helpful and accurate medical assistant. Answer questions based on the provided medical knowledge. Always cite your sources by referencing [Source N] in your answer.'

Why a system prompt with citation instructions?

The system prompt establishes two critical behaviors. First, it tells the model to ground its answers in the provided medical context rather than relying on training data. Second, the [Source N] citation format lets the frontend show which passages backed each claim.

• Now find the existing generate method inside the GenerationService class:

    def generate(self, prompt: str, max_new_tokens: int = 512) -> str:
        inputs = self.tokenizer(prompt, return_tensors='pt').to(self.model.device)
        with torch.no_grad():
            outputs = self.model.generate(**inputs, max_new_tokens=max_new_tokens, temperature=0.7, do_sample=True)
        return self.tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)

• Replace the entire generate method with the following two new methods. These construct a Llama 3.1 chat-format prompt and return both the generated text and a JSON string of source citations:

    def build_prompt(self, context: str, user_question: str) -> str:
        # Construct Llama 3.1 chat template with system context and user question
        return f'<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n{SYSTEM_PROMPT}\n\n{context}<|eot_id|><|start_header_id|>user<|end_header_id|>\n{user_question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n'

    def generate_with_sources(self, context: str, user_question: str, retrieved_passages: List[Dict]) -> Tuple[str, str]:
        # Build the full RAG prompt from context and question
        prompt = self.build_prompt(context, user_question)
        inputs = self.tokenizer(prompt, return_tensors='pt').to(self.model.device)
        with torch.no_grad():
            outputs = self.model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
        # Decode only the newly generated tokens
        response = self.tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
        # Package sources as JSON for frontend display
        sources_json = json.dumps([{'source_num': i+1, 'text': p['text'][:200], 'question': p['question']} for i, p in enumerate(retrieved_passages)])
        return response, sources_json

What do these methods do?

The build_prompt method assembles a Llama 3.1 instruction-format prompt. The system header contains the medical persona instructions plus the retrieved medical context and conversation history. The user header contains the actual question.

The generate_with_sources method calls build_prompt, runs inference, then packages the top-5 retrieved passages into a JSON string. Each source entry includes its number, a truncated text preview (first 200 characters), and the original question it came from.

• Save backend/app/services/generation.py.

Seeing a syntax error?

Make sure both new methods are indented inside the GenerationService class. The old generate method should be completely removed, not left alongside the new methods.

Help me debug this syntax error

✔️ Awesome, I've got everything!

Great. Double check you have saved your file before moving on.

ⓧ I'd like to double check the full code

import os
import json
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
from dotenv import load_dotenv
from typing import List, Dict, Tuple

load_dotenv()

BASE_MODEL = 'meta-llama/Llama-3.1-8B-Instruct'
ADAPTER_PATH = './adapters/medrag-llama-adapter'

SYSTEM_PROMPT = 'You are a helpful and accurate medical assistant. Answer questions based on the provided medical knowledge. Always cite your sources by referencing [Source N] in your answer.'

class GenerationService:
    def __init__(self):
        bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16)
        self.tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, token=os.getenv('HUGGINGFACE_TOKEN'))
        base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, quantization_config=bnb_config, device_map='auto', token=os.getenv('HUGGINGFACE_TOKEN'))
        self.model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
        self.model.eval()

    def build_prompt(self, context: str, user_question: str) -> str:
        return f'<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n{SYSTEM_PROMPT}\n\n{context}<|eot_id|><|start_header_id|>user<|end_header_id|>\n{user_question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n'

    def generate_with_sources(self, context: str, user_question: str, retrieved_passages: List[Dict]) -> Tuple[str, str]:
        prompt = self.build_prompt(context, user_question)
        inputs = self.tokenizer(prompt, return_tensors='pt').to(self.model.device)
        with torch.no_grad():
            outputs = self.model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
        response = self.tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
        sources_json = json.dumps([{'source_num': i+1, 'text': p['text'][:200], 'question': p['question']} for i, p in enumerate(retrieved_passages)])
        return response, sources_json

Connect the message endpoint to the full pipeline

The POST /chats/{chat_id}/messages endpoint currently saves the user message and returns a placeholder. You will now wire it to orchestrate the full RAG pipeline: retrieve passages, build context, generate a response, and persist everything to PostgreSQL.

• In backend/app/main.py, find the existing import lines at the top of the file:

from fastapi import FastAPI, Depends, HTTPException
from sqlalchemy.orm import Session
from . import models, schemas
from .database import engine, get_db
from typing import List

• Replace those imports with the following, which adds the three service imports:

from fastapi import FastAPI, Depends, HTTPException
from sqlalchemy.orm import Session
from . import models, schemas
from .database import engine, get_db
from .services.retrieval import RetrievalService
from .services.context_manager import ContextManager
from .services.generation import GenerationService
from typing import List

What are these imports?

Each import brings in one layer of the RAG pipeline. RetrievalService handles Pinecone search and cross-encoder re-ranking. ContextManager assembles retrieved passages and conversation history into a single context string. GenerationService loads the fine-tuned model and generates responses.

• Next, find these two lines that create the FastAPI app:

models.Base.metadata.create_all(bind=engine)

app = FastAPI(title='MedRAG API')

• Add the three service instantiation lines directly below app = FastAPI(title='MedRAG API'), so it looks like this:

models.Base.metadata.create_all(bind=engine)

app = FastAPI(title='MedRAG API')

retrieval_service = RetrievalService()
context_manager = ContextManager()
generation_service = GenerationService()

Why instantiate services at module level?

Creating the services at module level means the models (bi-encoder, cross-encoder, LLaMA) load once when the server starts. Each request reuses the same loaded models rather than reloading them per-call. This keeps inference latency low after the initial startup.

• Now find the send_message endpoint function at the bottom of the file:

@app.post('/chats/{chat_id}/messages', response_model=schemas.MessageResponse)
def send_message(chat_id: int, message: schemas.MessageCreate, db: Session = Depends(get_db)):
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    user_msg = models.Message(chat_id=chat_id, role='user', content=message.content)
    db.add(user_msg)
    db.commit()
    ai_msg = models.Message(chat_id=chat_id, role='assistant', content='[AI response placeholder]')
    db.add(ai_msg)
    db.commit()
    db.refresh(ai_msg)
    return ai_msg

• Replace the entire send_message function with the following, which orchestrates the full RAG pipeline:

@app.post('/chats/{chat_id}/messages', response_model=schemas.MessageResponse)
def send_message(chat_id: int, message: schemas.MessageCreate, db: Session = Depends(get_db)):
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    # Save the user message to PostgreSQL
    user_msg = models.Message(chat_id=chat_id, role='user', content=message.content)
    db.add(user_msg)
    db.commit()
    # Build conversation history from existing messages
    history = [{'role': m.role, 'content': m.content} for m in chat.messages]
    # Retrieve and re-rank relevant medical passages
    passages = retrieval_service.retrieve(message.content)
    # Combine passages and history into a context string
    context = context_manager.build_context(passages, history)
    # Generate a grounded response with source citations
    ai_text, sources_json = generation_service.generate_with_sources(context, message.content, passages)
    # Save the AI response with sources to PostgreSQL
    ai_msg = models.Message(chat_id=chat_id, role='assistant', content=ai_text, sources_json=sources_json)
    db.add(ai_msg)
    db.commit()
    db.refresh(ai_msg)
    return ai_msg

What is the pipeline doing step by step?

• The user's message is saved to PostgreSQL first, so it appears in conversation history.
• The history list pulls all existing messages from the chat for context windowing (the ContextManager takes the last 6).
• The RetrievalService encodes the query, fetches 20 candidates from Pinecone, and re-ranks with the cross-encoder to return the top 5.
• The ContextManager merges the top-5 passages and last 6 messages into a formatted string.
• The GenerationService builds the Llama 3.1 prompt, generates the answer, and packages sources as JSON.
• The AI response is saved with sources_json so the frontend can display which passages backed the answer.

• Save backend/app/main.py.

Import errors when the server restarts?

If you see ModuleNotFoundError for the service imports, confirm your project structure has backend/app/services/ as a package (it needs an __init__.py file inside the services folder). Also confirm your virtual environment is activated.

Help me debug this import error

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before moving on.

ⓧ I'd like to double check the full code

from fastapi import FastAPI, Depends, HTTPException
from sqlalchemy.orm import Session
from . import models, schemas
from .database import engine, get_db
from .services.retrieval import RetrievalService
from .services.context_manager import ContextManager
from .services.generation import GenerationService
from typing import List

models.Base.metadata.create_all(bind=engine)

app = FastAPI(title='MedRAG API')

retrieval_service = RetrievalService()
context_manager = ContextManager()
generation_service = GenerationService()

@app.post('/chats', response_model=schemas.ChatResponse)
def create_chat(chat: schemas.ChatCreate, db: Session = Depends(get_db)):
    db_chat = models.Chat(title=chat.title)
    db.add(db_chat)
    db.commit()
    db.refresh(db_chat)
    return db_chat

@app.get('/chats', response_model=List[schemas.ChatResponse])
def list_chats(db: Session = Depends(get_db)):
    return db.query(models.Chat).order_by(models.Chat.created_at.desc()).all()

@app.get('/chats/{chat_id}', response_model=schemas.ChatWithMessages)
def get_chat(chat_id: int, db: Session = Depends(get_db)):
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    return chat

@app.delete('/chats/{chat_id}')
def delete_chat(chat_id: int, db: Session = Depends(get_db)):
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    db.delete(chat)
    db.commit()
    return {'message': 'Chat deleted'}

@app.patch('/chats/{chat_id}', response_model=schemas.ChatResponse)
def rename_chat(chat_id: int, update: schemas.ChatUpdate, db: Session = Depends(get_db)):
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    chat.title = update.title
    db.commit()
    db.refresh(chat)
    return chat

@app.post('/chats/{chat_id}/messages', response_model=schemas.MessageResponse)
def send_message(chat_id: int, message: schemas.MessageCreate, db: Session = Depends(get_db)):
    chat = db.query(models.Chat).filter(models.Chat.id == chat_id).first()
    if not chat:
        raise HTTPException(status_code=404, detail='Chat not found')
    user_msg = models.Message(chat_id=chat_id, role='user', content=message.content)
    db.add(user_msg)
    db.commit()
    history = [{'role': m.role, 'content': m.content} for m in chat.messages]
    passages = retrieval_service.retrieve(message.content)
    context = context_manager.build_context(passages, history)
    ai_text, sources_json = generation_service.generate_with_sources(context, message.content, passages)
    ai_msg = models.Message(chat_id=chat_id, role='assistant', content=ai_text, sources_json=sources_json)
    db.add(ai_msg)
    db.commit()
    db.refresh(ai_msg)
    return ai_msg

Test the full pipeline end-to-end

With both files saved, it is time to restart the server and verify that medical questions produce grounded, cited responses.

• Restart the FastAPI server by running:

uvicorn backend.app.main:app --reload

The server will take longer to start than before because it is loading the LLaMA model, bi-encoder, and cross-encoder into memory. Wait until you see the startup complete message.

• Open your browser and navigate to http://localhost:8000/docs to open the Swagger UI.
• Click the POST /chats endpoint, click Try it out, and send this request body to create a test chat:

{"title": "Test Medical Chat"}

• Note the id in the response (likely 1 if this is your first chat).
• Scroll down to the POST /chats/{chat_id}/messages endpoint.
• Click Try it out, enter your chat ID, and send a medical question:

{"content": "What are the symptoms of type 2 diabetes?"}

• Check the response. You should see a JSON object with role set to assistant, a content field containing a medical answer with [Source N] citations, and a sources_json field listing the passages that backed the answer.

What should the response look like?

The content field contains a natural language answer referencing [Source 1], [Source 2], etc. The sources_json field is a JSON string containing an array of objects, each with source_num, text (a 200-character preview of the passage), and question (the original question that passage answered in MedQuAD).

• Test conversation awareness by sending a follow-up question in the same chat:

{"content": "How is it diagnosed?"}

The model should reference diabetes in its response even though you did not mention it in the follow-up. This proves the conversation history from PostgreSQL is being passed through the context window.

Getting a timeout or 500 error?

Generation with an 8B model can take 10-30 seconds depending on your GPU. If you are hitting a timeout, increase your client timeout. If you see a 500 error, check the server terminal logs for the actual exception traceback.

Common issues: the LoRA adapter path is wrong (verify ./adapters/medrag-llama-adapter exists relative to where you start the server), or CUDA is out of memory (close other GPU-using processes).

Help me debug this pipeline error

Your full RAG pipeline is live. A single API call now triggers retrieval from Pinecone, cross-encoder re-ranking, context assembly, and grounded generation with citations. Next up, you will build the Streamlit chat interface that gives this pipeline a professional frontend.

Build the Streamlit Chat Interface

Your backend can retrieve medical knowledge, re-rank it, generate answers, and persist everything to PostgreSQL. But without a frontend, interacting with it means manually crafting HTTP requests.

In this step, you will build a professional chat interface using Streamlit. The app connects directly to your FastAPI endpoints, giving you a familiar chat experience with sidebar navigation, conversation management, and expandable source citations.

In this step, get ready to:

• Create a Streamlit app with a sidebar layout and main chat area.
• Connect the UI to your FastAPI backend with API client functions.
• Add polish features including auto-generated titles, loading spinners, and timestamps.

Create the chat UI layout

Streamlit's chat elements let you build conversational interfaces entirely in Python. You will use st.chat_message for rendering message bubbles and st.chat_input for the text input box at the bottom.

• Create the frontend directory inside your project folder by running this command:

mkdir frontend

• Create the file frontend/app.py and add the imports and page configuration:

import streamlit as st
import requests
import json
from datetime import datetime

# Base URL for the FastAPI backend
API_BASE = 'http://localhost:8000'

# Configure Streamlit page settings
st.set_page_config(page_title='MedRAG', page_icon='🏥', layout='wide')

What does this code do?

The st.set_page_config call must be the first Streamlit command in your script. It sets the browser tab title, favicon, and uses a wide layout to give the chat area more space.

The API_BASE variable points to your running FastAPI server so all API calls route to the correct address.

Connect to the FastAPI backend

Each user action in the chat UI needs to talk to a backend endpoint. You will create thin wrapper functions that call your REST API and return JSON responses.

• Add the API client functions below the page config in frontend/app.py:

def create_chat():
    # Create a new chat session via the backend
    r = requests.post(f'{API_BASE}/chats', json={'title': 'New Chat'})
    return r.json()

def list_chats():
    # Fetch all existing chat sessions
    r = requests.get(f'{API_BASE}/chats')
    return r.json()

def get_chat(chat_id):
    # Get a single chat with its full message history
    r = requests.get(f'{API_BASE}/chats/{chat_id}')
    return r.json()

def send_message(chat_id, content):
    # Send a user message and receive the AI response
    r = requests.post(f'{API_BASE}/chats/{chat_id}/messages', json={'content': content})
    return r.json()

def delete_chat(chat_id):
    # Delete a chat session and all its messages
    requests.delete(f'{API_BASE}/chats/{chat_id}')

def rename_chat(chat_id, title):
    # Rename a chat session
    requests.patch(f'{API_BASE}/chats/{chat_id}', json={'title': title})

What does this code do?

Each function maps to a single FastAPI endpoint you built in Step 2. The requests library handles the HTTP calls.

The send_message function is the most important one. It triggers the full RAG pipeline on the backend: retrieval, re-ranking, context assembly, and generation. The function returns the AI response with source citations.

• Now add the sidebar UI below the client functions. This initializes session state and renders the chat list with a New Chat button and delete controls:

# Track the currently selected chat
if 'current_chat_id' not in st.session_state:
    st.session_state.current_chat_id = None

with st.sidebar:
    st.title('🏥 MedRAG')
    # Button to start a new conversation
    if st.button('+ New Chat', use_container_width=True):
        new_chat = create_chat()
        st.session_state.current_chat_id = new_chat['id']
        st.rerun()
    st.divider()
    # List all existing chats with delete buttons
    chats = list_chats()
    for chat in chats:
        col1, col2 = st.columns([4, 1])
        with col1:
            if st.button(chat['title'], key=f'chat_{chat["id"]}', use_container_width=True):
                st.session_state.current_chat_id = chat['id']
                st.rerun()
        with col2:
            if st.button('🗑', key=f'del_{chat["id"]}'):
                delete_chat(chat['id'])
                if st.session_state.current_chat_id == chat['id']:
                    st.session_state.current_chat_id = None
                st.rerun()

What does this code do?

Streamlit's st.session_state persists variables between reruns. Without it, clicking a button would reset the selected chat.

The sidebar lists every chat from the API. Each entry has two columns: the chat title (clickable to select) and a trash icon button (clickable to delete). After any action, st.rerun() refreshes the page to reflect the new state.

Add polish and UX features

The main chat area renders messages from the selected conversation. This section adds loading spinners during generation, expandable source citations, message timestamps, and auto-generated chat titles from the first message.

• Add the main chat area below the sidebar block in frontend/app.py:

if st.session_state.current_chat_id:
    chat_data = get_chat(st.session_state.current_chat_id)
    st.title(chat_data['title'])
    # Display all existing messages in the conversation
    for msg in chat_data.get('messages', []):
        with st.chat_message(msg['role']):
            st.markdown(msg['content'])
            # Show expandable source citations for assistant messages
            if msg.get('sources_json'):
                sources = json.loads(msg['sources_json'])
                with st.expander('📚 Sources'):
                    for s in sources:
                        st.write(f"**Source {s['source_num']}**: {s['text']}")
            # Display message timestamp
            st.caption(msg['created_at'])
    # Chat input at the bottom of the page
    if prompt := st.chat_input('Ask a medical question...'):
        with st.chat_message('user'):
            st.markdown(prompt)
        with st.chat_message('assistant'):
            # Show spinner while the RAG pipeline processes
            with st.spinner('Thinking...'):
                response = send_message(st.session_state.current_chat_id, prompt)
            st.markdown(response['content'])
            if response.get('sources_json'):
                sources = json.loads(response['sources_json'])
                with st.expander('📚 Sources'):
                    for s in sources:
                        st.write(f"**Source {s['source_num']}**: {s['text']}")
        # Auto-generate chat title from first message
        if len(chat_data.get('messages', [])) == 0:
            rename_chat(st.session_state.current_chat_id, prompt[:50])
        st.rerun()
else:
    st.title('Welcome to MedRAG 🏥')
    st.write('Create a new chat to get started.')

What does this code do?

When a chat is selected, the app loads all its messages and renders each one inside st.chat_message. The role field determines whether a user or assistant avatar appears.

The st.spinner('Thinking...') block shows a loading indicator while the backend runs retrieval, re-ranking, and generation. Once the response returns, source citations appear in a collapsible st.expander section.

If this is the first message in the conversation (messages list is empty), the app auto-renames the chat to the first 50 characters of the user's question. This gives each conversation a meaningful title in the sidebar.

• Save frontend/app.py.
• Launch the Streamlit app by running this command from your project root:

streamlit run frontend/app.py

• Your browser should open automatically at http://localhost:8501. You should see the welcome screen with the app title and the sidebar showing the MedRAG heading and a + New Chat button.
• Click + New Chat to create a conversation, type a medical question, and confirm you receive a response with expandable sources.

Seeing a connection error?

Make sure your FastAPI server is still running at http://localhost:8000 in a separate terminal. The Streamlit app calls your backend over HTTP, so both need to be running simultaneously.

If you see a "requests" import error, confirm your virtual environment is activated and that you installed requests in it.

Why is my Streamlit app not connecting to FastAPI?

✔️ Awesome, I've got everything!

Great. Double check you have saved frontend/app.py and that both your FastAPI server and Streamlit app are running.

ⓧ I'd like to double check the full code

import streamlit as st
import requests
import json
from datetime import datetime

API_BASE = 'http://localhost:8000'

st.set_page_config(page_title='MedRAG', page_icon='🏥', layout='wide')

def create_chat():
    r = requests.post(f'{API_BASE}/chats', json={'title': 'New Chat'})
    return r.json()

def list_chats():
    r = requests.get(f'{API_BASE}/chats')
    return r.json()

def get_chat(chat_id):
    r = requests.get(f'{API_BASE}/chats/{chat_id}')
    return r.json()

def send_message(chat_id, content):
    r = requests.post(f'{API_BASE}/chats/{chat_id}/messages', json={'content': content})
    return r.json()

def delete_chat(chat_id):
    requests.delete(f'{API_BASE}/chats/{chat_id}')

def rename_chat(chat_id, title):
    requests.patch(f'{API_BASE}/chats/{chat_id}', json={'title': title})

if 'current_chat_id' not in st.session_state:
    st.session_state.current_chat_id = None

with st.sidebar:
    st.title('🏥 MedRAG')
    if st.button('+ New Chat', use_container_width=True):
        new_chat = create_chat()
        st.session_state.current_chat_id = new_chat['id']
        st.rerun()
    st.divider()
    chats = list_chats()
    for chat in chats:
        col1, col2 = st.columns([4, 1])
        with col1:
            if st.button(chat['title'], key=f'chat_{chat["id"]}', use_container_width=True):
                st.session_state.current_chat_id = chat['id']
                st.rerun()
        with col2:
            if st.button('🗑', key=f'del_{chat["id"]}'):
                delete_chat(chat['id'])
                if st.session_state.current_chat_id == chat['id']:
                    st.session_state.current_chat_id = None
                st.rerun()

if st.session_state.current_chat_id:
    chat_data = get_chat(st.session_state.current_chat_id)
    st.title(chat_data['title'])
    for msg in chat_data.get('messages', []):
        with st.chat_message(msg['role']):
            st.markdown(msg['content'])
            if msg.get('sources_json'):
                sources = json.loads(msg['sources_json'])
                with st.expander('📚 Sources'):
                    for s in sources:
                        st.write(f"**Source {s['source_num']}**: {s['text']}")
            st.caption(msg['created_at'])
    if prompt := st.chat_input('Ask a medical question...'):
        with st.chat_message('user'):
            st.markdown(prompt)
        with st.chat_message('assistant'):
            with st.spinner('Thinking...'):
                response = send_message(st.session_state.current_chat_id, prompt)
            st.markdown(response['content'])
            if response.get('sources_json'):
                sources = json.loads(response['sources_json'])
                with st.expander('📚 Sources'):
                    for s in sources:
                        st.write(f"**Source {s['source_num']}**: {s['text']}")
        if len(chat_data.get('messages', [])) == 0:
            rename_chat(st.session_state.current_chat_id, prompt[:50])
        st.rerun()
else:
    st.title('Welcome to MedRAG 🏥')
    st.write('Create a new chat to get started.')

Your full-stack medical chatbot now has a professional interface. Next up, you will Dockerize the entire stack and prepare it for GitHub so anyone can reproduce your project with a single command.

Dockerize and Document for GitHub

Your RAG chatbot is working end-to-end locally. Now it is time to package the entire stack into a reproducible Docker Compose configuration so anyone can run it with a single command.

In this step you will create Dockerfiles for the backend and frontend, update Docker Compose to orchestrate all three services, write a professional README, and push everything to GitHub.

In this step, get ready to:

• Dockerize the FastAPI backend and Streamlit frontend with health checks and dependency ordering.
• Write a comprehensive README with an architecture diagram and setup instructions.
• Initialize a Git repository and push the project to GitHub.

Create Docker Compose for the full stack

Right now the FastAPI server and Streamlit app run directly on your machine. Dockerizing them means anyone with Docker installed can bring up the full stack without manually setting up Python environments.

• First, stop any running services (the FastAPI server, Streamlit, and the existing postgres container) with Ctrl+C in each terminal.
• Stop the existing Docker Compose postgres service by running:

docker compose down

• Create a backend/requirements.txt file that lists the backend dependencies by running:

pip freeze > backend/requirements.txt

• Create a frontend/requirements.txt file that lists the frontend dependencies by running:

echo "streamlit
requests" > frontend/requirements.txt

• Create the backend Dockerfile by adding this file at backend/Dockerfile:

FROM python:3.10-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

It starts from a lightweight Python 3.10 image, copies your requirements.txt to install dependencies, then copies all application code into the container.

The CMD instruction launches Uvicorn on port 8000, bound to all network interfaces so Docker can route traffic into the container.

• Create the frontend Dockerfile by adding this file at frontend/Dockerfile:

FROM python:3.10-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"]

What does this Dockerfile do?

The frontend Dockerfile follows the same pattern. The CMD runs Streamlit on port 8501 so Docker Compose can map that port to your machine.

• Now replace the contents of your existing docker-compose.yml with the full three-service configuration:

version: '3.8'
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    ports:
      - '5432:5432'
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U ${POSTGRES_USER}']
      interval: 10s
      timeout: 5s
      retries: 5
  backend:
    build: ./backend
    ports:
      - '8000:8000'
    env_file: .env
    depends_on:
      postgres:
        condition: service_healthy
    volumes:
      - ./adapters:/app/adapters
  frontend:
    build: ./frontend
    ports:
      - '8501:8501'
    environment:
      - API_BASE=http://backend:8000
    depends_on:
      - backend
volumes:
  postgres_data:

What does this configuration do?

• The postgres service uses a healthcheck that runs pg_isready so the backend waits until the database is fully ready to accept connections.
• The backend service builds from your backend/Dockerfile and mounts the adapters/ folder so the model weights remain on your host machine.
• The frontend service sets API_BASE=http://backend:8000 so that inside the Docker network, Streamlit can reach FastAPI by service name.
• The depends_on keys enforce startup order: postgres first, then backend, then frontend.

• Build and launch all three services by running:

docker compose up --build

• Wait for the logs to settle. You should see Uvicorn reporting it is running on port 8000 and Streamlit announcing it is available on port 8501.
• Open http://localhost:8501 in your browser. You should see the MedRAG chat interface.

Containers failing to start?

• If the backend container exits immediately, check your .env file has DATABASE_URL pointing to postgres as the host (not localhost). Inside Docker, services reference each other by service name.
• If you see a requirements.txt not found error, confirm you created the file inside the backend/ or frontend/ directory, not the project root.

Still stuck? Help me debug my Docker Compose setup for MedRAG

Write a professional README

A strong README is the first thing visitors see on your GitHub repository. It acts as the landing page for your project and signals professionalism to recruiters and collaborators.

• Create a README.md file in the project root with the following content:

# MedRAG: Full-Stack Medical RAG Chatbot

![Python](https://img.shields.io/badge/Python-3.10+-blue) ![FastAPI](https://img.shields.io/badge/FastAPI-0.100+-green) ![Streamlit](https://img.shields.io/badge/Streamlit-1.28+-red)

## Overview
A production-grade medical question-answering chatbot with persistent conversation history, semantic retrieval, cross-encoder re-ranking, and a QLoRA fine-tuned LLaMA 3.1 8B model.

## Architecture
```
Streamlit UI → FastAPI Backend → PostgreSQL (chat history)
                    ↓
              RetrievalService
              Pinecone (bi-encoder) → MedCPT Cross-Encoder Re-ranking
                    ↓
              GenerationService
              LLaMA 3.1 8B + QLoRA Adapter (local GPU)
```

## Features
- Full chat CRUD with persistent history
- Two-stage retrieval: dense bi-encoder + cross-encoder re-ranking
- QLoRA fine-tuned LLaMA 3.1 8B on MedQuAD
- Source citations in every response
- Docker Compose orchestration

## Setup
1. Clone the repo
2. Copy `.env.example` to `.env` and fill in credentials
3. Run `docker compose up --build`
4. Open http://localhost:8501

## Tech Stack
FastAPI, Streamlit, PostgreSQL, SQLAlchemy, Pinecone, sentence-transformers, MedCPT, LLaMA 3.1, PEFT/QLoRA, Docker

What makes this README effective?

• Badge shields at the top give instant visual context about the tech stack.
• The text-based architecture diagram shows how data flows without needing external image hosting.
• The Setup section gives anyone four clear steps to reproduce the project.

• Save README.md.

Prepare the GitHub repository

Before pushing to GitHub, you need a .gitignore to ensure sensitive files and large model weights stay out of version control.

• Create a .gitignore file in your project root with this content:

.env
adapters/
__pycache__/
*.pyc
.venv/
*.egg-info/
dist/
build/
.DS_Store
*.ipynb_checkpoints

Why exclude these files?

• .env contains your API keys. Never commit secrets to a repository.
• adapters/ holds multi-gigabyte model weights. Git is not designed for large binary files.
• __pycache__/ and *.pyc are auto-generated bytecode files that clutter diffs.
• .venv/ is your local virtual environment. Each collaborator creates their own.

• Go to github.com and create a new repository called medrag-chatbot. Do not initialize it with a README, license, or .gitignore.
• Initialize your local Git repository and push by running these commands:

git init -b main
git add .
git commit -m "Initial commit: full-stack MedRAG chatbot"
git remote add origin [[REPO_URL="https://github.com/YOUR_USERNAME/medrag-chatbot.git"]]
git push -u origin main

What do these commands do?

• git init -b main creates a new Git repository with main as the default branch name.
• git add . stages all files that are not excluded by .gitignore.
• git push -u origin main uploads your commit to the remote repository and sets up tracking.

• Refresh your GitHub repository page. You should see all your project files listed along with the rendered README at the bottom of the page.

Push rejected or authentication error?

• If GitHub rejects your push, make sure you did not initialize the remote repository with a README. A conflicting initial commit causes this.
• If you see an authentication error, you may need to set up a personal access token or SSH key. GitHub no longer accepts password authentication for Git operations.

Help me push my MedRAG project to GitHub

✔️ Awesome, I've got everything!

Great. Double check you have saved all files and your GitHub repo is showing the full project.

ⓧ I'd like to double check the full code

Here is the full docker-compose.yml:

version: '3.8'
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    ports:
      - '5432:5432'
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U ${POSTGRES_USER}']
      interval: 10s
      timeout: 5s
      retries: 5
  backend:
    build: ./backend
    ports:
      - '8000:8000'
    env_file: .env
    depends_on:
      postgres:
        condition: service_healthy
    volumes:
      - ./adapters:/app/adapters
  frontend:
    build: ./frontend
    ports:
      - '8501:8501'
    environment:
      - API_BASE=http://backend:8000
    depends_on:
      - backend
volumes:
  postgres_data:

Here is backend/Dockerfile:

FROM python:3.10-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Here is frontend/Dockerfile:

FROM python:3.10-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"]

Here is .gitignore:

.env
adapters/
__pycache__/
*.pyc
.venv/
*.egg-info/
dist/
build/
.DS_Store
*.ipynb_checkpoints

Here is README.md:

# MedRAG: Full-Stack Medical RAG Chatbot

![Python](https://img.shields.io/badge/Python-3.10+-blue) ![FastAPI](https://img.shields.io/badge/FastAPI-0.100+-green) ![Streamlit](https://img.shields.io/badge/Streamlit-1.28+-red)

## Overview
A production-grade medical question-answering chatbot with persistent conversation history, semantic retrieval, cross-encoder re-ranking, and a QLoRA fine-tuned LLaMA 3.1 8B model.

## Architecture
```
Streamlit UI → FastAPI Backend → PostgreSQL (chat history)
                    ↓
              RetrievalService
              Pinecone (bi-encoder) → MedCPT Cross-Encoder Re-ranking
                    ↓
              GenerationService
              LLaMA 3.1 8B + QLoRA Adapter (local GPU)
```

## Features
- Full chat CRUD with persistent history
- Two-stage retrieval: dense bi-encoder + cross-encoder re-ranking
- QLoRA fine-tuned LLaMA 3.1 8B on MedQuAD
- Source citations in every response
- Docker Compose orchestration

## Setup
1. Clone the repo
2. Copy `.env.example` to `.env` and fill in credentials
3. Run `docker compose up --build`
4. Open http://localhost:8501

## Tech Stack
FastAPI, Streamlit, PostgreSQL, SQLAlchemy, Pinecone, sentence-transformers, MedCPT, LLaMA 3.1, PEFT/QLoRA, Docker

Your full-stack medical RAG chatbot is now Dockerized, documented, and published on GitHub. You have a portfolio-ready project that demonstrates end-to-end ML system engineering.

Secret mission

Deploy a Live Demo to Streamlit Community Cloud

Your chatbot works locally, but nobody else can see it. In this mission, you'll make it shareable by creating a demo mode that runs without a local GPU and deploying to Streamlit Community Cloud. You'll also add a polished architecture diagram and a GIF demo to your README for maximum LinkedIn impact.

Clean Up Your Resources

Clean Up Your Resources

This project runs locally via Docker and uses Pinecone's free Starter plan, so there are no ongoing costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

• PostgreSQL database running in Docker (local, free).
• Pinecone index medrag-index (cloud, free Starter plan). Indexes are archived after 7 days of inactivity.
• Hugging Face model weights cached locally (~5GB for the 8B model in 4-bit quantization).
• Streamlit Community Cloud deployment (free tier).
• Local project files on your Desktop.

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with your MedRAG chatbot.

• Docker Compose remains running and serving your local stack.
• Your Pinecone index stays active as long as you query it at least once every 7 days.
• Your Streamlit Community Cloud app stays live at its public URL.
• Model weights remain in ~/.cache/huggingface/ (~5GB).

Pause - I'll come back to this later

Shut down running processes to free up memory while keeping all your data intact.

• Stop all Docker containers (preserves your PostgreSQL data in the Docker volume) by running:
• Your Pinecone index and Streamlit Cloud app remain live with no action needed.
• When you're ready to resume, run docker compose up from your project folder.

docker compose stop

Why stop instead of down?

The docker compose stop command halts your containers without removing them or their data volumes. Your PostgreSQL chat history is preserved and ready when you start back up.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below.

• Remove all Docker containers and data volumes by running:

docker compose down -v

• Delete your Pinecone index. Open a Python shell and run:

from pinecone.grpc import PineconeGRPC as Pinecone

pc = Pinecone(api_key="your-api-key-here")
pc.delete_index(name="medrag-index")

• Delete your Streamlit Community Cloud app. Go to your workspace at share.streamlit.io, click the overflow menu icon next to your MedRAG app, click Delete, and enter the required confirmation string.
• Delete cached Hugging Face model weights by running:

rm -rf ~/.cache/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct

• Delete the local project folder by running:

macOS

cd ~/Desktop
rm -rf medrag

Windows

cd ~\Desktop
rmdir /s /q medrag

Nice Work!

Nice Work!

You just built a full-stack medical RAG chatbot from scratch. This is a production-grade system combining ML engineering, backend development, and NLP system design in a single portfolio-ready application.

Here's what you accomplished:

• Built a FastAPI REST backend with full chat CRUD, PostgreSQL persistence via SQLAlchemy, and a professional Streamlit chat interface with conversation management.
• Created a two-stage retrieval pipeline with Pinecone for dense bi-encoder retrieval (~16K medical passages) and MedCPT Cross-Encoder re-ranking for precise result relevance.
• Fine-tuned LLaMA 3.1 8B with QLoRA on medical QA data and deployed it locally in 4-bit quantization for domain-grounded generation with source citations.
• Deployed a live demo to Streamlit Community Cloud with a demo mode for LinkedIn sharing.

Ready to quiz yourself?