Run Gemma 4 as Your Local AI Assistant

Set up Google's Gemma 4 to run offline on your PC and analyze documents.

Introduction

30 Second Summary

AI assistants like ChatGPT are powerful, but every message you send travels to a remote server. Your prompts, your documents, your images. All processed on someone else's computer.

In this project, you will install and run Google's Gemma 4 model directly on your Windows machine using Ollama. You will chat with it, analyze images, and build a Python script that uses the model to automatically analyze documents. Everything stays on your computer.

What You'll Build

A fully local AI assistant powered by Gemma 4 E4B that can chat, reason, understand images, and analyze documents with no cloud services, no API keys, and no data leaving your machine.

By the end of this project, you'll have:

  • A running Gemma 4 instance served by Ollama on your Windows PC, fully offline and private.
  • Hands-on experience with interactive chat sessions exploring text generation, reasoning with thinking mode, and image analysis.
  • A Python script that reads a text document and sends it to Gemma 4 for automated analysis, including summarization, key point extraction, and question answering.
  • A multi-document comparison tool that reads two documents and asks Gemma 4 to compare them side by side.

Are there any prerequisites?

None. This project includes setup instructions for every tool you need. You just need a Windows machine with at least 16 GB of RAM and about 10 GB of free disk space for the model download.

Install Python and Ollama

Before you can run Gemma 4 or write scripts to interact with it, you need two tools installed on your machine.

Python lets you write scripts that talk to the AI model programmatically. Ollama is the tool that actually downloads, manages, and runs the model on your hardware.

In this step, get ready to:
  • Install Python 3.13 and verify it works.
  • Install Ollama and verify it works.
  • Download the Gemma 4 E4B model to your machine.
Install Python 3.13

Python is the programming language you will use later to build scripts that communicate with Gemma 4.

  • Open the Python 3.13.13 Windows download page in your browser.
  • Scroll down to the Windows section.
  • Click Windows installer (64-bit) to download the installer file.
  • Run the downloaded python-3.13.13-amd64.exe file.

Check the box at the bottom before clicking Install

On the first screen of the installer, check the box that says Add python.exe to PATH at the bottom. This is critical. Without it, your terminal will not recognize the python command.

  • After checking the PATH box, click Install Now.
  • Wait for the installation to complete, then click Close.

Now verify that Python installed correctly.

  • Press the Windows key to open your search bar.
  • Type PowerShell and press Enter to open it.
  • Check your Python version by running this command:
python --version
  • You should see Python 3.13.13 printed in the terminal.
  • Check that pip (Python's package manager) is also available by running:
pip --version
  • You should see a version number and a file path. The exact version number does not matter.

I see version numbers

Python is ready to go. Keep this PowerShell window open for the next substep.

Command not found

This usually means the PATH checkbox was not checked during installation.

  • Close PowerShell.
  • Run the python-3.13.13-amd64.exe installer again.
  • Select Modify.
  • Click Next through the optional features screen.
  • On the Advanced Options screen, check Add Python to environment variables.
  • Click Install.
  • Open a new PowerShell window and try python --version again.

If you are still stuck, help me troubleshoot Python not being recognized in PowerShell.

Install Ollama

Ollama handles everything needed to run an AI model locally. It downloads the model weights, manages memory, and serves the model through an API.

  • Open the Ollama download page in your browser.
  • Click Download for Windows to download OllamaSetup.exe.
  • Run the downloaded OllamaSetup.exe file and follow the prompts to complete the installation.

Now verify that Ollama installed correctly.

  • Open a new PowerShell window. You need a fresh window so it picks up the new Ollama installation.
  • Check your Ollama version by running this command:
ollama --version
  • You should see a version number like ollama version is 0.21.1. Confirm it is 0.20.0 or later. This is the minimum version required for Gemma 4 support.

I see a version number

Ollama is installed and ready. Time to download the model.

Command not found

Make sure you opened a new PowerShell window after installing Ollama. Existing windows do not pick up new installations.

If a new window still does not work, try restarting your computer and running the command again. If you are still stuck, help me troubleshoot Ollama not being recognized in PowerShell.

What is Ollama?

Ollama is a tool that lets you run large language models on your own computer. It handles downloading model files, managing memory, and serving the model through a local API on port 11434. You interact with it through terminal commands or through code.

Pull the Gemma 4 E4B model

Ollama needs to download the model files before you can use Gemma 4. The E4B variant has 4.5 billion effective parameters and fits comfortably in 16 GB of RAM.

  • Download the Gemma 4 model by running this command in PowerShell:
ollama pull gemma4

What does this command do?

This downloads the default Gemma 4 E4B model from Ollama's model registry. The download is about 9.6 GB, so it will take several minutes depending on your internet speed.

You will see a progress bar as the model downloads. Wait for it to finish completely.

  • Verify the model is downloaded by running:
ollama list
  • You should see gemma4 listed in the output with its size and modification date.

Download stuck or failed?

If the download stalls or fails partway through, run ollama pull gemma4 again. Ollama resumes from where it left off. Make sure you have at least 10 GB of free disk space.

If you keep running into issues, help me troubleshoot the Gemma 4 download in Ollama.

Your tools are installed and the model is downloaded. Next up, you will start chatting with Gemma 4 and see what it can do.

Chat with Gemma 4

You have Ollama installed and Gemma 4 downloaded to your machine. The model is sitting on your hard drive, ready to go.

In this step, you will start an interactive chat session with Gemma 4 and explore what it can do. You will test text generation, try its built-in thinking mode for reasoning problems, and experiment with creative and coding tasks.

In this step, get ready to:
  • Start an interactive chat session with Gemma 4 and ask it questions.
  • Test thinking mode to see how the model reasons through logic puzzles.
  • Experiment with creative writing and code generation prompts.
Start your first chat

Ollama has an interactive chat mode that loads the model into memory and gives you a prompt where you can type messages back and forth. It works like a local version of ChatGPT, running entirely on your CPU.

  • Open PowerShell by pressing the Windows key, typing PowerShell, and pressing Enter.
  • Start an interactive chat session by running this command:
ollama run gemma4

What does this command do?

This loads the Gemma 4 E4B model into your RAM and opens an interactive prompt (shown as >>>). The model needs a few seconds to load the first time. Once you see the prompt, you can start typing messages.

You should see a >>> prompt appear after the model finishes loading.

  • Type the following question and press Enter:
What are the three most important things to know about running AI models locally?

The model generates its response word by word. On a CPU-only machine, expect the response to take several seconds to fully appear.

Why is it slower than ChatGPT?

Cloud AI services like ChatGPT run on powerful GPUs in data centers. Your model is running on your CPU, which is much slower for this type of work. The tradeoff is that your data never leaves your machine. Everything stays completely private.

  • Try a few more prompts to get a feel for the model. For example:
Explain what an API is in simple terms.
What is the difference between Python and JavaScript?

Model taking a very long time to respond?

The first response after loading the model is typically the slowest. Subsequent responses should be faster since the model is already in memory. If responses consistently take more than 60 seconds, make sure you do not have other memory-heavy applications open.

Help me troubleshoot slow Gemma 4 responses on my CPU-only Windows machine.

Test thinking mode

Gemma 4 has a built-in thinking mode where it reasons step by step before giving a final answer. Ollama enables thinking by default, so the model automatically engages this mode for complex questions.

  • In the same chat session, type this reasoning puzzle and press Enter:
I have 3 boxes. Box A has a red ball. I swap the contents of Box A and Box B, then swap Box B and Box C. Where is the red ball now?

You should see the model's internal reasoning appear before the final answer. The thinking section shows how the model works through the logic step by step.

What is thinking mode?

Thinking mode lets the model "show its work" before giving a final answer. Ollama handles this automatically through the model's chat template. You do not need to enable it manually. The model decides when to think based on the complexity of your question.

If you want to turn thinking off during a session, type /set nothink at the prompt. To turn it back on, type /set think.

  • Try another reasoning question to see thinking mode in action:
A farmer needs to cross a river with a wolf, a goat, and a cabbage. The boat can only carry the farmer and one item. If left alone, the wolf will eat the goat, and the goat will eat the cabbage. How does the farmer get everything across?

Watch how the model breaks the problem into steps and considers which combinations are safe at each crossing.

Not seeing the thinking section?

Thinking is enabled by default in Ollama. If you only see a final answer without reasoning steps, try typing /set think at the prompt to explicitly enable it. Then ask the reasoning question again.

Help me enable thinking mode for Gemma 4 in Ollama.

Experiment with creative and coding tasks

Gemma 4 can handle more than just factual questions. It can write creatively and generate code. Trying different types of prompts helps you understand the model's strengths and limitations.

  • Try a creative writing prompt:
Write a short poem about a computer that dreams of electric sheep.
  • Now try a coding task:
Write a Python function that checks if a string is a palindrome.

Notice how the model adjusts its style based on the type of request. Creative prompts produce flowing text. Coding prompts produce structured code with explanations.

  • Try one more prompt that combines reasoning with a practical task:
I'm planning a 3-day trip to Tokyo on a budget. Create a day-by-day itinerary with estimated costs.
  • When you are done exploring, type /bye and press Enter to exit the chat session.
/bye

Pro tip: keep it running

Ollama keeps the model loaded in memory for a few minutes after you exit. If you start a new session quickly with ollama run gemma4, it loads almost instantly because the model is still cached.

You have had your first conversation with a fully local AI model. Next up, you will use Gemma 4's multimodal capabilities to analyze images directly from the command line.

Analyze an Image with Gemma 4

In the last step, you chatted with Gemma 4 and explored its text generation and reasoning abilities. Now you are going to push the model further.

Gemma 4 E4B is a multimodal model, which means it can understand more than just text. You can feed it an image and ask questions about what it sees. In this step, you will send an image to Gemma 4 directly from the command line and explore how well it handles visual tasks like description, text extraction, and chart analysis.

In this step, get ready to:
  • Save a test image to your Desktop.
  • Send the image to Gemma 4 and ask it to describe what it sees.
  • Test OCR and detailed analysis on a more complex image.
Find or create a test image

Before you can test image analysis, you need an image file saved to a known location on your machine. Good test options include a screenshot of a webpage, a photo of a handwritten note, a chart, or a photo of a receipt.

  • Take a screenshot of any webpage you have open right now.
  • Press Windows+Shift+S to open the Snipping Tool.
  • Select the area you want to capture.
  • Click the notification that appears and save the screenshot to your Desktop as test-image.png.

Pro tip: pick a text-heavy image

Screenshots of webpages or documents work best for testing because they give the model both visual layout and text to extract. A photo of a receipt or handwritten note is also a great choice.

  • Confirm the file exists by checking your Desktop for test-image.png.

Can't find your screenshot?

If the Snipping Tool notification disappeared before you could save, press Windows+Shift+S again and retake the screenshot. This time, click the notification immediately and use Save As to save it to your Desktop.

Help me save a screenshot on Windows.

Send the image to Gemma 4

With your test image saved, you can now pass it directly to Gemma 4 inside the Ollama interactive chat. Ollama recognizes file paths in your prompt and automatically encodes the image for the model.

  • Open a new PowerShell window.
  • Start a chat session with Gemma 4 by running this command:
ollama run gemma4
  • At the >>> prompt, type the following and press Enter:
Describe what you see in this image C:\Users\[[USERNAME="your Windows username"]]\Desktop\test-image.png

What does this command do?

Ollama scans your prompt for file paths that point to image files. When it finds one, it reads the image, encodes it, and sends it to Gemma 4 alongside your text prompt. You should see Added image '...' appear before the model starts responding.

The model takes a few seconds to process the image and generate a response on CPU. You should see a detailed description of your screenshot.

  • Try a different prompt with the same image to test text extraction:
Extract all the text you can see in this image C:\Users\[[USERNAME="your Windows username"]]\Desktop\test-image.png
  • Try one more prompt to test topic identification:
What is the main topic of this screenshot? C:\Users\[[USERNAME="your Windows username"]]\Desktop\test-image.png

Notice how the model gives different responses depending on how you phrase your question, even though the image is the same.

Seeing "Added image" but no useful response?

If the model responds with generic text instead of describing your image, make sure the file path points to a real image file. Check that the file extension is .png, .jpg, or .webp.

If you see the model treating the path as regular text instead of loading the image, double-check that the path has no extra quotes or spaces around it.

Help me troubleshoot image input in Ollama.

Try OCR and detailed analysis

Simple descriptions are useful, but Gemma 4 can do much more with images. You can ask it to perform OCR (reading text from images), analyze charts, or break down complex visuals into structured information.

  • Find or create a more complex test image. A chart, infographic, or document screenshot works well.
  • Save it to your Desktop as complex-image.png.
  • If you are still in the Gemma 4 chat session, type the following prompt and press Enter:
Read all the text in this document image and organize it into bullet points C:\Users\[[USERNAME="your Windows username"]]\Desktop\complex-image.png
  • Try an analysis-focused prompt with the same image:
Analyze this image and describe any trends, patterns, or key information you see C:\Users\[[USERNAME="your Windows username"]]\Desktop\complex-image.png

How does image analysis work on CPU?

Image prompts take longer than text-only prompts because the model needs to process visual tokens in addition to text tokens. On your CPU, expect each image response to take 10-30 seconds depending on the image complexity. The model processes the image through a vision encoder with about 150 million parameters before reasoning about it.

  • Compare how the model handles your simple screenshot versus the more complex image.
  • When you are done exploring, type /bye and press Enter to exit the chat.

Model taking a very long time to respond?

Image processing on CPU is slower than text-only prompts. If a response takes more than 60 seconds, the image might be very large. Try resizing your image to a smaller resolution before sending it.

Help me speed up image analysis in Ollama.

You have just used a fully local AI model to understand and analyze images without sending any data to the cloud. Next up, you will write a Python script that automates document analysis using Gemma 4's API.

Analyze a Document with Python

So far you have been chatting with Gemma 4 by hand, typing prompts and reading responses one at a time. That works for quick questions, but what if you want to analyze a full document automatically?

In this step, you will write a Python script that reads a text file and sends it to Gemma 4 for analysis. The script uses Ollama's Python library to talk to the model programmatically.

In this step, get ready to:
  • Install the ollama Python package and set up a project folder.
  • Write a Python script that sends a document to Gemma 4 for automated analysis.
  • Run the script and experiment with different analysis prompts.
Install the ollama Python package

The ollama Python library wraps Ollama's REST API so you can call the model with a few lines of Python instead of raw HTTP requests.

  • Install the ollama package by running this command in PowerShell:
pip install ollama==0.6.1

What does this command do?

pip install ollama==0.6.1 downloads and installs version 0.6.1 of the official Ollama Python library. Pinning the version ensures your script behaves the same way every time.

  • Confirm the installation succeeded by checking for a line like Successfully installed ollama-0.6.1 in the output.

Seeing 'pip is not recognized'?

  • This means Python was not added to your system PATH. Reinstall Python and make sure you check Add python.exe to PATH at the bottom of the installer window before clicking Install Now.
  • After reinstalling, close and reopen PowerShell, then try the pip install command again.
  • Help me fix my pip installation
Create your project folder and sample document

Your script needs a document to analyze. You will create a project folder on your Desktop and add a sample text file to it.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and move into it by running these commands:
mkdir gemma4-project
cd gemma4-project
  • Confirm the folder was created by running:
ls

The folder is empty for now. You should see no files listed.

  • Open Notepad by pressing the Windows key, typing Notepad, and pressing Enter.
  • Paste a few paragraphs of text into Notepad. A news article, a Wikipedia section, or any text you find interesting works well.
  • Click File in the top menu bar.
  • Click Save As.
  • Navigate to Desktop > gemma4-project in the save dialog.
  • Type sample-doc.txt as the file name and click Save.
  • Confirm the file was saved by running this in PowerShell:
ls

You should see sample-doc.txt listed in the output.

Don't see sample-doc.txt?

  • Make sure you saved the file to the gemma4-project folder on your Desktop, not to another location like Documents.
  • Check that your PowerShell is in the right folder. Run pwd to see your current directory. It should end with gemma4-project.
  • Help me find my sample-doc.txt file
Write the document analysis script

Now for the exciting part. You will write a Python script that reads your document and sends it to Gemma 4 with instructions to summarize it, extract key points, and raise questions.

  • Open Notepad again by pressing the Windows key, typing Notepad, and pressing Enter.
  • Copy the following code and paste it into Notepad:
import sys
from ollama import chat

# Check that the user provided a file path when running the script
if len(sys.argv) < 2:
    print("Usage: python analyze.py <path-to-document>")
    sys.exit(1)

# Read the document file into a string
file_path = sys.argv[1]
with open(file_path, "r", encoding="utf-8") as f:
    document_text = f.read()

print(f"Loaded document: {file_path}")
print(f"Document length: {len(document_text)} characters")
print("Sending to Gemma 4 for analysis...\n")

What does this code do?

  • sys.argv captures command-line arguments. When you run python analyze.py sample-doc.txt, the file path becomes sys.argv[1].
  • The if len(sys.argv) < 2 check stops the script early if no file path was provided, so you get a helpful error message instead of a crash.
  • open(file_path, "r", encoding="utf-8") reads the entire file into a single string called document_text.

That handles loading the document. Now you need the part that sends it to Gemma 4.

  • Add the following code below what you already have in Notepad:
# Send the document to Gemma 4 with analysis instructions
response = chat(
    model="gemma4",
    messages=[
        {
            "role": "system",
            "content": "You are a document analyst. Analyze the provided document thoroughly and clearly."
        },
        {
            "role": "user",
            "content": f"""Analyze the following document and provide:
1. A brief summary (2-3 sentences)
2. Key points (bullet list)
3. Questions this document raises

--- DOCUMENT START ---
{document_text}
--- DOCUMENT END ---"""
        }
    ]
)

# Print the model's analysis
print(response.message.content)

What does this code do?

  • chat() is the main function from the ollama library. It sends messages to a model and returns the response.
  • The system message tells Gemma 4 what role to play. The user message contains the document text wrapped in clear markers so the model knows where the document starts and ends.
  • response.message.content pulls out the text of the model's reply.
  • Save the file as analyze.py inside your gemma4-project folder. Click File in the top menu bar.
  • Click Save As.
  • Navigate to Desktop > gemma4-project in the save dialog.
  • In the Save as type dropdown, select All Files (*.*).
  • Type analyze.py as the file name and click Save.

Pro tip: save as the right file type

Selecting All Files (*.*) in the save dialog prevents Notepad from adding .txt to the end of your filename. Without this, your file would be saved as analyze.py.txt and Python would not recognize it.

  • Confirm both files are in your project folder by running:
ls

You should see both analyze.py and sample-doc.txt listed in the output.

Awesome, I've got everything!

Double check you have saved analyze.py in your gemma4-project folder on your Desktop.

I'd like to double check the full code

Here is the complete analyze.py file for comparison.

import sys
from ollama import chat

# Check that the user provided a file path when running the script
if len(sys.argv) < 2:
    print("Usage: python analyze.py <path-to-document>")
    sys.exit(1)

# Read the document file into a string
file_path = sys.argv[1]
with open(file_path, "r", encoding="utf-8") as f:
    document_text = f.read()

print(f"Loaded document: {file_path}")
print(f"Document length: {len(document_text)} characters")
print("Sending to Gemma 4 for analysis...\n")

# Send the document to Gemma 4 with analysis instructions
response = chat(
    model="gemma4",
    messages=[
        {
            "role": "system",
            "content": "You are a document analyst. Analyze the provided document thoroughly and clearly."
        },
        {
            "role": "user",
            "content": f"""Analyze the following document and provide:
1. A brief summary (2-3 sentences)
2. Key points (bullet list)
3. Questions this document raises

--- DOCUMENT START ---
{document_text}
--- DOCUMENT END ---"""
        }
    ]
)

# Print the model's analysis
print(response.message.content)
Run the script and experiment with different prompts

Your script is ready. Time to see Gemma 4 analyze your document.

  • Make sure Ollama is running. If you closed it earlier, press the Windows key, type Ollama, and press Enter to start it.
  • In PowerShell, make sure you are inside your gemma4-project folder. Run pwd to check. If you are not there, navigate back by running:
cd ~/Desktop/gemma4-project
  • Run your analysis script by running this command:
python analyze.py sample-doc.txt

The script loads your document, sends it to Gemma 4, and prints the analysis. This takes several seconds on CPU, so be patient while the model processes.

What should I see?

You should see three lines confirming the document was loaded (file path, character count, and a "Sending to Gemma 4" message). After a pause of several seconds, the model's analysis appears with a summary, key points, and questions about the document.

Seeing an error when you run the script?

  • If you see ModuleNotFoundError: No module named 'ollama', the package did not install correctly. Run pip install ollama==0.6.1 again.
  • If you see FileNotFoundError, make sure sample-doc.txt is in the same folder as analyze.py. Run ls to verify both files are listed.
  • If you see a connection error, Ollama is not running. Start it by pressing the Windows key, typing Ollama, and pressing Enter.
  • If your file was saved as analyze.py.txt, Notepad added an extra extension. Rename it by running:
Rename-Item analyze.py.txt analyze.py

Help me debug my analyze.py script

Now try changing the analysis prompt to see different results from the same document.

  • Open analyze.py in Notepad. Right-click the file in File Explorer and select Open with, then Notepad.
  • Find the content string inside the user message. This is the line that starts with Analyze the following document.
  • Replace the analysis instructions with a different prompt. For example, try this:
            "content": f"""Read the following document and provide:
1. A sentiment analysis (is the tone positive, negative, or neutral?)
2. Any potential biases you detect
3. Three follow-up questions a reader might have

--- DOCUMENT START ---
{document_text}
--- DOCUMENT END ---"""
  • Save the file with Ctrl+S.
  • Run the script again by running this command:
python analyze.py sample-doc.txt

You should see a completely different analysis of the same document. The model now focuses on sentiment, bias, and follow-up questions instead of summarization.

Why does the prompt matter so much?

The same document produces very different outputs depending on what you ask for. This is prompt engineering in action. Clear, structured prompts with numbered instructions give you more organized, useful results than vague requests like "analyze this."

You now have a working Python script that automates document analysis with a local AI model. Next up, you will turn this into an interactive tool where you can ask follow-up questions about a document in real time.

Build an Interactive Document Q&A Tool

In the last step, you built a script that sends a document to Gemma 4 for a one-shot analysis. One-shot analysis is useful, but real document work is iterative.

You read a summary, then want to dig deeper into a specific section or ask a follow-up question. In this step, you will extend your script into an interactive Q&A tool that lets you have a back-and-forth conversation about a document.

In this step, get ready to:
  • Create a Python script that loads a document and lets you ask questions about it in a loop.
  • Maintain conversation context so the model remembers your previous questions and answers.
  • Test your Q&A tool with different types of documents.
Create the Q&A script

The key idea behind this script is conversation memory. Each time you ask a question, the script adds both your question and the model's answer to a running list of messages.

This way, Gemma 4 always has the full conversation history when answering your next question.

  • In your gemma4-project folder, create a new file called doc-qa.py.
  • Copy the following code into doc-qa.py:
import sys
from ollama import chat

# Check that the user provided a file path when running the script
if len(sys.argv) < 2:
    print("Usage: python doc-qa.py <path-to-document>")
    sys.exit(1)

# Read the document from the file path provided
file_path = sys.argv[1]
with open(file_path, "r", encoding="utf-8") as f:
    document_text = f.read()

print(f"Loaded document: {file_path}")
print(f"Document length: {len(document_text)} characters")
print("Type your questions below. Type 'quit' to exit.\n")

What does this code do?

  • sys.argv[1] grabs the file path you type after python doc-qa.py when you run the script.
  • The script reads the entire document into a string called document_text.
  • It prints the file name and character count so you can confirm the right document was loaded.
  • Save doc-qa.py.

Next, you will add the conversation setup. This is where the script tells Gemma 4 what role to play and feeds it the document.

  • Add the following code to the bottom of doc-qa.py, below the code you just pasted:
# Set up the conversation with a system prompt and the document as context
messages = [
    {
        "role": "system",
        "content": "You are a document analyst. Answer questions based only on the provided document. If the answer is not in the document, say so."
    },
    {
        "role": "user",
        "content": f"Here is the document to analyze:\n\n{document_text}"
    }
]

# Get an initial summary so the user knows the model has read the document
print("Analyzing document...\n")
response = chat(model="gemma4", messages=messages)
print(f"Assistant: {response.message.content}\n")

# Add the model's response to the conversation history
messages.append({"role": "assistant", "content": response.message.content})

What does this code do?

  • The messages list starts with two entries: a system prompt that tells Gemma 4 to act as a document analyst, and a user message containing the full document text.
  • The chat() function sends the messages to Gemma 4 and returns the model's response.
  • After printing the response, the script appends it to the messages list. This is how conversation memory works. The model sees everything that came before when answering the next question.
  • Save doc-qa.py.

Now add the interactive question loop that lets you keep asking follow-up questions.

  • Add this final section to the bottom of doc-qa.py:
# Start the interactive question loop
while True:
    # Prompt the user for a question
    question = input("You: ").strip()

    # Exit if the user types 'quit'
    if question.lower() == "quit":
        print("Goodbye!")
        break

    # Skip empty questions
    if not question:
        continue

    # Add the new question to the conversation history
    messages.append({"role": "user", "content": question})

    # Send the full conversation history to Gemma 4
    response = chat(model="gemma4", messages=messages)
    print(f"\nAssistant: {response.message.content}\n")

    # Add the model's answer to the conversation history for next time
    messages.append({"role": "assistant", "content": response.message.content})

How does the conversation loop work?

  • input("You: ") pauses the script and waits for you to type a question.
  • Each question gets appended to the messages list before calling chat(). Gemma 4 sees the system prompt, the document, and every previous question-answer pair.
  • The model's answer is also appended, so the conversation keeps growing. This is what makes follow-up questions like "tell me more about that" or "can you rephrase that?" work.
  • Save doc-qa.py.

Awesome, I've got everything!

Double check you have saved doc-qa.py before moving on.

I'd like to double check the full code

import sys
from ollama import chat

# Check that the user provided a file path when running the script
if len(sys.argv) < 2:
    print("Usage: python doc-qa.py <path-to-document>")
    sys.exit(1)

# Read the document from the file path provided
file_path = sys.argv[1]
with open(file_path, "r", encoding="utf-8") as f:
    document_text = f.read()

print(f"Loaded document: {file_path}")
print(f"Document length: {len(document_text)} characters")
print("Type your questions below. Type 'quit' to exit.\n")

# Set up the conversation with a system prompt and the document as context
messages = [
    {
        "role": "system",
        "content": "You are a document analyst. Answer questions based only on the provided document. If the answer is not in the document, say so."
    },
    {
        "role": "user",
        "content": f"Here is the document to analyze:\n\n{document_text}"
    }
]

# Get an initial summary so the user knows the model has read the document
print("Analyzing document...\n")
response = chat(model="gemma4", messages=messages)
print(f"Assistant: {response.message.content}\n")

# Add the model's response to the conversation history
messages.append({"role": "assistant", "content": response.message.content})

# Start the interactive question loop
while True:
    # Prompt the user for a question
    question = input("You: ").strip()

    # Exit if the user types 'quit'
    if question.lower() == "quit":
        print("Goodbye!")
        break

    # Skip empty questions
    if not question:
        continue

    # Add the new question to the conversation history
    messages.append({"role": "user", "content": question})

    # Send the full conversation history to Gemma 4
    response = chat(model="gemma4", messages=messages)
    print(f"\nAssistant: {response.message.content}\n")

    # Add the model's answer to the conversation history for next time
    messages.append({"role": "assistant", "content": response.message.content})

Seeing a syntax error when you save?

  • Make sure you pasted all three code chunks in order, one after the other. The conversation loop should come after the initial analysis block.
  • Check that the f-strings use matching quotes. Each f"..." must have a closing double quote.
  • Verify that the while True: block is not indented inside another block. It should be at the top level of the file.
  • Help me fix a syntax error in my doc-qa.py script
Run the Q&A tool

Time to take your interactive tool for a spin. You will run it against the sample-doc.txt file you created in the previous step.

  • Open PowerShell and navigate to your project folder by running:
cd C:\gemma4-project
  • Run the Q&A tool with your sample document by running this command:
python doc-qa.py sample-doc.txt

The script loads your document and sends it to Gemma 4 for an initial analysis. This first response takes several seconds on CPU.

Once the initial analysis prints, you will see a You: prompt waiting for your input.

Seeing 'FileNotFoundError'?

  • Make sure sample-doc.txt exists in your gemma4-project folder. You created this file in Step 4.
  • Check that your PowerShell is in the right folder. Run ls to see the files in your current directory. You should see both doc-qa.py and sample-doc.txt listed.
  • Help me fix a FileNotFoundError in my doc-qa.py script
Have a conversation about your document

With the tool running, you can ask follow-up questions. The model remembers everything you have discussed so far.

  • At the You: prompt, type a question about your document. For example:
What are the three most important points in this document?
  • Wait for the model to respond. On CPU, each answer takes several seconds to generate.
  • Ask a follow-up question that builds on the previous answer. For example:
Can you explain the second point in more detail?

The model knows which "second point" you mean because it remembers its own previous answer. This is conversation memory in action.

  • Try a few more questions. Here are some ideas:
Does the document mention any risks or downsides?
Summarize everything we've discussed so far in three bullet points.
  • When you are done, type quit and press Enter to exit.

Why does conversation memory matter?

Without conversation memory, each question would be treated as a completely new request. The model would not understand references like "tell me more about that" or "the second point."

By appending every question and answer to the messages list, you give Gemma 4 the full context of your conversation. The E4B model has a 128K token context window, so it can hold a very long conversation before running out of space.

  • Try running the tool again with a different document. Copy-paste a technical article, a meeting summary, a recipe, or even code documentation into a new .txt file and run the script against it.

Pro tip: notice the limits

Pay attention to where the model does well and where it struggles. It tends to be strong at summarization and extracting key points.

It can sometimes hallucinate details that are not in the document, especially for very specific factual questions. This is normal behavior for LLMs and a good thing to be aware of when using AI for document analysis.

You now have a fully interactive, local AI document analyst. Everything runs on your machine with no data leaving your computer. Next up, you will take on a challenge that pushes this tool even further.

Secret mission

Build a Multi-Document Comparison Tool

Your document analysis script works great for a single file. But what if you need to compare two documents side by side? In the real world, professionals constantly compare proposals, contracts, reports, and competing articles. Build a tool that reads two documents and asks Gemma 4 to analyze them together.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • Ollama (installed application serving the AI model).
  • Gemma 4 E4B model (approximately 9.6 GB stored locally by Ollama).
  • Python 3.13 (installed application).
  • The ollama Python package (installed via pip).
  • Project folder C:\gemma4-project\ with your Python scripts and sample documents.

Keep everything running

No action needed. Choose this if you are still actively building or want to keep experimenting with Gemma 4.

  • Ollama runs as a background service on Windows and uses minimal resources when no model is loaded.
  • The Gemma 4 model stays downloaded on disk (about 9.6 GB) and only loads into RAM when you run it.
  • Your Python scripts and project folder remain in place for future use.

Pause - I'll come back to this later

Free up disk space by removing the model while keeping your tools and scripts intact.

  • Stop any running Ollama chat sessions by typing /bye.
  • Remove the Gemma 4 model to reclaim about 9.6 GB of disk space. Run this in PowerShell:
ollama rm gemma4
  • Keep Ollama, Python, and your project folder installed. When you are ready to come back, just run ollama pull gemma4 to re-download the model.

Delete - I don't want to use this again

Remove all project resources and start fresh.

  • Remove the Gemma 4 model by running this in PowerShell:
ollama rm gemma4
  • Uninstall Ollama. Open PowerShell as an administrator and run:
winget uninstall ollama.ollama
  • Delete the Ollama data folder to remove any remaining model files. Open File Explorer and delete the .ollama folder inside your user home directory (e.g. C:\Users\YourName\.ollama).
  • Uninstall the ollama Python package by running:
pip uninstall ollama
  • Delete your project folder. Open File Explorer and delete C:\gemma4-project\.
  • Optionally, uninstall Python 3.13 from Settings > Apps > Installed apps. Only do this if you do not need Python for other projects.

Mission accomplished

That's a Wrap!

Nice work! You've set up a fully local AI assistant powered by Google's Gemma 4 E4B model, running entirely on your own Windows machine.

You've learned how to:

  • Install and configure Ollama to run a local LLM on a CPU-only machine with no cloud services or API keys.
  • Chat with Gemma 4 interactively, including thinking mode for step-by-step reasoning and multimodal image analysis for extracting information from images.
  • Build Python scripts that communicate with Gemma 4 through Ollama's API to automate document analysis and create an interactive document Q&A tool with multi-turn conversation memory.
  • Build a multi-document comparison tool that reads two documents and asks Gemma 4 to analyze their similarities and differences.

Ready to quiz yourself?