Build a Multi-Modal Form Processor

Extract printed text, handwriting, and bubble-sheet answers using OCR, ICR, and OMR.

Introduction

30 Second Summary

Every day, millions of forms get processed by hand. School exams, medical intake sheets, government applications. Someone reads the printed header, squints at the handwritten fields, and tallies up the bubbled-in answers. Each of those tasks needs a completely different recognition approach.

In this project, you will build a Python pipeline that extracts data from three types of document images using the right tool for each job: OCR for printed text, ICR for handwriting, and OMR for filled bubble sheets.

What You'll Build

You'll run your finished pipeline on three different document images and watch it return a single JSON object containing extracted printed text, decoded handwriting, and bubble-sheet answers.

By the end of this project, you'll have:

  • A printed-text extractor powered by Tesseract that returns every word alongside its confidence score and position on the page.
  • A handwriting recognizer using Microsoft's TrOCR transformer model that decodes cursive into clean digital text.
  • A unified pipeline that processes printed text, handwriting, and filled bubble sheets into one structured JSON output you can save and inspect.
  • Secret Mission: Add an auto-grading system that scores bubble-sheet answers against an answer key and flags low-confidence responses for human review.

Are there any prerequisites?

You need Python 3.10 or higher installed, a code editor (like VS Code or Cursor), and enough comfort with Python to write and run scripts.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up Your Environment

Your document recognition pipeline relies on three separate tools that each require their own installation. Tesseract is a system binary for printed text. PyTorch and the transformers library power the handwriting model. And marksense handles bubble-sheet reading.

Getting all of these wired up now means every later step just works. In this step you will install the Tesseract binary, create a Python project with all required packages, and generate sample images for testing.

In this step, get ready to:
  • Install the Tesseract OCR binary on your system.
  • Set up a Python project with all required dependencies.
  • Create sample images for testing each recognition method.
Install Tesseract OCR binary

Tesseract is the engine that reads printed text from images. Unlike Python packages, it is a standalone program that must be installed at the operating system level.

macOS

  • Install Tesseract using Homebrew by running:
brew install tesseract

What if I don't have Homebrew?

If your terminal says brew is not recognized, visit brew.sh and follow the one-line install command first. Then run the command above.

Windows

  • Open your browser and navigate to the UB Mannheim Tesseract wiki.
  • Download the 64-bit Windows installer (the file is named tesseract-ocr-w64-setup-*.exe).
  • Run the installer.
  • When prompted for components, make sure English language data is selected.
  • Leave the install path at the default: C:\Program Files\Tesseract-OCR.
  • Check Add to PATH when the installer offers it.

Missed the PATH checkbox?

You can manually add C:\Program Files\Tesseract-OCR to your system PATH. Search for Environment Variables in the Start menu, open the Path entry under System variables, and add the folder.

Linux

  • Install Tesseract using apt by running:
sudo apt install tesseract-ocr

After the install finishes, open a brand new terminal window and verify Tesseract is accessible.

  • Confirm the install by running:
tesseract --version

✔️ I see a version number

Tesseract is installed and available on your PATH. Move on to the next substep.

ⓧ Command not found

  • Make sure you opened a new terminal window after installing. Existing terminals do not pick up PATH changes.
  • On Windows, confirm C:\Program Files\Tesseract-OCR is listed in your system PATH environment variable.
  • On macOS, run brew info tesseract to confirm it was installed.

After fixing your PATH, close and reopen your terminal, then try tesseract --version again.

Create the project and install Python packages

With the system binary in place, you need a Python project folder, a virtual environment, and several packages.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder, move into it, and set up a virtual environment by running these commands:

macOS

mkdir document-recognition
cd document-recognition
python3 -m venv venv
source venv/bin/activate

Windows

mkdir document-recognition
cd document-recognition
python -m venv venv
venv\Scripts\activate

What is a virtual environment?

A virtual environment is an isolated Python installation inside your project folder. Packages you install here stay separate from your system Python, preventing version conflicts across projects.

Your terminal prompt should now show (venv) at the beginning, confirming the virtual environment is active.

  • Install the core packages by running:
pip install pytesseract Pillow requests
  • Install PyTorch (CPU-only) by running:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

Why CPU-only PyTorch?

The full PyTorch package with CUDA support is over 2 GB. The CPU-only build is much smaller and runs the TrOCR model fine for single-image inference.

  • Install the transformer library and the OMR package by running:
pip install transformers marksense

Before you run the verification command, do you think all three packages will import successfully?

  • Verify all imports work by running:
python -c "import pytesseract, transformers, marksense; print('All imports OK')"

You should see All imports OK printed to the terminal.

Seeing an ImportError?

  • Make sure your virtual environment is active. You should see (venv) in your prompt.
  • If pytesseract fails to import, confirm you ran pip install pytesseract inside the activated venv.
  • If marksense fails, check you are using Python 3.10+ (required by marksense).

Help me debug this import error.

Clone marksense examples and create sample images

The marksense library ships example bubble-sheet images and template JSON files in its repository. You also need a printed text image and a handwriting sample for the OCR and ICR demos.

  • Clone the marksense repository to get its example files by running:
git clone https://github.com/RoyAbra27/marksense.git marksense-repo
  • Confirm the clone succeeded by running:
ls marksense-repo/examples/

You should see folders including samples and templates.

Git not found?

If your terminal says git is not recognized, install Git from git-scm.com (Windows) or run xcode-select --install (macOS).

Now you will create a small Python script that generates a printed text sample and downloads a handwriting sample.

  • Create a new file called create_samples.py in your document-recognition folder.
  • Add the imports at the top of the file:
"""Generate sample images for testing the form processor."""
from PIL import Image, ImageDraw, ImageFont
from pathlib import Path
import requests
  • Below the imports, add the function that generates a printed text image:
def create_printed_text_sample():
    """Create a sample image with printed text."""
    Path("samples").mkdir(exist_ok=True)

    img = Image.new("RGB", (800, 400), "white")
    draw = ImageDraw.Draw(img)

    try:
        font = ImageFont.truetype("arial.ttf", 28)
    except OSError:
        font = ImageFont.load_default()

    text_lines = [
        "EXAM FORM - CS101 Introduction to Programming",
        "Student ID: 2026-0042",
        "Date: July 15, 2026",
        "Instructor: Dr. Sarah Chen",
        "Section: A (Morning)",
        "Time Allowed: 60 minutes",
    ]

    y_position = 30
    for line in text_lines:
        draw.text((30, y_position), line, fill="black", font=font)
        y_position += 50

    img.save("samples/printed_text.png")
    print("Created: samples/printed_text.png")

What does this function do?

  • It uses Pillow to create a blank white image (800x400 pixels).
  • It draws six lines of exam form text onto the image using a standard font.
  • The result is a clean printed-text image that Tesseract will read in the next step.
  • Below that function, add the download function and the main block:
def download_handwriting_sample():
    """Download a handwriting sample from the IAM dataset."""
    Path("samples").mkdir(exist_ok=True)

    url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02-00.jpg"
    response = requests.get(url)

    with open("samples/handwriting_sample.jpg", "wb") as f:
        f.write(response.content)
    print("Downloaded: samples/handwriting_sample.jpg")


if __name__ == "__main__":
    print("Generating sample images...")
    create_printed_text_sample()
    download_handwriting_sample()
    print("\nDone! Sample images are in the 'samples/' folder.")
    print("For OMR samples, use the marksense-repo/examples/ folder.")

What does the download function do?

  • It fetches a real handwriting sample from the IAM Handwriting Database (the same dataset TrOCR was trained on).
  • The if __name__ block runs both functions when you execute the script directly.
  • Save create_samples.py.
  • Run the script to generate your sample images:
python create_samples.py

You should see output confirming both files were created:

Generating sample images...
Created: samples/printed_text.png
Downloaded: samples/handwriting_sample.jpg

Done! Sample images are in the 'samples/' folder.
For OMR samples, use the marksense-repo/examples/ folder.
  • Confirm all sample files exist by running:
ls samples/

You should see printed_text.png and handwriting_sample.jpg in the output.

Download failed for the handwriting sample?

  • The IAM dataset server may be temporarily unavailable. Wait a minute and run python create_samples.py again.
  • If you are behind a corporate firewall, try connecting to a different network.

Help me troubleshoot the sample download.

✔️ Awesome, I've got everything!

Great. Make sure you have saved create_samples.py and that both sample images are in the samples/ folder.

ⓧ I'd like to double check the full code

Scroll up and confirm your create_samples.py contains all three sections in this order:

  • The imports block (4 lines starting with the docstring).
  • The create_printed_text_sample() function.
  • The download_handwriting_sample() function and the if __name__ main block.

If any section is missing, scroll up to the corresponding code block and paste it in.

Your environment is ready. You have Tesseract for printed text, TrOCR (via transformers) for handwriting, and marksense for bubble sheets. Next up, you will feed a printed document to Tesseract and watch it extract every word with confidence scores.

Extract Printed Text with Tesseract OCR

In Step 1, you installed Tesseract, set up your virtual environment, and generated sample images. Your toolkit is ready.

Now it is time to put Tesseract to work. You will write a Python script that reads your printed text image and extracts every word from it, along with confidence scores that tell you how certain the engine is about each detection.

In this step, get ready to:
  • Create an OCR script that extracts plain text from a printed image.
  • Add detailed extraction that returns per-word confidence scores and positions.
  • Examine what makes OCR output useful in data pipelines.
Create the simple text extraction script

The pytesseract library wraps the Tesseract binary you installed. Its simplest function, image_to_string(), takes an image and returns all the text it finds as a plain string.

  • Create a new file called ocr_demo.py in your document-recognition folder and paste the following code:
"""OCR: Extract printed text from images using Tesseract."""
import pytesseract
from PIL import Image
import sys

# Windows: uncomment and adjust if tesseract is not in PATH
# pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe"


def extract_printed_text(image_path):
    """Extract printed text from an image using Tesseract OCR."""
    image = Image.open(image_path)
    text = pytesseract.image_to_string(image, lang="eng")
    return text


if __name__ == "__main__":
    image_path = sys.argv[1] if len(sys.argv) > 1 else "samples/printed_text.png"

    print("=== Simple Text Extraction ===")
    text = extract_printed_text(image_path)
    print(text)

What does this code do?

Image.open(image_path) loads the image file into memory using Pillow.

pytesseract.image_to_string(image, lang="eng") sends the image to the Tesseract binary, which scans for English text and returns everything it finds as one string.

The __main__ block defaults to your printed text sample if you do not pass a custom image path.

  • Save ocr_demo.py.
  • Run the script by entering this command in your terminal:
python ocr_demo.py

You should see the full text from your exam form image printed to the terminal, including lines like EXAM FORM - CS101 Introduction to Programming and Student ID: 2026-0042.

Seeing 'TesseractNotFoundError'?

This means Python cannot find the Tesseract binary. If you did not check "Add to PATH" during installation, uncomment the pytesseract.pytesseract.tesseract_cmd line in your script and set it to your Tesseract install path.

If you did add it to PATH, close and reopen your terminal so the new PATH takes effect.

Help me fix a TesseractNotFoundError in pytesseract on Windows.

Add detailed extraction with confidence scores

Plain text extraction is useful, but production pipelines need more. Tesseract can also return confidence scores and bounding-box coordinates for every word it detects.

  • In ocr_demo.py, add the following function above the if __name__ == "__main__": line:
def extract_with_details(image_path):
    """Extract text with bounding box and confidence data."""
    image = Image.open(image_path)
    data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT)

    results = []
    for i in range(len(data["text"])):
        if data["text"][i].strip():
            results.append({
                "text": data["text"][i],
                "confidence": data["conf"][i],
                "x": data["left"][i],
                "y": data["top"][i],
                "width": data["width"][i],
                "height": data["height"][i],
            })
    return results

What does this function do?

pytesseract.image_to_data() returns a dictionary with keys like text, conf, left, top, width, and height. Each key maps to a list where index i corresponds to one detected word.

The loop filters out empty entries and builds a list of dictionaries. Each dictionary gives you the word, how confident Tesseract is about it, and exactly where on the page it sits.

  • Now add these lines at the end of your if __name__ == "__main__": block, below print(text):
    print("\n=== Detailed Extraction (with confidence) ===")
    details = extract_with_details(image_path)
    for item in details:
        print(f"  '{item['text']}' (confidence: {item['confidence']}%)")
  • Save ocr_demo.py.

Before you run this, what do you expect the confidence scores to look like for clean, machine-printed text?

  • Run the updated script:
python ocr_demo.py

You should see two sections. First, the plain text extraction from before. Second, a word-by-word breakdown where most words show confidence values of 90% or higher.

Seeing low confidence on every word?

Check that samples/printed_text.png exists and was generated correctly. Re-run python create_samples.py if the file is missing or corrupt.

If your system does not have arial.ttf, the sample script falls back to a default font. The text will still be legible to Tesseract.

Help me debug low OCR confidence scores on a generated sample image.

Understand the output structure

Look at the confidence scores in your terminal. On clean printed text, most words score 90% or above. That high confidence is what makes Tesseract reliable for machine-printed documents.

Why does position data matter?

Tesseract returns x, y, width, and height for each detected word. These coordinates tell you exactly where each word sits on the page.

This is what makes OCR useful in pipelines. You do not just get text. You get structured data with location, so downstream code can extract values from specific regions of a form (the student ID field, the date field) without parsing free text.

✔️ Awesome, I've got everything!

Double check you have saved ocr_demo.py and that both extraction sections print when you run it.

ⓧ I'd like to double check the full code

Your ocr_demo.py should contain these sections in order. Scroll up and confirm each is present:

  • Imports (pytesseract, Image, sys) and the commented-out Windows path line.
  • The extract_printed_text() function using image_to_string().
  • The extract_with_details() function using image_to_data() and building a list of result dictionaries.
  • The if __name__ == "__main__": block calling both functions and printing results.

You have a working OCR extractor that handles clean printed text with high accuracy. Next, you will see what happens when you throw handwriting at Tesseract. Spoiler: it does not go well.

Recognize Handwriting with TrOCR

In the previous step, Tesseract extracted clean printed text with high confidence scores. But real-world forms contain more than machine-printed characters.

Handwriting is fundamentally different from printed text. Characters vary wildly between writers, letters connect in unpredictable ways, and spacing is inconsistent. Tesseract was built for printed text, so running it on handwriting produces garbled output. In this step, you'll expose that limitation and then solve it with TrOCR, a transformer model purpose-built for handwriting recognition.

In this step, get ready to:
  • Try running Tesseract on a handwriting sample and observe the failure.
  • Apply Microsoft's TrOCR model for accurate handwriting recognition.
  • Compare the two approaches to understand when each tool fits.
Try Tesseract on handwriting

Tesseract uses pattern matching to identify printed characters. Each character is compared against a known alphabet. This works great on clean typefaces, but handwriting introduces curves, overlaps, and spacing that break the pattern-matching approach.

Let's see exactly what happens when you point Tesseract at a handwriting sample.

  • Create a new file called icr_demo.py in your document-recognition folder.
  • Add the imports and the Tesseract handwriting function by pasting this code:
"""ICR: Recognize handwritten text using TrOCR (with Tesseract comparison)."""
import pytesseract
from PIL import Image
from transformers import TrOCRProcessor, VisionEncoderDecoderModel
import requests
from pathlib import Path

# Windows: uncomment and adjust if tesseract is not in PATH
# pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe"


def try_tesseract_on_handwriting(image_path):
    """Attempt to read handwriting with Tesseract (will produce poor results)."""
    image = Image.open(image_path).convert("RGB")
    text = pytesseract.image_to_string(image, lang="eng")
    return text

What does this code do?

  • The imports pull in everything you'll need for both Tesseract and TrOCR (you'll use TrOCR shortly).
  • try_tesseract_on_handwriting() opens the handwriting image, converts it to RGB, and runs pytesseract.image_to_string() on it. This is the exact same function that worked perfectly on printed text.
  • Now add the main block below the function. This handles downloading the sample if needed and runs Tesseract on it:
if __name__ == "__main__":
    image_path = "samples/handwriting_sample.jpg"

    # Download sample if not present
    if not Path(image_path).exists():
        print("Downloading handwriting sample...")
        Path("samples").mkdir(exist_ok=True)
        url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02-00.jpg"
        response = requests.get(url)
        with open(image_path, "wb") as f:
            f.write(response.content)
        print(f"Saved to {image_path}\n")

    print("=== Tesseract on Handwriting ===")
    tesseract_result = try_tesseract_on_handwriting(image_path)
    print(f"Result: '{tesseract_result.strip()}'")
    print("(Notice the errors and garbled text)\n")

What does this code do?

  • The download check ensures the handwriting sample from the IAM dataset exists locally. You already downloaded it in Step 1, so this will be skipped.
  • The script then runs Tesseract on samples/handwriting_sample.jpg and prints whatever text it manages to extract.

Before you run this, what do you think Tesseract will produce when it tries to read handwriting? Will it be as clean as the printed text extraction from the last step?

  • Save icr_demo.py and run it:
python icr_demo.py

You'll see garbled, mostly incorrect output. Tesseract produces random characters, missing words, and jumbled text instead of anything readable.

This is the designed limitation. Tesseract's pattern-matching approach cannot handle the variability in handwritten characters. The same letter looks different every time someone writes it, and Tesseract has no way to account for that.

Seeing an error instead of garbled text?

  • If you see a TesseractNotFoundError, Tesseract is not in your PATH. Uncomment the pytesseract.pytesseract.tesseract_cmd line at the top of the file and adjust the path to match your Tesseract installation.
  • If the download fails, check your internet connection. The sample image is hosted by the IAM database.

Why isn't Tesseract recognizing my handwriting sample?

Apply TrOCR for accurate handwriting recognition

TrOCR is a transformer model from Microsoft that was trained on thousands of handwriting samples from the IAM dataset. Instead of matching against fixed patterns, it learned the underlying structure of handwritten characters.

  • In icr_demo.py, add the recognize_handwriting() function between try_tesseract_on_handwriting() and the if __name__ == "__main__": block:
def recognize_handwriting(image_path):
    """Recognize handwritten text using Microsoft's TrOCR model."""
    processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
    model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")

    image = Image.open(image_path).convert("RGB")
    pixel_values = processor(images=image, return_tensors="pt").pixel_values
    generated_ids = model.generate(pixel_values)
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

    return generated_text

What does this code do?

  • TrOCRProcessor handles image preprocessing. It resizes and normalizes the image into the format the model expects.
  • VisionEncoderDecoderModel is the actual transformer. It uses a vision encoder (BEiT) to read the image and a text decoder (RoBERTa) to generate the recognized text.
  • model.generate() runs autoregressive decoding, predicting one character at a time until the full text is produced.
  • batch_decode() converts the generated token IDs back into a human-readable string.
  • Now add the TrOCR call at the very end of your if __name__ == "__main__": block, after the Tesseract output lines:
    print("=== TrOCR on Handwriting ===")
    print("Loading TrOCR model (first run downloads ~350MB)...")
    trocr_result = recognize_handwriting(image_path)
    print(f"Result: '{trocr_result}'")
    print("(Much more accurate!)")

What happens on first run?

The first time you run this, from_pretrained() downloads the TrOCR model weights from Hugging Face (about 1.33 GB for the safetensors file). This is cached locally, so subsequent runs load instantly.

Processing takes a few seconds on CPU. The model reads the handwriting image and decodes it character by character.

  • Save icr_demo.py and run the complete script:
python icr_demo.py

You'll see both results printed side-by-side. Tesseract's garbled output appears first, followed by TrOCR's accurate transcription of the same handwriting image.

Model download taking too long?

  • The first download is about 1.33 GB. On a slow connection this can take several minutes. The model is cached locally after the first download.
  • If you see a connection timeout, check your internet and try again. Hugging Face Hub occasionally has high traffic.

Why is TrOCR so much better at reading handwriting than Tesseract?

Compare the two approaches

The terminal output makes the difference stark. Tesseract produces near-random characters on handwriting, while TrOCR reads it accurately.

When to use each tool

  • Use Tesseract for clean printed text. It is fast, lightweight, and returns confidence scores and bounding boxes for every word.
  • Use TrOCR for handwritten text. It is slower and heavier (requires a 1.33 GB model), but handles the variability that breaks Tesseract.

TrOCR works on single text-line images. This is a known limitation of the model. For multi-line handwritten documents, you would need to segment individual lines first before feeding each line to TrOCR separately.

Why TrOCR instead of other handwriting tools?

Alternatives like EasyOCR and PaddleOCR exist, but TrOCR consistently produces the most accurate results on English handwriting benchmarks. Its transformer architecture (vision encoder plus text decoder) mirrors how modern language models work, making it a natural fit for this pipeline.

✔️ Awesome, I've got everything!

Great. Double check you've saved icr_demo.py and that running it prints both the garbled Tesseract output and the accurate TrOCR transcription.

ⓧ I'd like to double check the full code

Your icr_demo.py should contain these sections, in order:

  • Module docstring and imports (pytesseract, PIL, transformers, requests, pathlib).
  • The commented-out tesseract_cmd config line.
  • try_tesseract_on_handwriting() function that opens the image and runs image_to_string().
  • recognize_handwriting() function that loads TrOCR and generates text.
  • if __name__ == "__main__": block that downloads the sample if missing, runs Tesseract, prints its garbled result, then runs TrOCR and prints the accurate result.

Scroll up and confirm each chunk is present in your file, in this order.

You now have proof that printed-text OCR and handwriting recognition are fundamentally different problems requiring different tools. Next up, you'll tackle a third recognition challenge: reading filled bubbles on an answer sheet.

Read Bubble-Sheet Answers with marksense

You now have two recognition methods working: Tesseract for printed text and TrOCR for handwriting. But neither can handle a filled bubble sheet.

Bubble sheets need a fundamentally different approach. Instead of recognizing character shapes, the system needs to detect which circles are filled based on pixel density. In this step, you will use marksense, a template-driven Optical Mark Recognition (OMR) library purpose-built for this job.

In this step, get ready to:
  • Write an OMR extraction script that reads bubble-sheet answers from an image.
  • Examine how the template-driven approach lets marksense know where bubbles are on a form.
Create the OMR extraction script

Unlike OCR and ICR which look at character shapes, OMR works by measuring how dark each bubble region is relative to a template that maps out the form layout. You give it a scanned image and a template JSON, and it returns structured answers.

  • Create a new file called omr_demo.py in your document-recognition folder.
  • Add the imports and core extraction function by pasting this code:
"""OMR: Read bubble-sheet answers using marksense."""
from marksense import read_form
import json
import sys


def read_bubble_sheet(image_path, template_path):
    """Read bubble sheet answers using marksense OMR."""
    result = read_form(image_path, template=template_path)
    return {
        "answers": result.answers,
        "confidence": result.confidence,
        "multi_marked": result.multi_marked,
    }

What does this code do?

The read_form() function from marksense takes two inputs: a scanned bubble-sheet image and a template JSON file that describes where each question's answer options are located on the form.

It returns a result object with three key attributes: result.answers (a dictionary mapping question IDs to detected answers), result.confidence (per-question confidence scores), and result.multi_marked (questions where multiple bubbles appear filled).

  • Now add the main block below the function to run the extraction and display results:
if __name__ == "__main__":
    image_path = (
        sys.argv[1]
        if len(sys.argv) > 1
        else "marksense-repo/examples/samples/quiz_filled_01.png"
    )
    template_path = (
        sys.argv[2]
        if len(sys.argv) > 2
        else "marksense-repo/examples/templates/quiz.json"
    )

    print("=== OMR Bubble Sheet Reading ===")
    print(f"Image: {image_path}")
    print(f"Template: {template_path}\n")

    results = read_bubble_sheet(image_path, template_path)

    print("Answers detected:")
    for question, answer in results["answers"].items():
        print(f"  {question}: {answer}")

    if results["multi_marked"]:
        print(f"\nMulti-marked questions (need review): {results['multi_marked']}")

    print(f"\nFull JSON output:")
    print(json.dumps(results, indent=2, default=str))

What's happening here?

The main block defaults to the example bubble-sheet image and quiz template that came with the marksense repository you cloned earlier. It calls read_bubble_sheet(), then prints each detected answer and finally outputs the full result as formatted JSON.

The multi_marked check identifies questions where the student may have filled more than one bubble. These are flagged for human review because the intended answer is ambiguous.

✔️ Awesome, I've got everything!

Great. Make sure you have saved omr_demo.py.

ⓧ I'd like to double check the full code

Your omr_demo.py should have two sections in order:

  • The imports and read_bubble_sheet() function (the first chunk you pasted).
  • The if __name__ == "__main__": block (the second chunk you pasted).

Scroll up and confirm both are present in your file with no extra code between them.

Before you run this, what do you think the output will look like? Do you expect letter answers like A, B, C, D, or something else?

  • Run the script by entering this command in your terminal:
python omr_demo.py

You should see structured output listing each question and its detected answer (Q1: A, Q2: D, etc.) followed by the complete JSON.

Seeing an import error for marksense?

  • Make sure your virtual environment is still activated. You should see (venv) at the start of your terminal prompt.
  • Confirm marksense is installed by running pip show marksense. If it is not found, run pip install marksense again.
  • Check that the file paths are correct. The marksense-repo/examples/ folder must exist with the sample and template files inside it.

Help me debug a marksense import error in my OMR script.

Examine the template-driven approach

The key insight of OMR is that all form-specific knowledge lives in the template JSON, not in the code. Your read_bubble_sheet() function is completely generic.

  • Open marksense-repo/examples/templates/quiz.json in your code editor to see what a template looks like.

What is an OMR template?

The template JSON describes the physical layout of the form: where each question's bubble options are located on the page, what values they map to (A, B, C, D), and how to align the scanned image to the expected layout.

This is what makes the approach powerful. To support a new form design, you write a new template file. The detection engine stays the same.

Look back at your terminal output from the script. Notice two features that make OMR production-ready:

  • Confidence scores tell you how certain the system is about each detected answer. High confidence means the filled bubble was clearly darker than the others.
  • Multi-mark flags identify questions where more than one bubble appears filled. These need human review because the student's intent is unclear.

Why template-driven instead of custom OpenCV code?

You could write 100+ lines of OpenCV threshold and contour detection to find filled circles. But that approach hard-codes the form layout in your code.

A template-driven tool separates the "how to detect marks" logic from the "where are the marks" layout. When your form design changes, you update a JSON file instead of rewriting detection code.

You now have all three recognition methods working independently. Next up, you will combine OCR, ICR, and OMR into a single unified pipeline that outputs one structured JSON result.

Build the Unified Pipeline

You now have three separate scripts, each tackling one type of document recognition. They work, but a real form processor needs to handle any combination of inputs in one pass and return a single structured result.

In this step, you'll bring OCR, ICR, and OMR together into a single FormProcessor class that loads the TrOCR model once and outputs unified JSON.

In this step, get ready to:
  • Create a FormProcessor class with methods for each recognition type.
  • Add a process_form() method that orchestrates all three and returns a unified dictionary.
  • Run the complete pipeline and save the results to form_results.json.
Create the FormProcessor class

The class needs to load the TrOCR model just once during initialization. Each individual demo script loaded the model from scratch every time, which wastes time when you process multiple documents.

  • Create a new file called pipeline.py in your document-recognition folder and add the imports plus the class initialization:
"""
Unified Document Recognition Pipeline
Combines OCR (printed text), ICR (handwriting), and OMR (bubble marks)
into a single structured output.
"""
import json
from pathlib import Path

import pytesseract
from PIL import Image
from transformers import TrOCRProcessor, VisionEncoderDecoderModel
from marksense import read_form

# Windows: uncomment and adjust if tesseract is not in PATH
# pytesseract.pytesseract.tesseract_cmd = r"C:\Program Files\Tesseract-OCR\tesseract.exe"


class FormProcessor:
    """Unified form processor combining OCR, ICR, and OMR."""

    def __init__(self):
        """Initialize the form processor with all recognition models."""
        print("Loading TrOCR model...")
        self.trocr_processor = TrOCRProcessor.from_pretrained(
            "microsoft/trocr-base-handwritten"
        )
        self.trocr_model = VisionEncoderDecoderModel.from_pretrained(
            "microsoft/trocr-base-handwritten"
        )
        print("Models loaded.")

What does this code do?

The imports pull in all four libraries you used in earlier steps: pytesseract for printed text, transformers for handwriting, marksense for bubbles, and Pillow for image handling.

The __init__ method loads TrOCR into memory once. Every subsequent call to recognize handwriting reuses the cached model instead of re-downloading it.

  • Save pipeline.py.
  • Verify the class initializes correctly by running this command:
python -c "from pipeline import FormProcessor; p = FormProcessor()"

You should see "Loading TrOCR model..." followed by "Models loaded." printed to the terminal. Since the model is already cached locally from earlier steps, this loads in a few seconds.

Seeing an ImportError?

  • Make sure you are in the document-recognition folder and your virtual environment is active.
  • Check that your imports match exactly. TrOCRProcessor and VisionEncoderDecoderModel both come from transformers.

Why can't Python find my pipeline module?

Now add the three recognition methods to the class. Each one wraps the logic you already wrote in the individual demo scripts.

  • In pipeline.py, add the following methods inside the FormProcessor class, directly below the __init__ method:
    def extract_printed_text(self, image_path):
        """Extract printed text using Tesseract OCR."""
        image = Image.open(image_path).convert("RGB")
        text = pytesseract.image_to_string(image, lang="eng")
        return text.strip()

    def recognize_handwriting(self, image_path):
        """Recognize handwritten text using TrOCR."""
        image = Image.open(image_path).convert("RGB")
        pixel_values = self.trocr_processor(
            images=image, return_tensors="pt"
        ).pixel_values
        generated_ids = self.trocr_model.generate(pixel_values)
        text = self.trocr_processor.batch_decode(
            generated_ids, skip_special_tokens=True
        )[0]
        return text

    def read_bubble_sheet(self, image_path, template_path):
        """Read bubble sheet answers using marksense OMR."""
        result = read_form(image_path, template=template_path)
        return {
            "answers": result.answers,
            "confidence": result.confidence,
            "multi_marked": result.multi_marked,
        }

What do these methods do?

  • extract_printed_text() opens an image and runs Tesseract on it. The .strip() removes trailing whitespace from the result.
  • recognize_handwriting() uses the pre-loaded TrOCR model (self.trocr_processor and self.trocr_model) instead of loading a fresh copy each time.
  • read_bubble_sheet() calls marksense's read_form() and returns the answers, confidence scores, and any multi-marked questions.
  • Save pipeline.py.
  • Test the printed text method by running:
python -c "from pipeline import FormProcessor; p = FormProcessor(); print(p.extract_printed_text('samples/printed_text.png'))"

You should see the extracted exam form text (EXAM FORM - CS101, Student ID, Date, etc.) printed to the terminal.

Not seeing any text output?

  • Make sure samples/printed_text.png exists. Run ls samples/ to check.
  • Check that the methods are indented inside the FormProcessor class. Python is whitespace-sensitive.

Help me debug why my FormProcessor methods aren't working.

The final piece of the class is the orchestration method. It calls all three recognition methods in sequence and assembles one unified output dictionary.

  • In pipeline.py, add the process_form() method below read_bubble_sheet(), still inside the FormProcessor class:
    def process_form(self, printed_image, handwriting_image, bubble_image, template_path):
        """Process a complete form with all three recognition methods."""
        output = {
            "ocr_printed_text": {},
            "icr_handwritten_text": {},
            "omr_bubble_answers": {},
        }

        print("  [1/3] Extracting printed text (OCR)...")
        output["ocr_printed_text"] = {
            "method": "Tesseract OCR",
            "text": self.extract_printed_text(printed_image),
        }

        print("  [2/3] Recognizing handwriting (ICR)...")
        output["icr_handwritten_text"] = {
            "method": "TrOCR (microsoft/trocr-base-handwritten)",
            "text": self.recognize_handwriting(handwriting_image),
        }

        print("  [3/3] Reading bubble sheet (OMR)...")
        omr_results = self.read_bubble_sheet(bubble_image, template_path)
        output["omr_bubble_answers"] = {
            "method": "marksense",
            "answers": omr_results["answers"],
            "flagged": omr_results["multi_marked"],
        }

        return output

What does process_form do?

This method is the orchestrator. It calls each recognition method in sequence and tags each result with the method that produced it.

The output dictionary has three keys: ocr_printed_text, icr_handwritten_text, and omr_bubble_answers. Each entry records which method was used, making the output self-documenting.

  • Save pipeline.py.
Run the complete pipeline and save results

The class is complete. Now add a main block that creates a FormProcessor, feeds it all three sample images, and saves the unified result to a JSON file.

  • At the bottom of pipeline.py (outside the class, at the top indentation level), add the main block:
if __name__ == "__main__":
    processor = FormProcessor()

    print("\nProcessing form...")
    result = processor.process_form(
        printed_image="samples/printed_text.png",
        handwriting_image="samples/handwriting_sample.jpg",
        bubble_image="marksense-repo/examples/samples/quiz_filled_01.png",
        template_path="marksense-repo/examples/templates/quiz.json",
    )

    print("\n" + "=" * 50)
    print("FORM PROCESSING RESULTS")
    print("=" * 50)
    print(json.dumps(result, indent=2, default=str))

    output_path = "form_results.json"
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2, default=str)
    print(f"\nResults saved to {output_path}")

What does the main block do?

  • It creates a FormProcessor instance, which loads TrOCR into memory.
  • It calls process_form() with all three sample images you created in earlier steps.
  • It prints the full JSON output to the terminal and saves a copy to form_results.json.
  • Save pipeline.py.

Before you run this, what do you expect the JSON output to contain? Think about what each method returns.

  • Run the unified pipeline:
python pipeline.py

You should see progress messages as each recognition method runs, followed by a JSON object containing all three sections of extracted data. The printed text section shows the exam form text. The handwriting section shows TrOCR's transcription. The bubble answers section shows detected answers for each question.

The terminal also confirms that results were saved to form_results.json.

Why one class instead of three scripts?

Loading TrOCR takes several seconds. By initializing the model once in __init__ and reusing it across calls, the pipeline processes multiple documents without repeated startup cost.

The unified output also makes downstream integration simpler. Any system consuming this data gets one JSON structure with clear method labels instead of three separate outputs to reconcile.

Pipeline crashing partway through?

  • If the crash happens at step [3/3] (OMR), check that marksense-repo/examples/samples/quiz_filled_01.png exists. Run ls marksense-repo/examples/samples/ to confirm.
  • If you see a file path error, check that you are running python pipeline.py from the document-recognition folder (not from inside a subfolder).
  • If Tesseract returns no text, make sure tesseract is still in your PATH. Open a new terminal and run tesseract --version to check.

Help me debug my pipeline.py script.

✔️ Awesome, I've got everything!

Make sure you have saved pipeline.py and confirmed that form_results.json was created.

ⓧ I'd like to double check the full code

Your pipeline.py should contain these sections in order. Scroll up and confirm each one is present:

  • Module docstring and imports (json, pathlib, pytesseract, PIL, transformers, marksense).
  • The FormProcessor class with __init__ loading TrOCR.
  • Three methods: extract_printed_text(), recognize_handwriting(), read_bubble_sheet().
  • The process_form() orchestration method.
  • The if __name__ == "__main__": block calling process_form() with all three sample images and saving to form_results.json.

Your pipeline now processes three different document types and returns one clean JSON structure. Next up, you'll extend it with auto-grading in the Secret Mission.

Secret mission

Add Auto-Grading with Confidence Flagging

Your pipeline extracts answers, but a teacher still has to check them manually. What if the pipeline could score the answers automatically and flag the ones it isn't sure about for human review?

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • The document-recognition project folder on your Desktop (all Python scripts, virtual environment, sample images, cloned marksense repository, and output JSON).
  • Tesseract OCR system binary installed via the Windows installer.
  • Hugging Face model cache (~350MB for the TrOCR model, stored outside your project folder).

Keep everything running

No action needed. Choose this if you plan to keep experimenting with your form processor or want to try new sample images.

  • Everything runs locally with no ongoing costs. Your project folder, Tesseract binary, and cached TrOCR model will remain on your machine.
  • To use the pipeline again, open a terminal, navigate to document-recognition, activate your virtual environment, and run python pipeline.py.

Pause - I'll come back to this later

Shut down any active terminal sessions to free up memory. Your files and installed tools stay in place for next time.

  • If your virtual environment is active (you see (venv) in your terminal prompt), deactivate it by running:
deactivate
  • Close any terminal windows you used for this project.
  • When you return, reactivate the virtual environment with venv\Scripts\activate (Windows) or source venv/bin/activate (macOS/Linux) from inside the document-recognition folder.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes your scripts, sample images, virtual environment, the cloned repository, Tesseract binary, and the cached model.

  • Delete the project folder. On macOS/Linux, run:
cd ~/Desktop
rm -rf document-recognition
  • On Windows, run:
cd ~\Desktop
Remove-Item -Recurse -Force document-recognition
  • Uninstall Tesseract OCR from your system. Open Settings, select Apps, then Installed apps. Search for Tesseract and click Uninstall.
  • Delete the Hugging Face model cache to reclaim ~350MB. On macOS/Linux, run:
rm -rf ~/.cache/huggingface/hub/models--microsoft--trocr-base-handwritten
  • On Windows, run:
Remove-Item -Recurse -Force "$env:USERPROFILE\.cache\huggingface\hub\models--microsoft--trocr-base-handwritten"
  • Confirm the project folder is gone by running ls ~/Desktop (macOS/Linux) or ls ~\Desktop (Windows). You should no longer see document-recognition listed.

Nice Work!

Nice Work!

Nice work. You just built a complete document recognition pipeline that extracts printed text, handwritten text, and bubble-sheet answers from form images using three distinct recognition technologies.

You've learned how to:

  • Extract printed text with Tesseract OCR and retrieve per-word confidence scores and position data for each detected word.
  • Discover Tesseract's limitation on handwriting and solve it with Microsoft's TrOCR transformer model for accurate Intelligent Character Recognition.
  • Combine OCR, ICR, and OMR into a unified FormProcessor pipeline that outputs structured JSON from multiple input types.
  • Secret Mission: Added auto-grading with answer-key comparison and confidence-based flagging for human review of uncertain responses.

Ready to quiz yourself?