AI Receipt Parsing with Gemini Flash

Replace brittle regex parsing with Gemini structured output in a MERN app.

Introduction

30 Second Summary

Every receipt looks different. If you've tried to extract data from receipts using pattern matching, you know how quickly those rules break when a new store formats things differently.

In this project, you will replace the regex-based receipt parsing in your expense tracker with a Gemini 2.5 Flash API call that returns structured JSON. The model extracts amount, date, merchant, and description from any receipt format, returning null for anything it can't confidently determine.

What You'll Build

A Gemini-powered receipt parser utility that replaces brittle regex with structured output, integrated into your existing Multer + Tesseract.js upload flow.

By the end of this project, you'll have:

  • A parseReceiptWithGemini.js utility module that takes raw OCR text and returns structured { amount, date, merchant, description } JSON with null for uncertain fields.
  • An updated Express route that swaps regex parsing for a single Gemini API call within your existing Multer + Tesseract.js pipeline.
  • A working end-to-end flow from receipt photo upload to auto-filled transaction form, with the merchant value mapping to your category field.
  • Secret Mission: Add confidence scores for each extracted field so your UI can flag low-confidence values for manual review.

Are there any prerequisites?

This project builds on a working MERN expense tracker that already uses Tesseract.js for OCR and Multer for receipt image uploads. You should have that flow working before starting here.

Install and Configure the Gemini SDK

Your expense tracker already extracts raw text from receipt photos using Tesseract.js. The problem is the regex that parses that text. It breaks on every new receipt format because regex relies on exact patterns rather than understanding language.

In this step, you'll install the Google Gen AI SDK and configure your API key so your app can talk to Gemini 2.5 Flash.

In this step, get ready to:
  • Install the @google/genai SDK in your Express backend.
  • Add your Gemini API key to environment variables.
  • Verify the SDK initializes without errors.
Install the SDK in your backend directory

The @google/genai package is the official Node.js SDK for calling Gemini models. It requires Node.js version 20 or later, which your environment already has.

  • Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
  • Type Terminal and press Enter to open it.
  • Navigate to your Express server directory (the folder containing your package.json with express, multer, and tesseract.js dependencies).
  • Install the Google Gen AI SDK by running this command:
npm install @google/genai
  • Verify the installation by opening your package.json and confirming you see @google/genai listed under dependencies.

Seeing permission errors or network issues?

If you see EACCES errors, try running the command with sudo on macOS or as Administrator on Windows.

If you see ENOTFOUND or timeout errors, check your internet connection and try again.

Help me troubleshoot this npm install error

Add the Gemini API key to your environment

Gemini requires an API key for authentication. You can create one for free in Google AI Studio with no billing required.

  • Go to Google AI Studio's API key page.
  • Sign in with your Google account if prompted.
  • Click Create API key.
  • Select a project (or create a new one) and confirm.
  • Copy the generated API key. You will only see it once, so copy it now.

Why store the key in .env?

API keys are secrets. You should never commit them to version control or share them. Storing the key in your .env file (which should already be listed in your .gitignore) keeps it out of your codebase.

  • Open your .env file in your code editor (this is the same file that holds your MongoDB URI and other environment variables).
  • Add this line anywhere in the file, replacing your-api-key-here with the key you just copied:
GEMINI_API_KEY=your-api-key-here

What does this line do?

Your existing require('dotenv').config() call at the top of your server entry point automatically loads every variable from .env into process.env.

Once this line is saved, your app can access the key as process.env.GEMINI_API_KEY without any extra configuration.

  • Save .env.

Not sure where your .env file is?

It should be in the root of your Express backend folder, the same directory as your package.json. If you do not have one yet, create a new file called .env there.

Help me find where to put my .env file

Verify the SDK loads correctly

A quick test confirms the SDK can initialize and read your API key from the environment.

  • Confirm the SDK loads by running this command from your Express backend directory:
node -e "require('dotenv').config(); const { GoogleGenAI } = require('@google/genai'); const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }); console.log('SDK loaded successfully');"
  • You should see SDK loaded successfully printed in your terminal.

What does this command do?

  • require('dotenv').config() loads your .env variables into process.env.
  • const { GoogleGenAI } = require('@google/genai') imports the SDK's main class.
  • new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY }) creates an instance that authenticates with your key.
  • If any step fails, Node.js throws an error instead of printing the success message.

Seeing an error instead of the success message?

If you see a message about not finding the module, make sure you are running the command from the same directory where you ran npm install.

If you see undefined printed or no output at all, check that your .env file is in the current directory and contains GEMINI_API_KEY with no typos.

Help me debug this SDK verification error

Your environment is ready. Next up, you'll build the Gemini receipt parser utility that replaces your regex logic.

Build the Gemini Receipt Parser Utility

Your SDK is installed and your API key is ready to go. Now it's time to build the core piece that replaces your regex parsing logic.

Regex patterns break on different receipt formats because they rely on exact character positions. Gemini understands natural language context, so it can extract structured data from any receipt layout and honestly return null when it cannot confidently determine a value.

In this step, get ready to:
  • Create the receipt parser utility with a JSON schema that enforces structured output.
  • Write the async function that calls Gemini and handles errors gracefully.
  • Test the utility with both a clean receipt and a messy OCR output.
Create the utility file with the structured output schema

Structured output forces the model to return JSON that matches an exact schema you define. Instead of hoping the model returns valid JSON, the responseJsonSchema config guarantees every response conforms to your field definitions.

  • Create a utils folder in your Express backend directory, then create a file called parseReceiptWithGemini.js inside it by running these commands:
mkdir -p utils
touch utils/parseReceiptWithGemini.js
  • Confirm the file was created by running:
ls utils

You should see parseReceiptWithGemini.js in the output.

  • Open utils/parseReceiptWithGemini.js in your editor and add the imports, SDK initialization, and schema definition:
const { GoogleGenAI } = require('@google/genai');

// Initialize the Gemini client with your API key
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });

// Define the JSON schema that Gemini must conform to
const receiptSchema = {
  type: 'object',
  properties: {
    amount: {
      type: ['number', 'null'],
      description: 'The total amount charged, as a number without currency symbols. Null if not confidently determined.',
    },
    date: {
      type: ['string', 'null'],
      description: 'The transaction date in YYYY-MM-DD format. Null if not confidently determined.',
    },
    merchant: {
      type: ['string', 'null'],
      description: 'The store or business name on the receipt. This will be used as the transaction category. Null if not confidently determined.',
    },
    description: {
      type: ['string', 'null'],
      description: 'A brief summary of the purchased items or services (e.g., "Groceries - milk, bread, bananas"). Null if not confidently determined.',
    },
  },
  required: ['amount', 'date', 'merchant', 'description'],
};

What does this code do?

The schema defines four fields that Gemini must extract from any receipt text. Each field uses type: ['string', 'null'] (or ['number', 'null']) which tells Gemini it can return null when it cannot confidently determine a value.

The required array ensures all four fields are always present in the response, even if their values are null. This gives you a predictable shape to work with in your frontend form.

Why nullable fields instead of guessing?

Your old regex either matched a pattern or returned nothing. Gemini could guess a value for any field, but a wrong guess is worse than no answer. Nullable fields let the model say "I'm not sure" so your form can prompt the user for manual entry instead.

  • Save utils/parseReceiptWithGemini.js.

Seeing a syntax error in your schema?

Make sure each property's type value is an array with square brackets, not a plain string. For example, type: ['number', 'null'] is correct. Using type: 'number' without the array and 'null' entry will prevent the model from returning null values.

help me fix a syntax error in my receiptSchema definition

Write the parsing function

The schema defines the shape of the response. Now you need the function that sends OCR text to Gemini and gets structured data back.

  • In utils/parseReceiptWithGemini.js, add the async function below the receiptSchema definition:
async function parseReceiptWithGemini(ocrText) {
  try {
    // Call Gemini with structured output config
    const response = await ai.models.generateContent({
      model: 'gemini-2.5-flash',
      contents: `You are a receipt parser. Extract structured data from the following OCR text.
Return null for any field you cannot confidently determine from the text.
For the merchant field, extract the store or business name exactly as it appears.
For the description, summarize the purchased items briefly.

OCR Text:
${ocrText}`,
      config: {
        responseMimeType: 'application/json',
        responseJsonSchema: receiptSchema,
      },
    });

    // Parse the JSON string response into an object
    return JSON.parse(response.text);
  } catch (error) {
    console.error('Gemini parsing error:', error.message);
    // Return null values on failure instead of crashing the request
    return { amount: null, date: null, merchant: null, description: null };
  }
}

module.exports = { parseReceiptWithGemini };

What does this function do?

  • responseMimeType: 'application/json' forces the model to output valid JSON instead of free-form text.
  • responseJsonSchema: receiptSchema constrains the JSON shape to your exact schema, so every response has the same four fields.
  • The contents string is the prompt that tells Gemini how to interpret the OCR text and when to return null.
  • The catch block returns a safe fallback object with all null values, preventing your Express route from crashing if the API call fails.
  • Save utils/parseReceiptWithGemini.js.

Seeing 'ai.models.generateContent is not a function'?

Make sure you're initializing with new GoogleGenAI({ apiKey: ... }), not new GoogleGenAI(apiKey). The constructor expects an options object with an apiKey property.

help me debug why ai.models.generateContent is not a function in my parseReceiptWithGemini utility

✔️ Awesome, I've got everything!

Great. Double check you've saved your file before moving on.

ⓧ I'd like to double check the full code

const { GoogleGenAI } = require('@google/genai');

const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });

const receiptSchema = {
  type: 'object',
  properties: {
    amount: {
      type: ['number', 'null'],
      description: 'The total amount charged, as a number without currency symbols. Null if not confidently determined.',
    },
    date: {
      type: ['string', 'null'],
      description: 'The transaction date in YYYY-MM-DD format. Null if not confidently determined.',
    },
    merchant: {
      type: ['string', 'null'],
      description: 'The store or business name on the receipt. This will be used as the transaction category. Null if not confidently determined.',
    },
    description: {
      type: ['string', 'null'],
      description: 'A brief summary of the purchased items or services (e.g., "Groceries - milk, bread, bananas"). Null if not confidently determined.',
    },
  },
  required: ['amount', 'date', 'merchant', 'description'],
};

async function parseReceiptWithGemini(ocrText) {
  try {
    const response = await ai.models.generateContent({
      model: 'gemini-2.5-flash',
      contents: `You are a receipt parser. Extract structured data from the following OCR text.
Return null for any field you cannot confidently determine from the text.
For the merchant field, extract the store or business name exactly as it appears.
For the description, summarize the purchased items briefly.

OCR Text:
${ocrText}`,
      config: {
        responseMimeType: 'application/json',
        responseJsonSchema: receiptSchema,
      },
    });

    return JSON.parse(response.text);
  } catch (error) {
    console.error('Gemini parsing error:', error.message);
    return { amount: null, date: null, merchant: null, description: null };
  }
}

module.exports = { parseReceiptWithGemini };
Test the utility in isolation

Before wiring this into your Express route, you should confirm the utility works on its own. Testing in isolation lets you catch prompt or schema issues without the noise of your full app stack.

  • Create a test file called test-gemini.js in the root of your backend directory by running:
touch test-gemini.js
  • Open test-gemini.js and add the following test script:
require('dotenv').config();
const { parseReceiptWithGemini } = require('./utils/parseReceiptWithGemini');

// A well-formatted receipt for testing happy path
const sampleReceipt = `WALMART SUPERCENTER
123 MAIN ST ANYTOWN USA
01/15/2025 14:32

MILK 2% GAL          2.99
WHEAT BREAD           3.49
BANANAS              1.29
SUBTOTAL             7.77
TAX                  0.62
TOTAL                8.39

VISA ****1234
THANK YOU FOR SHOPPING`;

// A messy, corrupted OCR output for testing null handling
const messyReceipt = `...rcpt...\ntota1 $5.2O\nth4nk y0u`;

async function test() {
  console.log('=== Test 1: Clear receipt ===\n');
  const result1 = await parseReceiptWithGemini(sampleReceipt);
  console.log('Parsed result:', JSON.stringify(result1, null, 2));

  console.log('\n=== Test 2: Messy/partial receipt ===\n');
  const result2 = await parseReceiptWithGemini(messyReceipt);
  console.log('Parsed result:', JSON.stringify(result2, null, 2));
}

test();

What does this test script do?

  • sampleReceipt is a well-structured Walmart receipt with clear amounts, a date, and item names. You should see all four fields filled in.
  • messyReceipt simulates corrupted OCR output with mangled characters and numbers. You should see null values for fields the model cannot determine.
  • require('dotenv').config() at the top loads your GEMINI_API_KEY from .env so the utility can authenticate.
  • Save test-gemini.js and run it:
node test-gemini.js

You should see two JSON outputs in your terminal. The first test should return filled values for all four fields (amount around 8.39, date as 2025-01-15, merchant as WALMART SUPERCENTER).

The second test should return null for most fields since the OCR text is too corrupted for confident extraction.

Seeing 'Gemini parsing error' in the output?

  • Check that your .env file has GEMINI_API_KEY set to a valid key from Google AI Studio.
  • Make sure require('dotenv').config() is the very first line in test-gemini.js so the key loads before the utility runs.
  • Verify your internet connection is active. The SDK needs to reach the Gemini API.

help me debug why my test-gemini.js script is showing a Gemini parsing error

✔️ Awesome, I've got everything!

Great. You've confirmed Gemini can parse receipts into structured JSON.

ⓧ I'd like to double check the full code

require('dotenv').config();
const { parseReceiptWithGemini } = require('./utils/parseReceiptWithGemini');

const sampleReceipt = `WALMART SUPERCENTER
123 MAIN ST ANYTOWN USA
01/15/2025 14:32

MILK 2% GAL          2.99
WHEAT BREAD           3.49
BANANAS              1.29
SUBTOTAL             7.77
TAX                  0.62
TOTAL                8.39

VISA ****1234
THANK YOU FOR SHOPPING`;

const messyReceipt = `...rcpt...\ntota1 $5.2O\nth4nk y0u`;

async function test() {
  console.log('=== Test 1: Clear receipt ===\n');
  const result1 = await parseReceiptWithGemini(sampleReceipt);
  console.log('Parsed result:', JSON.stringify(result1, null, 2));

  console.log('\n=== Test 2: Messy/partial receipt ===\n');
  const result2 = await parseReceiptWithGemini(messyReceipt);
  console.log('Parsed result:', JSON.stringify(result2, null, 2));
}

test();

Your Gemini receipt parser is working independently. Next up, you'll wire it into your Express route and React form so the full flow goes from receipt photo to auto-filled transaction.

Integrate Gemini into Your Express Route and React Form

Your Gemini parsing utility is built and tested. Now it's time to wire it into your actual application flow.

In this step, you'll swap out the regex parsing in your Express route for a single parseReceiptWithGemini call. Then you'll update your React form to handle the structured response, mapping merchant to your category field and leaving null fields empty for manual entry.

In this step, get ready to:
  • Replace regex parsing with Gemini in your Express receipt route.
  • Update your React form to auto-fill from Gemini's structured output.
  • Verify the end-to-end flow from receipt photo to auto-filled form.
Update your receipt scanning route

Your Express route currently takes the raw text from Tesseract.js and runs it through regex patterns to extract amounts, dates, and merchants. Those patterns break on unfamiliar receipt formats. You'll replace that entire regex block with a single function call.

  • Open your existing receipt/transaction route file (the one that handles receipt image uploads with Multer and runs Tesseract.js OCR).
  • At the top of the file, add this import alongside your existing requires:
const { parseReceiptWithGemini } = require('./utils/parseReceiptWithGemini');

What does this line do?

This imports the utility function you built and tested in the previous step. The relative path ./utils/parseReceiptWithGemini assumes your route file is in the same directory as the utils/ folder. Adjust the path if your route file lives in a subdirectory like routes/ (e.g., ../utils/parseReceiptWithGemini).

  • Find the section in your route handler where Tesseract.js finishes extracting text and you run regex patterns on the result.
  • Replace the entire regex parsing block with the following Gemini call and response:
// Replace your regex parsing logic with Gemini structured output
const parsedData = await parseReceiptWithGemini(ocrText);

// Return the structured JSON to your frontend
res.json(parsedData);

What does this code do?

The variable ocrText is the raw string output from your Tesseract.js OCR step. Whatever variable name you used for that text, pass it into parseReceiptWithGemini().

The response is a structured object with amount, date, merchant, and description fields. Each field is either a valid value or null if Gemini couldn't confidently extract it.

Match your existing variable name

If your Tesseract.js output is stored in a variable called something like text or extractedText, pass that variable name instead of ocrText. The function just needs a string of the raw OCR output.

  • Remove any leftover regex parsing code (the .match() calls, regex pattern definitions, and manual field extraction logic). The Gemini function handles all of this now.
  • Save your route file.

Seeing a "parseReceiptWithGemini is not a function" error?

Check that your require path matches your folder structure. If your route file lives inside a routes/ directory, the path needs to go up one level: ../utils/parseReceiptWithGemini.

Also confirm the destructuring matches. The function is exported as { parseReceiptWithGemini } with curly braces, not as a default export.

Help me debug my parseReceiptWithGemini import in my Express route file

Connect to your React form auto-fill

Your React component already sends the receipt image to the API and uses the response to fill form fields. Now you need to handle the new response shape, where any field might be null.

  • Open your React receipt upload component (the file where you handle the API response and set form state).
  • Find the section where you set form state from the API response.
  • Update it to only set fields that are not null, and map merchant to your category/title field. Follow this pattern:
// After receiving the API response (e.g., const data = response.data)
// Only set form fields that Gemini could confidently extract
// Leave null fields empty so the user can fill them manually
if (data.amount !== null) {
  // Set your amount field state with data.amount
}
if (data.date !== null) {
  // Set your date field state with data.date
}
if (data.merchant !== null) {
  // Map merchant to your category/title field state
  // e.g., setCategory(data.merchant) or setTitle(data.merchant)
}
if (data.description !== null) {
  // Set your description field state with data.description
}

Why check for null before setting state?

When Gemini can't confidently extract a field, it returns null instead of guessing. By skipping null fields, you leave those form inputs empty. The user sees exactly which fields need manual entry.

The merchant field maps directly to your transaction category or title. No transformation needed. Gemini extracts the store name (e.g., "WALMART SUPERCENTER") and you use it as-is.

  • Replace your existing response-handling logic with the null-checking pattern above, adapted to your specific state setter names.
  • Save your component file.

Form fields not updating?

Open your browser's developer console and log the API response before setting state. Confirm you're accessing the data at the right level. If you use Axios, the parsed data lives at response.data. If you use fetch, you need to call .json() on the response first.

Help me debug why my React form isn't auto-filling from the Gemini API response

Test the end-to-end flow

Time to verify the full pipeline. You'll upload a real receipt and watch it flow through Multer, Tesseract.js, and Gemini to auto-fill your transaction form.

  • Start your Express server and React dev server if they aren't already running.
  • Take a photo of a receipt (or use an existing receipt image on your computer).
  • Upload the receipt through your app's upload interface.
  • Watch the form fields. The amount, date, category (from merchant), and description should auto-fill with extracted values.

What should I see?

For a clear receipt, you should see all four fields populate automatically. The amount should be a number (no currency symbol). The date should be in YYYY-MM-DD format. The category should show the store name. The description should summarize the items.

If any field couldn't be determined, it stays empty so you can type the value manually. This is the null handling working as intended.

  • Try uploading a different receipt format (a restaurant receipt, a gas station receipt, or a handwritten one) to verify Gemini handles variety better than your old regex.

Why test with different formats?

This is the whole reason you replaced regex. Regular expressions need a specific pattern to match against. A grocery store receipt looks different from a restaurant receipt, which looks different from an online order confirmation.

Gemini understands the semantic meaning of receipt text. It can identify "TOTAL" whether it appears as "TOTAL:", "Total Due", "Amount Charged", or any other variation.

Getting null for all fields?

This usually means the OCR text being passed to Gemini is empty or garbled. Open your server logs and check what Tesseract.js is outputting. If the OCR text is mostly unreadable characters, the issue is with image quality. Try a clearer, well-lit photo.

Also verify your GEMINI_API_KEY is loaded correctly. If the key is missing, the error handler in parseReceiptWithGemini catches the failure and returns all nulls. Check your terminal for "Gemini parsing error" messages.

Help me debug why Gemini is returning null for all receipt fields

Secret mission

Add Confidence Scores

Your parser returns null for uncertain fields, but what about fields it extracts with low confidence? Extend your schema to return a confidence score (0.0 to 1.0) for each field so your UI can visually flag shaky extractions for manual review.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses the Gemini API free tier and runs locally, so there are no ongoing costs.

Resources you used:

  • utils/parseReceiptWithGemini.js utility file (new file in your backend).
  • test-gemini.js test script (new file in your backend).
  • @google/genai npm package (installed in your backend).
  • GEMINI_API_KEY environment variable in your .env file.
  • Gemini API key in Google AI Studio.

Keep everything running

No action needed. Choose this if you plan to keep using Gemini-powered receipt parsing in your expense tracker.

  • Your Gemini API free tier has no charges, so the API key can stay active indefinitely.
  • The @google/genai package remains installed in your backend's node_modules.
  • Your modified Express route and React component continue working as-is.

Pause - I'll come back to this later

Remove your API key from the environment to prevent accidental usage while keeping all code intact.

  • Open your .env file and delete (or comment out) the GEMINI_API_KEY line.
  • Your code stays in place. When you return, just add the key back to .env and everything works again.

Delete - I don't want to use this again

Remove all project resources created during this project and revert to your previous setup.

  • Delete the new utility and test files by running this from your backend directory:
rm utils/parseReceiptWithGemini.js test-gemini.js
  • Uninstall the Gemini SDK by running:
npm uninstall @google/genai
  • Open your .env file and delete the GEMINI_API_KEY line.
  • Delete your API key in Google AI Studio. Go to https://aistudio.google.com/apikey, find the key you created, and click the trash icon to delete it.
  • Revert your Express route to use your old regex parsing logic. Replace the parseReceiptWithGemini call with your previous regex-based extraction.
  • Revert your React receipt upload component to remove the confidence score visual indicators and restore the previous auto-fill logic without null handling.

Nice Work!

Nice Work!

You just upgraded your MERN expense tracker from brittle regex parsing to AI-powered receipt extraction with Gemini 2.5 Flash structured output.

What you learned:

  • Built a Gemini-powered receipt parser using structured output with nullable fields, guaranteeing schema-conformant JSON while returning null for uncertain data instead of wrong guesses.
  • Integrated the parser into your existing MERN stack flow (Multer upload to Tesseract.js OCR to Gemini) with minimal code changes, mapping the merchant field directly to your transaction category.
  • Tested the full end-to-end pipeline: receipt photo to OCR to structured JSON to auto-filled transaction form, handling varied receipt formats gracefully.
  • Secret Mission: Added confidence scores for each extracted field so your UI visually flags low-confidence extractions for manual review.

Ready to quiz yourself?