AI Stock Transaction Extractor

Add a Gemini Vision endpoint to extract stock trades from broker screenshots.

Introduction

30 Second Summary

Manually typing stock transactions from your broker app into a portfolio tracker is tedious and error-prone. What if you could just snap a screenshot and let AI do the data entry for you?

In this project, you will build an AI-powered image extraction feature for your existing stock portfolio tracker. You'll upload a broker screenshot, and your app will return structured transaction data ready to be saved.

What You'll Build

A new REST API endpoint that accepts broker screenshot uploads and uses Google Gemini Vision to extract stock transactions as structured JSON.

By the end of this project, you'll have:

  • A /api/extract-transactions POST endpoint that accepts image uploads and returns extracted stock data.
  • A Gemini Vision service integrated into your Spring Boot app using the Google GenAI Java SDK.
  • A structured JSON response with extracted stock names, prices, quantities, and transaction types.
  • Your feature deployed live on Render.com with secure environment variable management.
  • Secret Mission: Add confidence scoring and graceful error handling for blurry or non-stock images.

Are there any prerequisites?

This project builds on an existing Spring Boot portfolio tracker app already deployed to Render.com. You should have IntelliJ IDEA installed and a Java 21 JDK set up on your machine.

Get Your Gemini API Key

Your portfolio tracker app is ready to get a new superpower: AI-powered image understanding. To connect your app to Google's Gemini model, you need an API key that authenticates your requests.

In this step, you'll create a free Gemini API key, verify it works, and configure your IntelliJ project to use it securely through an environment variable.

In this step, get ready to:
  • Create a Gemini API key in Google AI Studio.
  • Test connectivity with a cURL command.
  • Add the key to your IntelliJ run configuration as an environment variable.
Create your API key in Google AI Studio

Google AI Studio is Google's developer portal for managing Gemini API access. You'll generate a free API key here that lets your Spring Boot app talk to Gemini.

  • Open your browser and go to the Google AI Studio API keys page:
  • https://aistudio.google.com/apikey
  • Sign in with your Google account if prompted.
  • Click Create API Key.
  • If asked to select a project, choose an existing Google Cloud project or let AI Studio create one for you.
  • Copy the generated API key and save it somewhere safe (like a text file on your Desktop).

What is Google AI Studio?

Google AI Studio is Google's interface for managing Gemini API keys and testing prompts. The free tier gives you access to Gemini models without requiring a credit card.

Can't create an API key?

If you see a permissions error, make sure you're signed into your personal Google account (not a work or school account that might have restrictions). Try creating a new Google Cloud project from within AI Studio if the default one doesn't work.

Help me troubleshoot my Google AI Studio API key creation

Test your API key with cURL

Before wiring the key into your Java app, confirm it works by sending a quick test request directly from your terminal.

  • Open a terminal in IntelliJ by clicking Terminal in the bottom toolbar (or press Alt+F12 on Windows / Option+F12 on macOS).
  • Test your API key by running this command (replace your-api-key-here with the key you just copied):
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash:generateContent" \
  -H "x-goog-api-key: your-api-key-here" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{
    "contents": [
      {
        "parts": [
          {
            "text": "Explain how AI works in a few words"
          }
        ]
      }
    ]
  }'

What does this command do?

  • It sends a POST request to the Gemini generateContent endpoint with the model gemini-3.5-flash.
  • The x-goog-api-key header authenticates your request using your API key.
  • The body sends a simple text prompt to verify the model responds.
  • You should see a JSON response containing the model's generated text. If you see a candidates array with text content inside, your key is working.

Seeing a 403 or permission error?

Double-check that you pasted the full API key with no extra spaces. Make sure the Generative Language API is enabled on your Google Cloud project (AI Studio does this automatically when you create the key).

If the error persists, try creating a new API key in a different Google Cloud project from AI Studio.

Help me fix a 403 error when testing my Gemini API key

Add the API key to your IntelliJ run configuration

Your app reads the API key from an environment variable at runtime. Setting it in IntelliJ's run configuration keeps the key out of your source code.

  • In IntelliJ, click the dropdown next to the green Run button in the top toolbar.
  • Select Edit Configurations....
  • Click on your Spring Boot application's run configuration (it should already exist from your portfolio tracker project).
  • Find the Environment variables field.
  • Click the browse icon (or the field itself) and add a new variable with name GEMINI_API_KEY and paste your API key as the value.
  • Click OK to save, then click Apply and OK to close the dialog.

Why an environment variable?

Storing the API key in an environment variable means it never appears in your code or gets committed to Git. Your GeminiService class will read it at startup using System.getenv("GEMINI_API_KEY").

Don't see the Environment variables field?

Click Modify options at the top of the configuration dialog, then check Environment variables to make the field visible.

Help me find the environment variables setting in IntelliJ IDEA

Your API key is live and your IntelliJ project knows how to find it. Next up, you'll add the Google GenAI SDK to your project so your Java code can call Gemini directly.

Add Google GenAI SDK to Your Project

Your API key is ready and configured in IntelliJ. Now it is time to give your Spring Boot app the ability to talk to Gemini.

The Google GenAI Java SDK is a single Maven dependency that provides a client for calling Gemini models. In this step, you will add the SDK to your project and create a service class that initializes the connection.

In this step, get ready to:
  • Add the Google GenAI SDK Maven dependency to your project.
  • Create a GeminiService class that connects to the Gemini API.
  • Verify everything compiles successfully.
Add the Maven dependency

The Google GenAI Java SDK lets your app communicate with Gemini models using a few lines of Java. You need to declare it as a dependency so Maven can download it.

  • In IntelliJ, open your pom.xml file from the project root.
  • Find the <dependencies> section and add the following dependency at the end, before the closing </dependencies> tag:
<dependency>
    <groupId>com.google.genai</groupId>
    <artifactId>google-genai</artifactId>
    <version>1.55.0</version>
</dependency>

What does this dependency do?

This pulls in the official Google GenAI Java SDK (version 1.55.0). It gives you the Client class for connecting to Gemini models and the types for building requests with images, text, and structured output schemas.

  • Save pom.xml.
  • IntelliJ may show a small floating icon or notification asking to reload Maven changes. Click the reload icon, or press Cmd+Shift+A (macOS) or Ctrl+Shift+A (Windows), type Reload All Maven Projects, and press Enter.
  • In the Maven tool window on the right side, expand Dependencies. You should see com.google.genai:google-genai:1.55.0 listed.

Don't see the dependency listed?

Make sure you pasted the dependency inside the <dependencies> block, not outside it. Also check for any red underlines in the XML that indicate a syntax error.

If Maven reports a resolution failure, confirm you have an internet connection. IntelliJ needs to download the library from Maven Central.

Help me fix a Maven dependency issue in IntelliJ

Create the GeminiService class

With the SDK available, you need a service class that initializes the Gemini client. This class will be the single point of contact between your app and the Gemini API.

  • In IntelliJ's Project panel on the left, navigate to src/main/java/com/yourapp/service/.
  • Right-click the service folder, select New, then Java Class.
  • Name it GeminiService and press Enter.
  • Replace the generated contents with the following code:
package com.yourapp.service;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.genai.Client;
import org.springframework.stereotype.Service;

@Service
public class GeminiService {

    private final Client client;
    private final ObjectMapper objectMapper;

    public GeminiService() {
        // Read the API key from the environment variable
        String apiKey = System.getenv("GEMINI_API_KEY");
        if (apiKey == null || apiKey.isBlank()) {
            throw new IllegalStateException("GEMINI_API_KEY environment variable is not set");
        }
        // Initialize the Gemini client with the API key
        this.client = Client.builder().apiKey(apiKey).build();
        // Jackson ObjectMapper for parsing JSON responses later
        this.objectMapper = new ObjectMapper();
    }
}

What does this code do?

  • @Service tells Spring Boot to automatically create and manage this class as a singleton bean.
  • System.getenv("GEMINI_API_KEY") reads the API key you configured in your IntelliJ run configuration in the previous step.
  • Client.builder().apiKey(apiKey).build() creates the Gemini client that you will use to send images and prompts to the model.
  • The ObjectMapper is prepared now so you can parse JSON responses from Gemini in the next step.
  • Save GeminiService.java.

Seeing red underlines on the imports?

If com.google.genai.Client shows as unresolved, Maven has not finished downloading the dependency yet. Go back and reload your Maven project. Wait for the progress bar in the bottom status bar to finish.

If com.fasterxml.jackson.databind.ObjectMapper is unresolved, your Spring Boot starter should already include Jackson. Confirm that spring-boot-starter-web is in your pom.xml dependencies.

Help me resolve import errors in my GeminiService class

Verify the project compiles

A quick compile confirms that Maven can resolve the new dependency and that your service class has no syntax errors.

  • Open the terminal in IntelliJ by clicking Terminal in the bottom toolbar (or press Alt+F12).
  • Compile your project by running:
mvn compile
  • You should see BUILD SUCCESS at the end of the output. This confirms Maven downloaded the Google GenAI SDK and your GeminiService class compiled without errors.

Seeing BUILD FAILURE?

If the error mentions google-genai not being found, check that your <dependency> block has the exact groupId com.google.genai and artifactId google-genai. A typo in either will cause a resolution failure.

If the error references GEMINI_API_KEY, that is a runtime error and should not appear during compile. Compile only checks syntax, not environment variables.

Help me fix my Maven build failure

✔️ Awesome, I've got everything!

Great. Double check you have saved both pom.xml and GeminiService.java.

ⓧ I'd like to double check the full code

Here is the dependency you should have added to pom.xml:

<!-- Add inside your existing <dependencies> section -->
<dependency>
    <groupId>com.google.genai</groupId>
    <artifactId>google-genai</artifactId>
    <version>1.55.0</version>
</dependency>

Here is the full GeminiService.java file:

package com.yourapp.service;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.genai.Client;
import org.springframework.stereotype.Service;

@Service
public class GeminiService {

    private final Client client;
    private final ObjectMapper objectMapper;

    public GeminiService() {
        String apiKey = System.getenv("GEMINI_API_KEY");
        if (apiKey == null || apiKey.isBlank()) {
            throw new IllegalStateException("GEMINI_API_KEY environment variable is not set");
        }
        this.client = Client.builder().apiKey(apiKey).build();
        this.objectMapper = new ObjectMapper();
    }
}

Your project now has the Gemini SDK wired in and a service class ready to make API calls. Next up, you will build the extraction logic that sends broker screenshots to Gemini and gets structured transaction data back.

Build the Transaction Extraction Logic

Your app now has the Gemini SDK wired up, but it doesn't know what to do with an image yet. You need a method that accepts raw image bytes, sends them to Gemini with a structured prompt, and gets back clean JSON with stock transaction data.

In this step, you'll build the core extraction logic. This means defining the shape of your response (a DTO), writing the multimodal prompt that tells Gemini exactly what to extract, and parsing the structured output back into Java objects.

In this step, get ready to:
  • Create a response DTO that defines the structure of extracted transaction data.
  • Write an extraction method that sends images to Gemini with a JSON schema for structured output.
  • Parse the structured JSON response from Gemini into typed Java objects.
Create the transaction response DTO

Before you call Gemini, you need a Java class that defines the shape of the data coming back. This DTO (Data Transfer Object) maps Gemini's JSON output to typed fields your app can work with.

  • In IntelliJ, right-click the com.yourapp package in your src/main/java folder.
  • Select New, then Package, and name it dto.
  • Right-click the new dto package, select New, then Java Class, and name it TransactionExtractionResponse.
  • Replace the file contents with the following code:
package com.yourapp.dto;

import java.util.List;

public class TransactionExtractionResponse {

    private List<StockTransaction> transactions;
    private String message;

    public TransactionExtractionResponse() {}

    public TransactionExtractionResponse(List<StockTransaction> transactions, String message) {
        this.transactions = transactions;
        this.message = message;
    }

    public List<StockTransaction> getTransactions() {
        return transactions;
    }

    public void setTransactions(List<StockTransaction> transactions) {
        this.transactions = transactions;
    }

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }

    // Inner class representing a single stock transaction
    public static class StockTransaction {
        private String stockName;
        private double stockPrice;
        private int quantity;
        private String transactionType;

        public StockTransaction() {}

        public String getStockName() {
            return stockName;
        }

        public void setStockName(String stockName) {
            this.stockName = stockName;
        }

        public double getStockPrice() {
            return stockPrice;
        }

        public void setStockPrice(double stockPrice) {
            this.stockPrice = stockPrice;
        }

        public int getQuantity() {
            return quantity;
        }

        public void setQuantity(int quantity) {
            this.quantity = quantity;
        }

        public String getTransactionType() {
            return transactionType;
        }

        public void setTransactionType(String transactionType) {
            this.transactionType = transactionType;
        }
    }
}

What does this code do?

This file defines two classes that model the API response:

  • TransactionExtractionResponse wraps a list of transactions plus a status message. This is the shape your REST endpoint will return to the caller.
  • StockTransaction is a static inner class holding the four fields Gemini will extract from each stock entry: stockName, stockPrice, quantity, and transactionType (BUY or SELL).
  • Save TransactionExtractionResponse.java.

Seeing a red underline on the package name?

Make sure the file is inside the dto sub-package under your main package. The package declaration at the top of the file must match the folder path exactly.

Help me fix my DTO package structure

Write the extraction and parsing logic

Now for the core AI integration. You'll add a method to GeminiService that sends an image to gemini-3.5-flash as multimodal content, tells it to respond in structured JSON, and parses the output into your DTO.

What is structured output?

Normally, an LLM returns freeform text. Structured output lets you pass a JSON Schema to Gemini so it guarantees the response matches that exact shape. No more regex parsing or hoping the model formats things correctly.

  • Open GeminiService.java in IntelliJ.
  • Replace the entire import section at the top of the file with the following imports:
package com.yourapp.service;

import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.genai.Client;
import com.google.genai.types.Content;
import com.google.genai.types.GenerateContentConfig;
import com.google.genai.types.GenerateContentResponse;
import com.google.genai.types.Part;
import com.yourapp.dto.TransactionExtractionResponse.StockTransaction;
import org.springframework.stereotype.Service;

import java.util.Base64;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

What are these new imports?

  • Content, Part, GenerateContentConfig, and GenerateContentResponse are Gemini SDK types for building multimodal requests.
  • JsonNode and TypeReference are Jackson classes for parsing the JSON response into your StockTransaction objects.
  • Inside the GeminiService class, add the extractTransactions method below the constructor by pasting the following code:
    public List<StockTransaction> extractTransactions(byte[] imageBytes, String mimeType) throws Exception {
        String base64Image = Base64.getEncoder().encodeToString(imageBytes);

        // Prompt telling Gemini what to extract from the image
        String prompt = """
                Analyze this broker/trading app screenshot and extract all stock transactions visible.
                For each transaction, extract the stock name, stock price, quantity, and whether it is a BUY or SELL transaction.
                If you cannot clearly read a value, skip that transaction entirely.
                Only extract data that is clearly visible in the image.
                """;

        // Build multimodal content with both text prompt and image bytes
        Content content = Content.fromParts(
                Part.fromText(prompt),
                Part.fromBytes(imageBytes, mimeType)
        );

        // Define the JSON schema that Gemini must conform to
        Map<String, Object> schema = buildTransactionSchema();

        // Configure structured JSON output
        GenerateContentConfig config = GenerateContentConfig.builder()
                .responseMimeType("application/json")
                .responseJsonSchema(schema)
                .build();

        // Call Gemini 3.5 Flash with the multimodal content and config
        GenerateContentResponse response = client.models.generateContent(
                "gemini-3.5-flash",
                content,
                config
        );

        // Parse the JSON response into StockTransaction objects
        String jsonResponse = response.text();
        JsonNode rootNode = objectMapper.readTree(jsonResponse);
        JsonNode transactionsNode = rootNode.has("transactions")
                ? rootNode.get("transactions")
                : rootNode;

        return objectMapper.convertValue(transactionsNode, new TypeReference<List<StockTransaction>>() {});
    }

What does this method do?

This method is the heart of your AI feature. Here's the flow:

  • It encodes the image bytes as Base64 and builds a multimodal Content object combining a text prompt with the raw image.
  • The GenerateContentConfig forces Gemini to respond in JSON that matches your schema. No freeform text, just structured data.
  • After calling generateContent, it parses the JSON response with Jackson. The rootNode.has("transactions") check handles both wrapped and unwrapped response formats gracefully.
  • Below the extractTransactions method, add the buildTransactionSchema helper method by pasting this code:
    private Map<String, Object> buildTransactionSchema() {
        // Define the properties for each stock transaction
        Map<String, Object> stockTransactionProperties = new HashMap<>();
        stockTransactionProperties.put("stockName", Map.of("type", "string", "description", "The name or ticker symbol of the stock"));
        stockTransactionProperties.put("stockPrice", Map.of("type", "number", "description", "The price per share"));
        stockTransactionProperties.put("quantity", Map.of("type", "integer", "description", "The number of shares"));
        stockTransactionProperties.put("transactionType", Map.of("type", "string", "enum", List.of("BUY", "SELL"), "description", "Whether this is a buy or sell transaction"));

        // Combine properties into a single transaction object schema
        Map<String, Object> stockTransactionSchema = new HashMap<>();
        stockTransactionSchema.put("type", "object");
        stockTransactionSchema.put("properties", stockTransactionProperties);
        stockTransactionSchema.put("required", List.of("stockName", "stockPrice", "quantity", "transactionType"));

        // Wrap in a top-level schema with a transactions array
        Map<String, Object> schema = new HashMap<>();
        schema.put("type", "object");
        schema.put("properties", Map.of(
                "transactions", Map.of(
                        "type", "array",
                        "items", stockTransactionSchema
                )
        ));
        schema.put("required", List.of("transactions"));

        return schema;
    }

What does the schema do?

This method builds a JSON Schema as a Java Map. When you pass this schema to Gemini via responseJsonSchema, the model is forced to output JSON that conforms to it exactly.

The enum on transactionType constrains the value to only BUY or SELL. Gemini cannot return anything else for that field.

  • Save GeminiService.java.
  • Verify the project still compiles by running this in your terminal:
mvn compile
  • You should see BUILD SUCCESS in the terminal output. This confirms that all your new imports, the DTO, and the service methods resolve correctly.

Seeing a compilation error?

  • If you see an error about StockTransaction not being found, double check the import at the top of GeminiService.java reads com.yourapp.dto.TransactionExtractionResponse.StockTransaction and that the DTO package matches your actual folder structure.
  • If you see an error about Content or Part, make sure you replaced ALL the old imports with the full import block shown above.

Help me fix my GeminiService compilation error

✔️ Awesome, I've got everything!

Great. Make sure both files are saved before moving on.

ⓧ I'd like to double check the full code

Here is the complete TransactionExtractionResponse.java:

package com.yourapp.dto;

import java.util.List;

public class TransactionExtractionResponse {

    private List<StockTransaction> transactions;
    private String message;

    public TransactionExtractionResponse() {}

    public TransactionExtractionResponse(List<StockTransaction> transactions, String message) {
        this.transactions = transactions;
        this.message = message;
    }

    public List<StockTransaction> getTransactions() {
        return transactions;
    }

    public void setTransactions(List<StockTransaction> transactions) {
        this.transactions = transactions;
    }

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }

    public static class StockTransaction {
        private String stockName;
        private double stockPrice;
        private int quantity;
        private String transactionType;

        public StockTransaction() {}

        public String getStockName() {
            return stockName;
        }

        public void setStockName(String stockName) {
            this.stockName = stockName;
        }

        public double getStockPrice() {
            return stockPrice;
        }

        public void setStockPrice(double stockPrice) {
            this.stockPrice = stockPrice;
        }

        public int getQuantity() {
            return quantity;
        }

        public void setQuantity(int quantity) {
            this.quantity = quantity;
        }

        public String getTransactionType() {
            return transactionType;
        }

        public void setTransactionType(String transactionType) {
            this.transactionType = transactionType;
        }
    }
}

And here is the complete GeminiService.java:

package com.yourapp.service;

import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.genai.Client;
import com.google.genai.types.Content;
import com.google.genai.types.GenerateContentConfig;
import com.google.genai.types.GenerateContentResponse;
import com.google.genai.types.Part;
import com.yourapp.dto.TransactionExtractionResponse.StockTransaction;
import org.springframework.stereotype.Service;

import java.util.Base64;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

@Service
public class GeminiService {

    private final Client client;
    private final ObjectMapper objectMapper;

    public GeminiService() {
        String apiKey = System.getenv("GEMINI_API_KEY");
        if (apiKey == null || apiKey.isBlank()) {
            throw new IllegalStateException("GEMINI_API_KEY environment variable is not set");
        }
        this.client = Client.builder().apiKey(apiKey).build();
        this.objectMapper = new ObjectMapper();
    }

    public List<StockTransaction> extractTransactions(byte[] imageBytes, String mimeType) throws Exception {
        String base64Image = Base64.getEncoder().encodeToString(imageBytes);

        String prompt = """
                Analyze this broker/trading app screenshot and extract all stock transactions visible.
                For each transaction, extract the stock name, stock price, quantity, and whether it is a BUY or SELL transaction.
                If you cannot clearly read a value, skip that transaction entirely.
                Only extract data that is clearly visible in the image.
                """;

        Content content = Content.fromParts(
                Part.fromText(prompt),
                Part.fromBytes(imageBytes, mimeType)
        );

        Map<String, Object> schema = buildTransactionSchema();

        GenerateContentConfig config = GenerateContentConfig.builder()
                .responseMimeType("application/json")
                .responseJsonSchema(schema)
                .build();

        GenerateContentResponse response = client.models.generateContent(
                "gemini-3.5-flash",
                content,
                config
        );

        String jsonResponse = response.text();
        JsonNode rootNode = objectMapper.readTree(jsonResponse);
        JsonNode transactionsNode = rootNode.has("transactions")
                ? rootNode.get("transactions")
                : rootNode;

        return objectMapper.convertValue(transactionsNode, new TypeReference<List<StockTransaction>>() {});
    }

    private Map<String, Object> buildTransactionSchema() {
        Map<String, Object> stockTransactionProperties = new HashMap<>();
        stockTransactionProperties.put("stockName", Map.of("type", "string", "description", "The name or ticker symbol of the stock"));
        stockTransactionProperties.put("stockPrice", Map.of("type", "number", "description", "The price per share"));
        stockTransactionProperties.put("quantity", Map.of("type", "integer", "description", "The number of shares"));
        stockTransactionProperties.put("transactionType", Map.of("type", "string", "enum", List.of("BUY", "SELL"), "description", "Whether this is a buy or sell transaction"));

        Map<String, Object> stockTransactionSchema = new HashMap<>();
        stockTransactionSchema.put("type", "object");
        stockTransactionSchema.put("properties", stockTransactionProperties);
        stockTransactionSchema.put("required", List.of("stockName", "stockPrice", "quantity", "transactionType"));

        Map<String, Object> schema = new HashMap<>();
        schema.put("type", "object");
        schema.put("properties", Map.of(
                "transactions", Map.of(
                        "type", "array",
                        "items", stockTransactionSchema
                )
        ));
        schema.put("required", List.of("transactions"));

        return schema;
    }
}

Your extraction logic is complete. The service can now accept raw image bytes, send them to Gemini, and get back typed Java objects. Next up, you'll expose this as a REST endpoint so external clients can upload screenshots.

Create the REST Endpoint

Your GeminiService can extract transactions from images, but there is no way for users to send images to it yet.

In this step, you will create a REST endpoint that accepts image uploads over HTTP. The controller validates the uploaded file, passes it to GeminiService, and returns the extracted transactions as JSON.

In this step, get ready to:
  • Create a REST controller with a POST endpoint for image uploads.
  • Wire the controller to GeminiService and return extracted transactions.
  • Add input validation for file type and size.
Create the TransactionExtractorController

A Spring Boot controller is the entry point for HTTP requests. It maps incoming requests to methods, handles input, and returns responses.

  • In IntelliJ, right-click the com.yourapp package in your src/main/java folder.
  • Create a new package called controller.
  • Inside the controller package, create a new Java class called TransactionExtractorController.
  • Replace the contents of the file with the following code:
package com.yourapp.controller;

import com.yourapp.dto.TransactionExtractionResponse;
import com.yourapp.dto.TransactionExtractionResponse.StockTransaction;
import com.yourapp.service.GeminiService;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

import java.util.List;

@RestController
@RequestMapping("/api")
public class TransactionExtractorController {

    private final GeminiService geminiService;

    // Spring Boot injects GeminiService automatically via constructor injection
    public TransactionExtractorController(GeminiService geminiService) {
        this.geminiService = geminiService;
    }

    @PostMapping("/extract-transactions")
    public ResponseEntity<TransactionExtractionResponse> extractTransactions(
            @RequestParam("image") MultipartFile image) {

        // Validate: reject empty uploads
        if (image.isEmpty()) {
            return ResponseEntity.badRequest()
                    .body(new TransactionExtractionResponse(null, "No image provided"));
        }

        // Validate: only allow PNG and JPEG image formats
        String contentType = image.getContentType();
        if (contentType == null || (!contentType.equals("image/png") && !contentType.equals("image/jpeg"))) {
            return ResponseEntity.badRequest()
                    .body(new TransactionExtractionResponse(null, "Only PNG and JPEG images are supported"));
        }

        // Validate: enforce 4MB size limit for Gemini base64 encoding
        if (image.getSize() > 4 * 1024 * 1024) {
            return ResponseEntity.badRequest()
                    .body(new TransactionExtractionResponse(null, "Image size must be under 4MB"));
        }

        // Call GeminiService to extract transactions from the image
        try {
            List<StockTransaction> transactions = geminiService.extractTransactions(
                    image.getBytes(), contentType);
            return ResponseEntity.ok(
                    new TransactionExtractionResponse(transactions, "Successfully extracted " + transactions.size() + " transactions"));
        } catch (Exception e) {
            return ResponseEntity.internalServerError()
                    .body(new TransactionExtractionResponse(null, "Failed to extract transactions: " + e.getMessage()));
        }
    }
}

What does this code do?

This controller handles the full lifecycle of a transaction extraction request:

  • @RestController and @RequestMapping("/api") register this class as an HTTP handler under the /api path.
  • @PostMapping("/extract-transactions") maps POST requests to /api/extract-transactions to this method.
  • @RequestParam("image") binds the uploaded file from the multipart form data to the image parameter.
  • The constructor accepts GeminiService as a parameter. Spring Boot automatically injects the service instance at startup.
Understand how the service wiring works

The controller delegates all AI logic to GeminiService. It only handles HTTP concerns like reading the upload, validating input, and formatting the response.

Why separate the controller from the service?

Keeping HTTP handling in the controller and AI logic in the service follows the single-responsibility principle. If you later swap Gemini for a different AI provider, only GeminiService changes. The controller stays the same.

The try/catch block wraps the service call so that any unexpected error from the Gemini API returns a clean 500 response instead of crashing the app.

Review the input validation

The three validation checks run before the image ever reaches Gemini. Each one returns an HTTP 400 (Bad Request) with a descriptive message so the caller knows exactly what went wrong.

Why these three checks?

  • Empty file check: Prevents sending a zero-byte payload to the API.
  • MIME type check: Gemini vision only processes image data. Sending a PDF or text file would produce garbage output.
  • Size check (4MB): Gemini enforces a 4MB limit on base64-encoded inline images. Catching oversized files here gives the user a clear message instead of a cryptic API error.

Increase the Spring Boot upload limit

Spring Boot defaults to a 1MB max file size. Since your endpoint allows up to 4MB, add this to your application.properties:

spring.servlet.multipart.max-file-size=5MB
spring.servlet.multipart.max-request-size=5MB

Setting it to 5MB gives headroom above your 4MB validation check.

  • Save TransactionExtractorController.java.
  • In your IntelliJ terminal, confirm the project still compiles by running:
mvn compile

You should see a BUILD SUCCESS message in the terminal output.

Seeing a compilation error?

  • Check that your package declaration matches the folder structure. If your app uses a different base package than com.yourapp, update the imports and package line accordingly.
  • Verify that GeminiService.java and TransactionExtractionResponse.java exist in the correct packages (service and dto respectively).
  • Make sure the controller package is inside your main application package so Spring Boot can find it during component scanning.

Help me fix my TransactionExtractorController compilation error

✔️ Awesome, I've got everything!

Great. Double check you have saved both TransactionExtractorController.java and application.properties.

ⓧ I'd like to double check the full code

package com.yourapp.controller;

import com.yourapp.dto.TransactionExtractionResponse;
import com.yourapp.dto.TransactionExtractionResponse.StockTransaction;
import com.yourapp.service.GeminiService;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

import java.util.List;

@RestController
@RequestMapping("/api")
public class TransactionExtractorController {

    private final GeminiService geminiService;

    public TransactionExtractorController(GeminiService geminiService) {
        this.geminiService = geminiService;
    }

    @PostMapping("/extract-transactions")
    public ResponseEntity<TransactionExtractionResponse> extractTransactions(
            @RequestParam("image") MultipartFile image) {

        if (image.isEmpty()) {
            return ResponseEntity.badRequest()
                    .body(new TransactionExtractionResponse(null, "No image provided"));
        }

        String contentType = image.getContentType();
        if (contentType == null || (!contentType.equals("image/png") && !contentType.equals("image/jpeg"))) {
            return ResponseEntity.badRequest()
                    .body(new TransactionExtractionResponse(null, "Only PNG and JPEG images are supported"));
        }

        if (image.getSize() > 4 * 1024 * 1024) {
            return ResponseEntity.badRequest()
                    .body(new TransactionExtractionResponse(null, "Image size must be under 4MB"));
        }

        try {
            List<StockTransaction> transactions = geminiService.extractTransactions(
                    image.getBytes(), contentType);
            return ResponseEntity.ok(
                    new TransactionExtractionResponse(transactions, "Successfully extracted " + transactions.size() + " transactions"));
        } catch (Exception e) {
            return ResponseEntity.internalServerError()
                    .body(new TransactionExtractionResponse(null, "Failed to extract transactions: " + e.getMessage()));
        }
    }
}

Your endpoint is ready to accept image uploads, validate them, and return structured transaction data. Next up, you will test it locally with a real broker screenshot.

Test Locally

You've built the full extraction pipeline. Now it's time to see it in action with real broker screenshots.

In this step, you'll start your app, hit the endpoint with a sample image, and confirm that Gemini returns structured JSON you can trust.

In this step, get ready to:
  • Start your Spring Boot app and confirm it boots without errors.
  • Upload a broker screenshot to your endpoint and verify the extracted JSON.
  • Test with multiple screenshot formats to confirm robustness.
Start your Spring Boot app locally

Your run configuration already has the GEMINI_API_KEY environment variable set from earlier. Starting the app from IntelliJ will pick it up automatically.

  • In IntelliJ IDEA, click the green Run button in the top toolbar (or press Ctrl+R on macOS or Shift+F10 on Windows).
  • Wait for the console to show that your app started on port 8080.

What should I see?

The IntelliJ console should display a Spring Boot banner followed by a line containing Started and your application class name. You should also see a line mentioning Tomcat started on port 8080.

App failing to start?

If you see an error about GEMINI_API_KEY not being set, double-check your IntelliJ run configuration has the environment variable added correctly.

If you see dependency resolution errors, run mvn compile in the terminal to confirm all dependencies download properly.

Help me debug my Spring Boot startup error

Upload a sample broker screenshot

You need a screenshot from a real stock broker or trading app that shows completed transactions. This could be from your own brokerage account, or a sample image showing stock names, prices, and quantities.

  • Save a broker screenshot (PNG or JPEG, under 4MB) to a location you can easily reference in the terminal.
  • Open a terminal window. In IntelliJ, click Terminal in the bottom toolbar.
  • Send the screenshot to your endpoint by running this cURL command:
curl -X POST -F "image=@[[SCREENSHOT_PATH="path/to/your/broker-screenshot.png"]]" http://localhost:8080/api/extract-transactions

What does this command do?

  • -F "image=@..." sends the file as a multipart form upload with the parameter name image. The @ symbol tells cURL to read from a file.
  • cURL automatically sets the content type to multipart/form-data, which is what your Spring Boot controller expects.
  • Check the JSON response in your terminal. You should see a transactions array with objects containing stockName, stockPrice, quantity, and transactionType fields.

What should I see?

A successful response looks like this:

{
  "transactions": [
    {
      "stockName": "AAPL",
      "stockPrice": 187.50,
      "quantity": 10,
      "transactionType": "BUY"
    }
  ],
  "message": "Successfully extracted 1 transactions"
}

The exact values depend on what your screenshot shows. The key thing is that you get back structured data matching the fields your DTO defines.

Getting an empty transactions array?

Make sure your screenshot clearly shows stock transaction data (stock names, prices, quantities). Gemini will return an empty array if it cannot identify any transactions in the image.

Also confirm the image is a PNG or JPEG file under 4MB. Other formats or oversized files will be rejected by the validation in your controller.

Help me debug my empty extraction response

Test with different screenshot formats

A single test proves the endpoint works. Testing with varied inputs proves it's robust enough for real users who will upload screenshots from different broker apps.

  • Test with a screenshot showing multiple stock transactions (3 or more stocks in a single image).
  • Test with a screenshot showing a single stock transaction.
  • If you have access to a different broker app or trading platform, test with a screenshot from that app to confirm layout flexibility.

No broker account?

You can search for sample stock trading screenshots online. Look for images that clearly show stock names, prices, and quantities in a table or list format. The clearer the text in the image, the better Gemini can extract it.

  • For each test, run the same cURL command with a different file path and verify that the response contains accurate stockName, stockPrice, quantity, and transactionType values.
  • Confirm that multi-stock screenshots return multiple objects in the transactions array.

Why test multiple formats?

Different brokers display transaction data in different layouts. Some use tables, others use card-style lists. Gemini's vision capabilities handle varied layouts well, but testing proves your prompt and schema are flexible enough for real-world use.

Your extraction endpoint is working locally and handling multiple screenshot formats. Next up, you'll deploy this feature live on Render.com so it's accessible from anywhere.

Deploy to Render.com

Your transaction extractor works locally. Now it is time to ship it to production so the feature is available on your live app.

Your GeminiService reads the API key from an environment variable at startup. On Render.com, you configure that variable through the dashboard so the key never touches your source code.

In this step, get ready to:
  • Add the GEMINI_API_KEY environment variable to your Render.com service.
  • Push your code changes to Git and trigger an auto-deploy.
  • Test the live endpoint with a real broker screenshot.
Add the GEMINI_API_KEY environment variable in Render.com

Your app reads GEMINI_API_KEY at runtime using System.getenv(). Render.com needs this variable configured in its dashboard so the deployed container can access it.

  • Open the Render Dashboard at dashboard.render.com.
  • Select your portfolio tracker service from the list.
  • Click Environment in the left pane.
  • Under Environment Variables, click + Add Environment Variable.
  • In the Key field, enter GEMINI_API_KEY.
  • In the Value field, paste the same Gemini API key you used locally.
  • From the save dropdown, select Save only.

Why Save only?

Choosing Save only stores the variable without triggering a deploy. You will push your code changes in the next substep, which triggers the auto-deploy. That single deploy picks up both the new code and the new environment variable.

  • Confirm the variable appears in your Environment Variables list with the key GEMINI_API_KEY.

Don't see the Environment section?

Make sure you clicked into your specific service first. The Environment option only appears in the left pane after selecting a service.

help me find the Environment section in my Render dashboard

Push your code changes to Git

Render.com auto-deploys whenever you push to your connected Git branch. Pushing your new files triggers a fresh build that includes the Gemini SDK and all your new classes.

  • Return to your terminal in IntelliJ IDEA.
  • Stage all your changes by running this command:
git add .
  • Commit the changes by running:
git commit -m "Add Gemini vision transaction extraction endpoint"
  • Push to your remote repository by running:
git push
  • Switch back to the Render Dashboard. You should see a new deploy starting automatically on your service's Events page.
  • Wait for the deploy to complete. The status changes to Live when it finishes.

How long does the deploy take?

Maven builds on Render typically take 2-5 minutes. The first deploy after adding the Gemini SDK may be slightly longer as Maven downloads the new dependency.

Deploy failed?

Check the deploy logs on the Events page. The most common causes are a missing GEMINI_API_KEY (the app throws IllegalStateException at startup) or a typo in pom.xml causing a build failure.

help me debug my Render.com deploy failure

Test the live endpoint

Your feature is live. Time to confirm the production endpoint works the same way it did locally.

  • Find a broker or trading app screenshot on your phone or computer. Any image showing stock transactions (stock names, prices, and quantities) works.
  • Send a test request to your live endpoint by running this command:
curl -X POST https://[[RENDER_URL="your-service-name.onrender.com"]]/api/extract-transactions \
  -F "image=@screenshot.png"

What does this command do?

This curl command sends a POST request with your image file to the live /api/extract-transactions endpoint. Replace screenshot.png with the actual path to your broker screenshot file.

You can also use Postman if you prefer a graphical interface. Set the method to POST, enter your Render URL, and attach the image under the Body tab as form-data with key image.

  • Verify the response contains a JSON object with a transactions array. Each transaction should have stockName, stockPrice, quantity, and transactionType fields matching what you see in your screenshot.

Getting a 500 error or empty response?

Double-check that your GEMINI_API_KEY value in Render matches the one you tested locally. Also confirm your image file path is correct in the curl command.

help me troubleshoot my live transaction extraction endpoint

Secret mission

Add Confidence Scoring and Error Handling

Your extraction endpoint works, but what happens when someone uploads a blurry screenshot or a photo of their lunch? Right now, the model might return garbage data. In this mission, you'll add confidence scoring to each extraction and teach the model to gracefully handle images it can't parse.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project uses the Gemini free tier and runs on your existing Render.com deployment, so there are no ongoing costs.

Resources you used:

  • Google AI Studio API key.
  • GEMINI_API_KEY environment variable on Render.com.
  • New Java source files added to your Spring Boot project (TransactionExtractionResponse.java, GeminiService.java, TransactionExtractorController.java).
  • google-genai Maven dependency added to pom.xml.

Keep everything running

No action needed. Choose this if you plan to keep using the AI extraction feature in your portfolio tracker app.

  • Your Gemini API key remains active on the free tier with no charges.
  • Your Render.com service continues to use the GEMINI_API_KEY environment variable for the extraction endpoint.
  • Your code stays deployed and functional.

Pause - I'll come back to this later

Remove the API key from Render.com so the endpoint stops working, but keep your code and API key intact for later.

  • In your Render.com dashboard, open your service's Environment settings.
  • Delete the GEMINI_API_KEY environment variable and save.
  • Keep your code as-is in your Git repository. When you want to resume, re-add the environment variable.

Delete - I don't want to use this again

Remove all project resources and revert your app to its previous state.

  • Delete your API key from Google AI Studio by visiting the API Keys page at https://aistudio.google.com/apikey, locating your key, and deleting it.
  • In your Render.com dashboard, open your service's Environment settings and delete the GEMINI_API_KEY environment variable.
  • Remove the new source files from your project by running:
rm src/main/java/com/yourapp/dto/TransactionExtractionResponse.java
rm src/main/java/com/yourapp/service/GeminiService.java
rm src/main/java/com/yourapp/controller/TransactionExtractorController.java
  • Open pom.xml and remove the google-genai dependency block:
<dependency>
    <groupId>com.google.genai</groupId>
    <artifactId>google-genai</artifactId>
    <version>1.55.0</version>
</dependency>
  • Commit and push your changes. Render will auto-deploy the reverted app.

That's a Wrap!

That's a Wrap!

Nice work! You've just built an AI-powered screenshot parser that turns broker images into structured portfolio data in seconds.

You've learned how to:

  • Built a multimodal AI integration using Gemini 3.5 Flash vision capabilities to extract data from images.
  • Created a structured data extraction pipeline that converts broker screenshots into JSON using a defined schema.
  • Integrated the Google GenAI Java SDK into an existing Spring Boot application and deployed it on Render.com with secure environment variable management.
  • Secret Mission: Added confidence scoring and graceful error handling for production readiness.

Ready to quiz yourself?