Predictive Object Tracking with IMU

Build a real-time tracker fusing ESP32 camera and gyroscope data with YOLOv8.

Introduction

30 Second Summary

Security cameras, self-driving cars, and military drones all need to track moving objects without losing sight of them. When the camera platform itself is moving, the problem gets exponentially harder because every object appears to jump between frames.

In this project, you will build a real-time perception system that streams camera and gyroscope data from an ESP32 microcontroller to your laptop, runs GPU-accelerated object detection, and uses a Kalman filter with inertial motion compensation to maintain persistent object tracking through camera motion.

What You'll Build

You can shake your camera platform, watch objects keep their identities on screen, and show a live terminal readout proving the sensor fusion works. This is the exact firmware-to-ML-inference bridge pattern used in defense ISR, autonomous navigation, and robotics perception systems.

By the end of this project, you'll have:

  • A live annotated video window showing bounding boxes with persistent track IDs and predicted trajectories overlaid on your ESP32 camera stream.
  • A real-time terminal dashboard printing detection count, active tracks, and angular rate so you can watch the system react as you move the camera.
  • A quantitative comparison where you shake the platform and measure how many tracks survive with IMU compensation versus without, proving the fusion works with numbers.
  • Secret Mission: Add track confidence scoring with temporal decay, simulating how real multi-target trackers handle occlusion in operational ISR systems.

Are there any prerequisites?

This project requires a Deneyap Kart 1A with a Deneyap Camera (OV2640), Arduino IDE configured with Deneyap board support, Python 3.11 installed, and an NVIDIA GPU laptop (RTX 3070Ti or similar) with drivers ready.

You should be comfortable with basic Python scripting and flashing firmware via Arduino IDE. Experience with numerical methods or state estimation is helpful but not required.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Set Up the Python Perception Environment

Your perception pipeline runs entirely on your laptop's GPU. Before any firmware touches the ESP32, you need GPU-accelerated inference working locally.

In this step, you will create an isolated Python environment and install PyTorch with CUDA support, Ultralytics YOLOv8, and OpenCV. Then you will verify that your RTX 3070Ti is accessible and that real-time object detection works at full speed.

In this step, get ready to:
  • Create a virtual environment and install PyTorch with CUDA, Ultralytics, and OpenCV.
  • Verify CUDA is accessible and run YOLOv8n inference on a test image.
  • Confirm real-time detection at 30+ FPS on your webcam stream.
Create the project environment and install packages

A virtual environment keeps your ML packages isolated from your system Python. This prevents version conflicts as you add heavy dependencies like PyTorch.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder, move into it, and set up a virtual environment by running these commands:
mkdir tracker-project
cd tracker-project
python -m venv venv
  • Confirm the folder and virtual environment were created by running:
ls

You should see a venv folder listed in the output.

  • Activate the virtual environment by running the command for your operating system:

macOS

source venv/bin/activate

Windows

venv\Scripts\activate

Linux

source venv/bin/activate

Your terminal prompt now shows (venv) at the beginning. This confirms you are working inside the isolated environment.

Don't see (venv) in your prompt?

Make sure you ran the activation command for your operating system. On Windows, use the backslash path. On macOS and Linux, use source.

If you see an execution policy error on Windows PowerShell, try running your terminal as Administrator or switch to Command Prompt.

Help me fix my virtual environment activation.

  • Install PyTorch with CUDA 11.8 support by running this command:
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

What does this command do?

This installs PyTorch 2.7.0 along with torchvision and torchaudio from a special index that includes CUDA 11.8 GPU binaries. The download is large (around 2-3 GB) because it includes the GPU compute libraries.

The install takes a few minutes depending on your connection speed. You will see a progress bar for each package.

  • Once PyTorch finishes, install Ultralytics (which includes OpenCV and NumPy as dependencies) by running:
pip install ultralytics

You should see a success message listing all installed packages including ultralytics, opencv-python, and numpy.

Seeing a dependency conflict?

If pip reports a version conflict, make sure your (venv) prefix is visible in the prompt. Installing without the virtual environment active can conflict with system packages.

Help me resolve pip dependency conflicts during my PyTorch and Ultralytics install.

Verify CUDA and run YOLOv8n on a test image

The packages are installed, but you need proof that your GPU is actually accessible. A single command confirms whether PyTorch can see your RTX 3070Ti through the CUDA driver.

  • Before you run the next command, predict: what should it print if your GPU is properly connected?
  • Check CUDA availability by running:
python -c "import torch; print(torch.cuda.is_available())"

It prints True. This confirms PyTorch can access your NVIDIA GPU through CUDA.

Seeing False instead of True?

This means PyTorch cannot find your CUDA drivers. Check that your NVIDIA drivers are up to date by running nvidia-smi in a separate terminal. If that command fails, you need to install or update your NVIDIA drivers first.

Also confirm you installed the CUDA 11.8 version of PyTorch (the command with --index-url), not the CPU-only version.

Help me get torch.cuda.is_available() to return True on my RTX 3070Ti.

Now run a real detection. YOLOv8n will auto-download its weights on first use and run inference on a sample image from the Ultralytics servers.

  • Run YOLOv8n on a sample image by running this command:
python -c "from ultralytics import YOLO; model = YOLO('yolov8n.pt'); results = model('https://ultralytics.com/images/bus.jpg'); print(f'Detected {len(results[0].boxes)} objects')"

What happens here?

The first run downloads the yolov8n.pt model weights (about 6 MB). It then downloads the sample bus image and runs inference on your GPU. The output tells you how many objects were detected in the scene.

You should see output mentioning the inference speed (in milliseconds) and a detection count of around 4-6 objects (buses, people).

Seeing a download error?

If the image URL fails to download, check your internet connection. The model weights and test image both require an active connection on first run.

Help me troubleshoot YOLOv8n model download issues.

Confirm real-time detection on your webcam

A single image proves the model works. Real-time detection on a live stream proves the GPU can sustain inference at the frame rates you need for the ESP32 pipeline.

  • Run the real-time webcam detection test by running this command:
python -c "
import cv2
from ultralytics import YOLO
import time

model = YOLO('yolov8n.pt')
cap = cv2.VideoCapture(0)

print('Press q to quit. Watching FPS...')
frame_count = 0
start = time.time()

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break
    results = model(frame, verbose=False)
    annotated = results[0].plot()
    frame_count += 1
    elapsed = time.time() - start
    fps = frame_count / elapsed if elapsed > 0 else 0
    cv2.putText(annotated, f'FPS: {fps:.1f}', (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
    cv2.imshow('YOLOv8n Test', annotated)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()
print(f'Average FPS: {fps:.1f}')
"

What does this script do?

  • Opens your laptop webcam with OpenCV.
  • Runs YOLOv8n on every frame using your GPU.
  • Draws bounding boxes and an FPS counter on the live feed.
  • Press q to quit and print the average FPS.

A window opens showing your webcam feed with colored bounding boxes around detected objects. The green FPS counter in the top-left corner should read 30+ FPS on your RTX 3070Ti.

  • Wave your hand or hold up an object in front of the camera to see bounding boxes appear and follow the object in real-time.
  • Press q to close the window once you have confirmed 30+ FPS.

Seeing low FPS or a black window?

If the FPS is below 30, confirm that your NVIDIA GPU is being used by running nvidia-smi in another terminal while the script runs. You should see a Python process using GPU memory.

If the window is black, your webcam may be in use by another application. Close other apps that might be using the camera (video calls, other camera apps) and try again.

Help me get real-time YOLOv8 webcam detection working on my GPU.

Your perception environment is ready. PyTorch sees your GPU, YOLOv8n detects objects in milliseconds, and real-time inference is running at full speed. Next up, you will flash the ESP32 as a dual-stream sensor node that feeds this pipeline with live camera frames and gyroscope data over WiFi.

Flash the Deneyap Kart as a Dual-Stream Sensor Node

Your Python perception environment is ready and verified. Now you need sensor data to feed it.

The Deneyap Kart 1A is your edge sensor platform. It needs to serve two simultaneous data streams over WiFi: MJPEG camera frames for visual detection, and JSON-formatted gyroscope angular rates for motion compensation. Without both streams running concurrently, you have no sensor data to fuse.

In this step, get ready to:
  • Flash the CameraWebServer firmware configured for the Deneyap Kart 1A camera pin mapping.
  • Add a second HTTP endpoint that serves live gyroscope data as JSON.
  • Verify both endpoints serve data simultaneously without blocking.
Load and configure the CameraWebServer example

The ESP32 Arduino core ships with a CameraWebServer example that handles MJPEG streaming. You need to configure it with the correct camera pin definitions for the Deneyap Kart 1A.

  • Switch to the Arduino IDE window where your Deneyap Kart 1A is connected.
  • Open the CameraWebServer example by clicking File in the menu bar.
  • Select Examples, then ESP32, then Camera, then CameraWebServer.
  • A new sketch window opens with the example code. Replace the entire contents with the firmware sections below.
  • Start the sketch with the library includes, camera pin definitions, and WiFi credentials by pasting this code:
#include "esp_camera.h"
#include <WiFi.h>
#include "esp_http_server.h"

// Deneyap Kart 1A camera pin definitions
#define PWDN_GPIO_NUM    -1
#define RESET_GPIO_NUM   -1
#define XCLK_GPIO_NUM    16
#define SIOD_GPIO_NUM     4
#define SIOC_GPIO_NUM     5
#define Y9_GPIO_NUM      15
#define Y8_GPIO_NUM      17
#define Y7_GPIO_NUM      38
#define Y6_GPIO_NUM      40
#define Y5_GPIO_NUM      42
#define Y4_GPIO_NUM       1
#define Y3_GPIO_NUM       2
#define Y2_GPIO_NUM      41
#define VSYNC_GPIO_NUM    6
#define HREF_GPIO_NUM     7
#define PCLK_GPIO_NUM    39

// WiFi credentials
const char* ssid = "your-wifi-ssid-here";
const char* password = "your-wifi-password-here";

// Server handles
httpd_handle_t stream_httpd = NULL;

What are these pin definitions?

Each #define maps a camera signal name (like XCLK, Y2-Y9) to the specific GPIO pin number on the Deneyap Kart 1A where that signal is physically routed.

These numbers come from the official Deneyap Kart 1A v2 board variant file in the ESP32 Arduino core. Using wrong pin numbers means the camera cannot communicate with the ESP32.

  • Below the pin definitions, add the MJPEG stream handler function:
// MJPEG stream handler
static esp_err_t stream_handler(httpd_req_t *req) {
  camera_fb_t *fb = NULL;
  esp_err_t res = ESP_OK;
  char part_buf[64];

  // Set multipart response type for continuous frames
  res = httpd_resp_set_type(req,
    "multipart/x-mixed-replace;boundary=frame");
  if (res != ESP_OK) return res;

  // Continuously capture and send JPEG frames
  while (true) {
    fb = esp_camera_fb_get();
    if (!fb) { res = ESP_FAIL; break; }
    size_t hlen = snprintf(part_buf, 64,
      "--frame\r\nContent-Type: image/jpeg\r\n"
      "Content-Length: %u\r\n\r\n", fb->len);
    res = httpd_resp_send_chunk(req, part_buf, hlen);
    if (res == ESP_OK)
      res = httpd_resp_send_chunk(req,
        (const char *)fb->buf, fb->len);
    if (res == ESP_OK)
      res = httpd_resp_send_chunk(req, "\r\n", 2);
    esp_camera_fb_return(fb);
    if (res != ESP_OK) break;
  }
  return res;
}

What does the stream handler do?

The stream_handler function sends a continuous multipart/x-mixed-replace HTTP response. Each part is a JPEG frame captured from the camera.

This is the standard MJPEG-over-HTTP pattern. The browser keeps the connection open and renders each new JPEG frame as it arrives, creating a live video feed.

  • Add the startStreamServer() function and an empty loop():
void startStreamServer() {
  httpd_config_t config = HTTPD_DEFAULT_CONFIG();
  config.server_port = 81;

  httpd_uri_t stream_uri = {
    .uri = "/stream",
    .method = HTTP_GET,
    .handler = stream_handler,
    .user_ctx = NULL
  };

  if (httpd_start(&stream_httpd, &config) == ESP_OK) {
    httpd_register_uri_handler(stream_httpd, &stream_uri);
    Serial.println("Stream server started on port 81");
  }
}

void loop() {
  delay(10000);
}

Why port 81?

The stream server runs on port 81 so it does not conflict with the main HTTP server on port 80. This is the same pattern the official ESP32 CameraWebServer example uses. The separation means a long-lived streaming connection on port 81 cannot block quick request-response endpoints on port 80.

  • Now add the setup() function. Start with the camera hardware configuration:
void setup() {
  Serial.begin(115200);

  // Camera configuration struct
  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.производ_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sccb_sda = SIOD_GPIO_NUM;
  config.pin_sccb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  config.frame_size = FRAMESIZE_QVGA;
  config.jpeg_quality = 10;
  config.fb_count = 2;
  config.grab_mode = CAMERA_GRAB_LATEST;
  • Directly below the camera config (still inside setup()), add the initialization, WiFi connection, and server start:
  // Initialize camera
  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    Serial.printf("Camera init failed: 0x%x\n", err);
    return;
  }
  Serial.println("Camera initialized");

  // Connect to WiFi
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println();
  Serial.print("WiFi connected. IP: ");
  Serial.println(WiFi.localIP());

  // Start the MJPEG stream server
  startStreamServer();
}

Why QVGA resolution?

QVGA (320x240) gives the ESP32 stable 20-30 FPS MJPEG streaming over WiFi while leaving CPU headroom for the IMU endpoint. Higher resolutions (VGA, SVGA) drop to 10-12 FPS and introduce latency that hurts real-time tracking.

The fb_count = 2 with CAMERA_GRAB_LATEST ensures the host always gets the most recent frame, not a stale buffered one.

  • Make sure your Deneyap Kart 1A is selected. Click Tools in the menu bar.
  • Under Board, confirm Deneyap Kart 1A is selected.
  • Click the Upload button (right arrow icon) to flash the firmware.
  • Wait for the upload to complete. You will see "Done uploading" in the Arduino IDE output.
  • Open the Serial Monitor by clicking Tools in the menu bar, then Serial Monitor.
  • Set the baud rate to 115200 in the dropdown at the bottom right of the Serial Monitor.
  • Press the reset button on the Deneyap Kart 1A. You should see WiFi connection dots, then an IP address printed.

Note the IP address the Serial Monitor prints. This is your ESP32's address on your local network.

  • Open a browser and navigate to the stream URL by entering http:// followed by your ESP32 IP address followed by :81/stream.

You should see a live MJPEG camera feed from the Deneyap Camera in your browser.

Not seeing the camera stream?

  • Check the Serial Monitor for error messages. If you see a camera init failure, verify the pin definitions match the GPIO numbers exactly.
  • Make sure your laptop and the ESP32 are on the same WiFi network.
  • If the page loads but shows no image, try refreshing. Some browsers need a moment to pick up the multipart stream.
  • help me troubleshoot my Deneyap Kart camera stream
Add the IMU gyroscope endpoint

The camera stream is working. Now you need the second data stream: gyroscope angular rates from the onboard LSM6DSM sensor. This endpoint returns JSON that the Python host reads in parallel with the camera frames.

  • Return to the Arduino IDE sketch.
  • Add the IMU library include at the top of the file, below the existing includes:
#include "lsm6dsm.h"

// Create IMU instance
LSM6DSM IMU;

What is lsm6dsm.h?

The lsm6dsm.h library is bundled with the Deneyap Arduino core. It provides the LSM6DSM class that communicates with the onboard gyroscope over I2C (address 0x6B). No external wiring needed.

  • Add the IMU HTTP handler function and the main server handle. Place these above setup():
// Main HTTP server for IMU endpoint (port 80)
httpd_handle_t main_httpd = NULL;

// IMU endpoint handler - returns gyro as JSON
static esp_err_t imu_handler(httpd_req_t *req) {
  // Read gyroscope values in degrees per second
  float gx = IMU.readFloatGyroX();
  float gy = IMU.readFloatGyroY();
  float gz = IMU.readFloatGyroZ();

  // Format as JSON
  char json_buf[128];
  snprintf(json_buf, sizeof(json_buf),
    "{\"gx\":%.2f,\"gy\":%.2f,\"gz\":%.2f}",
    gx, gy, gz);

  httpd_resp_set_type(req, "application/json");
  return httpd_resp_send(req, json_buf, strlen(json_buf));
}

What does the IMU handler return?

Each request to /imu reads the LSM6DSM gyroscope and returns a JSON object with three fields: gx, gy, and gz representing angular rates in degrees per second.

The default gyro range is 250 dps, which is more than enough to detect typical hand movements when holding the sensor platform.

  • Add the function to start the main HTTP server. Place it below startStreamServer():
void startMainServer() {
  httpd_config_t config = HTTPD_DEFAULT_CONFIG();
  config.server_port = 80;

  httpd_uri_t imu_uri = {
    .uri = "/imu",
    .method = HTTP_GET,
    .handler = imu_handler,
    .user_ctx = NULL
  };

  if (httpd_start(&main_httpd, &config) == ESP_OK) {
    httpd_register_uri_handler(main_httpd, &imu_uri);
    Serial.println("IMU server started on port 80");
  }
}
  • In setup(), add IMU initialization after the camera init block. Find the line Serial.println("Camera initialized"); and add below it:
  // Initialize IMU
  IMU.begin();
  Serial.println("IMU initialized");
  • Add startMainServer(); as the last line in setup(), after startStreamServer();.
  • Click Upload to flash the updated firmware.
  • Open the Serial Monitor and press reset on the board. You should see lines confirming camera init, IMU init, WiFi connection, stream server on port 81, and IMU server on port 80.
  • In your browser, navigate to http:// followed by your ESP32 IP address followed by /imu.

You should see a JSON response like {"gx":0.12,"gy":-0.34,"gz":0.56}. The values change each time you refresh.

Getting zeros or no response from /imu?

  • Make sure IMU.begin() is called before the server starts. Check the Serial Monitor for "IMU initialized".
  • Try rotating the board while refreshing /imu in the browser. The gyroscope only registers non-zero values during rotation.
  • help me debug my LSM6DSM gyroscope not returning data on the Deneyap Kart
Verify both endpoints work simultaneously

The real test is that both endpoints serve data at the same time without one blocking the other. The camera stream is a long-lived connection. The IMU endpoint is a quick request-response. They run on separate HTTP servers (port 81 and port 80) so they do not interfere.

  • Open the camera stream in one browser tab: http:// followed by your ESP32 IP address followed by :81/stream.
  • Open a second browser tab and navigate to http:// followed by your ESP32 IP address followed by /imu.
  • Refresh the /imu tab several times while the camera stream is running in the other tab.

Before you check the results, what do you expect to see? Will the camera stream freeze when you hit the IMU endpoint, or will both work independently?

Both should work without interference. The camera keeps streaming in one tab while the IMU returns fresh gyroscope JSON on every refresh in the other. Pick up the board and rotate it while refreshing the IMU tab. The gz value jumps noticeably during yaw rotation.

Camera stream freezes when you hit /imu?

  • This should not happen with the two-server architecture (port 80 and port 81). If it does, check that you have two separate httpd_handle_t variables and two separate httpd_start() calls with different ports.
  • help me fix my dual HTTP server setup on ESP32 where the stream blocks the IMU endpoint

✔️ Awesome, I've got everything!

Great. Make sure you have noted your ESP32's IP address. You will need it in the next step when your Python script connects to both endpoints.

ⓧ I'd like to double check the full code

Your complete sketch should contain these sections in order:

  • Library includes: esp_camera.h, WiFi.h, esp_http_server.h, lsm6dsm.h.
  • Pin definitions (PWDN through PCLK GPIO numbers).
  • WiFi credentials, LSM6DSM IMU instance, and server handle variables.
  • The stream_handler function for MJPEG on port 81.
  • The imu_handler function returning JSON with gx, gy, gz on port 80.
  • The startStreamServer() and startMainServer() functions.
  • The setup() function: camera init, IMU init, WiFi connect, start both servers.
  • An empty loop() with a delay.

Scroll up and verify each section is present in your sketch.

Your Deneyap Kart 1A is now a dual-stream sensor node. It serves live camera frames and real-time gyroscope telemetry simultaneously over WiFi. Next, you will pull those streams into Python and run YOLO object detection on the camera feed.

Build the Host-Side Detection Pipeline

Your ESP32 is streaming camera frames and gyroscope data over WiFi. Now you need a Python script on your laptop that pulls those frames, runs object detection on them, and shows you the results in real-time.

This step builds the baby version of the final system. Detection only, no tracking. You will see detections flicker wildly between frames with no identity persistence, no IDs, no continuity. That visible failure is the motivation for the Kalman filter in the next step.

In this step, get ready to:
  • Connect to the ESP32 MJPEG stream and parse frames with Python.
  • Run YOLOv8n inference on each frame with GPU acceleration.
  • Observe the stateless detection problem that motivates tracking.
Write the stream reader and model setup

OpenCV's VideoCapture cannot reliably handle the ESP32's chunked transfer encoding. The workaround is to read the raw byte stream with urllib and manually locate JPEG frame boundaries.

  • In VS Code, create a new file called detect.py in your project folder.
  • Add the imports, configuration, and stream-reading function by pasting this code:
import urllib.request
import cv2
import numpy as np
import time
from ultralytics import YOLO

# Configuration - replace with your ESP32's IP address
STREAM_URL = "http://[[ESP32_IP="your ESP32 IP address"]]:81/stream"

# Load YOLOv8n with CUDA acceleration
model = YOLO("yolov8n.pt")

# Byte buffer for MJPEG stream parsing
bytes_buffer = b''

def get_frame(stream):
    """Read bytes until a complete JPEG frame is found."""
    global bytes_buffer
    while True:
        bytes_buffer += stream.read(4096)
        # Find JPEG start (SOI) and end (EOI) markers
        start = bytes_buffer.find(b'\xff\xd8')
        end = bytes_buffer.find(b'\xff\xd9')
        if start != -1 and end != -1:
            jpg = bytes_buffer[start:end + 2]
            bytes_buffer = bytes_buffer[end + 2:]
            return jpg

What does this code do?

The ESP32 streams JPEG frames wrapped in a multipart/x-mixed-replace HTTP response. Each JPEG image starts with the bytes 0xFF 0xD8 (Start of Image) and ends with 0xFF 0xD9 (End of Image).

The get_frame function reads raw bytes from the stream, accumulates them in a buffer, and extracts complete JPEG frames by scanning for those markers. This is more reliable than cv2.VideoCapture for the ESP32's chunked HTTP transfer.

Add the main detection loop

Now you need the loop that connects to the stream, decodes frames, runs YOLO, draws bounding boxes, and displays everything in a live window.

  • Below the get_frame function in detect.py, add the main detection loop:
# Open the ESP32 MJPEG stream
stream = urllib.request.urlopen(STREAM_URL)
print("Stream connected. Press 'q' to quit.")

while True:
    t_start = time.time()
    # Get next JPEG frame from the stream
    jpg_bytes = get_frame(stream)
    frame = cv2.imdecode(
        np.frombuffer(jpg_bytes, dtype=np.uint8), cv2.IMREAD_COLOR)
    if frame is None:
        continue
    # Run YOLOv8n inference on GPU
    results = model(frame, device="cuda", verbose=False)
    # Annotate detections on the frame
    detection_count = 0
    for result in results:
        detection_count = len(result.boxes)
        for box in result.boxes:
            x1, y1, x2, y2 = map(int, box.xyxy[0])
            label = f"{model.names[int(box.cls[0])]} {float(box.conf[0]):.2f}"
            cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
            cv2.putText(frame, label, (x1, y1 - 10),
                        cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
    # Print FPS and detection count
    fps = 1.0 / (time.time() - t_start + 1e-9)
    print(f"FPS: {fps:.1f} | Detections: {detection_count}")
    # Display annotated frame
    cv2.imshow("ESP32 YOLOv8 Detection", frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
stream.close()
cv2.destroyAllWindows()

What does this code do?

  • Opens the ESP32 stream with urllib.request.urlopen and reads frames using the get_frame helper.
  • Decodes each raw JPEG into a NumPy array with cv2.imdecode.
  • Runs YOLOv8n inference on the GPU by passing device="cuda".
  • Iterates over result.boxes to draw a green rectangle and class label for each detection.
  • Prints per-frame FPS and detection count to the terminal so you can monitor performance.
  • Save detect.py.

✔️ Awesome, I've got everything!

Great. Double check you saved the file, then move on to running it.

ⓧ I'd like to double check the full code

Your detect.py should contain these two sections in order:

  • The imports, STREAM_URL config, model load, bytes_buffer declaration, and the get_frame function (from the first code block above).
  • The main while True detection loop with stream open, decode, YOLO inference, annotation, FPS print, and display (from the second code block above).

Scroll up and confirm each section is present, in order, with no extra code between them.

Observe the stateless detection problem

The script is complete. Time to run it and see what raw detection looks like without any tracking.

  • Make sure your ESP32 is powered on and connected to your WiFi network.
  • Run the detection script:
python detect.py

Before you look at the window, predict: when you move an object slowly across the camera's view, do you think it will keep the same label between frames, or get a fresh detection each time?

A live window opens showing your ESP32 camera feed with green bounding boxes flickering around detected objects. Your terminal prints FPS and detection count each frame.

  • Place an object (a cup, a phone, your hand) in the camera's view. Watch the bounding box appear.
  • Move it slowly across the frame. Notice: there is no ID, no continuity. Every single frame gets a fresh detection with no memory of the previous frame.
  • Now shake the ESP32 camera platform. Watch the boxes scatter chaotically. Detections appear and vanish with no coherence.

This is the stateless detection problem. The detector sees each frame in isolation. An object that is clearly the same thing across frames gets a brand new bounding box each time with zero identity persistence.

Why do detections flicker?

YOLOv8n processes each frame independently. It has no concept of "the same object I saw last frame." Small changes in position, lighting, or angle cause the bounding box to jump, resize, or disappear entirely for a frame. Without a state estimator to bridge frames, you get raw, memoryless detection.

The Kalman filter you build in the next step solves this by predicting where each object will be and matching new detections to existing tracks.

Not seeing a video window?

  • Check that your ESP32's IP address in STREAM_URL matches what the serial monitor printed when the board booted. The stream URL must include port 81 (not 80).
  • If you see a connection timeout, make sure your laptop is on the same WiFi network as the ESP32.
  • If the window opens but is blank, verify the ESP32 camera stream works in your browser first by visiting the stream URL directly.

Help me debug why my ESP32 MJPEG stream is not connecting in Python.

  • Press q in the video window to quit the script when you are done observing.

You now have a working detection pipeline pulling frames from your ESP32 and running GPU-accelerated YOLOv8 inference in real-time. The flickering you just witnessed is the exact gap that a state estimator fills. Next up, you will build a Kalman filter tracker that gives each detected object a persistent identity across frames.

Add the Kalman Filter Tracker

In the previous step, you watched detections flicker chaotically across the ESP32 camera feed. Every frame produced a fresh set of bounding boxes with no connection to the previous frame.

A Kalman filter solves this by giving detections memory. It maintains a state estimate for each tracked object, predicts where that object will appear next frame, and corrects its estimate when a matching detection arrives. Combined with Hungarian assignment for optimal detection-to-track matching, this turns your memoryless detector into a persistent multi-target tracker.

In this step, get ready to:
  • Implement a KalmanTrack class with a constant-velocity state model.
  • Build a TrackManager that uses Hungarian matching to associate detections with existing tracks.
  • Draw persistent track IDs and trajectory tails on the annotated frame.
Implement the KalmanTrack class

Each tracked object needs its own state estimator. The KalmanTrack class holds a 4-element state vector representing position and velocity, plus a transition matrix that advances position by velocity each frame.

  • Install SciPy (needed for Hungarian matching) by running:
pip install scipy

Why install SciPy separately?

SciPy ships as a dependency of Ultralytics, so it may already be present. Running the install command ensures the latest version is available and confirms the import path works.

  • Create a new file called tracker.py in the same directory as detect.py and add these imports at the top:
import numpy as np
import cv2
import urllib.request
import time
from scipy.optimize import linear_sum_assignment
from ultralytics import YOLO
  • Below the imports, add the KalmanTrack class:
class KalmanTrack:
    _next_id = 0

    def __init__(self, cx, cy):
        # State vector: [x, y, velocity_x, velocity_y]
        self.state = np.array([cx, cy, 0.0, 0.0])
        # Constant-velocity transition matrix
        self.F = np.array([[1,0,1,0],[0,1,0,1],[0,0,1,0],[0,0,0,1]], dtype=np.float64)
        self.id = KalmanTrack._next_id
        KalmanTrack._next_id += 1
        self.age = 0  # Frames since last detection match
        self.history = [(int(cx), int(cy))]

    def predict(self):
        # Advance position by velocity
        self.state = self.F @ self.state
        self.age += 1
        return self.state[:2]

    def update(self, cx, cy):
        # Snap to measured centroid, reset miss counter
        self.state[0] = cx
        self.state[1] = cy
        self.age = 0
        self.history.append((int(cx), int(cy)))
        if len(self.history) > 20:
            self.history.pop(0)

What does this code do?

The state vector [x, y, dx, dy] tracks both position and velocity for each object.

The transition matrix F applies the constant-velocity assumption: each predict step adds velocity to position. The update method snaps the state back to the actual measured centroid when a detection matches, resetting the miss counter.

The history list stores recent centroid positions for drawing trajectory tails later.

  • Save tracker.py.
  • Verify the class works by running this quick test:
python -c "from tracker import KalmanTrack; t = KalmanTrack(100, 200); print(t.predict()); print(t.state)"

You should see the predicted position [100. 200.] and the state vector with velocities still at zero.

Seeing an ImportError?

Make sure your virtual environment is active before running the test. Also confirm tracker.py is saved in the directory you are running from.

help me debug an ImportError when importing my tracker module

Build the TrackManager

The TrackManager orchestrates the full tracking cycle each frame: predict all existing tracks forward, match them to new detections using the Hungarian algorithm, update matched tracks, spawn new ones, and delete stale ones.

  • In tracker.py, below the KalmanTrack class, add the first part of the TrackManager:
class TrackManager:
    def __init__(self, max_age=5, match_threshold=100.0):
        self.tracks = []
        self.max_age = max_age
        self.match_threshold = match_threshold

    def predict_all(self):
        for track in self.tracks:
            track.predict()

    def match(self, detections):
        # Build Euclidean cost matrix between tracks and detections
        if len(self.tracks) == 0 or len(detections) == 0:
            return [], list(range(len(detections)))
        cost = np.zeros((len(self.tracks), len(detections)))
        for i, t in enumerate(self.tracks):
            for j, d in enumerate(detections):
                cost[i, j] = np.linalg.norm(t.state[:2] - d)
        # Hungarian optimal assignment
        row_idx, col_idx = linear_sum_assignment(cost)
        matched, unmatched_dets = [], set(range(len(detections)))
        for i, j in zip(row_idx, col_idx):
            if cost[i, j] < self.match_threshold:
                matched.append((i, j))
                unmatched_dets.discard(j)
        return matched, list(unmatched_dets)

What does the match method do?

It builds a cost matrix where each cell holds the Euclidean distance between a track's predicted centroid and a detection's centroid. Then linear_sum_assignment from SciPy finds the globally optimal assignment that minimizes total distance.

Pairs with cost above match_threshold (100 pixels) are rejected. This prevents a far-away detection from being incorrectly matched to an existing track.

  • Still inside the TrackManager class, add the lifecycle methods below the match method:
    def update_matched(self, matched, detections):
        # Correct matched tracks with detection centroids
        for i, j in matched:
            self.tracks[i].update(detections[j][0], detections[j][1])

    def create_new(self, unmatched_dets, detections):
        # Spawn new tracks for unmatched detections
        for j in unmatched_dets:
            self.tracks.append(KalmanTrack(detections[j][0], detections[j][1]))

    def delete_stale(self):
        # Remove tracks not seen for max_age frames
        self.tracks = [t for t in self.tracks if t.age <= self.max_age]

What do the lifecycle methods do?

These three methods complete the per-frame cycle. update_matched corrects each matched track with the fresh detection centroid. create_new spawns a new track for any detection that had no matching track. delete_stale removes tracks that went unmatched for 5 consecutive frames.

  • Save tracker.py.
  • Verify the TrackManager imports cleanly by running:
python -c "from tracker import TrackManager; m = TrackManager(); print('TrackManager ready, tracks:', len(m.tracks))"

You should see TrackManager ready, tracks: 0 confirming both classes load without errors.

Seeing an IndentationError?

The lifecycle methods (update_matched, create_new, delete_stale) must be indented at the same level as predict_all and match. They are methods of the TrackManager class, not standalone functions.

help me fix indentation in my TrackManager class

Wire up the main tracking loop and verify

With both classes in place, you need the same MJPEG stream reader and YOLO inference loop from detect.py, but now routing detections through the TrackManager and drawing persistent IDs on each frame.

  • Below the TrackManager class in tracker.py, add the stream reader and initialization code:
def read_stream(url):
    # Parse MJPEG stream manually (ESP32 chunked transfer)
    stream = urllib.request.urlopen(url)
    buf = b''
    while True:
        buf += stream.read(4096)
        start = buf.find(b'\xff\xd8')
        end = buf.find(b'\xff\xd9')
        if start != -1 and end != -1:
            jpg = buf[start:end+2]
            buf = buf[end+2:]
            frame = cv2.imdecode(
                np.frombuffer(jpg, dtype=np.uint8), cv2.IMREAD_COLOR)
            if frame is not None:
                yield frame

# Initialize model and tracker
model = YOLO("yolov8n.pt")
manager = TrackManager(max_age=5, match_threshold=100.0)
stream_url = "http://[[ESP32_IP="your ESP32 IP address"]]:81/stream"

What does this code do?

The read_stream generator uses manual JPEG boundary parsing (0xFFD8 start, 0xFFD9 end) because cv2.VideoCapture cannot handle the ESP32's chunked transfer encoding reliably.

The model and TrackManager are instantiated once before the loop begins. Replace the IP placeholder with the same address you used in detect.py.

  • Finally, add the main tracking loop below the initialization:
for frame in read_stream(stream_url):
    t0 = time.time()
    # Run YOLO detection
    results = model(frame, device='cuda', verbose=False)
    boxes = results[0].boxes.xywh.cpu().numpy()
    # Extract centroids
    detections = [(box[0], box[1]) for box in boxes]
    # Kalman predict -> match -> update cycle
    manager.predict_all()
    matched, unmatched = manager.match(detections)
    manager.update_matched(matched, detections)
    manager.create_new(unmatched, detections)
    manager.delete_stale()
    # Draw track IDs and trajectory tails
    for track in manager.tracks:
        cx, cy = int(track.state[0]), int(track.state[1])
        cv2.putText(frame, f"ID:{track.id}", (cx-10, cy-15),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
        for k in range(1, len(track.history)):
            cv2.line(frame, track.history[k-1], track.history[k],
                     (0, 255, 255), 2)
    # Display stats
    fps = 1.0 / (time.time() - t0 + 1e-6)
    cv2.putText(frame, f"FPS:{fps:.1f} Tracks:{len(manager.tracks)}",
                (10, 25), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255,255,255), 2)
    cv2.imshow("Tracker", frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cv2.destroyAllWindows()

What does the main loop do?

Each frame follows the predict-match-update cycle. After YOLO returns bounding boxes, centroids are extracted and fed through the TrackManager.

The drawing section renders each track's ID in green text and a yellow trajectory tail showing recent movement. FPS and active track count are overlaid in the top-left corner.

  • Save tracker.py.

Before you run this, what do you expect to see when you move an object slowly across the camera's view? Do you think the ID will stay the same or reset each frame like before?

  • Run the tracker by executing:
python tracker.py

A window titled "Tracker" appears showing the live ESP32 feed with green ID labels and yellow trajectory tails on detected objects. Move an object slowly across the frame and watch it keep the same ID.

Now shake the ESP32 camera platform quickly. Watch what happens to the track IDs.

The IDs reassign and trajectories break. That is the expected result. The Kalman predictor assumes the camera is stationary, so any platform motion looks like all objects jumping simultaneously, breaking every match.

Not seeing any tracks drawn?

Confirm your ESP32 is powered on and the stream URL matches the IP from your earlier step. Check the terminal for connection errors.

If you see detections but no IDs, verify that the TrackManager lifecycle methods are indented inside the class (not at module level).

help me debug why my tracker.py shows detections but no persistent track IDs

✔️ Awesome, I've got everything!

Great. Double check you have saved tracker.py and that you can see persistent IDs on slowly moving objects, with IDs breaking on camera shake.

ⓧ I'd like to double check the full code

Your tracker.py should contain these sections in order. Scroll up through the step and confirm each is present:

  • Imports (numpy, cv2, urllib.request, time, linear_sum_assignment, YOLO).
  • KalmanTrack class with __init__, predict, and update methods.
  • TrackManager class with predict_all, match, update_matched, create_new, and delete_stale methods.
  • The read_stream generator function with manual JPEG boundary parsing.
  • Model and TrackManager initialization.
  • Main loop: YOLO inference, predict-match-update cycle, draw IDs and tails, display with cv2.imshow.

You now have a working multi-target tracker that persists identities across frames. The gap is clear: the predictor assumes a stationary camera, so platform motion shatters every track. Next up, you will fuse the ESP32's gyroscope data directly into the Kalman prediction step to compensate for that rotation.

Fuse IMU Data for Motion Compensation

In the previous step you proved that a Kalman filter can give detections persistent identity across frames. But you also saw the fatal gap: shake the camera platform and every track shatters.

The Kalman predictor currently assumes the camera is stationary. It has no idea the platform moved, so it interprets all apparent motion as object motion. By reading the ESP32 gyroscope data in parallel and feeding angular velocity directly into the prediction step, you shift predicted positions by the expected pixel displacement from camera rotation. The system knows the platform moved, not the objects.

In this step, get ready to:
  • Add a background thread that polls the ESP32 IMU endpoint and stores angular rates in a thread-safe variable.
  • Convert gyroscope angular rate to pixel displacement and inject it into the Kalman prediction step.
  • Add a toggle key to compare track survival with and without IMU compensation.
Add a background thread to poll the IMU endpoint

Your ESP32 is already serving gyroscope data at its /imu endpoint. You need a separate thread that fetches this data continuously without blocking the main detection loop.

  • Open tracker.py in VS Code.
  • At the top of the file, add the following imports and IMU polling infrastructure:
import threading
import json
import time

# Thread-safe storage for latest IMU readings
imu_lock = threading.Lock()
imu_data = {"gx": 0.0, "gy": 0.0, "gz": 0.0, "timestamp": time.time()}

def poll_imu(esp32_ip):
    """Poll the ESP32 /imu endpoint at ~20Hz."""
    global imu_data
    url = f"http://{esp32_ip}/imu"
    while True:
        try:
            response = urllib.request.urlopen(url, timeout=0.5)
            data = json.loads(response.read().decode())
            with imu_lock:
                imu_data = {
                    "gx": data["gx"],
                    "gy": data["gy"],
                    "gz": data["gz"],
                    "timestamp": time.time()
                }
        except Exception:
            pass  # Skip failed reads, retry next cycle
        time.sleep(0.05)  # 50ms = ~20Hz polling rate

What does this code do?

The imu_lock is a threading.Lock that prevents the main loop and the background thread from reading/writing imu_data at the same time. Without it, you could read a half-updated dictionary.

The poll_imu function runs in an infinite loop, hitting the ESP32 /imu endpoint every 50ms. Each successful read overwrites the shared imu_data dict with fresh gyroscope values and a timestamp. Failed network reads are silently skipped so the thread never crashes.

Now you need to start this thread before the main loop begins. Find your main loop (the section where you initialize the TrackManager and begin reading frames), and add the thread launch just above it:

# Start IMU polling in a background daemon thread
imu_thread = threading.Thread(target=poll_imu, args=(ESP32_IP,), daemon=True)
imu_thread.start()
print("IMU polling thread started")

Why daemon=True?

A daemon thread exits automatically when the main program ends. Without this flag, closing the OpenCV window would leave the IMU thread running and the process hanging in your terminal.

  • Save tracker.py and run it briefly to confirm the thread connects:
python tracker.py

You should see IMU polling thread started printed in your terminal before the video window opens. The detection and tracking continues working as before.

Not seeing the IMU message?

Make sure the ESP32_IP variable matches the IP address your ESP32 printed to Serial Monitor when it connected to WiFi. The thread silently absorbs connection errors, so an incorrect IP will not crash the script but will produce no IMU data.

Confirm the ESP32 is powered on and connected to the same WiFi network as your laptop. Help me debug my IMU polling thread connection

Integrate IMU offset into the Kalman prediction

The OV2640 camera has a 66-degree horizontal field of view at 320 pixels wide. That means each degree of camera rotation shifts objects by approximately 4.85 pixels in the frame.

  • Below your poll_imu function, add the IMU-to-pixel conversion constants and helper:
# IMU-to-pixel conversion constants
FRAME_WIDTH = 320  # QVGA resolution
FOV_HORIZONTAL = 66.0  # OV2640 horizontal field of view in degrees
PIXELS_PER_DEGREE = FRAME_WIDTH / FOV_HORIZONTAL  # ~4.85 px/deg

def get_imu_pixel_offset(dt):
    """Convert gz angular rate to horizontal pixel displacement."""
    with imu_lock:
        gz = imu_data["gz"]
    # gz in deg/s * dt in seconds * pixels per degree
    dx_offset = gz * dt * PIXELS_PER_DEGREE
    return dx_offset

Where does this formula come from?

The LSM6DSM gyroscope reports angular velocity in degrees per second. Multiplying by dt (the time between frames) gives you degrees of rotation per frame. Multiplying by pixels-per-degree converts that rotation into the pixel shift objects would experience in the image plane.

At the default 250 dps range and 30 FPS, maximum displacement is about 40 pixels per frame. This comfortably fits within the Hungarian matching distance threshold.

Now you need to apply this offset inside your TrackManager.predict_all() method. Find the existing predict_all method in your TrackManager class:

def predict_all(self):
    for track in self.tracks:
        track.predict()
  • Replace it with this version that accepts imu_enabled and dt parameters and applies the IMU offset:
def predict_all(self, imu_enabled, dt):
    """Predict all tracks, applying IMU compensation if enabled."""
    dx_offset = get_imu_pixel_offset(dt) if imu_enabled else 0.0
    for track in self.tracks:
        track.predict()
        # Shift predicted x-position by camera rotation estimate
        track.state[0] += dx_offset

What changed?

Before the Hungarian matching step, every track's predicted x-position is shifted by the gyroscope-estimated pixel displacement. If the camera rotated 5 degrees to the right in one frame period, all predicted positions shift ~24 pixels to the left. The matcher then compares the shifted predictions to new detections and finds that objects are still close to their expected positions.

  • In your main loop, update the call to predict_all() to pass the new arguments. Find the line that calls track_manager.predict_all() and change it to:
# Calculate dt from last frame time
current_time = time.time()
dt = current_time - last_frame_time
last_frame_time = current_time

# Predict with IMU compensation
track_manager.predict_all(imu_enabled, dt)
  • Add last_frame_time = time.time() and imu_enabled = True before your main loop begins (next to where you start the IMU thread).
  • Save tracker.py and run it.
python tracker.py

Before you run this, think: what do you expect to happen now if you physically shake the ESP32 camera platform?

Shake the camera platform while objects are being tracked. Tracks should now hold their IDs through the motion instead of breaking apart. The gyroscope data is compensating for the camera rotation in real-time.

Tracks still breaking when you shake?

Check that imu_enabled is set to True before the main loop. Also verify the ESP32 /imu endpoint is returning live data by opening http://<ESP32_IP>/imu in your browser. You should see changing gz values when you rotate the board.

If gz is always 0, the LSM6DSM may not be initialized properly in your firmware. Reflash and check Serial Monitor for IMU initialization messages. Help me debug why my Kalman IMU offset is not working

Add a comparison mode with toggle key

To prove quantitatively that IMU compensation actually helps, you need a way to toggle it on and off and measure the difference in track survival during a shake test.

  • Add these tracking statistics variables before your main loop, alongside the other initializations:
# Comparison mode statistics
shake_test_active = False
shake_test_start = 0.0
SHAKE_TEST_DURATION = 10  # seconds
tracks_broken_no_imu = 0
tracks_broken_with_imu = 0
  • Inside your main loop, after the cv2.waitKey call that checks for the quit key, add the toggle and statistics logic:
# Toggle IMU compensation with 'i' key
if key == ord('i'):
    imu_enabled = not imu_enabled
    mode = "ON" if imu_enabled else "OFF"
    print(f"IMU compensation: {mode}")
    # Start a 10-second shake test
    shake_test_active = True
    shake_test_start = time.time()
    tracks_broken_no_imu = 0
    tracks_broken_with_imu = 0
    print("Shake test started! Shake the platform now.")

# Track broken-track count during shake test
if shake_test_active:
    elapsed = time.time() - shake_test_start
    if elapsed < SHAKE_TEST_DURATION:
        broken = track_manager.stale_count
        if imu_enabled:
            tracks_broken_with_imu += broken
        else:
            tracks_broken_no_imu += broken
    else:
        shake_test_active = False
        print(f"Shake test complete!")
        print(f"Tracks broken (no IMU): {tracks_broken_no_imu}")
        print(f"Tracks broken (with IMU): {tracks_broken_with_imu}")
        print(f"---")

How does the shake test work?

Press i to toggle IMU compensation on or off and simultaneously start a 10-second measurement window. During that window, the script counts how many tracks are deleted due to exceeding the 5-frame staleness threshold. Toggle again to measure with the other mode. The terminal prints a comparison at the end of each 10-second window.

  • You also need to expose the stale track count from your TrackManager. Find your delete_stale method and update it to store the count of deleted tracks:
def delete_stale(self, max_age=5):
    """Remove tracks not matched for max_age frames."""
    before = len(self.tracks)
    self.tracks = [t for t in self.tracks if t.age_since_update <= max_age]
    # Expose how many tracks were just deleted
    self.stale_count = before - len(self.tracks)
  • Add self.stale_count = 0 in your TrackManager.__init__ method.
  • Finally, add a real-time angular rate display to your terminal output. Inside your existing FPS print statement, replace it with this expanded version:
# Print frame stats with angular rate
with imu_lock:
    gz_display = imu_data["gz"]
print(f"FPS: {fps:.1f} | Detections: {det_count} | "
      f"Tracks: {len(track_manager.tracks)} | "
      f"gz: {gz_display:.1f} deg/s | "
      f"IMU: {'ON' if imu_enabled else 'OFF'}", end='\r')

What does this output show?

The terminal now shows a live readout of FPS, detection count, active tracks, angular rate from the gyroscope, and whether IMU compensation is active. When you shake the platform, you will see gz spike. With IMU on, the track count stays stable. With IMU off, it drops and resets.

  • Save tracker.py and run it:
python tracker.py

What do you predict will happen when you press i to disable IMU compensation and then shake the platform for 10 seconds?

Try the full comparison test. Place a few objects in view so the tracker has active tracks. Press i to switch IMU off and immediately shake the platform for 10 seconds. Note the Tracks broken (no IMU) count. Then press i again to switch IMU back on and shake for another 10 seconds. The Tracks broken (with IMU) count should be significantly lower.

Both counts look the same?

If the numbers are similar, check that you are shaking with enough angular velocity to exceed the matching threshold. Gentle tilts may not break tracks even without compensation. Try a firm rotation of 30-45 degrees back and forth.

Also verify that the gz value in your terminal readout is responding to motion. If it stays near 0, the IMU data is not reaching the main loop. Help me figure out why IMU compensation isn't making a difference

✔️ Awesome, I've got everything!

Great. Double check you have saved tracker.py and confirmed that tracks survive platform shaking with IMU enabled.

ⓧ I'd like to double check the full code

Scroll up and confirm these sections are present in tracker.py, in order:

  • Import block with threading, json, and time (plus your existing imports for urllib, cv2, numpy, scipy, ultralytics).
  • The imu_lock and imu_data globals, followed by the poll_imu function.
  • The FRAME_WIDTH, FOV_HORIZONTAL, PIXELS_PER_DEGREE constants and get_imu_pixel_offset function.
  • Your KalmanTrack class (unchanged from Step 4).
  • Your TrackManager class with the updated predict_all(self, imu_enabled, dt) and updated delete_stale exposing self.stale_count.
  • Thread launch (imu_thread), imu_enabled = True, last_frame_time = time.time(), and shake test variables before the main loop.
  • Main loop with dt calculation, predict_all(imu_enabled, dt) call, i key toggle, shake test statistics, and gz terminal readout.

Your tracker is now a genuine multi-sensor perception system. It fuses visual detections with inertial measurements to maintain state through platform motion. This is the same architecture used in ISR pods, autonomous drones, and robotic manipulation systems. Next up: the Secret Mission, where you add professional-grade track confidence decay.

Secret mission

Track Confidence with Temporal Decay

Real ISR trackers do not hold tracks forever on faith. They assign confidence scores that decay when a track goes unconfirmed, fading and eventually terminating lost targets. Can you add this occlusion-handling behavior to your tracker?

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally over WiFi and uses only your local GPU. There are no cloud resources or ongoing costs to worry about.

Resources you used:

  • Python virtual environment with PyTorch CUDA, Ultralytics, OpenCV, NumPy, and SciPy.
  • Project files including tracker.py and downloaded yolov8n.pt model weights.
  • ESP32 firmware flashed to your Deneyap Kart 1A.
  • Arduino IDE sketch files for the CameraWebServer with IMU endpoint.

Keep everything running

No action needed. Choose this if you plan to keep experimenting with the tracker, adding new features, or using the ESP32 sensor node for other projects.

  • Your Python virtual environment stays on disk ready for reactivation.
  • The ESP32 keeps its firmware and will serve the camera stream and IMU data whenever powered on and connected to WiFi.
  • The yolov8n.pt weights file stays in your project folder for instant inference without re-downloading.

Pause - I'll come back to this later

Shut down running processes to free up memory. Your files stay on disk for when you return.

  • Close the tracker.py script if it is still running by pressing q in the OpenCV window or Ctrl+C in the terminal.
  • Power off the Deneyap Kart 1A by disconnecting the USB cable.
  • Deactivate the Python virtual environment by running the following command:
deactivate
  • Your project folder, virtual environment, and model weights remain on disk ready to use when you return.

Delete - I don't want to use this again

Remove all project files, the virtual environment, and downloaded model weights from your machine.

  • Close the tracker.py script if running, then deactivate the virtual environment:
deactivate
  • Delete your project folder (which contains the virtual environment, tracker.py, and yolov8n.pt) by running the appropriate command for your operating system:

macOS

cd ~/Desktop
rm -rf [[PROJECT_FOLDER="your project folder name"]]

Windows

cd ~\Desktop
rmdir /s /q [[PROJECT_FOLDER="your project folder name"]]
  • Power off the Deneyap Kart 1A by disconnecting the USB cable. The firmware remains on the board but is inert without power. To fully clear it, flash any blank sketch from Arduino IDE.
  • Delete the Arduino sketch files for the CameraWebServer from your Arduino sketchbook folder if you no longer need them.

Nice Work!

Nice Work!

You just built a real-time multi-sensor perception system from scratch. Your Deneyap Kart 1A streams camera frames and gyroscope data over WiFi to a host running GPU-accelerated object detection with persistent, motion-compensated tracking.

You've learned how to:

  • Build a dual-stream ESP32 sensor node serving simultaneous MJPEG video and real-time IMU telemetry over WiFi to a host-side inference pipeline.
  • Run GPU-accelerated YOLOv8n detection on a live camera stream and implement a Kalman filter tracker from scratch that assigns persistent IDs and predicts object positions frame-to-frame.
  • Fuse gyroscope angular rates into the Kalman prediction step to compensate for platform rotation, quantifiably improving track survival during camera motion.
  • Secret Mission: Added temporal confidence decay with visual fading, simulating real ISR multi-target tracker occlusion handling.

Ready to quiz yourself?