Build a Spatio-Semantic Robot Memory

Give your Stretch 3 robot open-vocabulary 3D object memory with SigLIP and SAM 2.1.

Introduction

30 Second Summary

When you look for a coffee mug in a messy kitchen, you don't scan every surface pixel by pixel. Your brain maintains a spatial map that links objects to their locations by meaning, not raw coordinates.

In this project, you will build a spatio-semantic memory system for your Hello Robot Stretch 3 that works the same way. Your robot will scan its environment with the D435i camera, segment objects with SAM 2.1, encode them with SigLIP, and store everything in a queryable 3D voxel memory.

What You'll Build

Type "where is the coffee mug?" into your terminal and your robot's memory returns the object's 3D coordinates along with the exact camera frame where the mug was last seen.

By the end of this project, you'll have:

  • A natural language query system where you type any object name and the system finds the best-matching object in 3D space using cosine similarity search.
  • A semantic point cloud visualization color-coded by how similar each region is to your query, so you can see the robot's understanding of its environment at a glance.
  • A source image retrieval system that pulls up the exact camera frame where the robot last saw the matched object, confirming the result visually.
  • Secret Mission: Add dynamic memory updates so the system can track objects as they move in the real world, removing stale voxels and merging new observations.

Are there any prerequisites?

This project assumes you have a working Hello Robot Stretch 3 with a D435i head camera, existing Python code for capturing aligned RGB-D frames and retrieving the camera's global pose, and a GPU machine with the CUDA toolkit installed.

You should be comfortable writing Python and familiar with NumPy and OpenCV.

Before We Start

Before diving into code, let's lock in what you're about to build and why it matters for your Stretch 3.

Set Up the Environment and Dependencies

Your Stretch 3 already has the basics: pyrealsense2, OpenCV, NumPy, and your own kinematics library.

To build a spatio-semantic memory, you need three additional libraries: OpenCLIP for encoding images and text into a shared feature space, SAM 2.1 for automatic object segmentation, and Open3D for 3D point cloud visualization. Getting them all into one environment with compatible CUDA versions is the first real obstacle.

In this step, get ready to:
  • Create a Python environment and install PyTorch with CUDA support, OpenCLIP, and Open3D.
  • Clone the SAM 2 repository, install it, and download the model checkpoint.
  • Verify all imports work and confirm GPU availability.
Create a Python environment and install core packages

Each of these libraries has specific version requirements for PyTorch and Python.

A dedicated virtual environment keeps them isolated from your system Python and any other projects.

  • Create and activate a new Python virtual environment by running these commands:
python3 -m venv ~/dynamem-env
source ~/dynamem-env/bin/activate

What does this do?

python3 -m venv creates a fresh, isolated Python environment in the dynamem-env folder inside your home directory.

source ~/dynamem-env/bin/activate activates it so that every pip install and python command uses this environment instead of your system Python.

  • Confirm you are running Python 3.10+ inside the environment by running:
python --version

✔️ I see Python 3.10 or higher

Your environment is ready for the next installs.

ⓧ I see a version lower than 3.10

SAM 2 requires Python 3.10+ and will not install on older versions.

  • Install Python 3.10 or newer from python.org or via your system package manager.
  • Recreate the virtual environment using the new Python binary by running:
python3.10 -m venv ~/dynamem-env
source ~/dynamem-env/bin/activate
  • Run python --version again to confirm it now shows 3.10 or higher.

ⓧ Command not found

  • Make sure you activated the environment with source ~/dynamem-env/bin/activate.
  • If python3 is not installed at all, install Python 3.10+ from python.org.

PyTorch must match your system's CUDA version so the GPU can run tensor operations.

  • Check your CUDA version by running:
nvcc --version
  • Go to the PyTorch Get Started page.
  • Select Stable, your OS, Pip, Python, and the CUDA version that matches your nvcc output.
  • Copy the generated install command and run it in your terminal.

Why not just pip install torch?

A plain pip install torch may install a CPU-only build, which means none of the GPU-accelerated encoding will work.

The PyTorch installer page generates a command with the correct --index-url flag that points to CUDA-enabled wheels matching your hardware.

  • Verify PyTorch sees your GPU by running:
python -c "import torch; print(torch.cuda.is_available())"

You should see True printed.

Seeing False?

  • Make sure your NVIDIA driver is up to date. PyTorch ships its own CUDA runtime, but it still needs a compatible driver.
  • Double check that the CUDA version you selected on the PyTorch page matches or is compatible with your installed driver.
  • If you installed the CPU-only build by accident, uninstall it first with pip uninstall torch and reinstall using the correct command from pytorch.org.

Why is torch.cuda.is_available() returning False?

  • Install OpenCLIP and Open3D by running:
pip install open_clip_torch open3d==0.19.0

What are these packages?

open_clip_torch is the open-source implementation of CLIP and SigLIP models. You will use it to encode images and text into vectors that live in the same feature space.

open3d provides 3D visualization and point cloud data structures. You will use it to view your robot's environment as a colored 3D scene.

  • Confirm both installed correctly by running:
python -c "import open_clip; import open3d; print('OpenCLIP and Open3D OK')"

You should see OpenCLIP and Open3D OK printed with no errors.

Import errors?

  • If open_clip fails, check that you have PyTorch 2.6+ installed. OpenCLIP 3.3.0 requires torch >= 2.6.
  • If open3d fails, make sure your Python version is 3.10 or 3.11. Some Open3D wheels are not available for newer Python versions yet.

Help me fix my OpenCLIP or Open3D import error.

Clone SAM 2 and download the checkpoint

SAM 2 is not available as a standard pip package on PyPI.

You need to clone the repository and install it in editable mode so Python can find its modules.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Clone the SAM 2 repository and install it by running these commands:
git clone https://github.com/facebookresearch/sam2.git
cd sam2
pip install -e .

What does pip install -e . do?

The -e flag installs the package in editable mode. Instead of copying files into your environment's site-packages, Python reads directly from this cloned folder.

This means the SAM 2 modules are always available to your scripts without manually adjusting PYTHONPATH.

CUDA extension warning is safe to ignore

If you see a message like "Failed to build the SAM 2 CUDA extension" during installation, you can ignore it. Some post-processing functionality may be limited, but it does not affect the results in most cases.

  • Confirm the install succeeded by running:
python -c "import sam2; print('SAM 2 OK')"

You should see SAM 2 OK printed.

SAM 2 import failing?

  • Make sure you ran pip install -e . from inside the sam2 directory, not from a parent folder.
  • If you previously had an older SAM 2 install, uninstall it first with pip uninstall SAM-2, then run pip install -e . again from the repo root.

Help me fix my SAM 2 import error.

  • Download the SAM 2.1 small checkpoint by running these commands:
cd checkpoints
wget https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_small.pt
cd ..

Why this checkpoint?

SAM 2.1 checkpoints are improved versions of the original SAM 2 weights. The small variant balances speed and accuracy well for real-time robotics use.

  • Confirm the checkpoint file downloaded by running:
ls -lh checkpoints/sam2.1_hiera_small.pt

You should see a file roughly 200MB in size listed.

Download failed or file missing?

  • If wget is not available, use curl instead:
cd checkpoints
curl -L -o sam2.1_hiera_small.pt https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_small.pt
cd ..
  • Make sure you are inside the sam2 directory when running the download. The checkpoints folder should be directly inside sam2/.

Help me download the SAM 2.1 checkpoint.

Verify all imports and GPU access

Everything is installed. Before moving on, confirm that all three libraries can be imported in the same Python session and that the GPU is accessible.

  • Before you run the final check, what do you expect to see if everything installed correctly?
  • Run the full verification command:
python -c "import open_clip; import sam2; import open3d; print('All imports OK')"

You should see All imports OK printed with no errors.

  • Also confirm GPU access one more time by running:
python -c "import torch; print('CUDA available:', torch.cuda.is_available()); print('Device:', torch.cuda.get_device_name(0))"

You should see CUDA available: True followed by your GPU's name.

One of the imports failing?

  • If sam2 fails, confirm your terminal is still inside the sam2 directory or that the editable install registered correctly. Re-run pip install -e . from the sam2 repo root.
  • If open_clip or open3d fails, make sure your virtual environment is still active. Run which python and confirm it points to ~/dynamem-env/bin/python.

Help me fix my import errors after installing all dependencies.

Your environment is locked and loaded with every library the memory system needs. Next up, you will capture live RGB-D frames from your Stretch 3's D435i camera and turn them into a raw 3D point cloud.

Capture RGB-D Frames and Build a Raw 3D Point Cloud

Your Python environment is ready with all the libraries you need. Before layering on any semantic intelligence, you need to see your robot's world as raw geometry.

In this step, you will capture aligned RGB-D frames from the D435i head camera, save each frame's color image, depth map, and camera pose to disk, then backproject every depth pixel into 3D global coordinates. The result is a colored point cloud of your environment that you can rotate and explore in Open3D.

In this step, get ready to:
  • Write a capture script that saves RGB images, depth maps, and camera poses to disk.
  • Write a backprojection function that converts depth pixels into 3D global coordinates and merges multiple frames.
  • Visualize the merged point cloud in Open3D to confirm the geometry looks correct.
Write the capture script

The capture script grabs aligned RGB and depth frames from the D435i, records the camera's global pose at each frame using your existing kinematics library, and writes everything to disk. Later scripts will load these saved files instead of talking to the camera directly.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder and a subfolder for saved frames by running these commands:
mkdir -p dynamem/data/frames
cd dynamem
  • Confirm the folders were created by running:
ls data/

You should see a frames directory listed.

  • Create a file called capture.py inside the dynamem folder. Copy the following setup code into it:
import numpy as np
import cv2
import os

# Create output directory for saved frames
os.makedirs("data/frames", exist_ok=True)

# Number of frames to capture
NUM_FRAMES = 5

# ---- Your existing setup code goes here ----
# Initialize your pyrealsense2 pipeline for aligned RGB-D capture
# Initialize your pose/kinematics system
# Store your D435i camera intrinsics as fx, fy, cx, cy

What goes in the setup section?

Your existing pyrealsense2 initialization code belongs here. The project assumes you already have working code to start the D435i pipeline with aligned RGB and depth streams, and a pose/kinematics system that reports the camera's 4x4 global transformation matrix.

After your setup runs, you should have four values available: fx, fy, cx, and cy (the D435i's camera intrinsics).

  • Add the capture loop below your setup code in capture.py:
for i in range(NUM_FRAMES):
    input(f"Press Enter to capture frame {i}...")

    # ---- Your existing capture code goes here ----
    # Grab aligned RGB (H x W x 3, uint8) and depth (H x W, float32 meters)
    # Get the camera's 4x4 global pose matrix from your kinematics library
    # Store results in: rgb_image, depth_image, camera_pose

    # Save frame data to disk
    cv2.imwrite(f"data/frames/frame_{i}.png", rgb_image)
    np.save(f"data/frames/depth_{i}.npy", depth_image)
    np.save(f"data/frames/pose_{i}.npy", camera_pose)
    print(f"Saved frame {i}: RGB {rgb_image.shape}, depth {depth_image.shape}")

# Save intrinsics once (same for all frames from one camera)
np.save("data/frames/intrinsics.npy", np.array([fx, fy, cx, cy]))
print(f"Captured and saved {NUM_FRAMES} frames to data/frames/")

What does this code do?

The loop pauses at each iteration with input() so you can position the robot's head camera at a different angle before each capture. This gives you varied viewpoints that cover more of the room.

Each frame's RGB image is saved as a .png file, while the depth map and 4x4 pose matrix are saved as .npy files. NumPy's binary format preserves full floating-point precision for the depth values and pose matrices.

  • Save capture.py.
  • Run the capture script with your robot connected:
python capture.py

The script pauses before each frame and prints the shape of the saved data after each capture. After all 5 frames, it prints a confirmation line.

  • Confirm the files were saved by running:
ls data/frames/

You should see frame_0.png through frame_4.png, matching depth_N.npy and pose_N.npy files, and one intrinsics.npy.

Not seeing the saved files?

Make sure you ran the script from inside the dynamem folder. The script creates data/frames/ relative to wherever you run it.

If the script crashes during capture, check that your pyrealsense2 pipeline is initialized correctly and that your pose library returns a 4x4 NumPy array.

Help me debug my capture script

✔️ Awesome, I've got everything!

Great. Make sure capture.py is saved.

ⓧ I'd like to double check the full code

Your capture.py should contain two sections. Scroll up and confirm you have both:

  • The setup section with imports, directory creation, NUM_FRAMES = 5, and your existing initialization code.
  • The capture loop that calls input(), grabs frames, saves them with cv2.imwrite and np.save, and ends by saving intrinsics.npy.
Backproject depth to 3D global coordinates

Each depth pixel tells you how far away a surface is, but only from the camera's perspective. To combine views from different positions, you need to convert each pixel into a 3D point in global coordinates using the camera's intrinsics and its pose in the world.

  • Create a new file called backproject.py in the dynamem folder. Start with the imports and the backprojection function:
import numpy as np
import open3d as o3d
import cv2
import glob

def backproject_depth(depth_image, intrinsics, camera_pose):
    """Convert a depth image to 3D points in global coordinates."""
    fx, fy, cx, cy = intrinsics
    h, w = depth_image.shape

    # Create pixel coordinate grids
    u, v = np.meshgrid(np.arange(w), np.arange(h))

    # Filter out invalid (zero) depth values
    valid = depth_image > 0
    u = u[valid]
    v = v[valid]
    z = depth_image[valid]

    # Convert pixel coordinates to 3D camera-frame coordinates
    x = (u - cx) * z / fx
    y = (v - cy) * z / fy
    points_cam = np.stack([x, y, z], axis=-1)

    # Transform to global coordinates using the 4x4 pose matrix
    ones = np.ones((points_cam.shape[0], 1))
    points_hom = np.hstack([points_cam, ones])
    points_global = (camera_pose @ points_hom.T).T[:, :3]

    return points_global, valid

What does this function do?

The pinhole camera model defines how 3D points project onto 2D pixels. backproject_depth reverses that process. For each pixel (u, v) with a valid depth z, it computes the 3D camera-frame point as x = (u - cx) * z / fx and y = (v - cy) * z / fy.

It then multiplies each point by the 4x4 camera pose matrix to move it from camera coordinates into global coordinates. This lets you combine points from different viewpoints into one consistent 3D space.

  • Save backproject.py and run it to confirm the imports and function definition load cleanly:
python backproject.py

The script should exit silently with no errors. If you see an ImportError, double check that your environment has open3d installed.

  • Below the function in backproject.py, add the code that loads saved frames and merges them into point and color arrays:
# Load camera intrinsics saved by capture.py
intrinsics = np.load("data/frames/intrinsics.npy")
fx, fy, cx, cy = intrinsics

# Find all saved frames
frame_files = sorted(glob.glob("data/frames/frame_*.png"))
num_frames = len(frame_files)
print(f"Found {num_frames} saved frames")

all_points = []
all_colors = []

for i in range(num_frames):
    # Load saved frame data
    rgb = cv2.imread(f"data/frames/frame_{i}.png")
    rgb = cv2.cvtColor(rgb, cv2.COLOR_BGR2RGB)
    depth = np.load(f"data/frames/depth_{i}.npy")
    pose = np.load(f"data/frames/pose_{i}.npy")

    # Backproject depth pixels to 3D global coordinates
    points, valid_mask = backproject_depth(depth, (fx, fy, cx, cy), pose)

    # Get RGB colors for valid depth points (normalized to 0-1)
    colors = rgb[valid_mask].astype(np.float64) / 255.0

    all_points.append(points)
    all_colors.append(colors)
    print(f"Frame {i}: {len(points)} points")

What does this code do?

This section loads the intrinsics and each saved frame from disk. It calls backproject_depth on every frame to convert depth pixels into global 3D points, then collects the matching RGB colors.

OpenCV reads images in BGR format, so cv2.cvtColor converts to RGB. Colors are normalized to the 0-1 range because Open3D expects floating-point color values.

  • Save backproject.py and run it to verify the loading and backprojection work:
python backproject.py

You should see output showing how many frames were found and the number of 3D points generated per frame. Each frame typically produces tens of thousands to hundreds of thousands of points.

Seeing a FileNotFoundError?

Make sure you are running the command from inside the dynamem folder. The script looks for data/frames/ relative to your current directory.

If the error mentions intrinsics.npy, check that capture.py saved it. Run ls data/frames/ and confirm the file is there.

Help me debug my backprojection script

Visualize the merged point cloud

The point and color arrays from each frame need to be merged into a single Open3D point cloud object. This is the moment you see your robot's environment reconstructed in 3D.

  • Add the following code to the bottom of backproject.py, below the frame-loading loop:
# Merge all frames into a single set of points and colors
all_points = np.vstack(all_points)
all_colors = np.vstack(all_colors)
print(f"Total merged points: {len(all_points)}")

# Create an Open3D point cloud object
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(all_points)
pcd.colors = o3d.utility.Vector3dVector(all_colors)

# Open the 3D viewer
print("Opening 3D viewer... Close the window to exit.")
o3d.visualization.draw_geometries([pcd], window_name="Raw Point Cloud")

What does this code do?

It stacks the per-frame point and color arrays into single arrays with np.vstack, then wraps them in an Open3D PointCloud object. The draw_geometries function opens an interactive 3D viewer window.

Inside the viewer, you can rotate with left-click drag, zoom with the scroll wheel, and translate with Ctrl+left-click drag. Press H in the viewer for a full list of controls.

  • Save backproject.py.

Before you run the script, what do you think you will see in the viewer? A flat image? A wireframe mesh? Something else?

  • Run the full backprojection and visualization pipeline:
python backproject.py

A colored 3D point cloud of your environment appears in the Open3D viewer window. You can rotate around it and see the room from angles your robot never captured.

The geometry is raw and dense, but every point is just a color with no understanding of what it represents.

Viewer window not appearing?

If you are running over SSH, Open3D needs a display server. Make sure X11 forwarding is enabled (ssh -X) or run the script on a machine with a monitor attached.

If the window opens but shows nothing, your point cloud may be at an unexpected scale. Try zooming out with the scroll wheel. Depth values need to be in meters for the coordinates to make sense.

Help me get the Open3D viewer working

✔️ Awesome, I've got everything!

Great. Make sure backproject.py is saved.

ⓧ I'd like to double check the full code

Your backproject.py should contain three sections. Scroll up and confirm you have all of them in order:

  • Imports (numpy, open3d, cv2, glob) and the backproject_depth function that converts depth pixels to global 3D points.
  • The frame-loading loop that reads each saved frame, calls backproject_depth, and collects points and colors into lists.
  • The merge and visualization section that stacks arrays with np.vstack, creates an o3d.geometry.PointCloud(), and opens the viewer with draw_geometries.

You now have a geometric map of your robot's surroundings, built from real sensor data. It is colorful but semantically meaningless. In the next step, you will give each point a semantic identity by encoding image regions with SigLIP, so you can ask questions like "where is the coffee mug?" and get a 3D answer.

Add SigLIP Encoding for Semantic Features

Your merged point cloud from the previous step paints a colorful 3D picture of the room. It has geometry and color, but zero understanding of what anything actually is.

SigLIP is a vision-language model that encodes images and text into the same high-dimensional vector space. By encoding patches of each camera frame with SigLIP, every 3D point gains a semantic feature vector. Text queries like "where is the coffee mug?" then become a simple similarity search across those vectors.

In this step, get ready to:
  • Load SigLIP via OpenCLIP and write a patch encoding function.
  • Assign per-patch features to the 3D points in your point cloud.
  • Write a text query function and test it against the semantic point cloud.
Load SigLIP and write the patch encoding function

OpenCLIP provides a unified Python API for loading vision-language models. You will load the ViT-SO400M-14-SigLIP model, the same SigLIP variant that DynaMem uses on GPU. The model comes with a preprocessing transform that resizes and normalizes image inputs.

  • Create a new file called siglip_encode.py in your project folder, then add the following imports and model setup:
import numpy as np
import torch
import open_clip
from PIL import Image
import os

# Load SigLIP model and its image preprocessing transform
model, _, preprocess = open_clip.create_model_and_transforms(
    'ViT-SO400M-14-SigLIP', pretrained='webli'
)
model = model.cuda().eval()

# Patch extraction settings
PATCH_SIZE = 112
STRIDE = 56  # 50% overlap for dense coverage

What does this code do?

create_model_and_transforms downloads the SigLIP weights from HuggingFace (first run only) and returns the model plus its image preprocessing pipeline. The webli tag selects the pretrained checkpoint trained on Google's WebLI dataset.

model.eval() switches off training-specific behaviors like dropout. PATCH_SIZE and STRIDE control the sliding window that will divide each frame into overlapping image crops.

Next, you need a function that slides a window across an image and encodes each crop. Each crop becomes one feature vector that represents the semantic content of that region.

  • Add the following function below the constants in siglip_encode.py:
def extract_patch_features(image_path):
    """Divide an image into overlapping patches and encode each with SigLIP."""
    img = Image.open(image_path).convert('RGB')
    img_np = np.array(img)
    h, w, _ = img_np.shape

    patches = []
    centers = []

    # Slide a window across the image with overlap
    for y in range(0, h - PATCH_SIZE + 1, STRIDE):
        for x in range(0, w - PATCH_SIZE + 1, STRIDE):
            patch = Image.fromarray(img_np[y:y + PATCH_SIZE, x:x + PATCH_SIZE])
            patches.append(preprocess(patch))
            # Store the center pixel of each patch for backprojection
            centers.append((y + PATCH_SIZE // 2, x + PATCH_SIZE // 2))

    # Batch encode all patches on GPU
    patch_tensor = torch.stack(patches).cuda()
    with torch.no_grad():
        features = model.encode_image(patch_tensor)
        # L2-normalize so cosine similarity becomes a dot product
        features = features / features.norm(dim=-1, keepdim=True)

    return features.cpu().numpy(), centers

What does this function do?

The function slides a 112x112 pixel window across the image with 50% overlap (stride 56). Each crop is preprocessed and fed through SigLIP's image encoder via model.encode_image().

The center pixel coordinates of each patch are saved so you can map each feature vector back to a 3D point via depth backprojection. L2-normalization ensures that a simple dot product between vectors equals cosine similarity.

  • Save siglip_encode.py.
  • Verify the model loads and the function works by running this test command:
python -c "from siglip_encode import extract_patch_features; f, c = extract_patch_features('data/frames/frame_0.png'); print(f'Patches: {len(c)}, Feature dim: {f.shape[1]}')"

You should see a line like Patches: 120, Feature dim: 1152 (the exact patch count depends on your image resolution). The first run downloads the model weights, which may take a minute.

Seeing an Unknown model error?

SigLIP uses a timm-backed image encoder. Make sure you have the latest timm package installed. Run pip install --upgrade timm and try again.

If you see a CUDA out-of-memory error, your image may produce too many patches for a single batch. Try reducing the image resolution or processing patches in smaller batches.

Help me debug my SigLIP model loading issue.

Assign patch features to 3D points

Each patch center pixel has a known location in the image. Using the depth value at that pixel and the camera intrinsics, you can backproject the center to a 3D global coordinate. This is the same math your backproject.py already uses, applied only to patch center pixels.

  • Add the following backprojection function below extract_patch_features in siglip_encode.py:
def backproject_patch_centers(centers, depth, intrinsics, pose):
    """Convert patch center pixels to 3D global coordinates."""
    fx, fy, cx, cy = intrinsics
    points_3d = []
    valid_indices = []

    for i, (v, u) in enumerate(centers):
        z = depth[v, u]
        if z <= 0:
            continue
        # Camera-frame 3D coordinates from pinhole model
        x = (u - cx) * z / fx
        y = (v - cy) * z / fy
        point_cam = np.array([x, y, z, 1.0])
        # Transform to global coordinates using camera pose
        point_global = pose @ point_cam
        points_3d.append(point_global[:3])
        valid_indices.append(i)

    return np.array(points_3d), valid_indices

What does this function do?

For each patch center pixel, the function reads the depth value and applies the pinhole camera model to compute the 3D point in camera coordinates. It then multiplies by the camera pose matrix to get global coordinates.

Patches whose center pixel has zero or negative depth (no valid reading) are skipped. The valid_indices list tracks which patches survived so you can select the matching feature vectors later.

Now you need the main script body that processes every captured frame, extracts features, backprojects patch centers, and saves the result.

  • Add the following main processing loop at the bottom of siglip_encode.py:
# --- Main: process all captured frames ---
frames_dir = 'data/frames'
all_points = []
all_features = []

# Replace with your D435i camera intrinsics (fx, fy, cx, cy)
# These are the same values you use in backproject.py
fx, fy, cx, cy = 615.0, 615.0, 320.0, 240.0
intrinsics = (fx, fy, cx, cy)

frame_files = sorted(f for f in os.listdir(frames_dir) if f.startswith('frame_'))
for frame_file in frame_files:
    idx = frame_file.split('_')[1].split('.')[0]
    rgb_path = os.path.join(frames_dir, frame_file)
    depth = np.load(os.path.join(frames_dir, f'depth_{idx}.npy'))
    pose = np.load(os.path.join(frames_dir, f'pose_{idx}.npy'))

    # Encode patches and map to 3D
    features, centers = extract_patch_features(rgb_path)
    pts, valid = backproject_patch_centers(centers, depth, intrinsics, pose)
    all_points.append(pts)
    all_features.append(features[valid])

# Merge all frames into a single semantic point cloud
all_points = np.concatenate(all_points, axis=0)
all_features = np.concatenate(all_features, axis=0)
np.save('data/points_semantic.npy', all_points)
np.save('data/features_semantic.npy', all_features)
print(f'Encoded {len(all_points)} semantic points')
print(f'Feature dimension: {all_features.shape[1]}')

What does this code do?

The loop iterates every frame saved by capture.py. For each frame it extracts patch features with SigLIP, backprojects the patch centers into 3D global coordinates, and collects only the patches with valid depth.

After processing all frames, the points and features are concatenated and saved as .npy files. Each 3D point now has a position and a high-dimensional semantic vector living in the same space as text embeddings.

Update your intrinsics values

The placeholder intrinsics values (615.0, 615.0, 320.0, 240.0) are approximate D435i defaults. Replace them with the actual calibration values from your robot's setup. These should match the intrinsics you already use in backproject.py.

  • Update the intrinsics values to match your camera, then save siglip_encode.py.
  • Run the full encoding pipeline by running this command:
python siglip_encode.py

You should see output showing the total number of encoded semantic points and the feature dimension. Processing each frame through SigLIP takes a few seconds, so expect this to run for a minute or two depending on how many frames you captured.

Script crashes on a frame?

Check that your data/frames/ directory contains matching sets of frame_N.png, depth_N.npy, and pose_N.npy for each frame. A missing depth or pose file for any frame index will cause a file-not-found error.

Help me debug my siglip_encode.py script.

✔️ Awesome, I've got everything!

Double check you have saved siglip_encode.py and that the script printed the number of encoded points without errors.

ⓧ I'd like to double check the full code

Your siglip_encode.py should contain these sections in order. Scroll up and confirm each one is present:

  • Imports (numpy, torch, open_clip, PIL, os) plus model loading with create_model_and_transforms and PATCH_SIZE/STRIDE constants.
  • extract_patch_features(image_path) function with the sliding window loop and model.encode_image() call.
  • backproject_patch_centers(centers, depth, intrinsics, pose) function using the pinhole camera model.
  • Main processing loop that iterates frames, encodes, backprojects, concatenates, and saves to data/points_semantic.npy and data/features_semantic.npy.
Query the semantic point cloud

The semantic point cloud is built. Now you need a way to ask questions about it. SigLIP encodes text into the same vector space as images, so finding "the coffee mug" means encoding that phrase and finding whichever 3D point's feature is most similar.

  • Create a new file called query.py and add the following code:
import numpy as np
import torch
import open_clip

# Load the same SigLIP model for text encoding
model, _, _ = open_clip.create_model_and_transforms(
    'ViT-SO400M-14-SigLIP', pretrained='webli'
)
tokenizer = open_clip.get_tokenizer('ViT-SO400M-14-SigLIP')
model = model.cuda().eval()

# Load the semantic point cloud from siglip_encode.py output
points = np.load('data/points_semantic.npy')
features = np.load('data/features_semantic.npy')

def query_points(text):
    """Find the 3D point most similar to a text description."""
    tokens = tokenizer([text]).cuda()
    with torch.no_grad():
        text_feat = model.encode_text(tokens)
        text_feat = text_feat / text_feat.norm(dim=-1, keepdim=True)

    # Cosine similarity is a dot product since features are L2-normalized
    similarities = features @ text_feat.cpu().numpy().T
    best_idx = np.argmax(similarities)
    return points[best_idx], float(similarities[best_idx])

# Test with a query
query_text = 'coffee mug'
location, score = query_points(query_text)
print(f'Query: "{query_text}"')
print(f'Best match at: {location}')
print(f'Similarity score: {score:.4f}')

What does this code do?

get_tokenizer returns the text tokenizer matched to SigLIP. The query_points function tokenizes a text string and encodes it with model.encode_text() to produce a feature vector in the same space as the image features.

Since both the image and text features are L2-normalized, the dot product (features @ text_feat.T) equals cosine similarity. The function returns the 3D location of the point with the highest similarity score.

  • Save query.py.

Before you run this, take a moment to predict what you expect to see. Do you think the system will find the correct object? Will the similarity score be high or low?

  • Run the query script by running this command:
python query.py

You should see a 3D coordinate and a similarity score printed. The system does return a location for "coffee mug," which is exciting. The core concept works.

But look carefully at the results. If you try several different queries (swap coffee mug for other objects in your scene), you will notice that the high-similarity points scatter across the scene rather than clustering tightly on the actual object. Neighboring patches on the same object produce slightly different embeddings, and patches at object boundaries blend foreground and background.

Why are the results so noisy?

The per-patch approach encodes fixed-size crops of the image without any notion of object boundaries. A patch might contain half a mug and half a table, producing a feature that represents neither object cleanly.

This is exactly the problem that motivates the next step. Instead of encoding arbitrary patches, you will first segment the image into clean object masks and then encode each object as a whole.

Getting very low similarity scores?

Make sure your query text describes an object that is actually visible in the frames you captured. Try generic queries first, like chair or table, to confirm the pipeline is working before trying more specific objects.

Check that data/points_semantic.npy and data/features_semantic.npy exist and are non-empty. If siglip_encode.py reported zero encoded points, your intrinsics or depth data may be misconfigured.

Help me debug my query results.

✔️ Awesome, I've got everything!

Double check you have saved query.py and that running it printed a query, a best-match 3D coordinate, and a similarity score.

ⓧ I'd like to double check the full code

Here is the complete query.py for comparison:

import numpy as np
import torch
import open_clip

# Load the same SigLIP model for text encoding
model, _, _ = open_clip.create_model_and_transforms(
    'ViT-SO400M-14-SigLIP', pretrained='webli'
)
tokenizer = open_clip.get_tokenizer('ViT-SO400M-14-SigLIP')
model = model.cuda().eval()

# Load the semantic point cloud from siglip_encode.py output
points = np.load('data/points_semantic.npy')
features = np.load('data/features_semantic.npy')

def query_points(text):
    """Find the 3D point most similar to a text description."""
    tokens = tokenizer([text]).cuda()
    with torch.no_grad():
        text_feat = model.encode_text(tokens)
        text_feat = text_feat / text_feat.norm(dim=-1, keepdim=True)

    # Cosine similarity is a dot product since features are L2-normalized
    similarities = features @ text_feat.cpu().numpy().T
    best_idx = np.argmax(similarities)
    return points[best_idx], float(similarities[best_idx])

# Test with a query
query_text = 'coffee mug'
location, score = query_points(query_text)
print(f'Query: "{query_text}"')
print(f'Best match at: {location}')
print(f'Similarity score: {score:.4f}')

Your semantic point cloud answers plain-English queries about 3D space. That is a major milestone. The scattered, noisy results are the gap you will close next. In the next step, you will use SAM 2.1 to segment each frame into clean object masks before encoding, giving every object a single high-quality feature instead of dozens of noisy patch-level ones.

Integrate SAM 2.1 for Object-Level Segmentation

In the last step, you gave every 3D point a SigLIP feature vector by encoding small image patches. Text queries worked, but the results were scattered across the scene because neighboring patches on the same object got slightly different embeddings.

The fix is to segment the image into clean object masks first, then encode entire object crops instead of tiny patches. SAM 2.1 does exactly this. It produces 20-60 object masks per frame with zero prompts, and each mask isolates a single real-world object.

In this step, get ready to:
  • Write a segmentation script that automatically segments each RGB frame into object masks using SAM 2.1.
  • Update your encoding pipeline to use object-level crops instead of patches.
  • Re-run a text query and see dramatically cleaner, tightly clustered results.
Load SAM 2.1 and write an automatic mask generation function

SAM 2.1's automatic mask generator works by sampling a grid of point prompts across the entire image, predicting masks at each point, and filtering out duplicates. You do not need to provide any manual prompts.

  • Create a new file called segment.py in your project folder.
  • Add the following code to load the SAM 2.1 model and define a segmentation function:
import numpy as np
import torch
from sam2.automatic_mask_generator import SAM2AutomaticMaskGenerator

# Load SAM 2.1 model for automatic mask generation
mask_generator = SAM2AutomaticMaskGenerator.from_pretrained(
    "facebook/sam2.1-hiera-small",
    points_per_side=32,
    pred_iou_thresh=0.8,
    stability_score_thresh=0.95,
    min_mask_region_area=100,
)

What does this code do?

SAM2AutomaticMaskGenerator.from_pretrained() downloads the SAM 2.1 small checkpoint from Hugging Face and builds the mask generator in one call.

points_per_side=32 samples a 32x32 grid of point prompts across the image (1024 points total). pred_iou_thresh and stability_score_thresh filter out low-quality masks. min_mask_region_area removes tiny mask fragments smaller than 100 pixels.

  • Below the model loading code, add the segmentation function:
def segment_frame(rgb_image):
    """Segment an RGB image into object masks using SAM 2.1."""
    # SAM 2.1 expects HWC uint8 numpy array
    if rgb_image.dtype != np.uint8:
        rgb_image = (rgb_image * 255).astype(np.uint8)

    # Generate masks for all objects in the image
    masks = mask_generator.generate(rgb_image)

    # Extract binary segmentation masks
    binary_masks = [mask["segmentation"] for mask in masks]

    print(f"Found {len(binary_masks)} object masks in frame")
    return binary_masks

What does this function return?

mask_generator.generate() returns a list of dictionaries, each containing a segmentation key with a binary mask (a 2D boolean array the same height and width as the input image).

Each mask isolates one object. A typical indoor scene produces 20-60 masks depending on how many objects are visible.

  • Save segment.py.
  • Test the segmentation on one of your saved frames by running:
python -c "import cv2; from segment import segment_frame; img = cv2.imread('data/frames/frame_000_rgb.png'); img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB); masks = segment_frame(img); print(f'Total masks: {len(masks)}, first mask shape: {masks[0].shape}')"

You should see output showing the number of object masks found (typically 20-60) and the shape of each mask matching your image dimensions.

Not seeing any masks?

  • Make sure the SAM 2 repo was installed correctly with pip install -e . from inside the sam2 directory.
  • Confirm huggingface_hub is installed. The from_pretrained method downloads the checkpoint via Hugging Face.
  • Check that your image file path is correct and the image loads as a valid RGB numpy array.

Still stuck? Help me debug why SAM 2.1 automatic mask generation is returning no masks for my image.

Update encoding to use object-level crops instead of patches

Now that SAM 2.1 can carve each frame into object masks, you need to update siglip_encode.py to encode entire object crops instead of grid patches. This is the key change that fixes the noisy, scattered query results from the previous step.

  • Open siglip_encode.py.
  • Find the import section at the top of the file. Add the import for your new segmentation function:
from segment import segment_frame
  • Find the existing function that divides images into overlapping patches and encodes each patch. Replace it with a function that uses SAM 2.1 masks to extract and encode object crops:
def encode_frame_objects(rgb_image, model, preprocess, device):
    """Encode object-level crops from a single frame using SAM 2.1 masks."""
    # Segment the frame into object masks
    masks = segment_frame(rgb_image)

    object_features = []
    for mask in masks:
        # Get bounding box of the mask
        rows = np.any(mask, axis=1)
        cols = np.any(mask, axis=0)
        rmin, rmax = np.where(rows)[0][[0, -1]]
        cmin, cmax = np.where(cols)[0][[0, -1]]

        # Crop the RGB image to the mask bounding box
        crop = rgb_image[rmin:rmax+1, cmin:cmax+1].copy()

        # Zero out pixels outside the mask
        mask_crop = mask[rmin:rmax+1, cmin:cmax+1]
        crop[~mask_crop] = 0

        # Preprocess and encode the crop with SigLIP
        crop_pil = Image.fromarray(crop)
        crop_tensor = preprocess(crop_pil).unsqueeze(0).to(device)
        with torch.no_grad():
            feature = model.encode_image(crop_tensor)
            feature = feature / feature.norm(dim=-1, keepdim=True)

        object_features.append({
            "feature": feature.cpu().numpy().squeeze(),
            "mask": mask,
        })

    return object_features

What changed from the patch approach?

Instead of slicing the image into a fixed grid of small patches, you now use each SAM 2.1 mask to crop a single object region. Pixels outside the mask are zeroed out so SigLIP only sees the object itself.

The result is one feature vector per object instead of dozens of noisy patch-level vectors. Every 3D point that falls within the same mask now shares an identical feature, so query results cluster tightly on real objects.

  • Update the main encoding loop that processes all saved frames. Find the section where you iterate over frames and call the patch encoding function. Replace those calls with encode_frame_objects:
def assign_features_to_points(points_3d, object_features, depth_image, intrinsics, pose):
    """Assign object-level features to 3D points based on mask membership."""
    h, w = depth_image.shape[:2]
    features = np.zeros((len(points_3d), object_features[0]["feature"].shape[0]))

    for obj in object_features:
        mask = obj["mask"]
        feature = obj["feature"]
        # Assign this object's feature to all points that project into the mask
        for i, pt in enumerate(points_3d):
            # Project 3D point back to pixel coordinates
            pt_cam = np.linalg.inv(pose) @ np.append(pt[:3], 1.0)
            if pt_cam[2] <= 0:
                continue
            u = int(intrinsics["fx"] * pt_cam[0] / pt_cam[2] + intrinsics["cx"])
            v = int(intrinsics["fy"] * pt_cam[1] / pt_cam[2] + intrinsics["cy"])
            if 0 <= u < w and 0 <= v < h and mask[v, u]:
                features[i] = feature

    return features

How does feature assignment work now?

For each object mask, the function projects every 3D point back into the 2D image plane. If a point lands inside the mask, it receives that object's single SigLIP feature vector.

Points belonging to the same object all get the exact same feature, so there is zero noise within an object. This is the key improvement over the per-patch approach where every point got a slightly different embedding.

  • Save siglip_encode.py.
  • Re-run the encoding pipeline on your saved frames to rebuild the point cloud with object-level features:
python siglip_encode.py

You should see the segmentation counts printed for each frame (something like "Found 34 object masks in frame") as SAM 2.1 processes each image.

Seeing a CUDA out of memory error?

  • Add torch.cuda.empty_cache() after processing each frame. SAM 2.1 can accumulate GPU memory across iterations.
  • Try reducing points_per_side from 32 to 16 in segment.py. This samples fewer prompts and uses less memory.

Still running into GPU memory issues? Help me fix CUDA out of memory when running SAM 2.1 in a loop over multiple frames.

Re-run a text query and see the improvement

The moment of truth. Your point cloud now has object-level features instead of noisy patch features. Before you run the query, think about what you expect to see differently.

Do you think the "coffee mug" query will return one tight cluster of points, or will it still scatter across the scene like before?

  • Run the same query you used in the previous step by running:
python query.py "coffee mug"

The top result now clusters tightly around the actual mug location instead of scattering across the scene. The similarity score should also be higher because the feature vector represents the whole object, not a tiny patch.

Why is this so much better?

In the previous step, SigLIP encoded small image patches. A patch containing half a mug handle and half a table got a confused embedding that matched "mug" weakly. Dozens of these weak matches scattered across the scene made the results noisy.

Now, SAM 2.1 isolates the mug as a single object mask. SigLIP encodes the entire mug crop, producing one strong feature vector. Every point on the mug shares this identical vector, so they all score equally high and cluster together in 3D space.

Results still look scattered?

  • Make sure you re-ran siglip_encode.py after updating it. Old per-patch features may still be cached.
  • Check that segment_frame is being called inside the encoding loop. If the import failed silently, the old patch code might still be running.
  • Verify that the mask assignment is working by printing np.count_nonzero(features) after feature assignment. If it prints 0, no points are being assigned features.

Need help comparing the old and new results? Help me understand why my SAM 2.1 segmented query results still look noisy.

Your memory system now understands objects, not just pixels. Next up, you will voxelize the point cloud into a compact spatial data structure that makes queries fast and stores metadata like observation counts and source images.

Voxelize the Memory for Fast Querying

In the last step you integrated SAM 2.1 segmentation and saw query results snap from scattered noise to tight clusters on actual objects. The semantic quality is there now.

But the system still stores a separate feature vector for every single 3D point. That means tens of thousands of entries to search through on every query. Voxelizing the point cloud groups nearby points into fixed-size 3D cells. Each voxel stores one aggregated feature vector, an observation count, the source image ID, and a timestamp. Queries search over hundreds of voxels instead of tens of thousands of points.

In this step, get ready to:
  • Build a VoxelMap class that groups 3D points into fixed-size grid cells with aggregated features.
  • Add a query method that encodes text with SigLIP and finds the best-matching voxel by dot-product similarity.
  • Run the full end-to-end pipeline and verify accurate object localization with multiple text queries.
Build the VoxelMap class

A voxel is a 3D pixel. Just like an image grid groups light values into pixels, a voxel grid groups 3D points into fixed-size cubes. Each cube stores one representative feature vector instead of hundreds of redundant per-point vectors.

  • Create a new file called voxel_map.py in your project folder and add the following class definition:
import numpy as np
import time

class VoxelMap:
    def __init__(self, voxel_size=0.05):
        # Voxel size in meters (default 5cm)
        self.voxel_size = voxel_size
        # Maps voxel grid coordinates to stored voxel data
        self.voxels = {}

    def _to_key(self, point):
        # Convert a 3D coordinate to its voxel grid cell index
        return tuple((point[:3] / self.voxel_size).astype(int))

    def add_points(self, points, features, source_image_id):
        # Ingest 3D points with features into voxel grid cells
        timestamp = time.time()
        for i in range(len(points)):
            key = self._to_key(points[i])
            if key in self.voxels:
                v = self.voxels[key]
                # Update running weighted average of features
                n = v["obs_count"]
                v["feature"] = (v["feature"] * n + features[i]) / (n + 1)
                v["obs_count"] = n + 1
                v["source_image_id"] = source_image_id
                v["timestamp"] = timestamp
            else:
                # New voxel at the center of this grid cell
                center = (np.array(key) + 0.5) * self.voxel_size
                self.voxels[key] = {
                    "position": center,
                    "feature": features[i].copy(),
                    "obs_count": 1,
                    "source_image_id": source_image_id,
                    "timestamp": timestamp,
                }

What does this code do?

The VoxelMap class uses a Python dictionary keyed by integer grid coordinates. The default voxel size of 0.05 meters (5cm) means every 5cm cube of space becomes one entry.

When multiple points land in the same voxel, add_points computes a running weighted average of their feature vectors. It also tracks how many observations contributed to this voxel, which source image was most recent, and a timestamp for when the voxel was last updated.

  • Save voxel_map.py.
  • Confirm the import works by running this command:
python -c "from voxel_map import VoxelMap; vm = VoxelMap(); print(f'VoxelMap ready, voxel size: {vm.voxel_size}m')"

You should see VoxelMap ready, voxel size: 0.05m printed in the terminal.

Seeing an ImportError?

  • Make sure you saved the file as voxel_map.py in the same directory where you run the command.
  • Double-check for typos in the class name. The file must define a class called VoxelMap.
  • Help me debug my VoxelMap import error.
Add the query method to VoxelMap

The voxel map can ingest points now, but it has no way to answer questions yet. The query method encodes a text string with SigLIP's text encoder and finds the voxel whose feature vector is most similar.

  • In voxel_map.py, add the following query method inside the VoxelMap class, below the add_points method:
    def query(self, text, model, tokenizer, device="cuda"):
        # Encode the text query using SigLIP's text encoder
        import torch
        tokens = tokenizer([text]).to(device)
        with torch.no_grad():
            text_feat = model.encode_text(tokens)
            text_feat /= text_feat.norm(dim=-1, keepdim=True)
        text_feat = text_feat.cpu().numpy().squeeze()

        # Find the voxel with highest dot-product similarity
        best_score = -1
        best_voxel = None
        for voxel in self.voxels.values():
            feat = voxel["feature"]
            feat_norm = feat / np.linalg.norm(feat)
            score = np.dot(text_feat, feat_norm)
            if score > best_score:
                best_score = score
                best_voxel = voxel

        return {
            "position": best_voxel["position"],
            "score": float(best_score),
            "source_image_id": best_voxel["source_image_id"],
            "obs_count": best_voxel["obs_count"],
        }

What does the query method do?

The method tokenizes your text query (like "coffee mug") and runs it through SigLIP's text encoder via model.encode_text(). This produces a feature vector in the same latent space as the image features stored in each voxel.

It then loops over every voxel, normalizes each feature, and computes the dot product against the text feature. The voxel with the highest score is the best match. The return value includes that voxel's 3D position, the similarity score, the source image ID, and the observation count.

  • Save voxel_map.py.

Awesome, I've got everything!

Great. Make sure you saved voxel_map.py with both the add_points and query methods inside the VoxelMap class.

I'd like to double check the full code

The complete voxel_map.py is the two code blocks you built above, combined in order. Scroll up and confirm your file contains:

  • The imports (numpy, time) at the top.
  • The VoxelMap class with __init__, _to_key, add_points, and query methods, in that order.
  • The query method is indented inside the class, at the same level as add_points.
Run the full pipeline end-to-end

Now you have a complete VoxelMap class. Time to update query.py so it loads your saved point data into the voxel map and queries against voxels instead of raw points.

  • Replace the contents of query.py with the following code to load data and build the voxel map:
import numpy as np
import open_clip
import torch
import cv2
import os
from voxel_map import VoxelMap

# Load the SigLIP model and tokenizer
model, _, preprocess = open_clip.create_model_and_transforms(
    'ViT-SO400M-14-SigLIP', pretrained='webli'
)
model = model.to('cuda').eval()
tokenizer = open_clip.get_tokenizer('ViT-SO400M-14-SigLIP')

# Load saved point cloud data with features
points = np.load('data/points.npy')
features = np.load('data/features.npy')
image_ids = np.load('data/image_ids.npy')

# Build voxel map, grouping points by source image
voxel_map = VoxelMap(voxel_size=0.05)
for img_id in np.unique(image_ids):
    mask = image_ids == img_id
    voxel_map.add_points(points[mask], features[mask], int(img_id))

print(f"Compressed {len(points)} points -> {len(voxel_map.voxels)} voxels")

What does this code do?

  • It loads the SigLIP model (ViT-SO400M-14-SigLIP) via OpenCLIP so the query method can encode text.
  • It loads the 3D points, their SigLIP feature vectors, and the corresponding image IDs that your previous pipeline saved to .npy files.
  • It creates a VoxelMap with 5cm voxels and feeds all points in, grouped by their source image so each voxel tracks which image it came from.
  • Save query.py and run it to see the compression ratio:
python query.py

You should see output like Compressed 48320 points -> 412 voxels. Your exact numbers will differ, but the voxel count should be in the hundreds while the point count is in the tens of thousands.

Seeing a FileNotFoundError for the .npy files?

  • Make sure your previous pipeline (backproject.py and siglip_encode.py) saved points.npy, features.npy, and image_ids.npy to the data/ directory. Re-run the encode pipeline if they are missing.
  • Check that you are running query.py from the project root directory where data/ lives.
  • Help me find my saved numpy files for the voxel map.

Now add the query loop below the code you just wrote. This runs multiple text queries against the voxel map and shows the source image for each match.

  • Add the following code at the bottom of query.py, below the print line:
# Query the voxel map with natural language
queries = ["red cup", "laptop", "chair"]
for q in queries:
    result = voxel_map.query(q, model, tokenizer)
    pos = result["position"]
    print(f"'{q}' -> ({pos[0]:.2f}, {pos[1]:.2f}, {pos[2]:.2f}), "
          f"score: {result['score']:.3f}")

    # Display the source image where this object was last seen
    img_path = f"data/frames/rgb_{result['source_image_id']:04d}.png"
    if os.path.exists(img_path):
        img = cv2.imread(img_path)
        cv2.imshow(f"Match: {q}", img)
        cv2.waitKey(0)
cv2.destroyAllWindows()

What does the query loop do?

For each query string, it calls voxel_map.query() which encodes the text with SigLIP and searches all voxels for the best dot-product match. The loop prints the matched voxel's 3D position and similarity score.

It also displays the RGB image associated with the winning voxel. This is the last frame your robot captured where it saw that object. Press any key to advance to the next query result.

  • Save query.py.

Before you run this, think about what you expect to see. You had tens of thousands of points before. Now you have a few hundred voxels. Do you think the query results will be faster, slower, or about the same?

  • Run the full query pipeline by running:
python query.py

You should see the compression ratio printed first, followed by three lines showing each query, its matched 3D coordinates, and a similarity score. An OpenCV window pops up for each query showing the source RGB image where the matched object was last seen. Press any key to advance through each result.

What should I see?

The voxel count should be in the hundreds (compared to tens of thousands of points). Each query returns a single 3D coordinate tightly localized on the real object. The source images should show the actual scene where the object appears.

Swap the query strings to match objects actually in your robot's environment. If your room has a "water bottle" instead of a "red cup", change the list. The system handles any natural language description.

Queries returning wrong objects?

  • Make sure the query strings match objects your robot actually captured during the scan. If the mug is not in any frame, SigLIP will match the closest thing it can find.
  • If the source images look wrong, verify that data/frames/ contains the RGB images from your capture step with filenames like rgb_0001.png. The image ID format must match.
  • Help me debug my voxel map query results.

Your robot now has a complete spatio-semantic memory. It can scan its environment, segment objects, encode them with SigLIP, store everything in a fast voxel grid, and answer "where is the coffee mug?" with a 3D coordinate and the source image to prove it. Next up, the Secret Mission will make this memory dynamic so it can handle objects that move.

Secret mission

Add Dynamic Memory Updates

Your memory system works, but it is frozen in time. What happens when someone moves the coffee mug to a different table? Right now, your robot has no idea. In this secret mission, you will add dynamic updates so the voxel memory evolves as the world changes around your Stretch 3.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally on your GPU machine, so there are no cloud services or ongoing costs.

Resources you used:

  • Project Python files (capture.py, backproject.py, siglip_encode.py, segment.py, query.py, voxel_map.py).
  • Saved RGB-D frames and point cloud data in data/frames/.
  • Cloned SAM 2 repository (installed in editable mode).
  • Cached SAM 2.1 model weights (~200MB, downloaded by HuggingFace Hub to ~/.cache/huggingface/hub/).
  • Cached SigLIP model weights (~1.5GB, downloaded by OpenCLIP to ~/.cache/clip/).
  • Python virtual environment or conda environment with installed packages (PyTorch, OpenCLIP, Open3D).

Keep everything running

No action needed. Choose this if you are still actively building on your spatio-semantic memory system.

  • All model checkpoints, project files, and saved frame data remain in place for continued development.
  • Your virtual environment stays ready so you can re-run scripts without reinstalling anything.
  • Cached model weights in ~/.cache/ will persist across sessions. They take up approximately 2GB of disk space total.

Pause - I'll come back to this later

Free up active resources while keeping your files for later. There are no running processes to stop since all scripts run as one-off commands.

  • If you are using a virtual environment, deactivate it by running:
deactivate
  • If you are using a conda environment, deactivate it by running:
conda deactivate
  • Your project files, saved frames, and cached models stay on disk and are ready when you come back.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each step below to clean up everything.

  • Delete your project directory containing all Python files and saved frame data. Run the following from the parent directory of your project (replace the folder name with your actual project directory):
# macOS/Linux
rm -rf your-project-directory

# Windows (Command Prompt)
rmdir /s /q your-project-directory
  • Delete the cloned SAM 2 repository. If you cloned it inside your project directory, the step above already removed it. Otherwise, navigate to where you cloned it and run:
# macOS/Linux
rm -rf sam2

# Windows (Command Prompt)
rmdir /s /q sam2
  • Delete the cached SAM 2.1 model weights downloaded by HuggingFace Hub:
# macOS/Linux
rm -rf ~/.cache/huggingface/hub/models--facebook--sam2.1-hiera-small

# Windows (Command Prompt)
rmdir /s /q %USERPROFILE%\.cache\huggingface\hub\models--facebook--sam2.1-hiera-small
  • Delete the cached SigLIP model weights downloaded by OpenCLIP:
# macOS/Linux
rm -rf ~/.cache/clip

# Windows (Command Prompt)
rmdir /s /q %USERPROFILE%\.cache\clip
  • Delete your Python virtual environment. If you used venv, delete the environment folder:
# macOS/Linux (replace 'venv' with your environment folder name)
rm -rf venv

# Windows (Command Prompt)
rmdir /s /q venv
  • If you used a conda environment instead, remove it by running:
conda remove --name your-env-name --all

Nice Work!

Nice Work!

Nice work! You just built a full spatio-semantic memory system that turns your Stretch 3 into a robot that understands what objects are in its environment and where they are, queryable with plain English.

You've learned how to:

  • Build a spatio-semantic voxel memory from raw RGB-D frames, using SigLIP through OpenCLIP to embed both images and text in the same latent space for open-vocabulary object queries.
  • Integrate SAM 2.1 automatic segmentation to produce clean per-object semantic features instead of noisy per-patch encodings, dramatically improving query accuracy.
  • Construct a DynaMem-inspired VoxelMap that stores per-voxel semantic features, observation counts, source image IDs, and timestamps for fast 3D object querying over hundreds of voxels instead of tens of thousands of points.
  • Secret Mission: Add dynamic memory updates with stale voxel removal and re-observation checks, so the memory stays current as objects move in the real world.

Ready to quiz yourself?