Real-Time Tabletop Object Segmentation
Detect and cluster 3D objects on a table using RealSense and Open3D.
Introduction
30 Second Summary
Every robot that picks up objects from a table needs to answer a fundamental question first: what is on the table, and where exactly is each thing? This is the core perception challenge in robotics manipulation.
In this project, you will build a real-time 3D perception system that captures depth data from an Intel RealSense camera, detects the table surface mathematically, and isolates individual objects sitting on it.
What You'll Build
A three-module Python application that streams live RGBD data from an Intel RealSense camera, segments the tabletop plane, clusters individual objects, and renders everything in a real-time 3D viewer with color-coded segments and bounding boxes.
By the end of this project, you'll have:
- A RealSense camera module that streams aligned RGBD frames and produces Open3D point clouds.
- A detection pipeline that segments the dominant table plane via RANSAC and isolates objects via DBSCAN clustering.
- A real-time 3D visualizer rendering the segmented table, color-coded object clusters, and axis-aligned bounding boxes at interactive rates.
- Secret Mission: Replace axis-aligned bounding boxes with tighter oriented bounding boxes and print each object's estimated physical dimensions to the terminal.
Are there any prerequisites?
You need an Intel RealSense depth camera physically connected to your machine via USB. You also need Python programming proficiency and basic familiarity with 3D geometry concepts like planes, normals, and distance metrics.
Set Up the Project and Verify the Camera
Before you can segment objects on a table, you need a working connection to your Intel RealSense camera and the right processing libraries installed.
In this step, you will initialize a Python project with uv, install Open3D and pyrealsense2, and confirm the camera is streaming data.
In this step, get ready to:
- Initialize a Python project and install the three core dependencies.
- Verify the RealSense camera is detected and reporting device info.
- Create the empty module files for the application.
Initialize the uv project and install dependencies
The uv package manager creates an isolated virtual environment and locks dependency versions for reproducibility.
- Open a terminal by pressing Ctrl+Alt+T.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir realsense-tabletop && cd realsense-tabletop- Confirm the folder was created by running:
pwdYou should see ~/Desktop/realsense-tabletop in the output.
- Initialize the uv project by running:
uv initWhat does this command do?
uv init creates a pyproject.toml (your project's configuration file), a .python-version file (pinning Python 3.12), and a README.md.
- Install the three core libraries by running:
uv add open3d pyrealsense2 numpyWhat are these libraries?
- open3d provides 3D point cloud processing, visualization, and geometry algorithms.
- pyrealsense2 is the Python wrapper for Intel's RealSense SDK, giving you access to depth and color streams.
- numpy handles the numerical array operations needed to manipulate point data.
- Verify the virtual environment was created and dependencies are locked by running:
ls .venv/
cat uv.lock | head -20You should see the contents of the .venv/ directory listed, followed by the first 20 lines of your uv.lock file showing resolved package versions.
Seeing a resolution error?
- Make sure you have an active internet connection. Package installation requires downloading from PyPI.
- If you see a version conflict, check that your Python version is 3.12 by running python3 --version.
Help me troubleshoot my uv add errors for open3d and pyrealsense2 on Python 3.12
Verify the RealSense camera is detected
Before writing the full camera module, a quick sanity check confirms the hardware is connected and responding to the SDK.
- Create a file called test_camera.py by running:
touch test_camera.py- Open test_camera.py in your preferred text editor and paste the following code:
"""Quick sanity check to verify RealSense camera connectivity."""
import pyrealsense2 as rs
def main():
ctx = rs.context()
devices = ctx.query_devices()
if len(devices) == 0:
print("[ERROR] No RealSense device detected. Check USB connection.")
return
for dev in devices:
print(f"[OK] Found: {dev.get_info(rs.camera_info.name)}")
print(f" Serial: {dev.get_info(rs.camera_info.serial_number)}")
print(f" Firmware: {dev.get_info(rs.camera_info.firmware_version)}")
if __name__ == "__main__":
main()What does this code do?
- rs.context() creates a connection to the RealSense SDK runtime.
- ctx.query_devices() returns a list of all connected RealSense cameras.
- For each detected device, it prints the camera model name, serial number, and firmware version to confirm hardware communication is working.
- Save test_camera.py.
- Run the camera check by executing:
uv run python test_camera.pyYou should see the camera model name, serial number, and firmware version printed to the terminal.
Seeing 'No RealSense device detected'?
- Unplug the camera's USB cable and plug it back in. Use a USB 3.0 port (usually blue inside) for reliable detection.
- Make sure no other application is currently using the camera. Only one process can access the RealSense pipeline at a time.
- Check that the camera appears at the OS level by running lsusb and looking for an Intel entry.
Help me get my RealSense camera detected on Ubuntu
✔️ Awesome, I've got everything!
Great. Double check you have saved test_camera.py and the camera output shows your device info.
ⓧ I'd like to double check the full code
"""Quick sanity check to verify RealSense camera connectivity."""
import pyrealsense2 as rs
def main():
ctx = rs.context()
devices = ctx.query_devices()
if len(devices) == 0:
print("[ERROR] No RealSense device detected. Check USB connection.")
return
for dev in devices:
print(f"[OK] Found: {dev.get_info(rs.camera_info.name)}")
print(f" Serial: {dev.get_info(rs.camera_info.serial_number)}")
print(f" Firmware: {dev.get_info(rs.camera_info.firmware_version)}")
if __name__ == "__main__":
main()Create the project file layout
The application is split into three modules: one for camera access, one for detection logic, and one to tie them together in a live loop.
- Create the three empty module files by running:
touch camera.py detector.py app.py- Confirm the files exist by running:
lsYou should see app.py, camera.py, detector.py, and test_camera.py alongside pyproject.toml, uv.lock, and the .venv/ directory.
Why three separate files?
Splitting camera access, detection logic, and the main loop into separate modules keeps each file focused on one responsibility.
This makes it easy to swap components later. For example, you could replace the RealSense camera module with a simulated depth source without touching the detector.
Your project is ready to go. Next up, you will build the camera module that converts raw depth and color frames into 3D point clouds.
Build the Camera Module
Your project is initialized, dependencies are locked, and the camera is confirmed connected. Now it's time to build the module that actually talks to the hardware.
The RealSense camera produces two separate streams: a depth image (how far away each pixel is) and a color image (what each pixel looks like). Your camera.py module will fuse these two streams into a single 3D point cloud that downstream modules can process.
In this step, get ready to:
- Build the RealSenseCamera class with pipeline initialization and depth-color stream alignment.
- Implement the point cloud generation method that fuses RGBD frames into 3D geometry.
- Validate the module by capturing a live frame and confirming the point count.
Implement the RealSenseCamera class with pipeline initialization
A RealSense camera needs explicit configuration before it streams data. You need to tell it what resolution, format, and frame rate to use for each stream.
- Open camera.py in your editor and add the module header, imports, and class constructor:
"""
camera.py - Intel RealSense camera interface.
Handles pipeline initialization, depth-color frame alignment,
and conversion to Open3D point clouds.
"""
import numpy as np
import pyrealsense2 as rs
import open3d as o3d
class RealSenseCamera:
"""Manages the RealSense pipeline and produces aligned Open3D point clouds."""
def __init__(self, width=640, height=480, fps=30):
"""
Configure stream resolution and frame rate.
Args:
width: Frame width in pixels.
height: Frame height in pixels.
fps: Target frames per second for both streams.
"""
self.width = width
self.height = height
self.fps = fps
self.pipeline = rs.pipeline()
self.config = rs.config()
self.align = None
self.intrinsics = NoneWhat does this code do?
The constructor stores the desired resolution and frame rate, then creates two key pyrealsense2 objects.
- rs.pipeline() manages the camera's data flow. It handles connecting to the device, starting streams, and retrieving frames.
- rs.config() holds the stream settings you want (resolution, format, FPS). You pass it to the pipeline when you start.
- align and intrinsics are set to None for now. They get populated when you call start().
Now you need a method that actually activates the camera. The start() method configures both streams, starts the pipeline, and retrieves the camera's intrinsic parameters (the internal geometry of the lens).
- Add the start() method below the __init__ method:
def start(self):
"""
Start the RealSense pipeline with depth and color streams.
Configures depth (Z16) and color (RGB8) streams, starts the pipeline,
creates the alignment object, and retrieves camera intrinsics.
Raises:
RuntimeError: If the camera cannot be started (disconnected, in use, etc.)
"""
self.config.enable_stream(
rs.stream.depth, self.width, self.height, rs.format.z16, self.fps
)
self.config.enable_stream(
rs.stream.color, self.width, self.height, rs.format.rgb8, self.fps
)
profile = self.pipeline.start(self.config)
# Align depth frames to the color frame's viewport
self.align = rs.align(rs.stream.color)
# Store intrinsics for 3D back-projection
depth_stream = profile.get_stream(rs.stream.depth)
self.intrinsics = depth_stream.as_video_stream_profile().get_intrinsics()What does this code do?
- enable_stream() registers two streams: depth in Z16 format (16-bit unsigned integer for millimeters) and color in RGB8 format (standard 8-bit color).
- rs.align(rs.stream.color) creates an alignment processor. The depth and color sensors are physically offset on the camera, so each pixel's depth doesn't correspond to the same color pixel by default. Alignment remaps depth into the color frame's coordinate system.
- get_intrinsics() retrieves the camera's focal lengths (fx, fy) and principal point (ppx, ppy). These are needed later to back-project 2D pixels into 3D space.
- Save the file. You cannot verify yet because the class has no way to capture frames. The next chunk adds that.
Implement the get_pointcloud() method
With the pipeline running, each call to this method captures one synchronized depth+color frameset and converts it into an Open3D PointCloud object that the rest of your application can consume.
- Add the get_pointcloud() method below start():
def get_pointcloud(self):
"""
Capture one aligned frameset and convert to an Open3D PointCloud.
Returns:
o3d.geometry.PointCloud or None if a frame was dropped.
"""
frames = self.pipeline.wait_for_frames()
aligned_frames = self.align.process(frames)
depth_frame = aligned_frames.get_depth_frame()
color_frame = aligned_frames.get_color_frame()
if not depth_frame or not color_frame:
return None
# Convert frames to numpy arrays
depth_image = np.asanyarray(depth_frame.get_data())
color_image = np.asanyarray(color_frame.get_data())
# Wrap as Open3D images
depth_o3d = o3d.geometry.Image(depth_image)
color_o3d = o3d.geometry.Image(color_image)
# Create RGBD image (depth_scale=1000 converts mm to meters, truncate at 3m)
rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(
color_o3d,
depth_o3d,
depth_scale=1000.0,
depth_trunc=3.0,
convert_rgb_to_intensity=False,
)
# Build pinhole camera intrinsic from RealSense parameters
intrinsic = o3d.camera.PinholeCameraIntrinsic(
self.width,
self.height,
self.intrinsics.fx,
self.intrinsics.fy,
self.intrinsics.ppx,
self.intrinsics.ppy,
)
# Generate the colored point cloud
pcd = o3d.geometry.PointCloud.create_from_rgbd_image(rgbd, intrinsic)
return pcdWhat does this code do?
This method performs the full depth-to-3D conversion pipeline in five stages:
- wait_for_frames() blocks until a synchronized depth+color pair arrives from the sensor.
- self.align.process() warps the depth frame to match the color frame's viewport so every pixel lines up.
- The frames become NumPy arrays, then get wrapped as o3d.geometry.Image objects so Open3D can work with them.
- create_from_color_and_depth() fuses them into an RGBD image. depth_scale=1000.0 converts the raw millimeter values to meters. depth_trunc=3.0 discards anything beyond 3 meters to cut noise.
- create_from_rgbd_image() uses the stored camera intrinsics to back-project every pixel into a 3D point, producing the final colored point cloud.
The last pieces are a stop() method to release the hardware and a __main__ block so you can validate the module in isolation.
- Add the stop() method and the validation block below get_pointcloud():
def stop(self):
"""Stop the RealSense pipeline and release hardware resources."""
self.pipeline.stop()
if __name__ == "__main__":
# Quick validation: capture one frame and report point count
cam = RealSenseCamera()
try:
cam.start()
pcd = cam.get_pointcloud()
if pcd is not None:
print(f"[OK] Captured point cloud with {len(pcd.points)} points.")
else:
print("[WARN] Frame was dropped.")
except Exception as e:
print(f"[ERROR] {e}")
finally:
cam.stop()What does this code do?
- stop() cleanly shuts down the RealSense pipeline so the hardware is released for other processes.
- The __main__ block creates a camera, captures one frame, prints how many 3D points it contains, and shuts down. The try/finally ensures the pipeline is always stopped even if an error occurs.
- Save camera.py.
- Run the module to validate it captures a real frame by running:
uv run python camera.py- You should see output like [OK] Captured point cloud with 204800 points. (the exact number varies, but expect tens of thousands to hundreds of thousands for a 640x480 RGBD frame).
Seeing an error or zero points?
- If you see a RuntimeError about the device being unavailable, make sure no other application (like the RealSense Viewer) is using the camera. Only one process can access the pipeline at a time.
- If you see [WARN] Frame was dropped., run the script again. The first frame can occasionally be dropped while the sensor warms up.
- If the point count is very low (under 1000), check that objects are within 3 meters of the camera. Anything beyond that distance is truncated.
Why is my camera.py not producing a point cloud?
✔️ Awesome, I've got everything!
Great. Double check you've saved camera.py and confirmed a point count in the output.
ⓧ I'd like to double check the full code
"""
camera.py - Intel RealSense camera interface.
Handles pipeline initialization, depth-color frame alignment,
and conversion to Open3D point clouds.
"""
import numpy as np
import pyrealsense2 as rs
import open3d as o3d
class RealSenseCamera:
"""Manages the RealSense pipeline and produces aligned Open3D point clouds."""
def __init__(self, width=640, height=480, fps=30):
self.width = width
self.height = height
self.fps = fps
self.pipeline = rs.pipeline()
self.config = rs.config()
self.align = None
self.intrinsics = None
def start(self):
self.config.enable_stream(
rs.stream.depth, self.width, self.height, rs.format.z16, self.fps
)
self.config.enable_stream(
rs.stream.color, self.width, self.height, rs.format.rgb8, self.fps
)
profile = self.pipeline.start(self.config)
self.align = rs.align(rs.stream.color)
depth_stream = profile.get_stream(rs.stream.depth)
self.intrinsics = depth_stream.as_video_stream_profile().get_intrinsics()
def get_pointcloud(self):
frames = self.pipeline.wait_for_frames()
aligned_frames = self.align.process(frames)
depth_frame = aligned_frames.get_depth_frame()
color_frame = aligned_frames.get_color_frame()
if not depth_frame or not color_frame:
return None
depth_image = np.asanyarray(depth_frame.get_data())
color_image = np.asanyarray(color_frame.get_data())
depth_o3d = o3d.geometry.Image(depth_image)
color_o3d = o3d.geometry.Image(color_image)
rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(
color_o3d,
depth_o3d,
depth_scale=1000.0,
depth_trunc=3.0,
convert_rgb_to_intensity=False,
)
intrinsic = o3d.camera.PinholeCameraIntrinsic(
self.width,
self.height,
self.intrinsics.fx,
self.intrinsics.fy,
self.intrinsics.ppx,
self.intrinsics.ppy,
)
pcd = o3d.geometry.PointCloud.create_from_rgbd_image(rgbd, intrinsic)
return pcd
def stop(self):
self.pipeline.stop()
if __name__ == "__main__":
cam = RealSenseCamera()
try:
cam.start()
pcd = cam.get_pointcloud()
if pcd is not None:
print(f"[OK] Captured point cloud with {len(pcd.points)} points.")
else:
print("[WARN] Frame was dropped.")
except Exception as e:
print(f"[ERROR] {e}")
finally:
cam.stop()Your camera module is fully operational. It captures live RGBD data from the hardware and produces clean Open3D point clouds. Next up, you will build the detector module that finds the table surface and clusters individual objects sitting on it.
Build the Detector Module
Your camera module is streaming aligned RGBD frames and converting them to 3D point clouds. Now you need to make sense of those clouds.
A raw point cloud contains hundreds of thousands of points with no structure. The detector module will identify the table surface mathematically, filter out everything below it, and cluster the remaining points into individual objects. This is the core perception pipeline that turns raw sensor data into actionable scene understanding.
In this step, get ready to:
- Implement voxel downsampling and RANSAC plane segmentation to detect the table.
- Filter above-table points and cluster them into individual objects with DBSCAN.
- Validate the detector against a synthetic point cloud.
Implement voxel downsampling and RANSAC plane segmentation
A 640x480 depth frame produces up to 300,000 points. Processing all of them each frame would be too slow for real-time use. Voxel downsampling divides space into a 3D grid and keeps one representative point per cell, reducing the cloud to a manageable size while preserving its shape.
Once downsampled, RANSAC (Random Sample Consensus) finds the dominant plane in the cloud. It repeatedly picks 3 random points, fits a plane through them, counts how many other points lie close to that plane, and keeps the best fit after many iterations.
- Open detector.py in your editor and add the module header, imports, and class skeleton with configurable parameters:
"""
detector.py - 3D point cloud processing pipeline.
Handles voxel downsampling, RANSAC plane segmentation for tabletop detection,
and DBSCAN clustering for individual object isolation.
"""
import numpy as np
import open3d as o3d
class TableTopDetector:
"""Segments a tabletop surface and clusters objects above it."""
def __init__(
self,
voxel_size=0.005,
ransac_distance=0.01,
ransac_n=3,
ransac_iterations=1000,
dbscan_eps=0.02,
dbscan_min_points=10,
height_margin=0.01,
):
self.voxel_size = voxel_size
self.ransac_distance = ransac_distance
self.ransac_n = ransac_n
self.ransac_iterations = ransac_iterations
self.dbscan_eps = dbscan_eps
self.dbscan_min_points = dbscan_min_points
self.height_margin = height_marginWhat do these parameters mean?
- voxel_size=0.005 sets a 5mm grid for downsampling. Smaller means more detail but slower processing.
- ransac_distance=0.01 means a point within 1cm of the fitted plane counts as a table inlier.
- ransac_n=3 is the minimum number of points needed to define a plane (always 3 in 3D).
- dbscan_eps=0.02 and dbscan_min_points=10 control clustering sensitivity. Points within 2cm of each other can form a cluster, and a cluster needs at least 10 points.
- height_margin=0.01 rejects points within 1cm of the table surface, filtering noise at the boundary.
- Below the __init__ method, add the first half of the process() method that handles downsampling and plane segmentation:
def process(self, pcd):
if pcd is None or pcd.is_empty():
return None, [], []
# Reduce point count while preserving shape
pcd_down = pcd.voxel_down_sample(voxel_size=self.voxel_size)
if len(pcd_down.points) < 100:
return None, [], []
# RANSAC returns plane_model [a, b, c, d] for ax + by + cz + d = 0
plane_model, inliers = pcd_down.segment_plane(
distance_threshold=self.ransac_distance,
ransac_n=self.ransac_n,
num_iterations=self.ransac_iterations,
)
if len(inliers) == 0:
return None, [], []
# Split cloud into table (inliers) and everything else
table_pcd = pcd_down.select_by_index(inliers)
remaining_pcd = pcd_down.select_by_index(inliers, invert=True)
# Paint table green for visual distinction
table_pcd.paint_uniform_color([0.0, 0.8, 0.0])
if remaining_pcd.is_empty():
return table_pcd, [], []What does this code do?
The process() method takes a raw point cloud and returns three things: the table surface, a list of object point clouds, and a list of bounding boxes.
segment_plane() returns a plane equation [a, b, c, d] and the indices of points that lie on the plane. select_by_index() splits the cloud into table points (inliers) and non-table points (invert=True). The early-return guards handle degenerate cases gracefully.
- Save detector.py.
Why RANSAC instead of other plane detection methods?
RANSAC is the standard approach for dominant plane detection in unstructured point clouds. It requires no surface normals (avoiding an extra computation step), handles noisy sensor data gracefully, and works regardless of the table's orientation in the scene.
Filter above-table points and run DBSCAN clustering
With the table identified, you now need to isolate the objects sitting on top of it. The plane equation gives you a mathematical test: any point whose signed distance to the plane is positive (and above a minimum margin) is above the table.
Once filtered, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) groups nearby points into clusters without knowing how many objects exist in advance. Each cluster becomes one detected object.
- In detector.py, add the following code directly below the last line of the process() method (after return table_pcd, [], []). Keep the same indentation level inside the method:
# Compute signed distance from each point to the table plane
a, b, c, d = plane_model
points = np.asarray(remaining_pcd.points)
distances = a * points[:, 0] + b * points[:, 1] + c * points[:, 2] + d
# Keep only points above the table by at least height_margin
above_mask = distances > self.height_margin
above_indices = np.where(above_mask)[0]
if len(above_indices) == 0:
return table_pcd, [], []
objects_pcd = remaining_pcd.select_by_index(above_indices.tolist())
if objects_pcd.is_empty():
return table_pcd, [], []
# Cluster nearby points into individual objects
labels = np.array(
objects_pcd.cluster_dbscan(
eps=self.dbscan_eps,
min_points=self.dbscan_min_points,
)
)
if len(labels) == 0:
return table_pcd, [], []
max_label = labels.max()
if max_label < 0:
# All points classified as noise
return table_pcd, [], []
# Assign random colors and bounding boxes per cluster
object_pcds = []
bboxes = []
colors = np.random.rand(max_label + 1, 3)
for i in range(max_label + 1):
cluster_indices = np.where(labels == i)[0]
cluster_pcd = objects_pcd.select_by_index(cluster_indices.tolist())
cluster_pcd.paint_uniform_color(colors[i].tolist())
bbox = cluster_pcd.get_axis_aligned_bounding_box()
bbox.color = colors[i].tolist()
object_pcds.append(cluster_pcd)
bboxes.append(bbox)
return table_pcd, object_pcds, bboxesWhat does this code do?
- The signed distance formula ax + by + cz + d tells you which side of the plane a point is on. Positive means above the table.
- cluster_dbscan() returns a label array where each point gets a cluster number. Label -1 means noise (isolated points that don't belong to any object).
- Each cluster gets a random color via paint_uniform_color() and an axis-aligned bounding box that shares the same color for easy visual matching.
- Save detector.py.
Why DBSCAN instead of k-means?
DBSCAN does not require you to specify the number of clusters in advance. Since you never know how many objects are on the table, this is essential. It also naturally handles noise points (isolated outliers that don't belong to any object) by labeling them -1.
Seeing a syntax error?
- Make sure the new code is indented at the same level as the rest of process() (8 spaces or 2 tabs inside the class).
- Check that the code appears AFTER the if remaining_pcd.is_empty(): return table_pcd, [], [] block, not inside it.
Help me debug a syntax error in my detector.py process() method.
Validate the detector with a synthetic test
Before connecting this to the live camera, you can verify the logic works correctly using a synthetic point cloud. You will create a flat plane (simulating the table) plus two small clusters above it (simulating objects) and confirm the detector finds exactly 2 objects.
- At the bottom of detector.py (outside the class, at the top indentation level), add this test block:
if __name__ == "__main__":
print("[TEST] Creating synthetic point cloud...")
# Flat table at z=0 with slight noise
table_points = np.random.rand(5000, 3) * [1.0, 1.0, 0.0]
table_points[:, 2] = np.random.normal(0, 0.002, 5000)
# Two object clusters above the table
obj1 = np.random.rand(200, 3) * 0.05 + [0.3, 0.3, 0.1]
obj2 = np.random.rand(200, 3) * 0.05 + [0.7, 0.7, 0.15]
all_points = np.vstack([table_points, obj1, obj2])
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(all_points)
detector = TableTopDetector()
table, objects, boxes = detector.process(pcd)
print(f"[OK] Table points: {len(table.points) if table else 0}")
print(f"[OK] Objects detected: {len(objects)}")
for i, bbox in enumerate(boxes):
extent = bbox.get_extent()
print(f" Object {i+1}: {extent[0]*100:.1f} x {extent[1]*100:.1f} x {extent[2]*100:.1f} cm")What does this test do?
It constructs 5,000 points spread flat near z=0 (the synthetic table) plus two clusters of 200 points each placed at different positions above the surface. This mimics a real scene without needing the camera.
The bounding box get_extent() returns the width, depth, and height in meters. Multiplying by 100 converts to centimeters for readable output.
- Save detector.py.
- Run the detector test by executing this command:
uv run python detector.py- You should see output confirming 2 objects were detected, along with their approximate dimensions in centimeters (roughly 5 x 5 x 5 cm each since the synthetic clusters are 0.05m cubes).
Seeing 0 objects detected?
- Check that the if __name__ == "__main__": block is at the top level (no indentation), not inside the class.
- Verify that the above_mask = distances > self.height_margin line uses > (greater than), not <. The sign determines which side of the plane is "above."
- Make sure the object clusters use z-values above 0 (e.g. 0.1 and 0.15) so they sit above the synthetic table at z=0.
Help me debug why my detector finds 0 objects in the synthetic test.
✔️ Awesome, I've got everything!
Great. Double check you have saved detector.py and your test reports 2 objects detected.
ⓧ I'd like to double check the full code
Here is the complete detector.py file for comparison:
"""
detector.py - 3D point cloud processing pipeline.
Handles voxel downsampling, RANSAC plane segmentation for tabletop detection,
and DBSCAN clustering for individual object isolation.
"""
import numpy as np
import open3d as o3d
class TableTopDetector:
"""Segments a tabletop surface and clusters objects above it."""
def __init__(
self,
voxel_size=0.005,
ransac_distance=0.01,
ransac_n=3,
ransac_iterations=1000,
dbscan_eps=0.02,
dbscan_min_points=10,
height_margin=0.01,
):
self.voxel_size = voxel_size
self.ransac_distance = ransac_distance
self.ransac_n = ransac_n
self.ransac_iterations = ransac_iterations
self.dbscan_eps = dbscan_eps
self.dbscan_min_points = dbscan_min_points
self.height_margin = height_margin
def process(self, pcd):
if pcd is None or pcd.is_empty():
return None, [], []
pcd_down = pcd.voxel_down_sample(voxel_size=self.voxel_size)
if len(pcd_down.points) < 100:
return None, [], []
plane_model, inliers = pcd_down.segment_plane(
distance_threshold=self.ransac_distance,
ransac_n=self.ransac_n,
num_iterations=self.ransac_iterations,
)
if len(inliers) == 0:
return None, [], []
table_pcd = pcd_down.select_by_index(inliers)
remaining_pcd = pcd_down.select_by_index(inliers, invert=True)
table_pcd.paint_uniform_color([0.0, 0.8, 0.0])
if remaining_pcd.is_empty():
return table_pcd, [], []
a, b, c, d = plane_model
points = np.asarray(remaining_pcd.points)
distances = a * points[:, 0] + b * points[:, 1] + c * points[:, 2] + d
above_mask = distances > self.height_margin
above_indices = np.where(above_mask)[0]
if len(above_indices) == 0:
return table_pcd, [], []
objects_pcd = remaining_pcd.select_by_index(above_indices.tolist())
if objects_pcd.is_empty():
return table_pcd, [], []
labels = np.array(
objects_pcd.cluster_dbscan(
eps=self.dbscan_eps,
min_points=self.dbscan_min_points,
)
)
if len(labels) == 0:
return table_pcd, [], []
max_label = labels.max()
if max_label < 0:
return table_pcd, [], []
object_pcds = []
bboxes = []
colors = np.random.rand(max_label + 1, 3)
for i in range(max_label + 1):
cluster_indices = np.where(labels == i)[0]
cluster_pcd = objects_pcd.select_by_index(cluster_indices.tolist())
cluster_pcd.paint_uniform_color(colors[i].tolist())
bbox = cluster_pcd.get_axis_aligned_bounding_box()
bbox.color = colors[i].tolist()
object_pcds.append(cluster_pcd)
bboxes.append(bbox)
return table_pcd, object_pcds, bboxes
if __name__ == "__main__":
print("[TEST] Creating synthetic point cloud...")
table_points = np.random.rand(5000, 3) * [1.0, 1.0, 0.0]
table_points[:, 2] = np.random.normal(0, 0.002, 5000)
obj1 = np.random.rand(200, 3) * 0.05 + [0.3, 0.3, 0.1]
obj2 = np.random.rand(200, 3) * 0.05 + [0.7, 0.7, 0.15]
all_points = np.vstack([table_points, obj1, obj2])
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(all_points)
detector = TableTopDetector()
table, objects, boxes = detector.process(pcd)
print(f"[OK] Table points: {len(table.points) if table else 0}")
print(f"[OK] Objects detected: {len(objects)}")
for i, bbox in enumerate(boxes):
extent = bbox.get_extent()
print(f" Object {i+1}: {extent[0]*100:.1f} x {extent[1]*100:.1f} x {extent[2]*100:.1f} cm")Your detector module now handles the full perception pipeline: downsampling, plane segmentation, above-table filtering, and object clustering. Next up, you will wire the camera and detector together into a real-time visualization loop.
Build the Application Loop
You now have two independent modules: a camera that produces 3D point clouds and a detector that segments them into table + objects.
The missing piece is the real-time glue. You need a main loop that continuously grabs frames from the camera, feeds them through the detector, and pushes results into a live 3D viewer. The viewer must update without blocking the processing loop, so you get smooth interactive rates.
In this step, get ready to:
- Set up Open3D's non-blocking visualizer with component initialization.
- Implement the real-time capture-process-render loop.
- Handle graceful shutdown on Ctrl+C.
Set up the non-blocking Open3D visualizer
Open3D offers two visualization modes. The simple draw_geometries() call is blocking and only shows a static snapshot. For real-time updates, you need the Visualizer class which lets you poll events and push new data each frame without freezing the loop.
- In app.py, add the module header, imports, and initialization section by copying this code:
"""
app.py - Main execution loop.
Integrates camera and detector modules with Open3D's non-blocking
visualizer for real-time tabletop object segmentation.
"""
import sys
import numpy as np
import open3d as o3d
from camera import RealSenseCamera
from detector import TableTopDetector
def main():
"""Run the real-time tabletop segmentation pipeline."""
# --- Initialize Components ---
camera = RealSenseCamera(width=640, height=480, fps=30)
detector = TableTopDetector()
# Attempt to start the camera with graceful error handling
try:
camera.start()
print("[INFO] RealSense camera started successfully.")
except Exception as e:
print(f"[ERROR] Failed to start RealSense camera: {e}")
print("[ERROR] Please check that the camera is connected and not in use.")
sys.exit(1)
# --- Set Up Non-Blocking Visualizer ---
vis = o3d.visualization.Visualizer()
vis.create_window(window_name="Tabletop Object Segmentation", width=1280, height=720)
# Configure rendering
render_opt = vis.get_render_option()
render_opt.point_size = 2.0
render_opt.background_color = np.asarray([0.1, 0.1, 0.1])
# Track geometries currently in the scene for clean removal
current_geometries = []
first_frame = True
print("[INFO] Starting real-time processing. Press Ctrl+C to quit.")What does this code do?
- RealSenseCamera(width=640, height=480, fps=30) creates the camera interface using the same resolution your module was built for.
- The try/except around camera.start() catches connection failures (unplugged USB, camera in use by another process) and exits cleanly with sys.exit(1) instead of crashing with a traceback.
- The Visualizer opens a 1280x720 window with a dark background ([0.1, 0.1, 0.1]) so colored point clouds stand out clearly.
- current_geometries tracks what is currently displayed so you can remove old data each frame. first_frame tells the visualizer to auto-fit the camera on the very first render.
Why non-blocking instead of draw_geometries?
Open3D's draw_geometries() halts your Python process until you close the window. That works for inspecting a single snapshot, but a real-time perception pipeline needs to keep grabbing new frames while the window stays open.
The Visualizer class with poll_events() and update_renderer() gives you frame-by-frame control. Your loop stays in charge.
Implement the real-time processing loop and graceful shutdown
The main loop captures a frame, runs segmentation, and swaps the displayed geometries every iteration. You also need clean shutdown handling so the RealSense pipeline and OpenGL window release properly on exit.
- Directly below the code you just added (still inside main()), add the processing loop, shutdown handler, and entry point:
# --- Main Processing Loop ---
try:
while True:
# Capture a point cloud (handle transient frame failures)
try:
pcd = camera.get_pointcloud()
except Exception as e:
print(f"[WARN] Frame capture failed: {e}. Retrying...")
vis.poll_events()
vis.update_renderer()
continue
# If no valid frame, keep the visualizer alive and retry
if pcd is None or pcd.is_empty():
vis.poll_events()
vis.update_renderer()
continue
# Run segmentation
table_pcd, object_pcds, bboxes = detector.process(pcd)
# --- Update Visualizer ---
# Remove previously displayed geometries
for geo in current_geometries:
vis.remove_geometry(geo, reset_bounding_box=False)
current_geometries.clear()
# Add new geometries
new_geometries = []
if table_pcd is not None and not table_pcd.is_empty():
new_geometries.append(table_pcd)
new_geometries.extend(object_pcds)
new_geometries.extend(bboxes)
for geo in new_geometries:
vis.add_geometry(geo, reset_bounding_box=first_frame)
current_geometries = new_geometries
if first_frame and len(new_geometries) > 0:
first_frame = False
# Non-blocking render update
vis.poll_events()
vis.update_renderer()
except KeyboardInterrupt:
print("\n[INFO] Shutting down...")
finally:
camera.stop()
vis.destroy_window()
print("[INFO] Cleanup complete. Goodbye.")
if __name__ == "__main__":
main()What does this code do?
The loop has three main phases each iteration:
- Capture: get_pointcloud() is wrapped in its own try/except so a single dropped frame prints a warning and retries instead of crashing the whole app.
- Process: the detector returns the green table cloud, colored object clusters, and bounding boxes. If the frame is None or empty, the loop skips processing but still calls poll_events() to keep the window responsive.
- Render: all old geometries are removed (with reset_bounding_box=False so the camera viewpoint stays stable), then new ones are added. Only the first successful frame resets the bounding box to auto-fit the view.
The except KeyboardInterrupt catches Ctrl+C, and the finally block ensures the RealSense pipeline and OpenGL window always release cleanly.
- Save app.py.
Seeing an import error?
- Make sure camera.py and detector.py are in the same directory as app.py. The from camera import RealSenseCamera line looks for files in the current directory.
- Check that you run with uv run (not bare python) so the virtual environment with Open3D and pyrealsense2 is active.
Why can't Python find my camera or detector module?
Run the full application
Everything is wired together. Time to see the system in action.
- Launch the application by running:
uv run python app.py- An Open3D window titled Tabletop Object Segmentation should appear showing a live 3D point cloud.
- The table surface appears as a green plane.
- Objects on the table appear as distinct randomly-colored clusters with bounding boxes around them.
- You can click and drag in the window to rotate the 3D view.
- Press Ctrl+C in the terminal to exit cleanly.
Window opens but no point cloud appears?
- Verify the camera is working by running uv run python camera.py again. If it reports 0 points, the camera connection has dropped.
- Make sure the camera has a clear view of a table with objects. If the table is too far away (beyond 3 meters), all depth data is truncated.
- If you see [WARN] Frame capture failed repeating, try unplugging and re-plugging the camera USB cable.
My Open3D window is black or the point cloud isn't showing up. What should I check?
Only seeing a green plane, no objects?
- Place some objects on the table. They need to be at least 1cm above the table surface (the height_margin filter) and large enough for DBSCAN to cluster (10+ points after downsampling).
- Very flat objects like a sheet of paper may fall within the 1cm margin and get classified as part of the table.
✔️ Awesome, I've got everything!
Great. Double check you have saved app.py and that the visualization runs as expected.
ⓧ I'd like to double check the full code
Here is the complete app.py file:
"""
app.py - Main execution loop.
Integrates camera and detector modules with Open3D's non-blocking
visualizer for real-time tabletop object segmentation.
"""
import sys
import numpy as np
import open3d as o3d
from camera import RealSenseCamera
from detector import TableTopDetector
def main():
"""Run the real-time tabletop segmentation pipeline."""
camera = RealSenseCamera(width=640, height=480, fps=30)
detector = TableTopDetector()
try:
camera.start()
print("[INFO] RealSense camera started successfully.")
except Exception as e:
print(f"[ERROR] Failed to start RealSense camera: {e}")
print("[ERROR] Please check that the camera is connected and not in use.")
sys.exit(1)
vis = o3d.visualization.Visualizer()
vis.create_window(window_name="Tabletop Object Segmentation", width=1280, height=720)
render_opt = vis.get_render_option()
render_opt.point_size = 2.0
render_opt.background_color = np.asarray([0.1, 0.1, 0.1])
current_geometries = []
first_frame = True
print("[INFO] Starting real-time processing. Press Ctrl+C to quit.")
try:
while True:
try:
pcd = camera.get_pointcloud()
except Exception as e:
print(f"[WARN] Frame capture failed: {e}. Retrying...")
vis.poll_events()
vis.update_renderer()
continue
if pcd is None or pcd.is_empty():
vis.poll_events()
vis.update_renderer()
continue
table_pcd, object_pcds, bboxes = detector.process(pcd)
for geo in current_geometries:
vis.remove_geometry(geo, reset_bounding_box=False)
current_geometries.clear()
new_geometries = []
if table_pcd is not None and not table_pcd.is_empty():
new_geometries.append(table_pcd)
new_geometries.extend(object_pcds)
new_geometries.extend(bboxes)
for geo in new_geometries:
vis.add_geometry(geo, reset_bounding_box=first_frame)
current_geometries = new_geometries
if first_frame and len(new_geometries) > 0:
first_frame = False
vis.poll_events()
vis.update_renderer()
except KeyboardInterrupt:
print("\n[INFO] Shutting down...")
finally:
camera.stop()
vis.destroy_window()
print("[INFO] Cleanup complete. Goodbye.")
if __name__ == "__main__":
main()Secret mission
Oriented Bounding Boxes with Physical Dimensions
Your axis-aligned bounding boxes work, but they waste space. A coffee mug rotated 45 degrees gets a box that's much larger than the mug itself. Can you swap in tighter oriented bounding boxes and print each object's real-world dimensions in centimeters to the terminal?
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally using open-source libraries and your own hardware. There are no ongoing costs or cloud resources to worry about.
Decide whether to keep your project files for further experimentation or delete them entirely.
Resources you used:
- The realsense-tabletop project directory (contains all Python files, pyproject.toml, uv.lock, and the .venv/ virtual environment).
Keep everything running
No action needed. Choose this if you want to keep experimenting with segmentation parameters, try different voxel sizes, or test with new objects on your table.
- Your project files stay in the realsense-tabletop folder on your Desktop.
- The virtual environment in .venv/ keeps all dependencies isolated from your system Python.
- Run the app again anytime with uv run python app.py from inside the project directory.
Pause - I'll come back to this later
Since this project has no background processes or cloud services, pausing just means stopping the application if it is currently running.
- If the app is running, press Ctrl+C in the terminal to shut it down.
- Your project files and virtual environment will remain on disk until you decide to delete them.
Delete - I don't want to use this again
Remove the entire project directory. The virtual environment is self-contained inside the project folder, so everything gets cleaned up in one step.
- If the app is running, press Ctrl+C to stop it first.
- Navigate to your Desktop and remove the project folder by running:
cd ~/Desktop
rm -rf realsense-tabletop- Confirm the folder is gone by running:
ls ~/Desktop | grep realsenseYou should see no output, confirming the directory has been removed.
Nice Work!
Nice Work!
You just built a real-time 3D perception system from scratch. Your application streams live depth data from a RealSense camera, mathematically isolates the table surface, and identifies individual objects sitting on it.
What you learned:
- Built a modular RealSense camera interface that streams aligned RGBD data and converts it to Open3D point clouds.
- Implemented RANSAC plane segmentation to isolate the tabletop surface and DBSCAN density clustering to detect individual objects with randomized color coding and axis-aligned bounding boxes.
- Integrated camera and detector modules into a non-blocking real-time visualizer running at interactive frame rates with graceful shutdown handling.
- Secret Mission: Upgraded to oriented bounding boxes and printed real-world object dimensions in centimeters to the terminal.
Ready to quiz yourself?