Tabletop Object Detection with Open3D

Build a 3D perception pipeline to detect objects on a table using RANSAC and DBSCAN.

Introduction

30 Second Summary

Imagine a robot arm staring at a cluttered table, needing to figure out which objects are there and exactly where to grab them. That perception challenge is one of the hardest problems in robotics.

In this project, you will build a Python pipeline that takes raw 3D point cloud data and detects individual objects sitting on a table, outputting their exact position and size.

What You'll Build

A complete 3D object detection pipeline using Open3D that processes point cloud data through noise removal, table surface detection, and object clustering to produce bounding boxes for each detected object.

By the end of this project, you'll have:

  • A synthetic scene generator that simulates depth camera output from a robotic arm mounted above a table.
  • A preprocessing module that removes sensor noise and downsamples the point cloud for efficient processing.
  • A RANSAC-based table detector and DBSCAN-based object clusterer that segments the scene into individual objects with 3D bounding boxes.
  • Secret Mission: Add oriented bounding boxes to estimate object rotation for grasp planning.

Are there any prerequisites?

You should be comfortable with Python programming and NumPy array operations. Basic linear algebra knowledge (vectors and matrices) will help you understand the algorithms, but the project explains each one as you go.

Before We Start

Before We Start...

Set Up Open3D and Your Project

Your perception pipeline needs a library that can work with 3D point clouds. Open3D is the go-to Python library for this. It provides the algorithms you need to process, segment, and visualize 3D data from depth cameras.

In this step, you will create a project directory, install Open3D and its supporting libraries, and verify that 3D visualization works on your machine.

In this step, get ready to:
  • Create a project directory with a Python virtual environment.
  • Install Open3D, NumPy, and Matplotlib.
  • Confirm visualization works by rendering a test sphere.
Create a project directory and virtual environment

A virtual environment keeps this project's dependencies isolated from the rest of your system. This prevents version conflicts if you have other Python projects.

  • Open your terminal by pressing Ctrl+Alt+T.
  • Create the project folder and move into it by running these commands:
mkdir ~/perception_pipeline
cd ~/perception_pipeline
  • Create a Python virtual environment by running:
python3 -m venv venv
  • Activate the virtual environment by running:
source venv/bin/activate

You should see (venv) appear at the beginning of your terminal prompt. This confirms the virtual environment is active.

What is a virtual environment?

A Python virtual environment is an isolated folder that holds its own copy of pip and installed packages. When you activate it, any pip install command installs packages only into that folder, not system-wide.

This means you can have different versions of libraries for different projects without them interfering with each other.

Don't see (venv) in your prompt?

Make sure you ran the source venv/bin/activate command from inside the ~/perception_pipeline directory. If you are in a different folder, use the full path:

source ~/perception_pipeline/venv/bin/activate

Help me activate my Python virtual environment.

  • Install the project dependencies by running:
pip install open3d==0.19.0 numpy matplotlib

What does this command install?

  • open3d==0.19.0 is the 3D processing library that provides point cloud algorithms and visualization.
  • numpy provides fast array operations for generating and manipulating 3D coordinates.
  • matplotlib provides colormaps you will use later to color-code detected objects.
  • Verify Open3D installed correctly by running:
python -c "import open3d as o3d; print(o3d.__version__)"

✔️ I see 0.19.0

Open3D is installed and ready. Move on to creating your project files.

ⓧ I see an error

If you see ModuleNotFoundError, make sure your virtual environment is active. You should see (venv) in your prompt.

  • Re-activate the environment by running:
source ~/perception_pipeline/venv/bin/activate
  • Then retry the install command:
pip install open3d==0.19.0 numpy matplotlib

If you see a compilation or compatibility error, confirm you are running Python 3.10 by running python3 --version. Open3D 0.19.0 supports Python 3.8 through 3.12.

Create the project file structure

Your pipeline will live in two files. One generates synthetic point cloud data. The other runs the detection algorithms.

  • Make sure you are inside the ~/perception_pipeline directory.
  • Create both project files by running:
touch generate_scene.py tabletop_detection.py
  • Confirm the files exist by running:
ls

You should see generate_scene.py, tabletop_detection.py, and the venv folder listed in the output.

  • Open the ~/perception_pipeline folder in your IDE so you can see both files in the file explorer.

What are these files for?

  • generate_scene.py will create a synthetic point cloud that simulates what a depth camera mounted on your robot arm would capture.
  • tabletop_detection.py will contain the full detection pipeline: preprocessing, table segmentation, object clustering, and bounding box computation.

Files not showing in your IDE?

Make sure you opened the ~/perception_pipeline folder itself, not a parent directory. In VS Code, use File > Open Folder and navigate to perception_pipeline in your home directory.

Help me open my project folder in my IDE.

Test Open3D visualization

Open3D opens a GUI window to display 3D geometry. Before building the full pipeline, confirm that your display setup can render a 3D object.

  • Run this visualization test in your terminal:
python -c "
import open3d as o3d
mesh = o3d.geometry.TriangleMesh.create_sphere(radius=1.0)
mesh.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh], window_name='Open3D Test')
"

A window should open showing a smooth 3D sphere. You can rotate it by clicking and dragging. Close the window to return to your terminal.

What does this test do?

This creates a simple sphere mesh and renders it in an Open3D viewer window. If the window appears and displays a sphere, your graphics system is ready to handle the 3D visualizations you will use throughout this project.

No window appeared?

If you are running over SSH, Open3D visualization requires X11 forwarding or a local display. Reconnect with ssh -X to enable X11 forwarding, or run the project directly on your local Ubuntu machine.

If you see a GLX or display error, make sure your desktop environment is running and the DISPLAY environment variable is set.

Help me fix Open3D visualization on my Ubuntu setup.

Your environment is ready to go. Next up, you will generate a synthetic tabletop scene that simulates what a depth camera would capture in front of your robot arm.

Generate a Synthetic Tabletop Scene

Your environment is ready with Open3D installed and both project files created. Now you need data to work with.

A real depth camera mounted on a robot arm outputs thousands of 3D points representing everything in its field of view. In this step, you will write a scene generator that produces the same kind of raw data programmatically. This gives you full control over ground truth so you can verify your detection pipeline later.

In this step, get ready to:
  • Write a function that generates a synthetic tabletop scene with three objects and noise.
  • Visualize the raw point cloud to see what depth camera output looks like.
  • Save the scene to a PCD file for the detection pipeline to consume.
Build the scene generator

The generator creates points for a flat table surface plus three objects sitting on it (a cup, a book, and a ball). It also sprinkles random noise to simulate sensor artifacts from a real camera.

  • Open generate_scene.py in your IDE and paste the following code to start the scene generator:
import open3d as o3d
import numpy as np


def generate_tabletop_scene():
    """Generate a point cloud simulating a depth camera view of a table with objects."""
    np.random.seed(42)

    # Table surface: flat plane at z=0 with slight sensor noise
    n_table = 5000
    table_x = np.random.uniform(-0.4, 0.4, n_table)
    table_y = np.random.uniform(-0.4, 0.4, n_table)
    table_z = np.random.normal(0, 0.001, n_table)
    table_points = np.column_stack([table_x, table_y, table_z])

    # Object 1: Cylinder (cup) at position (0.15, 0.1)
    n_cyl = 800
    theta = np.random.uniform(0, 2 * np.pi, n_cyl)
    cyl_radius = 0.035
    cyl_x = cyl_radius * np.cos(theta) + 0.15
    cyl_y = cyl_radius * np.sin(theta) + 0.1
    cyl_z = np.random.uniform(0, 0.12, n_cyl)
    cyl_points = np.column_stack([cyl_x, cyl_y, cyl_z])

What does this code do?

The table is an 80cm x 80cm flat plane at z=0. The 5000 points have tiny random Z offsets (np.random.normal(0, 0.001)) to simulate real sensor noise on a flat surface.

The cylinder represents a cup: 800 points arranged in a circle of radius 3.5cm, extruded 12cm tall. The center is offset to position (0.15, 0.1) on the table so it does not sit at the origin.

  • You won't see any output yet. The next block completes the function with the remaining objects.
  • Add the remaining objects, noise, and the return statement directly below the cylinder code in generate_scene.py:
    # Object 2: Box (book) at position (-0.15, -0.1)
    n_box = 600
    box_x = np.random.uniform(-0.08, 0.08, n_box) + (-0.15)
    box_y = np.random.uniform(-0.05, 0.05, n_box) + (-0.1)
    box_z = np.random.uniform(0, 0.03, n_box)
    box_points = np.column_stack([box_x, box_y, box_z])

    # Object 3: Sphere (ball) at position (0.0, -0.2)
    n_sphere = 500
    sphere_radius = 0.04
    phi = np.random.uniform(0, np.pi / 2, n_sphere)
    theta = np.random.uniform(0, 2 * np.pi, n_sphere)
    sph_x = sphere_radius * np.sin(phi) * np.cos(theta) + 0.0
    sph_y = sphere_radius * np.sin(phi) * np.sin(theta) + (-0.2)
    sph_z = sphere_radius * np.cos(phi)
    sph_points = np.column_stack([sph_x, sph_y, sph_z])

    # Random noise (simulating sensor artifacts)
    n_noise = 100
    noise_x = np.random.uniform(-0.5, 0.5, n_noise)
    noise_y = np.random.uniform(-0.5, 0.5, n_noise)
    noise_z = np.random.uniform(0.15, 0.4, n_noise)
    noise_points = np.column_stack([noise_x, noise_y, noise_z])

    # Combine all points
    all_points = np.vstack([table_points, cyl_points, box_points, sph_points, noise_points])

    # Create Open3D point cloud
    pcd = o3d.geometry.PointCloud()
    pcd.points = o3d.utility.Vector3dVector(all_points)

    return pcd

What does this code do?

The box (book) is 16cm x 10cm x 3cm tall with 600 random points filling its volume. The sphere (ball) uses spherical coordinates to scatter 500 points over the top hemisphere of a 4cm-radius ball.

The 100 noise points float randomly between z=0.15 and z=0.4, simulating the stray readings a real depth sensor produces. Finally, np.vstack merges all point arrays and o3d.utility.Vector3dVector converts the NumPy array into an Open3D point cloud object.

  • Save generate_scene.py. The function is complete but not yet callable. The next section adds a main block to run it.

Seeing an indentation error?

Every line from # Object 2 through return pcd must be indented with exactly 4 spaces (inside the function body). Make sure there is no extra blank line between the cylinder block and the box block that breaks the indentation level.

help me fix indentation in my generate_scene.py

Visualize and inspect the raw scene

The function is ready but has no entry point. You need a main block that calls the function, saves the result, and opens a 3D viewer so you can inspect the scene.

  • Add this main block at the bottom of generate_scene.py (outside and below the function, with no indentation):
if __name__ == "__main__":
    pcd = generate_tabletop_scene()
    print(f"Generated point cloud with {len(pcd.points)} points")
    print(f"Bounding box: {pcd.get_axis_aligned_bounding_box()}")

    # Save to disk
    o3d.io.write_point_cloud("tabletop_scene.pcd", pcd)
    print("Saved to tabletop_scene.pcd")

    # Visualize
    o3d.visualization.draw_geometries(
        [pcd],
        window_name="Raw Tabletop Scene",
        width=1024,
        height=768,
    )

What does this code do?

The main block calls your generator function, prints the total point count and bounding box extents, then saves the cloud to a .pcd file. Saving happens before visualization because draw_geometries blocks execution until you close the window.

The visualization opens an interactive 3D viewer. You can rotate with left-click drag, zoom with scroll, and press H to see all available keyboard controls.

  • Save generate_scene.py.
  • Run the script by executing this command in your terminal (make sure your virtual environment is active):
python generate_scene.py
  • A visualization window titled "Raw Tabletop Scene" opens. You should see a flat surface with three distinct shapes (a tall thin cluster, a low flat cluster, and a round cluster) plus scattered dots floating above.
  • In your terminal, you should see output showing approximately 7000 points and the bounding box extents.
  • Close the visualization window when you are done inspecting.

Visualization window not appearing?

If you are running over SSH, Open3D visualization requires X11 forwarding. Connect with ssh -X or run the script on your local machine with a display attached.

If you see a ModuleNotFoundError, confirm your virtual environment is active by checking that your terminal prompt shows (venv) at the start.

help me get the Open3D visualization window to display

✔️ Awesome, I've got everything!

Great. Double check you have saved generate_scene.py and confirmed the visualization window displays the scene.

ⓧ I'd like to double check the full code

import open3d as o3d
import numpy as np


def generate_tabletop_scene():
    """Generate a point cloud simulating a depth camera view of a table with objects."""
    np.random.seed(42)

    # Table surface: flat plane at z=0 with slight sensor noise
    n_table = 5000
    table_x = np.random.uniform(-0.4, 0.4, n_table)
    table_y = np.random.uniform(-0.4, 0.4, n_table)
    table_z = np.random.normal(0, 0.001, n_table)
    table_points = np.column_stack([table_x, table_y, table_z])

    # Object 1: Cylinder (cup) at position (0.15, 0.1)
    n_cyl = 800
    theta = np.random.uniform(0, 2 * np.pi, n_cyl)
    cyl_radius = 0.035
    cyl_x = cyl_radius * np.cos(theta) + 0.15
    cyl_y = cyl_radius * np.sin(theta) + 0.1
    cyl_z = np.random.uniform(0, 0.12, n_cyl)
    cyl_points = np.column_stack([cyl_x, cyl_y, cyl_z])

    # Object 2: Box (book) at position (-0.15, -0.1)
    n_box = 600
    box_x = np.random.uniform(-0.08, 0.08, n_box) + (-0.15)
    box_y = np.random.uniform(-0.05, 0.05, n_box) + (-0.1)
    box_z = np.random.uniform(0, 0.03, n_box)
    box_points = np.column_stack([box_x, box_y, box_z])

    # Object 3: Sphere (ball) at position (0.0, -0.2)
    n_sphere = 500
    sphere_radius = 0.04
    phi = np.random.uniform(0, np.pi / 2, n_sphere)
    theta = np.random.uniform(0, 2 * np.pi, n_sphere)
    sph_x = sphere_radius * np.sin(phi) * np.cos(theta) + 0.0
    sph_y = sphere_radius * np.sin(phi) * np.sin(theta) + (-0.2)
    sph_z = sphere_radius * np.cos(phi)
    sph_points = np.column_stack([sph_x, sph_y, sph_z])

    # Random noise (simulating sensor artifacts)
    n_noise = 100
    noise_x = np.random.uniform(-0.5, 0.5, n_noise)
    noise_y = np.random.uniform(-0.5, 0.5, n_noise)
    noise_z = np.random.uniform(0.15, 0.4, n_noise)
    noise_points = np.column_stack([noise_x, noise_y, noise_z])

    # Combine all points
    all_points = np.vstack([table_points, cyl_points, box_points, sph_points, noise_points])

    # Create Open3D point cloud
    pcd = o3d.geometry.PointCloud()
    pcd.points = o3d.utility.Vector3dVector(all_points)

    return pcd


if __name__ == "__main__":
    pcd = generate_tabletop_scene()
    print(f"Generated point cloud with {len(pcd.points)} points")
    print(f"Bounding box: {pcd.get_axis_aligned_bounding_box()}")

    # Save to disk
    o3d.io.write_point_cloud("tabletop_scene.pcd", pcd)
    print("Saved to tabletop_scene.pcd")

    # Visualize
    o3d.visualization.draw_geometries(
        [pcd],
        window_name="Raw Tabletop Scene",
        width=1024,
        height=768,
    )
Confirm the saved point cloud file

When you ran the script, it automatically saved the scene to disk before opening the viewer. This .pcd file simulates having a recorded depth frame from your UR5e's camera.

  • Confirm the file was created by running:
ls tabletop_scene.pcd
  • You should see tabletop_scene.pcd listed in the output. This is the file your detection pipeline will load in the next step.

Why save to a PCD file?

In a real robotic system, your UR5e's depth camera (like an Intel RealSense) would publish point clouds as ROS 2 messages. Saving to a .pcd file decouples the perception pipeline from live hardware so you can develop and test offline.

The detection script you build next will load this file with o3d.io.read_point_cloud, treating it exactly like a recorded sensor frame.

File not found?

The script writes tabletop_scene.pcd to your current working directory. Make sure you ran python generate_scene.py from inside ~/perception_pipeline. If you ran it from a different directory, the file was created there instead.

help me find or regenerate my tabletop_scene.pcd file

Your synthetic scene is ready. Next up, you will load this raw data into a detection pipeline and clean it up with outlier removal and downsampling.

Preprocess the Point Cloud

In the previous step, you generated a synthetic tabletop scene and saved it to disk. You now have a raw point cloud with roughly 7000 points, including sensor noise scattered throughout.

Raw point clouds from depth cameras are messy. They contain random noise from sensor imperfections and have uneven point density. Before you can run any detection algorithms, you need to clean and normalize this data. That is what preprocessing does.

In this step, get ready to:
  • Build a loader function that reads point clouds from disk.
  • Write a preprocessing pipeline that removes noise, downsamples, and estimates surface normals.
  • Verify the pipeline by visualizing the cleaned point cloud with normals displayed.
Load and clean the data

The first job of your detection pipeline is loading the saved PCD file from disk and making it available as an Open3D PointCloud object. You will wrap this in a reusable function so the pipeline stays modular.

  • Open tabletop_detection.py in your IDE.
  • Add the imports and the load_point_cloud function by pasting this code:
import open3d as o3d
import numpy as np
import matplotlib.pyplot as plt


def load_point_cloud(filepath):
    """Load a point cloud from disk."""
    pcd = o3d.io.read_point_cloud(filepath)
    print(f"Loaded point cloud: {len(pcd.points)} points")
    return pcd

What does this code do?

The three imports bring in the libraries you need. open3d handles point cloud operations, numpy provides array math, and matplotlib supplies colormaps for later visualization steps.

The load_point_cloud function wraps o3d.io.read_point_cloud so you can load any PCD, PLY, or XYZ file with one call. It prints the point count so you always know how much data you are working with.

  • Save tabletop_detection.py.

You will not see output yet. The next chunk adds the preprocessing logic that makes the cloud usable.

Downsample for performance

Raw point clouds have uneven density and contain outlier points from sensor noise. Statistical outlier removal eliminates isolated noise points by checking each point's distance to its neighbors.

After cleaning, voxel downsampling divides the space into small cubes (voxels) and keeps one representative point per cube. This creates uniform density and reduces point count for faster processing downstream.

  • In tabletop_detection.py, add the preprocess function below load_point_cloud by pasting this code:
def preprocess(pcd, voxel_size=0.005, nb_neighbors=20, std_ratio=2.0):
    """Remove outliers, downsample, and estimate normals."""
    # Statistical outlier removal
    pcd_clean, inlier_indices = pcd.remove_statistical_outlier(
        nb_neighbors=nb_neighbors, std_ratio=std_ratio
    )
    n_removed = len(pcd.points) - len(pcd_clean.points)
    print(f"Outlier removal: removed {n_removed} noise points")

    # Voxel downsampling
    pcd_down = pcd_clean.voxel_down_sample(voxel_size=voxel_size)
    print(f"Downsampling: {len(pcd_clean.points)} -> {len(pcd_down.points)} points")

    # Estimate normals
    pcd_down.estimate_normals(
        search_param=o3d.geometry.KDTreeSearchParamHybrid(radius=0.02, max_nn=30)
    )
    print("Normals estimated")

    return pcd_down

What does this code do?

  • remove_statistical_outlier looks at each point's 20 nearest neighbors. If a point is more than 2 standard deviations away from the average neighbor distance, it gets flagged as noise and removed.
  • voxel_down_sample divides the 3D space into 5mm cubes and replaces all points in each cube with their centroid. This creates uniform density across the entire cloud.
  • estimate_normals computes a surface direction vector for each point using its local neighborhood. These normals are essential for the RANSAC plane fitting in the next step.

Why combine three operations in one function?

In a real robotic perception stack, preprocessing always runs as a single pass before any detection logic. Grouping outlier removal, downsampling, and normal estimation into one function mirrors how production ROS 2 perception nodes structure their pipeline.

  • Save tabletop_detection.py.
  • Test the preprocessing pipeline by running this command in your terminal:
python -c "from tabletop_detection import load_point_cloud, preprocess; pcd = load_point_cloud('tabletop_scene.pcd'); pcd_processed = preprocess(pcd); print(f'Final point count: {len(pcd_processed.points)}')"

You should see output showing the loaded point count (~7000), the number of noise points removed, and the downsampled point count (roughly 40-60% reduction from the cleaned total).

Seeing a ModuleNotFoundError?

  • Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt.
  • Confirm you are running the command from inside the ~/perception_pipeline directory where both Python files live.
  • Check that tabletop_scene.pcd exists in the same directory. Run ls *.pcd to verify.

Help me debug my preprocessing pipeline import error.

Estimate surface normals

Your preprocess function already computes normals as its final step. Now you will visualize the result to confirm everything works end to end.

What are surface normals?

A surface normal is a vector perpendicular to the local surface at each point. Normals encode the surface orientation, which is critical for plane fitting. The RANSAC algorithm you will use in the next step relies on normals to determine which points belong to the flat table versus raised objects.

  • Visualize the preprocessed cloud with normals by running this command:
python -c "
import open3d as o3d
from tabletop_detection import load_point_cloud, preprocess
pcd = load_point_cloud('tabletop_scene.pcd')
pcd_processed = preprocess(pcd)
o3d.visualization.draw_geometries([pcd_processed], window_name='Preprocessed Cloud', width=1024, height=768, point_show_normal=True)
"

A visualization window opens showing the cleaned, downsampled cloud with tiny lines extending from each point. Those lines are the surface normals.

Toggle normals in the viewer

Press N on your keyboard inside the Open3D viewer to toggle normal visualization on and off. This helps you confirm normals are present without the visual clutter.

  • Close the visualization window when you are done inspecting.

Visualization window not appearing?

  • Open3D visualization requires a display. If you are running over SSH, you need X11 forwarding enabled (connect with ssh -X).
  • On a local Ubuntu desktop, the window may appear behind other windows. Check your taskbar for an Open3D entry.

Help me get Open3D visualization working on my setup.

✔️ Awesome, I've got everything!

Great. Double check you have saved tabletop_detection.py with both functions in place.

ⓧ I'd like to double check the full code

Here is the complete tabletop_detection.py so far:

import open3d as o3d
import numpy as np
import matplotlib.pyplot as plt


def load_point_cloud(filepath):
    """Load a point cloud from disk."""
    pcd = o3d.io.read_point_cloud(filepath)
    print(f"Loaded point cloud: {len(pcd.points)} points")
    return pcd


def preprocess(pcd, voxel_size=0.005, nb_neighbors=20, std_ratio=2.0):
    """Remove outliers, downsample, and estimate normals."""
    # Statistical outlier removal
    pcd_clean, inlier_indices = pcd.remove_statistical_outlier(
        nb_neighbors=nb_neighbors, std_ratio=std_ratio
    )
    n_removed = len(pcd.points) - len(pcd_clean.points)
    print(f"Outlier removal: removed {n_removed} noise points")

    # Voxel downsampling
    pcd_down = pcd_clean.voxel_down_sample(voxel_size=voxel_size)
    print(f"Downsampling: {len(pcd_clean.points)} -> {len(pcd_down.points)} points")

    # Estimate normals
    pcd_down.estimate_normals(
        search_param=o3d.geometry.KDTreeSearchParamHybrid(radius=0.02, max_nn=30)
    )
    print("Normals estimated")

    return pcd_down

Your point cloud is now clean, uniformly dense, and has surface orientation data attached to every point. Next up, you will use RANSAC to find the table plane and separate it from the objects sitting on top.

Segment the Table with RANSAC

Your point cloud is now cleaned and downsampled. The next challenge is separating the table from the objects sitting on top of it.

A robot arm cannot grasp objects if it does not know where the table ends and the objects begin. RANSAC (Random Sample Consensus) solves this by finding the dominant flat surface in the point cloud and removing it, leaving only the objects behind.

In this step, get ready to:
  • Run RANSAC plane segmentation to detect the table surface.
  • Separate the table points from the object points.
  • Visualize the segmentation result to confirm clean separation.
Run RANSAC plane segmentation

RANSAC works by repeatedly picking random subsets of points, fitting a plane through them, and counting how many other points are close to that plane. After many iterations, it keeps the plane that had the most agreement.

Since the table is the largest flat surface, RANSAC will find it reliably even with noisy data.

  • In tabletop_detection.py, add the following segment_table function below the preprocess function:
def segment_table(pcd, distance_threshold=0.005, ransac_n=3, num_iterations=1000):
    """Use RANSAC to find the dominant plane (table) and separate objects."""
    plane_model, inliers = pcd.segment_plane(
        distance_threshold=distance_threshold,
        ransac_n=ransac_n,
        num_iterations=num_iterations,
    )
    a, b, c, d = plane_model
    print(f"Table plane: {a:.4f}x + {b:.4f}y + {c:.4f}z + {d:.4f} = 0")

    # Separate table from objects
    table_cloud = pcd.select_by_index(inliers)
    table_cloud.paint_uniform_color([0.8, 0.8, 0.8])

    objects_cloud = pcd.select_by_index(inliers, invert=True)
    print(f"Table: {len(table_cloud.points)} points | Objects: {len(objects_cloud.points)} points")

    return table_cloud, objects_cloud, plane_model

What does this code do?

This function performs two major tasks: finding the table plane and splitting the cloud into two groups.

  • segment_plane runs RANSAC with a 5mm distance_threshold. Any point within 5mm of the best-fit plane counts as a table point (inlier).
  • ransac_n=3 means RANSAC samples 3 points at a time to define a candidate plane (3 points uniquely define a plane in 3D).
  • select_by_index(inliers) extracts the table points. Passing invert=True gives you everything that is NOT the table (the objects).
  • paint_uniform_color([0.8, 0.8, 0.8]) colors the table gray so you can visually distinguish it from objects in the viewer.

Why RANSAC instead of a height threshold?

You could try removing all points below a certain Z value. But in a real robotics scenario, the table might be tilted, the camera might be at an angle, or the table height might be unknown.

RANSAC finds the dominant plane regardless of orientation or position. This is why every major robotic manipulation pipeline uses it for surface detection.

  • Save tabletop_detection.py.

Seeing a syntax error?

  • Make sure the function is at the same indentation level as preprocess (top-level, no extra indentation).
  • Check that you have a blank line between the end of preprocess and the start of segment_table.
  • Verify the f-string uses curly braces around variables like {a:.4f} with no extra spaces.

Help me debug my segment_table function

✔️ Awesome, I've got everything!

Great. Make sure you have saved tabletop_detection.py before moving on.

ⓧ I'd like to double check the full code

import open3d as o3d
import numpy as np
import matplotlib.pyplot as plt


def load_point_cloud(filepath):
    """Load a point cloud from disk."""
    pcd = o3d.io.read_point_cloud(filepath)
    print(f"Loaded point cloud: {len(pcd.points)} points")
    return pcd


def preprocess(pcd, voxel_size=0.005, nb_neighbors=20, std_ratio=2.0):
    """Remove outliers, downsample, and estimate normals."""
    # Statistical outlier removal
    pcd_clean, inlier_indices = pcd.remove_statistical_outlier(
        nb_neighbors=nb_neighbors, std_ratio=std_ratio
    )
    n_removed = len(pcd.points) - len(pcd_clean.points)
    print(f"Outlier removal: removed {n_removed} noise points")

    # Voxel downsampling
    pcd_down = pcd_clean.voxel_down_sample(voxel_size=voxel_size)
    print(f"Downsampling: {len(pcd_clean.points)} -> {len(pcd_down.points)} points")

    # Estimate normals
    pcd_down.estimate_normals(
        search_param=o3d.geometry.KDTreeSearchParamHybrid(radius=0.02, max_nn=30)
    )
    print("Normals estimated")

    return pcd_down


def segment_table(pcd, distance_threshold=0.005, ransac_n=3, num_iterations=1000):
    """Use RANSAC to find the dominant plane (table) and separate objects."""
    plane_model, inliers = pcd.segment_plane(
        distance_threshold=distance_threshold,
        ransac_n=ransac_n,
        num_iterations=num_iterations,
    )
    a, b, c, d = plane_model
    print(f"Table plane: {a:.4f}x + {b:.4f}y + {c:.4f}z + {d:.4f} = 0")

    # Separate table from objects
    table_cloud = pcd.select_by_index(inliers)
    table_cloud.paint_uniform_color([0.8, 0.8, 0.8])

    objects_cloud = pcd.select_by_index(inliers, invert=True)
    print(f"Table: {len(table_cloud.points)} points | Objects: {len(objects_cloud.points)} points")

    return table_cloud, objects_cloud, plane_model
Visualize the segmentation result

The function is written, but you need to see whether RANSAC actually found the table correctly. You will run a quick test that loads the cloud, preprocesses it, segments the table, and opens a 3D viewer.

  • Test the full pipeline so far by running this command in your terminal:
python3 -c "
from tabletop_detection import load_point_cloud, preprocess, segment_table
import open3d as o3d

pcd = load_point_cloud('tabletop_scene.pcd')
pcd_processed = preprocess(pcd)
table_cloud, objects_cloud, plane_model = segment_table(pcd_processed)

objects_cloud.paint_uniform_color([1, 0, 0])
o3d.visualization.draw_geometries(
    [table_cloud, objects_cloud],
    window_name='Table Segmentation Result',
    width=1024,
    height=768,
)
"

What should I see?

Your terminal output should show a plane equation with coefficients close to (0, 0, 1, 0). This indicates a horizontal plane at z=0, which is exactly where we generated the table.

The visualization window should show a flat gray surface (the table) with red clusters floating above it (the three objects). The separation should be clean with no red points on the table surface.

  • Close the visualization window when you are done inspecting the result.

What if the segmentation is poor?

The distance_threshold parameter controls sensitivity. A smaller value (like 0.002) means only points very close to the plane count as table. A larger value (like 0.01) is more tolerant of noise but might swallow low-lying objects.

For real sensor data from a RealSense camera, you would typically use 0.01 to 0.02 because depth sensors produce more noise than our synthetic data.

Visualization window not appearing?

  • Open3D visualization requires a display. If you are running over SSH, you need X11 forwarding enabled (ssh -X).
  • Make sure you are running the command from inside the ~/perception_pipeline directory where tabletop_scene.pcd is located.
  • Confirm your virtual environment is still activated. You should see (venv) in your terminal prompt.

Help me troubleshoot Open3D visualization

Your pipeline can now isolate objects from the table surface. Next up, you will use DBSCAN clustering to identify each individual object and compute its bounding box.

Cluster Objects and Compute Bounding Boxes

In the previous step, you used RANSAC to strip away the table surface and isolate the object points. The problem is that these object points are still one undifferentiated blob.

Your robot arm needs to know how many objects there are, where each one sits, and how big it is. DBSCAN (Density-Based Spatial Clustering of Applications with Noise) solves exactly this by grouping nearby points into clusters based on density.

In this step, get ready to:
  • Write a DBSCAN clustering function that separates individual objects and computes bounding boxes.
  • Build the main pipeline function that orchestrates the full detection flow.
  • Run the pipeline and verify 3 detected objects with their positions and sizes.
Write the DBSCAN clustering function

DBSCAN works by picking a point, finding all neighbors within a radius (eps), and expanding the cluster if enough neighbors exist (min_points). Points that don't belong to any cluster are labeled as noise.

  • In tabletop_detection.py, find the end of the segment_table function and add the following cluster_objects function below it:
def cluster_objects(objects_cloud, eps=0.02, min_points=10):
    """Use DBSCAN to cluster individual objects and compute bounding boxes."""
    labels = np.array(objects_cloud.cluster_dbscan(eps=eps, min_points=min_points))
    n_clusters = labels.max() + 1
    n_noise = np.sum(labels == -1)
    print(f"DBSCAN: found {n_clusters} objects ({n_noise} noise points)")

    # Color each cluster
    max_label = labels.max()
    colors = np.zeros((len(labels), 3))
    if max_label >= 0:
        cmap = plt.get_cmap("tab10")
        for i in range(len(labels)):
            if labels[i] >= 0:
                colors[i] = cmap(labels[i] / max(max_label, 1))[:3]
    objects_cloud.colors = o3d.utility.Vector3dVector(colors)

    # Compute bounding boxes
    bounding_boxes = []
    for i in range(n_clusters):
        cluster_indices = np.where(labels == i)[0]
        cluster = objects_cloud.select_by_index(cluster_indices.tolist())
        aabb = cluster.get_axis_aligned_bounding_box()
        aabb.color = (0, 1, 0)
        bounding_boxes.append(aabb)

        center = aabb.get_center()
        extent = aabb.get_extent()
        print(
            f"  Object {i + 1}: center=({center[0]:.3f}, {center[1]:.3f}, {center[2]:.3f}), "
            f"size=({extent[0]:.3f} x {extent[1]:.3f} x {extent[2]:.3f})m"
        )

    return labels, bounding_boxes

What does this code do?

This function takes the objects-only point cloud (table already removed) and separates it into individual objects.

  • cluster_dbscan(eps=0.02, min_points=10) groups points that are within 2cm of each other into clusters. Any point not near at least 10 neighbors is labeled as noise (-1).
  • The coloring section assigns a unique color from matplotlib's tab10 colormap to each cluster so you can visually distinguish objects.
  • For each cluster, get_axis_aligned_bounding_box() computes a box aligned to the X/Y/Z axes that tightly encloses the object. The center and extent give you the object's 3D position and size.

What is DBSCAN?

Unlike K-means (which requires knowing the number of clusters in advance), DBSCAN discovers clusters automatically based on point density. It also naturally handles noise points that don't belong to any object.

This makes DBSCAN ideal for robotics perception where you never know in advance how many objects are on the table.

  • Save tabletop_detection.py.

Seeing a NameError for plt?

Make sure the import matplotlib.pyplot as plt line is at the top of your file. It was added in the previous step when you first created tabletop_detection.py.

Help me debug a NameError in my clustering function.

Build the pipeline orchestrator

Now you need a main() function that chains every pipeline stage together: load, preprocess, segment table, cluster objects, and visualize the results.

  • Below your cluster_objects function, add the main() function and the entry point guard:
def main():
    print("=" * 60)
    print("TABLETOP OBJECT DETECTION PIPELINE")
    print("=" * 60)

    # Load
    print("\n--- Loading ---")
    pcd = load_point_cloud("tabletop_scene.pcd")

    # Preprocess
    print("\n--- Preprocessing ---")
    pcd_processed = preprocess(pcd)

    # Segment table
    print("\n--- Table Segmentation (RANSAC) ---")
    table_cloud, objects_cloud, plane_model = segment_table(pcd_processed)

    # Cluster objects
    print("\n--- Object Clustering (DBSCAN) ---")
    labels, bounding_boxes = cluster_objects(objects_cloud)

    # Final visualization
    print("\n--- Visualization ---")
    print("Displaying detected objects with bounding boxes...")
    print("(Close the window to exit)")
    o3d.visualization.draw_geometries(
        [objects_cloud, table_cloud] + bounding_boxes,
        window_name="Detected Objects with Bounding Boxes",
        width=1024,
        height=768,
    )


if __name__ == "__main__":
    main()

What does this code do?

  • main() calls each pipeline stage in sequence: load the PCD file, preprocess (denoise + downsample + normals), segment the table with RANSAC, then cluster the remaining objects with DBSCAN.
  • draw_geometries renders the colored object clusters, the gray table, and the green bounding boxes together in one interactive 3D window.
  • The if __name__ == "__main__": guard ensures main() only runs when you execute the file directly, not when importing it as a module.
  • Save tabletop_detection.py.
Run the full detection pipeline

Time to see the complete pipeline in action. The console output tells you exactly what a perception node would send to a manipulation planner: object count, positions, and sizes.

  • Run the full pipeline by executing this command in your terminal:
python tabletop_detection.py

What should I see?

Your console should print the full pipeline stages. Under the Object Clustering section, look for:

  • DBSCAN: found 3 objects confirming three separate clusters were detected.
  • Object 1 with a center near (0.15, 0.1) matching the cup position.
  • Object 2 with a center near (-0.15, -0.1) matching the book position.
  • Object 3 with a center near (0.0, -0.2) matching the ball position.

A 3D visualization window opens showing color-coded objects with green bounding boxes overlaid. This is the data your UR5e's MoveIt planner would consume to compute grasp poses.

  • Verify that 3 distinctly colored clusters appear in the visualization window with green boxes around each object.
  • Close the visualization window when you are done inspecting.

Seeing fewer than 3 objects?

If DBSCAN finds fewer clusters, two objects may be merging into one. Check that your eps value is 0.02 (2cm). A larger eps combines nearby objects into a single cluster.

Also confirm you ran generate_scene.py first to create the tabletop_scene.pcd file. Without it, the pipeline has no input data.

Help me debug why DBSCAN is finding the wrong number of clusters.

✔️ Awesome, I've got everything!

Great. Double check you have saved tabletop_detection.py and that your console shows 3 detected objects with positions and sizes.

ⓧ I'd like to double check the full code

Here is the complete tabletop_detection.py file at this point in the project:

import open3d as o3d
import numpy as np
import matplotlib.pyplot as plt


def load_point_cloud(filepath):
    """Load a point cloud from disk."""
    pcd = o3d.io.read_point_cloud(filepath)
    print(f"Loaded point cloud: {len(pcd.points)} points")
    return pcd


def preprocess(pcd, voxel_size=0.005, nb_neighbors=20, std_ratio=2.0):
    """Remove outliers, downsample, and estimate normals."""
    # Statistical outlier removal
    pcd_clean, inlier_indices = pcd.remove_statistical_outlier(
        nb_neighbors=nb_neighbors, std_ratio=std_ratio
    )
    n_removed = len(pcd.points) - len(pcd_clean.points)
    print(f"Outlier removal: removed {n_removed} noise points")

    # Voxel downsampling
    pcd_down = pcd_clean.voxel_down_sample(voxel_size=voxel_size)
    print(f"Downsampling: {len(pcd_clean.points)} -> {len(pcd_down.points)} points")

    # Estimate normals
    pcd_down.estimate_normals(
        search_param=o3d.geometry.KDTreeSearchParamHybrid(radius=0.02, max_nn=30)
    )
    print("Normals estimated")

    return pcd_down


def segment_table(pcd, distance_threshold=0.005, ransac_n=3, num_iterations=1000):
    """Use RANSAC to find the dominant plane (table) and separate objects."""
    plane_model, inliers = pcd.segment_plane(
        distance_threshold=distance_threshold,
        ransac_n=ransac_n,
        num_iterations=num_iterations,
    )
    a, b, c, d = plane_model
    print(f"Table plane: {a:.4f}x + {b:.4f}y + {c:.4f}z + {d:.4f} = 0")

    # Separate table from objects
    table_cloud = pcd.select_by_index(inliers)
    table_cloud.paint_uniform_color([0.8, 0.8, 0.8])

    objects_cloud = pcd.select_by_index(inliers, invert=True)
    print(f"Table: {len(table_cloud.points)} points | Objects: {len(objects_cloud.points)} points")

    return table_cloud, objects_cloud, plane_model


def cluster_objects(objects_cloud, eps=0.02, min_points=10):
    """Use DBSCAN to cluster individual objects and compute bounding boxes."""
    labels = np.array(objects_cloud.cluster_dbscan(eps=eps, min_points=min_points))
    n_clusters = labels.max() + 1
    n_noise = np.sum(labels == -1)
    print(f"DBSCAN: found {n_clusters} objects ({n_noise} noise points)")

    # Color each cluster
    max_label = labels.max()
    colors = np.zeros((len(labels), 3))
    if max_label >= 0:
        cmap = plt.get_cmap("tab10")
        for i in range(len(labels)):
            if labels[i] >= 0:
                colors[i] = cmap(labels[i] / max(max_label, 1))[:3]
    objects_cloud.colors = o3d.utility.Vector3dVector(colors)

    # Compute bounding boxes
    bounding_boxes = []
    for i in range(n_clusters):
        cluster_indices = np.where(labels == i)[0]
        cluster = objects_cloud.select_by_index(cluster_indices.tolist())
        aabb = cluster.get_axis_aligned_bounding_box()
        aabb.color = (0, 1, 0)
        bounding_boxes.append(aabb)

        center = aabb.get_center()
        extent = aabb.get_extent()
        print(
            f"  Object {i + 1}: center=({center[0]:.3f}, {center[1]:.3f}, {center[2]:.3f}), "
            f"size=({extent[0]:.3f} x {extent[1]:.3f} x {extent[2]:.3f})m"
        )

    return labels, bounding_boxes


def main():
    print("=" * 60)
    print("TABLETOP OBJECT DETECTION PIPELINE")
    print("=" * 60)

    # Load
    print("\n--- Loading ---")
    pcd = load_point_cloud("tabletop_scene.pcd")

    # Preprocess
    print("\n--- Preprocessing ---")
    pcd_processed = preprocess(pcd)

    # Segment table
    print("\n--- Table Segmentation (RANSAC) ---")
    table_cloud, objects_cloud, plane_model = segment_table(pcd_processed)

    # Cluster objects
    print("\n--- Object Clustering (DBSCAN) ---")
    labels, bounding_boxes = cluster_objects(objects_cloud)

    # Final visualization
    print("\n--- Visualization ---")
    print("Displaying detected objects with bounding boxes...")
    print("(Close the window to exit)")
    o3d.visualization.draw_geometries(
        [objects_cloud, table_cloud] + bounding_boxes,
        window_name="Detected Objects with Bounding Boxes",
        width=1024,
        height=768,
    )


if __name__ == "__main__":
    main()

Your pipeline now detects individual objects and reports their 3D positions and sizes. This is exactly the data a robotic manipulation planner needs to compute where to reach and how wide to open the gripper. In the Secret Mission, you will take this further by computing oriented bounding boxes that also tell the robot the angle to approach each object.

Secret mission

Oriented Bounding Boxes for Grasp Planning

Your axis-aligned bounding boxes tell the robot where objects are, but not how they are oriented. For a real pick-and-place task, your UR5e's gripper needs to know the object's principal axes to approach correctly. In this mission, you will add oriented bounding boxes using PCA to estimate object rotation and compute grasp approach vectors.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your project files for future experimentation, or delete them to free up disk space.

Resources you used:

  • Project directory ~/perception_pipeline (contains all Python scripts and the virtual environment).
  • Generated point cloud file tabletop_scene.pcd inside the project directory.
  • Python virtual environment venv with Open3D, NumPy, and Matplotlib installed.

Keep everything running

No action needed. Choose this if you plan to extend the pipeline, experiment with different RANSAC or DBSCAN parameters, or wrap this code as a ROS 2 node in a follow-up project.

  • Your project files stay in ~/perception_pipeline.
  • The virtual environment stays ready to activate with source venv/bin/activate.

Pause - I'll come back to this later

Deactivate the virtual environment to free up your shell, but keep all files on disk for later.

  • Deactivate the virtual environment by running:
deactivate

When you return, reactivate with:

cd ~/perception_pipeline
source venv/bin/activate

Delete - I don't want to use this again

Remove all project files, the virtual environment, and the generated point cloud data.

  • If you are currently inside the virtual environment, deactivate it first:
deactivate
  • Delete the entire project directory by running:
rm -rf ~/perception_pipeline
  • Confirm the directory is gone by running:
ls ~/perception_pipeline

You should see a message indicating the directory does not exist. This removes generate_scene.py, tabletop_detection.py, tabletop_scene.pcd, and the entire venv virtual environment with all installed packages.

That's a Wrap!

That's a Wrap!

Nice work! You just built a complete 3D object detection pipeline that takes raw point cloud data and outputs the position, size, and bounding box of every object on a table.

This is the exact perception module that powers real robotic pick-and-place systems. You now have a reusable codebase ready to wrap as a ROS 2 perception node for vision-guided manipulation.

You've learned how to:

  • Use RANSAC plane segmentation to detect and remove a table surface from a raw point cloud.
  • Apply DBSCAN clustering to segment individual objects and compute their 3D bounding boxes with position and size.
  • Build a preprocessing pipeline with statistical outlier removal and voxel downsampling to clean noisy sensor data for robust detection.
  • Secret Mission: Estimated object orientation using oriented bounding boxes and computed grasp approach vectors for robotic pick-and-place planning.

Ready to quiz yourself?