Build an Interactive 3D Shape Gallery

Build a visionOS app with RealityKit, custom ECS, and spatial gestures.

Introduction

30 Second Summary

Every object you pick up, rotate, and set down in real life is a spatial interaction. Spatial computing brings that same intuition to software, letting apps exist as 3D objects in the world around you.

In this project, you will build a visionOS app called ShapeGallery that combines a 2D control window with a volumetric 3D shape gallery. You will create interactive shapes that rotate continuously, change color on tap, and reposition with drag gestures.

What You'll Build

ShapeGallery is a visionOS app with an @Observable data model, a SwiftUI control window, and a volumetric window displaying four RealityKit shapes with custom animation and multi-gesture interaction.

By the end of this project, you'll have:

  • An @Observable data model shared via SwiftUI Environment, driving both a 2D control window and a volumetric 3D shape gallery.
  • A custom RotationComponent and RotationSystem using the Entity Component System pattern for continuous 3D rotation with quaternion math.
  • Multi-gesture spatial interaction including tap to randomize colors, drag to reposition shapes, and gaze-based hover effects on four procedurally generated 3D shapes.
  • A particle burst effect triggered on tap for extra visual polish (Secret Mission).

Are there any prerequisites?

You need a Mac with Apple Silicon and Xcode with the visionOS SDK installed. The visionOS Simulator is included with Xcode, so no Vision Pro hardware is required.

Set Up Your visionOS Project in Xcode

Your ShapeGallery app will run on visionOS, Apple's operating system for spatial computing. Before you can build any 3D shapes or gestures, you need an Xcode project configured for the visionOS platform.

In this step, you'll create the project from Xcode's visionOS template and run it in the Simulator. By the end, you'll have a working visionOS app displaying a default window on a simulated Apple Vision Pro.

What is visionOS?

visionOS is Apple's operating system for the Apple Vision Pro headset. It lets you build apps that mix 2D windows with 3D content in a spatial environment, using frameworks like SwiftUI and RealityKit.

In this step, get ready to:
  • Create a new visionOS project in Xcode.
  • Install the visionOS Simulator runtime.
  • Build and run the default app to verify your setup.
Create a new visionOS project

Xcode provides a visionOS App template that scaffolds the project structure for you. You'll configure a few settings to tell Xcode what kind of visionOS app you're building.

  • Press Cmd+Space to open Spotlight Search.
  • Type Xcode and press Enter to open it.
  • In the menu bar, click File.
  • Click New, then click Project.
  • Select the visionOS tab at the top of the template chooser.
  • Select App under the Application section.
  • Click Next.
  • Enter ShapeGallery in the Product Name field.
  • Select Window from the Initial Scene dropdown.
  • Check the Include a Reality Composer Pro project checkbox.
  • Leave the remaining settings at their defaults.
  • Click Next.
  • Choose your Desktop as the save location.
  • Click Create.

Your project opens in Xcode with the file navigator on the left showing the ShapeGallery source files.

What do these project settings mean?

  • Initial Scene: Window tells Xcode your app starts with a flat 2D window, just like a traditional app. You'll add a 3D volumetric window later in Step 3.
  • Include a Reality Composer Pro project adds a Reality Composer Pro content package to your project. This gives your app access to the RealityKit content pipeline for 3D assets.

Don't see the visionOS tab?

Make sure you're running a recent version of Xcode that includes the visionOS SDK. Developing for visionOS requires a Mac with Apple Silicon.

Help me figure out why the visionOS tab isn't showing in Xcode's template chooser.

Install the visionOS Simulator runtime

The visionOS Simulator lets you test your app on a virtual Apple Vision Pro without needing the actual hardware. You need to download the Simulator runtime before you can run your project.

  • In the menu bar, click Xcode.
  • Click Settings.
  • Select the Platforms tab.

I see visionOS listed

The visionOS Simulator runtime is already installed. Close the Settings window and move on to the next substep.

I don't see visionOS

  • Click the + button in the bottom-left corner of the Platforms panel.
  • Select visionOS from the list.
  • Click Download & Install.
  • Wait for the download to complete. This may take a few minutes depending on your internet connection.

Once the download finishes, you'll see visionOS listed under your installed platforms.

Download taking a while?

The visionOS Simulator runtime is several gigabytes. On slower connections, the download can take 10 to 15 minutes. Make sure you have a stable internet connection and enough free disk space.

Help me troubleshoot the visionOS Simulator download in Xcode.

Build and run your project

With the Simulator runtime installed, you can build and launch your app to make sure everything is wired up correctly.

  • In the Xcode toolbar near the top of the window, click the destination dropdown (it shows a device name next to your scheme).
  • Select Apple Vision Pro under the visionOS Simulators section.
  • Press Cmd+R to build and run the project.

The visionOS Simulator launches and displays a floating window with "Hello, World!" text. This is the default SwiftUI view that Xcode generated for you.

How do I navigate the visionOS Simulator?

  • Use the WASD or arrow keys to move the camera around the simulated space.
  • Click with your mouse to simulate a tap. On a real Apple Vision Pro, this maps to the gaze-and-pinch gesture.

Don't see the Hello World window?

  • Check the Xcode toolbar for a red error icon. If you see build errors, make sure the destination is set to Apple Vision Pro and not a physical device.
  • If the Simulator is open but the window is not visible, use the WASD keys to look around. The window may have appeared behind the camera.
  • If the Simulator fails to launch, go back to Xcode > Settings > Platforms and confirm the visionOS runtime finished downloading.

Help me troubleshoot why my visionOS app isn't running in the Simulator.

Your visionOS project is running in the Simulator. Next up, you'll replace that default Hello World view with an @Observable data model that drives the entire app.

Create the @Observable Data Model

Your visionOS project is set up and the default app runs in the Simulator. Now it's time to build the data layer that powers your entire shape gallery.

Every shape in the gallery needs a name, a color, and an emoji. You'll define these in a shared data model using the @Observable macro, then inject it into SwiftUI's environment so every view in the app can read from it.

In this step, get ready to:
  • Create a ShapeData struct and ShapeGalleryModel class using the @Observable macro.
  • Inject the model into the SwiftUI view hierarchy via the environment.
  • Display shape data in ContentView driven by the model.
Create the data model file

The data model is the single source of truth for your app. It defines what shapes exist and which one the user has selected.

  • In Xcode, right-click the ShapeGallery folder in the Project Navigator on the left.
  • Select New File... from the context menu.
  • Choose Swift File and click Next.
  • Name the file ShapeGalleryModel and click Create.
  • Replace the contents of ShapeGalleryModel.swift with the following code:
import SwiftUI
import Observation

// Each shape's display data
struct ShapeData: Identifiable {
    let id = UUID()
    let name: String
    let color: Color
    let emoji: String
}

// Shared data model using the @Observable macro
@Observable
class ShapeGalleryModel {
    // The four shapes in our gallery
    var shapes: [ShapeData] = [
        ShapeData(name: "Box", color: .blue, emoji: "🟦"),
        ShapeData(name: "Sphere", color: .yellow, emoji: "🟡"),
        ShapeData(name: "Cone", color: .red, emoji: "🔺"),
        ShapeData(name: "Cylinder", color: .green, emoji: "🟩")
    ]

    // Tracks which shape the user last tapped
    var selectedShapeName: String?
}

What does this code do?

  • ShapeData is a struct that holds the display info for one shape. It conforms to Identifiable so SwiftUI's ForEach can loop over shapes without needing an explicit id: parameter.
  • @Observable is a macro that makes every property on the class automatically observable. When shapes or selectedShapeName changes, any SwiftUI view reading that property re-renders automatically.
  • selectedShapeName is optional because no shape is selected when the app first launches.

What is @Observable?

@Observable is the modern Swift observation pattern, available from visionOS 1.0 onward. It replaces the older ObservableObject protocol that relied on Combine. With @Observable, you use @State and @Environment instead of @StateObject and @EnvironmentObject.

  • Save ShapeGalleryModel.swift with Cmd+S.
  • You won't see a change yet. The next substep wires this model into the app.

Awesome, I've got everything!

Great. Double check you've saved ShapeGalleryModel.swift before moving on.

I'd like to double check the full code

import SwiftUI
import Observation

struct ShapeData: Identifiable {
    let id = UUID()
    let name: String
    let color: Color
    let emoji: String
}

@Observable
class ShapeGalleryModel {
    var shapes: [ShapeData] = [
        ShapeData(name: "Box", color: .blue, emoji: "🟦"),
        ShapeData(name: "Sphere", color: .yellow, emoji: "🟡"),
        ShapeData(name: "Cone", color: .red, emoji: "🔺"),
        ShapeData(name: "Cylinder", color: .green, emoji: "🟩")
    ]

    var selectedShapeName: String?
}
Inject the model into the app

The model exists, but no view can access it yet. SwiftUI's environment system lets you create the model once and share it with every view in your app.

  • Open ShapeGalleryApp.swift in the Project Navigator.
  • Replace the full contents of the file with this code:
import SwiftUI

@main
struct ShapeGalleryApp: App {
    // Create the shared model as @State so SwiftUI manages its lifecycle
    @State private var model = ShapeGalleryModel()

    var body: some Scene {
        WindowGroup {
            ContentView()
                // Inject the model into the environment for all child views
                .environment(model)
        }
    }
}

What does this code do?

  • @State private var model creates a single instance of the model and gives SwiftUI ownership of its lifecycle. This keeps the model alive for as long as the app is running.
  • .environment(model) places the model into the SwiftUI environment. Any child view can retrieve it with @Environment without passing it manually through every view.
  • Save ShapeGalleryApp.swift.
  • You won't see a change yet. The next substep updates ContentView to read from the model.

Awesome, I've got everything!

Great. Double check you've saved ShapeGalleryApp.swift before moving on.

I'd like to double check the full code

import SwiftUI

@main
struct ShapeGalleryApp: App {
    @State private var model = ShapeGalleryModel()

    var body: some Scene {
        WindowGroup {
            ContentView()
                .environment(model)
        }
    }
}
Display shape data in ContentView

The model is created and injected. Now ContentView needs to pull the model from the environment and display each shape's emoji and name in a list.

  • Open ContentView.swift in the Project Navigator.
  • Replace the full contents of the file with this code:
import SwiftUI

struct ContentView: View {
    // Retrieve the shared model from the environment
    @Environment(ShapeGalleryModel.self) private var model

    var body: some View {
        VStack(alignment: .leading, spacing: 16) {
            // App title
            Text("Shape Gallery")
                .font(.largeTitle)
                .fontWeight(.bold)

            // Instructions subtitle
            Text("Tap a shape to change its color. Drag to reposition.")
                .font(.body)
                .foregroundStyle(.secondary)

            Divider()

            // List each shape from the model
            ForEach(model.shapes) { shape in
                HStack(spacing: 12) {
                    Text(shape.emoji)
                        .font(.title)
                    Text(shape.name)
                        .font(.title3)
                }
            }
        }
        .padding(32)
        .frame(minWidth: 300, minHeight: 350)
    }
}

What does this code do?

  • @Environment(ShapeGalleryModel.self) private var model retrieves the model you injected in ShapeGalleryApp.swift. This is the reading side of the environment injection you set up in the previous substep.
  • ForEach(model.shapes) loops over the four shapes in the model. Because ShapeData conforms to Identifiable, SwiftUI can track each item automatically.
  • Each HStack lays out the shape's emoji next to its name in a horizontal row.
  • Save ContentView.swift.
  • Build and run the app by pressing Cmd+R with the Apple Vision Pro Simulator selected as the target.
  • You should see a window with "Shape Gallery" as the title, the subtitle text, and four shapes listed with their emojis: Box, Sphere, Cone, and Cylinder.

Don't see the shape list?

  • Check that .environment(model) is applied to ContentView() in ShapeGalleryApp.swift. Without it, the @Environment property in ContentView will crash at runtime.
  • Make sure the type inside @Environment(ShapeGalleryModel.self) matches the class name exactly. A typo here causes a build error.
  • Verify that ShapeGalleryModel.swift is saved and listed under the ShapeGallery target in Xcode. Right-click the file, select Show File Inspector, and confirm the target membership checkbox is checked.

Still stuck? Help me debug why my @Observable model isn't showing data in ContentView.

Awesome, I've got everything!

Great. Double check you've saved ContentView.swift before moving on.

I'd like to double check the full code

import SwiftUI

struct ContentView: View {
    @Environment(ShapeGalleryModel.self) private var model

    var body: some View {
        VStack(alignment: .leading, spacing: 16) {
            Text("Shape Gallery")
                .font(.largeTitle)
                .fontWeight(.bold)

            Text("Tap a shape to change its color. Drag to reposition.")
                .font(.body)
                .foregroundStyle(.secondary)

            Divider()

            ForEach(model.shapes) { shape in
                HStack(spacing: 12) {
                    Text(shape.emoji)
                        .font(.title)
                    Text(shape.name)
                        .font(.title3)
                }
            }
        }
        .padding(32)
        .frame(minWidth: 300, minHeight: 350)
    }
}

Your data model is in place and driving the UI. Next up, you'll create a volumetric 3D window and fill it with four colorful shapes built from scratch with RealityKit.

Build the Volumetric 3D Shape Gallery

Your data model is wired up and the control window lists all four shapes. Now it's time to bring those shapes into 3D.

In this step, you'll create a volumetric window that displays four colorful 3D primitives using RealityKit. Each shape gets interaction components so it can respond to gaze and gestures later.

In this step, get ready to:
  • Create a RealityView with four procedurally generated 3D shapes.
  • Register a volumetric window in the app's scene definition.
  • Automatically open the 3D volume alongside the control window.
Create ShapeGalleryView with a RealityView

A RealityView is a SwiftUI view that hosts RealityKit 3D content. Its make closure runs once when the view first appears, and that's where you build your entities.

  • Right-click the ShapeGallery group in Xcode's Navigator sidebar.
  • Select New File.
  • Choose the Swift File template and name it ShapeGalleryView.
  • Click Create.
  • Replace the file's contents with this code:
import SwiftUI
import RealityKit

struct ShapeGalleryView: View {
    @Environment(ShapeGalleryModel.self) private var model

    var body: some View {
        RealityView { content in
            // Define properties for each shape
            let materialColors: [SimpleMaterial.Color] = [.blue, .yellow, .red, .green]
            let names = ["Box", "Sphere", "Cone", "Cylinder"]
            let xPositions: [Float] = [-0.2, -0.07, 0.07, 0.2]

            // Generate a different mesh for each primitive type
            let meshes: [MeshResource] = [
                .generateBox(size: 0.1),
                .generateSphere(radius: 0.05),
                .generateCone(height: 0.1, radius: 0.05),
                .generateCylinder(height: 0.1, radius: 0.05)
            ]

            // Create each entity with its mesh, material, and interaction components
            for i in 0..<4 {
                let material = SimpleMaterial(color: materialColors[i], isMetallic: false)
                let entity = ModelEntity(mesh: meshes[i], materials: [material])
                entity.name = names[i]
                entity.position = SIMD3<Float>(xPositions[i], 0, 0)

                // Required for gesture input detection
                entity.components.set(InputTargetComponent())
                entity.components.set(CollisionComponent(shapes: [ShapeResource.generateSphere(radius: 0.06)]))

                // Adds visual highlight on gaze
                entity.components.set(HoverEffectComponent())

                content.add(entity)
            }
        }
    }
}

What does this code do?

  • RealityView { content in ... } creates a SwiftUI view that hosts 3D content. The content parameter is the container you add entities to.
  • MeshResource generates the geometry for each shape. All measurements are in meters, so 0.1 means 10 centimeters.
  • SimpleMaterial applies a solid color. ModelEntity combines the mesh and material into a renderable object.
  • InputTargetComponent and CollisionComponent are both required for an entity to receive gesture input. Without either one, gestures silently fail.
  • HoverEffectComponent automatically highlights the entity when the user's gaze targets it. One line of code for built-in spatial feedback.
  • Save ShapeGalleryView.swift.

You won't see anything new yet. The app doesn't know about this view until you register a volumetric window for it.

Register the volumetric window and open it automatically

A visionOS app can display multiple windows. A regular WindowGroup creates a flat 2D window. Applying .windowStyle(.volumetric) turns it into a 3D volume that floats in the user's space.

  • Open ShapeGalleryApp.swift.
  • Find the closing brace of the existing WindowGroup block (the one containing ContentView()) and add this second window group below it:
WindowGroup(id: "ShapeVolume") {
    ShapeGalleryView()
        .environment(model)
}
.windowStyle(.volumetric)
.defaultSize(width: 0.6, height: 0.3, depth: 0.3, in: .meters)

What does this code do?

  • WindowGroup(id: "ShapeVolume") registers a named window that you can open programmatically using that ID.
  • .windowStyle(.volumetric) makes this window a 3D volume instead of a flat panel.
  • .defaultSize(width: 0.6, height: 0.3, depth: 0.3, in: .meters) sets the volume to 60 cm wide, 30 cm tall, and 30 cm deep.
  • .environment(model) injects the same shared data model into the volumetric view.
  • Save ShapeGalleryApp.swift.

The volume is registered, but nothing opens it yet. You need the control window to trigger the volume when it appears.

  • Open ContentView.swift.
  • Find the line:
@Environment(ShapeGalleryModel.self) private var model
  • Add this new environment property directly below it:
@Environment(\.openWindow) private var openWindow
  • Now find the .frame(minWidth: 300, minHeight: 350) modifier near the bottom of the view body.
  • Add this modifier directly after it:
.onAppear {
    openWindow(id: "ShapeVolume")
}

What do these additions do?

  • @Environment(\.openWindow) grabs a system action that lets you open any named window by its ID.
  • .onAppear fires once when the control window first renders. It calls openWindow(id: "ShapeVolume") to launch the volumetric window automatically.
  • Save ContentView.swift.

Awesome, I've got everything!

Make sure you've saved both ShapeGalleryApp.swift and ContentView.swift before running.

I'd like to double check the full code

Here is what ShapeGalleryApp.swift should look like at this point.

import SwiftUI

@main
struct ShapeGalleryApp: App {
    @State private var model = ShapeGalleryModel()

    var body: some Scene {
        WindowGroup {
            ContentView()
                .environment(model)
        }

        WindowGroup(id: "ShapeVolume") {
            ShapeGalleryView()
                .environment(model)
        }
        .windowStyle(.volumetric)
        .defaultSize(width: 0.6, height: 0.3, depth: 0.3, in: .meters)
    }
}

And here is what ContentView.swift should look like at this point.

import SwiftUI

struct ContentView: View {
    @Environment(ShapeGalleryModel.self) private var model
    @Environment(\.openWindow) private var openWindow

    var body: some View {
        VStack(alignment: .leading, spacing: 16) {
            Text("Shape Gallery")
                .font(.largeTitle)
                .fontWeight(.bold)

            Text("Tap a shape to change its color. Drag to reposition.")
                .font(.body)
                .foregroundStyle(.secondary)

            Divider()

            ForEach(model.shapes) { shape in
                HStack(spacing: 12) {
                    Text(shape.emoji)
                        .font(.title)
                    Text(shape.name)
                        .font(.title3)
                }
            }
        }
        .padding(32)
        .frame(minWidth: 300, minHeight: 350)
        .onAppear {
            openWindow(id: "ShapeVolume")
        }
    }
}
Run the app and verify both windows
  • Build and run the project by pressing Cmd+R.
  • Make sure the target is set to Apple Vision Pro in the Simulator toolbar.

Two windows should appear in the Simulator. The flat control window shows the shape list from Step 2. The volumetric window shows four colored 3D shapes floating in a row.

  • Move the mouse pointer over each shape in the volumetric window. You should see a subtle highlight appear on the shape you're pointing at.

That highlight is the HoverEffectComponent at work. In the Simulator, mouse hover stands in for the gaze tracking that would happen on a real Apple Vision Pro.

Only see the flat window?

  • Check that the id string in WindowGroup(id: "ShapeVolume") matches the string in openWindow(id: "ShapeVolume") exactly. A typo in either one causes a silent failure.
  • Make sure the second WindowGroup block is inside the body computed property, at the same level as the first WindowGroup. It should not be nested inside the first one.
  • Verify that .environment(model) is applied to ShapeGalleryView() inside the second window group. Without it, the app crashes because the view expects the model in its environment.
  • help me debug why my volumetric window isn't appearing in the visionOS Simulator

Shapes not visible in the volume?

  • The shapes may be behind the camera. Use the WASD keys to move the camera around in the Simulator until you can see them.
  • Confirm that content.add(entity) is inside the for loop. If it is outside the loop, only the last shape gets added.
  • my shapes are not showing up in the volumetric window even though the window appears

Your shapes are floating in 3D space, ready for interaction. Next up, you'll make them spin continuously using a custom Entity Component System with quaternion rotation math.

Add the Custom ECS Rotation System

Your volumetric window is alive with four colorful shapes floating in 3D space. They look great, but they're completely still.

In this step, you'll bring them to life with continuous rotation using RealityKit's Entity Component System pattern. ECS separates data (what an entity has) from behavior (what happens to it). It's the architecture pattern behind every production RealityKit app.

In this step, get ready to:
  • Create a custom RotationComponent that stores rotation data for each entity.
  • Build a RotationSystem that applies quaternion rotation every frame.
  • Register both and attach them to your shapes so they spin at different speeds.
Create the rotation component

The ECS pattern has two halves. The Component is a plain data container. It holds values but has zero behavior on its own.

What is the Entity Component System?

In ECS, entities are the objects in your scene (your box, sphere, cone, cylinder). Components are bags of data you attach to entities. Systems are loops that run every frame, reading and writing component data to create behavior.

This separation makes it easy to add new behaviors without rewriting entity classes. You just attach a new component and write a new system.

  • Create a new Swift file called RotationComponent.swift in your ShapeGallery project by pressing Cmd+N in Xcode, selecting Swift File, and clicking Next.
  • Name the file RotationComponent and click Create.
  • Replace all the default content with this code:
import RealityKit

// A custom ECS component that stores rotation data
struct RotationComponent: Component {
    // How fast the entity rotates (radians per second)
    var speed: Float = 1.0
    // Tracks the cumulative rotation angle
    var angle: Float = 0.0
}

What does this code do?

  • RotationComponent conforms to RealityKit's Component protocol, which tells RealityKit this struct can be attached to any entity.
  • speed controls how fast the entity rotates. Different values per shape will create varied animation.
  • angle accumulates over time so the rotation is smooth and continuous across frames.
  • Save RotationComponent.swift.

You won't see a change yet. The component is just data. The next file creates the system that reads this data and turns it into motion.

Create the rotation system

The System is the behavior half of ECS. It runs every frame, finds all entities that have a specific component, and applies logic to them.

  • Create a new Swift file called RotationSystem.swift by pressing Cmd+N again.
  • Name the file RotationSystem and click Create.
  • Replace all the default content with this code:
import RealityKit

// The system that processes all entities with a RotationComponent
struct RotationSystem: System {
    // Find every entity that has a RotationComponent attached
    static let query = EntityQuery(where: .has(RotationComponent.self))

    init(scene: RealityKit.Scene) {}

    func update(context: SceneUpdateContext) {
        for entity in context.entities(
            matching: Self.query,
            updatingSystemWhen: .rendering
        ) {
            // Read the current rotation data from the entity
            guard var rotation = entity.components[RotationComponent.self] else { continue }
            // Increment the angle based on speed and time since last frame
            rotation.angle += rotation.speed * Float(context.deltaTime)
            // Apply a Y-axis quaternion rotation to spin the entity
            entity.orientation = simd_quatf(angle: rotation.angle, axis: [0, 1, 0])
            // Write the updated angle back to the component
            entity.components.set(rotation)
        }
    }
}

What does this code do?

  • query uses an EntityQuery to filter the scene down to only entities that have a RotationComponent. The system ignores everything else.
  • update(context:) runs every frame. It multiplies speed by deltaTime (seconds since the last frame) so rotation stays smooth regardless of frame rate.
  • simd_quatf(angle:axis:) creates a quaternion rotation. The axis [0, 1, 0] means the entity spins around the Y-axis (like a top).
  • After updating the angle, the system writes the modified component back to the entity with entity.components.set(rotation). Without this line, the angle would reset every frame.

What is a quaternion?

A quaternion is a way to represent 3D rotation without the problems that come with using three separate angles (like gimbal lock). You give it an angle and an axis to rotate around, and it handles the math.

In this project, simd_quatf is Apple's quaternion type. Passing [0, 1, 0] as the axis means "rotate around Y," which spins shapes like a turntable.

  • Save RotationSystem.swift.

You won't see any rotation yet. The component and system exist but RealityKit doesn't know about them. The next substep registers them and attaches the component to your shapes.

Register and wire up the rotation

Custom ECS types must be registered with RealityKit before any scene loads. If you register them too late, RealityKit silently discards the component data.

  • Open ShapeGalleryApp.swift.
  • Find the line @State private var model = ShapeGalleryModel().
  • Add this init() block directly below it, before var body:
init() {
    // Register the custom ECS types before any scene loads
    RotationComponent.registerComponent()
    RotationSystem.registerSystem()
}

Why register in init()?

The App struct's init() runs before any scene or window loads. This guarantees RealityKit knows about your custom types before it encounters them in a RealityView.

Now you need to attach the RotationComponent to each shape entity so the system has something to process.

  • Open ShapeGalleryView.swift.
  • Inside the RealityView closure, find the array declarations at the top. Locate this line:
let xPositions: [Float] = [-0.2, -0.07, 0.07, 0.2]
  • Add this line directly below it to define a rotation speed for each shape:
let speeds: [Float] = [0.5, 0.7, 0.9, 1.1]
  • Now scroll down to the for loop body. Find this line:
entity.components.set(HoverEffectComponent())
  • Add this line directly below it to attach the rotation component with a per-shape speed:
entity.components.set(RotationComponent(speed: speeds[i]))

What does this line do?

Each shape gets a RotationComponent with a different speed: 0.5, 0.7, 0.9, and 1.1 radians per second. The box spins the slowest and the cylinder spins the fastest. This variation makes the gallery feel dynamic instead of mechanical.

  • Save both ShapeGalleryApp.swift and ShapeGalleryView.swift.
  • Build and run the app by pressing Cmd+R.

In the Simulator, you should see all four shapes spinning around their Y-axis at visibly different speeds. The box turns slowly while the cylinder spins noticeably faster.

Shapes not rotating?

  • Make sure RotationComponent.registerComponent() and RotationSystem.registerSystem() are inside the App's init(). If they're called after the scene loads, RealityKit ignores them.
  • Check that the entity.components.set(RotationComponent(speed: speeds[i])) line is inside the for loop, not outside it.
  • Verify that RotationSystem.swift uses .rendering in updatingSystemWhen:. Using a different value means the system won't run every frame.

Why aren't my shapes rotating in the visionOS Simulator?

Awesome, I've got everything!

Great. Double check you've saved all your files before moving on.

I'd like to double check the full code

Here are the two files you edited in this step. Compare them against your code.

import SwiftUI

@main
struct ShapeGalleryApp: App {
    @State private var model = ShapeGalleryModel()

    init() {
        RotationComponent.registerComponent()
        RotationSystem.registerSystem()
    }

    var body: some Scene {
        WindowGroup {
            ContentView()
                .environment(model)
        }

        WindowGroup(id: "ShapeVolume") {
            ShapeGalleryView()
                .environment(model)
        }
        .windowStyle(.volumetric)
        .defaultSize(width: 0.6, height: 0.3, depth: 0.3, in: .meters)
    }
}
import SwiftUI
import RealityKit

struct ShapeGalleryView: View {
    var body: some View {
        RealityView { content in
            let materialColors: [SimpleMaterial.Color] = [.blue, .yellow, .red, .green]
            let names = ["Box", "Sphere", "Cone", "Cylinder"]
            let xPositions: [Float] = [-0.2, -0.07, 0.07, 0.2]
            let speeds: [Float] = [0.5, 0.7, 0.9, 1.1]

            let meshes: [MeshResource] = [
                .generateBox(size: 0.1),
                .generateSphere(radius: 0.05),
                .generateCone(height: 0.1, radius: 0.05),
                .generateCylinder(height: 0.1, radius: 0.05)
            ]

            for i in 0..<4 {
                let material = SimpleMaterial(color: materialColors[i], isMetallic: false)
                let entity = ModelEntity(mesh: meshes[i], materials: [material])
                entity.name = names[i]
                entity.position = SIMD3<Float>(xPositions[i], 0, 0)
                entity.components.set(InputTargetComponent())
                entity.components.set(CollisionComponent(shapes: [ShapeResource.generateSphere(radius: 0.06)]))
                entity.components.set(HoverEffectComponent())
                entity.components.set(RotationComponent(speed: speeds[i]))
                content.add(entity)
            }
        }
    }
}

Your shapes are spinning at different speeds, driven entirely by the ECS pattern. Next up, you'll add tap and drag gestures so you can interact with these shapes directly in 3D space.

Add Tap and Drag Gesture Interaction

Your shapes are spinning at different speeds thanks to the custom rotation system. They look alive, but you can't interact with them yet.

In this step, you'll add two spatial gestures to your shapes. A tap gesture randomizes each shape's color, and a drag gesture lets you reposition shapes anywhere in the 3D volume.

In this step, get ready to:
  • Add a tap gesture that changes shape colors on tap.
  • Add a drag gesture that repositions shapes in 3D space.
  • Display the selected shape in the control window.
Add a tap gesture to change shape colors

Every entity that responds to gestures needs two components you already added in Step 3: InputTargetComponent and CollisionComponent. With those in place, you can attach a SpatialTapGesture to the RealityView that detects taps on any entity.

  • In ShapeGalleryView.swift, find the closing } of the RealityView block.
  • Add this .gesture() modifier directly after it:
.gesture(
    SpatialTapGesture()
        .targetedToAnyEntity()
        .onEnded { value in
            // Get the entity that was tapped
            let tappedEntity = value.entity

            // Pick a random color from the array
            let randomColor: SimpleMaterial.Color = [
                .red, .blue, .green, .yellow,
                .orange, .purple, .cyan, .magenta
            ].randomElement() ?? .white

            // Create a new material with the random color
            let newMaterial = SimpleMaterial(color: randomColor, isMetallic: false)

            // Swap the entity's material to the new one
            if var modelComponent = tappedEntity.components[ModelComponent.self] {
                modelComponent.materials = [newMaterial]
                tappedEntity.components.set(modelComponent)
            }

            // Update the shared data model with the tapped shape's name
            model.selectedShapeName = tappedEntity.name
        }
)

What does this code do?

  • .targetedToAnyEntity() tells SwiftUI to detect taps on any RealityKit entity that has both InputTargetComponent and CollisionComponent.
  • value.entity gives you the specific entity that was tapped, so you know which shape to update.
  • The code reads the entity's ModelComponent, swaps its material to a random color, and writes it back. This is the standard ECS read-modify-write pattern.
  • model.selectedShapeName = tappedEntity.name pushes the tapped shape's name into the shared data model, so the control window can display it.
  • Save ShapeGalleryView.swift.
  • Build and run the app with Cmd+R. Click on any shape in the volumetric window. You should see the shape's color change to a random new color.

Shape doesn't change color when you click?

  • Make sure the .gesture() modifier is chained after the RealityView { ... } closing brace, not inside the make closure.
  • Verify that each entity has both InputTargetComponent() and CollisionComponent from Step 3. Without both, gestures silently fail.
  • In the Simulator, click directly on a shape, not on empty space. The mouse click simulates a gaze-plus-pinch tap.

Help me debug why my SpatialTapGesture isn't working on entities in visionOS

Add a drag gesture to reposition shapes

The tap gesture uses .gesture(), but adding a second gesture requires .simultaneousGesture(). This tells SwiftUI to recognize both gestures at the same time instead of one blocking the other.

  • In ShapeGalleryView.swift, find the closing ) of the .gesture(SpatialTapGesture()...) block you just added.
  • Add this .simultaneousGesture() modifier directly after it:
.simultaneousGesture(
    DragGesture()
        .targetedToAnyEntity()
        .onChanged { value in
            // Convert the drag location to the parent entity's coordinate space
            let newPosition = value.convert(value.location3D, from: .local, to: value.entity.parent!)
            value.entity.position = newPosition
        }
)

What does this code do?

  • DragGesture() tracks finger movement (or mouse drag in the Simulator) across 3D space.
  • value.convert(value.location3D, from: .local, to: value.entity.parent!) translates the drag coordinates from the gesture's local space into the parent entity's coordinate system. Without this conversion, the shape would jump to unexpected positions.
  • Setting value.entity.position = newPosition moves the shape to follow the drag in real time.

Why simultaneousGesture instead of a second gesture modifier?

If you used .gesture() for both the tap and drag, only one would be recognized at a time. The .simultaneousGesture() modifier lets both gestures stay active on the same view. This is the standard pattern for multi-gesture handling in visionOS.

  • Save ShapeGalleryView.swift.
  • Build and run the app with Cmd+R. Click and drag a shape in the volumetric window. You should see it move to follow the drag.

Shape doesn't move when you drag?

  • Check that you used .simultaneousGesture() for the drag, not a second .gesture(). A second .gesture() overrides the tap gesture instead of running alongside it.
  • Make sure .onChanged is used instead of .onEnded. The drag needs to update position continuously while the gesture is happening.
  • In the Simulator, click and hold on a shape, then drag the mouse. A quick click registers as a tap, not a drag.

Help me debug why my DragGesture isn't repositioning entities in visionOS

Awesome, I've got everything!

Double check you've saved ShapeGalleryView.swift before moving on.

I'd like to double check the full code

import SwiftUI
import RealityKit

struct ShapeGalleryView: View {
    @Environment(ShapeGalleryModel.self) private var model

    var body: some View {
        RealityView { content in
            let materialColors: [SimpleMaterial.Color] = [.blue, .yellow, .red, .green]
            let names = ["Box", "Sphere", "Cone", "Cylinder"]
            let xPositions: [Float] = [-0.2, -0.07, 0.07, 0.2]
            let speeds: [Float] = [0.5, 0.7, 0.9, 1.1]

            let meshes: [MeshResource] = [
                .generateBox(size: 0.1),
                .generateSphere(radius: 0.05),
                .generateCone(height: 0.1, radius: 0.05),
                .generateCylinder(height: 0.1, radius: 0.05)
            ]

            for i in 0..<4 {
                let material = SimpleMaterial(color: materialColors[i], isMetallic: false)
                let entity = ModelEntity(mesh: meshes[i], materials: [material])
                entity.name = names[i]
                entity.position = SIMD3<Float>(xPositions[i], 0, 0)
                entity.components.set(InputTargetComponent())
                entity.components.set(CollisionComponent(shapes: [ShapeResource.generateSphere(radius: 0.06)]))
                entity.components.set(HoverEffectComponent())
                entity.components.set(RotationComponent(speed: speeds[i]))
                content.add(entity)
            }
        }
        .gesture(
            SpatialTapGesture()
                .targetedToAnyEntity()
                .onEnded { value in
                    let tappedEntity = value.entity
                    let randomColor: SimpleMaterial.Color = [
                        .red, .blue, .green, .yellow,
                        .orange, .purple, .cyan, .magenta
                    ].randomElement() ?? .white
                    let newMaterial = SimpleMaterial(color: randomColor, isMetallic: false)
                    if var modelComponent = tappedEntity.components[ModelComponent.self] {
                        modelComponent.materials = [newMaterial]
                        tappedEntity.components.set(modelComponent)
                    }
                    model.selectedShapeName = tappedEntity.name
                }
        )
        .simultaneousGesture(
            DragGesture()
                .targetedToAnyEntity()
                .onChanged { value in
                    let newPosition = value.convert(value.location3D, from: .local, to: value.entity.parent!)
                    value.entity.position = newPosition
                }
        )
    }
}
Display the selected shape in the control window

The tap gesture updates model.selectedShapeName every time you tap a shape. The control window doesn't show this yet, so let's connect it.

  • Open ContentView.swift.
  • Find the closing } of the ForEach block that lists the shapes.
  • Add the following code below it, before the closing } of the VStack:
Divider()

HStack {
    Text("Selected:")
        .font(.headline)
    // Show the selected shape name, or "None" if nothing is selected
    Text(model.selectedShapeName ?? "None")
        .font(.headline)
        .foregroundStyle(model.selectedShapeName != nil ? .primary : .secondary)
}

What does this code do?

  • model.selectedShapeName ?? "None" uses Swift's nil coalescing operator. If no shape has been tapped yet, the label shows "None". Otherwise it shows the tapped shape's name.
  • The .foregroundStyle() ternary switches between .primary and .secondary color to visually distinguish a selected shape from the empty state.
  • Because ShapeGalleryModel is @Observable, SwiftUI automatically re-renders this label whenever selectedShapeName changes. No manual notification code needed.
  • Save ContentView.swift.
  • Build and run the app with Cmd+R.
  • Click on a shape in the volumetric window. The control window should update to show "Selected: Box" (or whichever shape you tapped).
  • Click and drag a shape. It should follow the drag and reposition in 3D space.
  • Hover the mouse over a shape without clicking. You should see a subtle highlight effect from HoverEffectComponent.

Selected label not updating?

  • Make sure ContentView retrieves the model with @Environment(ShapeGalleryModel.self) private var model. The tap gesture in ShapeGalleryView and ContentView must share the same model instance.
  • Verify that ShapeGalleryApp.swift injects .environment(model) on both WindowGroup scenes. If one is missing, that view won't see updates.

Help me debug why my @Observable model isn't syncing between windows in visionOS

Awesome, I've got everything!

Double check you've saved ContentView.swift before moving on.

I'd like to double check the full code

import SwiftUI

struct ContentView: View {
    @Environment(ShapeGalleryModel.self) private var model
    @Environment(\.openWindow) private var openWindow

    var body: some View {
        VStack(alignment: .leading, spacing: 16) {
            Text("Shape Gallery")
                .font(.largeTitle)
                .fontWeight(.bold)

            Text("Tap a shape to change its color. Drag to reposition.")
                .font(.body)
                .foregroundStyle(.secondary)

            Divider()

            ForEach(model.shapes) { shape in
                HStack(spacing: 12) {
                    Text(shape.emoji)
                        .font(.title)
                    Text(shape.name)
                        .font(.title3)
                }
            }

            Divider()

            HStack {
                Text("Selected:")
                    .font(.headline)
                Text(model.selectedShapeName ?? "None")
                    .font(.headline)
                    .foregroundStyle(model.selectedShapeName != nil ? .primary : .secondary)
            }
        }
        .padding(32)
        .frame(minWidth: 300, minHeight: 350)
        .onAppear {
            openWindow(id: "ShapeVolume")
        }
    }
}

Secret mission

Add Particle Burst on Tap

Your shapes already react to taps by changing color. But what if they could also burst with sparks? Take your spatial interactions to the next level by adding a ParticleEmitterComponent burst effect whenever you tap a shape.

Before you go

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally using Xcode and the visionOS Simulator, so there are no ongoing costs.

Resources you used:

  • ShapeGallery Xcode project (local files on your Mac).
  • visionOS Simulator runtime (installed via Xcode).
  • RealityKit content package (bundled inside the Xcode project).

Keep everything running

No action needed. Choose this if you're still actively building or plan to extend your ShapeGallery app with more features.

  • Your Xcode project stays on disk and is ready to open anytime.
  • The visionOS Simulator runtime remains installed for future projects.
  • There are no background processes or ongoing costs to worry about.

Pause - I'll come back to this later

Shut down running processes to free up memory while keeping your project files for later.

  • Stop the running app in Xcode by clicking the Stop button or pressing Cmd+..
  • Quit the visionOS Simulator from the menu bar to free up system resources.
  • Your project files remain on disk. Open the ShapeGallery.xcodeproj file to pick up where you left off.

Delete - I don't want to use this again

Remove all project resources and start fresh.

  • Stop the running app and quit the visionOS Simulator.
  • Delete the ShapeGallery project folder from your Mac by moving it to the Trash.
  • Optionally, remove the visionOS Simulator runtime to reclaim disk space. Go to Xcode > Settings > Platforms, select the visionOS runtime, and click the delete button.
  • Keep Xcode installed if you plan to work on other Apple platform projects.

Mission accomplished

That's a Wrap!

Nice work! You just built a fully interactive visionOS spatial computing prototype from scratch. This is exactly the kind of work the Apple Spatial Platform team does every day.

You've learned how to:

  • Build a visionOS app with an @Observable data model driving both a SwiftUI control window and a volumetric 3D shape gallery.
  • Create a custom RotationComponent and RotationSystem using the Entity Component System pattern with simd_quatf quaternion math for continuous 3D animation.
  • Implement multi-gesture spatial interaction with SpatialTapGesture for color changes and DragGesture for repositioning 3D entities in space.
  • Add ParticleEmitterComponent burst effects that fire sparks when tapping a shape for visual polish.

Ready to quiz yourself?