Build a Local AI 3D Part Generator
Turn text prompts into interactive 3D models using a local LLM and Three.js.
Introduction
30 Second Summary
3D modeling tools are powerful, but they come with steep learning curves and complex interfaces. What if you could just describe what you want in plain English and watch it appear in 3D?
In this project, you will build a local web app that turns text descriptions into interactive 3D models using AI. You'll type prompts like "a table with 4 legs" and see 3D shapes appear on screen, all running on your own machine with zero cloud costs.
What You'll Build
A single-page web application that sends your text prompts to a local AI model, which returns structured data describing 3D shapes, positions, and colors that Three.js renders as an interactive 3D scene.
By the end of this project, you'll have:
- A working local AI 3D part generator that converts plain-text prompts into 3D models you can orbit and zoom.
- Hands-on experience calling a local LLM REST API with structured JSON output to enforce reliable responses from AI.
- An interactive Three.js 3D viewer loaded from a CDN using import maps with orbit controls for mouse interaction.
- Secret Mission: Add conversational refinement so you can say "make it taller" or "add 2 more legs" to iteratively evolve your design.
Are there any prerequisites?
This guide includes setup for all required tools. You just need a Windows PC with at least 8GB of RAM and Ollama installed. If you don't have Ollama yet, download it from ollama.com before starting.
Set Up Your Environment
Before you can generate 3D parts with AI, you need three things in place: a code editor to write your app, the AI model that will produce structured output, and a confirmed connection to the model's API.
In this step, you will install VS Code, create your project folder, download the qwen2.5-coder:3b model, and verify it responds to API requests.
In this step, get ready to:
- Install VS Code and create your project folder.
- Pull the qwen2.5-coder:3b model for structured JSON output.
- Verify the Ollama REST API is reachable at localhost.
Install VS Code
VS Code is the code editor you will use to write your single-page 3D app. It is free, lightweight, and works on every platform.
- Check if VS Code is already installed by pressing the Windows key to open your search bar.
- Type Visual Studio Code and check if it appears in the results.
✔️ I see Visual Studio Code in the results
VS Code is already installed. Click it to open it, then skip ahead to creating your project folder below.
ⓧ I don't see it
- Open your browser and go to https://code.visualstudio.com/.
- Click the large Download for Windows button.
- Run the downloaded installer and accept all the default options.
- Once the installer finishes, launch VS Code.
Installer not running?
Make sure you downloaded the correct version for your system (64-bit is the default for modern Windows). If your browser blocked the download, check your Downloads folder and manually double-click the .exe file.
Help me troubleshoot my VS Code installation.
Create your project folder
Every project needs a home. You will create a folder on your Desktop so it is easy to find later.
- Open a terminal inside VS Code by pressing Ctrl+` (the backtick key, above Tab).
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir ai-3d-generator
cd ai-3d-generator- Open this folder in VS Code by running:
code .What does this command do?
The code . command tells VS Code to open the current directory as your workspace. The dot (.) means "this folder." VS Code may open a new window with your ai-3d-generator folder in the sidebar.
- Confirm you see ai-3d-generator in the VS Code title bar or the Explorer sidebar on the left.
Don't see the folder name in VS Code?
If code . did not open a new window, go to File in the top menu bar, click Open Folder, and navigate to Desktop > ai-3d-generator. Click Select Folder.
Help me open my project folder in VS Code.
Pull the coding model
Ollama downloads models on demand. The qwen2.5-coder:3b model is a 3-billion parameter coding specialist that fits in 8GB of RAM and produces reliable structured JSON output.
- Open a new terminal in VS Code by pressing Ctrl+` if one is not already open.
- Download the model by running this command:
ollama pull qwen2.5-coder:3bWhat does this command do?
This downloads approximately 1.8GB of model weights to your machine. The model runs entirely on your CPU, so no GPU is required. Expect the download to take a few minutes depending on your internet speed.
- Wait for the download to complete. You will see a progress bar filling up.
- Once the download finishes, verify the model works by running:
ollama run qwen2.5-coder:3b "Say hello"- You should see a text response from the model (something like "Hello! How can I help you?").
- Type /bye and press Enter to exit the interactive session.
Seeing an error about model not found?
Make sure the Ollama application is running in the background. Look for the Ollama icon in your system tray (bottom-right of your taskbar). If it is not there, search for Ollama in the Windows search bar and launch it.
If you see a network error during the pull, check your internet connection and run the ollama pull command again. It resumes where it left off.
Help me fix my Ollama model download.
Verify the Ollama API is accessible
Your web app will communicate with the model through a REST API that Ollama runs on your machine at http://localhost:11434. Confirming this works now saves debugging time later.
- In your VS Code terminal, run this command to send a test request to the API:
curl http://localhost:11434/api/generate -d "{\"model\": \"qwen2.5-coder:3b\", \"prompt\": \"hi\", \"stream\": false}"- You should see a JSON response in your terminal containing a response field with a short text reply from the model.
What does this command do?
The curl command sends an HTTP POST request to Ollama's /api/generate endpoint. The JSON body tells Ollama which model to use, what prompt to send, and that you want the full response at once (stream: false) instead of word-by-word streaming.
This is the same pattern your web app will use to communicate with the model, except it will call the /api/chat endpoint with a system prompt and schema.
Seeing 'connection refused' or no response?
This means Ollama is not running. Look for the Ollama icon in your system tray and make sure it is active. If it is missing, search for Ollama in the Windows search bar and launch it, then try the command again.
If curl is not recognized, open your browser and navigate to http://localhost:11434 instead. You should see the text "Ollama is running" on the page.
Help me connect to the Ollama API.
Your environment is ready. You have VS Code open with your project folder, a local coding model downloaded, and a confirmed API connection. Next up, you will build the 3D viewer that displays the models your AI generates.
Build the 3D Viewer
Your local AI is ready to go. Now you need a visual interface where you can type a prompt and see the 3D result.
In this step, you will create a browser-based 3D viewport using Three.js. The page loads the library directly from a CDN using an import map, so there is no build step or package manager needed.
In this step, get ready to:
- Create the HTML file with a full-screen dark layout and import map.
- Build an interactive 3D scene with camera, lighting, and orbit controls.
Create the HTML file with the page layout
Your entire app lives in a single index.html file. This file contains the page structure, styling, and an import map that tells the browser where to find Three.js on a CDN.
- In VS Code, right-click inside the Explorer sidebar and select New File.
- Name the file index.html and press Enter.
- Paste the following code into index.html to set up the HTML structure, styling, and import map:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AI 3D Part Generator</title>
<style>
/* Reset default margins and use border-box sizing */
* { margin: 0; padding: 0; box-sizing: border-box; }
/* Full-screen dark layout with vertical flex */
body {
font-family: system-ui, sans-serif;
background: #1a1a2e;
color: #eee;
height: 100vh;
display: flex;
flex-direction: column;
overflow: hidden;
}
/* 3D viewport fills all available vertical space */
#viewport { flex: 1; }
/* Prompt bar pinned to the bottom */
#prompt-bar {
display: flex;
padding: 12px 16px;
background: #16213e;
border-top: 1px solid #0f3460;
gap: 8px;
align-items: center;
}
/* Text input stretches to fill the bar */
#prompt-input {
flex: 1;
padding: 10px 16px;
border: 1px solid #0f3460;
border-radius: 6px;
background: #1a1a2e;
color: #eee;
font-size: 14px;
outline: none;
}
#prompt-input:focus { border-color: #e94560; }
#prompt-input::placeholder { color: #666; }
/* Status text for loading indicators */
#status {
font-size: 12px;
color: #e94560;
min-width: 100px;
text-align: right;
}
</style>
</head>
<body>
<!-- 3D scene renders inside this div -->
<div id="viewport"></div>
<!-- Prompt bar at the bottom of the screen -->
<div id="prompt-bar">
<input type="text" id="prompt-input" placeholder="Describe a 3D part... (e.g. 'a table with 4 legs')" />
<span id="status"></span>
</div>
<!-- Import map tells the browser where to find Three.js modules -->
<script type="importmap">
{
"imports": {
"three": "https://cdn.jsdelivr.net/npm/three@0.184.0/build/three.module.js",
"three/addons/": "https://cdn.jsdelivr.net/npm/three@0.184.0/examples/jsm/"
}
}
</script>
<script type="module">
</script>
</body>
</html>What does this code do?
The CSS uses flexbox to create a two-part layout: the #viewport div stretches to fill the screen, and the #prompt-bar sticks to the bottom. Everything is styled with a dark color palette.
The import map is a browser-native feature that maps short names like 'three' to full CDN URLs. This lets you write clean import statements in your JavaScript without any build tools.
The empty <script type="module"> tag is where your Three.js code will go next.
- Save index.html by pressing Ctrl+S.
File not appearing in the sidebar?
Make sure you created the file inside the ai-3d-generator folder. If the Explorer sidebar is empty, click File then Open Folder and select the ai-3d-generator folder on your Desktop.
Help me find my project folder in VS Code.
Add the Three.js scene initialization
Three.js needs a scene (the 3D world), a camera (the viewpoint), a renderer (draws pixels to the screen), and lights (so objects are visible). You will set all of these up inside the module script.
- In index.html, find the empty <script type="module"> tag near the bottom of the file.
- Paste the following code between the opening and closing <script type="module"> tags:
import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';
// Grab references to DOM elements
const viewport = document.getElementById('viewport');
const promptInput = document.getElementById('prompt-input');
const status = document.getElementById('status');
// Create the 3D scene with a dark background
const scene = new THREE.Scene();
scene.background = new THREE.Color(0x1a1a2e);
// Set up a perspective camera positioned above and to the side
const camera = new THREE.PerspectiveCamera(60, viewport.clientWidth / viewport.clientHeight, 0.1, 1000);
camera.position.set(5, 4, 5);
camera.lookAt(0, 0, 0);
// Create the WebGL renderer and attach it to the viewport div
const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(viewport.clientWidth, viewport.clientHeight);
renderer.setPixelRatio(window.devicePixelRatio);
viewport.appendChild(renderer.domElement);
// Enable orbit controls for mouse-based rotation and zoom
const controls = new OrbitControls(camera, renderer.domElement);
controls.enableDamping = true;
// Add ambient light so all surfaces are partially lit
const ambientLight = new THREE.AmbientLight(0xffffff, 0.6);
scene.add(ambientLight);
// Add directional light for shadows and depth
const directionalLight = new THREE.DirectionalLight(0xffffff, 0.8);
directionalLight.position.set(5, 10, 5);
scene.add(directionalLight);
// Add a grid on the ground plane as a visual reference
const gridHelper = new THREE.GridHelper(10, 10, 0x0f3460, 0x0f3460);
scene.add(gridHelper);
// Animation loop: updates controls and re-renders every frame
function animate() {
controls.update();
renderer.render(scene, camera);
}
renderer.setAnimationLoop(animate);What does this code do?
- PerspectiveCamera simulates how human eyes see depth. The 60-degree field of view gives a natural look.
- WebGLRenderer draws the 3D scene to a <canvas> element that fills the viewport div.
- OrbitControls lets you click and drag to rotate, scroll to zoom, and right-click to pan around the scene.
- The animation loop calls renderer.render() every frame so any changes to the scene appear immediately.
- Save index.html.
- Open File Explorer (Windows) or Finder (macOS) and navigate to the ai-3d-generator folder on your Desktop.
- Double-click index.html to open it in your browser.
- You should see a dark 3D viewport with a grid floor and a text input at the bottom of the screen.
- Click and drag inside the viewport to orbit around the grid. Scroll to zoom in and out.
Seeing a blank dark screen with no grid?
Make sure you pasted the JavaScript code between the <script type="module"> and </script> tags, not outside them. The script must have type="module" for imports to work.
If you see errors in the browser console (press F12 to open DevTools), check that you have an active internet connection. The import map loads Three.js from a CDN, so it needs network access.
My 3D viewport is not rendering the grid or responding to mouse controls.
✔️ Awesome, I've got everything!
Great. Double check you have saved your file before moving on.
ⓧ I'd like to double check the full code
Here is the complete index.html at this point:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AI 3D Part Generator</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: system-ui, sans-serif;
background: #1a1a2e;
color: #eee;
height: 100vh;
display: flex;
flex-direction: column;
overflow: hidden;
}
#viewport { flex: 1; }
#prompt-bar {
display: flex;
padding: 12px 16px;
background: #16213e;
border-top: 1px solid #0f3460;
gap: 8px;
align-items: center;
}
#prompt-input {
flex: 1;
padding: 10px 16px;
border: 1px solid #0f3460;
border-radius: 6px;
background: #1a1a2e;
color: #eee;
font-size: 14px;
outline: none;
}
#prompt-input:focus { border-color: #e94560; }
#prompt-input::placeholder { color: #666; }
#status {
font-size: 12px;
color: #e94560;
min-width: 100px;
text-align: right;
}
</style>
</head>
<body>
<div id="viewport"></div>
<div id="prompt-bar">
<input type="text" id="prompt-input" placeholder="Describe a 3D part... (e.g. 'a table with 4 legs')" />
<span id="status"></span>
</div>
<script type="importmap">
{
"imports": {
"three": "https://cdn.jsdelivr.net/npm/three@0.184.0/build/three.module.js",
"three/addons/": "https://cdn.jsdelivr.net/npm/three@0.184.0/examples/jsm/"
}
}
</script>
<script type="module">
import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';
const viewport = document.getElementById('viewport');
const promptInput = document.getElementById('prompt-input');
const status = document.getElementById('status');
const scene = new THREE.Scene();
scene.background = new THREE.Color(0x1a1a2e);
const camera = new THREE.PerspectiveCamera(60, viewport.clientWidth / viewport.clientHeight, 0.1, 1000);
camera.position.set(5, 4, 5);
camera.lookAt(0, 0, 0);
const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(viewport.clientWidth, viewport.clientHeight);
renderer.setPixelRatio(window.devicePixelRatio);
viewport.appendChild(renderer.domElement);
const controls = new OrbitControls(camera, renderer.domElement);
controls.enableDamping = true;
const ambientLight = new THREE.AmbientLight(0xffffff, 0.6);
scene.add(ambientLight);
const directionalLight = new THREE.DirectionalLight(0xffffff, 0.8);
directionalLight.position.set(5, 10, 5);
scene.add(directionalLight);
const gridHelper = new THREE.GridHelper(10, 10, 0x0f3460, 0x0f3460);
scene.add(gridHelper);
function animate() {
controls.update();
renderer.render(scene, camera);
}
renderer.setAnimationLoop(animate);
</script>
</body>
</html>Your 3D viewer is live. Next up, you will connect this viewport to your local Ollama model so it can generate 3D shapes from text prompts.
Connect to Ollama
You have a 3D viewport with a grid floor and orbit controls, but nothing generates parts yet. The missing piece is a connection to your local LLM that can turn text descriptions into structured 3D data.
In this step, you will call the Ollama REST API with a JSON Schema that forces the model to return valid, predictable JSON describing 3D parts. This structured output approach means you never have to parse messy free-text responses.
In this step, get ready to:
- Define a JSON schema that describes valid 3D parts.
- Write a function that sends prompts to the local Ollama API.
- Wire the prompt input to trigger generation on Enter.
Define the JSON schema for 3D parts
Small language models are unreliable when asked to produce free-form code. A JSON schema acts like a contract: it tells the model exactly what shape the output must take.
- In index.html, scroll down inside the <script type="module"> block and find this line:
const gridHelper = new THREE.GridHelper(10, 10, 0x0f3460, 0x0f3460);
scene.add(gridHelper);- Below scene.add(gridHelper);, add the following schema definition:
// JSON schema that constrains the LLM to return valid 3D part data
const SCENE_SCHEMA = {
type: "object",
properties: {
parts: {
type: "array",
items: {
type: "object",
properties: {
type: { type: "string", enum: ["box", "cylinder", "sphere", "cone"] },
position: { type: "array", items: { type: "number" }, minItems: 3, maxItems: 3 },
size: { type: "array", items: { type: "number" }, minItems: 3, maxItems: 3 },
color: { type: "string" },
name: { type: "string" }
},
required: ["type", "position", "size", "color", "name"]
}
}
},
required: ["parts"]
};What does this code do?
This schema defines the exact structure of valid LLM output. Every response must be an object with a parts array.
- Each part has a type restricted to four primitives: box, cylinder, sphere, or cone.
- position and size are arrays of exactly 3 numbers (x, y, z coordinates and width, height, depth).
- color is a hex string and name labels the part for debugging.
You won't see a visible change yet. The next chunk uses this schema to instruct the LLM and constrain its output.
Write the generateParts function
Ollama's /api/chat endpoint accepts a system message (instructions for the model) and a user message (the prompt). By passing your schema in both the system prompt and the format parameter, you get grammar-constrained JSON output every time.
- Directly below your SCENE_SCHEMA definition, add the system prompt and the generateParts function by copying the following code:
// System prompt tells the model what to generate and embeds the schema
const SYSTEM_PROMPT = `You are a 3D part generator. Given a description, output a JSON object describing 3D parts.
Each part uses simple primitives: box, cylinder, sphere, or cone.
Position is [x, y, z] where y is up. Size is [width, height, depth].
Use realistic proportions. Place objects so y=0 is the ground.
Color should be a hex color string like "#ff0000".
JSON Schema for your response:
${JSON.stringify(SCENE_SCHEMA, null, 2)}`;
// Sends the user's prompt to the local Ollama API and parses structured output
async function generateParts(userMessage) {
status.textContent = 'Generating...';
const body = {
model: "qwen2.5-coder:3b",
messages: [
{ role: "system", content: SYSTEM_PROMPT },
{ role: "user", content: userMessage }
],
stream: false,
format: SCENE_SCHEMA,
options: { temperature: 0 }
};
try {
const response = await fetch('http://localhost:11434/api/chat', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body)
});
const data = await response.json();
const result = JSON.parse(data.message.content);
console.log(result);
if (result.parts && Array.isArray(result.parts)) {
status.textContent = `${result.parts.length} part(s)`;
} else {
status.textContent = 'No parts generated';
}
} catch (error) {
console.error('Error:', error);
status.textContent = 'Error - check console';
}
}What does this code do?
SYSTEM_PROMPT gives the model its role and embeds the full JSON schema so it knows the expected output format.
The generateParts function does several things:
- Sets the status to Generating... so the learner knows the request is in progress.
- Sends a POST to http://localhost:11434/api/chat with the model name, system message, user message, and the format parameter set to the schema.
- temperature: 0 makes the output deterministic so you get consistent results.
- Parses the response JSON content and logs it to the console for inspection.
Why both format and system prompt?
The format parameter enforces the schema at the grammar level, preventing the model from generating invalid JSON. Including the schema in the system prompt helps the model understand what the fields mean, producing more useful values.
You won't see a change in the browser yet. The next chunk connects the input field to this function.
Wire the prompt input to trigger generation
The generateParts function exists but nothing calls it yet. You need an event listener that fires when the learner presses Enter in the input field.
- Below the generateParts function's closing }, add the event listeners by copying the following code:
// Trigger generation when the user presses Enter
promptInput.addEventListener('keydown', (e) => {
if (e.key === 'Enter' && promptInput.value.trim()) {
generateParts(promptInput.value.trim());
promptInput.value = '';
}
});
// Keep the viewport responsive on window resize
window.addEventListener('resize', () => {
camera.aspect = viewport.clientWidth / viewport.clientHeight;
camera.updateProjectionMatrix();
renderer.setSize(viewport.clientWidth, viewport.clientHeight);
});What does this code do?
- The keydown listener checks for the Enter key and a non-empty input, then calls generateParts with the trimmed prompt and clears the input field.
- The resize listener recalculates the camera aspect ratio and renderer size whenever the browser window changes, preventing a stretched or cropped viewport.
- Save index.html.
- Refresh your browser page (or double-click index.html again if you closed it).
- Type a red cube in the prompt input at the bottom and press Enter.
- Open DevTools by pressing F12 and click the Console tab.
- After 10-30 seconds (CPU inference takes time), you should see a JSON object logged with a parts array containing one entry with type box and a red hex color value.
Why does it take so long?
The qwen2.5-coder:3b model is running entirely on your CPU. Without a GPU, each generation takes 10-30 seconds depending on your hardware. The Generating... status indicator lets you know the request is still in progress.
Seeing 'Error - check console'?
- Make sure Ollama is running. Open a terminal and run ollama list to confirm it responds.
- Check that you pulled the model. Run ollama pull qwen2.5-coder:3b if it is missing from the list.
- Verify the API is reachable by opening http://localhost:11434 in your browser. You should see a page that says "Ollama is running".
Still stuck? Help me debug an Ollama API connection error in my local 3D generator project.
✔️ Awesome, I've got everything!
Great. Double check you have saved index.html and confirmed a JSON response in the console.
ⓧ I'd like to double check the full code
Here is the full index.html as it should look at the end of this step:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AI 3D Part Generator</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: system-ui, sans-serif;
background: #1a1a2e;
color: #eee;
height: 100vh;
display: flex;
flex-direction: column;
overflow: hidden;
}
#viewport { flex: 1; }
#prompt-bar {
display: flex;
padding: 12px 16px;
background: #16213e;
border-top: 1px solid #0f3460;
gap: 8px;
align-items: center;
}
#prompt-input {
flex: 1;
padding: 10px 16px;
border: 1px solid #0f3460;
border-radius: 6px;
background: #1a1a2e;
color: #eee;
font-size: 14px;
outline: none;
}
#prompt-input:focus { border-color: #e94560; }
#prompt-input::placeholder { color: #666; }
#status {
font-size: 12px;
color: #e94560;
min-width: 100px;
text-align: right;
}
</style>
</head>
<body>
<div id="viewport"></div>
<div id="prompt-bar">
<input type="text" id="prompt-input" placeholder="Describe a 3D part... (e.g. 'a table with 4 legs')" />
<span id="status"></span>
</div>
<script type="importmap">
{
"imports": {
"three": "https://cdn.jsdelivr.net/npm/three@0.184.0/build/three.module.js",
"three/addons/": "https://cdn.jsdelivr.net/npm/three@0.184.0/examples/jsm/"
}
}
</script>
<script type="module">
import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';
const viewport = document.getElementById('viewport');
const promptInput = document.getElementById('prompt-input');
const status = document.getElementById('status');
const scene = new THREE.Scene();
scene.background = new THREE.Color(0x1a1a2e);
const camera = new THREE.PerspectiveCamera(60, viewport.clientWidth / viewport.clientHeight, 0.1, 1000);
camera.position.set(5, 4, 5);
camera.lookAt(0, 0, 0);
const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(viewport.clientWidth, viewport.clientHeight);
renderer.setPixelRatio(window.devicePixelRatio);
viewport.appendChild(renderer.domElement);
const controls = new OrbitControls(camera, renderer.domElement);
controls.enableDamping = true;
const ambientLight = new THREE.AmbientLight(0xffffff, 0.6);
scene.add(ambientLight);
const directionalLight = new THREE.DirectionalLight(0xffffff, 0.8);
directionalLight.position.set(5, 10, 5);
scene.add(directionalLight);
const gridHelper = new THREE.GridHelper(10, 10, 0x0f3460, 0x0f3460);
scene.add(gridHelper);
const SCENE_SCHEMA = {
type: "object",
properties: {
parts: {
type: "array",
items: {
type: "object",
properties: {
type: { type: "string", enum: ["box", "cylinder", "sphere", "cone"] },
position: { type: "array", items: { type: "number" }, minItems: 3, maxItems: 3 },
size: { type: "array", items: { type: "number" }, minItems: 3, maxItems: 3 },
color: { type: "string" },
name: { type: "string" }
},
required: ["type", "position", "size", "color", "name"]
}
}
},
required: ["parts"]
};
const SYSTEM_PROMPT = `You are a 3D part generator. Given a description, output a JSON object describing 3D parts.
Each part uses simple primitives: box, cylinder, sphere, or cone.
Position is [x, y, z] where y is up. Size is [width, height, depth].
Use realistic proportions. Place objects so y=0 is the ground.
Color should be a hex color string like "#ff0000".
JSON Schema for your response:
${JSON.stringify(SCENE_SCHEMA, null, 2)}`;
async function generateParts(userMessage) {
status.textContent = 'Generating...';
const body = {
model: "qwen2.5-coder:3b",
messages: [
{ role: "system", content: SYSTEM_PROMPT },
{ role: "user", content: userMessage }
],
stream: false,
format: SCENE_SCHEMA,
options: { temperature: 0 }
};
try {
const response = await fetch('http://localhost:11434/api/chat', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body)
});
const data = await response.json();
const result = JSON.parse(data.message.content);
console.log(result);
if (result.parts && Array.isArray(result.parts)) {
status.textContent = `${result.parts.length} part(s)`;
} else {
status.textContent = 'No parts generated';
}
} catch (error) {
console.error('Error:', error);
status.textContent = 'Error - check console';
}
}
promptInput.addEventListener('keydown', (e) => {
if (e.key === 'Enter' && promptInput.value.trim()) {
generateParts(promptInput.value.trim());
promptInput.value = '';
}
});
window.addEventListener('resize', () => {
camera.aspect = viewport.clientWidth / viewport.clientHeight;
camera.updateProjectionMatrix();
renderer.setSize(viewport.clientWidth, viewport.clientHeight);
});
function animate() {
controls.update();
renderer.render(scene, camera);
}
renderer.setAnimationLoop(animate);
</script>
</body>
</html>Your app can now talk to a local AI model and get structured 3D data back. Next up, you will turn that JSON data into actual 3D shapes rendered in your viewport.
Render LLM Output as 3D Parts
In the previous step, you proved that your local LLM can generate structured JSON describing 3D parts. The data is flowing, but it disappears into the console.
Now you will turn that JSON into actual 3D geometry. Each part the LLM describes becomes a real shape you can orbit and inspect in the viewport.
In this step, get ready to:
- Write a scene-clearing function that removes generated meshes without touching the grid or lights.
- Build a rendering pipeline that converts the LLM's JSON parts into Three.js meshes.
- Connect your existing API call to the renderer so prompts produce visible 3D output.
Write a clearScene function to remove generated meshes
Every time the user submits a new prompt, the old 3D shapes need to disappear before new ones appear. You need a way to track which meshes were generated by the LLM (so you can remove them) while leaving the grid and lights untouched.
- In index.html, find the line that says:
const gridHelper = new THREE.GridHelper(10, 10, 0x0f3460, 0x0f3460);
scene.add(gridHelper);- Add the following code directly below those lines to create a tracking array and the clearScene function:
// Track meshes generated by the LLM so we can remove them later
const generatedMeshes = [];
// Remove all LLM-generated meshes without touching the grid or lights
function clearScene() {
for (const mesh of generatedMeshes) {
scene.remove(mesh);
mesh.geometry.dispose();
mesh.material.dispose();
}
generatedMeshes.length = 0;
}What does this code do?
The generatedMeshes array acts as a registry of everything the LLM created. When you clear the scene, you only remove items in this array.
Calling .dispose() on geometry and material frees GPU memory. Without this, each generation would leak resources and eventually slow down the browser.
- Save index.html.
You will not see a change yet. The next chunk uses this function when rendering parts.
Write a createGeometry helper for each shape type
The LLM can return four primitive types: box, cylinder, sphere, and cone. You need a function that maps each type string to the correct Three.js geometry class, using the size array to set dimensions.
- In index.html, add the following function directly below the clearScene function you just wrote:
// Map a shape type string to the correct Three.js geometry
function createGeometry(type, size) {
const [w, h, d] = size;
switch (type) {
case 'sphere': return new THREE.SphereGeometry(w / 2, 32, 32);
case 'cylinder': return new THREE.CylinderGeometry(w / 2, w / 2, h, 32);
case 'cone': return new THREE.ConeGeometry(w / 2, h, 32);
case 'box':
default: return new THREE.BoxGeometry(w, h, d);
}
}What does this code do?
The size array from the LLM is destructured into width, height, and depth. Each geometry type uses these differently.
- SphereGeometry uses width/2 as the radius with 32 segments for smooth curves.
- CylinderGeometry uses width/2 for both top and bottom radii so you get a straight cylinder.
- BoxGeometry is the default fallback, using all three dimensions directly.
- Save index.html.
You will not see a change yet. The next chunk calls this helper to build visible meshes.
Write the renderParts function
This is the core rendering function. It takes the parts array from the LLM response, creates a Three.js mesh for each one, positions it in 3D space, and adds it to the scene.
- In index.html, add the following function directly below the createGeometry function:
// Turn the LLM's parts array into visible 3D meshes
function renderParts(parts) {
clearScene();
for (const part of parts) {
const geometry = createGeometry(part.type, part.size);
const material = new THREE.MeshStandardMaterial({
color: part.color || '#4488ff',
metalness: 0.3,
roughness: 0.7
});
const mesh = new THREE.Mesh(geometry, material);
mesh.position.set(part.position[0], part.position[1], part.position[2]);
scene.add(mesh);
generatedMeshes.push(mesh);
}
}What does this code do?
For each part in the array, the function creates geometry (using your helper), wraps it in a MeshStandardMaterial with the LLM's hex color, and positions the mesh using the [x, y, z] coordinates from the response.
The metalness: 0.3 and roughness: 0.7 values give shapes a subtle sheen that catches the directional light. Every mesh is also pushed into generatedMeshes so clearScene can find it later.
- Save index.html.
You will not see a change yet. The function exists but nothing calls it. The next edit wires it into the API response handler.
Connect generateParts to the renderer
The final connection is small but critical. Instead of logging the JSON to the console, you will call renderParts so the parsed data becomes visible geometry.
- In index.html, scroll down to your generateParts function and find these lines inside the try block:
const result = JSON.parse(data.message.content);
console.log(result);
if (result.parts && Array.isArray(result.parts)) {
status.textContent = `${result.parts.length} part(s)`;- Delete the console.log(result); line.
- Add renderParts(result.parts); inside the if block, just above the status update line. The result should look like this:
const result = JSON.parse(data.message.content);
if (result.parts && Array.isArray(result.parts)) {
renderParts(result.parts);
status.textContent = `${result.parts.length} part(s)`;- Save index.html.
- Refresh your browser. Type a table with 4 legs into the prompt and press Enter.
- Wait 10 to 30 seconds (CPU inference is slow). You should see 3D shapes appear in the viewport representing a table surface and legs. Orbit around them by clicking and dragging with your mouse.
Shapes not appearing?
- Check that you removed the console.log(result); line and added renderParts(result.parts); inside the if block (not outside it).
- Open DevTools (F12) and check the Console tab for error messages. A common issue is a typo in function names.
- Make sure the renderParts function is placed ABOVE the SCENE_SCHEMA variable and BELOW the gridHelper lines in the file.
Help me debug why my 3D shapes are not rendering after the LLM responds.
Why does it take so long?
Your model is running entirely on your CPU. A 3B parameter model generating structured JSON output takes 10 to 30 seconds on most machines. If you had a dedicated GPU, this would drop to 1 to 3 seconds.
✔️ Awesome, I've got everything!
Great. Double check you have saved your file before moving on.
ⓧ I'd like to double check the full code
Here is the complete index.html at this point. Compare it against your file to spot any differences.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AI 3D Part Generator</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: system-ui, sans-serif;
background: #1a1a2e;
color: #eee;
height: 100vh;
display: flex;
flex-direction: column;
overflow: hidden;
}
#viewport { flex: 1; }
#prompt-bar {
display: flex;
padding: 12px 16px;
background: #16213e;
border-top: 1px solid #0f3460;
gap: 8px;
align-items: center;
}
#prompt-input {
flex: 1;
padding: 10px 16px;
border: 1px solid #0f3460;
border-radius: 6px;
background: #1a1a2e;
color: #eee;
font-size: 14px;
outline: none;
}
#prompt-input:focus { border-color: #e94560; }
#prompt-input::placeholder { color: #666; }
#status {
font-size: 12px;
color: #e94560;
min-width: 100px;
text-align: right;
}
</style>
</head>
<body>
<div id="viewport"></div>
<div id="prompt-bar">
<input type="text" id="prompt-input" placeholder="Describe a 3D part... (e.g. 'a table with 4 legs')" />
<span id="status"></span>
</div>
<script type="importmap">
{
"imports": {
"three": "https://cdn.jsdelivr.net/npm/three@0.184.0/build/three.module.js",
"three/addons/": "https://cdn.jsdelivr.net/npm/three@0.184.0/examples/jsm/"
}
}
</script>
<script type="module">
import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';
const viewport = document.getElementById('viewport');
const promptInput = document.getElementById('prompt-input');
const status = document.getElementById('status');
const scene = new THREE.Scene();
scene.background = new THREE.Color(0x1a1a2e);
const camera = new THREE.PerspectiveCamera(60, viewport.clientWidth / viewport.clientHeight, 0.1, 1000);
camera.position.set(5, 4, 5);
camera.lookAt(0, 0, 0);
const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(viewport.clientWidth, viewport.clientHeight);
renderer.setPixelRatio(window.devicePixelRatio);
viewport.appendChild(renderer.domElement);
const controls = new OrbitControls(camera, renderer.domElement);
controls.enableDamping = true;
const ambientLight = new THREE.AmbientLight(0xffffff, 0.6);
scene.add(ambientLight);
const directionalLight = new THREE.DirectionalLight(0xffffff, 0.8);
directionalLight.position.set(5, 10, 5);
scene.add(directionalLight);
const gridHelper = new THREE.GridHelper(10, 10, 0x0f3460, 0x0f3460);
scene.add(gridHelper);
const generatedMeshes = [];
function clearScene() {
for (const mesh of generatedMeshes) {
scene.remove(mesh);
mesh.geometry.dispose();
mesh.material.dispose();
}
generatedMeshes.length = 0;
}
function createGeometry(type, size) {
const [w, h, d] = size;
switch (type) {
case 'sphere': return new THREE.SphereGeometry(w / 2, 32, 32);
case 'cylinder': return new THREE.CylinderGeometry(w / 2, w / 2, h, 32);
case 'cone': return new THREE.ConeGeometry(w / 2, h, 32);
case 'box':
default: return new THREE.BoxGeometry(w, h, d);
}
}
function renderParts(parts) {
clearScene();
for (const part of parts) {
const geometry = createGeometry(part.type, part.size);
const material = new THREE.MeshStandardMaterial({
color: part.color || '#4488ff',
metalness: 0.3,
roughness: 0.7
});
const mesh = new THREE.Mesh(geometry, material);
mesh.position.set(part.position[0], part.position[1], part.position[2]);
scene.add(mesh);
generatedMeshes.push(mesh);
}
}
const SCENE_SCHEMA = {
type: "object",
properties: {
parts: {
type: "array",
items: {
type: "object",
properties: {
type: { type: "string", enum: ["box", "cylinder", "sphere", "cone"] },
position: { type: "array", items: { type: "number" }, minItems: 3, maxItems: 3 },
size: { type: "array", items: { type: "number" }, minItems: 3, maxItems: 3 },
color: { type: "string" },
name: { type: "string" }
},
required: ["type", "position", "size", "color", "name"]
}
}
},
required: ["parts"]
};
const SYSTEM_PROMPT = `You are a 3D part generator. Given a description, output a JSON object describing 3D parts.
Each part uses simple primitives: box, cylinder, sphere, or cone.
Position is [x, y, z] where y is up. Size is [width, height, depth].
Use realistic proportions. Place objects so y=0 is the ground.
Color should be a hex color string like "#ff0000".
JSON Schema for your response:
${JSON.stringify(SCENE_SCHEMA, null, 2)}`;
async function generateParts(userMessage) {
status.textContent = 'Generating...';
const body = {
model: "qwen2.5-coder:3b",
messages: [
{ role: "system", content: SYSTEM_PROMPT },
{ role: "user", content: userMessage }
],
stream: false,
format: SCENE_SCHEMA,
options: { temperature: 0 }
};
try {
const response = await fetch('http://localhost:11434/api/chat', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body)
});
const data = await response.json();
const result = JSON.parse(data.message.content);
if (result.parts && Array.isArray(result.parts)) {
renderParts(result.parts);
status.textContent = `${result.parts.length} part(s)`;
} else {
status.textContent = 'No parts generated';
}
} catch (error) {
console.error('Error:', error);
status.textContent = 'Error - check console';
}
}
promptInput.addEventListener('keydown', (e) => {
if (e.key === 'Enter' && promptInput.value.trim()) {
generateParts(promptInput.value.trim());
promptInput.value = '';
}
});
window.addEventListener('resize', () => {
camera.aspect = viewport.clientWidth / viewport.clientHeight;
camera.updateProjectionMatrix();
renderer.setSize(viewport.clientWidth, viewport.clientHeight);
});
function animate() {
controls.update();
renderer.render(scene, camera);
}
renderer.setAnimationLoop(animate);
</script>
</body>
</html>Your AI 3D part generator is fully functional. You can describe any object and watch it appear in 3D space. Next up, you will take on the Secret Mission and add conversational refinement so you can iteratively improve your designs.
Secret mission
Add Conversational Refinement
Right now, every prompt starts from scratch. What if you could say "make it taller" or "add 2 more legs" and watch your design evolve? In this secret mission, you'll add conversation memory so the LLM builds on its previous output.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally on your machine. There are no cloud resources, no API subscriptions, and no ongoing costs.
Decide whether to keep your resources running, pause them to come back later, or delete them entirely.
Resources you used:
- The ai-3d-generator folder on your Desktop containing index.html.
- The qwen2.5-coder:3b model stored locally by Ollama.
Keep everything running
No action needed. Choose this if you plan to keep experimenting with your 3D part generator.
- Your index.html file stays on your Desktop and works anytime Ollama is running.
- The qwen2.5-coder:3b model stays downloaded and ready to use for this or other projects.
- There are no ongoing costs. Everything is local.
Pause - I'll come back to this later
Free up RAM by unloading the model from memory. Your files and model stay on disk for later use.
- Unload the model from memory by running this command in your terminal:
ollama stop qwen2.5-coder:3b- This frees the RAM the model was using. The model files remain on disk so you can use it again without re-downloading.
- Your ai-3d-generator folder stays on your Desktop. Open index.html in your browser anytime to pick up where you left off.
Delete - I don't want to use this again
Remove all project files and the downloaded model to reclaim disk space.
- Delete the project folder by running this command:
rm -rf ~/Desktop/ai-3d-generator- On Windows, run this instead:
rmdir /s /q %USERPROFILE%\Desktop\ai-3d-generator- Remove the qwen2.5-coder:3b model from Ollama by running:
ollama rm qwen2.5-coder:3b- This frees approximately 1.8GB of disk space used by the model.
That's a Wrap!
That's a Wrap!
Nice work! You just built a fully local AI 3D part generator that turns natural language into interactive 3D models, all without spending a cent or sending data to the cloud.
You've learned how to:
- Call the Ollama REST API with structured JSON output using the format parameter to get reliable, schema-compliant responses from a local LLM.
- Create an interactive Three.js 3D viewer with orbit controls loaded from a CDN using ES module import maps in a single HTML file.
- Connect an LLM to a 3D renderer, translating natural language prompts into visual output by parsing structured data into geometry, materials, and positions.
- Secret Mission: Added conversation history for iterative design refinement, letting you build on previous generations with follow-up prompts.
Ready to quiz yourself?