Agentic 3D Modeling with Procedural Control
Procedura turns a text prompt into an editable procedural assembly — a parametric program whose named parts are joined by typed mates, written with a frozen LLM and no 3D training. Left: the compiled geometry. Right: further objects coloured by their named modules, a part decomposition that comes free with the representation.
The output is not a point cloud or a soup of triangles. It is source code you
can open, edit, and re-compile — optionally with per-part PBR materials
(--paint) and articulation exported to OpenUSD/URDF (--motion).
Install
One command installs everything: Bun, the packages, a Manifold-capable OpenSCAD,
Blender, and a starter .env.
git clone git@github.com:SpatiaOS/Procedura.git && cd Procedura bash scripts/install-deps.sh
It is safe to re-run — anything already working is left alone — so it doubles as a "what am I missing?" check:
bash scripts/install-deps.sh --check # report only, install nothing bash scripts/install-deps.sh --no-blender # skip the ~350 MB download PREFIX=/opt bash scripts/install-deps.sh # install binaries somewhere else
| what | where it goes | why |
|---|---|---|
| Bun ≥ 1.3 | ~/.bun |
the whole pipeline is one TypeScript program |
| OpenSCAD (newest snapshot) | ~/opt/openscad |
compiles the generated program to a mesh |
| Blender (newest release) | ~/opt/blender/ |
the AO / parts-colour / PBR renders the critic looks at |
| Packages | node_modules/, web/node_modules/ |
pipeline + Studio dependencies |
.env |
repo root | copied from .env.example; add your endpoint and key |
Two things it deliberately does not do. It never installs NVIDIA Isaac Sim
(multi-GB and account-gated) — that is only needed for --motion physics
validation, and everything else runs without it. And it never touches an
existing .env.
OpenSCAD must have the Manifold backend. A pre-Manifold build does not fail — it falls back to CGAL and gets orders of magnitude slower. One
hull()measured 1774s under CGAL against 1.77s under Manifold, which turns a working run into one that looks hung for hours. The installer checks for--backendrather than merely "is openscad installed", because the version most distributions package is 2021.01 and predates it. Procedura makes the same check at startup and refuses to run without it unless you setPROCEDURA_ALLOW_CGAL_OPENSCAD=1.
Configuration
Procedura ships no keys and no hosted backend. It talks to whatever endpoint you configure, through two variables:
OPENAI_API_KEY=sk-... OPENAI_BASE_URL=https://api.openai.com/v1
Anything that speaks POST {base}/chat/completions with SSE works here — the
OpenAI API, OpenRouter, a corporate gateway, or a local vLLM / Ollama / LM
Studio server. For Gemini on its native GenAI path (where thinkingConfig and
first-class thought parts are available), set GEMINI_API_KEY /
GEMINI_BASE_URL instead and use a gemini: model key.
Model selection is just a string. Any id your endpoint serves works, whether or not this repo has heard of it:
PROCEDURA_MODEL=gpt-5.2 # via the openai provider PROCEDURA_MODEL=gemini:gemini-3-pro-preview # force the native Gemini transport
src/config/models.ts holds a small catalog of known-good presets and explains
the resolution order. .env.example documents every other knob — reasoning
effort, binary paths, timeouts, retry budgets.
Reference images are an input, not an assumption. Procedura will not call an
image API on your behalf: with no --image and no image model configured, a run
goes text-only rather than quietly spending on an endpoint you never named.
Supplying a reference is the single largest fidelity gain available, so pass
--image <path> when you have one — or name an image model and let the run
render its own:
PROCEDURA_IMAGE_MODEL=gpt-image-1 # unset = generation off; --image-model for one runIt reuses OPENAI_BASE_URL / OPENAI_API_KEY, so nothing else needs setting.
Usage
The default run
Nothing to configure past the endpoint. Plans the parts, builds them one at a time, refines the whole shape — from the prompt alone:
bun run scripts/procedura.ts -o outputs/daybed \
--prompt "a brutalist brass daybed with tapered legs"That is text-only, part-by-part, with no image API and no Blender in the draft loop, and it is what you get with no flags at all. It is the cheapest useful configuration, not the strongest one.
The best-quality run
Everything on. This is what the paper's results use, and every switch below buys accuracy at the cost of time or money:
# big plans mean long single calls; the default 600s cap is not enough export PROCEDURA_LLM_TIMEOUT_MS=1800000 export PROCEDURA_LLM_DEADLINE_MS=1800000 bun run scripts/procedura.ts -o outputs/gripper \ --image gripper_ref.png \ --3d-feedback \ --assembly \ --paint \ --motion --motion-urdf \ --max-steps 12 \ --prompt "a two-finger parallel robot gripper"
| switch | what it buys |
|---|---|
--image ref.png |
a reference to reconstruct — the single largest fidelity gain. Set PROCEDURA_IMAGE_MODEL instead to have the run render its own |
--3d-feedback |
the generator sees the build-so-far before each part instead of reading about it. One Blender pass per part |
--assembly |
parts join through real mating features — shared-nominal pegs/sockets, bolt patterns, snaps — rather than bare overlap |
--paint |
per-part PBR materials assigned by a vision call |
--motion --motion-urdf |
articulation planned, exported to OpenUSD + URDF, and validated headlessly in Isaac |
--max-steps 12 |
a longer refine budget; each cycle is render → critic → patch → gate, at 2 LLM calls |
Raise the timeouts rather than capping the part count — a long plan is the
pipeline working, and PROCEDURA_MAX_PARTS degrades the model to buy back time.
In between
# reconstruct a reference image you already have — the biggest single win bun run scripts/procedura.ts -o outputs/daybed --image ref.png --prompt "..." # generate the reference too (needs PROCEDURA_IMAGE_MODEL, or pass --image-model) bun run scripts/procedura.ts -o outputs/daybed --image-model gpt-image-1 --prompt "..." # long prompts, without fighting your shell bun run scripts/procedura.ts -o outputs/daybed --prompt-file my_prompt.txt # resume the refine loop on an existing run, or force a re-draft bun run scripts/procedura.ts -o outputs/daybed --max-steps 12 bun run scripts/procedura.ts -o outputs/daybed --prompt "..." --redo # the old monolithic draft: one call, no plan stage. Needs a reference bun run scripts/procedura.ts -o outputs/daybed --one-shot --image ref.png --prompt "..."
Over a dataset
The paper's batches run the shape and articulation together, then paint each finished case as a separate pass — so a paint failure never costs the geometry:
bun run scripts/batch.ts \ --prompts cases.jsonl --images \ --incremental \ --motion --validate --urdf \ --max-steps 3 \ --parallel 2 \ --out-root outputs/batch_v1 bun run scripts/paint.ts outputs/batch_v1/<case_id>
Keep --parallel at ~2 with --motion --validate: concurrent headless Isaac
instances exhaust GPU VRAM. Refine-only batches tolerate 8-12; each refine cycle
spawns Blender, so running every case at once will exhaust memory long before it
exhausts your rate limit.
Flags
| flag | meaning |
|---|---|
-o, --output DIR |
output directory (required) |
--prompt TEXT / --prompt-file PATH |
the text prompt |
--image PATH |
reconstruct this reference image |
--no-image |
text-only — the default when no reference is available |
--image-model M |
generate the reference with this model, for this run only |
--3d-feedback |
show the generator a render of the build-so-far before each part (off by default) |
--one-shot |
opt out of the part-by-part draft: one call, no plan stage |
--assembly |
inline the mating-feature library so parts join through real interfaces |
--paint |
material pass: a vision call assigns each part a PBR material |
--motion |
OpenUSD/Isaac articulation export |
--motion-urdf |
also emit URDF + per-link meshes |
--max-steps N |
refine-loop budget in edit cycles (default 6; 2 LLM calls each) |
--no-refine |
draft only — promote the draft to final.* |
--redo |
force a re-draft even if artifacts exist |
--agent-model / --scad-model / --paint-model / --motion-model |
per-stage model override |
bun run scripts/procedura.ts --help lists all of them, including the ablation
switches (--no-plan, --motion-aware, --motion-only) used in the paper.
What a run writes
image.png the reference (generated, or the one you passed)
plan.json _parts/NN_<name>/ --incremental: the part list + per-part artifacts
draft.scad draft.obj the drafted program and its mesh
final.scad final.obj after refine — the deliverable
final_summary.txt finish verdict + shipped connectivity
preview_final/ Blender AO render of the final mesh
final_materials.json --paint: part → colour / metalness / roughness
final_painted.scad/.obj/.mtl --paint, plus preview_painted/ (PBR Cycles render)
motion/final_motion.usda --motion, plus motion/urdf/ and motion/links/
_trajectory/procedura-<id>.jsonl every prompt, response, tool call and gate decision
_refine_steps/step_NNN/ one dir per refine cycle (render, diagnosis, edit, compile)
Every OBJ Procedura writes is normalized to a unit bounding box; STL is opt-in
(--export-stl).
Re-running a stage
paint.ts and motion.ts both take a finished run directory, so either stage
can be re-run — with a different model, or after a failure — without touching
the geometry:
bun run scripts/paint.ts <runDir> [--model KEY] [--refine-steps N] bun run scripts/motion.ts <runDir> [--model KEY] [--urdf] [--no-validate]
To compare runs across configurations — renders, source, metrics and trajectory, side by side per case:
bun run scripts/results-server.ts --root a=outputs/run_a --root b=outputs/run_b
Studio
A web front-end for the same pipeline: compose a run (prompt, optional reference image, preset or per-stage toggles), watch it build part by part, and inspect every intermediate artifact of any run — plan, parts, refine loop, final mesh in 3D, materials, articulation.
cd web && bun install && bun run start # http://localhost:8080
It spawns the CLI from the repo root and reads the same .env, so there is
nothing extra to configure. See web/README.md.
Acknowledgements
The vendored LLM harness in vendor/harness/ is an independent implementation
of architectural patterns from two MIT-licensed projects —
opencode (the four-axis
Protocol / Endpoint / Auth / Framing route model, the permission evaluator,
the cache-breakpoint policy) and
openclaw (the sandbox bridge, the
context engine, trajectory events). Credit for those designs belongs to them.
License
MIT. See LICENSE.
Procedura invokes OpenSCAD and Blender as external programs; both are GPL and
neither is redistributed here. The four Blender scripts under scripts/ use
Blender's bpy API and are MIT-licensed, which is GPL-compatible. No
third-party assets ship in this repo — the PBR studio environment is read from
your own Blender installation at render time.