zqc3117/Spotter

If our project helps you, please give us a star ⭐ on GitHub to support us.

Spotter adds a lightweight intervention layer between a frozen embodied model (System 1) and a VLM (System 2). Instead of having the VLM plan every step, the embodied model leads and keeps executing, while a local screener watches in parallel and the VLM steps in only to confirm and repair an error. The VLM's job becomes much simpler, so the whole framework works with a locally served, open-source Qwen and no privileged information.

Real robot (Franka Research 3), stacking the blue cup onto the pink cup. Shown at 2×, with the judge's thinking time shortened; the on-screen clock is real time.
Watch the full video on the project page.

• A lightweight intervention layer. Like a person who keeps a little attention on a task and focuses only when something looks wrong, Spotter lets execution flow continuously and brings in the VLM only on a confirmed error.
• A simpler job for the VLM. The VLM no longer drives the robot. It only has to notice a failure, repair it with a short primitive plan, and check the fix. A local Qwen with no access to the task's success signal is enough.
• Continuous execution, low overhead. The embodied model never waits for the VLM on the routine path: a successful episode takes only 13–16 s longer than the policy alone.

Existing VLM-led robot agents are serial: the VLM plans a segment, the embodied model executes it, and the VLM decides what comes next. This forces a dilemma:

• Short segments mean frequent VLM calls and high latency, even on easy tasks.
• Long segments mean a deviation during execution cannot be caught in time, and later actions keep going wrong.

Spotter removes this trade-off by supervising in parallel rather than in series.

With the same local Qwen and no privileged information on RoboCasa (1,200 episodes), a representative VLM-led agent falls below the policy alone (Cosmos Policy 67.9 → 65.2, π0.5 64.3 → 60.5), while Spotter still improves it (→ 71.7 and 68.4).

With Qwen, the judge is called less than once per successful episode on average, which adds only 13–16 s over the policy alone and takes about 70% less time than a VLM-led agent using the same Qwen.

Status (October 2026): please use RoboTwin 2.0 for now. The RoboCasa path has a rendering bug: in a small fraction of episodes the offscreen renderer returns black or noisy camera frames, which makes those episodes fail. A fix is being validated. Until it lands, the RoboCasa instructions in this README are hidden; RoboTwin 2.0 is unaffected and fully supported. Run it with one command, bash robotwin_eval.sh (see Evaluation).

RoboTwin 2.0 results and full logs (control vs. Spotter-Qwen, every episode's frames, prompts and replies): 50 tasks × 10 episodes.

• File Structure
• Environment Setup
• Model Preparation
• Simulator Assets and Episode Sets
• Evaluation with Released Checkpoints: RoboTwin 2.0
• Plugging In Your Own Embodied Model
• Outputs · Configuration Reference
• Acknowledgements · License · Citation

Spotter/
├── spotter.sh                        # one entry point: serve Qwen / the policy, run episodes, summarize
├── robotwin_eval.sh                  # one-command RoboTwin 2.0 evaluation (servers + Spotter arm + control + summary)
├── recovery_explore/                 # Spotter itself
│   ├── judge_driver_v7.sh            # supervision loop for one lane: screen -> judge -> repair -> verify
│   ├── env_service.py                # RoboCasa simulation service (RPC), started by launch_svc.sh
│   ├── env_service_robotwin.py       # RoboTwin 2.0 simulation service (same RPC surface, SAPIEN)
│   ├── robotwin_intervention.py      # RoboTwin repair primitives (dual arm); robodojo_intervention.py is its base
│   ├── robotwin_policy_server.py     # RoboTwin pi0.5 policy server (openpi websocket protocol)
│   ├── primitives.py                 # motion primitives the judge composes repairs from
│   ├── JUDGE_BRIEF_V7.md             # judge prompt; BRIEF_PNP.md / BRIEF_MECH.md / BRIEF_RT.md add task-family notes
│   ├── cli/                          # screener, judge channels (Qwen / GPT), harness, few-shot tools
│   ├── memory_bank/                  # frozen library of 77 general lessons (RoboCasa)
│   ├── memory_bank_robotwin_noreset/ # the RoboTwin 2.0 lessons
│   ├── episode_sets/                 # sall500.txt (paper, 1,200 episodes) · s500q96.txt (96, debugging) · rt50x10n.txt (RoboTwin 2.0, 500)
│   ├── fewshot_bank/                 # (downloaded) worked examples for the 1-shot setting; robotwin/ for RoboTwin 2.0
│   └── runs_<RUN>_<family>/          # (generated) results.jsonl, per-lane logs, frames shown to the judge
├── rpc/                              # π0.5 policy server, RPC protocol, RoboCasa rollout and collection scripts
├── cf_bench/                         # snapshot and replay tools for the error-correction study
├── env/                              # env.sh.example, requirements-robocasa.txt, requirements-pi05.txt
├── assets/                           # logo
└── LICENSE · THIRD_PARTY_NOTICES.md · LICENSES/

Qwen screener and judge. vLLM in its own environment, and a local env/env.sh:

python3 -m venv .venvs/vllm && .venvs/vllm/bin/pip install vllm
cp env/env.sh.example env/env.sh && $EDITOR env/env.sh

RoboTwin 2.0. Spotter runs on RoboTwin 2.0 (aloha-agilex dual arm, SAPIEN 3, 50 tasks, demo_randomized scenes); the family is robotwin.

RoboTwin's simulator and planner (SAPIEN 3, mplib, cuRobo) live in their own Python 3.10 environment, laid out as a portable runtime that env/env.sh points at:

<ROBOTWIN_RUNTIME>/
├── .venv/                                   # python 3.10: sapien 3.0.0b1, mplib 0.2.1, toppra, torch (CUDA build matching your GPU)
└── worktree/third_party/RoboTwin/           # the RoboTwin 2.0 tree
    ├── envs/curobo/src/                     # cuRobo sources (its CUDA kernels must match your torch)
    ├── assets/{objects,background_texture,embodiments}/   # RoboTwin 2.0 assets (TianxingChen/RoboTwin2.0 on the Hub)
    └── policy/pi0/checkpoints/pi05_robotwin2_clean_randomized/robotwin2/40000/   # the pi0.5 checkpoint

Install RoboTwin 2.0 and its assets as its README says, then set in env/env.sh:

export ROBOTWIN_RUNTIME=/path/to/robotwin-runtime            # or ROBOTWIN_ROOT + ROBOTWIN_PY
export ROBOTWIN_POLICY_PYTHON=/path/to/robotwin-policy-venv/bin/python   # a venv satisfying policy/pi05/requirements.txt (openpi, lerobot < 0.3)

The simulation renders with SAPIEN's ray tracer (Vulkan, RTX-class GPU); on RTX 5090 the optix denoiser is used because the default oidn fails there.

1) Qwen judge and screener.

huggingface-cli download Qwen/Qwen3.8-27B-FP8 --local-dir checkpoints/Qwen3.8-27B-FP8

Then set in env/env.sh:

export QWEN_MODEL=$PWD/checkpoints/Qwen3.8-27B-FP8
export VLLM_VENV=$PWD/.venvs/vllm

2) RoboTwin 2.0. The policy is the pi0.5 checkpoint motus-robotics/pi0.5_robotwin2, placed at policy/pi0/checkpoints/pi05_robotwin2_clean_randomized/robotwin2/40000/ inside the RoboTwin tree (see Environment Setup). For the 1-shot setting, download the RoboTwin example bank (33 MB) into recovery_explore/fewshot_bank/robotwin/:

curl -L https://github.com/zqc3117/Spotter/releases/download/v1.0/fewshot_bank_robotwin.tar.gz | tar -xz -C recovery_explore

GPT judge (optional). Set ENGINE=api together with OPENAI_RESPONSES_URL and OPENAI_API_TOKEN_FILE; the judge becomes GPT-6 Astra while the screener stays the local Qwen.

RoboTwin 2.0 assets are installed with RoboTwin itself (Environment Setup).

Episode sets ship with the repository, one TASK:SEED:EPISODE per line:

Each server runs in its own terminal after source env/env.sh.

1) Serve the Qwen screener and judge (one replica per GPU, ports 8301, 8302, ...).

bash spotter.sh judge --gpus 0,1,2,3
curl -s http://127.0.0.1:8301/v1/models        # check

With fewer GPUs, pass fewer ids (e.g. --gpus 0) and export QWEN_PORTS=8301 before running.

One command. robotwin_eval.sh starts Qwen and the pi0.5 policy if they are not already up, runs the Spotter arm and then the policy-only control on the same 500 episodes (rt500), and prints the summary. It uses the settings of the released results: 1.8× step budget, 1-shot, Qwen thinking in the repair turns with QWEN_THINK_BUDGET=5000.

bash robotwin_eval.sh --judge-gpus 0,1,2,3 --policy-gpu 4 --lanes 12

Add --dry-run to print the commands without running them. Each RoboTwin simulator takes about 8 GB of GPU memory, so plan about three lanes per 32 GB GPU. The judge's thinking dominates the time of the Spotter arm; more Qwen replicas (--judge-gpus) make it faster.

bash spotter.sh policy robotwin --gpu 4              # pi0.5 for RoboTwin, port 9100 (check: curl http://127.0.0.1:9100/healthz)
bash spotter.sh run robotwin                         # smoke: adjust_bottle, episode 35
bash spotter.sh run robotwin rt500 --lanes 12 --one-shot     # Spotter arm on the 500 evaluation episodes (1-shot)
bash spotter.sh run robotwin rt500 --lanes 12 --arm ctrl     # the policy-only baseline on the same episodes
bash spotter.sh summary

Results with this code. We re-ran the Spotter arms on RoboTwin 2.0 with this version of the code, on the 500 episodes of rt500 (1.8× step budget, 1-shot); the control is the policy alone on the same 500 episodes:

Because every run involves randomness, these numbers differ somewhat from the paper. We have uploaded the per-episode results, with every episode's frames, judge prompts and replies, to ModelScope: 50 tasks × 10 episodes.

Run options (bash spotter.sh run <family> [SET] [options]):

Both arms of an episode share the same scene, instruction and step budget:

spotter.sh run robotwin applies the RoboTwin column of the Configuration Reference; every setting can still be overridden through the environment.

Spotter never trains the policy, so a new embodied model only needs to be served and registered:

1. Serve it over the same RPC interface as rpc/policy_server_pi05.py: GET /health, and POST /v1/infer taking an InferenceRequest (task description, seed, observation images and state) and returning an InferenceResponse with an action chunk (see rpc/protocol.py).
2. Register the family next to cosmos and pi05: action dimension and steps per chunk in recovery_explore/cosmos_call.py and env_service.py, service launch in launch_svc.sh, port and defaults in spotter.sh.
3. Choose the supervision granularity: WINDOW (chunks per screening check) and TEL_CHUNKS (telemetry rows shown to the judge).
4. Run bash spotter.sh run <your-family> ....

recovery_explore/runs_<RUN>_<family>/
├── results.jsonl          # one line per episode
├── lane<N>/driver.log     # per-lane progress
└── lane<N>/jd-.../w<k>/   # frames shown to the judge in window k

bash spotter.sh summary            # all runs
bash spotter.sh summary robotwin_rt50x10n

spotter.sh run robotwin and robotwin_eval.sh apply these values; every one can be overridden through the environment.

Qwen thinks only in the repair turns of an intervention, up to QWEN_THINK_BUDGET (default 5000) tokens per call; the screener and the window verdict never think, and QWEN_REPAIR_THINK=0 turns thinking off everywhere, as in the paper.

Choosing the thinking budget. Set QWEN_THINK_BUDGET to what your Qwen servers can afford. A larger budget gives the repair plan more room but makes every repair turn slower; a smaller one is faster but cuts the thinking off before it finishes. Use the largest budget that keeps repair turns within ACT_DEADLINE (raise that too on slow GPUs or with many lanes per replica), and check after a run that most repair turns were not cut off. Each repair reply is saved as act_w*_a*_r*.json in the episode folder, with think_budget_hit: true when the thinking ran into the budget:

python3 - <<'EOF'
import glob, json
files = glob.glob("recovery_explore/runs_<RUN>_<family>/lane*/jd-*/act_w*.json")
hit = sum(bool(json.load(open(f)).get("think_budget_hit")) for f in files)
print(f"{hit}/{len(files)} repair turns hit the thinking budget")
EOF

If more than about half are cut, raise the budget (or add Qwen replicas so the larger budget stays fast). On RoboTwin 2.0, a budget of 2000 cut off the thinking in about 65% of the repair turns, and 5000 in about 44%.

How the judge is paced has its own switches, all off or conservative by default:

Changing STEP_BUDGET_SCALE, WINDOW or the policy family changes the experiment; compare runs by matched task/seed/episode, not only by aggregate success rate.

• RoboCasa: the simulation benchmark and assets we evaluate on.
• Cosmos Policy: the released RoboCasa checkpoint and its policy server.
• π0.5 and LeRobot: the RoboCasa π0.5 checkpoint and the library that loads it.
• Qwen and vLLM: the local screener and judge.
• RPent: parts of recovery_explore/ are adapted from it (Apache-2.0, see THIRD_PARTY_NOTICES.md).

This project is released under the MIT License. Some files in recovery_explore/ are adapted from RPent and remain under the Apache License 2.0; see THIRD_PARTY_NOTICES.md.

@article{li2026spotter,
  title   = {Spotter: Let the Embodied Model Lead, and the VLM Reflect for It},
  author  = {Li, Long and Zhao, Qichao and Yang, Yue and Xu, Fan and Wang, Zhe and Liew, Alan Wee-Chung and Qu, Chao and Shen, Heng Tao and Pan, Shirui},
  journal = {arXiv preprint arXiv:2609.36808},
  year    = {2026}
}