RomanSlack/jev-drone

An autonomous quadrotor flies a five-station obstacle course in MuJoCo using only its onboard camera. A small judgment model (TypeSafe's Jev) sits at ~2.5 Hz and decides what the situation means; everything time-critical stays in ordinary code.

The drone is the Skydio X2 from MuJoCo Menagerie

• a real airframe with four rotors, flown by a geometric controller. Nothing about the flight is scripted.

Jev is not a vision model. It takes JSON and returns typed answers with probabilities. So it cannot be the perception layer, and it cannot run at control rate. What it can do is answer small questions about a situation that ordinary code finds hard to phrase.

 500 Hz   geometric controller       flight.Pilot    thrust-priority mixing, slew-limited
  50 Hz   guidance + safety reflex   run.Guidance    ALWAYS owns safety
  15 Hz   camera -> symbolic scene   flight.Eye      depth + segmentation, no ground truth
~2.5 Hz   tactical judgment          tactics.py      Jev, advisory only

Classical CV turns the depth and segmentation buffers into a compact scene: five forward range sectors, the height of whatever is blocking the path, whether its top edge is even visible, and where the target is. Jev reads that and answers three questions in one call:

Code decides when to ask. On an open corridor with the target in view there is nothing to decide, so no call is made. Scenes are fingerprinted so an unchanged situation reuses the last judgment. A typical 65 s flight costs about 110 calls.

Code keeps the veto. A hard reflex layer runs at 50 Hz and overrides any judgment when something is close. Jev can propose climb, but code refuses it unless the obstruction's measured top edge is actually within the aircraft's climb ceiling.

Five stations, each breaking a different assumption:

1. Slalom - ordinary steering around pillars
2. Low beam - spans the whole corridor. No lateral gap exists at any width, so "go around" is not available. It must be flown over.
3. Turnstiles - arms sweeping across the lane. Moving, must be timed.
4. Sliding gate - a 3.2 m gap that slides sideways. Too tall to climb, so it must be threaded.
5. Cluster - dense pillars

The ablation is the same stack with the Jev call disabled, falling back to a greedy "steer toward the wider side" heuristic. That baseline is safe but stuck: it never crashes and it never gets past station 2, because going over an obstacle is not something it can express.

The baseline stops dead at station 2 every single time. With Jev the drone clears the entire course in about 47 s at up to 3.6 m/s:

x=19  beam0       t=15.7  z=3.0   climbed over
x=26  turnstiles  t=20.3  z=1.6   timed the sweeping arms
x=38  GATE        t=31.3  z=1.6   threaded the sliding 3.2 m gap
x=44  beam1       t=36.8  z=2.8   climbed over
x=50  cluster     t=41.3  z=1.6   lost the rover in the pillars
x=58  exit        t=46.6          re-acquired it and closed back to 3.7 m

80 calls over 65 s, 0.11 s median latency, 96k tokens.

The cluster is worth noting: the drone genuinely loses the rover there, flies to where it estimates the rover has got to, and picks it back up. That recovery is the difference between a demo and a system.

Honest caveats. The Jev column is a single 65 s run, not a seed-matched average. On an earlier, simpler arena a matched 3-seed comparison showed no advantage for Jev (it crossed 0/3, same as baseline) - the extra maneuvering cost target visibility. Run-to-run variance is large. The claim this repo supports is narrow and specific: the baseline is structurally incapable of the maneuver, and Jev supplies it. It is not "the model makes the drone better at everything".

Worth being precise, because it is easy to overclaim:

Code decides what is there; Jev decides what to do about it. Jev cannot see

• it takes JSON, not pixels. The defensible claim is "a judgment model in a live control loop at 3 Hz with 0.11 s median latency", not "the model does the perception".

On hand-built scenes, 6 of 7 correct with strong probabilities:

LOW BARRIER, all 5 blocked, top 0.45m  -> climb        p=0.93  risk=1.42
TALL PILLAR ahead, right wide open     -> gap_right    p=0.72  risk=1.53
TALL PILLAR ahead, left wide open      -> gap_left     p=0.40  risk=1.58
TARGET GONE 7s, wide open              -> reacquire    p=0.96  lost=0.85
BOXED IN, close on all sides, tall     -> brake        p=0.84  risk=1.89
ALL CLEAR, target dead ahead           -> hold_course  p=0.82  risk=0.46
BRIEF OCCLUSION 0.6s, path clear       -> (muddy, p=0.24)      lost=0.12

The one muddy case is instructive: the Choice was unconfident, but the Noul answered it cleanly (target_truly_lost=0.12). So the search behaviour is gated on the Noul, not on the Choice. Use the primitive that actually fits the question.

Jev initially refused to ever pick climb. That was correct. The state was five horizontal range sectors - it contained no vertical information at all, so "fly over it" was not an inferable option. Adding the obstruction's top-edge height, whether that top edge is visible to the camera, and the aircraft's own climb ceiling moved climb from never-chosen to p=0.93.

That was a state-design bug, not a model failure. It is the single most useful thing this project taught.

./setup.sh                      # venv + fetch the Skydio X2 model
cp .env.example .env            # then put your key in it

set -a && . ./.env && set +a
export MUJOCO_GL=glfw           # or egl on a headless box

.venv/bin/python run.py --seconds 65 --seeds 1              # Jev engaged
.venv/bin/python run.py --no-jev --fast --seconds 65 --seeds 0 1 2   # ablation (free)
.venv/bin/python run.py --seconds 65 --seeds 1 --video course.mp4    # + telemetry video
TRACE=1 .venv/bin/python run.py --seconds 65 --seeds 1      # print every judgment

A video run also writes course.mp4.tape.npy. Re-cut the visuals from that without flying again, and without spending any API credits:

.venv/bin/python replay.py course.mp4.tape.npy out.mp4 --from 12 --to 30

Every knob a human should review lives at the top of tactics.py: the three questions, the option rubrics, and every threshold that changes how the aircraft reacts to a judgment.

Most of the work was the simulator and the controller, not the model. These all cost real time:

• MuJoCo's map znear/zfar are fractions of stat.extent, not metres. In a 40 m arena znear=0.05 blinded the drone inside 1.93 m, so the safety reflex never once fired.
• Image-right is -y for a camera looking down +x, so the target bearing sign was inverted and the yaw loop became positive feedback: the drone turned away from what it was chasing.
• The camera sits inside the airframe, so the drone saw its own body as an obstacle at 3.94 m until own-body geoms were masked out.
• The target's mast is a separate geom from its hull. Identifying the target by geom instead of by body made the drone avoid its own target.
• Clipping negative motor commands destroys thrust and torque together. Thrust-priority mixing plus a slew-limited velocity command took peak tilt from 84 deg (tumble, unrecoverable) to a 40 deg transient.
• Anything you have already climbed over is not a threat. Obstacle pixels have to be filtered by height relative to the aircraft, or the reflex shoves you back off the wall you just cleared.
• The geometric attitude controller cannot recover from inverted flight, by construction. Its error term eR = 0.5*vee(Rd^T R - R^T Rd) has magnitude proportional to sin(theta): it peaks at 90 deg and falls to zero at 180 deg, so an upside-down aircraft sits at a stationary point with no restoring torque. That is the 179-degree lawn-dart. Swapping it for a quaternion error (magnitude sin(theta/2), monotonic all the way to 180 deg) gives maximal righting torque exactly when it is needed most - verified recovering from 179 deg. This one fix also took the obstacle course from 77 m to 88 m.
• A quadrotor cannot thrust downward, and the controller has to know that. On a hard commanded descent a_z + G goes negative, so the desired thrust axis b3 = a_des/|a_des| points at the floor and the controller faithfully commands an INVERTED attitude. The aircraft flips, drops into attitude recovery (which zeroes horizontal acceleration) and then drifts sideways with no lateral control at all - while the navigator is still correctly calling for a dodge. Clamping a_des[2] positive was worth more than every gain-tuning attempt combined.
• Lateral control needs a POSITION loop, not a velocity command. Commanding a lateral velocity and holding it until the next judgment gives the aircraft no notion of where to stop: it crosses the whole corridor and buries itself in the far wall. And the position it servos to has to be estimated: the wall measurement vanishes exactly when it matters (an obstacle fills the view), and a naive "assume centred" fallback had the aircraft convinced it was mid-tunnel while pinned against a wall at y=+5.5 m. Dead-reckon laterally and correct only on plausible wall readings: 0.07 m median error against ground truth.
• A lost target needs a world-frame estimate, not a bearing. A camera bearing is meaningless the moment the target leaves frame, so the drone swept blindly around a stale heading and sat still while the rover drove away. Fix: convert bearing+range+own pose into a world position, carry it forward with the observed velocity, and fly there. Measured against ground truth the estimate is good to 0.14 m and 0.12 m/s - and it must be differentiated over ~1 s, because over one 0.07 s camera frame pixel noise becomes tens of m/s and the drone extrapolates itself off the map.
• A reflex must brake for what is in your path, not what is beside you. Using the nearest obstacle across the whole forward cone meant walls 1.2 m to either side triggered a permanent brake, so any gap narrower than 2x the reflex radius was physically impossible to enter. The gate was not hard, it was forbidden. Splitting out path_ahead_m (the middle sectors only) fixed it.
• A wide lens helps you see the target but ruins threat assessment, because things 60 deg off the nose were never in the way. Separate the tracking FOV from the threat cone.
• Re-applying a heading correction at 50 Hz off a 15 Hz camera applies the same error three times and spins the aircraft. Latch an absolute setpoint per camera frame.
• Sim time is not wall-clock time. The sim ran 10x real time, so 152 of 182 judgment requests hit a full queue and the model influenced nothing. If you are putting a network call in a control loop, pace the sim to real time or you are not testing anything.

A separate, harder setup: a 620 m enclosed tunnel, 15 obstacles, chasing a car, with Jev as the only navigator - no reflex and no hand-written avoidance at all. Two graded Score questions ("how hard to steer, and which way", "should it change height") map straight onto the control command, so the answer is the steering signal rather than a label to act on.

Measured throughput, live in a 500 Hz control loop:

The honest reaction budget at 9 m/s: perception 0.03 s, Jev 0.118 s, decision age 0.024 s, and the airframe 0.70 s to translate 2 m sideways. The model is 13% of the loop; the vehicle is 80%. Making the decisions faster cannot help, and commanding the airframe harder is what makes it tumble.

Status: the aircraft is now stable (flies full episodes, never hits the floor, recovers from >90 deg), and in the best runs it dodged with zero collisions and 94% target visibility. But it does not reliably clear the whole tunnel - it tends to over-commit a dodge and end up against a wall. This is written up as a partial result, not a working system.

MIT. The Skydio X2 model comes from MuJoCo Menagerie under its own license.