Skip to content
Rotary Inverted Pendulum

The pipeline, end to end

How to take a freshly-built rig from “no policy at all” to “balances standalone on the Nano, no laptop tether”. Each step lists the command, the expected wall-clock cost, and what the next step depends on.

The end-to-end pipeline, in five stepsSteps 0 to 2 run in order: system identification, training in simulation, then fine-tuning and testing the teacher on the real rig, tethered to a laptop. If the teacher scores a balanced fraction of at least 0.85, steps 3 and 4 follow — distilling and gating a smaller student, then flashing and scoring it running standalone on the Nano. Stopping after step 2 leaves a working controller that needs the laptop attached.only to remove the tether0sysidmeasure the rig1train in simcurriculum, ~25 min2fine-tune + teston the real rig, tethered3distill + gateshrink the student4flash + scorestandalone on the Nanobalanced ≥ 0.85
  1. 0sysidmeasure the rig
  2. 1train in simcurriculum, ~25 min
  3. 2fine-tune + teston the real rig, tethered
  4. 3distill + gateshrink the student
  5. 4flash + scorestandalone on the Nano
Each step is idempotent and can be re-run on its own. Stopping after step 2 leaves a rig that balances with the laptop attached.

If you only need the upright/balance behaviour and are happy keeping the laptop attached: stop after step 2. Steps 3–4 only exist to remove the tether.

For how a single transition works, see the transition contract. For the decision log behind the hardware/firmware shape of the pipeline, see transport delay.

The canonical operating point

Section titled “The canonical operating point”

50 Hz, velocity mode, ±3.5 rad/s, K=4 frames, 4-tap actuator action smoothing, mirror augmentation, and observation-staleness DR over 2–20 ms. Every entry point — train_sac.py, curriculum_train.sh, distill_student.sh, finetune_async.py, run_policy.py and RLControl.ino — is set to this, so a bare run of these steps reproduces a champion-grade policy. This is verified against the champion’s config.json, not asserted: a bare train_sac.py run writes a config.json identical to the champion’s on all 17 shared keys.

The last two are recent and both are required to reproduce the current champion — it was trained with them, so a run without them is a different recipe:

  • Mirror augmentation stores each transition’s mirror image as well. The rig is exactly left/right symmetric but SAC breaks that symmetry on every run, which showed up as a persistent arm lean (why).
  • Staleness DR 2–20 ms spans the 15.6 ms sample→command latency measured on the rig; the older 2–10 ms range did not contain it, so no policy trained at the latency it deployed at (why).

Together they took a matched 300 s standalone capture to 1.000 balanced / 299.6 s / zero drops, with |action| 0.325 → 0.239, pendulum σ 2.53° → 1.90°, arm speed 1.18 → 0.89 rad/s and arm lean −17.4° → +4.6°.

Do not change one of them in isolation. Training, fine-tuning and deployment must agree, and run_config.check_config aborts on a mismatch rather than letting you deploy a rate-mismatched policy.

Action mode — velocity is the production mode

Section titled “Action mode — velocity is the production mode”

The validated pipeline uses velocity mode end-to-end: the policy’s action is a velocity setpoint (±3.5 rad/s), converted to accel commands each tick by a P-law tracking the controller’s own commanded-velocity integrator (never the quantised measured velocity — that injected a ±17 rad/s² dither, removed 2026-07-21), over the same CMD_SET_ACCEL firmware transport in every deployment. curriculum_train.sh, finetune_async.py, run_policy.py and RLControl.ino are all set for this mode; run the commands as written.

Legacy modes remain selectable for comparisons (--action-mode accel / position_delta, see the transition contract), and RLControl.ino no longer speaks position-delta.

train_sac.py’s own bare defaults are the canonical recipe too, so calling it directly no longer trains something different. Each legacy behaviour has an explicit opt-out: --action-mode accel, --no-firmware-obs-model, --no-mirror-augment, --reward-stillness-bonus-weight 0.

Prerequisites

Section titled “Prerequisites”
  • macOS / Linux dev box with arduino-cli, the arduino:avr core, and the AS5600 (RobTillaart) + FastAccelStepper libraries installed.
  • Python env set up per the Python tooling page. The project is uv-managed: prefix each command below with uv run (e.g. uv run python train_sac.py …, uv run bash curriculum_train.sh …), or activate the project venv once and drop the prefix. macOS only: commands that open the MuJoCo viewer (the --eval rollouts) must use mjpython, not python — e.g. uv run mjpython train_sac.py --eval ….
  • Rig wired with STEP on pin 9, DIR on pin 2, ENABLE on pin 5, AS5600 on I²C (A4/A5). Pin 9 is required by FastAccelStepper on ATmega328 (Timer1 OC1A — see the pin notes in LowLevelServer.ino) and works for AccelStepper too.

In the commands throughout these pages, replace /dev/cu.usbserial-1130 with whatever port arduino-cli board list shows for your Nano.

Re-running individual steps

Section titled “Re-running individual steps”

Every step is idempotent and can be re-run on its own:

Want toRe-runResume from
Tweak rewards or DR rangesstep 1scratch
Add real-rig datastep 2--resume-buffer
Re-distill (new teacher or transport)step 3existing teacher
More DAgger roundsstep 3bexisting --bc-dir
Re-flash with the same studentstep 4existing .h