The transition contract

What one (s, a, r, s') tuple in this project actually contains, end to end. Reference both for sim (pendulum_env.py) and real-device (real_env.py) environments — they’re identical by design so a sim-trained checkpoint can keep learning on the real rig without any adapter.

The setup in one paragraph

A rotary inverted pendulum (Furuta type). A horizontal arm rotates in the floor plane, driven by a stepper motor (the “motor” angle). At the end of that arm hangs a free-swinging pendulum (the “pendulum” angle). The goal is to swing the pendulum from hanging-down to upright and hold it there. Every 10 ms the policy sees the current state, picks a control nudge, and the world moves on.

Conventions and frame

Pendulum angle θ: 0 means upright, ±π means hanging-down. We use θ rather than the raw MuJoCo joint angle φ everywhere in the observation and reward, because it makes “the goal” trivially θ=0. Internally θ = wrap_pi(φ − π).
Motor angle: 0 is the calibrated centre at startup. Positive = counter-clockwise looking down. Mechanical hard stops at ±135°; we clamp commanded targets to ±125° so the policy never asks for a stop hit.
Control rate: configurable (control_freq_hz). Sim physics integrates at 1 kHz under the hood; real env paces wall-clock to match. The choice of rate is rig-specific and bounded by motor bandwidth and pendulum dynamics — see control_rate_selection.md. The runtime enforcement of whatever rate is chosen is described in async_control_architecture.md.

Observation `s` — what the policy sees (5 floats)

s = \bigl[\,\text{motor\_pos},\ \sin\theta,\ \cos\theta, \ \text{motor\_vel},\ \text{pendulum\_vel}\,\bigr]

Component	Units	Range	Meaning
`motor_pos`	rad	±2.36 (= ±135°)	Current arm angle from centre
`sin(θ)`	—	[−1, 1]	Sine of pendulum-from-upright
`cos(θ)`	—	[−1, 1]	Cosine of pendulum-from-upright. = 1 at the goal, = −1 hanging down
`motor_vel`	rad/s	±200	Arm angular velocity
`pendulum_vel`	rad/s	±200	Pendulum angular velocity

Why sine + cosine instead of θ directly? θ wraps at ±π, so the policy network would see a discontinuity right next to one of its operating points (hanging down). (sin θ, cos θ) is a continuous unit-circle embedding — no jump, no gradient cliff.

Frame stacking (obs_history_len, default 1). The observation can be the concatenation of the last K 6-dim frames (oldest → newest; at reset the stack is seeded with K copies of the initial frame). Since each frame carries prev_action, K > 1 gives the policy observation AND action history at once: it can filter the firmware’s ~±0.4 rad/s velocity-quantisation spikes itself, infer the per-episode θ-bias from its own arm drift (a memoryless policy can only be robust to the bias, never estimate it), and account for the ~17 ms in-flight command. K must match across training, fine-tuning, and deployment — it changes the obs shape, is recorded in config.json, and the deploy/fine-tune entrypoints inherit it from the checkpoint automatically. The on-device path (distill.py → RLControl.ino) predates this knob and only supports K = 1 until both grow matching frame buffers.

Positions-only frames (obs_include_velocities=False, --drop-velocity-obs). With stacking in place the velocities can be dropped from the frames entirely ([motor_pos, sin θ, cos θ, prev_action]); the policy derives its own velocity estimate from the position history. This removes the finite-difference window — and its ±0.4 rad/s quantisation spikes and any filter lag — from the loop, and shrinks the sim/real observation gap to the position channels (which sim already quantises to the encoder LSB under DR). Velocities are still read from the firmware for the reward, the rest detection, and the deploy logs — they just aren’t shown to the policy. Requires obs_history_len >= 2 (use 4); recorded in config.json and inherited from the checkpoint by deploy/fine-tune.

How s is built:

Sim (pendulum_env.py::_obs): read qpos/qvel directly from MuJoCo. With domain randomisation on, we additionally quantise the pendulum angle to AS5600 LSB (12-bit, ~0.0015 rad) and inject small Gaussian noise on positions and velocities — so the sim observation pipeline matches what the real rig actually delivers.
Real (real_env.py::_build_obs): poll the LowLevelServer over serial → un-flip the firmware’s sign convention → finite-difference the velocities and run them through a 20 Hz low-pass filter. The filter is critical: raw finite-difference at 100 Hz on a noisy encoder is unusable.

Action `a` — what the policy outputs (1 float)

a \in [-1,\, 1]

Action mode. The env interprets a two ways, selected by action_mode (--action-mode):

accel (default): a maps to angular acceleration, integrated to a capped velocity and then a position target. Mirrors moveByAcceleration on the rig (CMD_SET_ACCEL).

position_delta: a maps to a per-tick motor-target delta — the mode described below, and the one RLControl.ino runs on-device (CMD_SET_TARGET / moveTo).

This section describes position_delta. Training, fine-tuning, and deployment must use the same mode.

In position_delta mode the action is not a torque or a position. It’s a normalised delta applied to the motor’s commanded target each step:

motor_target ← clip(motor_target + a · max_action_delta_rad, ±125°)

So the policy steers the motor by issuing per-step nudges. The clip enforces the soft motor limit so the policy literally cannot command a hard-stop hit, even if it wants to.

max_action_delta_rad is one of two coupled knobs (the other is control_freq_hz). Their product is the slew rate in rad/s, which must respect the motor’s bandwidth — see control_rate_selection.md for the rationale and recipe.

This action representation has two key benefits:

The stepper firmware (AccelStepper) accepts position targets, not torques. Mapping action directly to a position-delta matches the hardware interface.
Smooth-by-construction: between consecutive steps the commanded position can change by at most 0.1 rad, regardless of policy craziness. This bounds the worst-case actuator slew rate and protects the motor from policy exploration during training.

Transition dynamics — going from `s` to `s'`

In sim (per step):

Apply the action delay queue (DR samples a 0–N-step delay scaled to bracket the rig’s measured transport delay; the action that takes effect now might be the one the policy chose several steps ago — modelling serial RTT + AccelStepper ramp).
Update the commanded motor_target with the (delayed) action.
Apply motor first-order lag (DR samples τ ∈ [0, 10] ms): the motor_applied value fed to MuJoCo trails the commanded target by an exponential of time-constant τ. With τ=0 it’s instantaneous.
Step MuJoCo physics for one control period at 1 ms substeps.
Read out the new state, build s', compute reward.
Episode terminates if the motor hits the hard stop at ±135° (incurs a −5 penalty); truncates after episode_length_s (default 8 s).

In real (per step):

Update the commanded motor_target with the action (no action delay queue — the real hardware is the delay).
Send set_target(motor_target) over serial.
Sleep until the next tick (paces wall-clock to control rate).
Read state from the rig: (time_us, motor_pos, pendulum_pos).
Finite-diff + low-pass filter the velocities.
Build s', compute reward.
Terminate on |motor_pos| ≥ 135° (firmware also enforces this in hardware as a backstop); truncate after episode_length_s (default 6 s during fine-tuning).

The key sim-to-real bridge is that the firmware’s transport delay, acceleration ramp, and stepper friction are what the sim’s action_delay_steps, motor_tau_s, and joint friction parameters are modelling. Domain randomisation samples those over plausible ranges each episode so the policy has seen enough variation to handle the real point.

Reward `r` — what we’re rewarding

The current reward is the standard Quanser quadratic-cost form (common in Furuta-pendulum literature):

r = -\left[\, \theta^2 + k_{\dot\theta}\,\dot\theta^2 + k_{\alpha}\,\alpha^2 + k_{\dot\alpha}\,\dot\alpha^2 + k_{a}\,a^2 \,\right]

where:

$\theta$ = pendulum-from-upright (rad). The dominant term: $\theta^2$ is 0 at the goal and $\approx \pi^2 \approx 9.87$ at hanging-down.
$\dot\theta$ = pendulum_vel. Small $k_{\dot\theta} = 0.001$ weight discourages spinning through upright forever.
$\alpha$ = motor_pos (rad, sim’s misnamed-from-Quanser variable). $k_\alpha = 0.5$ keeps the policy near centre.
$\dot\alpha$ = motor_vel. $k_{\dot\alpha} = 0.005$ discourages frantic arm motion.
$a$ = action $\in [-1,\, 1]$ . $k_a = 0.05$ light penalty for jerky control.

Alive terms (added 2026-07-21, audit finding F1). On top of the quadratic cost, the default training reward now adds:

\begin{aligned} r &\mathrel{+}= k_\text{alive offset} && \text{(default 15.0)} \\ r &\mathrel{+}= k_\text{upright alive} \cdot \mathbf{1}\!\left[\, |\theta| \le 15^\circ \ \text{and}\ |\dot\theta| \le 2\ \text{rad/s} \,\right] && \text{(default 5.0)} \end{aligned}

Rationale: the purely non-positive cost combined with hard-stop termination made early termination attractive — hanging for a full 8 s episode costs ≈ −4000, while crashing into the rail costs a few hundred total, so “drive into the wall” was a strong local optimum (the observed stage-2/3 training collapses). The constant offset is chosen above the worst realistic per-step cost (~14), making per-step reward non-negative so terminating always forfeits value. The velocity-gated upright bonus pays only for caught balance — a pendulum swinging through the upright band at speed earns nothing — using the same gates as analyze_deploy.py’s honest balance metrics, so training optimises exactly what deployment certifies. Set both to 0 (--reward-alive-offset 0 --reward-upright-alive-weight 0) to recover the legacy canonical reward; the values are recorded in config.json and must match between training and fine-tuning.

With the alive terms at 0 the reward is purely non-positive — max 0 when fully balanced still at centre with no motor activity, around −10 per step at hanging-down. SAC handles negative rewards fine, and the all-negative signal makes “less negative” gradient toward upright unambiguous — but see the termination caveat above.

What the policy actually learns to do:

Far from upright: Pump the arm back and forth. The θ² term rewards getting upright; the α² and α̇² penalties keep the pumping bounded so it doesn’t slam into the limits.
Near upright: Hold still. The dominant θ² term goes near zero there, leaving only the small velocity/action penalties as residuals — so the policy is rewarded for any state close to (θ=0, θ̇=0, α=0, α̇=0).

If the policy gets the pendulum near upright but not still, the cost is dominated by k_θ̇·θ̇². If it balances but with the arm wandering, the cost is dominated by k_α·α². These weights are what shape the policy from “wobbly catch” toward “smooth hold”.

Episode boundaries

Event	What happens	When
Reset	Sim places pendulum hanging-down with small noise, motor at random `±0.7·motor_safe_limit`. Real disengages motor, waits `reset_settle_s` for pendulum to coast to rest, re-engages at current motor position.	Every episode
Termination	`terminated = True`, reward gets a final `−5` penalty. Episode boundary.	Hard-stop hit (`
Truncation	`truncated = True`, reward unaffected. Episode boundary.	Time limit reached (8 s sim, 6 s real default)

Truncation just bookkeeps the time limit — the value-of-future is still estimated normally. Termination signals the value should drop to zero (“game over”) and is reserved for the hard-stop bad outcome.

Why this representation works

A few non-obvious choices, called out:

(sin θ, cos θ) over θ: avoids the wraparound discontinuity. The policy sees a smooth manifold, not a step function.
Action as delta-in-target, not absolute target: by integrating the policy output, we let the policy steer rather than hop. Slew rate is bounded; motor can never be commanded to teleport.
Reward purely negative: simpler optimisation surface than mixed positive/negative. SAC’s entropy bonus handles exploration; we don’t need positive shaping bonuses.
Same env for sim and real: zero translation cost on checkpoint-load. The replay buffer in Phase 4 fine-tuning fills with real (s, a, r, s’) tuples that look identical in shape to the sim ones the policy already learned from.

Where things live in code

File	What it does
`pendulum_env.py`	The full sim env — MJCF model, DR, action delay, motor lag, reward. The canonical reference.
`real_env.py`	Hardware version. Deliberately mirrors `pendulum_env.py`’s observation, action, and reward exactly.
`run_policy.py`	Deployment-only client. Same observation pipeline as `real_env.py`, no learning.
`async_control.py`, `finetune_async.py`	Runtime that produces transitions during fine-tuning at strict rate. Internals out of scope here — see `async_control_architecture.md`.
`finetune_real.py`	Deprecation shim → forwards to `finetune_async.main`.

Read pendulum_env.py::step and pendulum_env.py::_obs together for the canonical sim transition; read real_env.py::step and real_env.py::_build_obs to see the same flow against hardware.

The transition contract

The setup in one paragraph

Conventions and frame

Observation s — what the policy sees (5 floats)

Action a — what the policy outputs (1 float)

Transition dynamics — going from s to s'

Reward r — what we’re rewarding