Robots need skills.
Humans already have them.

Meanwhile, humans upload millions of hours of skilled behavior for free. Central question: can human motion serve as the reward signal for RL?

How should we define the reward?

Every block is standard except the reward. For the skills we care about, nobody knows how to write it by hand.

Could the video itself be the reward?

Reward as pixel comparison?

reward  =  −‖

human video · frame t

−

simulator render · frame t

‖²  ?

Comparing raw pixels is tough: different appearance, viewpoint and embodiment; noisy, prone to reward hacking (the policy exploits the metric), and it says little about how to move.

Or compare in a latent space?

reward  =  −‖ φ(

human video · frame t

) − φ(

simulator render · frame t

) ‖²

φ is a frozen encoder pre-trained on human video (recall Lecture 6), e.g. R3M (Ego4D). Inverse RL has long pursued this: learning the reward, or the features it is built on, from demonstrations. Rewards from XIRL and VIP point in this direction.

Invariant to appearance and viewpoint, and an active line of work. But the structure stays implicit: hard to interpret, easy to hack, silent about bodies, hands and objects. Can we make it explicit? Can we extract more structured signals from the video first?IRL: Ng & Russell, 2000; Abbeel & Ng, 2004 · R3M: Nair et al., 2022 · XIRL: Zakka et al., 2021 · VIP: Ma et al., 2023

We can 4D-fy videos

As in Lecture 8 (HMR 2.0, HaMeR, WHAM, HaWoR), off-the-shelf models lift ordinary video into 3D geometry moving through time: bodies, hands and objects.

A 4D reconstruction is a trajectory of states, and a trajectory of states can serve as a reward function: reward the robot for tracking it.

Foundation models are 4D-fying for us

With GPT-6 Astra, real2sim (rebuilding a real scene as a simulation) is collapsing from a research problem into a prompt: video or photos in, physics-ready 3D & 4D scenes out.

The references RL needs are becoming one prompt away: much of what follows is getting cheaper.

Imitation as a tracking reward

Instead of engineering task rewards, reward the agent for matching a reference motion inside a physics simulator.

the demonstration

r^imitate — from the body

r^task — from the world

Drop either term and RL finds the wrong optimum. Figures: DexMachina, Mandi et al., 2025.

The video is human, the robot is not

Left: the person. Middle: a fitted SMPL body. Right: the humanoid reference after retargeting. Proportions, joint limits and degrees of freedom all differ.

Kinematic retargeting

01 — Extract 3D landmarks / keypoints

Fit a parametric model (SMPL for bodies, MANO for hands) and read out its joint keypoints: trajectories x_i,t the robot must reproduce. (SMPL — Loper et al.)

02 — Kinematic retargeting / inverse kinematics (IK)

JᵀJ⁺DLS

StepRunTrack

Damping λ

q^*_1:T = argmin_q Σ_t,i ‖FK_i(q_t) − s·x_i,t‖² + λ‖q_t − q_t−1‖²
s.t.  q_min ≤ q_t ≤ q_max

Joint angles q_t that put the robot's links on the keypoints, solved frame by frame (differential IK), with λ tying each frame to the previous one. FK_i: forward kinematics of link i; s: human-to-robot scale. (mink — Zakka, 2026)

This is pure kinematics: making q_1:T dynamically feasible is RL's job.

Human shape ≠ robot shape: reshape before retargeting

Landmarks from a human-proportioned body land in places the robot cannot reach. Fitting the human model to the robot's shape first makes the IK targets attainable.

Bodies · raw human  |  shape-fitted human  |  humanoid

Hands · morphometric optimization → residual RL → policy

Kinematic retargeting breaks physics

Skating & collision costs

Grounding can happen inside the IK solver: add costs that pin contacts and forbid penetration.

Both costs as used in VideoMimic, Allshire et al., 2025.

Skating & collision costs are blind to relations: where the hand sits relative to the box, or the body relative to the terrain. OmniRetarget preserves the whole interaction, not only the contacts.

The interaction mesh

Delaunay tetrahedralization over body joints (blue) and points sampled on the object & environment (green); target (robot) left, source (human) right.

Laplacian deformation energy

L(p_t,i)  =  p_t,i − Σ_{j ∈ 𝒩(i)} w_ij · p_t,j

E_L  =  Σ_i ‖ L(p^source_t,i) − L(p^target_t,i) ‖²

The Laplacian coordinate is a keypoint's offset from the average of its mesh neighbors (uniform w_ij = 1/|𝒩(i)|), so it encodes local relative geometry. Human keypoints are first rescaled by the robot-to-human height ratio; E_L absorbs the remaining proportion mismatch. Minimize E_L (+ a smoothness term) over the robot configuration q_t, with collision avoidance, joint & velocity limits and no foot skating as hard constraints: skating and collisions become constraints, not costs.

IK costs only approximate physics. Instead: roll the motion out in a physics simulator and search for the controls that reproduce the demo. Whatever comes out is feasible by construction.

Closed-loop, or it falls

• The tracking reward. Reward = how closely the simulated character's joint configuration & velocities match the reference clip at each timestep, plus an optional task term, trained with standard RL (PPO).
  r^imitate_t = ω^config r^config_t + ω^velocity r^velocity_t + ω^ee r^end-effector_t + ω^CoM r^CoM_t,
  e.g.  r^config_t = exp[−2 Σ_j ‖q̂_t,j ⊖ q_t,j‖²] ↓ why exp(·)?
  q_t,j: simulated rotation of joint j · q̂_t,j: reference rotation · ⊖: quaternion difference · ω: scalar weights
• Reference State Initialization. Start episodes at random points along the clip, so the policy learns the landing of a backflip before it can reach it. This removes the exploration bottleneck.
• Early Termination. End the episode on failure (e.g., the torso hits the ground), so hopeless states stop polluting the data and lying on the floor never becomes an optimum.

The tracking reward alone is not enough: RSI and ET are what make dynamic skills learnable.

The harder the skill, the more they matter

Backflip — a dynamic skill

Without ET, falling floods the data and return plateaus: the policy settles for a partial flip. Without RSI, it never experiences the landing until it can already reach it. Only RSI + ET learns the full backflip.

Walk — an easy skill

For walking, every variant converges, though without ET (green) it needs several times more samples. The tricks matter far less when exploration is easy: every state is a few steps from a good one.

Learning curves from Fig. 11, Peng et al., 2018. The tracking reward defines the task; RSI & ET decide whether RL can solve it.

Imitation shapes the throw, the task aims it

both terms: a baseball pitch, aimed at the target

without the imitation term

Imitation-only policies hit the target in 5% of throws and 19% of strikes; with both terms, 75% and 99%. Drop either term and RL finds the wrong optimum. Figs. 7–8, Table 4, Peng et al., 2018.

From one reference to many

DeepMimic and SFV train one policy per clip. Two routes lead to a single controller that holds an entire motion library.

Both routes end in one network; they differ in where the RL happens.

SONIC: one policy, any reference

A single tracker trained on ~700 h of mocap follows whatever reference it is given, whatever produced it.

Same weights in every clip, real Unitree G1. Videos from nvlabs.github.io/GEAR-SONIC, Luo, Yuan, Wang, et al., NVIDIA, 2026.

Residual vs. non-residual control

Body tracking lets the policy output the whole action. Many dexterous-hand works instead learn a correction on top of the retargeted reference.

r_t  =  r^task_t  (object pose tracking)

• DeepMimic, applied to the object. Exponential tracking reward, early termination, time step in the state; but the reference is an object pose trajectory (from mocap, animators or expert policies), not a body motion.
• TCDM benchmark. 50 tasks, 34 objects, 3 hands (Adroit, D'Hand, D'Manus). Reward, termination and hyper-parameters are identical across tasks: only the exemplar changes.
• Non-residual. The policy outputs the full hand action directly (slide 30, left).

Object tracking is embodiment-agnostic: the same exemplar defines the task for any hand, the idea DexMachina builds on later. Dasari et al., 2023 · pregrasps.github.io

One pre-grasp beats a full hand reference

01 — Start from a pre-grasp

A planner moves the hand to one pre-grasp pose (from mocap via IK, teleop, hand labels or a grasp predictor); RL starts from there. The manipulation version of DeepMimic's reference state initialization.

02 — Final success, TCDM-30

Without a pre-grasp, even DeepMimic-style fingertip tracking stalls near 0.2; with it, every method reaches ≈0.8–0.85. Extra hand supervision adds nothing.

Fig. 3, Dasari et al., 2023. As with RSI and ET in DeepMimic: in dexterous manipulation, exploration is the bottleneck.

YouTube as the demonstration source. Mine piano-performance videos + MIDI: fingertip trajectories from the video become the r^imitate, pressed keys the r^task.

r^imitate — distance between the robot's fingertips and the human pianist's fingertips extracted from the video: it tells the policy which finger plays which key.

r^task — the piano's state: reward for pressing the right keys, and only the right keys, read off the target MIDI.

Notes alone are not enough

Baseline: task reward only, correct notes with no human reference. IK: kinematic replay of the human fingertips, no RL. Ours: IK warm start + residual RL (slide 30) with imitation and task rewards. Fig. 3, Qian et al., 2024.

a_t = a^I_t + Δa^R_t    r_t = r^hand_t + r^object_t + r^contact_t

• Imitate the hand first. A generalist hand-only imitator is trained with PPO on large mocap data (wrist and finger tracking), with no object in the scene.
• Then a residual per task. A zero-initialised residual adds the corrections the object needs; its reward adds object following and contact force terms (slide 30, right).
• Physics relaxation curriculum. Start with no gravity and high friction, then restore them gradually. Plus RSI and early termination with a shrinking object threshold.
• DexManipNet. 3.3K episodes on the Inspire hand from FAVOR + OakInk-V2 (pen capping, bottle unscrewing), deployed on real hands.

Hand-first: the human hand motion is the base action, and RL only learns what the object adds. Li et al., 2025 · maniptrans.github.io

• Functional retargeting. From one human hand-object demo (from ARCTIC, a Lecture 8 reading), learn a policy whose main reward is reproducing the object's trajectory (as in TCDM), with auxiliary hand-motion and contact terms from the retargeted demo: long-horizon, bimanual tasks with articulated objects.
• Virtual object controllers with decaying strength. At first the object is driven toward its targets "by magic", and the policy only needs to follow along; the assist decays on a curriculum until real contacts carry the task.
• Cross-embodiment by design. The object trajectory is embodiment-agnostic, so the same demo trains Allegro, Inspire, XHand and Schunk hands, and the benchmark doubles as a functional comparison of hand hardware.

Object-first: tracking alone is not enough, and the decaying virtual-controller curriculum makes long-horizon dexterity learnable. Mandi et al., 2025

r_t = r^task_t + r^imit_t + r^wrench_t

• Match what the contacts can do. Contact location alone does not fix the object's motion. Reward the robot when its contacts can push the object the way the human's contacts could; this transfers across hand shapes.
• Combines both recipes. Residual on the retargeted motion (as ManipTrans) and annealed virtual object controllers (as DexMachina).
• Scale. 4,739 bimanual tasks from mocap and in-house videos; 82% success on 1,831 tasks with one set of hyper-parameters. Real transfer on two Sharpa hands.

Contact-first: the human tells the robot what forces to apply, not where to put each finger. Zhu et al., 2026 · Tab. 1 & ablation · project page

One recipe, simulation to hardware

Box, mixer and capsule machine run in both sim and real; the long-horizon sim clips are sped up 2×. Zhu et al., 2026 · videos from the project page

Randomize the simulator, or identify it

real robot   randomized sims   identified sim  ·  In practice: Sys-ID to centre, DR around it.

Also: online adaptation (RMA) · teacher–student / asymmetric critic · learned sim–real residuals (ASAP) · noise & latency models.  Tobin et al., 2017 · Peng et al., 2018 · Tan et al., 2018 · Hwangbo et al., 2019

Sys-ID matches the trajectory, DR covers it

Toy PD-controlled joint (inset: one motor + link, dark = real, orange = sim, teal = randomized sims), I q̈ = K_p(q^cmd − q) − (K_d + b) q̇, simulated live. DR samples inertia, friction and actuator delay. "Covered" = real measurements within ±0.04 rad of some simulated rollout.

How wide? Let the policy earn every widening

With a memory (LSTM), the policy learns to identify the world it is in while acting: sys-ID happens inside the network. OpenAI et al., Rubik's Cube, 2019 · Handa et al., DeXtreme, ICRA 2023 (ADR on GPU)

Identify the actuators, randomize the rest

Still a gap? Learn a correction from a few real rollouts (ASAP).  For small hobby servos, actuator fidelity is most of the sim2real gap (Microduck).

Trained in simulation, walking in the real world

Randomize in sim, then go zero-shot

Both train in simulation only and transfer zero-shot, with no real-world fine-tuning. SimToolReal randomizes the objects (procedural tools); OmniReset randomizes the starting states and, for the RGB student, the visuals. arXiv 2602.16863 · arXiv 2603.15789 · simtoolreal.github.io · OmniReset

Privileged teacher, deployable student

The same split runs through this lecture: RMA and HORA infer the hidden physics; Dactyl and DeXtreme learn the pose from rendered images; RotateIt adds simulated touch. Lee et al., 2020 · Kumar et al., RMA, 2021 · Qi et al., HORA, 2022 · Chen et al., Visual Dexterity, 2023 · Qi et al., RotateIt, CoRL 2023

Two ways to distill: whose states do we learn on?

DAgger only works because the teacher can be asked anywhere: in simulation the privileged state exists at every state the student visits. With a human teacher, that is expensive; with a sim teacher, it is free.