Choosing a foothold

Pupper's foot is half a tread: test whether terrain information helps before choosing a sensor or a foothold planner

Choosing footholds from terrain perception and then placing the feet there is an established line of work on 9–90 kg quadrupeds. At Pupper v3's size it has been done with motion capture and known terrain (LittleDog, Solo-12), and once with a camera on the robot. LittleDog, whose leg links (7.5 + 10 cm) match Pupper's 0.1725 m leg, crossed obstacles of 10–10.8 cm with laser-scanned terrain and motion capture at 7–13 cm/s (Kalakrishnan et al., ICRA 2010 and IJRR 2011). In 2009 it crossed rocks of 6 cm and 12 cm relief with a stereo camera on the robot, updated twice a second, without motion capture; no trial counts were given (Kolter, Kim and Ng, ICRA 2009). Since then the field has split in two. Explicit pipelines pick a foothold on a height map and a controller tracks it. Implicit pipelines give a policy terrain heights or depth and let it place its feet. Same-paper comparisons agree on where each wins:

• On dense terrain such as stairs the two tie once both see the terrain: RLOC 93.0 % against 93.2 % for an end-to-end height-scan policy. An end-to-end attention policy also beat the DTC hybrid on stairs, 98.6 % against 92.2 %.
• Explicit footholds win on sparse terrain (stepping stones, gaps, beams) when the planner models the supports correctly: DTC crossed a 0.1 m gap 10 of 10 times where an end-to-end policy failed.
• They lose when the planner's support model is too strict: on 12 cm stones the attention policy scored 85.8 % against DTC's 0.0 %.
• Knowing the right foothold does not solve short treads by itself. A humanoid tracking exact foothold targets climbed a 0.13 m rise on 0.300 m treads in 12 of 100 trials against 46 on 0.375 m treads, and riser contact caused 70 and 23 of the falls (Mind Your Steps, RSS 2026).

Shappy's blind policy fails where foot placement matters. own3_h1 crosses 5 cm stairs 15–19 times in 100 on 80 mm treads but 92–94 times on 160–240 mm treads. It crosses 4–5 cm box fields 15–72 times depending on the box layout. Scaled by leg length, ETH's 17 × 29 cm stair tests become 4.7 × 8.1 cm stairs for Pupper, which is the printed course. On ANYmal a terrain map raised step-ups of 0.25 and 0.38 leg lengths from 70 to 100 % and from 0 to 90 %; for Pupper those steps are 4.4 and 6.6 cm.

What is new at Pupper's size is the foot. A 40 mm foot on an 80 mm tread leaves its centre a ±2 cm band, one foot diameter wide, against about four on ANYmal's stair tests and six on A1's. Each error source of a foothold pipeline, as published, would take half or all of that band:

• foothold tracking, 2.3 cm (DTC);
• state estimation, 3–4 cm (ANYmal-B);
• leg-odometry drift between the camera's last view of the ground and a hind-foot touchdown, 1.4–1.6 cm;
• 100 ms of latency at 0.5 m/s, 5 cm.

Pupper's head camera sees each front foot's next foothold at about 1 mm per pixel when the foot lifts off. It gives no depth, loses the touchdown spot 20–70 ms before contact, and last sees a hind foot's landing ground 0.56–0.61 s before that foot lands.

For Shappy the order is therefore:

1. In simulation, give the actor the height scan the critic already receives. Compare it with an identically trained blind twin on 5 cm stairs with 80 mm treads, 4–6 cm boxes and new Pupper-scale sparse lanes. This is the G3 diagnostic already listed in terrain_training/MULTI_TERRAIN.md, not yet run. It decides whether terrain information raises Pupper's ceiling at all, for roughly half an hour to an hour of training per arm.
2. Only if it does, model in the same pipeline what a real sensor would deliver (field of view, memory, noise, latency), and then choose the sensor. A RealSense D405 on the nose, tilted 45–60° down, is the route with a small-robot precedent. Monocular depth from the existing fisheye on the Hailo-8L is untested on any legged robot.
3. Explicit foothold targets only if sparse terrain becomes a goal, or if the VLA layer is meant to command footsteps.
4. For V1.0, reserve a downward-pitched depth-camera mount and an Orin-class compute slot. The small LiDARs suit V1.0, not Pupper.

Sources and per-paper numbers are in five notes in notes/research_notes/Perceptive foothold selection/:

• model_based_foothold_selection.md: planners, scores, front ends, hardware results, compute, explicit against implicit.
• learned_foothold_policies.md: learned selectors, foothold-conditioned policies, sparse-foothold policies, rewards, the head-to-head table, footholds as a command interface.
• sensing_for_footholds_small_robots.md: depth cameras, LiDAR, stereo, monocular and fisheye depth, mapping, odometry and memory, Pupper's camera.
• training_and_evaluation_mjx.md: simulator and Brax support, scan geometry, noise models, target generation, evaluation design, and the code changes in this repository.
• pupper_camera_geometry.md: what the head camera and a D405 see of the footholds, computed from the official CAD and MJX model. The script pupper_camera_geometry.py and its output sit beside it.

The earlier survey, notes/reports/Multi terrain quadruped policy training.md, already covers perceptive policies in general (height maps, depth and parkour pipelines, RGB, blind against perceptive). This report does not repeat it.

Explicit foothold selection has used one decomposition since LittleDog

Every explicit pipeline found has the same four steps:

1. Terrain becomes a 2.5D height map or a set of convex surfaces.
2. A Raibert-style heuristic proposes a nominal foothold.
3. A search or optimiser moves that foothold to a nearby cell that scores well.
4. A controller places the foot there.

The scores have hardly changed in 15 years:

• slope and roughness at the scale of the foot;
• distance to edges, at 1.5–2.5 foot sizes;
• step height and kinematic reach;
• shin and knee collision at the scale of the leg. Once the other scores worked, knee collision became the most frequent stair failure in the ETH papers.

Cells were 0.5–1 cm on LittleDog and 4 cm on HyQ and ANYmal, about 0.4–1 foot diameter. At Pupper's size that means 1–2 cm cells (computed). Shappy's training heightfield has 2 cm cells, and the critic's scan samples every 3 cm.

Learned variants replace one stage at a time:

• the foothold evaluator (a HyQ CNN of 8,238 parameters chose footholds the heuristic called safe in 99.74–99.88 % of cases, 15–200 times faster);
• the selector (DeepGait, RLOC, Visual-Locomotion, SafeSteps);
• the tracker (DTC, Raibo).

Foothold-conditioned trackers reach their targets to within a few centimetres: 2.3 cm on ANYmal on flat ground (DTC), 5.3 cm on Raibo hardware and 3.7 cm in simulation. DTC took 14 days to converge and was "not more sample efficient" than end-to-end RL.

Compute splits into cheap local parts and expensive global ones. Every figure with a named computer comes from x86 or Jetson hardware; nothing has been measured on a Raspberry Pi.

Explicit footholds pay on sparse terrain; on stairs a policy that sees the heights does as well

Three readings follow.

1. Once both arms see the terrain, dense terrain shows little difference. In RLOC's table perception supplies most of the stairs gain (72.2–77.0 % with the planner blind in xy, 92.3–92.5 % with a heuristic, 92.7–93.6 % learned), and learning supplies the bricks gain. The end-to-end policy's lower bricks score came with heights sampled every 0.1 m against the planner's 0.02 m.
2. Explicit selection wins where its model of the supports is right and unstructured RL fails to explore, as on gaps and stones. It loses where that model is too narrow, for example supports below its steppability threshold. On the stone courses QuadPiPS's elevation map merged the stones into "one large surface".
3. The model-based pipelines' own stair failures were knee and shin contact going up, and over-extension or a lost view of the lower treads going down. The knee failures shrank when the legs were modelled (TAMOLS, Grandia's knee distance field), not when footholds improved; TAMOLS's knees still "occasionally collide with the edges".

Short treads have little direct evidence.

• Mind Your Steps is the clearest case: exact footholds, failure by riser contact. The authors write that on short treads "the swing foot must lift nearly vertically to clear the riser".
• With perception disabled, DTC's ANYmal crossed 0.18 × 0.29 m stairs 3 times up and 3 times down. By leg length that is Pupper's 5 cm rise on 80 mm treads, so the ceiling need not be informational. But n = 3, ANYmal's actuators differ, and its tread is 4.8 foot diameters against Pupper's 2.0.
• On Go2, StairMaster's depth policy climbed 55° hollow stairs 19 of 30 times. A blind policy (HIMLoco), Unitree's built-in controller and the depth-based Extreme Parkour policy each scored 0 of 30. Hollow stairs have no risers to feel, which confounds the blind comparison.

Rewards that read the terrain under the foot are the cheapest placement lever, and they serve implicit policies too:

• Removing a feet-edge term from a Lite3 depth policy dropped real stepping stones from 5 of 5 to 3 of 5, and 0.7 m gaps from 5 of 5 to 0 of 5 (START).
• Without PointWalker's edge term, edge landings rose from 11.5–18.7 % to 32.4–57.2 %.
• Extreme Parkour's edge term cut mean edge contacts per step from 0.08 to 0.03.

AME-2 deliberately uses no foothold reward, because near-edge footholds and knee contacts help climbing. On Shappy's mixed field that argues for gating an edge term to stair tiles. At Pupper's scale the band would be about ±2 cm from the nosing and the riser, since the published 2.5–5 cm bands would cover the whole 80 mm tread.

How foothold targets are generated repeats Shappy's main failure:

• Uniformly sampled step targets produced policies that "simply learn to stand on the first step" (ALLSTEPS).
• An end-to-end stepping-stone baseline "immediately collapses to standing" (Walk the PLANC).
• Raibo's generator, which adds difficulty only when the tracker succeeds, passed more than 95 % where a fixed-order curriculum stalled at 4.3 of 10 stones.

No paper relabels a policy's own footholds in hindsight.

As a command interface, footholds bought precision and coverage, in simulation and without perception:

• Mind Your Steps' bridge: 100 % with foothold commands against 36.6 % with velocity commands.
• A contact-goal quadruped trained only to walk forward and backward walked sideways when commanded, where the velocity policy hesitated (Omar and Khadiv, L4DC 2026).
• One route fits Shappy's velocity-only actor without retraining. Taouil et al. (IROS 2023) learned how Aliengo's black-box velocity controller moves its feet (footstep error 1.99 cm), then ran A* over velocity commands on a foot cost map. It crossed 10–15 cm gaps and a 6 cm step in 3 of 3 attempts per scenario, with motion-capture odometry.
• No perceptive policy trained across several bodies was found.

At Pupper's size perception should matter, and placing the foot is hardest

Shappy's failures sit where terrain information has helped other robots.

• At 0.5 m/s own3_h1 strides 24.3 cm per gait cycle, three 80 mm treads (project records). How often its feet land near a nosing has not been measured; the per-touchdown metrics proposed below would show it.
• By leg length the printed course is ETH's 17 × 29 cm staircase (4.7 × 8.1 cm), and the map raised ANYmal's step-ups at 4.4 and 6.6 cm Pupper-equivalent (computed).
• With perfect terrain knowledge LittleDog, a robot with Pupper's leg length, crossed 2.5–3 times the rise Shappy's blind policy manages. It did so in a slow static walk at 7–13 cm/s, against the 39 cm/s own3_h1 walks at a 0.5 m/s command. Without perception, LittleDog absorbed planks up to 4 cm, Shappy's blind level.

The foot does not shrink with the robot.

• Pupper's foot is 0.23 leg lengths across, against about 0.10 for ANYmal C and Go2 (computed).
• In foot units the 80 mm tread is 2.0 foot diameters. That equals the hardest supports any quadruped paper trains on: He et al.'s 12 cm stones, where DTC scored 0 %, and MARG's 9 cm beams.
• In leg units the tread is 0.46 leg lengths, wider than any ANYmal stone.
• Each tread spans the whole lane, so placement on Shappy's course is a fore-aft problem only.

The error budget against the ±2 cm band:

By inference, explicit placement on 80 mm treads needs about 1 cm of end-to-end error. The smallest published figure found is DTC's 2.3 cm, for tracking alone on flat ground. A learned controller can still feel for the riser when the map is wrong. In DTC's simulations, success held constant for map drift below 0.1 m and then fell linearly. With a gap hidden from the map, DTC passed where the model-based controllers failed.

What Pupper can see, and with what

The head camera, computed from the official v3.5 CAD and the MJX model:

These figures assume an equidistant lens with the listed 0.76 mm focal length. Upstream's double-sphere calibration of the Pupper camera implies 0.79–0.85 mm, which would make the resolution 4–11 % finer. The near limit depends on where the lens's optical centre sits; a ruler under the nose of the real robot settles it.

Three consequences follow.

1. Resolution is not the limit for the front feet.
2. Depth is missing.
3. Placing the hind feet needs at least 0.6 s of terrain memory. The official-size policy's 20-step history covers 0.4 s, and Brax PPO has no recurrent policy.

Removing the GRU memory from START's depth policy cut real stepping-stone success from 5 of 5 to 1 of 5. StairMaster's authors write that narrow treads "disappear from the sensor's field of view prior to the hind legs engaging with them".

Sensor options:

The passive stereo cameras (D405, Gemini 305) were specified on textured targets. Single-colour printed treads may leave holes in the depth image, which a printed pattern on the course or a camera with a projector avoids.

On the computer:

• elevation_mapping_cupy needs CUDA.
• A CPU elevation mapper reports about 10 ms per 30,000-point scan on a Jetson Orin CPU (FastDEM, self-reported).
• Nothing has been measured on a Raspberry Pi 5.
• Binning one 48 × 48 depth frame (2,304 points) into a scan of a few hundred cells is a small workload, not measured.
• Upstream Pupper has no estimator of base position, so a map carried by odometry needs a new estimator. One visual-inertial odometry implementation took 49 ms per frame on one Raspberry Pi 4 core.
• The ROS 2 sensor packages are released as Ubuntu binaries and would have to be built from source on Pupper's Raspberry Pi OS.
• The Hailo-8L's published rates (SCDepthV3 145 FPS, Depth Anything V2-Small 32.4 FPS) were measured on a PCIe Gen3 x4 host, and the Pi 5 offers one PCIe 2.0 lane.

For V1.0 at Go2 size the picture changes. A D435i or Gemini 335/336 (75–99 g, about 3 W) pitched down, as QuadPiPS pitched its Go2 camera 45°, plus an Orin-class module is the published template. A LiDAR's ±2 cm is about an eighth of a 15–16 cm step, and Go2 itself carries the L2.

Camera-only routes

Work that uses cameras and no depth sensor or LiDAR falls into three groups. Entries marked "earlier note" are detailed in notes/research_notes/Multi terrain quadruped policy training/perceptive_policies.md, section 3.

One RGB camera, with the policy placing the feet implicitly:

• Loquercio, Kumar and Malik, ICRA 2023 (arXiv:2211.03785; earlier note).
  • Setup: a Unitree A1 with a monocular RGB camera. A blind policy was trained in simulation. A visual module was trained on the robot, labelled with the terrain the robot later felt through proprioception (cross-modal supervision), so no RGB simulation was needed.
  • Results: stairs up to 19 cm and curbs up to 20 cm. Success rose from 40 % to 100 % with under 30 minutes of real-world data.
• LucidSim, CoRL 2024 (arXiv:2411.00083; earlier note).
  • Setup: a Go1 with a budget webcam, trained on images generated from the simulator's geometry, running on a Jetson AGX Orin.
  • Results: real stairs 10 of 10, hurdles 73–87 %.
• GaussGym (arXiv:2510.15352; venue not found; earlier note). It renders Gaussian splats inside IsaacGym and trains a policy over DINOv2 features with an LSTM; the paper reports "initial zero-shot transfer" to real stair climbing.
• VR-Robo (RA-L 2025) and QuadVerse (2026) use RGB for navigation over a velocity policy, not for foot placement (earlier note).

Two cameras (passive stereo), with explicit footholds or gait adaptation:

• LittleDog (Kolter, Kim and Ng, ICRA 2009), described in the first section above.
• HyQ (Bazeille et al., "Quadruped robot trotting over irregular terrain assisted by stereo-vision", Intelligent Service Robotics 7(2):67–77, 2014; DOI 10.1007/s11370-014-0147-9, checked in Crossref). According to the abstract as summarised by a search engine, the full text not read, the robot used stereo and inertial sensing: stereo sized obstacles to gauge roughness, which set step height and speed, and images steered it toward a visual target. Footholds were not chosen explicitly, and the abstract gives no numbers.
• Atlas (Fallon, Marion et al., "Continuous humanoid locomotion over uneven terrain using stereo fusion", Humanoids 2015; DOI 10.1109/HUMANOIDS.2015.7363465, checked in Crossref). A dense terrain map built from a passive stereo camera fed a footstep planner, and the robot walked a DARPA Robotics Challenge-style course without stopping; the abstract gives no trial counts.

One wide-angle camera, perception only:

• OneOcc (CVPR 2026 per the arXiv comment; arXiv:2511.03571) predicts semantic occupancy around a quadruped from a single panoramic camera, with a module that corrects for gait motion. It reports perception benchmarks and no locomotion control.

What this means for Pupper:

• No work uses a fisheye camera to place feet.
• All the monocular results ran on 12–15 kg robots with Jetson-class computers. Whether their image encoders reach a useful rate on a Raspberry Pi 5 with the Hailo-8L is unmeasured.
• Loquercio's route fits Pupper best on paper. Its labels come from the robot's own later contacts, and Shappy's blind policy already finds risers by contact. Its cost is walking time on the real robot and a training loop beside it.
• A second IMX296 on the Pi 5's other camera connector would give LittleDog's stereo route a modern form: about 1.4–5.6 mm of depth error at 0.3 m in theory with a 50 mm baseline. Matching speed on the Pi 5 is unmeasured, and the scene needs texture.

Why one camera is rarely used to place feet, read from the sources above:

• Footholds need metric geometry to within about the foot band, and one camera gives relative depth.
  • The depth networks in the Hailo Model Zoo output unitless depth.
  • The best zero-shot metric models reach AbsRel 6–9 % on NYUv2, about 2–3 cm at 0.3 m.
  • Models trained on perspective images lose most of their accuracy on fisheye images: UniDepth δ1 0.364, against 0.852 for Depth Any Camera.
  • A D405 specifies ≤ 1 % RMS within 0.5 m.
• On the robots studied, a depth camera is a small addition: 0.5–2.3 % of robot mass on Solo-12, Mini Cheetah, A1, Lite3 and Go2.
• RGB is hard to simulate. Loquercio et al. write: "Since simulating RGB is hard, we necessarily have to learn vision in the real world." LucidSim needed a generative image model; trained with texture randomization alone, its policy climbed real stairs in 5 of 10 trials, against 10 of 10.
• In the one same-robot comparison (LucidSim on Go1, real trials), depth matched RGB on geometry and RGB won on recognising objects:

• (Inference) Explicit foothold work comes from model-based groups that build metric maps from LiDAR or depth, while RGB work is end-to-end and places feet implicitly. Choosing foothold coordinates from one camera falls between the two.

Reasons that apply to Pupper and not to larger robots (inference):

• The camera and the Hailo-8L are already on the robot. A D405 adds about US$272 (reseller price, unverified) to a US$2,480 kit, plus a USB 3 device and a driver built from source on Raspberry Pi OS.
• The VLA layer already reads RGB from this camera. A locomotion policy that consumes the same image would join the two layers through one sensor; this is untested anywhere.
• A 3 kg robot can collect on-robot self-supervised data, Loquercio's route, more cheaply and safely than a 12–50 kg robot.
• The fisheye sees the ground from 7 cm ahead of the front feet and to the sides. A nose-mounted D405 at 60° covers 71–78 % of the next front footholds.

None of these reasons improves the geometry. A camera-only study on Pupper therefore asks how much of a perfect height scan's gain the existing camera recovers, compared with a D405. Simulation gives the first estimate before any hardware work:

• the scan actor sets the upper bound;
• the same actor trained with monocular-like errors added to its scan (a per-frame scale error of 6–9 % and smooth distortions) gives the camera's expected share.

What it takes in Shappy's pipeline

The simulator and trainer already support the cheapest route.

Height scan:

• Shappy's analytic height lookup matches MuJoCo's surface to 0.0003 mm. It beats every ray option: in the pinned MJX-JAX 3.15, mjx.ray has no heightfield intersection.
• The current 15 × 7 scan at 3 cm stops 1.15 cm inside the feet's lateral track (feet at ±10.15 cm) and does not reach the hind feet's swing.
• The tooling note suggests a 2 cm grid from −0.20 to +0.36 m and ±0.14 m: 435 points, 25 % more first-layer weights.

Brax 6d03937:

• A noisy scan in the actor, with the privileged critic kept, needs environment code only.
• Distilling into a feed-forward student is built in (online DAgger).
• Recurrent students and estimators trained alongside PPO, as in SoloParkour, PIE and PUMA, would need new agent code.

Depth rendering:

• MuJoCo Warp's batch renderer handles heightfields. A merged pull request reports 2.22 M frames/s at 64 × 64 for 4,096 worlds on an unnamed GPU, against about 60,000 frames/s that 10 Hz depth would need.
• The cost lies in the image network: MuJoCo Playground trained PPO on pixels at 31.3k steps/s against 403k for stepping and rendering alone.
• The 222° fisheye cannot be rendered, so a student would train on a pinhole crop.

Problems found in the code:

• --actor privileged_state mixes 21 non-terrain values into the actor and would not restore a 720-input policy, so isolating terrain information needs a terrain-only actor key.
• The box course in eval_stairs.py has no height map, so a scan would read flat ground there.
• Brax's automatic reset reuses a cached first observation, which gives one stale scan per episode.
• On the robot, neural_controller hard-codes 36 inputs per step. That must change anyway, since it already rejects the 46-per-step behaviour policies.

Evaluation should count events per touchdown, not only crossings: touchdowns within 2 cm of an edge, riser contacts, and missteps. Identical 100-trial runs differ by up to 10–15 passes on one lane, which is what binomial sampling predicts. Telling 79 % from 89 % takes about 208 trials per arm, and 79 % from 84 % about 943 (computed).

What to do for Shappy

The position, in order:

1. A separating test in simulation, first; it needs no hardware.
   • Arms, from one start (own3_h1 with its first layer widened by zero rows) and one field, seed set and step budget:
     • the blind twin;
     • the actor with a terrain-only, base-relative height scan;
     • the same plus a riser-distance edge term gated to stair tiles.
   • Control: on stair lanes, run the scan actor with a flat lane's scan substituted. If its passes fall to the blind twin's, the gain came from the scan.
   • Field: it must contain the failing geometry (4–6 cm rises on 70–120 mm treads, box fields) at a share the policy can still win. Earlier runs on fields mostly beyond the policy's reach (stairs_v3, stairs_v4, ft_r8, cur_v1, own4_v1) learned to stop at the first riser.
   • Evaluation:
     • stairs of 4–6 cm on 80, 100 and 130 mm treads;
     • 4–6 cm box fields in three layouts, with a height function for the box course;
     • new lanes where a blind gait fails by construction: stones of 8–16 cm with 4–8 cm gaps, and trenches of 4–11 cm.
     • At least about 200 trials per lane, with per-touchdown edge and riser counts. At that size a difference of about 10 points near 80 % is resolved.
2. If the scan helps: a perception model in the same pipeline, then the sensor.
   • The model: restrict the scan to what the head camera or a D405 at 45–60° sees, add memory for the hind feet, and add noise and latency at measured or datasheet values.
   • Whether the gain survives this decides whether a sensor is worth mounting.
   • The default sensor is a D405 on the nose, with a printed texture on the course.
   • The fisheye with Hailo-8L monocular depth is a separate research question. It can be checked on the assembled robot against the printed course, whose geometry is known from its CAD: can riser heights of 3–5 cm be measured to ±1–2 cm at 0.2–0.4 m, with scale taken from the known camera height?
3. Explicit foothold targets only for sparse terrain or a footstep interface. Two routes fit:
   • A GPU target generator inside the jitted Brax step: footholds snapped to precomputed safe cells, a DTC-style touchdown reward and the gait clock behaviors.py already has. A CPU optimiser in the loop does not fit a jitted environment at about 300k steps/s.
   • A Taouil-style planner over the current velocity policy, which needs no retraining.
4. V1.0. Reserve a front depth-camera mount pitched down (QuadPiPS used 45° on its Go2), power for about 3 W, and a slot for an Orin-class module. A LiDAR, if one is wanted for mapping and navigation, belongs on V1.0.

What each check would settle:

As research, the explicit pipeline itself would not be new; planners and trackers exist from 2.5 kg LittleDog to 90 kg HyQ. Three questions are unmeasured at Pupper's size:

• How placement error and the value of perception scale down to a 3 kg robot whose foot is half a tread. A blind, scan and real-sensor comparison on Pupper, in simulation and then on the printed course, would answer it.
• Whether a monocular fisheye on Pi-class hardware can carry foothold perception. No legged robot has tried.
• Whether footstep commands serve a VLA better than velocity commands. The evidence so far is blind and from simulation.

Conclusion

A robot that sees the ground, picks where each foot goes and puts it there describes 15 years of quadruped work. That work leaves a clear rule: explicit foothold selection pays where footholds are sparse, and on stairs a policy that sees the terrain heights does as well without a planner. Short treads add a second rule: knowing where to step does not tell the leg how to clear a riser that close. Pupper sits where perception should matter: its blind policy fails at 5 cm on 80 mm treads and passes on longer ones, and other robots gained most from maps at these relative heights. It also sits where foot placement is hardest. Its foot is half a tread, so every centimetre of mapping, odometry, latency and tracking error counts. Its camera sees the next front footholds sharply but gives no depth, and the hind feet land on ground it saw 0.6 s earlier.

The project already has what the deciding experiment needs: an exact height scan in the critic, a Brax pipeline that can pass it to the actor, and a test course with the failing geometry. Running that comparison first costs a few hours of GPU time. It tells whether a sensor is worth mounting before any sensor is bought, and it is the result a later paper on perception at small scale would start from.

Source notes, scripts and their outputs: notes/research_notes/Perceptive foothold selection/ in the Shappy repository.