Architecture and Shappy
Shappy's world model is one module of a larger architecture, and most of the others are not built yet
The goal set on 2026-10-28 is a robot with a persistent self that walks, talks with people and carries out varied tasks. LeCun's 2022 architecture for autonomous intelligence gives that goal a module list that can be checked one module at a time. Mapped onto Shappy, the walking policy covers the reactive part of the actor and the training reward covers part of the cost. The JEPA world model, the perception encoder, the memory and the link to the dialogue layer do not exist yet. The first step is collecting data the world model can learn from, and the data recorded so far cannot serve that purpose. The figure at the top puts the paper's Figure 2 on the left and Shappy's state for each module on the right, with the same colour for the same module.
The modules and what Shappy has
The paper's architecture has a configurator that configures all other modules for the task, a perception module that estimates the current state of the world, a world model that predicts future states from action sequences the actor proposes, a cost module made of an immutable intrinsic cost (pain, pleasure, hunger) and a trainable critic that predicts future intrinsic cost, a short-term memory that keeps the current and predicted world states with their intrinsic costs, and an actor that proposes action sequences and outputs the first action of the one with the lowest estimated future cost.
The figure at the top left is Figure 2 of Y. LeCun, "A Path Towards Autonomous Machine Intelligence", version 0.9.2, 2022 (OpenReview, id BZ5a1r-kVsf), reproduced for reference. The copyright is the author's; reuse terms were not checked, so the figure is not for use outside the project.
| Module | Role in the paper | Shappy counterpart | State on 2026-10-28 |
|---|---|---|---|
| Configurator | Sets the other modules for the current task | The dialogue and VLA layer that sends body commands to the walking policy | Designed by the team in the progress slides; not connected to the robot |
| Perception | Turns sensor readings into a state representation | A JEPA encoder small enough to run on the Pi 5, reading the camera, IMU and joint states | Not started; the robot has a person detector only |
| World model | Predicts the next state from the current state and an action | An action-conditioned predictor trained on the lab server skim, run on the Pi | Not started |
| Cost | A fixed intrinsic cost (what the agent avoids) and a critic that predicts future cost | The penalty terms of the walking reward (torque, joint limits, body tilt, falls); no critic | Reward terms exist in training; nothing on the robot computes them yet |
| Actor | Proposes actions, either by reaction (mode 1) or by planning through the world model (mode 2) | The walking policy at 50 Hz is mode 1; planning over speed commands would be mode 2 | Mode 1 exists; mode 2 not started |
| Short-term memory | Keeps the current and predicted world states and their intrinsic costs | A runtime buffer on the robot of recent states, predictions and costs. The recordings on skim are training data for the world model, a separate thing | Not started |
The walking policy stays as mode 1
The paper describes two perception-action modes. In mode 1 the actor computes an action directly from the perceived state, and "this reactive process does not make use of the world model nor of the cost"; in mode 2 the actor plans by imagining action sequences through the world model and picking the one with the lowest predicted cost. The architecture keeps the fast reactive policy and adds planning above it. Here that means the trained walking policy keeps running at 50 Hz, and the world model is used one level up, to choose speed commands by imagining their outcome. This matches the project's structure, where a high-level layer sends commands to the low-level policy, and it keeps the world model off the control loop that must not stall.
What the world model needs from the data
An action-conditioned JEPA learns to predict the next representation from the current one and the action taken, so the recording must pair what the camera saw with what the robot did, on one clock. The recordings made so far do not:
- the frames carry burned-in detection boxes and a timer, so the encoder would learn the overlays;
- they are 640 × 640 at 5 frames per second, with no action, joint, or IMU record next to them;
- most runs are one person being followed, a narrow range of behaviour.
The recording to build keeps the plain camera frames at about 10 frames per second, the policy's joint targets and the speed command at the control rate, joint and IMU states, and the policy file and run notes, all on one clock. Sessions should include driven runs that vary speed, turning and stopping, and some runs on slippery or obstructed ground so that surprises appear in the data. Frames stored as H.264 video and states as a timestamped table should come to roughly 1–2 GB per hour (an estimate, not a measurement); the Pi has 97 GB free, and the upload path to skim already exists. The cost terms are computed from the same states, except that motor current has never reached the Pi in the official software; whether that holds for the installed image has not been checked.
Order of work
- Build the recording and check on a short session that frames and actions line up in time.
- Pass that data through a pretrained JEPA encoder and see whether the representation of fisheye, shaking footage is usable. If it is not, the plan changes before more data is collected.
- Train the predictor on skim, then test it first as a surprise detector (does prediction error rise before a slip or a fall).
- Define the cost terms and a critic, then try planning over speed commands, then connect the dialogue layer.
Limits of this note
The module roles and the two modes were checked against the paper's text (version 0.9.2) on 2026-10-28. The cards on the right list the modules without Fig. 2's arrows, so the information flow has to be read from the paper's figure. The status column describes this repository and the robot as of that date; the robot was offline, so nothing here was measured on it.
Source notes, scripts and their outputs: notes/research_notes/Autonomous intelligence architecture and Shappy/ in the Shappy repository.