Survey: do larger models and longer training help reinforcement learning as they help behaviour cloning?

Reinforcement learning grew with model size and training only when its task grew too (motion-capture breadth, many bodies, harder terrain sampling). On a fixed task, wider PPO networks destabilised and continuing a converged policy gained nothing, in the literature and in the multi-terrain walking policy (mt_c).

• Reported gains came with broader targets: a humanoid tracker grown from 1.2 M to 42 M parameters on up to 100 M motion frames lifted unseen-motion success from 98.0 % to 99.6 %, while its velocity-tracking baseline stopped improving beyond 8 GPUs; PPO on ANYmal collapsed as its MLP grew from 2.4 M to 80 M parameters
• In Shappy's runs mt_c's evaluation reward stayed at 16.8–17.2 from 100 M to 1 B steps. All eight continuations ended at or under their start (1.5 to 12.3 points under on the trained lanes; 41 under for an actor twice as wide at an unchanged step size, which collapsed)
• Keeping the critic and the curriculum levels cut a continuation's loss from 12 points to 1.5 but gained nothing. Read afterwards from the logs (not a registered reading): four of seven plain continuations had one to four updates after which more than half the training robots fell, against none for the state-kept run and the three narrow-bell runs
• The actor has slack on flat ground: halving the rank of hidden layers 2–4 cost 0–8 % of walking speed. The small-command dead zone follows the reward geometry (standing keeps 85 % of the tracking term at a 0.2 m/s command); only a narrow yaw bell with 20 % small commands moved the turn threshold (0.4 to 0.22 / 0.30 rad/s), at the cost of the forward and sideways responses
• Outside the policy the symptom is fixed in simulation: a minimum turn command of 0.45 rad/s, a pulsed command or a follower gain of 6 centred 48, 48 and 47 of 48 trials against 16 of 48 for the upstream follower. Registered and queued: replicate controls, two full retrains, a full-length wide actor at half the learning rate, a frozen-start state-kept continuation, and the three bells with small commands