Behaviour commands: a residual policy for the commands

In one shared network, holding the default walk suppressed some of the commands. A residual policy, the multi-terrain policy frozen plus a second network that acts only when a behaviour is commanded, keeps the default walk exactly and follows the commands; with the graded step reward its training stays stable.

• Default walk, 100 trials, 4 cm stairs / 20° slope / 7 cm curb: residual policy 85 / 93 / 59 (it is the multi-terrain policy exactly), shared network with commands 47 / 69 / 43
• Commands on flat ground, residual policy: high step 7.3 cm at an 8 cm command, 4 Hz gait command 4.07 Hz, pitch error 4.1° at 10°; the crouch not yet (16.6 cm at 13 cm)
• Falls during training 8 % with the graded step reward against 21 % with the clipped one; exploration noise at most 4.3 against 11.4
• 4 Hz gait on 3 cm stairs / 15° slope: 29 / 26 of 30 with the graded reward against 5 / 2 with the clipped one
• Holding the default walk in one shared network: at weight 1 the 4 Hz gait stays and the high step goes (5.5 cm), at weight 10 both go
• Correction: an evaluation fault had hidden the 4 Hz and pitch commands from these policies; all numbers here are from the fixed evaluation
• Trained longer (1 B steps): commands held on flat ground (swing 7.4 cm, 4 Hz gait 3.97 Hz, pitch error 3.7°); with commands, 3 cm stairs and 15° slope passed in 6 of 8 tests (pitch not yet); falls during training 4 %
• Body height command still short (15.9 cm at 13 cm); a stronger height term is running