NEW OAT + SUBTASK · ACTION LOSS WEIGHT 0.1 · 5K CHECKPOINT
After the swaps · fixed-frame check
Weight = the OAT action-loss multiplier during training. Total loss = subtask loss + weight × OAT action loss. The subtask-loss weight is 1 for every model.
New: weight 0.1 from the start. Continued: weight 1 through 4k, then weight 0.1 from 4k to 5k. VLM only: subtask training with no action loss.
Each answer uses the exact final hidden-cup frame and its video history, with no response deadline. The replay table below uses answers returned while the video keeps playing. These are separate evaluation methods.
| Model | 5 swaps · 30 demos | 6 swaps · 15 demos | 4 cups · 2 demos |
|---|---|---|---|
| New OAT + subtask · weight 0.1 · 5k | 28/30 · 93.3% | 11/15 · 73.3% | 0/2 · 0.0% |
| Original OAT + subtask · weight 1 · 4k | 20/30 · 66.7% | 6/15 · 40.0% | 0/2 · 0.0% |
| Continued OAT + subtask · weight 0.1 · 5k | 22/30 · 73.3% | 7/15 · 46.7% | 0/2 · 0.0% |
| VLM + subtask only · no action loss · 4k | 28/30 · 93.3% | 12/15 · 80.0% | 0/2 · 0.0% |
New 5k checkpoint: First 15: 14/15 · Next 15: 14/15. All 30 five-swap demos were excluded from training. The 4k references have fewer training steps than the new 5k model.
Offline action error on the 30 moving demos: 0.0638 using the model’s own subtask answer, versus 0.1170 for the earlier weight-0.1 continuation at 5k. With the correct subtask supplied, error is 0.0476 versus 0.0480. These are normalized mean squared errors, not robot success rates.
| Action MSE · lower is better | Correct subtask supplied | Model’s own subtask |
|---|---|---|
| Original OAT + subtask · weight 1 · 4k | 0.048091 | 0.115501 |
| Continued OAT + subtask · weight 0.1 · 5k | 0.048003 | 0.116964 |
| New OAT + subtask · weight 0.1 · 5k | 0.047577 | 0.063836 |
See all 47 final predictions and open recordings
| Set / test | Episode | Model answer | Ground truth | Result |
|---|---|---|---|---|
| 5 swaps · Test 01 | 242 | pick left | Cup 3 · left | Correct |
| 5 swaps · Test 02 | 226 | pick left | Cup 3 · left | Correct |
| 5 swaps · Test 03 | 210 | pick left | Cup 3 · left | Correct |
| 5 swaps · Test 04 | 194 | left | Cup 3 · left | Correct |
| 5 swaps · Test 05 | 177 | middle | Cup 2 · middle | Correct |
| 5 swaps · Test 06 | 161 | middle | Cup 2 · middle | Correct |
| 5 swaps · Test 07 | 145 | middle | Cup 2 · middle | Correct |
| 5 swaps · Test 08 | 129 | middle | Cup 2 · middle | Correct |
| 5 swaps · Test 09 | 112 | right | Cup 1 · right | Correct |
| 5 swaps · Test 10 | 96 | right | Cup 1 · right | Correct |
| 5 swaps · Test 11 | 80 | right | Cup 1 · right | Correct |
| 5 swaps · Test 12 | 64 | right | Cup 1 · right | Correct |
| 5 swaps · Test 13 | 47 | pick left | Cup 1 · right | Incorrect |
| 5 swaps · Test 14 | 31 | left | Cup 3 · left | Correct |
| 5 swaps · Test 15 | 13 | middle | Cup 2 · middle | Correct |
| 5 swaps · Test 16 | 235 | pick left | Cup 1 · right | Incorrect |
| 5 swaps · Test 17 | 218 | pick left | Cup 3 · left | Correct |
| 5 swaps · Test 18 | 202 | left | Cup 3 · left | Correct |
| 5 swaps · Test 19 | 186 | left | Cup 3 · left | Correct |
| 5 swaps · Test 20 | 169 | middle | Cup 2 · middle | Correct |
| 5 swaps · Test 21 | 153 | middle | Cup 2 · middle | Correct |
| 5 swaps · Test 22 | 137 | middle | Cup 2 · middle | Correct |
| 5 swaps · Test 23 | 121 | middle | Cup 2 · middle | Correct |
| 5 swaps · Test 24 | 103 | right | Cup 1 · right | Correct |
| 5 swaps · Test 25 | 88 | pick right | Cup 1 · right | Correct |
| 5 swaps · Test 26 | 72 | pick right | Cup 1 · right | Correct |
| 5 swaps · Test 27 | 56 | pick right | Cup 1 · right | Correct |
| 5 swaps · Test 28 | 39 | pick left | Cup 3 · left | Correct |
| 5 swaps · Test 29 | 22 | left | Cup 3 · left | Correct |
| 5 swaps · Test 30 | 6 | middle | Cup 2 · middle | Correct |
| 6 swaps · Test 01 | 0 | right | Cup 1 · right | Correct |
| 6 swaps · Test 02 | 1 | left | Cup 3 · left | Correct |
| 6 swaps · Test 03 | 2 | right | Cup 1 · right | Correct |
| 6 swaps · Test 04 | 3 | middle | Cup 2 · middle | Correct |
| 6 swaps · Test 05 | 4 | left | Cup 2 · middle | Incorrect |
| 6 swaps · Test 06 | 5 | left | Cup 3 · left | Correct |
| 6 swaps · Test 07 | 6 | pick middle | Cup 2 · middle | Correct |
| 6 swaps · Test 08 | 7 | pick middle | Cup 2 · middle | Correct |
| 6 swaps · Test 09 | 8 | pick right | Cup 1 · right | Correct |
| 6 swaps · Test 10 | 9 | pick left | Cup 3 · left | Correct |
| 6 swaps · Test 11 | 10 | pick right | Cup 2 · middle | Incorrect |
| 6 swaps · Test 12 | 11 | pick right | Cup 1 · right | Correct |
| 6 swaps · Test 13 | 12 | right | Cup 3 · left | Incorrect |
| 6 swaps · Test 14 | 13 | middle | Cup 2 · middle | Correct |
| 6 swaps · Test 15 | 14 | pick right | Cup 2 · middle | Incorrect |
| 4 cups · Test 01 | 15 | pick left | Cup 1 · rightmost | Incorrect |
| 4 cups · Test 02 | 16 | pick right | Cup 2 · second from right | Incorrect |
OFFLINE MODE 2 · CUP CHECKPOINTS · THINKING OFF
Round 1 and Round 2 replay the same demos independently at 1× speed. Each score uses answers returned by the final hidden-cup frame. Scores appear when a full round finishes; otherwise the table shows how many demos have been checked.
| Model | 5 swaps · 30 demos | 6 swaps · 15 demos | 4 cups · 2 demos |
|---|
Models keep their trained camera sampling. The older OAT checkpoints are marked _brian and use their original 30-frame history and 0/96 latent-slot settings. Offline mode 1 contains earlier independent-query checks. Four-cup exact scoring can reach at most 1/2 here: one target is an inner cup that left/middle/right cannot identify exactly.
CONTINUOUS REPLAY · TWO ROUNDS
Loading recordings…
BLOCK POSITION SEQUENCE
Raw model output
MODEL'S OVERHEAD HISTORY
0 framesThe model encodes video frames in pairs. A new answer can add two history slots or refresh frames already shown.
SEQUENCE
FINAL PREDICTION
PendingGROUND TRUTH