THINKING EXPERIMENT

What does thinking cost?

Measured generation speed and recorded cup predictions. Native thinking produces text before the answer. The existing cup-evaluation results remain on their own page.

Speed benchmark

First-token time includes visual encoding and prompt processing. Per-token speed measures subsequent token delivery. Output limits include both reasoning and the final answer. Response time is the median actual duration; outputs may stop before the limit. “Final answers” requires a finished answer after thinking closes; ending inside the reasoning does not count.

Model	Thinking	Output limit	Trials	ms / token	tokens / s	First token	Response time	Final answers

Explore a recorded response

Final query only · no live GPU calls

ModelEvaluation setEpisodeThinkingRecorded request

Cross-model scores also reflect the different prompts and sampling settings shown below. “Exact format” uses the existing location parser: left, middle, right, or pick + location. “Final location” also accepts a final standalone label or explicit “Answer: left/middle/right” line after an explanation. Neither score guesses from reasoning text. Ambiguous four-cup middle answers and unfinished responses do not count as correct.

CameraOpen full evaluation ↗

FINAL PREDICTION

Full final answer · verbatim

Task / question given to the model

Generated reasoning · verbatim

Full generated output

Time per generated token

First token shown separately; hover a point for its text

Select a response to inspect its measured token timings.