How GUI Agents Receive Instructions: An Exploratory Mapping of 85 Benchmark Families
Task Specification, Video Demonstrations and Evaluation Methods
Ankur Shrestha, Vertov, United States. 14 November 2026.
How benchmarks for GUI and computer-use agents give the agent its task, and where video and recorded demonstrations fit. Coded with AI assistance; no independent human recoding was performed.
- 85
- benchmark families read in full (147 components)
- 80 of 81
- video-targeted components give the goal as text (64 of 66 in the random comparison group)
- 9 of 81
- video-targeted components include a video or demonstration in the instruction (1 of 66 in the comparison group)
Preprint, released with its data under CC BY 4.0.