Vertov Research
How GUI Agents Receive Instructions: An Exploratory Mapping of 85 Benchmark Families
Task Specification, Video Demonstrations and Evaluation Methods
- Author
- Date
- 14 November 2026
- Type
- Preprint, summarized on this page
- DOI
- 10.5281/zenodo.24217329
- License
- CC BY 4.0
Abstract
Benchmarks for GUI and computer-use agents tell the agent what to do in different ways, and some now supply video or recorded demonstrations. We separate three constructs that descriptions of benchmarks often mix: how the goal is specified, what the agent observes while acting, and how completion is judged and what evidence can be inspected afterwards. We coded them for a declared, non-exhaustive frame of 101 candidate benchmarks: 61 whose abstracts mention video, demonstration or recording (stratum V) and a seeded random sample of 40 others (stratum C). Eighty-five benchmark families (147 components) could be read in full and are analyzed, each positive value tied to a verbatim quote, a location and a document hash (checked absences and not-applicable cases are recorded without a quote). Coding was done by AI agents, checked mechanically against the stored sources and audited by a second AI pass; no independent human recoding was performed, and the study was not preregistered. In both strata the goal is given as natural-language text in almost every component (80 of 81 in V, 64 of 66 in C). A video, recording or demonstration trajectory is part of the specification in 9 of 81 V components (8 families) and 1 of 66 C components, together with text in all but one, the primary condition of VideoGUI's high-level planning track, whose coding as a recording-only specification rests on one judgment about an outcome preview presented with a generic question wrapper. More V components are coded as using video as authoring input (18 of 81, some thinly evidenced) than as a specification channel. The contribution is a coding scheme, a quote-evidenced public registry and a reproducible pipeline for checking how benchmarks specify tasks; counts describe this sample, not the field.
Keywords: GUI agents; computer-use agents; benchmarks; task specification; video; demonstration; meta-research.
- 85
- benchmark families read in full, 147 components
- 80 of 81
- video-targeted components give the goal as text (64 of 66 in the random comparison group)
- 9 of 81
- video-targeted components put a video or demonstration in the instruction (1 of 66 in the comparison group)
- 18 of 81
- video-targeted components use a video only as authoring input, which the agent never receives (0 of 66 in the comparison group)
What the study found
The task is almost always told in words
Of the 81 components in the video-targeted group, 80 give the goal as natural-language text. So do 64 of the 66 in the random comparison group. The video-targeted group was chosen because the benchmarks' abstracts mention video, demonstration or recording, which makes it the place where a video instruction is most likely to turn up.
Channels that carry the task goal, by stratum
- V: video-targeted stratum
- C: random comparison stratum
Bars show the share of components, with the count k/n and the percentage beside each. Components without a determinable value: V 0 of 81, C 0 of 66.
A video in the instruction is rare, and usually comes with text
A video, recording or demonstration is part of the instruction in 9 of 81 video-targeted components (8 families) and in 1 of 66 comparison components. Text accompanies it in 9 of those 10.
The one without text is the primary condition of the high-level planning track in VideoGUI, where the agent gets a preview of the finished result and a generic question. Whether that wrapper counts as task text is a judgment call, and the study says so: read it as text and the count of recording-only instructions falls from 1 of 81 to 0 of 81. The paper reports the current coding and the alternatives side by side.
Where video does appear, it is mostly behind the scenes
More video-targeted components use a video as authoring input, material the benchmark's builders used to create tasks and that the agent never receives (18 of 81, in 12 families), than as part of the instruction (8 of 81, in 7 families). Some of the authoring-input codings rest on thin evidence, and the paper says which.
Role of any video or recording, by stratum
- V: video-targeted stratum
- C: random comparison stratum
Bars show the share of components, with the count k/n and the percentage beside each. Components without a determinable value: V 0 of 81, C 0 of 66.
42 of 81 video-targeted components use no video in any role, against 59 of 66 in the comparison group. Calling a benchmark a video benchmark says little about whether the agent is ever given a recording.
Video the agent sees while acting is a different thing
In 8 of 80 video-targeted components with a determinable value (and 1 of 66 comparison components) the agent is shown video as part of what it perceives while it works. Screenshots are far more common. Watching video while acting changes what the agent can see, not what it is asked to do, and the two sit in largely different benchmarks. That is why the study codes them as separate constructs.
What the agent observes while acting, by stratum
- V: video-targeted stratum
- C: random comparison stratum
Bars show the share of components, with the count k/n and the percentage beside each. Components without a determinable value: V 1 of 81, C 0 of 66.
Limits to keep in mind
- A sample, not the field. The frame is declared and not exhaustive. The video-targeted group was picked by keyword, which over-includes, and the comparison group is small. The counts describe the 85 families read and do not estimate how common any channel is across GUI-agent benchmarks.
- Not preregistered. Every result is exploratory, and the frame was defined after the screening and pilot results had been seen.
- AI-assisted coding. AI agents did the extraction and classification. No independent human recoding was performed, so the paper reports no agreement statistic between people.
- Judgment calls. What counts as a video, a demonstration or a preview is a coding decision, and a few borderline cases move a count by one or two, as the VideoGUI case shows.
- Unread sources. 12 of 101 candidates could not be read in full (4 coded from repositories only, 8 pending), mostly recent unpublished work.
- A preprint. This page summarizes a preprint. The PDF and the Zenodo record carry the full paper, with every table, the references and the appendices.
Publication details
The full paper, with every table, the reference list and the appendices, is the PDF above and the Zenodo record. This page is a summary of it and does not replace it. The study is a preprint.
- Manuscript (PDF)https://vertov.app/research/task-specification-in-gui-agent-benchmarks/shrestha-how-gui-agents-receive-instructions.pdf
- Manuscript recordDOI 10.5281/zenodo.24217329
- Dataset recordDOI 10.5281/zenodo.24217767
- Dataset: registry, evidence register and audit filesDOI 10.5281/zenodo.24217767
- Codebook and manual (in the dataset record)https://doi.org/10.5281/zenodo.24217767
- Analysis code (in the dataset record)https://doi.org/10.5281/zenodo.24217767
Registry of 85 GUI and computer-use agent benchmark families, version run1-v0.2.1. 147 components in 85 benchmark families, each coded value with a verbatim quote, a location and a document hash. AI-coded and quote-checked; no independent human recoding was performed.
First posted 14 November 2026. Registry version run1-v0.2.1, codebook 0.4.0-run1. Errata: none recorded.
Source files and sha256 behind the numbers on this page
- 13-manuscript/manuscript-draft.md: 4ea97fa87f2c670396f2a9d1af0ae0cd9b7f99de80e834482d9c9844877df606
- T01_flow.csv: 2a114633086c7d33f1c40ce3c8d54890a86404c4f1deb89e44b185c13b701332
- T02_code_frequencies.csv: 885f993eccb66c3faf5e94930090fb9decd2ba2582faf481c954718971c65e37
- T03_derived_cells.csv: 78d72807c82624c13ebbc0513b5483caa9869c7cc8566b414a7ee967923f486e
- T04_video_demo_history_dialogue_components.csv: b5fac7a396da9963eb1ec108b0a0805685a2780e9d19dc9aaff9ad01bdabc474
- T05_evidence_status_by_variable.csv: 0b35e21732e45b6bb97cef1eeae9553f690d0872123628bf8fdb4d5c106b788a
- T06_quote_check.csv: c83a1320aca6a7ba3f09e39851753afe53207fe91d7766a62869b7ccd5a40dc7
- T07_video_role_vs_observation.csv: e2368ed665b40d1e16aab8d8f8c51ca00617abe6820e026f56ee276099825a27
- T09_stated_task_counts.csv: 0763fe8c819006aa15783d93bacd407454b5f0349187aaa18f1cfe602ebad781
- T10_run1_vs_pilot_ai_ai.csv: c82cb70a6f080453e144c86bb3c97c124dde4bf066f27b9b0c63a3a945a7942e
- T11_audit_verify.csv: 4948b4524f1b295d430f55722cb45611f6decd0b7e36597a32e09893308b068c
- T12_audit_blind_agreement.csv: cf869cbdd54c2832a3ce1df011c16eb6e3f8ab182ae61df5a1d4e29ad0f25175
- T13_sensitivity.csv: 003288f4aa703e4572692ef904ae5c3a4bf3764eaf600db143562a5056573fb8
- audit_disagreements.csv: 5b4798d128ba9d867f81aa91831fcf7945034eda5d76aefe0173dc4436f02627
- 16-public-registry/VERSION.md: 3e44263f051cf1e2246312d6985382900fb1f420733f5e39f212a0104b3eec4d
- 13-manuscript/figures/fig1-framework.svg: 1f4ab0443cdab7caae1acb908e0bd5b7c5a2e09115017f09988420ede52f00a7
- 13-manuscript/figures/fig2-selection.svg: 0fe6d250920306b6ed7ce7011ce0199abf0b3bca6fb6577d9b978cacc1ddce3a
- 13-manuscript/figures/og-a14.png: 619229a79e5531b8b1822de9267d86c6d3fe2770154c8348bfded4b38f872ac3
Disclosures
As stated in the paper.
- Competing interests. The author is affiliated with Vertov (https://vertov.app), a company developing screen-recording and AI-agent collaboration software. Vertov has a commercial interest in the subject of this research.
- Use of AI. Claude (Anthropic) was used to assist with literature research, data extraction and classification, analysis, verification checks and manuscript preparation. The methods and limitations of AI-assisted coding are described in Sections 3 and 6.
- Funding. This research was funded by Vertov. No external funding was received.
- Author contributions. Ankur Shrestha: Conceptualization, methodology, investigation, analysis, writing and project supervision.
Cite as
Shrestha, A. (2026). How GUI Agents Receive Instructions: An Exploratory Mapping of 85 Benchmark Families: Task Specification, Video Demonstrations and Evaluation Methods. Preprint. Vertov Research. https://doi.org/10.5281/zenodo.24217329. https://vertov.app/research/task-specification-in-gui-agent-benchmarks
@misc{shrestha_how_gui_agents_receive_instructions,
author = {Shrestha, Ankur},
title = {How GUI Agents Receive Instructions: An Exploratory Mapping of 85 Benchmark Families: Task Specification, Video Demonstrations and Evaluation Methods},
year = {2026},
note = {Preprint},
publisher = {Vertov Research},
doi = {10.5281/zenodo.24217329},
url = {https://vertov.app/research/task-specification-in-gui-agent-benchmarks}
}RIS
TY - GEN
AU - Shrestha, Ankur
TI - How GUI Agents Receive Instructions: An Exploratory Mapping of 85 Benchmark Families: Task Specification, Video Demonstrations and Evaluation Methods
PY - 2026
PB - Vertov Research
N1 - Preprint
DO - 10.5281/zenodo.24217329
UR - https://vertov.app/research/task-specification-in-gui-agent-benchmarks
ER -