OmniACT

OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web

Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, Ruslan Salakhutdinov

FIG. 01 See the screen → Ground the task → Write the script → Score against reference
Identifier
arXiv:2402.17553
Published
Feb 27, 2024
Authors
7
Type
Benchmark
Area
Practical evaluations

In one paragraph

Can an AI agent look at a screen and write the script that completes a task, from "play the next song" to sending an email? OmniACT is a benchmark of desktop and web tasks for answering that. The strongest baseline, GPT-4, reached only 15% of human proficiency.

of human proficiency reached by the strongest baseline (GPT-4)
15%
of human proficiency reached by the strongest baseline (GPT-4)
tasks span native apps and websites
desktop + web
tasks span native apps and websites
agents output runnable scripts, not descriptions
executable
agents output runnable scripts, not descriptions

FIG. 02: Mechanism

How it works

  1. See the screen

    The agent gets a screenshot of a desktop app or website, plus a natural-language task.

  2. Ground the task

    It has to find the right UI elements on screen, such as buttons, fields and menus.

  3. Write the script

    It outputs an executable action script (clicks, typing, hotkeys) that would complete the task.

  4. Score against reference

    The generated script is compared with a human-written reference, both for the action sequence and for where on screen each action lands.

FIG. 03: Interactive figure

You are the agent.

Read the screen, click through the task, and compare your generated script against the reference.

Loading interactive demo…

FIG. 04: Notes

The problem

For computers to work for us, agents have to do more than chat. They need to operate software the way people do: read the screen, find the button and act. Before OmniACT, most agent benchmarks covered only web pages or used simplified environments.

The approach

OmniACT pairs screenshots of real desktop applications and websites with natural-language tasks. The tasks range from one step (“Play the next song”) to several, such as composing and sending an email. The agent must generate an executable script that completes the task. Scoring compares the script with a reference, both for the actions taken and for where on screen they land.

What we found

Even the strongest baseline, GPT-4, reached only 15% of human proficiency at generating executable scripts. The gap shows how hard visually grounded computer control is, and it gives a clear target for multimodal agents. The work is a collaboration between WRITER and Carnegie Mellon University.

Reference

Cite this work
@misc{kapoor2024omniact,
  title={OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web},
  author={Raghav Kapoor and Yash Parag Butala and Melisa Russak and Jing Yu Koh and Kiran Kamble and Waseem AlShikh and Ruslan Salakhutdinov},
  year={2024},
  eprint={2402.17553},
  archivePrefix={arXiv},
  primaryClass={cs.AI}
}