In a loop: 1) perception - the agent receives a screenshot (and optionally an accessibility tree/DOM) and interprets it with a vision-language model; 2) reasoning/planning - it decides the next step toward the goal; 3) action - it performs a GUI operation (click at specific coordinates, type text, scroll, shortcuts); 4) observe the effect and repeat until the task is complete. The agent often runs in an isolated environment (VM/browser), can use the user's logged-in accounts, and combines GUI control with tool/API calls where available.
Automating computer tasks usually required API integrations or brittle RPA scripts that break when interfaces change. A CUA operates at the GUI layer like a human, so it can handle apps without APIs and adapt to interface changes.
Analyzing screenshots (and optionally DOM/accessibility tree) with a vision-language model to understand the interface.
Deciding the next steps toward the goal based on the screen state.
The action set: click (coordinates), typing, scrolling, keyboard shortcuts.
Official
An isolated environment with a browser and file system, often with access to user accounts.
Official
A CUA acts with the user's privileges (logged-in accounts), so mistakes or hijacking can cause real harm.
Malicious page content can try to hijack the agent and induce unwanted actions.
Multi-step GUI tasks are prone to compounding errors; a single wrong click can derail the whole flow.
In October 2024 Anthropic released Computer Use, in which Claude controls a computer via screenshots and GUI actions - a breakthrough in popularizing CUAs.
In January 2025 OpenAI introduced the Operator agent based on a CUA model, performing browser tasks; Google was developing Project Mariner.
Specialized, commercial CUAs such as Hark Handoff emerged, competing on benchmarks (Online-Mind2Web) and lowering cost per token.
Resolution of the virtual screen and screenshots fed to the model. Anthropic recommends XGA (1024x768), configured via WIDTH/HEIGHT env vars; OpenAI uses display_width/display_height. Too high a resolution degrades click accuracy.
The environment the agent operates in (e.g. a browser or an operating system). In the OpenAI API the environment parameter accepts values such as browser/mac/windows/ubuntu.
The iteration budget of the observe-decide-act loop per task. It caps cost and the risk of looping, at the expense of completing long tasks.
The observe-decide-act loop is inherently sequential: each GUI action depends on the screen state after the previous action, so steps within a single task cannot be parallelized (only multiple independent tasks/agents can run in parallel).
A CUA is an orchestration layer (an observe-decide-act loop) on top of a vision-language model; the agent logic itself is hardware-independent, while compute requirements sit with the underlying model (GPU/TPU).
