Glossary
What is computer use in AI?
Computer use is an AI capability where a model operates a computer as a person does: it looks at screenshots, then moves the mouse, clicks and types. Anthropic released computer use for Claude in October 2024, and OpenAI offers a similar agent. It lets agents work with any app, not only ones with APIs.
How it works
- The agent receives a screenshot.
- The model decides an action: click at a position, type text, scroll, press a key.
- The action runs; a new screenshot follows.
- Repeat until done or the agent needs a human.
Good and bad at
| Good | Harder |
|---|---|
| Apps with no API | Fast, precise tasks |
| Varied, unfamiliar interfaces | Long sequences without errors |
| Following visual cues | Tiny controls, dense screens |
Where an API or structured page access exists, it's usually faster and more reliable than pixels.
Safety
Anthropic recommends running computer use in a virtual machine or container with minimal privileges, avoiding access to sensitive accounts, and asking a human to confirm meaningful actions. Content on screen can contain injected instructions.
Example
An agent uses computer use to fill a desktop app's export dialog that has no API, then switches to a browser extension's structured page access to submit a web form, which is faster.
FAQ
Is computer use the same as RPA?
Similar goal; RPA follows scripted steps, computer use lets a model decide.
Is computer use safe?
Use sandboxes, limited permissions and human confirmations.
Which models support computer use?
Anthropic's Claude and OpenAI's computer-using agent, among others.