Claude Can Now See Your Screen and Operate the Mouse — "Computer Use" Launches in Beta
On October 22, 2024, Anthropic released “computer use” in API beta, a new capability that lets Claude view a computer screen, move the cursor, click, and type text. This was the first time a frontier AI model offered this kind of capability in public beta, and it was announced alongside an upgraded Claude 3.5 Sonnet and the new Claude 3.5 Haiku model.
Details
- Computer use (beta): Lets developers have Claude operate software the way a human would — viewing the screen while moving the cursor, clicking, and typing. Anthropic itself described it as “experimental — at times cumbersome and error-prone” at this stage
- Upgraded Claude 3.5 Sonnet: Pushed its score on the SWE-bench Verified coding benchmark from 33.4% to 49.0%, surpassing reasoning-focused models and specialized systems, while keeping the same price and speed as the previous Sonnet
- Claude 3.5 Haiku: Matches the speed of the previous-generation Haiku while delivering performance comparable to Claude 3 Opus on many benchmarks, scoring 40.6% on SWE-bench Verified
- Availability: Available immediately via the Anthropic API, Amazon Bedrock, and Google Cloud Vertex AI. Companies like Replit and Cognition were among the first to test it for automation tasks
- Caveats: Scrolling and dragging were still noted as unstable, and Anthropic recommended starting with lower-risk tasks
What happened next
This “computer use” feature later became the starting point for Claude directly operating browsers and terminals to autonomously complete tasks. It went on to serve as the foundational technology behind Anthropic’s agentic product lineup, including “Claude Code” in 2025.