// CLAUDE AGENTS · DESKTOP AUTOMATION

Claude Cowork & Computer Use: OS Automation at Scale

Deploy autonomous desktop coworkers powered by Claude 5.5 Computer Use: inspecting user interfaces via screenshots, clicking native UI elements, filling forms across legacy ERPs, and automating workflows where APIs do not exist.

1. The Computer Use Paradigm

Anthropic's Computer Use API represents a breakthrough in digital process automation. Rather than requiring bespoke API integrations for every legacy desktop application, Claude directly interacts with graphical user interfaces (GUIs) just like a human knowledge worker:

sequenceDiagram
    autonumber
    participant Agent as Claude 5.5 Agent
    participant Bridge as Computer Use Bridge
    participant Desktop as OS / Virtual Desktop
    participant Human as Human Supervisor

    Agent->>Bridge: Action: take_screenshot()
    Bridge->>Desktop: Capture FrameBuffer
    Desktop-->>Bridge: Raw PNG Screen Data
    Bridge-->>Agent: Scaled Screen Image (1024x768)
    Agent->>Agent: Extended Thinking: Locate Submit Button at (540, 380)
    alt Action is Sensitive (e.g. Wire Transfer)
        Agent->>Human: Request Approval: Click Wire Transfer?
        Human-->>Agent: Approved
    end
    Agent->>Bridge: Action: mouse_click(x=540, y=380)
    Bridge->>Desktop: Dispatch Synthetic OS Click Event
    Desktop-->>Bridge: State Updated

2. Coordinate Scaling & Screen Normalization

Computer Use scales high-resolution displays down to standard aspect ratios (e.g., 1024x768 or 1280x800) to optimize vision tokens while retaining fine typographic details. The client library automatically normalizes synthetic click coordinates:

Step 1 · Frame Capture

Desktop Capture

Captures virtual framebuffer (X11 / Wayland / Windows Desktop) via PyAutoGUI or scrot.

Step 2 · Scaling

Aspect Normalization

Downscales image while preserving pixel aspect ratio to fit within Claude's vision context budget.

Step 3 · Coordinate Map

Inverse Projection

Maps Claude's predicted target (x, y) back to native monitor resolution coordinates before executing mouse input.

3. Tutorial: Python Desktop Automation Controller

Here is an architectural implementation of a safe, containerized desktop automation loop using the Anthropic computer tool declaration:

python/computer_use_controller.py

import base64
import os
from anthropic import Anthropic

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

# Official Computer Use tool declaration
computer_tool = {
    "type": "computer_20241022",
    "name": "computer",
    "display_width_px": 1024,
    "display_height_px": 768,
    "display_number": 1
}

def capture_screen_base64() -> str:
    """Simulate capturing a local desktop screenshot"""
    # In production, use pyautogui.screenshot() or Xvfb capture
    return "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="

def execute_desktop_action(action: str, coordinate=None, text=None):
    print(f"[OS EVENT]: Action={action}, Coordinate={coordinate}, Text={text}")
    # In production, dispatch via pyautogui: pyautogui.click(x=coordinate[0], y=coordinate[1])
    return capture_screen_base64()

print("Computer Use Controller ready for enterprise coworker tasks.")

1. The Computer Use Paradigm

2. Coordinate Scaling & Screen Normalization

3. Tutorial: Python Desktop Automation Controller

Deploy Production Claude Agent Systems