Computer Use: Let Astra Work Your Screen

The Architecture: Screenshot, Plan, Act

Computer use in GPT-6 Astra runs a perception-action loop. At each step, the model takes a screenshot of the current screen state, reasons about what it sees, decides on the next action — a click, keystroke, scroll, or text input — executes that action, then takes a new screenshot and repeats. The loop continues until the task is complete or the model determines it cannot proceed.

This design is called visual grounding: the model works from what it can see, not from an abstract representation of the interface. It does not use accessibility APIs or DOM inspection; it reads the screen as a user would, which means it handles any application — legacy software, non-web UIs, unfamiliar interfaces — without integration work on your part.

The key benchmark for visual grounding quality is ScreenSpot Pro, which measures a model's ability to locate a specific element in a professional application screenshot. Astra scores 92.7% on ScreenSpot Pro, compared with 76.9% for GPT-5.6 Sol. That 16-point improvement explains the efficiency gain: when the model misidentifies a button target, it executes the wrong action, has to recognize the error, and try again. Higher grounding accuracy means fewer corrective loops.

The overall effect on task completion time is significant: Astra averages 40 minutes per OSWorld 2.0 task, versus 75 minutes for GPT-5.6 Sol. The improvement is not primarily that Astra is faster per action — it is that Astra needs fewer corrective actions per task. It identifies UI elements correctly more often, executes the right action, and progresses rather than doubling back.

How to Enable It in ChatGPT

Computer use is available on ChatGPT Plus (via Work and Codex), Pro, Business, and Enterprise plans. Free-tier users do not have access. The specific access path depends on your plan:

• ChatGPT Plus — computer use is accessible through Work and Codex, not the standard Chat interface. Open ChatGPT, switch to Work, and select Astra as the model.
• ChatGPT Pro, Business, Enterprise — Astra with computer use is available in the standard Chat interface. Select Astra from the model picker.

Computer use requires the ChatGPT desktop app — it is not available in the browser. The desktop app needs two system permissions to function:

• Screen Recording — allows ChatGPT to capture screenshots of the current screen state
• Accessibility — allows ChatGPT to send click, keyboard, and scroll events to applications

On macOS, both permissions are granted in System Settings → Privacy and Security. The app will prompt for them automatically on first use.

To invoke computer use in a session, mention @Computer in your message, or reference a specific application directly — for example, "open Salesforce and update the deal stage for Acme Corp." Once a computer-use task is running, the ChatGPT interface shows a live activity log: each action the model takes is labeled — "clicked Login", "typed search query", "scrolled down" — so you can follow what it is doing in real time. A pause button is available throughout; you can interrupt and take over at any point.

Task Categories It Handles Well

Astra's computer use is most reliable in task categories with predictable UI patterns and clear success criteria. Six categories consistently perform well in practice:

The common thread across these categories is deterministic success criteria: either the form was submitted correctly, or it was not. Either the meeting was created with the right attendees, or it was not. Computer use performs worst on tasks where success is ambiguous, where the correct output requires domain judgment, or where the model cannot verify completion by reading the screen.

Safety Scope and Hard Refusals

GPT-6 Astra is the first model OpenAI has rated as reaching the Critical level under its Preparedness Framework for cybersecurity capability. That rating means that with the right tools and access, the model can independently discover previously unknown security flaws and develop working exploits across well-protected systems — without a human guiding each step.

This capability drives a set of hard refusals that apply unconditionally to computer use:

• Creating proof-of-concept exploits for any vulnerability, disclosed or otherwise
• Exploit discovery against systems the user does not have authorization to test
• Accessing or navigating restricted government or defense systems
• Any agentic task that triggers the cybersecurity safety check at the API level

In ChatGPT, a safety check interrupts the task outright — the model stops and declines. In the API, a cybersecurity safety check stops the agentic task rather than passing control to the developer. There is no mechanism to override these checks programmatically through standard plan access.

OpenAI runs a separate program called OpenAI Daybreak for organizations that need access to defensive security workflows — vulnerability validation, malware analysis, and detection engineering. This requires a separate application and is not available through standard Plus, Pro, Business, or Enterprise plans.

For ordinary computer use tasks, the Critical rating does not create additional restrictions. The hard refusals are narrowly scoped to offensive security work. General-purpose automation — form filling, research, CRM updates, scheduling — runs without safety-related interruptions.

When It Gets Stuck

Even at 72.6% on OSWorld 2.0, computer use fails on more than one in four tasks. Understanding the common failure modes helps you write prompts that avoid them, and recognize when to intervene rather than wait.

Unexpected popups and dialogs. A dialog that appears mid-task — a software update prompt, a privacy consent banner, a session timeout warning — breaks the expected sequence. Astra's default behavior is to attempt dismissal and continue; it usually succeeds with simple OK/Cancel dialogs. It fails on dialogs that require a judgment call it cannot make from the screen alone, such as an update that might break a dependency. Those cases produce a pause and a request asking what to do. The fix: give explicit instructions upfront — "dismiss any update prompts or notification banners without installing."

CAPTCHAs and bot detection. Astra will not attempt to solve visual CAPTCHAs. When one appears, the model stops, reports it, and waits. The practical fix is to use authenticated sessions or accounts that do not trigger CAPTCHA challenges on the network or device you are running from.

Authentication walls. Computer use cannot enter passwords on your behalf unless you provide them in the prompt. The model does not have access to your stored credentials. For tasks that cross authentication boundaries, either be already logged in before invoking computer use, or include the credentials explicitly in the prompt — with awareness that they will appear in your ChatGPT session history.

Ambiguous UI states. Some applications render inconsistently: a button appears grayed out but is clickable, a form looks submitted but has not been, a modal is visible but the model cannot determine its z-order. When Astra reads the screen and cannot determine the current state confidently, it pauses and describes what it sees. These pauses are useful signal — if the model is uncertain about an interface, the interface is doing something that is genuinely ambiguous. The prompt strategy is to give explicit recovery instructions: "if you see a loading spinner for more than 10 seconds, reload the page and try again."