Share a desktop through explicit targets and handovers

An assistant preparing a report should be able to read an authorised browser tab while its user writes in another application. That is a useful promise of computer-use agents: delegated work can proceed without occupying the whole desktop. The design has to distinguish the resource being addressed from the shared input devices needed to operate it.

OpenGCUA, a research report published on 15 October, explores agents that combine visual interaction with code and application interfaces. Its real-device design binds operations to explicit resources such as windows and tabs. It also distinguishes background delivery from operations requiring foreground input. (Technical report, sections 4.1–4.3)

For a workplace technology owner, the opportunity is parallel progress on a person’s machine. The architecture decision is when the agent can work against its own target, when it must borrow the pointer or keyboard, and what happens when the person changes the underlying work.

Keep the target independent of desktop focus

Consider a hypothetical analyst preparing an internal supplier comparison. The assistant reads approved supplier pages and prepares a draft workbook. The analyst continues writing meeting notes. Switching to the notes application should not redirect the assistant’s next observation or edit.

The proposed task record should identify the browser session, selected tab and destination workbook. A window title alone may be ambiguous when several documents have similar names. Bind an operation to a verified resource and retain that binding across the task. If the resource disappears, resolve it again from evidence rather than choosing whichever window is now active.

The SDK documentation describes separate accessibility and visual interfaces. An accessibility tree exposes application elements and their properties as structured data. A screenshot supplies pixels. These observations can complement each other, but an element reference and a screen coordinate are different kinds of target.

Choose the observation that answers the current question. Reading a specific tab may be sufficient to extract its displayed supplier terms. Inspecting the entire desktop could instead capture the analyst’s unrelated notes. If the task needs visual layout, request a view of the selected resource where supported. Keep the observation scope visible in the record so a reviewer can establish which application supplied the evidence.

Target binding has a limit. It identifies where an operation belongs. It does not freeze that application’s contents while the model reasons, or prove that a later edit reached the intended field.

Make foreground input a bounded handover

Reading a background window does not establish that typing into it will leave the foreground application untouched. The paper’s action-delivery discussion explicitly makes background operation dependent on the target and platform. Window-relative coordinates alone do not establish background execution.

For the hypothetical supplier task, use a supported application interface to prepare the workbook where possible. If a native dialogue requires foreground input, the service should explain the operation and request a bounded handover. The analyst needs to know which application will receive input, what will change and how to take control back.

A proposed operating sequence is:

1. Prepare. Finish work that can run against explicit resources without shared foreground input. Record the remaining operation and its target.
2. Acquire. Obtain the agreed handover, then observe the target again. Do not execute coordinates retained from an earlier desktop arrangement.
3. Act and check. Perform the bounded operation and inspect the application’s resulting state. A delivered keystroke is not a saved workbook.
4. Release. Return control and report the confirmed result, any partial change and the next unresolved step.

This sequence is a design proposal, not a protocol verified in OpenGCUA. It should be enforced by the trusted desktop service around the model. A timer alone is insufficient: after an interruption, the next action needs a renewed view of the target and confirmation that the handover still applies.

illustrative fixture

Separate target binding from control of the desktop

Swipe or scroll for the full diagram →

Choose the target and coordinate shared input separately.

Observed

OpenGCUA describes explicit resource bindings and platform-dependent foreground or background action delivery.

Inference

A desktop service should announce and bound any period in which an agent needs the shared pointer or keyboard, then refresh the target before continuing.

Evidence boundary

Whether the intended application can preserve user work, detect conflicting changes and recover correctly on the proposed device.

The user should be able to reclaim input promptly. Cancellation also needs a precise meaning. Stopping further keystrokes does not reverse an edit already applied or cancel an operation already accepted by the application. Preserve the partial state and reconcile it before resuming.

Background work can still conflict with the user’s work

An assistant can avoid moving the pointer and still overwrite a person’s change. In the hypothetical example, the analyst might correct a supplier name in the same workbook while the assistant prepares an update from an earlier copy. Both actors address the correct document, yet the later write can erase the correction.

Prefer a separate draft or an application mechanism that detects a changed version before accepting a write. Where the application exposes no reliable conflict check, use an agreed period of exclusive editing or return a proposed change for the analyst to apply. Background delivery solves input interference. The document’s editing mechanism must handle competing changes.

This is distinct from the archive’s external-effect stop conditions, which determine whether an action is permitted. A permitted edit can still collide with another permitted edit. It also differs from repairing an agent plan: the desktop can change even when the plan itself remains valid.

Trial the coordination before sharing a working machine

The released harness documentation marks confirmation gating as unfinished and advises using machines whose state can be lost. The release is a research preview. Its SDK coverage table also marks platform adapters as preview, with Windows planned. Do not infer deployable workplace controls from the proposed architecture.

The paper’s real-device table covers four macOS tasks using one selected run per model. It does not establish reliability under ordinary workplace interruptions. (Evaluation scope, section 4.3.3) No agent experiment was run for this article.

Use a disposable test environment and synthetic documents to assess the intended application route. Change focus during a background read, move or close the target, interrupt a foreground operation and edit the destination before the agent writes. Check target identity, preservation of the person’s work, partial-state reporting and successful resumption. Measure time spent waiting for the desktop alongside total completion time and review effort.

The workplace technology owner should decide whether shared-device operation offers enough useful progress to justify that coordination. If foreground access dominates or conflicts cannot be detected, choose a separate agent workspace or an explicit editing handover. Promise concurrent work only for the application paths that the service can demonstrate.