Read app structure as JSON
Snapshots expose roles, labels, windows, and actionable references through native accessibility APIs. This gives the caller a structured view of supported application controls.
Agent Desktop
Agent Desktop exposes desktop apps through operating-system accessibility trees. Explore Vercel Labs’ CLI and C library to design observe–act–verify workflows, handle stale element references, and check platform permissions before automating a task.
Make a desktop step observable and verifiable
Name the application, intended record, and fields or controls the task may affect. Prepare a desktop-workflow request that observes current targets and checks saved results, with runtime integration, platform permissions, and final approval kept explicit before automation proceeds.
Name the app, starting state, expected file or result, and permission constraints.
Copy the request into a new Floatboat conversation and provide the required material.
Download FloatboatYour request for Floatboat
Preview and copy the request. It is not sent automatically from this page.
Reference: Agent Desktop https://github.com/vercel-labs/agent-desktop Read the original documentation and check the current environment and dependencies. Propose any new installation or tool connection before proceeding. My task: Read Agent Desktop’s installation and snapshot documentation. For my monthly invoice-export task, map the application states, required permissions, snapshot references, and export verification. Start with a dry-run plan; do not grant permissions or operate the application yet. Expected deliverable: An observe/act/verify sequence, including stop conditions and evidence for each completed step.
About Vercel Labs’ Agent Desktop
Agent Desktop is an open-source desktop automation tool from Vercel Labs. Its CLI and C library expose app structure through operating-system accessibility APIs, returning JSON snapshots and references to actionable elements. The calling agent chooses the next step; Agent Desktop supplies observation and control rather than its own planning model.
The core interaction is to inspect a snapshot, act on an element reference, and inspect the result. References belong to their snapshot, so an outdated or ambiguous target calls for renewed observation. Accessibility actions are headless by default; enabling physical input is a separate choice, and exact capabilities differ across operating systems.
For workflows organized in Floatboat, the project provides a reference for desktop steps that need observable targets and explicit verification. Examples include entering prepared data into an application and checking the resulting state. Reusing the tool requires a compatible integration and the necessary permissions; a directory listing does not connect it to Floatboat.
Observe, Act, and Verify Desktop State
Agent Desktop exposes application structure, scoped targets, and action results as different evidence. Review how JSON snapshots and element references support control, then distinguish a successful command from confirmation that the intended state was actually saved in the application.
Act on scoped element references
Clicks and value changes target references from an observed snapshot. Stale or ambiguous references require another snapshot instead of silently retargeting an action.
Inspect errors before retrying
Command envelopes describe success or a structured error. The documented retry disposition helps the caller distinguish a safe retry from an action that needs investigation.
Use Cases for Agent Desktop
Application entry, changing lists, and runtime adapters need current desktop evidence. Explore Agent Desktop's control model through target localization and saved-value checks, while keeping approval and final verification in the workflow that surrounds each call.
Transfer reviewed information into an app
Transferring information into an application starts with validated source data and a defined set of permitted fields. Locate the actual controls before entering values, then read back what was saved. Stop when the values or target do not match the intended record instead of continuing through an uncertain desktop state.
Recover after a list changes
A changed list can invalidate an earlier row position or control reference. If an item disappears or the target becomes ambiguous, take a fresh snapshot and locate it again using the intended row's identifying criteria. Retain the new action evidence instead of relying on stale coordinates.
Embed control in an existing runtime
An agent runtime that already owns planning and task context can compare CLI calls with the C library as desktop-control interfaces. Map the needed calls to existing permissions and runtime boundaries. Keep approval decisions and verification of the resulting desktop state in the surrounding workflow, regardless of the adapter chosen.
Diagnose an unsuccessful desktop action
A failed control call may reflect an inaccessible target, changed application state, or an unmet platform condition. Inspect the returned error and current observation before retrying, then compare the result with the target application's actual state. Keep any permission change or approval decision in the surrounding workflow instead of treating repeated calls as a substitute for understanding the failure.
Check Agent Desktop platform access
Desktop control depends on platform permissions, the target application's accessibility exposure, and the observation data collected. Check those conditions for the actual operating system and display environment before using screenshots, clipboard contents, or model-backed helpers in a task.
Confirm permissions and session access
macOS requires Accessibility permission. Windows control is constrained by process integrity level; Linux needs an accessible graphical session and the documented platform dependencies.
Check the exact app and display system
Validate the target app’s accessibility exposure. Linux X11 and Wayland differ for input, screenshots, and clipboard operations; read the current platform capability table.
Limit sensitive observation data
Snapshots, clipboard contents, screenshots, and traces can contain private material. Keep the collected evidence scoped to the task and check any optional model-backed helper separately.
Related Software Projects
Compare the workflows, operating requirements, and source licenses of these projects.
Agent Desktop FAQ
Is Agent Desktop an agent harness or a model?
It is a tool an agent can invoke, delivered as a CLI and C library. The surrounding agent still owns planning, model calls, approvals, and the decision to continue.
How does it differ from Agent Browser?
Agent Desktop targets desktop applications through OS accessibility APIs. Agent Browser focuses on browser automation. Choose the tool according to where the task’s controls actually live.
What happens when an element reference becomes stale?
References are scoped to snapshots. On a stale or ambiguous target, observe again and use a reference from the new snapshot rather than assuming the old reference still identifies the intended element.
Does headless mean it runs without a graphical environment?
No. Here it means default actions use accessibility APIs without taking focus or moving the pointer. Linux still needs the user’s graphical session, and physical-input capabilities have additional platform conditions.
Can all three operating systems perform the same actions?
No. The documentation lists platform-specific differences for mouse input, screenshots, clipboard access, notifications, and system surfaces. Check the current table for the actual environment.
Does a Skill installation include desktop permissions?
No. Instructions explain how an agent should use the tool. The executable, its platform helpers, OS permissions, and any host integration remain separate setup requirements.