agent desktop
- Repo stars 3,082
- Author repo grok-cli
- Domain
- AI
- Author-declared agents
- No explicit declaration found; this is not inferred or tested compatibility
- Static check
- 88 / 100 · heuristic scan, not runtime safety proof
- Runtime evidence
- Not run; no isolated runtime test evidence is currently recorded
- Author / version / license
- @superagent-ai · no license declared
- Token usage
- Lean
- Setup complexity
- Guided setup
- External API key
- Not required
- Operating systems
- macOS
- Runtime requirements
- No explicit requirements detected
- Detected file/system behavior
-
- Read-only
- Shell exec
- Network behavior
- Local-only
- Install commands
- None (reference only)
Heads up: Tool permissions are not declared; actual access depends on the runtime.
# agent-desktop
Use this skill when the task involves the host macOS desktop rather than repository files, shell output, or browser-only verification.
This project uses `agent-desktop` on the host. Prefer the built-in `Computer` sub-agent for these tasks instead of trying to drive the desktop with plain shell commands.
## When to use it
- The user wants to inspect or interact with a native macOS application.
- The task needs app launch, window focus, accessibility snapshots, semantic element refs, clicks, typing, keypresses, or scrolling.
- A browser-only workflow is not enough, or the target is not a web page.
## Requirements
- `agent-desktop` must be installed and its native binary available.
- On macOS, the terminal app running `grok` needs Accessibility permission.
- Refs from `computer_snapshot` are only valid until the next snapshot.
## Preferred flow
1. Delegate to `task` with `agent: "computer"` unless the current agent already has the `computer_*` tools and the task is tiny.
2. Use `computer_launch`, `computer_list_windows`, or `computer_focus_window` to get the target app/window ready.
3. Start with `computer_snapshot`.
4. Pick one ref-based action.
5. After any UI transition, snapshot again before reusing refs.
## Tool guidance
- `computer_snapshot`: primary observation tool; prefer `interactive_only`.
- `computer_click`: use `ref` values from the latest snapshot whenever possible.
- `computer_type`: requires a target ref and is preferred over coordinate typing.
- `computer_press`: use for shortcuts and special keys such as `Enter`, `Tab`, `cmd+space`, or `cmd+k`.
- `computer_scroll`: pass an element ref, not raw coordinates.
- `computer_get`: read text, value, bounds, role, or states from a ref.
- `computer_screenshot`: use for visual confirmation or when the accessibility tree is not enough.
## Reliability rules
- Use a snapshot -> act -> snapshot loop.
- Prefer refs over coordinates.
- Refs go stale after the next snapshot; do not reuse them blindly.
- Keep actions reversible and low risk unless the user explicitly asked for something destructive.
- If the accessibility tree is poor or ambiguous, use `computer_screenshot` for confirmation and explain the limitation.
## Blockers
Stop and report clearly if:
- Accessibility permission is missing.
- `agent-desktop` is unavailable or its native binary is missing.
- The target app/window cannot be found.
- Refs are stale or ambiguous after a UI transition.
- The requested action could be destructive and the user did not explicitly ask for it.
Sections and files
Listed in source order.
sections -> When to use it → Requirements → Preferred flow → Tool guidance → Reliability rules → Blockers
terms -> Use this skill when the task involves the host macOS desktop rather than repository files, shell output, or browser-only verification. · This project uses agent-desktop on the host. · - The user wants to inspect or interact with a native macOS application. · - agent-desktop must be installed and its native binary available. · 1. Delegate to task with agent: "computer" unless the current agent already has the computer tools and the task is tiny. · - computersnapshot: primary observation tool; prefer interactiveonly. · - Use a snapshot -> act -> snapshot loop. · - Accessibility permission is missing.
files/cmd -> agent-desktop · Computer · grok · computer_snapshot · task · agent: "computer" · computer_* · computer_launch
body sha256 -> 7d473211e896
When to use it
The user wants to inspect or interact with a native macOS application. The task needs app launch, window focus, accessibility snapshots, semantic element refs, clicks, typing, keypresses, or scrolling. A browser-only workflow is not enough, or the target is not a web page.
Requirements
agent-desktopmust be installed and its native binary available. On macOS, the terminal app runninggrokneeds Accessibility permission. Refs fromcomputer_snapshotare only valid until the next snapshot.Preferred flow
Delegate to
taskwithagent: "computer"unless the current agent already has thecomputer_*tools and the task is tiny. Usecomputer_launch,computer_list_windows, orcomputer_focus_windowto get the target app/window ready. Start withcomputer_snapshot. Pick one ref-based action. After any UI transition, snapshot again before reusing refs.Tool guidance
computer_snapshot: primary observation tool; preferinteractive_only.computer_click: userefvalues from the latest snapshot whenever possible.computer_type: requires a target ref and is preferred over coordinate typing.computer_press: use for shortcuts and special keys such asEnter,Tab,cmd+space, orcmd+k.computer_scroll: pass an element ref, not raw coordinates.Reliability rules
Use a snapshot -> act -> snapshot loop. Prefer refs over coordinates. Refs go stale after the next snapshot; do not reuse them blindly. Keep actions reversible and low risk unless the user explicitly asked for something destructive. If the accessibility tree is poor or ambiguous, use
computer_screenshotfor confirmation and explain the limitation.