Skip to main content

Remote Computer Use

Use @assemblyline-agents/computer-use when the agent runs on Railway, Fly, Docker, or another host but must operate a specific Mac. Use @assemblyline-agents/peekaboo only when the Assembly Line Node runtime itself runs on that Mac.

The remote path has three independently auditable pieces:

  1. The agent uses a static transport: "relay" connection and a deployment- scoped binding.
  2. Assembly Line Computer Host, installed by Assembly Line Builder, runs Peekaboo on the Mac and maintains an outbound WebSocket.
  3. A Cloudflare Worker and one Durable Object per Mac route encrypted request and response envelopes.

Availability: Assembly Line Builder (the desktop app that installs Computer Host and manages pairing) and the openeve-computer-relay Workers project are distributed separately from this repository and are not part of the npm packages. This page documents the agent-side contract they plug into.

The middle service is needed because the hosted agent cannot normally open a connection into a Mac behind NAT, sleep/wake, or a consumer firewall. Both ends initiate outbound TLS connections to a stable public rendezvous point. Do not expose Peekaboo MCP, a local HTTP wrapper, or a desktop port directly to the internet.

Security Boundary​

Pairing creates a separate P-256 key pair and random bearer credential for each deployment binding. Assembly Line derives direction-specific AES-256-GCM keys with ECDH and HKDF. Tool arguments and results are encrypted between the agent runtime and Computer Host; the relay sees device and binding identifiers, message timing, and ciphertext sizes, but not plaintext tool data.

The relay stores SHA-256 token digests, rejects expired or replayed envelopes, caps payload and pending-request sizes, and supports immediate binding revocation. Computer Host additionally enforces an exact accessibility-first tool allowlist, refuses actions while the Mac is locked, and applies one of three local write policies:

  • always: show a native approval prompt for every write.
  • deny: allow inspection but reject all writes.
  • never: allow writes without a local prompt; reserve this for a deliberately provisioned, trusted Mac account.

The Assembly Line connection policy is a second gate. New connections start with write: false; use write: { approval: "always" } until the agent's authority is well understood.

The binding contains the agent-side private key and is a secret. Keep it in Assembly Line Builder's Keychain-backed secret store and deployment environment. Never put it in source, a prompt, a skill, tool arguments, logs, or relay config.

Set Up A Hosted Agent​

Add the plugin to the agent:

assembly-line add computer-use ./agent

The generated connection is explicit and read-only:

import { defineComputerUseConnection } from "@assemblyline-agents/computer-use";

export default defineComputerUseConnection({
  access: { read: true, write: false }
});

In Assembly Line Builder:

  1. Add Assembly Line Computer Use to the agent on the Bind & Build screen.
  2. Enter the relay URL and the relay registration token for the Mac's first registration.
  3. Choose the local write policy, then select Install & pair.
  4. Grant Accessibility and Screen Recording to the signed Assembly Line Builder/ Computer Host application when macOS asks.
  5. Sync or deploy the production secrets. Builder stores ASSEMBLY_LINE_COMPUTER_USE_BINDING in Keychain and writes the local .env with mode 0600. A custom relay also uses ASSEMBLY_LINE_COMPUTER_USE_RELAY_URL.

Builder bundles a pinned, checksum-verified universal Peekaboo release and installs a background LaunchAgent. The registration token is sent to the relay only during first registration and is not part of the deployment binding.

Enable connection writes only after pairing succeeds:

export default defineComputerUseConnection({
  access: {
    read: true,
    write: { approval: "always" }
  }
});

Agent approval and Mac approval are independent. A write runs only when both policies permit it. Revoking the binding in Builder removes the local Keychain/.env material and causes the relay to reject it immediately; deploy a new binding before the agent can use that Mac again.

If pairing fails, re-check the relay URL and registration token, confirm the deployed environment carries ASSEMBLY_LINE_COMPUTER_USE_BINDING, and see Troubleshooting for deploy preflight and connection-secret failures.

Host The Relay​

The relay is a separate, self-hostable Cloudflare Workers project named openeve-computer-relay. It uses a hibernatable WebSocket Durable Object per device and SQLite storage for token hashes, binding revocation, replay protection, and metadata-only audit events.

Set a long random Worker secret, deploy, and use the resulting /v1 URL and secret during first pairing:

pnpm install
pnpm exec wrangler secret put ADMIN_TOKEN
pnpm test
pnpm exec wrangler deploy

Use HTTPS outside local development. Rotate the bootstrap secret through Workers secrets; do not place it in wrangler.jsonc, the agent, or the Mac host configuration. The relay is deliberately not an MCP server and never receives the binding's E2EE private key.

Operating Limits​

The current tool surface is accessibility-tree-first. It includes inspection, application/window/menu/dialog control, clicking, typing, hotkeys, scrolling, dragging, set_value, and accessibility actions. Screenshot/vision, recording, shell, clipboard/paste, browser-specific Peekaboo tools, configuration, cleanup, and nested autonomous agents are excluded.

Computer Host must be online in an unlocked macOS 15+ user session. A locked, offline, revoked, denied, expired, or approval-rejected result is a hard policy boundary, not an instruction to find another route around the host.