Skip to main content

Runtime And Deployment

Assembly Line separates authoring from execution:

agent folder -> compiler -> .assembly-line artifact -> runtime host -> durable runs

The compiler validates a folder and emits a deterministic manifest. The runtime loads the compiled artifact, executes runs, records events, stores context and artifacts, manages approvals and human input pauses, and delivers final responses idempotently.

Contents:

CLI Commands​

Run the CLI against an agent root:

assembly-line <command> <agent-root>

(For a monorepo checkout, Getting Started explains the in-repo pnpm assembly-line form.)

CommandPurpose
initScaffold a minimal agent folder.
addInstall a plugin and wire its provider contribution into gateway.ts or channels/ (--role <role> disambiguates multi-role plugins like docker; --no-install prints the install command instead of running it; --pm overrides the lockfile-detected package manager).
validateCheck required files, exports, schemas, routes, automations, lifecycle handlers, connections, subagents, skills, and gateway bindings.
manifestPrint or write the compiled manifest.
buildEmit the .assembly-line/ runtime artifact.
devValidate, build, and run a local dev check. With --watch, serve locally and rebuild + restart on the same port when the agent changes.
runExecute one local run from a message and optional tool/input.
evalBuild once and run isolated evals/*.json golden cases through the production runtime path, with fingerprinted experiment artifacts, deterministic and custom assertions, optional LLM judges, repetitions/retries, and baseline regression gates.
serveStart the compiled Node runtime locally.
runs cancel|suspend|resumeCancel, suspend, or resume a run on a deployed agent over the authenticated HTTP API (--url, --token).
agent disable|enableDisable or enable new channel ingress on a deployed agent over the authenticated HTTP API (--url, --token).
channels wirePrint or apply channel provider ingress URLs after deploy; Telegram webhooks are set by API when credentials are present.
checkpoints compactDry-run or apply checkpoint retention cleanup for the configured state adapter. Omit --apply for a safe report-only run.
auth codexRun Codex device authentication through the selected deploy publisher and its persistent credential volume.
modelsList provider/model specs from the local model catalog (--provider <name> filters).
deploy --dry-runBuild a deployment plan and report missing setup without publishing.
deployPublish or prepare the target declared by gateway.ts or --target.
help [command]Show usage for all commands or one command (--help also works per command).

Common flags:

FlagPurpose
--root <path>Agent root override.
--out <dir>Build artifact directory.
--message <text>Message for dev or run.
--tool <name>Force a local tool call.
--input <json>Tool input JSON.
--approveAllow or resume approval-gated tool execution.
--dry-runPrint deploy plan or checkpoint cleanup impact without applying changes.
--applyFor checkpoints compact, delete the reported checkpoint rows.
--target <name>Override gateway deploy target.
--onceBoot-check long-lived commands once, then exit.
--port <number>Port for serve, dev --watch, or local deploy.
--watchKeep dev serving and rebuild + restart on agent changes.
--no-model-checkSkip the validate model catalog check.
--url <baseUrl> / --token <token>Deployed agent base URL and admin token for runs and agent (fallbacks: ASSEMBLY_LINE_URL, ASSEMBLY_LINE_ADMIN_TOKEN).
--migration-command <bin>Run artifact migrations before hosted deploy.

Run assembly-line help <command> for the full per-command flag list.

Project environment​

CLI commands load .env from the selected agent root before validation, build, local execution, and deployment planning. This works even when the CLI is invoked from a different directory:

assembly-line run /absolute/path/to/agent --message "hello"
assembly-line serve /absolute/path/to/agent --port 3000
assembly-line deploy /absolute/path/to/agent --target local --serve

For dev, run, serve, and a serving local deploy, non-empty .env values are added directly to the local runtime process environment. They are not copied to a secret store or written into the .assembly-line build artifact. The validate, manifest, build, and local deploy-planning paths use the same resolved environment for required-variable preflight.

Values already present in the invoking shell or host environment take precedence over .env. A declared key with an empty value does not satisfy required-environment preflight. Hosted values are only copied to a provider secret store when deploy --sync-secrets is explicitly used.

Build Artifact​

assembly-line build emits:

.assembly-line/
  manifest.json
  agent-revision.json
  Dockerfile
  package.json
  preflight.json
  route-table.json
  schedules.json
  automations.json
  server/
    boot.json
    boot.js
    sources.json
    source-metadata.json
    source-map.json
  assets/
  migrations/

During the rename compatibility window, an existing legacy .openeve/ artifact directory is still read; having both .openeve/ and .assembly-line/ present at once is an error.

Important files:

  • manifest.json - complete compiled agent contract.
  • agent-revision.json - deterministic source/config revision.
  • package.json - artifact dependency declaration and start script (node server/boot.js).
  • preflight.json - required env and provider setup.
  • route-table.json - HTTP channel routes.
  • automations.json - canonical schedule- and event-triggered automation registrations.
  • schedules.json - deprecated schedule registration compatibility metadata.
  • server/boot.js - production Node runtime boot entrypoint. It loads manifest.json and server/sources.json, constructs production runtime adapters from gateway.ts, and serves the full HTTP API — not a health-only stub.
  • deployment.json - created by assembly-line deploy after a local or provider publish/prepare operation, not by plain build.

The artifact is generated output and should not be committed.

Build artifacts support two package modes:

  • Local mode is the default inside a monorepo checkout. It writes file:./vendor dependencies for Assembly Line workspace packages, copies those packages once under vendor/, links them into node_modules, and copies the runtime dependency closure needed for no-install local artifact smoke tests. Optional native dependencies are filtered to the generated Docker target (Linux x64 with glibc) plus the current build host, so the same local artifact remains runnable for smoke tests without copying every platform binary.
  • Release mode is for published package deployments and is the default when the toolchain is installed from npm (i.e. when no packages/ workspace layout is present). You can also force it with packageMode: "release" on buildAgent() or ASSEMBLY_LINE_ARTIFACT_PACKAGE_MODE=release. The artifact package.json pins the published @assemblyline-agents/* versions (all share one fixed version) and omits local vendor/ and node_modules/ copies; each published package's own external dependencies are resolved transitively by the package manager. The generated Dockerfile then installs dependencies through normal package-manager semantics.

Runtime Lifecycle​

For each run, the runtime:

  1. Persists the run before execution starts.
  2. Builds and stores a context bundle.
  3. Starts the model loop or forced tool call.
  4. Sends the model request without local accounting gates, then observes the provider response and attempts to persist its source-backed usage.
  5. Creates delivery obligations for final responses.
  6. Sends deliveries idempotently through the channel adapter.
  7. Marks the run completed, failed, suspended, cancelled, waiting for input, or waiting for approval.

Failed final deliveries are durable. When a channel send keeps failing retryably, the obligation is deferred onto a durable delivery queue (delivery.deferred, status pending with backoff) instead of going terminal, and a delivery worker drains it later — including after a crash or in another replica (Postgres leases use for update skip locked). Exhausted or non-retryable deliveries end as delivery.failed with failedAt.

Recovery is staleness-guarded and conservative. Executing runs heartbeat updatedAt (default 30s, ASSEMBLY_LINE_RUN_HEARTBEAT_MS); only runs stuck in created/running past max(5min, 4x heartbeat) are swept, and each candidate is claimed through an idempotency key so concurrent replicas never double-recover. The sweep completes already-delivered runs without re-sending, cancels tool calls before side effects start, attempts one in-place resume when a harness continuation checkpoint exists, enqueues a real pending delivery for runs that reached a model response but not delivery, and marks everything else failed (with run.failed) instead of pretending it completed. listenNodeRuntime runs recoverIncompleteRuns() once at boot and startBackgroundWorkers() keeps the delivery, sandbox-sync, conversation-turn mailbox, and orphan-recovery workers running until the server closes.

Usage Accounting And Observability​

Usage accounting is behavior-neutral observability. It never reserves expected tokens or cash, estimates a request, rejects a provider call, or changes an agent response based on local cost state. A response_completed event is normalized into the ledger using the provider response ID when available, so retries are idempotent. If ledger persistence fails, the provider response still completes; the durable model event and runtime warning expose the observation gap.

Every record carries the run, parent run, stable agent/revision, subagent, iteration, provider, requested and response model, provider request ID, billing mode, UTC occurrence time, native input/output/cache-write/cached/ reasoning tokens, currency, receipt hash, and sanitized provider receipt. Actual cash uses integer cost_micros and only accepts provider_reported/provider_reconciled provenance. Local catalog-price math is discarded. Unavailable cash remains null, never $0 or an estimate.

The ledger has three non-additive record kinds:

  • transaction: one attributable model response; this is the default report.
  • control_total: a provider organization/activity bucket used to verify completeness.
  • adjustment: reserved for explicit accounting corrections.

OpenRouter usage accounting exposes native token counts and charged credits. The Pi bridge follows the generation ID to obtain the settled receipt; a temporarily unavailable receipt is stored as unavailable and retried by reconciliation through the provider's generation endpoint. OPENROUTER_MANAGEMENT_KEY additionally imports the last 30 completed UTC days of activity totals.

The Codex app-server supports both ChatGPT and API-key authentication. Before each turn, Assembly Line asks app-server which rail is active. ChatGPT mode snapshots provider rate-limit/credit state for reporting only; API-key mode is labeled api_key. Snapshot failures and depleted states do not become local gates. App-server token notifications are provider-reported. It does not expose a per-turn dollar charge, so Assembly Line records that cash as unavailable rather than zero or a price-table guess.

For API-key Codex cash and organization-wide completeness, POST /usage/reconcile calls OpenAI's Admin Usage and Costs APIs with ASSEMBLY_LINE_OPENAI_ADMIN_KEY (or OPENAI_ADMIN_KEY). Completions usage and cost buckets become separate provider control totals. They receive an agentId only when an operator supplies a dedicated mapping: API-key or project for token controls, and project for cash controls because the Costs API does not group by API key. An organization invoice is never proportionally allocated to runs. This means token accounting remains exact per Codex run, while exact cash is per run only for providers that issue transaction receipts and otherwise remains exact at the provider control-total grain.

The reconciliation report returns transaction totals, provider control totals, signed variances, separate unreconciled token/cash counts, and exact run-token/run-cost coverage for any [from,to) USD period. Schedule the authenticated reconciliation operation after the provider's settlement delay (48 hours by default for OpenAI). Control totals are excluded from ordinary transaction summaries, so reconciling never doubles reported spend.

A transport failure after the durable request_started event produces an unobserved transaction with zero measured tokens, null cash, and unavailable provenance. Crash recovery does the same for interrupted requests. These records express uncertainty; they are not charged against a local quota. A missing or failing usage facet emits degradation warnings but never blocks provider execution or response delivery.

Node Runtime HTTP API​

Auth model​

In dev mode, local inspection and API-run endpoints are open for quick iteration. In production, the Node host fails boot unless control-plane auth is configured with a host-provided auth policy or ASSEMBLY_LINE_ADMIN_TOKEN. /manifest, /routes, /conversations, /conversations/:id/messages, /runs, /usage, /runs/:id, /runs/:id/events, /runs/:id/timeline, and /runs/:id/stream require Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN> when the built-in token policy is used. Conversation rename/archive routes use the same authenticated agent-control policy. POST /conversations/:id/turns uses the authenticated run-create policy and, like POST /runs, is disabled in production by default; set ASSEMBLY_LINE_ENABLE_API_RUNS=true only for deployments that intentionally expose authenticated API-triggered runs. The resume and operator endpoints use the same admin auth but are not gated by ASSEMBLY_LINE_ENABLE_API_RUNS — paused and active runs must remain operable even when API-triggered run creation is off. Resumes are safe to replay: each one claims the run's per-pause idempotency key, so a double-submit executes the gated tool at most once, even across replicas.

Compiled channel routes, such as /slack/events or /message, are also mounted from the manifest route table. In production, first-party provider helper routes remain public so the provider can call them, but their normalizers must verify signatures or shared secrets before accepting a turn. Generic defineChannel() HTTP routes require a host auth policy or Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN>; the raw message fallback is dev-only unless the host has authenticated the request.

Endpoints​

EndpointPurpose
GET /healthLiveness plus agent revision.
GET /healthzLiveness plus agent revision.
GET /readyzReadiness: 200 normally, 503 while the runtime is draining during graceful shutdown.
GET /manifestCompiled manifest.
GET /routesCompiled route table.
GET /conversationsList durable conversations for this stable agent. Query params: limit (default 40, max 100), cursor (opaque, from the response's nextCursor), archived (true for archived only, all for both; default active only), subject, and channel (default direct).
GET /conversations/:id/messagesRead an ordered, paginated transcript for an agent-owned conversation. Query params: before (opaque message cursor) and limit (default 50, max 100).
PATCH /conversations/:idRename or archive a conversation with {"title":"…","archived":true}.
POST /conversations/:id/turnsSend a turn through the built-in direct transport. Accepts JSON or multipart attachments and supports the same "stream": true SSE lifecycle as POST /runs.
POST /runsStart a local/API run. Pass "stream": true to receive the whole run as server-sent events (lifecycle events, token deltas, then a final run_result).
GET /runsQuery run summaries. limit query param defaults to 200 (max 1000).
GET /runs/:idInspect one run timeline.
GET /runs/:id/eventsInspect raw run events.
GET /runs/:id/timelineInspect grouped timeline.
GET /runs/:id/streamAttach to a run's live SSE stream: replays the durable event log (SSE id: is the event sequence, so Last-Event-ID reconnects resume where they left off), then tails live events including ephemeral model.response_delta tokens, and ends with stream_end after a terminal event. Works for runs started by any channel, schedule, or client.
GET /usageQuery source-backed usage. Defaults to transaction summaries grouped by agent/provider/model/billing mode/cost source/currency/day. view=records returns records. Filters: from, to, runId, parentRunId, agentId, agentRevision, subagent, provider, model, responseModel, billingMode, currency, tokenSource, costSource, and recordKind; groupBy controls summary dimensions. Currency is always an aggregate boundary so unlike currencies are never added together.
POST /usage/reconcileAuthenticated agent-control operation that imports provider control totals and retries pending receipts. Body: {"provider":"openai","from":"ISO","to":"ISO?","providerLabel":"optional"}; provider may instead be openrouter. Provider credentials come from runtime environment variables, never the request or ledger.
POST /runs/:id/approveResume a run paused on tool approval (waiting_for_approval): runs the gated tool and re-enters the harness. Returns the post-resume run summary. 404 if the run is unknown, 409 if it is not waiting for approval or a resume is already in flight.
POST /runs/:id/answerResume a run paused on ask_question (waiting_for_input) with {"answer": "..."}: splices the answer as the tool result and continues. 400 without an answer, 404/409 as above.
POST /runs/:id/cancelCancel a parked run synchronously (200) or request cooperative cancellation of a running run (202).
POST /runs/:id/suspendRequest cooperative suspension of a running run (202). Other statuses return 409.
POST /runs/:id/resumeResume a deliberately suspended run from its latest compatible harness continuation (200; 404/409 for unknown or wrong-status runs).
GET /agent/controlRead the durable ingress control for the stable agent identity.
POST /agent/disable / POST /agent/enableDisable or enable new channel ingress (200) and append a control-plane audit event.
GET/POST /assembly-line/automations/tickTrigger due static and dynamic time-based automations from a gateway or cloud scheduler.
POST /assembly-line/automations/eventsSubmit a trusted normalized provider event for matching event automations.
GET/POST /assembly-line/scheduler/tickDeprecated alias for /assembly-line/automations/tick.
GET/POST /assembly-line/connections/callbackComplete connection authorization callbacks.

During the rename compatibility window, the legacy /openeve/* spellings of these internal routes remain served as aliases with the same authentication and rate limits. For the OAuth connection callback specifically, both /assembly-line/connections/callback and the legacy /openeve/connections/callback are served; callback-URL construction still emits the legacy path until the Mirror-side registration cutover, after which it flips to the new path.

Direct conversations​

direct is Assembly Line's built-in, provider-neutral client transport. It is the right channel name for a first-party app, internal console, custom web UI, or mobile client chatting with an agent. custom remains appropriate only for a developer-defined channel adapter with its own route, normalization, delivery, and trust boundary.

The direct API is control-plane HTTP, not a public provider webhook. Production boot requires host authentication, transcript reads require admin-read authority, mutations require agent-control or run-create authority, and conversation ids are checked against the runtime's stable agent scope before messages can be read or added. A client should send Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN> (or credentials accepted by the host-provided auth policy) over HTTPS and must never embed an admin token in a public browser bundle.

Conversations persist across agent revisions when agent.id is stable. Their messages and metadata live in the configured state adapter. Inbound attachment bytes are normalized into the configured private blob adapter before transcript metadata is returned; generated files use the same private storage boundary. Object storage is not made public unless application code explicitly writes a public blob.

Create and stream a turn:

curl -N https://agent.example/conversations/01JTHREAD/turns \
  -H "Authorization: Bearer $ASSEMBLY_LINE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message":"Summarize the attached brief","stream":true}'

The SSE response includes durable lifecycle events, ephemeral model deltas, a final run_result, and stream_end. If the connection drops after the run id is known, reattach through GET /runs/:id/stream with Last-Event-ID. Use an Idempotency-Key header or a stable body eventId when a client may retry the initial POST.

Operator Controls​

Suspend and cancel are cooperative at the model-request boundary. The intent is written atomically to the durable run row, and the executing replica's heartbeat observes it; cross-replica detection latency is therefore at most ASSEMBLY_LINE_RUN_HEARTBEAT_MS (30 seconds by default). A tool already executing is allowed to finish, preserving a consistent side-effect record. Cancel wins a race with suspend.

Suspension applies only to running runs. It persists a harness continuation, moves the run to suspended, and keeps it outside orphan recovery while still bounding its active checkpoints. POST /runs/:id/resume claims a per-generation idempotency key and re-enters the normal continuation path. Parked approval/input/connection runs can be cancelled synchronously; pending tool records become cancelled.

The agent control is an ingress kill switch, not a process kill switch. Disabled channel ingress returns 503 without Retry-After and consumes neither capacity nor the provider event's idempotency key. In-flight runs, explicit resumes, schedules, and operator access continue. The setting is scoped by stable agent.id when present (otherwise the compiled revision). Use FileStateAdapter, PostgresStateAdapter, or another durable RuntimeSettingsStore; a missing settings facet falls back to memory and will not survive restart.

Remote CLI equivalents use the same authenticated HTTP API:

assembly-line runs suspend <runId> --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line runs resume <runId> --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line runs cancel <runId> --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line agent disable --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line agent enable --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"

ASSEMBLY_LINE_URL and ASSEMBLY_LINE_ADMIN_TOKEN are the flag fallbacks.

Request bodies are capped before parsing. The default limit is 10 MiB; set ASSEMBLY_LINE_HTTP_MAX_BODY_BYTES or NodeRuntimeServerOptions.maxRequestBodyBytes only when a deployment intentionally accepts larger webhook payloads.

Provider channel routes remain public HTTP routes because providers such as Slack, Telegram, Teams, and Discord interactions call them directly. Those routes must rely on their channel adapter's signature or token verification before accepting a turn. Long-lived provider ingress such as Discord Gateway is started by the Node host through startIngress() and feeds the same idempotent accepted-turn path as HTTP channels.

/assembly-line/automations/tick (and its deprecated scheduler alias) accepts optional now and lookbackMs values in the query string or JSON body. In production, set ASSEMBLY_LINE_SCHEDULER_SECRET and send it as Authorization: Bearer <secret> or x-assembly-line-scheduler-secret. Without a configured secret, the endpoint only accepts dev-mode runtime requests.

Scheduler startup comes from gateway.scheduler:

  • adapter("local") starts an in-process loop with the Node host.
  • adapter("gateway") registers time-based automations but relies on the endpoint or host code calling runDueAutomations().
  • adapter("postgres") starts the polling loop and coordinates duplicate workers through Postgres-backed state/idempotency. Pair it with state: adapter("postgres").

Durability Workers​

listenNodeRuntime runs recoverIncompleteRuns() once at boot, then calls runtime.startBackgroundWorkers() to keep the delivery worker, sandbox-sync worker, conversation-turn mailbox worker, and periodic orphan sweep running; all four stop when the server closes. Each worker has an env kill-switch (ASSEMBLY_LINE_DELIVERY_WORKER, ASSEMBLY_LINE_SANDBOX_SYNC_WORKER, ASSEMBLY_LINE_CONVERSATION_TURN_WORKER, ASSEMBLY_LINE_RUN_RECOVERY), the heartbeat and sweep cadence are tunable (ASSEMBLY_LINE_RUN_HEARTBEAT_MS, ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS), and the delivery queue has lease, batch, attempt, and interval knobs. The canonical tables are in the Configuration Reference: Durability workers and recovery and Delivery queue.

Model-Call Resilience​

Every model request runs inside a retry-and-deadline envelope. Retryable failures (408/429/5xx, provider overload, network errors) are retried with jittered exponential backoff — Retry-After hints are honored and capped — while fatal failures (invalid API key, authentication, invalid request) fail the run immediately with the durable reason model.request_failed. A stream that stops producing events is aborted by an inactivity watchdog and retried. Retries are visible as model.request_retried events and warn-level log lines. When the retry budget is exhausted on a transient error, the run is left running with a durable run.execution_error event so the orphan sweep resumes it from the latest continuation checkpoint — one transient outage never terminally fails a run. Knobs: ASSEMBLY_LINE_MODEL_MAX_RETRIES, ASSEMBLY_LINE_MODEL_TIMEOUT_MS, ASSEMBLY_LINE_MODEL_MAX_RETRY_DELAY_MS, ASSEMBLY_LINE_MODEL_STREAM_IDLE_TIMEOUT_MS.

Run Deadline And Tool Timeouts​

The run heartbeat doubles as a wall-clock deadline enforcer: when an active execution segment exceeds ASSEMBLY_LINE_MAX_RUN_DURATION_MS (default 1 h), the in-flight model request is aborted via AbortSignal and the run fails with reason run.max_duration_exceeded. Parked runs (approvals, human input) hold no budget — the clock only ticks while the run actively executes, and it resets on resume. Tool execute calls are raced against ASSEMBLY_LINE_TOOL_TIMEOUT_MS (per-tool timeoutMs on the definition overrides it), and model-supplied bash timeouts are clamped to ASSEMBLY_LINE_BASH_TIMEOUT_MAX_MS — timeoutMs: 0 falls back to the default rather than disabling the timeout.

Terminal Outcomes And Recovery Fidelity​

Every run that reaches failed records a machine-readable terminalReason and human-readable terminalError on the run record (queryable without scanning the event log; also on the run.failed event payload). Reasons include model.request_failed, run.max_iterations_exceeded, run.max_duration_exceeded, output.validation_exhausted, tool.execution_failed, connection.tool_failed, trigger.*_failed, and run.orphaned. Terminal outcomes (run.completed/run.failed/ run.cancelled) are always logged at a single choke point, whichever code path produced them; response content is never logged, only its size.

Crash recovery reads the newest continuation checkpoint when reconciling a run that died after its final model response: the actual answer is delivered (recovered: true, contentRecovered: true) and the "response was interrupted" notice is reserved for genuinely missing checkpoints. When a crash interrupted a tool batch, the resume surfaces already-completed tool outputs and interrupted-tool warnings to the model as a recovery report so completed side effects are not blindly re-executed. Dynamic schedules that fail repeatedly back off exponentially and are auto-disabled after ASSEMBLY_LINE_SCHEDULE_MAX_FAILURES consecutive failures (an operator-visible schedule.disabled_after_failures control event is recorded).

Concurrency And Rate Limiting​

Accepted provider turns are first written to a durable per-conversation FIFO mailbox. One turn per (agent, conversation) may be running or parked; later turns wait, while distinct conversations can use the full global concurrency budget in parallel. Provider routes therefore acknowledge valid durable work even when all run slots are busy instead of relying on webhook redelivery for backpressure. Postgres enforces the active-turn exclusion across replicas.

Direct brand-new runs still pass through the bounded semaphore before run state is written. When its in-memory admission queue is full the runtime rejects direct work with RunCapacityError, and the Node host maps that to HTTP 429 with a whole-second Retry-After on POST /runs. Resumes (approvals, human input, connection callbacks, orphan-recovery continuations, and scheduler continuations of an existing run) never queue behind the limit — queueing a resume behind the run it unblocks would deadlock — but still count toward the drain performed by graceful shutdown. A terminal resume also releases the next mailbox turn for that conversation.

Ingress rate limiting is a token bucket applied after auth on provider-channel, run-create, and scheduler routes (health and admin routes are exempt). It is off by default and enabled either through NodeRuntimeServerOptions.rateLimit (see the customization guide) or through ASSEMBLY_LINE_INGRESS_RATE_LIMIT and ASSEMBLY_LINE_RUNS_RATE_LIMIT (capacity/refillPerSecond form, e.g. 60/10); ASSEMBLY_LINE_RUNS_RATE_LIMIT also seeds the run-resume and run-control buckets unless those are configured separately. ASSEMBLY_LINE_MAX_CONCURRENT_RUNS bounds simultaneously executing brand-new runs (createProductionRuntimeOptions defaults it to 16 outside dev) and ASSEMBLY_LINE_MAX_QUEUED_RUNS (default 0) lets excess runs wait for a slot. The canonical table is Concurrency and rate limiting.

Rate-limited requests receive 429 { "error": "Rate limited." } with a Retry-After header.

Rate-limit buckets are keyed by route class and client address. Behind a reverse proxy or load balancer, set ASSEMBLY_LINE_TRUST_PROXY=true so the first X-Forwarded-For hop is used as the client address; without it, every proxied request shares one bucket keyed by the proxy's address, so a single noisy client can exhaust the limit for everyone.

Graceful Shutdown​

listenNodeRuntime returns a NodeRuntimeHandle with an idempotent shutdown({ timeoutMs? }) and a closed promise. The shutdown sequence: mark draining (/readyz starts answering 503 so load balancers stop routing new traffic; /health//healthz stay 200 for liveness) -> close the HTTP listener and stop channel ingress -> stop the scheduler -> stop background workers -> wait for in-flight runs up to the timeout -> flush the telemetry sink -> close the state adapter (Postgres ends its pool when it created it). Runs still executing at the timeout are abandoned safely: orphan recovery repairs them on the next boot.

assembly-line serve and assembly-line deploy --serve install SIGTERM/SIGINT handlers (installSignalHandlers from @assemblyline-agents/node): the first signal drains gracefully and exits 0; a second signal exits 1 immediately.

The drain timeout is ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS (default 30000), and ASSEMBLY_LINE_SIGNAL_HANDLERS=false prevents handler installation; see Graceful shutdown.

State And Blob Storage​

Local development uses file-backed state and local blob storage. Production state should use Postgres:

import { defineGateway } from "@assemblyline-agents/core";
import { neonPostgres } from "@assemblyline-agents/postgres";
import { r2Blob } from "@assemblyline-agents/s3";

export default defineGateway({
  state: neonPostgres(),
  blob: r2Blob()
});

Postgres stores runs and atomic control intents, events, messages, conversations, tool calls, approvals, delivery obligations, schedules, schedule run lifecycle status, memory indexes, file catalog records, sandbox leases, idempotent usage receipts, exact micro-dollar/token aggregates, run queues, idempotency keys, learned skills, dynamic schedules including trigger metadata, dynamic connections, runtime settings, and control-plane audit events. PostgreSQL provides durable multi-replica usage observability; file/in-memory accounting is process-local.

Connection grants and OAuth authorization sessions use the durable state adapter when it implements those stores. The Postgres adapter does, so Postgres-backed production deployments do not need a separate file encryption secret for connection credentials.

When a production Node deployment uses file-backed connection credential stores, set ASSEMBLY_LINE_CONNECTION_STORE_SECRET or ASSEMBLY_LINE_SECRET to a stable secret of at least 32 characters. The built-in local development fallback is accepted only with devMode: true; production boot rejects missing, short, or known development secrets.

Blob storage stores context bundles, attachments, extracted text, generated artifacts, and sandbox sync bundles. Blob records are private by default. S3/R2 *_PUBLIC_BASE_URL is used only when code explicitly writes a blob with visibility: "public"; context bundles, memory files, and inbound attachments should remain private.

Sandbox Sync​

Sandbox-backed file tools write through provider workspaces, but durable production persistence remains Assembly Line state and blob sync. When a run finishes with dirty sandbox files and async sync is enabled, the runtime retains the dirty sandbox instead of deleting it, queues a sync job, and lets final delivery complete. The sync worker first calls adapter connect() and then wake() so paused, stopped, detached, or otherwise retained warm sessions can still be synced. After a successful sync the session is marked clean and disposed through the adapter's clean lifecycle path.

Hosted sandbox adapters do not use provider snapshots as the normal persistence path. Snapshots are created only when the configured sandbox snapshot policy requests them and the provider SDK exposes snapshot creation.

Deploy Targets​

Every non-local target follows one CLI sequence: resolve the publisher, run optional target preflight and preparation hooks, sync secrets when requested, run artifact migrations once (locally or through the publisher), publish, then write the final receipt with migration status. Built-in option precedence is CLI flag, environment variable, gateway.ts option, then provider default. Community publishers use this path too and remain compatible when they omit the newer optional hooks.

Environments​

--env <name> selects the deploy environment (default: development, or the manifest's gateway.deploy.options.environment). Every environment holds an isolated deployment of the same agent:

  • Identity. The default environment keeps its legacy, unscoped provider resource names, so existing deployments are never orphaned. Any other environment gets environment-scoped resources: Docker suffixes container and volume names (openeve-<agent>-test) and requires an explicit --port in serve mode; Fly derives <app>-<environment> when the app name comes from the manifest (an explicit --fly-app is always respected verbatim); VPS derives <environment>.<domain> — DNS for that subdomain (a wildcard or an explicit record) is your prerequisite; Railway passes the environment through natively, so the named Railway environment must exist. Explicit identity flags always win over derivation.
  • Receipts. Each environment's canonical receipt is written to .assembly-line/deployments/<environment>.json. The legacy .assembly-line/deployment.json keeps its historical last-deploy-wins behavior for existing readers; new readers should prefer the environment-scoped file.
  • Teardown. assembly-line deploy <agentRoot> --env <name> --destroy removes the environment's provider resources on every built-in target (docker, fly, railway, vps, local). Durable data — volumes, databases, the VPS deployment directory — survives unless --purge-data is passed; Fly and Railway require it explicitly, because destroying a Fly app or deleting a Railway environment always removes the volumes and databases inside it. VPS destroy never touches shared host infrastructure (the Caddy edge, the shared Postgres cluster) or customer-owned external databases. Destroying the default environment additionally requires --force.

Composed with assembly-line eval --url, this is the ephemeral test-environment recipe:

assembly-line deploy agent --env test --sync-secrets --secrets-from .env.test
assembly-line eval agent --url https://<test-gateway> --token $ASSEMBLY_LINE_ADMIN_TOKEN
assembly-line deploy agent --env test --destroy --purge-data

Local​

Use local deploy for developer machines or long-lived VMs:

assembly-line deploy agent --target local --serve --port 3000

Railway​

adapter("railway") or railwayDeploy() publishes the built artifact through the @assemblyline-agents/railway deploy publisher and Railway CLI.

Required:

  • RAILWAY_TOKEN or authenticated Railway CLI
  • Linked project/service, or --railway-project and --railway-service
assembly-line deploy agent \
  --target railway \
  --railway-project prj_x \
  --railway-service svc_y

When the gateway uses state: railwayPostgres(), deployment first inspects the selected Railway environment. It reuses the Postgres database service when present, otherwise provisions one with railway add --database postgres, and sets the Assembly Line service's DATABASE_URL to ${{Postgres.DATABASE_URL}} before railway up. The deploy receipt records whether the database was created or reused. Use railwayPostgres({ databaseService: "name", provision: false }) to require a specific existing database service without automatic creation. Automatic creation uses Railway's default Postgres service name. A local DATABASE_URL is not required for this auto-provisioned path; the ordinary Postgres, Neon, Supabase, and provision: false paths still require one during deployment preflight.

Syncing secrets to the target​

By default deploy sets nothing on the remote service — you configure variables in the provider dashboard. Pass --sync-secrets to push your local secrets as part of the deploy: the CLI reads the project .env (or --secrets-from <path>), overlays declared runtime requirements from the resolved host environment, and calls the publisher's syncSecrets before publishing, so the first deploy boots with them. Undeclared host variables are never copied. Only key names are logged, never values; empty keys are skipped. Supported on railway (railway variables --set), fly (flyctl secrets set), vps (an atomic remote 0600 environment file), and docker (held in memory for the deploy and handed to serve-mode docker run through the child process environment with value-less --env KEY flags — never on argv or disk; image-only builds never bake secrets). The local target reports that sync is unsupported and leaves secrets to you.

Credentials required by the selected deploy adapter remain local and are not copied into the agent runtime. For example, RAILWAY_TOKEN authorizes the Railway CLI and FLY_API_TOKEN authorizes flyctl; runtime requirements such as model, channel, state, blob, and connection credentials are eligible for remote sync.

assembly-line deploy agent --target railway --sync-secrets

The deploy receipt (written to .assembly-line/deployments/<environment>.json, with the legacy .assembly-line/deployment.json mirroring the most recent deploy) separates deploymentUrl — the reachable service URL when the provider CLI reports one, otherwise null — from dashboardUrl, the provider's management console, so the two are never conflated.

Docker​

adapter("docker") builds the compiled artifact as a Docker image through the @assemblyline-agents/docker deploy publisher and can run it locally.

Served deployments use a stable openeve-<agent-slug> container name. Codex artifacts also use the stable openeve-<agent-slug>-data volume mounted at /data; the slug comes from agent id, then name, then the agent folder.

assembly-line deploy agent \
  --target docker \
  --docker-image assembly-line/my-agent \
  --serve \
  --port 3000

Fly​

adapter("fly") writes a minimal fly.toml and deploys the artifact with flyctl through the @assemblyline-agents/fly deploy publisher.

Codex artifacts provision the app-scoped openeve_data volume in the selected region. Fly deploys are limited to single-Machine apps because Fly volumes are Machine-local; deploy and auth fail clearly when an existing app has more than one Machine.

Required:

  • FLY_API_TOKEN
  • --fly-app or FLY_APP_NAME
assembly-line deploy agent \
  --target fly \
  --fly-app my-agent \
  --fly-region iad

The supported deployment contract has credential-free parity checks: pnpm smoke:deploy:docker exercises a real local build, run, recreation, remote command, and persistent volume; pnpm smoke:deploy:fly parses generated configuration with the installed Fly CLI and verifies the publisher's CLI surface using an intentionally invalid token. Neither check creates hosted resources. pnpm smoke:deploy:fly:live is the opt-in hosted exit gate: it creates a uniquely named app, deploys through the Assembly Line publisher, verifies HTTP health, secret sync, SSH execution, and /data persistence across a redeploy, then destroys the app and volume and verifies their absence. It requires an authenticated flyctl session and may incur brief provider usage.

Generic VPS​

vpsDeploy() from @assemblyline-agents/vps deploys to an AMD64 Ubuntu 24.04, Ubuntu 26.04, or Debian 12 host. Hetzner, Hostinger, DigitalOcean, OVH, Vultr, and similar hosts use the same workload publisher.

Hetzner hosts can be created or adopted with assembly-line hosts bootstrap. The bootstrap verifies that the public key matches the private key selected by identityFileEnv, creates an openeve sudo user, installs Docker Engine and Compose, enables UFW, fail2ban, unattended upgrades, provider backups, delete and rebuild protection, and a Hetzner Firewall. SSH is restricted to --ssh-source <CIDR> unless --allow-global-ssh is explicitly supplied. The command waits for cloud-init and the security services, pins the SSH host key, then writes the inventory entry. Existing servers are never rebuilt implicitly. Adoption of an existing named server additionally requires --host-key-sha256 obtained from the provider console or another trusted path; Assembly Line will not establish trust from an in-band key scan alone.

export ASSEMBLY_LINE_PRODUCTION_EU_SSH_KEY="$PWD/keys/assembly-line_ed25519"

assembly-line hosts bootstrap \
  --provider hetzner \
  --host production-eu \
  --server assembly-line-production-eu \
  --inventory ./assembly-line.hosts.json \
  --ssh-public-key ./keys/assembly-line_ed25519.pub \
  --identity-file-env ASSEMBLY_LINE_PRODUCTION_EU_SSH_KEY \
  --location ash \
  --server-type cpx32 \
  --ssh-source 198.51.100.10/32 \
  --expected-region ash

The server must have Docker Engine, Docker Compose, flock, curl, ss, seccomp, AppArmor, and root SSH or passwordless sudo. Host Postgres mode also requires OpenSSL and systemd. Register it by name in assembly-line.hosts.json:

{
  "version": 1,
  "hosts": {
    "production-eu": {
      "address": "203.0.113.10",
      "ssh": {
        "user": "deploy",
        "port": 22,
        "identityFileEnv": "ASSEMBLY_LINE_PRODUCTION_EU_SSH_KEY",
        "hostKeySha256": "SHA256:replace-with-the-pinned-fingerprint"
      },
      "provider": {
        "kind": "hetzner",
        "resourceId": "optional-server-id",
        "region": "ash"
      }
    }
  }
}

Inventory precedence is --vps-hosts-file, ASSEMBLY_LINE_VPS_HOSTS_FILE, then the nearest assembly-line.hosts.json found upward from the agent root. During the rename compatibility window a legacy openeve.hosts.json is still discovered (the new name is searched first at each level). The SSH private key path comes from identityFileEnv; the key and host address are never written to deployment receipts. Host-key scanning must match the pinned SHA-256 fingerprint before strict SSH is allowed.

import { defineGateway, adapter } from "@assemblyline-agents/core";
import { vpsDeploy } from "@assemblyline-agents/vps";

export default defineGateway({
  deploy: vpsDeploy({
    host: "production-eu",
    domain: "agent.example.com",
    aliases: ["www.agent.example.com"],
    expectedRegion: "ash",
    resources: { cpus: 1, memory: "1g", pids: 256 },
    database: { mode: "host" },
    monitoring: { enabled: true, diskFreeMinimumMb: 5120 }
  }),
  runtime: adapter("node"),
  state: adapter("postgres"),
  blob: adapter("r2"),
  sandbox: adapter("e2b")
});

VPS deployment requires a stable agent.id, Node runtime, Postgres state, S3/R2 blobs, a hosted sandbox, ASSEMBLY_LINE_ADMIN_TOKEN, DNS for the configured primary domain and aliases. Local state/blob storage and local or Docker-socket sandboxes are hard preflight failures. ASSEMBLY_LINE_VPS_ALERT_WEBHOOK_URL is an optional notification destination; health checks continue to run and record failures in systemd/journald when it is unset. expectedRegion compares the configured intent with inventory and warns about likely user, Photon, database, or sandbox latency.

Use --sync-secrets on the first deploy. Repeat deploys can reuse the complete remote 0600 environment without copying runtime credentials back to the operator machine; the publisher validates required keys remotely before database setup, migrations, and activation. The VPS publisher sets ASSEMBLY_LINE_PUBLIC_URL to the configured primary HTTPS domain on every secret sync so callbacks and generated public links cannot retain a prior provider's hostname.

assembly-line secrets diff agent --target vps --env production compares required and configured key names without returning remote values. If .env is absent, --sync-secrets still reads declared runtime variables from the command environment. An explicitly requested missing --secrets-from file is an error.

assembly-line deploy agent \
  --target vps \
  --vps-host production-eu \
  --vps-domain agent.example.com \
  --sync-secrets \
  --env production

Each agent receives a dedicated hardened non-root container, ingress network, data network, /data volume, secret file, domain ownership record, and database identity. Runtime containers are read-only, drop all capabilities, set no-new-privileges, carry CPU/memory/PID/log limits, and never mount the Docker socket. A bounded, non-executable /app/.openeve tmpfs holds generated runtime module-cache files without making the application root writable. A trusted shared Caddy container joins each ingress network but agents do not join one another's networks.

Releases use immutable revision-labelled images and inactive blue/green slots. Preparation, activation, rollback, and domain transfer are separate operations:

# Build the inactive slot, sync secrets, run migrations, but do not route traffic.
assembly-line deploy agent --target vps --env production --sync-secrets --prepare-only

# Authenticate a Codex-backed prepared release if applicable.
assembly-line auth codex agent --target vps --env production --prepared

# Activate exactly the persisted prepared revision.
assembly-line deploy agent --target vps --env production --activate

# Restore the previous runtime/route without changing durable state.
assembly-line deploy agent --target vps --env production --rollback

# Reconcile the primary domain and aliases without rebuilding.
assembly-line deploy agent --target vps --env production --domains-only

Activation rejects stale prepared metadata, waits for container /readyz, transactionally claims all domain aliases, reloads Caddy, verifies public readiness, records the prior slot, and only then removes the old runtime. Failures restore both the prior route and prior domain ownership. The host retains the active and previous revision directories and prunes older release directories and images.

Before a state cutover, use the durable maintenance fence:

assembly-line agent quiesce --url https://agent.example.com
assembly-line agent status --url https://agent.example.com
# perform the verified transfer
assembly-line agent resume --url https://agent.example.com

Quiescence stops new ingress, schedules, delivery work, sandbox-sync work, and run recovery, then waits for the reported in-flight run count to reach zero. The state survives process restarts.

database.mode: "external" requires DATABASE_URL and writes a deployment ownership marker before migrations, refusing reuse by another agent. database.mode: "host" runs one private Postgres cluster and creates a separate database/login role per agent. Host mode requires the ASSEMBLY_LINE_VPS_BACKUP_BUCKET, ASSEMBLY_LINE_VPS_BACKUP_REGION, ASSEMBLY_LINE_VPS_BACKUP_ACCESS_KEY_ID, and ASSEMBLY_LINE_VPS_BACKUP_SECRET_ACCESS_KEY secrets; optional ASSEMBLY_LINE_VPS_BACKUP_ENDPOINT selects a custom S3-compatible endpoint and ASSEMBLY_LINE_VPS_BACKUP_RETENTION_DAYS defaults to 30. A daily systemd timer creates a compressed dump, verifies the uploaded object, and enforces retention. A weekly timer downloads the newest backup and restores it into a scratch database. A deployment-scoped restore script remains on the VPS; it requires the literal --replace-confirmed argument, takes a fresh backup, and automatically restores the pre-restore database if the requested restore fails. Migration commands connect to the private cluster through a temporary fingerprint-pinned SSH tunnel; the tunnel closes as soon as the migration command finishes.

To move an external database such as Neon into host mode, quiesce the source first and keep its URL in an environment variable:

export NEON_DATABASE_URL='postgresql://...'
assembly-line state migrate-postgres agent \
  --env production \
  --source-url-env NEON_DATABASE_URL \
  --source-quiesced \
  --replace-target

The transfer uses version-matched containerized clients, rejects an older target major, takes an offsite and local pre-transfer backup, verifies the uploaded dump checksum, restores into the isolated role/database, and compares normalized schema plus exact per-table row counts. Any restore or verification failure automatically restores the pre-transfer target. Receipts contain the source environment-variable name and dump hash, never the URL.

Postgres images require explicit numeric tags and default to postgres:17.10-alpine. PostgreSQL 18+ uses the official image's /var/lib/postgresql volume layout; 17 and earlier use /var/lib/postgresql/data. Change the configured image only through:

assembly-line state upgrade-postgres agent --env production --confirm-upgrade

The upgrade pulls and verifies the image major, creates logical globals and per-database custom dumps, restores into a new versioned volume, compares exact row counts, and retains the stopped previous container and volume for rollback. Ordinary deploys refuse an image mismatch instead of silently upgrading.

The host monitor runs every five minutes and checks the active container, public /readyz, Postgres, backup/restore-verification units and timers, and free disk space. Failures are recorded by systemd/journald and are also posted to ASSEMBLY_LINE_VPS_ALERT_WEBHOOK_URL when it is configured.

This is trusted-owner process isolation, not hostile multi-tenant isolation. Use separate VMs or microVMs for mutually untrusted tenants. V1 schedules one AMD64 replica per agent; ARM64, high availability, multi-replica scheduling, automatic workload deletion, and non-Hetzner provider bootstrapping are deferred.

Codex (Preview)​

Preview — this surface may change without notice.

An agent whose primary model starts with openai-codex/ uses the official Codex CLI session. That session may be authenticated through ChatGPT entitlements/credits or an OpenAI API key; Assembly Line reads app-server's credential-free account mode and labels every receipt with the active billing rail. The Node host starts the official Codex app-server and lets the Codex CLI resolve its own authentication. Assembly Line never reads or exports the OAuth cache, and the Codex built-in shell, filesystem, patch, web, app, MCP, and subagent tools are disabled; app-server dynamic tools call back through Assembly Line's durable tool and approval path.

The generated container includes a pinned compatible @openai/codex CLI, sets CODEX_HOME=/data/codex, and puts the artifact's local npm binaries on PATH. The compiler stamps two requirements into every Codex artifact: persistent /data storage and remote command execution. Planning rejects deploy targets that do not advertise both persistent-storage and remote-exec, and auth also requires the publisher to implement runRemoteCommand.

Deploy and authenticate​

Deploy and authenticate in two explicit steps:

assembly-line deploy agent --target railway --env production
assembly-line auth codex agent --target railway --env production
assembly-line auth codex agent --target railway --env production --status

assembly-line auth codex runs codex login --device-auth, then verifies it with codex login status. The remote Codex CLI performs the browser/device exchange and writes its own cache below /data/codex; Assembly Line never reads, copies, logs, syncs, or records the OAuth token in environment variables, provider secret stores, artifacts, or deployment receipts.

Railway, Docker, Fly, and VPS satisfy the same contract:

  • Railway creates or reuses a service volume mounted at /data and executes the Codex commands with Railway SSH.
  • Docker creates or reuses openeve-<agent-slug>-data, mounts it at /data for the stable served container, and uses a one-shot interactive container with the same image and volume for login or status. Build-only deploys can therefore authenticate without starting the HTTP service.
  • Fly creates or reuses the app-scoped openeve_data volume in the effective region, writes the /data mount to fly.toml, and uses Fly SSH. Codex auth requires a deployed single-Machine app because Fly volumes are Machine-local.
  • VPS uses the active immutable image plus the agent's dedicated /data volume, environment, and data network for one-shot device login/status containers.

Credential trust boundary​

The runtime service and its persistent volume are part of the credential trust boundary. Restrict provider and shell access, treat snapshots/backups as credential-bearing, and revoke the session when the deployment is retired. Do not upload a developer-machine auth.json; direct device authentication avoids duplicating that local credential.

For a personal ChatGPT subscription, use this route only on a trusted runtime whose persistent cache was authenticated with assembly-line auth codex. Do not copy a personal CODEX_HOME into a shared image or expose it as a service credential. The Codex CLI persists the app-server thread there so an Assembly Line approval, human-input, connection, or crash-recovery resume can continue the same model thread.

For headless automation, OpenAI documents managed CODEX_ACCESS_TOKEN use for eligible Enterprise workspaces. For general hosted application traffic, use openai/* with OPENAI_API_KEY instead. See OpenAI's Codex authentication guide and plan availability.

printenv CODEX_ACCESS_TOKEN | codex login --with-access-token

The deployment image must include a current codex executable; an outdated CLI fails with an update hint.

ChatGPT rate-limit and credit snapshots are captured only as observational provider receipts; they never gate a request. API-key mode is billed through the OpenAI API account and can be reconciled to the OpenAI organization Usage/Costs APIs. Codex app-server reports per-turn tokens but not a per-turn dollar charge, so exact cash remains unavailable at run grain unless the provider adds such a receipt. See Usage Accounting And Observability.

Migrations​

If the build artifact contains migration files under .assembly-line/migrations, hosted deploys require a migration runner:

assembly-line deploy agent \
  --migration-command ./scripts/run-assembly-line-migrations

The migration process receives:

  • ASSEMBLY_LINE_ARTIFACT_ROOT
  • ASSEMBLY_LINE_AGENT_REVISION
  • ASSEMBLY_LINE_DEPLOY_ENV
  • ASSEMBLY_LINE_MIGRATION_FILES

The Postgres adapter records schema migrations with id, checksum, description, package version, and applied time.

Preflight​

Use dry-run deploys before publishing:

assembly-line deploy agent --target railway --dry-run

Preflight requirements are inferred from:

  • gateway.ts adapters.
  • Static connection files.
  • Static channel files.
  • The model provider prefix in agent.ts.
  • Artifact deployment requirements such as persistent directories and remote execution.

For every non-local target, planning refuses sandbox: adapter("local") unless ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_IN_PRODUCTION=true explicitly acknowledges the unconfined execution risk. Local state and local blob storage are allowed, but the plan warns that container replacement can lose them.

Static stdio MCP connections launch their configured binary on the Node runtime host and are closed during graceful runtime shutdown. The command is never routed through a shell. Ensure the binary, working directory, and OS permissions exist on every replica. In particular, @assemblyline-agents/peekaboo only operates when the Node host itself is a permitted macOS 15+ machine; deploying that agent to a Linux container does not create remote access back to the developer's Mac. Peekaboo declares local and darwin host requirements, so an incompatible deployment plan is rejected before publishing and a non-macOS runtime rejects the connection before process launch. Remote computer access uses the separate @assemblyline-agents/computer-use connection, Assembly Line Builder's Mac Computer Host, and an end-to-end encrypted relay; it is not a mode of this stdio plugin. The hosted runtime requires ASSEMBLY_LINE_COMPUTER_USE_BINDING; a self-hosted relay can also set ASSEMBLY_LINE_COMPUTER_USE_RELAY_URL. See Remote Computer Use.

Model provider env keys:

PrefixEnv
anthropic/ANTHROPIC_API_KEY
cerebras/CEREBRAS_API_KEY
deepseek/DEEPSEEK_API_KEY
fireworks/FIREWORKS_API_KEY
google/GOOGLE_API_KEY
groq/GROQ_API_KEY
mistral/MISTRAL_API_KEY
openai-codex/Codex CLI login; no model API-key env requirement
openai/OPENAI_API_KEY
openrouter/OPENROUTER_API_KEY
together/TOGETHER_API_KEY
xai/XAI_API_KEY

LiveKit voice-call tools and connections use LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET. Outbound phone-call defaults can also use LIVEKIT_OUTBOUND_TRUNK_ID and LIVEKIT_VOICE_AGENT_NAME. defineLiveKitConnection() contributes the required LiveKit env to preflight.

Provider setup details are in Adapters.

Production Checklist​

  • Use a stable agent.id for agents with learned skills or durable state.
  • Use Postgres for hosted durable state.
  • Set ASSEMBLY_LINE_CONNECTION_STORE_SECRET or ASSEMBLY_LINE_SECRET if production uses file-backed connection credential stores instead of Postgres-backed stores.
  • Use S3-compatible blob storage or R2 for attachments and artifacts.
  • Use Docker or a hosted sandbox for untrusted code and shell work. The production Node helper refuses the local sandbox unless ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_IN_PRODUCTION=true.
  • Use deploy --dry-run and resolve all required preflight items.
  • Keep model provider, channel, database, blob, sandbox, and deploy credentials out of the agent folder.
  • For openai-codex/*, install the Codex CLI in the runtime image and follow the subscription/managed-token boundary above; never bake a personal CODEX_HOME into a shared image.
  • Set ASSEMBLY_LINE_ADMIN_TOKEN or provide a host auth policy before production boot.
  • Set ASSEMBLY_LINE_ENABLE_API_RUNS=true only when authenticated API-created runs are intended.
  • Set ASSEMBLY_LINE_BASH_TOOL_MODE=approval or disabled for agents that should not get direct shell access. The default is enabled in every runtime mode. Embedders can gate any tool by name via RuntimeOptions.coreToolPolicy (e.g. { write: "disabled" }).
  • Set TELEGRAM_WEBHOOK_SECRET for Telegram channels and either PHOTON_WEBHOOK_SIGNING_SECRET/PHOTON_SIGNING_SECRET or PHOTON_INGRESS_TOKEN/PHOTON_WEBHOOK_BEARER_TOKEN for Photon channels before production boot.
  • Bound run concurrency (ASSEMBLY_LINE_MAX_CONCURRENT_RUNS; createProductionRuntimeOptions defaults to 16 outside dev) and enable ingress rate limiting (ASSEMBLY_LINE_INGRESS_RATE_LIMIT, ASSEMBLY_LINE_RUNS_RATE_LIMIT) on internet-facing hosts.
  • Set ASSEMBLY_LINE_TRUST_PROXY=true when the host sits behind a reverse proxy or load balancer so rate limits key on the real client address.
  • Point load-balancer readiness at GET /readyz (drains to 503 during shutdown) and liveness at /health; deliver SIGTERM for deploys so in-flight runs drain within ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS.
  • Verify /health, /readyz, authenticated /manifest, channel routes, and authenticated /runs after deploy.
  • Make side-effect tools idempotent and approval-gated where appropriate.