Skip to main content

Assembly Line Framework Guide

Assembly Line is a filesystem-first framework for durable AI agents. An agent folder declares what the agent is; the compiler turns that folder into a manifest and runtime artifact; the runtime executes runs durably through pluggable adapters.

This page explains the concepts and contracts. For command tables, the artifact tree, the HTTP API, and deploy targets, see Runtime And Deployment. For every define* shape and ASSEMBLY_LINE_* variable, see the Configuration Reference.

Contents:

Agent Folder Convention​

Only instructions.md and agent.ts are required.

agent/
  instructions.md
  agent.ts
  context.ts
  gateway.ts
  skills/
  tools/
  channels/
  automations/
  automation-handlers/
  connections/
  evals/
  sandbox/
  subagents/
  hooks/
  resolvers.ts
  instrumentation.ts

lib/ and playbooks/ are not reserved Assembly Line conventions. App helpers can live wherever the app normally keeps source code.

agent.ts​

agent.ts exports defineAgent({ ... }).

import { defineAgent } from "@assemblyline-agents/core";

export default defineAgent({
  model: "openai/gpt-5.4-mini",
  reasoning: "medium",
  description: "A portable Assembly Line agent.",
  defaultTools: ["echo"]
});

The compiler statically extracts the model, reasoning level, description, default tools, and context declaration. Tool implementations, secrets, databases, deployment providers, and blob stores do not belong in agent.ts.

reasoning is optional and accepts "off", "minimal", "low", "medium", "high", or "xhigh". Assembly Line defaults to "medium" for Pi model runs when this field is omitted.

Assembly Line reads manifest-facing config through the TypeScript AST. This applies to agent.ts, gateway.ts, tools, channels, schedules, trigger handlers, connections, sandbox definitions, and subagent agent.ts files. Manifest-affecting values must be statically readable: a default-exported define*({ ... }) helper call, an exported identifier that resolves to that call or object, top-level const indirection for literal strings, arrays, and objects, shorthand properties, imported helper aliases, as const, and satisfies are supported. The compiler never executes config code during manifest generation, so dynamic expressions cannot affect the manifest.

Agent Engine​

Pi is the primary engine. The model loop is pi — open source, minimal, and proven — adapted in @assemblyline-agents/pi and constructed directly by the runtime. Agents never select a harness: agent.ts has no harness: slot (declaring one fails validation with harness-not-configurable). The Node host does have a built-in preview provider route: selecting openai-codex/* uses @assemblyline-agents/codex and OpenAI's official Codex app-server, authenticated by the Codex CLI. Subagents run through Pi as well; their definitions can narrow model, tools, workspace, and connections but cannot select another engine.

The runtime talks to pi only through the internal AgentHarness contract in @assemblyline-agents/core. This seam is not a public extension point, but it is deliberately engine-shaped and load-bearing: the runtime imports no pi types, continuation state is opaque JSON owned by the engine adapter (so persisted runs never depend on pi internals), and all durability (tool records, approvals, checkpoints, events, usage, spans) stays in the runtime. Embedders and tests can replace the engine for every run through RuntimeOptions.agentHarness, or register host-owned provider-prefix routes through RuntimeOptions.modelHarnesses — the durability test suite drives scripted engines through exactly this seam. A sibling seam, RuntimeOptions.toolStubs, replaces named tool executions with canned implementations (approval gates and tool-call recording still apply); the eval runner's case-level mocks build on it. The resolved model spec is stored with the continuation so paused and recovered runs resume through the same route. Subagents reuse the Pi loop, including follow-ups via persisted continuations; there is no public subagent-harness extension point.

Tool batches run in parallel by default: when the model emits several tool calls in one response, parallel-safe tools execute concurrently. Pause-capable tools — approval-gated tools, ask_question, connection tools, and the deferred tool_call bridge — carry execution: "sequential" on their descriptors, which serializes any batch that contains one. A pause stops the batch wherever it happens: later sequential calls are skipped with an explicit { skipped: true } result instead of executing after the run parked, and a dynamic pause inside a parallel batch (a custom tool calling ctx.askQuestion) still ends the active loop execution without another model request.

The engine also reports token deltas (response_delta events). They are ephemeral: the runtime fans them out to live run-stream subscribers (runtime.subscribeRunStream(runId, listener), or the Node host's GET /runs/:id/stream SSE endpoint) and never writes them to the durable event log.

context.ts​

Agents do not need context.ts. When it is absent, Assembly Line uses defaultContext().

defaultContext() builds a Flue-shaped context bundle with trusted instructions, active event, bounded recent history, memory filesystem shape, file manifest, current attachments, a compact skill and capability catalog, always-on tool summaries, channel metadata, and trust boundaries. Deferred tool schemas and skill bodies are not injected up front. Files, screenshots, webpages, tool output, search results, memory, and attachments are context, not instructions.

Image and supported video attachments are model-native by default. After attachment intake stores the private bytes, the runtime hydrates bounded PNG, JPEG, GIF, WebP, MP4, MPEG, MOV, and WebM inputs at the harness boundary. Pi sends native image blocks when the selected model advertises image input. For OpenRouter models, Pi verifies the live model metadata and sends complete videos through OpenRouter's native video_url content type only when input_modalities includes video.

Assembly Line never silently substitutes sampled frames for native video. If the selected model or harness does not support video, the turn returns an explicit limitation without making a model request or extracting frames. Frame-based review remains an explicit user-approved FFmpeg operation. The Codex app-server input protocol accepts text and images, but not video, so openai-codex/* follows that explicit limitation path. Image-bearing MCP tool results stay typed rather than being flattened into JSON text, and base64 payloads are omitted from observability content events.

Every runtime model prompt also receives a small stable Assembly Line filesystem contract immediately after instructions.md. It names resource paths and their mutability: /memory is durable memory accessed directly through memory tools by default, /workspace is writable workspace for generated artifacts and modified copies when a sandbox exists, /history is read-only conversation history, and /files is read-only input context whose manifest.json should be inspected before reading file contents. Hosted sandboxes establish /workspace as a physical shell cwd, so core file tools and shell commands may use the same absolute paths. The trusted Local adapter maps those paths onto a temporary host directory; Docker is the local parity path for absolute shell semantics. These paths are projected only when sandbox-backed tools need them; the contract does not inject memory or file contents into every turn.

Prompt layout and prompt caching​

The system prompt is cache-stable by construction: it contains only conversation-invariant content (instructions, the filesystem contract, and compact JSON for the skill index, core tool summaries, capabilities, channels, and trust boundaries) and stays byte-identical across the turns of a conversation. Per-turn data — the active event's channel context, prompt context, trigger target, and attachment metadata — rides on the turn's user message as an OpenEve turn context: block, with the user's text last. Timestamps never enter the model-visible prompt. This keeps provider prompt caches hitting on every follow-up; the per-request cacheReadRatio span attribute on ai.streamText reports cache effectiveness.

Conversation transcript resume and compaction​

After a successful conversation run, the runtime checkpoints the harness's final continuation (conversation.transcript) and stamps a pointer on the conversation record. The next turn in that conversation resumes the real transcript — assistant turns and tool calls intact — instead of rebuilding context from flattened recent history. The pointer is an acceleration layer, never the source of truth: any load, harness-version, or trim failure falls back to the flattened-history path (evented as context.transcript_fallback), and durable conversation messages and /history are unchanged.

Before resume the harness trims the stored transcript in two layers. First, tool-result bodies outside the recent ~20k-token tail are capped with a restorable marker (re-run the tool or read the sandbox file to recover the full output; no model call). Second, when the transcript still exceeds the model context window minus a reserve, older full turns are summarized into a structured context checkpoint (pi-agent-core's summarizer), keeping the recent tail verbatim and never separating a tool result from its call; re-compactions update the previous summary instead of stacking summaries, and the run is notified to persist durable facts under /memory. Compactions are evented as context.transcript_compacted with the pre-compaction token estimate. Stored media (base64 images/video) never replays on conversation resumes. Trimming runs at resume time between runs; mid-run growth is bounded by maxIterations.

Custom context can extend the default:

import { defaultContext, defineContext } from "@assemblyline-agents/core";

export const customContext = defineContext({
  kind: "custom",
  name: "customContext",
  extends: defaultContext({ recentHistory: { maxMessages: 5 } })
});

Context policy is trusted app-runtime code and is recorded in the manifest with source attribution.

gateway.ts​

gateway.ts is the portable stack declaration. It is tiny and declarative:

import { adapter, defineGateway } from "@assemblyline-agents/core";

export default defineGateway({
  deploy: adapter("railway"),
  runtime: adapter("node"),
  state: adapter("postgres"),
  blob: adapter("r2"),
  sandbox: adapter("daytona"),
  scheduler: adapter("gateway")
});

Runtime host, state, blobs, sandbox, scheduler, connections, and observability are independent choices. Deploy adapters host the runtime process; they do not force a state/blob/sandbox provider.

Like agent.ts, gateway.ts is parsed as TypeScript syntax rather than executed. Use statically readable defineGateway({ ... }) declarations, adapter("kind"), or known provider helper calls such as railwayDeploy(), vpsDeploy(), neonPostgres(), railwayPostgres(), supabasePostgres(), r2Blob(), and dockerSandbox(). Dynamic expressions are ignored unless they resolve to top-level literals the compiler can validate.

Scheduler choices are explicit. adapter("local") starts an in-process polling loop for development or single-process hosts. adapter("gateway") does not start a loop; a cloud scheduler, platform cron, or gateway worker calls the runtime tick endpoint or runDueAutomations(). adapter("postgres") starts the same polling loop but expects Postgres state so multiple workers coordinate through shared idempotency and dynamic-automation leases.

Production state is Postgres. Neon is the default hosted path; Railway, Supabase, and local/custom Postgres are presets, and the adapter is adapter("postgres") in every case because each provider exposes standard Postgres. On Railway deploys, railwayPostgres() provisions or reuses a managed Railway Postgres service and wires its private DATABASE_URL reference before publishing the runtime. Blob storage is S3-compatible, with R2 as a first-class preset/wrapper. Railway, Docker, Fly, and generic VPS deployment are supported through their official provider packages. All satisfy the same artifact-level persistent-storage and remote-execution contract, and deploy choice does not imply a state/blob/sandbox vendor. See Deploy Targets.

See Adapters for the current adapter list, helper functions, and provider environment variables.

Tools​

Each file in tools/ becomes one model-facing tool. The filename is the tool name. Tool descriptors include name, description, input schema, optional output schema, approval policy, and model-output projection.

Tool code runs in the trusted app runtime by default. It only uses the sandbox through ctx.getSandbox().

Assembly Line starts model runs with exactly the core harness tools: read, write, edit, delete, list, grep, bash, ask_question, spawn_subagent, load_skill, tool_search, tool_describe, and tool_call. Authored tools, plugin tools, MCP tools, and connection tools are deferred behind tool_search -> tool_describe -> tool_call; the runtime still logs, approves, traces, and executes the underlying tool name. bash is enabled by default in every runtime mode unless ASSEMBLY_LINE_BASH_TOOL_MODE=approval or ASSEMBLY_LINE_BASH_TOOL_MODE=disabled is set, or the host passes the equivalent core tool policy (which gates any tool by name).

Every core harness tool is a replaceable slot: an authored tools/<name>.ts with a built-in's name overrides it (spread builtInToolDefaults from @assemblyline-agents/runtime to wrap instead of rewrite), and a disableTool() default export removes it, with unknown names failing the build. See Customizing Agents.

Tools can optionally declare capability metadata, but most apps should rely on inference:

capability: {
  visibility: "deferred",
  execution: "direct",
  namespace: "billing",
  tags: ["invoice", "customer"],
  aliases: ["receivables"]
}

visibility can be auto, always, deferred, skill, or hidden; execution can be auto, direct, sandbox, or both. auto defers to inference. These fields feed deferred discovery metadata. Core harness tools are not configurable by agent authors.

All file, memory, skill, workspace, and artifact work goes through the sandbox filesystem. /history and /files are read-only; /memory, /skills, and /workspace are writable according to policy. The core file tools lazily acquire and hydrate the sandbox for requested paths.

import { approvalRequired, defineTool } from "@assemblyline-agents/core";

export default defineTool({
  description: "Record a note after approval.",
  inputSchema: { type: "object", properties: { note: { type: "string" } }, required: ["note"] },
  needsApproval: approvalRequired("Recording a note is a durable side effect."),
  async execute(input, ctx) {
    await ctx.emit("note.recorded", { idempotencyKey: ctx.idempotencyKey("note") });
    return { recorded: true, note: input.note };
  }
});

toModelOutput can expose a bounded, safe projection while the rich result remains available in the durable tool log. See tools/*.ts for the full field reference.

Skills And Self-Improvement​

Skills live under skills/<name>/SKILL.md. Assembly Line parses skill frontmatter and builds compact skill capabilities in the manifest. Runs see skill names and descriptions up front; the body is loaded on demand through the core model-facing load_skill tool or the host-driven loadSkill(runId, skillName) API. Loading a skill reads from the durable skill store, returns instructions, and does not reveal additional tools. If a sandbox already exists, the selected skill may also be projected into /skills/<name>/SKILL.md for filesystem inspection.

Self-improvement means one thing: the agent authoring and editing its own skills as it learns how to do things. It is configured by selfImprovement in agent.ts and is on by default. Runtime automation and connections are deliberately separate concerns with their own config blocks (dynamicAutomations, dynamicConnections) and their own clearly-named tool APIs — they are not part of "self-improvement." Setting every block off restores fully static, manifest-only behavior.

export default defineAgent({
  // Stable logical identity for durable learned state across revisions/deploys:
  id: "travel-concierge",
  model: "openai/gpt-5.4-mini",
  // The agent learning by writing/editing skills:
  selfImprovement: { writable: true, writeApproval: false },
  // Separate, clearly-named concerns:
  dynamicAutomations: { dynamic: true, approval: false },
  dynamicConnections: { dynamic: false, approval: true, allowedHosts: [] }
});

Skills are a durable folder (the self-improvement surface). On boot, compiled skills/ seed a durable SkillStore, scoped by stable agent.id when configured and tracked by a content hash so redeploys upgrade pristine seeded skills and preserve anything the agent or user changed. Each run puts only the skill index in the prompt: name, description, and path. Full bodies live at /skills/<name>/SKILL.md and are read, edited, created, or deleted with the normal file tools. When selfImprovement.writable is off, /skills writeback is blocked.

Tools reach these concerns through three distinct context APIs — ctx.selfImprovement (skills), ctx.automationManager, and ctx.connectionManager — and the runtime exposes saveSkill, listSkills, dispatchAutomationEvent, runDueSchedules, saveConnectionDefinition, and related methods for host-driven use. Durable stores are provided by the state adapter (Postgres) or fall back to local JSON files under the artifact root. See examples/self-improving-agent and Customizing Agents.

Channels​

Channels normalize platform entrypoints and delivery behavior. HTTP-capable channels declare a route and methods; the compiler emits a route table.

import { defineChannel } from "@assemblyline-agents/core";

export default defineChannel({
  transport: "http",
  route: "/message",
  methods: ["POST"]
});

The default raw HTTP message fallback is dev-only unless the Node host has authenticated the request. Production provider routes should use a helper or custom normalizeHttp() that verifies the provider request before accepting a turn.

Channel files own platform event semantics, not durable state schema or sandbox lifecycle.

Assembly Line ships one-line helpers for Slack, Discord, Telegram, Microsoft Teams, and Photon-style agent communication channels. Provider helpers preserve the same channel contract: verify the incoming event, normalize to ChannelTurn, use provider delivery IDs for idempotency where available, and send replies through provider APIs. GitHub is available separately as an authenticated connection package for repository tooling; it is not an inbound channel.

For retried webhook providers, channel modules can return kind: "accepted" to acknowledge the HTTP request before the model turn completes. Use the provider's stable delivery id as the idempotency key. For Slack Events API channels, verify the request signature, normalize the event, return a 2xx response immediately, and set idempotencyKey to Slack's event_id so retries do not start duplicate turns.

Accepted turns enter a durable FIFO mailbox keyed by stable agent identity and normalized conversation id. Only one turn in a conversation can be running or parked at a time; later messages wait. Different conversations lease independently and consume the ordinary global run-concurrency budget in parallel. A parked approval, input, connection, or suspended run keeps its conversation closed until it reaches a terminal state. This rule is enforced by the runtime after normalization, so channel modules define conversation boundaries but do not implement their own queues.

Channel modules can also augment context after ACK and before default context bundle construction. Slack uses this hook to bridge a bounded set of recent Assembly Line Slack conversations for the same user, so a user can follow up from a thread in a DM without merging every Slack surface into one transcript or fetching workspace-wide Slack context.

Channel modules can also export startIngress(ctx, emit) for long-lived provider listeners. The Node host starts these listeners beside the scheduler, restarts provider-owned listeners through adapter code, and stops them on server shutdown. emit.accepted({ turn, idempotencyKey, idempotencyScope }) feeds Gateway-style events into the same durable, idempotent run path used by accepted HTTP webhooks. Discord uses this for Gateway DMs, mentions, and thread messages.

Connections​

Connections declare required external capabilities, scopes, subject mapping, and whether they are required. Raw secrets and refresh tokens stay outside the agent folder and model context. Live MCP, OpenAPI, HTTP, and sandbox CLI connection tools are searched through tool_search; concrete schemas are not visible until tool_describe selects them. MCP supports request-policy-governed Streamable HTTP and static, directly spawned stdio processes. Sandbox CLI connections preserve the same connection policy and scoping while invoking reviewed arguments in the active run sandbox, where short-lived materialized credentials may be projected when explicitly configured. Dynamic connections are URL-only and cannot launch host or sandbox processes. Tools and channels consume connection handles from runtime context.

Dynamic connections are a separate, gated concern. With dynamicConnections.dynamic on (off by default), a tool calling ctx.connectionManager can persist a new MCP/HTTP/OpenAPI connection into a ConnectionDefinitionStore; stored definitions merge into the connection registry and become discoverable through tool_search. Credentials route to the existing encrypted grant store through host APIs or authorization flows, never model-visible tool input, the agent folder, sandbox, or model context. Saving is approval-gated and restricted to dynamicConnections.allowedHosts; remote tool descriptions are treated as untrusted data. Agents never author trusted tool code — a CLI a user hands the agent is run in the sandbox and wrapped in a skill; net-new typed tools remain a reviewed source change.

Automations​

Automations declare durable work started by either a schedule trigger or a normalized provider event. Schedule triggers use the existing cron dispatcher. Event triggers enter through runtime.dispatchAutomationEvent(), a verified channel normalizer, a long-lived channel listener, or authenticated POST /assembly-line/automations/events. Both paths reserve stable idempotency keys, honor run capacity, and execute the same target and lifecycle contract. Time-based triggers are dispatched by runtime.runDueAutomations() and /assembly-line/automations/tick.

Trusted prepare/finalize code lives in automation-handlers/ and is referenced through lifecycle.handler. Dynamic time-based automations use dynamicAutomations and ctx.automationManager; dynamic event subscriptions remain reviewed source because they own provider authentication and subscription policy. See automations/.

Sandbox​

Sandbox files declare the agent computer selection. The core contract supports file reads/writes, shell execution, and optional provider snapshots. The local adapter is for trusted dev/test work; Docker is the supported local isolation baseline; Daytona, E2B, and Modal are supported hosted sandbox choices. Sandboxes are acquired lazily when a sandbox-backed tool or capability asks for one.

Hosted sandbox paths share one physical, versioned namespace rooted at /workspace. Providers validate shell cwd and file-API agreement after create, connect, and wake; runtime manifests fence older contract versions from reuse. Providers reject traversal and return canonical absolute paths such as /workspace/report.txt from listings. /runtime is retired and rejected. Local sandbox execution emulates the logical namespace in a temporary host directory and is trusted developer or self-managed execution only; use Docker, Daytona, or E2B when untrusted code needs an isolation boundary.

Snapshots are a scarce infrastructure checkpoint, not the normal turn persistence mechanism. Production runs should use Assembly Line state/blob sync for memory, messages, tool traces, and /workspace artifacts, and keep fresh run sandboxes ephemeral. The default snapshot policy is never, so a hosted sandbox run does not create a remote snapshot unless the agent explicitly opts in.

Sandbox files may declare a snapshot policy:

export default defineSandbox({
  adapter: "daytona",
  image: "node:22",
  snapshot: {
    mode: "manual",
    retainLast: 3,
    reason: "operator-requested checkpoint"
  }
});

Supported modes are never, manual, on_failure, and always. manual only captures when run metadata includes an explicit sandbox snapshot request. always is intended for short-lived debugging or controlled checkpoint jobs, not chat turns. ASSEMBLY_LINE_SANDBOX_SNAPSHOT_MODE, ASSEMBLY_LINE_SANDBOX_SNAPSHOT_RETAIN_LAST, and ASSEMBLY_LINE_SANDBOX_SNAPSHOT_REASON can override policy at runtime.

Compiler Output​

assembly-line build emits the .assembly-line/ artifact; the canonical file tree and per-file descriptions live in Runtime And Deployment → Build Artifact. Two contracts matter conceptually:

  • agentRevision is a deterministic hash of source paths, source hashes, and relevant config. Rebuilding unchanged source produces the same revision; changing source changes it.
  • The manifest contains instructions, agent definition, context policy, gateway config, tools, capabilities, skills, channels, schedules, trigger handlers, connections, sandbox declarations, subagents, instrumentation source, route table, schedule metadata, preflight requirements, validation results, package versions, and source hashes.

For the CLI commands that produce and run the artifact, see the CLI reference.

Durability Guarantees​

Persistence model​

The runtime persists the run before execution starts. It records context bundle creation, sandbox acquisition, harness/model events, tool requests, approval pauses, tool execution, sandbox events, memory sync, delivery obligations, delivery sends, and final status. Replay reconstructs a run from durable events, current recovery checkpoints, tool calls, and delivery records.

Checkpoints are recoverability state, not the permanent audit log. Active runs keep a bounded tail of overwrite-style harness.continuation recovery checkpoints per run/name while preserving pause and step data needed by the live run. Terminal-run checkpoint cleanup is explicit operator maintenance: assembly-line checkpoints compact <agentRoot> dry-runs by default and --apply deletes only checkpoint rows older than the configured TTLs. Events, messages, tool calls, delivery obligations, idempotency keys, files, memory, schedules, and approvals remain separate durable records. Large checkpoint payloads are gzip-compressed into the configured blob adapter with SQL retaining a small content-addressed pointer/hash/size record, and replay/resume hydrates them transparently. Checkpoint cadence, retention, and TTL knobs are in the model loop reference.

HITL resume​

Human approval, ask_question, and deliberate suspension states are represented durably, and resuming them re-enters the model loop. resumeApproval(runId) executes the approved tool and feeds its result back to the harness as the pending tool result, so the model continues reasoning from where it paused; resumeInput(runId, answer) does the same with the human's answer, so the agent that asked the question actually sees it; resumeSuspended(runId) restores the latest compatible continuation. If no usable continuation checkpoint exists (pre-upgrade runs, or a harness that returned none), HITL resume degrades to completing the run directly and records a harness.resume_degraded event so the fallback is observable. Every resume path first claims a per-generation idempotency key, so a double-clicked approval, a double-submitted answer, a replayed OAuth callback, or a repeated suspended-run resume cannot execute twice across replicas. Final delivery is an idempotent delivery obligation.

Durable steps​

Authored tools and trigger handlers can wrap expensive or side-effect-adjacent substeps in ctx.step(key, fn). A completed step stores its JSON result as a durable_step.completed checkpoint; if the same run reaches the same key again, the runtime emits durable_step.replayed and returns the stored result instead of running fn. This is intentionally scoped to one run and uses the existing checkpoint store rather than a workflow table. If a process dies inside fn before completion is checkpointed, the body may run again, so external writes still need ctx.idempotencyKey(...) or a destination-level dedupe key.

Delivery queue​

A delivery only reports success when a real channel sender ran (or when the channel legitimately has no sender: dev-no-sender in dev mode, local-no-sender for local channels that declare no required env). If a channel module fails to load at delivery time, the delivery fails as retryable instead of being silently marked sent, and the failure is logged.

Final delivery is backed by a durable queue, mirroring sandbox sync. On queue-capable state adapters the completion path creates the obligation already leased by the inline sender (status sending with a lease token), so a delivery worker on this or any other replica can never lease and send the same obligation concurrently; success settles through the token-enforced completeDeliverySend, and a crash mid-send leaves the row leased until expiry, after which the worker retries it. The in-process fast path retries transient failures first (ASSEMBLY_LINE_DELIVERY_RETRY_ATTEMPTS, default 2). If it still fails retryably and the state adapter supports the queue (in-memory, file, and Postgres all do), the obligation is deferred instead of terminally failed: status back to pending with exponential backoff (nextAttemptAt, attemptCount, lastError), a delivery.deferred event, and the run still completes with deliveryDeferred metadata. A delivery worker (runDueDeliveries() / startDeliveryWorker()) recovers expired sending leases, leases due obligations (Postgres uses for update skip locked, so multiple replicas are safe), re-sends through the channel module using the persisted payload (channels fall back to payload.delivery when the original turn is gone), and settles each attempt with delivery.sent, delivery.retrying, or — after maxAttempts — a terminal delivery.failed with failedAt. Non-retryable failures go terminal immediately. Lease, batch, attempt, and interval knobs are in the delivery queue reference.

Orphan recovery​

Orphaned-run recovery is staleness-guarded and multi-replica safe. Every executing run has a heartbeat (default 30s, ASSEMBLY_LINE_RUN_HEARTBEAT_MS) that bumps updatedAt through the state adapter's touchRun — a guarded, column-scoped write that never touches status and never resurrects a terminal run, so a heartbeat racing a completion cannot clobber it. recoverIncompleteRuns() only touches runs stuck in created/running longer than max(5min, 4x heartbeat) (override with staleAfterMs), and claims each candidate through a run:recovery idempotency key before acting. Recovery policy, in order: runs with delivery.sent complete without re-sending; tool calls requested but never started are cancelled before side effects; tools that were mid-execution at crash time (tool.execution_started without a settle event) are marked failed-as-interrupted (tool.execution_interrupted) and, when the continuation is parked on that tool, the resume hands the model an explicit { interrupted: true } tool result instead of silently re-driving it into re-execution; runs with a persisted harness.continuation (or legacy pi.continuation) checkpoint — written after every model iteration by default — get one in-place resume attempt through the normal harness resume path (run.recovery_resume_attempted, capped, then degrade); runs that reached a model response but not delivery first adopt any existing delivery obligation before creating one (using the same final-delivery idempotency key the original attempt would have used), so crash recovery can never double-send (deliver-after-model-response); everything else is marked failed with a run.failed event rather than being misreported as completed. The sweep runs once at host boot and periodically via startBackgroundWorkers() (default every 60s, ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS).

Resume is checkpoint-restart, not deterministic event replay, and tool execution is therefore at-least-once: a crash in the window between a tool's side effect and the next persisted continuation can lead the model to request the tool again on resume. The runtime narrows this window with per-iteration checkpoints, bounded active checkpoint retention, the interrupted-tool guard above, and ctx.step(...) completed-step replay, but cannot close it for arbitrary external side effects. Authored tools that perform non-idempotent external writes should use the ctx.idempotencyKey(...) helper (or their own dedup key derived from the tool-call id) so a re-executed call is a no-op at the destination. Crashed in-flight model requests are simply re-issued from the last continuation; the cost is extra tokens, never lost state.

runtime.startBackgroundWorkers() starts the delivery, sandbox-sync, conversation-turn mailbox, and orphan-recovery workers, and returns a controller with stop(). The Node host (listenNodeRuntime) wires all of this automatically and each worker has an env kill-switch; see the durability workers reference.

Sandbox sync​

Assembly Line does not acquire a sandbox before every turn. Channel lifecycle events and final text delivery run without sandbox hydration. The first core file or shell tool lazily acquires the sandbox, then hydrates only requested paths or bounded candidate sets:

/memory
/history
/files
/workspace

The local adapter materializes projected paths as normal files under its temporary sandbox root. /memory, /skills, and /workspace remain writable; /history and /files are marked read-only after hydration. Minimal hydration writes indexes, bounded history, file manifests, runtime context, and selected resources instead of every skill or memory document. Agents should write generated artifacts, transformed files, reports, scripts, and modified copies of input files under /workspace. Use Docker rather than Local when a development test depends on physical absolute /workspace shell paths.

Sandbox acquisition is provider-neutral and ordered: connect to the current live sandbox for the agent/conversation/project key, wake the warm sandbox recorded in Postgres, then create a fresh sandbox. Reconnect, provider lookup, and snapshot restore are skipped when the recorded filesystem-contract version is obsolete; versioned provider names prevent a replacement from colliding with the old resource. The foreground path creates a durable session row and sync obligation before mutating side effects, but final delivery and the next message never wait for filesystem scanning or canonical writeback.

Sandbox sync is a durable queue, not just in-process cleanup. Mutating sandbox tools enqueue a sandbox_sync_job before the side effect. The worker leases jobs, retries with backoff, recovers expired leases and dirty sessions through recoverIncompleteRuns(), writes /memory/** to durable memory documents, writes/deletes /skills/*/SKILL.md through the durable skill store, stores /workspace/** blobs and file catalog records, and records blocked failures as blocked_requires_operator without undoing final delivery. Dirty sandboxes are retained or paused until writeback completes. Operators can call sandboxSyncDiagnostics(), inspectSandboxSyncJob(jobId), and retrySandboxSyncJob(jobId) for counts, due/leased/expired/blocked status, manifests, attempts, and manual retry. Inline mode, lease, batch, and attempt knobs are in the sandbox sync reference.

Security boundaries​

Security boundaries are intentionally conservative. Memory, history, files, webpages, search results, and tool output are untrusted context, not instructions. /files and /history are read-only projections; write generated or transformed outputs under /workspace. Skills are trusted instructions from the durable skill store and are loaded one selected skill at a time. Sandboxes remain isolated execution environments, and projection is an allowlisted copy of selected resources, not ambient host filesystem access. See Architecture for the full trust-boundary map.

Observability​

Agent-authored hooks/ modules subscribe to the same durable event stream (observe-only, firing after each event persists; a thrown handler is logged and swallowed), and resolvers.ts declares per-run model/instructions/tool-set choices that are recorded as run.capabilities_resolved events before they apply — see Customizing Agents.

Agent Runs-style observability is available without instrumentation.ts through the run, event, tool, checkpoint, delivery, usage, subagent, and grouped run query contracts. The Node host exposes GET /runs, GET /runs/:id, GET /runs/:id/events, GET /runs/:id/timeline, and arbitrary-period GET /usage queries so a dashboard or CLI can inspect persisted runs without replaying provider calls. Usage is behavior-neutral: only provider-reported or reconciled cash is stored, unavailable values remain null, and provider aggregate control totals are compared rather than added to transactions. Observation failures are warned but do not gate model execution or delivery.

Every terminal run outcome becomes a structured log line at one choke point, no matter which code path (loop, workers, recovery) produced it: run.completed/run.cancelled at info, run.failed at warn with its machine-readable reason. Failed runs also carry terminalReason/terminalError on the run record itself, so "list failed runs by reason" needs no event-log scan. Model retries (model.request_retried) and errors that escape a run (run.execution_error, left non-terminal for crash recovery) are durably evented and logged with the same discipline. Response content never appears in logs — only sizes; content capture remains a separate, opt-in telemetry concern.

Optional OpenTelemetry-shaped sinks receive an Eve-style span hierarchy. Each completed turn emits ai.assembly-line.turn as the parent span, with child spans for model steps (ai.streamText), tool calls (ai.toolCall), subagents, sandbox commands, memory sync, and delivery sends. The assembly-line.run and assembly-line.tool span names are also emitted for compatibility. Spans carry trace/span IDs plus agent revision, run, session/conversation, turn, channel, model, tool, sandbox, delivery, status, usage (input/output/cached tokens, cost), and error attributes where available.

Telemetry is configured in agent/instrumentation.ts — the single home, auto-discovered and run once at startup. @assemblyline-agents/otlp provides createOtlpSink / createOtlpSinkFromEnv to export these spans over OTLP/HTTP as OpenTelemetry GenAI semantic conventions to any OTLP backend (Langfuse, Phoenix, Grafana, Honeycomb, ...); see Customizing Agents → Observability.

If instrumentation.ts exports defineInstrumentation({ setup }), the runtime runs setup({ agentName, manifest, env }) before the first agent turn. recordInputs, recordOutputs, captureContent, and functionId describe capture/export preferences; Assembly Line defaults to usage-level capture (token/cost/model metadata, no message bodies) and bounded previews rather than full payloads. If setup() returns a telemetry sink, the runtime uses it unless a host supplied telemetry programmatically. Exported telemetry remains supported as the compatibility fallback.

State And Blob Adapters​

The state contract stores runs, run events, checkpoints, tool calls, delivery obligations, and replay data. The Postgres package provides migrations plus a driver-neutral query(sql, params) adapter. The Node production host wires that adapter to DATABASE_URL connections through pg; Neon, Railway, Supabase, local Postgres, and custom Postgres use the same schema. The schema covers agents, revisions, conversations, messages, runs, run events, checkpoints, tool calls, approval gates, delivery obligations, schedules, schedule runs, memory file indexes, file catalog records, sandbox leases, sandbox snapshots, idempotent usage receipts, exact micro-dollar aggregates, run queue entries, and idempotency keys.

Postgres memory search uses indexed full-text search for keyword/exact/hybrid modes. Semantic search is optional and needs a MemoryEmbeddingProvider: embedding-backed search turns on automatically when a provider is configured only when the state adapter reports that its vector backend is available, and ASSEMBLY_LINE_MEMORY_EMBEDDINGS_ENABLED can disable the feature. Postgres enables that capability only when optionalMigrations includes 003_openeve_memory_embeddings_pgvector (or is true); the database must provide pgvector. Run runMemoryEmbeddingBackfill() after enabling it to populate stale or missing embeddings. Deployments without the optional backend keep deterministic full-text and portable lexical fallback behavior without querying pgvector tables.

Postgres migrations are recorded in openeve_schema_migrations with id, checksum, description, package version, and applied time. PostgresStateAdapter.migrate() is idempotent and rejects checksum drift; planMigrations() reports pending/applied/skipped-optional/checksum-mismatch state without applying SQL. Existing memory file indexes can be promoted into state-backed memory documents with backfillMemoryDocumentsFromFileIndexes() when the blob adapter can read the indexed blob keys.

The blob contract stores context bundles, attachments, extracted text, generated artifacts, and sandbox sync bundles. The S3 package implements the blob contract against S3-compatible storage and ships R2/AWS/MinIO-style helpers. The R2 package is a compatibility wrapper and in-memory test bucket.