Troubleshooting

Common failures, what they mean, and how to fix them. Setup steps live in Getting Started; every environment variable referenced here is defined in the Configuration Reference.

Quick Index

Error	Section
assembly-line: command not found	CLI Errors
Unknown command: <cmd>. Run "assembly-line help" for usage.	CLI Errors
Agent root not found: <path>	CLI Errors
Hint: the model prefix in agent.ts decides the required env var ...	Model And Provider Key Errors
Model catalog unavailable.	Model And Provider Key Errors
Production OpenEve Node runtime requires OPENEVE_ADMIN_TOKEN ...	401, 403, And 429 From The HTTP API
401 admin_auth_required / 403 admin_auth_invalid	401, 403, And 429 From The HTTP API
403 API run creation is disabled.	401, 403, And 429 From The HTTP API
429 Rate limited. / 429 Run capacity exhausted. Retry later.	401, 403, And 429 From The HTTP API
File-backed connection credential stores require OPENEVE_CONNECTION_STORE_SECRET ...	Connection Store Secrets
<bin> not found. Install it, or pass the matching --*-bin flag	Deploy Preflight Failures
Migration files are present, but no migration runner was configured.	Deploy Preflight Failures
Failed to load plugin package ...	Deploy Preflight Failures
403 scheduler_unauthorized	Webhooks And Scheduled Runs Rejected
Production provider ingress requires webhook authentication: ...	Webhooks And Scheduled Runs Rejected

Install And Build Failures

• pnpm install fails or picks the wrong pnpm. The repo pins pnpm@10.x through the packageManager field. Use Corepack instead of a globally installed pnpm: corepack enable, then corepack pnpm install.
• Engine errors during install. Assembly Line requires Node >=22.19.0 (declared in engines). Check node --version and upgrade before reinstalling.
• pnpm assembly-line fails with a module-not-found error. The CLI runs from built output; run pnpm build first. This also applies after pulling changes that touch any packages/*/src.
• Typecheck or tests fail on a fresh clone. Run in order: pnpm install, pnpm build, pnpm check, pnpm test.

CLI Errors

• assembly-line: command not found. The global assembly-line binary comes from the published @assemblyline-agents/sdk npm package. When it is not installed — including whenever you work from a source checkout — run the same commands through the repo workspace instead: pnpm assembly-line <command> resolves the built @assemblyline-agents/cli binary (run pnpm build first).
• pnpm assembly-line prints nothing useful. Run pnpm assembly-line help for the full command list or pnpm assembly-line help <command> (or pnpm assembly-line <command> --help) for per-command flags.
• Unknown command: <cmd>. Run "assembly-line help" for usage. Check the spelling against pnpm assembly-line help; commands are init, add, validate, manifest, build, dev, run, eval, serve, channels, checkpoints, auth, runs, agent, deploy, models, and help.
• Agent root not found: <path>. The positional argument (or --root) must point at an agent folder. Run assembly-line init <path> to scaffold one.

Model And Provider Key Errors

• A model run fails before the first response. The model prefix in agent.ts decides the required env var: openai/* needs OPENAI_API_KEY, anthropic/* needs ANTHROPIC_API_KEY, and so on (full table in Runtime And Deployment). The CLI prints the same hint when a provider auth error reaches the top level: Hint: the model prefix in agent.ts decides the required env var (e.g. openai/* needs OPENAI_API_KEY).
• openai-codex/* says the Codex CLI is missing. Install a current Codex CLI in the same environment that runs the Node host, then run codex login.
• openai-codex/* reports an authentication failure. Run codex login status, then codex login if needed. This prefix does not use OPENAI_API_KEY; it uses the CLI-owned ChatGPT session. Assembly Line does not read or copy the OAuth cache.
• openai-codex/* says dynamic tools or the experimental API are not supported. Update the Codex CLI. The preview bridge requires app-server's dynamic-tool protocol and deliberately does not fall back to a private HTTP endpoint.
• validate warns unknown-model. The model in agent.ts is not in the local model catalog. It is a warning, never an error — custom providers still work at runtime — so first check for typos. pnpm assembly-line models lists the catalog (--provider openai filters); --no-model-check skips the check entirely.
• assembly-line models prints Model catalog unavailable. The catalog comes from @assemblyline-agents/pi; make sure the workspace is built (pnpm build). When the catalog cannot load, validate silently skips the model check rather than failing.

Build Artifacts And .assembly-line Churn

• A command fails while rebuilding an example's .assembly-line/ directory. Point the artifact somewhere disposable with --out /private/tmp/openeve-minimal (supported by build, run, dev, serve, and deploy), or delete the generated .assembly-line/ directory and rebuild.
• Example artifacts pile up. pnpm clean:artifacts removes ignored examples/**/.assembly-line (and legacy examples/**/.openeve) directories; pnpm clean removes package dist/ output; pnpm clean:tmp removes openeve-* temp directories.
• Generated files show up in git status. .assembly-line/, dist/, node_modules/, and env files are local build output and should stay out of commits.
• A rebuild did not restart dev --watch. Restarts only happen when the agent revision changes; the CLI prints Rebuilt: no manifest change; keeping the running server. for cosmetic edits. While the folder is invalid, the previous server keeps serving and the CLI prints the validation issues until the folder is valid again.

401, 403, And 429 From The HTTP API

• Production boot fails with Production OpenEve Node runtime requires OPENEVE_ADMIN_TOKEN or a host-provided auth policy to protect control-plane routes. Set ASSEMBLY_LINE_ADMIN_TOKEN (or pass an auth policy from host code). Local assembly-line serve runs in dev mode, where inspection endpoints are open; a deployed artifact boots in production mode.
• 401 admin_auth_required / 403 admin_auth_invalid. /manifest, /routes, and the /runs* inspection endpoints require Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN> in production.
• 403 API run creation is disabled. Production POST /runs is off by default. Set ASSEMBLY_LINE_ENABLE_API_RUNS=true and authenticate with the admin token; leave it off unless API-triggered runs are intended.
• 429 { "error": "Rate limited." }. Ingress rate limiting is enabled (ASSEMBLY_LINE_INGRESS_RATE_LIMIT / ASSEMBLY_LINE_RUNS_RATE_LIMIT or the host rateLimit option). The Retry-After header says when to retry.
• 429 Run capacity exhausted. Retry later. The run concurrency cap (ASSEMBLY_LINE_MAX_CONCURRENT_RUNS, production default 16) rejected a direct brand-new run such as POST /runs or a legacy synchronous custom channel. Accepted provider webhooks are durably queued per conversation and do not return this capacity error.
• Load balancer keeps routing during deploys. Point readiness at GET /readyz — it returns 503 { "draining": true } during graceful shutdown — and liveness at /health or /healthz, which stay 200. Deliver SIGTERM so in-flight runs drain within ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS (default 30s); a second signal exits immediately.

Connection Store Secrets

File-backed connection credential stores are encrypted, and production boot validates the secret:

• File-backed connection credential stores require OPENEVE_CONNECTION_STORE_SECRET or OPENEVE_SECRET outside dev mode. Set one of the two, or use Postgres state (the Postgres adapter implements the grant stores directly and needs no file secret).
• ... must be at least 32 characters outside dev mode. The secret has a hard 32-character minimum in production.
• ... cannot use the local development secret outside dev mode. The built-in dev fallback value is rejected in production; generate a real secret.

Rotating the secret makes previously encrypted grant files unreadable — plan rotation as a re-authorization event.

Deploy Preflight Failures

• deploy --dry-run reports missing setup. Each preflight item names a required env var or provider setup step inferred from gateway.ts, channels, connections, and the model prefix. Satisfy every required item before publishing.
• Railway: requires RAILWAY_TOKEN (or an authenticated Railway CLI) and a linked project/service or explicit --railway-project and --railway-service.
• Docker: requires Docker locally; the image tag comes from --docker-image or ASSEMBLY_LINE_DOCKER_IMAGE.
• Fly: requires FLY_API_TOKEN and --fly-app or FLY_APP_NAME (Fly deploys require --fly-app or FLY_APP_NAME.).
• VPS: requires a named assembly-line.hosts.json entry, a verified SHA256: SSH host-key fingerprint, the inventory-selected identity-file environment variable, supported AMD64 Ubuntu 24.04/26.04 or Debian 12, Docker Compose, and available ports 80/443. Create or adopt a Hetzner host with assembly-line hosts bootstrap; it does not report success until cloud-init, Docker, UFW, fail2ban, and unattended upgrades are active. First deploys need --sync-secrets unless a complete remote runtime environment already exists.
• SSH host-key fingerprint mismatch: verify the new fingerprint through the provider console or another trusted path. Do not replace the pin based only on the key returned by the same network connection.
• A VPS release rolls back after readiness: inspect the blue/green runtime containers and openeve-caddy on the host. Assembly Line restores the previous route when container health, Caddy validation/reload, or public HTTPS /readyz fails.
• A primary domain or alias is already owned: another deployment has the host-local ownership claim. Confirm the intended owner, then update the owning gateway and run deploy --domains-only; never delete ownership files merely to bypass the collision. Domain reconciliation retains old claims until the new Caddy route passes public readiness and rolls new claims back on failure.
• A prepared release cannot activate: --activate is revision-fenced. Run it from the same agent revision that produced --prepare-only. If another prepare superseded it, prepare the intended revision again.
• Postgres image mismatch: ordinary deployment refuses to recreate the cluster under a different image. Review downtime and backups, then run assembly-line state upgrade-postgres <agentRoot> --confirm-upgrade.
• Postgres transfer verification failed: Assembly Line compares normalized schema and exact per-table counts and restores the pre-transfer target on failure. Keep the source quiesced, inspect the reported failure, and do not resume traffic until a later verified transfer succeeds.
• <bin> not found. Install it, or pass the matching --*-bin flag. The deploy path shells out to railway, docker, flyctl, or the VPS SSH client; install the binary or point --railway-bin/--docker-bin/--fly-bin at it.
• Migration files are present, but no migration runner was configured. The artifact contains .assembly-line/migrations; pass --migration-command or set ASSEMBLY_LINE_MIGRATION_COMMAND.
• Community plugin providers: a provider-package-unresolved validation warning means the compiler could not import the packageName given to adapter(kind, opts, { package }) from the agent root, so preflight falls back to a generic requirement. Install the package where the agent builds. At boot, the Node host re-resolves it and fails with a specific error when the package is missing (Failed to load plugin package ...), does not export openeveProvider, or has no registration for the role/kind. See Authoring Plugin Providers.

Webhooks And Scheduled Runs Rejected

• 403 scheduler_unauthorized on /assembly-line/automations/tick (or its deprecated scheduler alias). Set ASSEMBLY_LINE_SCHEDULER_SECRET and send it as Authorization: Bearer <secret> or x-assembly-line-scheduler-secret. Without a configured secret the endpoint only accepts dev-mode requests.
• Production boot fails with Production provider ingress requires webhook authentication: .... A channel declared ingress secrets (ingress.requiredSecretEnv, any-of groups) and none of its groups is fully set — for example Telegram needs TELEGRAM_WEBHOOK_SECRET, and Photon needs a signing secret or a bearer token. Dev mode logs a warning instead of failing. Set the secrets named in the error.
• Webhooks return 401 even though boot succeeded. Boot checks that the secrets exist; each request is still verified by the channel module (for example Telegram compares x-telegram-bot-api-secret-token constant-time). Make sure the provider-side webhook config sends the same secret.

Runs Stuck, Deliveries Missing, Sandbox Writes Lost

• Deliveries sit in pending and never send. The delivery worker drains the durable queue; check that ASSEMBLY_LINE_DELIVERY_WORKER is not set to false/0 and that the host called startBackgroundWorkers() (listenNodeRuntime does this automatically).
• A run completed but the user got no message yet. A retryable send failure defers the delivery instead of failing the run: the run event log shows delivery.deferred with the error, attempt count, and nextAttemptAt, and the worker retries with backoff up to ASSEMBLY_LINE_DELIVERY_QUEUE_MAX_ATTEMPTS (default 5) before a terminal delivery.failed.
• Runs stuck in running after a crash or redeploy. Orphan recovery sweeps runs stale past max(5min, 4x heartbeat): already-delivered runs complete, runs with a continuation checkpoint get one resume attempt, runs with a model response get a real pending delivery, and everything else is marked failed. It runs at boot and every ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS; the kill-switch is ASSEMBLY_LINE_RUN_RECOVERY=false.
• Sandbox writes (memory, skills, /workspace files) not persisting. The sandbox-sync worker (ASSEMBLY_LINE_SANDBOX_SYNC_WORKER) performs the writeback; use sandboxSyncDiagnostics() and inspectSandboxSyncJob(jobId) to see due/leased/expired/blocked jobs, and retrySandboxSyncJob(jobId) after fixing a blocked one.
• A sandbox reports /home/user, /root, or another cwd instead of /workspace. Current built-in hosted adapters reject that session during initialization; do not rewrite command strings or add a symlink in agent code. Confirm the deployment contains the current adapter packages and inspect its provider metadata for openeve.filesystemContractVersion. Sandboxes without the current version are deliberately not reconnected.
• workingDirectory must be /workspace. Remove the alias or set it to /workspace. The field cannot remap absolute paths embedded in shell commands. For local testing of those absolute paths, use Docker instead of the Local adapter.
• Kill-switches for debugging: ASSEMBLY_LINE_DELIVERY_WORKER, ASSEMBLY_LINE_SANDBOX_SYNC_WORKER, and ASSEMBLY_LINE_RUN_RECOVERY each accept false/0. Every background behavior has one; see the tables in Runtime And Deployment.

Still Stuck?

Inspect the durable record: .assembly-line/manifest.json, .assembly-line/preflight.json, and .assembly-line/route-table.json for compile-time surprises; GET /runs/:id/events (or the state file in dev) for runtime behavior — every model step, tool call, pause, delivery attempt, and recovery action is recorded as an event.