Skip to main content

evals/

evals/ is the convention for an agent's golden dataset. Assembly Line ships the runner and case contract; cases and scoring opinions remain developer-owned. Eval files are excluded from manifest.files, sources.json, and agentRevision, so changing test data does not create a deployment revision.

Use one JSON file per case:

{
  "name": "Escalates an unsafe request",
  "description": "The agent should ask for approval before the write.",
  "tags": ["high-risk"],
  "input": {
    "message": "Apply the account change",
    "approve": false
  },
  "expect": {
    "status": "waiting_for_approval",
    "toolsCalled": ["update_account"],
    "toolsNotCalled": ["delete_account"],
    "maxCostUsd": 0.05
  }
}

name defaults to the filename stem (including subfolders: cases are discovered recursively, so evals/billing/refund.json defaults to billing/refund; the evals/evaluators/ subtree is never treated as cases). A useful starting suite balances normal, edge, ambiguous, and high-risk cases.

Case contract​

input accepts:

  • message for a single-turn case, or turns for a scripted conversation (exactly one of the two);
  • channel and recentHistory (multi-turn cases apply recentHistory to the first turn only; later turns hydrate history from the conversation itself);
  • tool and toolInput for a forced authored-tool run;
  • approve to override the command-wide --approve setting.

A case may also declare top-level mocks (canned tool results), fixtures (starting sandbox/memory state) — see below — and repetitions (integer 1..100): how many times this case runs per suite, winning over the --repetitions flag. Each repetition produces its own report entry carrying repetitionsPlanned, so clients can render a per-case pass distribution rather than a single binary.

expect accepts:

  • status: any durable run status, including waiting states;
  • output.contains, output.matches, or output.equals;
  • output.json.valid, output.json.jsonSchema, and output.json.fields using dot paths or JSON Pointer paths;
  • toolsCalled and toolsNotCalled;
  • trajectory, for ordered, exact, present, or forbidden tool-call checks;
  • evaluators, for developer-owned scoring modules;
  • maxCostUsd across the case's recorded model usage;
  • judge: { "criteria": "...", "model": "provider/model", "threshold": 0.7 }.

When agent.ts declares outputSchema, output.json assertions read the already-validated RunAgentResult.output. Otherwise the assertion layer parses the response as JSON.

Multi-turn conversations​

input.turns scripts a conversation. Every turn is its own durable run, and all turns share one conversation: the runtime hydrates recent history and reuses the conversation-scoped sandbox session exactly as a production channel turn would, so behavior across turns (memory of earlier answers, files written in turn one and read in turn two) is evaluated for real.

{
  "name": "Clarifies before acting",
  "input": {
    "turns": [
      {
        "message": "Change the plan on my account",
        "expect": { "toolsCalled": ["ask_question"] }
      },
      {
        "message": "The pro plan, for user@example.com",
        "approve": true,
        "expect": { "status": "completed", "toolsCalled": ["update_account"] }
      }
    ]
  },
  "expect": {
    "status": "completed",
    "toolsNotCalled": ["delete_account"],
    "judge": { "criteria": "The agent asked before acting and confirmed the change." }
  }
}

Each turn accepts message, approve, tool, and toolInput, plus an optional per-turn expect limited to the deterministic assertions (status, output, toolsCalled, toolsNotCalled, trajectory, maxCostUsd); per-turn failures are reported prefixed turn N:. judge and evaluators stay case-level.

The case-level expect evaluates the conversation as a whole: status, output, and response come from the final turn; toolsCalled, toolsNotCalled, and trajectory span every turn in order; maxCostUsd covers the summed cost. The judge receives the full transcript, and results record a turns array (per-turn status, response, tools, cost, and run id). The --timeout budget applies per turn. A turn that ends waiting (for approval or input) parks that run; the next turn starts a new run in the same conversation — in-conversation approval resumption is not simulated, so use per-turn approve to model the granted-approval path.

Tool mocks​

mocks.tools replaces named tool executions with canned results, so cases that would otherwise hit external services run deterministically and side-effect free. The authored tool module is never imported; approval gates, tool-call recording, and run events still apply, so trajectory and approval assertions keep working against mocked tools.

{
  "mocks": {
    "tools": {
      "search_accounts": { "result": { "accounts": [{ "id": "a1" }] } },
      "flaky_api": { "results": [{ "attempt": 1 }, { "attempt": 2 }] },
      "billing_api": { "error": "upstream unavailable" }
    }
  }
}

Each entry declares exactly one of result (every call), results (consumed per call; the last repeats), or error (the tool call throws). Sequences reset for every attempt and repetition; because per-run sequence cursors are in-memory, a crash-resumed run or a subagent child run restarts its sequence. Built-in sandbox tools can be mocked by name too; connection tools cannot — they execute for real wherever the suite runs.

Single-turn mocks also work against remote gateways that enable ASSEMBLY_LINE_ENABLE_EVAL_RUNS (see Remote gateways below). Multi-turn cases keep their mocks local so sequences span turns.

Fixtures​

fixtures seeds starting state before the first turn, for cases that assume the agent has already accumulated files or memory:

{
  "fixtures": {
    "sandbox": { "notes.txt": "existing workspace file" },
    "memory": { "/memory/USER.md": "Prefers concise answers." }
  }
}

sandbox paths are seeded into the case's sandbox session (relative paths land under /workspace). memory paths must start with /memory/ and are written to the memory store in the same scope the case's runs read from. Cases with sandbox fixtures (like multi-turn cases) run conversation-scoped so every turn sees the seeded session.

Tool trajectories​

toolsCalled answers only whether a tool appeared. Use trajectory when order, arguments, completion status, or returned values matter:

{
  "expect": {
    "trajectory": {
      "mode": "ordered",
      "steps": [
        {
          "tool": "search_accounts",
          "arguments": { "email": "person@example.com" },
          "argumentsMatch": "partial",
          "status": "completed"
        },
        {
          "tool": "update_account",
          "arguments": { "plan": "pro" },
          "result": { "updated": true }
        }
      ]
    }
  }
}

Modes are:

  • contains: every expected step must match a different call, in any order;
  • ordered: expected steps must appear in order, with unrelated calls allowed;
  • exact: call count, order, and every expected step must match;
  • forbid: no listed step may match.

Argument and result matching is recursive and partial by default. Set argumentsMatch or resultMatch to exact when extra object fields should fail the case.

Custom evaluators​

Put custom scoring logic in evals/evaluators/<name>.ts (JavaScript and MTS are also accepted). The framework owns loading, validation, and reporting; the agent developer owns the scoring opinion. Wrap the module in defineEvaluator from @assemblyline-agents/cli/eval, or export a plain object with satisfies EvalEvaluator:

import type { EvalEvaluator } from "@assemblyline-agents/cli/eval";

export default {
  name: "citation-quality",
  async evaluate({ actual, config }) {
    const minimum = typeof config.minimum === "number" ? config.minimum : 1;
    const count = (actual.response.match(/https:\/\//gu) ?? []).length;
    return {
      key: "citations",
      score: Math.min(1, count / minimum),
      pass: count >= minimum,
      comment: `${count} citations found`
    };
  }
} satisfies EvalEvaluator;

Reference it from a case:

{
  "expect": {
    "evaluators": [
      {
        "name": "citation-quality",
        "config": { "minimum": 2 },
        "metric": "citations",
        "minScore": 0.8
      }
    ]
  }
}

The evaluator context has: evalCase (the case), actual (the actual response, parsed output, and per-turn records for conversations), timeline (the final turn's run timeline), timelines (one timeline per turn), config (the case-owned JSON config), repetition, attempt, and complete — a model completer bound to the suite's judge configuration:

const graded = await complete({
  prompt: "Rate the citations in this answer...",
  systemPrompt: "Return PASS or FAIL.", // optional
  model: "anthropic/claude-sonnet-5"     // optional; defaults to the judge model
});
// graded.text, graded.costUsd (accounted into the case's judge cost)

This makes model-based evaluators (rubric panels, pairwise comparisons, ensemble judging) first-class without wiring a provider client; the scoring opinion still lives entirely in the evaluator. An evaluator returns one metric or an array of uniquely named metrics. A metric can expose a numeric score, JSON value, boolean pass, and a short comment. requirePass defaults to true. Evaluators are eval-only code and are not packaged into the deployed agent.

Running evals​

assembly-line eval ./agent
assembly-line eval ./agent --filter escalation
assembly-line eval ./agent --tag high-risk --concurrency 4
assembly-line eval ./agent --repetitions 3
assembly-line eval ./agent --json > eval-report.json

Flags:

  • --judge-model overrides ASSEMBLY_LINE_EVAL_JUDGE_MODEL; a case-level model wins over both. Without any override, the agent model is used.
  • --timeout is the per-run timeout in seconds (default 120), applied to each turn of a conversation case. A timeout requests cooperative cancellation and reports an errored case.
  • --repetitions runs every valid case multiple times (default 1); a case-level repetitions field wins for that case. Malformed case files are reported once rather than repeated.
  • --retry-attempts is the number of retries after the first transient provider failure (default 2); --retry-backoff-ms sets the initial exponential delay (default 250 ms). Set retry attempts to 0 to disable it.
  • --fail-fast stops scheduling new cases after the first failure or error.
  • --approve enables gated tools unless input.approve overrides it. An approving turn records tool.approval_requested plus tool.approval_auto_resolved durably and executes the gated tool in the same tick — approval assertions keep working, without parking the run.
  • --json emits the stable CI report and suppresses the human table.
  • --url <gatewayUrl> runs the cases against a deployed gateway instead of the in-process runtime (see below). --token supplies the admin token, falling back to ASSEMBLY_LINE_ADMIN_TOKEN.

Cases run sequentially by default. Every repetition and retry gets fresh file state, blob, and sandbox roots under .assembly-line/eval-results/ in the build artifact. The runtime uses record-only delivery: it creates and settles the normal durable delivery obligation but never invokes a channel sender.

Remote gateways​

--url points the suite at a deployed runtime: runs are created through the node host's POST /runs (the target must set ASSEMBLY_LINE_ENABLE_API_RUNS=true), graded from GET /runs/:id/timeline, and cancelled on timeout through POST /runs/:id/cancel. Assertions, trajectories, judges, and cost metrics grade the deployed runtime's real timeline.

The runner probes the target's /healthz capabilities before sending anything eval-specific. Servers that set ASSEMBLY_LINE_ENABLE_EVAL_RUNS=true (or allowEvalRuns) advertise eval-runs and accept a per-run eval block on POST /runs: single-turn tool mocks, approval auto-resolve, and record-only delivery all work remotely, and the block is persisted into the run's durable input — every stubbed run is auditable in run detail and marked by its tool.approval_auto_resolved events. Against servers without the capability, cases that need mocks or auto-approval are refused upfront with the reason rather than silently degraded: an old server ignoring a stub it never received cannot execute a real tool by accident.

Still refused remotely regardless of capability: multi-turn conversations, seeded recentHistory, and fixtures — they depend on local adapters or run() inputs the HTTP surface does not accept. Case metadata tagging is not transmitted remotely.

Security notes. ASSEMBLY_LINE_ENABLE_EVAL_RUNS is separate from ASSEMBLY_LINE_ENABLE_API_RUNS because stubbed runs are a testing surface: enable it on dedicated test environments (assembly-line deploy agent --env test), not on production. The normal run-create authentication (admin token or host auth policy) still applies. Connection tools are never stubbable and auto-approval makes gated connection tools really execute — do not point remote evals at agents holding live production connections.

For local (in-process) runs, each judged attempt also appends an eval.judged event to the run's durable event log with the score, threshold, pass, and reasoning — the verdict travels with the run, not just the report. Remote runs record verdicts only in the report.

Retries are deliberately narrow. They cover common provider conditions such as HTTP 408/425/429/5xx responses, connection resets, DNS retry signals, timeouts reported by a provider, rate limits, and temporarily unavailable or overloaded services. Assertion failures, malformed judge responses, local evaluator failures, and eval case timeouts are not retried. A transient judge call is retried without rerunning the agent.

The report distinguishes assertion failures (failed) from execution errors (errored, such as missing provider credentials, model failures, malformed case files, judge parse errors, or timeouts). Either produces a non-zero exit code. Costs are split between agent runs and judge calls, with per-tag results included in both human and JSON output.

Experiment artifacts​

Every invocation creates a new experiment directory and never overwrites an old one:

.assembly-line/eval-results/<experiment-id>/
├── experiment.json
├── results.json
├── evaluator-cache/
└── cases/

experiment.json is written before cases run. It records the framework and agent revisions, selected case count, planned execution count, safe execution configuration, and SHA-256 suite/config fingerprints. The suite fingerprint covers selected case bytes and bundled source for referenced custom evaluators, including local imports. Environment values and provider credentials are never captured. results.json is written once after the run and contains the manifest, per-execution outcomes and metrics, costs, and artifact paths. A crash therefore leaves an inspectable manifest without pretending the experiment completed.

Baselines and regression gates​

Compare a run with a prior results.json, experiment.json, or experiment directory:

assembly-line eval ./agent \
  --baseline .assembly-line/eval-results/<experiment-id> \
  --max-regressions 0 \
  --max-cost-increase-percent 20

For each case present in both experiments, Assembly Line compares the fraction of repetitions that passed. A lower fraction is a regression; a higher fraction is an improvement. Added and removed case IDs are listed separately. The default gate allows zero regressions. The cost-growth gate is opt-in; a positive cost against a zero-cost baseline fails it as an unbounded percentage increase. Failed gates set the CLI exit status to non-zero even when every current case passes. This keeps framework policy neutral: teams choose the baseline, repetition count, and acceptable cost budget.

CI​

Store provider credentials in the CI secret store, never in case files. A minimal gate is:

assembly-line eval ./agent --json \
  --baseline .assembly-line/eval-results/<approved-experiment-id>

For deterministic PR checks, omit expect.judge; run judge cases in a separate job when model variability or provider availability should not block every code change. Repetitions are also opt-in because they multiply runtime and provider cost; use them for nondeterministic or release-critical suites rather than every fast local check.