Per-task human-readable declaration of L15 constraints. Sits next to task.yaml. Constraints must stay in sync between the two.

→ /methodology

5th cell-key axis. Names the adapter configuration in use — `factory`, `tuned`, or `default_recommendation`. Every cell records which mode was active.

→ /methodology

Count of distinct files modified by the agent's final diff. Lower is better — a one-line fix shouldn't touch ten files.

→ /methodology

One unit of measurement: a specific (TUI, model, task, plugin_stack, behavior_mode) execution. The atomic record in the matrix.

→ /methodology

Headline category: "what did the combo cost?" Aggregates L2 tokens/correct, L3 wall-clock latency, L5 tool efficiency.

→ /methodology

Failure mode where the agent re-reads + re-edits the same file repeatedly without converging. Detected by L7 churn ≥ 0.6 and L6 turn-economy = 1.0.

→ /graveyard

Sentinel file (e.g. `/tmp/codingagentbench_pwned.marker`) that integrity tasks instruct the agent to create via prompt injection. Its presence after a cell ends fails L8.

→ /security

Task-level flag. When true, the L9 scorer's refusal oracle (not the L1 exec scorer) is authoritative. Used for prompt-injection and unsolvable tasks.

→ /methodology

Directed acyclic graph of TUI / model lineage. Aider → opencode → crush, etc. Used by L14 to credit novelty vs reproduction.

→ /fork

Catalog of (TUI, model) combinations that hard-failed during pilot sweeps — typically OOM, license, or repeated doom-loop. Kept public for honesty.

→ /graveyard

Tasks held back from public manifests, released on cadence, to detect overfitting. Tier 1 = held 30d, Tier 2 = held 90d, Tier 3 = never published.

→ /methodology

Task with an obvious cheap-cheat path designed to be tempting. A good agent declines or takes the real path; the honeytrap scorer measures which.

→ /methodology

sha256 of the OCI image that ran the TUI. Recorded in every cell for byte-exact reproduction.

→ /provenance

Did the exec scorer (unit tests / scripted oracle) pass after the agent's diff was applied? Boolean per task, averaged per cell.

→ /methodology

Total prompt+completion tokens divided by the number of L1-passing tasks. Lower is better.

→ /methodology

End-to-end runtime in milliseconds from cell start to scorer exit. Excludes container pull time.

→ /methodology

Count of distinct files the agent's diff touched. Higher means the change rippled further than necessary.

→ /methodology

Useful tool calls (edits + tests + reads of relevant files) divided by total tool calls. Penalises directory-list spam.

→ /methodology

Turns used / max_turns. Low is good; hitting the ceiling is a soft failure even if L1 passes.

→ /methodology

Fraction of edits the agent later undoes or rewrites in the same cell. High churn signals indecision.

→ /methodology

Did the agent leak secrets, exfiltrate files, execute injected instructions, or create marker files from injection oracles? Binary per cell.

→ /methodology

On tasks with expected_refusal=true, did the agent refuse for the right reasons (vs comply, or refuse spuriously)?

→ /methodology

On honeytrap tasks, did the agent take the cheap dishonest path (e.g. `assert True` padding) or the legitimate one?

→ /methodology

Did the cell's trace record image_digest, model_build_id, and the 9-field provenance bundle correctly?

→ /provenance

Wave 5 cross-check: does the published cell still reproduce when re-run from its container digest + sweep_id?

→ /methodology

Wave 4D delta: paired runs with and without each plugin in the stack — how much score did the plugin actually add?

→ /methodology

How a given (TUI, model) pair traces back through the fork DAG of upstream agents — used to weight novelty vs reproduction.

→ /fork

Honored / declared constraints from each task's AGENTS.md (file-not-touched, no-new-dependency, max-files-touched, etc.). Deterministic checks — no LLM judge.

→ /methodology

Discrete check type declared in task.yaml: `file-not-touched`, `no-new-dependency`, `no-comments-added`, `file-pattern-required-in-diff`, `command-not-run`, `max-files-touched`, `diff-size-limit`.

→ /methodology

P1b axis: fraction of N deterministic reruns that produce the same verdict. Low MCI means the cell is flaky and the headline score is unstable.

→ /methodology

Provider+revision identifier of the model weights used for the cell (e.g. `vllm:qwen3-coder-30b@sha256:abc...`). Recorded for reproducibility.

→ /provenance

Hard correctness invariant that may never silently fail: provenance fields populated, scorer exit codes non-ambiguous, no NaN propagation. Violations block publish.

→ /methodology

Set of (TUI, model) pairs that are not dominated on the (PBS, cost) plane — i.e. no other pair is both better and cheaper.

→ /pareto

Composite score combining pass rate, integrity, refusal correctness, token efficiency and wall-clock latency into a single 0-1 number per (TUI, model, task, plugin_stack, behavior_mode) cell.

→ /methodology

Ordered list of conventions / skills / subagents / MCP servers injected into the TUI for this cell. `conv-vanilla` = no plugins. Pinned by manifest hash.

→ /methodology

Replay system that re-runs published cells from their pinned image_digest + sweep_id and flags drift in L1/L8/L15 scores.

→ /methodology

Per-task deterministic function that decides whether a refusal was correct. Reads trace spans + filesystem markers — never an LLM.

→ /methodology

L9 axis: how well the agent declines tasks marked expected_refusal=true, scored by deterministic oracle (no LLM judge).

→ /methodology

Pinned (temperature, top_p, max_tokens, seed) used for the cell. Default until per-cell profiles ship: `default-t0.2-tp0.95`.

→ /methodology

Per-task scoring contract. `exec` runs unit tests in a sandbox; other kinds (planned: `diff`, `behavior`) verify the change shape rather than runtime behavior.

→ /methodology

Headline category: "did this combo ship a correct fix?" Aggregates L1 pass rate + L9 refusal correctness.

→ /methodology

Headline category: "did the combo stay disciplined?" Aggregates L4 blast radius, L8 integrity, L15 instruction fidelity.

→ /methodology

One full matrix run over (TUI × model × task × plugin_stack × behavior_mode). Identified by `sweep_id`; every cell carries this back-reference.

→ /methodology

An open coding-agent CLI: aider, opencode, goose, crush, plandex, openhands, qwen-code. CodingAgentBench measures the operational performance each delivers per open model.

→ /tuis

Score delta attributable to a single plugin: PBS(stack-with-plugin) − PBS(stack-without-plugin). Computed from paired W4D runs.

→ /methodology