Status: shipped benchmark contract with failure attribution.
The benchmark score is the substrate. On top of it, Orivoxx records a bounded trace, attributes a failure to one of 20 layer × mode cells, and proves the attribution with a counterfactual replay. Attribution and replay are deterministic and are specified in docs/ATTRIBUTION_SPEC.md and docs/OBSERVABILITY_SPEC.md. packages/benchmarks implements the task registry, rulesets, scoring configuration, and pure score function; packages/harness implements the adapters and Docker sandbox. The scorecard and leaderboard are live in apps/web. The exact model/provider is configuration, not a benchmark invariant.
This document defines what each bench measures, how it is scored, and what it deliberately does not measure. The last part is as important as the first: a deterministic benchmark is only honest if its boundaries are explicit.
PROJECT_SPEC.md, not a hidden unit-test file. The agent is told what to build.Question: did the agent build what the spec asked for?
What it measures: the fraction of declared requirements that pass their deterministic checks. Requirement types include required/forbidden files, required/forbidden text markers, required/forbidden dependencies, and required static source references.
How it is scored (shipped):
critical, high, medium, low).PASS earns the requirement’s weight; FAIL earns zero; UNVERIFIED earns zero and is reported separately so it cannot be mistaken for a pass.earned weight / total weight, normalized to 0–100.critical failure can cap the overall score or mark the run BLOCKED, depending on the task’s scoring configuration. This is a per-task policy, not a global rule.What it does not measure:
Current state: the rule engine and verdict logic exist. A versioned benchmark task registry and scoring configuration do not. The TaskBoard fixtures are the first usable task candidate.
Question: did the agent avoid known vulnerable patterns?
What it measures: rule pass-rate over a versioned security ruleset. Example rule families:
The existing engine can express several of these as text_not_contains, dependency_absent, file_absent, and source_reference_exists checks. Richer data-flow rules are not implemented and would require a new rule type.
How it is scored (shipped):
What it does not measure:
Current state: security requirements are declared per task and scored by the deterministic engine. Both shipped tasks include a named security ruleset (shell-execution dependency, environment file, dynamic evaluation). There is no vulnerability database and no data-flow analysis.
Question: is the produced code structurally understandable?
What it measures: graph metrics over the captured repository:
How it is scored (shipped):
What it does not measure:
Current state: shipped in packages/benchmarks (scoring/import-graph, scoring/structure). The builder parses JS/TS-family imports statically, resolves relative specifiers inside the captured inventory, and computes cycles, fan-out, orphan ratio, and module count. Tests and mocks are excluded. Only resolvable relative imports create edges; unsupported or alias imports are reported as warnings, never invented.
Question: how much of the result comes from the harness rather than the model?
What it measures: two or more agent configurations run against the same task version and same ruleset version, with the model held constant where possible. The comparison isolates the effect of the harness: tool set, retry policy, context management, stopping conditions.
How it is scored (shipped):
What it does not measure:
Current state: no harness, no adapters, no runner. This is a stretch bench for after the core platform works.
Every result must record:
| Field | Why it is required |
|---|---|
taskId + taskVersion |
The spec and starting repository can change between versions. |
rulesetVersion |
The same repository can score differently under a changed rule set. |
adapterId + adapterVersion |
The harness is part of the measured system. |
scoringConfigVersion |
Weights, thresholds, and caps change the score. |
repositoryFingerprint |
Ties the result to the exact captured artifact. |
sandboxImage |
Tool and runtime differences can change agent behaviour. |
modelId |
Reported for attribution, not as the sole unit of comparison. |
A result missing any of the first five fields is not a valid leaderboard entry.
(taskVersion, rulesetVersion, scoringConfigVersion, repositoryFingerprint) must produce the same score.UNVERIFIED, never PASS.The score is a pure function of declared rule outcomes, structural metrics, and a versioned scoring configuration. It makes no model calls.
benchScore = 100 * sum(severityWeight[req] * pass[req]) / sum(severityWeight[req])
overall = 0.60 * specScore + 0.25 * securityScore + 0.15 * maintainabilityScore
pass[req]: 1 for PASS, 0 for FAIL, DRIFT, and UNVERIFIED. UNVERIFIED is reported separately and never counted as a pass.FAIL/DRIFT caps overall at 59.9; a critical security FAIL/DRIFT caps it at 49.9. The cap is applied after the weighted sum and recorded in capsApplied. UNVERIFIED earns zero but does not trigger a cap: missing evidence is not a proven failure.taskVersion, rulesetVersion, adapterVersion, scoringConfigVersion, and repositoryFingerprint are mandatory on every result. A result missing any of them is rejected by ScoreResultSchema and is not leaderboard-valid.The import graph is built statically from JS/TS-family files. Test and mock files are excluded. Only relative imports that resolve to a captured module become edges; bare package imports are external and are not counted. Framework entrypoints (app/**/{page,layout,route,error,...}, pages/**, middleware, instrumentation, common src/index|main files) are not orphans.
cycles = strongly connected components with >1 module, plus self-imports
maxFanOut = maximum distinct internal modules imported by one module
orphanRatio = modules with zero internal in-edges and not an entrypoint / moduleCount
moduleCount = captured JS/TS-family modules, excluding tests and mocks
For cycles, maxFanOut, and orphanRatio, with published good and warn thresholds:
metricScore = 100 if value <= good
= 100 - 30 * (value - good) / (warn - good) if good < value <= warn
= max(0, 70 - 30 * (value - warn) / warn) if value > warn
moduleCount scores 100 inside the published [min, max] range, then degrades linearly to 60 at zero modules or at twice the maximum. maintainabilityScore is the mean of the four metric scores, rounded to one decimal. Default thresholds (scoringConfigVersion 1.0.0): cycles good 0 / warn 1, max fan-out good 8 / warn 12, orphan ratio good 0.10 / warn 0.25, module count 3–60.
Suppose a task has 10 requirements:
| Bench | Requirements | Earned weight | Total weight | Bench score |
|---|---|---|---|---|
| Spec | 6 | 26 | 32 | 81.3 |
| Security | 3 | 12 | 16 | 75.0 |
| Maintainability | metrics | — | — | 90.0 |
overall = 0.60 * 81.3 + 0.25 * 75.0 + 0.15 * 90.0 = 48.78 + 18.75 + 13.50 = 81.0
If the failed requirement is a critical spec failure, the 59.9 cap applies and the result records capsApplied: ["critical-spec-failure"].
Weights live in the task’s scoring configuration, not in code. The UI must show the per-bench breakdown and the rule-level deductions. A single number without the breakdown is not sufficient evidence.