Orivoxx

Orivoxx — Benchmark Suite

Status: shipped benchmark contract with failure attribution.

0. Attribution layer

The benchmark score is the substrate. On top of it, Orivoxx records a bounded trace, attributes a failure to one of 20 layer × mode cells, and proves the attribution with a counterfactual replay. Attribution and replay are deterministic and are specified in docs/ATTRIBUTION_SPEC.md and docs/OBSERVABILITY_SPEC.md. packages/benchmarks implements the task registry, rulesets, scoring configuration, and pure score function; packages/harness implements the adapters and Docker sandbox. The scorecard and leaderboard are live in apps/web. The exact model/provider is configuration, not a benchmark invariant.

This document defines what each bench measures, how it is scored, and what it deliberately does not measure. The last part is as important as the first: a deterministic benchmark is only honest if its boundaries are explicit.

Design principles

  1. Deterministic scoring. No LLM in the score. Rules are declared, versioned, and reproducible.
  2. Model-agnostic. The same task can score Claude Code, Codex CLI, OpenClaw, or a custom Python agent through an adapter.
  3. Spec-first. The task is a PROJECT_SPEC.md, not a hidden unit-test file. The agent is told what to build.
  4. Explainable. Every point lost maps to a named rule, a requirement, an evidence record, and a source path/line where available.
  5. Versioned. A score is only meaningful alongside its task version, ruleset version, and adapter version.

1. Spec-Conformance Bench

Question: did the agent build what the spec asked for?

What it measures: the fraction of declared requirements that pass their deterministic checks. Requirement types include required/forbidden files, required/forbidden text markers, required/forbidden dependencies, and required static source references.

How it is scored (shipped):

What it does not measure:

Current state: the rule engine and verdict logic exist. A versioned benchmark task registry and scoring configuration do not. The TaskBoard fixtures are the first usable task candidate.

2. Security-Pattern Bench

Question: did the agent avoid known vulnerable patterns?

What it measures: rule pass-rate over a versioned security ruleset. Example rule families:

The existing engine can express several of these as text_not_contains, dependency_absent, file_absent, and source_reference_exists checks. Richer data-flow rules are not implemented and would require a new rule type.

How it is scored (shipped):

What it does not measure:

Current state: security requirements are declared per task and scored by the deterministic engine. Both shipped tasks include a named security ruleset (shell-execution dependency, environment file, dynamic evaluation). There is no vulnerability database and no data-flow analysis.

3. Maintainability Bench

Question: is the produced code structurally understandable?

What it measures: graph metrics over the captured repository:

How it is scored (shipped):

What it does not measure:

Current state: shipped in packages/benchmarks (scoring/import-graph, scoring/structure). The builder parses JS/TS-family imports statically, resolves relative specifiers inside the captured inventory, and computes cycles, fan-out, orphan ratio, and module count. Tests and mocks are excluded. Only resolvable relative imports create edges; unsupported or alias imports are reported as warnings, never invented.

4. Harness Bench (stretch)

Question: how much of the result comes from the harness rather than the model?

What it measures: two or more agent configurations run against the same task version and same ruleset version, with the model held constant where possible. The comparison isolates the effect of the harness: tool set, retry policy, context management, stopping conditions.

How it is scored (shipped):

What it does not measure:

Current state: no harness, no adapters, no runner. This is a stretch bench for after the core platform works.

Reproducibility

Every result must record:

Field Why it is required
taskId + taskVersion The spec and starting repository can change between versions.
rulesetVersion The same repository can score differently under a changed rule set.
adapterId + adapterVersion The harness is part of the measured system.
scoringConfigVersion Weights, thresholds, and caps change the score.
repositoryFingerprint Ties the result to the exact captured artifact.
sandboxImage Tool and runtime differences can change agent behaviour.
modelId Reported for attribution, not as the sole unit of comparison.

A result missing any of the first five fields is not a valid leaderboard entry.

Determinism rules

Score aggregation (shipped)

The score is a pure function of declared rule outcomes, structural metrics, and a versioned scoring configuration. It makes no model calls.

benchScore   = 100 * sum(severityWeight[req] * pass[req]) / sum(severityWeight[req])
overall      = 0.60 * specScore + 0.25 * securityScore + 0.15 * maintainabilityScore

Maintainability score (shipped)

The import graph is built statically from JS/TS-family files. Test and mock files are excluded. Only relative imports that resolve to a captured module become edges; bare package imports are external and are not counted. Framework entrypoints (app/**/{page,layout,route,error,...}, pages/**, middleware, instrumentation, common src/index|main files) are not orphans.

cycles       = strongly connected components with >1 module, plus self-imports
maxFanOut    = maximum distinct internal modules imported by one module
orphanRatio  = modules with zero internal in-edges and not an entrypoint / moduleCount
moduleCount  = captured JS/TS-family modules, excluding tests and mocks

For cycles, maxFanOut, and orphanRatio, with published good and warn thresholds:

metricScore = 100                                       if value <= good
            = 100 - 30 * (value - good) / (warn - good)  if good < value <= warn
            = max(0, 70 - 30 * (value - warn) / warn)    if value > warn

moduleCount scores 100 inside the published [min, max] range, then degrades linearly to 60 at zero modules or at twice the maximum. maintainabilityScore is the mean of the four metric scores, rounded to one decimal. Default thresholds (scoringConfigVersion 1.0.0): cycles good 0 / warn 1, max fan-out good 8 / warn 12, orphan ratio good 0.10 / warn 0.25, module count 3–60.

Worked example

Suppose a task has 10 requirements:

Bench Requirements Earned weight Total weight Bench score
Spec 6 26 32 81.3
Security 3 12 16 75.0
Maintainability metrics — — 90.0

overall = 0.60 * 81.3 + 0.25 * 75.0 + 0.15 * 90.0 = 48.78 + 18.75 + 13.50 = 81.0

If the failed requirement is a critical spec failure, the 59.9 cap applies and the result records capsApplied: ["critical-spec-failure"].

Weights live in the task’s scoring configuration, not in code. The UI must show the per-bench breakdown and the rule-level deductions. A single number without the breakdown is not sufficient evidence.

Known limitations