Benchmark the agent, not just the model.
Most agent leaderboards are model leaderboards in disguise. They fix one harness, one prompt, one tool set, and one stopping policy, then compare foundation models on a static task suite. The resulting rank tells you how those models performed under that one configuration. It does not tell you how well an agent — the compound system — performs when the harness changes.
That gap matters because real agent performance is won or lost in the harness: how tools are described, how failures are retried, how context is compacted, when the loop stops, and whether the agent verifies its own work before declaring success. A strong model with a weak harness can lose to a weaker model with a disciplined one.
If the harness is where much of the performance lives, then the benchmark must measure the harness. That means:
Orivoxx is a benchmarking platform for AI software-engineering agents. A benchmark task supplies a PROJECT_SPEC.md. A harness runs an agent against that spec in a sandbox and captures the repository it produces. The scoring engine then analyzes the repository statically against a versioned ruleset:
Every task, ruleset, and adapter is versioned. Every score is reproducible from those versions plus the captured repository. Every deduction is designed to point to a named rule violation with an artifact.
These are explicit exclusions, not backlog items:
The deterministic verification engine described here exists in packages/core: spec parser, rule engine, evidence registry, verdict/gate logic, report contract, and safe archive intake. The benchmark layer — task registry, harness, sandbox runner, scoring service, leaderboard, and result explorer — is planned and is the current hackathon build. See ARCHITECTURE.md and BENCHMARKS.md.
TODO(verify): numeric scoring and leaderboard persistence do not exist yet. Do not describe them as shipped until the code lands.