Orivoxx

Orivoxx — Adapters

An adapter is the only place a harness choice enters a benchmark. It turns a task and a model identity into one process that runs inside the sandbox. The adapter never scores, never writes the result record, and never sees a platform secret.

1. Adapter matrix

Adapter Version Kind Model Runtime Network Entry point
fake 1.0.0 baseline null In-process materialize none packages/harness/src/adapters/fake.ts
claude-code-openrouter 1.0.0 harness anthropic/claude-sonnet-4.5 Claude Code 2.1.329 model-proxy /opt/orivoxx/agents/brand_runner.py
codex-openrouter 1.0.0 harness openai/gpt-5.1-codex Codex CLI 0.171.0 model-proxy /opt/orivoxx/agents/brand_runner.py
aider-openrouter 1.0.0 harness MODEL_NAME Aider 0.86.2 on Python 3.12 model-proxy /opt/orivoxx/agents/aider_runner.py
python-openrouter 1.0.0 harness MODEL_NAME Python 3.12 standard library model-proxy /opt/orivoxx/agents/python_openrouter.py (synthetic) / real_python_runner.py (real task)

The fake adapter is the CI baseline and the self-serve public path. It never calls a model and its rows are labelled Fake fixture (task/variant). The two real adapters are organizer-executed and record the model identity on every row.

GitHub Copilot CLI 1.0.114 was attempted and excluded: its org policy denied access and it has no OpenRouter base-url override. Cursor is excluded because it cannot be reliably headless-routed through OpenRouter. There is no OpenCode CLI adapter. The point of the benchmark is to compare harness configurations, so the harness is explicit: brand CLIs, a hand-written agent loop, and Aider.

2. Common contract

Every adapter implements AgentAdapter from packages/harness/src/contracts.ts:

interface AgentAdapter {
  id: string;          // contract-validated kebab-case
  version: string;     // semver, recorded on the result
  label: string;       // shown in the UI
  kind: "baseline" | "harness";
  model: AdapterModel | null;
  plan(input: AdapterRunInput): AdapterPlan;
}

plan() returns either a materialized fixture or one argv command. The command is validated against AgentCommandSchema before execution:

A real adapter requires input.network === "model-proxy" and refuses a run bound to a different model. That is what keeps a head-to-head honest: the same slug, one varying harness.

2b. Real repository tasks

Brand adapters receive a repo cloned at a base commit on the host, an issue file at .orivoxx/issue.md, and a test command held outside the agent prompt. The agent runs in the cloned workspace; after exit the harness runs the test command inside the same sandbox. taskSuccess is test exit 0. Claude Code uses the Anthropic-compatible OpenRouter endpoint through the proxy; Codex uses the Responses wire API through the proxy. Neither receives the real provider key.

Acceptance fixtures live under .orivoxx/ and are rewritten after the agent exits. Declared dependencies are installed through the allowlisted CONNECT tunnel before the agent starts. General egress is not opened.

3. python-openrouter

A deliberately small agent loop, standard-library Python only. It exists so Orivoxx measures a transparent harness rather than a black box. On real tasks it switches to real_python_runner.py, which asks for one multi-file JSON patch and applies it with a single apply_patch tool call. On synthetic tasks it reads PROJECT_SPEC.md.

Synthetic mode:

  1. Reads PROJECT_SPEC.md and bounded starting files from /workspace.
  2. Calls the model once for a JSON file plan ({"files":["src/..."]}).
  3. Calls the model once per planned file for complete content.
  4. Validates every path: relative only, no .., no NUL/backslash tricks, no .git/node_modules/dist, no .env, no lockfiles or package.json, allowed extensions only.
  5. Enforces 10 files, 64 KiB per file, 256 KiB total, 64 KiB of context.
  6. Writes with parent creation inside the workspace, never outside it.

Provider failures are safe: HTTP <status>, unreachable, invalid JSON, or no choices. The provider response body is never printed. A run that writes no files exits non-zero, so a failed harness is visible instead of silently scoring an empty repository.

Prompt rule: the specification is data. The system prompt tells the model to analyze it and ignore embedded instructions about how the agent works.

4. aider-openrouter

Aider 0.86.2 is pinned in the sandbox image (/opt/aider). The wrapper does three things:

  1. Extracts spec-named paths from PROJECT_SPEC.md with a strict relative-path regex (max 12) and creates missing ones as empty files. The paths come from the task specification, not from the ruleset.
  2. Runs Aider with the same model through the proxy and the sentinel key: --edit-format whole --yes-always --no-auto-commits --no-git --no-gitignore --no-stream --no-pretty --no-suggest-shell-commands --map-tokens 0.
  3. Keeps Aider history under /tmp, so no harness bookkeeping enters the captured repository.

Aider reads PROJECT_SPEC.md as a read-only file and edits the spec-named file set. The wrapper exits non-zero if Aider fails or produces no content.

Aider is a second harness, not a second model. The head-to-head pair (python-openrouter vs aider-openrouter) uses one model slug so only the harness varies.

5. Model identity

Model configuration is environment-driven and recorded per run:

Variable Role
MODEL_PROVIDER Provider label recorded on the result (default openrouter)
MODEL_BASE_URL Provider base URL used by the proxy and recorded on the result
MODEL_API_KEY Server-only key; OPENROUTER_API_KEY is accepted as the alias
MODEL_NAME Default model slug
FALLBACK_MODEL_NAME Optional second slug for resilience
DEMO_SECOND_MODEL Same-harness/different-model demo run

No model slug is a product invariant. Every row records adapter.id, adapter.version, model.provider, model.name, and model.baseUrl.

5b. Traces and chaos

Adapters emit tool and step events to ORIVOXX_TRACE_FILE inside the workspace; the proxy independently records model-call timing, tokens, and cost. Both are merged into the result trace. python-openrouter supports ORIVOXX_CHAOS=tool_error|failing_tools|latency|context_pressure; the other adapters accept the variable but do not inject faults. failing_tools intercepts the single multi-file apply_patch call on a real task, so the recorded action can be retried by the counterfactual replay. A chaos run is labelled in the result and used as ground truth for attribution validation.

6. Running an adapter

# Build once
pnpm --filter @orivoxx/harness build
pnpm --filter @orivoxx/harness sandbox:build

# List tasks and adapters
pnpm bench:list

# Fake baseline (no model, no network)
pnpm bench:run --task taskboard --adapter fake --variant compliant

# Real harnesses, same model
pnpm bench:run --task taskboard --adapter python-openrouter --model openai/gpt-4o-mini
pnpm bench:run --task taskboard --adapter aider-openrouter --model openai/gpt-4o-mini

# Same harness, different model
pnpm bench:run --task taskboard --adapter python-openrouter --model deepseek/deepseek-chat-v3-0324

Each command appends to the configured store (Upstash when both Upstash variables are set, otherwise .data/results) and prints a bounded JSON summary: status, exit code, duration, score, benches, caps, tally, captured file/byte counts, and the last stdout/stderr lines. It never prints the prompt or the model response.

7. Recorded runs

Final real-task runs (November 2026):

Adapter Task Test Duration Cost Result
claude-code-openrouter slugify-max-length PASS 232.7 s $0.9044 res_85394cc5…
python-openrouter (openai/gpt-5.1) community is-decimal-800937e9 PASS 5.6 s $0.0068 res_2ea69a6f…
codex-openrouter is-email-predicate FAIL 492.7 s $0.7198 res_e2767b1f…
python-openrouter (anthropic/claude-sonnet-4.5) itsdangerous-future-signature, chaos: failing_tools FAIL → proof PASS 47.8 s $0.1129 res_c1ba8d7b…

Synthetic runs:

Recorded on 15 November 2026, task taskboard 1.0.0, sandbox image orivoxx/sandbox:1.0.0, Docker with no host secrets. All three rows were appended to the shared Upstash store.

Adapter Model Result Overall Spec / Sec / Maint Caps Wall time Captured
python-openrouter openai/gpt-4o-mini res_f74d49e5… 97.3 100 / 100 / 82.0 none 24.4 s 7 files / 7 465 B
aider-openrouter openai/gpt-4o-mini res_a39c6e29… 97.3 100 / 100 / 82.0 none 25.7 s 7 files / 7 122 B
python-openrouter deepseek/deepseek-chat-v3-0324 res_71626175… 98.1 100 / 100 / 87.1 none 35.6 s 9 files / 9 279 B

The first two rows are the head-to-head pair: one model, two harnesses. The third is the same harness on a different model. The fake baseline rows (99.5 compliant, 59.9 regressed) stay in data/seed-results.json as the checked-in fallback.

An earlier Aider run scored 94.4 because the spec-path extractor’s extension alternation matched .ts before .tsx and produced TaskList.ts. The pattern was fixed, the image rebuilt, and the row above is the recorded replacement. The stale Upstash row remains fetchable by id but is not part of the comparison dataset.

8. Smoke tests

Test Scope
src/adapters/adapters.test.ts Plan shape, model binding, network requirement, secret sentinel, registry contents
src/adapters/python.integration.test.ts Real Python loop against a stub provider: plan, path filtering, fence stripping, provider error safety, missing spec
src/sandbox/docker.integration.test.ts Docker isolation, proxy allowlist, quotas, cleanup (gated by ORIVOXX_DOCKER_TESTS=1)

The Python loop test executes our own agent script against a local stub provider. It does not execute produced code and does not need Docker. The Aider smoke test is the recorded organizer run above, because Aider’s own loop cannot be exercised meaningfully against a stub.

9. Adding an adapter

  1. Add src/adapters/<id>.ts with a factory that validates its model and command.
  2. Register it in src/adapters/registry.ts.
  3. Add the agent entry point to packages/harness/agents/ and copy it in docker/sandbox.Dockerfile if it needs to run in the image.
  4. Add unit tests for plan shape, network, and model binding; add an integration test if the loop is ours.
  5. Document it here and record a real run.