Running log for the autonomous build. Newest entries at the bottom. This is a record, not an approval gate.
Inputs received: defaults accepted; LICENSE holder “Orivoxx Contributors”; push and Vercel deployment authorized; OpenRouter only; Nebius demoted; adapters fake, python-openrouter, aider-openrouter; OpenCode CLI removed as an adapter; reference video at demo/reference/reference.mp4 (confirmed present, 8.8 MB); Docker/package install/network permissions granted.
Environment verified: Linux root shell; Node 22 default but Node 24.22.1 available at ~/.nvm/versions/node/v24.22.1; pnpm 10.34.5; git 2.43.0; ffmpeg 6.1.1 with libx264/aac; Python 3.12.3; Docker 29.1.3 daemon running; Playwright Chromium cached; network egress to npm/GitHub/Vercel/OpenRouter/ElevenLabs; ~19 GB free; gh authenticated as onyebuchidaniel60; vercel authenticated as onyebuchidaniel60-1034; no Status.txt.
Baseline commit: docs: reposition Orivoxx as an agent benchmark platform (19 doc files + .gitignore, HEAD before this build 364a112).
MODEL_PROVIDER=openrouter, MODEL_BASE_URL=https://openrouter.ai/api/v1, MODEL_API_KEY falls back to OPENROUTER_API_KEY. Nebius adapter remains optional/demoted.openai/gpt-4o-mini, deepseek/deepseek-chat-v3-0324, deepseek/deepseek-v3.2, deepseek/deepseek-v4.1-flash-1017:batch.MODEL_NAME=openai/gpt-4o-mini; fallback deepseek/deepseek-chat-v3-0324; demo second model deepseek/deepseek-chat-v3-0324; head-to-head harness pair uses deepseek/deepseek-v3.2 so both harnesses share one model.UNVERIFIED = 0 earned, reported separately; critical spec cap 59.9, critical security cap 49.9.ResultStore interface; local append-only JSON under .data/results; Upstash Redis in deployment; checked-in seed results. 90-day artifact retention; scorecard JSON only, no full source artifacts.PROJECT_SPEC.md stays historical; each task owns a versioned spec; docs/BENCHMARK_TASK_TEMPLATE.md added.LICENSE (MIT).AGENTS.md: rule 3 now permits sandboxed execution and states the static-scoring invariant; provider section moved to OpenRouter; added score/sandbox/version invariants.IMPLEMENTATION_PLAN.md to P0–P11; appended the old verifier plan under ## Historical (verifier era, superseded) with demoted headings.AI_HANDOFF.md.packages/benchmarks.packages/benchmarks)Changed files
packages/benchmarks/ new workspace package: package.json, tsconfig.json, tsconfig.typecheck.json, eslint.config.mjs, vitest.config.ts.src/contracts/scoring.ts, src/contracts/task.ts, src/contracts/index.ts — Zod contracts for semver, bench/severity weights, structural thresholds, scoring config, task manifest, ruleset, task definition, task summary.src/fingerprint.ts — canonical JSON + SHA-256 ruleset fingerprint.src/tasks/build.ts — deterministic-only enforcement, bench assignment validation, spec/starting-file agreement, ruleset construction.src/tasks/registry.ts — deterministic registry with semver ordering, pinned tasks, duplicate/empty rejection.src/tasks/taskboard/ — task spec (9 deterministic requirements), starting repo, compliant/regressed fixtures, generated TS.src/tasks/linkstash/ — second task (12 requirements: 9 spec, 3 security), starting repo, compliant/regressed fixtures, generated TS.src/files.ts — normalized project files: path validation, exclusion, secret redaction, fingerprint, language detection.src/scoring/config.ts — scoringConfigVersion 1.0.0: spec 60 / security 25 / maintainability 15, severity 8/4/2/1, caps 59.9/49.9, structural thresholds.scripts/generate-task-assets.mjs — regenerates task asset modules from specs/starting/fixtures.docs/BENCHMARK_TASK_TEMPLATE.md — task authoring contract.packages/core/src/archive/redact.ts — added OpenRouter key redaction category (sk-or-v1-…), needed because produced/starting files may embed keys.Verification
pnpm --filter @orivoxx/core test → 16 files, 217 tests passed.pnpm --filter @orivoxx/benchmarks test → 2 files, 26 tests passed.pnpm --filter @orivoxx/benchmarks typecheck → exit 0.pnpm --filter @orivoxx/benchmarks lint → exit 0.pnpm --filter @orivoxx/benchmarks build → exit 0, dist emitted (gitignored).Decisions
text_contains marker (assertProjectOwner), so the benchmark keeps a model-free score path. Recorded in the task spec preamble.PROJECT_SPEC.md stays historical; each task owns its own PROJECT_SPEC.md inside packages/benchmarks.tsconfig.typecheck.json maps @orivoxx/core to source with rootDir: ../..; tsconfig.json (build) resolves the built package. This avoids emitting core files into the benchmarks package.next/server imports and unsafe patterns.Known issues
packages/core build currently emits .js/.d.ts/.map into src when run without a clean; stray files were deleted. Root cause is the prebuild chain invoking tsc with a rootDir that resolves oddly under pnpm -r; tracked for P3 when the package build graph is revisited.packages/harness)Changed files
packages/harness/ new workspace package with build/typecheck/lint/vitest config.src/contracts.ts — AdapterRunInput, AgentCommand, AdapterRunOutput, CapturedFile, artifact stats, AgentAdapter, AgentExecutor, hard limits, and the sandbox env allowlist. Command env rejects unknown names, provider-shaped secrets, and non-sentinel API keys.src/capture.ts — bounded recursive artifact capture: path validation, exclusion denylist, symlink rejection, per-file/aggregate/file-count ceilings, UTF-8 and binary checks, secret redaction, deterministic repository fingerprint, conversion back to NormalizedFile for static scoring.src/workspace.ts — run workspace creation, empty-directory precondition, safe starting-file/produced-file writes with containment checks, guarded cleanup.src/run.ts — runAdapter() orchestration and buildAdapterRunInput() from a versioned task; bounded/redacted stdout/stderr summaries; status derivation; output schema validation.src/adapters/fake.ts — baseline adapter that materializes a compliant or regressed task fixture without a model.src/adapters/registry.ts — adapter lookup and summaries (only fake until P6).Verification
pnpm --filter @orivoxx/harness test → 3 files, 32 tests passed.pnpm --filter @orivoxx/harness typecheck → exit 0.pnpm --filter @orivoxx/harness lint → exit 0.pnpm --filter @orivoxx/harness build → exit 0.Decisions
OPENAI_API_KEY / AIDER_OPENAI_API_KEY inside the sandbox may only hold the orivoxx-sandbox-proxy sentinel; the real provider key never enters a container. This lets Aider-style harnesses start while the allowlisted proxy injects the real credential host-side.null and the label names the fixture variant, so the leaderboard can never mistake it for an agent result.Known issues
fake is runnable end to end.Changed files
packages/core/src/report/pipeline.ts — extracted computeRepositoryFingerprint() (path + content fingerprints) so the harness and the scorer share one repository identity.packages/harness/src/capture.ts — capture fingerprint now delegates to the core function.packages/benchmarks/src/contracts/score.ts — Zod result contract: version fields, bench scores, structural metrics, requirement outcomes, caps, unverified IDs. Rejects duplicate requirement IDs, a cap without a reason, and an overall above its cap.packages/benchmarks/src/scoring/score.ts — pure scoreRun(): severity-weighted bench scores, UNVERIFIED earns zero without capping, critical-failure caps at 59.9/49.9, deterministic ordering, version/fingerprint enforcement.packages/benchmarks/src/scoring/structural.ts — static import graph: module set, resolved relative edges, Tarjan SCC cycle count, max fan-out, entrypoint-aware orphan ratio, published degradation curves.packages/benchmarks/src/scoring/evaluate.ts — static evaluation entry point: core checks → outcomes → structural metrics → score. No produced code is executed.packages/benchmarks/src/tasks/registry.ts — registry now rejects a task without at least one spec and one security requirement.@orivoxx/benchmarks, including determinism, ordering, UNVERIFIED, caps, weight changes, version-field rejection, structural thresholds, fixture evaluation, and an import-graph proof test.docs/BENCHMARKS.md — published maintainability formula and corrected cap reason names.Verification
pnpm --filter @orivoxx/benchmarks test → 6 files, 63 tests passed.pnpm --filter @orivoxx/benchmarks typecheck → exit 0.pnpm --filter @orivoxx/benchmarks lint → exit 0.pnpm --filter @orivoxx/benchmarks build → exit 0.pnpm --filter @orivoxx/core test → 217 tests passed.pnpm --filter @orivoxx/harness test → 32 tests passed.99.5 (spec 100, security 100, maintainability 96.7, no cap); regressed 59.9 (spec 52.4, security 100, maintainability 90, critical-spec-failure cap).Deterministic proof
import-graph.test.ts walks the real import graph from score.ts, evaluate.ts, structural.ts, and round.ts. It fails if the graph reaches @orivoxx/harness, apps/web, Nebius, OpenAI, Anthropic, undici, node:http(s), node:net, node:dns, or node:child_process; if the source contains fetch(, process.env, Math.random, Date.now, new Date, or child_process; or if it depends on anything outside @orivoxx/core, node:path, and zod.Decisions
FAIL and DRIFT trigger critical caps; UNVERIFIED does not. Missing evidence already earns zero; treating it as a proven critical failure would overstate certainty.Known issues
Changed files
packages/harness/src/sandbox/docker.ts — SandboxRunner (alias of AgentExecutor) and DockerExecutor: image preflight, workspace ownership, network setup, proxy sidecar lifecycle with readiness probe, spawn-based container run, timeout kill, bounded output, guaranteed cleanup. ensureSandboxNetworks() exported for tests and tooling.packages/harness/src/sandbox/proxy.ts — allowlisted model proxy: /v1-only path allowlist, traversal/encoded-traversal rejection, GET/POST only, 1 MiB request budget (declared and streamed), 2 MiB response budget, 60 s upstream timeout, credential strip/inject, header allowlist, pinned target origin, no-store.packages/harness/src/sandbox/proxy-server.ts — sidecar entry point reading MODEL_BASE_URL, MODEL_API_KEY, PROXY_PORT.packages/harness/src/sandbox/index.ts — sandbox exports.packages/harness/docker/sandbox.Dockerfile — node:24-bookworm-slim, ca-certificates, git, python3 + venv, uid/gid 10001, HOME=/tmp.packages/harness/scripts/build-sandbox.mjs — image build; packages/harness/scripts/write-proxy-package.mjs — writes dist/sandbox/package.json (type: module) so the sidecar runs the ESM build from a read-only mount.packages/harness/package.json — sandbox:build script; build now emits the proxy package marker.packages/harness/src/errors.ts — sandbox error codes (sandbox_unavailable, sandbox_image_missing, sandbox_proxy_misconfigured, sandbox_proxy_build_missing, sandbox_run_failed).packages/harness/src/sandbox/proxy.test.ts — 18 unit tests.packages/harness/src/sandbox/docker.integration.test.ts — 12 Docker tests gated by ORIVOXX_DOCKER_TESTS=1.docs/SANDBOX.md — container profile, network topology, proxy policy, workspace lifecycle, timeouts, operations, test evidence.docs/ARCHITECTURE.md — shipped-status corrections for packages/harness, packages/benchmarks, apps/runner (superseded), scoring boundary, execution flow.Verification
pnpm --filter @orivoxx/harness sandbox:build → image orivoxx/sandbox:1.0.0 built (id 771393837211).pnpm --filter @orivoxx/harness test → 4 files, 50 tests passed; 12 integration tests skipped without the gate.ORIVOXX_DOCKER_TESTS=1 pnpm --filter @orivoxx/harness test → 5 files, 62 tests passed in ~39 s.pnpm --filter @orivoxx/harness typecheck → exit 0.pnpm --filter @orivoxx/harness lint → exit 0.pnpm --filter @orivoxx/harness build → exit 0; dist/sandbox/proxy-server.js + package.json present.network: none): status completed, exit 0, 1020 ms, 8 captured files / 7 904 bytes, score 99.5 (spec 100, security 100, maintainability 96.7), 9/9 PASS, no cap, fingerprint b6a8435d66ca568f….docker ps -a --filter name=orivoxx- empty; no host files created outside the workspace.Deterministic boundary proof
packages/benchmarks to packages/harness; evaluateRepository consumes NormalizedFile[] produced by toNormalizedFiles(output.files).output.repositoryFingerprint equals score.repositoryFingerprint.Sandbox proofs
HOME, TMPDIR, NODE_ENV, plus schema-allowlisted adapter variables. MODEL_API_KEY, MODEL_BASE_URL, and OPENROUTER_API_KEY are absent; host secret values never appear in the dump. OPENAI_API_KEY inside the container is exactly the orivoxx-sandbox-proxy sentinel.network: "model-proxy", /healthz returns 200, a proxied POST reaches a sibling fake-provider container at /api/v1/chat/completions with Authorization: Bearer <provider key> injected host-side, while a direct request to that sibling container and a raw 1.1.1.1 request both fail./etc/passwd, /escape-root.txt, and /workspace/../escape.txt return EROFS/EACCES/EPERM; /tmp is writable; nothing appears on the host.timedOut, no exit code, container removed); 4 KiB output budget truncates a 2 MiB stream; pids-limit 64 yields EAGAIN for a 300-process fork loop; memory 512 MB, 1 CPU, 64 MB /dev/shm, 256 file descriptors.orivoxx-run-* or orivoxx-proxy-* container survives any test.Decisions
SandboxRunner is exported as the platform’s only permitted executor type; DockerExecutor is its single implementation. This satisfies the plan’s “SandboxRunner interface plus Docker implementation” without inventing a parallel abstraction.docs/SANDBOX.md §5. Root filesystem is still --read-only with a bounded tmpfs /tmp./v1/<suffix> → ${MODEL_BASE_URL}/<suffix>, so adapters use the standard OpenAI base URL and the provider base path (e.g. /api/v1 on OpenRouter) is configuration, not adapter code.host.docker.internal: host-gateway is deliberately not added, so a compromised agent cannot probe host services.ORIVOXX_DOCKER_TESTS=1 but are part of the required verification for this phase and for any sandbox change.Known issues
python-openrouter adapter that wants requests must use the standard library or the image must be extended and versioned; adapter dependencies are pinned in the image, not installed at run time.chown on bind mounts; prepareWorkspace falls back to chmod -R a+rwX. Linux organizer machines are the supported path.502 response_too_large.Changed files
packages/results/ — new package.
src/contracts.ts — BenchmarkResult and LeaderboardEntry Zod contracts, buildBenchmarkResult, tallyRequirements, deriveBenches, toLeaderboardEntry. A superRefine enforces internal consistency: denormalized overall/benches/tally/capsApplied/version fields must equal the embedded scorecard, and a completed run must exit 0.src/store.ts — ResultStore interface (append, get, list, listByTask), ResultStoreError codes, compareLeaderboardEntries (score desc, earlier createdAt, then resultId), rankLeaderboard, clampLimit.src/local.ts — LocalResultStore: one immutable file per result under .data/results, writeFile with flag: "wx" so a duplicate id cannot overwrite evidence.src/upstash.ts — UpstashResultStore: single Lua EVAL writes the result document plus rank:all and rank:task:<id> indexes atomically; an existing id aborts the whole append. Bounded ZRANGE with overscan, per-member validation, store_unavailable on transport/HTTP failures without leaking a response body.src/seed.ts — SeedResultStore: read-only bundled rows; an invalid row is dropped with a seed_res_<n>_invalid diagnostic instead of failing the page.src/read.ts — readLeaderboard/readResult fallback reads with an explicit degraded flag; createResultStore factory (Upstash when both env vars exist, local otherwise).src/id.ts, src/samples.ts, src/index.ts, package/tsconfig/eslint/vitest config.contracts.test.ts, ordering.test.ts, local.test.ts, seed.test.ts, upstash.test.ts — 53 tests.packages/harness/src/diff.ts — buildProducedDiff: bounded unified diff between the starting repository and the captured repository, LCS-based edit script, byte/file/lines-per-file budgets, truncated flag, deterministic file order. src/diff.test.ts — 11 tests.packages/harness/src/benchmark.ts — runBenchmark: adapter run → capture → evaluateRepository → diff → buildBenchmarkResult. src/benchmark.test.ts — 5 tests.packages/harness/scripts/record-seed.mjs — records data/seed-results.json from real Docker sandbox runs of the fake baseline.packages/harness/scripts/publish-results.mjs — organizer publisher: appends recorded results to the configured store, skips duplicates.data/seed-results.json — 4 rows (taskboard 99.5 / taskboard 59.9, linkstash 99.3 / linkstash 49.9), each a real recorded sandbox run.apps/web/src/server/env.ts — model provider env (MODEL_PROVIDER, MODEL_BASE_URL, MODEL_API_KEY with OPENROUTER_API_KEY alias, MODEL_NAME, FALLBACK_MODEL_NAME, DEMO_SECOND_MODEL), Upstash env, ORIVOXX_DATA_DIR. Nebius stays as an optional demoted block.apps/web/src/server/results.ts — server-only read layer: bundled seed store, store factory, webLeaderboard, webResult, getSeedStatus, fallback policy.apps/web/src/app/api/v1/leaderboard/route.ts — GET /api/v1/leaderboard with taskId/limit validation, no-store.apps/web/src/app/api/v1/results/[id]/route.ts — GET /api/v1/results/:id, fixed 404 envelope for unknown and malformed ids.apps/web/src/app/api/v1/health/route.ts — non-secret provider and leaderboard status.apps/web/next.config.ts, apps/web/tsconfig.json, apps/web/vitest.config.mts, apps/web/package.json — build/runtime wiring.docs/RESULTS_RETENTION.md — stored fields, exclusions, store layout, 90-day retention, privacy, API surface.docs/ARCHITECTURE.md — results package, read APIs, shipped statuses..gitignore — .data/.src/server/results.test.ts, leaderboard/route.test.ts, results/[id]/route.test.ts — 18 tests.Verification
pnpm -r test → core 217, benchmarks 63, results 53, web 18, harness 66 passed + 12 Docker tests skipped.pnpm -r typecheck → all packages and the web app pass.pnpm -r lint → all packages and the web app pass.pnpm -r build → all packages and next build pass; routes /api/v1/health, /api/v1/leaderboard, /api/v1/results/[id] are dynamic (ƒ).node packages/harness/scripts/record-seed.mjs → 4 rows written from real Docker sandbox runs.SeedResultStore load of data/seed-results.json → 4 rows, 0 diagnostics; first row res_1ab24b38… has a 3 611-byte diff over 4 files.node --env-file=.env.local packages/harness/scripts/publish-results.mjs data/seed-results.json → 4 appended to Upstash, 0 duplicates, 4 rows visible.next start, port 3101) against Upstash:
GET /api/v1/health → {"model":{"provider":"openrouter","name":"openai/gpt-4o-mini","configured":true},"leaderboard":{"store":"upstash","seed":{"available":true,"rows":4}}}GET /api/v1/leaderboard → store: "upstash", degraded: false, 4 rows ordered 99.5, 99.3, 59.9, 49.9.GET /api/v1/results/res_1ab24b38… → 200, cache-control: no-store, x-content-type-options: nosniff, 9 requirement outcomes, 3 611-byte diff.{"error":{"code":"not_found","message":"No result with that id is stored."}}.?limit=999 → 400 invalid_query without echoing the input.store: "seed", 4 rows, so the read path works from seed data alone.Decisions
ResultStoreError codes are stable and safe to expose.-overall with the leaderboard JSON as the member, so ZRANGE is already ranked and the tie-break is completed in memory with a 50-row overscan.process.cwd().degraded: true; empty local store → seed, labelled sample data; production without a shared store → seed, read-only. An Upstash store that answers with zero rows stays empty (an empty shared store is a real state, not an error)..env.local; apps/web/.env.local is a gitignored symlink to it so next dev and next start see the same values. Vercel uses dashboard env vars.apps/web/tsconfig.json no longer aliases @orivoxx/core to TypeScript source. The app consumes the built packages (dist), which keeps .js specifiers in package output resolvable by Turbopack.@orivoxx/benchmarks and @orivoxx/results; pnpm -r build orders the build topologically.Known issues
list(). That is fine for a demo-scale directory; the shared store is indexed.Files
packages/harness/agents/python_openrouter.py — stdlib agent loop: plan call, one call per file, path/size validation, safe provider errors.packages/harness/agents/aider_runner.py — spec-path extractor + pinned Aider 0.86.2 wrapper (whole edit format, no git, history in /tmp).src/adapters/python.ts, src/adapters/aider.ts, src/adapters/model.ts, registry + exports.src/adapters/adapters.test.ts, python.integration.test.ts, aider.integration.test.ts.docker/sandbox.Dockerfile — pip-installs aider-chat==0.86.2 into /opt/aider, copies agent entry points; build context moved to the package root with .dockerignore.scripts/run-benchmark.mjs — organizer CLI; bench:list / bench:run / bench:publish root scripts.docs/ADAPTERS.md, docs/FRONTEND_BENCHMARK_SPEC.md (P7 contract, written before UI work).Verification
orivoxx/sandbox:1.0.0 (1.44 GB).openai/gpt-4o-mini → 97.3 (100/100/82), 9/9 PASS, 24.4 s, res_f74d49e5….openai/gpt-4o-mini → 97.3 (100/100/82), 9/9 PASS, 25.7 s, res_a39c6e29….deepseek/deepseek-chat-v3-0324 → 98.1 (100/100/87.1), 9/9 PASS, 35.6 s, res_71626175…..ts winning over .tsx in the spec-path regex; fixed, image rebuilt, rerun recorded.Decisions
PROJECT_SPEC.md, never the ruleset, so the harness does not leak rubric-only paths.bench:run is organizer-only; the public app has no arbitrary-command field.Known issues
Input is not a terminal warning to stderr.Files
docs/FRONTEND_BENCHMARK_SPEC.md — list-first UI contract: tokens, routes, page contracts, status vocabulary, states, a11y, responsive, component inventory.apps/web/src/components/ — app-shell, site-nav, status-chip, score-block, cap-banner, version-strip, tally-chips, requirement-list, code-block, diff-view, leaderboard-table, task-card, empty-state, submit-form, compare-panel./, /benchmarks, /benchmarks/[taskId], /submit, /results/[resultId], /leaderboard, /compare, /privacy, not-found; dark-only globals.css tokens; metadata.tasks, tasks/[taskId], runs (fake self-serve), results (list alias).src/server/runs.ts — fake run: materialize → capture → score → append.Verification
pnpm --filter web test → 6 files, 32 tests pass.next build pass; 14 routes compiled./, /benchmarks, /benchmarks/taskboard, /leaderboard, /compare, /submit, /privacy → 200; unknown result → 404.POST /api/v1/runs {taskboard, compliant} → 201 res_b07ed478…, overall 99.5, store upstash./results/<id> renders score, requirement outcomes, and diff; /leaderboard shows python, aider, and fake rows; /compare?task=taskboard defaults to python vs aider with Same model.Decisions
Known issues
/docs is linked from nav but arrives in P9.Files
src/server/rate-limit.ts — @upstash/ratelimit sliding window (10 requests / 10 min), 2 s timeout, hashed client id (no raw IP), fail-closed in production when the shared limiter is missing or unreachable.src/server/origin.ts — same-origin check for mutating requests, including APP_BASE_URL and forwarded host/proto, plus sec-fetch-site.src/server/runs.ts — two-run concurrency ceiling per instance.src/app/api/v1/runs/route.ts — origin → rate limit → 4 KiB body → schema → task → slot; safe envelopes with requestId; rate headers.src/middleware.ts — x-request-id on every request/response.next.config.ts — CSP, HSTS, X-Frame-Options: DENY, nosniff, referrer policy, permissions policy.hardening.test.ts + route abuse tests (origin, production fail-closed, request id, concurrency).Verification
x-request-id present.forbidden_origin; missing Origin → 403.x-request-id echoed, x-ratelimit-remaining decreasing; 11th request → 429 rate_limited with retry-after.Decisions
rate_limit_unavailable for mutations rather than allowing unlimited runs. Read routes stay available.@upstash/ratelimit; the wrapper treats a thrown error as unavailable, and a missing limiter is denied in production. The 2 s timeout is a bounded fallback, not a silent bypass: response headers expose remaining quota.Known issues
'unsafe-inline') because Next injects inline bootstrapping; a nonce-based policy is future work.Files
README.md — rewritten: TOC, true badges, hero, problem, three benches, 8 real screenshots, architecture, quickstart, real-run commands and recorded results, adding a task, adding an adapter, deterministic/sandbox guarantees, retention, API, structure, testing, hackathon, license, acknowledgements.docs/assets/screens/*.png — 9 Playwright captures from the production build (home, benchmarks, task, submit, leaderboard, compare, result scorecard, requirement explorer, capped result).apps/web/src/content/docs.ts + /docs and /docs/[slug] — sidebar, breadcrumb, search, 10 sections from Overview to Hackathon note.apps/web/scripts/screenshots.mjs — repeatable capture.ARCHITECTURE.md status, BENCHMARKS.md maintainability shipped, HACKATHON.md current shipped/roadmap, PRIVACY_NOTICE.md accuracy, AI_HANDOFF.md.Verification
node_modules, no .env.local, no .git): pnpm install --frozen-lockfile, pnpm -r build, pnpm -r typecheck, pnpm -r test all pass. Totals: core 217, benchmarks 63, results 53, harness 82 (+12 skipped), web 44 = 459 passing./, /leaderboard, /docs, /api/v1/health → 200; leaderboard store: seed, 4 rows, baselinesOnly: true.Decisions
Sample.Known issues
docs/FRONTEND_DESIGN_SPEC.md remains as historical visual reference; the benchmark UI contract is docs/FRONTEND_BENCHMARK_SPEC.md.Files
demo/orivoxx-demo.mp4 — 2:38.8, 1920×1080, 30 fps, H.264/AAC, −16.1 LUFS, −1.8 dBTP, 8.5 MiB.demo/thumbnail.png, demo/thumbnail.jpg — result-scorecard cover.demo/SCRIPT.md, demo/REFERENCE_NOTES.md, demo/README.md.demo/scripts/narration.mjs, generate-audio.mjs, generate-piper-audio.mjs, record-scenes.mjs, assemble.mjs.demo/audio/scene-01..08.wav (narrration), demo/audio/elevenlabs/ (3 evidence renders)..gitignore excludes demo/build/ and demo/recordings/.Verification
ffprobe: duration 158.828 s, video h264 1920×1080 30/1, audio aac 48 kHz stereo.Decisions / deviation
T2sZgOl8ol4kf4u6yM2z is a library voice; the free plan returns HTTP 402 paid_plan_required. Remaining quota was 144/10,000 characters; three scenes consumed it and scene 4 required 390. Reset is 12 Dec 2026, after the deadline.en_US-lessac-medium for all eight scenes, one consistent voice, so the video is complete. The three ElevenLabs renders are preserved in demo/audio/elevenlabs/. generate-audio.mjs remains for a future rebuild if credits are restored. This is a recorded deviation from the ElevenLabs requirement, forced by an external quota.Known issues
Files
e2e/benchmark-flow.spec.ts — submit → run → score → leaderboard; regressed cap; keyboard toggle; 404; dark-only + overflow at 1440 and 390 px.playwright.config.ts, e2e/global-setup.ts — production server on :3300 with a throwaway local store and no shared Redis.SUBMISSION.md — Devpost field draft.README.md — live URL, deployment section, E2E command, correct repository URL.apps/web/src/components/app-shell.tsx — correct GitHub link.apps/web/src/server/results.ts — removed the production shortcut that made a local-store result unreadable; local results now win over the seed in every environment.apps/web/src/server/rate-limit.ts — explicit ORIVOXX_ENV=test escape hatch for E2E only.vercel project settings: root apps/web, Next.js preset, install/build from the repo root.Verification
pnpm -r test → 459 passed: core 217, benchmarks 63, results 53, harness 82, web 44.pnpm -r typecheck, pnpm -r lint, pnpm -r build → pass.ORIVOXX_DOCKER_TESTS=1 pnpm --filter @orivoxx/harness test → 94 passed (12 Docker).pnpm test:e2e → 5 passed in 19.1 s.packages/benchmarks/src/scoring/import-graph.test.ts passes; no provider/adapter/network import in the scoring source./, /benchmarks, /leaderboard, /compare, /submit, /docs, /docs/scoring, /privacy → 200./api/v1/health → model configured, store upstash, seed 4./api/v1/leaderboard → 3 harness rows (98.1 / 97.3 / 97.3), baselinesOnly: false.X-Frame-Options: DENY, nosniff, x-request-id present.POST /api/v1/runs same-origin → 201, rate-limit headers, store upstash; cross-origin → 403..env.local and synthetic test fixtures contain key-shaped strings; no real secret tracked.Decisions
ORIVOXX_ENV=test is the only way to bypass production fail-closed rate limiting, and Vercel never sets it.apps/web with install/build commands that run from the repo root; output .next.Known issues / open submission items
SUBMISSION.md is the draft.demo/README.md.git push (done in the audit sequence) and the human should confirm the MIT copyright holder.Reference studies (mandatory, before code)
demo/REFERENCE_NOTES.md — 17 cuts / 8.8s mean, ≈65% dissolves / ≈30% hard cuts, 138.3 WPM, 1.18s median pause, word-synced lower-third at 96% height, continuous bed ~5 dB under narration, URL-only outro.docs/UI_REFERENCE_NOTES.md — Coderia: #0B0B09 warm black, #E1E1DB text, #17CF97 accent, Instrument Serif 60–70px + Geist 14px, 90px section rhythm, 1280px width, fade/translate motion.docs/APP_HOME_REFERENCE_NOTES.md — Pass /app: six substantive numbered sections, embedded artifacts, orientation/action balance; Orivoxx plan 8 sections.docs/LANDING_REFERENCE_NOTES.md — Pass /: problem → solution → how → features → CTA; Orivoxx plan 13 sections.Shipped
packages/core/src/contracts/observability.ts — trace event union, bounds, metrics derivation, tests.packages/core/src/contracts/attribution.ts — 5×4 taxonomy, evidence, intervention, proof, reliability schemas.packages/results/src/aggregate.ts — mean/std, pass@k, layer distribution, chaos sensitivity, trend.packages/harness: proxy model trace, Docker trace mount, workspace trace/action collection, chaos env, runAdapterDetailed, attribution + replay proof in runBenchmark.packages/attribution — 12 rules, 12 labeled fixtures (precision 1.0, recall 1.0), unavailable/unattributed paths.packages/replay — deterministic replay, retry/stop/replacement interventions, no-intervention invariant, 5 tests.apps/web: marketing landing /, app home /app, product route group, Coderia tokens/Instrument Serif, attribution card, six-metric grid, trace page, replay page, reliability leaderboard, compare deltas, docs sections, API routes (attribute, replay, compare).demo/: new ElevenLabs narration (8 scenes, with-timestamps), ASS karaoke captions, dissolves, original ambient bed, 2:21.5 video.Real evidence
res_64068a9c11d744aa8c0391efa0bc04a4 — 49.7, harness × recovery, retry@2 → 98.3 PASS, 6 replayed steps.res_0427d9f5bf014f78ab8f630e8f101735 — 49.9, same cell, retry → 59.9 insufficient.res_df4fc568db41468ba499223509bcdd7e — 96.4 baseline, 45 trace events, 15,438 tokens, $0.0028.Verification
pnpm -r typecheck, pnpm -r lint, pnpm -r build pass (build requires AI_AGENT/AGENT/OPENCODE unset to avoid the Next security nudge).https://orivoxx.vercel.app: /, /app, /benchmarks, /leaderboard, /compare, attributed result, /replay/:id, /docs/attribution, /api/v1/health → 200; public fake run 201; cross-origin 403.Decisions
/app. Product routes use the app shell; marketing uses its own shell.trace_unavailable rather than inventing attribution.Known issues
middleware.ts is deprecated in Next 16.4.2 in favour of proxy; warning only.Brand agents
claude-code-openrouter (Claude Code 2.1.329, anthropic/claude-sonnet-4.5) and codex-openrouter (Codex CLI 0.171.0, openai/gpt-5.1-codex) to the sandbox image and adapter registry. Both route through the existing allowlisted proxy: Claude via ANTHROPIC_BASE_URL/sentinel token, Codex via a generated config.toml with the Responses wire API and sentinel OPENAI_API_KEY.real_aider_runner.py) when .orivoxx/issue.md is present.python-openrouter remains in source but is no longer in the headline registry.Real tasks
packages/benchmarks/src/tasks/real.ts — real-task schema (repoUrl, baseCommit, issueText, testCommand, passCondition, visibility, status, validation).is-decimal, is-hexadecimal, is-alphabetical, is-word-character, is-whitespace-character, is-number, each pinned to a real commit. Test commands are single node -e assertions that fail at base.real-registry.ts loads disk → remote → curated; real-score.ts scores test exit 0 as primary, with security/maintainability flags.real-task.ts clones on the host at the exact commit, strips .git, and bounds text files; the agent and test run in Docker.taskSuccess: true.test_failed_after_patch, patch_apply_failed_unhandled, test_run_skipped, test_timeout, issue_ambiguous_vs_test.Community ingestion
apps/web/src/server/community-tasks.ts — Upstash task keys, validation queue, per-user/global rate limits, private visibility, curator promotion/reject./api/auth/github, callback, logout) with signed session cookie when GITHUB_CLIENT_ID/SECRET exist; HttpOnly anonymous UUID fallback otherwise. viewer() hashes the anonymous token.POST /api/v1/tasks, GET ?mine=true, real-task detail with validation events, PATCH /promote, POST /reject./tasks, /tasks/new, /tasks/[taskId] (3s polling), /tasks/mine; app home card; leaderboard includes curated task filters and owner task list.packages/harness/scripts/validate-task.mjs; six gates in the same sandbox, license via GitHub API with LICENSE fallback.is-decimal-d36f6021 validated (all six gates pass), promoted by curator, public at /tasks/is-decimal-d36f6021.is-decimal-263f95c1 returns 403 for a second anonymous user and 200 for its owner.rate_limited.github.com/login/oauth/authorize with the correct callback.Verification
pnpm -r typecheck, pnpm -r lint, pnpm -r build pass (build with agent env unset).https://orivoxx.vercel.app: /tasks, /tasks/new, /tasks/mine, community task, real result, /leaderboard, /api/v1/health → 200; private task API → 403 for a second user.Decisions / limitations
res_64068a9c… (harness × recovery, 49.7 → 98.3).Proxy and package managers
registry.npmjs.org, registry.yarnpkg.com, pypi.org, files.pythonhosted.org. All other CONNECT targets return 403. General egress stays closed.anthropic-version, anthropic-beta, and openai-beta; the proxy injects the provider key as both Authorization and x-api-key./tmp tmpfs to 256 MiB so npm/pip caches fit. Workspace installs persist in the bind-mounted workspace.pip/pip3 to the test-command allowlist and HTTPS_PROXY/HTTP_PROXY/npm/pip cache variables to the adapter env allowlist.Harder real tasks
slugify-max-length (index.js + index.d.ts), is-email-predicate (source/index.ts + source/types.ts), itsdangerous-future-signature (exc.py + timed.py + init.py).installCommand and .orivoxx/ acceptance fixtures to the real-task schema. The harness rewrites fixtures after the agent exits so the acceptance test cannot be weakened.real.ts.Attribution proof
failing_tools to the chaos vocabulary and a real-task Python runner with a single multi-file apply_patch tool call and one repair retry.res_c1ba8d7b435e4ae6b514138458fab8b7: python (chaos: failing_tools) on itsdangerous-future-signature.
harness × recovery, rule patch_apply_failed_unhandled, evidence step 2./replay/res_c1ba8d7b….slugify-max-length (100.0, $0.9044). Codex failed is-email-predicate with a real incomplete patch (0.0, model × decision, no proof). A community is-decimal task passed with python-openrouter + openai/gpt-5.1 (100.0, $0.0068).Security and Next.js
eslint-config-next from 16.4.2 to 16.4.6. pnpm --filter web build now passes both with and without AI_AGENT/AGENT/OPENCODE/OPENCODE_SESSION_ID set.ORIVOXX_RUNS_DISABLED=1; mutating run and task routes return 503 before rate limiting or model work. Tested.child_process use outside packages/harness; no NEXT_PUBLIC_ secrets; OAuth/session/anonymous cookies are HttpOnly, SameSite=Lax, Secure in production; no provider keys or raw prompts in stored results..pytest_cache, __pycache__, .mypy_cache, and .ruff_cache to the static exclusion list.failing_tools to the reliability profile schema.Verification and docs
/app → submit community task → six gates → curator promotion → real sandboxed agent → result → leaderboard → attribution → replay. Task is-decimal-800937e9, result res_2ea69a6f….docs/screenshots/.README.md for the final state: brand agents, real tasks, 5×4 taxonomy, community gates, quickstart, architecture, screenshots, limitations.SUBMISSION.md proof/agent/task/limitations sections.Known issues
Video
missing_permissions (text_to_speech). Per the Phase 5 stop rule, the video was not re-recorded. The existing demo/orivoxx-demo.mp4 (2:21) is the pre-final cut; the final script and recording are blocked on a valid ElevenLabs key with TTS permission.demo/ files were modified in this pass.Next.js and deployment
next, eslint-config-next, @next/env). Build passes with and without AI_AGENT/AGENT/OPENCODE/OPENCODE_SESSION_ID.d77079c to https://orivoxx.vercel.app; /, /app, /tasks, /tasks/new, /tasks/mine, all three curated task pages, the community task page, both result pages, the replay page, /leaderboard, /api/v1/health → 200; /api/auth/github → 307./api/v1/tasks/is-decimal-263f95c1 → 403 for a fresh anonymous cookie.Fresh clone quickstart
git clone --depth 1 + pnpm install --frozen-lockfile + pnpm -r build + pnpm -r typecheck + pnpm --filter web build all pass. The pnpm -r build step is required because the web app consumes workspace dist output; README updated.Final data
is-decimal-800937e9, result res_2ea69a6f1de543c481d945793129d3bf, python-openrouter + openai/gpt-5.1, 100.0, $0.0068.res_85394cc5a29a4c89b0f7613e229ebd99, slugify-max-length, 100.0, $0.9044.res_c1ba8d7b435e4ae6b514138458fab8b7, python (chaos: failing_tools), 0.0 → 100.0, harness × recovery.res_e2767b1f393b4adb8a0d74cdeeb0afca, is-email-predicate, 0.0, model × decision, no proof.Tests
ORIVOXX_DOCKER_TESTS=1), including 12 integration tests.pnpm -r typecheck and pnpm -r lint pass with no warnings.Open items
Proxy package checks (sandbox)
npm install express inside the validation sandbox → exit 0 (12.6s).python3 -m pip install --target .deps requests inside the validation sandbox → exit 0 (6.8s).urllib.request.urlopen("https://example.com") through the same proxy → URLError: Tunnel connection failed: 403 Forbidden (exit 1). General egress remains blocked.Claude Code on the fresh community task
anthropic/claude-haiku-5.5 ran the fresh community task is-decimal-800937e9 in 11.8s for $0.0015 and passed at 100.0 (res_74101d58…). This completes the E2E requirement to run Claude Code against the newly validated task without using the exhausted Sonnet budget.Phase 1: missing real tasks
packages/benchmarks/src/tasks/real.ts, but taskSummaries() only called the synthetic listTasks() registry. The web /tasks page imported CURATED_REAL_TASKS directly, while /api/v1/tasks, /app, /benchmarks, and the landing page used the synthetic-only summaries.apps/web/src/server/tasks.ts to merge the three curated real tasks (kind: "real", repo, language, license) with the synthetic tasks. TaskCard and the app home now link real tasks to /tasks/:id; /benchmarks shows both families./api/v1/tasks now returns all five tasks, with the real three first.6565664 / latest. Live proof:
/api/v1/tasks, /, /app, /tasks, /benchmarks, and /leaderboard all render the tasks.Phase 2: final demo video
SOYHLrjzK2X1ezoPC6cr (“Harry”) voice. The .env.local library voice returns HTTP 402 on this plan, so the render overrides it to the premade voice. The single-scene test file was deleted after verification.demo/SCRIPT.md and demo/scripts/narration.mjs to the final story: Problem, Solution, Six Metrics + four chaos modes, real tasks/Claude run, harness × recovery, 0.0 → 100.0 replay proof, community tasks/six gates, close.demo/orivoxx-demo.mp4, 134.7s (2:14.7), H.264/AAC 1920×1080/30, −16.2 LUFS integrated, −1.6 dBTP true peak, 0 caption overlaps, 8.9 MiB.demo/README.md and demo/thumbnail.{png,jpg}.Final count after the catalogue regression test
/api/v1/tasks returns real:is-email-predicate, real:itsdangerous-future-signature, real:slugify-max-length; /, /app, /tasks, /benchmarks, and /leaderboard each contain all three.0.0 → 100.0: 134.7s, −16.2 LUFS, −1.6 dBTP, 0 caption overlaps./promo route (apps/web/src/app/promo/page.tsx) with framer-motion, Instrument Serif headings, Geist body, the Coderia palette, a slow green orb, and a browser mockup.demo/recordings/raw-01.webm … raw-05.webm and served trimmed copies from apps/web/public/demo/.pNInz6obpgDQGcFmaJgB, premade). Single-scene test returned HTTP 200. Built one promo-audio.mp3 with a 1.7s intro silence, per-scene gaps, and a 1.2s tail; wrote the shared timeline.json./promo in one Playwright take at 1920×1080. The scene clock uses wall time, not audio.currentTime; headless audio playback drifts, so the wall-clock timeline keeps the visual cuts aligned with the separately muxed narration.-ss 4.55), muxed with the promo narration and a low ambient bed, normalized to −16.3 LUFS / −1.5 dBTP.demo/orivoxx-demo.mp4, 130.1s (2:10.1), H.264/AAC 1920×1080/30, 12.1 MB.assemble.mjs, record-scenes.mjs, generate-audio.mjs, generate-piper-audio.mjs) and old scene recordings.demo/README.md, demo/SCRIPT.md, thumbnails, and SUBMISSION.md.pitch/deck.html, a 7-slide 16:9 deck using the Coderia palette and Instrument Serif/Geist.pitch/render.mjs (Playwright) and rendered pitch/Orivoxx_Deck.pdf: 7 pages, 960 × 540 pt (13.333 × 7.5 in), 2.3 MB.harness × recovery cell is highlighted; the other taxonomy rules are listed without a proof claim.