Status: target design for the HackCulture AI Global Hackathon build. This document separates what exists in the repository today from what is planned.
Legend: [SHIPPED] = implemented and covered by the current source/tests. [PARTIAL] = contracts or scaffolding exist but the runtime path does not. [PLANNED] = design only; no implementation in this repository yet.
apps/web Next.js 16 shell
├── / dark landing placeholder
├── /api/v1/health safe provider/leaderboard status
├── /api/v1/leaderboard ranked result rows (seed fallback, no-store)
└── /api/v1/results/:id full stored result by id (seed fallback)
packages/benchmarks versioned benchmark tasks and the deterministic score
├── tasks registry, TaskBoard, Linkstash, fixtures, generated assets
├── scoring scoreRun, static structural metrics, evaluateRepository
├── contracts task/ruleset/scoring-config/score Zod schemas
└── fingerprint canonical ruleset fingerprint
packages/harness the only component that executes an agent
├── adapters adapter contract, registry, fake + python + aider
├── sandbox Docker executor, allowlisted model proxy
├── capture bounded artifact capture + repository fingerprint
├── diff bounded unified produced-code diff
├── benchmark run → capture → score → immutable result
└── run run lifecycle, command/env validation, summaries
packages/attribution deterministic failure attribution rules
packages/replay counterfactual replay + proof
packages/results append-only result storage
├── contracts BenchmarkResult/LeaderboardEntry, consistency invariants
├── local one immutable JSON file per result
├── upstash atomic Lua append + sorted indexes
├── seed bundled read-only fallback rows
└── read fallback reads, degraded flag, store factory
packages/core pure TypeScript verification engine
├── archive bounded in-memory ZIP intake, path checks, redaction
├── spec-parser PROJECT_SPEC.md requirement parser
├── inventory eligible file metadata
├── checks deterministic rule engine (7 rule types)
├── evidence server-generated evidence registry
├── verdict precedence rules and gate
├── report schema-validated report builder
├── graph/contracts graph schema only (no builder)
├── contracts Zod contracts for reports, assessments, explanations
├── verify-input verify-request validation and consent refinement
└── fixtures TaskBoard regressed/compliant snapshots
What this means in practice:
packages/harness runs an adapter in Docker, captures the repository, scores it, and builds an immutable result.ResultStore interface, a bundled seed fallback, and read APIs.┌──────────────────────────────────────────────────────────────────────┐
│ Public benchmark UI │
│ task catalogue · submit agent · run status · result explorer │
│ leaderboard · score breakdown · evidence viewer │
└──────────────────────────────┬───────────────────────────────────────┘
│ same-origin HTTPS
┌──────────────────────────────▼───────────────────────────────────────┐
│ apps/web (Next.js, Vercel) │
│ benchmark API · score API · leaderboard read API · explanation API │
│ auth-less public read · rate limits · safe error envelope │
└───────────────┬──────────────────────────────┬───────────────────────┘
│ │
┌───────────────▼──────────────┐ ┌────────────▼──────────────────────┐
│ packages/benchmarks [PLANNED]│ │ packages/harness [PLANNED] │
│ task registry + versions │ │ agent adapters │
│ ruleset versions │ │ sandbox orchestration │
│ scoring configuration │ │ artifact capture │
│ leaderboard aggregation │ │ run metadata │
└───────────────┬──────────────┘ └────────────┬──────────────────────┘
│ │
│ ┌──────────▼──────────────────────┐
│ │ apps/runner [PLANNED, optional] │
│ │ isolated execution service │
│ │ sandbox image + quotas │
│ └──────────┬──────────────────────┘
│ │ produced repository
┌───────────────▼──────────────────────────────▼───────────────────────┐
│ packages/core [SHIPPED — scoring rubric] │
│ spec parser · deterministic rules · evidence · verdict · report │
│ static analysis only; never executes the produced repository │
└──────────────────────────────┬───────────────────────────────────────┘
│ optional explanation context
┌──────────────────────────────▼───────────────────────────────────────┐
│ Nebius Token Factory / Nemotron [PARTIAL] │
│ explanations only; never scoring, never verdicts │
└──────────────────────────────────────────────────────────────────────┘
| Component | Responsibility | Status |
|---|---|---|
packages/benchmarks |
Versioned task registry, ruleset manifests, scoring configuration. | [SHIPPED] |
packages/harness |
Agent adapter interface, run lifecycle, Docker sandbox, allowlisted model proxy, artifact capture, adapter versioning. | [SHIPPED] |
packages/attribution |
12 deterministic layer × mode rules over trace + scorecard; evidence required. | [SHIPPED] |
packages/replay |
Memoized counterfactual replay over recorded actions; deterministic proof. | [SHIPPED] |
packages/results |
Append-only result storage, reliability aggregation, leaderboard ordering, Upstash/local stores, seed fallback. | [SHIPPED] |
apps/runner |
Optional isolated execution service. | [SUPERSEDED] — real runs execute through packages/harness on the organizer machine |
apps/web |
Public benchmark UI, task catalogue, fake run submission, result explorer, leaderboard, comparison, in-app docs. | [SHIPPED] — optional explanations are not shipped |
packages/core |
Deterministic scoring rubric over the captured repository. | [SHIPPED] — engine only |
Flow for one organizer-run benchmark run:
packages/harness creates an empty run workspace, writes the task’s starting repository, and starts a hardened container with no platform secrets (docs/SANDBOX.md).packages/benchmarks runs the deterministic ruleset and the static import graph over the captured text. No produced code is executed.scoreRun() maps rule outcomes to the task score. The result records task/ruleset/adapter/scoring-config versions and the repository fingerprint.The scoring engine is synchronous and stateless [SHIPPED]. Real benchmark runs are not: an agent run takes minutes and needs isolation, so it runs as a separate organizer-executed command that writes a result to the store [SHIPPED] in packages/harness. A public run queue is [PLANNED]; nothing in the public web app executes an agent today.
The scoring function is:
score = f(taskVersion, rulesetVersion, capturedRepository, scoringConfigVersion)
packages/benchmarks implements it: scoreRun() maps deterministic requirement outcomes and static structural metrics to a 0–100 score using the task’s versioned scoring configuration. evaluateRepository() runs the checks and the static import graph first. The model is not an input.
Properties:
A deterministic FAIL is final. Missing or invalid evidence is UNVERIFIED, never PASS. Unknown requirement IDs, paths, or evidence references from any external source are rejected before they can affect a result.
There are two distinct execution contexts, and the documentation must keep them separate:
| Context | What runs there | What it may access | What it must never access |
|---|---|---|---|
Harness sandbox [SHIPPED] |
The agent process and its declared tools | The task’s starting repository, an ephemeral run workspace, one allowlisted model proxy | Platform API keys, the leaderboard store, other runs’ artifacts, the host filesystem, unrestricted network |
Scoring engine [SHIPPED] |
Pure TypeScript analysis | The captured, normalized repository in memory | The network, a shell, the agent process, persistent storage |
This separation is a strength, not a limitation. The scoring engine cannot be compromised by malicious code in the agent’s output because it never executes that code. It parses source with the TypeScript parser API and evaluates declarative checks. The harness is the only component that executes anything, and it does so in a disposable sandbox with explicit quotas. See docs/SANDBOX.md for the container profile, the network topology, and the proxy allowlist.
The existing engine rules still apply to the scoring engine:
Approved: adding a harness that executes agents was approved in Phase 0 and is recorded in
AGENTS.md. That rule continues to apply to the scoring engine, which never executes produced code. The sandbox boundary above is the shipped replacement for the harness.
| Verifier-era component | New role in the benchmark platform |
|---|---|
Spec parser (packages/core/src/spec-parser) |
Parses each benchmark task’s PROJECT_SPEC.md into versioned requirements and checks. |
Deterministic rule engine (packages/core/src/checks) |
The scoring rubric. Each check is a named, reproducible rule. |
Verdict engine (packages/core/src/verdict) |
Precedence layer that turns rule outcomes into per-requirement statuses and a gate. |
Evidence registry (packages/core/src/evidence) |
Makes every deduction traceable to a rule, a requirement, a path, and a line where available. |
Archive intake (packages/core/src/archive) |
Normalizes and secures the repository artifact captured from the harness. |
Report contract (packages/core/src/report) |
Base for the planned benchmark result artifact; the benchmark result adds task/ruleset/adapter versions and a numeric score. |
Graph contracts (packages/core/src/graph/contracts) |
Structural metric schema. The builder and coupling/cycle metrics are planned. |
Nebius adapter (apps/web/src/server/nebius) |
Optional explanation service only. Never scoring. |
Map-first dashboard design (FRONTEND_DESIGN_SPEC.md) |
Visual reference for the planned result explorer. The current UI is a landing shell. |
Design-level shape for a benchmark result. This is not implemented; field names are a proposal for the code pass.
{
"resultId": "string",
"taskId": "string",
"taskVersion": "string",
"rulesetVersion": "string",
"scoringConfigVersion": "string",
"adapterId": "string",
"adapterVersion": "string",
"agentLabel": "string",
"modelId": "string",
"harnessMetadata": {},
"sandboxImage": "string",
"startedAt": "ISO-8601",
"completedAt": "ISO-8601",
"repositoryFingerprint": "sha256",
"score": 0,
"scoreBreakdown": [],
"ruleOutcomes": [],
"findings": [],
"artifactRef": "string"
}
The taskVersion, rulesetVersion, adapterVersion, scoringConfigVersion, and repositoryFingerprint fields are mandatory for reproducibility. A result missing any of them is not a valid leaderboard entry.
[SHIPPED] Node.js 24 LTS, pnpm workspace, TypeScript strict mode.[SHIPPED] Next.js 16 App Router on Vercel for the public UI and read APIs (currently a shell plus health endpoint).[PARTIAL] @xyflow/react + @dagrejs/dagre are installed, but the graph builder and result-explorer UI are planned.[SHIPPED] Zod for external contracts.[SHIPPED] fflate for bounded ZIP parsing.[PARTIAL] openai Node SDK against https://api.tokenfactory.nebius.com/v1/, model nvidia/Nemotron-3.5-Super-VL, for explanations only. The current adapter is a smoke test.[PLANNED] Upstash Redis REST + @upstash/ratelimit for shared rate limits. The package is not installed and no limiter is wired.[SHIPPED] Vitest for unit/integration tests. [PARTIAL] Playwright is installed as a dev dependency but has no test suite yet.These need a human decision before the code pass:
apps/runner is a separate deployable or a local-only harness for the hackathon demo.