Benchmark Results

This page tracks current measurements from the benchmark suite. Numbers are reproducible — the harness is deterministic up to model-provider non-determinism, and the same scenario JSON + distractor seed always produces the same memory pool. Regenerate any table below with node harness/summarize.js results/<file>.json and the statistics with node harness/analyze.js results/<file>.json.

Run metadata

Why single-shot? The model receives the task plus whatever the arm injects (nothing, a retriever's top-k, brain's session-start payload, or the oracle) and replies in one turn. This isolates memory's effect — there is no agentic file exploration to rediscover conventions — and keeps token counts clean and comparable (no cache-read inflation). It mirrors the Mem0 / LongMemEval framing.

Scenario A — Noisy Project Folder (retrieval under 1000 distractors)

Headline. Under a 1000-distractor haystack, brain is the only retrieval method whose memories let the model succeed — both BM25 (Recall@5 0.33) and a local vector store (Recall@5 0.00) fail to surface the three oracle memories, and the model fails with them. Brain retrieves 2/3 (Recall@5 0.67), which is enough to pass, and it does so at the lowest tokens-per-success of any passing arm (3,199 — below the oracle's 3,572 and far below the 8k/60k dumps). Disabling the pinned tier drops brain to 67% and doubles its Recall miss — the ablation that isolates pinning's value.

Scenario B — Three Sessions, One Decision (continuity)

Headline. Postgres was decided across three sessions despite a discarded Mongo prototype. Brain recalls it perfectly (Recall@5 1.0) and passes at the fewest tokens-per-success of the memory arms. Here the haystack is small (100 distractors), so plain BM25 also retrieves the decision and passes — on this scenario brain's edge is efficiency, not correctness.

Scenario C — The Contradiction Test (tabs → spaces → tabs)

Headline — an honest null. Everything clusters near 67%. Indentation (tabs vs spaces) is a noisy signal to grade from a single-shot text reply, and the arm that does best is dump-all-chrono, which simply concatenates all three versions in time order so the latest (tabs) wins. Brain retrieves the final decision (Recall@5 1.0) but doesn't convert that into a clear pass-rate win here. We report it rather than hide it.

Scenario D — Skill Progressive Disclosure (token efficiency)

Headline. Loading just the one relevant skill passes 100% at 1,580 tokens-per-success — ~31% leaner than dumping every skill body (2,289). The index-only (L0) and no-skills arms fail: a single-shot model can't act on a skill index the way an agent would (read the index, then load the matching SKILL.md). That on-demand load is exactly what the skills tier automates — and when it fires, it's both correct and the cheapest passing arm.

Scenario F — Abstention (no confabulation)

Headline — a second honest null. Asked to set up a deployment pipeline with no deployment target in memory, DeepSeek abstains correctly with or without brain (100%) — the base model already declines to invent a target. A noisy keyword retriever, which injects irrelevant top-k, actually drags it down to 67%. Memory neither helps nor hurts abstention in this case.

Reading the suite

Brain wins where it is designed to — retrieval under heavy noise (A) and procedural-skill efficiency (D) — passing where the BM25 and vector baselines fail, at the lowest token cost among passing arms. It is efficient and competitive on continuity (B), and neutral on contradiction (C) and abstention (F), where the base model needs no memory. That mixed, baseline-anchored picture — floor, oracle ceiling, real retriever baselines, and brain ablations on every scenario — is the point: the wins are attributable to the memory system, and the nulls are reported honestly.

Raw JSON →

Retrieval scoring calibration — in-vivo probe (July 2026)

The problem. BM25 relevance was normalized per-query by the top score, so the best available match for any query reported relevance 1.0 — including queries the brain knew nothing about. A four-term nonsense control query returned ten results against a real 177-memory brain, the top one scoring a confident-looking composite of 0.679. An agent trusting those scores injects noise into its context.

The fix (post-beta.30). Three coordinated changes: relevance is scaled by IDF-weighted query coverage (matching one term of a four-term query caps relevance near 0.25, however strong that single match); explicit-query recall applies a relevance floor so zero-relevance memories can't pad the top-N on strength alone; and spreading-activation sources are relevance-gated in query mode — activation spreads outward from the query-relevant set, so a strong-but-irrelevant clique can't rescue itself past the floor, while associates of genuinely relevant memories still surface.

The probe. A real working brain (177 memories, 1,245 association edges), 20 tag-derived topical queries with known target memories, plus the nonsense control. Same brain, same queries, before and after:

Coverage scaling did not just fix the calibration — it improved ranking: the two probe queries whose targets previously ranked #2 moved to #1, because partial-match distractors no longer outrank full-coverage targets.

Caveats. n=20, a single brain, self-run, and tag-derived queries favor the index's weighted tag field. The honest headline is the calibration change — irrelevant queries now return honestly low scores or nothing — with the ranking improvement as directional evidence pending a controlled re-run (below).

What's next

• Re-run the controlled suite (Scenario A especially) on the calibrated scoring — coverage scaling and the relevance floor should sharpen brain-full's Recall@5 under the 1000-distractor haystack.
• More runs (5–10/arm) for statistical-confidence error bars on the token metric.
• Scenario E (continual coding, write-side memorize) validated end-to-end.
• A second agent under test (e.g. a stronger reasoning model) to separate memory effects from base-model competence.