# Methodology One pinned reference agent runs every task against every provider identically; LLM judges grade the answers and are themselves audited against human labels before anything publishes. Nothing is blended into one number: every index is a pure quality construct, and cost and latency are separate published axes. ## Principles - A task passes only if it addresses the question, is correct, is current where that applies, and is supported by what the tool actually returned. An unsupported or fabricated answer is a scored failure. - The agent onboards itself from each product's own documentation. We do not write a custom integration or otherwise engineer the setup: any engineering on our side would interfere with the test. - The scored task splits are private and rotate per release, so providers cannot train on the test. - Judges never run uncalibrated: labels from at least two humans set the agreement ceiling, and the judge's agreement against them is published with every release. - No provider can pay for results, methodology changes, placement, or listing. Paid engagements never touch public scores; the full rules live on the [governance](/governance) page. ## Memory The [Agentic Memory Index](/) applies the same measurement to agent memory tools. 12 memory tools measured on simulated multi-week company brain operations. This page is the method; the full data lives on the index page. One simulated working relationship. The agent is onboarded once, then stores and recalls across 56 sessions, and facts stored early are asked for again dozens of sessions later. ### Task Bank | Task type | What it tests | | --- | --- | | Direct Recall | Remembering a stated fact | | Synthesis | Combining facts from multiple conversations | | Updated Facts | Using the current version of a fact | | Time Relationships | Reasoning about when things happened | | False Memory Check | Avoiding claims unsupported by memory | | Preferences | Remembering expressed preferences | | Conflicting Facts | Handling incompatible facts | | Erasure | Respecting requests to forget | | Long Term | Remembering facts after later conversations | The bank holds 150 scored probes per tool. ### Judging and Calibration The verdict enum is shared with search, with two relabels: a fabricated memory is an assertion about something never stored, and a provider infrastructure fault voids the affected runs instead of scoring them. Two independent humans, Labeller A and Labeller B, labeled the same 60-answer calibration set and agreed on 59/60 (Cohen's kappa 0.953). That is the ceiling any judge can reach. Our judge matched Labeller B on 59/60 (kappa 0.953) and Labeller A on 60/60 (kappa 1). One disagreement was a grounding judgment on a date inference. Judge vs humans 98.3% kappa 0.953 · 59/60 Human-human ceiling 98.3% kappa 0.953 · 59/60 ### Cost and Speed Cost / 1,000 questions uses all questions as its denominator. Tool fees and pre-load are excluded. Speed is end-to-end answer time with the same pinned agent model on every tool, published alongside one-time setup, indexing lag, and per-operation latency. The full charts, per-tool numbers, and the leaderboard live on the [Agentic Memory Index](/) page. ## Versioning - Kind weights and the judge (model and prompt) are pinned per index version; any change ships as a new, recalibrated version. - Task splits are private and rotate per release; saturated sources are retired with published rationale. - Changes and corrections are logged in the [changelog](/changelog), never silent. ## Lineage The judge design adapts the Universal Verifier's published calibration principles (Rosset & González Fernández et al., arXiv:2604.06240); hurdle and signed grounding follow Mercor's ACE. Stay up to date with agent memory Get new memory test results and updates by email. Keep me posted I agree to receive emails from Verging Labs about new benchmarks and product updates. I can unsubscribe at any time. Verging Labs Compare independent test results to see where each memory system works well and where it fails. We’re building the lab that measures every part of an AI agent, starting with memory. No provider pays for placement, ordering, or scores. [Free Memory Check](/memory-check) Questions, corrections, or a tool we should benchmark: [contact@verginglabs.com](mailto:contact@verginglabs.com)