Methodology
One pinned reference agent runs every task against every provider identically; LLM judges grade the answers and are themselves audited against human labels before anything publishes. Nothing is blended into one number: every index is a pure quality construct, and cost and latency are separate published axes.
Principles
- A task passes only if it addresses the question, is correct, is current where that applies, and is supported by what the tool actually returned. An unsupported or fabricated answer is a scored failure.
- The agent onboards itself from each product's own documentation. We do not write a custom integration or otherwise engineer the setup: any engineering on our side would interfere with the test.
- The scored task splits are private and rotate per release, so providers cannot train on the test.
- Judges never run uncalibrated: labels from at least two humans set the agreement ceiling, and the judge's agreement against them is published with every release.
- No provider can pay for results, methodology changes, placement, or listing. Paid engagements never touch public scores; the full rules live on the governance page.
Memory
The Agentic Memory Index applies the same measurement to agent memory tools. 12 memory tools measured on simulated multi-week company brain operations. This page is the method; the full data lives on the index page.
Task Bank
| Task type | What it tests |
|---|---|
| Direct Recall | Remembering a stated fact |
| Synthesis | Combining facts from multiple conversations |
| Updated Facts | Using the current version of a fact |
| Time Relationships | Reasoning about when things happened |
| False Memory Check | Avoiding claims unsupported by memory |
| Preferences | Remembering expressed preferences |
| Conflicting Facts | Handling incompatible facts |
| Erasure | Respecting requests to forget |
| Long Term | Remembering facts after later conversations |
The bank holds 150 scored probes per tool.
Judging and Calibration
The verdict enum is shared with search, with two relabels: a fabricated memory is an assertion about something never stored, and a provider infrastructure fault voids the affected runs instead of scoring them.
Two independent humans, Labeller A and Labeller B, labeled the same 60-answer calibration set and agreed on 59/60 (Cohen's kappa 0.953). That is the ceiling any judge can reach. Our judge matched Labeller B on 59/60 (kappa 0.953) and Labeller A on 60/60 (kappa 1). One disagreement was a grounding judgment on a date inference.
Cost and Speed
Cost / 1,000 questions uses all questions as its denominator. Tool fees and pre-load are excluded.
Speed is end-to-end answer time with the same pinned agent model on every tool, published alongside one-time setup, indexing lag, and per-operation latency. The full charts, per-tool numbers, and the leaderboard live on the Agentic Memory Index page.
Versioning
- Kind weights and the judge (model and prompt) are pinned per index version; any change ships as a new, recalibrated version.
- Task splits are private and rotate per release; saturated sources are retired with published rationale.
- Changes and corrections are logged in the changelog, never silent.
Lineage
The judge design adapts the Universal Verifier's published calibration principles (Rosset & González Fernández et al., arXiv:2604.06240); hurdle and signed grounding follow Mercor's ACE.