Methodology
One pinned reference agent runs every task against every provider identically; LLM judges grade the answers and are themselves audited against human labels before anything publishes. Nothing is blended into one number: every index is a pure quality construct, and cost and latency are separate published axes.
Principles
- A task passes only if it addresses the question, is correct, is current where that applies, and is supported by what the tool actually returned. An unsupported or fabricated answer is a scored failure.
- The scored task splits are private and rotate per release, so providers cannot train on the test.
- Judges never run uncalibrated: labels from at least two humans set the agreement ceiling, and the judge's agreement against them is published with every release.
- No provider can pay for results, methodology changes, placement, or listing. Paid engagements never touch public scores; the full rules live on the governance page.
Search
The Agentic Search Index measures the web-search and retrieval tools AI agents use. The pinned reference agent runs a bounded loop, max 8 search calls per task.
Judge Calibration
Two independent humans grading the same answers agreed 95.8% of the time (Cohen's kappa 0.917, n=48). That is the ceiling any judge can reach. Our judge matched the human consensus 94.1% of the time (kappa 0.85, n=34). A second judge model agreed with the pinned judge's pass-fail call 95.1% of the time across 122 overlapping cases. 3,267 runs were executed and 3,263 verdicts were scored after voiding harness-fault runs.
| Comparison | n | Agreement | Kappa |
|---|---|---|---|
| Human vs human (ceiling) | 48 | 95.8% | 0.917 |
| Judge vs human consensus | 34 | 94.1% | 0.85 |
| Judge vs second judge model | 122 | 95.1% pass-fail | n/a |
Task Bank
| Kind | Tasks | What it tests |
|---|---|---|
| Static Facts | 45 | Short factual questions with one indisputable answer. |
| Dynamic Facts | 28 | Questions whose answers change with the world. |
| Multi-Source | 40 | Questions requiring integration of multiple sources. |
| Deep Research | 8 | Hard multi-constraint discovery questions. |
The freshness generator behind the Dynamic Facts kind is private. Fabricated support scores minus 1: fabricated confidence is worse than an honest miss. Provider errors and timeouts count against the provider. Task-bank seeds: SimpleQA-Verified (MIT), FRAMES and WebWalkerQA (Apache-2.0).
Verdict Buckets
| Verdict | Meaning |
|---|---|
| Correct Answer | Answered the task correctly. |
| Question Not Addressed | Never addressed the question asked. |
| Incorrect Answer | Answered confidently, but incorrectly. |
| Outdated Answer | Answered with out-of-date information. |
| Fabricated Citation | Cited a source that does not support the claim. |
| Provider API Error | The provider call itself failed. |
Baselines
A no-search baseline (the identical pinned agent with search disabled) scored 16.7 on the same bank.
Cost Method
Cost per 1,000 successful answers = billed provider fees across the whole workload (all calls, including retries and failed tasks) plus payload ingestion priced at the pinned shopper's input rate (Claude Opus 4.8, $5.00 per million tokens), divided by successful tasks.
Worked example (Firecrawl): $0.01 list price/call x 2.4 avg searches per task / 0.84 pass rate = $0.0376 per successful answer ($37.57 per 1,000).
Memory
The Agentic Memory Index applies the same measurement to agent memory tools. Eight memory tools measured on simulated multi-week company brain operations. This section summarizes the method; the full data lives on the index page.
Task Bank
| Task type | What it tests |
|---|---|
| Direct Recall | Facts stored once and asked for directly. |
| Updated Facts | Facts that changed after storage; the newest value must win. |
| Thread Growth | Threads that grew across sessions; the accumulated state must come back. |
| Synthesis | Answers assembled from several stored memories. |
| Long Term | Facts stored weeks of simulated time before the probe. |
| False memory check | Probes about things never stored; the honest answer is that nothing is known. |
The bank holds 272 scored probes per tool: 200 positive probes and 72 never-stored honesty probes. The store every tool answers from is built through 42 simulated multi-week operator relationships plus two long arcs, and early facts are re-probed at five checkpoints across 56 working sessions.
Judging and Calibration
The verdict enum is shared with search, with two relabels: a fabricated memory is an assertion about something never stored, and a provider infrastructure fault voids the affected runs instead of scoring them.
Two independent humans labeled the same 66-answer calibration set and agreed 91% of the time before adjudication (Cohen's kappa 0.63); every disagreement traced to one rubric clause. The judge pipeline matched the primary labeler 98.5% of the time (kappa 0.95) across the full set.
Claude Code built-in memory runs the same bank as an unranked reference and scored 67.7.
Cost and Speed
Cost per 1,000 successful answers counts agent token spend to store and answer (CLI-metered at API-equivalent rates), the entry paid tier amortized at its capacity through measured usage, and embedding spend where billed separately.
Speed is end-to-end answer time with the same pinned agent model on every tool, published alongside one-time setup, indexing lag, and per-operation latency. The full charts, per-tool numbers, and the leaderboard live on the Agentic Memory Index page.
Versioning
- Kind weights and the judge (model and prompt) are pinned per index version; any change ships as a new, recalibrated version.
- Task splits are private and rotate per release; saturated sources are retired with published rationale.
- Changes and corrections are logged in the changelog, never silent.
Telemetry Semantics
- Liveness is not "returns 200": every watched surface carries an expected status set, and a 200 with the wrong content records as a soft-404, not presence.
- Blocked is not down: bot walls are recorded as blocked and never counted as downtime.
- Percentiles come from rollups only; a single-sample latency is never displayed. Speed and liveness are measured with unpaid probes from a single vantage point (our probe region). Treat them as our view, not global ground truth.
- The watcher never pays and never authenticates: unpaid, unauthenticated requests only.
Lineage
The judge design adapts the Universal Verifier's published calibration principles (Rosset & González Fernández et al., arXiv:2604.06240); hurdle and signed grounding follow Mercor's ACE.