Search Methodology
One pinned reference agent runs every task against every provider identically; LLM judges grade the answers and are themselves audited against human labels before anything publishes. Nothing is blended into one number: every index is a pure quality construct, and cost and latency are separate published axes.
Principles
- A task passes only if it addresses the question, is correct, is current where that applies, and is supported by what the tool actually returned. An unsupported or fabricated answer is a scored failure.
- The scored task splits are private and rotate per release, so providers cannot train on the test.
- Judges never run uncalibrated: labels from at least two humans set the agreement ceiling, and the judge's agreement against them is published with every release.
- No provider can pay for results, methodology changes, placement, or listing. Paid engagements never touch public scores; the full rules live on the governance page.
Search
The Agentic Search Index measures the web-search and retrieval tools AI agents use. The pinned reference agent runs a bounded loop, max 8 search calls per task.
Judge Calibration
Two independent humans grading the same answers agreed 95.8% of the time (Cohen's kappa 0.917, n=48). That is the ceiling any judge can reach. Our judge matched the human consensus 94.1% of the time (kappa 0.85, n=34). A second judge model agreed with the pinned judge's pass-fail call 95.1% of the time across 122 overlapping cases. 3,267 runs were executed and 3,263 verdicts were scored after voiding harness-fault runs.
| Comparison | n | Agreement | Kappa |
|---|---|---|---|
| Human vs human (ceiling) | 48 | 95.8% | 0.917 |
| Judge vs human consensus | 34 | 94.1% | 0.85 |
| Judge vs second judge model | 122 | 95.1% pass-fail | n/a |
Task Bank
| Kind | Tasks | What it tests |
|---|---|---|
| Static Facts | 45 | Short factual questions with one indisputable answer. |
| Dynamic Facts | 28 | Questions whose answers change with the world. |
| Multi-Source | 40 | Questions requiring integration of multiple sources. |
| Deep Research | 8 | Hard multi-constraint discovery questions. |
The freshness generator behind the Dynamic Facts kind is private. Fabricated support scores minus 1: fabricated confidence is worse than an honest miss. Provider errors and timeouts count against the provider. Task-bank seeds: SimpleQA-Verified (MIT), FRAMES and WebWalkerQA (Apache-2.0).
Verdict Buckets
| Verdict | Meaning |
|---|---|
| Correct Answer | Answered the task correctly. |
| Question Not Addressed | Never addressed the question asked. |
| Incorrect Answer | Answered confidently, but incorrectly. |
| Outdated Answer | Answered with out-of-date information. |
| Fabricated Citation | Cited a source that does not support the claim. |
| Provider API Error | The provider call itself failed. |
Baselines
A no-search baseline (the identical pinned agent with search disabled) scored 16.7 on the same bank.
Cost Method
Cost per 1,000 successful answers = billed provider fees across the whole workload (all calls, including retries and failed tasks) plus payload ingestion priced at the pinned shopper's input rate (Claude Opus 4.8, $5.00 per million tokens), divided by successful tasks.
Worked example (Firecrawl): $0.01 list price/call x 2.4 avg searches per task / 0.84 pass rate = 3.76 cents per successful answer ($37.57 per 1,000).
Versioning
- Kind weights and the judge (model and prompt) are pinned per index version; any change ships as a new, recalibrated version.
- Task splits are private and rotate per release; saturated sources are retired with published rationale.
- Changes and corrections are logged in the changelog, never silent.
Lineage
The judge design adapts the Universal Verifier's published calibration principles (Rosset & González Fernández et al., arXiv:2604.06240); hurdle and signed grounding follow Mercor's ACE.