Read a Verging Memory CI report

A Verging Memory CI regression report answers three questions:

• Is this release ready?
• What changed since the previous release?
• What evidence explains each failure?

For installation and submission, see Integrate Verging Memory CI with GitHub Actions.

Start with the summary

The report opens with the information needed to understand its scope and result:

• Tested names the release and, when available, the previous release used for comparison.
• Release verdict is Ready or Not ready, followed by the reason.
• Agent setups names the harness and model configurations tested.
• Test suites names the suites covered by this release.
• Environments tested counts each test suite exercised in each agent setup once. It shows the breadth of testing performed.
• Stage says whether this is a preliminary or final report.

For example, two agent setups tested across three suites is six Environments tested.

The version shown in the report is the vendor_version label submitted with the release. Verging Labs tests the product served at your Test endpoint when testing starts, so deploy that version before submitting the release.

Test suites

A test suite covers a situation your memory product must handle:

• Onboarding. Can a fresh agent set up your product from its published documentation, store a memory, and recall it in a new session? The report shows whether setup succeeded and how long it took.
• Core Recall. Does the agent recall stored facts, keep corrections current, combine related facts, and avoid inventing answers?
• Preference Adherence. Does the agent follow standing user instructions when it acts?
• Truth Maintenance. Does the agent return the current record when sources conflict, facts change, or the user asks it to forget something?
• History Migration. Can the agent use imported history for recall, synthesis, updates, date reasoning, and honest abstention?
• Memory at Scale. Does recall remain reliable as the memory store grows?
• Long-Horizon Retention. Does memory remain reliable across extended use, growing history, and changes in how information is organized?

When suites is omitted, a release covers all suites chosen for the account during onboarding that every named agent setup supports and has set up. On a release naming several agent setups, anything left out is named on the receipt with the reason and the release that covers it. Onboarding is supported on Claude Code and Hermes agent setups. Long-Horizon Retention is available on a separately ordered tier.

Release verdicts and comparisons

Ready means the selected suites found no release-blocking failure. Not ready names the failure and, on a report covering several agent setups, the setup where it occurred.

A regression is a test that passed on the previous release and fails on this one. A newly added test that fails is marked as new rather than as a regression, but it can still make the release Not ready. Tests without a grade do not count as passes or failures.

Comparison is tracked separately for each test suite in each agent setup. The first time Verging Labs tests a suite on a setup is that suite's activation, so there is no earlier result for that combination. Its report shows the measured Accuracy, Speed, and Cost, and its verdict is based on Accuracy.

After activation, each suite compares with its own most recent delivered result on the same setup. A suite omitted from the current release does not appear and is never treated as removed or regressed. A suite added to the current release is shown as newly covered when it has no earlier result. Results never compare across suites or agent setups.

See Agent setups for activation and what to do when a setup changes.

Results at a glance

The summary shows results by test suite and, when several agent setups were tested, by setup:

• Accuracy shows how many tests passed, out of the tests counted as a pass or a failure.
• Pending shows every test counted as neither: tests still awaiting a final grade, and tests the previous release never finally graded. Accuracy and Pending always add up to the same total, in the summary and in each agent setup's section.
• Speed shows elapsed testing time.
• Cost shows testing cost on the reference rate card described below.

If a model provider is unavailable before an Environment completes, the preliminary report identifies the affected Environment. Verging Labs reruns it and includes the completed result in the Final Report.

The detailed Accuracy section groups results by test type and shows what regressed, what improved, and what was already failing. Speed and Cost are reported for the smallest unit that can be measured directly; suites whose tests share agent sessions are measured as a whole.

Failed tests and evidence

Test outcomes use these labels:

• Correct Answer means the answer matched the stored fact.
• Setup Did Not Work means a fresh agent could not get the product working from its published documentation.
• No Answer means the fact was stored but the agent said it did not know.
• Outdated Answer means the agent returned an earlier value instead of the current one.
• Wrong Answer means the answer was incorrect but based on information the memory returned.
• Made Up An Answer means the answer was not supported by information the memory returned.
• Provider Service Error means the memory service errored while answering after its allowed retries.

Every failed test includes an Evidence link. The evidence shows the question, expected answer, answer given, relevant calls to the memory product, and the stored facts needed to answer. Values shaped like credentials are redacted.

Occasionally the report cannot safely attribute a call excerpt to one test. In that case the question, expected answer, and answer given remain available, and the missing excerpt does not affect the verdict.

Several agent setups in one report

A release covering several agent setups still produces one report. Results at a glance summarizes each setup, followed by a detailed section for each one. The Release verdict reflects the least-ready setup, and each failure names the setup where it occurred.

Special report states

Storage-only

A storage-only release covers write acknowledgements, time to readiness, and whether write behavior changed from the previous release. It asks no test questions and therefore has no Accuracy verdict or grading corrections. It is delivered once as a final report.

Storage-only testing is unavailable during a suite's activation. It covers one agent setup at a time, and every selected suite must already be set up there.

Testing stopped before retrieval

If the product rejects every write in the first suite attempted, no question can be asked. The report is final on delivery and its verdict is Not ready because testing could not proceed beyond storage. Suites that never started are shown as not tested.

Preliminary and final reports

When some tests still need grading, the preliminary report arrives after testing and shows those tests as Pending. The final report replaces it after grading and explains what changed.

If nothing needs further grading, the first report may already be final. A test that remains ungraded in the final report is identified and counts as neither a pass nor a failure.

Speed and Cost

Speed and Cost can vary even when the product does not change. The report calls a movement a regression or improvement only when it falls outside expected variation. Movement within that range is reported without changing the release verdict.

Cost uses recorded token counts and a consistent reference rate card. Both releases in a comparison are priced on the same current card, so a provider price change does not appear as a product regression. These figures are a comparison basis, not necessarily the amount the customer pays its model provider.

Machine-readable results

diff.json contains the report data for CI gates and further analysis:

• release_verdict is the report's overall ready or not_ready decision.
• release_verdict_reasons names the dimensions responsible.
• verdict reports Accuracy comparison status.
• cost_verdict reports Speed and Cost comparison status.
• pending_tests and not_compared_tests identify the tests the Pending column counts; ungraded_tests identifies tests still without a final grade in the final report.

On a first or otherwise non-comparable release, cost_verdict is not_compared and the overall verdict is based on Accuracy.

diff.json also includes 95% grading-uncertainty intervals. They describe uncertainty in grading, not normal variation in the product between releases.

See API setup and Agent setups.