RAG Regression GateSearch checks for staff assistants

Keep the right policy within reach.

Before you update an internal help assistant, check which sources it brings back.

Open the exampleSet up your own checks

Saved test results for 6 example staff questions

How long should raw customer event records be retained?

Question 3 of 6
Hold this update

The current policy is still found, but the outdated proposal now ranks above it.

Before update

Top sources returned for this question.

  1. Approved retention policy

    Current

    Delete raw customer event records after 30 days.

  2. 2022 proposal

    Unapproved proposal

    365 days · never approved.

After update

Top sources returned for this question.

  1. 2022 proposal

    Unapproved proposal

    365 days · never approved.

  2. Approved retention policy

    Current

    Delete raw customer event records after 30 days.

See the full evidence

All six checks pass, but the update is still held.

6 of 6 question checks pass. The overall order of documents still got worse, so a simple pass count would miss this.

Download the data (JSON)Read the full report

Checks which documents are found, not whether answers are correct. Six questions is a quick check, not a measure of accuracy.

Showing the first two sources. Open the evidence below for every retrieved source and its exact text.

Follow the evidence.

These are saved search-test results. You can explore them here; an engineer runs new checks on your documents. This is not a live assistant.

Read every retrieved source

Before update · all 3 retrieved sources

  1. Approved retention policy

    The approved retention standard deletes raw customer event records after 30 days.

    retention-current

  2. 2022 proposal

    A superseded 2022 proposal suggested retaining raw customer event records for 365 days; it was never approved.

    retention-superseded

  3. Annual leave requests

    Annual leave requests should normally be submitted 16 days before travel.

    leave-hard-negative

After update · all 3 retrieved sources

  1. 2022 proposal

    A superseded 2022 proposal suggested retaining raw customer event records for 365 days; it was never approved.

    retention-superseded

  2. Approved retention policy

    The approved retention standard deletes raw customer event records after 30 days.

    retention-current

  3. Annual leave requests

    Annual leave requests should normally be submitted 16 days before travel.

    leave-hard-negative
What does this test tell us?

RAG means retrieval-augmented generation: an assistant finds source documents before writing an answer. This gate checks that search step, not the answer.

It checks whether the expected sources are found and where they rank. Finding the right policy somewhere is different from putting it first.

This example compares 11 illustrative documents and 6 questions. It uses a 0.02 absolute drop threshold with the statistical significance check off. That makes it a small smoke test, not an accuracy benchmark.

A pass means no regression was detected under these settings. It does not prove an improvement, a safe release or correct generated answers. The existing refund result already puts an ended pilot first.

Inspect the technical result and reproduce it

Candidate: candidate-chunk-12. Top 3 retrieved sources per question.

Rank of the first relevant source (MRR): 0.9167 before, 0.8333 after. fail.

Quality of the source order (nDCG): 0.9385 before, 0.8770 after. fail.

The full report includes every metric, case check, configuration, source fingerprint and recorded resource cost. Cost is reported, not gated.

Open the full offline report

From an installed checkout, reproduce the selected result:

rag-gate gate --dataset fixtures/policies.json --k 3 --baseline configs/baseline.json --candidate configs/candidate-chunk-12.json --no-bootstrap --output reports/comparison.json --html reports/report.html

Reports are unsigned. Content fingerprints identify recorded inputs; they do not prove authorship. Use evidence from a trusted run.

A check before the change.

  1. Choose questions that matter.

    Use questions your staff ask. Label which documents support each answer, including conflicting or outdated guidance.

  2. Compare the two versions.

    The engine runs the same questions against your current and proposed search settings. It records the evidence each version finds.

  3. Review what moved.

    Inspect source order, missing context and measured drops. Keep the report with the proposed change.

Bring your own questions.

A team member can review the evidence. An engineer connects the command-line gate to your documents and release checks.

Your data stays where you run the tool. This website has no upload form, accounts or model service.

Set up the gateDownload the labelled example
Engineer setup and test settings

Clone the repository and run these commands from its root:

python -m pip install -e .
rag-gate gate --dataset fixtures/policies.json --k 3 \
  --baseline configs/baseline.json \
  --candidate configs/candidate-chunk-12.json \
  --no-bootstrap --output reports/comparison.json \
  --html reports/report.html

The deliberate regression returns exit 1. Replace the fixture with your labelled questions and documents, and choose the configs before testing. Do not tune on the test questions.

For a representative evaluation, remove --no-bootstrap. The default gate requires both a drop over 0.02 and a paired 95% bootstrap interval excluding zero. Invalid or incomparable input gives no verdict and exit 2.

Core retrieval uses Python’s standard library. Optional dense models require separate installation and explicit permission to download.