Comparable run reports · schema 2 · metric schema 1 · tool 0.1.0
Candidate regresses
mrr_at_k, ndcg_at_k
baseline-bm25 → candidate-chunk-12 · policies · 11 documents · 6 questions · k=3
Threshold only · bootstrap off. Maximum absolute drop: 0.02. Small evaluation: six or fewer questions. Treat this as a smoke test, not an accuracy benchmark.
Matching recorded source identities
Corpus: c54ae8f79ab12b2b8fa24ecdb4076bdeba0d275fb04414237a6920689a10e121
Queries: 74d2fa56bae560ed52e233e78a3641cd64dcb94ea3ef7cfb3b35fe753a0e6c43
Labels: 01b8f709643da815dab341a3ea84ee719026bd0e29d87e00b5383a405d5f8239
Algorithm: sha256-canonical-json-v1. Fingerprints bind recorded contents, not authorship: these unsigned reports must come from a trusted run.
Baseline config: 9fef4df77f9a8240ce0cc3c043bbea715a970b45bfa91031c5aefb9a87309d93
Candidate config: 0d52411cd0ab9b22863d6df32a0d3501bb2fdc4db22d09f567241315f7bacb51
Retrieval metrics
| Metric | Baseline | Candidate | Delta | Confidence interval | Result |
|---|---|---|---|---|---|
| hit_at_k | 1.0000 | 1.0000 | +0.0000 | Not estimated | ok |
| mrr_at_k | 0.9167 | 0.8333 | -0.0834 | Not estimated | FAIL |
| ndcg_at_k | 0.9385 | 0.8770 | -0.0615 | Not estimated | FAIL |
All other metrics (5)
| Metric | Baseline | Candidate | Delta | Confidence interval | Result |
|---|---|---|---|---|---|
| recall_at_k | 1.0000 | 1.0000 | +0.0000 | Not estimated | ok |
| context_coverage_at_k | 1.0000 | 1.0000 | +0.0000 | Not estimated | ok |
| citation_precision | 1.0000 | 1.0000 | +0.0000 | Not estimated | ok |
| answers_fully_supported_rate | 1.0000 | 1.0000 | +0.0000 | Not estimated | ok |
| hard_negative_top1_rate | 0.0000 | 0.0000 | +0.0000 | Not estimated | ok |
Not applicable (no qualifying labels): None.
Question evidence
6 of 6 candidate case checks passed. Ranking metrics can regress even when every case passes.
parental-leave · case check passed · MRR 1.0000 → 1.0000
How much paid parental leave is available?
Labelled relevant: leave-current · Required context: leave-current
Baseline
MRR: 1.0000 · nDCG: 1.0000 · case check: pass
leave-currentleave-hard-negative
Missing context: None
Citations not retrieved: None
Candidate
MRR: 1.0000 · nDCG: 1.0000 · case check: pass
leave-currentleave-hard-negative
Missing context: None
Citations not retrieved: None
api-key-rotation · case check passed · MRR 1.0000 → 1.0000
How often should production API credentials be rotated?
Labelled relevant: keys-current · Required context: keys-current
Baseline
MRR: 1.0000 · nDCG: 1.0000 · case check: pass
keys-currentleave-hard-negative
Missing context: None
Citations not retrieved: None
Candidate
MRR: 1.0000 · nDCG: 1.0000 · case check: pass
keys-currentleave-hard-negative
Missing context: None
Citations not retrieved: None
conflicting-retention-evidence · case check passed · MRR 1.0000 → 0.5000
How long should raw customer event records be retained?
Labelled relevant: retention-current · Required context: retention-current, retention-superseded
Baseline
MRR: 1.0000 · nDCG: 1.0000 · case check: pass
retention-currentretention-supersededleave-hard-negative
Missing context: None
Citations not retrieved: None
Candidate
MRR: 0.5000 · nDCG: 0.6309 · case check: pass
retention-supersededretention-currentleave-hard-negative
Missing context: None
Citations not retrieved: None
refund-window-with-conflict · case check passed · MRR 0.5000 → 0.5000
How many days after delivery can a customer request a refund?
Labelled relevant: refund-current · Required context: refund-current, refund-superseded
Baseline
MRR: 0.5000 · nDCG: 0.6309 · case check: pass
refund-supersededrefund-currentretention-current
Missing context: None
Citations not retrieved: None
Candidate
MRR: 0.5000 · nDCG: 0.6309 · case check: pass
refund-supersededrefund-currentretention-current
Missing context: None
Citations not retrieved: None
expense-deadline · case check passed · MRR 1.0000 → 1.0000
When are expense claims due?
Labelled relevant: expenses · Required context: expenses
Baseline
MRR: 1.0000 · nDCG: 1.0000 · case check: pass
expenses
Missing context: None
Citations not retrieved: None
Candidate
MRR: 1.0000 · nDCG: 1.0000 · case check: pass
expenses
Missing context: None
Citations not retrieved: None
sev1-escalation · case check passed · MRR 1.0000 → 1.0000
How quickly must a severity-one incident be escalated?
Labelled relevant: incident-handbook · Required context: incident-handbook
Baseline
MRR: 1.0000 · nDCG: 1.0000 · case check: pass
incident-handbookoncall-rotaexpenses
Missing context: None
Citations not retrieved: None
Candidate
MRR: 1.0000 · nDCG: 1.0000 · case check: pass
incident-handbookexpensesoncall-rota
Missing context: None
Citations not retrieved: None
Measured cost · not gated
Memory is the process peak, including earlier work in the same process. Compare isolated runs for cost decisions.
| Run | Build seconds | p50 ms | p95 ms | Peak RSS MB |
|---|---|---|---|---|
| baseline | 0.00 | 0.01 | 0.01 | 27.0 |
| candidate | 0.00 | 0.01 | 0.02 | 27.0 |
Reproduce the comparison
Run from the same working directory with the same local files and tool version. Downloads remain opt-in.
rag-gate gate --dataset fixtures/policies.json --k 3 --baseline configs/baseline.json --candidate configs/candidate-chunk-12.json --no-bootstrap --output reports/comparison.json --html reports/report.htmlDownload comparison JSON