A golden dataset is a versioned set of representative queries, eligible documents, relevance judgments, slices, and exclusions. It turns retrieval changes into testable hypotheses. Measure recall@k, precision@k, reciprocal rank, nDCG, no-result behavior, latency, and policy violations; preserve a held-out set and inspect failures instead of optimizing one average score.
Sample real, privacy-reviewed queries plus designed edge cases. Define the unit: chunk, document, product, or answer evidence. Use grades such as 0 irrelevant, 1 marginal, 2 relevant, 3 ideal. Give annotators the authorized candidate universe and adjudicate disagreements. Record labeler, timestamp, guideline version, and notes.
Do not label only results returned by the current system; pool candidates from lexical, semantic, hybrid, and manual discovery to reduce blind spots.
Report query-level distributions and slice tables, not only averages.
Version queries, labels, source snapshot, embedding contract, SQL, ANN parameters, fusion, reranker, and code. Separate development for tuning and test for final decisions. When the test set becomes familiar, rotate or add a new holdout.
Hand-calculate metrics for three tiny queries, then compare with evaluation code. Introduce an intentionally swapped rank and confirm MRR/nDCG change. Re-run a frozen baseline twice. Use bootstrap confidence intervals or paired tests when differences are small and decisions are material.
Create a retrieval evaluation protocol for [domain]. Define query sampling, privacy review, candidate pooling, graded labels, adjudication, development/test split, formulas, slices, policy gates, latency/cost gates, uncertainty, and an error-analysis template.
Verification contract: Independently hand-check a sample of labels and metric calculations; no security violation may be traded for aggregate relevance.