AI Capstone: Governed Citation-First RAG
Build a multi-tenant knowledge assistant that answers only from current authorized evidence, cites exact source versions, and abstains when support is missing. The capstone combines structured extraction, chunking experiments, versioned embeddings, hybrid retrieval, reranking, row-level security, bounded context, prompt-injection defense, structured generation, citation validation, evaluation, observability, deletion, and recovery.
What is the mission?
Employees ask policy and procedure questions. The system must retrieve only documents their groups may access, distinguish current from obsolete versions, answer with claim-level citations, and refuse unsupported requests. Permission revocation and document deletion must propagate through retrieval, cache, citations, and logs within a defined SLO.
What are the user stories?
- Ask a conversational question and receive a concise evidence-based answer.
- Open every citation at the exact authorized source location.
- See an explicit “insufficient evidence” outcome.
- Correctly handle exact codes and semantic paraphrases.
- Allow administrators to inspect version, latency, and quality traces without seeing unnecessary content.
What architecture will you build?
Rendering diagram...
What are the milestones?
- Define source, document-version, chunk, ACL, model, embedding, job, and audit schemas.
- Build golden extraction fixtures with tables, pages, Unicode, and malicious instructions.
- Compare at least two chunking policies on 100 reviewed queries.
- Establish exact retrieval and RLS denial baselines.
- Add HNSW, FTS, RRF, and optional reranking with ablations.
- Implement structured citation-first API with deterministic providers.
- Integrate approved real providers behind versioned interfaces.
- Add context-injection tests, abstention, and citation entailment review.
- Add source-update, revocation, tombstone, and cache-invalidation flows.
- Load-test, restore, fail over, and rehearse rollback.
What are the evaluation gates?
- zero unauthorized chunks, answers, caches, or citations;
- held-out recall/nDCG targets by language and source type;
- groundedness and citation correctness reviewed under a written rubric;
- no-evidence abstention meets defined precision/recall;
- prompt injection cannot alter tool/policy behavior;
- p95 stage and end-to-end latency fit the budget;
- revocation/deletion and freshness SLOs pass;
- recovery meets RPO/RTO and preserves extension compatibility.
What is the threat model?
Test indirect prompt injection, ACL tampering, cross-tenant caches, invented citations, oversized context, malicious files, unapproved provider egress, sensitive logging, stale replicas, poisoned feedback, model drift, and denial of wallet through large requests.
How do you verify it?
Produce an evidence pack: data-flow map, threat model, role/RLS SQL, golden dataset, experiment manifests, ablations, quality report, red-team cases, latency/cost trace, deletion proof, backup restore, incident runbooks, limitations, and signed launch decision.
What breaks in production?
- A high answer score hides one authorization violation.
- Citations exist but do not support claims.
- New document versions mix with old chunks.
- Model calls receive data outside approved policy.
- Feedback becomes training/evaluation truth without review.
AI pair-work prompt
Be a skeptical RAG launch reviewer. For each artifact I provide, distinguish claim from evidence. Require authorization tests, exact/ANN/hybrid ablations, citation support, injection tests, deletion traces, stage SLOs, cost, recovery, and known limitations. Return only the next three highest-risk gaps.
Verification contract: Launch requires independent reproduction of the security and quality gates on the release candidate and dataset snapshot.
Check your understanding
- Why evaluate retrieval and answer quality separately?
- Which deletion paths must be tested?
- What belongs in the final evidence pack?
Official references