Retrieval-augmented generation begins with governed ingestion, not a vector query. A reliable pipeline authorizes and scans sources, extracts structured text, creates semantically coherent chunks with lineage and access metadata, embeds them with a versioned model, evaluates retrieval quality, and supports reprocessing and deletion without mixing incompatible representations.
For each document: tenant, owner, source, content hash, detected type, scan result, parser version, text hash, chunker version, embedding model/version, vector dimensions, access policy, retention, status, and timestamps.
For each chunk: document/version ID, tenant, ordered position, source locator such as page and heading, exact text, token estimate, content hash, access metadata, embedding version, and searchable fields.
Start with natural structural boundaries—headings, paragraphs, list blocks, table regions, or code units—then combine within a measured token range with limited overlap. Fixed character windows are a baseline, not a universal best practice. Store heading context separately rather than duplicating huge prefixes in every chunk.
Build a small retrieval dataset before optimizing: representative questions, expected relevant source passages, tenant/access context, and hard negatives. Compare chunking strategies using recall at k, ranking quality, citation usefulness, index cost, and answer performance.
Start with a PostgreSQL vector extension when operational simplicity and scale fit; it keeps metadata, tenant scope, and transactions close. Move to a dedicated search/vector service only when measured scale, hybrid-search capability, latency, availability, or operations justify it. The abstraction should express retrieval, not a vendor API.
Never mix vectors with incompatible models or dimensions in one assumed space. Write new chunks/embeddings under a new index version, evaluate them in shadow, then atomically switch the active version. Keep rollback until the new version is proven. Use document hashes to avoid duplicate cost.
Poor ingestion cannot be fixed by a better prompt. RAG quality, privacy, and freshness originate here. An inaccessible source must never create an accessible chunk, and deletion requests must reach embeddings, caches, generated summaries, and evaluation copies according to policy.
Take five support documents with headings and repeated footers. Specify two chunking strategies and a 15-question retrieval dataset including three cross-tenant negatives.
Acceptance criteria: every chunk maps to a source location, repeated noise is removed, inaccessible chunks never appear, strategies are compared with recorded metrics, and deletion removes every derived record.