An AI retrieval service needs ordinary database telemetry plus ingestion freshness, retrieval quality, policy violations, model/config versions, and answer evidence metrics. Define user-centered service-level indicators and objectives, alert on actionable symptoms, trace stage latency safely, run synthetic golden queries, and prepare incident playbooks for stale data, quality drift, leakage, overload, and provider failure.
Database: connections, pool wait, query latency, errors, locks, CPU, memory, I/O, WAL, replication lag, dead tuples, vacuum, index size/use. Ingestion: queue age, throughput, retries, terminal failures, source-to-visible lag, re-embedding completeness. Retrieval: result count, exact recall sample, relevance metrics, filters, stage latency, cache behavior. RAG: context tokens, citation validation, groundedness evaluation, abstention, model errors, cost. Security: denied access, policy errors, secret detection, deletion SLO.
Avoid labels such as raw query text, user ID, document ID, or tenant ID when they create high-cardinality or privacy risk.
Example: “99% of authorized search requests return at least the required candidate count within 800 ms over 28 days, while sampled ANN recall@10 remains at least 0.95 and cross-tenant violations remain zero.” Quality and security do not fit perfectly into one availability ratio, so use multiple explicit gates.
Run a stable canary query set, sample exact-versus-ANN comparisons, track query and document distributions, monitor result/no-result rates by approved slice, and compare versioned configurations. Drift can come from content, users, models, ANN maintenance, filters, or labels.
Inject synthetic failures: queue stall, high replica lag, provider throttling, low ANN effort, invalid index, access-policy error, and stale cache. Confirm the expected metric, alert, dashboard, owner, runbook, and recovery evidence.
Turn this architecture and product promise [paste] into SLIs, SLOs, error budgets, metrics, privacy-safe labels, dashboards, alerts, synthetic probes, drift tests, and runbooks. Include exact/ANN quality and deletion/authorization gates.
Verification contract: Fire one controlled failure per alert and prove an operator can identify the version, scope, owner, and safe mitigation from telemetry alone.