Model routing maps a task’s capability, risk, data, latency, and quality requirements to an evaluated deployment. A router must enforce eligibility before optimization: unsupported tools, disallowed regions, missing structured output, or sensitive-data restrictions eliminate candidates. Among eligible choices, use measured task performance and explicit fallback semantics—not model fame.
You will create an eligibility-first router, define fallback rules, and evaluate routing as part of the system rather than an invisible cost trick.
The deployment catalog should record provider, model, region, capabilities, evaluated scores, latency distribution, cost proxy, status, and policy labels. Refresh volatile facts from configuration and official sources.
Do not send sensitive content to several providers merely to race them unless policy explicitly permits that data multiplication.
A fallback must preserve the task’s required capabilities and controls. Never fall from a structured, approval-aware tool path to free-form chat and call it graceful degradation. If no eligible route exists, return a clear unavailable/queued/human-review outcome.
Failure injection: Mark the preferred deployment unhealthy. The router should select only a fully eligible alternative or stop. It must not relax data residency or tool-policy requirements.
Replay representative cases against candidate deployments and routing policies. Score selection accuracy, final task quality, policy compliance, latency, cost, fallback frequency, and churn. Shadow evaluation may compare outputs without exposing side effects.
Implement a deterministic router over a fake deployment catalog and property-test that disallowed data classes never select forbidden providers.