Capacity controls keep a service useful under normal bursts, abusive traffic, and dependency slowdown. Caches reduce repeated work; rate limits shape arrival; concurrency limits cap in-flight pressure; queues absorb bounded bursts; backpressure slows producers; and load shedding rejects low-priority work before the whole system collapses. Each control needs tenant fairness and failure policy.
Use multiple layers. A generous request limit does not control a single request that launches 100 model calls.
Reserve estimated tokens/cost atomically before a call, reject or downgrade when the budget is unavailable, and reconcile actual usage afterward. Scope by verified tenant, user, feature, and time window. Keep durable billing/audit totals in PostgreSQL; Redis may provide a fast admission counter that reconciles with truth.
Return 429 for caller-specific limits and include a safe Retry-After where meaningful. Use 503 for temporary service capacity failure. Do not reveal another tenant's usage.
Cache only an explicitly safe representation. Include tenant, authorization variation, input hash, prompt/schema/model/policy/index versions, and locale where relevant. Set TTL and jitter, cap value size, protect against stampede, and define invalidation. Never let a cache hit bypass current authorization for a consequential or newly revoked resource.
Provider slowdown increases in-flight work, which exhausts server connections, queue depth, and cost. Apply admission control before spending, reserve high-risk-review capacity, and make lower-cost fallbacks explicit. Graceful degradation might disable AI drafting while ticket CRUD remains available.
Design controls for normal ticket reads, AI drafts, and batch evaluations. Simulate a provider slowdown from 1 to 20 seconds and a tenfold traffic burst.
Acceptance criteria: CRUD latency stays within its goal, queues remain bounded, premium policy is explicit rather than accidental, spend cannot exceed the hard quota, and clients receive actionable errors.