Operate, Improve, and Migrate Agent Systems
Agent operations is a continuous loop: observe outcomes, investigate traces, label failures, update requirements and cases, change one controlled layer, compare against baseline, canary, monitor, and roll back when necessary. Model or provider migration is a capability and behavior change, not a string replacement; replay contracts before shifting production traffic.
What will you be able to do?
You will run an improvement cadence, diagnose drift, and design a staged migration that preserves safety and business invariants.
What is the improvement loop?
Do not change model, prompt, tools, and policy together unless an emergency requires it; otherwise attribution becomes impossible.
What kinds of drift occur?
- user/task distribution changes;
- knowledge and business policy changes;
- model/provider behavior or availability changes;
- tool/MCP schemas change;
- prompt/skill/plugin assets drift;
- evaluation sets become unrepresentative; and
- operators develop workarounds that signal product gaps.
Monitor inputs and outcomes without collecting unnecessary sensitive content.
How do you migrate providers or models?
- Inventory required capabilities and provider-native extensions.
- Update adapter in isolation.
- Pass deterministic adapter contracts.
- Replay sanitized traces with stubbed tools.
- Run full quality/safety/latency/cost evaluation.
- Shadow read-only traffic when policy permits.
- Canary low-risk tasks with no cross-provider side-effect retry.
- Increase traffic gradually with rollback thresholds.
- Preserve old session completion or explicitly migrate supported state.
- Retire credentials, sessions, traces, and dependencies under policy.
Failure injection: Force a migration fallback after a mutation response is lost. The system must reconcile the original provider/tool attempt before any new provider continues.
What should an operating review include?
SLOs, evaluation trend, top failure clusters, unsafe/blocked proposals, cost per success, approval latency/fatigue, provider/tool incidents, drift indicators, upcoming changes, and completed regression cases.
What mistakes should you avoid?
- Auto-upgrading models without evaluations.
- Migrating active sessions by copying text alone.
- Running dual providers on sensitive data without policy.
- Keeping deprecated tools/credentials “just in case.”
- Optimizing feedback that is loud but unrepresentative.
Check your understanding
- Why is migration not a model-name change?
- Which change sequence preserves attribution?
- What must be retired after migration?
Expert extension
Write a migration plan from one adapter to the other with capability gaps, evaluation thresholds, session policy, canary stages, rollback triggers, and deletion evidence.
Official references