Progressive Delivery Overview

You've deployed applications to Kubernetes using ArgoCD. That works for many scenarios, but you've faced a critical limitation: all-or-nothing deployments. When you apply a new Deployment, Kubernetes rolls out the new version to all replicas at once. If there's a bug in that new version, all users see it. You've swapped a working version for a broken one in seconds.

In production, this is unacceptable. You need progressive delivery—the ability to deploy new versions safely by rolling them out gradually, watching them carefully, and rolling back if problems appear.

This chapter introduces two complementary approaches to progressive delivery: canary deployments (gradually shift traffic to new version) and blue-green deployments (run both versions, switch all traffic instantly). We'll then introduce Argo Rollouts, the Kubernetes-native tool that automates these patterns.

By the end of this chapter, you'll understand why AI agents especially benefit from progressive delivery (behavior changes are subtle), the mechanics of canary vs blue-green strategies, and how Argo Rollouts implements them.

Why Progressive Delivery Matters for AI Agents

Before we dive into mechanics, let's establish why progressive delivery is non-negotiable for AI-powered services.

Traditional Deployments: All-or-Nothing Risk

When you deploy a new version of your FastAPI agent with a standard Kubernetes Deployment:

Output:

Kubernetes replaces old pods with new ones gradually (rolling update). But here's the problem: the entire system is running v2.0 within minutes. If v2.0 has a subtle bug—maybe it incorrectly calculates priorities, or misses edge cases in task decomposition—every single user-facing request encounters that bug.

For traditional services (static content, CRUD APIs), this is manageable. You monitor error rates, and if something breaks, you rollback.

For AI agents, this is catastrophic.

The AI Agent Problem: Subtle Failures

AI agent behavior changes are fundamentally different from traditional software bugs.

Traditional bug: "Function throws exception when input is null"

• Detection: Immediate (exception logged)
• Impact scope: Specific input condition
• Fix: Patch the function

Agent behavior change: "New prompt template makes agent more aggressive in scheduling tasks"

• Detection: Gradual (users notice subtly different task prioritization over hours)
• Impact scope: ALL requests, ALL users, cascading downstream effects
• Fix: Requires investigation, retraining, prompt refinement

With all-or-nothing deployments, you can't distinguish between:

• "This version is working well, let's keep it"
• "This version works but behaves differently than expected"
• "This version is broken"

Progressive delivery solves this by allowing gradual, observable rollouts where you can:

1. Deploy to a small percentage of users and observe behavior
2. Run experiments (e.g., canary traffic) and compare metrics
3. Rollback instantly if something goes wrong
4. Promote gradually as confidence increases

This is especially critical for agent behavior because the failures are:

• Subtle: Not obvious in logs or error rates
• Wide-impact: Affect all users when fully deployed
• Observable through metrics: Task completion rates, execution times, user satisfaction

Progressive delivery lets you catch these before they reach everyone.

The Two Approaches to Progressive Delivery

There are two main strategies for safely deploying new versions: canary and blue-green. They solve the same problem but with different tradeoffs.

Canary Deployments: Gradual Traffic Shift

Concept: Deploy the new version alongside the old version, then gradually shift traffic from old to new. Monitor metrics at each step. If something goes wrong, rollback by returning all traffic to the old version.

Visual model:

Process:

1. Deploy new version: Run new version in parallel with old
2. Route percentage of traffic: Send X% to new, (100-X)% to old
3. Monitor metrics: Watch error rates, latency, task completion in the canary traffic
4. Increase traffic: If metrics are healthy, increase percentage to new version
5. Promote or rollback: If metrics degrade, immediately return all traffic to old version

Example timeline for a 5-step canary:

If something goes wrong:

Advantages:

• Gradual validation: Real traffic validates new version before full deployment
• Early detection: Catch behavior changes while only affecting small percentage of users
• Flexibility: Adjust timing and percentages per deployment

Disadvantages:

• Dual-stack overhead: Both versions run simultaneously, increasing resource usage
• Stateful behavior complexity: Requests from same user may hit different versions
• Requires metrics: You must have alerting/metrics to detect problems in canary traffic

Blue-Green Deployments: Instant Switch

Concept: Run two complete versions of your application (blue and green). All traffic currently goes to blue. Deploy green with the new version, fully test it, then switch ALL traffic to green instantly. If green has problems, switch back to blue.

Visual model:

Process:

1. Deploy green: Full parallel deployment of new version
2. Pre-flight validation: Synthetic tests, health checks, smoke tests (NO real traffic)
3. Traffic switch: Route 100% traffic from blue to green instantly
4. Observe: Watch metrics in green
5. Rollback or promote: If something goes wrong, switch back to blue. If healthy, scale down blue.

Example timeline for blue-green:

If something goes wrong:

Advantages:

• Zero-downtime: Traffic never interrupted, just switched
• Instant rollback: If something goes wrong, switch back in seconds
• No dual-stack overhead: Only two full replicas (one active, one hot)
• Clear before/after: Tests run before any real traffic hits new version

Disadvantages:

• Large resource footprint: Must run both versions fully (2x resource usage)
• Limited real-traffic validation: Tests are synthetic, not real user patterns
• Stateless requirement: Works best for stateless services (agents are typically stateless)

Canary vs Blue-Green: Which to Choose?

There's no universal "better" strategy. The choice depends on your constraints:

For AI agents, canary deployments are often superior because:

• Subtle behavior changes require real traffic to validate
• Gradual exposure minimizes impact if something goes wrong
• Metrics collection lets you detect agent behavior differences (task completion, latency, satisfaction)

Introducing Argo Rollouts

Now that you understand the concepts, let's introduce the tool: Argo Rollouts.

Argo Rollouts is a Kubernetes controller that automates progressive delivery. Instead of writing your own canary/blue-green logic, you declare a Rollout resource (similar to Deployment) with a strategy (canary or blue-green) and steps (traffic percentages, timing, analysis).

Argo Rollouts then:

• Manages the old and new versions
• Routes traffic according to your strategy
• Monitors metrics and health checks
• Promotes or rolls back based on analysis

Basic structure:

Output:

When you apply this Rollout:

Argo Rollouts takes over:

1. Deploys v2.0: Creates pods with new version
2. Shifts traffic: At step 1, routes 20% to new version, 80% to old
3. Monitors analysis: Every 30 seconds, checks error_rate metric
4. Promotes or waits: If analysis passes, waits 5 minutes, then moves to next step
5. Full promotion: After all steps pass analysis, promotes v2.0 to 100%

If analysis fails at any step:

You can immediately rollback:

And Argo Rollouts returns all traffic to the previous version.

The Rollout CRD: Key Concepts

Let's map the Rollout resource to the concepts you know:

Deployment vs Rollout (conceptual comparison):

Output:

Output:

Key differences:

Integration with ArgoCD: The Full Picture

You might be wondering: "I'm already using ArgoCD to deploy applications. How does Argo Rollouts fit in?"

ArgoCD deploys resources to Kubernetes. Argo Rollouts is a resource that ArgoCD can deploy.

The architecture looks like:

Example workflow:

1. You commit a new version to Git:

   bash

   WrapCopy

   git commit -m "feat: improve task decomposition in agent" git push origin main

2. GitHub Actions CI/CD pipeline runs:

   • Builds new image: myregistry/task-agent:v2.0
   • Tests it, pushes to registry
   • Updates rollout.yaml: Changes image to v2.0

3. ArgoCD detects Git change:

   bash

   WrapCopy

   $ argocd app get task-agent Name: task-agent Namespace: default Status: OutOfSync Repository: https://github.com/you/agent-repo

4. ArgoCD syncs (applies rollout.yaml):

   bash

   WrapCopy

   $ argocd app sync task-agent SYNCED - Rollout task-agent created with image v2.0

5. Argo Rollouts controller takes over:

   • Starts canary deployment
   • Shifts traffic: 20% → 50% → 100%
   • Monitors metrics
   • Completes deployment or rolls back

This is the GitOps + Progressive Delivery integration: Git is your source of truth, ArgoCD keeps the cluster in sync with Git, and Argo Rollouts automates safe deployment strategies.

Canary vs Blue-Green Revisited: Implementation with Argo Rollouts

Now let's see how each strategy looks as a Rollout resource.

Canary Rollout

Output:

Blue-Green Rollout

Output:

Key Concepts to Remember

• Canary deployments: Gradual traffic shift (10% → 100%) using real traffic for validation. Best for AI agents.
• Blue-green deployments: Instant traffic switch (0% → 100%) after synthetic tests. Best for zero-downtime with stateless services.
• Argo Rollouts: Kubernetes controller that automates these strategies and monitors metrics for auto-promotion/rollback.
• AI Agent Need: Subtle behavior changes require gradual validation and instant rollback capabilities.

Try With AI

Setup: Open your agent repository in your editor. You'll work with Argo Rollouts concepts.

Prompt 1 - Understanding Your Current Risk

Ask AI: "I currently deploy my FastAPI agent using a standard Kubernetes Deployment with RollingUpdate strategy. Explain what happens if the new version has a subtle bug in task prioritization logic—how quickly will all users see it, and what's my rollback process?"

Prompt 2 - Canary vs Blue-Green Decision

Ask AI: "My agent processes user requests in isolation (no session state). For deploying a new agent version with improved prompt templates, should I use canary or blue-green strategy? Explain the tradeoffs for my use case."

Prompt 3 - Translating Concepts to Manifests

Ask AI: "Show me a Rollout manifest for a canary deployment of an agent with 3 replicas. The canary should shift traffic: 20% for 5m → 50% for 5m → 100%. Include a simple success metric (error rate should stay below 1%)."

---

Reflect on Your Skill

You built a gitops-deployment skill in Chapter 0. Test and improve it based on what you learned.

Test Your Skill

Identify Gaps

Ask yourself:

• Did my skill include Argo Rollouts configuration with canary steps?
• Did it handle analysis templates for automated promotion/rollback?

Improve Your Skill

If you found gaps: