USMAN’S INSIGHTS
AI ARCHITECT
  • Home
  • About
  • Thought Leadership
  • Book
Press / Contact
USMAN’S INSIGHTS
AI ARCHITECT
⌘F
HomeBook
HomeBookShipping to 100% of Users Is a Bet You'll Eventually Lose
Previous Chapter
TLS Termination with CertManager
Next Chapter
Autoscaling with HPA VPA KEDA
AI NOTICE: This is the table of contents for the SPECIFIC CHAPTER only. It is NOT the global sidebar. For all chapters, look at the main navigation.

On this page

60 sections

Progress0%
1 / 60

Muhammad Usman Akbar Entity Profile

Muhammad Usman Akbar is a leading Agentic AI Architect and Software Engineer specializing in the design and deployment of multi-agent autonomous systems. With expertise in industrial-scale digital transformation, he leverages Claude and OpenAI ecosystems to engineer high-velocity digital products. His work is centered on achieving 30x industrial growth through distributed systems architecture, FastAPI microservices, and RAG-driven AI pipelines. Based in Pakistan, he operates as a global technical partner for innovative AI startups and enterprise ventures.

USMAN’S INSIGHTS
AI ARCHITECT

Transforming businesses into autonomous AI ecosystems. Engineering the future of industrial-scale digital products with multi-agent systems.

30X Growth
AI-First
Innovation

Navigation

  • Home
  • Book
  • About
  • Contact
Let's Collaborate

Have a Project in Mind?

Let's build something extraordinary together. Transform your vision into autonomous AI reality.

Start Your Transformation

© 2026 Muhammad Usman Akbar. All rights reserved.

Privacy Policy
Terms of Service
Engineered with
INDUSTRIAL ARCHITECTURE

Traffic Splitting Patterns

Module 7 takes the agent you built in Module 6 and turns it into a production cloud service. You'll containerize the stack, orchestrate it on Kubernetes, automate delivery, and operate it with observability, security, and cost controls. The goal: a reliable Digital FTE that runs 24/7 for real users.

Prerequisites: Modules 4-6. You need a working agent service to deploy.

You've deployed version 2 of your Task API. It passed all tests in staging. But staging is not production. Real users behave differently than test suites. Network conditions vary. Edge cases appear that no one anticipated. Shipping a new version directly to 100% of users is gambling—if something breaks, everyone breaks.

Traffic splitting lets you deploy with confidence. Send 5% of traffic to the new version. Watch the error rates. If errors spike, roll back instantly. If metrics look good, increase to 25%, then 50%, then 100%. This is progressive delivery—reducing risk by validating in production with real traffic, while keeping most users safe on the proven version.

This lesson teaches three deployment patterns: canary (gradual rollout), blue-green (instant switch), and A/B testing (feature experiments). You'll implement each using HTTPRoute traffic weights and header-based routing. By the end, you'll deploy new versions without the 3 AM panic of a full cutover gone wrong.


Understanding Traffic Splitting

Traffic splitting divides incoming requests between multiple backend services. Gateway API implements this through HTTPRoute backendRefs with weight fields.

How Weights Work

yaml
backendRefs: - name: service-v1 port: 8000 weight: 90 - name: service-v2 port: 8000 weight: 10

Weights are relative. In this example:

  • Total weight: 90 + 10 = 100
  • service-v1 receives: 90/100 = 90%
  • service-v2 receives: 10/100 = 10%

Weights don't need to sum to 100:

yaml
backendRefs: - name: service-v1 weight: 9 - name: service-v2 weight: 1

This produces the same 90/10 split (9/10 = 90%, 1/10 = 10%).

Three Deployment Patterns

PatternDescriptionUse When
CanaryGradual rollout with increasing trafficNew features, risky changes
Blue-GreenInstant switch between environmentsDatabase migrations, breaking changes
A/B TestingRoute specific users to test versionFeature experiments, beta programs

Canary Deployment

Canary deployment releases changes to a small subset of users first. If the canary version fails, only that subset is affected. The name comes from coal mining—canaries detected toxic gases before miners were harmed.

Initial Canary Setup (5%)

Deploy version 2 alongside version 1:

bash
# Deploy canary version kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: task-api-canary namespace: task-api spec: replicas: 1 selector: matchLabels: app: task-api version: canary template: metadata: labels: app: task-api version: canary spec: containers: - name: task-api image: task-api:v2 ports: - containerPort: 8000 --- apiVersion: v1 kind: Service metadata: name: task-api-canary namespace: task-api spec: selector: app: task-api version: canary ports: - port: 8000 EOF

Output:

text
deployment.apps/task-api-canary created service/task-api-canary created

Create HTTPRoute with 5% canary traffic:

yaml
apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: task-api-canary namespace: task-api spec: parentRefs: - name: task-api-gateway rules: - matches: - path: type: PathPrefix value: /api/ backendRefs: - name: task-api-stable port: 8000 weight: 95 - name: task-api-canary port: 8000 weight: 5

Apply:

bash
kubectl apply -f task-api-canary-route.yaml

Output:

text
httproute.gateway.networking.k8s.io/task-api-canary created

Verify Traffic Split

Send requests and check distribution:

bash
for i in {1..100}; do curl -s http://localhost:8080/api/version | jq -r '.version' done | sort | uniq -c

Output:

text
95 v1 5 v2

Approximately 5% of requests reach the canary.

Progressive Rollout Stages

As confidence grows, increase canary traffic:

Stage 1: 5% (initial validation)

yaml
backendRefs: - name: task-api-stable weight: 95 - name: task-api-canary weight: 5

Stage 2: 25% (broader validation)

yaml
backendRefs: - name: task-api-stable weight: 75 - name: task-api-canary weight: 25

Stage 3: 50% (confidence building)

yaml
backendRefs: - name: task-api-stable weight: 50 - name: task-api-canary weight: 50

Stage 4: 100% (full rollout)

yaml
backendRefs: - name: task-api-canary weight: 100

After full rollout, rename canary to stable and remove the old deployment.

Monitoring During Canary

Watch error rates between versions:

bash
# Compare error rates kubectl logs -l version=stable -n task-api --tail=100 | grep -c ERROR kubectl logs -l version=canary -n task-api --tail=100 | grep -c ERROR

Output:

text
3 # stable errors 12 # canary errors - 4x higher!

If canary shows significantly higher errors, rollback immediately.


Canary Rollback

When canary metrics look bad, rollback is one YAML change away.

Immediate Rollback

Shift all traffic back to stable:

yaml
apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: task-api-canary namespace: task-api spec: parentRefs: - name: task-api-gateway rules: - matches: - path: type: PathPrefix value: /api/ backendRefs: - name: task-api-stable port: 8000 weight: 100 - name: task-api-canary port: 8000 weight: 0

Apply:

bash
kubectl apply -f task-api-canary-route.yaml

Output:

text
httproute.gateway.networking.k8s.io/task-api-canary configured

Traffic instantly stops going to canary. Requests in flight complete, but no new requests reach the problematic version.

Verify Rollback

bash
for i in {1..20}; do curl -s http://localhost:8080/api/version | jq -r '.version' done | sort | uniq -c

Output:

text
20 v1

All traffic now goes to stable.

Cleanup Failed Canary

After rollback, clean up the failed deployment:

bash
kubectl delete deployment task-api-canary -n task-api kubectl delete service task-api-canary -n task-api

Output:

text
deployment.apps "task-api-canary" deleted service "task-api-canary" deleted

Blue-Green Deployment

Blue-green deployment maintains two identical environments. One serves production traffic (blue), the other sits idle with the new version (green). When ready, you switch all traffic instantly.

When to Use Blue-Green

ScenarioPattern
Database schema changesBlue-green (clean cutover)
Breaking API changesBlue-green (all-or-nothing)
Regulatory complianceBlue-green (auditable switch)
Gradual feature rolloutCanary (progressive)
Performance testingCanary (compare metrics)

Setting Up Blue-Green

Deploy both environments:

bash
# Blue environment (current production) kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: task-api-blue namespace: task-api spec: replicas: 3 selector: matchLabels: app: task-api environment: blue template: metadata: labels: app: task-api environment: blue spec: containers: - name: task-api image: task-api:v1 ports: - containerPort: 8000 --- apiVersion: v1 kind: Service metadata: name: task-api-blue namespace: task-api spec: selector: app: task-api environment: blue ports: - port: 8000 EOF
bash
# Green environment (new version, idle) kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: task-api-green namespace: task-api spec: replicas: 3 selector: matchLabels: app: task-api environment: green template: metadata: labels: app: task-api environment: green spec: containers: - name: task-api image: task-api:v2 ports: - containerPort: 8000 --- apiVersion: v1 kind: Service metadata: name: task-api-green namespace: task-api spec: selector: app: task-api environment: green ports: - port: 8000 EOF

Output:

text
deployment.apps/task-api-blue created service/task-api-blue created deployment.apps/task-api-green created service/task-api-green created

Initial State: Blue Active

Route all traffic to blue:

yaml
apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: task-api-bluegreen namespace: task-api spec: parentRefs: - name: task-api-gateway rules: - matches: - path: type: PathPrefix value: /api/ backendRefs: - name: task-api-blue port: 8000 weight: 100 - name: task-api-green port: 8000 weight: 0

Switch to Green

When green is validated and ready:

yaml
apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: task-api-bluegreen namespace: task-api spec: parentRefs: - name: task-api-gateway rules: - matches: - path: type: PathPrefix value: /api/ backendRefs: - name: task-api-blue port: 8000 weight: 0 - name: task-api-green port: 8000 weight: 100

Apply:

bash
kubectl apply -f task-api-bluegreen-route.yaml

Output:

text
httproute.gateway.networking.k8s.io/task-api-bluegreen configured

Traffic instantly switches to green. To rollback, swap weights back.

Blue-Green Rollback

If green has issues, instant rollback:

yaml
backendRefs: - name: task-api-blue weight: 100 - name: task-api-green weight: 0

The old environment is still running—just flip the switch.


A/B Testing with Headers

A/B testing routes specific users to different versions. Unlike canary (random percentage), A/B testing is deterministic—users with certain attributes always see the test version.

Header-Based Routing

Route beta users to the experimental version:

yaml
apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: task-api-ab namespace: task-api spec: parentRefs: - name: task-api-gateway rules: # Beta users see experimental version - matches: - path: type: PathPrefix value: /api/ headers: - name: x-beta-user value: "true" backendRefs: - name: task-api-experimental port: 8000 # Everyone else sees stable version - matches: - path: type: PathPrefix value: /api/ backendRefs: - name: task-api-stable port: 8000

Apply:

bash
kubectl apply -f task-api-ab-route.yaml

Output:

text
httproute.gateway.networking.k8s.io/task-api-ab created

Test A/B Routing

bash
# Regular user sees stable curl http://localhost:8080/api/version

Output:

text
{"version": "stable", "features": ["core"]}
bash
# Beta user sees experimental curl -H "x-beta-user: true" http://localhost:8080/api/version

Output:

text
{"version": "experimental", "features": ["core", "ai-suggestions", "smart-tags"]}

Premium Tier Routing

Route premium users to dedicated infrastructure:

yaml
apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: task-api-premium namespace: task-api spec: parentRefs: - name: task-api-gateway rules: # Premium API keys get dedicated backend - matches: - path: type: PathPrefix value: /api/ headers: - name: x-api-tier value: "premium" backendRefs: - name: task-api-premium port: 8000 # Standard users share the main backend - matches: - path: type: PathPrefix value: /api/ backendRefs: - name: task-api-standard port: 8000

Premium users get:

  • Dedicated pods (no noisy neighbors)
  • Higher rate limits
  • Priority during high load

Combining Patterns

Real deployments often combine patterns. Canary the new version to 10% of standard users while all premium users stay on stable.

Canary with Premium Protection

yaml
apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: task-api-combined namespace: task-api spec: parentRefs: - name: task-api-gateway rules: # Premium users ALWAYS get stable (no experiments) - matches: - path: type: PathPrefix value: /api/ headers: - name: x-api-tier value: "premium" backendRefs: - name: task-api-stable port: 8000 weight: 100 # Standard users get canary rollout - matches: - path: type: PathPrefix value: /api/ backendRefs: - name: task-api-stable port: 8000 weight: 90 - name: task-api-canary port: 8000 weight: 10

Premium users are protected—they never see the canary. Standard users get the gradual rollout.


Monitoring Traffic Split Results

Traffic splitting without monitoring is flying blind. You need to see error rates, latency, and throughput per version.

Prometheus Queries

Error rate by version:

promql
sum(rate(http_requests_total{status=~"5.."}[5m])) by (version) / sum(rate(http_requests_total[5m])) by (version)

Latency by version (p99):

promql
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (version, le))

Requests per second by version:

promql
sum(rate(http_requests_total[5m])) by (version)

Quick Health Check Script

bash
#!/bin/bash # compare-versions.sh echo "Comparing stable vs canary..." # Error rates STABLE_ERRORS=$(kubectl logs -l version=stable -n task-api --tail=1000 | grep -c ERROR) CANARY_ERRORS=$(kubectl logs -l version=canary -n task-api --tail=1000 | grep -c ERROR) echo "Stable errors: $STABLE_ERRORS" echo "Canary errors: $CANARY_ERRORS" if [ "$CANARY_ERRORS" -gt $((STABLE_ERRORS * 2)) ]; then echo "WARNING: Canary error rate is more than 2x stable!" echo "Consider rollback." fi

Run:

bash
chmod +x compare-versions.sh ./compare-versions.sh

Output:

text
Comparing stable vs canary... Stable errors: 3 Canary errors: 2

Canary looks healthy—proceed with rollout.


Exercises

Exercise 1: Create a Canary Deployment

Set up 10% canary traffic:

bash
kubectl apply -f - <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: exercise-canary namespace: default spec: parentRefs: - name: task-api-gateway rules: - matches: - path: type: PathPrefix value: /api/ backendRefs: - name: task-api-stable port: 8000 weight: 90 - name: task-api-canary port: 8000 weight: 10 EOF

Verify:

bash
kubectl get httproute exercise-canary -o yaml | grep -A6 backend Refs

Expected Output:

text
backendRefs: - name: task-api-stable port: 8000 weight: 90 - name: task-api-canary port: 8000 weight: 10

Exercise 2: Configure A/B Testing

Route beta users to a test version:

bash
kubectl apply -f - <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: exercise-ab namespace: default spec: parentRefs: - name: task-api-gateway rules: - matches: - path: type: PathPrefix value: /api/ headers: - name: x-beta-user value: "true" backendRefs: - name: task-api-beta port: 8000 - matches: - path: type: PathPrefix value: /api/ backendRefs: - name: task-api-stable port: 8000 EOF

Test (if services exist):

bash
# Beta user curl -H "x-beta-user: true" http://localhost:8080/api/version # Regular user curl http://localhost:8080/api/version

Exercise 3: Blue-Green Switch

Create blue-green configuration and switch environments:

bash
# Initial: blue active kubectl apply -f - <<EOF apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: exercise-bluegreen namespace: default spec: parentRefs: - name: task-api-gateway rules: - matches: - path: type: PathPrefix value: /api/ backendRefs: - name: task-api-blue port: 8000 weight: 100 - name: task-api-green port: 8000 weight: 0 EOF

Switch to green (modify and reapply):

bash
kubectl patch httproute exercise-bluegreen -n default --type=merge -p '{ "spec": { "rules": [{ "matches": [{"path": {"type": "PathPrefix", "value": "/api/"}}], "backendRefs": [ {"name": "task-api-blue", "port": 8000, "weight": 0}, {"name": "task-api-green", "port": 8000, "weight": 100} ] }] } }'

Expected Output:

text
httproute.gateway.networking.k8s.io/exercise-bluegreen patched

Exercise 4: Rollback Canary

Practice rollback by setting canary weight to 0:

bash
kubectl patch httproute exercise-canary -n default --type=merge -p '{ "spec": { "rules": [{ "matches": [{"path": {"type": "PathPrefix", "value": "/api/"}}], "backendRefs": [ {"name": "task-api-stable", "port": 8000, "weight": 100}, {"name": "task-api-canary", "port": 8000, "weight": 0} ] }] } }'

Verify:

bash
kubectl get httproute exercise-canary -o yaml | grep -A6 backend Refs

Expected Output:

text
backendRefs: - name: task-api-stable port: 8000 weight: 100 - name: task-api-canary port: 8000 weight: 0

Reflect on Your Skill

You built a traffic-engineer skill in Lesson 0. Based on what you learned about traffic splitting:

Add Deployment Pattern Templates

Your skill should generate configuration for each pattern:

Canary template:

yaml
# Template: canary-deployment apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: {{ service }}-canary namespace: {{ namespace }} spec: parentRefs: - name: {{ gateway }} rules: - matches: - path: type: PathPrefix value: {{ path_prefix }} backendRefs: - name: {{ stable_service }} port: {{ port }} weight: {{ stable_weight }} - name: {{ canary_service }} port: {{ port }} weight: {{ canary_weight }}

Blue-green template:

yaml
# Template: blue-green backendRefs: - name: {{ service }}-blue port: {{ port }} weight: {{ blue_weight }} - name: {{ service }}-green port: {{ port }} weight: {{ green_weight }}

A/B testing template:

yaml
# Template: ab-testing rules: - matches: - path: type: PathPrefix value: {{ path_prefix }} headers: - name: {{ header_name }} value: {{ header_value }} backendRefs: - name: {{ test_service }} port: {{ port }} - matches: - path: type: PathPrefix value: {{ path_prefix }} backendRefs: - name: {{ stable_service }} port: {{ port }}

Decision Framework for Your Skill

QuestionIf YesIf No
Need instant switch?Blue-greenCanary
Testing specific users?A/B testingCanary
Database migration?Blue-greenCanary
Gradual risk reduction?CanaryBlue-green

Rollback Guidance

Your skill should include rollback commands:

bash
# Canary rollback: set canary weight to 0 kubectl patch httproute {{ route }} --type=merge -p '{"spec":...}' # Blue-green rollback: swap weights # From green (100) back to blue (100)

Try With AI

Generate Canary Configuration

Ask your traffic-engineer skill:

text
Using my traffic-engineer skill, generate a canary deployment configuration for my Task API: - Stable service: task-api-stable (port 8000) - Canary service: task-api-canary (port 8000) - Initial split: 95% stable, 5% canary - Namespace: task-api - Gateway: task-api-gateway - Path prefix: /api/

What you're learning: AI generates traffic splitting configurations. Review the output—are weights correct? Is the path matching appropriate? Do the service names match your request?

Evaluate and Extend

Check AI's output:

  • Do weights sum correctly for the intended split?
  • Is the HTTPRoute in the correct namespace?
  • Are parentRefs pointing to your gateway?

If you need progressive stages, ask:

text
Now generate the HTTPRoute for stage 2: 75% stable, 25% canary.

What you're learning: AI adapts configurations through iteration. Compare the stage 1 and stage 2 outputs—only weights should change.

Add A/B Testing Layer

Extend for premium user protection:

text
Add a rule BEFORE the canary split that routes x-api-tier: premium users to task-api-stable with 100% weight. Premium users should never see the canary.

What you're learning: Rule ordering matters. AI should place the premium rule first so it matches before the general canary rule.

Generate Rollback Command

Ask for operational commands:

text
Generate the kubectl patch command to rollback this canary to 0% traffic while keeping 100% on stable.

What you're learning: AI generates operational commands, not just YAML. Review the patch syntax—is it correct for your HTTPRoute?

Safety Note

Traffic splitting affects real users in production. Always test configurations with kubectl apply --dry-run=client first. Start canary deployments with small percentages (1-5%) and monitor error rates before increasing. Keep rollback commands ready—you may need them at 3 AM.