USMAN’S INSIGHTS
AI ARCHITECT
  • Home
  • About
  • Thought Leadership
  • Book
Press / Contact
USMAN’S INSIGHTS
AI ARCHITECT
⌘F
HomeBook
HomeBookWhy Your App Crashes Before the Database Is Ready
Previous Chapter
Sync Strategies and Policies
Next Chapter
ApplicationSets Scaling Deployments
AI NOTICE: This is the table of contents for the SPECIFIC CHAPTER only. It is NOT the global sidebar. For all chapters, look at the main navigation.

On this page

41 sections

Progress0%
1 / 41

Muhammad Usman Akbar Entity Profile

Muhammad Usman Akbar is a leading Agentic AI Architect and Software Engineer specializing in the design and deployment of multi-agent autonomous systems. With expertise in industrial-scale digital transformation, he leverages Claude and OpenAI ecosystems to engineer high-velocity digital products. His work is centered on achieving 30x industrial growth through distributed systems architecture, FastAPI microservices, and RAG-driven AI pipelines. Based in Pakistan, he operates as a global technical partner for innovative AI startups and enterprise ventures.

USMAN’S INSIGHTS
AI ARCHITECT

Transforming businesses into autonomous AI ecosystems. Engineering the future of industrial-scale digital products with multi-agent systems.

30X Growth
AI-First
Innovation

Navigation

  • Home
  • Book
  • About
  • Contact
Let's Collaborate

Have a Project in Mind?

Let's build something extraordinary together. Transform your vision into autonomous AI reality.

Start Your Transformation

© 2026 Muhammad Usman Akbar. All rights reserved.

Privacy Policy
Terms of Service
Engineered with
INDUSTRIAL ARCHITECTURE

Sync Waves and Resource Hooks

You've learned how to manage sync policies in Chapter 9. Now you face a critical challenge: deployment order matters. Your PostgreSQL database migration must run before your FastAPI application deploys. Your health checks must pass before marking the deployment as healthy. Your rollback must run cleanup jobs when deployment fails.

This is where sync waves and resource hooks solve real problems. Waves order resource creation sequentially. Hooks define what happens at critical sync points: before, during, and after deployment.

The Order Problem

Consider this scenario. You're deploying version 2 of your agent with a database schema change:

  1. Your CI pipeline builds the new container
  2. ArgoCD syncs the manifests
  3. ArgoCD creates: ConfigMap, Secret, Deployment, Service all at once
  4. The Deployment starts immediately without migrations
  5. The container crashes (no database columns it expects)

Without hooks, your application fails because migrations never ran. With hooks:

  1. CI pipeline builds the new container
  2. ArgoCD syncs the manifests
  3. ArgoCD creates Secret, ConfigMap (wave 0)
  4. ArgoCD runs migration Job as PreSync hook (before sync)
  5. ArgoCD creates Deployment, Service (wave 1)
  6. ArgoCD verifies readiness (hook with HookSucceeded deletion policy)

The difference is simple: waves group resources by creation order. Hooks define executable steps around those waves.

Sync Waves Explained

A sync wave is an annotation that tells ArgoCD: "Create this resource in this order, then move to the next wave."

Wave Numbers and Ordering

Waves execute in ascending numerical order:

  • Wave -1 → Created first (infrastructure preparation)
  • Wave 0 → Created second (main infrastructure: config, secrets, volumes)
  • Wave 1 → Created third (applications: deployments, services)
  • Wave 2 → Created fourth (post-deployment: monitoring, cleanup)

Key principle: ArgoCD waits for each wave to fully sync and stabilize before moving to the next wave. A resource in wave 1 cannot start until all wave 0 resources are healthy.

Annotating Resources for Waves

The annotation syntax is simple:

yaml
metadata: annotations: argocd.argoproj.io/sync-wave: "0"

Example: ConfigMap and Secret in Wave 0

yaml
# Config must exist before app starts apiVersion: v1 kind: ConfigMap metadata: name: agent-config annotations: argocd.argoproj.io/sync-wave: "0" data: agent_model: gpt-4 max_tokens: "4096" --- # Secrets in same wave as config apiVersion: v1 kind: Secret metadata: name: agent-secrets annotations: argocd.argoproj.io/sync-wave: "0" type: Opaque data: api_key: YWJjMTIz # base64 encoded

Output:

text
Wave 0 resources created and stabilized - ConfigMap agent-config: Ready - Secret agent-secrets: Ready

Example: Deployment in Wave 1

yaml
# App deployment waits for wave 0 apiVersion: apps/v1 kind: Deployment metadata: name: agent-api annotations: argocd.argoproj.io/sync-wave: "1" spec: replicas: 1 selector: matchLabels: app: agent-api template: metadata: labels: app: agent-api spec: containers: - name: api image: myregistry.azurecr.io/agent-api:v2.1.0 ports: - containerPort: 8000 envFrom: - configMapRef: name: agent-config - secretRef: name: agent-secrets

Output:

text
Wave 0 complete. Proceeding to Wave 1. - Deployment agent-api: Creating replicas (waiting for wave 0) - Service agent-api: Created Pod: agent-api-xxxxx ready

Resource Hooks: PreSync, PostSync, SyncFail

A resource hook is a Kubernetes Job or Pod that runs at specific points in the sync lifecycle. Unlike regular resources, hooks don't stay running—they execute and exit.

Hook Types and When to Use Them

Hook TypeWhenExample
PreSyncBefore any resources syncDatabase migrations, data preparation
SyncDuring normal resource creationCustom initialization (rarely used)
PostSyncAfter sync completes successfullyNotifications, smoke tests, warmup
SyncFailWhen sync failsAlerting, rollback notifications
PostDeleteWhen ArgoCD deletes resourcesCleanup, deregistration

Hook Deletion Policy

After a hook finishes, what happens to the Job/Pod?

yaml
argocd.argoproj.io/hook-deletion-policy: [Hook Succeeded|Hook Failed|Before Hook Creation]
PolicyDeletes WhenUse Case
HookSucceededHook succeedsOne-time jobs (migrations)
HookFailedHook failsKeep failed jobs for debugging
BeforeHookCreationBefore hook runsCleanup old instances (prevent duplicates)

PreSync Hooks: Running Database Migrations

A PreSync hook runs before any resources sync. This is where you run database migrations.

Complete Database Migration Example

yaml
# Service account for migration job apiVersion: v1 kind: ServiceAccount metadata: name: db-migrate namespace: agents --- # Role for migration job apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: db-migrate namespace: agents rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- # Bind role to service account apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: db-migrate namespace: agents subjects: - kind: ServiceAccount name: db-migrate namespace: agents roleRef: kind: Role name: db-migrate apiGroup: rbac.authorization.k8s.io --- # PreSync hook: Database migration job apiVersion: batch/v1 kind: Job metadata: name: db-migration namespace: agents annotations: argocd.argoproj.io/hook: PreSync argocd.argoproj.io/hook-deletion-policy: HookSucceeded argocd.argoproj.io/sync-wave: "-1" spec: backoffLimit: 1 template: spec: serviceAccountName: db-migrate restartPolicy: Never containers: - name: migrate image: myregistry.azurecr.io/agent-api:v2.1.0 command: - sh - -c - | timeout 60 bash -c 'until python -c "import psycopg2; psycopg2.connect(host='$DB_HOST',user='$DB_USER',password='$DB_PASSWORD',dbname='$DB_NAME')" 2>/dev/null; do sleep 2; done' cd /app pip install -q alembic psycopg2-binary alembic upgrade head python -c " import psycopg2 conn = psycopg2.connect(host='$DB_HOST',user='$DB_USER',password='$DB_PASSWORD',dbname='$DB_NAME') cursor = conn.cursor() cursor.execute(\"SELECT version FROM alembic_version;\") version = cursor.fetchone() print(f'✓ Migration completed. Version: {version[0]}') cursor.close() conn.close() " env: - name: DB_HOST valueFrom: secretKeyRef: name: db-secrets key: host - name: DB_USER valueFrom: secretKeyRef: name: db-secrets key: username - name: DB_PASSWORD valueFrom: secretKeyRef: name: db-secrets key: password - name: DB_NAME valueFrom: secretKeyRef: name: db-secrets key: database

Output (on successful migration):

text
PreSync: Running database migration Job: db-migration created - Waiting for migration to complete... - Migration command: alembic upgrade head - ✓ Migration completed. Version: 5 Job deleted (HookSucceeded policy) PreSync complete. Proceeding to sync.

Output (on migration failure):

text
PreSync: Running database migration Job: db-migration created - Waiting for migration to complete... - ERROR: Cannot find column 'agent_id' in agents table Job NOT deleted (job failed, waiting for manual investigation) Sync FAILED - PreSync hook failed

PostSync Hooks: Notifications and Smoke Tests

A PostSync hook runs after sync completes successfully. Use this for:

  • Notifications (Slack, email, webhook)
  • Smoke tests (basic health checks)
  • Warming up caches
  • Database reseeding

Example: Slack Notification on Deployment

yaml
# Secret containing webhook URL apiVersion: v1 kind: Secret metadata: name: slack-webhook namespace: agents type: Opaque stringData: webhook-url: https://hooks.slack.com/services/YOUR/WEBHOOK/URL --- # PostSync hook: Send Slack notification apiVersion: batch/v1 kind: Job metadata: name: notify-deployment namespace: agents annotations: argocd.argoproj.io/hook: PostSync argocd.argoproj.io/hook-deletion-policy: HookSucceeded argocd.argoproj.io/sync-wave: "2" spec: backoffLimit: 1 template: spec: restartPolicy: Never containers: - name: notify image: curlimages/curl:latest command: - sh - -c - | WEBHOOK_URL=$(cat /var/run/secrets/slack/webhook-url) DEPLOYMENT_IMAGE=$(kubectl get deployment agent-api -n agents -o jsonpath='{.spec.template.spec.containers[0].image}') REPLICA_READY=$(kubectl get deployment agent-api -n agents -o jsonpath='{.status.readyReplicas}') curl -X POST "$WEBHOOK_URL" \ -H 'Content-Type: application/json' \ -d "{ \"text\": \"Agent API Deployment Complete\", \"blocks\": [ { \"type\": \"section\", \"text\": { \"type\": \"mrkdwn\", \"text\": \"*Agent API Deployed Successfully*\n*Image:* \`$DEPLOYMENT_IMAGE\`\n*Ready Replicas:* $REPLICA_READY\" } } ] }" volumeMounts: - name: slack-secret mountPath: /var/run/secrets/slack readOnly: true volumes: - name: slack-secret secret: secretName: slack-webhook items: - key: webhook-url path: webhook-url

Output:

text
PostSync: Running notifications Job: notify-deployment created - Sending Slack notification... - curl: Slack API responded with 200 OK ✓ Deployment notification sent Job deleted (HookSucceeded policy) Sync complete.

SyncFail Hooks: Alerting on Failure

A SyncFail hook runs when the sync process fails. Use this for alerting and rollback notifications.

yaml
apiVersion: batch/v1 kind: Job metadata: name: sync-failed-alert namespace: agents annotations: argocd.argoproj.io/hook: SyncFail argocd.argoproj.io/hook-deletion-policy: HookFailed argocd.argoproj.io/sync-wave: "3" spec: backoffLimit: 0 template: spec: restartPolicy: Never containers: - name: alert image: curlimages/curl:latest command: - sh - -c - | WEBHOOK_URL=$(cat /var/run/secrets/slack/webhook-url) curl -X POST "$WEBHOOK_URL" \ -H 'Content-Type: application/json' \ -d "{ \"text\": \"ALERT: Agent Deployment Failed\", \"blocks\": [ { \"type\": \"section\", \"text\": { \"type\": \"mrkdwn\", \"text\": \":warning: *Deployment Failed*\nCheck ArgoCD UI for details. Cluster state may be inconsistent.\" } } ] }" volumeMounts: - name: slack-secret mountPath: /var/run/secrets/slack readOnly: true volumes: - name: slack-secret secret: secretName: slack-webhook

Output (when sync fails):

text
Sync FAILED: Invalid image reference SyncFail: Running alert hook Job: sync-failed-alert created - Sending failure alert to Slack... - curl: Slack API responded with 200 OK ✓ Failure alert sent Cluster is in UNKNOWN health state (check manually)

Hook Deletion Policies in Detail

HookSucceeded (Default for One-Time Jobs)

yaml
argocd.argoproj.io/hook-deletion-policy: Hook Succeeded

Deletes the Job immediately after it succeeds. Use this for database migrations and one-time setup jobs.

HookFailed (Keep Failed Jobs for Debugging)

yaml
argocd.argoproj.io/hook-deletion-policy: Hook Failed

Deletes the Job only if it fails. If it succeeds, the Job persists. Use this to keep successful runs for audit trails.

BeforeHookCreation (Cleanup Old Instances)

yaml
argocd.argoproj.io/hook-deletion-policy: Before Hook Creation

Deletes the previous hook instance before creating a new one. Use this for notification hooks that run on every sync.

Complete Example: Multi-Wave Deployment with Hooks

Here's a realistic example combining waves and hooks:

yaml
# Wave -1: Prepare (migrations) apiVersion: batch/v1 kind: Job metadata: name: db-migration namespace: agents annotations: argocd.argoproj.io/hook: PreSync argocd.argoproj.io/hook-deletion-policy: HookSucceeded argocd.argoproj.io/sync-wave: "-1" spec: template: spec: restartPolicy: Never containers: - name: migrate image: myregistry.azurecr.io/agent-api:v2.1.0 command: ["alembic", "upgrade", "head"] --- # Wave 0: Infrastructure (config, secrets) apiVersion: v1 kind: ConfigMap metadata: name: agent-config annotations: argocd.argoproj.io/sync-wave: "0" data: log_level: INFO --- # Wave 1: Application apiVersion: apps/v1 kind: Deployment metadata: name: agent-api annotations: argocd.argoproj.io/sync-wave: "1" spec: replicas: 2 template: spec: containers: - name: api image: myregistry.azurecr.io/agent-api:v2.1.0 --- # Wave 2: Post-deployment smoke tests apiVersion: batch/v1 kind: Job metadata: name: smoke-tests namespace: agents annotations: argocd.argoproj.io/hook: PostSync argocd.argoproj.io/hook-deletion-policy: HookSucceeded argocd.argoproj.io/sync-wave: "2" spec: template: spec: restartPolicy: Never containers: - name: test image: curlimages/curl:latest command: ["curl", "-f", "http://agent-api:8000/health"]

Output (complete sync with waves):

text
ArgoCD Sync Started Wave -1: PreSync Preparation Job: db-migration running... ✓ Wave 0: Infrastructure ConfigMap: agent-config created... ✓ Wave 1: Application Deployment: agent-api created (waiting for replicas)... ✓ Wave 2: Post-Sync Verification Job: smoke-tests running... ✓ Sync SUCCESSFUL Application Status: Healthy

Troubleshooting Failed Hooks

When hooks fail, ArgoCD stops the sync. Here's how to diagnose:

Check Hook Job Logs

bash
# List jobs in the namespace kubectl get jobs -n agents # View logs from failed migration kubectl logs -n agents job/db-migration # Describe the job to see event messages kubectl describe job -n agents db-migration

Output:

text
NAME COMPLETIONS DURATION AGE db-migration 0/1 5s 5s LOGS: ERROR: Cannot connect to database at 192.168.1.10:5432 Connection refused - check DATABASE_URL environment variable EVENTS: Normal Created 5s batch Created pod: db-migration-xxxxx Warning Failed 1s batch Pod failed (exit code 1)

Common Issues

IssueCauseFix
Hook never runsWave number too highUse sync-wave: "-1" for PreSync critical jobs
Hook runs but failsMissing sensitive dataVerify secretKeyRef or ConfigMap exist in Wave 0
Job persistsWrong deletion policyChange to HookSucceeded for one-time migrations
Multiple jobs hangingMissing cleanupUse BeforeHookCreation to delete legacy instances

Reflect on Your Skill

You built a gitops-deployment skill in Chapter 0. Test and improve it based on what you learned.

Test Your Skill

bash
Using my gitops-deployment skill, generate manifests with sync waves to deploy a database before an application. Does my skill use annotations like argocd.argoproj.io/sync-wave correctly?

Identify Gaps

Ask yourself:

  • Did my skill include resource hooks for PreSync and PostSync operations?
  • Did it handle database migrations using hooks?

Improve Your Skill

If you found gaps:

bash
My gitops-deployment skill doesn't include hook configurations. Update it to add Pre Sync hooks for schema migrations and Post Sync hooks for smoke tests.