Secrets Management for GitOps

You've deployed applications with ArgoCD, configured sync policies, and mastered ApplicationSets. But you haven't stored any secrets in Git yet—and for good reason. Committing API keys, database passwords, or authentication tokens to version control is a security breach waiting to happen.

GitOps requires everything to be in Git, but secrets are the exception: they must never be stored in plaintext in version control. This chapter teaches you patterns for handling secrets safely while maintaining the GitOps principle of "Git as truth."

By the end of this chapter, you'll understand:

1. Why plaintext secrets in Git are dangerous
2. How to encrypt secrets using Sealed Secrets (Bitnami approach)
3. How to sync secrets from external stores using External Secrets Operator
4. When to use HashiCorp Vault integration through ArgoCD plugins
5. Best practices for API key rotation and access control
6. How to deploy applications that need secrets without committing those secrets

The Problem: Secrets in Git

Imagine your Python FastAPI agent needs:

• OpenAI API key (OPENAI_API_KEY=sk-...)
• Database password (DB_PASSWORD=secure123!)
• JWT signing key (JWT_SECRET=abc123xyz...)

In non-GitOps workflows, you might:

1. Store secrets in a .env file (never commit to Git)
2. Pass them via environment variables at deployment time
3. Manage them manually in each environment

But GitOps says everything should be in Git—including the specification of what secrets your application needs. This creates a tension:

Requirement 1: Everything must be versioned in Git (GitOps principle) Requirement 2: Secrets must never be in plaintext in Git (security principle)

These aren't contradictory if you encrypt secrets at rest (in Git) while keeping them decrypted at runtime (in the cluster).

Why Plaintext Secrets in Git Are Dangerous

1. Permanent history: Once committed, secrets exist in Git history forever (even if you delete them)
2. Broad access: Everyone with repository access can read plaintext secrets
3. Audit trail: No way to know who accessed or rotated secrets
4. Accidental exposure: Easy to commit .env files, API keys in comments, etc.

Even private repositories are risky—developers accidentally grant too many permissions, contractors get access, or repositories are acquired in acquisitions where permissions aren't immediately revoked.

Sealed Secrets: Encrypt Secrets with the Cluster Key

Sealed Secrets (by Bitnami) is the simplest approach for Kubernetes-native secret management.

How Sealed Secrets Works

1. Cluster generates a key pair during Sealed Secrets installation
2. You encrypt plaintext secrets using the public key
3. Encrypted secrets go into Git (they look like gibberish)
4. ArgoCD applies the encrypted YAML to the cluster
5. The controller automatically decrypts using the private key
6. Your application reads the plaintext Secret from the cluster

The private key never leaves the cluster, so only your cluster can decrypt the secrets.

Install Sealed Secrets

First, install the Sealed Secrets controller:

Output:

Verify it's running:

Output:

Create an Encrypted Secret

You need the kubeseal CLI tool to encrypt secrets locally:

Output (after install):

Now, encrypt a secret. First, create a plaintext Kubernetes Secret manifest:

Encrypt it:

Output:

Examine the encrypted output:

Output:

The values are unreadable. This is safe to commit to Git.

Commit and Deploy

Commit the sealed secret to Git:

Output:

Create an ArgoCD Application that includes this sealed secret:

Output (after sync):

Your application now has access to plaintext secrets without them being stored in Git:

Output:

The secret is decrypted in the cluster. Your Deployment mounts it:

Your container sees the plaintext environment variables.

External Secrets Operator: Sync from External Stores

Sealed Secrets work well for small, static secrets. But if you use AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault, External Secrets Operator keeps those systems as the single source of truth and syncs them into Kubernetes.

How External Secrets Works

1. You store secrets in an external vault (AWS Secrets Manager, Vault, etc.)
2. External Secrets creates a Kubernetes Secret that mirrors the vault
3. ArgoCD manages the ExternalSecret CRD (which goes in Git)
4. The controller automatically syncs updates from vault to cluster
5. Your application reads the mirrored Secret just like before

Install External Secrets Operator

Output:

Create a SecretStore (Vault Example)

A SecretStore defines how to connect to your external vault:

Output (after apply):

Create an ExternalSecret

An ExternalSecret specifies which vault secrets to sync:

Apply it:

Output:

The External Secrets controller immediately syncs from Vault. Check the Secret:

Output:

Your Deployment mounts it exactly like a Sealed Secret:

The difference: secrets now stay synchronized with Vault. If you rotate a secret in Vault, the Kubernetes Secret updates automatically within 1 hour (or immediately with a webhook trigger).

When to Use Which Approach

For this course, Sealed Secrets is sufficient. For production deployments in enterprises, External Secrets + Vault is standard.

Best Practices for API Key Management

1. Never Commit Secrets

Create a .gitignore rule:

2. Rotate Secrets Regularly

For OpenAI, Anthropic, and other provider keys:

1. Set an expiration date in your notes (e.g., rotate every 90 days)
2. Create a new key in the provider dashboard
3. Update the Sealed Secret or Vault
4. Delete the old key from the provider
5. Restart the deployment so containers pick up the new key

Example rotation for an OpenAI key:

Output:

3. Limit Secret Scope

Use Kubernetes namespace isolation:

• production namespace: Sealed secrets for production API keys
• staging namespace: Sealed secrets for staging API keys (different keys)
• development namespace: Sealed secrets for dev API keys (lower security)

Each namespace has its own Sealed Secret, encrypted with the cluster key, but never mixed.

4. Audit Secret Access

For production deployments, use Vault with audit logging:

Vault logs every secret access. Check the logs:

Output:

5. Separate Secrets by Environment

Production secrets should NOT be the same as development secrets:

If a development key is compromised, production remains secure.

Complete Example: FastAPI Agent with Secrets

Here's a full example deploying your agent with Sealed Secrets:

Step 1: Create plaintext secret

Step 2: Encrypt it

Output:

Step 3: Commit encrypted version

Output:

Step 4: Create ArgoCD Application

Step 5: Create Deployment that uses the secret

Apply everything:

Output:

Comparison: Secret Patterns

Try With AI

Setup: You have an OpenAI API key and a PostgreSQL connection string that need to be deployed with your FastAPI agent.

Part 1: Initial Request

Ask AI: "I need to securely store my OpenAI API key and PostgreSQL connection string in Kubernetes. I want to keep them encrypted in Git. Should I use Sealed Secrets or External Secrets Operator? What are the tradeoffs?"

Part 2: Critical Evaluation

Review AI's response. Ask yourself:

• Does AI explain when each approach is appropriate?
• Which approach seems simpler for your current setup?
• What assumptions did AI make about your infrastructure?

Part 3: Share Your Constraints

Tell AI your constraints: "We're learning Gitops in a Minikube environment, so we need something that doesn't require external services. What's the minimal setup for Sealed Secrets?"

Part 4: Refinement

Ask AI to help you: "Generate the kubeseal encryption steps and the YAML manifest for my FastAPI agent Deployment that reads these secrets."

Part 5: Final Check

Compare your result:

• Can you trace how secrets flow from plaintext → kubeseal → encrypted YAML → Kubernetes Secret → environment variables?
• Would this approach work for rotating your OpenAI key in 90 days?
• Could you adapt this for External Secrets later?

---

Reflect on Your Skill

You built a gitops-deployment skill in Chapter 0. Test and improve it based on what you learned.

Test Your Skill

Identify Gaps

Ask yourself:

• Did my skill include External Secrets Operator configuration for Vault?
• Did it handle secret rotation strategies and best practices?

Improve Your Skill

If you found gaps: