Secrets Management

In January 2023, CircleCI disclosed a security incident affecting thousands of customer projects. Attackers had accessed stored secrets—API keys, OAuth tokens, SSH keys—because they were stored as environment variables visible in build logs and process listings. The incident affected companies from startups to enterprises, requiring emergency credential rotation across their entire infrastructure.

Your Task API needs database credentials, API keys, and service tokens. How you store and consume these secrets determines whether a container escape becomes a minor incident or a catastrophic breach. Environment variables—the most common approach—are also the most dangerous because they appear in process listings, crash dumps, and debugging output.

This lesson teaches you the secure pattern: volume mounts for secret consumption. You'll also understand where Kubernetes Secrets fit in the broader secrets hierarchy, from development conveniences to production-grade external secret managers.

---

The Problem with Environment Variables

Before learning the secure pattern, understand why the common pattern fails.

Why Teams Use Environment Variables

Environment variables are convenient:

The secret value becomes accessible via $DATABASE_PASSWORD in your application. Simple, readable, supported by every framework. But fundamentally insecure.

Environment Variable Attack Surface

A single debugging command can expose your production database password:

Output:

---

Volume Mounts: The Secure Alternative

Volume mounts project secrets as files inside the container. This approach eliminates most environment variable attack vectors.

How Volume Mounts Work

Your application reads credentials from files instead of environment variables. This simple change eliminates five of the six exposure vectors above.

Security Comparison

Volume mounts keep secrets out of the execution environment entirely. They exist only as files with restricted permissions.

---

Creating Kubernetes Secrets

Kubernetes Secrets store sensitive data as key-value pairs. Values are base64-encoded (NOT encrypted) in etcd.

Method 1: Create from Literals

For individual values, use --from-literal:

Output:

Verify the Secret exists:

Output:

The values are base64-encoded. Decode to verify:

Output:

Base64 is NOT Encryption

Base64 encoding is reversible by anyone. It provides zero security—only encoding convenience. Anyone with kubectl get secret access can decode your credentials instantly. Kubernetes stores Secrets in etcd, which should have encryption-at-rest enabled in production.

Method 2: Create from Files

For certificates, keys, or complex credentials, use --from-file:

Output:

The --from-file approach is essential for multi-line values like TLS certificates:

Output:

---

Consuming Secrets via Volume Mount

Mount your Secret as a volume in the pod spec:

Key elements:

Apply the deployment:

Output:

Verify Volume Mount

Check that secrets are mounted correctly:

Output:

Read the secret value:

Output:

Application Code Pattern

Your application reads credentials from files:

---

Mounting Specific Keys

You can mount individual keys to specific file paths:

This is useful when your application expects specific filenames.

---

The Secrets Management Hierarchy

Kubernetes Secrets are the foundation, but production environments need more:

When to Use Each Level

Sealed Secrets Overview

Sealed Secrets (Bitnami) encrypts secrets with a cluster-specific key. The encrypted SealedSecret resource is safe to commit to git:

Only the controller running in your cluster can decrypt it. Different clusters have different keys, so secrets stay cluster-specific.

External Secrets Operator Overview

External Secrets Operator (ESO) syncs secrets from external stores into Kubernetes:

ESO creates a standard Kubernetes Secret that your pods consume via volume mount—the same pattern you learned above. The difference is where the source of truth lives (external vault) and that ESO handles rotation automatically.

---

Task API Secret Configuration

Here's the complete Secret and volume mount configuration for Task API. Create task-api-secrets.yaml:

Apply the configuration:

Output:

---

Reflect on Your Skill

Test your cloud-security skill against secrets management:

1. Does your skill default to volume mounts instead of environment variables?
2. Does your skill set defaultMode: 0400 for restrictive file permissions?
3. Does your skill include the readOnly: true flag on volumeMounts?
4. Does your skill warn about base64 encoding vs encryption?
5. Does your skill mention when to escalate to Sealed Secrets or ESO?

If any answers are "no," update your skill with the patterns from this lesson.

---

Try With AI

Practice secrets management patterns and troubleshooting.

Prompt 1:

What you're learning: Common volume mount debugging. The issue could be namespace mismatch (Secret in different namespace than pod), Secret key names don't match expected paths, or the Secret was created after the pod started. The skill should walk through verification steps: checking Secret exists in same namespace, verifying key names, and restarting the pod if Secret was created late.

Prompt 2:

What you're learning: Migration pattern from environment variables to volume mounts. Notice how the conversion requires both volumeMount entries and volume definitions. The application code also needs updating to read from files instead of environment variables.

Prompt 3:

What you're learning: How to select the appropriate level in the secrets hierarchy based on requirements. Rotation and audit requirements point to External Secrets Operator with a backend like HashiCorp Vault. Multi-cluster also favors centralized management. The skill should explain why K8s Secrets and Sealed Secrets don't meet these requirements.

Security Reminder

Never commit plaintext secrets to git, even temporarily. Use kubectl create secret imperatively or Sealed Secrets for GitOps workflows. Base64 encoding provides zero security—anyone with cluster access can decode your secrets instantly.