Production Secrets & Configuration

In the previous Chapter, you deployed your Task API to DOKS with a basic configuration. But production services need more than hardcoded values. Your AI agent needs database credentials, API keys for external services, and environment-specific settings that change between staging and production.

Storing these directly in Deployment manifests creates security risks and operational headaches. Every configuration change requires a new container image or manifest edit. Worse, secrets end up in version control where anyone with repository access can read them.

Kubernetes solves this with two primitives: Secrets for sensitive data and ConfigMaps for non-sensitive configuration. This lesson teaches you to use both effectively, including the critical image pull secret that authenticates your cluster to pull images from GitHub Container Registry.

---

What You'll Learn

This lesson covers 5 concepts organized into three groups:

Prerequisites:

• Previous Chapter completed (Task API deployed to DOKS)
• kubectl configured and connected to your cluster
• GitHub personal access token with read:packages scope

Time Estimate: 25 minutes

• Concepts: 15 minutes
• Exercises: 8 minutes
• Try With AI: 2 minutes

---

Concept 1: Kubernetes Secrets

A Kubernetes Secret stores sensitive data like passwords, tokens, and certificates. Unlike ConfigMaps, Secrets are base64-encoded (not encrypted by default, but protected from casual viewing) and can be restricted with RBAC.

Creating a Secret from Literal Values

Output:

Viewing Secret Structure (Not Values)

Output:

The values are base64-encoded, not encrypted. Anyone with kubectl get secret access can decode them:

Output:

Security Note: Base64 is encoding, not encryption. For true at-rest encryption, enable Kubernetes secrets encryption or use a secrets manager like HashiCorp Vault or cloud provider secrets (AWS Secrets Manager, GCP Secret Manager).

---

Concept 2: Image Pull Secrets for Private Registries

Your Task API container lives in GitHub Container Registry (GHCR), which requires authentication. Without an image pull secret, Kubernetes cannot pull your private images.

Creating the Image Pull Secret

Output:

Where:

• $GITHUB_USERNAME is your GitHub username
• $GITHUB_TOKEN is a personal access token with read:packages scope

Verifying the Secret

Output:

Referencing in Deployment

Add imagePullSecrets to your Deployment spec:

Without this reference, pod creation fails with ImagePullBackOff:

Output (without secret):

---

Concept 3: ConfigMaps for Non-Sensitive Configuration

ConfigMaps store configuration data that isn't sensitive: feature flags, service URLs, logging levels. Unlike Secrets, ConfigMaps appear in plain text and are easier to inspect.

Creating a ConfigMap

Output:

Creating ConfigMap from a File

For complex configuration, use a file:

Output:

Viewing ConfigMap Contents

Output:

---

Concept 4: Configuration Hierarchy Pattern

Production applications need layered configuration: defaults that work everywhere, environment overrides for staging vs production, and secrets that are never committed to version control.

The Three-Layer Pattern

Environment variables from higher layers override lower layers.

Implementing in Deployment

Output (kubectl describe pod):

Why This Order Matters

If LOG_LEVEL appears in both ConfigMap and as a hardcoded env var, the ConfigMap value wins because envFrom is processed after env. Secrets in envFrom are processed last, so they override everything.

---

Concept 5: Verifying Secret and ConfigMap Injection

Deploying configuration is only half the battle. You must verify that containers receive the expected values.

Check Pod Environment Variables

Output:

Verify All Environment Variables

Output:

Troubleshooting Missing Variables

If expected variables are missing, check:

1. ConfigMap/Secret exists:

   bash

   WrapCopy

   kubectl get configmap task-api-config kubectl get secret task-api-secrets

2. Deployment references are correct:

   bash

   WrapCopy

   kubectl get deployment task-api -o yaml | grep -A10 env

From ```

3. Pod events for errors:

   bash

   WrapCopy

   kubectl describe pod -l app=task-api | grep -A20 Events

Output (missing ConfigMap):

---

Common Mistakes

1. Forgetting imagePullSecrets for Private Registries

Fix: Always include imagePullSecrets when using private registries:

2. Putting Secrets in ConfigMaps

Fix: Use Secrets for sensitive data:

3. Using Wrong Secret Type for Docker Registry

Fix: Use docker-registry type:

4. Forgetting to Restart Pods After Secret Update

Updating a Secret doesn't automatically restart pods:

Fix: Restart the deployment:

---

Exercise 6.1: Create Application Secrets

Create secrets for your Task API with database credentials and an API key.

Instructions:

1. Create the secret:

   bash

   WrapCopy

   kubectl create secret generic task-api-secrets \ --from-literal=DATABASE_PASSWORD=my-secure-password \ --from-literal=API_KEY=your-test-api-key

2. Verify creation:

   bash

   WrapCopy

   kubectl get secret task-api-secrets

Expected output: Secret listed with Opaque type and 2 data items.

---

Exercise 6.2: Create Image Pull Secret for GHCR

Set up authentication for GitHub Container Registry.

Instructions:

1. Create a GitHub personal access token with read:packages scope at https://github.com/settings/tokens

2. Create the secret (replace with your values):

   bash

   WrapCopy

   export GITHUB_USERNAME=your-username export GITHUB_TOKEN=ghp_your_token_here kubectl create secret docker-registry ghcr-secret \ --docker-server=ghcr.io \ --docker-username=$GITHUB_USERNAME \ --docker-password=$GITHUB_TOKEN

3. Verify:

   bash

   WrapCopy

   kubectl get secret ghcr-secret -o jsonpath='{.type}'

Expected output: kubernetes.io/dockerconfigjson

---

Exercise 6.3: Create ConfigMap and Deploy with Full Configuration

Create a ConfigMap and update your deployment to use the three-layer configuration pattern.

Instructions:

1. Create ConfigMap:

   bash

   WrapCopy

   kubectl create configmap task-api-config \ --from-literal=LOG_LEVEL=info \ --from-literal=ENABLE_METRICS=true

2. Apply updated deployment with all references (save as task-api-full.yaml):

   yaml

   WrapCopy

   apiVersion: apps/v1 kind: Deployment metadata: name: task-api spec: replicas: 1 selector: matchLabels: app: task-api template: metadata: labels: app: task-api spec: imagePullSecrets: - name: ghcr-secret containers: - name: task-api image: ghcr.io/your-username/task-api:latest ports: - containerPort: 8000 envFrom: - configMapRef: name: task-api-config - secretRef: name: task-api-secrets

3. Apply and verify:

   bash

   WrapCopy

   kubectl apply -f task-api-full.yaml kubectl exec -it deployment/task-api -- printenv | grep -E "(LOG_LEVEL|DATABASE)"

Expected output: Both ConfigMap and Secret values visible in pod environment.

---

Try With AI

Part 1: Initial Request

Your Task API needs additional secrets for a new feature: Redis connection string and a webhook signing secret. Ask AI for help:

"I need to add Redis and webhook secrets to my Kubernetes deployment. Create kubectl commands to add REDIS_URL and WEBHOOK_SECRET to my existing task-api-secrets."

What you're learning: How to extend existing secrets without recreating them from scratch.

Part 2: Critical Evaluation

Review AI's response. Ask yourself:

• Did it use kubectl create secret with --dry-run=client -o yaml | kubectl apply -f - to update existing secrets?
• Or did it suggest deleting and recreating (which causes downtime)?
• Does the approach preserve existing secret values?

Part 3: Configuration Hierarchy

Ask AI about a more advanced pattern:

"I want to use external-secrets operator to pull secrets from AWS Secrets Manager instead of creating them with kubectl. Is this better than what I'm doing now?"

What you're learning: The difference between simple kubectl-based secrets (good for learning) and enterprise patterns (external-secrets operator).

Part 4: Validation

Verify the secrets work by checking your pod:

If the variable appears with the expected value, your secret management is working.

---

Reflect on Your Skill

You built a multi-cloud-deployer skill in an earlier Chapter. Test and improve it based on what you learned.

Test Your Skill

Identify Gaps

Ask yourself:

• Does my skill cover creating Kubernetes secrets with kubectl?
• Does it include image pull secrets for private registries like GHCR?
• Does it demonstrate the three-layer configuration pattern?
• Does it include verification steps using kubectl exec?

Improve Your Skill

If you found gaps: