Visualization with Grafana

Your Prometheus server is collecting thousands of metrics every 30 seconds. The data exists. But when your Task API starts throwing 500 errors at 3 AM, can you answer these questions in under 60 seconds?

• Is the error rate actually increasing, or was it a blip?
• Which endpoint is failing?
• When did the problem start?
• Is it correlated with increased traffic or resource saturation?

Raw PromQL queries won't cut it. You need dashboards that surface the answers instantly—before your users notice and before your on-call engineer has their first coffee.

Grafana transforms your Prometheus metrics into operational intelligence. This lesson teaches you to build dashboards that make metrics actionable: panels for the 4 golden signals, variables for multi-service filtering, and community dashboard imports that save hours of configuration.

---

The 4 Golden Signals: What Every Dashboard Needs

Google's Site Reliability Engineering book defines four golden signals that every service dashboard must display. These aren't arbitrary—they're the minimum information needed to diagnose most production issues:

Your Task API dashboard will include all four signals. When something breaks, you'll look at the dashboard and know within seconds whether it's a latency spike, error surge, traffic overload, or resource exhaustion.

---

Grafana Dashboard Architecture

Before creating panels, understand how Grafana structures dashboards:

Everything in Grafana is ultimately JSON. You can create dashboards through the UI, but the underlying model is a JSON document. This matters because:

1. Version control: Export dashboards as JSON and store in Git
2. Reproducibility: Deploy identical dashboards across environments
3. Automation: Generate dashboards programmatically

---

Creating Your First Dashboard: Task API Golden Signals

Let's build a dashboard for the Task API metrics you instrumented in Lesson 2.

Step 1: Access Grafana

If you installed kube-prometheus-stack via Helm, Grafana is already running:

Output:

Open http://localhost:3000 in your browser. Default credentials are admin / prom-operator (or whatever you set in Helm values).

Step 2: Create a New Dashboard

1. Click the + icon in the left sidebar
2. Select Dashboard
3. Click Add visualization

You'll see an empty panel with a query editor.

Step 3: Build the Latency Panel (P95)

The first golden signal is latency. Configure your panel:

Query (PromQL):

Panel Settings:

• Title: P95 Latency
• Panel type: Time series
• Unit: seconds (s)
• Legend: P95

Click Apply to add the panel to your dashboard.

Output (Visual): The panel displays a line graph showing the 95th percentile latency over time. Spikes indicate periods when 5% of requests took longer than usual.

Step 4: Build the Traffic Panel (Requests/Second)

Query (PromQL):

Panel Settings:

• Title: Request Rate
• Panel type: Time series
• Unit: requests/sec (reqps)
• Legend: Requests/sec

Output (Visual): A line showing request volume over time. Traffic patterns reveal usage spikes (lunch hour, batch jobs) and help contextualize errors.

Step 5: Build the Error Rate Panel

Query (PromQL):

Panel Settings:

• Title: Error Rate
• Panel type: Gauge
• Unit: percent (0-100)
• Thresholds:
  • Green: 0-1
  • Yellow: 1-5
  • Red: 5-100

Output (Visual): A gauge showing current error percentage. Green means healthy; yellow is warning territory; red requires immediate attention.

Step 6: Build the Saturation Panel (CPU)

Query (PromQL):

Panel Settings:

• Title: CPU Saturation
• Panel type: Gauge
• Unit: percent (0-100)
• Thresholds:
  • Green: 0-70
  • Yellow: 70-85
  • Red: 85-100

Output (Visual): A gauge showing CPU utilization as percentage of limits. High saturation (>85%) indicates your pods are resource-constrained and may need scaling.

---

Dashboard Variables: Multi-Service Filtering

Hardcoding namespace="default" in every query limits reusability. Dashboard variables let users filter dynamically.

Creating a Namespace Variable

1. Click the gear icon (Dashboard settings) in the top right
2. Select Variables in the left menu
3. Click Add variable

Configure:

• Name: namespace
• Type: Query
• Data source: Prometheus
• Query: label_values(kube_namespace_labels, namespace)
• Regex: Leave empty (or filter with /(default|production|staging)/)
• Multi-value: Enable
• Include All option: Enable

Output: A dropdown appears at the top of your dashboard. Selecting a namespace filters all panels.

Creating a Service Variable

Add another variable:

• Name: service
• Type: Query
• Data source: Prometheus
• Query: label_values(task_api_requests_total{namespace="$namespace"}, service)

This variable depends on the namespace selection—Grafana refreshes the service list when namespace changes.

Using Variables in Queries

Update your panel queries to use variables:

Now the same dashboard works for any service in any namespace.

---

The Dashboard JSON Model

Everything you built through the UI exists as JSON. Export your dashboard:

1. Click the share icon (top right)
2. Select Export
3. Toggle Export for sharing externally
4. Click Save to file

Here's a simplified version of the Task API Golden Signals dashboard:

Store this JSON in your Git repository under observability/dashboards/task-api-golden-signals.json. Deploy it via ConfigMap with the kube-prometheus-stack's dashboard provisioning.

---

Importing Community Dashboards

Don't reinvent the wheel. Grafana.com hosts thousands of community dashboards. For Kubernetes monitoring, the most popular include:

Import a Dashboard

1. In Grafana, click + → Import
2. Enter dashboard ID: 6417
3. Click Load
4. Select your Prometheus datasource
5. Click Import

Output: The "Kubernetes Pods" dashboard appears with pre-built panels for CPU, memory, network, and filesystem metrics per pod.

Customize the Imported Dashboard

Community dashboards are starting points. Customize for your needs:

1. Clone before modifying: Click Settings → Save As to create a copy
2. Update variables: Add your namespace/service variables
3. Add custom panels: Include your application-specific metrics
4. Remove noise: Delete panels you don't need

---

Dashboard Design Best Practices

Follow these patterns for dashboards that scale:

1. Organize with Rows

Group related panels into collapsible rows:

• Overview row: High-level health (error rate, latency, traffic)
• Resources row: CPU, memory, network saturation
• Breakdown row: Per-endpoint, per-pod details

2. Consistent Units

Always specify units in panel settings:

• Latency: seconds (s) or milliseconds (ms)
• Memory: bytes (bytes) with auto-scaling (Ki, Mi, Gi)
• Percentages: percent (percent)
• Rates: requests/sec (reqps)

3. Meaningful Thresholds

Set thresholds based on SLOs:

• Green: Within SLO (e.g., under 500ms latency)
• Yellow: Approaching SLO breach (e.g., 500-800ms)
• Red: SLO violated (e.g., over 800ms)

4. Include Time Context

Add annotations for deployments and incidents:

This marks when deployments occurred, correlating code changes with metric changes.

5. Link Related Dashboards

Add dashboard links so operators can drill down:

• Service overview → Pod details
• Cluster metrics → Node metrics
• Application metrics → Trace viewer (Jaeger)

---

Deploying Dashboards as Code

For production, deploy dashboards via Kubernetes ConfigMaps:

The kube-prometheus-stack's Grafana sidecar watches for ConfigMaps with the grafana_dashboard: "1" label and auto-imports them.

Output:

Within seconds, the dashboard appears in Grafana without manual import.

---

Try With AI

Now collaborate with AI to extend your dashboard capabilities.

Setup: You have the Task API Golden Signals dashboard from this lesson. You want to add per-endpoint breakdowns and a summary table.

Prompt 1: Per-Endpoint Latency Breakdown

What you're learning: Using by (endpoint) in PromQL aggregations to create multi-line charts where each line represents a different endpoint.

Prompt 2: Summary Table Panel

What you're learning: Grafana's Table panel type with instant queries (not range queries) and value mappings for status colors.

Prompt 3: Dashboard for Multiple Services

What you're learning: Dashboard architecture patterns. The answer typically involves a hierarchy: fleet overview (all services on one page) linking to service-specific dashboards with full detail.

Safety note: When importing or creating dashboards, avoid exposing them publicly without authentication. Grafana dashboards can reveal infrastructure details. Always deploy behind authentication and consider read-only viewer roles for shared access.

---

Reflect on Your Skill

You built an observability-cost-engineer skill in Lesson 0. Test and improve it based on what you learned about Grafana.

Test Your Skill

Identify Gaps

Ask yourself:

• Did my skill produce valid Grafana JSON with proper schema version?
• Did it include all 4 golden signals with appropriate panel types?
• Did it configure dashboard variables for namespace/service filtering?
• Did it set meaningful thresholds based on SRE best practices?

Improve Your Skill

If you found gaps:

By the end of this lesson, your skill should generate production-ready Grafana dashboards that surface the 4 golden signals with proper filtering and thresholds.