Performance Engineering and the Testing Pyramid

Performance engineering starts with user-visible budgets and measurement, then improves the actual bottleneck. A trustworthy backend uses fast unit tests for rules, real integration tests for infrastructure, contract tests for boundaries, a small end-to-end suite, load and resilience tests for capacity, security tests for abuse, and AI evaluations for probabilistic quality.

What will you be able to do?

• define latency, throughput, concurrency, memory, and cost budgets;
• profile application, database, network, and model bottlenecks;
• build a balanced test portfolio;
• run realistic load tests without confusing local results with production capacity.

What should be measured first?

For each operation define workload shape, payload, dataset size, concurrency, warm/cold condition, success criteria, latency percentiles, error rate, resource use, and cost. Measure API time broken into queue, database, retrieval, provider, and serialization. Benchmark one controlled change at a time.

Common improvements include correct indexes, bounded result sets, eliminating N+1 queries, reusing connections, moving CPU work to workers, caching measured hot safe reads, reducing AI context, selecting an evaluated smaller model, and controlling concurrency. Adding web workers does not fix an exhausted database pool.

What is the testing portfolio?

Use real PostgreSQL for transaction and query semantics. Use Redis integration tests for TTL and atomic scripts. Keep most AI application tests on deterministic fakes; run provider contract smoke tests separately and evaluations before release.

How do you test performance responsibly?

Warm the system intentionally, generate realistic tenant/data distribution, include writes and slow dependencies, ramp gradually, watch saturation, preserve environment details, and stop before harming shared systems. Never load-test production without explicit authorization and safeguards.

Why does this matter in AI-native systems?

Semantic quality and runtime performance trade off through model choice, context size, retrieval depth, reranking, and tool steps. Optimize against a Pareto frontier of quality, latency, cost, and safety rather than one metric.

Common mistakes

• Chasing requests per second while tail latency and errors collapse.
• Mocking SQL and believing transaction behavior is tested.
• End-to-end tests cover everything and become slow/flaky.
• Test suite calls live models on every commit.
• Load test has one tenant and misses fairness/noisy-neighbor behavior.
• Cache makes the benchmark fast while serving stale unauthorized data.

AI Pair-Programmer Prompt

Exercise

Build a test matrix and load model for ticket listing plus AI drafting. Include 100 tenants, one noisy tenant, provider slowdown, Redis loss, and a database query regression.

Acceptance criteria: tests identify the query regression, capacity controls protect other tenants, quality gates remain enforced, and a written result distinguishes bottleneck evidence from speculation.

Knowledge check

1. Why use real PostgreSQL integration tests?
2. Does adding workers always increase capacity?
3. What four dimensions should AI optimization balance?

---

Answers

1. Mocks cannot reproduce SQL, constraints, isolation, locks, query plans, or driver behavior.
2. No; downstream pools, CPU, memory, and dependencies may already be saturated.
3. Quality, latency, cost, and safety.

Completion checklist

• Performance claims have reproducible measurements.
• Test layers match the risks they can prove.
• AI optimization preserves evaluation and safety gates.

Primary references

• pytest
• k6 documentation
• PostgreSQL monitoring