Performance work fails quietly when the measurement is theatrical. A chart goes up; a narrative forms; nobody can reproduce the run three weeks later.
This note is a short checklist I use when I am trying to learn something real about a system — whether a C++ microbenchmark on Linux or a batch pipeline that “feels” slower after a refactor.
Start with a question, not a knob
“Make it faster” is not a question. Better:
- Which percentile matters?
- Under what input shape?
- Against which baseline commit?
- On which machine and power settings?
If you cannot answer those, you are not measuring yet — you are improvising.
Capture the environment
At minimum:
- CPU model, core count, memory
- OS and kernel
- Compiler and flags
- Frequency governor / turbo behavior (when relevant)
- Commit SHA and build ID
Boring metadata is what turns a result into something you can argue with later.
Prefer paired comparisons
Run A and B under the same harness. Randomize order if the machine warms up. Report distributions, not a single heroic number. If the difference is smaller than run-to-run noise, say so.
hypothesis → harness → baseline → change → remeasure → write it downWatch for self-deception
Common traps:
- Optimizing a microbenchmark the production path never hits
- Letting the compiler delete your workload
- Confusing mean latency with tail latency
- Cherry-picking the best of N runs
The fix is social as much as technical: write the hypothesis before you see the chart.
What “good enough” looks like
You do not need a journal paper. You need:
- A reproducible command
- A before/after comparison
- A one-paragraph interpretation that admits uncertainty
That standard transfers. It is the same discipline that keeps data platforms honest when someone asks whether a pipeline change “improved quality.”
Further reading
- Related project notes: Systems Performance Lab
- TODO: Add links to papers or posts you actually recommend