Backups are a feature: pg_dump and cron done properly
Retention, verification, and the restore drill you should run this month.
The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.
The problem
I want to be honest about the trade-offs here. The approach I settled on solves the problem I had, not every problem of this shape, and there is a clear point where you should reach for something more heavy-duty.
The first version worked, which is exactly why it was hard to see how wrong it was. Everything looked fine in the demo, and everything looked fine until traffic arrived on a Tuesday afternoon and the graph started doing something I'd never seen before.
def load_config(path: str) -> dict:
with open(path, encoding="utf-8") as f:
return {k.strip(): v.strip() for k, v in
(line.split("=", 1) for line in f if "=" in line)}First attempt
Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
A tip for a practical recommendation.
A better model
Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
- Latency at p50, because most users live there
- Latency at p99, because someone always lives there
- Error rate, because a fast failure is still a failure
- Saturation, because averages hide the cliff
Results
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.