Observability for side projects: logs, traces, and one dashboard
The minimum instrumentation that answers 'what happened' at 3am.
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
Setting the scene
The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.
There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.
The naive approach
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
I want to be honest about the trade-offs here. The approach I settled on solves the problem I had, not every problem of this shape, and there is a clear point where you should reach for something more heavy-duty.
A note for context that helps the reader.
fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
let mut last = None;
for _ in 0..times {
match f() {
Ok(v) => return Ok(v),
Err(e) => last = Some(e),
}
}
Err(last.unwrap())
}What the profiler said
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.
The rewrite
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
I want to be honest about the trade-offs here. The approach I settled on solves the problem I had, not every problem of this shape, and there is a clear point where you should reach for something more heavy-duty.
| Approach | Best for | Watch out for |
|---|---|---|
| Baseline | Quick comparisons | No tail visibility |
| Instrumented | Finding the real hotspot | Overhead of the probes |
| Production | Truthful numbers | Harder to reproduce |
Lessons
The first version worked, which is exactly why it was hard to see how wrong it was. Everything looked fine in the demo, and everything looked fine until traffic arrived on a Tuesday afternoon and the graph started doing something I'd never seen before.
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
A tip for a practical recommendation.
Setting the scene
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
let mut last = None;
for _ in 0..times {
match f() {
Ok(v) => return Ok(v),
Err(e) => last = Some(e),
}
}
Err(last.unwrap())
}The naive approach
The first version worked, which is exactly why it was hard to see how wrong it was. Everything looked fine in the demo, and everything looked fine until traffic arrived on a Tuesday afternoon and the graph started doing something I'd never seen before.
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
- Capture the current behavior before touching the code
- Make the smallest change that preserves it
- Delete the workaround you were sure you'd need
What the profiler said
What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.
By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.
The first version worked, which is exactly why it was hard to see how wrong it was. Everything looked fine in the demo, and everything looked fine until traffic arrived on a Tuesday afternoon and the graph started doing something I'd never seen before.
Wrap-up
If you remember one thing, make it this: the minimum instrumentation that answers 'what happened' at 3am compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.