Posts

Serene post template

2026-10-31T00:00:00+00:00

This is a living template for future posts. Copy this file, remove the sections you do not need, and replace the example text with your own.

Headings and inline formatting

Use headings to organize a long post. Text can be bold, italic, ~~struck through~~, or inline code. You can add links, automatic URLs such as https://www.getzola.org, and images.

Lists

Unordered list:

A short item
An item with emphasis
- A nested item

Ordered list:

Research the problem
Implement the solution
Measure the result

Task list:

Write the first draft
Review the draft
Publish the post

Blockquotes and alerts

A normal blockquote is useful for quoting a person, a paper, or a useful idea.

Use a note for context that helps the reader.

Use a tip for a practical recommendation.

Use important for a requirement the reader should not miss.

Use warning for a risky operation or a likely mistake.

Use caution for a destructive or irreversible operation.

Tables

Feature	Use it for	Enabled here
Markdown	Most writing	Yes
KaTeX	Equations	Yes
Mermaid	Diagrams	Yes
Components	Richer blocks	Yes

Code blocks

Inline code is useful for short names. Fenced code blocks support syntax highlighting:

from pathlib import Path

def word_count(path: str) -> int:
    return len(Path(path).read_text().split())

Code annotations are also supported:

1fn main() {
2    let message = "Hello, Serene"; // [!code highlight]
3    println!("{message}");
4}

When documenting Tera syntax, wrap the example in raw so Zola does not evaluate it:

{{ <figure src="/img/example.png" alt="Example" /> }}

Images

The figure component adds dimensions, alt text, and a caption:

A local image can be colocated with the post. Captions support Markdown.

You can also use regular Markdown images:

Mathematics

Set math = true in the front matter to enable KaTeX. Inline math looks like $E = mc^2$. Display math uses double dollar signs:

$$ \int_0^1 x^2,dx = \frac{1}{3} $$

Mermaid diagrams

Set mermaid = true to render diagrams:


flowchart LR
    A[Draft] --> B{Review}
    B -->|Revise| A
    B -->|Approve| C[Publish]

Serene components

Use a styled quote when a citation should stand out:

Make every section earn its place.

— A useful writing rule

Use a detail block for optional context:

Show the implementation notes

This content is hidden until the reader opens it. It is useful for caveats, long explanations, or implementation details that would interrupt the main flow.

You can embed a YouTube video with a video ID:

Footnotes

Footnotes keep supporting details out of the main sentence.¹ You can also reuse a named footnote.²

This is a regular footnote.

Footnotes can contain links, such as Zola's documentation.

A final checklist

Before publishing a real post:

Replace the title, description, date, and tags.
Keep only the front matter flags you actually need.
Give every image useful alt text and explicit dimensions when using figure.
Test math, diagrams, embeds, and links with zola serve.
Remove this tutorial post when it is no longer useful.

Small now, sharp later: API versioning without regrets

2026-10-27T00:00:00+00:00

Additive changes, date-based versions, and deprecations users can plan around.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Where I started

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

What went wrong

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

I want to be honest about the trade-offs here. The approach I settled on solves the problem I had, not every problem of this shape, and there is a clear point where you should reach for something more heavy-duty.

A note for context that helps the reader.

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

The fix

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

What I'd do differently

The first version worked, which is exactly why it was hard to see how wrong it was. Everything looked fine in the demo, and everything looked fine until traffic arrived on a Tuesday afternoon and the graph started doing something I'd never seen before.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Where I started

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

A note for context that helps the reader.

What went wrong

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

What surprised me most was how small the final change was. Most of the work was understanding the problem precisely enough that the fix became obvious, and that understanding took far longer than the fix itself.

CREATE INDEX CONCURRENTLY idx_orders_created_at
    ON orders (created_at DESC)
    WHERE status = 'open';

The fix

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

What I'd do differently

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

By the end of this post you should be able to reproduce the whole thing yourself in under an hour, including the parts I got wrong the first time — those are marked so you can skip them if you're in a hurry.

Wrap-up

If you remember one thing, make it this: additive changes, date-based versions, and deprecations users can plan around compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Killing my dev laptop fan: a Docker desktop postmortem

2026-10-06T00:00:00+00:00

Filesystem mounts, VM memory, and the settings that finally cooled the machine.

The story so far

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

The measurement

A note for context that helps the reader.

The change

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

Numbers

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A very long title to test how the post list wraps entries when someone writes something extremely wordy and unhelpful

2026-09-16T00:00:00+00:00

An intentionally long headline to check list wrapping and heading balance.

Background

Read the error message twice before changing anything
A second point, kept short on purpose

How it works

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: an intentionally long headline to check list wrapping and heading balance is a decision you make once and revisit deliberately.

Concurrency models compared: goroutines, threads, and promises

2026-08-26T00:00:00+00:00

Three ways to do two things at once, and the cost of each.

I kept a log of every decision during this project, and reading it back a month later was humbling: the choices that aged well were the boring ones, and the clever ones all came with a footnote I hadn't written yet.

A short history

The design

A note for context that helps the reader.

const debounce = (fn, ms = 200) => {
  let t;
  return (...args) => {
    clearTimeout(t);
    t = setTimeout(() => fn(...args), ms);
  };
};

Implementation notes

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Failure modes

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

When not to use this

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A destructive or irreversible operation.

A short history

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Capture the current behavior before touching the code
Make the smallest change that preserves it
Delete the workaround you were sure you'd need

The design

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Wrap-up

If you remember one thing, make it this: three ways to do two things at once, and the cost of each compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Backups are a feature: pg_dump and cron done properly

2026-08-05T00:00:00+00:00

Retention, verification, and the restore drill you should run this month.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

The problem

def load_config(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        return {k.strip(): v.strip() for k, v in
                (line.split("=", 1) for line in f if "=" in line)}

First attempt

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A tip for a practical recommendation.

A better model

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

Results

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Making peace with plain old HTML forms

2026-07-16T00:00:00+00:00

Progressive enhancement, no client state, and forms that work before JS loads.

Where I started

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Read the error message twice before changing anything
A second point, kept short on purpose

What went wrong

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: progressive enhancement, no client state, and forms that work before js loads is a decision you make once and revisit deliberately.

Agent tool design: less is more

2026-06-25T00:00:00+00:00

Fewer, sharper tools beat a giant toolbox the model has to browse.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A short history

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

def load_config(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        return {k.strip(): v.strip() for k, v in
                (line.split("=", 1) for line in f if "=" in line)}

The design

Something the reader should not miss.

Implementation notes

Capture the current behavior before touching the code
Make the smallest change that preserves it
Delete the workaround you were sure you'd need

Failure modes

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

What my first production incident taught me about logging

2026-06-04T00:00:00+00:00

The log line I needed didn't exist. Now every service I own has it.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

The story so far

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

The measurement

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

A risky operation or a likely mistake.

The change

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Capture the current behavior before touching the code
Make the smallest change that preserves it
Delete the workaround you were sure you'd need

Numbers

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Frontend performance: the 100ms budget

2026-05-15T00:00:00+00:00

Where input latency actually goes, and the measurements that locate the thief.

The story so far

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

The measurement

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A note for context that helps the reader.

const debounce = (fn, ms = 200) => {
  let t;
  return (...args) => {
    clearTimeout(t);
    t = setTimeout(() => fn(...args), ms);
  };
};

The change

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Numbers

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

What remains

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A risky operation or a likely mistake.

The story so far

def load_config(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        return {k.strip(): v.strip() for k, v in
                (line.split("=", 1) for line in f if "=" in line)}

The measurement

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Capture the current behavior before touching the code
Make the smallest change that preserves it
Delete the workaround you were sure you'd need

The change

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Wrap-up

If you remember one thing, make it this: where input latency actually goes, and the measurements that locate the thief compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

The day I learned to fear feature flags

2026-04-24T00:00:00+00:00

Dead branches, stale config, and the cleanup ritual nobody scheduled.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Setting the scene

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
    let mut last = None;
    for _ in 0..times {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => last = Some(e),
        }
    }
    Err(last.unwrap())
}

The naive approach

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

A note for context that helps the reader.

What the profiler said

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

The rewrite

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Notes on reading code you didn't write

2026-04-03T00:00:00+00:00

A repeatable method for entering an unfamiliar repository and finding the truth.

Setting the scene

Read the error message twice before changing anything
A second point, kept short on purpose

The naive approach

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: a repeatable method for entering an unfamiliar repository and finding the truth is a decision you make once and revisit deliberately.

Search that works without Elasticsearch

2026-03-14T00:00:00+00:00

SQLite FTS5, trigram matching, and when to stop and hire a search engine.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Where I started

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

CREATE INDEX CONCURRENTLY idx_orders_created_at
    ON orders (created_at DESC)
    WHERE status = 'open';

What went wrong

A tip for a practical recommendation.

The fix

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

What I'd do differently

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

A tiny HTTP server in Rust, line by line

2026-02-21T00:00:00+00:00

Tokio, a socket loop, and the pieces you'd otherwise take for granted.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The problem

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

First attempt

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A note for context that helps the reader.

fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
    let mut last = None;
    for _ in 0..times {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => last = Some(e),
        }
    }
    Err(last.unwrap())
}

A better model

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Results

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Trade-offs

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Something the reader should not miss.

The problem

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

First attempt

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

A better model

Wrap-up

If you remember one thing, make it this: tokio, a socket loop, and the pieces you'd otherwise take for granted compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Slicing features so each PR is reviewable in 20 minutes

2026-01-31T00:00:00+00:00

Vertical slices, feature flags, and reviews that catch things.

A short history

Read the error message twice before changing anything
A second point, kept short on purpose

The design

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: vertical slices, feature flags, and reviews that catch things is a decision you make once and revisit deliberately.

Async Python is not a silver bullet

2026-01-11T00:00:00+00:00

Where asyncio shines, where it hurts, and when threads were the right answer.

The problem

First attempt

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A note for context that helps the reader.

const debounce = (fn, ms = 200) => {
  let t;
  return (...args) => {
    clearTimeout(t);
    t = setTimeout(() => fn(...args), ms);
  };
};

A better model

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Results

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Trade-offs

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

A tip for a practical recommendation.

The problem

CREATE INDEX CONCURRENTLY idx_orders_created_at
    ON orders (created_at DESC)
    WHERE status = 'open';

First attempt

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

A better model

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Wrap-up

If you remember one thing, make it this: where asyncio shines, where it hurts, and when threads were the right answer compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Why I write design docs before code

2025-12-21T00:00:00+00:00

One page, three sections, and the arguments it saves a month later.

The problem

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Latency at p50, because most users live there
A second point, kept short on purpose

First attempt

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: one page, three sections, and the arguments it saves a month later is a decision you make once and revisit deliberately.

Performance budgets for people who ship

2025-11-30T00:00:00+00:00

A budget you can enforce in CI beats a wish you revisit never.

Context

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Capture the current behavior before touching the code
A second point, kept short on purpose

The plan

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: a budget you can enforce in ci beats a wish you revisit never is a decision you make once and revisit deliberately.

The long tail of accessibility fixes nobody assigns

2025-11-10T00:00:00+00:00

Focus rings, labels, and the audit checklist that fits in an afternoon.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The problem

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

# rebuild only what changed
docker build \
  --cache-from "$IMAGE:latest" \
  --tag "$IMAGE:${GIT_SHA}" \
  .

First attempt

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A note for context that helps the reader.

A better model

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

Results

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Webhooks done right: retries, signatures, and idempotency

2025-10-20T00:00:00+00:00

The three properties that separate webhooks that just work from ones you fear.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Context

The plan

A note for context that helps the reader.

fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
    let mut last = None;
    for _ in 0..times {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => last = Some(e),
        }
    }
    Err(last.unwrap())
}

What shipped

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Follow-ups

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Context

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A note for context that helps the reader.

The plan

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

What shipped

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

Follow-ups

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Wrap-up

If you remember one thing, make it this: the three properties that separate webhooks that just work from ones you fear compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Terminal workflows: the tools I refuse to give up

2025-09-29T00:00:00+00:00

The small utilities that quietly save an hour a day.

Background

# rebuild only what changed
docker build \
  --cache-from "$IMAGE:latest" \
  --tag "$IMAGE:${GIT_SHA}" \
  .

How it works

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A risky operation or a likely mistake.

In practice

Capture the current behavior before touching the code
Make the smallest change that preserves it
Delete the workaround you were sure you'd need

Edge cases

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

How I keep a learning journal that I actually reread

2025-09-09T00:00:00+00:00

A flat file, five questions, and a weekly review that survives real life.

The problem

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Capture the current behavior before touching the code
A second point, kept short on purpose

First attempt

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: a flat file, five questions, and a weekly review that survives real life is a decision you make once and revisit deliberately.

SQLite is all you need (until it isn't)

2025-08-19T00:00:00+00:00

Where a single-file database wins outright, and the exact line where it stops.

The story so far

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

# rebuild only what changed
docker build \
  --cache-from "$IMAGE:latest" \
  --tag "$IMAGE:${GIT_SHA}" \
  .

The measurement

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A destructive or irreversible operation.

The change

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

Numbers

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

From monolith to modules without the microservices tax

2025-07-29T00:00:00+00:00

Boundaries, dependency rules, and extracting a service only when the metrics say so.

Background

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

How it works

A note for context that helps the reader.

fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
    let mut last = None;
    for _ in 0..times {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => last = Some(e),
        }
    }
    Err(last.unwrap())
}

In practice

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Edge cases

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Alternatives considered

A destructive or irreversible operation.

Background

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
    let mut last = None;
    for _ in 0..times {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => last = Some(e),
        }
    }
    Err(last.unwrap())
}

How it works

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Capture the current behavior before touching the code
Make the smallest change that preserves it
Delete the workaround you were sure you'd need

In practice

Wrap-up

If you remember one thing, make it this: boundaries, dependency rules, and extracting a service only when the metrics say so compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Reading the standard library for fun

2025-07-09T00:00:00+00:00

What you learn about a language by reading the code everyone else already trusts.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

A short history

Capture the current behavior before touching the code
A second point, kept short on purpose

The design

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: what you learn about a language by reading the code everyone else already trusts is a decision you make once and revisit deliberately.

CSS container queries changed how I build components

2025-06-18T00:00:00+00:00

Components that know their own width, and the layout code that disappeared.

The problem

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

const debounce = (fn, ms = 200) => {
  let t;
  return (...args) => {
    clearTimeout(t);
    t = setTimeout(() => fn(...args), ms);
  };
};

First attempt

Something the reader should not miss.

A better model

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

Results

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

The deployment pipeline that broke in prod (and how I fixed it)

2025-05-28T00:00:00+00:00

A green build, a broken release, and the missing piece between them.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Setting the scene

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

The naive approach

A note for context that helps the reader.

const debounce = (fn, ms = 200) => {
  let t;
  return (...args) => {
    clearTimeout(t);
    t = setTimeout(() => fn(...args), ms);
  };
};

What the profiler said

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

The rewrite

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Lessons

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

A risky operation or a likely mistake.

Setting the scene

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

The naive approach

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

What the profiler said

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Wrap-up

If you remember one thing, make it this: a green build, a broken release, and the missing piece between them compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Rust ownership explained with a library book checkout counter

2025-05-07T00:00:00+00:00

Borrowing, moves, and lifetimes through one analogy that finally stuck.

Context

def load_config(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        return {k.strip(): v.strip() for k, v in
                (line.split("=", 1) for line in f if "=" in line)}

The plan

A tip for a practical recommendation.

What shipped

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Capture the current behavior before touching the code
Make the smallest change that preserves it
Delete the workaround you were sure you'd need

Follow-ups

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Making friends with the Linux OOM killer

2025-04-17T00:00:00+00:00

Reading dmesg, understanding overcommit, and choosing sane memory limits.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Context

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

def load_config(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        return {k.strip(): v.strip() for k, v in
                (line.split("=", 1) for line in f if "=" in line)}

The plan

A destructive or irreversible operation.

What shipped

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

Follow-ups

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Prompt engineering is just interface design

2025-03-27T00:00:00+00:00

Treating prompts as APIs — versions, contracts, and tests included.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Where I started

Capture the current behavior before touching the code
A second point, kept short on purpose

What went wrong

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: treating prompts as apis is a decision you make once and revisit deliberately.

An honest review of my AI agent architecture after six months

2025-03-06T00:00:00+00:00

What held up, what I ripped out, and where the token bills went.

Background

How it works

A note for context that helps the reader.

const debounce = (fn, ms = 200) => {
  let t;
  return (...args) => {
    clearTimeout(t);
    t = setTimeout(() => fn(...args), ms);
  };
};

In practice

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Edge cases

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Alternatives considered

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Something the reader should not miss.

Background

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

# rebuild only what changed
docker build \
  --cache-from "$IMAGE:latest" \
  --tag "$IMAGE:${GIT_SHA}" \
  .

How it works

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Capture the current behavior before touching the code
Make the smallest change that preserves it
Delete the workaround you were sure you'd need

In practice

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Wrap-up

If you remember one thing, make it this: what held up, what i ripped out, and where the token bills went compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Connection pooling in five minutes

2025-02-14T00:00:00+00:00

What pgbouncer actually pools, and the two settings that matter most.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Setting the scene

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Latency at p50, because most users live there
A second point, kept short on purpose

The naive approach

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: what pgbouncer actually pools, and the two settings that matter most is a decision you make once and revisit deliberately.

Why your background jobs keep failing at 3am

2025-01-24T00:00:00+00:00

Retries, idempotency, and the quiet ways outboxes fall out of sync.

The problem

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

First attempt

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

A tip for a practical recommendation.

A better model

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

Results

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Observability for side projects: logs, traces, and one dashboard

2025-01-03T00:00:00+00:00

The minimum instrumentation that answers 'what happened' at 3am.

Setting the scene

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

The naive approach

A note for context that helps the reader.

fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
    let mut last = None;
    for _ in 0..times {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => last = Some(e),
        }
    }
    Err(last.unwrap())
}

What the profiler said

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

The rewrite

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Lessons

A tip for a practical recommendation.

Setting the scene

fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
    let mut last = None;
    for _ in 0..times {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => last = Some(e),
        }
    }
    Err(last.unwrap())
}

The naive approach

Capture the current behavior before touching the code
Make the smallest change that preserves it
Delete the workaround you were sure you'd need

What the profiler said

Wrap-up

If you remember one thing, make it this: the minimum instrumentation that answers 'what happened' at 3am compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Writing tests for code you're afraid to touch

2024-12-14T00:00:00+00:00

A characterization-testing recipe for legacy code with no safety net.

Background

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

def load_config(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        return {k.strip(): v.strip() for k, v in
                (line.split("=", 1) for line in f if "=" in line)}

How it works

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A note for context that helps the reader.

In practice

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

Edge cases

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

CI pipelines that finish before the coffee does

2024-11-23T00:00:00+00:00

Caching, matrix builds, and the discipline of a pipeline under four minutes.

Where I started

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

What went wrong

A risky operation or a likely mistake.

The fix

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

What I'd do differently

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

What happens when you type a URL into a browser

2024-11-02T00:00:00+00:00

DNS, TCP, TLS, and rendering — the full trip, told once, properly.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The problem

First attempt

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A note for context that helps the reader.

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

A better model

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Results

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Trade-offs

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A note for context that helps the reader.

The problem

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
    let mut last = None;
    for _ in 0..times {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => last = Some(e),
        }
    }
    Err(last.unwrap())
}

First attempt

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

A better model

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Wrap-up

If you remember one thing, make it this: dns, tcp, tls, and rendering compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

React state management in 2026: the boring answer

2024-10-13T00:00:00+00:00

Most apps need less state than they have, and a store is rarely the missing piece.

Context

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

The plan

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

A destructive or irreversible operation.

What shipped

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

Follow-ups

TypeScript types are a programming language

2024-09-22T00:00:00+00:00

Discriminated unions, branded types, and the point where types start computing.

Context

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

The plan

A note for context that helps the reader.

CREATE INDEX CONCURRENTLY idx_orders_created_at
    ON orders (created_at DESC)
    WHERE status = 'open';

What shipped

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Follow-ups

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Context

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A destructive or irreversible operation.

The plan

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

What shipped

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

Follow-ups

Wrap-up

If you remember one thing, make it this: discriminated unions, branded types, and the point where types start computing compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Rate limiting from scratch: token bucket in 40 lines

2024-09-01T00:00:00+00:00

A small, correct token bucket, and the edge cases that make it interesting.

Where I started

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

const debounce = (fn, ms = 200) => {
  let t;
  return (...args) => {
    clearTimeout(t);
    t = setTimeout(() => fn(...args), ms);
  };
};

What went wrong

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Something the reader should not miss.

The fix

Capture the current behavior before touching the code
Make the smallest change that preserves it
Delete the workaround you were sure you'd need

What I'd do differently

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

The cost of a cache: measuring before optimizing

2024-08-12T00:00:00+00:00

Hit rates, eviction, and the latency numbers behind a cache that made things worse.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Where I started

What went wrong

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A note for context that helps the reader.

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

The fix

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

What I'd do differently

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Where I started

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A risky operation or a likely mistake.

What went wrong

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

def load_config(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        return {k.strip(): v.strip() for k, v in
                (line.split("=", 1) for line in f if "=" in line)}

The fix

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

What I'd do differently

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Wrap-up

If you remember one thing, make it this: hit rates, eviction, and the latency numbers behind a cache that made things worse compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Git bisect saved me four hours

2024-07-22T00:00:00+00:00

How a two-minute ritual turned a mysterious regression into a single bad commit.

The problem

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Read the error message twice before changing anything
A second point, kept short on purpose

First attempt

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: how a two-minute ritual turned a mysterious regression into a single bad commit is a decision you make once and revisit deliberately.

Docker images 60% smaller: multi-stage builds in practice

2024-07-01T00:00:00+00:00

A practical walkthrough from a 1.2GB image to a 460MB one without changing app code.

Setting the scene

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

CREATE INDEX CONCURRENTLY idx_orders_created_at
    ON orders (created_at DESC)
    WHERE status = 'open';

The naive approach

A note for context that helps the reader.

What the profiler said

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

The rewrite

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Building a retrieval pipeline that doesn't lie to you

2024-06-11T00:00:00+00:00

Chunking, reranking, and evaluation for a RAG system that admits when it doesn't know.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A short history

The design

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A note for context that helps the reader.

CREATE INDEX CONCURRENTLY idx_orders_created_at
    ON orders (created_at DESC)
    WHERE status = 'open';

Implementation notes

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Failure modes

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

When not to use this

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Something the reader should not miss.

A short history

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

The design

Wrap-up

If you remember one thing, make it this: chunking, reranking, and evaluation for a rag system that admits when it doesn't know compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Understanding embeddings without the math degree

2024-05-21T00:00:00+00:00

Vectors, similarity, and why your search results suddenly got better.

Background

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

How it works

A tip for a practical recommendation.

In practice

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

Edge cases

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Shipping a side project in 30 days: what actually mattered

2024-04-30T00:00:00+00:00

Scope, deadlines, and the one weekly ritual that kept the project alive.

Setting the scene

const debounce = (fn, ms = 200) => {
  let t;
  return (...args) => {
    clearTimeout(t);
    t = setTimeout(() => fn(...args), ms);
  };
};

The naive approach

A risky operation or a likely mistake.

What the profiler said

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

The rewrite

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Things I wish someone told me before my first internship

2024-04-10T00:00:00+00:00

Nobody expects you to know the codebase. They expect you to ask good questions.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Where I started

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Read the error message twice before changing anything
A second point, kept short on purpose

What went wrong

Everything above is the short version. The full write-up of how this played out is a story for another day — for now, the takeaway is simple: nobody expects you to know the codebase. they expect you to ask good questions is a decision you make once and revisit deliberately.

Debugging a memory leak with nothing but /proc

2024-03-20T00:00:00+00:00

A weekend incident, a growing RSS number, and no profiler available on the box.

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Where I started

def load_config(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        return {k.strip(): v.strip() for k, v in
                (line.split("=", 1) for line in f if "=" in line)}

What went wrong

A destructive or irreversible operation.

The fix

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Read the error message twice before changing anything
Reproduce it with the smallest input you can
Write the failing test first, even if it feels ceremonial
Only then reach for the fix

What I'd do differently

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Postgres indexes: the only guide I wish I had

2024-02-28T00:00:00+00:00

B-tree, partial, covering, and GIN indexes with the queries that motivated each one.

The problem

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

First attempt

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A note for context that helps the reader.

fn retry<T>(mut f: impl FnMut() -> Result<T>, times: u32) -> Result<T> {
    let mut last = None;
    for _ in 0..times {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => last = Some(e),
        }
    }
    Err(last.unwrap())
}

A better model

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

Results

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Trade-offs

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A tip for a practical recommendation.

The problem

const debounce = (fn, ms = 200) => {
  let t;
  return (...args) => {
    clearTimeout(t);
    t = setTimeout(() => fn(...args), ms);
  };
};

First attempt

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

A better model

Wrap-up

If you remember one thing, make it this: b-tree, partial, covering, and gin indexes with the queries that motivated each one compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

A gentle introduction to async Rust for people who know sync Python

2024-02-08T00:00:00+00:00

Futures, executors, and await, explained against the backdrop of Python's asyncio.

Where I started

The numbers below are from my own machine and my own workload. Treat them as a method, not a benchmark: the point is how the measurement was taken, not the exact figures it produced.

What went wrong

A note for context that helps the reader.

[deploy]
strategy = "rolling"
healthcheck = "/healthz"
max_unavailable = 1

The fix

The best engineering advice I've received is unglamorous: make it measurable, make it reversible, and only then make it fast.

What I'd do differently

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

Approach	Best for	Watch out for
Baseline	Quick comparisons	No tail visibility
Instrumented	Finding the real hotspot	Overhead of the probes
Production	Truthful numbers	Harder to reproduce

Where I started

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

A note for context that helps the reader.

What went wrong

const debounce = (fn, ms = 200) => {
  let t;
  return (...args) => {
    clearTimeout(t);
    t = setTimeout(() => fn(...args), ms);
  };
};

The fix

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

What I'd do differently

Wrap-up

If you remember one thing, make it this: futures, executors, and await, explained against the backdrop of python's asyncio compounds. The cost of getting it right early is a afternoon; the cost of getting it late is a migration.

Why I stopped fighting CSS specificity and started nesting

2024-01-18T00:00:00+00:00

Specificity wars are a symptom of structure, not selectors — nesting with layers fixed it for me.

A short history

# rebuild only what changed
docker build \
  --cache-from "$IMAGE:latest" \
  --tag "$IMAGE:${GIT_SHA}" \
  .

The design

Tools matter less than the feedback loop they enable. A fast, boring loop beat every sophisticated setup I tried, and I've kept it ever since, even as the rest of the stack changed twice over.

A tip for a practical recommendation.

Implementation notes

Latency at p50, because most users live there
Latency at p99, because someone always lives there
Error rate, because a fast failure is still a failure
Saturation, because averages hide the cliff

Failure modes

There is a version of this article that opens with theory. I decided against it, because the theory is easy to find and the motivation is not — so let's start with the thing that actually broke.

Posts

Serene post template

Headings and inline formatting

Lists

Blockquotes and alerts

Tables

Code blocks

Images

Mathematics

Mermaid diagrams

Serene components

Footnotes

A final checklist

Small now, sharp later: API versioning without regrets

Where I started

What went wrong

The fix

What I'd do differently

Where I started

What went wrong

The fix

What I'd do differently

Wrap-up

Killing my dev laptop fan: a Docker desktop postmortem

The story so far

The measurement

The change

Numbers

A very long title to test how the post list wraps entries when someone writes something extremely wordy and unhelpful

Background

How it works

Concurrency models compared: goroutines, threads, and promises

A short history

The design

Implementation notes

Failure modes

When not to use this

Further reading

A short history

The design

Wrap-up

Backups are a feature: pg_dump and cron done properly

The problem

First attempt

A better model

Results

Making peace with plain old HTML forms

Where I started

What went wrong

Agent tool design: less is more

A short history

The design

Implementation notes

Failure modes

What my first production incident taught me about logging

The story so far

The measurement

The change

Numbers

Frontend performance: the 100ms budget

The story so far

The measurement

The change

Numbers

What remains

The story so far

The measurement

The change

Wrap-up

The day I learned to fear feature flags

Setting the scene

The naive approach

What the profiler said

The rewrite

Notes on reading code you didn't write

Setting the scene

The naive approach

Search that works without Elasticsearch

Where I started

What went wrong