An Email Is an Event, Not a State

Last week I shipped shipment-status emails for a storefront I'm building — the "your order is on its way" kind. The feature took a couple of days. The week after it, I deleted most of the emails it sent.

That sounds like a failure. It was the most valuable part of the work, and the reasoning transfers to anything that turns stored state into a message: push notifications, Slack alerts, webhooks, SMS. So here is what watching one real order taught me — including two bugs no test I would have thought to write could have caught.

Some context, kept deliberately vague because it's client work: the storefront hands orders to a third-party logistics vendor that fulfils them line by line. Order four things and they can ship two today, cancel one tomorrow, and send the last next week. Our side is a Next.js app plus a standing worker process, coordinating through Postgres. There is no message broker — the order status column is the queue, claimed with FOR UPDATE SKIP LOCKED. The worker sweeps on a timer: read the vendor, write what changed, maybe send an email.

1. Never decide a side effect from a diff you're about to destroy

The obvious design is a diff. You have the line's stored status, you have the vendor's answer; if they differ, email the customer and store the new value.

Try to order those two writes and the design falls apart.

Store first, then send, and the diff is gone the instant you need it. If the send fails — SMTP timeout, provider hiccup, process killed — nothing records that an email was owed. The next sweep compares stored to vendor, sees agreement, and stays silent forever. A dropped email becomes permanently undetectable.

Send first, then store, and a crash between them sends the same email again on the next sweep. You've traded silent loss for duplicates.

The fix was to stop asking "what changed?" and start asking "what have I already spoken for?" A table of sent emails records, per row, the lines that email covered. The pending email is then a pure function of stored state and coverage:

export function pendingNotification(
  items: readonly NotifiableItem[],
  sent: readonly EmailCoverage[],
): PendingNotification | null;

Nothing in there is a diff. The worker writes the new statuses first, then asks this function what is still owed. Sending is claim → send → release on failure: insert the coverage row (a unique index is the guard), send, and delete the row if the send failed, so the batch stays owed and the next sweep retries it. An unsent email is a missing row — a thing you can query, alert on and retry — rather than a difference that stopped existing.

It also fixed a case I hadn't thought about, for free. The vendor moves a line from shipped to invoiced; both mean "gone" to us, so both map to the same internal status. A diff over their raw strings would have emailed twice about one parcel. Coverage looks at the line, finds it already spoken for, and says nothing.

The bug inside the fix

The coverage table's unique key started as (orderId, status). That's the key the order-confirmation email uses, and confirmation emails are one per order, so it looked like consistency.

Except one order now owes several shipped emails. The first batch's row occupied (orderId, "shipped"), so every later batch's claim hit the unique constraint — and a unique violation on claim is indistinguishable from "already sent", which is the success path. Ship two lines on Monday and two on Wednesday, and Wednesday's email was silently dropped as a duplicate.

The key is now (orderId, status, batchKey), where batchKey is the batch's line IDs, sorted so it is stable however the lines come back.

If your dedup key is coarser than your unit of work, dedup becomes data loss — and it is the quietest possible failure, because suppression is supposed to look like success.

2. A view describes a state; an email announces an event

The vendor gave us a mapping from their internal statuses to four words a shopper should see. Thirteen or so states collapse into Confirmed, Processing, Shipped, Cancelled.

I read that mapping as a specification for emails. It isn't. It's a specification for vocabulary, and I wired one email per state.

Then I watched a real order on staging. It sat in the vendor's open state — received, nobody has touched it — for two days. The first sweep after deploy duly emailed the customer: we're preparing your order. Which was, on the evidence, false, and in any case carried nothing the confirmation email hadn't already said an hour after checkout.

I cut open. A few days later I cut pick and pack too, for the same reason one level up: the order status page already shows Processing, and a message that adds nothing is how a shopper learns to ignore the one that matters. The final rule is narrow:

Email when a line ships, or when a line never will, plus one message once every line is resolved. Nothing else.

The page kept all four words. That's the distinction that ended the per-status argument, and the one I'll carry to the next system: a page is a view and describes a state; an email is an interruption and announces an event. They read the same data and answer different questions. Anything the page already shows has to clear a much higher bar to also arrive in someone's inbox.

The test I'd give any notification I design from now on: did this change the outcome for the person receiving it? A parcel leaving changed the outcome. An item being cancelled changed the outcome. Warehouse staff picking a box off a shelf did not.

3. Any state-derived notifier owes a backfill on its first run

This one is structural, and I nearly shipped it to production.

Coverage rows only exist once an email has been sent. So on the day a new notifying status deploys, every order already in that state owes an email immediately — the system cannot tell "this just happened" from "this has been true for a fortnight", because the only thing it knows is that nothing has spoken for it yet.

On staging, the first sweep after deploy sent 28 emails in one go. In production that would have been the entire open order book, at once, all telling people something they mostly already knew.

The property is inherent to deriving at-least-once notifications from state, and the fix isn't to go back to diffing — it's to recognise that introducing a notification is a data migration. Backfill coverage rows for existing records as a deploy step, so the notifier starts life believing the back catalogue has already been spoken for.

If you're adding a notification to a system that already has live data, that's the question to ask before you merge: what does the first sweep think it owes?

4. Two bugs that only a real order could show

Both of these were found by pushing one order through the whole pipeline on staging and reading the logs. Neither was a unit-test-shaped mistake.

An alert must not park the work

When the vendor cancels a line, a human has to decide about a refund — that is genuinely not the worker's call. So the poll logged an alert and returned early, leaving the order parked for a person.

On every sweep. Not just the first one.

Which meant an order with one cancelled line never advanced its status again. Its page showed a "Processing" pill above two lines that had already shipped, and would have done so forever, while the shipped email waited on a sweep that returned before reaching it. One order needing a human judgement about $40 froze every fact the system knew about the other three items.

The correction was to do the work first and colour the outcome afterwards: write the statuses, derive the emails, then alert and park. Parking a decision must never park the facts. A state machine's job is to record what is true, and "a human owes us a judgement" is not a reason to stop recording.

The alert also now fires once, when the cancellation is new, rather than on every sweep. An alert nobody has acted on yet, re-firing every few minutes, buries the next real one.

A capped queue plus a permanently-failing item equals starvation

The poll does one bulk read, then spends a per-order call on up to 50 unresolved orders. It took them oldest-first, which felt like the fair choice — first in, first served, nothing gets left behind.

It is exactly the wrong choice, because an item that can never complete never leaves the queue. Staging had a block of old fixture orders the vendor permanently 404s. They were unresolved, they were the oldest things there, and so they occupied the head of a 50-item window on every single sweep, forever. A real order placed that morning was never chased. Not delayed — never reached, on every sweep, indefinitely.

Oldest-first is safe only when work eventually drains. Add a cap and a class of items that cannot drain, and the cap becomes a wall. The claim now reads newest-first, because a recently submitted order is where fulfilment is actually moving, and the bulk read still covers the older ones without spending a per-order call on them.

Worth checking in your own system: does anything take a fixed-size window off the front of a sorted set of pending work? If so, what happens when the front of that set holds something that will never succeed?

The short version

Six things I'd hand to someone building this next time:

1. Derive side effects from what you've done, not from what changed. A diff is destroyed by the write that follows it.
2. Dedup at the granularity of the unit of work. A coarse key turns suppression into silent loss.
3. A view describes a state; an email announces an event. If the page already shows it, it probably isn't an email.
4. Introducing a notification is a data migration. Ask what the first sweep thinks it owes.
5. An alert must never park the work it is alerting about. Record the facts, then flag the decision.
6. Capped queues starve. Order by where progress actually happens, not by who arrived first.

And the one that produced all six: run a real thing through the whole system and watch it. Every item on that list came from reading logs on staging as one order moved, not from a failing test. The tests were green the whole time — they were testing the design, and the design was what was wrong.

$ tail -f — mehdi chamiani, digital architect