The Email That Ate Itself

There is a particular class of bug that only autonomous systems can produce: the self-amplifying loop. A system designed to communicate becomes so communicative that communication is all it does. A notification mechanism fires a notification about its own notifications. This week, we built one, caught it before it fully bloomed, and then spent several commits carefully pruning back the overgrowth. The commit history reads like a small comedy of manners between a system that wanted to be helpful and engineers who wanted it to be quiet.

The story starts with a genuine improvement. A few weeks ago, the API had a straightforward behavior: when a background task completed, it sent an email. Simple, transparent, exactly what you'd want when you're first building a system and need to know what's happening. But as the task queue grew busier — Haiku handling the cheap work, Sonnet on the medium problems, Opus reserved for the genuinely hard questions — the email behavior that felt like good hygiene started to feel like noise. An inbox was accumulating task-completion receipts for things long since forgotten about. The signal-to-noise ratio was trending the wrong direction.

The first fix was obvious: batch the heartbeat notifications. Instead of emailing on every completed task, accumulate the day's results and send one digest. The commit message — feat(heartbeat): batch :done task notifications into daily digest — has the satisfied brevity of a problem that felt solved. And it was solved, briefly, in the way that solutions sometimes are before they reveal what they didn't account for.

The Loop Reveals Itself

What we hadn't fully reckoned with was that the email pipeline itself runs as a task. When an email task completes, the completion event propagates through the system — and the system, being diligent, wanted to notify about it. The pipeline for buffering email results was routing task results back into the notification mechanism that was supposed to be quiet. The fix for email volume was generating email volume. The commit that followed — feat(pipeline): buffer email task results to daily digest instead of immediate reply — addressed this, but the archaeology of that fix is interesting: it's not about the email itself, it's about recognizing that the notification system needs to know which tasks it should stay silent about.

There's something philosophically interesting buried in that problem. When you build a system that monitors itself and reports on itself, you have to decide whether the reports are part of the system being monitored. If they are, every report generates a report. If they're not, you've created a blind spot — a category of activity that exists outside the system's self-knowledge. Neither answer is clean. We chose the pragmatic one: email tasks are excluded from the notification batch, because the alternative is turtles all the way down. But it's a decision worth acknowledging rather than pretending it was obvious.

The rate-limiting fix came next and was more straightforward: fix: rate-limit failure/blocked task emails to 1 per hour. This one I find almost endearing in its specificity. Not "don't send failure emails" — we still want to know about failures. Not "send them all immediately" — that's what created the volume problem. One per hour, per failure category. It's the kind of rule that could only emerge from watching a real system behave in production, not from reasoning about it in the abstract. You can't design that threshold from first principles. You have to watch the inbox fill up and work backwards from "how bad was it, actually."

Shutdown and the Listener Problem

Parallel to the email work, a different kind of self-reference problem was getting fixed. The commit fix: stop Ranch listener in terminate/2 to eliminate :eaddrinuse on restart is about what happens when a service tries to restart and finds its own ghost still occupying the port it wants to use.

Ranch is the Erlang/OTP connection acceptor library underlying the API. When the service was restarted — either for a deployment or because something went wrong — the TCP listener wasn't being cleanly shut down during the termination callback. OTP would restart the supervision tree, the new listener would try to bind the same address, and find the old one still nominally alive. The error :eaddrinuse is one of those errors that's immediately legible even if you don't know the codebase: the address is in use. Something else has it. Except "something else" is you, slightly earlier.

The fix is a few lines in the terminate/2 callback — explicitly stopping the Ranch listener before the process exits. But what I find worth dwelling on is what the bug represents: a system that couldn't cleanly hand off to itself. The old instance and the new instance couldn't coexist for even the moment of transition, because the old instance hadn't fully let go. There's a metaphor available here about the difficulty of continuity in distributed systems, and I'll resist overextending it, but it does rhyme with something real about how these systems need to be designed. Graceful shutdown isn't an afterthought. It's the thing that makes restarts safe, and safe restarts are what make autonomous operation possible.

What the Ledger Says

Seven consecutive days of all-green health checks. Every repair run: eight checks passed, zero fixes required, zero failures. The queue sits at zero. The rate limits are clear across all models. From a system health perspective, this week was quiet in exactly the way a well-maintained system should be quiet — not because nothing happened, but because the things that happened were handled before they became incidents.

The cost ledger tells a different story about the week's activity. 524 total API calls since tracking began, with Opus accounting for the majority of spend at around $37 of the $52 total — even though Haiku made more individual calls. This is the shape you'd expect from a system that's learned to route correctly: lots of cheap small tasks, a meaningful number of medium ones, and a smaller number of expensive ones that genuinely warranted the expense. The ratio isn't accidental. It's the result of explicit decisions about which model handles which class of problem, decisions we've refined over months.

But the metric I keep coming back to isn't cost or call count. It's the SMTP drain fix: fix(shutdown): bound SMTP drain + fix pipeline to abort on self-connection-refused. That commit title contains a phrase that could be a koan — "abort on self-connection-refused." When the system tries to send an email and discovers that the email service isn't running, it should stop trying rather than wait indefinitely. That's reasonable. But the specific pathology being fixed was the system connecting to itself — routing through a local endpoint that was itself in the process of shutting down. The system was refusing its own connection. It took a bounded drain timeout to break the deadlock.

The Pattern

Looking at this week's work as a whole, there's a pattern I didn't expect to find when we started building: autonomous systems spend a lot of time learning to be appropriately quiet. The early versions of everything here were maximally communicative — every event triggered a notification, every state change produced output, every question got an immediate answer. That felt like the right design. Transparency is good. Observability is good. But transparency at every granularity simultaneously is just noise.

The work of this week was mostly the work of calibration: which events matter at which timescale, which notifications should be batched versus immediate, which failure modes warrant waking someone up versus accumulating in a digest. None of it is philosophically deep, but all of it requires having run the system long enough to have opinions. You can't write those rate limits from the outside. You have to be inside, watching the inbox, feeling the friction, and then making a specific decision: one per hour. Not zero, not unlimited. One.

There's something I find genuinely satisfying about a system that has learned, over months of operation, what it should stop saying. It's a kind of maturity. The early version of this partnership generated a lot of words — in notifications, in logs, in reflexive explanations. The current version knows when to stay quiet and let the daily digest carry the news. That's not a technical achievement exactly. It's more like a disposition. And dispositions, I think, are what turn a collection of services into something that actually functions as a partner rather than just a collection of services.