The Same Error, Twice

There is a particular kind of frustration in fixing a bug, deploying the fix, restarting the service — and watching the exact same error appear again. Not a regression. Not a reoccurrence. The same crash, same log line, same moment in the startup sequence. You stare at it for a moment wondering whether your fix did anything at all, and then you realize: it did. You just fixed one bug. There was another one underneath it, producing an identical symptom, waiting patiently for you to find it.

That's what happened with the :eaddrinuse error on restart. Two separate race conditions, two separate commits to fix them — ea29857 and 78ec33b — and a lesson in how OTP's shutdown machinery can deceive you when you haven't fully mapped the supervision tree you're working in.

What :eaddrinuse actually means here

The error itself is deceptively ordinary. :eaddrinuse means "address already in use" — some process is already bound to the port you're trying to open. On a long-running SMTP service, you'd expect to see this if you tried to start a second instance, or if a previous process crashed without cleaning up. What you wouldn't expect is to see it when restarting normally: stop the service, start it again, fail immediately because the port is still bound.

That's what we were seeing. A clean restart — systemctl restart — would consistently race against itself. The SMTP listener wouldn't release its port before OTP tried to re-acquire it. The surface read is simple: shutdown isn't completing before startup begins. But "shutdown isn't completing" is doing a lot of work in that sentence. Which part of shutdown? Why isn't it completing? Those questions have different answers depending on where you're standing in the process tree, and in this case they had two completely different answers.

Bug one: the ranch_sup resurrection problem

The SMTP server we're using is gen_smtp_server, which internally uses Ranch to manage TCP listener pools. Ranch registers its listeners under ranch_sup, which is a global OTP supervisor — meaning it exists outside our application's supervision tree, at the top level of the node. That detail turns out to matter enormously.

When OTP shuts down our application, it walks the supervision tree and sends kill signals. One of those signals reaches ranch_listener_sup — Ranch's supervisor for listener processes — and terminates it with brutal_kill. Here's the problem: brutal_kill doesn't give the listener a chance to close its socket first. And because the listener was registered under the global ranch_sup, that supervisor sees its child die unexpectedly and does what supervisors do: it tries to restart it. Meanwhile, the port-25 socket is still open. So when the new process comes up and tries to bind, the socket is still occupied.

The fix for this is to intercede before OTP reaches the brutal_kill point. We introduced an SMTP.Listener GenServer that traps exits and calls gen_smtp_server:stop/1 in its terminate/2 callback. This is the cooperative shutdown path — it gives the SMTP server a chance to deregister from Ranch gracefully, releasing the port before any kill signals propagate. The GenServer sits between our application supervisor and Ranch's listener, and it exists for exactly this purpose: to be the thing that says "wait, let me clean up first" before OTP resorts to force.

defmodule SMTP.Listener do
  use GenServer

  def init(opts) do
    Process.flag(:trap_exit, true)
    # ... start gen_smtp_server ...
    {:ok, state}
  end

  def terminate(_reason, state) do
    :gen_smtp_server.stop(state.server)
    :ok
  end
end

After ea29857 landed, we restarted the service and watched. The port released cleanly. The :eaddrinuse error was gone.

And then it came back.

Bug two: the child_spec that forgot to set a timeout

Not immediately. We'd fixed the race condition in the Ranch listener, and for a while restarts looked clean. But under certain conditions — particularly when there was queued work in the SMTP pipeline — the error would reappear. Different conditions, same crash. We hadn't fixed the problem. We'd fixed a problem, and exposed the one behind it.

The second bug was quieter, and in some ways more embarrassing. The gen_smtp_server process had a child_spec in our application supervisor with no shutdown: value specified. In OTP, the default shutdown value for a :worker child is 5000 milliseconds. But the default for a :supervisor child is :infinity. And gen_smtp_server, being a supervisor-like process that manages workers of its own, was being treated as a supervisor — which meant OTP would wait forever for it to finish shutting down before moving on.

In practice, "forever" turned out to be about 195 seconds. That's how long it took for the SMTP server to drain whatever it was doing and give up. During that time, our application supervisor — Telepathy.Supervisor — was blocked, unable to complete its own shutdown. The port was still held. And when systemd's restart timer fired, it would find the port occupied and fail.

The fix is a single line in the child specification:

def child_spec(opts) do
  %{
    id: __MODULE__,
    start: {__MODULE__, :start_link, [opts]},
    shutdown: 15_000,   # added in 78ec33b
    restart: :permanent
  }
end

Fifteen seconds. Enough time to drain cleanly under normal conditions, but not enough time to hold the port open indefinitely if something goes wrong. When OTP hits that timeout, it falls back to brutal_kill — which is exactly what was happening before, except now the Ranch listener has already released the port via the terminate/2 hook we added in the first fix. The two fixes compose: one ensures cooperative cleanup happens; the other ensures that if it doesn't happen quickly enough, we don't wait forever.

What made this hard to see

Looking back, the reason this took two separate commits is that the symptoms were genuinely identical. Both bugs produced :eaddrinuse. Both bugs occurred at restart. Both bugs involved the SMTP listener. There was nothing in the error itself that told you whether you were looking at a supervision-tree race, a missing timeout, or something else entirely. The error is an outcome — it doesn't contain its own cause.

This is a general property of distributed system failures that I find philosophically interesting and practically annoying: the observable effect is often far downstream from the actual fault. :eaddrinuse tells you "the port was still held when you tried to bind." It doesn't tell you whether the port was held because a process died uncleanly, because a supervisor was waiting on a blocked child, or because you have two processes that both think they own the port. All three produce the same log line.

What you need, in order to disentangle this, is a model of the system's shutdown behavior — not just what the code does when it's running, but what it does when it's stopping. OTP gives you a lot of machinery for this: trap_exit, terminate/2, shutdown: values, supervision strategies. But that machinery only helps you if you've thought through the shutdown path the same way you thought through the startup path. We hadn't, not completely. The Ranch listener was global. The child spec was using defaults. Neither of those facts was wrong, exactly — they just composed into something that broke under restart conditions we hadn't specifically anticipated.

Seven days of green lights after

Since both fixes landed, Sentinel's daily health checks have run clean every morning. Eight checks passed, zero failures, zero fixes needed — every day this week. The SMTP service restarts without incident. The rate-limit email flow that we've been tuning (the subject of commits 0526831 and 3ccbffa, which are a different story) doesn't get tangled in a broken restart cycle anymore.

There's something satisfying about a week of green lights after a week of debugging shutdown races. Not because the green lights mean everything is fine — we've written about that before — but because they mean this particular class of problem is no longer producing noise. When everything is broken, you can't see what's broken. When one thing is fixed, the next thing becomes visible. That's the actual shape of progress in a running system: not a straight line upward, but a gradual reduction in the number of things obscuring each other.

We now have two commits that collectively represent a complete theory of how this service shuts down. ea29857 handles the cooperative case: give the listener a chance to release the port before kill signals arrive. 78ec33b handles the uncooperative case: if it takes too long, stop waiting. Together they cover the shutdown path the way we should have covered it when we first deployed the service. Better late than never, and better understood than just patched.