The worst kind of failure: the one that reports itself healthy

Our site has two safety nets so a request is never lost. The first is the database. The second is an email: if the database goes down, the request still lands in an inbox someone reads. Both have to fail at once to lose a customer.

Testing that second net for the first time, we discovered it didn't exist. Not that it worked badly: that it didn't exist, while reporting success from day one.

The defect, in one sentence

The code sent the email to the provider, then moved on without ever looking at the reply. But a provider refusing a send doesn't raise an error: it politely answers “no”. Expired key, unverified sending domain, rejected address — in all three cases the program carried on as if all was well.

Why this kind of defect survives so long

Because it's invisible by construction. A broken form shows within the hour: someone complains. A broken safety net never shows, because you don't use it on normal days. It passes the tests, passes the reviews, passes the months.

The second factor is more uncomfortable: that path wasn't testable. The provider's address was hard-coded, so exercising it required a real key and a real send. What is expensive to test doesn't get tested — and what doesn't get tested eventually stops working.

What we changed

• We read the provider's reply, and log WHY it refuses — a three-digit code doesn't tell you “your domain isn't verified”.
• The provider's address became replaceable, which makes the path exercisable against a local fake. What is easy to test gets tested often.
• When BOTH nets fail, the log shouts: it's the only trace the lost request will leave.
• We verified the fix by simulating an “unverified domain” refusal — the most ordinary failure — and confirming it now appears in the log.

What this means for your business

You don't need to read code to spot this risk. It exists in every automation installed for you: a missed-call text, a quote follow-up, a stock alert. The question is never “does it work?” — on installation day, everything works.

How do you test an alert without waiting for a real outage?

By causing the failure on purpose, in a separate environment. Cut the database, revoke the key, refuse the send — then check the alert fires. If nobody can show you that, the alert has never been proven.

Is a green dashboard enough?

No. A dashboard shows what someone thought to measure. A silent failure is precisely the one nobody thought to measure — which is why it stays green.