Postmortem: How Stale MX Records Created an Email Black Hole

The migration was done. Green checkmarks across the board. The cutover to Google Workspace was celebrated as a success on Thursday night, and Friday morning was deceptively calm. Then the first ticket landed.

It wasn't a flood, just a trickle. A user reported that a key vendor received a bounce-back. Another user couldn't receive a password reset link to their corporate account sent from a personal one. The reports were sporadic, inconsistent, and maddeningly difficult to reproduce. Mail was flowing for the most part, but every so often, a message would simply vanish into a black hole, leaving behind only a cryptic Non-Delivery Report (NDR) on the sender's side.

We were staring at a classic intermittent failure, the worst kind of problem in a system as complex as global email delivery. It wasn't a total outage, which is easy to spot, but a subtle, probabilistic failure that threatened to undermine the entire migration project.

The Phantom Menace: When SPF and DKIM Pass but Mail Fails

Our first instincts took us to the usual suspects: SPF and DKIM. A domain's email authentication policies are the bedrock of deliverability. A mistake here is the most common cause of post-migration pain. We immediately checked the domain's SPF record, defined in RFC 7208. It was perfect, containing the `_spf.google.com` include and nothing else. All legacy IP addresses and includes from the old provider were gone.

DKIM, governed by RFC 6376, was next. We sent test messages and inspected the headers. The `Authentication-Results` field was exactly what we wanted to see: `dkim=pass header.d=yourdomain.com`. Google was signing correctly, and the public key in the DNS `._domainkey` record was valid. DMARC (RFC 7489) was in `p=none` monitoring mode as planned for the transition, so it wasn't actively rejecting anything. From an authentication perspective, everything was clean.

This is where a less experienced team might get stuck. The authentication results told a story of success. The emails that *were* getting delivered had pristine headers. This wasn't a systemic misconfiguration of the authentication protocols. It was something else, something happening before authentication could even be evaluated. The problem wasn't with the *content* of the message, but with the *routing* to the destination mailbox server.

The 'Dig' That Revealed Everything

When you've ruled out authentication, the next stop on the OSI model is the transport and network layer. For email, that means DNS. Specifically, the Mail Exchanger (MX) records that tell the world where to send email for your domain. On a hunch, one of our engineers ran a simple command: `dig yourdomain.com MX`.

The result was our 'aha' moment. Staring back at us was a list of records from two different email providers.

yourdomain.com. 3600 IN MX 1 ASPMX.L.GOOGLE.COM.
yourdomain.com. 3600 IN MX 5 ALT1.ASPMX.L.GOOGLE.COM.
yourdomain.com. 3600 IN MX 5 ALT2.ASPMX.L.GOOGLE.COM.
yourdomain.com. 3600 IN MX 10 mx1.legacy-provider.net.
yourdomain.com. 3600 IN MX 20 mx2.legacy-provider.net.

The problem was immediately obvious. The priority 1 and 5 records pointed to Google Workspace, as they should. But the priority 10 and 20 records pointed to the old mail host we had just migrated away from. During the pre-migration checklist, someone had correctly *added* the new Google records but had failed to *delete* the old ones. They were left behind, a ticking time bomb.

How a Sending MTA Interprets a Split-Brain DNS

The Logic of MTA Fallback

To understand why this caused intermittent failures, you have to think like a sending Mail Transfer Agent (MTA), like Postfix or a large provider's outbound mail gateway. When an MTA wants to deliver a message to `user@yourdomain.com`, it performs an MX lookup.

It receives the list of hosts and their priorities. The MTA's logic is simple and resilient: always try the host with the lowest priority number first. In our case, that's `ASPMX.L.GOOGLE.COM` at priority 1. Under normal circumstances, the MTA connects to Google's server, delivers the email via SMTP, and its job is done. For 99% of the emails, this is exactly what was happening.

The Failure Scenario Unfolds

So where did the failure come from? The internet is not a perfectly reliable network. A sending MTA might fail to connect to `ASPMX.L.GOOGLE.COM` for any number of transient reasons: a temporary routing issue, a dropped packet, or even the receiving server being momentarily too busy to accept a new connection. This is normal. When this happens, the MTA's retry logic kicks in.

Instead of giving up, a well-behaved MTA will try the next-lowest priority server in the list. Seeing the Google priority 5 servers were also an option, it would try those. But what if, by some fluke, it decided to fall back to the priority 10 record, `mx1.legacy-provider.net`? The MTA has no idea this is a 'legacy' provider; to the machine, it's just another valid, advertised mail exchanger for the domain.

The MTA successfully connected to the legacy server. But that server, having been decommissioned, no longer had a mailbox for `user@yourdomain.com`. It correctly responded with a permanent failure: `550 5.1.1 User unknown`. The sending MTA, receiving this hard bounce, would not try any other MX records. It had received a definitive 'no' from a legitimate, advertised mail server. It then generated the NDR that was sent back to the original sender. This explained the intermittency perfectly. It only happened when a sending MTA failed to connect to Google's servers and fell back to the legacy ones.

The Decommissioning and Validation Checklist

The fix was simple, but the cleanup required discipline. Leaving orphaned DNS records is a common but dangerous oversight. A proper migration plan must treat the decommissioning of old records with the same importance as the creation of new ones.

Immediate Remediation

The first step was, of course, to log in to the DNS provider and delete the `mx1.legacy-provider.net` and `mx2.legacy-provider.net` records. This was the surgical fix that stopped the bleeding. Once the change was saved, we had to contend with DNS propagation.

Post-Change Validation

We didn't just assume the problem was solved. We used multiple external DNS lookup tools from different geographic locations to confirm that the old records were gone from the global DNS cache. We also lowered the TTL (Time To Live) on the remaining MX records to 300 seconds temporarily, allowing for faster changes if any other issues were discovered.

Finally, we set up monitors to specifically watch for SMTP errors related to `User unknown` and monitored inbound mail flow logs closely for the next 24 hours. The trickle of support tickets stopped. The black hole was closed.

The Cardinal Sin: Never Mix MX Records Across Providers

The core lesson here is architectural. MX records from different, independent email systems should never coexist for a single domain. It's not a form of redundancy; it's a state of conflict.

Using multiple MX records is standard practice for high availability, but only when all the listed servers belong to the *same email system*. Google's `ASPMX.L.GOOGLE.COM`, `ALT1.ASPMX.L.GOOGLE.COM`, and `ALT2.ASPMX.L.GOOGLE.COM` all talk to the same backend infrastructure. If the primary is down, the alternates can accept mail and queue it for the same set of mailboxes. They share a common state.

When you mix providers, you break this shared state. The legacy provider's MX server has no knowledge of the mailboxes on Google Workspace. To that server, the user truly does not exist. A sending MTA, following the rules of SMTP, has no choice but to trust that rejection. You are actively advertising a valid path to a guaranteed delivery failure.

You are not creating a 'backup' mail server. You are creating a fork in the road where one path leads off a cliff. — A very stressed-out Lead Engineer

The takeaway

A migration's success isn't measured at the moment of cutover, but in the stability that follows. This incident was a painful reminder that DNS hygiene is not a one-time setup task. It's a lifecycle management process. Decommissioning is just as critical as deployment. Your migration plan should have an explicit, mandatory line item: 'Delete all legacy MX, SPF, and DKIM records,' with a verification step to prove it was done.

This kind of configuration drift is precisely what automated monitoring is for. A simple check for multiple, top-level email providers in a domain's MX records could have flagged this issue before a single user was impacted. It's a check we've since built into our own processes, and it's the kind of continuous validation that platforms like MailSleuth.AI can provide, turning a multi-hour incident into a proactive, one-minute fix.