Anatomy of a Silent Outage: When an SPF `include` Goes Dark

The first alert wasn't from a dashboard. It was a Slack message from Sales: 'Hey, did BigCo get the proposal? They said they never received it.' One-off delivery problems are business as usual. But then a second report came in. And a third. Our open rates on a recent campaign looked unusually low, but not zero. Something was wrong, but it wasn't a hard-down outage. It was worse: a slow, silent bleed.

Initial checks showed green lights. Our primary mail gateways were up. DKIM signing was flawless. Our DMARC policy was at `p=reject`, and the aggregate reports trickling in from Google, Microsoft, and others showed no significant spike in authentication failures. By all conventional measures, our email posture was solid. Yet, a growing volume of our outbound mail was simply disappearing into the ether.

This is the story of a failure mode that lives in the gaps between RFC specifications and real-world MTA behavior. It’s a breakdown caused not by a misconfiguration on our end, but by a vendor's silent disappearance from the DNS, and how it spawned an SPF `PermError` that most of our security stack was blind to.

The Illusion of Safety: Why DMARC Stayed Quiet

The first place every security analyst looks for email trouble is DMARC reporting. DMARC, defined in RFC 7489, is our industry's best tool for domain-level email validation. It correlates SPF and DKIM results, checks for alignment, and tells receivers what to do with failures. Our `p=reject` policy should have meant any mail failing both SPF and DKIM would be blocked, and we'd get a report about it. So why were our RUA reports clean?

The answer lies in the specific type of SPF failure we were experiencing. SPF, defined in RFC 7208, doesn't just have `pass` and `fail`. It has a whole spectrum of results: `neutral`, `softfail`, `temperror`, and the one at the heart of our incident, `permerror` (Permanent Error). A `PermError` signals a configuration problem so severe that the SPF record can't be correctly interpreted. Common causes include syntax errors, invalid mechanisms, or exceeding the 10 DNS lookup limit.

Crucially, RFC 7208 states that a `PermError` should be treated with caution, often defaulting to a non-committal result or even being treated like a `Fail` by some interpreters. This ambiguity is the key. Many large mail providers, when faced with a `PermError`, don't want to risk dropping legitimate mail due to a recoverable misconfiguration. Instead of treating it as a hard `fail`—which DMARC would count and report—they treat it as an indeterminate state, effectively `None` or `TempError`. The email doesn't pass SPF, but it doesn't definitively fail either. Since our DKIM was still passing, DMARC alignment was a moot point, and the message sailed past the DMARC check, only to be judged harshly on its other merits (like the ambiguous SPF result) by downstream spam filters, leading to inbox delivery failure or silent drops.

Chasing Ghosts in DNS: The `dig` Investigation

Finding the Faulty Chain

With DMARC reports offering no clues, we had to go manual. The problem affected multiple outbound services, all of which pointed to one common denominator: a marketing automation platform we'll call 'EmailVendor'. Their setup guide had us add `include:spf.emailvendor.com` to our domain's SPF record. This is standard practice; it delegates part of our SPF authentication to the third party who sends mail on our behalf.

My first step was to query our own SPF record.

$ dig txt ourdomain.com

;; ANSWER SECTION:
ourdomain.com. 3600 IN TXT "v=spf1 include:_spf.google.com include:spf.emailvendor.com ~all"

It looked fine. The syntax was correct. The problem wasn't in our record itself. The next step is to test the chain. I queried the first `include`.

`dig txt _spf.google.com` returned a valid list of Google's netblocks. No problem there. Then I checked the suspect.

The Smoking Gun

`dig txt spf.emailvendor.com`... and there it was. Not a `PermError`, not a `softfail`, but something much simpler. The command returned nothing. No `TXT` record. No `SOA` record indicating authority. Just a blank, empty `NXDOMAIN` response from the DNS resolvers. The subdomain we were instructed to `include` simply ceased to exist.

This is a catastrophic failure. When a mail receiver's SPF processor tries to evaluate our record, it performs a DNS lookup for `spf.emailvendor.com` as part of the `include` mechanism. When that lookup fails entirely, the processor can't proceed. It throws a `PermError` because the record is unresolvable. The chain is broken. This wasn't a subtle misconfiguration; the vendor had seemingly decommissioned a critical piece of their infrastructure without notice.

The Blast Radius: SPF as Supply Chain Risk

The discovery that `spf.emailvendor.com` was a ghost immediately clarified the scope of the incident. This wasn't just about marketing campaigns. We used that vendor for transactional email, too. Password resets, welcome emails, security alerts—all of it was being sent via their infrastructure.

This highlights a fundamental, often overlooked aspect of email security: your SPF record is a map of your digital supply chain. Each `include` statement is an expression of trust. You are not just delegating email sending; you are tethering your domain's reputation to the operational discipline of another company. If their DNS admin makes a mistake, if they get acquired and bungle the infrastructure migration, if they simply go out of business—it's your deliverability that pays the price.

In our case, the vendor had been acquired months prior, and this DNS record was a casualty of a clumsy platform consolidation. They had sent out an advisory email, of course. But it was a low-priority 'product update' notification that landed in a shared marketing inbox, never triaged as a critical operational change. A single missed email from a vendor led to a significant deliverability incident for us.

A Playbook for Proactive SPF Auditing

Waiting for users to report missing emails is not a monitoring strategy. This incident forced us to build a proactive defense against SPF record degradation. You can't control your vendors, but you can aggressively monitor the dependencies you have on them.

Audit the `include` Chain Manually

At a minimum, you must periodically walk your own SPF record just as a receiving mail server would. Use `dig` or an online SPF validator to trace every `include`, `a`, `mx`, and `redirect` mechanism. Does every DNS query resolve? Do any of them point to records that are themselves close to the lookup limit? Document this chain of dependencies. Know who owns each link.

Beware the 10-Lookup Limit

Our issue was a non-existent record, but a far more common `PermError` comes from exceeding the 10 DNS lookup limit imposed by RFC 7208. A single `include` might seem harmless, but if that included record itself contains more includes, the lookup count can spiral quickly. This is called SPF record flattening, and it's a constant battle. A vendor might add a new `include` to *their* record, suddenly pushing *your* record over the limit without you changing a thing. This is another reason to monitor dependencies, not just your own record.

Automate the Audit

Manual checks are good for deep dives, but they don't scale. The only sustainable solution is automated monitoring. This can be as simple as a cron job running a shell script that uses `dig` to query each dependency in your SPF record and alerts you on `NXDOMAIN`, `SERVFAIL`, or other DNS errors. It should also count the total lookups required to resolve the full record and warn you when you're approaching the limit (say, 8 or 9 lookups).

The takeaway

The immediate fix was simple: we removed `include:spf.emailvendor.com` from our SPF record and transitioned all critical mail streams to another provider. The post-incident work was more profound. We now treat our SPF record not as a static configuration file, but as a living dependency tree that requires constant, automated validation.

This incident was a stark reminder that email authentication is not 'set and forget.' Your domain's reputation is only as strong as the weakest link in your authentication chain—and often, that weak link is a third-party service you don't control. Don't wait for your sales team to be your primary alerting system. Automate your monitoring, validate your dependencies, and assume your vendors will eventually fail. Tools like MailSleuth.AI's SPF record validator can help automate this process, but the principle is what matters: own your dependencies, or they will own you.