Beyond Bad Grammar: Hunting AI-Generated Phishing

An urgent wire transfer request hits your CFO’s inbox. It’s from a known vendor, the language is pitch-perfect, and it references a real project. There are no spelling mistakes, no weird formatting. Everything about the text screams legitimate. Yet, something feels off. This is the new front line of email security, where the most dangerous phish isn't the one with bad grammar—it's the one with none.

For years, we taught users to spot phishing by looking for linguistic errors. Awkward phrasing, typos, and cultural mistranslations were reliable red flags. Large language models (LLMs) have incinerated that playbook. Attackers can now generate flawless, context-aware, and even stylistically-mimicked prose at scale. The tells we relied on are gone.

This means the SOC analyst's job—and the threat hunter's—has moved deeper into the stack. If the content of the email is a perfect mask, we have to rip the mask off and inspect the machinery underneath. The real clues are no longer in the body content; they're in the headers, the infrastructure, and the subtle metadata that AI can't easily fake.

The Unnatural Valley of Perfect Prose

The first shift in mindset is recognizing that 'perfect' can be a signal of its own. Human communication is messy. We use slang, create typos, and have individualistic tics in our writing. AI-generated text, particularly from base models with minimal prompt engineering, is often sterile. It’s grammatically flawless but lacks a human fingerprint.

Think of it as an 'unnatural valley' of prose. It's so clean it feels synthetic. The tone might be slightly too formal or too generic for the supposed sender. An email asking for an urgent favor might lack the chaotic punctuation or brevity a real, stressed executive would use. It reads less like a person and more like a press release.

This gets more complicated with targeted Business Email Compromise (BEC) attacks. With a few samples of a target's real emails, an attacker can fine-tune an LLM to mimic their style, vocabulary, and even common mistakes. But this is still difficult to execute perfectly. The AI might replicate the style but fail to reference a recent, private conversation, creating a social disconnect that a savvy recipient might notice. The core takeaway remains: the absence of error is now, itself, a data point to consider.

Ghost in the Machine: Metadata and Structural Clues

When the prose is a dead end, the next place to hunt is the email's structure and metadata. Every email is more than just text; it's a collection of structured data governed by RFCs, and AI-generation tools often cut corners here.

Anomalous Threading and Timestamps

Real email conversations have a history baked into their headers. The `Message-ID`, `In-Reply-To`, and `References` headers create a logical chain. In a sophisticated AI campaign that spoofs a reply, these headers have to be perfect. Often, they aren't. An attacker might generate a subject line that starts with "Re:" but forget to include the `In-Reply-To` and `References` headers of the original message. To a mail client, this isn't a reply; it's the start of a new conversation. That’s a huge red flag.

The `Received` headers are another goldmine. As an email traverses mail transport agents (MTAs), each one stamps the message with a timestamp. You can trace the entire path. Look for impossible transit times—seconds to cross continents—or hops through unusual geographic locations. If an email supposedly from your US-based vendor routes through a server in a country you never do business with, it deserves scrutiny.

The Tell-Tale MIME Structure

Beneath the rendered email is the Multipurpose Internet Mail Extensions (MIME) structure. This defines how the text, HTML, and attachments are pieced together. Emails sent from standard clients like Outlook or Gmail have a characteristic, often complex, MIME structure. They might include `text/plain` and `text/html` parts, calendar invites as `text/calendar`, and have specific boundary markers.

Emails generated by simple Python scripts or basic APIs, which are common for launching AI-driven campaigns, often have an unnaturally simple MIME structure. You might see only a single `text/html` part with no plain-text alternative, or a generic, computer-generated boundary string. It's the digital equivalent of a forged document with the wrong kind of paper. It looks right on the surface, but an expert can feel the difference.

Following the Money Trail: Infrastructure as a Fingerprint

An attacker with a perfect AI lure still faces a fundamental obstacle: email delivery. Sending a message that lands in the inbox requires infrastructure, and infrastructure leaves a trail. This is often the weakest link in an AI-assisted attack chain.

Threat actors favor two approaches: spin up new domains or compromise existing ones. A brand-new domain, registered hours before the campaign launch, is a classic indicator. It will have no history, no reputation. Even if the attacker correctly configures SPF (RFC 7208) and DKIM (RFC 6376), allowing the email to technically pass authentication, the sending domain's youth and the IP's lack of a positive reputation are massive signals for any modern email gateway.

Alternatively, attackers use credential stuffing or vulnerabilities to take over legitimate email infrastructure—a small business's Microsoft 365 tenant or an unused account on a commercial email service provider like AWS SES. In this case, the authentication might be technically valid. But other signals emerge. Is this sender's IP address block associated with a residential ISP or a cloud provider not typically used for corporate email? Is the Autonomous System Number (ASN) a known source of spam? The reputation of the infrastructure is a community-defended moat that AI can't just write its way across.

Reading the Tea Leaves: Header Analysis for AI Threats

This is where the real deep-dive investigation happens. The `Authentication-Results` header is the final report card from the receiving mail server, and learning to read it is a non-negotiable skill for a security analyst.

The DMARC Alignment Problem

A common mistake is seeing `dkim=pass` and `spf=pass` and assuming the email is legitimate. This is where DMARC (RFC 7489) alignment becomes critical. DMARC doesn't just check if SPF and DKIM passed; it checks if they passed for the *correct domain*. The domain in the `From:` header—the one the user sees—must align with the domain used for SPF authentication or the domain in the DKIM signature (`d=`).

Authentication-Results: mx.google.com; dkim=pass header.d=bounces.evil-send.net; spf=pass smtp.mailfrom=attacker@evil-send.net; dmarc=fail (p=REJECT sp=REJECT dis=REJECT) header.from=yourbank.com

In the example above, DKIM and SPF both passed for the attacker's domain (`evil-send.net`), but the `From:` header showed `yourbank.com`. Because the domains don't align, DMARC fails, and the receiving server knows this is a forgery. This is the single most powerful defense against domain spoofing, and it's a purely technical check that is immune to how well-written the email body is.

Broken Forwarding Chains and ARC

Legitimate email forwarding, like a calendar invite sent from a vendor that gets forwarded to your team, famously breaks SPF. Some mail servers also make minor modifications (like adding a footer) that break the DKIM body hash. This is a headache. To solve it, the Authenticated Received Chain (ARC), defined in RFC 8617, was created. ARC creates a cryptographic seal (`ARC-Seal`) over the original authentication results, preserving them across hops.

When you, as an analyst, see an email with failing SPF and DKIM, your next move should be to check for a valid ARC chain. If a valid `ARC-Seal` and `ARC-Authentication-Results` are present, it's strong evidence that the message was legitimate at its origin and simply broke during transit. Conversely, if an email looks like a forward but has no ARC headers, it's suspicious. Attackers launching complex campaigns are less likely to bother with correctly implementing ARC, making its absence a potential tell.

Hardening the Inbox: Practical Defenses

Defending against AI-generated phishing isn't about buying a magic AI detection tool. It’s about doubling down on foundational email security protocols and augmenting them with behavioral intelligence. The best defense is a layered one.

First, enforce DMARC. Moving from monitoring (`p=none`) to a quarantine or reject policy (`p=quarantine` or `p=reject`) is the single most effective step to prevent direct domain spoofing. Once DMARC is in place, implement MTA-STS (RFC 8461) to force opportunistic TLS encryption for your inbound mail, preventing downgrade attacks that could allow an attacker to intercept or modify emails in transit.

Second, look at behavior and context. Modern security platforms don't just analyze one email; they analyze relationships. Is this the first time this sender has ever emailed the CFO? Is this sender's domain only a day old? Is it normal for your vendor's accounting department to send invoices as encrypted ZIP files with a link to a password on Pastebin? The AI can write a perfect email, but it lives in a vacuum. It lacks the organizational context that exposes its requests as anomalous.

Finally, the payload is still a weak point. The email body is just the lure. The attacker still needs you to click a link, open an attachment, or scan a QR code. Aggressively analyze these payloads. Un-shorten URLs, detonate files in a sandbox, and inspect QR codes that lead to credential harvesting pages. The sophistication of the lure has increased, but the underlying mechanics of the attack often remain the same.

The takeaway

The rise of AI-powered phishing forces us to stop being content experts and become systems analysts. We can no longer trust what the email says. We have to investigate what the email *is*. The evidence is in the transport chain, the authentication layers, and the infrastructure reputation—not the prose.

Manually piecing together `Received` hops and DMARC alignment for every suspicious email is unsustainable for a busy SOC. This is why modern analysis platforms, including our own MailSleuth.AI, are built to automate that grunt work, surfacing the key header and infrastructure anomalies for the analyst. The game has moved from spotting human error to finding logical flaws in a machine's work. The winning question is no longer 'Does this email *read* right?' but 'Does this email *work* right?'