Phishing is not what it used to be. The tell-tale signs that security awareness training taught people to look for, such as clumsy grammar, generic greetings and odd formatting, are disappearing. Attackers now use generative AI to write fluent, personalized messages in any language, and deepfake audio and video to impersonate executives and colleagues.

The result is an arms race. Attackers use AI to produce more convincing lures at greater volume and variety than ever, and defenders must respond with AI-driven detection, automation and processes that do not depend on a human spotting a fake. The future of email security is not about finding typos. It is about staying ahead in an AI-versus-AI contest while designing controls that still work when someone is fooled.

The numbers behind AI-powered phishing

Several studies show how quickly the threat has changed, although vendor figures should be read as indicators rather than precise measurements.

• Volume. Email security firm SlashNext reported a 1,265 percent increase in malicious phishing emails in the year after ChatGPT’s launch in late 2022.
• Prevalence of AI. KnowBe4’s 2025 Phishing Threat Trends Report found that 82.6 percent of phishing emails it analyzed between September 2024 and February 2025 showed some use of AI, and that 76.4 percent of campaigns used polymorphic techniques, generating variations of the same message to evade filters.
• Effectiveness. A study led by Harvard researcher Fred Heiding and colleagues found that fully AI-automated spear-phishing emails achieved a 54 percent click-through rate among participants, matching emails written by human experts and far above the 12 percent rate for generic phishing emails. The sample was small, 101 participants, but the finding is striking: AI now matches expert social engineers at a fraction of the cost.
• Impact. IBM’s 2025 Cost of a Data Breach report identified phishing as the most common initial attack vector, and put the global average cost of a data breach at 4.44 million dollars.

Why conventional defenses are falling behind

Traditional email security relies heavily on reputation and signatures: known bad senders, known malicious links, known phrases. AI undermines each of these. Every message can be unique, so there is no repeated pattern to block. Messages can reference real projects, colleagues and vendors scraped from public sources, so they pass the “does this make sense?” test. And thousands of variants can be launched in minutes.

Awareness training built around spotting mistakes is losing its footing for the same reason. Meanwhile, social engineering increasingly targets the controls meant to stop it. Attackers use real-time phishing proxies to capture one-time MFA codes, bombard users with push prompts, or call the help desk to have MFA reset. Deepfake voice and video add another layer: in early 2024 an employee at engineering firm Arup transferred about 25 million dollars after a video call in which the other participants were AI-generated impersonations of colleagues. For the broader picture of how AI changes each stage of an attack, see AI-powered threats: cyberattacks are getting smarter.

Fighting AI with AI

Defenders have access to the same technology, and it is effective when applied to the right problems.

• Behavioral anomaly detection. Modern email security uses machine learning to model normal communication patterns: who writes to whom, in what tone, with what kinds of requests. A first-time sender asking finance to change bank details, or a CEO emailing from an unusual location at 2 a.m., stands out even when the text is flawless.
• Language analysis. AI models can judge intent, such as urgency, secrecy or payment requests, rather than matching keywords, which catches business email compromise messages that contain no links or attachments.
• Automated threat intelligence. When one user reports a phishing message, automation can find and remove every similar message across all mailboxes in seconds, shrinking attacker dwell time.
• Adaptive security operations. AI-assisted triage in the security operations center groups related alerts, summarizes incidents and recommends responses, so a small team can keep pace with a high volume of attacks.

Controls that work even when someone is fooled

AI detection will never be perfect, so the strongest defenses are designed on the assumption that a convincing message or call will eventually get through. These measures address that:

1. Deploy phishing-resistant MFA. Passkeys and FIDO2 security keys cannot be relayed through a fake login page, which neutralizes credential phishing regardless of how convincing the email is.
2. Verify payments and changes out of band. Any request to move money, change bank details or reset credentials must be confirmed through a known phone number or a separate channel, never through the channel the request arrived on.
3. Harden help desk procedures. Require strong identity verification before resetting passwords or MFA, and log every reset for review.
4. Make reporting easy. A one-click report button, with fast feedback, turns employees into sensors for the automated response described above.
5. Update training. Teach staff that a perfect message, a familiar voice or a video call is not proof of identity, and that following the verification procedure is always acceptable.
6. Authenticate your email domains. Enforce SPF, DKIM and DMARC so attackers cannot send email that appears to come from your own domain.

The trade-off is friction. Callback procedures slow down urgent payments, stricter email filtering occasionally quarantines a legitimate message, and security keys require distribution and support. Those costs are modest and predictable. A single successful business email compromise or deepfake-enabled transfer can cost more than years of prevention, and many cyber insurers now ask specifically about MFA and payment verification controls when setting coverage.

Our article on social engineering and deception in 2025 covers the human side of these defenses in more depth.

Frequently asked questions

Can AI detection tools reliably spot AI-written phishing?

Not by detecting AI-written text alone, since legitimate emails are increasingly AI-assisted too. The better approach is behavioral: flagging unusual senders, requests and patterns regardless of who or what wrote the message.

Is security awareness training still worth it?

Yes, but its focus should shift from spotting errors to following procedures. Employees should know that verification steps apply to every sensitive request, no matter how convincing it looks or sounds.

How do we protect against deepfake voice or video calls?

Use pre-agreed verification procedures for payments and sensitive requests, such as calling back on a number already on file or using a code word, and never approve a transaction based only on a call or video meeting.

Stay ahead in the arms race

Delana Technologies helps organizations modernize email security, deploy phishing-resistant authentication and build verification processes that hold up against AI-generated deception. Explore our cybersecurity and compliance services, call 239.414.5126 or contact us.

---

Sources: SlashNext, State of Phishing Report (2023); KnowBe4, Phishing Threat Trends Report (March 2025); Heiding et al., “Evaluating Large Language Models’ Capability to Launch Fully Automated Spear Phishing Campaigns” (arXiv 2412.00586); IBM, Cost of a Data Breach Report 2025; CNN reporting on the Arup deepfake fraud (May 2024).