The last couple of years have felt different in security, though it took time to understand why. There wasn’t a single moment you could point to. No headline that captured it. Just a growing sense that something subtle had shifted.
When generative AI became widely accessible in late 2022, it felt more like a novelty than a threat. Useful, sometimes impressive, occasionally concerning, but not something that fundamentally changed how attacks worked. Early signs in 2023 reinforced that perception. Phishing volumes increased, content generation sped up, and low-effort campaigns multiplied. These were familiar patterns, just accelerated.
Over time, the pattern became harder to ignore.
I spent a significant amount of time trying to understand what was actually happening beneath the surface. I didn’t read every report end to end, but I paid close attention to the ones that mattered — incident write-ups, vendor disclosures, academic research, and, occasionally, dark-web discussions where criminals were far more honest about what worked and what didn’t. When you stack enough of those signals together, a clear trajectory emerges.
This analysis draws from intelligence documents, documented incidents, and roughly $30–40 billion in tracked losses across G7 nations. The data isn't perfect. Some cases overlap. Some losses are estimates. But when the same patterns appear across vendor reports, academic studies, law enforcement disclosures, and occasional dark-web marketplace intelligence, the signal becomes hard to dismiss.
AI didn’t arrive fully formed as a weapon. It was adopted gradually, first as an aid, then as a multiplier, and eventually as something approaching an operator. Capabilities that once required time, skill, and coordination collapsed into a single workflow. Tasks that used to limit attackers quietly stopped doing so.
Two years ago, realistic, adaptive, high-volume campaigns required significant effort. Today, much of that effort has disappeared. Not because attackers suddenly became more capable, but because AI absorbed much of the complexity.
Why This Period Matters
None of these attack techniques are new. Phishing, voice fraud, malware, and social engineering have existed for decades.
What changed is the simultaneous removal of three constraints:
- Cost collapsed. Operations that required $50,000–$100,000 in infrastructure and expertise now cost around $5,000 — an 80–90 percent reduction. Criminal AI subscriptions start at $200 per month.
- Time collapsed. Week-long campaign preparation compressed into minutes. Trial-and-error loops that once took days now complete in seconds.
- Skill collapsed. Capabilities that required years of expertise became accessible through point-and-click interfaces and natural language commands.
When cost, time, and skill constraints collapse simultaneously, attack volume and sophistication scale exponentially. That's what makes 2023–2025 structurally different from prior automation waves. The techniques aren't new. The absence of limiting factors is.
What changed wasn't just phishing.
Voice cloning bypassed authentication systems. Deepfake video fooled finance teams on conference calls. Malware began dynamically generating and adapting evasion techniques in real time. Multi-modal attacks combined text, voice, and video to overcome every layer of verification simultaneously. By September 2025, Anthropic assessed one nation-state operation as reaching high degrees of autonomy — estimated at ~80–90% of lifecycle steps executed without direct human input — while humans retained approval gates and managed operational security.
This blog is Part 1 of a three-part series. The goal here is to establish the timeline — who adopted AI first, how usage evolved across multiple attack domains, and where the shift from experimentation to real weaponization occurred. Part 2 will examine the technical vulnerabilities that made this progression possible. Part 3 will focus on what defenders can realistically do now that many of the old assumptions no longer apply.
Key Terms Used in This Analysis
Autonomy
The percentage of attack lifecycle steps executed by AI without direct human input per decision. Humans may still provide strategic direction, approval gates, and operational security oversight.
Example: Anthropic assessed GTG-1002 as operating at an estimated ~80–90% autonomy — meaning most reconnaissance, vulnerability scanning, and exploitation steps occurred without requiring human decision-making between each action, though humans retained oversight and approval authority at key decision points.
Weaponization
AI integrated into live operational attack workflows, not just content generation or research. Weaponized AI executes actions directly—scanning networks, deploying malware, exfiltrating data — rather than generating reports for humans to act on.
Example: PROMPTSTEAL queries LLMs in real time during active infections to generate evasion techniques.
Multi-Modal Attack
Operations combining multiple AI-generated media types (text, voice, video) in a single coordinated effort.
Example: UNC1069 used AI-generated Spanish text for social engineering, deepfake video to impersonate executives, and voice synthesis for phone verification — overcoming language barriers and visual trust signals simultaneously.
Operator vs Multiplier
A multiplier accelerates human-led tasks (2023: AI helped attackers work faster). An operator executes tasks autonomously (2025: AI conducts reconnaissance and exploitation while humans provide strategic oversight).
The shift from multiplier to operator is the central thesis of Part 1.
1. The Early Days: When AI Was Just Noise
The shift did not announce itself.
There was no alert, no clear inflection point, no moment when anyone in security could confidently say that everything had changed. Instead, there were small inconsistencies. Phishing emails looked slightly cleaner than usual. Malware behaved just off-pattern. Incidents felt familiar but moved faster than expected.
At the time, these signals were easy to dismiss. Security teams encounter anomalies every day, and most of them mean nothing.
Looking back, this was the first signal, but it did not register as such when it was happening.
November 2022: When the Door Quietly Opened
When ChatGPT was released in late 2022, it wasn’t framed as a security event. It was a product launch. A novelty. A glimpse of what conversational AI might look like in the future. For defenders, there was no immediate reason to worry. The model had guardrails. It refused malicious requests. It hallucinated. It made obvious mistakes.
Attackers noticed something else.
What mattered wasn’t that the model was perfect. It wasn’t. What mattered was that it was accessible, fast, and free. Within weeks, phishing volumes surged dramatically. Not because the messages were sophisticated, but because they were effortless to produce. Language barriers disappeared overnight. Grammar stopped being a limiting factor. Time stopped being a constraint.
This wasn’t weaponization yet.
It was acceleration.
And at this stage, defenders still had the upper hand.
2023: AI as a Crutch, Not a Threat
Throughout 2023, AI stayed firmly in the category of “useful but flawed.” Criminals used it the same way many legitimate users did: drafting emails, translating content, summarizing data, speeding up research.
Studies from that period showed AI-generated phishing was still significantly less effective than human-crafted messages. Faster, yes. Convincing, no.
From a defensive perspective, this reinforced the right instincts. Detection models adapted. Security awareness training evolved. Familiar signals were still present, shallow context, generic phrasing, subtle awkwardness. AI had not learned nuance yet.
Most importantly, humans were still clearly in control.
AI helped attackers move faster, but it did not replace them. That distinction mattered. The threat felt manageable.
What we underestimated was how much leverage speed alone provides when everything else stays constant. We assumed quality would remain the deciding factor. We did not yet see how quickly scale would rewrite the rules.
2. The Shift: When Scale Started to Matter
By early 2024, something subtle had changed.
AI was no longer confined to surface-level acceleration. It started to appear in parts of operations where repetition mattered more than creativity. This shift was not obvious in any single incident. It emerged slowly, across intelligence reports, vendor disclosures, and patterns that only became visible when you stepped back far enough.
That’s when the first serious signals appeared.
Early 2024: AI Becomes Operational
In February 2024, Microsoft and OpenAI published the first public attribution of nation-state actors using AI in real operations. Five groups were named. At the time, the findings were deliberately cautious. AI was described as an assistant, not an operator, used for research, coding help, and OSINT acceleration, not autonomous deployment.
That framing was accurate, but incomplete.
What mattered wasn’t what the models were doing yet. It was where they were being introduced.
Reconnaissance pipelines. Vulnerability research. Malware development workflows. In those environments, AI didn't need to be exceptional. It needed to be tireless.
Nation-state groups treated AI like a capable intern. Humans still made the decisions. Humans still executed attacks. But the slow, repetitive groundwork compressed dramatically.
This was the moment scale quietly entered the equation.
Criminal Ecosystems Catch Up
Organized cybercrime moved even faster.
While nation-states experimented carefully, criminal markets iterated aggressively. Dark web tooling matured.
Jailbreak techniques became reliable.
Independent analysis of GitHub identified 285 documented jailbreak repositories (November 2024–November 2025). Separate testing by Cisco Talos found multi-turn jailbreak attacks achieved success rates between 25.86% and 92.78% across different open-weight models, with Mistral Large-2 and Alibaba Qwen3-32B reaching the highest vulnerability at 92.78% (November 2025). Success rates vary by model, safety tuning, and evaluation criteria (single-shot vs multi-turn, target policy, and test harness), so these figures describe the state of public tooling and test results— not a universal bypass guarantee.
At the same time, unrestricted models eliminated the need to bypass guardrails entirely. WormGPT, FraudGPT, DarkBERT. Purpose-built for crime, no jailbreak required. Open-source repositories multiplied. Tools that once required deep expertise became usable through point-and-click interfaces. None of this made headlines on its own, but together it lowered the barrier to entry at an alarming rate.
What changed was not sophistication.
It was throughput.
Single operators could now run multiple campaigns in parallel. Research that once took hours collapsed into minutes. Trial-and-error loops tightened. Failure stopped being costly.
At this stage, humans were still in control of execution. AI accelerated preparation and iteration, but decision-making remained human-led.
The economics shifted faster than most noticed. Operations that once required $50,000–$100,000 in infrastructure and expertise over six months now cost around $5,000. Criminal AI subscriptions started at $200 per month. The barrier to entry collapsed by an estimated 80–90 percent.
Signals We Underestimated
Looking back, the warning signs were there.
Jailbreak success rates improved rapidly. Open-weight models degraded under sustained multi-turn interaction. Context windows expanded far beyond what most security evaluations accounted for. At the same time, the volume of AI-assisted attacks increased without a corresponding spike in obvious indicators.
Defenders noticed attacks moving faster, but speed alone rarely triggers alarms. We are trained to look for novelty, not acceleration. We focus on new techniques, not the quiet removal of constraints.
By the end of 2024, AI had not replaced human attackers. But it had already reshaped the economics of offense. Preparation became cheap. Iteration became effortless. Scale stopped being limited by headcount.
The system was under stress long before it visibly broke.
3. March 2025: When AI Surpassed Humans
March 2025 is easy to overlook in hindsight. There was no dramatic breach. No singular campaign that dominated headlines. But from a defensive perspective, this was the moment one of our last assumptions stopped being true. For the first time, AI-generated phishing outperformed human-crafted phishing in a controlled, large-scale study. Not marginally. By 24 percent.
Two years earlier, the opposite had been true. In 2023, AI phishing was roughly 31 percent less effective than human efforts. The swing between those two points is the part that matters. Over roughly 24 months, the gap did not close. It inverted. A 55-point swing occurred without any corresponding shift in how most defenses were designed.
This was the first time an AI system surpassed humans at a task we considered inherently human.
Beyond Phishing: The Broader Pattern
The March 2025 phishing breakthrough wasn't isolated.
It signaled something larger. AI was crossing effectiveness thresholds in multiple domains at roughly the same time. Phishing was simply the first case where measurement was straightforward.
Voice Cloning and Deepfake Fraud
By 2025, voice cloning had matured from a proof-of-concept threat to an operational one. More than 3,500 documented incidents. Over $1 billion in confirmed losses according to combined FBI and Europol reporting. Training requirements dropped from hours of audio to under a minute. Quality became indistinguishable from human speech in controlled tests.
The most significant single case occurred in February 2024. A finance team at a multinational corporation participated in what they believed was a routine video conference with their CFO and four other executives. They verified identities visually. They heard familiar voices. They followed standard approval procedures.
All five participants on the call were deepfakes. The attackers walked away with $25.6 million.
By November 2025, the attack surface expanded beyond financial fraud. The first documented voice cloning compromise of an IT helpdesk occurred in Spain. An attacker used a cloned employee voice to request system access. The threat was no longer confined to wire transfers.
Malware Generation
Nation-state actors began deploying AI-powered malware frameworks operationally.
Russia's APT28 weaponized a system called PROMPTSTEAL, which queries open-source language models in real time to adapt evasion behavior. Google's Threat Intelligence Group documented five similar families by late 2025: PROMPTFLUX, PROMPTSTEAL, FRUITSHELL, PROMPTLOCK, QUIETVAULT. These systems don't rely on pre-programmed evasion logic. They generate it dynamically, making signature-based detection increasingly unreliable.
Multi-Modal Operations
North Korea's UNC1069 group conducted what appears to be the first fully multi-modal AI attack. Text generation for social engineering. Deepfake video for visual verification. Voice synthesis for audio confirmation. The operation targeted cryptocurrency executives and overcame both language barriers and visual trust signals in a single coordinated effort.
The phishing tipping point mattered because it was measurable.
But it wasn't unique. AI was surpassing human effectiveness across the entire attack lifecycle. March 2025 was simply the moment we could no longer ignore the pattern.
Phishing has always been treated as a human problem. Language, tone, timing, context. Defenders built controls around the idea that attackers would eventually make mistakes. Unnatural phrasing. Cultural mismatches. Sloppy personalization.
Those assumptions were quietly invalidated.
At the same time, the underlying technology changed in ways that compounded the impact. Long-context models arrived. Context windows expanded from thousands of tokens to hundreds of thousands, and in some cases beyond a million. That did not just make phishing better. It made it scalable.
An AI model could ingest inboxes, public profiles, and documents in a single pass. It could map relationships. Track conversations. Generate unique, highly personalized messages for large target sets at once. What used to be careful, manual social engineering became a batch process.
Quality and scale crossed the threshold together.
The Quiet Collapse of a Defensive Advantage
This is where the defender advantage began to erode structurally.
Not because attacks became more creative, but because they became indistinguishable from legitimate communication at scale. Linguistic signals lost reliability. Fatigue disappeared. Human error was no longer a limiting factor.
At the time, this shift didn’t feel catastrophic. Phishing was already a problem. Losses were already high. From the outside, March 2025 looked like another data point in a long-running trend.
From the inside, it was the moment the system tipped.
Everything that followed, autonomy, economics, industrialized attacks, was downstream of this break.
4. From Assistance to Autonomy
Once AI crossed the human effectiveness threshold, the rest unfolded quietly and quickly. Not because attackers suddenly became more ambitious, but because restraint stopped making economic sense.
By mid-2025, the criminal AI ecosystem had matured into a functioning dark web AI marketplace generating an estimated $20–40 million annually. For a few hundred dollars a month, attackers could access tools that automated research, generated malware variants, personalized phishing at scale, and adapted in real time. Entry costs dropped dramatically. Skill stopped being the limiting factor.
At that point, humans were no longer the most efficient part of the operation.
Autonomy wasn’t a leap. It was optimization.
What Made Autonomy Possible
Long-context models provided scale, but they weren't sufficient for true operational autonomy.
That required infrastructure.
The GTG-1002 operation relied on what's called Model Context Protocol, a system that allows AI to access external tools. Not just analyze text, but invoke network scanners, web scrapers, and exploitation frameworks directly. The AI didn't just recommend actions. It executed them.
Retrieval-Augmented Generation played a similar role. Instead of relying entirely on context windows, these systems query external knowledge bases in real time. Exploit databases. CVE repositories. Attack technique documentation. The effective knowledge base becomes unlimited.
Agent orchestration frameworks like LangChain and AutoGPT tie it all together. Reconnaissance leads to vulnerability discovery. Vulnerability discovery generates exploits. Exploits deploy payloads. Payloads enable lateral movement. Each step feeds into the next without human coordination.
Long context plus tool access plus orchestration.
That combination is what enabled the autonomy threshold.
Part 2 will examine why these systems turned out to be far less secure than their designers expected.
September 2025: Crossing a Visible Threshold
That optimization crossed a visible threshold in September 2025.
Anthropic assessed one Chinese state-sponsored operation (GTG-1002) as conducting cyber operations with an estimated ~80–90% autonomy — meaning most reconnaissance, network mapping, asset discovery, vulnerability scanning, exploit selection, and deployment steps occurred without direct human input between actions, while humans retained approval authority and managed operational security across roughly 30 target organizations.
The operation ran at machine speed. Thousands of requests. Multiple per second. Execution that would have been physically impossible for human operators to coordinate manually.
Humans retained final approval authority and managed operational security. But the work itself, the actual execution, had become machine-driven.
Several researchers later questioned whether this constituted true operational autonomy, citing limited disclosure and the continued presence of human decision points.
That criticism is valid.
The significance isn’t that fully autonomous cyber operations are now routine. They aren’t. The significance is that the technical and economic conditions required for autonomy are largely in place. The remaining gaps are narrower than many defenders assume.
Once execution becomes autonomous, scale becomes unbounded. Single operations fragment into dozens of seemingly unrelated incidents. Controls built around human-paced threats begin to fail, not because they are poorly designed, but because their assumptions no longer hold.
Why This Matters for Security Teams
If you work in a SOC, the most important takeaway is this: You are no longer racing human attackers. You are racing orchestration pipelines that run at machine speed, never get tired and never repeat the same mistake twice.
And we haven’t even hit the peak yet.
Where We Go Next
Part 2 will dig into the technical reality behind all this. The jailbreak crisis. The failures in model context isolation. The first C2-less autonomous malware families. All the mechanisms that allowed attackers to leap forward while defenders were still updating playbooks from 2021.
Part 3 will be the hard one. We will talk about what defenders can actually do now, in a world where attacker speed is measured in seconds, not hours.
For now, the takeaway is simple: AI didn’t just accelerate cybercrime. It changed the nature of it. It turned tools into weapons, and attackers into operators of increasingly autonomous systems.
We’re not fighting humans alone anymore.
We’re facing systems that already operate at machine speed — and with each model generation, less human intervention is required to keep them running.
To be clear, defenders are using AI too. Threat detection. Incident response. Vulnerability management. The tools are available to both sides.
The asymmetry isn't capability. It's economics.
A single attacker with AI can target thousands of organizations simultaneously, iterating in parallel, adapting in real time. Defense doesn't scale the same way. That imbalance is what makes this shift structural, not temporary.
---
Data Limitations
We acknowledge several limitations:
- Financial loss figures rely on victim self-reporting (underreporting likely)
- Attribution for some incidents (especially hybrid actors) remains uncertain
- GTG-1002 autonomy claims lack independent verification
- Dark web user counts are estimates based on marketplace analytics
Reference
- Anthropic. (2025, November 13). Anthropic disrupts state-sponsored cyber operation. Anthropic Security Blog.
- Hoxhunt. (2023, 2025). AI Phishing Effectiveness Studies. Multiple reports.
- SPRF India. (2025, July). Digital Threats Analysis. Research report.
- SlashNext. (2023). Generative AI & Cybersecurity and related press coverage on post-ChatGPT phishing volumes (e.g., Decrypt).
- Microsoft Threat Intelligence & OpenAI. (2024, February 14). Staying ahead of threat actors in the age of AI. Microsoft Security Blog.
- Google Threat Intelligence Group. (2025, November). Advances in threat actor usage of AI tools. GTIG Report, 18 pages.
- Zscaler ThreatLabz. (2024). 2024 ThreatLabz AI Security Report. 536.5B AI/ML transactions analyzed (Feb-Dec 2024).
- ENISA. (2024). Threat Landscape Report 2024. European Union Agency for Cybersecurity.
- Cisco Talos. (2024, November). Death by a Thousand Prompts: Multi-Turn Jailbreak Success Rates. Research paper.
- Author's analysis. (2025). Dark web AI marketplace intelligence. Based on Group-IB, Recorded Future, Trend Micro data.
- Dark web pricing intelligence. (2025, June–November). WormGPT marketplace analysis.
- Author's analysis. (2025, November 24). Long Context Window Models – Security Implications Analysis.