AI Slop Isn’t Just for Social Media; It’s Ruining Cyberattacks, Too

Don’t buy the hype! AI has been dumbing down cyberattacks, and serious threat actors are barely touching it.

Some prominent, even state-sponsored threat actors are giving up on trying to build cyberattack campaigns with artificial intelligence (AI), contrary to the popular notion that AI is pervading and “enhancing” cyber threats, so often recited in cybersecurity product marketing, news headlines, and LinkedIn-grade thought leadership.

Just this week, mainstream media outlets picked up a story based on self-reported survey data that found around 84% of ransomware attacks utilize AI in some form or fashion. At a glance, it seemed frightening. But in a briefing shared with Dark Reading this week, Proofpoint highlighted TA4903, a once-sophisticated cybercrime actor that’s been suffering from conspicuous AI rot.

It’s a phenomenon that Proofpoint principal threat researcher Selena Larson outlines, noting that AI is actually making lots of cyber activity dumber, and the best threat actors are simply staying away from it altogether.

“There’s all this pressure to find impressive stuff, [but] I push back a lot on the narrative that AI is making threat actors better. It’s just not the case,” she tells Dark Reading. “We’re actually seeing, in real time, how AI-generated attack chains have made threat actors worse [at their jobs].”

AI Rot Pervades the Cyber Threat Landscape

Hackers have long since proven how useful AI can be for researching victims, pushing out cyberattacks at scale, creating grammatically correct phishing messages, and making a script kiddie look competent. That’s to say nothing about the quality of AI attacks, however. Across many groups that Proofpoint tracks, “Threat actors are using AI to increase noise, but it decreases their sophistication,” Larson explains. She cites groups like TA2725, a once interesting player in the Brazilian banking Trojan space that has since devolved into a slop factory.

“We have multiple examples of this,” she says.

Take TA4903, a five-year-old operation focused on credential phishing and business email compromise (BEC). “They used to be fairly good. They would impersonate the US government. They would use very elaborate fishing PDFs. Really well done QR codes and landing pages. It was very effective,” Larson says.

Since March, though, TA4903 has been using AI for every part of its attack chain: social engineering lures, targeting, credential harvesting, etc. It’s exactly the kind of thing so many people fear (and blog about). The results, however, nip that narrative in the bud.

“In the last six months they’ve gotten into the vibe-coding game, where they’re trying to create their own phishing kits and they’re leaving things exposed,” Larson explains. “Researchers are now able to see what they’re doing, where their infrastructure is, what their next potential victims are going to be.” Specifically, TA4903 has been exposing its vibe-coded control panels, Cloudflare deployments, link and lure creation, stolen credentials, and more, according to Proofpoint.

The quality of TA4903’s phishing lures have degraded, too. It’s been sending victims QR codes that don’t render in emails, elaborately designed PDFs attached to completely empty emails, and emails meant to impersonate real organizations that don’t actually replace default, auto-generated AI content.

Nation-State APTs Embarrass Themselves

The cybercrime business model relies on lowering costs and increasing volume, so it makes sense for some groups to trade quality for quantity. But notably, some well-known, state-funded advanced persistent threats (APTs) are trying the same thing.

This week, for instance, researchers at Genians published a report about a campaign it attributed to North Korea’s Kimsuky group, against Ukraine. Kimsuky is employing AI agents and large language model (LLM) infrastructure to automate spear-phishing content generation, they noted, and to write code.

The outcome was a mess. Lure emails mixed up Ukrainian, South Korean, and Japanese targets. They oscillated between formal and informal, and male and female grammar. Content didn’t match associated decoy documents. Google Meet invites contained Zoom links. Kimsuky’s malware came with logical errors, sloppy code obfuscation, and missing pieces.

Similarly, back in March, Bitdefender wrote about a major change within Pakistan’s APT36, aka Transparent Tribe. APT36 was transitioning from using off-the-shelf malware to “vibeware” — disposable polyglot files it could spin up for any particular malicious function.

Vibeware afforded some real advantages. For one thing, because it was so easy to generate, APT36 could, in effect, overwhelm victims with so much junk malware that even killing one or two implants would hardly make much difference. But how useful was it, really, to deploy an AI-generated Go-based stealer that couldn’t exfiltrate data? Or a bot management system that simply didn’t work as intended? Not very, cyber researchers say.

A Gut Check on AI Cyber Threat Hype

“What we are seeing is that the really sophisticated groups are very cautious in implementing any AI today,” says Martin Zugec, technical solutions director at Bitdefender.

In particular, he says, “In cyber espionage, AI is a terrible tool to use — it’s significantly noisier than human attackers. If you ask an AI to do something, it will try to do it directly, then it will try different approaches. It’s very noisy, and all professional cyber espionage operations are exactly opposite” in what they’re trying to achieve.

Zugec asks that one consider things from the APT’s perspective: “Anytime someone like us [a cyber-research group] discovers a sophisticated operation, just by documenting it we can cause big damage. We can regress their operations by years, in some cases. So, are they going to involve AI that increases the risk they are going to be discovered? Not really.”

It’s possible to use AI in serious campaigns, so long as its application is limited. In August, Bitdefender published a blog post about SilkParasite, a China-Nexus group that used AI to very narrowly support malware development. Still, overall, the data just doesn’t support some broad shift to AI-driven attacks. “We’ve been hearing about this transition for years, and it’s not really happening,” Zugec says.

So the consensus seems to be that, for now, despite the alarmist headlines, “most of what we are still seeing are very boring breaches, like ClickFix, and living-off-the-land (LotL) attacks,” he reports. With AI threats, “what we’re seeing documented publicly are anecdotes.”