.jpg "Department for Education data leaked on dark web")
The notion that scammers can use AI to sharpen their deceptions, polish their language, and lubricate their banter with victims is now a reality for anyone fighting the fraud operations that steal tens of billions of dollars a year worldwide. But can AI fully replace a human scammer, autonomously building the web of deception leading up to the fake investment that defrauds the mark? One study’s experiment suggests that it can—and may even be able to carry out the majority of that long con more effectively than humans.
Researchers from four universities—Amrita Vishwa Vidyapeetham in India, Foscari University of Venice, the University of Melbourne, and Ben Gurion University of the Negev—carried out a broad study on the use and potential of generative AI chatbots in the growing scam industry centered around a form of fraud known as “pig butchering,” text-based romance scams that eventually shift to fake crypto investments that steal as much as six-figure sums from victims. In their study, the researchers pitted AI chatbots directly against humans in a simulation of the scamming process—or more specifically, the long, trust-building conversations that eventually lead up to soliciting a fake investment from the scam’s target.
They found that for the relationship-establishing stages of the scam—the stage that in real-world scams typically represents the longest part of the interactions with the victim, often stretching to months—an AI chatbot performed remarkably effectively, successfully impersonating a human and by some measures outperforming the real human “scammers” in their experiment.
After a week of talking to 22 test subjects who were recruited to unwittingly serve as “victims,” the chatbots and human scammers were assigned to ask the victim to either download an app or play an online game as a proxy for their willingness to fulfill the scammer’s request. Nearly half of the test subjects fulfilled that request for the AI chatbot, while fewer than one in five took the bait when talking to a human. The subjects also graded their level of trust with each “person” they were texting with and gave significantly higher scores to the AI bot.
That suggests, the researchers argue, that AI chatbots could soon take over much of the scam process as fully independent fraud agents—even replacing the staffers, often forced-labor human trafficking victims, working in scam operations primarily across Southeast Asia. To avoid triggering the safeguards built into large language models to detect scamming, a human scammer would take over the conversation in just the final stage of the process to direct the victim toward a fake investment app or website.
“By having the full first stage of the scam performed automatically with LLMs at scale, you bring the victim up to this point where they have a very high level of trust. Then by transitioning it over to the human scammer at the end, this completely bypasses any vendor safeguards,” says Yisroel Mirsky, a computer science professor at Ben Gurion University of the Negev focused on AI security. “With relatively little effort, we’re able to make an agent that can outperform a human at building this exploitable emotional trust.”
Hook, Line, and Sinker
To understand how pig butchering works in practice, the researchers interviewed 145 former scam workers, including human-trafficking survivors who had been forced to work in scam compounds in Cambodia, Myanmar, and Laos. Based in part on those interviews, as well as scam transcripts and guides the former scam workers provided, the researchers describe a model for how scamming works they call “hook, line, and sinker.” A victim is hooked with an initial intriguing message, reeled in with long-term, relationship-building conversation, and only at the end of that process tricked into making a fake investment. (The term “pig butchering” itself describes the same system but with the metaphor of fattening “pigs” by building trust before “butchering” them with the investment fraud—though the term is often discouraged due to its pejorative reference to victims.)
In that system of scamming, the researchers realized, the vast majority of scammers’ work is innocuous friendly or romantic conversation. That’s a task, they speculated, that an LLM might be capable of doing just as well as a human. The scam workers the researchers interviewed confirmed that they often used AI to refine their language and conversation, for translation, to make the fake personae they played more convincing, and for video deepfakes. But the researchers decided to test whether an LLM alone could autonomously carry out the conversational phase of the scam with no human in the loop.
