A few years ago, artificial intelligence (AI) voice cloning was merely a mildly entertaining novelty. Songs produced by renowned artists such as Drake and Ariana Grande were circulating on the web. However, concerns about the AI technology were validated as scams involving AI voice cloning began to emerge. In April of last year, a family in Arizona, United States, was coerced into paying ransom for a fictitious kidnapping orchestrated by an AI-cloned voice. These scams were not isolated incidents. The accessibility of AI voice replicas also led to the spread of disinformation.
In January, users of 4chan flocked to AI text-to-speech tools to generate hate speech, including instances where Ben Shapiro, a liberal democratic journalist, was falsely portrayed as making racist comments against Democrat lawmaker Alexandra Ocasio-Cortez, and Emma Watson, the actress from Harry Potter, was depicted reading a segment from Mein Kampf.
Why is India a primary target for scams involving AI message clones?
Similar incidents have been reported in India. According to a report titled “The Artificial Imposter” published in May of last year, 47% of surveyed Indians either fell victim to AI-generated voice scams or knew someone who had. This percentage is nearly twice the global average of 25%. India ranked highest on the list of countries with the most victims of AI message scams. While some cases went undetected, others were uncovered. For instance, a resident of Lucknow fell prey to a cyberattack in December where AI was used to replicate the voice of the victim’s sibling, urging the victim to transfer money using UPI. In another case from August, a man from Haryana was swindled out of $30,000 by scammers who impersonated the victim’s friend in urgent need of money due to an accident using AI apps.
What are speech deepfakes and how do they operate?
It has been observed that individuals in India are particularly susceptible to such frauds. McAfee reported that 66% of Indian respondents admitted they would promptly respond to a call or message requesting urgent financial assistance, especially if the caller claimed to be a family member such as a parent, spouse, or child. Scammers exploit a sense of urgency to bypass these vulnerabilities. The report indicated that 86% of Indians shared their voice data or personal information online monthly, making them susceptible to such manipulations.
How are voice imitators created?
Perpetrators simply need to upload a target’s voice clip to an online platform capable of replicating the voice, with minor variations in intonation. Several online programs, including Murf, Resemble, and Speechify, offer this service. These platforms typically offer a free trial period, with subscription fees ranging from under \(15 for basic plans to \)100 for premium options.
In addition, renowned tech companies are also involved in the AI voice market. For instance, Meta recently introduced SeamlessM4T, an open-source multilingual model that can interpret nearly 100 languages from speech or text and provide real-time translations. Apple incorporated a voice cloning feature into the iPhone 17 to aid individuals at risk of losing their voice due to a chronic illness.
Conclusion and Future Outlook
The rapid proliferation and ease of use of these tools have raised concerns. The U.S. Federal Trade Commission (FTC) initiated a Voice Cloning Challenge to address the detection and prevention of manipulated voice technology. Additionally, a proposed Imitation Rule aims to combat fraudulent voice cloning. However, advancements in relational intelligence continue to outpace regulatory efforts. The industry is projected to grow exponentially, with the global market for voice cloning applications expected to reach \(1.2 billion by 2022 and potentially \)5 billion by 2032, with a compound annual growth rate (CAGR) ranging from 15% to 40%.
