In late January 2024, X experienced an inundation of innovative, artificially created photos featuring Taylor Swift, known as deepfakes. While celebrities have often been targets of image leaks and cyber assaults, this instance stood out due to the utilization of artificial intelligence.
Taylor Swift’s fan base swiftly reported the manipulated images, leading to their removal from the author’s account within a day of existence. Despite the platform’s regulations against unauthorized exposure, the images managed to gain popularity in that brief timespan. Subsequently, an announcement from the propaganda research firm Graphika disclosed that the deepfakes originated from 4chan. Individuals on the platform had actively encouraged each other to produce sexually suggestive deepfakes involving prominent female celebrities, aiming to circumvent regulations on explicit content.
Regrettably, Swift’s encounter is not an isolated incident. Xochitl Gomez, a 17-year-old Marvel actress, shared her challenges in having deepfake content removed from X during an episode of the podcast “The Press,” highlighting the detrimental impact on her mental well-being. Gomez and Swift are just two examples among numerous women who have recently faced the distressing proliferation of sexualized deepfakes.
Siwei Lyu, a Computer Science Professor at the University of Buffalo, remarked to The Daily Beast, “People have always used media to try and defame individuals; what has evolved is the heightened visibility it now garners.”
Towards the end of the preceding year, the AI image generation platform CivitAI gained popularity for its “Bounties” feature, incentivizing users to produce deepfakes in exchange for virtual rewards. As reported by 404 Media, the majority of bounty submissions featured individuals, including non-celebrity women and private citizens.
Experts foresee a worsening scenario, particularly with the increasing utilization of such systems by online incel communities. Henry Ajder, an authority on algorithmic and AI technologies, noted the close association between platforms like CivitAI and the proliferation of harmful content. He emphasized that CivitAI has become a hub for diverse content, including sexually explicit material, due to its specific user base seeking such content.
Ajder described CivitAI’s technology as versatile, catering to various purposes. While some platforms are explicitly designed for producing unauthorized pornographic content, CivitAI’s tools have gained traction within cultural circles, facilitated by platforms like Reddit and 4chan.
The pace of technological advancement has outstripped regulatory efforts, leaving institutions struggling to keep pace with emerging challenges.
Hera Husain, chairperson of Chayn—an organization supporting victims of gender-based violence and stress—highlighted the accessibility and ease of using deepfake technology, enabling individuals to materialize their fantasies, albeit with severe repercussions. She cautioned that while individuals may perceive it as harmless, the implications for affected individuals are profound.
Incel culture has also been influenced by deepfakes, with research indicating that AI-generated companions may exacerbate the dangers posed by incels. These virtual constructs could potentially shape and distort incels’ perceptions of an “ideal woman,” leading to concerning behaviors. Incels may resort to algorithmic means when faced with unattainable standards, as exemplified in the cases of Swift and Gomez.
As Belinda Barnet, a senior media lecturer at Swinburne University, remarked, “Governments are struggling to keep pace with technological advancements.”
A “distinct upheaval”
The repercussions of such developments are particularly pronounced in global contexts, where individuals may face additional challenges due to entrenched patriarchal norms. In some traditional societies, even a fabricated deepfake of a woman could incite severe repercussions, including familial censure or violence. Ajder elucidated that societal reactions are often driven by malice rather than genuine belief in the authenticity of such images, representing a unique form of harm akin to “justice porn.”
The proliferation of AI technologies further complicates efforts to counter extremism and radicalization, particularly in regions with language barriers like South Asia. Husain emphasized the challenges in combating incel extremism, both ethically and from a policy standpoint.
Lyu highlighted the diverse impacts of global free speech and technology regulations, noting the contrasting approaches in countries like the United States and China. While the U.S. upholds freedom of speech, China imposes stringent restrictions on technology use, illustrating the complex interplay between technology and governance.
The widespread adoption of AI technologies has amplified the potential for harm, with malicious actors leveraging these tools to perpetrate abuse and sow discord. Ajder underscored the detrimental impact on women, perpetuation of stereotypes, and reinforcement of distorted ideologies within incel culture.
The trajectory suggests a challenging road ahead, with incel culture wielding technology as a potent weapon to propagate harmful narratives and deepen societal divides.
