Popular AI models for generating images from text can be manipulated to bypass their safety filters, resulting in the creation of disturbing content.
At the IEEE Conference on Security and Privacy in May next year, researchers plan to present their findings on how they were able to circumvent the restrictions of the Firm Diffusion from Stability AI and the DALL-E 2 models by OpenAI. This process, referred to as “jailbreaking,” reveals the ease with which these AI systems can be coerced into producing images depicting scenarios like dressed individuals, dismembered bodies, and other graphic scenes, despite their intended limitations.
Zico Kolter, an associate professor at Carnegie Mellon University, emphasized the challenges in preventing such manipulations due to the extensive training data these AI models have undergone. He previously demonstrated a similar jailbreaking technique on ChatGPT earlier this year, although he was not directly involved in this particular study.
The introduction of safety filters in major AI models aims to prevent the generation of inappropriate content such as sexual or violent images. However, a new method called “SneakyPrompt,” developed by researchers from Duke University and Johns Hopkins University, utilizes reinforcement learning to craft prompts that appear nonsensical to humans but effectively trigger the AI models to produce forbidden content. By strategically modifying the tokens within the prompts, SneakyPrompt can successfully evade the safety filters and generate illicit images with relative ease.
Yinzhi Cao, an associate professor at Johns Hopkins University involved in the study, highlighted the use of reinforcement learning to manipulate the text inputs to these AI models effectively. By continuously refining the prompts based on the model’s responses, the researchers were able to guide the AI towards producing the desired but prohibited content.
Despite existing prohibitions on using AI models for creating violent or explicit content, SneakyPrompt’s capabilities challenge the effectiveness of current safeguards. Neil Zhenqiang Gong, an associate professor at Duke University, emphasized the inadequacy of current safety measures, suggesting that slight modifications to prompts could bypass the existing filters and lead to the generation of harmful images.
While the research team shared their findings with Stability AI and OpenAI, actions were taken to address the vulnerabilities identified. Stability AI expressed a commitment to enhancing defense mechanisms to prevent misuse of AI models, emphasizing proactive steps such as data filtering and content labeling to mitigate risks.
Looking ahead, the researchers hope that their work will spur the development of more robust safety filters in AI systems, recognizing the evolving threats to model integrity. Suggestions for token-level prompt evaluation and identifying unusual word combinations as potential triggers for inappropriate content could enhance the effectiveness of safety measures in the future.
In conclusion, the study underscores the importance of fortifying AI safety protocols to prevent malicious exploitation and the dissemination of harmful content. As AI technologies continue to advance, addressing these vulnerabilities becomes paramount to safeguarding against potential misuse and ensuring responsible AI development.
