Following a research study by 404 Media that uncovered the utilization of the text-to-image program in generating images that could potentially be classified as child pornography, OctoML has stated that it has terminated its business partnership with Civitai. Despite Civitai recently utilizing OctoML’s octai for image generation, 404 Media disclosed on Saturday that OctoML opted to sever all connections with Civitai. This decision contrasts with earlier indications suggesting a continuation of collaboration with Civitai while implementing new measures to mitigate the creation of harmful imagery.
Internal communications indicated that OctoML was cognizant of certain individuals at Civitai producing sexually explicit material, including depictions of children and nonconsensual images of real individuals, as reported by 404 Media on December 5. Subsequently, OctoML reportedly implemented an automated filter to block the generation of any NSFW content on Civitai before announcing its disassociation publicly. In light of the recent investigation, Civitai has also introduced additional moderation techniques, such as the mandatory integration of Citai Safe Helper (Minor), designed to prevent the model from generating images of children upon detection of mature themes or keywords, as detailed by 404 Media.
Civitai, supported by Andreessen Horowitz, has faced criticism for its “bounties” feature, which incentivizes users to create realistic images of real individuals in exchange for rewards. In November, 404 Media uncovered the use of this feature to produce sexually explicit deepfakes involving various individuals, predominantly females, including celebrities and ordinary people. Civitai has since implemented a filter to prevent the creation of NSFW content featuring specific celebrities, as reported by 404 Media.
Nevertheless, OctoML, a user of Amazon Web Services’ infrastructure, has opted to disengage from the partnership. In a statement to 404 Media on Saturday, OctoML declared, “We have chosen to end our business relationship with Civitai,” citing a commitment to upholding the safe and ethical utilization of AI.
