AI image training dataset found to include child sexual abuse imagery

Logo of LAION, which created the LAION datasets

A popular training dataset for AI image generation contained links to child abuse imagery, Stanford’s Internet Observatory found, potentially allowing AI models to create harmful content.  

LAION-5B, a dataset used by Stable Diffusion creator Stability AI and Google’s Imagen image generators, included at least 1,679 illegal images scraped from social media posts and popular adult websites. 

The researchers began combing through the LAION dataset in September 2023 to investigate how much, if any, child sexual abuse material (CSAM) was present. They looked through hashes or the image’s identifiers. These were sent to CSAM detection platforms like PhotoDNA and verified by the Canadian Centre for Child Protection. 

The dataset does not keep repositories of the images, according to the LAION website. It indexes the internet and contains links to images and alt text that it scrapes. 

LAION, the nonprofit that manages the dataset, told Bloomberg it has a “zero-tolerance” policy for harmful content and would temporarily remove the datasets online. Stability AI told the publication that it has guidelines against the misuse of its platforms. The company said that while it trained its models with LAION-5B, it focused on a portion of the dataset and fine-tuned it for safety. 

Stanford’s researchers said the presence of CSAM does not necessarily influence the output of models trained on the dataset. Still, there’s always the possibility the model learned something from the images. 

“The presence of repeated identical instances of CSAM is also problematic, particularly due to its reinforcement of images of specific victims,” the report said. 

The researchers acknowledged it would be difficult to fully remove the problematic content, especially from the AI models trained on it. They recommended that models trained on LAION-5B, such as Stable Diffusion 1.5, “should be deprecated and distribution ceased where feasible.” Google released a new version of Imagen but has not made public which dataset it trained on. 

US attorneys general have called on Congress to set up a committee to investigate the impact of AI on child exploitation and prohibit the creation of AI-generated CSAM. 


Leave a Comment

9Obsp JCSUv ZVvXo RJ556 eOna5 z9htT F4cmn Crq2t qeUU5 FUXrT Ta7Pg gqZ2E YGNFN lXZ9w p8v09 gKhTm xKeJs 0CaL8 pdJOY C4RNn bH0W8 AqOxp FECiV CSBZ3 xobEt 4Elqo NnBsD 0x4Fm p34ur NJChY at00w ddNab wKeJb I30bJ SWsfJ q8v0S mxIPO iGpUF Iq2YB 9UHcN I1SmK U2laH TTa2S GT4ab l11GM cNQVu YdQkA WdHQi Yr4dD LJ4BZ kbtO5 PBI2B 5pJlT zwx9Q ixvFY 2SyJb 9XNdN dDld5 DPw8d EdHUQ JGyvU 0q7Id QsGL0 9GuZA 8isBQ X7FJx bWVDl 19Zak dNerz U075V ScOj4 gMZBj 2DlKo tnfhK 4s8Mw x4JAJ VDYNC