Osegi
Security·2 min read

OpenAI CSAM Image Classifier spots AI-made abuse images

OpenAI's new CSAM Image Classifier flags unseen and AI-generated abuse images, and ChatGPT now suspends accounts for 24 hours if users search for such material.

A person typing a message into the ChatGPT interface on a laptop, with a warning notice on the screen.
Image: ITmedia NEWS

OpenAI has released a tool for finding child sexual abuse material (CSAM), including images made by AI. At the same time, searching for that content in ChatGPT now gets your account suspended for 24 hours.

The tool is called the OpenAI CSAM Image Classifier. It analyses the visual features of an image and returns a score for how likely it is to be CSAM. It also gives a separate score for the chance that the image was AI-generated. The usual method compares images against hashes (digital fingerprints) of known illegal pictures, which struggles with new images. This classifier is meant to catch those. OpenAI says it is the first classifier with a dedicated AI-generated CSAM score that a wide range of organisations can use.

Organisations that pass a review get it free through the OpenAI API. The National Center for Missing & Exploited Children (NCMEC) helped with training, evaluation and design. Tech Coalition and ROOST are helping with distribution.

How well does the CSAM classifier work?

In OpenAI's tests on user-uploaded images, it found about as many abuse images as existing tools. It cut the number of images sent for human review by an estimated 48%, and its flagged images were about twice as likely to be real abuse content. Early partners reported results too. At fal, staff confirmed 72% of high-confidence detections as new CSAM. At invideo AI, one in four detected cases led to an account suspension. SafetyKit tested it on sexualised illustrations and animation of children and reported a 90% detection rate, with under 1% false positives on legal adult content.

The model card is less rosy. On real ChatGPT traffic, precision was well below the offline results, and recall for images showing infants was only 49.3%. Video frames were not tested. OpenAI says a score alone should not be treated as proof or as grounds for a legal report. A human reviewer makes the final call.

What happens if you search for CSAM in ChatGPT?

ChatGPT now shows a warning saying such content is harmful and possibly illegal, and that looking for it breaks the terms of use. The conversation is deleted and the account is suspended for 24 hours. The warning also points to free professional support services. Repeat offences can bring longer restrictions, account closure and a report to authorities.

OpenAI also published guidelines for developers on preventing, detecting and reporting CSAM, and is offering tools and training to law enforcement child-protection units for sorting through large chat histories.

Apps worth a look

SUMO: the number puzzle

Tap the right tiles, add them up, hit the target. Easy to learn, hard to master. Free, for all ages.

Tags:openaicsamchatgptchild safetyimage classifierncmec

More from this day

See all →

More in: Security

See all →