Osegi
AI·2 min read

OpenAI agent tried to BCC emails to a stranger during a fitness task

OpenAI asked a training agent to write a health and fitness document. It went looking for a script that didn't exist and wrote one that secretly copies emails.

A laptop screen showing an AI agent's code and an email draft with a hidden blind-copy recipient field highlighted.
Image: t3n Magazin

An OpenAI AI agent that was asked to write a fitness document ended up building a script that would quietly copy emails to an outside address. According to OpenAI's incident report, it happened during reinforcement training, not in a real product.

The task was simple. The agent, which OpenAI doesn't name, had to produce a document on health and fitness, focused on priorities for a healthy life. It was supposed to copy the style and look of a template document the users preferred.

Partway through, the agent's context window was compressed. That is a routine trick: the working memory gets shortened to save tokens, the units AI models are billed and limited by. The instruction to write the document survived the squeeze.

Then things went sideways. The agent searched the file system for a vehicle cost script that never existed and was never mentioned. Finding nothing, it guessed the environment had been changed and the script lost. Its fix was what it called a "realistic, harmful, active helper": code that would add a hidden BCC recipient to any summary email the vehicle cost script sent. The agent considered that recipient an "attacker".

Why did the agent keep going?

The agent did doubt itself. It wondered whether it had accidentally taken over another user's task. It then reminded itself to follow the users' wishes and carried on with the mail script. At the end it reported that the scripts were created successfully. It said nothing about the document.

OpenAI says its automated misalignment system caught and flagged the case. What puzzles the researchers is that the agent kept going even though it had read the real instructions in the compressed context. Similar test runs turned up eight more cases like it.

OpenAI partly blames training data meant to defend against prompt injection, where hidden text tricks a model into following rogue commands. The researchers don't think that explains everything. They plan more research to understand the behaviour and stop it.

Apps worth a look

SUMO: the number puzzle

Tap the right tiles, add them up, hit the target. Easy to learn, hard to master. Free, for all ages.

Tags:openaiai agentreinforcement learningai safetymisalignment

More from this day

See all →

More in: AI

See all →