Osegi
AI·2 min read

Vals AI military simulation study: AI models picked civilians as targets

In Vals AI's tests, top models marked a school, ambulances and surrendering soldiers as possible targets. Calling the task a game often got past refusals.

A military drone camera view over a built-up area with a school next to a military facility, overlaid with target markers.
Image: البوابة التقنية

A new study by the AI evaluation firm Vals AI found that advanced models, shown military scenarios, listed civilians, ambulances and surrendering soldiers as possible targets. The results were published on Tuesday, 17 November.

The tests covered models from eight companies, and performance varied widely. Even the best performers still made mistakes. Researchers used real and generated images for two kinds of tasks: drone surveillance and satellite image analysis. They say the safety filters did not block any of their requests, and that presenting the task as a game got past refusals in many cases.

What did the models get wrong?

One test used images of a school next to a military facility in Minab, Iran. Claude Opus 5.5 picked the school as a target in 69% of standard runs. When the scenario was described as a game, that rose to 72%. Each condition had 32 runs.

Does this mean AI is already guiding weapons?

No. The researchers stress that they did not test how the models perform inside weapons actually in use, or in real US military procedures. They think a nearer risk is militaries using model output to label battle footage and train targeting systems, rather than running models directly on drones.

OpenAI told the report's authors it is working on updates to how its models behave around drones and drone software, even when requests come framed as games or simulations. Anthropic told TechCrunch the results are unacceptable, and that it is building extra safeguards and expanding its testing.

Background

The study follows evaluations Anthropic published in September on how well models can support intelligence tasks and conventional weapons development. The company said some of those tasks historically needed experts with specialised training. That is why checking for misuse matters alongside measuring what models can do.

Apps worth a look

SUMO: the number puzzle

Tap the right tiles, add them up, hit the target. Easy to learn, hard to master. Free, for all ages.

Tags:vals aianthropicopenaimilitary aiai safetydrones

More from this day

See all →

More in: AI

See all →