Apple Watch learns new hand gestures in a minute, study finds

How the test worked

The team first trained a model on about 2,760 hours of public wrist-movement data from 381 people, mostly broad activities such as sitting, walking and exercising. Ten participants then wore an Apple Watch Series 10 and each picked seven everyday activities, including stirring coffee, drawing a heart in the air, applying hand sanitizer, opening a bottle, tying a shoelace and filing nails.

With the 6-axis encoder, a single 2.56-second clip per activity gave a balanced accuracy of 75.2%. Five clips, about 13 seconds, raised it to 86.7%, and 20 clips, about 51 seconds, to 89.3%. A full minute pushed accuracy past 90%.

Everything runs on the watch. Training on a new set of activities takes less than a second, and a prediction takes about 24 milliseconds, with no phone or server needed. In one test pooling 70 activities in a single model, accuracy stayed above 82%. The core model is adapted from LSM-2, a wearable sensor model from Google researchers, and one team member is a Google research scientist, so the approach could also reach Wear OS watches.

Possible uses and limits

The researchers see gestures triggering apps or shortcuts, such as starting music when you stir coffee or logging a workout when you do bicep curls. They also point to passive tracking of symptoms like tremors and to factory systems that follow assembly steps.

The system struggles with small, sustained movements that look alike to the sensors, such as writing, scrolling on a phone or applying lotion, and with multi-step routines like making a latte. The study was short and controlled, and battery impact was not measured. Training tools are not yet publicly available.