Researchers presented the WearableQA benchmark for evaluating data analysis from smartwatches

A research group presented WearableQA — a specialized benchmark for testing the ability of large language models to interpret data from fitness trackers. During testing, 14 popular models analyzed their skills in working with pulse, sleep quality, and physical activity indicators over a six-month period.

The results showed a systemic gap in AI capabilities: models demonstrate low accuracy when performing basic computational tasks, such as finding trends or fixing anomalies in numerical arrays. In particular, GPT-4o successfully coped with calculations in only 25% of cases. At the same time, the same models show twice as high results in clinical data interpretation, formulating conclusions about health risks.

The authors of the study note that the current LLM architecture is better adapted to semantic analysis than to precise processing of time series. This creates risks when using AI as a personal medical assistant, since the model can make correct conclusions based on incorrectly interpreted or incorrectly calculated original data.

The WearableQA benchmark allows developers to assess the robustness of models to errors when working with personal telemetry. The obtained data indicate the need to implement specialized tools for preliminary processing of numerical indicators before transferring them to a language model.