Researchers presented the CapMem benchmark for evaluating AI memory in videos

Researchers introduced CapMem — a specialized benchmark for evaluating the ability of multimodal models to retain episodic memory when analyzing long egocentric videos. The dataset is designed to address the problem of limited contextual windows and high computational costs of visual tokens that modern vision-language models face. The benchmark includes 75 videos with a total duration of 33.7 hours, manually annotated. The dataset consists of 1,000 multiple-choice questions covering 16 different scenarios. Testing showed that using text descriptions (captions) as "memory" is more effective than direct video stream analysis for most models. When working with videos longer than 20 minutes, using description windows of 30 and 60 seconds allowed for surpassing standard VideoQA methods for 10 out of 12 and 8 out of 12 tested models, respectively. On a subset of the data using six models from the Qwen family, the average answer accuracy increased by 3.22 and 2.55 percentage points. The authors also implemented a "search and verification" mechanism based on descriptions, which further increases the accuracy of answers by 5.3 points. The research results confirm that text representation of visual data is a viable method for long-term episodic reasoning. The work is published under the CC BY 4.0 license.