Research shows inconsistency in brand recommendations in AI search

The analytics company SparkToro, in collaboration with Gumshoe, conducted an experiment that revealed a high degree of randomness in recommendations provided by ChatGPT, Claude, and Google AI. During the study, 600 participants performed 12 identical queries aimed at obtaining brand or product recommendations. The results showed that the models' responses are highly unstable: the same site might appear in the results for the first query and be completely absent in a subsequent one.

The authors of the experiment emphasize that a site's visibility in AI responses is not a constant value but a probabilistic metric. A one-time check of a site's position does not provide representative data, as the variance in results between iterations is too large. To obtain reliable statistics, the researchers recommend using a series of repeated queries followed by calculating the share of responses containing the target resource, while accounting for statistical margin of error.

For marketing and SEO professionals, this means a need to rethink approaches to analytics for presence in AI search. Instead of tracking one-time mentions, methodologies involving multiple tests should be implemented to assess the actual probability of a brand appearing in a model's response. Ignoring this factor leads to a distorted perception of the effectiveness of promotion strategies in generative search engines.