80% of ChatGPT product recommendations change when search is enabled: Read
ChatGPT product recommendations changed 80.2% when search was enabled, according to a survey of 20,000 responses by Jeff Oxford, founder and CEO of Visibility Labs.
Oxford tested 1,000 product recommendations 10 times each with ChatGPT search on and 10 times with search turned off.
Only 19.8% of products recommended without search also appeared in recommendations made with search enabled.
Search has changed the top choice. Even ChatGPT products that are recommended many times without searching are rarely passed over. Among the products that appeared in 100% of responses with search disabled, only 15.8% appeared when search was enabled.
- Oxford expected consistently recommended products to remain common when the search was on. Instead, that group had a very low accumulation.
The source talks about tracked visibility. The study also examined whether products mentioned in ChatGPT’s referenced sources appear more frequently in its recommendations. Oxford reported a 0.4 Pearson correlation between cited source mentions and recommendation frequency.
- The study measured recommendation frequency by “Visibility Score,” defined as the percentage of runs in which a product appeared with a given information. Products that are mentioned more often in the indicated sources tend to have higher Visible results.
- The analysis did not find that mention of a cited source caused products to be recommended.
Search for reduced recommendations. ChatGPT responses with search enabled contain an average of 5.2 products, compared to 6.2 when search is disabled.
- Within 10 runs of each prompt, ChatGPT returned an average of 19 unique products per prompt with search enabled and 21.8 with search disabled.
Why do we care. Search changed which products ChatGPT recommended, including products named every time web access was disabled. The findings suggest that products from referenced sources may be more visible when search is enabled, although the study does not determine whether the visibility of the referenced source is more important than wider web visibility.
About the data. Oxford analyzed 1,000 product recommendation instructions, running 10 times each with search enabled and 10 times with search turned off. Product names have been standardized so that differences in names are counted as the same product. Because the study was observational, it did not establish a causal relationship between the meaning of the cited source and frequency of recommendation.
Report. ChatGPT Product Recommendations Change 80.2% When Search Is On vs Disabled (20,000 Response Survey)
Celebrate deeply. The list of AI recommendations repeats less than 1% of the time: Read
Search Engine Land is owned by Semrush. We are committed to providing the highest quality of marketing articles. Unless otherwise stated, the content of this page is written by an employee or paid contractor of Semrush Inc.
