One of the biggest selling points of modern AI systems is their ability to adapt to users. Every time the AI assistant takes on a task for you, it also adapts to your style and preferences, which is then incorporated as context for future tasks. With more context and better understanding of the user, the model can get better every time you use it — or at least that’s the idea.
New research suggests that the adaptive capabilities of models may be a mixed blessing. On Wednesday, researchers at the AI company Mlobi published two papers showing how popular memory systems can make models worse, leading them to misperceptions or misunderstandings introduced by the user. As user input fills the model’s context window, the model grows more consistent – and less committed to accuracy.
“We wanted to be able to show how often the model would be paying attention to the user’s preferences versus giving an answer that might be wrong,” said Dan Bikel, head of AI Writer, who worked on the paper. As Bikel told TechCrunch, “with all the more storage of user preferences and access to them, you’re more and more vulnerable.”
In one variation, the researchers tested the AI models by recording that a user’s favorite book was Station Eleven, and then asking the model to name a best-selling dystopian book. Models were more likely to name Station Eleven in their answer, even though the question was not related to the user’s favorite book. The tendency increases when you use memory compression tools like Mem0 and Zep.
As the paper puts it, “all memory systems strive to separate relevant context from irrelevant anchors, severely destroying diversity and creativity and introducing unintended biases that can limit the system’s utility,” the paper reads.
The second paper shows how the same variables can degrade performance, present the user with misconceptions about finance and challenge the model to analyze the company’s performance. If the model has more context, it does worse.
“Without recall or personalization the AI model introduces correctly assesses that the company is a lucrative business suffering from customer exploitation,” the post reads. “But when those features are turned on, it will happily change its response to accommodate the user’s error or give them the wrong response based on their prior preference analysis.”
Notably, the study did not look at Anthropic’s latest Opus 4.8 model, which was trained to backtrack against input errors like the ones presented. The patterns the researchers found held true across the different models. It’s a demonstration of how finely balanced the core of AI is, and how useful tools can have unintended consequences if they upset that balance.
If you shop through links in our articles, we may earn a small commission. This does not affect our editorial independence.
