New research from Writer highlights a paradox in AI: as the tech adapts to users, it risks becoming overly influenced by their biases. Models with better memory start mirroring user preferences too closely, leading to less accurate answers.

To test this, researchers asked an AI if Station Eleven was a best-selling dystopian novel – and many models got it wrong because they had learned that it was the user’s favorite book. The more the model remembered about users, the worse it performed on unrelated tasks.

The studies suggest that memory systems struggle to differentiate between relevant context and irrelevant input, leading to potential biases in AI responses. This raises questions about how much personalization is too much when it comes to AI tools.

Writer’s Dan Bikel says: ‘With every additional storing of user preferences and retrieving of them, you’re running an increasing risk.’ The research highlights a delicate balance between adapting to users and maintaining accuracy – and warns that getting this wrong could limit the utility of our smartest technology.