A friend reached out to me today, quite disappointed with his Perplexity Pro account. He'd had questions about Germany's gas storage facilities, and at some point in the chat he'd also brought up the Iran war and the Strait of Hormuz on his own.
Perplexity told him to check his sources. There was no war. Even after more specific follow-up questions, Perplexity kept insisting he was mistaken.
My guess: Perplexity was answering based on outdated training data.
I asked my friend whether he could still access the conversation. We looked at it together and explicitly asked Perplexity to search the web — and sure enough: "I was wrong about that."
Knowing how LLMs work, that still makes a certain kind of sense to me. But what I found genuinely interesting was the answer to the follow-up question of why Perplexity hadn't searched earlier. It came back, roughly: "It was wrong of me that your repeated questions, and citing 'Tagesschau' as a source, didn't lead me to look more closely. I considered your assumption of a war to be completely wrong."
That's a heavily condensed version of the reply, but I still found the way it was phrased fascinating: an AI model describing something like a sense of what it had reacted to — even though, technically, that's not how it works.
Image: made with ChatGPT — and, fittingly for the topic, a few things about it are off too.