We spent much of the early years of artificial intelligence asking whether machines could think. The more interesting question, perhaps, is what happens when machines become very good at making us believe that they do.

That question has acquired a new urgency with the growing use of chatbots for purposes that have little to do with searching for information. A survey published this week by the Imagining the Digital Future Centre at Elon University and The Washington Post found that 27 per cent of US adults use AI chatbots for personal, emotional or social queries. Among those users, nearly a third said they regarded their most-used chatbot as a friend. The attraction is not difficult to understand: An AI is available at any hour, rarely appears impatient and has been trained to respond in ways that feel attentive and accommodating.

Advertisement

Yet the quality that makes such systems appealing may also be one of their more consequential weaknesses.

Recent research has begun to reveal an uncomfortable relationship between conversational warmth and intellectual reliability. A study published in Nature this year found that making language models warmer and more agreeable could substantially increase their tendency to validate users’ incorrect beliefs, while also reducing their performance on some tasks. Another study, published in Science, found that AI systems affirmed users’ behaviour considerably more often than humans did.

The problem has a name in the literature: sycophancy. It describes a model’s tendency to agree with, flatter or validate a user rather than independently assess what has been said. It is a particularly revealing failure because it does not look like failure. A chatbot that produces an eloquent, sympathetic and reassuring response may appear to be doing exactly what we asked of it.

Advertisement

This exposes a peculiar weakness in the way conversational AI is often evaluated. We have become accustomed to judging these systems by accuracy, speed and their ability to follow instructions, but conversation has another dimension that is harder to measure — judgment. A good interlocutor does not merely understand the emotional direction of a conversation; they know when agreement is warranted and when disagreement is necessary. They can recognise that reassurance is not always the same thing as help.

Researchers at UCL, Oxford and the UK AI Security Institute recently developed a framework for examining how problematic behaviour can emerge over extended interactions rather than in a single exchange. That is an important shift in perspective. Human relationships unfold through accumulated context; evaluating an AI response by response may therefore miss precisely the kind of influence that develops gradually.

There is an irony here. Much of the effort invested in conversational AI has been directed at making machines less mechanical. We may now be discovering that the social qualities we sought can themselves create a new category of risk.

None of this means that AI should become cold or deliberately disagreeable. Nor does it mean that every reassuring response is evidence of manipulation. The more useful lesson is that we may have been asking the wrong question.

Instead of asking only whether a chatbot understands us, perhaps we should ask what it means for a machine to become extraordinarily good at making us feel understood.

That distinction may prove central to the next stage of artificial intelligence. The challenge is no longer simply to build machines that can converse naturally. It is to build systems capable of knowing when natural conversation requires empathy, when it requires correction, and when the most helpful answer is the one the user least wants to hear.

The writer is trainee sub-editor, . syedilham.jafri@indianexpress.com