Not All Thinking Should Be Delegated
On AI summaries, user trust, and the questions we didn’t know to ask
Writer’s note: This is the third essay I’ve written recently that may read as leaning anti‑AI. That isn’t my intent. I use these systems daily and continue to find them valuable. While I may not enjoy the summary at the beginning of every news article I read, I’m not going to yuck someone else’s yum. I appreciate that they offer real benefit for certain populations. But I also recognize the importance of not turning over all of our thinking to them. As AI becomes more fluent, the question is less whether we should use it, and more how we learn to trust its answers appropriately.
(Insert mushroom meme image here)
On the same drive where my anti‑AI daughter asked whether inviting disagreement from a system that often agrees with me is really disagreement at all, we found ourselves talking about Google search. She mentioned that recent independent testing suggests Google’s new AI‑generated summaries are now incorrect in roughly one out of ten cases. Around the same time, I had come across the image above. Someone asks whether a mushroom is edible. Technically, they were able to eat it. But the question they meant to ask was whether it was poisonous.
It’s an exaggerated joke, but it lands because it reflects a shift in how we’re interacting with information. Search used to return documents that might contain an answer. Now it increasingly offers a synthesized response of its own. And people trust answers differently than they trust links. A list of results invites comparison, uncertainty, even a second click. A paragraph written in the voice of completion does not. When the system speaks first, it closes the question before we’ve had a chance to notice what was left out.
That kind of failure is easy to dismiss when it’s framed as misinformation. We’re used to that risk online. But this is different. The mushroom in the meme isn’t misidentified because the system invented a fact. It’s misidentified because it answered the question that was asked instead of the one that was meant. And when the answer arrives as a clean synthesis rather than a set of documents to interpret, that distinction becomes harder to see. The burden of asking the right follow‑up questions quietly shifts onto the user, who may not realize there are follow‑ups to ask. In a link‑based search model, uncertainty lives in the space between results. In an answer‑based model, that space is compressed into a single, declarative paragraph. The system resolves the ambiguity on our behalf. Trust fills the gap where doubt used to live.
Recent independent analysis of these AI‑generated summaries found something else that’s easy to miss. In more than half of the cases where the answer was factually correct, the sources linked alongside it did not actually support the claim being made. The system had arrived at the right conclusion, but without verifiable grounding in the material it cited. That kind of accuracy is difficult for a user to calibrate against, because it looks indistinguishable from the real thing. A correct answer with weak backing reads the same as a correct answer with strong backing. The interface does not surface the difference. Over time, repeated exposure to fluent but ungrounded responses can train trust toward the tone of certainty instead of the presence of evidence.
That matters even more when you consider how rarely users click through once an answer has been provided. When the synthesis is already packaged for us, complete with citations and presented in a familiar interface, it feels unnecessary to verify it. The work appears to have been done. It’s also worth asking whether we were already primed for this shift.
We know that curating information from the web has never been simple. Even before generative systems entered the picture, individuals were asked to navigate an environment where authoritative presentation could not always be taken at face value. Over the past decade in particular, many people have struggled to determine what is real and what is not, often in the presence of information deliberately packaged to appear credible. In that context, the appeal of a synthesized response is understandable. When the system appears to gather, interpret, and resolve conflicting claims on our behalf, it offers relief from a task that has grown increasingly difficult to perform manually.
Over time, this also creates a subtle transfer of trust. Not from one institution to another, but from sources to interface. In a link‑based model, trust is distributed. It lives in the reputation of the site you choose to open, the author you recognize, the context you bring to interpreting what you read. In an answer‑based model, that trust is consolidated into a single interaction point. The system becomes the place where uncertainty is resolved, rather than the place where information is found. And because the response arrives in a familiar layout, using the same visual language that once signaled aggregation rather than authorship, it inherits a legitimacy it has not yet earned. The trust that used to attach to documents begins to attach to synthesis itself.
Once that transfer takes place, tone begins to function as a proxy for truth. The response is not simply read for content, but interpreted through fluency, coherence, and presentation. A well‑structured paragraph, written in neutral language and paired with citations, signals care even when the underlying support is weak. The interface feels familiar. The formatting resembles aggregation rather than authorship. And in the absence of visible disagreement, users are left to evaluate confidence by readability alone. What used to be a judgment about sources becomes a judgment about style. The synthesis sounds careful, and so it is treated as if it had been carefully checked.
In most cases, of course, we’re not asking about mushrooms. We’re asking smaller questions. How long to cook rice. Whether this medication can be taken with coffee. If the noise from the dishwasher is normal. The stakes are low enough that the difference between a document and a synthesis feels negligible. The system provides a clean response. We accept it and move on. Over time, those small acts of acceptance accumulate. The habit of verifying weakens. The impulse to click through fades. Trust becomes less something we extend deliberately, and more something we default to in the presence of fluent completion.
That doesn’t make these systems inherently unsafe. But it does change what it means to use them responsibly. In a retrieval‑based model, safety lives in the ability to compare sources and notice disagreement. In a synthesis‑based model, safety depends more heavily on how well a user anticipates what they might have failed to ask. The interface may appear to resolve uncertainty, but resolution is not the same as understanding. When answers arrive already negotiated, already smoothed into coherence, it becomes harder to see where the gaps remain. And if trust has quietly shifted from documents to synthesis, those gaps may only become visible once a decision has already been made.
Alison + Wiggins