A colleague recently asked me to review an email before they sent it out. The email provided an explanation for a tracking problem a client was having. It was plausible, well-written, and wrong.
I was not surprised to find that the explanation was provided by AI. As a person who manages an analytics practice, I have been reckoning with the positives and negatives of LLMs for a while. One clear positive is that we can do work a lot faster. One persistent negative is that the answers we get from AI can be confidently and catastrophically incorrect. The catchphrase that has been popularized to counter this problem is “human in the loop.” But in the example above, one human in the loop wasn’t enough.
I recently came across some fascinating research that explores why this might be. It is described in the paper, “Thinking—Fast, Slow, and Artificial: How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender,” by Steven D. Shaw and Gideon Nave. The paper describes its key concept as follows:
“…cognitive surrender is an uncritical abdication of reasoning itself. It reflects not merely the use of external assistance, but a relinquishing of cognitive control: the user accepts the AI’s response without critical evaluation, substituting it for their own reasoning.”
The paper explores this concept by means of three behavioral experiments, each looking at aspects of decision making with AI. The findings helped me in my thinking about how to safely and effectively incorporate AI into my analytics workflows and tool building.
Overview of the experiments
The authors ran three experiments with a total of 1,372 participants. Each experiment had participants answer a set of seven questions that were designed to require deliberative reasoning.
Some study participants worked alone. Others had access to ChatGPT. The latter were free to use it or not, but either way they had to enter the answer themselves.
The researchers also manipulated how ChatGPT responded with hidden seed prompts so that, on a randomly chosen half of the questions, the AI assistant returned the wrong answer — delivered confidently, with a short, plausible-sounding rationale.
Each experiment compared the accuracy rates of participants with and without access to AI, as well as when AI gave correct and incorrect answers.
- The first experiment had no time constraints and no immediate feedback after a participant answered a question
- In the second experiment, participants had limited time to answer each question, and a countdown clock to heighten a sense of time-pressure
- In the third experiment, participants were given immediate feedback on each question and received a small financial incentive to answer correctly
Result one: worse than no help at all
With no time constraints and no feedback, participants with access to AI did a lot better than the no-AI group when AI gave the right answer, but significantly worse when AI gave the wrong answer.
Accuracy on reasoning problems, with and without an AI assistant
When the assistant was wrong, people did worse than people with no assistant at all
Note that the authors were manipulating the accuracy of the AI answers, so the takeaway here is not about how often AI gets things wrong—the experiment was getting at our propensity to trust AI, whether it is right or wrong. One way of looking at these results is:
- If 46% of answers were right without AI and only 32% were right when AI gave a wrong answer, accuracy dropped by 14 percentage points against a 46% baseline. In other words, roughly 30% of people who could have gotten the answer right on their own repeated the AI’s wrong answer instead (0.14 ÷ 0.46 ≈ 30%).
And the actual truth is a bit worse than that, since participants with access to AI didn’t necessarily consult it for a given question. The results showed that when they did ask AI for an answer, they accepted the wrong answer 80% of the time. Note that given the 46% baseline of accuracy, these were clearly hard questions.
What people did with the AI’s answer once they asked for it
Grey = overrode the AI. Four out of five people passed along a confidently wrong answer.
An aside: flawed data makes error rates worse
While the study set the accuracy rate of AI at about 50%, recent benchmarks show LLM accuracy rates above 90%. So one could argue that people putting too much trust in AI is a relatively small problem, and it gets even smaller with every new model release.
That’s a reasonable argument in some problem domains, but I don’t think those benchmarks should provide comfort to data analysts.
A key aspect of our job — possibly THE key aspect — is to understand, minimize and contextualize flaws and limitations in data.
As an example, say we’re reporting on a campaign and phone calls are up sharply. The AI reads the data, notes the lift, connects it to the creative refresh that went live the same week, and writes a clean, well-reasoned paragraph about it.
Every step of that reasoning is sound. But it’s wrong, because some of those calls were generated by a fraudulent ad network partner inflating conversion numbers. And this is not hypothetical; it happened to me recently.
Bot traffic, broken tagging, privacy controls, and myriad other factors make marketing data a prime breeding ground for red herrings. As a result, an LLM taking data at face value will often produce very plausible but false conclusions.
We can and should provide LLMs with context that describes data limitations, but I can tell you anecdotally that when it comes to marketing data interpretation, LLMs don’t come close to 90% accurate, even with context.
Result two: it gets worse under pressure
The second experiment put people under time pressure. Everyone got worse, and the surrender pattern held. Honestly, that result feels obvious to me — of course rushing people makes them lean harder on the fast answer.
But while perhaps obvious, this result provides a caution for organizations that are making staff cuts or increasing productivity expectations due to “AI efficiency.” If you tell an analyst to provide more answers more quickly, you may get what you asked for at the expense of accuracy.
Result three: don’t trust, verify
In the third experiment, access to an LLM and accuracy of LLM answers mirrored the conditions of the previous experiments, but the researchers paid one set of participants for each correct answer and told them immediately whether they’d gotten each answer right or wrong. A second set received no incentives or feedback.
For the incentivized participants, rejection of wrong answers roughly doubled, from the 20% we saw in the first experiment to 42% in the third. People also got better at telling good advice from bad—they followed the AI more when it was right and less when it was wrong.
Across all conditions, incentives + feedback improved accuracy rates significantly, and the group that received wrong answers from AI continued to bring up the rear.
Accuracy when correct answers were paid and scored
Incentives and immediate feedback lifted every condition — but the wrong-answer group still lagged well behind.
One hopes that being employed as an analyst counts as a form of accuracy incentive, but in any case I think it is neither feasible nor wise to incorporate pay-per-correct-answer into an analyst’s compensation. Unfortunately, the authors did not conduct a feedback-only experiment, but we can reasonably surmise that incorporating a feedback loop into analytics processes will result in accuracy improvements.
Overcoming cognitive surrender
Analytics QA is a new job function
These experiments show that humans are prone to rubber-stamping what AI gives them, so just making sure that AI work passes by a human is not enough. Reviewing AI outputs needs to be in people’s job descriptions and time allocation, and not everyone is suited to this type of work. It is a form of QA, which takes patience, attention to detail, and an enthusiasm for cleaning up other people’s messes, and now LLMs’ messes too. To QA analytics work, that person also needs to:
- Understand the problem domain the data describes, e.g., if they are analyzing Google Search Console data, how well do they understand SEO?
- Understand the inherent limitations and flaws in the data. In my experience, this mostly comes from knowing the mechanics of how the data was collected.
- Have a broad understanding of the forces of marketing. Most of the time, the data we are analyzing provides a narrow window into a larger picture. If we don’t know what’s going on outside that window, we tend to overstate the significance of what we see. This is an even bigger problem for LLMs than it is for humans.
- Understand statistics enough to know when a result is meaningful. An LLM is designed to provide an answer, significant or otherwise.
- Know the intended audience—sometimes a result is factually accurate, but whiffs because it doesn’t match the audience’s technical knowledge or desire for detail.
I check those boxes pretty well, but I get a little sweaty when I think about having to QA AI output at scale. Cognitive surrender starts to sound kind of tempting.
Bake review steps into the analysis process
Add review language to project instructions, AGENTS.md, or wherever you standardize context. Below are examples of prompts—I came up with them while thinking about this study, so I haven’t had the chance to test them in the wild. I’m sure I’ll be making adjustments as I do.
Ask for the plan before the analysis:
“Before you analyze anything, give me your approach, your metric definitions, and your assumptions—then stop. Also tell me what you’d want to see that you may not have access to.”
Make the LLM separate inferences from observations:
“For each conclusion, separate what the data directly shows from what you’re inferring and what you’re assuming about how the data was collected. When you use causal language, say what would be needed to support it.”
Show the steps and methods after the analysis:
“With any result, show me the query you ran, the row count it returned, and the totals for each dimension you grouped by. If you filtered or excluded anything, show the count before and after.”
The point is to design your process so the analyst has visibility into how the work is being done, as it is being done. This positions AI as a collaboration partner rather than a crutch.
Aspire to better rather than faster
Lastly, the time-pressure experiment reinforces my POV that we should be incorporating AI into our workflows in ways that prioritize better over faster. I recently read an article where an analytics consultant described three analytics teams’ journeys with AI. In a nutshell, one failed and one succeeded. The third stumbled, then used AI to improve documentation and knowledge sharing. The result was that the team became more resilient. I love that.
An LLM can be a powerful tool when writing code and doing analysis, but if we focus on replacing rather than augmenting good people doing good work, we are surrendering more than cognition.
I recommend reading the paper itself for a more complete picture, as well as some interesting findings that I didn’t cover: Thinking—Fast, Slow, and Artificial: How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender
Every month, I share articles and resources I’ve found that help me do analysis better with AI. Check out my newsletter and subscribe here.
