Fish Food 660: Think-prompt-think
Lemon juice and LLMs, the AI bubble, the death of the corporate job, how people are really using ChatGPT, and the decline of free play
This week’s provocation: Invisible ink and visible ignorance
In 1995, McArthur Wheeler walked into two Pittsburgh banks in broad daylight and robbed them with his face uncovered. When police arrested him hours later using surveillance footage, Wheeler was genuinely shocked. He'd rubbed lemon juice on his face (the same substance used in invisible ink) believing that it would make him invisible to cameras. He'd even tested his theory by taking a Polaroid of himself (though he likely aimed the camera poorly) and when no face appeared in the frame, he felt confident in his plan.
This rather peculiar case caught the attention of a Cornell psychologist who, over lunch, mentioned the story to his graduate student. The case was so unusual that the two of them wondered if Wheeler had somehow lacked the ability to recognise just how wrong he was in his belief. So they designed a series of experiments testing Cornell undergraduates on various skills (logical reasoning, grammar, humour recognition) and then asked them to estimate how well they'd performed compared to their peers. The results revealed a striking pattern. Those who scored in the bottom quartile consistently overestimated their performance by about 50 percentile points. The least competent people, it seemed, were the least aware of their incompetence.
David Dunning and Justin Kruger had just discovered what has become known as the ‘Dunning-Kruger effect’. As a follow up to their initial experiment the two psychologists brought poor performers back and taught them the skills they lacked. As these participants became more competent, they also became more accurate at assessing their own abilities. The metacognitive ability to evaluate one's performance, they realised, requires the same expertise needed to perform well in the first place. It's like a catch-22 of competence in that those who most need to improve are least equipped to realise it.
Despite what’s on the visual above, this is not about stupidity. Rather, the effect reveals the blind spots that we all have in domains where we lack expertise. Even experts fall prey to it when they venture outside their specialties. The ‘Nobel Disease’ (or Nobelitis) for example, is the phenomenon where Nobel Laureates venture outside their traditional area of expertise and make embarrassingly wrong pronouncements. Linus Pauling, after winning two Nobel Prizes including one for Chemistry, became convinced that massive doses of Vitamin C could cure cancer and common colds, despite overwhelming evidence to the contrary.
The flip side of the Dunning-Kruger story emerged from the same Cornell experiments, but it arguably took longer for people to appreciate its significance. In those original studies, whilst poor performers drastically overestimated their abilities, the top performers consistently underestimated theirs (students who scored in the 87th percentile, for example, guessed they were merely in the 68th). The most competent people systematically sell themselves short. In effect, they realise how much they don’t know, and how much there is still to learn about a topic. They are like experienced hikers who've climbed high enough to see the entire mountain range stretching out before them, while the beginners in the foothills see only the single peak in front of them.
Competent people often suffer from what is sometimes called the ‘false consensus effect’ in reverse. When something comes easily to you, you assume it comes easily to everyone. The mental operations that feel automatic to an expert become invisible to them. It’s like a native speaker who can't explain the grammar rules they follow perfectly. The more skilled you become, the harder it is to remember what it felt like not to know. There's an almost tragic irony here: incompetence breeds confidence while competence breeds doubt. The very expertise that makes someone exceptional also makes them acutely aware of their limitations.
The AI field presents a perfect Dunning-Kruger storm. LinkedIn is currently full of AI gurus who are confidently proclaiming about the future and who seem to know enough to see AI's potential but not enough to recognise its brittleness. This can sometimes feel a bit like having weather forecasting dominated by people who've only seen sunshine. Meanwhile, the researchers who genuinely understand the current fragilities and the future challenges often sound almost apologetic when discussing AI's capabilities.
This Economist piece from last month (£) talks about the risks of cognitive offloading and how, if we accept answers at face value, we stop questioning. AI can make us feel like we know more than we actually know about any topic without us even taking the trouble to think too hard about it. But the piece also makes the point that none of this is inevitable. How we use AI is at least as important as the technology itself.
In June, MIT released the (well-discussed) findings of a limited study into the cognitive cost of using LLMs. They divided a group of participants into three groups, an LLM group, a search engine group, and a ‘brain-only’ group, and tasked each with writing an essay whilst monitoring their brain activity using EEG. The finding which many people jumped on at the time is that the groups had significantly different neural connectivity patterns:
‘Brain connectivity systematically scaled down with the amount of external support: the Brain‑only group exhibited the strongest, widest‑ranging networks, Search Engine group showed intermediate engagement, and LLM assistance elicited the weakest overall coupling.’
In other words, when we outsource tasks that require thought to an LLM, our level of engagement, our ownership of the results, and our ability to recall the outputs all reduce significantly.
But there was another interesting finding from that study which garnered much less attention at the time. After three rounds of completing tasks with no tech support the brain-only group were then asked to use an LLM and when they did that they showed a significant increase in brain connectivity across all EEG frequency bands: ‘AI-supported re-engagement invoked high levels of cognitive integration, memory reactivation, and top-down control’.
This is why what Francois Grouiller calls a ‘Think-Prompt-Think’ approach is so important for integrating AI into processes and tasks, particularly those that require careful consideration like strategy and problem solving. Start with just your brain - think about what it is that you are actually trying to do, form hypotheses, consider options, identify assumptions that you may have. Then bring in the LLM to open up new ways of thinking, reframe a problem, source and summarise inputs, challenge, extend and fine-tune. And then step back from the outputs, bring in your human judgement and intuition to assess, improve, edit, finesse, and add your own voice.
It’s so tempting to go straight to the LLM and type in your question, to use it passively like a search engine or an answer machine but the Dunning-Krugeresque risks are very real. This is why qualities such as intellectual humility, curiosity, as well as critical thinking will be so needed in the era of AI. We need to work harder than ever to remove the status barriers from admitting when we don’t know something, and to welcome challenge, constructive debate and healthy conflict. The real advantage comes when AI supports rather than replaces cognitive engagement.
Interestingly, I recently had my first request from a client for a session with their team on ‘Critical Thinking in the Era of AI’ (I’m delivering it next week). Some leaders, it seems, are already aware of the risks associated with cognitive outsourcing and are taking steps to prepare their teams to work with LLMs as a true thought partner but sadly, I suspect that this is an exception rather than the norm.
One final reflection on this. A good learning or problem-solving process uses feedback loops to correct misperceptions or misdirection. But AI development is moving so fast that normal forms of competence calibration become difficult. By the time you’ve accurately assessed your understanding of current AI, the technology has already evolved beyond that understanding. In this environment, the considered human-AI-human approach that I’m advocating for here becomes more important than ever. Otherwise we’re all at risk of becoming McArthur Wheelers in our own way, rubbing our faces with outdated knowledge and hoping it makes us invisible to our own ignorance.
Rewind and catch up:
Pioneers, Settlers and Town Planners
AI, creativity, and lived cultural philosophies
How AI supercharges strategy and planning
Using AI for simulation and scenario planning in strategy
Image: By 忍者猫 - Own work, CC0
If you do one thing this week…
A couple of related and really interesting perspectives on where we’re at with AI right now, and where it’s going. This essay from Jerry Neumann proposes that rather than creating fortunes for investors AI will actually follow the shipping containerisation model, with new value spread very thinly. AI, he argues, currently looks more like the end of the ICT wave rather than the beginning of a new one. Pair that with Azeem Azhar’s sensible framework looking at whether we are in an AI bubble (TL:DR not quite yet).
Photo by Zdeněk Macháček on Unsplash
Links of the week
‘I keep meeting people who describe their jobs using words they'd never use in normal conversation. They attend meetings about meetings. They create PowerPoints that no one reads, which get shared in emails no one opens, which generate tasks that don't need doing’. I only got round to reading it this week (after it being well shared on Substack) but Alex McCann’s post on ‘The Death of the Corporate Job’ captured what he called ‘the hidden economy of nonsense’ in a way that seemed to resonate with a lot of people. So much so that he wrote a part two in which he summarises some key themes from all the feedback and comments he got on the original post
OpenAI released the findings of what they say is the largest study to date into how people are actually using ChatGPT. By July 2025, 18 billion messages were being sent each week by 700 million users, and the most common use case categories are various applications for writing and ‘practical guidance’, which is interesting because some of the latter (like how-to advice) is directly replacing search (HT Dave Chaffey)
This looks really good - Claude (which I find myself using more and more) can now create and edit files.
Interesting new research (based on 4,000 users of AI) that suggests that 80% of people would be receptive to advertising on AI platforms to support free access but only if it was contextually relevant and clearly labelled
A new paper from Carat (and their CSO Matt Willifer) looking at ‘how marketers can harness AI-powered algorithms to drive growth whilst maintaining the human insight that creates a competitive advantage’. I liked the delineation between different types of algorithms (brand, relationship, growth, companion)
This week I came across this essay by Psychologist Peter Gray, arguing that we are depriving children of one of the most essential areas of development - free play. This is the kind of unsupervised play where the child makes up the game and the rules, and that has been shown to build confidence, resilience, and autonomy. Rather than filling every moment of a child’s time with over-scheduling, adult direction and digital immersion, he argues that we should develop more social spaces and unstructured free time for children. Very thought-provoking.
And finally…
Russell Davies (who wrote the brilliant ‘Everything I Learned in Life I Learned From Powerpoint’) is generously setting up a ‘Presentation Club’ where people can practice presentations in a ‘safe, supportive, welcoming space’. The first one is online, on November 5th. Great idea.
Weeknotes
This week I’ve still been overseas but I had an interesting (and challenging in some ways) week working with a space tech business (they make technology that goes into satellites). The theme was focused on creating high-performing teams which is an endlessly fascinating subject and something I’d like to do more on. Next week I’m back in the UK for a few days, running a workshop with an agency on how they can integrate AI into strategy and planning, and then off to Venice to work with Trinity Business School and a group of Brazilian Credit Cooperative leaders.
Thanks for subscribing to and reading Only Dead Fish. It means a lot. This newsletter is 100% free to read so if you liked this episode please do like, share and pass it on.
If you’d like more from me my blog is over here and my personal site is here, and do get in touch if you’d like me to give a talk to your team or talk about working together.
My favourite quote captures what I try to do every day, and it’s from renowned Creative Director Paul Arden: ‘Do not covet your ideas. Give away all you know, and more will come back to you’.
And remember - only dead fish go with the flow.