This week’s provocation: Transformation and the ROI of AI
Last week I was asked to take part in a Cambridge Judge Business School panel on AI transformation. It was a pretty wide ranging discussion but there was a useful lens on this given by the latest McKinsey State of AI 2025 report which had just come out. There were some interesting findings about enterprise adoption of AI including (predictably perhaps) that whilst adoption is near universal it is also shallow, and that many businesses remain stuck in the pilot phase. 88% of businesses for example, now report regular AI use in at least one business function, but usage is often concentrated in a few functions, and many organisations are still piloting use cases.
The ‘bottom-up trap’ and ‘top-down fantasy’
I think it’s easy to get caught up in the hype and believe that every business will be left behind if they don’t move faster with AI adoption. Talking a deliberate, steady approach to integrating potentially transformational technologies is not entirely a bad thing. But I do want to call out one disconnect which I think is a bigger potential challenge than the speed at which organisations are going. This is what Karl Yeh has called the ‘bottom-up trap’ and the ‘top-down fantasy’ problem. Companies, he says, often fall foul of a ‘bottom-up trap’ where local experiments stay siloed and struggle to scale beyond pockets of enthusiasm, efforts lack alignment with enterprise priorities limiting impact and investment, early adopters risk burnout without visible leadership support and so momentum fizzles as pilots fail to embed into workflows or decision-making. Equally the ‘top-down fantasy’ happens when senior leaders announce grand visions without a grounding in practical use cases, strategies assume compliance and overlook frontline realities and adoption barriers, cultural resistance grows when employees feel imposed upon rather than involved, and transformation plans stall when rhetoric outpaces capability building.
This disconnect between senior leadership intent, organisational strategy and the reality of how stuff actually gets done and what will actually help it to be done better is perhaps one of the key reasons why so many organisations are struggling to see a return on AI investments. That MIT study from a few months back that showed that 95% of generative AI pilot projects fail to produce measurable return or impact has come in for some criticism (around methodology) but there are plenty of other studies that show just how challenging it is to prove ROI on AI investments. The McKinsey survey found that over 80% of businesses were seeing no EBIT impact despite some benefits being seen in business units. Interestingly, of 25 attributes that were tested, workflow redesign had the biggest single EBIT impact and yet less than a quarter of businesses in the survey (21%) have fundamentally redesigned workflows and less than 20% had actually tracked GenAI KPIs at all (the top correlated practice for impact on the bottom-line).
Johnson and Johnson’s AI transformation is an interesting counterpoint to this (I’ve written up a longer case study on this here if you’re interested). They began in 2022 with a ‘thousand flowers bloom’ approach which invited ideas for where AI could bring benefit from right across its global business, and then facilitated open, safe-to-fail experimentation, overseen by a centralised AI governance board which ensured feasibility and alignment. This led to a burst of curiosity and creativity that resulted in 900 AI projects, supporting ground-up buy in, aligned experimentation and capability building as teams learned how AI could be (and could not be) applied to their domains.
The challenge then of course was fragmentation and duplication. So last year J & J pivoted to focus on the 10-15% of these use cases which were generating around 80% of the measurable value, concentrating investment in high-impact areas. They disbanded the central AI governance board, distributing the governance to core business units (commercial, R & D, supply chain) ensuring contextual and domain accountability, resource optimisation, and alignment to specific business goals such as accelerated drug discovery, and enhancing sales effectiveness. This new approach meant that they could focus on scaling the highest-value use cases with specific propositions including AI sales copilots, predictive models and simulations in drug discovery and supply chain risk management.
The J & J case study is a good example of starting broad, then narrowing to bring focus and impact, starting with central governance which then devolves as local initiatives scale, creating learning loops involving learning from failed initiatives, and treating it as an evolving capability, moving from AI projects to AI-powered processes.
I’d like to finish this reflection with a few thoughts on assessing ROI, which seems to be an area which many businesses are particularly struggling with. I think it can be useful here to focus on both hard and soft measures (as articulated here by IBM). The former are perhaps the more obvious measures that are easier to get to. But there is real value in also tracking soft measures relating more to things like decision-quality, customer and employee engagement and innovation velocity. Here’s my initial list:
Hard Measures (quantifiable financial and operational impact)
Cost reduction: Track savings from automation, reduced labour, or lower error rates compared with baseline costs.
Productivity gains: Measure increased output per employee or reduced cycle time using process metrics.
Revenue uplift: Attribute incremental sales or cross-sell opportunities driven by AI-powered recommendations or dynamic pricing.
Process efficiency: Compare throughput, defect rates, and operational downtime before and after AI implementation.
Customer acquisition cost: Evaluate change in marketing efficiency or conversion rates due to AI targeting or optimisation.
Resource utilisation: Quantify improvements in equipment uptime, inventory optimisation, or logistics efficiency.
Time to insight: Measure speed of data analysis or decision turnaround relative to prior benchmarks.
Model accuracy and impact: Track model performance improvements that drive measurable business outcomes (e.g., fraud detection rate).
Soft Measures (qualitative or more intangible benefits)
Decision quality: Conduct pre- and post-implementation surveys to assess confidence, speed, and evidence-based reasoning in leadership decisions.
Customer satisfaction: Use NPS, CSAT, or sentiment analysis to detect improvement in perceived experience due to AI-enabled personalisation.
Employee engagement: Track adoption rates, satisfaction surveys, and task satisfaction when AI tools are introduced.
Innovation velocity: Measure frequency of new product ideas, time-to-prototype, or new insights generated using AI.
Brand perception: Use social listening or brand tracking studies to gauge sentiment shifts linked to AI-enhanced services.
Risk reduction: Assess improved compliance, reduced incidents, or more accurate risk scoring as proxies for AI-driven foresight.
Decision traceability: Track auditability and transparency in decision logs to measure governance maturity.
The softer, more intangible measures are harder to quantify, so useful techniques may include the use of balanced scorecards (combining quant KPIs with qual assessments), outcome mapping (attempting to define a cause and effect between AI use cases and strategic goals), decision reviews (using controlled before-and-after case comparisons), AI adoption metrics and qual feedback, and other proxy measures (e.g. speed-to-insight).
Back in May I wrote about using the classic Design Thinking framework of Desirability, Viability and Feasibility (DVF) to assess the value of AI initiatives. Feasibility relates to how easily something can be created, Viability meaning sustainable profitability or benefit to the business, and desirability relating to whether end users actually want what you’re creating. I mentioned then that the main focus seems to be going on feasibility (‘let’s do something with AI, what can we do?’) and viability (‘how can we use AI to maximise benefit to the business?’). It’s remarkably easy to forget about the ‘desirability’ of initiatives, or what will actually work for and be attractive to employees and/or customers. If we are to solve the bottom-up trap and top-down fantasy we need to be able to experiment widely but also scale in a focused, aligned and inclusive way to ensure that we’re bringing employees and customers on the journey with us.
Rewind and catch up:
How to be Interested (Part Two)
In Praise of Working at the Edge
The Future of Agencies in the Age of AI
Photo by Luke Jones on Unsplash
Photo by Nick Fewings on Unsplash
If you do one thing this week…
There was an interesting deep dive this week by Daniel Parris into how much AI audiences will accept in movies and music. He makes the point that the answer is seemingly more nuanced than most reporting suggests. In movies for example, research respondents ‘…were generally supportive of using artificial intelligence for visual effects and dialogue translation, but overwhelmingly opposed generative AI replacing human actors and screenwriting.’ Polls, he says, show more support for the use of assistive AI (which supports human creativity without generating entire artwork) but there’s strong disapproval for art produced without meaningful human involvement. A lot of what he says makes intuitive sense, and yet this week we also had the news that a completely AI generated song is topping Billboard’s Country Digital Song Sales chart. Perhaps there is a difference here between the beliefs and opinions that people are reporting and the reality of a track that seems to have broad appeal regardless of who or what created it? Perhaps it’s really all about how good the finished artwork is? 🤷♂️
Links of the week
Meta have released a new white paper on the future of agencies - it’s nicely complementary to my piece on the same topic from a couple of weeks ago, but is particularly interesting as it uses examples of innovative agency approaches from APAC. Some more obvious examples including outcomes or subscription based pricing, and plugging directly into platforms, but there’s also some more innovative examples such as dramatically faster onboarding and serving SMEs at scale (HT Pete Buckley). Meanwhile, lots of non-WPP people have been crowing (a bit unfairly) about WPP hiring McKinsey to look at their strategy
Sangeet Paul Chowdray on AI, jobs, and what we misunderstand about complementarity (the idea that ‘humans and AI specialize in what each does best, jointly producing more value than either could alone’). When AI complements humans ‘it does not preserve existing jobs; it reallocates human “advantage’”, redefining which skills matter, displacing some forms of work through irrelevance rather than just automation, and concentrating the gains made from the shift. Sangeet also co-authored (with John Winsor) an HBR piece this week on that MIT ‘95% of AI projects fail’ stat, and why asking how you can be in the 5% is the wrong question to ask. We need to think bigger about how we can reimagine the system of work (how control flows through the value chain, where decisions sit and so on). I’ve been reading quite a bit of Sangeet’s writing recently including his new book
In this interview Sam Altman talks about how ChatGPT will eventually do advertising but ‘not like the pay-to-rank, keyword-auction model we know today’. Instead it will (probably) look more like an affiliate model. HT Paul Frampton-Calero
Frank Chimero’s thought-provoking talk on creative agency in the age of AI takes a different angle to the Daniel Parris post I mentioned above, describing three different ways of thinking about the relationship between human creativity and AI (besides the machine, into the machine, beyond the machine) HT Dave Tallon
With AI we’ve reached a critical fork in the road in education and the way we learn, and this is an opportunity to change it fundamentally. Hard to disagree with this.
A handy short video on how to (simply) do JSON prompting (particularly useful for image and video generation)
Well this is a revelatory way of doing long multiplication (from Japan)
And finally…
‘Consulting Slop is an AI-powered strategy generator that produces firm-grade strategic thinking tailored to your company’. A fun takedown which uses AI to generate consulting decks in the style of the big firms, from the folk at Nobl.
Weeknotes
This week I’ve been working in Muscat with Oman’s national electricity provider and their largest telecoms network. Abdulaziz, my ‘fixer’ whilst out here, has told me a lot about Omani traditions and how people live here. On a Thursday night (equivalent to our Friday night) he gets together with his friends and they drive out into the desert, set themselves up atop a sand dune, and cook a meal watching the stars and having a laugh. How wonderful is that? Next week I’ll be flying back to the UK midweek to do an IPA AI in Advertising course and a session with Imperial College Business School.
Thanks for subscribing to and reading Only Dead Fish. It means a lot. This newsletter is 100% free to read so if you liked this episode please do like, share and pass it on.
If you’d like more from me my blog is over here and my personal site is here, and do get in touch if you’d like me to give a talk to your team or talk about working together.
My favourite quote captures what I try to do every day, and it’s from renowned Creative Director Paul Arden: ‘Do not covet your ideas. Give away all you know, and more will come back to you’.
And remember - only dead fish swim with the stream.