Artificial Intelligence and Machine Learning Benefits? Is there any Evidence?
27/01/2020

The world is awash with AI and ML programmes and projects – and marketing hype!
There is clearly a place for such applications – we heard recently about AI (actually it’s good old sophisticated image-recognition) beating experienced doctors in identifying breast cancer, we’ve heard about its use in some automated driving situations, etc.
However, although many organisations are developing and deploying these applications, it appears the world has yet to see any hard evidence that they are delivering benefits to stakeholders (apart from the developers and suppliers of such systems) either in terms of cost-savings/avoidance or increasing effectiveness / income.
There is overwhelming material out there about what they can deliver in the future – but not a lot on what has been delivered so far.
Our evidence for the lack of evidence comes from our own observations, as well as Dominic Cummings’ recent blogvert where he refers to a paper by Makridakis, Spiliotis and Assimakopoulos.

Some golden nuggets:
- “These studies contributed to establishing two major changes in the attitudes towards forecasting: First, it was established that methods or models, that best fitted available data, did not necessarily result in more accurate post sample predictions (a common belief until then)”
- “Second, the post-sample predictions of simple statistical methods were found to be at least as accurate as the sophisticated ones. This finding was furiously objected to by theoretical statisticians …”
- “A problem with the academic ML forecasting literature is that the majority of published studies provide forecasts and claim satisfactory accuracies without comparing them with simple statistical methods or even naive benchmarks. Doing so raises expectations that ML methods provide accurate predictions, but without any empirical proof that this is the case.”
· “In addition to empirical testing, research work is needed to help users understand how the forecasts of ML methods are generated (this is the same problem with all AI models whose output cannot be explained). Obtaining numbers from a black box is not acceptable to practitioners who need to know how forecasts arise and how they can be influenced or adjusted to arrive at workable predictions.”
- “A final, equally important concern is that in addition to point forecasts, ML methods must also be capable of specifying the uncertainty around them, or alternatively providing confidence intervals. At present, the issue of uncertainty has not been included in the research agenda of the ML field, leaving a huge vacuum that must be filled as estimating the uncertainty in future predictions is as important as the forecasts themselves.”
Work by Genevera Allen, Associate Professor at the Rice University in Houston, Texas also questions the lack of evidence – and, more importantly, the apparent lack of interest in even attempting to collect this evidence!

Some golden nuggets:
- ” … scientists must keep questioning the accuracy and reproducibility of scientific discoveries made by machine-learning techniques until researchers develop new computational systems that can critique themselves.”
- “The question is, ‘Can we really trust the discoveries that are currently being made using machine-learning techniques applied to large data sets?'” Allen said. “The answer in many situations is probably, ‘Not without checking,’ ….”
- “A lot of these techniques are designed to always make a prediction, they never come back with ‘I don’t know,’ or ‘I didn’t discover anything,’ because they aren’t made to.”
- “But there are cases where discoveries aren’t reproducible; the clusters discovered in one study are completely different than the clusters found in another,” she said. “Why? Because most machine-learning techniques today always say, ‘I found a group.’ Sometimes, it would be far more useful if they said, ‘I think some of these are really grouped together, but I’m uncertain about these others.'”
When we blogged about Matthew Syed’s “Black Box Thinking” in 2016, if you don’t have a properly constructed “closed loop” (in an engineering sense, and “open loop” in the Syed sense) learning system when experimenting with new applications, then we have no idea if we’ve actually improved things!
So, like Dilbert, you may want to be wary about where and how you apply AI and ML! And don’t start with the coffee machine!
Categories & Tags:
Leave a comment on this post:
You might also like…
How do I reference… a newspaper article in the APA7 style?
If you're using newspaper content in your work, you may be wondering how to reference it. Is it exactly the same as a journal article reference? Well, it's pretty similar. Here's a short guide. To ...
Resource trial: Writefull
Throughout June, Library Services are running a trial for Writefull which provides tools to help with academic writing. Writefull's support includes proofreading, spelling and grammar checking your work. It can also help you craft your ...
Working smarter, cleaner and greener: The future of manufacturing and materials
Everything we own, use or interact with in life starts with a material or materials. But even we in the industry admit that materials are a huge part of what has led us to where ...
Hubert Ovie Madise: My group design project
'Hubert Ovie Madise, what have you been up to the past ten weeks?' The Cranfield School of Water, Energy and Environment (SWEE) Group Design Project (GDP) module - that's what! The SWEE GDP module ...
The importance of big ideas: How small businesses can maximise their impact
Every year the Bettany Centre for Entrepreneurship hosts our annual entrepreneurship conference VentureDay. We are proud to have some small and medium enterprise (SME) owners on staff, including Heni Cloake and Gabriela Pearson, who share ...
How do I access the full-text of Harvard Business Review (HBR)?
This is one of the most frequently asked questions in the School of Management Library, presumably because HBR is such a key management journal and is renowned worldwide. The short answer is via EBSCO Business ...