Performance Reporting Measures vs Performance Management Measures – Part 5
12/02/2020
Sophisticated Statistical Treatment of Measures – Part ii
You may have read my previous blogs comparing Performance Reporting Measures vs Performance Management Measures.
Performance reporting is littered with measures that may appear to carry meaning for some people, but in our observations, have been misleading and impenetrable to many. And certainly don’t help understanding nor how to improve!
Here are some examples of reporting measures that we introduced previously:
- % items completed: % implies a ratio – with a numerator and denominator. E.g. % Repairs Completed defined by (Number of Repairs Completed / Total Number of Repair Calls) * 100
- % completed within some timeframe: E.g. From a previous blog’s A&E Figures, we saw % A&E attendants seen in 4 hours or under.
- Complicated Measure Combinations: E.g. % Forecast Accuracy in Supply-chain
- Applying sophisticated statistical treatment to raw performance measures that only stats specialists can read: E.g. Exponentially weighted moving averages
- Statistical representation of a population of people or things: E.g. Electric Car Use by Country
This week we’ll look at a second example of the sophisticated statistical treatment of simple performance measures – moving average charts
Moving Average Charts
So we already know from our previous 5 blogs on this subject that % measures present problems, and measures that use % success within a constraint (usually time) / target inhibit understanding, complicated measure combinations are dangerous, and how out of context correlation can be misleading – so we’ll not revisit them here.
So why can the sophisticated statistical treatment of simple performance measures, in this case moving averages, confuse or mislead?
First, we ask, what is a moving average. In fact there are several kinds of moving average, and Wikipedia does a pretty good job of explaining what they are:
https://en.wikipedia.org/wiki/Moving_average
Remember the rule of thumb from our previous blogs – the more sophisticated the algorithm you apply to a measure, the more distant you become from what is actually going on. Let’s look at the simplest moving average, which is referred to, funnily enough, as a Simple Moving Average.
Simple Moving Averages can often be used to smooth a series of data points, for example if the number of orders for a particular vaccine peak before and during the flu season, and drop in between, financial types might want to look at some averaged result in order to establish budgets / forecasts. We’ve seen in previous blogs how some crimes (e.g. burglary) are seasonal, volumes of people attending A&E are seasonal, GDP on a monthly basis bounces up and down over the year, and so on. Statisticians often resort to smoothing these results by taking a 3, 6 or 12 month rolling average in order to try to establish potential trends and forecast forward.
Why can such statistical treatment potentially mislead, particularly when we’re trying to improve processes and systems?
We’ll work with a simple 6-month moving average as illustrated in the chart below:
The critical thing to note is that more often than not, these moving average charts are presented out of context (i.e. without a chart of the underlying data). And this is the main problem as we’ll see below. But first, let’s explain the chart.
The chart above is an (unweighted or simple) 6-month moving average. The first point on the 6-month moving average is 6 months in, since, obviously, it needs 6 months (November 2013 through April 2014) to calculate a 6-month moving average for April 2014. The 6-month moving average for May 2014 is calculated from December 2013 through May 2014, the 6-month moving average for June 2014 is calculated from January 2014 through June 2014 and so on.
You’ll note that the latest result in October 2014 is the highest in the series of points. It might get some people excited about trying to find the cause of this. But let’s have a look at the underlying data – and we might draw a different conclusion.
This chart is a simple time-series chart of crimes per month from November 2013 through October 2014. Note the significant drop in April 2014!
You’ll note that the last month (October 2014) on the time-series chart below is not unusually high, but, as we saw above, the last result in the moving average chart is the highest of any of the previous 6 results. This is not because October 2014 is unusual, it is because the calculation for October 2014 6-month moving average uses May 2014 through October 2014, and the very low result of April 2014 has dropped out of the calculation. We have observed in our travels, that these spurious “highest results” (or the opposite “lowest results”) can get people pretty agitated and can kick off wasteful investigations around what happened in the latest month, when, in fact, the issue was, in this case, 6 months earlier!
Our recommendation is, in order to maintain context, you ALWAYS display moving average charts (or any other kind of sophisticated statistically treated measure) with their corresponding time-series chart.
That would keep Dilbert happy!
We’ll use the next and final part in this series, to look at statistical representations of populations and when they are useful and when they are not! Dilbert would be super-impressed!
Categories & Tags:
Leave a comment on this post:
You might also like…
Navigating the World of Robotics: My Journey at Cranfield University
Hey there, I'm Manideep, and I'm thrilled to share my experience pursuing an MSc in Robotics at Cranfield University. Let me take you through my journey and how Cranfield became the ...
Exploring safer and smarter airports with the Applied Artificial Intelligence MSc group design project
Artificial intelligence (AI) technologies have experienced rapid development in recent years, spanning from large language models (LLMs) to generative artificial intelligence (GAI). These cutting-edge advancements have significantly impacted various aspects of ...
My aerospace manufacturing journey at Cranfield University
Hey there, I'm Abhishek and I wanted to share my journey into aerospace manufacturing, guided by my experiences at Cranfield University. Let's dive into how this remarkable institution shaped my career ...
Changes to Library Services over Easter, 29 March – 1 April
Libraries on the Cranfield site Both Kings Norton Library and the School of Management Library (Building 111, first floor) will be open 24/7 over the Easter weekend. You will be able to use the study ...
How to present well as a group
You will have put a lot of work into your research or project and want to show everyone what you have achieved or discovered, so you need to impart this knowledge as clearly as possible. ...
Finding Financial Times articles in Factiva
If you are looking for Financial Times articles, look no further than Factiva! But be aware, there is a one-month embargo on content which means the most recent month is unavailable online. You can find the ...