Workshop: Pessimists, realists and liars: why benefits forecasts are always wrong
Claire Dellar, NHS Digital, Benefits expert, educator, change & P3M professional
Claire Dellar skilfully mixed Agatha Christie with Terry Pratchett and even references to Donald Trump to humorously engage her audience on the sources of risk, uncertainty and bias, why they arise and what you can do to mitigate against them.
“Why harrow oneself by looking on the worst side? Because it is sometimes necessary.” Agatha Christie, Sad Cypress
What are the sources of risk, uncertainty and bias?
Claire looked at different mindsets and the sources of error associated with each and reminded us that what people perceive often overrules what they would sit down and logically conclude.
Weak links in the chain of causality of benefits lead to assumptions, which are basically risks, but this is still better than a “hydra-digital estimation” or wet finger in the air approach.
Sometimes the organisation itself encourages lying or over-optimism, for example, risk averse organisations and the attitude of “tell us everything before we release the money” encourages the liars.
Another source of risk, uncertainty and bias relates to the type of project or programmes. For infrastructure-type projects, the starting point for the business case is often cost-effectiveness, but to gain approval for the project the business case needs to be formulated to achieve a specified hurdle rate (1).
In addition, these projects use people to deliver the outputs who don’t realise the benefits, who cannot control who uses and exploits the outputs. However, the people who are going to exploit the outputs cannot deliver them! So, there is a void between user needs and the delivery of the outputs - which realise the benefits.
On the other hand, entrepreneurial-type projects have benefit forecasts which are aspirational. For example, you cannot force someone to spend their ‘freed-up’ time with their family or at the gym and you cannot force business to relocate around a new road.
(1) In capital budgeting, hurdle rate is the minimum rate that a company expects to earn when investing in a project. Hence the hurdle rate is also referred to as the company's required rate of return or target rate. In order for a project to be accepted, its internal rate of return must equal or exceed the hurdle rate.
Ways to address and mitigate against risk, uncertainty and bias
Mature organisations are more likely to be able to challenge pet-projects and develop real options to solve a problem.
Good practice is to take benefits forecasts from the business case and make them the targets by which you measure you are still on track. This increases the level of accountability from the business case stage through the entire lifecycle of the project or programme.
Claire introduced the audience to several techniques for forecasting actual benefits, namely; Reference Class Forecasting, where the size of the evidence base does matter, the Delphi Technique, where is it not about size but what you do with the results that matters, and Stochastic Modelling and Predictions, which she compared to Terry Pratchett’s “resistentialism” which he defines as:
"the tendency of garden hoses, no matter how carefully one coils and stores them, to unloop themselves overnight and tie the bicycle to the lawnmower"
Claire introduced a confidence grid used by NHS Digital to assess how confident you are that you will see the benefits when you think you will. It’s alright to have low confidence in a benefit but don’t understate this in the business case!
Claire also reminded us to be brave and stop when it’s time to stop; fail fast rather than flogging a dead horse.
Setting tolerances for most likely benefits realisation profile and best and worst-case profiles is another tool which can be used to evaluate predicted benefits, as well as monitoring during benefits realisation. Again, it is important to actually monitor against these and be willing to make decisions around whether to continue.
Some final points to consider
Claire acknowledged as good practice, the continued search to uncover new or emergent benefits, whilst recognising that emergent benefits in a non-related area cannot be used to justify a business case that doesn’t meet the originally stated benefits.
Finally, the role of portfolio management is to set priorities and risk appetite, to evaluate multiple projects and programmes and sometimes to say “stop!” Which brings us nicely back to the earlier point levels of organisational maturity; the truth and lies of benefits forecasting.
No one was more surprised than Claire when her background in experimental economics actually turned out to be useful!
Claire’s enthusiastic delivery and deep knowledge of her subject makes her an ideal presenter and seminar leader specialising in benefits realisation and change.
An accomplished public speaker, Claire has presented at conferences, seminars and workshops and she is a guest lecturer at Leeds University. She contributed to the Managing Benefits™ textbook and is a contributing editor of the Change Activation Toolkit and co-author of the NHS Digital benefits realisation training courses, which are spreading throughout the NHS national bodies.
Chair of the national Health Family PDC Specific Interest Group on benefits, Claire works full time for NHS Digital, where she has trained hundreds of people in practical benefits management. Her sanctuary is a cottage in Snowdonia, from which she emerges each week to visit health and care organisations across the country, infecting them with her passion for benefits-led change.
Write up by Alexis Blades
Benefits Management SIG committee member
Claire Dellar: Mindmap
Back to Benefits Summit 2017 mainpage