In his excellent book, “Fluke”, Brian Klaas quotes the eighteenth-century philosopher David Hume, who says that most of our understanding of cause and effect is based solely on our past experiences. As he puts it, “probability is founded on the presumption of a resemblance betwixt those objects, of which we have had experience and those of which we have had none.”

Probabilities can be useful but can also be misleading and suggest false precision. The future can be dramatically different from past patterns, which can have profound impacts on risk analysis and management. Where risks are being assessed based on repeatable processes, such as stable business operations, the use of historical data to predict future problems is relatively safe. For example, mean-times to failure of critical components or projects that are very similar to ones done previously. But in situations where the venture includes a degree of “innovation”, such as a new project utilising groundbreaking technology, or a new product development, we are in dangerous territory if we put too much value on past data. (See box on ‘Xerox line model’ right).

Brian Kalas references another good summary of this effect by John Kay and Mervyn King in their book, “Radical Uncertainty”. They suggest that probabilities can only be successfully used in situations where “the possible outcomes are well defined, the underlying processes that give rise to them change little over time, and there is a wealth of relevant historical information”. Unfortunately, for pretty much every innovative project that is undertaken, these assumptions don’t apply.

The ‘Xerox line’ model, derives from an analogy which considers the photocopier giant’s use of knowledge and information in two stages. Information which is below the line is effectively copies or re-used versions of older work. While information above the line is where new and truly transformative or original ideas lie.

More on the Xerox line and its use in AI.

At De-RISK, our SDA (Strategic Delivery Assurance) methodology relies on process rigour to bring reliability and predictability to programme delivery. We are very careful about how we use probability as part of our process.

The science philosopher Ian Hacking states that there are two main types of probability statements. Either frequency type probability or belief type probability. The former uses historical data, while it is the latter that we need to capture if we are to successfully assess and manage risk in innovative projects.

SDA focuses on belief type probabilities to assess the risk in the largescale, complex programmes and uses probability very carefully to support this approach. For example:

1. The use of the positive “assumption-based” language in interviewing techniques leads the programme stakeholders to think ahead in terms of the risks within the assumptions they are making, both explicit and implicit. This is far more effective than the negative “risk-based” language that is normally employed in traditional risk management processes, which tends to make people defensive and unwilling to think ahead. Read more on the psychology behind traditional risk management approaches here.
2. The benefit of having both “contingency” and “disaster” estimates, ie 4-point estimating (rather than the limiting 3-point estimating used in techniques such as QSRA, see our blog ‘why QSRA doesn’t work and how to fix it’) allows people to think at both extremes of “bad” without turning every risk into a “Black Swan” event. In SDA, the extremes of “bad” can therefore be modelled in one place and then benchmarked to decide which extreme is appropriate.
3. When sharing the results of our SDA methodology on a specific project, we do use “probability” to explain the likely chances of success, but this is couched in the terms of confidence in a particular outcome and not to suggest false precision. It is the relative values that become the focus, eg “you may only have a 5% chance of success but if you manage these 23 assumptions it theoretically rises to 90%” putting the emphasis on the relative value rather than the numbers to drive appropriate behaviours as its primary aim.