Why QSRA Doesn’t Work – and how to fix it
The real reasons why applying Quantitative Schedule Risk Analysis doesn’t guarantee programme success.
Many programme managers and consultants are familiar with QSRA, virtually an “industry standard” for large scale infrastructure programmes. The aim is to try and establish the schedule risk (ie the probability of meeting key milestones and delivery dates) and therefore ensure risks are mitigated and ultimate project success is secured.
Yet the theory is not often supported by reality in practice:
QSRA traditionally works by:
- Extracting a “sub-set” of the programme plan (typically Level 1)
- Using the rolled-up durations for the “mean” durations for 3-point estimates
- Then adding “best case” and “worst case” estimates to create 3 estimates for each element of the plan
- Monte Carlo analysis is then run on the subset plan to produce probability distributions and allow a predicted probability of success to be extracted
- Identified risks are then “tagged” (loosely) to the estimates to explain the spread and their impact and probability is then estimated
Following this process, QSRA often gives a high probability of success. And yet, experience shows that programmes still miss milestones by a significant amount. Take the UK Crossrail programme, which used QSRA extensively and yet missed virtually every major milestone by considerable margins. Eg: https://learninglegacy.crossrail.co.uk/documents/managing-schedule-risk/
So why is QSRA’s effectiveness and accuracy so far from satisfactory?
The premise is flawed from the very start because QSRA is based on the “current” plan, which assumes that this plan is “correct”. In reality, the original plan has often been “squeezed” to make it fit the required timescales eg the end milestone is fixed and the tasks that make it up are squeezed until they add up to the required answer.
This means that when the durations are extracted for the Monte Carlo Analysis, the “mean” durations are often smaller than the team’s original estimate. For example, the original estimate for the design might have been 8 weeks, but to make it fit the overall programme, management applied pressure to squeeze this down to 4 weeks. The addition of minimum and maximum durations to create the 3-point estimate corrects this situation a little but, statistically, not nearly enough.
Next, identifying and quantifying the critical path through the programme is the focus of QSRA. Whilst this is appropriate in a static situation, it often does not recognise that there are multiple “potential critical paths” and the longest one may be a different critical path at different times in the programme.
Large workshops are normally used to populate the data ie minimums, maximums, risks, probabilities etc. These workshops can take a long time with many points of view and gaining agreement can prove very difficult, often leading to guesswork and a general, catastrophic, lack of rigor. In particular, the risk register is only loosely linked to the estimates, so the spread of data is only partially explained.
As a consequence, each QSRA exercise normally takes weeks and the desire to repeat it frequently during long programmes is low, despite the fact that it should be a critical requirement as the programme and risks are continuously changing.
Is a more accurate method of analysis possible?
The answer lies in a more strategic extraction of the programme schedule and a combination of qualitative and quantitative assessment of current assumptions which are rigorously captured.
Over many years, De-RISK has developed a more strategic and accurate approach to programme delivery assurance. The resulting Strategic Delivery Assurance methodology has been used on large scale programmes to establish the schedule risk (ie the probability of meeting key milestones) and what needs to be done to manage the delays or take opportunities.
“Our experience has shown a far higher degree of predictability, process effectiveness and accuracy. For example, where our SDA methodology has predicated delays, and mitigating actions are not followed through, the predicted delays materialise.” – Keith Baxter, De-RISK Managing Director
A Strategic Delivery Assurance Approach can bridge the gap where QSRA fails:
Strategic Delivery Assurance or SDA works by building a “strategic” extraction of the programme schedule ie the Potential Critical Path Network (PCPN) in a short workshop with senior programme representatives. The PCPN shows the potential critical paths through the schedule only and is not dictated by the structure of the current plans.
The “bricks” (chunks of activity) that make up the PCPN are each allocated to “brick owners” (ie the best person to estimate the durations at this point in the programme) who are interviewed in 1-2-1 interviews to provide four estimates from most optimistic to “disaster scenario” and also rigorously capture the assumptions that need to hold to manage each component of the brick.
The 4-point estimates in SDA allows contingency and disaster scenarios to be expressed in each brick rather than just a “worst case”
Monte Carlo analysis is then run on the PCPN to produce probability distributions and allow a % of success to be extracted for both “contingency” and “disaster” scenarios – for each potential critical path.
Results are benchmarked to check appropriateness of estimates relative to the current phase of the programme which is very powerful way of judging overall optimism or pessimism within the team.
And the exercise to get the % probability of success, and the assumptions that need to be managed in order to deliver the programme on time, takes days rather than weeks.
Importantly, it is worth noting that SDA can be used in conjunction with QSRA to strengthen/replace specific elements of the process and tools if the programme is already heavily invested in the QSRA process.
The Strategic Edge of SDA over QSRA:
Download the full PDF document for further reading – PDF DOWNLOAD HERE