Measuring Innovation and Transformation Success
By the end of this module you will be able to tell the difference between delivering a programme and delivering its promised benefits, repeat a digital maturity assessment honestly against its own baseline, revisit a digital business case with eighteen months of real numbers instead of a forecast, and combine all of it into one scorecard a board can trust precisely because it isn't flattering.
Digital Transformation Benefits Calculator
Enter the investment, expected annual benefit and implementation cost, and see the ROI, payback period and three-year net benefit.
10.1Benefits Realisation versus On-Time, On-Budget Delivery
A programme that finishes on time and within budget has answered one question well and left another one entirely open. Project success has traditionally been judged against what is often called the iron triangle — cost, time and quality — but Atkinson (1999) makes the case that this is a weaker test than it looks: cost and time are, at best, two guesses made when least is known about the project, and quality is "a phenomenon", an emergent property that different stakeholders judge differently, rather than a fixed line to hit. A programme can satisfy all three and still fail the test that actually matters to the people who funded it — did the organisation get what it paid for?
That second test is benefits realisation. The Project Management Institute (2019) defines benefits realisation management as the discipline of identifying, planning, tracking and confirming that the benefits promised in a business case are actually delivered — a distinct activity from delivering the project's outputs, and one that typically runs for longer. Most digital benefits do not appear the day a system goes live; cost savings, revenue growth and customer-experience gains accumulate as adoption spreads and old habits fall away, often over months or years after the project itself has formally closed and its delivery team has moved on. That time lag creates an accountability gap unless someone is explicitly made responsible for it: a project manager can reasonably be judged on delivery, but benefits realisation needs an owner — usually the business sponsor — who is still answerable for the numbers long after the delivery team's job is done.
Conflating the two measures is not a harmless simplification; it lets a programme declare victory the moment it goes live, before the only evidence that actually matters — did the benefit show up — has had time to exist. Separating them is what makes a later, honest "did it work" conversation possible at all.
10.2Re-Measuring Digital Maturity
A baseline is only useful if someone goes back and measures against it. Module 1 scored Thornfield's digital maturity at 2.1 out of 5 across strategy and leadership, technology and data, customer experience, and innovation culture — a four-dimension model consistent with the strategy-led view of maturity in Kane, Palmer, Phillips, Kiron and Buckley's (2015) research. Re-measuring maturity means returning to that identical model, dimension by dimension, rather than quietly redesigning the scorecard to flatter whatever has actually changed; a maturity assessment that measures different things at baseline and re-measurement cannot honestly report a before-and-after figure at all, because the two numbers are not describing the same thing.
Interpreting the result is where the discipline gets harder than the arithmetic. A score that has risen but still sits below its target can mean several quite different things, and the re-assessment's job is to say which: the organisation may simply be behind where it should be; it may be on a longer, perfectly sound path than the roadmap first assumed, with the shortfall a matter of timing rather than direction; or the original target may itself have been set with more optimism than evidence. Each explanation calls for a different response — accelerating effort, extending the timeline, or revising the target — and a re-assessment that doesn't distinguish between them leaves leadership no better placed to decide than before it was commissioned.
There is also a quieter risk worth naming: whoever runs the re-assessment usually has some stake in the result looking like progress, which creates a subtle pressure to score generously. Guarding against this means being explicit, before the scores are seen, about what evidence justifies moving a dimension up or down — the same discipline that made the original baseline credible in the first place.
10.3The Financial Case Revisited
Module 2 built Thornfield's digital business case around the structure Ward and Peppard (2002) set out: costs, and benefits split into those that are quantified in money, those that are measurable but not easily monetised, and those that are observable only qualitatively. Revisiting the financial case means returning to that same structure with actuals in place of forecasts — recalculating return on investment, payback period and net benefit on what has actually happened, not on what the original business case projected would happen.
A gap between the original projection and the actual result is not, by itself, evidence that the original case was wrong. Flyvbjerg (2006), drawing on his team's studies of large transport infrastructure projects, argues that cost and benefit forecasts are systematically distorted by optimism bias — the ordinary human tendency to underestimate cost and overestimate benefit — and, in some cases, by strategic misrepresentation, where a sponsor shades the numbers to get a proposal approved. A digital business case carries the same two risks on a smaller scale. So a shortfall against target has at least two live explanations that call for different responses: the original estimate may have been too optimistic, in which case the lesson is about how the next business case should be built; or the benefit may simply be arriving more slowly than planned, in which case the lesson is about adoption, not about the estimate. Distinguishing between the two requires looking at the mechanism behind the shortfall, not just the size of the gap.
10.4Reporting to Leadership, Honestly
A board scorecard tends to collect metrics of genuinely different kinds — financial, maturity, customer, team — and the temptation is to let the best-looking number carry the room. Customer Net Promoter Score is a useful case study in why that temptation should be resisted. Reichheld (2003) proposed NPS as a single figure calculated from the percentage of customers who are promoters (likely to recommend) minus the percentage who are detractors, answering one recommendation question on a 0–10 scale, and argued it was a cleaner proxy for customer loyalty than longer satisfaction surveys. The simplicity is also the limitation: compressing an entire distribution of customer sentiment into one number discards the texture a longer survey would have kept, and NPS benchmarks vary enough by sector that a score which looks poor in one industry would be considered strong in another — a single number read without its sector context, or without the qualitative feedback behind it, is easy to misread in either direction.
The same caution applies to combining NPS with the other metrics on a transformation scorecard: maturity, cost-to-serve and team engagement are measured in different units, on different scales, for different reasons, and no single one of them should be allowed to stand in for the whole picture — a strong engagement figure does not offset a weak financial return, and a strong financial return does not excuse a maturity score that still lags its target. The discipline that makes a scorecard trustworthy is reporting every metric against the target set for it at the outset, not against a revised, more comfortable goal chosen after the fact because the original one looks unflattering. A board that sees the original targets alongside the actuals can do its own weighing of what matters most; a board shown only the metrics that look good, or targets quietly softened to match the result, cannot.
Thornfield's Eighteen-Month Scorecard
Eighteen months into Project Northstar, Thornfield Logistics Group assembles its first full transformation scorecard for the board, built on exactly the structure this module teaches: actuals reported against original targets, not against a softened substitute.
Digital maturity, re-measured against the identical four-dimension model used for the 2.1/5 baseline in Module 1, now stands at 3.4 out of 5 — short of the 4.0 target set in the original roadmap, but ahead of the pace that roadmap assumed at this stage, helped by the Pathfinder Replacement Squad delivering its minimum viable booking module by its third sprint, even though the full replacement is not yet finished. Customer Net Promoter Score has moved from 24 to 33, against a target of 40; read alongside sector benchmarks and the qualitative feedback behind it, rather than as a bare number, Sofia Marsh's team judges the gain real; in this illustration it is concentrated among SME customers using the new tracking view, with little movement yet among larger accounts who have not adopted it. Cost-to-serve is down 5%, against the 8% target set out in the original £18m business case — a shortfall the team traces to slower-than-planned adoption among rural-route drivers, the same pattern Module 3 found in the telematics pilot, rather than to any error in the original benefit estimate. Team engagement across Project Northstar's wider team, measured at 74% twelve months in (up from 58% at the start) in Module 9, is carried into this scorecard unchanged, with no target recorded for it, since the next engagement survey falls after this reporting point.
No figure is reframed to look better than it is, and none is dropped because it falls short. The board accepts the scorecard not because every target was met, but because it can see exactly what was promised eighteen months ago, what has actually happened, and a credible account of why — ahead of schedule on maturity, behind on NPS and cost-to-serve, and visibly improved on the engagement of the team delivering it. That is what a credible transformation report looks like: not a win, and not a failure, but an honest account a board can actually use to decide what happens next.
Knowledge Check
Why can a programme that finishes on time and on budget still fail the test that actually matters to the people who funded it?
Because the iron triangle of cost, time and quality (Atkinson, 1999) measures delivery, not benefit; benefits realisation management (Project Management Institute, 2019) is the separate discipline of tracking whether the benefits promised in the business case actually land, often long after the project itself has closed.Thornfield re-measures its digital maturity at 3.4 out of 5, against a 4.0 target. Why is this not straightforwardly "behind"?
Because the re-assessment, using the identical dimensions as the 2.1/5 baseline, finds Thornfield ahead of the pace the original roadmap assumed at this stage, even though it has not yet reached the target figure — a gap to target can reflect timing rather than underperformance, and the two explanations call for different responses.Why should a shortfall against an original financial target, such as Thornfield's 5% cost-to-serve reduction against an 8% target, not automatically be read as an error in the original business case?
Flyvbjerg (2006) found that project forecasts are systematically distorted by optimism bias and sometimes strategic misrepresentation, but a shortfall can equally reflect slower-than-planned adoption rather than a flawed estimate; the two explanations are distinguished by examining the mechanism behind the gap, not by the size of the gap alone.Why does reporting a mixed result against original targets, rather than reframing it to look better, make a scorecard more credible to a board, not less?
Because a board shown the original targets alongside the actuals — as Thornfield's board is, including a Net Promoter Score of 33 against a target of 40 (Reichheld, 2003) — can judge the gap for itself; reframing any target after the fact, or letting one flattering metric substitute for the others, removes the board's ability to do that judging honestly.Go back to Module 2, where Thornfield's business case split its benefits into those quantified in money, those measurable but not easily monetised, and those observable only qualitatively, and where the first driver of change was three major customers leaving over poor real-time visibility. In the worked example, the NPS gain from 24 to 33 is concentrated among SME customers using the new tracking view. Which kind of benefit is it, and why does the drivers analysis make it more than a soft number?
It is a measurable benefit that is not easily monetised (Ward and Peppard, 2002), so it belongs on the scorecard in its own units rather than being forced into money or left out; and because the platform bet was chosen to answer the customer-visibility gap that drove customers away, a gain among the customers actually using the tracking view is direct evidence on the reason the £18m was invested, while the lack of movement among larger accounts that haven't adopted it shows where that reason is still unanswered.
Resources & Further Practice
- Project Management Institute (2019): Benefits Realization Management: A Practice Guide. Newtown Square, PA: Project Management Institute — standardised definitions and a life-cycle framework for benefits realisation, the discipline this module's first lesson distinguishes from on-time, on-budget delivery.
- Atkinson (1999): Atkinson, R. (1999) 'Project management: cost, time and quality, two best guesses and a phenomenon, it's time to accept other success criteria', International Journal of Project Management, 17(6), pp. 337–342 — the critique of the iron triangle used to open this module.
- Flyvbjerg (2006): Flyvbjerg, B. (2006) 'From Nobel Prize to project management: getting risks right', Project Management Journal, 37(3), pp. 5–15 — optimism bias and strategic misrepresentation in forecasts, the two explanations a shortfall has to be tested against.
- Kane et al. (2015): Kane, G.C., Palmer, D., Phillips, A.N., Kiron, D. and Buckley, N. (2015) 'Strategy, Not Technology, Drives Digital Transformation', MIT Sloan Management Review and Deloitte University Press — the maturity research behind the baseline this module re-measures.
- Reichheld (2003): Reichheld, F.F. (2003) 'The one number you need to grow', Harvard Business Review, 81(12), pp. 46–54 — the original Net Promoter Score article; read alongside independent commentary on NPS's sector-benchmark variation and its loss of detail versus a fuller satisfaction survey before relying on it as a sole customer metric.
- Ward and Peppard (2002): Ward, J. and Peppard, J. (2002) Strategic Planning for Information Systems, 3rd edn. Chichester: Wiley — the quantified, measurable and qualitative benefit categories, recalled from the digital strategy module.
- Further practice: Run the Digital Transformation Benefits Calculator with your own programme's actual investment and benefit figures, then carry your completed scorecard into Module 11, where it becomes one input to the full transformation roadmap.
Reflect on Your Transformation Scorecard
- What I did: name the programme you measured, the benefits and owners in your tracking plan, how your maturity re-score compared with its baseline, and the metrics and targets on your scorecard.
- What I learned: note one gap you first read as failure that turned out to be a matter of timing or an optimistic target, or the reverse, and the evidence that told you which.
- What I'll change: write one habit you will take into reporting change at work or in the organisation you studied, for example setting out the evidence needed to move a score before anyone sees the result.
Did this module make sense?
Thanks for saying so: the Faculty sees every “not really”.
Which part lost you? (optional, anonymous)
Thanks: that tells the Faculty exactly where to look.
In the meantime:
Key terms in this module
- Benefits realisation
- The discipline of tracking whether the benefits promised in a business case were actually delivered, by whom and by when — distinct from simply delivering the project on time and budget.
- Digital maturity model
- A staged framework for assessing and re-assessing an organisation's digital capability over time, used here as a before/after measure rather than a one-off audit.
- Net Promoter Score (NPS)
- A customer-loyalty metric calculated from the percentage of promoters (likely to recommend) minus the percentage of detractors, on a 0–10 recommendation question — widely used but a single number that should be read alongside qualitative feedback.
Thanks. Good to hear.