Hirst AcademyInnovation and Change in Marketing
← Course overview
Module 10

Measuring Innovation and Transformation Success

Estimated time: 5–6 hours · Closes the loop from the digital business case built in the digital strategy module by measuring what was actually delivered, including the team and leadership outcomes alongside the financial ones.

By the end of this module you will be able to tell the difference between delivering a programme and delivering its promised benefits, repeat a digital maturity assessment honestly against its own baseline, revisit a digital business case with eighteen months of real numbers instead of a forecast, and combine all of it into one scorecard a board can trust precisely because it isn't flattering.

Interactive Tool

Digital Transformation Benefits Calculator

Enter the investment, expected annual benefit and implementation cost, and see the ROI, payback period and three-year net benefit.

Open the tool →
10.1

Benefits Realisation versus On-Time, On-Budget Delivery

A programme that finishes on time and within budget has answered one question well and left another one entirely open. Project success has traditionally been judged against what is often called the iron triangle — cost, time and quality — but Atkinson (1999) makes the case that this is a weaker test than it looks: cost and time are, at best, two guesses made when least is known about the project, and quality is "a phenomenon", an emergent property that different stakeholders judge differently, rather than a fixed line to hit. A programme can satisfy all three and still fail the test that actually matters to the people who funded it — did the organisation get what it paid for?

That second test is benefits realisation. The Project Management Institute (2019) defines benefits realisation management as the discipline of identifying, planning, tracking and confirming that the benefits promised in a business case are actually delivered — a distinct activity from delivering the project's outputs, and one that typically runs for longer. Most digital benefits do not appear the day a system goes live; cost savings, revenue growth and customer-experience gains accumulate as adoption spreads and old habits fall away, often over months or years after the project itself has formally closed and its delivery team has moved on. That time lag creates an accountability gap unless someone is explicitly made responsible for it: a project manager can reasonably be judged on delivery, but benefits realisation needs an owner — usually the business sponsor — who is still answerable for the numbers long after the delivery team's job is done.

Conflating the two measures is not a harmless simplification; it lets a programme declare victory the moment it goes live, before the only evidence that actually matters — did the benefit show up — has had time to exist. Separating them is what makes a later, honest "did it work" conversation possible at all.

Learning ObjectiveDistinguish benefits realisation from on-time, on-budget project delivery, and explain why a programme can succeed by one measure and fail by the other.
Key ConceptsThe iron triangle of cost, time and quality (Atkinson, 1999); benefits realisation management as the discipline of tracking whether promised benefits are actually delivered, by whom and by when (Project Management Institute, 2019); benefits that land only after project closure; sponsor accountability for benefits versus project-manager accountability for delivery.
Practical ExampleSuppose Project Northstar's telematics pilot at 3 of Thornfield's 22 depots finishes on time and within its budget — a clean delivery by the iron triangle's own measures — but whether it actually produced the customer-visibility and cost-to-serve benefits the original business case promised is a separate question the delivery record cannot answer on its own.
ExerciseFirst read the worked example at the end of this module to see the benefits your tracking plan will eventually have to report on. Then draft a benefits-tracking plan for a change programme in your own organisation: list each promised benefit, who owns proving it was delivered, and when — not necessarily at project closure — it can honestly be measured. If you don't have one to use, draft it for Project Northstar's cost-to-serve and customer-visibility benefits, with COO Daniel Osei as sponsor, and say which role (for example, Head of Marketing Sofia Marsh, or a depot operations lead you define) you would make answerable for each benefit, labelling that choice as your own assumption.
10.2

Re-Measuring Digital Maturity

A baseline is only useful if someone goes back and measures against it. Module 1 scored Thornfield's digital maturity at 2.1 out of 5 across strategy and leadership, technology and data, customer experience, and innovation culture — a four-dimension model consistent with the strategy-led view of maturity in Kane, Palmer, Phillips, Kiron and Buckley's (2015) research. Re-measuring maturity means returning to that identical model, dimension by dimension, rather than quietly redesigning the scorecard to flatter whatever has actually changed; a maturity assessment that measures different things at baseline and re-measurement cannot honestly report a before-and-after figure at all, because the two numbers are not describing the same thing.

Interpreting the result is where the discipline gets harder than the arithmetic. A score that has risen but still sits below its target can mean several quite different things, and the re-assessment's job is to say which: the organisation may simply be behind where it should be; it may be on a longer, perfectly sound path than the roadmap first assumed, with the shortfall a matter of timing rather than direction; or the original target may itself have been set with more optimism than evidence. Each explanation calls for a different response — accelerating effort, extending the timeline, or revising the target — and a re-assessment that doesn't distinguish between them leaves leadership no better placed to decide than before it was commissioned.

There is also a quieter risk worth naming: whoever runs the re-assessment usually has some stake in the result looking like progress, which creates a subtle pressure to score generously. Guarding against this means being explicit, before the scores are seen, about what evidence justifies moving a dimension up or down — the same discipline that made the original baseline credible in the first place.

Learning ObjectiveRepeat a digital maturity assessment against its own baseline and interpret what moved, what didn't, and why a lower-than-target score is not automatically bad news.
Key ConceptsRe-assessing the identical dimensions used at baseline, not a redesigned scorecard; strategy and leadership as a dimension in its own right (Kane et al., 2015); distinguishing "ahead of schedule but short of target" from simply behind; the risk of an assessor unconsciously inflating a self-assessment under pressure to show progress.
Practical ExampleThornfield's 18-month re-assessment repeats the same four-dimension model used for its 2.1/5 baseline and finds overall digital maturity at 3.4/5 — short of the 4.0 target set for the roadmap, but ahead of where that roadmap expected the organisation to be at this stage.
ExerciseUsing the maturity baseline you scored for your own organisation in Module 1, re-score it against the identical dimensions and state, for each, whether the gap to target reflects being genuinely behind, being on a longer path than first drawn, or the original target having been set too high. If your baseline did not set a target for each dimension, set one now (labelled as your own) before you re-score. Before you score, write down what evidence would justify moving each dimension up or down. If you don't have one to use, take Thornfield's 2.1 baseline (weakest on technology and data and on innovation culture) and its 3.4 re-measurement; the case gives only the overall figures, so label any dimension scores you set as your own assumptions.
10.3

The Financial Case Revisited

Module 2 built Thornfield's digital business case around the structure Ward and Peppard (2002) set out: costs, and benefits split into those that are quantified in money, those that are measurable but not easily monetised, and those that are observable only qualitatively. Revisiting the financial case means returning to that same structure with actuals in place of forecasts — recalculating return on investment, payback period and net benefit on what has actually happened, not on what the original business case projected would happen.

A gap between the original projection and the actual result is not, by itself, evidence that the original case was wrong. Flyvbjerg (2006), drawing on his team's studies of large transport infrastructure projects, argues that cost and benefit forecasts are systematically distorted by optimism bias — the ordinary human tendency to underestimate cost and overestimate benefit — and, in some cases, by strategic misrepresentation, where a sponsor shades the numbers to get a proposal approved. A digital business case carries the same two risks on a smaller scale. So a shortfall against target has at least two live explanations that call for different responses: the original estimate may have been too optimistic, in which case the lesson is about how the next business case should be built; or the benefit may simply be arriving more slowly than planned, in which case the lesson is about adoption, not about the estimate. Distinguishing between the two requires looking at the mechanism behind the shortfall, not just the size of the gap.

Learning ObjectiveCompare actual ROI, payback and net benefit against the original business case, and decide what a shortfall means for the programme rather than for the spreadsheet.
Key ConceptsRevisiting the quantified, measurable and qualitative benefit categories set out in the original business case (Ward and Peppard, 2002); optimism bias in the original forecast versus a genuine delivery shortfall (Flyvbjerg, 2006); recalculating ROI and payback on actuals rather than the original projection.
Practical ExampleThornfield's cost-to-serve reduction stands at 5% eighteen months in, against the 8% target in the original £18m business case — a gap the finance team traces mainly to slower-than-planned adoption among rural-route drivers, not to an error in the original benefit estimate.
ExerciseRe-run the Digital Transformation Benefits Calculator using actual investment and benefit figures from a programme in your own organisation, and write two or three sentences on what the gap between the original and actual numbers is actually telling you: optimism in the original estimate, or a benefit arriving more slowly than planned. If you don't have one to use, start from the tool's illustrative defaults for Thornfield's £18m case, reduce the annual benefit to reflect cost-to-serve at 5% against an 8% target, and label that reduction as your own assumption, since the case gives no money figure for the shortfall.
10.4

Reporting to Leadership, Honestly

A board scorecard tends to collect metrics of genuinely different kinds — financial, maturity, customer, team — and the temptation is to let the best-looking number carry the room. Customer Net Promoter Score is a useful case study in why that temptation should be resisted. Reichheld (2003) proposed NPS as a single figure calculated from the percentage of customers who are promoters (likely to recommend) minus the percentage who are detractors, answering one recommendation question on a 0–10 scale, and argued it was a cleaner proxy for customer loyalty than longer satisfaction surveys. The simplicity is also the limitation: compressing an entire distribution of customer sentiment into one number discards the texture a longer survey would have kept, and NPS benchmarks vary enough by sector that a score which looks poor in one industry would be considered strong in another — a single number read without its sector context, or without the qualitative feedback behind it, is easy to misread in either direction.

The same caution applies to combining NPS with the other metrics on a transformation scorecard: maturity, cost-to-serve and team engagement are measured in different units, on different scales, for different reasons, and no single one of them should be allowed to stand in for the whole picture — a strong engagement figure does not offset a weak financial return, and a strong financial return does not excuse a maturity score that still lags its target. The discipline that makes a scorecard trustworthy is reporting every metric against the target set for it at the outset, not against a revised, more comfortable goal chosen after the fact because the original one looks unflattering. A board that sees the original targets alongside the actuals can do its own weighing of what matters most; a board shown only the metrics that look good, or targets quietly softened to match the result, cannot.

Learning ObjectiveCombine financial, maturity, customer and team metrics into one scorecard a board can trust, reported against original targets rather than reframed to look better.
Key ConceptsNet Promoter Score methodology and its known limits — a single number that compresses a loyalty distribution, with benchmarks that vary by sector (Reichheld, 2003); combining metrics of different kinds — financial, maturity, customer, team — without letting one flattering number stand in for the rest; reporting against the targets set at the outset, not a softened substitute.
Practical ExampleThornfield's board scorecard presents maturity, NPS and cost-to-serve against their original targets, and team engagement against its starting figure since no target was set for it, rather than reframing any of them against an easier goal, and the board accepts the mixed result because the reporting is legible against what was promised eighteen months earlier.
ExerciseUsing the worked example below as your model, build a one-page transformation scorecard for a programme in your own organisation: list each metric, its original target, its current actual, and one honest sentence on what the gap means — resisting the temptation to replace any original target with an easier one. If you don't have one to use, rebuild Thornfield's four-metric scorecard and go beyond the worked example: for each of the three metrics with a target, say whether the board should accelerate effort, extend the timeline or revise the target, and why.
Worked Example

Thornfield's Eighteen-Month Scorecard

Eighteen months into Project Northstar, Thornfield Logistics Group assembles its first full transformation scorecard for the board, built on exactly the structure this module teaches: actuals reported against original targets, not against a softened substitute.

Digital maturity, re-measured against the identical four-dimension model used for the 2.1/5 baseline in Module 1, now stands at 3.4 out of 5 — short of the 4.0 target set in the original roadmap, but ahead of the pace that roadmap assumed at this stage, helped by the Pathfinder Replacement Squad delivering its minimum viable booking module by its third sprint, even though the full replacement is not yet finished. Customer Net Promoter Score has moved from 24 to 33, against a target of 40; read alongside sector benchmarks and the qualitative feedback behind it, rather than as a bare number, Sofia Marsh's team judges the gain real; in this illustration it is concentrated among SME customers using the new tracking view, with little movement yet among larger accounts who have not adopted it. Cost-to-serve is down 5%, against the 8% target set out in the original £18m business case — a shortfall the team traces to slower-than-planned adoption among rural-route drivers, the same pattern Module 3 found in the telematics pilot, rather than to any error in the original benefit estimate. Team engagement across Project Northstar's wider team, measured at 74% twelve months in (up from 58% at the start) in Module 9, is carried into this scorecard unchanged, with no target recorded for it, since the next engagement survey falls after this reporting point.

No figure is reframed to look better than it is, and none is dropped because it falls short. The board accepts the scorecard not because every target was met, but because it can see exactly what was promised eighteen months ago, what has actually happened, and a credible account of why — ahead of schedule on maturity, behind on NPS and cost-to-serve, and visibly improved on the engagement of the team delivering it. That is what a credible transformation report looks like: not a win, and not a failure, but an honest account a board can actually use to decide what happens next.

Knowledge Check

  1. Why can a programme that finishes on time and on budget still fail the test that actually matters to the people who funded it?
    Because the iron triangle of cost, time and quality (Atkinson, 1999) measures delivery, not benefit; benefits realisation management (Project Management Institute, 2019) is the separate discipline of tracking whether the benefits promised in the business case actually land, often long after the project itself has closed.
  2. Thornfield re-measures its digital maturity at 3.4 out of 5, against a 4.0 target. Why is this not straightforwardly "behind"?
    Because the re-assessment, using the identical dimensions as the 2.1/5 baseline, finds Thornfield ahead of the pace the original roadmap assumed at this stage, even though it has not yet reached the target figure — a gap to target can reflect timing rather than underperformance, and the two explanations call for different responses.
  3. Why should a shortfall against an original financial target, such as Thornfield's 5% cost-to-serve reduction against an 8% target, not automatically be read as an error in the original business case?
    Flyvbjerg (2006) found that project forecasts are systematically distorted by optimism bias and sometimes strategic misrepresentation, but a shortfall can equally reflect slower-than-planned adoption rather than a flawed estimate; the two explanations are distinguished by examining the mechanism behind the gap, not by the size of the gap alone.
  4. Why does reporting a mixed result against original targets, rather than reframing it to look better, make a scorecard more credible to a board, not less?
    Because a board shown the original targets alongside the actuals — as Thornfield's board is, including a Net Promoter Score of 33 against a target of 40 (Reichheld, 2003) — can judge the gap for itself; reframing any target after the fact, or letting one flattering metric substitute for the others, removes the board's ability to do that judging honestly.
  5. Go back to Module 2, where Thornfield's business case split its benefits into those quantified in money, those measurable but not easily monetised, and those observable only qualitatively, and where the first driver of change was three major customers leaving over poor real-time visibility. In the worked example, the NPS gain from 24 to 33 is concentrated among SME customers using the new tracking view. Which kind of benefit is it, and why does the drivers analysis make it more than a soft number?
    It is a measurable benefit that is not easily monetised (Ward and Peppard, 2002), so it belongs on the scorecard in its own units rather than being forced into money or left out; and because the platform bet was chosen to answer the customer-visibility gap that drove customers away, a gain among the customers actually using the tracking view is direct evidence on the reason the £18m was invested, while the lack of movement among larger accounts that haven't adopted it shows where that reason is still unanswered.

Resources & Further Practice

  • Project Management Institute (2019): Benefits Realization Management: A Practice Guide. Newtown Square, PA: Project Management Institute — standardised definitions and a life-cycle framework for benefits realisation, the discipline this module's first lesson distinguishes from on-time, on-budget delivery.
  • Atkinson (1999): Atkinson, R. (1999) 'Project management: cost, time and quality, two best guesses and a phenomenon, it's time to accept other success criteria', International Journal of Project Management, 17(6), pp. 337–342 — the critique of the iron triangle used to open this module.
  • Flyvbjerg (2006): Flyvbjerg, B. (2006) 'From Nobel Prize to project management: getting risks right', Project Management Journal, 37(3), pp. 5–15 — optimism bias and strategic misrepresentation in forecasts, the two explanations a shortfall has to be tested against.
  • Kane et al. (2015): Kane, G.C., Palmer, D., Phillips, A.N., Kiron, D. and Buckley, N. (2015) 'Strategy, Not Technology, Drives Digital Transformation', MIT Sloan Management Review and Deloitte University Press — the maturity research behind the baseline this module re-measures.
  • Reichheld (2003): Reichheld, F.F. (2003) 'The one number you need to grow', Harvard Business Review, 81(12), pp. 46–54 — the original Net Promoter Score article; read alongside independent commentary on NPS's sector-benchmark variation and its loss of detail versus a fuller satisfaction survey before relying on it as a sole customer metric.
  • Ward and Peppard (2002): Ward, J. and Peppard, J. (2002) Strategic Planning for Information Systems, 3rd edn. Chichester: Wiley — the quantified, measurable and qualitative benefit categories, recalled from the digital strategy module.
  • Further practice: Run the Digital Transformation Benefits Calculator with your own programme's actual investment and benefit figures, then carry your completed scorecard into Module 11, where it becomes one input to the full transformation roadmap.
Checkpoint

Reflect on Your Transformation Scorecard

  • What I did: name the programme you measured, the benefits and owners in your tracking plan, how your maturity re-score compared with its baseline, and the metrics and targets on your scorecard.
  • What I learned: note one gap you first read as failure that turned out to be a matter of timing or an optimistic target, or the reverse, and the evidence that told you which.
  • What I'll change: write one habit you will take into reporting change at work or in the organisation you studied, for example setting out the evidence needed to move a score before anyone sees the result.