Originally published in the Cutter IT Journal.
The Hurdle Of IT ROI
Lies, Damn Lies & Business Cases
Business cases for IT projects experience a high level of mistrust and suspicion. Executives and senior managers tend to regard their claims with a jaundiced eye and a healthy doubt as to whether the benefits that are being claimed can be realized – and even whether or not the identified costs are in line. Much of this mistrust comes from previous experience of IT projects – with overruns, delays and a frequent failure to fully realize their promised goals. The questions that must be asked are:
- Are the suspicions regarding business cases reasonable?
- Do the underlying problems lie with the business case, the projects they represent or how the organization approaches those projects?
- What can be done to increase the confidence in IT business cases going forward?
The reality is that the suspicions and mistrust of business cases seems to be not only well placed but also well founded. What also appears important, however, is that the problems are not unique to IT business cases – they just seem to be amplified in an IT context, just as cost and schedule overruns are. Part of the challenge lies with the greater uncertainty associated with IT projects – that we have to get further into the project before we can have better certainty of the costs – but there are larger issues that seem to exacerbate how projects and business cases are viewed.
Hurdles, Then A Marathon
The overall approach taken to projects creates some of the greatest hurdles associated with effective business case development. There has long been a tendency to place a larger up-front emphasis on approving a project, with much less focus on subsequent reviews during the project or assessments post-completion. The tendency towards a greater emphasis on due diligence up front has only increased in recent years – as corporate decisions come under increasing scrutiny internally and externally, there is a strong bias towards ‘getting it right’ up front.
This intense scrutiny at the start of the project tends to be counterbalanced by a comparatively lax process of on-going review as the project progresses. Once a project received the green light (although that process may take months) it is exceptionally rare that an organization will ever raise the red light (or even a cautionary yellow) unless there are serious problems being encountered with the project execution. In analyzing the reasons why projects do get stopped, however, the vast majority are due to management or technical failure to deliver the product of the project. Far fewer are stopped because of questions regarding business value or the attainability of outcomes.
Justification Or Decision Making Tool?
How business cases are typically used, and the manner in which they are developed, is also a source of suspicion and concern. At its core essence, a business case should be a tool for decision making expressed in financial terms. More often, however, the role of the business case is a sales document or means of justification to rationalize a decision that has already been made. Rather than asking whether a project should be done, or evaluating the different paths by which an outcome might be realized, it becomes a record of the assumptions that are being made to support the project choice.
A Thousand Points Of Light – But One Estimate
Finally, and most particularly challenging for IT projects, there is a strong emphasis on getting the estimate right up front – of both costs and benefits. By the time a project decision is made, there is a strong desire on the part of senior management to have costs accurately reflected, and an equally strong desire for IT project staff to be able to get them accurate. This desire for accuracy, however, is not supported by the reality of most IT projects. By their nature, they have a higher level of uncertainty, and a greater investment required in defining the solution before a level of confidence can be articulated.
Figure 1 – Use of success measures in project organizations
The Implications Of Structure
While the points discussed above underscore the difficulties that are particular to developing business cases for IT projects, the reality is that there are some more fundamental underlying issues that relate not so much to the types of projects as they do to the mindset and structure of our organizations. The most significant of these issues, as illustrated in Figure 1, is that formal measurement of project success simply isn’t done in most organizations. Based upon research conducted by Interthink Consulting during the period 1997-2003, involving a survey of project management practices of more than 550 organizations and over 2,000 individual practitioners, just slightly more than a quarter of organizations today formally define and measure project – and by extension, investment – success.
Not formally managing project results or benefits attainment is reflective of a larger reality that the majority of organizations today do not have strong measurement cultures. Many, if not most, organizations tend to operate on a relatively intuitive basis. While the use of metrics and statistical practices is slightly more common in the realm of operations management, projects – despite their focus on budgets, schedule and resource effort – are often measured based upon gut feel.
Creating A Culture Of Benefits Realization
Facing The Hurdles
The primary driver in shifting the mindsets discussed above is developing a better means of benefits realization. Some discussion of just what this means, however, would probably be helpful at this point. For any investment that we make, there is clearly some return that we are hoping to gain from it. Understanding what this return has been, however, is a different exercise than simply knowing or believing it is there. While we may intuitively view the results of a project as a change for the better, there is a difference between thinking that you have obtained a benefit and knowing you have. This distinction is very much like that of Schrodinger’s cat – you may have attained the benefits of a project, but unless you go and look you cannot be certain that you have truly realized them.
How we approach planning, the formalized measures we do and don’t use, the way in which we view projects – and indeed the very accuracy of our business cases – are all influenced by how accountable we feel for demonstrating the results and outcomes at project completion.
First Step, A Change In Approach
Our ability to create a framework and culture of benefits realization first depends upon having in place a formalized process. Establishing a benefits realization culture, however, requires the introduction of a level of formality of management and decision making that will be very different for many organizations. It requires looking at the business case not as a sales tool or even as a means of making an isolated decision about a project – it becomes a commitment on the part of the organization to how it wants to operate, what it wants to look like and the cost expectations associated with those future operations. In essence, it becomes an input into a future budget cycle that defines what the future cost and revenue structure associated with that capability will be.
The implications on future budget cycles are probably the most significant impacts that a formal benefits realization process will have. If we commit that a project will deliver a level of revenue, or reduce specific costs, or realize a defined level of productivity, then those benefits should represent an implicit promise to adjust future budgets by the same amount once the project is complete. What this means is that – external factors assumed to be equal – your future cost structure and revenues should be a product of the projects you do today. The business cases associated with those projects should directly translate into the budgetary expectations of future years.
To fully understand and quantify these implications, organizations must adopt a full lifecycle focus in defining their business cases. The scope of costs cannot simply be those associated with the project, but must also address the on-going operating costs associated with both implementing the new capability and changing or retiring any existing capabilities. It must articulate what the expected cost structure is going forward – including any costs associated with managing problems that the project isn’t planning on eliminating (or in fact will create, even if only in the short term!)
The largest result that will occur – certainly in the initial period after an organization announces a stated intention to review its business case results – is a very real change in the benefits from those being articulated in business cases today. Most project business cases will tend to slide towards much more conservative numbers, on the philosophy of ‘I will only commit to what I know for certain I can attain’. Some of this change is indicative of what are probably overstated benefits in current business cases – the benefits cited are often those that are needed to justify the project going forward, rather than the ones that can be realistically attained. The negative impact of this shift, however, will as often be a more conservative approach to benefits identification in an effort to minimize accountability. This is where organizations must realize a fine balance in the early stages – pushing for benefits that are still realistically attainable and justifiable in terms of changing the business, without pushing so far that the promised benefits become unrealistic and unattainable.
The other shift – both as a cause and effect – will be the change in level of scrutiny associated with projects – both during as well as after projects are completed. According to the same research cited earlier, there is currently a very low involvement on the part of senior management through the course of the project. As accountability – and therefore scrutiny – increases on the results of the project and ensuring that benefits are being realized, the immediate result will be much more involvement on the part of sponsoring executives in ensuring the benefits can be realized. Again, there are positive and negative implications. On the positive side, senior management involvement has been demonstrated to deliver better project results; to counteract the potential negative impact of excessive scrutiny and interference, however, the role that senior management plays in the project – and their associated responsibilities and accountabilities – must be clearly defined.
Finally, organizations should expect to see an increase in the number of projects that are cancelled prior to completion, for the very sound reason that they cannot deliver on their benefits. As discussed earlier, in the majority of organizations the scrutiny surrounding projects occurs at the outset. Once a project is approved, it tends to stay approved, regardless of any material changes that occur within the project or the business it is being done for. Where there is a focus on ensuring that benefits can be realized post-implementation, the scrutiny on whether value can be attained will increase exponentially. This is not a bad thing, but in fact a very good thing – today, most organizations have more projects underway than they have the people or money to complete them. Where decisions to cancel a project are not made, a Darwinian process of ‘survival of the fittest’ begins to emerge. Cancelling a project because it no longer makes sense or its benefits can be fully realized is the healthiest decision an organization can make.
Towards A Measurement Culture
The most important change that occurs in moving to an environment where benefits realization is an accepted and integral part of the management structure is one of setting expectations. There needs to be clear understanding regarding how investment decisions are made going forward, the criteria for those investment decisions and the expectations of accuracy of estimates at each stage in a project.
A significant aspect of these considerations is deliberately building into the process an expectation of refinement of the costs and benefits of a project over time. There is a very strong bias – particularly within IT projects – to develop as accurate an estimate as possible as early as possible. While the goal is admirable, the ability to realize this goal is not. There are too many variables in any project to define – at the outset – what the budget will be with a sufficient degree of accuracy. Where business cases are being evaluated to decide not just whether to proceed with a project but also to commit to an expectation of realizable benefits at completion, the pressure to provide accurate estimates will grow exponentially. Where there is not an adoption of progressive refinement of the estimates, the result will be far more work being done in advance of the initial business case to develop a more accurate estimate that the project sponsor and team can commit to. While this may seem reasonable on the face of it, the reality is that in doing so we are moving work that normally occurs later in the lifecycle to the front-end of the project. Not only is this more work – and more cost – before the organization decides a project has sufficient merit to proceed, but it narrows an important decision-making window within the project. Rather than progressively evolving the strategy as requirements become better understood, hard commitments are made early that constrain subsequent thinking and work.
Setting the expectation of progressive refinement will not necessarily result in immediate compliance, however, Organizations should anticipate resistance to the concept of progressive refinement of estimates when it is initially introduced – project managers that have been burned in the past will tend to shy away from committing to any number until they have confidence in it. While at a pre-business case stage an estimate with a 300% confidence range (the project could range from $250K to $1MM) allows a customer to decide whether the investment is even appropriate for the expected return, most consulting companies – and many internal IT organizations – will strongly resist articulating a number with that range of confidence. Estimating the benefits is even worse – if only because historically most organizations have placed so little emphasis on evaluating their accuracy over time. The source of this resistance will typically be the project teams and mid-level managers responsible for the execution of the projects, not senior management. Overcoming this challenge requires setting and reinforcing expectations about the accuracy of benefits, including visibly allowing for changes to project budgets within the initially defined ranges as the project progresses and the estimates become more refined.
The most significant means of ensuring movement to a culture of measurement that supports benefits realization is to define or re-define the process of project initiation and post-project implementation and evaluation, and to consistently define expectations in the context of that process. There should be clear guidelines for the articulation of benefits, planning for benefits realization, and the roles and responsibilities for attainment of those benefits. These will often be unique to the organization, its structure and its strategy. Clearly communicating in the context of this process on a consistent basis – for every project – is the single best means of establishing trust in the process, ensuring people are comfortable with their responsibilities and accountabilities, and as a result complying with it.
Finally, The Mechanics
While the mindset of benefits realization is critical, a positive commitment to managing and ensuring attainment of the project results can be just as easily derailed without the mechanics in place to manage the process. What most organizations have seen even where they have successfully cleared the hurdle of acceptance of a change is an inability to support realization of that change because they simply don’t have the data necessary to effectively support the process.
The reporting capabilities necessary to truly understand organization performance are core to the benefits realization process. Where a measurement culture already exists – whether through a quality management, balanced scorecard or Six Sigma approach – there is a greater likelihood that at least the capability will exist for measurement and reporting. What remains to be seen is whether the underlying metrics that are already being used support the dimensions required to manage the benefits realization of any given project. Where existing measures can be leveraged they should be, but the organization must also be willing to define new measures and approaches where the existing capabilities are inappropriate.
Some of the barriers to the measurement of benefits realization exist not because there are no measures, but in fact because there are too many measures. Where the same metric exists within many processes or systems, with different inputs and means of categorization, reconciliation of the resulting data to provide a meaningful picture of what is happening can be a far greater effort than the actual project we are trying to support. Measures will vary based upon their definitions, when in time the information is being tracked, the level of detail it is being captured at and how it is being categorized. Project managers that have tried to secure an accurate picture of their project costs from internal accounting systems will understand this challenge intimately – payable systems can be two or three weeks (or sometimes months!) behind in reflecting actual costs incurred, just because of the timing around receipt, processing, approval and payment of invoices.
As well, there is every likelihood that organizations will need to redefine the actual measures being used in order to allow them to align more with the expected behaviours that any project is designed to establish. The process of revisiting and refining measures may well occur – and at the very least needs to be evaluated – for every project.
Reinforcing these philosophies requires a change to how business cases are typically documented. Each business case needs to be able to articulate not just the benefits that will be realized, but how those benefits will be assessed and evaluated. We need to include a plan for measurement and management – including where we need to realign internal measures or systems in order to be able to ensure that the behaviours we are seeking can be seen, evaluated and realized. The business case should clearly define any assumptions made regarding how benefits would be allocated, particularly in situations (increased revenue being the most common) where a change in performance of what is being measured cannot always be fully attributed to the project being discussed. The resulting measurement approach must be accepted as part of the business case approval – as well as the actual costs and benefits themselves. Without doing so, projects run the risk of arguing about their degree of success after the fact simply because how to measure it cannot be agreed to.
Critical Success Factors
While the dimensions of implementing an effective approach to benefits realization can be far reaching, there are a few core critical success factors that – if kept in mind – make attainment of these outcomes much easier to realize:
- Allow time to fully realize the change. Any change will have resistance. Moving to an environment that creates such a clear level of accountability for significant project investments will have more resistance than most. Allow time to get better at the process of measuring before creating consequences for not realizing results.
- Set expectations for the approach, and consequences for not utilizing it. While consequences for not realizing investment results may be less appropriate for early projects, what is inappropriate is not attempting to measure and evaluate them. Not only do clear expectations of adopting a process of benefits realization need to be set, there have to be clear consequences for not utilizing it. Allowing some projects to by-pass the requirement is the surest means of ensuring its failure.
- Expect changes in what is being measured – based on what can be measured, and the implications of the measures. The measures necessary to demonstrate realization of a business case will often change by project, and specific projects may need to develop new and different ways of measuring and demonstrating outcomes that have not been seen before. While there needs to be confidence that the measures being proposed will meaningfully reflect the intended results of the project, there needs to be openness to new measures being proposed.
- Be realistic about timeframes associated with project completion and benefits realization. The dominant project pressure in most organizations is one of schedule. In this environment, there is a strong pressure on accelerated delivery, even at the cost of compromising the project. A focus on benefits realization opens a window on understanding the trade-offs associated with accelerating project delivery – in terms of both costs and benefits delivery. It is essential that these trade-offs be consciously assessed and evaluated.
- Recognize that an approach to benefits realization requires a shift in mindset more than any other change. There needs to be a process of collaboration throughout the organization, ensuring that the roles and responsibilities that the projects, sponsors, executives and business units have are clearly defined. This is particularly important as the perspective moves from one of ‘project’ to ‘investment’ – with a corresponding level of co-operation between what happens inside the project and after the project to ensure outcomes are attained.
- Place emphasis on why projects are being done. Possibly the greatest contribution that a culture of benefits realization makes is one of genuinely understanding why an investment is being made. This clarity creates a far greater – and sometimes different – understanding of why projects are being done, and the critical success factors those projects must deliver if the benefit can be realized.
While the concept of benefits realization is a simple one, its actual attainment is far more challenging. That the challenge is a difficult one is not a sufficient reason not to address and implement an effective measurement approach to manage investment outcomes. For all the complexity associated with doing so, it is a far greater cost for organizations to continue to make investments whose return cannot be measured than to implement a process of measuring them.