Re-examining Project Risk

Beyond Size, Structure and Technology, today’s systems require management of integration and market risks

By Doug Brockway
February 2018

In September 1981 the Harvard Business Review published an article by F. Warren McFarlan on the “Portfolio Approach to Information Systems.”  McFarlan described three determinants of project risk: 

• Size (how big an effort is this?)
• Structure (how well are the objectives defined?), and
• Technology (does the technology, and our understanding of it, match the task?)

In advance of implementation he advocated assessing project risk singly (should we build this system/app?) and as a portfolio (what does this do for us overall?).  He advocated adjusting project management approaches based on the nature of the project.

The model was developed in the days when almost all-important systems were in-house, mainframe systems and when mini-computers, local networks, and inter-company systems were just beginning their breakout.  It has no awareness of the challenges brought by PCs, hand-helds, the Internet, Social, Mobile, Analytics or the Cloud (ISMAC). As a result, two more determinants of risk are added in this paper:

• Integration (how do we orchestrate all that is part of a modern application ecosystem?)
• Market (how do we penetrate and stay in the target market?

In sum, the framework and its main points remain relevant. Here’s a synopsis with some updates.

Risk Determinants

McFarlan cites three determinants of a given project or portfolio risk:  size, structure and technology.  There are at least two more forms to add: Market participation risk and Integration risk.  In each of these both the absolute measure of a determinant and the relative measure to the company or use must be considered in evaluating risk.

Size

The size of a project can be described through its dollar expense, the staff needed to develop it, the time taken, and the number of people and departments affected.  Projects can also be sized by the amount of “stuff” that is delivered. At that time KLOCs (thousands of lines of code) were also used, but rarely today.  Function Points (there are at least 5 types) are most prominent.  Many companies count use cases.  In agile methodologies teams evaluate the size of deliverables by analogy, refining estimates, sprint-to-sprint, based on the team’s known experience, cadence, and demonstrated “velocity.”

As a rule, a project with 25 people assigned is riskier than one with 5.  A project to be used by a department of 50 people is less risky than one used by 3,000 customers.  For a company experienced in projects affecting four to six thousand customers, a 3,000 customer project may be lower risk than one that is built for a department of 250 by a firm with an average user group is 50.  Absolute size and size relative to a company’s experience materially affect risk.

Many systems are still built in-house but for even the smallest company externally facing systems are the rule.  In bringing them to life companies must select and manage vendors to partner with in building almost all significant applications and systems.  In doing so a company’s past experience in vendor selection at this size of project is a significant risk.  If you’ve never hired someone for a 3,000 external-user application, only 250 internal user systems, the risk rises.

Structure

In McFarlan’s terminology “structure” refers to the “structure” of project outputs.  In highly structured projects they are defined completely, from the moment of conceptualization.  Today, as then, with such clearly stated design and deliverables project risk is reduced. Usually these are small projects or very plain vanilla projects to replace or build basic operational systems.

In contrast, on projects where the users cannot reach a consensus on what the outputs should be, where the outputs shift “almost weekly,” the risk goes up. In practical terms this means that any system of meaningful business impact has moderate to high risk.  It also means that any larger scaled project carries extra risk.  No matter how much analysis and use-case development are applied at the front end of a multi-month project, the definition of the most optimal outcome will shift before the project finishes.  The longer the project and/or the more it is directly used to conduct business, the greater the actual need shifts away from initial design.

How systems are developed has changed somewhat.  Approaches like agile development and, when the project size is large, overarching concepts like “release trains,” Kanban and the Scaled Agile Framework (SAFe) make it easier to keep delivering on well “structure” projects while keeping the organization’s “design aim” tracking the moving target. 

Agile breaks projects down into two-week sprints (iterations), each with a defined required output, each output accepted by someone from business management.  Sprints are small projects that are highly structured and, by McFarlan’s analysis, low risk.  If the actual needed deliverable requires more time than that a “release train” is defined; a set of sprints, one following the other, to produce the needed system in 6-12 sprints.  Each sprint has known deliverables. A working system or enhancement is the result.

As the broader project team and the business learns, the definition of later sprints in the release train tend to change over time but, importantly, once a given sprint starts its structured objective does not.  The risk is managed within the sprints.  Release trains are managed by business owners and IT architects to ensure that the overall objectives and the technical results match with company strategy and need.  Using approaches like SAFe, for larger or on-going efforts, the business strategy informs which release trains to build and in a lean, Kanban method, resource availability limits how many are in process concurrently.

Technology

Traditionally companies focused on their internal experience with the technology being used.  As the project team’s familiarity with the technology decreased the risk went up.  If you didn’t know the technology, you hired someone who did. In today’s ISMAC-world there are so many technologies and trends you need to hire someone who can hire someone who knows the technology. 

For larger efforts you’ll need to hire someone to hire many someones to cover all the needed bases. And those bases, the underlying technologies, are changing faster than ever before.  A company’s absolute experience with the technologies drives risk as well as its absolute experience in vendor management in any given field.  And, as before, a company and industry’s relative experience with a technology and a vendor’s relative experience say much about the risk taken in pursuing a project or a portfolio.

Integration Risk (new)

Current systems are usually built by third parties, often in whole or in part in other countries, as often as not an amalgam of systems, services, modules and widgets, and frequently must either interact directly with customers and suppliers and a mobile workforce (especially the business-critical projects). These factors create an additional integration risk within and across a set of systems and ecosystems.

Into the 1980’s systems tended to be supportive of given business functions.  They typically stayed within the bounds of a given processes or organization.  They rarely went outside the organization.  Virtually none were conceived and delivered as the business itself, the interaction with buyers, suppliers and competitors.  In today’s world some systems, apps or applets are aimed at a known set of users in a known organization or set of organizations and markets. For these prior experience is rich. Performance expectations are understood and regularly have been met. Such systems have a different risk profile than a revenue generating product’s user experience.

Even targeted, small revenue generating systems can quickly go viral and, in the process, break the systems responsiveness and availability.  Specific skills and experience in technologies and vendors is needed to mitigate this potential. Efforts intended to be transformative can flop.  Engineering insight is needed to design a system that can be seamlessly and economically scaled down.

A system of significant impact must often deal with multiple jurisdictions on intellectual property, personal data and related security, standards of customer service, rights of access and more.  A project with few of these issues and in known jurisdictions for the issues it has is far less risky than one with many, often in new jurisdictions, and where the growth of the consuming population is faster than planned. Certain vendors are prepared for this, others merely claim it.  The development team must be able to determine the difference.

Market Risk (new)

The original five elements remain relevant but when McFarlan developed his approach to risk systems were internally focused.  They supported business efficiency and, on occasion, had impact on revenues.  Today, systems are often an automated reflection of our business strategies. They are the method with which we interact with customers and suppliers, do transactions, and win in the marketplace. In cases like eBay or Amazon the systems ARE the marketplace. As a result, the 21^st century risks include issues around intellectual property, competitors’ activities, market disruptors, and so-called “whole product” challenges (Crossing the Chasm). Many systems today define how, when and where we do business and with whom.  Modern systems often carry market or business risks.

If a project is part of market-facing capabilities or closely tied to them it bears the “strategic risk” of becoming irrelevant if the business model doesn’t work out, the revenues needed don’t appear, or unexpected competitors arise.  Management of these risks require participation, insight and direction from the company’s resources that own or are responsible for defining and attacking markets.

Most current systems are designed with “compliance risk” issues in mind.  A lender checks to ensure all the disclosures are made.  A trucker checks to ensure that drivers get sufficient rest.  A chemical company reports that all outputs, useful and waste, are accounted for.  This changes the makeup of teams (they require people versed in relevant regulations and laws, current and likely), the testing that must be done, and adds to future maintenance and enhancement obligations.

Because modern systems are either the business itself or directly enable it they bring scrutiny regarding “operational risk.”  Contingency plans for what to do if the system doesn’t scale or is sporadic in its performance or availability are more urgently needed and more stringently examined.  These factors increase the need for efficiency in design.  As companies pursue total quality initiatives quality must be designed in which leads to changes in development methods and team composition.

Lastly, by being part of the product modern systems bear “reputational risk” and reputational hurdles.  If a competitor adds a useful feature to its e-commerce experience others must follow suit.  You can’t enter a market if you can’t reasonably mimic the feature/function provided by dominant player. Amazon in retailing is the most obvious example. In judging a market facing project or portfolio’s risk you must judge your ability to play at all and for the long haul.

Assessing and Managing Risk

Many companies conduct reviews of projects wherein the IT and business managers independently evaluate the project and then discuss how to improve next time. They also develop questionnaires to be used as part of deciding to fund a project.  In all cases keeping and analyzing the history and trends of risk scores and results is key to developing customized knowledge of how your company/group deals with project risk and what to do about it.

Systems management suites often have risk assessment components.  An ITIL v3 Change Management process must evaluate an array of risks before it accepts any change and supporting software has support for this. These risk assessment modules should be customizable to the kinds of risk you intend to measure and manage.  If not, you’ll need to decide whether to develop a process outside the tool or not.

Many project management techniques and methods assume waterfall development:  business case, business design, technical design, coding, testing, implementation.  This was, essentially, the only way systems were built when McFarlan did his study which lists sixteen methods of either planning projects or integrating the efforts and expectations of delivery and user teams.  Items like milestone phases, systems specification standards and post audit procedures are wide-spread now as are user project managers, steering committees, progress reports and user ownership.

Often better than waterfall are DevOps and agile approaches. A delivery team that practices DevOps fully is producing completed, debugged, implementable code as they go.  This has an enormous, beneficial impact on project and cumulative portfolio risk. Any organization that is properly using agile approaches has “product owner” reviews of deliverables at the end of each sprint and a team retrospective.  Together it is known if the output met specifications, if the process used worked properly, and what will be done to improve both.  As sprint follows sprint follows sprint these retrospectives have a Kaizen-like effect, continuously reducing risk.  Similar retrospectives occur during and more exhaustively at the end of release trains.  Many companies are having broader success applying Scaled Agile-like frameworks across release trains, programs and strategic initiatives.

Warren McFarlan’s model still resonates.  If you could only ensure that you measured and managed project size, structure and technology you will have made important strides in project and portfolio risk.  But today’s systems have a wider array of risks to manage.  Understanding the one’s you face and developing an analytic sense across projects of risk trends and mitigation success will lead to better systems in place and the confidence to do more impactful things in the future.