Innovation is a process. Like it or not, you won't get results overnight. It can take months or years to bring an idea to practice, depending on its complexity. In this blog post, I will explain why the innovation management process cannot be seen separately from its context, the roles to play, the methods available, the different types of innovation projects, the process itself, and the outcomes to expect.
Creativity
Let us first address the myth that creativity is at the core of the innovation process. It is not, uncertainty is. While we often associate innovation with creativity, uncertainty is what sets the process apart. When developing and implementing new solutions, you are exploring new territories. That requires embracing the unknown, dealing with uncertainty, lots of guts, and stamina.
Creativity is what sparks the process. During the process, it also comes in handy when thinking of new solutions and solving (unexpected) problems. So creativity is not unimportant. However, the innovation process is more a matter of diligent execution, than unbridled creativity.
Thus, more than creativity, it is the ability to deal with uncertainty that defines successful innovators. So be forewarned, planning for uncertainty will consume more of your time and effort than thinking creatively.
Context
The innovation process takes place within an organization. That is what sets it apart, from for instance creating a startup. Both processes involve bringing an innovative idea to practice, but the context is very different. A startup is building a new offering from scratch. An established organization has existing customers, capabilities, vendors, reputation, etc. that can be built on, or that can make change more difficult.
How the innovation process is organized, depends on how your organization works. Moore identified two different innovation processes, one for organizations that produce complex systems, like system integrators such as Northrop Grumman, the other for organizations that have volume operations, like consumer good manufacturers such as Proctor & Gamble.
In his article, 'Strategy and Your Stronger Hand', Moore shows that while the steps of the process are similar, the location and execution are very different. He claims that these two business models cannot be combined for that reason.
These different business models also dictate whether a centralized or decentralized innovation process works better (source).
The innovation process for volume operations
The more common innovation process is that of volume operations, which concerns industries like electronics, car manufacturing, food processing etc.. It is the best understood and most written about innovation process for which most methods and software tools are designed. In these companies, innovation is a continuous endeavor that has its own dedicated processes, people, etc. Simply, because in the consumer good market creating new products to improve and replace the current ones is key to survival.
Because the large volumes these companies deal with, the focus on efficiency is very important in these bulk operations. As such, there is no room for playing around and testing that is an integral part of development activities. Therefore, in these organizations, the innovation process has its own function and is housed separately from operations.
The innovation process for complex systems
The more common operation model is that of the complex systems. Nearly all service organizations and most product companies delivering tailored solutions fall under the complex systems paradigm.
Because the operational processes in these organizations are more flexible than those of volume operations, there is more room to develop and test new things as part of ongoing operations. However, these ongoing operations are often set up to react to client demands, not to proactively address clients' concerns and conquer new markets. The change from reacting to taking proactive action is what makes the innovation process clash with operations in this context.
Compared to the volume operations model, the innovation process for complex systems resembles more a batch process. When the opportunity or the need is there, these organizations innovate. It is far from a continuous process and typically these organizations don't have -nor can afford - a dedicated innovation unit with its own staff and processes.
As we have found in our research, integrating the innovation process within operations is a must in most of these organizations. First, because the state-of-the-art knowledge resides with the providers on the work floor.
Second, the scalability is limited. If you only serve a few clients, you cannot afford to staff an innovation unit and recoup the costs of these "unbillable' employees, unless of course, you operate in a capital intensive industry.
As a result, housing the innovation process in a dedicated innovation unit does not result in more profits in organizations with a complex system business model.
Domain vs innovation experts driving the innovation process
The context also impacts who is engaged in the innovation process. If the innovation process is housed in a dedicated innovation unit, your innovation experts will know all about the innovation process but typically lack domain knowledge of all the products, technologies, and disciplines represented in the organization.
If the innovation process is housed within ongoing operations, the process will be driven by people with lots of domain knowledge, but limited innovation expertise.
In the organizations we work with - law firms, engineering firms, health care providers, consultancies, etc. - domain knowledge is much more difficult to come by than innovation expertise. We can relatively easily teach a team how to innovate, however, it takes years to understand a particular area of the law, a health care specialty, or an engineering domain.
Not surprisingly, in these latter cases, the business model resembles that of complex system providers, which makes it more effective to have these service providers drive the innovation process.
Projects
All the projects in the innovation process together form the firm's innovation portfolio. The sum of your projects should enable you to execute your innovation strategy. And to spread risks, you want to have a balanced portfolio of innovation projects that consists of small and large investments, and short and long-term returns.
Small, short term projects are called incremental innovations. These can be process improvements, additional features, or simple additions to current offerings.
More fundamental changes require typically more time and budget to develop. These are called radical innovations. Research endeavors, or advanced development, often fall outside the scope of the innovation process. As research has the aim to find something new, which may or may not lead to a commercial solution.
Disruptive innovation is a special kind of innovation project, as it aims to replace the organization's current offerings and thereby destroying the current capabilities and value of the firm. It is well documented, that such disruptive innovation projects need protection high in the organization to stand a chance of survival and success.
There are different methods for different kinds of projects, see below. One practical problem I have encountered is that innovators typically don't set out to engage in incremental or radical innovation. They have an idea or problem to solve. If you ask them, they all will say that their idea is really novel, as a badge of honor. However, how radical or disruptive the solution actually is, often only becomes clear during the innovation process, which makes it a challenge to select the right method and approach at the start of each innovation project.
Portfolio management
Portfolio management is a process by itself, to ensure that the sum of projects is more than the parts. Most portfolio analyses show graphs similar as the one below, where the position visualizes the increasing uncertainty and duration of the innovation projects, size indicates the resources required for the project, and colors are used to indicate different involved business units. Such graphs provide a very insightful snapshot of an organization's portfolio of projects.
Another way of classifying incremental and radical innovation projects is by identifying projects as horizon 1, 2 and 3 projects.
The value of dividing projects in different horizons is to spread the risk and ensure there is some return from innovation activities, even in the short term.
Process
Compared to all the other topics, the process itself is remarkably consistent for all types of organizations and innovation projects.
Probably, because there are certain tasks that each project team has to accomplish. Generally speaking, these tasks are:
- Describe the project
- Validating the concept
- Obtaining funding for the development
- Develop the solution
- Implement the solution (with a first user)
- Obtain paying clients
- Scale the solution
Validation
Especially for the first set of tasks, there are many different terms: ideation, fuzzy-front end, customer-discovery, incubation, etc. All of these have the same basic premise, at the start of the innovation process you have an idea that is basically worthless until it is validated. It takes work to figure out if an idea is desirable, valuable, and feasible, no matter what method you use.
It pays off to invest time and effort in these early phases, because the better you validate a project in the early stages, the higher the likelihood of success later on.
Scaling
In organizations with volume operations, scaling is a given.
However, in organizations that produce complex systems, scaling requires deliberate effort, because there are limited existing drivers to leverage or scale new offerings. Nevertheless, it is of utmost importance to a successful innovation process to scale successful new offerings, as it is the only way to make a return on investment on the entire portfolio of innovation activities. No organization can afford to invest in the development of a new offering and then not commercialize it to its full potential.
Duration
The duration of innovation projects can vary greatly from just a few months to years for more complex innovation projects.
Over time, the process certainly has become faster. The table below is based on older research data, but it still is indicative:
| 1995 Categories | Average 1995 (weeks) | Average 2004 (weeks) | 2004 Categories |
| New-to-the world | 181 | 104 | New-to-the world |
| New product lines | 126 | ||
| Major revisions | 78 | 62 | More innovative |
| Incremental | 33 | 29 | Incremental |
Newer methods, see below, seem not have made much difference in overall development timelines. Instead, they mostly have improved the accuracy with which we vet projects. Time to market is obviously important, however, it is irrelevant if you are not working on the right solution.
Methods
There are many methods to guide innovation projects through the innovation process. Below are a few of the most common approaches.
The stage-gate model is the bigger framework, that most organizations use to provide structure to their innovation process.
Discovery-driven-growth and lean-startup allow for better processes and decision making at each stage of the innovation process because both methods embrace uncertainty as a given.
Lean or six-sigma and similar approaches have proven their value in the realm of incremental innovations. These are approaches that allow for data-driven improvement actions, while quickly cycling through plan, do, check, act cycles.
Design thinking has become the cornerstone of product development, where observation of current users has proven invaluable to make better decisions in the design process, which ultimately leads to better solutions.
Agile has been adopted from software development, where iterating and trying things is so cheap, that it presents its own challenges.
Stage-gate
The stage-gate methodology is probably the oldest and most used concept to manage the innovation process. The idea is simple. The innovation process is divided in separate stages and acts as a funnel. Because of all the uncertainties, innovation projects are approved to proceed from one stage to the next if they meet the stage criteria. Many projects are started, while few projects - only the most promising ones - make it through all the gates.
The stage-gate process also facilitates portfolio management, as it will tell you where your projects are in the process.
Discovery driven growth
Rita Gunter- McGrath and Ian MacMillan were the first to come up with a better way of dealing with uncertainties in the innovation process. Instead of planning and executing, they suggested that a better approach would be expressing and testing uncertainties. They call this discovery-driven planning:
Unlike traditional project management, in which much is known, discovery-driven planning forces managers to articulate what they don’t know. It sets the stage for disciplined learning during the innovation process.
Lean startup
Central to the lean startup methodology is the assumption that startup companies invest their time into iteratively building products or services to meet the needs of early customers. The lean startup method has been applied to companies, so they can reduce market risks and sidestep the need for large amounts of initial project funding and expensive product launches and failures
Similar to discovery-driven growth, the process is a combination of business-hypothesis-driven experimentation, iterative product releases, and validated learning.
While lean startup was built on discovery-driven planning, it is the better-known methodology.
Lean six sigma
Other than the word 'lean', Lean Six Sigma has nothing to do with the lean startup method. In general, when referring to lean, those in the entrepreneurship world think of Lean startup, while those in corporations think about lean manufacturing and lean six sigma, the method to improve existing products and processes.
Lean Six Sigma is a combination of approaches to reduces process defects and waste, through a reduction in variation. PDCA stands for the Plan - Do - Check - Act cycle. So similar to discovery-driven-growth it uses an iterative approach to get to the best solution.
Because of its focus on the current process and the reduction of variation, Lean six sigma is often seen as a hindrance to more radical innovative solutions.
Design thinking
Design thinking is a process for creative problem solving that puts the human experience at its core.
It consists of three main pillars:
- Empathy — Understanding the needs of those you’re designing for.
- Ideation — Generating a lot of ideas. Brainstorming is one technique, but there are many others.
- Experimentation — Testing those ideas with prototyping.
While it can also be used in the earlier stages, to validate the problem, its approach is especially valuable in the development phase, when the solution is being created.
Agile
The software development community has developed its own approach to product development, and this method has made its way into the innovation process as well.
Similar to Lean six sigma, Agile is also an iterative approach that is performed in a collaborative environment by self-organizing teams. In this case, it was developed with the goal to create high-quality software in a cost-effective and timely manner, while meeting stakeholders' changing needs.
Which method to use when?
In general, I would say, use stage gating as the larger framework. Use discovery-driven planning throughout, while complementing this with techniques from the lean startup approach in the early phases, and design-thinking during the development phase. Use Agile for the development of software solutions and lean six-sigma to improve on what is already out there.
In any case, having a structured process is better than having none. Without a structured approach to innovation, you will be reinventing the wheel with each project, which is costly and frustrating for innovation teams. Worse, your innovation process will never get better because you have nothing to compare.
Even if you only innovate once every so often, it pays off to formalize the process. If you don't have the expertise in-house, you can always rely on solutions like our Programs to provide your teams with a simple yet rigorous innovation process.
Outcomes
Innovation is an investment in the future. Investments should pay off, but in many organizations, they don't. For the simple reason, that management is too focused on individual projects and short-term results. A sustainable return will come from a portfolio of innovation efforts, will take time to establish, and will require a very disciplined innovation process. Especially in complex systems organizations, where there are fewer opportunities for big gains to make up for the many small losses that occur as part of the essential experimentation.
High failure rates
Nevertheless, the high failure rates in the innovation process are shocking.
95% of all product innovations fail, according to Clayton Christensen - a Harvard Business School professor and expert in disruptive innovation.
While the accuracy of the previous numbers can be debated, according to a McKinsey Global Innovation Survey, only 6% of executives are satisfied with their innovation performance.
What’s more, according to the 2017 PwC Innovation Benchmark, 54% of innovating organizations have trouble bridging the gap between innovation strategy and the larger business strategy.
So clearly there is room to improve on the innovation process.
Good money after bad?
In general, innovation projects need to be fostered as part of the innovation process, until the new offering has similar profitability margins as the existing offers. Otherwise, innovative offerings won't survive and maybe killed prematurely.
Decision-makers are often overconfident and impatient, expecting returns within the same fiscal year. That is not realistic, as anything beyond simple me-to types of new offerings typically takes at least a year to develop. Patience is a virtue in the innovation process.
That does not mean that you need to keep investing in ongoing innovation projects. Quite on the contrary. Only keep supporting teams that are doing the work and learning from what they are doing. As long as a team is diligent in its execution and completes its milestones, the funding should continue. They may hit a dead end and have to restart, pivot, or redo a milestone, that is all part of the process.
From what I have seen, innovation projects are sustained too long and fail because of overconfidence, reliance on past experience, gut-feel decisions, over-commitment, and favoritism. Mistakes that can largely be avoided when applying a disciplined innovation process, using the methods suggested above.
Return on investment
To create a return on investment, decision-making in the innovation process should be based on outputs and outcomes, and not based on speculations about the final results. As the latter is not guaranteed until the very end, thereby providing no handles to manage the innovation process.
I cannot say it enough, you need a diligent process to be efficient and effective in your innovation efforts - so you are not wasting valuable time and resources during the process.
Last but not least, you need to maximize the profit from winners. While this sounds obvious, many innovators love the process of innovating so much, that they see less value in creating a repeat offering for other clients. From an organizational perspective, the return on investment comes from these repeat offerings and therefore should be an integral part of the innovation process.
Success!
