Innovation is a process. Like it or not, results rarely appear overnight. It can take months or even years to bring an idea into practice, depending on its complexity.
Today this process is becoming even more important. Organizations are experimenting with new technologies like Generative AI, automation, and digital services. Many ideas are launched, but only a few ultimately succeed. Without a disciplined innovation process, it becomes very easy to invest time and money in initiatives that never deliver meaningful results.
In this blog post, I explain why the innovation management process cannot be separated from its context. We will look at the roles involved, the methods available, the different types of innovation projects, the process itself, and the outcomes organizations can realistically expect.
Creativity
Let us first address a persistent myth: that creativity is at the core of the innovation process. In reality, uncertainty is what defines innovation.
We often associate innovation with creativity and breakthrough ideas. But the real challenge is not coming up with ideas — it is figuring out which ideas are worth pursuing when the outcome is still highly uncertain.
When developing and implementing new solutions, organizations are exploring unknown territory. That requires dealing with uncertainty, testing assumptions, and having the stamina to continue learning when things do not work as expected.
Creativity is what sparks the process. During the process, it also comes in handy when thinking of new solutions and solving (unexpected) problems. So creativity is not unimportant. However, the innovation process is more a matter of diligent execution, than unbridled creativity.
Thus, more than creativity, it is the ability to deal with uncertainty that defines successful innovators. So be forewarned, planning for uncertainty will consume more of your time and effort than thinking creatively.
Why innovation processes fail
Many organizations invest in innovation but struggle to produce meaningful results. This often happens for three reasons.
First, innovation efforts are disconnected from the organization's strategic priorities. Teams may generate many ideas, but few translate into initiatives that truly matter for the business.
Second, organizations underestimate the level of uncertainty involved. Traditional project management assumes that the end result is known in advance, while innovation projects require testing assumptions and learning along the way.
Third, innovation efforts often lack clear ownership. When responsibilities are unclear, projects move slowly or stall completely.
A structured innovation management process helps organizations overcome these challenges by clarifying roles, defining decision points, and ensuring that learning happens quickly.
What actually defines an innovation process?
When people talk about an innovation process, they often think of a creative spark followed by a sequence of stages or gates.
In practice, successful innovation depends on much more than creativity or a set of steps. A functioning innovation process requires several elements working together:
- Context — the organizational environment in which innovation takes place
- Roles — innovators, facilitators, and decision-makers
- Projects — the initiatives that together form the innovation portfolio
- Methods — tools that help teams deal with uncertainty
- Process — the stages through which ideas become real offerings
- Outcomes — the long-term returns innovation is expected to generate
Only when these elements work together can innovation efforts consistently produce results.
The sections below explore each of these components in more detail.
Context
The innovation process takes place within an organization. That is what sets it apart, from for instance creating a startup. Both processes involve bringing an innovative idea to practice, but the context is very different. A startup is building a new offering from scratch. An established organization has existing customers, capabilities, vendors, reputation, etc. that can be built on, or that can make change more difficult.
How the innovation process is organized, depends on how your organization works. Moore identified two different innovation processes, one for organizations that produce complex systems, like system integrators such as Northrop Grumman, the other for organizations that have volume operations, like consumer good manufacturers such as Proctor & Gamble.
In his article, 'Strategy and Your Stronger Hand', Moore shows that while the steps of the process are similar, the location and execution are very different. He claims that these two business models cannot be combined for that reason.
These different business models also dictate whether a centralized or decentralized innovation process works better.
Modern technologies are making this distinction even more relevant. In many professional service organizations — law firms, consultancies, engineering firms, and healthcare providers — innovation increasingly involves digital tools, automation, and AI capabilities. These organizations rarely have the scale to build large innovation departments, which means innovation must be integrated into ongoing operations.
The innovation process for volume operations
The more common innovation process is that of volume operations, which concerns industries like electronics, car manufacturing, food processing etc.. It is the best understood and most written about innovation process for which most methods and software tools are designed. In these companies, innovation is a continuous endeavor that has its own dedicated processes, people, etc. Simply, because in the consumer good market creating new products to improve and replace the current ones is key to survival.
Because the large volumes these companies deal with, the focus on efficiency is very important in these bulk operations. As such, there is no room for playing around and testing that is an integral part of development activities. Therefore, in these organizations, the innovation process has its own function and is housed separately from operations.
The innovation process for complex systems
The more common operation model is that of the complex systems. Nearly all service organizations and most product companies delivering tailored solutions fall under the complex systems paradigm.
Because the operational processes in these organizations are more flexible than those of volume operations, there is more room to develop and test new things as part of ongoing operations. However, these ongoing operations are often set up to react to client demands, not to proactively address clients' concerns and conquer new markets. The change from reacting to taking proactive action is what makes the innovation process clash with operations in this context.
Compared to the volume operations model, the innovation process for complex systems resembles more a batch process. When the opportunity or the need is there, these organizations innovate. It is far from a continuous process and typically these organizations don't have -nor can afford - a dedicated innovation unit with its own staff and processes.
As we have found in our research, integrating the innovation process within operations is a must in most of these organizations. First, because the state-of-the-art knowledge resides with the providers on the work floor.
Second, the scalability is limited. If you only serve a few clients, you cannot afford to staff an innovation unit and recoup the costs of these "unbillable' employees, unless of course, you operate in a capital intensive industry.
As a result, housing the innovation process in a dedicated innovation unit does not result in more profits in organizations with a complex system business model.
Domain vs innovation experts driving the innovation process
The context also impacts who is engaged in the innovation process. If the innovation process is housed in a dedicated innovation unit, your innovation experts will know all about the innovation process but typically lack domain knowledge of all the products, technologies, and disciplines represented in the organization.
If the innovation process is housed within ongoing operations, the process will be driven by people with lots of domain knowledge, but limited innovation expertise.
In the organizations we work with - law firms, engineering firms, health care providers, consultancies, etc. - domain knowledge is much more difficult to come by than innovation expertise. We can relatively easily teach a team how to innovate, however, it takes years to understand a particular area of the law, a health care specialty, or an engineering domain.
Not surprisingly, in these latter cases, the business model resembles that of complex system providers, which makes it more effective to have these service providers drive the innovation process.
Projects
All the projects in the innovation process together form the firm's innovation portfolio. The sum of your projects should enable you to execute your innovation strategy. And to spread risks, you want to have a balanced portfolio of innovation projects that consists of small and large investments, and short and long-term returns.
Small, short term projects are called incremental innovations. These can be process improvements, additional features, or simple additions to current offerings.
More fundamental changes require typically more time and budget to develop. These are called radical innovations. Research endeavors, or advanced development, often fall outside the scope of the innovation process. As research has the aim to find something new, which may or may not lead to a commercial solution.
Disruptive innovation is a special kind of innovation project, as it aims to replace the organization's current offerings and thereby destroying the current capabilities and value of the firm. It is well documented, that such disruptive innovation projects need protection high in the organization to stand a chance of survival and success.
There are different methods for different kinds of projects, see below. One practical problem I have encountered is that innovators typically don't set out to engage in incremental or radical innovation. They have an idea or problem to solve. If you ask them, they all will say that their idea is really novel, as a badge of honor. However, how radical or disruptive the solution actually is, often only becomes clear during the innovation process, which makes it a challenge to select the right method and approach at the start of each innovation project.
Portfolio management
Portfolio management is a process by itself, to ensure that the sum of projects is more than the parts. Most portfolio analyses show graphs similar as the one below, where the position visualizes the increasing uncertainty and duration of the innovation projects, size indicates the resources required for the project, and colors are used to indicate different involved business units. Such graphs provide a very insightful snapshot of an organization's portfolio of projects.
Another way of classifying incremental and radical innovation projects is by identifying projects as horizon 1, 2 and 3 projects.
The value of dividing projects in different horizons is to spread the risk and ensure there is some return from innovation activities, even in the short term.
For many organizations, the challenge is not generating ideas but choosing where to focus. Portfolio management helps decision-makers balance short-term improvements with longer-term opportunities, while ensuring that resources are not spread too thin across too many initiatives.
Process
Compared to the other topics, the innovation process itself is remarkably consistent across industries and organizations.
Every innovation project must accomplish a similar set of tasks: understanding the problem, validating the opportunity, developing the solution, and bringing it to market.
What differs between organizations is not the steps themselves, but how they are executed, who performs them, and how decisions are made along the way.
Generally speaking, innovation teams need to complete the following steps:
- Describe the project
- Validating the concept
- Obtaining funding for the development
- Develop the solution
- Implement the solution (with a first user)
- Obtain paying clients
- Scale the solution
Validation
Especially for the first set of tasks, there are many different terms: ideation, fuzzy-front end, customer-discovery, incubation, etc. All of these have the same basic premise, at the start of the innovation process you have an idea that is basically worthless until it is validated. It takes work to figure out if an idea is desirable, valuable, and feasible, no matter what method you use.
It pays off to invest time and effort in these early phases, because the better you validate a project in the early stages, the higher the likelihood of success later on.
Scaling
In organizations with volume operations, scaling is a given.
However, in organizations that produce complex systems, scaling requires deliberate effort, because there are limited existing drivers to leverage or scale new offerings. Nevertheless, it is of utmost importance to a successful innovation process to scale successful new offerings, as it is the only way to make a return on investment on the entire portfolio of innovation activities. No organization can afford to invest in the development of a new offering and then not commercialize it to its full potential.
Duration
The duration of innovation projects can vary greatly from just a few months to years for more complex innovation projects.
Over time, the process certainly has become faster. The table below is based on older research data, but it still is indicative:
| 1995 Categories | Average 1995 (weeks) | Average 2004 (weeks) | 2004 Categories |
| New-to-the world | 181 | 104 | New-to-the world |
| New product lines | 126 | ||
| Major revisions | 78 | 62 | More innovative |
| Incremental | 33 | 29 | Incremental |
Newer methods, see below, seem not have made much difference in overall development timelines. Instead, they mostly have improved the accuracy with which we vet projects. Time to market is obviously important, however, it is irrelevant if you are not working on the right solution.
Methods
Today, many organizations combine multiple innovation methods rather than relying on a single framework. For example, stage-gating can provide overall structure, while lean startup techniques help test assumptions early and agile development accelerates implementation.
There are many methods to guide innovation projects through the innovation process. Below are a few of the most common approaches.
The stage-gate model is the bigger framework, that most organizations use to provide structure to their innovation process.
Discovery-driven-growth and lean-startup allow for better processes and decision making at each stage of the innovation process because both methods embrace uncertainty as a given.
Lean or six-sigma and similar approaches have proven their value in the realm of incremental innovations. These are approaches that allow for data-driven improvement actions, while quickly cycling through plan, do, check, act cycles.
Design thinking has become the cornerstone of product development, where observation of current users has proven invaluable to make better decisions in the design process, which ultimately leads to better solutions.
Agile has been adopted from software development, where iterating and trying things is so cheap, that it presents its own challenges.
Stage-gate
The stage-gate methodology is probably the oldest and most used concept to manage the innovation process. The idea is simple. The innovation process is divided in separate stages and acts as a funnel. Because of all the uncertainties, innovation projects are approved to proceed from one stage to the next if they meet the stage criteria. Many projects are started, while few projects - only the most promising ones - make it through all the gates.
The stage-gate process also facilitates portfolio management, as it will tell you where your projects are in the process.
Discovery driven growth
Rita Gunter- McGrath and Ian MacMillan were the first to come up with a better way of dealing with uncertainties in the innovation process. Instead of planning and executing, they suggested that a better approach would be expressing and testing uncertainties. They call this discovery-driven planning:
Unlike traditional project management, in which much is known, discovery-driven planning forces managers to articulate what they don’t know. It sets the stage for disciplined learning during the innovation process.
Lean startup
Central to the lean startup methodology is the assumption that startup companies invest their time into iteratively building products or services to meet the needs of early customers. The lean startup method has been applied to companies, so they can reduce market risks and sidestep the need for large amounts of initial project funding and expensive product launches and failures
Similar to discovery-driven growth, the process is a combination of business-hypothesis-driven experimentation, iterative product releases, and validated learning.
While lean startup was built on discovery-driven planning, it is the better-known methodology.
Lean six sigma
Other than the word 'lean', Lean Six Sigma has nothing to do with the lean startup method. In general, when referring to lean, those in the entrepreneurship world think of Lean startup, while those in corporations think about lean manufacturing and lean six sigma, the method to improve existing products and processes.
Lean Six Sigma is a combination of approaches to reduces process defects and waste, through a reduction in variation. PDCA stands for the Plan - Do - Check - Act cycle. So similar to discovery-driven-growth it uses an iterative approach to get to the best solution.
Because of its focus on the current process and the reduction of variation, Lean six sigma is often seen as a hindrance to more radical innovative solutions.
Design thinking
Design thinking is a process for creative problem solving that puts the human experience at its core.
It consists of three main pillars:
- Empathy — Understanding the needs of those you’re designing for.
- Ideation — Generating a lot of ideas. Brainstorming is one technique, but there are many others.
- Experimentation — Testing those ideas with prototyping.
While it can also be used in the earlier stages, to validate the problem, its approach is especially valuable in the development phase, when the solution is being created.
Agile
The software development community has developed its own approach to product development, and this method has made its way into the innovation process as well.
Similar to Lean six sigma, Agile is also an iterative approach that is performed in a collaborative environment by self-organizing teams. In this case, it was developed with the goal to create high-quality software in a cost-effective and timely manner, while meeting stakeholders' changing needs.
Which method to use when?
In general, I would say, use stage gating as the larger framework. Use discovery-driven planning throughout, while complementing this with techniques from the lean startup approach in the early phases, and design-thinking during the development phase. Use Agile for the development of software solutions and lean six-sigma to improve on what is already out there.
In any case, having a structured process is better than having none. Without a structured approach to innovation, you will be reinventing the wheel with each project, which is costly and frustrating for innovation teams. Worse, your innovation process will never get better because you have nothing to compare.
Even if you only innovate once every so often, it pays off to formalize the process. If you don't have the expertise in-house, you can always rely on solutions like our Programs to provide your teams with a simple yet rigorous innovation process.