I’m the old guy in the room. There, I’ve admitted it. I’ve been around software development for a long time. As a consultant, I’ve built custom solutions for companies large and small. I’ve worked in product companies that were, and remain, leaders in their field. I’ve led development teams building successful internal tools and external products. I’ve been a member or facilitator of business teams for a broad range of process improvements: standardizing internal controls for Sarbanes-Oxley purposes, refining guidelines for how people should work together, introducing complex applications into organizations and improving workflows that result in signed high-value contracts for product purchases and implementations.
Along the way, I’ve used many product and software development methodologies: Waterfall, Spiral, SCRUM, XP, Six Sigma and Lean just to name a few. What I have found is nearly every business and software development methodology has value when applied to the right type of project and with the right amount of discipline. The amount of value realized depends on many things including the problem domain, the project participants' skill and discipline and the availability of the customer to address issues when discovered.
Question: What’s the best methodology?
Answer: It depends.
Creating a worthwhile software product or a sustainable business is hard. The number of interacting elements is staggering. Here are a few questions I consider.
- What is the complexity of the company—startup, multi-product behemoth?
- What is the size of the market for this product?
- What is the competitive landscape?
- How will we make money from this venture?
- What do we need to deliver so that potential customers will even look at our handiwork?
- How much will this cost us? When will those costs occur? How will we fund this?
- How do we entice the right people to come aboard or stay on board?
In the early decades of software development, this emerging industry followed engineering models that served the world well throughout modern history. Collectively, these methodologies can be described as “anticipatory.” The primary assumptions behind anticipatory methodologies are that:
- We know what the ultimate consumers of the product want.
- With enough thinking and planning, we can design an infrastructure that will support the product.
- The cost of change rises sharply as time passes.
Hey, it worked for the pyramids, right?
As time passed, however, the software development community found that many large projects failed because those assumptions proved to be largely incorrect. Product consumers have no idea what they want, and they change their minds constantly. So predicting what customers want is . . . impossible. Building an infrastructure that can support all the features we don’t yet know about is incredibly expensive. The final assumption about the sharply rising cost of change, however, did prove to be true.
There have been many attempts to address the underlying issues that cause failures in product development, whether they are software or non-software products. The attempts that are proving to be successful turn those earlier assumptions on their heads:
1. Make implementation a part of requirements discovery—Potential consumers of a new product idea do not always know precisely what they need. Learning a little about what we need to build gives everyone visibility into what needs to be done and the costs of doing it. We suggest possibilities and gauge their reaction.
2. Incrementally grow the system—It isn’t possible to predict the entire infrastructure that is needed to support our product. We should only build what we absolutely need at the time we need it.
3. Flatten the cost curve—The cost of change must remain relatively flat across the entirety of the product development and maintenance. An implementation that makes it increasingly more difficult to make changes over time drives up the overall cost and increases the risk of breaking existing features—which fuels the need to get the architecture right very early on in the project, when we know the least about what we are trying to accomplish. A more lean, flexible system where the cost of making a change is roughly the same whether making it earlier or later in the project is more desirable. It also allows us to defer decisions until we know more and removes the unrealistic need to predict the future.
These newer approaches attempt to be more flexible and resilient than the earlier anticipatory methodologies. For me, the most difficult yet critical issue is to flatten that cost curve. If you can keep the cost curve flat, then not knowing what to build today is less of a concern, since the cost to build it later, when you know more, is the same.
I am fascinated with the many ways that humans have created to make the difficult bits and pieces of creating new products move forward in the best possible manner. My plan is to address all of those aspects over a series of blog posts.
Next blog post: Extreme Programming—plusses and minuses.