There is a problem in the Agile community. Pundits tell us to prioritize our backlogs to generate the best possible ROI. But no agile teams that I’ve ever worked for or with do this. So the pundits must be wrong, because Agile continues to provide stellar results for a lot of product teams. This post is the first of three posts to explain both why the pundits are wrong and how to fix it, and is a precursor to my talk Converting Business Value into Actual Money at the Agile-08 conference on Tue Aug 5th. In this post, I’ll cover the basics of backlog prioritization and the problems with attempting to prioritize based on financial metrics.

Backlog Prioritization 101

In its simplest terms, your backlog is the discrete set of deliverables that your team must create. Regardless of your particular flavor of Agile, all successful Agile projects (and, truthfully, all successful projects, Agile or otherwise) rely on the effective prioritization of work. To prioritize effectively, product managers must identify set of attributes for backlog items that will be used for prioritization, assign them a value, and then sort the list based on the values of these attributes.

The pundits say that you should value your backlog based on ROI, NPV, net cash, time for payback, or any number of other financial metrics that boil down to comparing revenue vs costs for individual backlog items. In all of the years I’ve been doing Agile, I’ve never seen a product backlog where each backlog item slated for the release has an associated ROI analysis. Heck—I’ve never seen a backlog where more than a very few items have an ROI.(The previous statement does not hold true for portfolio backlogs, which are often prioritized for ROI based on sensible market analysis).

Product backlog items that do have revenue associated with them tend to be directly associated with contractual obligations, often referred to as Non-Recurring Engineering (NRE). This kind of ROI is simple: customer customer so-and-so is willing to pay $x for this feature, it will cost $y to implement it, and since $x is >= $y we’ll do it. Except that in quite a number of cases product teams will do it even is $x <= $y, because customer so-and-so is either a really big customer, or they rationalize that if customer so-and-so wants it other customers will want it, or that they need the revenue, or, because only these items have ROI calculations, they bubble up to the top because they have more data, regardless of whether they are the best choice.

When the performance gap between espoused theory and actual practice is large you need to look harder at both to understand what’s causing the gap. Sometimes, practioners are simply “street smarter” than their more academically motivated counterparts, and the espoused theory has to change to better match the realities of practice. At other times, the practice is just perpetuating old-thinking, and the theorists have a better approach. In this case, I think that street smart Agile practitioners are smarter than those Agile pundits who spend most of their time teaching theories instead of learning from practice.

The Problems With Prioritizing Your Backlog Based on ROI

There are several reasons why prioritizing your product backlog based on ROI is problematic. Here are just a few. Feel free to add your own.

• To establish a realistic ROI, you need to engage in appropriate market research. Unfortunately, most market research that can establish realistic ROI, such as a conjoint analysis or a willingness-to-pay study, is usually based on relatively large chunks of functionality that may or may not correlate to what is on the backlog. The unfortunate side-effect is that the release planning process, in which the development team negotiates the actual features with product management, can result in changes to the tested features that are so large that they original research is invalidated.
• In attempting to solve this problem, many Agile teams are adopting the Incremental Funding Methodology (IFM) popularized in the book Software By Numbers by Denne and Cleland-Haung. Simplifying quite a bit, IFM is based on using market research to establish Minium Marketable Features (MMF) that can be used to maximize NPV. While I love this approach in theory, because it celebrates direct customer input, I haven’t found any examples of this being used in practice. I suspect that one reason is that the method, while relatively easy to explain, has a significant amount of overhead, and defensible NPV needs to be based on defensible revenue data.
• The IFM also suffers from the same problem as much traditional market research in Agile projects: the backlog items tested are created before release planning, but the backlog items implemented are changed during release planning. This isn’t to say that the concept of market research in an agile project is fatally flawed. It isn’t. We need more market research in the Agile community, not less. We just need market research tuned to Agile projects, something I’ll cover in Part 3 of this series.
• A related challenge is that Agilists prefer to manage relatively small user stories, but customers tend to value groups of related and often interdependent stories that together solve complex problems (something Jeff Patton refers to as “task” focus). And while any good Agile Product Manager hates interdependent backlog items, the reality is that sometimes backlog items really are interdependent. To illustrate, one Enthiosys client was building regulatory compliance software in the trucking industry. Many of their customers had large operations in the US and smaller operations in Canada and Mexico. The NPV for the MMF “provide compliance software in all geographic markets” would be positive, but the NPV for the MMF for each individual backlog item (one for each market) would be negative.
• Creating, executing, and analyzing a statistically significant market research project can take longer than a release cycle for a good agile team. In one client project, Enthiosys participated in a comprehensive conjoint analysis for a rather large system. In this case, it was the right choice for the client, but the overall project took about 7 months—far longer than the release cycle of most Agile teams. And the expense ($100K’s) means that it is highly unlikely that any company would fund more than one of these projects per year.
• Statistically significant market research may be antagonistic to Agile. Consider that the co-creation of value, which involves that active participation of companies and their customers, is evolving to the point where many market segments don’t expect to be asked to respond to traditional surveys. Instead, they expect to be allowed to collaborate with the company—and ther customers—to jointly develop specifications and solutions to complex problems, something our client SAP (among others) is pioneering through their Collaboration Workspace.
• Agilists celebrate our ability to respond to change rather than following a rigorous plan. Which sounds great until you’re asking a product manager who just invested a lot of time and money in her market research to reshuffle the backlog based on new information. They’ll be torn. The market research says that they should stick to the plan. The new information says that it should change. The greater the investment in the market research, the more challenging it is to make the change.

I suspect that even with all of these problems, pundits continue to recommend this approach because it sounds easy, it creates long-term, and therefore, expensive consulting projects, and it makes Agile easier to sell to MBA-trained senior executives (“See? We’re prioritizing by business value, in terms that you’ve been trained to understand—NPV. Just don’t ask me to show you how I got that number, OK?). And, since I don’t see much emphasis on rigorously testing if the projected NPV matches the actual NPV, and what to do if they don’t, I’m admittedly skeptical of just how well ROI based prioritization methods perform over time (which is, if you’ll notice, the core aspect of NPV; money sooner is better).

In summary, consistently prioritizing your backlog for ROI is a fools errand. In part two of this series, I’ll discuss how to create sets of attributes for prioritization that every Agile Product Manager can leverage. In part three of this series, I’ll show how to gather the data for these attributes and how to use them to prioritize your backlog for sustained success.