As part of my continuing interview series with thought leaders in the agile industry, here is an interview with Ryan Martens, CTO of Rally Software Development.  Ryan is a devout member of the software industry since the early 1980’s. In the mid-90’s, Ryan moved into consulting and re-engineering using Internet technologies with two different companies and numerous clients. His last start-up was acquired by BEA Systems where Ryan directed product management for BEA’s Portal. Since 2002, Ryan has focused his efforts on changing the software industry through Rally Software. Every day, Ryan drives Rally toward becoming a sustainable business, while working to help customers realize the benefits of software agility. Ryan is a board member on the Agile Alliance, Colorado Conservation Trust, Entrepreneurs’ Foundation of Colorado and the Knight Foundation, as well as being a husband, father and farmer in Boulder, Colorado. 

 Aside from all of this goodness, Ryan is just a flat out great guy, a good friend, and very approachable.  Our team works closely with him and his folks at Rally to bring agile practices to the forefront of our company's development activities.  I'd like to offer my thanks to Ryan for all the help he's given us and for taking the time out of his busy schedule to share his insights about agile software development.

Q. You have had the opportunity to work with many software development teams in your career.  Is there something special that seems to define or set apart high-performance agile teams?

A. Cool tools/technologies, cool customers, trust and collaboration have made for fun, high performing teams in my career at 9 software companies and two consulting firms. Being successful enough to be able to continue to play the fun game was the reward. With regards to the special things that set apart high-performance teams, I do not claim to be the expert. Like most, I knew it when I was in it. But thanks to an Agile University instructor and friend, Christopher Avery, we have the research to really put our finger on answers to that question. His research (see great-teams.com) points to three top things that I would totally concur with:

• Trust
• Goodwill/Collaboration
• Clarity in Purpose

Q. The flip side of that coin is equally important.  What would say are the common pitfalls that cause agile teams to fail or be ineffective?

A. I assume you can plot teams on a normal distribution curve, and the great ones and the ones that fail make up the tip and tail of the curve. The ones that fail violate the things that make a great team. Another friend Luke Hohmann, Enthiosys.com, describes the anti-pattern as a “J-O-B.” Whenever it just feels like work, stuff-for-money, or you’re working with people who are painful, it becomes a J-O-B. When it becomes a JOB, you lose the determination to inspect and adapt agile, and you plateau as an agile amateur and ultimately fall back and typically apart.

Q. There are many agile tools in the market place today.  What are your thoughts on the necessity and value of tooling for agile teams?

A. All teams need agile tools, techniques and methods to reinforce the structure and discipline of agile. In the talk that Ron Jefferies and I gave at Agile 2007, we talked about three types of “tools.” There are tools that help you reduce the cost of iterating, tools that help increase your visibility within and across teams, and role-specific tools that help accomplish specific tasks. Of course, there is a continuum of low to high tech in all of these categories.

• Tools that help you reduce the cost of iterating a check-in, a story, or an entire increment include testing tools, mock-up tools, development frameworks, refactoring IDE’s, and testing frameworks that help you reduce your batch sizes. As your batch sizes of running tested features decrease, your agility level increases.
• Tools that help you increase your visibility across the lifecycle like task boards, agile project management and integrated build tools increase in value as the project grows larger. These tools build collaboration, measurement, discipline and trust to make performance more objective.
• Role-specific tools like backlog prioritization, specific types of test tools, and modeling tools help make steps in the agile process more effective. The need for these is varied based on the people in those roles and the technology in use of the project.

You have to think of the tools, techniques and methods as working to support building trust, collaboration and clarity in purpose as well agility.

Q. Your company actively promotes agile practices for software development.  How has Rally adopted agile practices and what value have you recognized from it?

A. As we have grown, we have matured our agile practices and disciplines to manage the growth in development, the business and our customer base. We followed an incremental adoption approach to agile, and the value we’ve recognized has steadily increased. Here are the rough descriptions of the transitions that our team went through.

Step 1 Started Out Iterating – Pull together a new team of 8 into a state of iteration “flow.” We set a two week iteration cycle and worked to stabilize the process so that we could deliver a consistent level of iteration acceptance. We used internal demos to steer our iterations and we gained increasing alignment, component-level quality and company-level visibility. However, we had little feedback that we were building something that was “really” valuable and our testing practices were not good enough to achieve 100% iteration acceptance.

Step 2 Increased our Agility to “Pull” – This came after we had a handful of customers and we moved the single team into a “pull” state. This included roadmap planning based on feedback, release planning to steer iterations and better acceptance testing tools and techniques. We used customer feedback to help prioritize and shape our backlog and better automation to reduce the size of defect pile we had to clean-up before release. We worked hard at this state to get to a zero-defect mentality both inside the iteration and with regards to overall technical debt. As a result, we achieved a consistent level of 90% iteration and 100% release acceptance levels.

Step 3 Scaled to Multiple Scrum Teams - We split the 20 person team that was working in a state of “pull” into multiple scrum teams to make meetings and communication more effective. We instituted scrum-of-scrum and product council meetings and built out the integrated build management system. We also instituted a stop-the-line philosophy with regard to broken builds. Set a goal of 90% successful fully integrated builds. We gained velocity in the individual teams, but struggled with the new hierarchy to steer a multi-team program. We broadened our product portfolio to include two products.

Step 4 Extended Agility Outside the Dev Team – We extended agility out to support and closed the loop with regard to feedback and defect management. By using our Agility Suite to link Salesforce.com and Rally, we tracked the lifecycle and cycle time of customer request. As a result, we gained customer intimacy and trust through increased communication and visibility into our backlog.

Step 5 Vision and Roadmapping – We then extended agility up to synchronize vision and roadmap across the company. Increased our discipline in business planning to help guide and synchronize our product vision and roadmap planning efforts. Also baked hack-a-thons into our regular release calendar on the last week of every release. We gained better alignment across the company and better day-to-day decision making across a distributed company of 75+ employees.

Step 6 Pull System Testing Forward into the Nightly Build - Working to move performance and security suites into the nightly process for all product editions and deployment options. Refactoring acceptance test regression suites to run in parallel and reduce run time by a factor of 10. Increasing the quality of the system and reducing cost of iterating with a faster regression suite.

Q. How does Rally gather and prioritize user stories or requirements from their large user community?

A. We prioritize at multiple levels and with multiple tools:

Daily tasks based on stand-up and any issues in production

• Iteration stories, every 2 weeks, based on iteration rank of ready items from the current release and any patch releases
• Release epics, every 8 weeks, based on value rank from roadmap, critical deals, market pressures, feature request voting by customers, hack-a-thon efforts and technical debt
• Product roadmap themes, every 6 months, based on strategic rank from market rhythms, market threats, market opportunities and the stage of each individual product in its lifecycle
• Product line resource allocation, every 6 months, based on vision, profit and loss, core purpose and business SWOT across the portfolio

Q. In terms of the scalability of Scrum and agile, what major differences have you observed between small-team scrum implementations and large distributed multi-team Scrum implementations?

A. The major difference between small teams and large distributed teams is the forced level of discipline. Many small teams can implement agile somewhat. With limited complexities, these teams can run with less discipline and fewer skilled “craftsmen” and still regularly ship quality software.

Large teams must increase their agile disciplines to manage the scale and distribution complexities or risk falling back to old ways. As a result, the level of tooling to manage cross-team communication, dependencies and signaling teams is a big difference.

Q. For organizations that are just starting out on their agile journey, what advice can you give in terms of cultural shifts they may need to make to support their agile teams?

A. Your blog post from mid-January said it best - take baby steps, celebrate the success and don’t bank all your savings from agile successes. Make sure to reinvest parts of your savings back into a continuous quest to learn and get better. This is a game of regular and continuous improvement.

Internally, we’ve seen great success by adopting agile practices using agile techniques (incremental, iterative, fully inclusive, guided by inspection and adaptation). Don’t try to bite off too much too fast.

Q. Your organization actively engages in and promotes Green IT practices.  Can you briefly discuss simple things every IT organization can do to green their business?

A. I think greening is just like adopting agile. We need to get our linear business process to become more feedback driven and sustainable. In this regard, the simplest things are not necessarily the right things. To green, you must get on a curve of incremental adoption. The right thing to do is to get folks from around the company to volunteer, set regular meetings, charter the group, baseline your issues and start picking off the visible, impactful and easy items. You need to build success and demonstrate value. It is really easy to “greenwash,” just like it is easy to “agilewash.”

Some easy things all of us can do:

• CF bulbs
• Motion lights
• HVAC timers
• PC power management software
• Building upgrades with the help of owner and city zoning
• Purchase Renewable Energy Credits for your server farm or travel
• Flex hours to support car pooling
• Work at home solutions

Q. How do you see agile practices contributing to the greening effort?

A. As I stated above, adopting agile and greening are both incremental improvement journeys, so teams that are on the expert curve to agility are in a good place to be successful in greening. The driving principles behind agile are the Lean principles of:

• Eliminate waste
• Empower teams
• Amplify learning
• Build integrity in
• See the whole
• Deliver fast
• Delay decisions

These are the same driving principles necessary to really create a sustainable company or industry. Fundamentally, I believe we have two perceptions wrong in the high-technology world.

• • 1. We have product mentality that drives us to follow a linear product lifecycle of taking materials, making products, transferring ownership and having customers discard them as waste. By moving software to a service model driven by agile, we can shift the power to software service providers and begin to change the entire high-technology industry to a more sustainable service model.
    2. We believe building software is a deterministic process and should be managed with traditional critical path methods of project management.

I have submitted this as a topic to Agile 2008. If you want to hear it, please jump in and add a comment here - http://submissions.agile2008.org/node/2100.

Q. In general terms, what do you think the future holds for agile software development?

A. Huge growth! This approach to software development is critical in a world with:

• Higher customer expectations for immediate action and quality
• Embedded computing everywhere
• The computer as collaboration environment
• Sustainable businesses networks
• More employees in software/IT because it is a sustainable and globally distributed market

A look beyond agile at the program level and into business agility and reliable innovation

• Closer connection to “the customer” through Web 2.0 communities
• Better tools to get the cost of iterating and current set-based development down
• Extended into other knowledge work areas

I must confess that I have learned the hard way that scrum may not be suited for or welcomed by everyone. I, as the scrum master, have already sat through two retrospectives which basically turned out to be a platform for playing the blame game.

I can't go in to the details, but based on a request from my boss, I had to work with an unconvinced (about scrum) and non-involved product owner who botched up the majority of effort (I say majority since there were other issues too, but none as big as the one stated).

Since it was 'his' project, I couldn't give him the boot given that I was a consultant scrum master for this team. So, at the end of the second sprint (this team is still in trials and they didn't meet the done definition in both sprints), I have come to accept that some people simply cannot adapt to the mindset needed to work inside the scrum framework.

I have decided not to recommend another sprint for this team and that its better for them to continue to work the 'control-tower' way with the product owner in charge.

So, do you folks think I took the right decision?

Let's face it, no matter where you work, fires flare up from time to time.  Some are serious and need to be addressed, some are just smoldering and can be put off for a short time, and some are just people yelling fire when there is no fire.  Some organizations are perpetually in fire drill mode and can't break the habit.  So, if you're an agile organization and your agile teams are committed to completing their iteration tasks, how do you fight these fires without interrupting your agile teams?  Even if you allow a buffer of time within your iteration planning for handling unexpected requests, you still run the risk of distracting your development teams with potentially harmful context switching. 

An interesting solution may be to assemble an agile fire fighting team.  Essentially, the team would be composed of developers from different project teams.  The fire fighting team would be assembled for two-week iterations (synchronized with your project iterations) and team members would rotate in and out of the team.  One team member would be the fire captain.  The fire captain is responsible for handling fire drill requests, triaging them, prioritizing them, and working with the fire fighting team to task them out. During their rotation on the fire fighting team, team members respond to urgent requests and work on them the same way they would any other task in an iteration, committing to address them by the conclusion of the iteration.  What if there are no fires to fight?  Well, don't get too excited...it's not off to the beach for the fire fighting team.  If there are no fires, team members can address defects that may exist on their current project backlogs, they can work on documentation tasks (I knew you'd love that), or they can use the time as part of their hackathon innovation allowance. 

My personal preference is that organizations work as hard as they can to move beyond the fire drill mode.  I think it is harmful to your development teams, reduces overall quality and value you are delivering to your customers, and ultimately it will impact you organization in a negative way. That said, if you can't wean yourself off the fire drill mentality, consider putting together a fire fighting team.

On a daily basis, agile teams should be answering three basic questions: (1) What did you work on yesterday? (2) What are you working on today? and (3) Do you have any impediments to accomplishing your tasks?  That third one is a key question, and one that is answered differently depending on the maturity of your team.  What is an impediment anyway?  It can be a multitude of things, from organizational issues to technical resources, from physical impediments to technical ones.  But how good is your team at identifying impediments and voicing them?  I think that depends on the maturity of the team.  There are always the easy impediments that are surfaced when you first start doing agile.  It's the subtle impediments that are tougher to identify.

I heard something recently from one of our agile teams when I listened in on a few of their daily stand-up meetings. For three meetings in a row, the same developer reported that he worked the same task all three days and was planning on working on it again.  One of the other developers picked up on this and said, "You only estimated 6 hours for that thing.  Are you having trouble with it?  Is there something we can do to help you out?"  There are two things I really liked about this comment.  First it showed concern about the estimate, but there was no blame attached.  It wasn't "Hey, you said it was going to take 6 hours and now you're 3 days in.  Are you really working or are you playing around?"  It expressed real concern.  And secondly, it asked the right question: "Are you having a problem with accomplishing your task?".  Even though the first developer wasn't reporting an impediment, the second was making sure than there wasn't an impediment.  This led to a discussion between the developers about why this task was taking so long.  It turned out that the first developer was waiting on live data (not staged) from one of our clients and couldn't make much progress without the live data.  I was really glad to hear this conversation take place between the developers.  It clearly illustrated the maturity of the team members to work collaboratively to identify impediments to reaching their iteration goal.