Measuring agile success

Measuring Agile Success: A look at Metrics

In the just released VersionOne 9th Annual State of Agile Survey, they asked how people measure success on a day-to-day basis. Respondents provided 23 different answers, and the number one selection was Velocity (59%) followed by Iteration burndown (51%) and Release burndown (39%).

Half way down the list, at number 11, Budget vs Actual cost is listed at 22%. It is not surprising to see that people value success using non-monetary criteria. After all, the Agile Manifesto focuses on Working Softare as one of its values, and yet the respondents still listed the Ability to change Organizational Culture (with 44%) as the highest barrier to furtherAgile adoption.

Observing this key metric, it became apparent to me that the Manifesto’s 7th Principle, “Working software is the primary measure of progress” was really coming through on the survey even while people were saying that they are struggling culturally with change.

A closer look at Budget vs. Actual

In a recent client interaction, a consultant of ours was invited to a meeting with a Project Manager to discuss project status and reporting. It was explained that the Comptroller was expecting a monthly update of budget vs. actual dollars spent for the project. What seemed like a straight forward request, turned out to generate a lot of discussion due to one key row on the report: Phase.

You see, this report wanted the project phases listed by where the money was being spent: Analysis, Design, Build, Test or Deploy. These phases work perfectly when the project is following a Waterfall methodology, but when adhering to Agile, what do you put in the Phase column?

After considerable time spent discussing this reporting issue, we decided to go to the source and we scheduled a call with the Comptroller. Again, taking a cue from the Manifesto by valuing Individuals and Interactions over Process and Tools. We wanted to ask him directly what he was looking for from our project.

He explained the report to us and then we explained how our project was adhering to Scrum and what that meant. After explaining to him that we would most likely list all of the phases in one month of budget vs. actual’s, the conference call got quiet as we waited for questions.

The real question

“What I need to understand is, are we on track from a budget perspective and is the business getting what they are paying for?” Now we were talking.

We suggested a burndown that would show how the project was spending the budget each month. We also explained Story Points and agreed that what was important for this report was a summary of software delivered based on business value. While a Story Point burndown would have been the best indicator of delivery, in the end, it was decided that a summary of progress would suffice.

One meeting was all it took.

What started as a tense discussion as we unsuccessfully tired to communicate an Agile project using a Waterfall template, led to a clear understanding of expectations on all sides. As we were finalizing the details, the Comptroller had one more question for us.

“So, if I understand this correctly, the business will continue to feed Stories to the team and the team will continue to deliver until the budget has been spent? This puts it in the businesses hands to decide what they want for the money they have been allocated.”

Exactly!