Ask the Right Questions
Before getting into what metrics you should be using, it is first important to know why you are using them. If you are looking to measure individual performance or compare performance between teams, stop. These agile metrics will not help you. Using metrics in that way will only cause problems. Now, can you figure out who is producing more than another on a team? Sure. Should you? Absolutely not. Why? Because doing so doesn’t help you understand the whole picture of how the team is doing, nor the department (in the cases where you have multiple feature teams).
Each work item, or on a larger scale, feature, is unique with its own set of technical challenges. Additionally, each member of a team has their own skill sets and technical abilities. Comparing one person against another, or one team against another is logically flawed as they are inherently different. You are asking the wrong questions. Rather than looking at individuals or individual teams, you need to look at the whole. So, how is the team performing as a whole? How is the department performing as a whole? What is the quality of work from that team? Metrics answer some questions, but not all.
The Best Measure of Performance
The greatest measure you can have for a team, or department, is actual feedback from the people the team is serving. There isn’t a magic metric or number that can give you the same quality of feedback that comes from talking to a person. Because, in the end, its about People over Processes. So, if you are on the outside looking in, talk to the people that the team serves and get that valuable feedback, then share it.
If you are on a team, someone should be getting the feedback from the customers/end users and sharing that feedback. That’s the whole point of business agility, gathering feedback early and often and adjusting to improve.
So, the metrics that I will be outlining below are only metrics to assess the process. If you want to assess the people or a team, you’ll have to talk to people and do the legwork to get an accurate picture.
Agile Metrics that Matter
The metrics below are not the only ones that are out there. But, these are the ones that are probably the most useful in determining how a team is working together within their process.
Burn Up Chart
Chris Adams, at the Modern Analyst provides a great overview of what a Burn Up Chart is and why you would want to use it. If your team has set iterations, then this chart will be useful in determining how much work was completed over the course of time in comparison to the overall work needed for a release of a particular feature. In contrast to the Burn Down Chart, this chart gives you the added ability to identify scope creep within an established block of time. Knowing that the scope has changed is useful in determining how the team has performs with change.
Creating a Burn Up Chart
To setup this chart, place your units of time on the horizontal axis. On the vertical axis, increment the story points that your team is working with. As each unit of time is completed, plot the total number of work completed by the team as well as the total amount of work available for the established block of time. As a result, you will have two lines. A horizontal (or mostly horizontal) line representing the amount of work for the feature. Your second line is an upward-sloping line representing the total amount of work completed after each unit of time.
Cumulative Flow Diagram
This diagram shows how work moves through your team’s established process. Ideally you would have bands that slope upwards in equal increments. In practice, you will see bulges and contractions within the graph that indicates where you are experiencing bottlenecks. Identifying these areas will help you analyze a teams flow and areas for improvement. There are various patterns that arise within a cumulative flow diagram and recognizing. A great primer on those patterns can be found here.
Velocity
Velocity is a way to track story points over time if your team uses iterations. This metric isn’t complicated, but you should be looking for incremental improvements over time. It is important to note that this metric will eventually plateau. The plateau indicates the optimal level that your team operates. What you should really be looking out for is dips or trending downward lines over time and diagnose the reason behind those issues.
Caveats to velocity
Velocity only works if you are working within an iteration-based approach. If you use iterations, you can quantify velocity as it is a function of completed work over time. Without the iterations, you are measuring work over some arbitrary time box which doesn’t really get you anything. The reason for calculating velocity is being able to forecast your teams ability to complete future work. Absent a non-changing iteration, there’s no way to make that forecast.
Another caveat to velocity is that it is a historical indicator, not a target. In his Leading Agile blog post, David Nicolette outlines that if you make velocity a target, story points simply get adjusted to hit the target without actually increasing the team’s ability to handle more complexity in the same amount of time.
Lead Time & Cycle Time
In any agile framework, the goal is to release early and often. The quicker you can obtain feedback from a user, the better your product. Lead time measures your teams ability to get something to the end user or customer from request to delivery; its calculated from the end user’s perspective.
Cycle Time, in contrast, calculates time from the perspective of a developer. This metric looks at the time period from when a developer picks up a work item to when the developer resolves that item.
Kanban-based teams typically use these metrics as there is no set iteration (so velocity is useless). Teams look to optimize their flow to improve their lead time. But, like velocity, at some point the line will plateau as there are limits to what a team can get through without any internal changes to physical make-up of the team.
Production Bug & Issue Count
Being quick doesn’t count unless you are delivering quality software; that’s not agile (that’s the “working software” part of the manifesto). So, to measure quality, a prime example is being able to measure bugs that stem from a released feature or work item. This is simply a count over time. The idea is that your team should be adequately testing and verifying that the software works as intended prior to releasing anything to production. That said, it may be unrealistic to fully anticipate every scenario or integration point that the complete increment would encounter. So, bugs will arise. In those instances, how quickly you can handle the bugs is also something to consider as a quality metric.
Bugs vs. Issues
Just like Lead Time and Cycle Time are closely related metrics but vary depending on the perspective, bug and issue counts also are closely related but vary based on perspective. Bugs are issues from encountered from the customer or end-user’s perspective. They are defects in code. Issues are those things that are uncovered after the developer initially finishes their development work but hasn’t been released to production. In other words, issues are what is uncovered in the QA portion of the process. Issues don’t necessarily have to deal with code defects either. It could also be tied to missed requirements. Whatever the root cause, the issue is meant to highlight items that need to be resolved for working software to be produced.
Both of these counts should be trending downward over time. As your team gets more experience working together and working in the code base, issues and bugs will decrease.
A Final Note
This list is not exhaustive look at agile metrics. There are more out there. The key take away is to know how to use them and in which framework they apply. Agile metrics are meant to give you data as to how the team is performing in terms of process. In terms of value and performance, feedback will be your greatest measure. Also, it is important to note that these metrics are serve the team.
Agile teams are self-organizing, which to me, means they can make key decisions and govern themselves. They establish their own rules, monitor their progress, and are accountable to each other for the way the team operates. Dictating processes via standardization negates the notion of autonomy and trust that should exist with agile teams.
So, measure with caution, compare with care, and validate your conclusions as to performance with human feedback.
