Call today: 866.652.9866

Be Careful What You Measure: 10 Metrics For Agile Teams

by Steve Ostermiller
An old GE voltage meter displaying a high voltage reading

People behave according to how they're measured.

The metrics we implement are usually based on good intentions, but the unintended consequences can be counterproductive. Before you start implementing measurements on performance or output, consider the consequences of the metrics you come up with before implementing them.

Many traditional metrics are output-driven and overlook the desired outcomes. An output is something like "the number of widgets produced per day." An outcome might be "how well did this new widget meet our customer's needs?" Agile teams focus on desired outcomes more than measured output.

Metrics can be powerful tools for planning, inspecting, adapting and understanding progress over time. Rates of success or failure can let a scrum team know whether it needs to make changes or keep doing more of what works. Time and cost numbers can highlight the benefits of agile projects and provide support for an organization's financial activities. Metrics that quantify people's satisfaction can help a scrum team identify areas for improvement with customers and with the team itself.

Again, before you implement these, or any other metrics, consider what your desired outcome is, then tailor a metric to encourage that desired outcome. But, if the metric seems to be doing more harm than good, pivot. Always inspect and adapt. Metrics should encourage behavior that aligns with your values and desired outcomes.
Here are ten key metrics (some familiar, some maybe not) you might consider to help guide your agile teams.

Return on Investment

Return on investment (ROI) is income generated by the product less project costs: money in versus money out. ROI is fundamentally different in agile projects than it is in traditional projects. Agile projects have the potential to generate income with the first release and can increase revenue with each new release.

(Remember that ROI can also be calculated based on other value measurements than revenue. For example, many internal or not-for-profit projects have their own definition of value--perhaps cost savings, time savings, or some other user perceived value, which can usually be quantified and substituted in the revenue portion of the ROI calculation.)

To fully appreciate the difference between ROI on traditional and agile projects, consider a single project and compare the results in each approach. Both approaches have the potential to generate $100,000 in income every month when all the requirements are finished. Both projects have a monthly development cost of $80,000. Under a traditional approach, all requirements will be gathered, defined, designed, developed, tested and deployed together. Let's say deployment happens after six months. To make a long math story short, this means the first positive dollar of ROI won't be realized until about 11 months after the project started, which is 5 months after deployment.
Under an agile approach, new functionality can be delivered as early as right after the first iteration, or sprint. If income can be generated after the first month, let's say, the breakeven point happens much sooner, increasing the overall ROI on the project, as much as three-fold.

ROI metrics help justify projects from the start because companies can fund projects based on ROI potential. Organizations can track ROI for individual projects as well as for the organization as a whole.
For a more in-depth breakdown of this ROI calculation, and for more about capital redeployment as it relates to ROI, check out our book, Agile Project Management For Dummies.

Satisfaction Surveys

On agile projects, a scrum team's highest priority is to satisfy the customer early and often. At the same time, the scrum team strives to motivate individual team members and promote sustainable development practices.
A scrum team can benefit from digging deeper into customer and team member experiences. One way to get measurable information about how well a scrum team is embodying agile principles is through satisfaction surveys:

  • Customer satisfaction surveys: Measure the customer’s experience with the project, the process, and the scrum team. The scrum team can use customer survey results to examine processes, continue positive practices, and adjust behavior as necessary.
  • Team satisfaction surveys: Measure the scrum team members’ experience with the organization, the work environment, processes, other project team members, and their work. Everyone on the scrum team can take team surveys. Customer survey results over time can provide a quantitative look at how the scrum team is maturing as a team.

Survey results will be more honest and freely given if the organization fosters a culture of openness, transparency, and support for learning. Also, avoid leading questions and encourage open ended questions to uncover responses to your questions you might not have considered possible.

Defects in Production

Defects are a part of any project. However, testing and fixing them can be time-consuming and costly, especially when they reach production. Agile approaches help development teams proactively minimize defects.
As development teams iterate through the development of a requirement, they test and find defects. The sprint cycle facilitates fixing those defects immediately before they reach production. Ideally, defects in production don't occur due to automated testing and continuous integration.

Tracking defect metrics can let the development team know how well it's preventing issues and when to refine its processes. To track defects, it helps to look at the following numbers:

  • Build defects: If the development team uses automated testing and continuous integration, it can track the number of defects at the build level in each sprint. By understanding the number of build defects, the development team can know whether to adjust development processes and environmental factors to be able to catch defects even sooner in their development process.
  • User acceptance testing (UAT) defects: The development team can track the number of defects the product owner finds when reviewing completed functionality in each sprint. By tracking UAT defects, the development team and the product owner can identify the need to refine processes for understanding requirements. The development team can also determine whether adjustments to automated testing tools are necessary.
  • Release defects: The development team can track the number of defects that make it past the release to the marketplace.
  • Delay in defect discovery: Development teams can also track the number of days between user story acceptance and defect discovery. The fewer days passed since a developer worked on the functionality, the lower the cost to fix the defect.

The number of defects and whether defects are increasing, decreasing, or staying the same are good metrics to spark discussions on project processes and development techniques at sprint retrospectives.

Sprint Goal Success Rates

One way to measure agile project performance is the rate of achieving the sprint goal.

Before reading further, let's make something clear about sprint goals. The sprint may not need all the requirements and tasks in the sprint backlog to achieve the goal. However, a successful sprint should have a working product increment that fulfills the sprint goals and meets the scrum team's definition of done: developed, tested, integrated, documented and approved.

Throughout the project, the scrum team can track how frequently it succeeds in reaching the sprint goals and use success rates to see whether the team is maturing or needs to correct its course. Sprint success rates are a useful launching point for inspection and adaptation.

Time to Market

Time to market is the amount of time an agile project takes to provide value by releasing working functionality to users. Time to market helps organizations recognize and quantify the ongoing value of agile projects. Time to market is especially important for companies with revenue-generating products because it aids in budgeting throughout the year. It's important also if you have a self-funding project -- a project being paid for by the income from the product.

Organizations may perceive value in a couple of ways:

  • When a product directly generates income, its value is the money it can make.
  • When a product is for an organization's internal use, its value will be the employees' ability to use the product and will contain subjective factors based on what the product can do.

When measuring time to market, consider the following:

  • Measure the time from the project start until you first show value.
  • Some scrum teams deploy new product features for use at the end of each sprint. For scrum teams that release with every sprint, the time to market is simply the sprint length, measured in days.
  • Other scrum teams plan releases after multiple sprints and deploy product features in groups. For scrum teams that use longer release times, the time to market is the number of days between each release.

Lead and Cycle Times

Lead time is the average amount of time between receiving a request for a requirement and delivering it finished. Cycle time is the average time between when development on a requirement begins and when it is delivered.

Agile teams work in a lean environment, one that seeks to eliminate waste. Constraints exist in every flow of work or stream of creating value. Agile teams continually seek ways to identify and remove constraints to maximize the flow of work through their system.

Lead and cycle time provide not only a measurement of where bottlenecks may exist but also expectations for stakeholders regarding how long a request they submit may take to be completed, on average.
If the lead time for a particular scrum team is 45 days but the cycle time is only 5 days, this discrepancy may alert the team to evaluate their planning and product backlog refinement process to see how they might tighten the 40-day difference. Likewise, if the lead time is 45 days and the cycle time is 40 days, the team may want to evaluate their development workflow for bottlenecks. In any case, scrum teams should always be looking at removing constraints to decrease both lead and cycle time appropriately.

Cost of Change

Agile leaders and teams embrace change for the customer's competitive advantage. But acceptance of change should not mean acceptance of unnecessary costs associated with changes. As agile teams inspect and adapt the product and their processes, their goal should be to continuously minimize the effect of change.

Increasing product flexibility is a common way of reducing the cost of change. With software development, using a service oriented architecture (SOA) strategy allows agile teams to make each component of an application independent of others, so that the entire system doesn't require changes when one component must be changed. Development, testing, and documentation require significantly less effort.

In manufacturing, standardization and modularization of parts has allowed car manufacturers such as Toyota and more recently WikiSpeed to build cars more quickly and with less wasteful rework due to incompatibility.

Value stream mapping is a common technique for identifying constraints in a system or a workflow. By visualizing (on a whiteboard, for instance) each step in a process, agile teams can identify where introducing changes forces the most stress or cost on its processes. When a constraint is identified, the scrum master and other organizational change agents can work to remove that constraint to decrease the cost of future changes in the system.

Team Member Turnover

Agile projects tend to have higher morale. One way of quantifying morale is by measuring turnover. Although turnover isn't always directly tied to satisfaction, it can help to look at the following metrics:

  • Scrum team turnover: Low scrum team turnover can be one sign of a healthy team environment. High scrum team turnover can indicate problems with the project, the organization, the work, individual scrum team members, burnout, ineffective product owners forcing development team commitments, personality incompatibility, a scrum master who fails to remove impediments, or overall team dynamics.
  • Company turnover: High company turnover, even if it doesn't include the scrum team, can affect morale and effectiveness. High company turnover can be a sign of problems in the organization. As a company adopts agile practices, it may see turnover decrease.

When the scrum team knows turnover metrics and understands the reasons behind those metrics, it may be able to take actions to maintain morale and improve the work environment. If turnover is high, start asking why.

Skill Versatility

Mature scrum teams are typically more cross functional than less mature scrum teams. By eliminating single points of failure in a scrum team, you increase its ability to move faster and produce higher-quality products. Tracking skill versatility allows scrum teams and functional managers to gauge the growth of cross-functionality.

When starting out, capture the existing skills and levels contained at each of the following organizational structures:

  • Per person skills and levels
  • Per team skills and levels
  • Per organization skills and levels

Over time, as each person increases his or her quantity and level of skills, the constraints and delays due to skill gaps disappear. Agile teams are about skills, not titles. You want team members who can contribute to the sprint goal each and every day without the risk of single points of failure.

Manager-to-Creator Ratio

Larger organizations likely have developed a heavy middle layer of managers. Many organizations haven't figured out how to function well without multiple managers handling personnel, training, and technical direction on development issues. However, you need to strike the right balance of managers and individuals who produce product.

Remember, every dollar spent on someone who manages organizational processes is a dollar not spent on a product creator.

Track your manager-to-creator ratio to identify bloat and ways to minimize the investment you're making in people who don't create product.
 
For more details on these metrics, check out our book, Agile Project Management For Dummies.
If you want to improve your metrics to become more agile, contact an advisor today.