People behave according to how they’re measured.
The metrics we implement usually originate from good intentions, but the unintended consequences can be counterproductive. Before you implement agile metrics for performance or output, consider the implications of these measurements.
Many traditional metrics are output-driven and overlook desired outcomes. An output is something like “the number of widgets produced per day.” An outcome might be, “how well did this new widget meet our customer’s needs?” Agile teams focus on desired outcomes more than measured output.
Agile metrics are powerful tools
Metrics can be powerful tools for planning, inspecting, adapting, and understanding progress over time. Rates of success or failure let a scrum team know whether it needs to make changes or keep doing more of what works. Time and cost can highlight the benefits of agile projects and support an organization’s financial activities. Metrics that quantify people’s satisfaction help a scrum team identify areas for improvement with customers and the team.
Again, before you implement these or any other metrics, consider your desired outcome. Then tailor a metric to encourage that desired outcome. But, if the metric seems to be doing more harm than good, pivot. Always inspect and adapt. Metrics should encourage behavior that aligns with your values and desired outcomes.
Here are 10 key agile metrics (some familiar, some maybe not) to consider.
Return on investment
Return on investment (ROI) is income generated by the product less project costs: money in versus money out. ROI is fundamentally different in agile projects than in traditional projects. Agile projects have the potential to produce income with the first release and can increase revenue with each new release.
(Remember, you can calculate ROI based on other value measurements than revenue. For example, many internal or not-for-profit projects have their own definition of value. Perhaps cost savings, time savings, or some other user-perceived value, which is usually quantifiable and substitutable in the revenue portion of the ROI calculation.)
To fully appreciate the difference between ROI on traditional and agile projects, consider a single project, and compare the results in each approach. Both approaches have the potential to generate $100,000 in income every month after requirement completion. They each have a monthly development cost of $80,000.
Under a traditional approach, you would gather, define, design, develop, test, and deploy requirements. Let’s say deployment happens after six months. To make a long math story short, you won’t realize the first positive dollar of ROI until 11 months into the project, five months after deployment.
With an agile approach, new functionality is available as early as right after the first iteration or sprint. If you can generate income after the first month, the breakeven point happens much sooner. This boosts the overall ROI on the project, as much as three-fold.
ROI metrics help justify projects from the start because companies can fund projects based on ROI potential. Organizations can track ROI for individual projects as well as for the organization as a whole.
For a more in-depth breakdown of this ROI calculation and more about capital redeployment as it relates to ROI, check out our book, Agile Project Management For Dummies.
Satisfaction surveys
On agile projects, a scrum team’s highest priority is to satisfy the customer early and often. At the same time, the scrum team strives to motivate individual team members and promote sustainable development practices.
A scrum team can benefit from digging deeper into customer and team member experiences. One way to get measurable information about how well a scrum team embodies agile principles is through satisfaction surveys.
- Customer satisfaction surveys: Measure the customer’s experience with the project, the process, and the scrum team. The scrum team can use survey results to examine processes, continue positive practices, and adjust behaviors, as necessary.
- Team satisfaction surveys: Measure the scrum team members’ experience with the organization, the work environment, processes, other project team members, and their work. Everyone on the scrum team can take team surveys. Customer survey results over time can provide a quantitative look at how the scrum team is maturing.
Survey results will be more honest and freely given if the organization fosters a culture of openness, transparency, and support for learning. Also, avoid leading questions and encourage open-ended questions to uncover responses to your questions you’ve not contemplated.
Defects in production
Defects are a part of any project. However, testing and fixing them can be time-consuming and costly, especially when they reach production. Agile approaches help development teams proactively minimize defects.
As development teams iterate through the development of a requirement, they test and find defects. The sprint cycle facilitates fixing those defects immediately before they reach production. Ideally, defects in production don’t occur due to automated testing and continuous integration.
Tracking defect metrics can let the development team know how well it’s preventing issues and when to refine processes. To track defects, it helps to look at the following numbers:
- Build defects: If the development team uses automated testing and continuous integration, it can track the number of errors at the build level in each sprint. By understanding the number of build defects, the development team gathers insight on whether to adjust development processes and environmental factors to catch them sooner.
- User acceptance testing (UAT) defects: The development team can track the defects the product owner finds when reviewing completed functionality in each sprint. By tracking UAT defects, the development team and the product owner can conclude if they should refine processes for understanding requirements. The development team can also determine whether adjustments to automated testing tools are necessary.
- Release defects: The development team can monitor the number of defects that make it past the release to the marketplace.
- Delay in defect discovery: Development teams can also track the number of days between user story acceptance and defect discovery. The fewer days that pass since a developer worked on the functionality, the lower the cost to fix the defect.
Whether defects are increasing, decreasing, or staying the same are useful metrics to spark discussions on project processes and development techniques at sprint retrospectives.
Sprint goal success rates
One way to measure agile project performance is the rate of achieving the sprint goal.
Before reading further, let’s make something clear about sprint goals. The sprint may not need all the requirements and tasks in the sprint backlog to achieve the goal. However, a successful sprint should have a working product increment that fulfills the sprint goals and meets the scrum team’s definition of done: developed, tested, integrated, documented, and approved.
Throughout the project, the scrum team can observe how frequently it succeeds in reaching the sprint goals and use success rates. This information provides clarity on whether the team is maturing or needs to correct course. Sprint success rates are a useful launching point for inspection and adaptation.
Time to market
Time to market is the amount of time an agile project takes to provide value by releasing working functionality to users. Time to market helps organizations recognize and quantify the ongoing value of agile projects. It’s especially crucial for companies with revenue-generating products. This metric aids in budgeting throughout the year. It’s also important to have a self-funding project (a project being paid for by the income from the product).
Organizations may perceive value in a couple of ways:
- When a product directly generates income, its value is the money it can make.
- When a product is for an organization’s internal use, its value will be the employees’ ability to use the product. It will contain subjective factors based on what the product can do.
When measuring time to market, consider the following:
- Measure the time from the project start until you first show value.
- Some scrum teams deploy new product features for use at the end of each sprint. For scrum teams that release with every sprint, the time to market is simply the sprint length, measured in days.
- Other scrum teams plan releases after multiple sprints and deploy product features in groups. For scrum teams that use longer release times, the time to market is the number of days between each release.
Lead and cycle times
Lead time is the average amount of time between receiving a request for a requirement and delivering it finished. Cycle time is the average time between when development on a requirement begins and delivery.
Agile teams work in a lean environment, one that seeks to eliminate waste. Constraints exist in every flow of work or stream of creating value. Agile teams continually seek ways to find and remove constraints to maximize workflow through the system.
Lead and cycle time provide a measurement of where bottlenecks may exist as well as stakeholder’s expectations regarding how long a submitted request takes to complete, on average.
If the lead time for a scrum team is 45 days, but the cycle time is only five days, this discrepancy may alert them to evaluate the planning and product backlog refinement process. Once aware, they can see how they might tighten the 40-day difference. Likewise, if the lead time is 45 days, and the cycle time is 40 days, the team may evaluate development workflow for bottlenecks. In any case, scrum teams should always be proactively removing constraints to decrease lead and cycle time appropriately.
Cost of change
Agile leaders and teams embrace change for the customer’s competitive advantage. But acceptance of change should not mean acceptance of unnecessary costs associated with it. As agile teams inspect and adapt the product and processes, their goal should be to minimize the effect of change continuously.
Increasing product flexibility is a common way of reducing the cost of change. With software development, using a service-oriented architecture (SOA) strategy allows agile teams to make each component of an application independent of others. Doing so means that the entire system doesn’t require changes when one element must change. Development, testing, and documentation require significantly less effort.
In manufacturing, standardization and modularization of parts allows car manufacturers, such as Toyota and more recently WikiSpeed, to build cars quicker and with less wasteful rework due to incompatibility.
Value stream mapping is a common technique for pinpointing constraints in a system or workflow. By visualizing (on a whiteboard, for instance) each step in a process, agile teams can identify where introducing changes forces the most stress or cost on processes. When you recognize a constraint, the scrum master and other organizational change agents can work to remove it, decreasing the cost of future changes in the system.
Team member turnover
Agile projects tend to have higher morale. One way of quantifying morale is by measuring turnover. Although turnover isn’t always directly tied to satisfaction, it can be valuable to look at the following metrics:
- Scrum team turnover: Low scrum team turnover can be one sign of a healthy team environment. High scrum team turnover can indicate problems with the project, the organization, the work, individual scrum team members, burnout, ineffective product owners forcing development team commitments, personality incompatibility, a scrum master who fails to remove impediments, or overall team dynamics.
- Company turnover: High company turnover, even if it doesn’t include the scrum team, can affect morale and effectiveness. It can be a sign of problems in the organization. As a company adopts agile practices, it may see turnover decrease.
When the scrum team knows turnover metrics with context, it may take actions to maintain morale and improve the culture. If turnover is high, start asking why.
Skill versatility
Mature scrum teams are typically more cross-functional than less mature ones. By eliminating single points of failure in a scrum team, you increase their ability to move faster and produce higher quality products. Tracking skill versatility allows scrum teams and functional managers to gauge the growth of cross-functionality.
When starting out, capture the existing skills and levels contained at each of the following organizational structures:
- Per person skills and levels
- Per team skills and levels
- Per organization skills and levels
Over time, as each person enhances the quantity and level of skills, the constraints and delays due to skill gaps disappear. Agile teams are about skills, not titles. You want team members who can contribute to the sprint goal every day without the risk of single points of failure.
Manager-to-creator ratio
Larger organizations usually have a heavy middle layer of managers. Many haven’t figured out how to function well without multiple managers handling personnel, training, and technical direction on development issues. However, you need to strike the right balance of managers and individuals who produce product.
Remember, every dollar spent on someone who manages organizational processes is a dollar not spent on a product creator.
Track your manager-to-creator ratio to identify bloat and ways to minimize the investment you’re making in people who don’t create product.
Ready to implement more impactful agile metrics?
For more details on these agile metrics, check out our book, Agile Project Management For Dummies.
If you want to improve your team’s agility, contact an advisor today.
