Even a great metric, if used alone, might lead to tunnel vision, and incentivize teams to maximize that metric at the expense of all else.
Definitions of metrics change with the audience, so does the meaning of success.
One of the biggest challenges in my recent role was aligning the right audience to appropriate metrics. Many people, by looking at the inappropriate set of metrics, were unnecessarily putting their efforts into tracking, measuring, and correcting certain behavior/process/practice at the workplace.
One great example is, in an agile setup, a manager in the company tracking a scrum team’s story sizes against the time taken for their completion. This can certainly be a metric. The scrum team may internally track to optimize their definition of story points/sizes to use the learning in the future. But this metric can not be a tool for an external audience, like the manager following it.
Metrics:
A metric should derive the questions that lead to actionable insights. It is very important that the metrics are thoughtfully aligned and restricted to the audience. Not all metrics are appropriate for all audiences. Not all metrics tell you valuable stories.
Understand that, as a human, we may tend to tag unconscious bias to the metrics. That is why the metrics themselves should be able to share as clearer stories as they can.
Metrics should not just be numbers, they should be the starting point of a conversation about anchors and engines to success.
Many times people think that a metric is Me-Trick the data. That is exactly what the metric is NOT.
The collection of metrics data should be transparent to avoid the willing or unwilling rigging.
As we all know, the journey of data is Data >> Information >> Knowledge >> Wisdom, and if we minimize human intervention, we can read the best story the data wants to tell.
A metric should be easy to calculate and understand. Metrics, which are overly complex or not fully understood, even if they provide good insights, are not useful in guiding day-to-day activities.
Vanity Vs Actionable Metrics
Everyone loves the metrics that look good. Vanity metrics are those that look good, but fail to provide guidance for the next steps. A great example is the number of views on a promotional video. This can’t actually tell you what is going to happen next. This can’t tell you exact steps on how to turn those views into prospects.
A good metric gives you actionable insight, well in time. A delayed insight, i.e. lagging indicator, can tell you a story to retrospect, but often fails to provide corrective measures before the failure occurs. You need strong leading indicators on your radar. The lead indicators (a.k.a. lead measures) often predict the success of lag measures (lag indicators). Weight loss is an example. How much weight you actually lost over a span of time is your lag measure. You will know about it only when you can’t do anything about it. The lead measures to the weight loss goal could be calorie intake per day, calorie burn per day, and active time per day. Control the lead measures to better predict the lag measure.
Other examples of lagging indicators are customer satisfaction, % of work delivered in a sprint, corporate profit and revenue in a quarter, and the number of accidents on a construction site.
Examples of leading indicators include customer confidence, backlog readiness and availability of the team, stock price, and the % use of safety equipment at the construction site.
One should be observing a good composition of actionable metrics (lagging and leading measures) to monitor the overall health of the path to success and take timely actions as needed to reach the goals.
The vanity metrics can be captured to decorate your reports and dashboards, but don’t give it much importance. Your time is precious.
Control Limits:
The control limits of your chart represent your process variation and help indicate when your process is out of control.
Not all metrics, especially related to human performance and mind-works, can follow a linear line. The upper and lower control limits should be based on the acceptable variation in the process. An intervention may be needed when control limits are crossed.
An example is, the capacity vs commitment of a team in a sprint. As a Scrum Master, I do not intervene with my team as long as they are committing the work within +/-15% of their capacity for the sprint.
More than the data, one should be looking at the signals. Sometimes the signals are hidden in the pattern, not in the data points. An example is a team delivering 100% of committed items for 6 consecutive sprints, regardless of the unplanned PTOs and flying-in urgent production fixes. Though there is no control limit crossing, the signal is that the team is actually under-committing to avoid failure.
Outliers:
Outliers are the data points situated abnormally away from the pattern. Since these are abnormal behaviors, there is no standard protocol to reflect on them. It is up to the metrics and the analyst to derive action items if any. There could be signals in the leading indicators that may help to assess the situation and to respond to potential questions raised by the outliers.
Industry Examples:
1. A manager’s metrics
There are few managers finding pride in digging into the minute-to-minute life of human ‘resources’. They pull metrics to find lines of code per day per coder, numbers of checked-in code, resource idle time, punch time, scrutinizing timesheets, etc. These metrics can be helpful, but most of the time these metrics are used to emit pain.
In this case, success can be defined differently, and a good set of metrics can be introduced. Better example metrics to emit positivity and make a positive influence could be short-term and long-term goals of associates, predictability of team, production bugs delivered by the team, the business value generated by a team over time, employee availability, and 360-degree feedback.
2. Employee turnover
If many employees leave a company in a year, they have high staff turnover. If the workforce is comparatively stable and employees in the company tend to stick around, they have a low employee turnover.
Situation: A mid-sized company had a 0.8% turnover in the past year.
In general, if the top management or a prospective candidate sees that the company turnover is only 0.8% in the past year, they may have a quick impression that the company is doing great on this part. They may not be wrong. But this single metric could be misleading. The larger picture could be different.
One should also slice and dice the turnover trend to support or challenge the initial impression. Here are two tandem metric examples.
Metric #1: Employee turnover by the length of service
Indicator: Though the turnover is low, the length of service of the leaving employees may give a different perspective.
Result: What if most of the newly joined employees quickly leave the company?
Signal: The company culture may not be generative and welcoming to new ideas.
What it means to the prospective candidate: The prospective candidates may cautiously accept the offer.
What it means to the management: The management may have multiple action items since this is about the company’s culture and sustainable growth forward. Potential questions to respond could be: Do we have enough challenges for the generative minds? Are we open to new ideas and market trends? What can we do to retain the new people we hire?
Metric #2: Employee turnover by position/level
Indicator: Though the turnover is low, the position or hierarchical level of the leaving employees may tell a new story.
Result: What if most junior employees quickly leave the company?
Signal: The company culture may be bureaucratic. Command and control, bullying could be happening in the company.
What it means to the prospective candidate: It’s alarming. Take a very cautious step ahead. Make the right decision.
What it means to the management: Changing the managerial culture could be a time-taking and highly tiring initiative, but if realized the need, it is a must-do thing. Potential questions to respond could be: Is our culture bureaucratic? Which areas/departments or people, in particular, are supporting the command-and-control style of management? How can we induce confidence in the associates so they may talk fearlessly? How can we provide a safe environment to all working for the company?
Take away:
Look at the big picture!
Determine the metrics that define the real success of your undertaking.
Don’t fall prey to vanity metrics, but realize what actionable metrics may help you find the right path to your goal.
A metric should be used in tandem with other metrics.
Some questions must be considered while introducing metrics to the team/group or organization.
Why does this metric matter to us?
Who is the appropriate audience?
How can we restrict potential misuse?
How will the metrics be collected and distributed to the target audience?
What is the collection and distribution cadence of the metrics?
How can we make our measurements transparent?
Watch this blog
Watch this blog for an upcoming article on:
Agile metrics
Comments