What I love about Quality is that it’s one of the most empirical and philosophical concepts in Engineering, than can be driving both interesting and controversial conversations at the same time, with the different opinions coming for people with very diverse backgrounds. As our team is made of: Devs, Product, UI/UX, SEO, Marketing – everyone has on their agenda their own understanding of this fuzzy concept. Now that everybody wants to join the Road for Quality (Assistance), it is important first to make a pitstop: to understand, as a team, where we aim to arrive, what would be our destination, to be able to know how to tackle and prepare for our journey.
Kicking-off
When you’re starting in a green field from a quality perspective, I’ve seen people being very eager to share possible solutions to problems, and, at the same time, at the opposite pole, facing themselves problems while trying to prioritize what are the first milestones to achieve or having difficult times taking steps for improvement. And there is absolutely no blame game here: how can one get things moving without understanding first where one stands, and then where they’d ideally want to head. Even more, how can one have the feeling if things are improving at all? Here’s where Metrics and KPIs chip in.
During 2019 I got intrigued while reading “Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations”, which, apart from many things, defines 4 modern, simple and straightforward KPIs to look after:
- Lead Time: the time difference between a customer making a request to the request being fulfilled.
- Deployment Frequency: frequency of desirable batches that are delivered to the final customers.
- Mean Time to Restore: how quickly can the team recover from failures.
- Change Fail Percentage: the total amount of failures, as they are defined, divided by the total amount of deploys for a specific period of time.
items which I’ve measured for around 2 months, after an initial alignment with the whole team. I’m not going to dig too much into Accelerate KPIs in this post, as I’d rather explain my full experience with it in a separate one.
First Reactions
As I was making the results of these measurements public, the first reactions from the team started to arrive:
- Can I see how are you measuring X?
- I see that we’re improving lately with in the Y metric, it must be because we’ve followed up on that important action from the past retro!
- Why are we measuring W just like this? Wouldn’t it be better to have an extra metric here that can give us a better overview on what’s happening?
- I don’t think that Z even should be measured at all!
Although the first natural instinct behind all of these questions might be that some people are questioning the whole approach, reflecting it all up from an Quality Assistance perspective, I found out the first nice surprise: the team is communicating out loud that they want to be part of the quality journey. It took me a while to realize, but that click is what changed my hat to the facilitator one and made me later on follow-up:
- Then what are we interested to measure?
- How can we measure all of these as a team with the current processes?
- What should be our aim to reach for each one of them?
What I’ve learned
- Start by measuring something. It might not be ideal, it might not be perfect, but this can be at any point improved if found relevant. Dedicate more energy into kicking this effort off and then you can focus on things like automating the collection of data or building up ultra-correct formulas to calculate. At the beginning, a minimal setup will give you already a huge base to work and start with.
- The key point on KPIs and Metrics is that the team believes in them, and the way they are measured. Without this there’s no easy way to propel full team ownership for actions and improvement points.
- It’s more important that KPIs and Metrics are measured, rather than who owns the responsibility. The realities of different organizations can bring up different types of ownership: QA Lead/Manager owned, Upper Management owned, shared ownership. In Quality Assistance, it’s more important that we facilitate the proper set up of these with alignment of all involved parties.
PS: For the curious ones, we’ve decided that we have as well some other interesting things to measure right now, apart from Accelerate KPIs. As we’re in the beginning with the approach, we don’t mind trying out different metrics and as we don’t find things relevant anymore, scratch them off our list.