A quick and easy guide to measurement

The most common question I get asked is, "How do we demonstrate an ROI for this project?" and that's almost exclusively for projects already built and in production. My standard is to ask, "What are you currently measuring?" which generally elicit vague references to uptime or eyes wandering around the room trying to find something comforting to focus on. It's almost like naughty schoolchildren, but in this case, they've blown tens of millions on something that no one can justify. As a rule of thumb, 95% or more of the projects I see have no meaningful set of measurements.

I'm sure that somewhere, there is a beautiful world where projects are measured given the wealth of books and conferences on this topic. I'm sure I'm just unlucky as I've yet to find that world. I would be very grateful if someone pointed me in the right direction or bought me a train ticket.

Deciding on what you will measure matters because it will change the type of system that you will build. A measure for getting people quickly from high to low altitude will encourage the development of one system - for example, personal solid fuel rockets. However, a measure for doing this safely will promote the development of another - for example, a parachute.

If measurement matters, then how do we build systems with no measures? Contrary to popular belief, investment choices in corporations are often made on a whim, for which a vast amount of paperwork is subsequently created to justify that whim. A typical path is as follows:-

An executive reads some HBR (Harvard Business Review) articles or discusses with some peers about a technology change, like big data or AI. They become enamoured by this change, believing it to be the future - this is the whim that must now be justified. A group is directed to examine this. The group can come up with any conclusion as long as the whim is the correct answer. The group first determines the "cost" of building something that meets the whim. An ROI calculation is then applied to the cost to determine how much value the thing must create. A group (possibly the same) then collects data to justify this value. Ideally, a worthy tome of data, assumptions, financial figures, graphs and justifications for building the thing is created. This is known as the business case. The thing is then approved and built.

If the project is seen as successful, then all is good. The business case gathers dust or is probably lost to history. The problems start if the project is seen as rocky rather than prosperous. The danger is that someone then asks questions. If the assumptions in the business case (assuming the business case can be found) are flawed, then this is usually when someone turns up at my door asking, "How do we demonstrate an ROI for this project?"

They have no measurement, no baselines, nothing except cost and a business case that wasn't worth the paper it was written on - especially if that paper was manufactured by expensive management consultants in the form of a PowerPoint presentation or "ChatPPT", as I like to call it. ChatGPT is at least cheaper.

I'd like this to stop. I want everyone to at least put some basic measurements in place. Even if those measurements aren't perfect, then I'd have something to work on in the future.

Let us start with a cheap and cheerful process for working out some initial measures for a system. This should take, at most, a couple of hours. For this post, I will use an example of a common platform (known as CP) for the development of applications in a Government related agency.

The information you'll need is usually in the minds of many people, so gather many people involved in the project—ideally 6 or 7. Getting 6 or 7 people to agree on a time to meet up online is often the hardest and most time-consuming part of this process. If you complete this step, you're already a winner!

Ask the question, "Where can this system provide value"?

Answers should be on a postcard with a self-addressed envelope ... no, wait ... a Miro board with post-it notes are fine. You can do this all online. You'll get a mix of half-answers, vague answers, consultant answers and measures. That's good as well. In Figure 1, I've shown the answers that my group came up with.

Once you have your lists of words, ask the group to categorise them into themes. These themes are our potential perspectives to examine the system. We will choose one or more perspectives to explore depending on the group size. With 6 or 7 people, we will pick one theme as our perspective.

Make sure that you give people time to discuss these themes. If the collection step took 15 minutes, give the group 30 minutes to categorise - see Figure 2

A couple of points are worth making here.

1) Yes, we're going to map. That's the next step. You choose perspectives because... well ... imagine that no one has ever mapped Paris. You send a group out to map Paris; they return with a map. You now ask a question such as "What's the most important feature of Paris?" and you get the response "Da Giuseppe Pizza" because they've mapped Paris from the perspective of "nice places to eat Pizza".

When you map a space out, add a description for what you are mapping, the perspective you are mapping from and the date. We will look at building measurements for this common platform, so we need to understand our viewpoint (i.e. the current map from our perspective) before asking, "What shall we measure?". If you have a large enough group, you can map and then aggregate multiple perspectives, but that's a more complicated process.

2) Allow time for people to categorise and choose a perspective. Don't rush this bit. In the above example, it was during this discussion that the theme of "assurance" became prominent. Whilst a common platform can help speed up delivery from a technology perspective, governance systems often need to be revised. I know of one organisation with an assurance process requiring over 40 signatures before a system goes live. No amount of fiddling with technology will fix that problem.

You start by identifying the system's broader users, their needs, what components are involved and how evolved those components are. Often, you'll find stakeholders who never actually touch the system but have a strong need associated with it. For example, the sponsor of the standard platform was interested in reducing the assurance burden of building applications.

These maps can be quick and easy; use Post-it notes on Miro, etc - see Figure 3.

Once we have a map, we can ask our question. In our case, we want to know "What should we measure?". That question has two parts - "What could we measure?" and "Is this measure worthwhile?"

The first part is super easy - ask the group to put possible measures on the map, i.e. use the map as a guide to what we could measure - see Figure 4.

Once you have these measures, we place them on a graph of "ease to implement" versus "value to stakeholders". Make sure you give plenty of time for this as it usually creates helpful discussions of the practicality of the measure - see Figure 5.

At this point, we choose our measures. Almost everyone goes for high-value and easy-to-implement and then usually sticks around the high-value mark, moving up to the more difficult-to-implement measures.

From no measure to a list of measurements that we considered of value and relatively easy to implement in 2 hours. Those measures were connected (through the map) to users' needs. We could now baseline this before even starting to build the system.

Of course, if you're going to spend tens of millions of dollars, then obviously you're a significant person and have better things to do than waste two hours working on measures. Trust me, you don't. Spend the time.

Once you have measures connected from the system to user needs and hence the things that users value, you can then look to turn that into financial value. What is the value of speeding up assurance mechanisms? This is a lot easier than it sounds. The hard part is the bit you've just done - finding what to measure.

A metric should only exist because it helps you to answer a question that supports you in achieving a goal. When a group turns up with a system they have built but with no metrics, if often hints at a deeper problem - there is a lack of clarity on the goal of the system.

The mapping exercises in figures 3 & 4 are mostly about discovering what the goals of the system really are. You might, at first glance, think that a common platform is all about delivering common components but in this case, the goal is more about simplification and reduction of the assurance processes. Hence many of the metrics that bubbled up are connected to assurance. Common components are just an aid in achieving this goal.

Maps are a view on the landscape to which we can ask questions. In this case, the hidden question we are asking is "What is the goal of the system". Of course, you can't just ask that question or you'll get the stock answer to whatever story has been built around the system - "The goal is to provide common components" or "The goal is to speed up development". You have to investigate and this is what maps enable you to do in a collaborative fashion.

Maps, like views, like metrics, help you answer questions that support you in achieving your goals. If they didn't, there wouldn't be much point to them. My goal was to "Find the goal of their system" and enable them to "Find metrics that supported their goal". Maps helped.

In most organisations, I find a terribly lazy assumption that when we build a system that there exists someone who actually knows what the goals of the system are :- that person may have moved on, those goals may never have been explicitly stated and the true nature of the system may have been hidden. Experience teaches me that the actual goals for a system are only known in the minority of cases.

This post was about getting to some form of measurement - any measurement, as imperfect as that might be - for those of us mortals who aren't blessed to live in that world where people instinctively measure things. The steps are based upon the assumption that no single person has the answer, and it uses three simple questions.

Q1 : "Where can this system provide value"?

Q2 : "What could we measure?"

Q3 : "Is this measure worthwhile?"

Behind the scenes, you may also find that the maps help clarify what the goal of the system is. Don't start there though, start with the questions because you can often find the actual goal of the system is not the stated goal.