The interrelation between goals, metrics and biases

The first step in setting a goal comes down to a question that is as basic as it is sometimes difficult to answer: what do you want to achieve? The objective is set by first identifying what you want to achieve and why. This question sometimes has a complex answer. It may depend on the type of project we undertake or the parties involved. However complicated it may be to define it, it is vital to achieving an objective.  

With this thought, Fran Avila, Regional Director at Codurance Spain, began a meetup to discuss choosing the right metrics to evaluate the success of the objectives we establish.

Setting objectives

First, we must determine what the problem is. If we cannot state a problem, we will not be able to decide what steps to take.

A problem well posed is a problem half solved.

- Charles Kettering 

Objectives are sometimes stated in general terms such as: "I want to achieve a good quality platform for my users." But 'good quality' may have a different meaning for many people, so we must be able to define or break down the objective into clear and achievable goals.

Continuing with this example, one could elaborate:
Goal 1: reliability - the system does not suffer interruptions during use.
Goal 2: flexibility - that the system adapts to work peaks.
Goal 3: regulation - that data collection complies with regulations.
And so on, until the big goal is met: to provide a good quality platform for customers.

This way, we pinpoint where the problem lies and why it needs to be solved, and leads us to focus on three key aspects:

1. If something is essential, it is observable.
2. If it is observable, it is quantifiable (or there is a range of possible values).
3. If it falls within a range, it is measurable.

What is a metric and what is it used for?

A quantitatively expressed reduction of uncertainty based on one or more observations.

- Douglas W. Hubbard

Metrics should reduce our uncertainty about what we measure, i.e., they should provide valuable information that allows us to take actionable steps. So says Fran: "If you are not going to make any decisions with the metrics you have, then don't measure." First, you must know where the decisions you want to make are oriented and what you are interested in learning. Then, measure to obtain the data that will help you choose the best possible option. 

A metric does not have to be exact but indicative enough to allow us to make a decision. For example, I will buy a computer screen that must fit on a table, so I need to know that the table is larger than the screen. But I don't need to measure the table in millimeters; it is enough to see that it is more extensive and will be useful for my purpose. The cost of the metric should never be greater than the cost of the decision you are going to make.

What is the value of a metric? VI = VSbi - VSai. The value of information is equal to the value of the situation before having that information minus the value of the situation after having that information. If it does not change, the value of the information is zero. And, in a perfect case, it would eliminate the value of the situation before. Any metric that does not contribute to eliminating the uncertainty you had before is useless.

Biases to avoid

Fran presented several traps we can fall into when choosing what data to look for or measure to make decisions:

Confirmation bias - Seek information that confirms the decision you wanted to make or have already made. Taking data that ensures what you want to hear. To overcome it, try to get information that disproves your theory and gives you another point of view to make a complete decision.

Procedural bias - When the procedure conditions the result of the information we will obtain. For example, we will have different results in surveys depending on how the questions are asked or which people answer them. To avoid this, it is necessary to allow people from different contexts and with different opinions to intervene.

McNamara's Fallacy - The fact of not being able to measure certain data can cause us to fall into an oversimplification that generates biased and erroneous conclusions. Thus, my picture of reality focuses only on what I can measure:

1. Measure what is easily measurable.
2. Ignoring what is not easily measured or giving an arbitrary quantitative value.
3. Assume that what is not easily measured is not essential.
4. Asserting that what cannot be easily measured does not exist.

Summary

Fran explained that the way they use metrics that bring value to their projects could be outlined as follows:

• Why am I doing this? - Objective, what do I want to achieve?
• How do I do it? - What are we going to measure?
• What do you know so far? - What information do I have?
• How are we going to use the information? - How am I going to act on that information? How am I going to calibrate the different elements?
• What decision do we make? - Execute the action to achieve the why.

We thank Fran for this delivery and invite you to visit our Insights page to find more content to inspire you.