Monday, February 8, 2010

When is enough, enough? The art of counting jelly beans.

I was reading an article in National Geographic Blog about counting jelly beans that in a way boiled down what my general belief about metrics (see also another relevant post about metrics). Also, I have lately been involved in MESA’s metrics working group and all of this had me thinking…

The current practice in regards to metrics in operational environments especially in manufacturing is the typical engineering tactic - we need to break down the complexity of the system and figure out what metrics we need. Well one approach is to take a look at what other companies are doing – as in MESA paper we try to figure out if there are any best practices. But is this the right approach for any company or for you? I say no! Metrics are a way to represent patterns in the complex systems that enhance our understanding of its dynamics. We can look at one metric and say we are doing good or bad, which works well if you are a computer since we can take a decision based on this metric. Yet unlike computers, we humans are much better at using our knowledge and understanding to detect patterns and react to them. So if you give us a set of metrics that help us interpret what the system is doing then we do much better. That is exactly what the term "Actionable Intelligence" means.

There are a few important factors that I believe are important:

  • A few metrics used independently are never going to give you the true picture of what is going on.
  • Simply copying what other companies are doing will not work either, since metrics should be driven by what you are trying to achieve – and that changes with time.
  • The Accuracy of the metrics are not necessarily important. What! Yes, I just said it; accuracy is not that important. Let’s think about a jelly bean counting competition, you really don’t needs to know the exact number but be the one that is closest to the precise number, in order to win. Michael Schumacher (if you do not know, he was a 7 time Formula 1 GP winner) once said “… you don’t have to drive your fastest to win, but faster than the guy behind you”.
So why do we obsess about having so much data, detail and precision? Why are we always trying to quantify the unquantifiable, as in 6-Sigma. Maybe it is our upbringing as engineers, that constant quest for details. Really all we need is an indication when things will go wrong and when they are going well? This is of course not to say that sometimes we do have to be very precise as well as accurate, such as in engineering tasks, designs, etc. The art of it all is to know when we are precise enough.

In a blog post Stephen Few states “most poor decisions are caused by lack of understanding, not lack of data”. This is exactly my point, we spend so much time working on detail. For example, how much the ocean water level will rise over the next period of time. Some say a few meters some say 30 meters in the next 100 years. But the point here is that even in a moderate scenario calculation; Venice, New Orleans and the SF Bay Area delta, among other places, will be under water. So if you live there – guess what? You are in trouble, regardless of accuracy.

1 comment:

Anonymous said...

Very interesting ideas!