Just enough data

You may have seen awareness campaigns about the dangers of air or water pollution. Programmes to improve nutrition to curb obesity and diabetes. Vaccination camps to ensure that children are properly innoculated against debilitating conditions. How do governments decide where to deploy limited budgets so they have the greatest impact on population health? This is not something we think about often (or at all), but accurate and consistent data on disease and death rates are critical for policy makers to spend their energies and budgets in the most effective manner.

Epic Measures by Jeremy Smith is the story of Chris Murray’s perseverance in tabulating the world’s illness and mortality, in order to help save lives. I chanced upon it while working with Resolve to Save Lives on the open source hypertension management project Simple, where some of my colleagues recommended it.

As the book unfolded, the enormity of the challenge involved struck me. It is no easy task to attribute a death to a specific cause, particularly when there are multiple chronic and acute conditions involved. Even more so when you expect the numbers to neatly total up and inform policy decisions directed towards improving population health. Similarly, disability can have a grave impact on the quality of life: to what extent does a bad back affect your quality of life relative to poor vision?

As a Product Manager, I have seen a lot of recent focus on data-driven decision making, and to support it, data collection. A huge amount of effort is invested in instrumenting and collecting all available data, trying to improve the precision of these measurements, and finally to set up dashboards to analyze and vizualise what you have.

If you’re trying to follow the user journey of each of your users and “solve” all of their problems, you might as well join Chris Murray in his quest!

Instead, first define what you are trying to do, and then come up with an acceptable metric to measure it. Acknowledge that all metrics are approximations, and that they will fall short (as numbers always do), when they try to represent reality. Also ask yourself if a more precise measurement will really make a significant difference to outcomes. If not, move on. You can always revisit the metrics as your product and consumers evolve, but frequent churn with definitions, and parsing of greater quantities of data in more and more complex ways will invariably cost you more than it’s worth.

tl;dr Recognize that data collection and visualization is complex, and takes effort. More importantly, recognize that more data is not necessarily the answer.

Next
Next

Inbox zero