Do not discount what you cannot measure


Bogus quantification attempts to compress complex problems and analyses into single observations.

“When you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.” This observation scores high on the list of silly remarks made by clever men, to which its author, Lord Kelvin, made several sterling contributions. The knowledge possessed by Plato and Shakespeare, Austen and Darwin was neither meagre nor unsatisfactory.

Bogus quantification attempts to compress complex problems and analyses into single observations. The annual report of the United Nations Human Development Programme is an entirely admirable endeavour. The agency focuses on three broad dimensions of development – health, education and material standard of living – and reports on progress towards these goals among UN members.

Then the curse of Kelvin strikes. Health is measured easily enough. Take life expectancy at birth, subtract 25 years, then divide by 60. This score has a one-third weight in the final total. Educational development is measured as a combination of attainment – adult literacy – and opportunity – enrolment. Material standard of living is measured by gross domestic product per head at purchasing power parity. But differences in GDP are much larger than differences in life expectancy, so you prevent this measure from swamping the whole calculation by using the logarithm of GDP rather than the level. Finally, you give equal weight to each of the three components and come up with your ratings. Iceland, ironically, comes top, and Sierra Leone bottom.

If you must undertake an exercise of this kind, then what the UNDP does is sensible enough. But suppose I thought that more weight should be given to health and less to per capita income? Or that a measure of human development should include freedom of speech, or democracy, or religious tolerance, or religious observance? The problem is not just that reasonable people would have different views on these things, but that there are no objective criteria by which such disagreement could ever be conducted, far less resolved.

If Iceland did not come near the top of the ranking and Sierra Leone near the bottom, we would not change our assessments of Iceland or Sierra Leone: we would change the index. If we were to revise our view of Iceland, it would not be because we were told that its HDI score had fallen to 0.96: it would be because we were told it was a poorer country than we had previously thought. The index of human development is not telling us anything we have not already told the index of human development.

The French government has just published a report, supervised by a distinguished panel of economic experts, on the measurement of national income. President Nicolas Sarkozy is miffed by repeated criticism of French economic performance, since the quality of life in France is plainly very high. You can find evidence of this, for example, in the number of foreigners who visit the country every year. Much of the difference between France and the US in GDP per head is explained by the French penchant for longer lunches, longer holidays and earlier retirement.

It is not difficult to invent an index that would represent these facts. Such measures do not, however, add anything to the knowledge contained in the preceding paragraph. If anything, they reduce it. Just as the statement “Iceland has an HDI score of 0.99” tells us less, not more, than the sentence “Icelandic people are, by global standards, rich, healthy and well educated”.

In Peter Weir’s film Dead Poets Society, Robin Williams portrays a charismatic teacher obliged to teach from a text by J. Evans Pritchard. Mr Pritchard explains that “if a poem’s score for perfection is plotted along the horizontal of a graph, and its importance is plotted on the vertical, then calculating the total area of the poem yields the measure of its greatness”. Williams tells his pupils to tear these pages from the book, and goes on to inspire them with a genuine love of literature. We should approach bogus quantification in the same way.

Print Friendly, PDF & Email