The tube map

A few years ago some friends invited me to dinner at their house in Hyde Park gardens.  At the time I was living in Oxford and didn’t know London very well, so I asked them what the nearest tube station was and they told me it was Lancaster Gate. On the day I got the train from Oxford to Paddington station and, on arrival, consulted the tube map. A quick look told me that I should go two stops on the circle line, change at Notting Hill Gate and then go two stops on the central line. This is what I did and it took me around twenty minutes. In my ignorance I regarded my journey as successful and indeed repeated it for a number of years without much consideration. Had I consulted a surface map instead of the tube map, however, I would have immediately realised my mistake. What I was actually doing was travelling a mile and a half west only to then retrace my journey by travelling back, a mile and a half east. From Paddington station you can actually walk to Hyde Park Gardens in under five minutes.

You might think that what I am implying is that the tube map is a bad model, as it is doesn’t accurately represent the London underground. Actually, the opposite is true. The London tube map is an inspired piece of graphic design. It was constructed in the 1930’s by an electrical engineer called Harry Beck and was regarded as such a brilliant innovation that it was adopted in London and copied around the world. Tens of millions of people have reached their destinations using that map. It is a brilliant model; it’s just the wrong model to use if you want to get from Paddington station to Hyde Park Gardens.

I know that now, because I live in London and have, over a number of years, acquired a general knowledge of the city. This enables me to make judgments about when I need to use the tube map, and when I don’t. This is true for most Londoners. The point that I am making is not to say that models are useless, models are indispensable, but judgment and experience are required in order to select the correct model for a particular circumstance. Often there won’t even be one correct model; there will be several different models with more or less relevance. This is the antithesis of an increasingly prevalent, approach to decision making that I will term ‘bogus rationality’. This is where, in a world in which we cannot know everything, we simply make it up and then make our decisions on that basis.

Franklin’s gambit

A discussion of the history of rational decision making might begin with Benjamin Franklin, who, in 1772, wrote a letter to, the English chemist, Joseph Priestley on that subject. He said,

“Divide half a sheet of paper by a line into two columns, writing over the one Pro and over the other Con…  When I have got them all together in one view, I endeavour to estimate the respective weights… I have found great advantage for this kind of equation, in what may be called moral or prudential algebra.”^[i]

That is the mechanism of what we believe to be rational decision making. I suspect, however, that Franklin had his tongue in his cheek, given that he also said, “So convenient a thing it is to be a reasonable creature, since it enables one to find or make a reason for everything one has a mind to do.”^[ii] This remains an insightful and relevant observation of human nature, given that a great deal of what we currently consider to be rational decision making, or evidence-based policy, has precisely that character.

Bogus models

In fact it is a wide-spread characteristic of modern life, but I would like to focus on a particular area, in which it has become, not just absurd, but detrimental. It is a phenomenon I describe as ‘bogus modelling’. Three of the best, or worst, examples of this are firstly, WebTag; a framework used for appraising transport projects, most recently the high speed rail link. It’s Scottish equivalent, STag, was used to justify the Edinburgh tram project, whose appraisal was executed as poorly as the project itself. Second, are public sector comparator models, which are used to assess potential PFI projects, and finally; value at risk modelling (VAR), which is used for risk management in banks. It would be an understatement to suggest that these models have not been wholly successful. 

Bank risk management, based on VAR, led to the most comprehensive collapse of the banking sector that we have seen in a century. PFI projects, despite meeting the requirements of these value for money comparators, are set to cost tax payers billions in funding costs. The UK is admired around the world for the quality of its transport modelling, but is certainly not admired around the world for the quality of its transport infrastructure. Despite this, all of these models are not only still in routine use, but they are more or less compulsory. In order to obtain funding for a transport project or a PFI scheme, you have to undertake an appraisal or assessment of this kind. The Basel rules, that could not prevent a banking crisis the first time round, are being multiplied in the misguided belief that they will prevent the next one. The real world failures are significant but have not, as yet, lessened the use of these models.

All of these bogus models can be identified by a common structure. They begin by considering how you would make a particular decision if you had complete and perfect knowledge of the world, now and in the future.  Having done that, they then look at cost-benefit ratios for transport projects, costs of PFI projects compared to an alternative, and for VAR modelling, the loss you would make on a bad day and the associated probability. As a process, it is incredibly data intensive, for obvious reasons. The problem is, very little of the relevant data is actually known. The solution? Make it all up.

To get an idea of what this invented data actually includes, we can use some examples from transport modelling. In the world of WebTag, an individual’s time is given a monetary value depending on the mode of transport by which they travel. There are thirteen different categories. So, as a taxi passenger, your time is worth £44.69 per hour, whereas the taxi driver’s time is considered much less valuable, at £9.77 per hour (2002 prices)^[iii]. Absurd as it might seem to put such a precise value on time in the present, the model demands that this level of precision continues into the future. Growth projections make it possible to predict how valuable time will be in 2052, to the penny. If you would also like know how many people will be travelling in a car in 2036, WebTag will provide an answer. This ensures that every cell in the spreadsheet can be filled and that at the end of the process, numbers will be provided.      Most objective observers would conclude that this exercise has gotten a little out of hand.

What is wrong with these approaches?

A fundamental problem is that, since most of the numbers are invented, they can usually be selected to deliver the desired result. In the case of the Edinburgh Tram, it requires five minutes on the back of an envelope to demonstrate that it is a wasteful project. However, it actually took a firm of consultants several years and several million pounds to conclude that it was, in fact, an excellent project. It is important to note that almost all of these types of appraisals are conducted by a small number of firms, for whom this is their sole business. As with all businesses, their success depends on them delivering what their customers want.

The way in which lack of knowledge about the future is addressed is unrealistic by most people’s standards. It is assumed to be essentially similar to the present, except for certain mechanical projections of demand, income etc. I have no idea how you will be getting about in 2052, it might be by personal flying platform or by horse and cart, and nor would I attempt to guess. A model, however, expects that you will still be travelling in the exact same way; the only thing that will have changed is the value of your time.

There is a critical question in all of this, which is; what is the terminal value of a project? Consider the very first cost-benefit analysis of a UK transport project. It was conducted in the 1960’s to assess the potential value of the Victoria Line. The assessment period covered fifty years, so it was assumed that there would be no benefits beyond 2011 at all.^[iv] In the 1960’s the Victoria Line cost £90m to build; today it would probably be closer to £10bn. Even discounting that figure back fifty years, would still indicate that it is more valuable today than it as ever been. The same is true for many of the tube lines. But for some transport projects fifty years would be far too long a period of assessment, relative to their realistic life expectancy. It is a critical issue that is essentially ignored.

The prescription of a universal template obstructs the proper use of judgment and experience. But more than that, the cost of these exercises actually gets in the way of intelligent public debate. In the case of the high speed train project, a group of local authorities opposed to the scheme, commissioned their own assessment using a different firm of transport consultants, but the same standard WebTag model. It is probably not a coincidence that their study came to the opposite conclusion. These exercises have cost millions of pounds and yet the debate surrounding the project is stagnant, narrowly focused and bogged down in detail.

Why do we engage in these exercises?

       Firstly, there is a misconceived search for objectivity. Governments are under pressure to find an objective, analytic process for decision making via a mechanism that can be universally defended. Secondly, rationality and quantification are being confused. Lord Kelvin famously said, that if “you cannot measure it… your knowledge is of a meagre and unsatisfactory kind.”^[v] This remark was engraved on the Social Sciences building at Chicago University. Frank Knight, a Chicago School economist who walked past it on a daily basis, reportedly said in response, “…and if you can’t measure it, measure it anyway.”^[vi] That is what is being done here.

There are also significant entry barriers that have been constructed to ensure the continuing employment of people associated with this process, primarily; consultants, civil servants and risk managers. The group of firms that build these models is small, and the only realistic way to enter the industry is by hiring from within them. So, not only does this act as an entry barrier but also as a business opportunity. The vested interest is obvious.

What should we do instead?

I have spent much of my career building models of one kind or another and I consider them to be a necessary part of life. But a model can never be a true representation of the world and nor should we expect it to be. A good model is a purpose- specific simplification of the world, like the tube map. Its usefulness is in its relevance to the problem at hand, not its comprehensiveness. The real purpose of a model is to identify the key factors that ought to be influencing an assessment. For example, in the case of high speed rail, a critical element is how valuable it would be to passengers, to reduce their journey time by half an hour. Having framed the issue in this way, it then becomes an exercise in gathering evidence. Calculations involving the value of time projected indefinitely into the future are a possible reality check, they are not, however, a sound basis for decision making. What these exercises typically require, in order to be successful, is further detailed research, political judgment or experience of similar projects.

Quantification can be a helpful analytical tool, but if it is too precise it becomes meaningless. For example, if you can only know enough about a project to say that it will cost more than £1m but less than £1bn, that is considerably more useful than giving a bogus estimate of £43bn, based on fabricated numbers. This kind of thinking allows for greater flexibility and enables a more piecemeal assessment of individual components rather than black box analysis. In relation to the high speed train we might want to ask questions like; is Euston the best place for it to terminate? Do we really need the expensive tunnel? Etc.

Above all we should abandon completely the concept of a standard template that can be applied to every problem with similar characteristics. That is precisely the mistake I made in using the tube map to get from Paddington station to Hyde Park Gardens. The reason that these templates, such as WebTag, remain in use is largely due to the commercial and professional interests of the people involved.

I am strongly in favour of quantification, modelling and evidence based policy. What I am against is bogus modelling that in my view discredits all of these things. These are all tools that are essential for good policy making but the skill of a policy analyst is in identifying the models and evidence that are relevant. We must not confuse a model with the world that is being used to describe.  It was a polish philosopher, Alfred Korzybski, who put it best when he said, ‘the map is not the territory’^[vii]. 

^[i] Franklin, B., (1956), Mr. Franklin: A Selection from His Personal Letters, New Haven, CT, Yale University Press.

^[ii] Franklin, B., (1889), The Autobiography of Benjamin Franklin, London, G.P Putnam’s Sons.

^[iii] Department For Transport, (2011) http://www.dft.gov.uk/webtag/documents/expert/unit3.5.6.php#012

^[iv] C. D. Foster and M. E. Beesley, (1963), “Estimating the Social Benefit of Constructing an Underground Railway in London”, Journal of the Royal Statistical Society, Vol. 126, No. 1 (1963), p46-93.

^[v]  Kelvin, W. T., (1883), “Electrical Units of Measurement” , Popular Lectures, Vol. I, p73.

^[vi] Merton, R. K., D. L. Sills, and S. M. Stigler, (1984), “The Kelvin Dictum and Social Science: An Excursion into the History of an Idea.” Journal of the History of the Behavioral Sciences, Vol. 20,  p319-331.

^[vii] Korzybski, Alfred, (1931), ‘A non-Aristotelian system and its necessity for rigour in mathematics and physics’, a paper presented before the American Mathematical Society at the New Orleans, Louisiana meeting of the American Association for the Advancement of Science, 28 December. Reprinted in Science and Sanity, (1993), p747-61.