The maths may be simple but intuition is more use

Probability theory works well for a limited class of problems, but the real world is much more open-ended and there is usually fundamental uncertainty about both the nature of the outcomes and the process that gives rise to them.

Our ancestors gambled around camp-fires on the savannahs thousands of years ago. The ancient Greeks had games of chance, embryonic financial markets and some of the finest mathematicians who have ever lived. But they did not discover the rather simple arithmetic of probability and the methods we use today are no more than 300 years old.

Even today, many people struggle to learn probability theory and experts make mistakes in applying it. This can have disastrous results, as in the imprisonment of innocent mothers whose children had been victims of unlikely, but not that unlikely, sequences of accidental death. Stephen Jay Gould is only one of many science writers to have observed that human minds are not well adapted to dealing with issues of probability.

The two-box problem, which I described last week, is one in which a knowledge of probabilities seems to be a hindrance rather than a help. The puzzle offers you the choice of two boxes, one containing more money than the other. Once you have made a decision, you are shown what is in your preferred box. Do you stick with your original choice, or switch?

You have so little information that there seems no rational basis for decision. Yet the dilemma is real. In our personal lives, in business and finance, in employment and in house-hunting, we repeatedly encounter two-box problems – should we stick with what we know or switch to something about which we know much less?

The extraordinary feature of the two-box problem is that there is a strategy that seems better than always switching or always sticking, one that beats random choice even in a situation of almost total ignorance.

Before the game starts, focus on a sum of money. It does not matter what the amount is – say, £100. The “threshold strategy” is to switch if the box you choose contains less than £100 and to stick if it contains more. The threshold strategy gives you a better-than-even chance of getting the larger sum. It does so for any value of the threshold you choose.

If both boxes have less than £100, or more than £100, then the probability that you get the larger sum from your random choice remains one-half. But if one box has less than £100 and the other has more than £100, adopting the threshold strategy makes sure you get the larger sum. Since there is at least a possibility that the amounts in the boxes lie in this range, the threshold strategy must increase your chance of winning.

So how should you set the threshold? There are two criteria. How likely is it that the box with the smaller amount of money has an amount below, but not too much below, the figure you set? There may not be much benefit in setting a very high or very low threshold. So if you have some idea, however vague, about the range of possible contents, you can tweak the threshold strategy to your advantage. Second, choose a threshold in the range of sums of money that would make a real difference to you: if £20,000 would not transform your life but £50,000 would, then go for £50,000. The range from £25,000 to £50,000 is the range in which the benefit from switching might be greatest.

The curious feature of the threshold strategy is that the maths is surprising but the intuition is familiar. In real-life search problems the principle of “be realistic but look for something that will make a difference” represents typical behaviour. In the version of the two-box problem I described last week, where you had the additional information that one box contained twice as much money as the other, there seemed always to be an argument for switching; the potential gain is always twice the seemingly equally likely potential loss. But this conclusion is wrong. Once more, our intuition runs ahead of our mathematical understanding.

Probability theory works well for a limited class of – mostly artificial – problems, such as coin tossing and roulette. But the real world is much more open-ended and there is usually fundamental uncertainty about both the nature of the outcomes and the process that gives rise to them. Perhaps the reason we do not use probability theory much is that it is not all that useful.