Sophistication of method is used to torture data to reveal conclusions that do not obviously follow from them, but which fit either the researchers’ preconceptions or the sponsor’s policy objectives, or both.
Like me, you probably disapprove of drinking and driving and dislike it when people smoke. So, like me, you may have been impressed by reports that reducing Britain’s limit on blood alcohol content for drivers would save several hundred lives a year and that the ban on smoking in public places has significantly reduced the incidence of heart attacks among non-smokers. But then I looked at the evidence for these claims.
Most analysis of the association between drinking and road accidents is based on investigation of blood alcohol levels in victims of fatal road accidents. One conclusion jumps out from these data. Most people who have been drinking and are killed on the road are not close to the limit. They are drunk. Two-thirds of victims of fatal road accidents who were over the legal limit had blood alcohol levels more than twice the legal limit. Many of them had consumed much more.
The claim that reducing the legal limit would save many lives is entirely based on the belief that such a measure would have a big effect on the behaviour of people who are already well above the legal limit. The principal evidence for this claim is a study undertaken in South Australia. But the incidence of drunk driving in South Australia was then 10 times the current figure in the UK. Australia used to have a culture of the drive-in bottle shop, which would fill your car boot with cans of beer. A raft of policies have sensibly addressed the issue. The social context is so different that Australian experience is almost irrelevant.
Surveys show that most people in Britain underestimate how much they can drink and yet drive legally, and it seems unlikely that most people breaking the law today do not know they are doing so. Britain’s drink-driving problem is that a minority of people, mostly but not exclusively young men, sometimes drink far too much, and some of them drive. They cause accidents and injuries, mostly but not exclusively to themselves. It is not obvious, or even likely, that the best method of dealing with this issue is to reduce the legal limit they are already disregarding.
What of the claim that the smoking ban reduced the incidence of what lay people call heart attacks in England by 2.4 per cent? This follows the more extravagant claim that the similar ban in Scotland had an effect of 17 per cent. The evidence for the former proposal is weak, and the latter claim is implausible. The incidence of heart attacks is falling steadily in both countries by around 3 to 4 per cent per year, with fluctuations but no obvious break in this trend. Better health education and reduction in smoking are part of the explanation.
In the year after the English ban, the fall in reported cases was 4.3 per cent. The effect claimed for the ban is the difference between the actual figure and that predicted by a model constructed by the researchers. In Scotland there was an above average fall in heart attacks – though nearer 7 per cent than 17 per cent – immediately after the ban, but this seems to have been a freak, since it was reversed the following year. The researchers derived their conclusions from a specially compiled dataset constructed over a limited period.
The studies I have cited are carefully referenced and use advanced statistical techniques. But sophistication of method is used to torture data to reveal conclusions that do not obviously follow from them, but which fit either the researchers’ preconceptions or the sponsor’s policy objectives, or both.
Bad arguments do not necessarily invalidate the causes in which they are deployed. People should not drink and drive. Smoking is unpleasant and perhaps harmful to non-smokers. But these observations do not justify blurring the distinction between genuine scientific analysis and propaganda disguised as science. Policy should follow evidence, not evidence policy. It is time to reassert the principle that research must pursue the truth wherever it leads: the principle on which the social and economic progress of the past few centuries has depended.
