John Kay - The wisest choices depend on instinct and careful analysis

The wisest choices depend on instinct and careful analysis

MoneyballMichael Lewis’s 2003 book on the science of picking baseball teams, was perhaps written to distract himself from his usual work of attacking the financial services industry. Even after downloading the rules of baseball, I still could not fully understand what was going on. But I caught the drift: sabermetrics, the statistical analysis of the records of players, proved a better guide than the accumulated wisdom of experienced coaches.

Another lesson, important for business strategy, was the brevity of the benefits gained by the Oakland A’s, Lewis’s sporting heroes. If the only source of competitive advantage is better quantitative analysis – whether in baseball or quant strategies in the financial sector – such an advantage can be rapidly and accurately imitated.

At the same time, another genre of books proclaims the virtues of instinctive decision-making. Malcolm Gladwell’s Blink (2005) begins with accounts of how experts could identify the Getty kouros – a statue of naked youth purported to be of ancient Greek provenance and purchased in 1985 for $9m – as fake immediately, even though it had supposedly been authenticated through extended scientific tests.

Gary Klein, a cognitive psychologist has for many years monitored the capabilities of experienced practical decision makers – firefighters, nurses and military personnel – who make immediate judgments that are vindicated by the more elaborate assessments possible only with hindsight.

Of course, there is no real inconsistency between the two propositions. The experienced coaches disparaged by sabermetrics enthusiasts were right to believe they knew a lot about spotting baseball talent; they just did not know as much as they thought they did. The art experts and firefighters who made instantaneous, but accurate, judgments were not hearing voices in the air. But no expert can compete with chemical analysis and carbon dating in assessing the age of a work of art.

There are two ways of reconciling expertise with analysis. One takes the worst of both worlds, combining the overconfidence of experience with the naive ignorance of the quant. The resulting bogus rationality seeks to objectivise expertise by fitting it into a template.

It is exemplified in the processes by which interviewers for jobs, and managers who make personnel assessments, are required to complete checklists explaining how they reached their conclusion using prescribed criteria.

In reality, the interviewer or manager forms a judgment about an individual and completes the form to ensure consistency with the decision. This is a waste of time but no worse; more serious damage is done when an initial evaluation is influenced not just by the judgment of the appraiser but by the ease with which the evaluation can be defended by reference to the imposed criteria.

Similar risks arise when models are developed based on elaborate spreadsheets, and the modellers then fill empty cells by guessing what the missing numbers should be. They will, as a rule, seek expert advice on making up the numbers; but since most are known at best only within wide ranges they can readily be selected to yield whatever answer the problem-setters wanted – one consistent with the conclusion the decision maker has already reached. And so we have a large consultancy business of transport modellers, environmental experts, risk managers and impact assessment modellers, the front line of an army that has turned evidence-based policy into policy-based evidence.

These procedures cloak often casual instinctive assessments in an appearance of objective justification. Instead of the worst of both worlds we should seek the best, combining the value of experience or judgment with the data-processing capabilities of information technology.

We will never succeed in evaluating works of art, choosing candidates, managing risk, without the skills that can be acquired only through experience; but that experience can always be enhanced by the power of data analysis and the implementation of scientific techniques.

True expertise can never provide a full objective justification of the judgments that emerge; to believe that it could is to misunderstand the nature of true expertise.

 

[ RSS feed for comments on this post. ]