Trying to assess lecturers, doctors, and teachers is difficult. But that doesn’t mean it cannot, or should not, be done.
In the last ten years, research assessment has been introduced in British universities. Academics have to report on the quantity and quality of their publications, and the quality of each department is graded and published.
Many of my Oxford colleagues, in common with dons at other universities, bitterly resent these innovations. They argue that the need to fill in forms to respond to these inquiries distracts them from their real work – of teaching and research. They claim that the assessment of research quality is inevitably a subjective business, and can only really be done by the researcher himself. And they say that since they are already doing as good a job as could possibly be done, the whole activity is essentially pointless.
There is a small amount of truth in these criticisms. It is certainly true that the research assessment exercise is bureaucratic and emphasises those aspects of research which can be measured – like the number of publications – at the expense of those which are less quantifiable – whether the research made any significant contribution to the sum of human knowledge. It is also true that it is often difficult to judge the significance of research. People outside, or even inside, the field may not understand, and the importance of a discovery may not be apparent at the time.
I am sure that the merchants of Pisa, their feet firmly on the ground, thought Galileo was wasting his time and their money when he climbed the leaning tower to drop two balls to the ground. However the theory of gravitation explained why it was that their feet were firmly on the ground: and eventually enabled them to leave the ground by making possible the development of aviation and rocket science.
But you have heard these arguments before. You heard them from monopoly utilities, who claimed that any proposal to introduce competition into telecommunications or aviation or gas supply was an insulting suggestion that they weren’t doing a good job. You heard them from schools, which claimed that league tables comparing their performance diminished the professional status of education. You heard them from doctors and nurses, who said that time spent filling in forms would be better spent with their patients. You heard them from financial services companies which claimed that regulation was a bureaucratic interference which got in the way of their real job of selling pensions to as many people as possible.
And you hear it today from business people who complain about the burgeoning industry of corporate governance. The need to provide information to people outside the business consumes time and resources which would be better spent beating international competition. What right, or capacity, do people outside the business have to sit in judgement anyway? And, since managers are already performing honestly, ethically and effectively, the whole activity is offensive and unnecessary
And yet the introduction of greater external accountability did seem to affect the behaviour of my university colleagues. Star researchers are now prized, and poached : duds are more rigorously reviewed: departments set their plans by reference to research ratings. Indeed, it was necessary for research quality assessment to be followed by teaching quality assessment, otherwise the emphasis on research within universities might have been even more excessive than before. And publishing league tables also made schools and hospitals try harder, independent procedures made the police more careful about complaints, regulatory interventions led financial service companies to tighten up on their compliance procedures.
I suppose that clerics are kept up to the mark by their ultimate accountability to God, but the affairs of Lincoln Cathedral suggest that a little bit of earthly supervision by the Church of England could do no harm. Certainly I doubt that either senior managers or academics are the only exceptions to the general rule that effective external accountability for your performance tends to improve it. And one of the interesting lessons from these assessments is that they work even if no action follows from them – as with the school league tables. Shame or kudos is itself an effective spur. Still, it is better if action does follow, and the university system, which redistributes research funds to follow ratings, serves well in this regard.
So arguments that say we – teachers, health professionals, managers – are too busy and too admirable to be externally assessed should be dismissed for the self-serving cant they are. But it is right to be wary of what Hampel calls box-ticking. This is the kind of assessment encouraged by standards such as ISO 7000, essentially related to process rather than substance. It assesses lectures by reference to whether they defined and achieved their objectives, not on whether the objectives were of any relevance. But even so, do not dismiss box-ticking entirely. If the lecture had no objectives, it is unlikely that any objectives were achieved.
And it is also necessary to make the criteria relevant – you cannot assess the value of Galileo’s discovery by the length of the publication in which he described it. Do not allow doctors to divert attention from clinical outcome – whether the patent died or got better – to the side issue of how long the patient had to wait before one or the other happened. And beware Goodhart’s law – whatever you try to control or measure changes its meaning. If you measure waiting lists by the numbers who have waited more than a year, you will find that a lot of people wait for eleven months.
But most of all, never believe that people will objectively assess themselves. The documents that we write to assess our own courses are as vacuous as the self-congratulatory platitudes which fill the corporation’s annual report. The greatest weakness of corporate accountability is that both groups of monitors – auditors and non-executive directors – are effectively appointed by those they monitor, which does not encourage rigour in selection or scrutiny. Still, that is because corporate accountability has been going longest and managers have learned how to capture the assessment system. I expect that dons, teachers and doctors will learn to do the same.
