There are some people that if they don’t know, you can’t tell ‘em.

– Louis Armstrong

My friend Ian Musgrave, research pharmacologist at the University of Adelaide and Panda’s Thumb regular, sent me a paper with the apparent desire of assassinating me by hypertensive crisis. The paper is called “The role of evidence in alternative medicine: Contrasting biomedical and anthropological approaches” by Dr Christine Ann Barry (full PDF file here). It appeared in the journal Social Science and Medicine around Christmas time.

What Barry means by “contrasting biomedical and anthropological approaches” is actually a list of reasons why alternative practitioners don’t need to worry about evidence. Here is a sample from her abstract:

It is proposed that the promotion of differently constructed modes of evidence can be used to legitimise alternative medicine by widening the definition of what works in therapy, and offering a critique of what people feel is lacking from much of orthodox medical care.

Barry is suggesting that the standards of biomedical research be relaxed for alternative medicine. Actually, this isn’t saying anything new. The standards for alternative medicine are pretty much non-existent as things stand — they’re almost entirely woven from testimonials and received wisdom. What Barry is proposing is the National Health Service in the UK devote more resources to alternative medicine despite the lack of evidence for its effectiveness.

Thus anthropological evidence can be used to open a debate about what one should be measuring as evidence of alternative medicine efficacy, and whether one should be measuring it at all.

Yes, you read it correctly. Barry is questioning whether the effectiveness of alternative medicine should even be measured before the NHS drains an ocean of gold into it.

Barry’s main target is the randomised control trial.

I do not wish to discredit the notion of the RCT. In its purest ideological form, the concept of offering patients only therapeutic interventions that have been proven to work is unquestionably sensible and morally correct. Where the problems arise is the imperfection of the RCT tool as an arbiter of what works, because it measures the wrong things or the wrong populations. The real world clinical context is different to the trial laboratory. The RCT can sometimes become a victim of hubris. Just being the ‘gold standard’ is not enough—it is still an imperfect tool. Even the most elegantly designed trial with statistically beneficial sample sizes and clever protocols for blinding usually measure only a subset of symptoms and therapeutic effects (those that are short-term and easiest to measure). The production of scientific evidence is a social as well as a scientific process. There is no such thing as The Evidence, just competing bodies of evidence.

Or, to put it another way, randomised control trials (RCTs) are fine for biomedicine but shouldn’t apply to alternative therapies.

Despite claiming to respect RCTs and evidence, Barry’s argument is condemnatory.

Alternative medicine in the past has shown little interest in producing RCT evidence. Its proponents are less embedded within a science-based epistemology, there is no money available, and there has been an awareness of the limitations of such methodology for studying complex individualised treatments (Long, Mercer, & Hughes, 2000). However, increasing integration requires alternative therapists to start to play the ‘evidence’ game.

Collecting evidence is a “game” which alternative practitioners rarely pursue because they are “less embedded within a science-based epistemology.” To paraphrase a Bush aide, alternative practitioners work outside “the reality-based community” and have no need for “solutions [that] emerge from your judicious study of discernible reality.”

Barry’s paper is open to many criticisms, but I leave it to interested readers to draw their own conclusions. I am more interested using Barry’s paper as a spur to describing the importance of randomised control trials, the so-called “gold standard” of medical evidence, why they matter, and why they cannot be discarded just because they don’t suit the interests of alternative therapists and the deconstructive anthropologists who support them.

A brief history of clinical trials

Randomised control trials, despite the implications of Barry, are not a traditional part of biomedical research. The first control trial was performed by James Lind in 1747, the first randomised control trial took place in 1948, and the first trial to keep the principal investigators at arm’s length from the randomisation process took place in 1967. The evolution of the RCT had nothing to do with politicisation of medical care. Each step took place because scientists recognised flaws in earlier trials and developed techniques to make their findings more trustworthy.

If we search history, the first reported clinical trial was an accidental study reported in the Book of Daniel, Chapter 1: 1-16. Around the year 600 BCE, King Nebuchadnezzar II of Babylon conquered Jerusalem and took captive the Israelites. To make them suitable as courtiers, he ordered the young Israelite noblemen to partake of the royal Babylonian diet of meat and wine, but this went against Jewish custom and “Daniel determined not to contaminate himself by touching the food and wine assigned to him by the king, and he begged the master of the eunuchs not to make him do so.” The eunuch was naturally reluctant to disobey Nebuchadnezzar’s order, especially if Daniel became sickly and “dejected”, but Daniel was a persuasive man. He said,

Submit us to this test for ten days. Give us only vegetables to eat and water to drink; then compare our looks with those of the young men who have lived on the food assigned by the king and be guided in your treatment of us by what you see.

After ten days, Daniel and his friends “looked healthier and were better nourished than all the young men who had lived on the food assigned them by the king. So the guard took away the assignment of food and the wine they were to drink and gave them only the vegetables.”

This was a fortunate observation based on a clash of religious convictions rather than a deliberate attempt to learn, but it was still a control trial. Attempts to understand the universe go back as far as human history, but for the most part it was Natural Philosophy rather than Science that led the way. The natural philosophers observed the world and drew inferences from their observations, from received wisdom, and from the logical rigour of mathematics. It was not until the scientific revolution that scholars learned that there was an even more powerful tool: the matching of observations to theoretical predictions by planned interrogation of the natural world. The short word for this is experiment!

From Daniel’s Biblical trial, it took another 2000 years until natural philosophers worked out that they could plan to test the world and apply their tests with reproducible methodical strategies. It took biomedical science another 250 years to catch up with the physical sciences. This was partly due to the incredible complexity of organisms and ecosystems, but it is also partly due to the historical conservatism of medical authorities.

One of the difficulties in deciding if a treatment works is that a person’s health changes over time. We have immune systems that fight infections. Cuts and broken bones heal. Grief abates. It has been very difficult to wean the general public off its abiding faith in the power of antibiotics to cure viral infections (viruses are completely immune to antibiotics). Most viral infections will resolve without any intervention at all in five to seven days and most people don’t bother to see a doctor until they have been unwell for two or three days. And yet, I have been told many times by faithful patients that “every time I have antibiotics, I get better in a day or two.” It is very difficult to convince many of them that what they experienced was the natural course of the illness rather than the curative effect of treatment. One patient of mine with an ordinary everyday cough jumped out of his chair when I suggested letting the illness take its course. He had suffered through a genuine bacterial pneumonia as a young adult and, on being advised not to use antibiotics, he stalked up and down the consulting room, wringing his hands and shouting, “But I will die! I will DIE!”

These patients are not fools: it is an inescapable human behaviour to see causes in coincidences, and it affects scientists just as much as lay-people. What scientists do that separates them from most of humanity is that they use experiments to test their beliefs. The classic experimental technique, in physics as well as in medicine, is to use a control as a baseline for comparison. There is historical doubt about whether Galileo ever actually performed his famous experiment of dropping weights from the Tower of Pisa. Whether real or myth, the story is illustrative of control trials. By dropping two weights at once, Galileo was able to show that the rate of falling was not affected by confounding effects such as wind speed, sunlight, or rain, as these factors ought to be the identical at the same time and place. Any differences in the rates of falling then had to be due to fundamental physical properties of the objects rather than different conditions. Galileo could have described his findings in modern parlance as showing no difference between the control weight and the heavier weight.

The first planned control trial in medicine was James Lind’s 1747 experimental “proof” that oranges and lemons cured scurvy aboard the HMS Salisbury. Lind chose a number of scurvy-afflicted sailors on board his ship and administered different treatments to them. The effect of citrus fruits was dramatic: one of the men with scurvy recovered enough to return to active duty within six days. The other treatments failed to show any improvement. The British Navy (eventually) put this finding to great use, and Lind played no small part in the British Empire’s dominance of world trade.

By today’s standards, Lind’s study so deeply flawed that it would probably be refused publication in any medical journal. (I say “probably” because even poorly designed studies can work their way into the top journals like the Lancet and the British Medical Journal if the editorial team decides to promote a controversial paper over its shortcomings.) Lind gave each treatment to only two sailors and he did not record how he decided which sailor got which treatment. Also, he did not use standardised outcome measures. Lind was lucky that the effect of citrus was so dramatic that his findings were correct despite his flawed trial method. If Edward Jenner had used this method to study cowpox and smallpox, he would have found nothing.

Whatever the flaws in Lind’s trial, he earned a place in history for being the first medical scientist to understand why control trials were needed. His name has been honoured in the James Lind Library, a superb online resource devoted to the history of clinical trials in medicine. The most heroic aspect of Lind’s character was that he was prepared to acknowledge the flaws in his studies so that others could expand on his experiences. Many of his methodical flaws were first described in publication by Lind himself.

Others did learn from Lind. They used larger study populations so that they could be more confident that their findings were due to a real effect rather than chance, and they developed randomisation. When doctors allocate patients to the different arms of a trial, they tend to select sicker people to have the intervention and healthier people to have the control treatment. If the treatment group shows no improvement, it is impossible to know if that is because the treatment did not work or because the people being treated were sicker in the first place. To get around this problem, scientists stopped allocating subjects to trial arms themselves and started using random methods. Hence the randomised control trial, or RCT.

The first true RCT did not take place until 1948, when the British Medical Research Council examined the use of streptomycin to treat pulmonary tuberculosis. As can be seen by the gaps between those two dates, biomedical researchers were very slow to take up the challenges laid out by Lind. Since then, however, study designs have evolved at breakneck speed. Randomisation protocols, crossover trials, systematic reviews, meta-analyses, and nested trials have all developed over a span of sixty years, along with the statistical tools that go with them, in a sort of Cambrian explosion of medical research techniques.

Barry never mentions it in her paper, but another important design feature is blinding. When scientists refer to randomised control trials as RCTs, they almost invariably mean “prospective double-blinded randomised control trials.” Unless there are compelling reasons why prospective double-blinding was impossible to implement, retrospective or unblinded findings are usually condemned to distrust and obscurity.

To understand why blinding is so critical to modern medical research, we’ll take a detour into physics.

(Next: Blinding, Blondlot and N-rays…)