In defence of randomised control trials [3/4]
Demolishing the deconstruction
Through long experience, I have come to the sad estimation that most claims to have “deconstructed” something, whatever the status of deconstruction as a theoretical pursuit, in practice amount to little more than attempts to undermine credibility by unsupported assertions of hidden political, gender, or cultural bias. Barry’s paper fails to rise above my prejudice.
What Barry fails to appreciate is that RCTs have developed over a long time specifically to reduce the effect of bias on results, but she has come to the opposite conclusion.
Macro-analyses suggest that the response of biomedicine at the system level, to the growing popularity of alternative medicine, can be interpreted as politically motivated. Mike Saks suggests RCT evidence is being used strategically by biomedicine’s medical associations, to reduce potential threat from alternative medicines, feeding into incorporationist and assimilationist policies (Saks, 1996).
In the world of Barry’s deconstructive anthropology, the rise of the RCT had more to do with controlling alternative therapists with biomedical “rhetoric” than with finding out that streptokinase saved lives and flecainide killed people.
The main points of her argument (there are many more sub-arguments) can be summarised as:
- RCTs are not perfect.
- RCTs do not account for the placebo effect.
- RCTs are often criticised within biomedical circles for poor design.
- RCTs measure simple outcomes rather than complex outcomes.
- RCTs are controlled by large drug companies with no interest in non-patentable medicines.
- The evidence from RCTs is not routinely implemented even by orthodox practitioners.
- RCTs are reductive while alternative therapies are holistic.
- Homeopaths treat different patients with different treatments, even for the same biomedical diagnosis, so treatment cannot be randomised.
- Recruiting subjects for RCTs in alternative medicine is difficult because these patients often refuse to take the chance of being in the placebo arm and they say they will continue to believe in alternative medicine regardless of evidence.
- The rise in RCTs has been a political move by orthodox practitioners to control medical resources.
Broadly speaking, these objections belong to two strategies: criticising the limitations of RCTs and criticising the application of RCTs to alternative medicine. The limitations of RCTs are as relevant to biomedical practice as to alternative medicine, so I will address these first. In order to keep this post from becoming even longer, I will focus on only a few of Barry’s claims. There are rejoinders to all of them which I will expand on if there is sufficient interest.
“The RCT can sometimes become a victim of hubris. Just being the ‘gold standard’ is not enough — it is still an imperfect tool.” Indeed RCTs are imperfect, although it is hard to see how they suffer from hubris, a quality usually attached to tragic heroes like Oedipus and Agamemnon. The purpose of RCTs is to reduce bias, but there are so many ways that bias can creep in that even the most fastidious study cannot entirely eliminate it. To choose one example: a not uncommon bias in drug trials comes from comparing a new drug to an older, established drug — only to “underpower” the control arm by prescribing a low dose of the older drug to be compared against a high dose of the new drug. The abstract of such a paper may well conclude that the new drug is more effective than the older drug, and in terms of the raw data the conclusion would be justified. The problem is in the design. There are many other ways an RCT can fail, and as Barry points out, when scientists perform systematic reviews of the available research, most published RCTs are rejected as being of poor quality.
Another limitation is that the RCT is only the “gold standard” for one particular type of study. If you want to know whether a given treatment works better than placebo or existing treatment, the double-blind RCT is best. But there are plenty of other kinds of useful knowledge. To know the prevalence of syphilis in the community, the appropriate trial would be a population study. To assess a new test for HIV, then the appropriate study would be a diagnostic trial. Even in intervention trials, there are situations where RCTs are not feasible, such as when a disease is so rare that there is no possibility of recruiting enough subjects or where the study cannot be performed ethically. New drugs are almost never tested on pregnant women.
While RCTs are not perfect, they remain the best design for intervention studies. (Barry refers to systematic review as having a higher status, but this is a matter of some controversy and even so, it is impossible to perform a systematic review of RCTs without a population of RCTs in the first place.) To paraphrase Churchill on democracy: No one pretends that the RCT is perfect or all-wise. Indeed, it has been said that RCTs are the worst form of intervention study except all those other forms that have been tried.
“The placebo concept has powerful healing properties not fully acknowledged in RCT methodology.” On this point, Barry is just plain wrong. Far from disregarding the placebo effect, RCTs developed because of it. The “placebo concept” came directly out of the experience and research of biomedical scientists rather than alternative therapists, and each incremental improvement in the methodology of RCTs followed a deeper understanding of the ways in which the placebo effect could influence outcomes.
The basic purpose of the RCT is not to eliminate the placebo effect but to check whether the intervention has any benefit above placebo. After all, if the treatment is no better than placebo, then why pay for it? Why expose patients to side effects? It is becoming more common for RCTs to abandon placebo in the control arm and instead use an existing treatment. One reason for this is ethical: it is hard to justify giving patients placebo if there is already an effective treatment, especially for life-threatening diseases. Another reason is practical: if there is already an effective treatment, then nobody particularly wants to know if the new treatment is merely better than placebo. The important question is whether it is better than a current treatment known to be better than placebo. As mathematicians like to put it, this reduces the problem to one already solved.
The avoidance of placebo effect is a common criticism of orthodox medicine, but this objection is based on the bizarre assumption that the placebo effect only works for alternative treatments. In my own practice, I am delighted to exploit the placebo effect. If I am confident that a treatment will improve a patient’s condition, I make sure that the patient is aware of the reasons for my confidence. The patient expectation will be greater, and as we know, this will elicit a strong placebo effect on top of the pharmacological or surgical effect.
Encouraging a placebo benefit is one thing. Giving pure placebo is another. If a doctor knows that a treatment is ineffective and yet tells a patient that it has curative powers, a strong placebo response may result, but is this not a case of deceiving the patient? How is it possible to knowingly mislead patients while offering informed consent?
“There are…ongoing extensive debates about EBM [Evidence Based Medicine] among biomedical practitioners.” Well, of course there are. Since RCTs and scientific studies in general are not perfect, there is always going to be debate about the value of specific studies. (It is striking how much this argument has in common with creationist arguments. To creationists, small gaps in the understanding of evolution are counted as fatal flaws, and debate among evolutionary biologists on specific mechanisms such as neutral drift is taken as evidence of a “theory in crisis” rather than the to-and-fro of a thriving scientific discipline.)
One of Barry’s criticisms is particularly telling:
Another example of attempts to control the impact of alternative medicine, through defensive strategies, is evidenced by the impassioned hostile response to RCTs published in mainstream medical journals. Such trials usually produce a furore of response from individual doctors and scientists. In the British Medical Journal, Taylor et al. (2000) published an overview of four trials of homeopathy in perennial allergic rhinitis. There were an unusually large number (61) of electronic rapid letter responses, mostly dismissing these positive findings, often using emotively charged phrases: ‘The Emperor’s New Therapy’, ‘Homeopathy is ridiculous. Preposterous’, ‘Like the sprinkling of holy water on vampires’ (see responses on www.BMJ.com after Taylor et al., 2000).
It does not seem to occur to Barry that the paper may have been flawed and that at least some of the criticism was warranted. In fact, I remember this paper when it was first published. It went straight into my personal file of poor statistical techniques to appear in respected journals. Barry does not even make a gesture towards defending the quality of the study. It appears to be beside the point to her. Barry assumes that criticism of this paper is ipso facto a political attack and any reasoning behind the criticism is irrelevant.
The original paper (Taylor et al, BMJ 321:471-476, 2000) can be found here. Rather than explain in great detail why this paper deserved some of its criticism, I will summarise it as succinctly as possible: The study claims to have shown that homeopathic treatments successfully treat allergic rhinitis (hay fever). The authors studied two outcomes, nasal air flow and patients’ subjective reports. While they found a significant improvement in air flow, they did not find any difference in subjective response. They still managed to produce a statistically significant result by mixing in their results with previously published studies which were known to show some statistical change. That is, they carried out a double-blind RCT but when the results were not impressive, they presented it as a meta-analysis and tossed in a known positive dataset instead of data from a proper systematic review…a technique that suggests to me the application of homoeopathic techniques to research, complete with serial dilution in a statistically significant solvent and succussion of data with each iteration.
Detail of a table from Taylor et al, 2000. The data collected in this study (my emphasis) shows no difference between treatment and control. Only by combining the results with pre-existing data did the outcome show any improvement — and then only marginally. Also note that the top two data sets show overlapping confidence intervals on the left, and yet when they are converted to pooled data on the right they have acquired statistical significance.Although much of the criticism was beside the point, several correspondents did raise fair concerns about the quality of the analysis. I refer interested readers to the original correspondence to read for themselves. Anyone who takes the time to do so will realise that Barry has committed more than one sin here. Not only has she declined to address the question of whether the criticism was warranted, she has seriously misrepresented it. Of the 61 letters in the archive, only 14 use emotive language. The quotes Barry pulled from the letters come from only two letters. Two of the quotes come from the “Emperor’s New Therapy” letter, which certainly is an emotive tirade, and the third quote was taken out of context. Here is the full letter about vampires and holy water:
If homeopathic remedies were produced according to their own strict formula, they would be chemically inert, with something like a single molecule of active ingredient per tankful of carrier substance. If they produce results significantly better than placebo, something must be fudged: maybe a violation of randomization or blinding protocols, or use of homeopathic test substances vastly more concentrated than they were purported to be, or the unsuspected presence of other active material in the carrier.
I don’t believe that a substance chemically indistinguishable from water has mystical healing powers any more than I believe that holy water sprinked on vampires causes them to burn. I don’t believe in any form of magic. Science certainly doesn’t have every answer, but we abandon it for magic at our extreme peril.
While this is a little confrontational, it is hardly the stuff of abuse, and it is important to realise that the author is not dismissing the results per se but pointing to possible biomedical explanations and design limitations that would accommodate the findings within our well-tested knowledge of chemistry.
Meanwhile, Barry declines to mention the fact that another 12 letters use emotive language and sometimes personal abuse or threats to the skeptics of homoeopathy. Here are a few samples of invective, each from a different letter: “It’s time governments supported scientists who have moved on, as opposed to those who should be moved on”; “ludicruous and hypocritical”; “a little foolish”; “double standards”; “the behemoths of modern medicine are just as obtuse as those of 200 years ago”; “it is customary in any public discussion forum, [a] to know at least something about the subject, and [b] to have something useful to say” (this to someone who was asking tough but genuine questions); “arrogant (and too often incompetent) conventional therapists”; “like a thundering, narrow-minded Pope of old”; “hateful prejudice”; and “armchair philosophy.” One correspondent claimed that skepticism of homoeopathy is a strong a warning sign of being a drug company stooge and even demanded of another correspondent that he “satisfy BMJ readers…that his motives are truly honourable,” as if questioning the motives of a skeptic is sufficient to overturn Avagadro’s number.
Contrary to Barry, more than three-quarters of the letters were polite and considered from both skeptics and defenders alike. Of the 14 emotive letters, twelve were from homoeopathy defenders and only two were from skeptics. For abusive letters, the odds are eleven to one. Far from being an example of orthodox medicine using emotive language to suppress homoeopathy, this exchange of shows that people of varying experiences and viewpoints can have a constructive debate while ignoring abusive letters — which were almost exclusively directed in the opposite direction to that implied by Barry.
I came to this conclusion using a politicised scientific rhetoric known as counting.
Revealing flaws and misrepresentations in Barry’s paper, however, does nothing to address the central question she raises. Should standard biomedical research techniques be applied to alternative therapies?
12 People have left comments on this post
Chris,
Can I just say that the “deconstructive” approach you criticise has also run riot for twenty years now in the social sciences. Deconstruction itself, as concieved by Derrida, is one thing, but the common garden variety postmodernism that claims that such “instrumentally rationalist” ideas such as truth, evidence, objectivity and so on are simply authoritarian inventions designed to bolster the dominant discourse, involves an eviscerating conflation of a number of things and ultimately destroys deconstruction as Derrida conceived it. When one of my students comes up with the position that there is no such thing as “truth” I ask them whether they believe the holocaust actually happened. It tends, at least, to make them think.
In defence of Derrida, Terry Eagleton has explained that, “Deconstruction, the philosophical method he [Derrida] promoted, means not destroying ideas, but pushing them to the point where they begin to come apart and expose their latent contradictions. It meant reading against the grain of supposedly self-evident truths, rather than taking them for granted. ”
Having said all this, it seems to me that there IS a critique to be made of western medicine(don’t you?), though perhaps this is not the place to make it.
Rjurik
Hi, Rjurik.
I agree. The basic idea behind deconstruction, as far as I can see through the myriad of different definitions, is to challenge the underlying assumptions of a piece of text, film, etc. And I think this is not only an interesting exercise, but sometimes extremely valuable. This is why my opening paragraph talks only of deconstruction as it is often practised rather than dismissing it out of hand.
I think the problem with deconstruction is twofold. The first is that many people who are not particularly interested in debating a point on its merits find it tempting to make broad generalisations about the biases of the author(s) and leave it at that. The actual problem is never addressed, only the author’s right to speak about it. And as we all know, even the most vile, biased, reprehensible writer can get some things right. William Shockley, a hardcore racist, was still a great physicist who earned his Nobel Prize. To me, most deconstruction is putting the cart before the horse. When I know something is wrong, then I’m fascinated to know how people came to believe it, and that’s where deconstruction comes in. But not the other way around.
I think the second problem is the word “deconstruction” sounds too much like “destruction.” I see deconstruction in much the same way that I see dissection by a pathologist. When someone has died, it is a valuable exercise to cut them open and find out why. This applies equally to ideas or books. While dissection is a physically destructive process (obviously the body is not intact afterwards), the purpose is to learn something useful. Likewise, deconstruction can be a destructive process (it can damage a writer’s reputation — not too many people quote Kipling without qualification anymore), and unless the purpose is to learn something useful then it’s just destruction for the joy of it.
As an example of a good deconstruction (that was never called that as far as I know), I recommend John Kessel’s evisceration of the moral universe in Orson Scott Card’s fiction.
As for critiques of Western medicine, I am all for them. There are plenty of practices worth critiquing. Too many to list here, in fact. What I want, though, is thoughtful and constructive critiquing rather than special pleading for those who want the credibility of biomedical science without the rigour that established that credibility in the first place.
I’d also point out that the most powerful and thoughtful critiques come from within the field itself. This only really started with Lind’s trial, and has only become an established pattern for the last 50 years or so. Before that, medical knowledge was largely dominated by received authority and was just as prone to authoritarian errors as any other field of endeavour.
To the best of my knowledge, “anthropological” or “social science” investigations of science and medicine have failed to create any useful critique that was not already well-established by scientists and doctors themselves. For instance, Thomas Kuhn, author of The Structure of Scientific Revolutions, was a physicist and he grounded his arguments in his knowledge of science. This is one of the reasons why the book has never been out of print since 1962, while the shallow and scientifically-ambivalent Life Among the Scientists has already been forgotten despite many glowing reviews on its release in 1990.
Is this the place for critiques? Possibly not in this comment thread, but I’d be delighted to post any interesting pieces.
Thanks for this Chris. I’m really enjoying it.
I agree with Rjurik that there is potential for a good critique of science, but I also agree that a lot of the critiques I’ve read, in the arts field, are pretty lazy. As you say, they make arguments about empiricism and black-and-white notions that I suspect are unrealistic pictures of how the majority of scientists think and work…
Chris,
when we next sit down we’ll have to talk about Kuhn, because it’s been some years since I read him. Lakatos seemed more compelling to me, as a philosopher of science when I read all that stuff. But it has been, well, quite some time.
It also seems to me that a critique of science should be as much about the “practice” of it - i.e. the economy of science (something like 70% of scientists (correct me if I’m wrong, which I probably am) work for the military industrial complex. Just as the economy of drug companies/ drug distribution distorts the actual research and development, the kinds of drugs we use, and so on. What happens, then, is that the actual science or medicine - the research, the theories, and so on - involved is “overdetermined” (to get theoretical and use a phrase of Louis Althusser’s) or more simply, influenced, by this set of social relations.
What needs to be defended then is the actual science - from these influences. It’s not the science that needs to be jettisoned so much as the social set-up. The mistake of many “postmodernists” is to conclude that western science etc is thus to be thrown out “tout court.”
Rjurik
I agree about Kuhn. He was not a compelling writer, particularly in the Structure of Scientific Revolutions which is essentially a dry, systematic description of his thoughts on the matter. Far more compelling is his lesser-known The Copernican Revolution, which covers a lot of the same ground but has the added advantage of being tied to one particular subject area and including the fascinating stories of Copernicus and Galileo. I’ve only read about Lakatos, not his original work.
I am skeptical of that 70% claim, even if one tweaks the figures by including scientists who are now in managerial positions and one uses a very broad definition of the military-industrial complex. But regardless of the actual figure, there are plenty of opportunities to critique the way science and medicine is practised without, as you say, throwing trouts at the tennis.
The thing about Kuhn that he’s usually criticised for is his relativism. There seems little within his theory to describe “advances” rather than simply “replacements.” Again, it’s been a while, but I thought Lakatos seemed to deal with this problem.
I did enjoy The Copernican Revolution and also Koestler’s The Sleepwalkers which is less accurate but wonderfully written.
Rju
I think that Kuhn has been misread on this point, largely because he considered it self-evident and didn’t address it. Kuhn is not a strong relativist and he has been critical of the extreme post-modern view in recent times. What he was saying was that there was a problem with the then-standard view that science progressed by ever-finer approximations, just adding decimal points to the fundamental constants. Kuhn knew that finer approximation or “working within the paradigm” is what happens most of the time in science, but every now and then a revolutionary theory comes along that completely abandons the previous worldview. Einstein’s universe is not a more precise version of Newton’s, it rests on completely different axioms.
As for Koestler, he did write beautifully, but I found the inaccuracies in THE SLEEPWALKERS hard to forgive as they were well within his power to check, and even worse, were accompanied by a constant, tub-thumping self-righteousness. Factual error and moral superiority make an unhappy combination.
Koestler made his name with his fictional attack on Stalinism in DARKNESS AT NOON, where his moral indignation was warranted (especially since at the time there were many Western academics bending over backwards to excuse Stalin’s crimes) and where any errors of fact were dwarfed by (and largely caused by) the Soviet Union’s own disregard for truth. I suspect, but don’t know, that Koestler’s critique of the progressivist view of science was really an attack on the Marxist version of progress, and it skewed his vision.
And I really should read some Lakatos. Any advice on where to start?
Hi Chris,
I’ve enjoyed your critique of Barry’s article, and want to respond on a few points, but might wait until you post the fourth section before doing so fully. For now I had a couple of comments about your discussion with Rju. I agree with both of you about the sloppiness of much self-proclaimed “deconstruction”, and also agree that Barry’s paper (though she only throws the term in once) participates in this. This is one of the reasons Derrida disassociated himself from the term over the last couple decades, and is evident in the fact that you can (with some accuracy) portray “deconstruction” as consisting of (1) ad hom attacks and (2) senseless destruction, when in fact (1) Derrida was noted for destabilising notions of authorship and intention (and it was his Anglo-American detractors who resorted to such tactics) and (2) deconstruction was coined explicitly as a modification of Heidegger’s Destruktion in order to disrupt the term’s violence, and was always practised with an ethico-political “purpose” in mind.
As an aside, I’d also want to suggest that some aspects of what might properly be called deconstruction are in fact applicable not only to texts and ideas but also to social practices, experiments and the scientific production of knowledge. But if we’re going to talk philosophy (or anthropology) of science - you guys mentioned Kuhn and Lakatos - then a perhaps stronger strand that enables the critique of (as Rju points out) the social dimension of science and medicine comes (again) from the French through (Bachelard, Althusser but more strongly) Foucault and Latour. The latter is central to what is now called “actor-network theory” (with plenty of its own distillation and bastardisation, of course). In his best work I think he demonstrates the possibility of critiquing scientific practice as a non-scientist participant observer. The other key figure is Haraway, who is a biologist, and who annoyed a great number of primatologists and other scientists with her standpoint feminism.
Anyway, look forward to the final installment.
Matt
Hi, Matt. The fourth instalment is up, so feel free to comment away. One point I should make at this juncture: this isn’t really intended to be a broad swipe at deconstruction itself. I’ve used the term “deconstructive anthropologist” to distinguish this approach from the scientific anthropologists, but I think I need to come up with a better term because I don’t have a problem with deconstruction as a principle. (Likewise, I wouldn’t describe myself as a post-modernist but I can still appreciate that there was some very interesting work in early post-modernism before it became infested with anti-rationalists looking for cheap rhetorical ploys.)
In the next instalment I used the term “anti-rationalist” to describe Barry, but I’m not even sure that’s quite right as she claims not to want to abandon RCTs altogether. I guess I’m still grasping for the right phrase.
Thanks Chris. There’s probably an argument for letting them have the term. But terminological discussions are boring. Ultimately - after Barry has been taken to task - the real issue is going to be strength of the critiques that deconstruction proper, and other more rigorous “postmodernist” or whatever-we-call-them anthropologists, might make of biomedicine, RCTs and so on.
BTW on Lakatos, see Lakatos ed. Criticism and the Growth of Knowledge, and also Essays in honour of Lakatos, ed. Feyerabend and pals. But I think you’d find much more of interest in Foucault’s The Birth of the Clinic.
1 Trackback(s)
Sorry, comments for this entry are closed at this time.