It took a while. Five years after Ars’ Chris Lee pointed out that the authors of a homeopathy paper did little more than offer “magic”
The work described in the now withdrawn paper involved a small clinical trial for depression with three groups of participants. One group received a standard treatment, another a placebo. The third group received a homeopathic remedy – which means they received water. According to the analysis in the paper, the water was more effective than either placebo or standard treatment. But as Chris noted in his original critique, the authors jump to the conclusion that treating people with water must therefore be effective.
The problem with this is that it ignores some equally viable explanations, such as a statistical fluke in a very small study (only about 45 people per group), or that it was the time spent with the homeopathic GP that made the difference. , not the water. These are problems with the interpretation of the results rather than with the data. (This probably explains why the paper ended up being published by PLOS ONE, where reviewers are asked to simply look at the data quality instead of the significance of the results.)
This does not mean that there were no potential problems with the data. According to the withdrawal announcement, other researchers criticized specific aspects of the research, prompting the journal to convene a panel of three editors, an external academic and a statistical expert. They considered one of the issues Chris noted – inability to rule out a placebo effect from the homeopathic process. But they also examined how the authors chose different “treatments” that involved variations in the preparation of the water. There were also questions about how the participants had initially been diagnosed.
The authors of the original paper were given the chance to respond, and they did. But the PLOS ONE committee found their response inadequate, leading to withdrawal.
In response to their refusal, the authors say they have provided more than enough information to anyone skilled in homeopathy to repeat the study – they all apparently wanted to know exactly how to prepare water based on a patient’s symptoms.
But their response also gives kind of play away. “The PLOS ONE editors did not explain in what ways they considered our study design to be inadequate,” they wrote. “Rather, they simply said it because the homeopathic treatments included [different] strengths of the homeopathic medicines [aka water]any perceived positive effect must have been a placebo effect. “For the authors, their study design was intended to rule out a placebo effect so that any difference could show the effect of homeopathy. They are sorry that the editors did not see things that way.
And rather than the specific complaints about the method, this is the real problem here. Control groups do not tell you anything about the specific mechanism that drives any changes in the experimental group. They just let you identify when experimental conditions give a different result. That cause of this difference is a matter of interpretation, informed by what we know from other scientific studies. If you see a difference, consider all the scientifically plausible mechanisms to account for it.
Based on what we know from other work, PLOS ONE editors are right in considering “homeopathy generates a stronger placebo effect than a pill” as a likely mechanism. And they are right in not considering “water behaving magically” as distantly probable. The authors of the paper prefer the latter, so that their work does not belong in a scientific journal.
What is somewhat frustrating is that the editors do not bother to pamper it. Instead, their withdrawal notice focuses largely on experimental details, as if the paper could be corrected by elaborating on the Materials and Methods section. It gives a misleading picture of the problems here. While there is some consolation from achieving the right result – the paper is officially withdrawn – it would be more helpful if the result came for the right reasons.