A large clinical trial might be said to resemble an ocean liner…Very occasionally there is a shipwreck – Simon Wessely
Sir Simon Wessely is apparently still hawking cruises on a wrecked ship that can’t be salvaged. I urge refunds.
After a long career, Wessely is in the twilight of his influence and relevance. His tired defense of the design of the PACE trial suggests that he is out of touch with contemporary thinking about psychotherapy trials and risks to their validity. But he still chastises those who disagree with his assessment of the PACE trial.
I invite readers to read and decide….
If you haven’t read Julie Rehmeyer’s excellent article in STAT Bad science misled millions with chronic fatigue syndrome. Here’s how we fought back, you should at least bookmark it for reading later. A mathematics and science writer who happens to be a patient, Julie Rehmeyer’s piece has proven to be the right article at the right time to bring the controversy into the public eye over chronic fatigue syndrome (hereafter CFS/ME)* and what is widely seen as the demolished credibility of the 5 million pound PACE clinical trial of cognitive behavioral therapy (CBT) and graded exercise therapy (GET) for chronic fatigue syndrome. The article has been rightfully republished and is receiving wide commentary in conventional and social media.
Rehmeyer’s article’s uniformly warm reception was marred only by a chilly comment from Sir Simon Wessely that began:
Sorry to spoil the party but some cold facts are necessaey [sic]
This blog post is the first in a series. I will respond to some of Wessely’s odd pronouncements about evaluating clinical trials and his dismissive defense of the PACE trial.
Wessely starts by lecturing his readers that they should check the PACE trial against the CONSORT checklist, which you can find here.
The PACE trial remains an excellent trial and a model of how to deliver a complex intervention RCT. Read the 2012 Lancet paper again. Check it against the CONSORT statement. You will see it is 100% compliant.
This is a puzzling suggestion. The CONSORT checklist evaluates whether particular aspects of a clinical trial are reported in an article, not whether those aspects were competently implemented in the trial. For instance, an author could say that “Data for patients who did not respond to treatment as we hoped were discarded.” That would meet the criteria for disclosing whether or not analyses were conducted on an intention-to-treat basis, but it would be a gross violation of best research practices. As we’ll see, simply relying on CONSORT to decide whether the PACE trial was properly done, would miss some egregious questionable research practices.
Wessely’s comment echoes a similarly condescending comment in his earlier Mental Elf blog, to which he provided a link in his comment on Julie’s article:
I am struck that some of the critics are not familiar with the fundamental strengths of the randomised control trial, and why medicine continues to value it so highly. Likewise, some show unfamiliarity with the core methodological components that contribute to the integrity of a clinical trial, and whose violation calls into question the findings, as compared to what one might call secondary less important features. In other words, what distinguishes a good trial whose results are likely to be sound from one in which there is a definite risk of bias.
Wessely simply doesn’t get it. The PACE trial is now being scrutinized by a large international audience who won’t tolerate being patronized.
Chris Chambers recently remarked:
What’s happened instead is that technology has empowered the new generation to speak freely and publicly on scientific issues, to critique poor quality science and practices, to bust fraud, and to break the bounds of peer review. Twitter in particular has shaken the traditional academic hierarchy to its core. On Twitter a PhD student with thousands of followers suddenly has a greater voice than an Ivy League professor who might have no social media presence at all.
Here are a few of the aspects of the PACE trial comparison of the active treatments – (CBT) and (GET)– to a control condition that Wessely encourages us to ignore.
- Inadequacy of the control group. All four of the conditions in the trial involved providing patients with “standardized specialty medical care” (SSMC). One condition provided only SSMC and served as the control/comparison for evaluating the CBT and GET.
The condition is grossly inadequate as a control group because it is deficient at the basic level of contact time – patients assigned to SSMC received only 3 medical sessions of 30 minute duration. In contrast, patients assigned to CBT or GET have access to these three medical sessions of SSMC plus 15 sessions of either CBT or GET.
Standardised Specialist Medical Care SSMC will be given to all participants. This will include visits to the clinic doctor with general, but not specific advice, regarding activity and rest management, such as advice to avoid the extremes of exercise and rest, as well as pharmacotherapy for specific symptoms and comorbid conditions. SSMC is standardised in the SSMC Doctor’s Manual. As well as this, SSMC participants, like all other participants, will already have received the Patient Clinic Leaflet (PCL). The PCL is a generic leaflet explaining what CFS/ME is, its likely causes, and available treatments. There will be no additional therapist involvement.
So, the patients assigned to SSMC got a pamphlet. In general, clinical and health psychologists researchers are convinced that getting a pamphlet is an inert intervention. So much so, that pamphlets are routinely provided as control inventions where researchers are intent on making their active intervention appear effective. And routinely criticized as an inadequate control condition.
In contrast to SSMC, the active treatments were delivered with a strong induction of positive expectations. Alem Matthees, the patient who obtained release of the PACE data with a FOI remarked in an email to me:
The CBT manuals for PACE assert with confidence that the therapy was safe and powerful (etc), and aimed the therapies at changing patients’ perceptions about their symptoms. Similarly, the GET manuals stated exercise reverses the pathophysiology responsible for symptoms and that most patients feel “much better” after therapy. No such equivalent in the other manuals. So it is difficult to separate any genuine benefit from methodological artefacts arising from placebo response and other reporting bias. If we assume subjective measures are important (which they are) and there is some genuine benefit (which there probably is), we must still consider the use of objective measures.
- The study was not blinded. Patients assigned to either CBT or GET knew they were getting more treatment. In contrast, patients assigned to the control group received only the SSMC. They could see that they have gone to the bother of signing up for clinical research with the expectation that they would get more than SSMC, but now they are being left in that treatment with the added burden of all the research assessments.
The lack of blinding potentiates the problems of a control condition lacking the frequency and intensity of contact provided with the active treatment.
- Bolstering of positive expectations with a newsletter. A newsletter sent to patients while enrollment in the trial was still ongoing strengthened positive expectations and increased a sense of obligation from patients assigned to the control group. This effort was not specified in the original protocol. If this were a drug trial being scrutinized by the FDA, this would be a blatant protocol violation.
I noted in an earlier blog:
Before the intervention phase of the trial was even completed, even before accrual of patients was complete, the investigators published a newsletter in December 2008 directed at trial participants. An article appropriately reminds participants of the upcoming two and one half year follow-up. But then it acknowledges difficulty accruing patients, but that additional funding has been received from the MRC to extend recruiting. And then glowing testimonials appear on p. 3 of the newsletter about the effects of their intervention.
“Being included in this trial has helped me tremendously. (The treatment) is now a way of life for me, I can’t imagine functioning fully without it. I have nothing but praise and thanks for everyone involved in this trial.”
“I really enjoyed being a part of the PACE Trial. It helped me to learn more about myself, especially (treatment), and control factors in my life that were damaging. It is difficult for me to gauge just how effective the treatment was because 2007 was a particularly strained, strange and difficult year for me but I feel I survived and that the trial armed me with the necessary aids to get me through. It was also hugely beneficial being part of something where people understand the symptoms and illness and I really enjoyed this aspect.”
Taken together with the acknowledgment of the difficulty accruing patients, the testimonials solicit expression of gratitude and apply pressure on participants to endorse the trial by providing a positive evaluation of their outcome in the self-report measures they were provided. Some minimal effort is made to disguise the conditions from which the testimonials come. However, references to a therapist and, in the final quote above, to “control factors in my life that were damaging” leave no doubt that the CBT and GET favored by the investigators is having positive results.
Adequacy of control groups is of crucial importance. The US Agency for Healthcare Research and Quality (AHRQ) undertook a comprehensive systematic review and meta-analysis of meditation programs for psychological stress and well-being. I covered the agency’s report and the JAMA: Internal Medicine article in a recent blog post, Mindfulness research’s huge problem with uninformative control groups. The AHRQ report and JAMA article concluded that the widespread impression that meditation is an effective way of reducing stress and improving well-being largely comes from trials with inadequately matched control groups. When meditation, including mindfulness, iscompared to suitable active treatments, there is insufficient evidence of any superiority.
The same could be said for the PACE trial, but I’m just getting started. In my next blog post, I will take a critical look in a second comment on Julie’s article at Simon Wessely’s endorsement of outcomes switching as an admirable feature of the interpretation of clinical trials. What he says:
In essence though they decided they were using a overly harsh set of criteria that didn’t match what most people would consider recovery and were incongruent with previous work so they changed their minds – before a single piece of data had been looked at of course. Nothing at all wrong in that- happens in vast numbers of trials. The problem arises, as studies have shown, when these chnaged [sic] are not properly reported. PACE reported them properly. And indeed I happen to think the changes were right – the criteria they settled on gave results much more congruent with previous studies and indeed routine outcome measure studies of which there are many.
In a subsequent blog post, I’ll be arguing that Simon’s justification is factually incorrect and inconsistent with best research practices. Until my next post, for a condemnation of outcomes switching in pharmaceutical trials, see my recent blog post, Study protocol violations, outcomes switching, adverse events misreporting: A peek under the hood. I report on criticism by prominent senior psychiatrists of bad research practices in trials of antidepressants. After writing that blog post, I signed a letter with the senior psychiatrists demanding retraction of one of these papers the blog discussed. The paper reported data from a ghost written trial of antidepressants characterized by protocol violations, outcomes switching and adverse events misreporting. I’ve heard no evidence that the PACE trial was ghostwritten, but I think there is ample evidence of its sharing these other sins.
**I am using this term with some reservation, because it’s familiar to readers and because that is how a set of diverse conditions is named in the bulk of the scientific and in the media. But for an excellent critique of the term, see another excellent article in STAT, Why we shouldn’t call it ‘chronic fatigue syndrome’ . I expect we will see a retirement of the term “chronic fatigue syndrome.”