“I’m in competition with literally hundreds of stories every day, political and economic stories of compelling interest…we have to almost overstate, we have to come as close as we came within the boundaries of truth to dramatic, compelling statement. A weak statement will go no place.”
—-Journalist interviewed for JA Winsten, Science and media: the boundaries of truth
Hyped, misleading media coverage of a study in Lancet of CBT for persons with unmedicated schizophrenia left lots of clinicians, policymakers, and especially persons with schizophrenia and their family members confused.
Did the study actually showed that psychotherapy was as effective as medication for schizophrenia? NO!
Did the study demonstrate that persons with schizophrenia could actually forgo medication with nasty side effects and modest effectiveness and just get on with their life with the help of CBT? NO!
In this blog post, I will scrutinize that media coverage and then distribute blame for its inaccuracies.
At PLOS Mind the Brain, I’ve been providing a detailed analysis of this complex study that was not particularly transparently reported. I will continue to do so shortly. You can consult that analysis here, but briefly:
The small-scale, exploratory study was a significant, but not overwhelming contribution to the literature. It showed that persons with unmedicated schizophrenia could be recruited to a clinical trial for psychotherapy. But it also highlighted the problems of trying to conduct such a study.
- Difficulties getting enough participants resulted in a very mixed sample combining young persons who had not yet been diagnosed with schizophrenia but who were in early intervention programs with older patients who were refusing medication after a long history of living with schizophrenia.
- The treatment as usual combined settings with enhanced services including family therapy and cognitive therapy with other settings where anyone who refused medication might be denied any services. The resulting composite “treatment as usual” was not very usual and so did not make for a generalizable comparison.
- Many of the participants in both the CBT and control group ended up accepting medication before the end of the trial, complicating any distinguishing of the effects of the CBT versus medication.
- The trial was too small, had too many participants lost from follow up to be used to determine effect sizes for CBT.
- But, if we must, at the end of the treatment, there were no significant differences between persons randomized to CBT and those remaining in routine care!
The official press release from University of Manchester was remarkably restrained in its claims, starting with its title
Cognitive therapy “safe and acceptable” to treat schizophrenia
There were no claims of comparisons with medication.
And the press release contained a quote from the restrained in tentative editorial in Lancet written by Oliver Howes from the Institute of Psychiatry, London:
“Morrison and colleagues’ findings provide proof of concept that cognitive therapy is an alternative to antipsychotic treatment. Clearly this outcome will need further testing, but, if further work supports the relative effectiveness of cognitive therapy, a comparison between such therapy and antipsychotic treatment will be needed to inform patient choice. If positive, findings from such a comparison would be a step change in the treatment of schizophrenia, providing patients with a viable alternative to antipsychotic treatment for the first time, something that is sorely needed.”
But unfortunately the rest of this excellent editorial was, like the Lancet report of the study itself, locked behind a pay wall. Then came the BBC.
“Talk therapy as good as drugs for schizophrenia.”
Inexplicably, when the BBC story was accessed a few days later, the headline had been changed to
“Talk therapy moderately effective for schizophrenia.”
There was no explanation, and the rest of the new item was not changed. Creepy. Orwellian.
The news item contained a direct quote from Tony Morrison, the principal investigator for the study:
We found cognitive behavioural therapy did reduce symptoms and it also improved personal and social function and we demonstrated very comprehensively it is a safe and effective therapy.
Wait a minute: Was it really demonstrated very comprehensively that CBT was an effective therapy in this trial? Are we talking about the same Lancet study?
The quote is an inaccurate summary of the findings of the study. But it is quite consistent with misrepresentations of the results of the study in the abstract. Here as elsewhere in the media coverage that would be rolling out, almost no one seemed to scrutinize the actual results of the study, only buy into the abstract.
Shilling at Science Media Centre: Thou shalt not shill.
Science Media Centre ran a briefing of the study for journalists and posted an
Expert reaction to CBT for schizophrenia
A joint quote [how you get a joint quote?] from Professor Andrew Gumley, University of Glasgow and Professor Matthias Schwannauer, University of Edinburgh proclaimed the study “groundbreaking and landmark.” They praised its “scientific integrity,” citing the study’s pre-registration that prevented “cherrypicking” of findings to put the study in the best light.
As I’ll be showing in my next Mind the Brain post, the “pre-registration” actually occurred after data collection had started. Preregistration is supposed to enforce specificity of hypotheses and precommit the investigators to evaluate particular primary outcomes at particular times But preregistration of this trial avoided designation of the particular timepoint at which outcome would be assessed. So it did not prevent cherry picking.
Instead, it allowed the authors to pick the unusual strategy of averaging outcome across five assessments. What they then claimed to represent the results of the trial was biased by
- a last assessment point when most patients were not longer followed,
- many of the improved patients were on medication,
- and results affected by a deterioration of the remaining control patients, not improvement in the intervention patients.
Again, let me remind everybody, at the end of the intervention period—the usual time point for evaluating a treatment, there were no differences between CBT and treatment as usual.
Actually, these two guys were quite familiar with the study because they are collaborators with Tony Morrison in a new study that is exactly the kind that they ended proposing as the needed next step. They weren’t unfamiliar, they were hiding a conflict of interest in praising the study and calling for what they were already doing with Tony Morrison. Come on, Andy and Matt, no shilling allowed!
As it has done in the past when there is a lot of media hype, NHS Choices offered an exceptionally sophisticated, restrained assessment of the study. This source missed the lack of differences between intervention and control group at the end of the treatment, but it provided an exceptionally careful review of the actual study not just its abstract. It ended up catching a number of other important limitations that almost nobody seemed to be noticing. And it warned
However, importantly, it does not show that it is better or equivalent to antipsychotic medication. The participants continued to have moderate levels of illness despite receiving therapy.
Media coverage got more exaggerated in its headlines.
Wired.UK.com: “Talking therapy could help schizophrenic sufferers that refuse drugs”
This story added a provocative direct quote from Tony Morrison in a sidebar:
“Without medication the results were almost exactly the same as the average effect size you see in antipsychotic drug trials”
This is simply an untrue and irresponsible statement.
TopNews Arab Emirates: CT Acts as Great Treatment for Schizophrenia Spectrum Disorder Patients Unwilling to take Antipsychotics—
Some of the most misleading headlines appeared at sources that consumers would think they could trust
Medical News Today Cognitive therapy ‘an effective treatment option’ for schizophrenia
West (Welfare, society, territory): Cognitive therapy better than drugs for schizophrenia.
GPonline.com: CBT for schizophrenia an effective alternative to antipsychotics, study finds
Nursing Times: Study suggests drug-free therapy hope for schizophrenia
And the prize goes to AAA’s Science for the most horrible misrepresentation of the study and its implications
Science: Schizophrenia: Time to Flush the Meds?
So who’s to blame? Lots of blame to go around
It has been empirically demonstrated that lots of the distortion in medical and scientific journalism starts with distorted abstracts. That was certainly the case here, where the abstract gave a misleading portrayal of the findings of the study that persisted unchallenged and even got amplified. The authors can be faulted, but so can Lancet for not enforcing CONSORT for abstracts or even their requirement of trial design and primary outcomes be preregistered.
The authors should be further faulted for their self-promoting, but baseless claims that their study indicated anything about a comparison between cognitive therapy and neuroleptics. The comparison did not occur and the sample in this study was very different than the population studied it in research on neuroleptic medication.
Furthermore, if Lancet is going to promote newsworthy studies, including in this case, with podcasts, they have a responsibility to take down the pay walls keeping readers from examining the evidence for themselves. BMJ has already adopted a policy of open access for clinical trials and meta-analysis. It is time other journals follow suit.
For me, when the most striking things about media coverage is its boring, repetitive sameness. Lazy journalists simply churnaled or outright plagiarized what they found on the web. The competition was not in content, but in coming up with outrageous headlines.
It is absolutely shameful some of the most outrageous headlines were associated with sources that ostensibly deserve respect. In this instance, they do not. And someone should reel their marketing departments or whoever chooses their headlines.
There is a lot to blame to go around, but there is also room for some praise.
NHS Choices gets a prize for responsible journalism and careful research that goes beyond what the investigators themselves say about their study. I recommend the next time the media gets into a frenzy about particular medical research, that consumers run and look at what NHS Choices has to say.
Stay tuned for my upcoming PLOS Mind the Brain post that will continue discussion of this trial. Among other things, I will show that the investigator group knew what they were doing in constructing such an inappropriate control/comparison group. They gave evidence that they believed that a more suitable comparison with befriending or simple supportive counseling might not have revealed an effect. It is a pity, because I think that investigators should go for appropriate comparisons rather than getting an effect.
I will identify the standards that the investigator group for this trial has applied to other research. I will show that if they apply the same standards to their own study, it is seriously deficient except as a small preliminary, exploratory study that cannot be used to estimate the effects of cognitive therapy. But the study is nonetheless important in showing how hard it will be to do a rigorously controlled study for a most important question.