Sometimes authors misapply principles of evidence-based medicine appraisal when their interests and livelihood depend on getting it wrong. This could serve as an example.
Leichsenring, F., Luyten, P., Hilsenroth, M. J., Abbass, A., Barber, J. P., Keefe, J. R., … & Steinert, C. (2015). Psychodynamic therapy meets evidence-based medicine: a systematic review using updated criteria. The Lancet Psychiatry, 2(7), 648-660
Falk Leichsenring and Sven Rabung have trying for years to get the “evidence supported” label attached to psychodynamic psychotherapy (PP), especially long term psychodynamic psychotherapy (LTPP). They have received surprising assists from editors of prestigious journals like JAMA and British Journal of Psychiatry. They have also faced some withering criticisms of their breaking of basic rules in conducting and interpreting meta analyses and systematic reviews.
I teamed up with Aaron T. Beck and others in a detailed critique entitled Is Longer-Term Psychodynamic Psychotherapy More Effective than Shorter-Term Therapies? of their earlier efforts in JAMA. We faulted the meta-analysis and systematic review for its
- Computational mistakes.
- Improbably large effect sizes.
- Integration of randomized and nonrandomized trials.
- Ignoring of the uniformly poor quality of the studies included in the review.
Some of the studies involved comparing PP to conditions in which patients did not receive therapy of any kind.
For another succinct methodological critique see
Littell, J. H., & Shlonsky, A. (2011). Making sense of meta-analysis: A critique of “effectiveness of long-term psychodynamic psychotherapy”. Clinical Social Work Journal, 39(4), 340-346.
Leichsenring and Rabung’s reply was not responsive to our critique, but they charged I had not disclosed a conflict of interest:
[This] critique reveals a bias that is obviously related to their own approach, although they do not disclose their conflicts of interest. One wonders whether CBT advocates who have a large financial stake in propagating the view that their treatment is superior, or the only evidence-based approach, can be objective about these scientific issues. Thus, the paper by Bhar et al.  may serve as an excellent example of an investigator allegiance effect. We regret that in this discussion, research is in danger of being used as a weapon to defeat a real or imagined rival. Although directed against and psychodynamic psychotherapy, such a misuse can do harm to the public image of psychotherapy in general.
Yup, Falk and Sven, I spend all my time writing CBT self-help books and doing workshops, when I am not, I do what I can to harm the public image of psychotherapy.
Actually, while I consider Dr. Beck a friend, he considers me one of his most persistent critics.
Like other psychotherapies, PP seeks the label of “evidence-based” in order to secure third-party payments. The problem is that PP is longer term and more intensive than other psychotherapies claiming that label. The added length and intensity may not be cost-effective. For some conditions, such as eating disorders, the dodo bird verdict may not actually hold for PP. For instance, cognitive behavioral therapy that is less intensive but more goal- rather than insight- oriented may be more effective than PP for bulimia.
A larger issue however is that practitioners of PP have historically tended toward suspicion and even hostility toward attempts to fit their approach into the rigor (that they consider false rigor) of a randomized trial. They tend to favor psychodynamic formulations over the formal diagnostic categories and standardized measurements used in research. PP does not have a strong research tradition. So, the quantity and quality of evidence that can be mustered for its efficacy is less than for other therapies
Leichsenring and Rabung’s systematic review and meta-analysis in JAMA was accompanied by a guardedly positive editorial entitled Psychodynamic Psychotherapy and Research Evidence: Bambi Survives Godzilla?
The opening of the editorial explained
In a now classic 1982 article, Parloff surveyed the results of psychotherapy research evidence and its relevance for policy makers and treatment reimbursement decisions, characterizing that encounter as “Bambi meets Godzilla.” He concluded that although research evidence in psychotherapy outcome at that time was “extensive and positive,” it was not responsive to the policy makers’ central question, “What kinds of psychotherapy are most effective for what kinds of problems?”
The editorial concluded:
For now, the question is: Does this new meta-analysis mean that LTPP has survived the Godzilla of the demand for empirical demonstration of its efficacy? The answer is a qualified yes. The meta-analysis was carefully performed and yielded a within-group effect size of 0.96 (95% confidence interval [CI], 0.87-1.05) for pretreatment-postreatment overall outcomes, which would be considered a large effect.
Dear JAMA editorialist, you don’t use within-group effect sizes in meta-analyses and you shouldn’t use them to evaluate psychotherapies. Everyone knows that they uniformly inflate estimates of efficacy over more appropriate between-group effect sizes. Tsk, Tsk.
After the critical response to their JAMA article, Leichsenring and Rabung published a redoing of it in British Journal of Psychiatry. They justified this redundant publication as being an update. Actually, nine of the ten studies reviewed in the BJP article were included in the previous JAMA paper. The tenth study was irrelevant to the evaluation of LTPP versus other psychotherapies and, besides, its publication was early enough to be included in the JAMA review.
How did Leichsenring and Rabung get their terribly flawed article into JAMA with an accompanying editorial? How did such an overlapping article then get into BJP? A little help from friends?
Leichsenring and Rabung have just published another review in The Lancet Psychiatry. Its title suggests an echoing of the JAMA editorial’s Bambi meets Godzilla theme Psychodynamic therapy meets evidence-based medicine.
Leichsenring, F., Luyten, P., Hilsenroth, M. J., Abbass, A., Barber, J. P., Keefe, J. R., … & Steinert, C. (2015). Psychodynamic therapy meets evidence-based medicine: a systematic review using updated criteria. The Lancet Psychiatry, 2(7), 648-660.
The abstract suggests past problems have been overcome:
Psychodynamic therapy (PDT) is an umbrella concept for treatments that operate on an interpretive-supportive continuum and is frequently used in clinical practice. The use of any form of psychotherapy should be supported by sufficient evidence. Efficacy research has been neglected in PDT for a long time. In this review, we describe methodological requirements for proofs of efficacy and summarise the evidence for use of PDT to treat mental health disorders. After specifying the requirements for superiority, non-inferiority, and equivalence trials, we did a systematic search using the following criteria: randomised controlled trial of PDT; use of treatment manuals or manual-like guidelines; use of reliable and valid measures for diagnosis and outcome; adults treated for specific mental problems. We identified 64 randomised controlled trials that provide evidence for the efficacy of PDT in common mental health disorders. Studies sufficiently powered to test for equivalence to established treatments did not find substantial differences in efficacy. These results were corroborated by several meta-analyses that suggest PDT is as efficacious as treatments established in efficacy. More randomised controlled trials are needed for some mental health disorders such as obsessive-compulsive disorder and post-traumatic stress disorder. Furthermore, more adequately powered equivalence trials are needed.
But, alas, post publication peer review to the rescue. The credibility of this article was demolished in a brief, but effective PubMed Commons commentary by Stefan G. Hofmann, Nora Esser, and Giovanbattista Andreoli.
The study by Leichsenring and colleagues highlights the importance of considering the quality of the studies that are included in a meta-analysis when evaluating the results. The Cochrane Collaboration’s Tool (Higgins et al., 2011) is a commonly-used instrument to quantify the risk of bias using the following criteria: allocation sequence concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective outcome reporting. We analyzed the 64 randomized controlled trials of manual-guided PDT for specific mental disorders that were used in the review by Leichsenring et al (see Table 1). Thirty studies showed risk biases in sequence generation, 54 in allocation concealment, and 31 in the blinding conditions. Only one of the studies showed no obvious biases. Our results suggest that the studies included in Leichsenring’s meta-analysis were of poor quality, essentially invalidating the authors’ results and making the findings meaningless. Table 1: http://issuu.com/gvand/docs/quality_ratings_of_studies_in_leich/1 Table 2: http://issuu.com/gvand/docs/description_and_results_of_studies/1 References: Higgins, J.P., Altman, D.G., Gøtzsche, P.C., Jüni, P., Moher, D., Oxman, A.D., Savovic, J., Schulz, K.F., Weeks, L., Sterne, A.C., Cochrane Bias Methods Group, Cochrane Statistical Methods Group (2011). The Cochrane Collaboration´s tool for assessing risk of bias in randomised trials. RESEARCH METHODS & REPORTING, 343.)
We should always skeptical when authors clearly with a dog in the fight undertake systematic reviews and meta-analyses evaluating their favorite treatments. It’s best to get a couple of degrees of separation. A group that included John Ioannidis is provided such a review of LTPP. The results were not favorable:
The recovery rate of various mental disorders was equal after LTPP or various control treatments, including treatments without a specialized psychotherapy component. Similarly, no statistically significant differences were found for the domains target problems, general psychiatric problems, personality pathology, social functioning, overall effectiveness or quality of life.
Control conditions were heterogeneous and frequently of low quality, e.g. without a specialized psychotherapy component. If anything, this suggests that LTPP is often compared against relatively ineffective “straw man” comparator… LTPP comparisons to specialized non-psychodynamic treatments, like dialectical behavior therapy and schema-focused therapy, suggest that LTPP might not be particularly effective.
My past blog posts about Leichsenring and Rabung’s systematic reviews and meta analyses
Flawed, Biased Review Article in British Journal of Psychiatry. December 16, December 16, 2011
Is Long-Term Psychodynamic Psychotherapy Worthwhile? January 14, 2012