What We Need to Do to Redeem Psychotherapy Research

[Update] This blog post has now been expanded into an article in Journal of Evidence-Based Psychotherapies that can be downloaded at ResearchGate. Commentaries on this article by E. David  Klonsky and Bruce Thyer are available at the journal site.

This post serves as a supplement to one in PLOS Mind the Brain, Salvaging Psychotherapy Research: a Manifesto. The Mind the Brain post declares

We need to shift the culture of doing and reporting psychotherapy research. We need to shift from praising exaggerated claims about treatment and faux evidence generated to promote opportunities for therapists and their professional organizations.  Instead, it is much more praiseworthy to provide  robust, sustainable, even if more modest claims and to call out hype and hokum in ways that preserve the credibility of psychotherapy.

The current post provides documentation in the form of citations and further links for the points made there concerning the need to reform the psychotherapy research literature.

Many studies considered positive, including those that become highly cited, are basically null trials.

spin noTwo examples of null trials became highly cited because of spin.

Bach, P., & Hayes, S. C. (2002). The use of acceptance and commitment therapy to prevent the rehospitalization of psychotic patients: a randomized controlled trial. Journal of consulting and clinical psychology, 70(5), 1129.

Discussed in these blog posts:

More on the Acceptance and Commitment Therapy Intervention That Failed to Reduce Re-Hospitalization.

Study Did Not Show That Brief Therapy Kept Psychotic Patients Out of Hospital

Here is another trial spun and dressed up:

Dimidjian, S., Hollon, S. D., Dobson, K. S., Schmaling, K. B., Kohlenberg, R. J., Addis, M. E., … & Jacobson, N. S. (2006). Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of consulting and clinical psychology, 74(4), 658.

The Dimidjian et al trial launched interest in behavior activation as a Third Wavelipstick-pig psychotherapy and has been cited times, almost always uncritically. What could possibly be wrong with the study? I will have to blog about that sometime, but check it out, using the link to the PDF that I provided. Hint: whatever happened to the obviously missing presentation for main time x treatment interactions for the primary outcome? What was emphasized instead and why?

positive spin 2Spin starts in abstracts

Discussed in a pair of blog posts :

Investigating the Accuracy of Abstracts: An Introduction

Dissecting a Misleading Abstract

When controls are introduced for risk of bias or investigator allegiance, affects greatly diminish or even disappear.

An example is

Jauhar, S., McKenna, P. J., Radua, J., Fung, E., Salvador, R., & Laws, K. R. (2014). Cognitive-behavioural therapy for the symptoms of schizophrenia: systematic review and meta-analysis with examination of potential bias. The British Journal of Psychiatry, 204(1), 20-29.

For an interesting discussion of how much meta-analyses of the same literature can vary in conclusions whether or not risk of bias and investigator allegiance are taken into account, see

Meta-Matic: Meta-Analyses of CBT for Psychosis

Conflicts of interest associated with authors having substantial financial benefits at stake are rarely disclosed in the studies that are reviewed or the meta-analyses themselves.

I recently blogged about these two articles

Sanders, M. R., Kirby, J. N., Tellegen, C. L., & Day, J. J. (2014). The Triple P-Positive Parenting Program: A systematic review and meta-analysis of a multi-level system of parenting support. Clinical psychology review, 34(4), 337-357.


Sanders, M. R., & Kirby, J. N. (2014). Surviving or Thriving: Quality Assurance Mechanisms to Promote Innovation in the Development of Evidence-Based Parenting Interventions. Prevention Science, 1-11.


Critical analysis of a meta-analysis of a treatment by authors with financial interests at stake


Are meta-analyses done by promoters of psychological treatments as tainted as those done by Pharma?

And here.

Sweetheart relationship between Triple P Parenting and the journal Prevention Science?

There are low thresholds for professional groups such as the American Psychological Association Division 12 and governmental organizations such as the US Substance Abuse and Mental Health Services Administration (SAMHSA) declaring treatments to be “evidence-supported.”

I blogged about this.

Troubles in the Branding of Psychotherapies as “Evidence Supported”

Professional groups have conflicts of interest in wanting their members to be able to claim the treatments they practice are evidence-supported.

I have blogged about the Society for Behavioral Medicine a number of times

Faux Evidence-Based Behavioral Medicine at Its Worst (Part I)

Faux Evidence-Based Behavioral Medicine Part 2

Does psychotherapy work for depressive symptoms in cancer patients?

Some studies find differences between two active, credible treatments

This does not happen very often, but here is such a study:

Poulsen, S., Lunn, S., Daniel, S. I., Folke, S., Mathiesen, B. B., Katznelson, H., & Fairburn, C. G. (2014). A randomized controlled trial of psychoanalytic psychotherapy or cognitive-behavioral therapy for bulimia nervosa. American Journal of Psychiatry, 171(1), 109-116.

that I blogged about

When Less is More: Cognitive Behavior Therapy vs Psychoanalysis for Bulimia

Bogus and unproven treatments are promoted with pseudoscientific claims.

Here is a website offering APA approved continuing education credit for Somatic Experiencing

Somatic Experiencing® is a short-term naturalistic approach to the resolution and healing of trauma developed by Dr. Peter Levine. It is based upon the observation that wild prey animals, though threatened routinely, are rarely traumatized. Animals in the wild utilize innate mechanisms to regulate and discharge the high levels of energy arousal associated with defensive survival behaviors. These mechanisms provide animals with a built-in ‘’immunity’’ to trauma that enables them to return to normal in the aftermath of highly ‘’charged’’ life-threatening experiences.

Declarations of conflicts of interest are rare and exposure of authors who routinely failed to disclose conflicts of interest is even rarer.

I blogged about this:

Sweetheart relationship between Triple P Parenting and the journal Prevention Science?

Departures from preregistered protocols in published reports of RCTs are common, and there is little checking of discrepancies in abstracts from results that were actually obtained or promised in preregistration.

Here is a notable recent example about which I blogged here, here and here.

Morrison, A. P., Turkington, D., Pyle, M., Spencer, H., Brabban, A., Dunn, G., … & Hutton, P. (2014). Cognitive therapy for people with schizophrenia spectrum disorders not taking antipsychotic drugs: a single-blind randomised controlled trial. The Lancet, 383(9926), 1395-1403.

See also

Milette, K., Roseman, M., & Thombs, B. D. (2011). Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: a systematic review. Journal of psychosomatic research, 70(3), 205-217.

Specific journals are reluctant to publish criticism of their publishing practices.

Cook JM, Palmer S, Hoffman K, Coyne JC. Evaluation of clinical trials appearing in Journal of Consulting and Clinical Psychology: CONSORT and beyond. The Scientific Review of Mental Health Practice. 2007;5,69-80.

This article attempted to point out shortcomings in the reporting of clinical trials In Journal of Consulting and Clinical Psychology and was first submitted there and rejected. As you can see, we in no way intended to bash the journal, but to highlight the need for adopting and enforcing CONSORT. The article was sent out to review to two former editors who understandably took issue with its depiction of the quality of clinical trials they had accepted for publication. Fortunately, we were able to publish this article elsewhere when JCCP rejected it.

Those of us around on the listservs in the early 2000s can recall how aggressively APA resisted adoption of CONSORT. Finally APA relented with

Guidelines seek to prevent bias in reporting of randomized trials

But it contained an escape clause. All authors had to do was fail to declare their study was a RCT. But making that disclosure is part of adhering to CONSORT!.

Authors of APA journal articles who call a clinical trial a “randomized controlled trial” (RCT) are now required to meet the basic standards and principles outlined in the Consolidated Standards of Reporting Trials (CONSORT) guidelines as part of an effort to improve clarity, accuracy and fairness in the reporting of research methodology.

We complained and the escape clause was eliminated, even if enforcement of CONSORT remained spotty.

Coyne, J. C., Cook, J. M., Palmer, S. C., & Rusiewicz, A. (2004). Clarification of clinical trial standards. Monitor on Psychology: A Publication of the American Psychological Association, 35(11), 4-8.

If a title or abstract of a paper reporting a RCT does not explicitly state “randomized clinical trial,” there is risk it will be lost in any initial search of the literature. We propose that if editors and reviewers recognize that a study reports a randomized clinical trial, they will require that authors label it as such and that they respond to the CONSORT checklist.


No more should underpowered in exploratory pilot feasibility studies be passed off as RCTs when they achieve positive results.

An excellent discussion of this issue can be found in

Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry, 63(5), 484-489.


Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role and interpretation of pilot studies in clinical research. Journal of psychiatric research, 45(5), 626-629.

Evaluations of treatment effects should take into account prior probabilities suggested by the larger literature concerning comparisons between two active, credible treatments. The well-studied treatment of depression literature suggests some parameters.

Cuijpers, P., & van Straten, A. (2011). New psychotherapies for mood and anxiety disorders: necessary innovation or waste of resources?. Canadian journal of psychiatry. Revue canadienne de psychiatrie, 56(4), 251.

On the one hand, there is a clear need for better treatments, as mood and anxiety disorders constitute a considerable burden for patients and society. Further, modelling studies have shown that current treatments can reduce only one-third of the disease burden of depression and less than one-half of anxiety disorders, even in optimal conditions.2

However, there are already dozens of different types of psychotherapy for mood and anxiety disorders, and there is very little evidence that the effects of treatments differ significantly from each other. In depression, we found that interpersonal psychotherapy is somewhat more effective than other therapies,3 but differences were very small (Cohen’s d < 0.21) and the clinical relevance is not clear. In the field of anxiety disorders, there is evidence that relaxation is less effective than cognitive-behavioural therapy, but there is very little evidence for significant differences between other therapies.

We think that new therapies are only needed if the additional effect compared with existing therapies is at least d = 0.20. Larger effect sizes are not reasonable to expect as 0.20 is the largest difference between therapies found until now. Further, this effect needs to be empirically demonstrated in high-quality trials.

However, to show such an effect of 0.20 we would need huge numbers. A simple power calculation shows that this would require a trial of about 1000 participants (STATA[Statacorp, College Station, TX] sampsi command). As a comparison, the large National Institute of Mental Health Treatment of Depression Collaborative Trial examining the effects of treatments of depression included only 250 patients.


Adverse events and harms should routinely be reported.

Vaughan, B., Goldstein, M. H., Alikakos, M., Cohen, L. J., & Serby, M. J. (2014). Frequency of reporting of adverse events in randomized controlled trials of psychotherapy vs. psychopharmacotherapy. Comprehensive Psychiatry.

Meta-analyses of psychotherapy should incorporate p-hacking techniques.

This is discussed in

Lakens, D., & Evers, E. R. (2014). Sailing From the Seas of Chaos Into the Corridor of Stability Practical Recommendations to Increase the Informational Value of Studies. Perspectives on Psychological Science, 9(3), 278-292.

evidence based

5 thoughts on “What We Need to Do to Redeem Psychotherapy Research

  1. doesn’t the dodo hypothesis pretty much hold up? hasn’t wampold etal shown that? isn’t it sufficient to say psychotherapy is a moderately effective treatment much of the time and leave it at that. do we need more studies in this area? hasn’t psychotherapy reached it’s end point of usefulness?


    • I think that Wampold and colleagues overstate the equal effectiveness of all therapies. There’s been lots of criticism of their particular comparisons on which they base their conclusion. Certainly for OCD, PTSD, and some eating disorders, some treatments are better than others. And I think that following some kind of manual with some kind of time limit within which treatment is evaluated is an important alternative to just trying everything without a specified limit to the treatment.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s