Serious errors in Stephen Lewandowsky’s paper in PLOS One

Stephan Lewandowsky

Stephan Lewandowsky

Comment left at my blog post, Further insights into war against data sharing: Science Media Centre’s letter writing campaign to UK Parliament:

Lewandowsky’s modus operandi, played out previously over the period 2010-2013 in a different context, and now in replay goes as follows:

  1. Find a group of people I disagree with.

  2. Poke them with a sharp stick, making vague unsubstantiated allegations.

  3. Wait for the inevitable response.

  4. Feign surprise at the response, and continue goading (see his comment at Nature, “This surprised us…”) and encouraging speculation.

  5. Monitor and collect the angry responses.

  6. Write a paper saying that this group of people are prone to irrational conspiracist ideation.

As reported in José Duarte’s Minors, Lewandowsky, and Ceremonial Ethics:

s200_joe.duarte

Enter a caption

José Duarte

plos oneThe Lewandowsky, Gignac, and Oberauer paper in PLOS ONE has been substantially corrected. I had alerted the journal last fall that there were serious errors in the paper, including the presence of a 32,757-year-old in the data, along with a 5-year-old and six other self-reported minors. The paleoparticipant in particular had knocked out the true correlation between age and the conspiracy belief items (the authors had reported that there was no correlation between age and those items.) See my original essay on this paper, what the bad data did to the age correlations, and lots of other issues here.

Deeply troubling issues remain. The authors have been inexplicably unwilling to remove the minors from their data, and have in fact retained two 14-year-olds, two 15-year-olds, a 16-year-old, and a 17-year-old. This is strange given that the sample started with 1,001 participants. It is also wildly unethical.

To provide some context, here’s the timeline:

October 4, 2013: A layperson alerted Lewandowsky – on his own website – that there was a 32,757-year-old and a 5-year-old in his data.

There was no correction. Recall that he had reported analyses that included age variable in the paper, and that these analyses were incorrect because of the 32,757-year-old.

August 18, 2014: On the PLOS ONE page for the paper, I alerted the authors to the 32,757-year-old, the 5-year-old, and the six other minors in their data (along with several other problems with the study.)

There was no correction.

September 22, 2014: I contacted PLOS ONE directly and reported the issue. I had waited over a month for the authors to correct their paper after the notification on August 18, but they had mysteriously done nothing, so it was time to contact the journal.

August 13, 2015: Finally, a correction was published. It is comprehensive, as there were many errors in their analyses beyond the age variable.

I’d like to pause here to say that PLOS ONE is beautiful and ethically distinctive. They insisted that the authors publish a proper correction, and that it thoroughly address the issues and errors in the original. They also placed a link to the correction on top of the original paper. The authors did not want to issue a proper correction. Rather, Lewandowsky preferred to simply post a comment on the PLOS ONE page for the paper and call it a corrigendum. This would not have been salient to people reading the paper on the PLOS ONE page, as it requires that one click on the Comments link and go into the threads. Notably, Lewandowsky’s “corrigendum” was erroneous and required a corrigendum of its own… It was also strangely vague and uninformative.

A serious ethical issue remains – they kept the minors in their data (except the 5-year-old.) They had no prior IRB approval to use minors, nor did they have prior IRB approval to waive parental consent. In fact, the “ethics” office at the University of Western Australia appears to be trying to retroactively approve the use of minors as well as ignoring the issue of parental consent. This is ethically impossible, and wildly out of step with human research ethics worldwide. It also cleanly contradicts the provisions of the Australian National Statement on Ethical Conduct of Human Research (PDF). In particular, it contradicts paragraphs 4.2.7 through 4.2.10, and 4.2.12. The conduct of the UWA ethics office is consistent with all their prior efforts to cover up Lewandowsky’s misconduct, particularly with respect to Lewandowsky’s Psych Science paper, which should be treated as a fraud case. UWA has refused everyone’s data requests for that paper, and has refused to investigate. Corruption is serious problem with human institutions, one that I increasingly think deserves a social science Manhattan Project to better understand and ameliorate. UWA is a classic case of corruption, one that mirrors those reported by Martin.

Here is the critical paragraph regarding minors in the PLOS ONE correction:

“Several minors (age 14–17) were included in the data set for this study because this population contributes to public opinions on politics and scientific issues (e.g. in the classroom). This project was conducted under the guidelines of the Australian National Health and Medical Research Council (NH&MRC). According to NH&MRC there is no explicit minimum age at which people can give informed consent (as per https://www.nhmrc.gov.au/book/chapter-2-​2-general-requirements-consent). What is required instead is to ascertain the young person’s competence to give informed consent. In our study, competence to give consent is evident from the fact that for a young person to be included in our study, they had to be a vetted member of a nationally representative survey panel run by uSamp.com (partner of Qualtrics.com, who collected the data). According to information received from the panel provider, they are legally empowered to empanel people as young as 13. However, young people under 15 are recruited to the panel with parental involvement. Parental consent was otherwise not required. Moreover, for survey respondents to have been included in the primary data set, they were required to answer an attention filter question correctly, further attesting to their competence to give informed consent. The UWA Human Rights Ethics Committee reviewed this issue and affirmed that “The project was undertaken in a manner that is consistent with the Australian National Statement of Ethical Conduct in Human Research (2007).”

The above may be difficult for people to parse and unpack. Here are the essentials we can extract from it:

1. There was no prior IRB approval for the use of minors. (UWA’s review was retroactive, amazingly.)

2. Parental consent was not obtained for minors who were at least 15 years of age.

3. Obtaining parental consent for 13 and 14-year-olds was delegated to a market research company. However, the term “consent” is not used in this case. Rather, the authors claim that the market research company recruited these kids with “parental involvement”. It’s not clear what this term means.

4. The UWA “ethics” committee is attempting to grant retroactive approval for the use of minors and the lack of parental consent, as well as the delegation of consent obtainment to a market research company. They cite the National Statement of (sic) Ethical Conduct in Human Research, even though it contains no provision for retroactive approvals or cover-ups. In fact, the Statement does not contemplate such absurdities at all.

5. In the opening sentence, they claim they intended to use minors. I don’t believe that for a minute. If they intended to use minors, they would have sought IRB approval to do so (before conducting their study.) Second, if they intended to use minors, they would not have only seven minors. Third, there is no mention of minors in the original paper, nor any mention of this notion of high school classrooms as the public square.

Facts 1 through 4 are revolutionary. This is an ethical collapse. Researchers worldwide would be stunned to hear of this. No IRB approval for the use of minors? No parental consent? A new age threshold of 15 for parental consent, and 13 for participation? Delegating parental consent to a market research company? An IRB acting as a retroactive instrument? An IRB covering up the unapproved use of minors? I’m not sure we’ve ever encountered any one of these things. Having all of these happen at the same time is a singularity, an ethical event horizon that dims the sun.

Notably, their citation of the NH&MRC page is a sham. The page makes no mention of age or minimum ages. It ultimately defers to Chapter 4.2, which takes for granted that there is IRB approval to use minors, as well as parental consent. (See the Respect and Standing Parental Consent sections.) It does not contemplate a universe where IRB approval is not obtained. It’s extremely disturbing that staff at UWA would try to deceive the scientific community with a sham citation.

I contacted UWA about these issues some months ago. As far as I can tell, they refuse to investigate. It’s as though their ethics office is specifically designed to not investigate complaints if they think they can escape scrutiny and legal consequences. Mark Dixon of the UWA anti-ethics office said the following in an e-mail:

“However, this project was designed for a general demographic. Surveys targeted to a general population do not prohibit the collection of data from minors should they happen to respond to the survey.”

“You are probably aware that the survey written up in the article was an online survey, where consent is indicated by the act of taking the survey.”

“Inclusion or omission of outliers, such as the ‘5 year old’ and the ‘32,000 year old’, are reasonable scholarship when accompanied by explanatory notes. However, it would be unusual to actually delete data points from a data-set, so I don’t understand your concern about the remaining presence of such data-points in the data-set.”

“You expressed concern that the survey “… did not even ask participants for their ages until the end of the study, after participation and any “consent” had been secured”. Demographic information is routinely collected at the end of a survey. This is not an unusual practice.”

To say that these statements are alarming is an understatement. He thinks research ethics doesn’t apply to online studies. He thinks we don’t need to obtain consent for online studies, that simply participating is consent. He thinks 5-year-olds and 32,757-year-olds are “outliers” and that it is reasonable to retain them (is he aware that the age variable was analyzed?) He thinks researchers can ask someone’s age at the end of a study. This person retains the title “Associate Director (Research Integrity)”, yet he appears to know nothing of research or research integrity. The best explanations here are that he has no training in human research ethics and/or he’s corrupt. This is such an extraordinary case.

For lay readers, let me note the following:

1. An online study is a study like any other study. The same research ethics apply. There’s nothing special about an online study. Whether someone is sitting in front of a computer in a campus lab, or in their bedroom, the same ethical provisions apply.

2. We always require people to be at least 18 years of age, unless we are specifically studying minors (which would require explicit IRB approval).

3. We always include a consent form or information letter at the start of an online study. This form explains the nature of the study, what participants can expect, how long it should take, what risks participation may pose to the participant, any compensation they will receive, and so forth. Notably, the form will explicitly note that one must be at least 18 to participate.

4. We always ask age up front, typically the first page after a person chooses to participate (after having read the consent or information sheet.)

5. We always validate the age field, such that the entered age must be at least 18 (and typically we’ll cap the acceptable age at 99 or so to prevent fake ages like 533 or 32,757.) All modern survey platforms offer this validation feature. A person cannot say that they are 5 years old, or 15 years old, and proceed to participate in an IRB-approved psychology study (where approval to use minors was not granted.)

We can’t usually do anything about people who lie about their ages (relatedly, I doubt the 5-year-old age was accurate, but it won’t matter in the end.) This has nothing to do with online studies. The manner in which people report their age is the same in online and in-person studies – they’re seated in front of a computer and self-report their age. But if someone tells us that they are minors, their participation ends at that moment. Because of this, there should never be minors or immortals in our data. If we wanted to card people, we could – both in-person and online (see Coursera’s ID validation) – but there’s a trade-off between age verification and anonymity.

6. Note that DIxon’s first claim contradicts the Correction’s claim that they intended to use minors. Note also that Dixon’s claim reduces to: If approval to use minors is not obtained, that’s when you’re allowed to use minors. He’s saying that in a study not intended for minors, you’re allowed to have them in your data. That would open an interesting and catastrophic loophole – we could always use minors simply by not asking for approval to use them…

At this point, I think PLOS ONE should just retract the paper. The paper is invalid anyway, but we can’t have unapproved – or retroactively approved – minors in our research. UWA is clearly engaged in a cover-up, and their guidance should not inform PLOS ONE‘s, or any journal’s, decisions. This exposes a major structural ethical vulnerability we have in science – we rely on institutions with profound conflicts of interest to investigate themselves, to investigate their own researchers. We have broad evidence that they often attempt to cover up malpractice, though the percentages are unclear. Journals need to fashion their own processes, and rely much less on university “finders of fact”. We should also think about provisioning independent investigators. The standards in academic science are much lower than in the private sector (I used to help companies comply with Sarbanes-Oxley.) In any case, UWA’s conduct deserves to be escalated and widely exposed, and it will be. This is far from over – we can’t ignore the severity of the ethical breaches here, and we won’t.

I doubt minors were harmed in this case, but institutions that would do what these people did are at risk of putting minors – and adults – in harm’s way. Throwing out the rules regarding minors in research is a trivial recipe for a long list of bad outcomes, as is clinging to the nonsense that research ethics don’t apply to online studies. These people are a disgrace and a menace.

APPENDIX A: UWA Personnel Who Were Alerted to the Minors

Murray Maybery, Head of the Department of Psychology: I contacted him on August 20, 2014, informing him of the minors in the data, as well as other issues. He did not respond.

Vice-Chancellor Paul Johnson: I contacted him (and Maybery again) on January 19, 2015, informing them of the minors, and requesting the full data. Neither of these men responded.

I tried again on February 15, 2015, and Maybery pointed me to the stripped data file for the related Psych Science study, essentially mocking my explicit request for the full data (I had made clear that the available data was stripped, yet he still referred me to it without comment. There’s some real ugliness in Perth.) Everyone knows about the stripped data (Lewandowsky stripped the age variable from the Psych Science study data file, along with other variables like gender and his Iraq War conspiracy item, an item whose inclusion weakens his theories – the paper says his minimum cutoff age was 10. He won’t release the full data.) Maybery ignored my reports on the presence of minors, had nothing to say about it.

Johnson never responded at any point, and was revealed to be cartoonishly corrupt in a similar episode. More recently, he was in the media for revoking Bjorn Lomborg’s job offer because Lomborg likes to do cost-benefit analyses on climate change mitigation, and this is evidently controversial. This whole situation is extraordinary – only once before in my life have I encountered corruption that stilled and chilled me the way UWA has. I used to be a Court Appointed Special Advocate (CASA) for abused, abandoned, or neglected children in CPS care, and I’ll write about an episode from that experience soon.

I’ll update the list as events dictate, and document everything more thoroughly in other venues. I’ll publish similar reports on Psych Science and APS’ conduct when informed of minors and other issues, and in other venues. I haven’t posted much on these issues lately because my own work comes first, but I would never let these issues go – I prefer to be slow and methodical on these issues anyway, to give people a long time to do their jobs, to take my time to understand what I’m looking at, etc.

APPENDIX B: QUICK VALIDITY NOTE

The study was invalid because of the low quality and bias of the items, as well as the fact that some of the authors’ composite variables don’t actually exist.

In social psychology, our constructs don’t always exist in the sense that trees, stars, and the Dallas Cowboys exist. They’re often abstractions that we infer based on sets of survey items or other lower-level data. I could put together a scale and call it Free Market Views, but that doesn’t necessarily mean I’m measuring free market views. It has to be validated. People should read my items and make sure they actually refer to free market views. They should make sure my items are not biased. They should make sure that my items are clear, single-barrelled, and understandable to virtually all participants. They should run factor analyses to see if my items are measuring the same thing, or perhaps measuring more than one thing. Well, I should do all those things, but so should other people.

Journalists should be mindful of this reality: Just because someone says they’ve measured free market views, and says that “free market views” predict this and that, doesn’t make it so. Remember, this is not like trees, stars, and the Dallas Cowboys. You need due diligence. You need to look at someone’s items and data and think about it, reason about it.

So what happened here? Well, there were five purported free market items. A simple factor analysis reveals that there are two strong factors in those five items. The strongest factor would credibly be called Environmentalism. The following items load on this factor:

Free and unregulated markets pose important threats to sustainable development.

The free market system is likely to promote unsustainable consumption.

The free market system may be efficient for resource allocation but it is limited in its capacity to promote social justice.

The second factor carries these items:

An economic system based on free markets unrestrained by government interference automatically works best to meet human needs.

The preservation of the free market system is more important than localized environmental concerns.

Both factors include environmentalist items, but the first factor includes two of them, those two items load more heavily on that factor than the third item does, and that factor explains 43 percent of the variance in scores, while the second factor explains 25 percent. The second factor is distinct in carrying what I’d call the extremist items: unrestrained free markets “automatically work best” to meet (unspecified) “human needs”; and preservation of free markets is more important than (unspecified) localized environmental concerns.

The whole scale is a mess because the items are a mess. For example, this item is not only double-barrelled (which we avoid), but the barrells point in opposite directions (which I’ve never seen before):

The free market system may be efficient for resource allocation but it is limited in its capacity to promote social justice.

Which barrell wins? It’s too complicated on several grounds. The term “social justice” is a leftist concept. The rest of us don’t use it, don’t grant its conception of justice, its definition of the good. In other words, not everyone thinks social justice is just. Perhaps more importantly, people who don’t use the term might not know what it means. Lastly, agreement with the double-barrelled item was scored as disagreement with free markets, yet I suspect many pro-market types would agree with it (if they know what social justice entails.)

The scale in general is politically biased. It’s written in the language of the left, particularly in the language of environmentalists, with concepts like “sustainability” — another abstraction that non-environmentalists will not necessarily grant as a valid or useful concept, or even understand. It’s not nearly as bad the invalid SDO and RWA scales, but it’s decently bad. We could never assume we’re getting a clean read on free market attitudes with this scale. It’s got too much environmentalism, left-wing language, and poor item construction. It should never be used. We don’t really know what we’re looking at when we see a correlation between this scale and some other scale or view – our best guess is that we’re looking at the relation between environmentalism and Y.

pattern matrixHere’s the pattern matrix from a PCA with Direct Oblimin rotation, eigenvalues greater than one. I included all the political items, not just the purported “free market” scale, because we can’t assume a priori how we should slice them. It looks like we’d drop the communism item (it’s probably on an island because no one talks about communism anymore) and perhaps the socialism item for similar reasons. This is stuff that researchers would normally do before reporting, or even analyzing, substantive results. Evidently, Lewandowsky, Gignac, and Oberauer didn’t do the work – they just asserted composite variables that include the items they want to include, but that’s not normally how social psychology works:

Substantive note: The correction does not address one of the substantive errors in the original. Gender is the largest predictor of GMO attitudes. They never reported this, but rather implied that gender did no work. A lot of times boring variables like age and gender explain a lot of variance, and in this case gender explained more than any other. (Women trusted GMOs less, using Lewandowsky’s primitive linear correlations on the scale index. It’s unclear whether women actually distrusted GMOs – i.e. where the women clustered on the items. A correlation doesn’t tell you this. A bad researcher would say “women distrusted GMOs” given a negative correlation coefficient, without specifying descriptives or their actual, substantive placement on the scale, which could in fact be pro-GMO, just less pro than men.)

 

Further insights into war against data sharing: Science Media Centre’s letter writing campaign to UK Parliament

plos oneMy requesting the PACE trial data is much simpler than it is being portrayed. The PACE investigators promised the data would be available upon request as a condition for publishing in PLOS One. No one forced Peter White and colleagues to publish in an open access journal committed to data sharing, but by doing so they incurred an obligation. So, they should simply turn over the data.

Of course, providing me with the data would involve the risk of my analyses exposing what they have been hiding or falsely claiming.

The PACE investigators have thus far refused to turn over the data. They have treated my request as falling under a Freedom of Information request, which it does not. But with my request conceptualized in this way, they denied it because they deemed my motives in seeking the data “vexatious,” a strategy that they have used with other requests.

The PACE investigators are playing hardball.

Stephan Lewandowsky

Stephan Lewandowsky

Peter White previously enlisted Stephan Lewandowsky to disseminate a misrepresentation of the PACE investigators’ commitment to transparency:

 

Some requests were complied with, so long as they did not compromise medical confidentiality, future publications, or academic safe space to deliberate…Our deliberate policy, to help allay concerns about the trial, was to be as transparent as possible regarding what we did, while also protecting medical confidentiality and our staff and patient [p 17].

We know of course this isn’t true. The collusion of Lewandowsky in this caper undercuts his claim that he and Dorothy Bishop are not ‘intimately familiar with the details’ of the PACE Trial. (See comments on their Nature article, Research integrity: Don’t let transparency damage science) Lewandowsky’s claim was made in reaction to the two of them being criticized for lumping those who seek the PACE data, like me, with science deniers who must be resisted:

Orchestrated and well-funded harassment campaigns against researchers working in climate change and tobacco control are well documented. Some hard-line opponents to other research, such as that on nuclear fallout, vaccination, chronic fatigue syndrome or genetically modified organisms, although less resourced, have employed identical strategies.

Is there a remarkable coincidence in Lewandowsky and Bishop attempting to make data sharing so complicated, just when the PACE investigators are urgently seeking reasons not to share their data? Here is some evidence that it is not a coincidence.

Dorothy Bishop is an adviser to the Science Media Centre  founded by Simon Wessely. The SMC is coordinating a letter writing campaign to Parliament instigated by Peter White attempting to get an exclusion from the Freedom of Information Act for request for data.

What was revealed by a response to a request made under the Freedom of Information Act

A FOIA request produced an email from Fiona Fox of the Science Media Centre dated October 30, 2013:

Those of you, like the SMC, a worried about the malicious use of FOIA against researchers may like to see this and use it in your own efforts. I’ve just been briefed about [redacted] FOIing a number of universities about the primate research [redacted] is a convicted animal rights extremists!!! Surely this was never the intent of FOI!!!

Cheers

Fiona

The email is attached to a further thread of emails. The source for the October 28, 2013 email is redacted but the CC to Fiona Fox is revealed along with the subject letter:

Dear all.

As you can read, QMUL are lobbying MPs to get a good FOIA exemption passed into law that properly defends research. Please use this as a basis for asking your own Universities and other likely lobbyists to get in touch with their own or your own friendly MPs.

Best wishes,

Peter

I think when you read the letter template, you can see that there is no doubt that the source is Queen Mary University of London’s Peter White.

The template letter that is attached for sending to Parliament attempts specifically to get an exemption for the PACE trial:

Dear XXX,

The Intellectual Property bill currently going through Parliament includes a proposed exemption for current research from disclosure under the Freedom of Information Act requests and would prevent the premature release of data in academic research.

We very much welcome this necessary exemption as we believe that it will do much to avoid the significant risks to academic freedom presented under the current pre-publication exemption of the Act. We also believe that it will help ensure the ability and willingness of researchers to engage in free inquiry into important areas of research.

We are writing to you because we are concerned that the bill as it stands may not fully clarify when the exemption from requests applies. In many cases, a research project proceeds via a series of publications, rather than one alone, progressively investigating and reporting outcomes. This, for example, is the case with the P/VCE Trial, an important study into the safety and efficacy of various treatments for Chronic Fatigue Syndrome in which QMUL researchers have participated for the last 10 years. Here there have been separate papers, published according to a defined research and publication strategy, that have addressed, or will address in sequence the main results of the effectiveness and safety of the treatments, their cost-effectiveness, long-term follow-up and the mediators of the treatments to explain how they work. [Emphasis added]

It is not immediately clear from the bill as it stands whether information from such a continuing series of publications within a given research project would fall within the exemption beyond the first publication, since it may be argued that a “report of the research” has already been published under the present wording of proposed Section 20.

We would therefore suggest the following slight change to clarify “the programme” and “publication… of a report”:

“the programme is continuing with a view to the publication, by a public authority or any other person, of a report of the research that either includes the requested data or the analysed results thereof.” [Emphasis in the original]

We hope that you will raise this issue in Parliament so that the bill may be ensured to have its intended and necessary effect.

With best regards

XXX

My previous blog post, Glimpses into the assault on data sharing discussed how the UK Independent Commission on Freedom of Information issued a call for evidence. Peter White, submitted testimony to the UK Independent Commission on Freedom of Information some time between October 9 and December 7, 2015:

“Section 22a of the Act is insufficient protection for science into controversial subjects, and requires that the research is on-going, so is irrelevant to completed research. We need science in the UK to be protected or it will continue to be damaged as this trial has been (other examples include climate change science, and research into the health effects of tobacco). Exempting Universities from the FOIA would achieve that. Exempting scientific research data produced by Universities and other higher educational institutes might be a workable alternative.”

Another blog post, King’s College London stalls some more, reiterating refusal to release the PACE trial data discusses a press release dated December 18, 2015 in which Kings College, London reiterated its refusal to release the PACE trial data.

King’s College, London had not communicated with me directly, but I learned of the press release from a tweet by Simon Wessely. I recommend following Simon on Twitter @WesselyS to get the latest updates on the maneuvering by the PACE investigators to resist sharing their data.

The press release indicated:

We are…concerned for the rights and welfare of trial participants. Participants did not give consent to the public release of their data when they entered the trial. In particular we are concerned to ensure that there is no risk of misuse of the data such as through inadvertent personal identification. The scientists who have already received data have all signed a formal confidentiality agreement, approved by the independent PACE Trial Steering Committee, which required that they respect the confidential nature of the data, and keep them secure, as agreed with trial participants when they consented to take part. We stand by our decisions to decline two recent applications for trial data as we believe that they did not meet these requirements.

We are currently seeking further ethical and scientific advice, as well as the advice of patients, on how best to provide independent decisions about appropriate access to relevant data in a way that balances the rights of trial participants, and future progress of the trial analysis and follow up, with the public interest in releasing trial data.”

The Nature comment by Stephan Lewandowsky and Dorothy Bishop expands on these concerns. Of course, that’s only a coincidence.

In future blog posts, I will be discussing more about the Science Media Centre’s well-orchestrated campaign to protect the claims of the PACE investigators’ interpretation of their data, including by attacking the motives and character of those who criticize it and request the data for independent reanalysis.

The motive is simple. If the claims of the PACE investigators are as credible, then disabled people who are getting payments from the government can be required to get cognitive behaviour therapy as a condition for continuing their payments. Of course, it has been gradually unfolding that CBT or Graded Exercise Therapy (GET) can not get chronic fatigue/myalgic encephamyelitis patients back to work. And a course of GET makes many of them more disabled. But who cares when requirement that these patients be in such treatment can be used to deny social welfare payments to them?

smc media campaign

Science Media Centre organized media campaign inaccurately portraying benefits of CBT

A future blog post will also reveal the government and corporate funding sources  for the Science Media Centre, including energy company BP, Proctor & Gamble, Rolls-Royce and pharmaceutical company Astra Zeneca. Another blog post will discuss the origins of the smearing of critics of the PACE trial as science deniers. I’ll explain why we are being lumped with animal rights activists and climate change deniers, but not opponents of fracking.

I’m sure that many readers were as confused as I was by Dorothy Bishop’s apparent about-face from a November blog, Who’s afraid of Open Data to the recent Nature commentary with Stephan Lewandowsky. In November Bishop was saying

“Fears about misuse of data can be well-justified when researchers are working on controversial areas where they are subject to concerted attacks by groups with vested interests or ideological objections to their work. There are some instructive examples here and here. Nevertheless, my view is that such threats are best dealt with by making the data totally open. If this is done, any attempt to cherrypick or distort the results will be evident to any reputable scientist who scrutinises the data. This can take time and energy, but ultimately an unscientific attempt to discredit a scientist by alternative analysis will rebound on those who make it.  In that regard, science really is self-correcting. If the data are available, then different analyses may give different results, but a consensus of the competent should emerge in the long run, leaving the valid conclusions stronger than before.”

I think this seeming contradiction could be resolved if Dorothy Bishop publicly clarified her relationship to the Science Media Centre, including where she stands, as an adviser, on its campaign against data sharing.

 

Is psychodynamic therapy an evidence-supported treatment? Critical look at a systematic review raises doubts.

Sometimes authors misapply principles of evidence-based medicine appraisal when their interests and livelihood depend on getting it wrong. This could serve as an example.

Leichsenring, F., Luyten, P., Hilsenroth, M. J., Abbass, A., Barber, J. P., Keefe, J. R., … & Steinert, C. (2015). Psychodynamic therapy meets evidence-based medicine: a systematic review using updated criteria. The Lancet Psychiatry, 2(7), 648-660

freudFalk Leichsenring and Sven Rabung have trying for years to get the “evidence supported” label attached to psychodynamic psychotherapy (PP), especially long term psychodynamic psychotherapy (LTPP). They have received surprising assists from editors of prestigious journals like JAMA  and British Journal of Psychiatry. They have also faced some withering criticisms of their breaking of basic rules in conducting and interpreting meta analyses and systematic reviews.

I teamed up with Aaron T. Beck and others in a detailed critique entitled Is Longer-Term Psychodynamic Psychotherapy More Effective than Shorter-Term Therapies? of their earlier efforts in JAMA. We faulted the meta-analysis and systematic review for its

  • Computational mistakes.
  • Improbably large effect sizes.
  • Integration of randomized and nonrandomized trials.
  • Ignoring of the uniformly poor quality of the studies included in the review.

Some of the studies involved comparing PP to conditions in which patients did not receive therapy of any kind.

For another succinct methodological critique see

 Littell, J. H., & Shlonsky, A. (2011). Making sense of meta-analysis: A critique of “effectiveness of long-term psychodynamic psychotherapy”. Clinical Social Work Journal, 39(4), 340-346.

Leichsenring and Rabung’s reply was not responsive to our critique, but they charged I had not disclosed a conflict of interest:

[This] critique reveals a bias that is obviously related to their own approach, although they do not disclose their conflicts of interest. One wonders whether CBT advocates who have a large financial stake in propagating the view that their treatment is superior, or the only evidence-based approach, can be objective about these scientific issues. Thus, the paper by Bhar et al. [7] may serve as an excellent example of an investigator allegiance effect. We regret that in this discussion, research is in danger of being used as a weapon to defeat a real or imagined rival. Although directed against and psychodynamic psychotherapy, such a misuse can do harm to the public image of psychotherapy in general.

Yup, Falk and Sven, I spend all my time writing CBT self-help books and doing workshops, when I am not, I do what I can to harm the public image of psychotherapy.

Actually, while I consider Dr. Beck a friend, he considers me one of his most persistent critics.

dead_dodo_200Like other psychotherapies, PP seeks the label of “evidence-based” in order to secure third-party payments. The problem is that PP is longer term and more intensive than other psychotherapies claiming that label. The added length and intensity may not be cost-effective. For some conditions, such as eating disorders, the dodo bird verdict may not actually hold for PP. For instance, cognitive behavioral therapy that is less intensive but more goal- rather than insight- oriented may be more effective than PP for bulimia.

A larger issue however is that practitioners of PP have historically tended toward suspicion and even hostility toward attempts to fit their approach into the rigor (that they consider false rigor) of a randomized trial. They tend to favor psychodynamic formulations over the formal diagnostic categories and standardized measurements used in research. PP does not have a strong research tradition. So, the quantity and quality of evidence that can be mustered for its efficacy is less than for other therapies

Bambi-meets-Godzilla-513d504906daa_hires-218x300Leichsenring and Rabung’s systematic review and meta-analysis in JAMA was accompanied by a guardedly positive editorial entitled Psychodynamic Psychotherapy and Research Evidence: Bambi Survives Godzilla?

The opening of the editorial explained

In a now classic 1982 article, Parloff surveyed the results of psychotherapy research evidence and its relevance for policy makers and treatment reimbursement decisions, characterizing that encounter as “Bambi meets Godzilla.” He concluded that although research evidence in psychotherapy outcome at that time was “extensive and positive,” it was not responsive to the policy makers’ central question, “What kinds of psychotherapy are most effective for what kinds of problems?”

The editorial concluded:

For now, the question is: Does this new meta-analysis mean that LTPP has survived the Godzilla of the demand for empirical demonstration of its efficacy? The answer is a qualified yes. The meta-analysis was carefully performed and yielded a within-group effect size of 0.96 (95% confidence interval [CI], 0.87-1.05) for pretreatment-postreatment overall outcomes, which would be considered a large effect.

Dear JAMA editorialist, you don’t use within-group effect sizes in meta-analyses and you shouldn’t use them to evaluate psychotherapies. Everyone knows that they uniformly inflate estimates of efficacy over more appropriate between-group effect sizes. Tsk, Tsk.

After the critical response to their JAMA article, Leichsenring and Rabung published a redoing of it in British Journal of Psychiatry. They justified this redundant publication as being an update. Actually, nine of the ten studies reviewed in the BJP article were included in the previous JAMA paper. The tenth study was irrelevant to the evaluation of LTPP versus other psychotherapies and, besides, its publication was early enough to be included in the JAMA review.

How did Leichsenring and Rabung get their terribly flawed article into JAMA with an accompanying editorial? How did such an overlapping article then get into BJP? A little help from friends?

Leichsenring and Rabung have just published another review in The Lancet Psychiatry. Its title suggests an echoing of the JAMA editorial’s Bambi meets Godzilla theme Psychodynamic therapy meets evidence-based medicine.

Leichsenring, F., Luyten, P., Hilsenroth, M. J., Abbass, A., Barber, J. P., Keefe, J. R., … & Steinert, C. (2015). Psychodynamic therapy meets evidence-based medicine: a systematic review using updated criteria. The Lancet Psychiatry, 2(7), 648-660.

The abstract suggests past problems have been overcome:

Psychodynamic therapy (PDT) is an umbrella concept for treatments that operate on an interpretive-supportive continuum and is frequently used in clinical practice. The use of any form of psychotherapy should be supported by sufficient evidence. Efficacy research has been neglected in PDT for a long time. In this review, we describe methodological requirements for proofs of efficacy and summarise the evidence for use of PDT to treat mental health disorders. After specifying the requirements for superiority, non-inferiority, and equivalence trials, we did a systematic search using the following criteria: randomised controlled trial of PDT; use of treatment manuals or manual-like guidelines; use of reliable and valid measures for diagnosis and outcome; adults treated for specific mental problems. We identified 64 randomised controlled trials that provide evidence for the efficacy of PDT in common mental health disorders. Studies sufficiently powered to test for equivalence to established treatments did not find substantial differences in efficacy. These results were corroborated by several meta-analyses that suggest PDT is as efficacious as treatments established in efficacy. More randomised controlled trials are needed for some mental health disorders such as obsessive-compulsive disorder and post-traumatic stress disorder. Furthermore, more adequately powered equivalence trials are needed.

PubMed CommonsBut, alas, post publication peer review to the rescue. The credibility of this article was demolished in a brief, but effective PubMed Commons commentary by Stefan G. Hofmann, Nora Esser, and Giovanbattista Andreoli.

The study by Leichsenring and colleagues highlights the importance of considering the quality of the studies that are included in a meta-analysis when evaluating the results. The Cochrane Collaboration’s Tool (Higgins et al., 2011) is a commonly-used instrument to quantify the risk of bias using the following criteria: allocation sequence concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective outcome reporting. We analyzed the 64 randomized controlled trials of manual-guided PDT for specific mental disorders that were used in the review by Leichsenring et al (see Table 1). Thirty studies showed risk biases in sequence generation, 54 in allocation concealment, and 31 in the blinding conditions. Only one of the studies showed no obvious biases. Our results suggest that the studies included in Leichsenring’s meta-analysis were of poor quality, essentially invalidating the authors’ results and making the findings meaningless. Table 1: http://issuu.com/gvand/docs/quality_ratings_of_studies_in_leich/1 Table 2: http://issuu.com/gvand/docs/description_and_results_of_studies/1 References: Higgins, J.P., Altman, D.G., Gøtzsche, P.C., Jüni, P., Moher, D., Oxman, A.D., Savovic, J., Schulz, K.F., Weeks, L., Sterne, A.C., Cochrane Bias Methods Group, Cochrane Statistical Methods Group (2011). The Cochrane Collaboration´s tool for assessing risk of bias in randomised trials. RESEARCH METHODS & REPORTING, 343.)

We should always skeptical when authors clearly with a dog in the fight undertake systematic reviews and meta-analyses evaluating their favorite treatments. It’s best to get a couple of degrees of separation. A group that included John Ioannidis is provided such a review of LTPP. The results were not favorable:

The recovery rate of various mental disorders was equal after LTPP or various control treatments, including treatments without a specialized psychotherapy component. Similarly, no statistically significant differences were found for the domains target problems, general psychiatric problems, personality pathology, social functioning, overall effectiveness or quality of life.

And furthermore,

Control conditions were heterogeneous and frequently of low quality, e.g. without a specialized psychotherapy component. If anything, this suggests that LTPP is often compared against relatively ineffective “straw man” comparator… LTPP comparisons to specialized non-psychodynamic treatments, like dialectical behavior therapy and schema-focused therapy, suggest that LTPP might not be particularly effective.

 My past blog posts about Leichsenring and Rabung’s systematic reviews and meta analyses

Flawed, Biased Review Article in British Journal of Psychiatry. December 16, December 16, 2011

Is Long-Term Psychodynamic Psychotherapy Worthwhile?  January 14, 2012

Psychoanalysts claim long-term psychoanalytic psychotherapy more effective than shorter therapies.  February 4, 2014

Bambi meets Godzilla: Independent evaluation of the superiority of long-term psychodynamic therapy  May 24, 2014

 

 

 

 

 

Ten suggestions to the new associate editors of Psychological Science

Steps associate editors could take to reduce journal’s emission of faulty science.

Mustang Ranch

Night staff at the Mustang Ranch

I would no sooner  become an associate editor for a journal of ill repute anymore than I would become the night manager for the Mustang Ranch. In both cases I might be able to affect positive change, but my aims would be so counter to the prevailing business models that the effort would be just too frustrating for everybody.

Okay, it’s unfair to classify Psychological Science as a journal of ill repute, even if the journal has taken some recent knocks to its reputation. Some problems were related to inconsistent thoroughness of peer review. Others have been compounded by carelessly letting poor science get through peer review and then making a big fuss about it in press releases. Both sets of problems are consistent with a business model favoring newsworthiness and claims of scientific breakthrough over what is robust and trustworthy.

Brent Roberts and Ayse Uskul   are new PS Associate Editors. I don’t believe I have met either of them, but I hear nothing but good about them. I congratulate them on their new positions. I hope they are up to the task of restoring the credibility of PS.

Despite the discontent with PS that I share with many others, I would see it as ungrateful and uncool for Roberts and Uskel to publicly denounce the past editors or editorial policies of the journal

10 suggestionsHowever, I do have ten suggestions for changes that they should immediately, even if discreetly, introduce. Impatient critics need to appreciate that change will take time to become apparent in published articles. Another “sadness impairs color perception”  article  could well be in the pipeline. If so, the journal will have to take another drubbing on PubPeer. Yet to be fair, we need to assume that the influence of Roberts and Uskel is taking time to appear and that such articles are not their doing.

Here are my ten suggestions, which I hope that we will soon see evidence of being implemented.

  1. Abandon Pink Floyd rejections

all in all bricxk in the wallAn outgoing editorial by a recent PS first editor gloated about how he built the reputation of the journal. Notoriously, he heavily relied on desk rejections, i.e, not sending manuscripts out for review and depriving authors of any useful feedback. Among the reasons:

The Pink Floyd Rejection: Most triaged papers were of this type; they reported work that was well done and useful, but not sufficiently groundbreaking. So the findings represented just another brick in the wall of science.

We all know what can happen when the representation of psychological science is deprived of too many well-crafted bricks in the wall. More importantly, we also need to appreciate the considerable inducement to questionable research practices by junior and senior psychologists when they are compelled to demonstrate that their solid work is not simply another brick in the wall.

  1. Adopt the Pottery Barn rule.

you break, you buyThe American chain of high-end mall stores may not actually have a Pottery Barn rule , but journals should advertise and adhere to a Pottery Barn Rule.

Once a journal has published a study, it becomes responsible for publishing direct replications of that study. Publication is subject to editorial review of technical merit but is not dependent on outcome. Replications shall be published as brief reports in an online supplement, linked from the electronic version of the original.

Currently, attempted replications of articles that appear in PS are herded into groups and led off to ghettos. This strategy serves to punish those so foolish as to attempt to improve the trustworthiness of the journal. The visibility of such large assembled groupings also serves to tarnish the reputation of the journal. Note that attempted replications of articles originally appearing in Psychological Science contributed heavily to the proportion of failed replications in the report of the Open Science Collaboration: Psychology (OSC) article in Science.

  1. Adopt CONSORT, the Consolidated Standards for Reports of Clinical Trials,  and require completion of a checklist modifying to fit the kinds of studies being submitted to the journal and the details needed for the independent interpretation interpretation.

Journals of the Association for Psychological Science (APS) are the only credible journals publishing psychological studies that have not formally adopted CONSORT.

The first requirement of CONSORT is that titles and abstracts of studies reporting interventions be prominently labeled as randomized trials. My colleagues and I critiqued an article in PS that did not even mention that it was a clinical trial until the supplementary materials. The article was basically a poorly reported randomized clinical trial (RCT) with null findings, although dressed up to look like something very different. Its publication in PS was exploited in promotion of expensive workshops and other positive psychology merchandise. If reviewers had been initially alerted that this was a RCT, they would have been more likely to notice obvious flaws in the conduct, reporting, and interpretation of this trial.

CONSORT is a useful guide to what to include in reporting clinical and laboratory studies where research participants are randomized to conditions that differ in manipulations intended to affect adaptational outcomes. Beyond that, CONSORT represents good reporting of experimental psychology, something often lacking in PS – accounting for all subjects who were randomized, allotment concealment, blinding of experimenters, and primary and secondary outcomes.

Studies appearing in PS do not typically involve interventions with clinical populations. Nonetheless, more frequently than noticed, PS articles involve manipulations intended to demonstrate how human well-being can be improved. The authors then go on to generalize to what can be accomplished in everyday environments. Such articles typically consist of reports of single or a few underpowered studies that are poorly reported in the PS article, but nonetheless are subsequently used to make broad generalizations in press releases and subsequent articles in the popular and business media. Adherence to CONSORT would facilitate independent evaluation of the claims being made because of greater transparency about the methods and results.

  1. Adopt CONSORT for abstracts

Readers form strong, even immutable opinions about articles after only scanning abstracts. There is a funneling process from finding a title in an electronic bibliographic search or citation in another article or press release to retrieving and reading the actual article. Much of the progression stops with the abstract.

PS abstracts are about as uninformative as a Taco Bell advertisement when it comes to allowing independent evaluation and comparison. CONSORT has an extension checklist for abstracts. Even if the PS editors are not keen on adopting CONSORT for abstracts, they could at least encourage more informative abstract by adopting the traditional IMRAD format.

Something needs to be done about woefully uninformative abstracts in PS that foster misunderstanding of what the articles actually contain. There is evidence from other sources that misrepresentations of scientific articles in the media are often tied directly to exaggerated claims by the authors in abstracts. Reviewers should therefore be encouraged to check the correspondence to what is said in the abstract and was contained in the article.

  1. Require disclosures of conflict of interest.

Some authors seek a greased path from an early-release PS article to an op ed piece in the New York Times to TED talks and then on to lucrative corporate speaking engagements or promotion of self-help books. Such a path of course brings temptations to report results serving to fan the hype and suppress the rest. Readers deserve routine disclosures of conflicts of interest so they can decide for themselves whether an APS article deserves particular scrutiny because of what is at stake professionally more financially for the authors.

  1. Prioritize what articles will receive press releases and vet what is said about them.

There have been some self-inflicted embarrassments to PS from press releases attracting scrutiny to articles, the credibility of which cannot sustain such attention.

Don’t depend on being able to create an amnesia in readers scrubbing the Internet of all press releases. Some readers are on to you and preserve links that can be put back on the web.

Searching for old PS press releases can be an entertaining game. There are more out there, but one can get a good chuckle from the 2011 press release entitledLife is one big priming experiment…” which declared:

One of the most robust ideas to come out of cognitive psychology in recent years is priming. Scientists have shown again and again that they can very subtly cue people’s unconscious minds to think and act certain ways.

Ah, yes, 2011 was when PS was in its prime.

  1. Lower the threshold for accepting letters to the editor critical of articles that you have published.

There is a risk that the de facto place for finding critical commentary on PS articles will become PubPeer. Critics can find the advantages of anonymity, peer support, and a wonderful process in which the next comment builds on previous ones. However, PS ceding post publication peer review commentary to this outlet leaves a lot at risk to both the authors of the paper being criticized in the reputation of journal itself. PS is also ceding a long of control: editors can no longer monitoring comments will give the authors of the precise paper the traditional last word.

Maybe as an expansion of the Pottery Barn rule, PS could accept responsibility for publishing reasonable criticisms if they publish an article.

More generally, PS should accept the post publication peer review is not going to go away and should find ways of better harnessing it and improving the quality of the science PS presents the world.

  1. Exercise the same peer review and tone policing of authors’ responses to critical letters to the editor as you do for the letters themselves.

I’ve talked to other people who have attempted to submit letters to the editor. We find we share a common complaint. We have had excessive fussing editing of our letters for tone, only to receive gratuitous nastiness from authors who flagrantly ignore the points that we have made. This becomes a further incentive to move post publication peer review elsewhere. It also invites counterattack in the social media.

  1. Comply with the Committee on Publication Ethics (COPE) and establish explicit procedures for appeals of editorial decisions and stick to those procedures.

The journal endorsed COPE procedures, so now live up to them, starting with publishing procedures. Authors disputing what they construe to be unfair and arbitrary editorial decisions from PS editors currently have no recourse in terms of a published set of procedures by which they can make an appeal.

Again, talking to a number of people who have submitted letters to the editor, I have found the associate editors become very defensive about decisions that they have made, getting overly protective of their friends and cronies among the authors being criticized, and arbitrarily and summarily reject appeal. Unfair opportunities are offered to authors who would been criticized to censor and even veto publication of criticism.

I know, it’s tough being an associate editor, but in many situations it’s best to recluse oneself from what should be orderly, laid out, fair procedure of appeal.

  1. Replace the badges program with warning labels for articles that do not (1) preregister,(2)  place key materials in an open access repository, or (3) make their data available without interested parties going through a freedom of information act request or proving that their interest in the data is not vexatious.

Really, at the APS headquarters in or in your own universities are badges awarded for flushing after using the toilet? Adopting these practices should become so commonplace that badges become silly. Articles don’t have these features should be less frequent, but shameworthy.

Readers may have noted some embarrassment to the current badge system in all the uproar over the sadness impairs color perception article. When criticized in post-publication peer-review, the authors took down the data for which they had a badge for having uploaded and re-uploaded it with curious evidence of manual changes.

 

What reviewers can do to improve the trustworthiness of the psychotherapy literature

Reviewers arise, we have only an untrustworthy psychotherapy literature to lose.

Psychotherapy researchers have considerable incentives to switch outcomes, hide data, and spin reports of trials to get published in prestigious journals, promote their treatments in workshops, and secures future funding. The questionable research practices that permeate psychotherapy research cannot be changed without first challenging the questionable publication practices that allow and encourage them.

Journals must be held responsible for the untrustworthiness of what they publish concerning the efficacy and effectiveness of psychotherapy. When journals publish unreliable findings, they are failing not only their readers, but patients, clinicians, and policymakers.

Yet there are institutional agendas supporting and encouraging the questionable research practices of psychotherapy researchers. Unreliable, but newsworthy reports of “breakthrough” findings contribute more early citations than honest, transparent reporting of findings that are inevitably more modest than the illusions that questionable research practices and poor reporting can create. Early citations of articles lead to higher impact factors, which, rightly or wrongly, are associated with more prestige and the ability to attract reports of more ambitious, better resourced trials, even if the reliability of the reports are in question.

Editors of journals often endorse responsible practices such as registration of trials, publishing of protocols, and CONSORT (Consolidated Standards for Reporting Clinical Trials), but do little to enforce these practices in request for revisions and editorial decisions.

Reviewers can nonetheless lead the reform of the psychotherapy literature by making their own stand for responsible reporting.

The burden of getting a better psychotherapy literature may depend on reviewers’ insistent efforts, particularly in the likelihood that journals for which they review are lax or inconsistent in enforcing standards, as they often are.

When reviewers are given no explicit encouragement from the journals, they should not be surprised when their recommendations are overruled or when they do not get further requests for reviews after holding authors to best practices. But reviewers can try anyway and decline further requests for reviews from journals that don’t enforce standards.

cheshire cat4Recently I tried to track the progress of a psychotherapy trial from (a) its registration to (b) publishing of its protocol to (c) reporting of its outcomes in the peer-reviewed literature.

Reports of the trial had been reported in at least two articles. The first reportin Psychosomatic Medicine ignored the primary outcomes declared in the protocol.

Journal of Psychosomatic Research published another report that did not acknowledge registration, minimally cited the  first paper without noting its results, and hid some important shortcomings of the trial.

Together, these two papers entered a range of effect sizes for the same trial into the literature. Neither article by itself indicating which should be considered the primary outcome and they compete for this claim. When well done, meta-analyses should be limited to a single effect size per study. Good luck to anyone undertaking the bewildering task of determining which of the outcomes, if any, of the reported in these two papers should be counted.

Overall, detecting the full range of problems in this trial and even definitively establishing that the two reports were from the same trial, took considerable effort. The article in JPR did not give details or any results of the first report of the trial in PM. Although the PM article and JPR claimed to adhere to CONSORT in its reporting, it provided no flow chart of participants moving from recruitment through follow-up. That flowchart was included in the PM article but not in JPR. Yet even in PM,  the authors failed to discuss that the flowchart indicated substantial lower retention of patients randomized to treatment as usual (TAU). A reader also had to scrutinize the tables in both articles to recognize the degree to which substantial differences in baseline characteristics influenced the outcome of the trial and limited its interpretability. This was not acknowledged by the authors.

Overall, figuring out what happened in this trial took intense scrutiny, forensic attention to detail, and a certain clinical connoisseurship. Yet that is what we need to evaluate what contribution to the literature it provided, and with what important cautions because of its limitations.

There were shortcomings in the peer review of these two articles, but I don’t think that we can expect unpaid reviewers to give the kind of attention to detail that I gave in my blog. Yet, we can expect reviewers to notice basic details related to the trustworthiness of reporting of psychotherapy trials than they now typically do.

If reviewers don’t catch certain fundamental problems that may be hiding in plain sight, there are unlikely to be detected by subsequent readers of the published paper. It is notoriously difficult to correct errors once they are published. Retractions are almost nonexistent. APA journalssuch as Journal of Consulting and Clinical Psychology or Health Psychology that are the preferred outlets for many investigators publishing psychotherapy trials are quite averse to publishing critical letters to the editor

Anyone who has tried to published letters to the editor criticizing articles in these journals knows that editors set a high bar for even considering any criticism. Authors being criticized often get veto over what gets published about their work, either by being asked directly or by simply refusing to respond to the criticism. Some journals still hold to the policy that criticism cannot be published without a response from authors.

It also isn’t even clear that the authors of the original papers have to undergo peer review of their responses to critics. One gets doubts from the kinds of ad hominem attacks that are allowed from them and from authors’ general tendency to simply ignore the key points being made by critics. And authors get the last word, with usually only a single sequence of criticism and response allowed.

The solution to untrustworthy findings in the psychotherapy literature cannot depend on the existing, conventional system of post publication peer review for correction. Rather, something has to be done proactively to improve the prepublication peer review

A call to arms

If you are asked to review manuscripts reporting psychotherapy trials, I invite you to join the struggle for a more trustworthy literature. As a reviewer, you can insist that manuscripts clearly and prominently cite:

  • Trial registration.
  • Published study protocol.
  • All previously published reports of outcomes.
  • What are the reports might subsequently be in the works.

Author should provide clear statements in both the cover letter and the manuscript whether it is the flagship paper from the project reporting primary outcomes outcomes.

Reviewers should double check the manuscript against electronic bibliographic sources such as Google Scholar and Pub Med to see if other papers from the are not being reported. Google Scholar can often provide a way of identifying reports that don’t make it into the peer-reviewed literature as reported in Pub Med or have not yet made it to listings in Pub Med.

Checking is best done by entering the names of all authors into a search. It’s often the case that order of authors change between papers and authors are added or dropped. But presumably there will be some overlap.

Reviewers should check for the consistency between what is identified as outcomes in a manuscript being reviewed with what was registered and what was said in the published protocol. Inconsistencies should be expected, but reviewers should but insist these be resolved in what could be a major revision of the manuscript. Presumably, as a reviewer, you can’t make final recommendations for publication without this information been prominently available within the paper and you should encourage the editor to withhold judgment.

Reviewers should alert editors to incomplete or inaccurate reporting and consider recommending a decision of “major revisions” where they would otherwise be inclined to recommend “minor revisions” or outright acceptance.

It can be a thankless task to attempt to improve the reliability of what is published in the psychotherapy literature. Editors won’t always like you, because you are operating counter to their goal of getting newsworthy reports into their journals. But the next time, particularly if they disregard your critique, you can refuse to review for them and announce that you are doing so in the social media.

Update 1 (January 15, 2016 8:30 am): The current nontransparent system of prepublication peer review requires reviewers to keep confidential the review process and not identify themselves as having been involved after the fact. Yet, consistent with that agreement of confidentiality, reviewers are still free to comment on published papers. When they see that journals have ignored their recommendations and allowed the publication of untrustworthy reports of psychotherapy trials, what options do they have?

They can simply go to PubPeer and post a critique of the published trial without identifying themselves as having been a reviewer. If they are lucky, they will get a thread of post publication peer review commentary going that will influence the subsequent perception of the trial’s results. I strongly recommend this procedure. Of course, if they would like, disappointed reviewers can write a letter to the editor, but I’ve long been disillusioned with the effectiveness of that approach. Taking that route is likely to only leave them disappointed and frustrated.

Update (January 15, 2016 9:00 am):

While I was working on my last update, an announcement about the PRO appeared on Twitter. I reviewed it, signed on, and find its intent quite relevant to what I am advocating here. Please consider signing on yourself.

The Peer Reviewers’ Openness (PRO) Initiative is, at its core, a simple pledge: scientists who sign up to the initiative agree that, from January 1 2017, will not offer to comprehensively review, or recommend the publication of, any scientific research papers for which the data, materials and analysis code are not publicly available, or for which there is no clear reason as to why these things are not available. To date, over 200 scientists have signed the pledge.

 

just say noReviewers, just say no to journals and editors not supporting registration, transparent reporting, and, importantly the sharing of data required by readers motivated to reevaluate for themselves what is being presented to them.

David Cameron’s call for “more mature” conversation about mental health should get more evidence-based

Focusing on reducing suicide in pregnancy suggests a serious misdirection of scarce funds.

UK Prime Minister’s plans may make a dysfunctional system of depression care even more dysfunctional.

UK Prime Minister Cameron will pledge extra mental health support [http://www.bbc.com/news/uk-politics-35276854 ] with a call for a “more mature” conversation about mental health. The BBC News coverage of this forthcoming announcement gave a few details:

Specialist care for new mothers with mental health problems and support for anorexic teenagers will gain money allocated in the Autumn Statement.

The PM will also pledge more support in hospital A&E departments.

He will also announce new parenting support and plans to demolish some of England’s worst council estates.

He is expected to say mental illness is not contagious and is “nothing to be frightened of”.

“As a country, we need to be far more mature about this. Less hushed tones, less whispering; more frank and open discussion,” he will say.

“We need to take away that shame, that embarrassment, let people know that they’re not in this alone, that when the clouds descend, they don’t have to suffer silently.

“I want us to be able to say to anyone who is struggling, ‘talk to someone, ask your doctor for help and we will always be there to support you’.”

The specific measures expected, which will apply in England only, are:

  • £290m up to 2020 to give 30,000 more women each year access to specialist mental health care before and after giving birth, including through classes
  • £247m over the next five years so that every hospital has mental services in their Accident and Emergency unit
  • A new waiting time target for teenagers with eating disorders, which will track the number of patients being seen within a month of being referred
  • A target that at least half of people experiencing psychosis for the first time should be treated within two weeks

Who could take issue with this initiative? Well, the sparse details provided in the BBC article suggest a commitment to making popular, but ineffectual pitches that are counter to available evidence.

The BBC news coverage prominently features a photo which seems to indicate misdirection in Cameron’s efforts.

cameron pregnant woman

Cameron obviously did not consult with mental health consumer groups who strongly object to head-holding depictions of persons with mental disorders.

The statement that “women with serious psychiatric disorders during pregnancy are at an increased risk of suicide” will gain support from the uninformed, but skirts a wealth of contradictory evidence about suicide, mental disorder and pregnancy.

Compared to otherwise matched other women of childbearing age who are not giving birth, pregnant women have no more depression.

In terms of absolute risk, pregnancy and early motherhood are periods of low risk periods for suicide. Going back to the classic work of Emile Durkheim, there is the strong suggestion (and considerable subsequent evidence) that having responsibility for a young child actually reduces suicidality though it can be a source of considerable stress especially for economically disadvantaged mothers.

George Brown’s classic work suggests that having three children under the age of six at home puts women at risk for depression, but this work did not show that put them at risk for suicide.

Programs to reduce depression in pregnant women and women who have just delivered often focus on routinely screening them for depression. Although popular with policymakers, there is a lack of evidence that screening for depression improves depression outcomes in this population and certainly no evidence that reduces suicidality.

Screening programs result in more referrals to mental health services, but not uptake of services or improvement in depression outcomes. They are much better in generating employment for mental health professionals than improving the well-being of women.

Depression screening programs for young women, to the extent to which they have any effects at all, increase the mislabeling of personal and social problems as a clinical disorder. Why is that?

A self-report questionnaire or touchscreen screening instrument is not an adequate basis of diagnosis. An elevated score only suggests increased likelihood that the person is depressed. A questionnaire does not provide a valid diagnosis. This requires a structured conversation with a specific probing of symptoms. Evidence is that primary care physicians are unwilling or unable to follow-up the large number of women screening positive for depression, most of whom will prove to be false positives.

A heightened score on a depression screening instrument is a report of considerable distress, but not necessarily a clinical disorder, in terms of indicating number and severity of depressive symptoms. Everyone experiencing misery is not suitable for mental health care, the evidence base for which requires appropriate diagnosis.

Most heightened scores on a depression questionnaire do not indicate a need for specialty mental health services. Heightened scores may indicate a need for personal and social difficulties.  Further conversation is required to determine precisely what is needed and how any needed services can be accessed. Having to deal with false positive screens takes personnel away from having these conversations.

I lacking interest of time or training to probe heightened scores, primary care personnel may simply accept a positive score on a screening instrument as indication of clinical disorder. This overidentification of patients as depressed is confusing to those who know that they are not clinically depressed, but directs others into treatment that they don’t need.

There is considerable controversy about the safety and relative efficacy of antidepressants during pregnancy and breast-feeding. Particularly with women who have a past history of severe depression and suicidality, continuation or initiation of antidepressant treatment may be warranted, but with extra monitoring. In general, however, decision-making about whether to accept antidepressants or psychotherapy is best undertaken in an extended conversation with a woman in which professionals provide information and address myths and misunderstandings.

Many women who screen positive and are found to have clinical depression have already been identified, but are receiving no or inadequate care.

The prospects for a newly screened and identified depressed women getting adequate and timely treatment are low.

In the UK, a woman identified by a screening program, properly diagnosed, and referred to psychotherapy can expect to wait 8 months before receiving treatment.

What would be a better use of scarce funds than programs for detecting and treating depression in pregnant women and new mothers? Improve the overall quality and access to primary care and specialty mental health depression treatment in this age range. Otherwise, scarce resources will be withdrawn from already poorly functioning services and women will simply be put on waiting list or offered unnecessary treatment with antidepressants after an inadequate discussion of treatment options.

So Prime Minister Cameron, what are you going to do to increase the opportunities for conversations with women at risk for depression to ensure that they get appropriate treatment when they need it? Your current system is quite dysfunctional and you may be about to make it more dysfunctional.

See

Time to screen postpartum women for depression and suicidality to save lives?

Please read if you are visiting my Facebook page [updated]

I appreciate all the Facebook friend requests that I have recently received. Some of the increased volume of requests is tied to my activism concerning the PACE trial of cognitive behaviour and graded exercise therapy for chronic fatigue syndrome/myalgic encephalomyelitis (commonly hashtagged as #MECFS).

fb logofb logofb logoMy Facebook page provides a front row seat, where visitors can get a first view of not only my blogging, but direct actions I take to challenge the:

  • Bad research practices of PACE ranging from its poor design as a clinical trial, failure to deliver what was promised in the trial registry, and shamefully bad statistical analysis.
  • Bad publication practices that put the flawed science of PACE into prestigious journals without adequate peer review.
  • Extraordinary publicity machine that has led UK media to utterly fail as a filter on the PACE investigators’ exaggerated claims and unreported conflicts of interest.
  • Strange plutocratic politics of the UK that seriously compromise the funding, peer review, and dissemination of research that directly impacts on public health and individual well-being.

Anyone can come to my publicly accessible Facebook page and find the latest news to either read in, get as takeout or to freely distribute. Anything that I place on my Facebook page is meant to be distributed elsewhere, although I appreciate it when people acknowledge where they got it.

But long before I  paid any attention to PACE, my Facebook page was already developing quite a following. My page is very different from most Facebook pages because my frequent posts are distinctive and diverse and often uncategorizable ways. Except maybe that they pull together an irreverence, a skepticism about what I read in the media and the scientific literature, and an insistence on best evidence as a way of resolving important controversies.

I continue to welcome friend requests from people who don’t see the relevance of the trial PACE or ME/CFS to their own lives. I expect that they will still make up the bulk of visitors to my Facebook page.

Regardless, I welcome all the new friends wherever they come from, as well as visitors wandering through without being interested in joining in.

However, my Facebook page remains just that – my own Facebook page. It has a much broader function. I don’t want it to become a general message board where people post information they want to reach a larger audience. I also want to preserve the diversity of those who come to my Facebook page to comment and to take away information to share elsewhere. I want to attract such traffic with a diversity of posts.

So, expect me to continue to post a broad range of material that I find interesting. Much of it comes from social media gathered in my efforts to write about various topics, both in blogs and in scientific articles.

I don’t always agree with what I post. My posting does not necessarily represent endorsement. Some material is chosen because it so outrageously offends my sense of good science or social justice. Some of what I post is satirical, but devilishly so hard to distinguish what is presented as proof in the larger media. Some of what I post to simply ambiguous as to whether it is true or satire challenges visitors to rethink a sharp distinction that they may hold.

It’s understandable that the influx of new Facebook friends often includes people who don’t understand what I am up to. Often innocently, they sometimes try to postings on my page, not noticing that is decidedly not a general message board.

I need to get clear about that.

Expect me to increasingly remove material that others place on my FB page, even when it’s quite good. I do not want to encourage the idea that this is a general message board or that its focus is narrower than I would intend.

[Update: I have now chosen a FB setting that allows me to moderate what is posted on my wall. Thanks to those who suggested this.]

I hope implementing this process will not cause great offense. I also think that, in particular, members of the ME/CFS community can appreciate that I more effectively serve their interests, which is one of my goals, if I involved broadest possible audience in the broadest possible definition of the issues posed by the PACE trial, namely, bad science allowed by bad publication practices, and promoted with harm to the patient population that is supposed to help.

Recently I posted at Mind the Brain an article that had absolutely no direct relevance to chronic fatigue syndrome or the PACE trial. Some people were confused and asked “how is this relevant to CFS?” The simple answer is it was not.

I should also be clear for those who don’t know already – PLOS One is squaring off with its lawyers against the PACE investigators and their lawyers who are resisting given me the data promised by the PACE investigators having published in PLOS One.

It’s also no secret that Richard Horton,* editor of The Lancet has put pressure on the PLOS management to quiet me down at both at their blog site and in social media. Yup, Horton demanded that PLOS clamp down not only what I upload to their blog, but what I say on Twitter and Facebook.

plos oneThat is one of the many reasons I think it is best for both PLOS One in its struggle with the PACE investigators for the data I requested if I avoid the appearance of a conflict of interest in using PLOS Mind the Brain for activism concerning the PACE data. So, I’ve taken the PACE discussion to my alternative blog Quick Thoughts where I have to worry less about making PLOS nervous during this legal struggle.

I strongly promote PLOS and work long hours for free as an academic editor for PLOS One because I am committed to the values that it represents. I’m quite critical of PLOS when I feel its journals depart from those values. Ultimately, I am more loyal to those values than to PLOS.

Some people get a bit confused by my past going public with criticisms of PLOS and I have had to explain our complex relationship. I believe that going public was the thing to do. When issues were satisfactorily resolved, I said something that I often say on Twitter: PLOS is by no means perfect, but it generally does a good job of acknowledging and trying to fix its mistakes.

PLOS and I are close, but we do not sleep in the same bed.

PLOS is well aware I also can use the Quick Thoughts blog to target PLOS if it doesn’t deliver on the data or retract the paper. I don’t think it’ll come to this, but my Facebook wall certainly could be a place where a general strike of the over 4000 PLOS academic editors could be organized – if PLOS dropped the ball and weakened their data sharing policy.

Stay tuned, if I ever had to resort to such tactics, you’d probably see it announced first on my Facebook wall. But I doubt it will happen.

Finally, taking the discussion off Mind the Brain also allows me to poke fun at the laughable Richard Horton without having to worry as much about him pressuring PLOS. Visit my Facebook wall and you will be seeing more discussions of Horton Hatches the Egg.

If you don’t know much about Horton and the Egg, here is a vintage full length cartoon. Ponder the many ways in which it could be relevant here.

horton hatches

Watch 1942 cartoon here

*Richard Horton is @richardhorton1 on Twitter. His Twitter account announces

richard horton twitter handle.PNG

Now, ponder what that could possibly mean. From what direction is Horton attacking the present?

An update [January 9, 2016, 9:42 am]

I appreciate the attention that this modest blog post has received in the social media, including more requests to become my friend on Facebook. This further attention suggests some updating of what I previously posted.

First, there have been numerous requests that if my Facebook page is not going to be devoted to debunking the PACE trial or providing information about chronic fatigue syndrome/myalgic encephalomyelitis**, I should start another page that has that specific focus.

The problem with this suggestion is that I simply don’t have the time and I’m just one person. I do all this on my own time with uncertain payoff. I have enough work already just keeping up with continued academic output and the workshops, as well as the blogging and the curating of my presence on social media. And though I’m giving a lot of attention to certain issues, I don’t want to – as some patient-activists warned me – fall down the rabbit hole. I want to keep a broad focus.

However, I make my material freely available, and so if someone else wants to set up a distinctive page, they can feel free to do so. They can make use of my material, hopefully giving credit for its source, but they can’t claim that they represent me or that I have approved what they do. I think that is what some patient-oriented Facebook pages are already doing, and that is fine.

Secondly, I appreciate all the requests to be Facebook friends. I have no illusion about what being a Facebook friend means – with almost 4800 friends, I can’t claim to have met many or most of you.

You are not necessarily offering much in asking to be my friend on Facebook, nor are you expecting much. If you flood me with “happy birthday’s”, I will be appreciative but I don’t expect we will be sharing birthday cake.

http://tinyurl.com/jjr63j8

But if becoming a Facebook friend means simply expressing interest in what I post and possible interest in sharing it or commenting, fine.

I have been mostly agreeing to becoming friends with people who ask, but I sometimes get some strange requests. In this category, I would include requests from women who don’t seem to have any connection to me and have no evidence of shared interests. If I try to look at their Facebook pages, all I see are some attractive and sometimes provocative photos and no other content. If I make a mistake of agreeing to be a friend on Facebook, I begin getting some overly personal back channel messages, some persistent to the point where I have to defriend or block somebody.

So, if you send me a new request be friends on Facebook, it helps if your homepage indicates some shared interests. If not, I’ll have to decide whether to accept your FB friendship on the basis of you being friends already with people who are my FB friends. That will work for most people.

But some people are brand-new to Facebook and don’t have such connections. For them I would recommend dropping me a brief note, indicating why they want to be friends or what the connection is.

I think that will be easier on all of us.

**PS. An e-book? I’m wondering if one solution for managing some of the material I’ve been producing concerning the PACE trial and MECFS is for me to assemble an inexpensive e-book with material and additional commentary and resources. Is there an interest in that?