Spokesperson clarifies that American Psychological Association does not endorse routine data sharing

I provide an update of the recent blog post about a researcher requesting a small set of variables to verify claims appearing in an article published in an APA journal. My update involves a further exchange of emails between this researcher and the APA publications office.

In the new exchange:

  1.  APA verifies that it has a weak policy concerning authors having to share data and it does not really endorse routine sharing of data.
  2.  APA indicates that any re-analyses of data requested from an article published in APA Journal have to stick strictly to reproducing analyses in the published article, and not go further without the express permission of the original authors.
  3.  Basically, analysis of data provided in response to a request can only be used to check specific statistical calculations and not to conduct further analyses that might shed light on the appropriateness of the original author’s conclusions.

That is usually not where any clarification of controversies will be found.

I present further evidence of contradiction and even hypocrisy in what APA says about data sharing.

I make three nominations to a Wall of Shame of those who resist correction of potential untrustworthiness in the psychological literature by penalizing and threatening those who request data for checking by reanalysis.

reusing-data-makes-me-research-parasite In an editorial entitled Data sharing,  the editors of New England Journal of Medicine have condemned as “research parasites” researchers who seek to test alternatives to authors’ hypotheses or new exploratory hypotheses with data shared from published articles. The APA seems to have much the same attitude.

Improving the trustworthiness of psychology depends crucially on the work of such research parasites.

Why? For many areas of psychology the scope and expense of research projects make replication initiatives impractical. Efforts to improve the trustworthiness of psychology involves insisting on completeness and transparency in what is published, but also on routine data sharing from published papers.  Well aware of the limitations of review of manuscripts before they are published, we need independent, post-publication review with access to the data. APA’s position is making any scrutiny more difficult.

This  incident again demonstrates the institutional resistance to data sharing and the institutional support available to authors who want to protect the claims from independent scrutiny.

I wonder what these authors were hiding by presenting obstacles to accessing their article. We won’t get to see.

After all, Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results.

 What was reported so far

  1.  In the recent guest blog post, Professor Christopher Ferguson of Stetson University described the extraordinary resistance he experienced when he attempted to obtain a few variables from an author to decide for himself if her claims held up about the effects of exposure to media violence on youth violence.
  2.  To summarize:
  3.  Chris first asked for data sets from two articles which were needed to check the claims in one of the articles.
  4. The spokesperson for the Flourishing Families Project, from where the data came, objected. She considered what Chris was proposing to be new analyses rather than merely fact-checking the original paper.
  5.  Chris revised his request to a more modest set of variables across 3 time periods, plus some covariate/control variables.  Maybe  a total of14-15 variables.
  6.  In response, he received an invoice for $450 and a contract that he was required to sign to obtain the data. The contract stipulated that Chris could only reproduce exact analyses presented in the article and would have to obtain permission of the Flourishing Families Project to publish any results. The contract carried the penalty of ethical charges if these terms were not met.
  7.  Chris launched a GoFundMe drive and raised the $450, but also contacted APA Ethics Committee.
  8.  While Chris was waiting for a response from APA, he received a new contract from the spokesperson for the Flourishing Families Project. The revised contract seemed to relax requirements that Chris get permission from the project to publish results. But the contract retained the restriction that he could only reproduce analyses contained in the article. It introduced a threat of a letter to Chris’s Dean if he did not accept this restriction.
  9.  Chris then received a letter from the APA Ethics Committee explaining that it is not appropriate for authors to restrict independent re-analyses needed to confirm their conclusions.
  10. A victory? No, not really. The spokesperson for the Flourishing Families Project then wrote to Chris, stating that her previous letter had been written in consultation with the APA Journals Office and General Counsel. Wow, brought in the lawyers. That should scare Chris.

The big picture issues

An earlier post at Stat How researchers lock up their study data with sharing fees had nailed the key issues in the author’s response to Chris:

The story highlights a potentially uncomfortable aspect of data-sharing: Although science is unquestionably a public good, data often are proprietary. Drug companies, for example, spend millions upon millions of dollars on clinical trials for their experimental products. While the public certainly has a right to know the results of those trials — favorable or not — researchers who want access to the data to conduct their own studies can’t reasonably expect the original investigators not to recoup the costs of sharing it.

Okay, but:

 Trying to recoup costs is fine in the abstract, but if it’s used as just another way to avoid sharing data, then it’s deeply objectionable.

And:

One thing about the Ferguson-BYU example is clear: We need explicit policies. Winging it will work about as well in this arena as it does in, say, presidential debates. Existing rules about data sharing, if they even exist, are vague and institution-specific, and permit researchers to erect obstacles, financial or otherwise, that give them effective veto power over the use of their data.

Now, a further exchange between Chris and APA

The next chapter in this story started with an email from Rosemarie Sokol-Chang, who identifies herself as Publisher, APA Journals (Acting). The full email is reproduced below, but I want to highlight:

“Certainly there is a group of scholars and organizations that advocate for open sharing of data – free and without restriction. While these movements exist, the extent to which to share data is up to the author, and in this case, the author chose not to freely share it.”

I guess we can count APA out of a “group of scholars and organizations…”

 The actual emails

 Subject: APA Journals – data share request

Dear Chris,

Jesse and I met to discuss your request to reuse data, in light of the letter you received from the Ethics Committee. From our read of the letter, we (Journals and General Counsel) are not interpreting the code differently from the Ethics Committee – that is, data should be released to those who “seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose…”

We understand the claim you are making regarding data sharing, but that extends beyond the ethical code APA set and follows. Certainly there is a group of scholars and organizations that advocate for open sharing of data – free and without restriction. While these movements exist, the extent to which to share data is up to the author, and in this case, the author chose not to freely share it. Per Standard 8.14:

“Psychologists who request data from other psychologists to verify the substantive claims through reanalysis may use shared data only for the declared purpose. Requesting psychologists obtain prior written agreement for all other uses of the data.”

I understand this is not the outcome that you want, but the author is complying with the current APA Ethics Code and the APA Journals policy of sharing data for verification.

All best,

Rose

Rosemarie Sokol-Chang, PhD
Publisher, APA Journals (Acting)
American Psychological Association
750 First Street NE
Washington, DC 20002

Decide for yourself, but I think this email indicates that APA is contradicting earlier communication, but denying it is doing so.

Chris replied:

From: Chris Ferguson (Psychology Professor) [mailto:cjfergus@stetson.edu]
Sent: Wednesday, November 30, 2016 6:01 PM
To: Raben, Jesse <jraben@apa.org>; Sokol-Chang, Rose <RSokol-Chang@apa.org>
Subject: Re: APA Journals – data share request

hi Jesse (and Rose):

Thanx for being willing to continue to dialogue with me on this.  Unfortunately I don’t think we’re at “apples to apples” yet.  To be clear (I thought I had been, but if not then I apologize) my intent is, consistent with 8.14, to verify the substantive claims of the original paper not to do “anything I want”…I don’t think I *could* do anything else with this dataset as it has so few variables in it.

However, my interpretation of both 8.14 as well as the letter I received from the Ethics Committee is that, while I must test the same hypotheses as the original paper, I am not restricted to the original analyses.  The Ethics Committee letter appears rather clear on this in fact as they say “Thus, the Committee feels that Standard 8.14(a) promotes the sharing of data for reanalysis, whether this be a straight replication of the published analysis or not.”  This makes a great deal of sense because, of course, the initial analyses may be wrong…and it makes no sense for a verification effort not to check this.

Let me ask you a few pointed questions.

1.) If I were to discover that a variable had been miscalculated in the original dataset, would I be able to recalculate it and rerun the analyses with the corrected variable?

2.) If I discovered that the original analyses were misspecified…would I be able to rerun a corrected analyses with proper specifications?

3.) If I were to learn that the analytic approach itself were inappropriate for the data, would I be able to test the substantive claims using alternate, more appropriate analyses.

4.) If I discovered other, unforeseen, errors in the data or analyses, would I be able to report these?

As I read the current contract, the answer to each of these questions would be “no.”  If I am incorrect in my interpretation (and I may be) I think it would be important for the contract to make clear what I *can* do, particularly in light of the unpleasant language in it threatening ethical complaints and calls to my dean.

Thank you for your consideration.

Chris

And then Rosemarie Sokol-Chang responded:

Hi Chris,

The intent of the 8.14 as we apply it to authors is to offer a check of the validity of what was reported. If you were to receive the data, and run the same analyses, and get different results – we would want the scientific literature cleaned up in light, so that the article didn’t persist with inaccurate results. If you were to find that the data looked “fishy” – which happens rarely but there are some big-name cases of numerous retractions by the same author – this is something APA Journals would also want to know to be able to take measures to clean up the record. This is the “verification” step.

Replication is duplicating an entire experiment – you’d be collecting new data following the same method. Reanalysis is using the same data set – and whether or not a requestor can use a data set to run any particular analyses not reported in the manuscript is ultimately up to the author.

All best,

Rose

Rosemarie Sokol-Chang, PhD

Publisher, APA Journals (Acting)

American Psychological Association

750 First Street NE

Washington, DC 20002

rchang@apa.org202-336-5667

www.apa.org/pubs/journals

Déjà vu all over again

 APA has been here before. See:

The APA and Open Data: one step forward, two steps back?

And what APA really meant in:

Access to Archives of Scientific Psychology Data

One of my earliest blog posts ever was about a study from this department. Did a Study Really Show that Abstinence Before Marriage Makes for Better Sex Afterwards? Requesting data was not an option  in 2011. I don’t think  getting a look at the data was needed to establish the patent absurdity of this study’s methodology and conclusions.

 Three Nominations for the Routine Data Sharing Wall of Shame

use the wall of shame.jpg
sarah-coyne

Sarah Coyne blocked efforts to independently verify her claims about effects of media violence on children.

laura

Laura Padilla-Walker got APA General Counsel involved and raise threat of going to Chris’s Dean.

rosemaire-soklol-chang

Rosemarie Sokol-Chang withdrew what had been apparent APA support for Chris’s request and clarified that APA does not support routine data sharing.

Postscript

Is it really wrong to shame researchers who are not willing to share data or do so with unnecessary obligations?

If you can earn badges for uploading new data, why shouldn’t we also give badges for sharing old data? – Uli Schimmack on FB. November 29, 2016

A bad response to the crisis of untrustworthiness in clinical psychological science

surpriseA new Call for Papers establishes a place for failed replications and null findings in clinical psychology in an American Psychological Association journal. Unfortunately, the journal lacks an impact factor, despite the journal having been publishing for decades.

There are lots of reasons that establishing such a ghetto where failed replications and null findings can be herded and ignored is a bad idea. I provide nine. I’m sure there are more.

But the critical issue in the creation such ghettos is that they reduce pressure on the APA vanity journal,  Journal of Consulting and Clinical Psychology to reform questionable publication practices and routinely accept replications and null findings.

 Clinical psychology is different

  • The untrustworthiness in clinical psychological science is serious, but different than that of personality and social psychology, and the crisis it poses requires different solutions.
  • There is little harm to not been able to replicate personality and social psychology studies, beyond to the credibility of those fields and the investigators within them.
  • However, untrustworthy findings in clinical psychology – whether they are exaggerated or simply false – can translate into ineffective and even harmful services being delivered, along with poor commitment of scarce resources to where they are needed less.
  • Personality and social psychologists can look to organized mass replication efforts to assess the reproducibility of findings in their fields. However, such efforts are best undertaken with Internet-recruited and student samples using surveys and simple tasks.
  • Mass replication efforts are less suitable for key areas of clinical psychology research, which often depends on expensive clinical trials with patients and extended follow-up. Of course, research and clinical psychology benefits from independent replication, but it is unlikely to occur on a mass basis.

Efforts to improve the trustworthiness of clinical psychology should have progressed more, but they have not.

Clinical psychology has greater contact than personality and social psychology with the biomedical literature, where untrustworthy findings can have more serious implications for health and mortality.

In response to repeated demonstrations of untrustworthy findings, medical journals have mandated reforms such as preregistration, CONSORT checklists for reporting, transparency of methods and results using supplements, declarations of conflicts of interest, and requirements for the routine sharing of data.  Implementation of these reforms in medical journals is incomplete and enforcement is inconsistent, with clear signs of resistance from some prestigious journals. Note for instance, the editor of the New England Journal of Medicine warning that routine sharing of data from clinical trials would produce “research parasites” who would put the data to different purposes than intended by the original authors.

While many of these reforms have been nominally endorsed by specialty clinical psychology journals, they are largely ignored in the review and acceptance of manuscripts. For instance, a recent systematic review published in JCCP  of randomized trials published in the most prestigious clinical psychology journals in 2013 identified 165 RCTs. Of them,

  • 73 (44%) RCTs were registered.
  • 25 (15%) were registered prospectively.
  • Of registered RCTs, only 42 (58%) indicated registration status in the publication.
  • Only 2 (1% of all trials) were registered prospectively and defined primary outcomes completely.

Apparently not only are investigators failing to register their trials, editors and reviewers ignore whether registration has occurred and don’t bother to check whether what is reported in a manuscript is inconsistent with what is proposed in a registration.

Questionable research practices in clinical psychology

The crisis in clinical psychological science lies in its evidence base:

  • RCTs are underpowered, yet consistently obtain positive results by redefining the primary outcomes after results are known.
  • Typical RCTs are small, methodologically flawed study conducted by investigators with strong allegiances to one of the treatments being evaluated.
  • Treatment preferred by investigators are a better predictor of the outcome of RCTs than the specific treatment being evaluated.

Questionable publication practices in clinical psychology

Questionable research practices (QRPs) in clinical psychology are maintained and amplified by questionable publication practices (QPPs).

The premier psychology journal for publishing randomized trials is Journal of Consulting and Clinical Psychology. It is a vanity journal with a strong confirmation bias and a distinct aversion to publishing null findings and replications. Until recently, letters to the editor were not even allowed. When the ban was relaxed a few years ago, a high bar was set for accepting them. Statistics about the rate of acceptance of letters to the editor are not available, but accounts from colleagues suggest that criticisms of basic flaws in articles that have been published are suppressed. JCCP is not a journal hospitable to post-publication peer review.

Publication of flawed studies in JCCP go on detected and unannounced, except through alternative post publication peer review, outside the journal, such as PubMed Commons comments and blogging.

Although the term “Pink Floyd rejection” was originally developed by an outgoing editor of the Association for Psychological Science’s Psychological Science, it captures well the editorial practices of JCCP.

pink floyd study -page-0

Call for Brief Reports: Null Results and Failures to Replicate

An APA press release announced:

Journal of Psychotherapy Integration will start publishing a new recurring brief reports section titled, “Surprise, Surprise: Interesting Null Results and Failures to Replicate.”

In an era when findings from psychological science are called into question, it is especially important to publish carefully constructed studies that yield surprising null results and/or failures at replicating “known” effects.

The following 2012 article published in Journal of Psychotherapy Integration is a good example of a paper that would be appropriate for this section:

DeGeorge, J., & Constantino, M. (2012). Perceptions of analogue therapist empathy as a function of salient experience. Journal of Psychotherapy Integration, 22, 52-59.

Submitted manuscripts should not exceed 2500 words, including references. Manuscript should be submitted electronically through the journal’s submission portal under Instructions to Authors.

Please note in your cover letter that you are submitting for this brief reports section. We look forward to your submissions!

What’s wrong with this resting place for failures to replicate and null findings?

  1. Authors undertaking replications, regardless whether they succeed in confirming past findings, are entitled to a journal with an impact factor.
  2. The title Journal of Psychotherapy Integration adds nothing to electronic bibliographic searches because “psychotherapy integration” is not what failures to replicate and null findings necessarily represent. Locating particular articles in electronic bibliographic searches is often fortuitous. Readers’ decisions to click on a title   to examine the abstract  depend on their recognizing the relevance of the article from the title of the journal in which it is published.
  3. The title to this special section is demeaning. If it is a joke, it will soon wear thin.
  4. Failures to replicate and null findings are not necessarily “surprises” given the untrustworthiness of the clinical psychology literature.
  5. Reasons for the failure to replicate previously published clinical trials often lie in the conduct and reporting of the original studies themselves. Yet having been granted “peer-reviewed” status in a more prestigious journal, the original articles are automatically granted more credibility than the failure to replicate them.
  6. A word limit of 2500 is hardly adequate to describe methods and results, yet there is no provision for web-based supplements to present further details. The value in failures to replicate and null findings lies in part in the ability to make sense of the apparent discrepancy with past studies. Confining such papers to 2500 words reduces the likelihood that the discussion will be meaningful.
  7. The existence of such a ghetto to which these papers can be herded takes pressure off the vanity JCCP to reform its publication practices. Editors can perceive when studies are likely to be failed attempts at replications or null findings and issue desk rejections for manuscripts with a standard form letter suggesting resubmitting to the Journal of Psychotherapy Integration.
  8. pottery barn ruleProviding such a ghetto is APA’s alternative to acceptance of a Pottery Barn rule, whereby if JCCP publishes a clinical trial, it incurs an obligation to publish attempted replications, regardless of whether results are consistent with the study being replicated.
  9. Without journal reform, publication in JCCP represents a biased sampling of evidence for particular psychotherapies with a strong confirmation bias.

Clinical psychology doesn’t need such silliness

Initiatives such as this call for papers are a distraction from the urgent need to clean up the clinical psychology literature. We need to confront directly  JCCP‘s policy of limiting publication to articles that are newsworthy and that claim to be innovative, at the expense of being robust and solid clinical psychological science.

Some personality and social psychologists involved in the replication initiative have received recognition and endorsement from the two professional organizations competing for the highest impact factors in psychology, Association for Psychological Science and American Psychological Association. Those of us who’ve continue to call in the social media for reform of the vanity journals, are often met with a flurry of negative response from the replicators who praise the professional organizations for their commitment to open psychological science

Have the replicators sold out the movement to reform psychology by leaving the vanity journals intact? As I’ve argued elsewhere, compromises worked out for replicability project may adversely affect efforts to improve the trustworthiness of clinical  psychologicalscience, even if the stakes are higher.

 

 

 

How competition between APS and APA for prestige fueled the crisis of reproducibility in psychology

This blog post is truly a quick thought that needs refinement. I invite comments and criticisms, and particularly suggestions for more links and details to what I believe is an important message. Please correct me where you believe I am wrong or exaggerating.

 Counting papers is “a great way to measure whether a scientist puts ambition ahead of curiosity, scientific rigor and the quest for knowledge

stop_the_hype_by_thegoldenbox-d9ewr6xAssociation for Psychological Science (APS)   has strived for the distinction of being the more scientifically oriented of the two main organizations for American psychologists. APS was born out of an intense political struggle within its current rival, American Psychological Association (APS). Although historically the APA had eminent scientists as its presidents, a group of ambitious entrepreneurial psychologists were taking over. Their goal, and this “dirty dozen” succeeded, was to establish and accredit lucrative, freestanding professional training programs conferring PsyD degrees.

Once scorned, and confronted by seemingly insurmountable odds, the following fourteen professional psychologists, affectionately called The Dirty Dozen, changed for all time the face of the American Psychological Association (p. 270)

Unlike the existing PhD degrees, these degrees did not require dissertations, but could serve as the basis for licensure of clinical psychologists. These professional training schools de-emphasized research, as compared to traditional PhD programs, and the graduates were going to quickly make up the bulk of new members to APA. This made the previously dominant research-oriented psychologists uncomfortable. The shift in power was among the reasons many left. Some formed an alternative, first known as the American Psychological Society.

APS was politically and financially disadvantaged by its size, but also by the existing prestige of APA journals.  Offered at considerable discounts as part of the privileges of membership, they were an incentive for sticking with APA.

The stage was set for an intense competition between APS and APA for the claim to having the most prestigious journals.

Prestige is quantified by Journal Impact Factor (JIF). Despite the numerous deficiencies of JIFs has measures of scholarly accomplishment, they are widely accepted as measures of scientific accomplishment by hiring and promotion committees of universities and funding agencies.

Candidates competing for jobs, promotion, and grants can be quantitatively compared on the basis of the JIFs of the journals in which they publish, even if these numbers are distorted measures of actual accomplishment and easily gamed.

The exact algorithms by which JIFs are calculated are obscure. The JIFs reported by the authoritative  INCITES Journal Citation Reports cannot be precisely replicated. However, JIFs are somehow a quantification of the number of times on average a paper is likely to be cited within two years after it appears in print.

The validity of JIFs as predictors of how many times a particular paper will be cited is at its lower level since JIFs first came into use.

JIFs are easily gamed.

The notion of papers being published at a particular point in time is tied to an earlier period in which articles were primarily printed on paper, bundled in journals that were mailed an ultimately bound in volumes on library shelves.

The advent of Internet-based open access journals  with immediate availability on the web posed a challenge to traditional journals of the for-profit publishers and traditional professional organizations.

There was an open access advantage the traditional journals had to overcome. The response to this challenge was the creation of “early view” articles available on websites before articles were actually assembled into paper journals.

JIFs somehow calculated by the number of times a paper cited within two years after it “appears” in a journal. So, availability of “early view” articles could start the clock early in terms of accumulating citations before paper journals actually were physically available.

Researchers who will potentially cite a particular article X can quickly turn out another article Y with a citation of X that will contribute to the JIF of the Journal in which article X appeared.

The key is for these researchers to become aware as quickly as possible of the availability of “early view” articles that are candidates for citing.

Press releases and other notifications in the social media become particularly important for getting the articles cited that will raise JIFs.

In the competition between APS and APA, press releases become particularly important for raising JIFs.

But not just press releases, rather, press releases that report innovative and breakthrough studies that will get echoed and churnaled by lazy journalists across social and conventional media.

The innovative and the breakthrough study, or at least one that can be tortured into having the having these characteristics, gets favored over more transparently reported, trustworthy and durable studies.

Notoriously, an outgoing editor of APS’s Psychological Science   proudly declared that he had rejected over 6000 submissions in his five years as editor without the manuscript is going out to reviewers

Potential authors also wondered why papers are declined without full review (i.e., “triaged,” reflecting a policy to devote most editorial resources to papers that were considered stronger candidates for publication). That was an easier question to answer: Given the volume of papers that fell in this category (more than 6,000 submissions over 5 years), a few prototypes emerged. We even gave them names (which reflected the age of the editorial staff):

At the top of his three reasons was:

The Pink Floyd Rejection: Most triaged papers were of this type; they reported work that was well done and useful, but not sufficiently groundbreaking. So the findings represented just another brick in the wall of science.

Psychological science has no monopoly on “Pink Floyd rejections.” APA’s Journal of Personality and Social Psychology relies on the strategy for its competition with APS, but its editors just do not brag about it.

pink floyd study -page-0Bricks-in-the wall studies are denigrated. These include most research that is well conducted and transparently reported. Also included are null results and replications of previously published studies, particularly those that do not yield positive results, are not welcomed.

reject as Pink Floyd study

But such “bricks in the wall” are likely more trustworthy than the over 50% of Journal of Personality and Social Psychology and Psychological Science articles evaluated in the Open Science Collaboration’s (OSC) Replication Project: Psychology that did not reproduce with the same strength of effects.

APS and APA are in a vicious competition to accumulate citations by promoting hype. This contributes to the reproducibility crisis in psychology for which replication of initiatives are showy, but ineffectual solutions.

I have no interest in restoring the prestige of APA after lots of blows to it, including its protection of psychologists who facilitated and participated in “enhanced interrogation.”

But I’m pained by how its competitor for the loyalty of psychologists is so damaging itself by deliberate questionable publication practices. It is soiling its brand  with questionable and even junk science. How about we get a hashtag campaign going for APS – #stopthehype.