An open letter to the Cochrane Collaboration: Bill Silverman lies a-moldering in his grave

To whom it may concern:

cochraneI’m writing to ask you to consider the implications of having authors conduct a systematic review with the Cochrane collaboration brand attached who have ties to an industry which would benefit from particular conclusions. These same conclusions would personally enrich some of the authors professionally and personally.

Larun L, Odgaard-Jensen J, Brurberg KG, Chalder T, Dybwad M, Moss-Morris RE, Sharpe M, Wallman K, Wearden A, White PD, Glasziou PP. Exercise therapy for chronic fatigue syndrome (individual patient data) (Protocol).  Cochrane Database of Systematic Reviews 2014, Issue 4. Art. No.: CD011040. http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD011040/abstract

The Cochrane Collaboration allowing this authorship for the systematic review is egregious enough. However your acceptance of this configuration has a role in a monumental battle over the sharing of data promised as a condition for publication in PLOS One. In a reply to my request, they state my request for the data is vexatious and motivated by the intent to damage their credibility as professionals and the conclusions of their PLOS One paper. They cite their involvement in the Cochrane review as evidence of their willingness to share patient-level data with more reasonable, collaborative people. They do not say so explicitly, but they imply their data will be shared with those willing to commit to protecting the credibility of their conclusions.

So, we find ourselves on opposite sides in the battle for data sharing. I think you are either not maintaining proper oversight of your operations or simply being embarrassingly hypocritical in your position.

I write to you as one of the authors that you honored with the Bill Silverman award for a paper in The BMJ,

Roseman Michelle, Turner Erick H, Lexchin Joel, Coyne James C, Bero Lisa A, Thombs Brett D et al. Reporting of conflicts of interest from drug trials in Cochrane reviews: cross sectional study BMJ 2012; 345 :e5155 [http://www.bmj.com/content/345/bmj.e5155.full ]

bill cochraneI might be tempted to give you back your US$1000 except my esteemed co-authors and I gave the money entirely to our graduate student first author. It would be unfair to punish her for your hypocritical behavior. Yet, from what I know of Bill Silverman, he would heartily approve of my giving you the money back, if it were mine to give.

Because this is an open letter that will be read by an international audience, I will draw upon your description of who Bill Silverman was and what he represented to the collaboration, i.e., a troublemaker.

 William (Bill) Silverman (1924-2004) was one of the founders of American neonatal medicine. He was honoured repeatedly as one of the pioneers in his specialty; however, he often evoked somewhat contradictory responses amongst his colleagues because he was in the habit of raising troubling questions about the scientific basis and ethics of his and their practices. Like many of the people who have helped to establish Cochrane, Bill Silverman could be regarded as a ‘troublemaker’. As he reiterated frequently, however, criticism is a form of troublemaking that can help to drive progress. Furthermore, criticism should not be limited to examining the work of others, but should also include self-criticism.

The prize

…Acknowledges explicitly Cochrane’s value of criticism, with a view to helping to improve its work, and thus achieve its aim of helping people make well-informed decisions about health care by providing the best possible evidence on the effects of healthcare interventions. The Cochrane Steering Group approved the establishment of the Prize in 2007, and it was awarded for the first time in 2008.

The study that earned us the award examined

 Systematic reviews of drug interventions published in 2010 in the Cochrane Database of Systematic Reviews.

We found

Only 16 of the 151 Cochrane reviews (11%, 7% to 17%) provided any information on trial author-industry financial ties or trial author-industry employment. Information on trial funding and trial author-industry ties was reported in one to seven locations within each review, with no consistent reporting location observed

We concluded

Most Cochrane reviews of drug trials published in 2010 did not provide information on trial funding sources or trial author-industry financial ties or employment. When this information was reported, location of reporting was inconsistent across reviews.

Among other things, we recommended

That the Cochrane Collaboration reconsider its position that trial funding and trial author-industry financial ties not be included in the risk of bias assessment. The 2008 version of the Cochrane handbook listed “inappropriate influence of funders” (section 8.14.1.6) (for example, data owned by industry sponsor) as a potential source of bias that review authors could optionally incorporate in the “other sources of bias” domain of the Cochrane risk of bias tool.37 The 2011 version of the handbook, however, argues that “vested interests” should not be included in the risk of bias assessment, which “should be used to assess specific aspects of methodology that might be been influenced by vested interests and which may lead directly to a risk of bias” (section 8.15.1.5).38 As previously noted,22 empirical criteria are generally used to select items (for example, sequence generation, blinding) that are included in assessments of risk of bias,38 48 including evidence of a mechanism, direction, and likely magnitude of bias. Empirical data show that trial funding by pharmaceutical companies and trial author-industry financial ties are associated with a bias towards positive results even when controlling for other study characteristics6 8 49 50 and, thus, meet these criteria. One concern might be that including conflicts of interest from included trials in the risk of bias assessment could result in “double counting” of potential sources of bias. However, ratings in the risk of bias table are not summed to a single score, and inclusion of risk of bias from conflicts of interest could reflect mechanisms through which industry involvement can influence study outcomes6 that are not fully captured by the current domains of the risk of bias tool (random sequence generation, allocation concealment, blinding of participants and staff, blinding of outcome assessment, incomplete outcome data, selective reporting, and other sources of bias). Furthermore, even if all relevant mechanisms were to be assessed, the degree of their influence may not be fully captured when reviewers only have access to the relatively brief descriptions of trial methods that are provided in most published reports. Inclusion of conflicts of interest from included trials in the risk of bias assessment would encourage a transparent assessment of whether industry funded trials and independently conducted trials reach similar conclusions. It would also make it explicit when an entire area of research has been funded by industry and would benefit from outside scrutiny.

Consistent with my past experience criticizing the collaboration, you graciously received our criticism. And in the case of The BMJ article, you did something different as a result, you modified your Risk of Bias Assessment to take trial authors’ conflict of interest into account.

In the name of Bill Silverman, I ask you to take this review away from the current authors. You had the opportunity to know Dr.Silverman better than I do. What would he do?

I await your response.

Sincerely,

scanned signature-page-0-1

Jim Coyne

 

 

Advertisements

5 thoughts on “An open letter to the Cochrane Collaboration: Bill Silverman lies a-moldering in his grave

  1. In the acknowledgement of the protocol paper for looking at exercise treatments for ME they have an acknowledgement for
    “2013 via Peter D White’s academic fund (Professor of Psychological Medicine, Centre for Psychiatry, Wolfson Institute of Preventive Medicine, Barts and The London School of Medicine and Dentistry, Queen Mary University of London).”
    Who as well as being one of the protocol authors would be a beneficiary of a good review having promoted GET as a treatment for ME and done outcome switching for the PACE trial resisting attempts for others to get the data.

    Like

  2. I’m not sure the cochrane risk of bias assessment is up to much. Have a look at the assessment for PACE in
    http://onlinelibrary.wiley.com/enhanced/doi/10.1002/14651858.CD003200.pub4
    Here they class the risk of incomplete outcome data and reporting bias as low for PACE despite protocol changes. On being challenged by Tom Kindlon they reply

    “The Cochrane Risk of Bias tool enables the review authors to be transparent about their judgments, but due to the subjective nature of the process it does not guarantee an indisputable consensus. You particularly mention the risk of bias in the PACE trial regarding not providing pre-specified outcomes however the trial did pre-specify the analysis of outcomes. The primary outcomes were the same as in the original protocol, although the scoring method of one was changed and the analysis of assessing efficacy also changed from the original protocol. These changes were made as part of the detailed statistical analysis plan (itself published in full), which had been promised in the original protocol. These changes were drawn up before the analysis commenced and before examining any outcome data. In other words they were pre-specified, so it is hard to understand how the changes contributed to any potential bias. The relevant paper also alerted readers to all these changes and gave the reasons for them. Overall, we don’t think that the issues you raise with regard to the risk of selective outcome bias are such as to suspect high risk of bias, but recognize that you may reach different conclusions than us.”

    So according to Cochrane if a protocol for a non-blinded trial is changed prior to analysis then that still counts are pre-specified! even though the authors would have a good idea of what is going on.

    They also say its ok to change the measures of success as long as you don’t change what is being measured. That is like me saying success of an exercise program is walking a mile a day and then changing it to be walking a bit more than before – its ok because I’m measuring distance in each case.

    Like

  3. “This illustrates what we feel should become a cardinal rule: the need to separate the clinical evaluation of innovations from their innovators, who irrespective of any of their endeavours to be ‘neutral’ have a substantial investment, whether emotional, perhaps financial, or in terms of professional or international status, in the successful implementation of their idea. It is noteworthy, but perhaps incidental, that the finding of the Japanese trial in favour of external immobilisation is in contrast to a lack of differences between external and internal immobilisation found by the other two randomised trials, both of which were at high risk of bias only from lack of blinding.”
    http://www.cochranelibrary.com/editorial/10.1002/14651858.ED000081

    Like

  4. Reblogged this on ToTo NeuroImmunologisk Kurativ Behandling and commented:
    Blogganbefaling: James C Coyne sender åpent brev til Cochrane Collaboration 6mar2016.

    Coyne setter spørsmåltegn om publikasjonen og oversiktsstudien, Den nye Cochrane-oversikten som er en oppdatering av en oversikt fra 2004 av ; Larun L, Odgaard-Jensen J, Brurberg KG, Chalder T, Dybwad M, Moss-Morris RE, Sharpe M, Wallman K, Wearden A, White PD, Glasziou PP. Exercise therapy for chronic fatigue syndrome (individual patient data) (Protocol). Cochrane Database of Systematic Reviews 2014, Issue 4. Art. No.: CD011040, og enkelte av forfatters interressekonflikt med en industri som kan ha fordel av bestemte konklusjoner. Coyne refererer til en tidligere sak og gjennomgang hvor budskap er Risiko for Bias vurdering ved å ta med forskningsstudiene forfatternes interessekonflikt i betraktning.

    Fra Kunnskapssenteres sider 11feb2015 kan vi lese fælgende:
    Den nye Cochrane-oversikten er en oppdatering av en oversikt fra 2004 som viste at treningsbehandling hadde lovende resultater hos voksne med CFS/ME. Siden den gang er det publisert flere studier som vurderer effekt og sikkerhet av trening som behandling for voksne med CFS/ME.

    Forskerne sammenfattet resultatene fra åtte randomiserte kontrollerte studier som inkluderte til sammen 1518 voksne med CFS/ME.

    Fem studier ble utført i Storbritannia og de tre andre fra Australia, New Zealand og USA.

    Pasientene hadde vært syke mellom 2,3 og 7 år og de kunne ikke være sengeliggende eller i rullestol.

    I sju av studiene bestod behandlingen av aerobe treningsformer som å gå, svømme eller sykle. Én studie tok for seg anaerob trening, for eksempel tøyning eller styrkeøvelser. Treningsbehandlingen gikk over 12 til 26 uker, og begynte vanligvis med fem til 15 minutter per dag, ofte med gradvis økning over tid.

    Les mer: http://www.kunnskapssenteret.no/nyheter/treningsbehandling-kan-hjelpe-me-pasienter

    Omtale av Cochrane-oversikt – 2015, 11feb2015 endret 5mai2015:

    Hva er denne informasjonen basert på?

    Forfatterne av Cochrane-oversikten gjorde systematiske søk i aktuelle forskningsdatabaser i mai 2014, og fant åtte randomiserte kontrollert studier med til sammen 1518 personer som de inkluderte i oversikten. Diagnosen CFS/ME ble satt ved bruk av to ulike verktøy.

    I tre studier ble Center for Disease Control sine kriterier brukt, og i fem andre studier ble Oxford-kriteriene brukt.

    Treningsbehandlingen varte fra 12 til 26 uker. Syv studier benyttet variasjoner av aerob treningsbehandling somturgåing, svømming, sykling, dansing eller blanding av disse aktivitetene. Intensiteten varierte fra lav til ganske høy. I én studie ble det benyttet anaerob trening.

    Personene i kontrollgruppen fikk enten vanlig behandling eller avspenning, eller en kombinasjon av disse. To studier sammenliknet effekten av treningsbehandling med kognitiv atferdsterapi (CBT), en så på kognitiv terapi, en så på støttende samtale, en om pacing, en om farmakologisk behandling, og en om kombinasjonsbehandling. Oppfølgingstiden varierte.

    Treningsbehandling ble hovedsakelig gitt av en erfaren fysioterapeut, men i enkelte av studiene var det sykepleiere som sto for terapien. Det var relativt mange av deltakerne som også hadde andre typer plager eller tilstander, hovedsakelig depresjonssymptomer. Denne oversikten er en oppdatering av en tidligere Cochrane-oversikt.

    Les mer: http://www.kunnskapssenteret.no/publikasjoner/bedre-helse-med-treningsbehandling-for-personer-med-kronisk-utmattelsessyndrom

    Hvis du har spørsmål eller kommentarer til Cochrane-oversikten om CFS/ME kan du sende e-post til: cfsme@kunnskapssenteret.no

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s