John Ioannidis has a way of stirring the pot just enough to get people running in to see what’s cooking, but just short of sending many of them fleeing out the door, horrified by what they find. In a manner that should be studied very carefully by aspiring authors and wannabe John Ioannidises, he provides enough documented examples of what would otherwise be outrageous points. Sure, this usually doesn’t rise to level of a systematic review, and sometimes can lead to accusations of overstating the case. But these are nonetheless substantiated, solid evidence-based critiques.
He and his colleagues recently raised the question: Should NIH pull the plug on underperforming big ideas? in a October 4, 2016 Viewpoint in JAMA and now their article has produced an exchange of letters.
Excerpts of the recommendations of the October 16 paper:
Mechanisms should be in place to sunset underperforming initiatives. In the current environment, scientists are pigeonholed in a narrow discipline and are penalized by study sections if they exit their specific niche.
NIH deinvestment in preclinical research promises that clearly do not deliver will allow more funding to be directed toward work of clear public health importance and for imaginative biomedical research that is truly innovative and not constrained by current narratives.
To arrive at these conclusions, they brutally cut through the promises of six topics what is called personalized or precision medicine, plus stem cell therapy and electronic records.
As an example, stem cell therapy was called to task:
The prospects of effective treatment based on stem cells have been challenged in comprehensive reviews of the available trials. For instance, in congestive heart failure, improvements in cardiac function have been observed only in industry-sponsored studies, and a positive relationship has been noted between effect size and the number of experimental design flaws..
The sweeping conclusion:
None of these popular topics has had any measurable effect on population mortality, morbidity, or life expectancy in the United States. The improvements of the past decades in these outcomes, which have been substantial but are now stalling, have largely reflected improvement in nonmedical aspects of everyday life and the operation of broad-based public health and classic prevention efforts, such as curtailing smoking, that are undervalued as outmoded and old-fashioned by the narrative. The anticipation that improvements in medical care and outcomes derived from big ideas will reduce costs also seems unlikely given the high costs of applying targeted therapeutic interventions to small numbers of people based on complex and expensive technologies, as well as the inevitable overdiagnosis and overtreatment that follows from more intensive monitoring.
In getting there, the authors really nailed the anxieties of US medical school-based researchers who are confronted by such priorities and who have to earn their salary from NIH grant support:
Apparently a large number of scientists either believe in the potential of these topics or feel compelled to work on them, recognizing that these topics constitute a major locus of important science, financial support, recognition, and prospects for a successful career.
The viewpoint elicited this response:
..Although we agree that progress in subcellular biology (especially genetics) and health information technology (especially EHRs) has not yet achieved the anticipated promise, we believe that it is too early to judge the success or failure of these 8 big ideas on the basis of whether they have achieved “measurable reductions in mortality and morbidity.” If one were to judge all biological or information technology–related research using these criteria, researchers would be limited to studying ways to increase adoption of known public health interventions including: getting the general population to eat less and exercise more; reducing the incidence of drunk driving; increasing the use of seat belts, child restraints, and bicycle helmets; and reducing cardiovascular disease risk through aspirin, blood pressure control, cholesterol reduction, and smoking cessation.
To which Joyner, Ioannids, and Paneth replied:
Drs Miller and Sittig challenge our claim that the big ideas that have long dominated biomedical research have underperformed. Interestingly, they admit that morbidity and mortality can be reduced by “getting the general population to eat less and exercise more; reducing the incidence of drunk driving; increasing the use of seat belts, child restraints, and bicycle helmets; and reducing cardiovascular disease risk through aspirin, blood pressure control, cholesterol reduction, and smoking cessation.” Indeed, these interventions constitute a terrific research agenda! Why not devote more research funds to finding out how to make these interventions more efficient?
This is a familiar argument: what appears to be reductions in morbidity and mortality by medical advances actually being due to public health measures and socioeconomic change. But it needs restating here in the face of wild claims about the sufficiency of high tech “big ideas.” It also can be seen as a renewed call for basic health behavior and behavioral research that increasingly being neglected as not sexy enough without being attached to some dubious biomarker or unnecessary brain science.
Note: Nature, News, and Comment recently speculated that John Ioannidis was under consideration by the Trump administration as a possible replacement of Frances Collins as head of the National Institute of Health. While I find that prospect exciting, I have to dismiss this as just wishful thinking or even fake news.