Systematic review shows no improvement in quality of mindfulness research in 16 years

Should we still take claims about mental health benefits of mindfulness with a grain of  salt? A systematic review by one of mindfulness training’s key promoters suggests maybe so.

saltCritics have been identifying the same weaknesses in mindfulness research for almost two decades. This review suggests little improvement in 16 years the quality of randomized trials for mental health problems.

This study examined 171 articles reporting RCTs for:

(a) active control conditions, (b) larger sample sizes, (c) longer follow-up assessment, (d) treatment fidelity assessment, (e) reporting of instructor training, (f) reporting of ITT samples.

What was missed

Whether articles reporting RCTs had appropriate disclosure of financial or other conflicts of interest. COI pose significant risk of bias, especially when they are not reported.

This article discloses authors’ interests. One of the authors, Richard Davidson is a prominent promoter of mindfulness training.  A Web of Science search of Davidson RJ and mindfulness yielded 26 articles from 2002 to 2016. It would be interesting to check in see if these consistent weaknesses in mindfulness research are mentioned in these articles. To what extent do RCTs with Davidson as an author had these weaknesses, like being underpowered?

Critic: You say financial interests or other investments in a treatment are a risk of bias. Yet, this article is critical of mindfulness research. Wouldn’t you expect a more positive appraisal of the literature because of the authors having a confirmation bias?

Not necessarily. Conflicts of interest are a risk of bias, but don’t discredit an author, They only alert readers to be skeptical. Furthermore, the weaknesses in this literature are so pervasive, it would be difficult to put a positive spin on them.  Besides calling attention to specific weaknesses that need to be addressed in future research can become part of a pitch for more research.

The article

Goldberg SB, Tucker RP, Greene PA, Simpson TL, Kearney DJ, Davidson RJ. Is mindfulness research methodology improving over time? A systematic review. PLOS One. 2017 Oct 31;12(10):e0187298.

End of paper conclusion:

In conclusion, the 16 years of mindfulness research reviewed here provided modest evidence that the quality of research is improving over time. There may be various explanations for this (e.g., an increasing number of novel mindfulness-based interventions being first tested in less rigorous designs; the undue influence of early, high-quality studies). However, it is our hope that demonstrating this fact empirically will encourage future researchers to work towards the recommendations here and ultimately towards a clearer and scientifically-informed understanding of the potential and limitations of these treatments.

From the abstract


The current systematic review examined the extent to which mindfulness research demonstrated increased rigor over the past 16 years regarding six methodological features that have been highlighted as areas for improvement. These feature included using active control conditions, larger sample sizes, longer follow-up assessment, treatment fidelity assessment, and reporting of instructor training and intent-to-treat (ITT) analyses.

Data sources

We searched PubMed, PsychInfo, Scopus, and Web of Science in addition to a publically available repository of mindfulness studies.

Study eligibility criteria

Randomized clinical trials of mindfulness-based interventions for samples with a clinical disorder or elevated symptoms of a clinical disorder listed on the American Psychological Association’s list of disorders with recognized evidence-based treatment.

Study appraisal and synthesis methods

Independent raters screened 9,067 titles and abstracts, with 303 full text reviews. Of these, 171 were included, representing 142 non-overlapping samples.


Across the 142 studies published between 2000 and 2016, there was no evidence for increases in any study quality indicator, although changes were generally in the direction of improved quality. When restricting the sample to those conducted in Europe and North America (continents with the longest history of scientific research in this area), an increase in reporting of ITT analyses was found. When excluding an early, high-quality study, improvements were seen in sample size, treatment fidelity assessment, and reporting of ITT analyses.

Conclusions and implications of key findings

Taken together, the findings suggest modest adoption of the recommendations for methodological improvement voiced repeatedly in the literature. Possible explanations for this and implications for interpreting this body of research and conducting future studies are discussed.

Competing interests

RD is the founder, president, and serves on the board of directors for the non-profit organization, Healthy Minds Innovations, Inc. In addition, RD serves on the board of directors for the Mind and Life Institute. This does not alter our adherence to PLOS ONE policies on sharing data and materials

The variables examined in the systematic review

Six methodological features that have been recommended in criticisms of mindfulness research [10–12. 14]. These include: (a) active control conditions, (b) larger sample sizes, (c) longer follow-up assessment, (d) treatment fidelity assessment, (e) reporting of instructor training, (f) reporting of ITT samples.

…We graded the strength of the control condition on a five-tier system. We defined specific active control conditions as comparison groups that were intended to be therapeutic [17]. More rigorous control groups are important as they can provide a test of the unique or added benefit a mindfulness intervention may offer, beyond non-specific benefits associated with the placebo effect, researcher attention, or demand characteristics [11,14]. Larger sample sizes are important as they increase the reliability of reported effects and increase statistical power [11]. Longer follow-up is important for assessing the degree to which treatment effects are maintained beyond the completion of the intervention [10]. Treatment fidelity assessment allows an examination of the degree to which the given treatment was delivered as intended [12]. Treatment fidelity is commonly assessed through video or audio recordings of sessions that are coded and/or reviewed by treatment experts [18]. We coded all references to treatment fidelity assessment (e.g., sessions were recorded and reviewed, a checklist measuring adherence to specific treatment elements was completed). Relatedly, reporting of instructor training increases the likelihood that the treatment that was delivered by qualified individuals [12], which should, in theory, influence the quality of the treatment provided. Lastly, the reporting of ITT analyses involves including individuals who may have dropped out of the study and/or did not complete their assigned intervention [12]. Generally speaking, ITT analyses are viewed to be more conservative estimates of treatment effects [19,20], and are preferred for this reason.


Unethical: Why Bristol University SMILE trial should not have been conducted with chronically ill children

If it should have been conducted at all, the first clinical trial of Lightning Process should not have been conducted with chronically ill children. Safety and efficacy had neither been established with adults, nor healthy children.

Phil Parker, developer of the trademarked, commercial Lightning Process package claims that is not a psychological treatment, but a physical one. He further claims its scientific basis is an amalgamation of neurolinguistic programming, osteopathy, and life coaching.

not psychologicalThere is no plausible scientific mechanism by which Lightning Process would work. A small minority of vulnerable adult and child patients with chronic fatigue syndrome are drawn to the false claims of the health benefits lightning process, which the UK Advertising Standards Authority now forbids posting on the web. This is a truth-in-advertising-problem, not justification for launching a clinical trial. 

Researching the Lightning Process is no more ethically and scientifically justified than researching Prince Charles’s claims that (organic) coffee enemas can slow progression of cancer. Yet, the logic is the same that has been used to justify the SMILE trial.

Protecting-children-520x350Pediatrician Esther Crawley should undergo remedial ethics training and I recommend an excellent source below.

No parents should consent to their children participating in clinical trials of Bristol University, until a transparent independent inquiry reports how and why the SMILE trial was approved.

A belated praise to the parents who stood up against Professor Esther Crawley and all her nastiness. She should apologize to you.

Required reading for Professor Crawley and relevant administrative staff of Bristol University

Ethical Conduct of Clinical Research Involving Children. Edited by Marilyn J. Field and Richard E. Behrman. 448 pp., illustrated. Washington, D.C., National Academies Press, 2004. $57. ISBN 0-309-09207-8

Although the price is listed as US$57. A free downloadable PDF is available here 

No excuse, Professor Crawley, overcoming your being ethically challenged can start with some free reading.

An excellent summary is here Institute of Medicine (US) Committee on Clinical Research Involving Children; Field MJ, Behrman RE, editors. Ethical Conduct of Clinical Research Involving Children. Washington (DC): National Academies Press (US); 2004. Summary.

Ethical Conduct of Clinical Research Involving Children.






Some key excerpts

The special ethical and regulatory protections for children may preclude potentially important clinical studies that would be approved for adult participation. This prospect can put pressure on those involved in developing or reviewing studies that include infants, children, or adolescents. A strong system of protections for adult and child participants in research will provide support and guidance for all involved to help them fulfill their legal and ethical responsibilities in such situations.


Notwithstanding the expected benefits of policies to increase the amount of research involving infants, children, and adolescents, some caution is appropriate. Unlike most adults, children usually lack the legal right and the intellectual and emotional maturity to consent to research participation on their own behalf. Their vulnerability demands special consideration from researchers and policymakers and additional protections beyond those provided to mentally competent adult participants in research.


As discussed later in this chapter, instances of unethical research practices involving children have prompted public criticism and concern that has contributed to the development of current federal regulations to protect both child and adult participants in research. Since the 1960s, policymakers, researchers, research institutions, and research sponsors have taken a number of steps to strengthen ethical standards and policies for human research and to create formal programs, including institutional review boards (IRBs), to approve and monitor research. Clinical studies funded, conducted, or regulated by the government are now subject to a (mostly) common set of provisions for the protection of human participants in research, including special protections for children. One result is that some potentially important clinical studies that would be approved for adult participation cannot be approved for participation by children.

a9062388703d9f42237ea708b2e088f1--protective-boyfriend-quotes-protective-quotesParents’  “intimate and profound duty to protect and promote their child safety and well-being in research”

…The committee recognizes the important role of parents. They have a most intimate and profound duty and desire to protect and promote their child’s safety and well-being in research, as in all realms of life. Chapter 5, in particular, has discussed how investigators, IRBs, and others can effectively and compassionately support parents in fulfilling their responsibilities and, thereby, help them to feel that they have done the right thing for their child, whatever their choices about the child’s participation in research. Once parents have agreed to their child’s participation in research, they—and older children and adolescents—may sometimes have crucial responsibilities for following the research protocol (e.g., administering medicines or bringing the child in for research appointments). Investigators need to make sure that parents and older children and adolescents understand any such responsibilities before they agree to research participation and that they have appropriate support in adhering to the protocol during the course of the research

Selected Recommendations

Recommendation 4.1: In evaluating the potential harms or discomfort posed by a research protocol that includes children, investigators, and reviewers of research protocols should

Interpret minimal risk in relation to the normal experiences of average, healthy, normal children;

Focus on the equivalence of potential harms or discomfort anticipated in research with the harms or discomfort that average, healthy, normal children may encounter in their daily lives or experience in routine physical or psychological examinations or tests;

Consider the risk of harms or discomfort in relation to the ages of the children to be studied; and

Assess the duration as well as the probability and magnitude of potential harms or discomfort in determining the level of risk.

In Section 406 of 45 CFR 46, federal regulations permit research that involves a minor increase over minimal risk without the prospect of direct benefit if the research involves children with a disorder or condition, is likely to yield vital knowledge about that disorder or condition, and entails research experiences that are reasonably similar to those that such children encounter in certain other situations. Consistent with the interpretation of minimal risk, the interpretation of this level of research risk should not allow a higher threshold of risk for children who are exposed to more risk in other aspects of their lives (Recommendation 4.2). Also, consistent with the language of the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, which defined this standard in 1977, the risk allowed under this category can be only slightly above minimal risk.

Recommendation 4.3: In determining whether proposed research involving a minor increase over minimal risk and no direct benefit can be approved, the term condition should be interpreted as referring to a specific (or a set of specific) physical, psychological, neurodevelopmental, or social characteristic(s) that an established body of scientific evidence or clinical knowledge has shown to negatively affect children’s health and well-being or to increase their risk of developing a health problem in the future.

Recommendation 4.1 In evaluating the potential harms or discomfort posed by a research protocol that includes children, investigators, and reviewers of research protocols should

Interpret minimal risk in relation to the normal experiences of average, healthy, normal children;

Focus on the equivalence of potential harms or discomfort anticipated in research with the harms or discomfort that average, healthy, normal children may encounter in their daily lives or experience in routine physical or psychological examinations or tests;

Consider the risk of harms or discomfort in relation to the ages of the children to be studied; and

Assess the duration as well as the probability and magnitude of potential harms or discomfort in determining the level of risk.

Parents should have been warned about unprofessional providers in Esther Crawley’s SMILE trial

The providers of Lightning Process in the SMILE trial were not credentialed health professionals and are not bound by ethical codes.

giphyEsther Crawley’s overdue publication of results from the SMILE trial  deserves close scrutiny for lots of reasons. But especially for what is said about the supervision and qualifications of Lightning Process practitioners who delivered the treatment.

Readers can easily gloss over a seemingly straightforward description of these practitioners in the paper. Yet, a little probing and checking of sources should cause some alarm among the parents who agreed to their children participating in the trial.

The details are alarming enough to cause worry about participating in any clinical trial involving Esther Crawley. She does not look after the safety and rights of children in her care.

This trial was approved by a number of committees at Bristol University. Assured by those well-publicized approvals, parents undoubtedly assumed that their children were protected from risks and needless exposure to quack treatments or providers not bound by enforceable ethical codes. The children did not have these protections, which should give pause to anyone contemplating allowing their children to participate in a trial regulated by Bristol University.

What was said in the published paper

LP practitioners have completed a diploma through the Phil Parker Training Institute in Neurolinguistic Programming, Life Coaching and Clinical Hypnotherapy. This diploma is examined through written and practical examinations and is accredited by the British Institute of Hypnotherapy and NLP. Following the diploma, LP practitioners undertake a further course to learn the tools and delivery required for the LP after which they must pass both a practical and written examination. Practitioners undertake supervision and continuous professional development in order to further develop their skills and knowledge. They are regulated by the register of LP practitioners, adhere to a code of conduct and there is a Professional Conduct Committee that oversees complaints and professional practice issues.

Let’s isolate and probe some of these statements.

LP practitioners have completed a diploma through the Phil Parker Training Institute in Neurolinguistic Programming, Life Coaching and Clinical Hypnotherapy.

advanced practitionerComment: This is does not represent anything approaching a degree from an accredited university or training institute. Phil Parker has been sanctioned by the UK Advertising Standards Authority for false claims about treating and even curing complex medical conditions. Yet the SMILE trial presented the Lighting Process as a treatment to the parents and children.

It would be fascinating to debrief the parents to discover what they assumed in consenting to their children’s participating in the trial. Did they think the children were being treated? Were they informed that the ASA forbade billing Lightning Process as a treatment? That the treatment providers were not chartered or qualified medical or mental health providers?

Neurolinguistic Programming has been widely discredited as quackery and is no longer is discussed in the scientific or clinical literatures.

Life Coaching is so vague a term as to be meaningless. There is no regulation or consistent specification of what is done or forbidden to be done in life coaching.

Clinical Hypnotherapy is not a restricted, independently recognized, nor chartered class of provider. Anyone can call themselves a clinical hypnotherapist. Phil Parker’s institute issues meaningless certificates that be hung on providers’ walls to confuse consumers.

This diploma is examined through written and practical examinations and is accredited by the British Institute of Hypnotherapy and NLP.

 Comment: Sounds impressive, huh? The ‘written and practical examinations’ involve showing adeptness in delivering a quack treatment that involves making false claims and offering pseudoscientific explanations to patients- messages defined by those who sell the training products.  Ability to meet these criteria involves taking trademarked training courses in quackery. Any deficiencies in knowledge or practice that examiners perceive are remedied by taking more courses. Essentially, this is a pyramid scheme from which Phil Parker benefits financially.

Reassured that providers are accredited by “British Institute of Hypnotherapy and NLP?” Go visit the website and you can readily see that it is an extension of Phil Parker’s marketing effort, not something independent or otherwise chartered or regulated. The site offers courses in referrals to people who have completed courses.

At the website a rotating variety of pseudoscientific explanations are provided with claims of dramatic effects of hypnosis that are at odds with the evidence. Visitors are led to believe that hypnotherapy is effective for reducing blood pressure and stopping smoking:

Hypnotherapy can help you kick the habit of smoking into touch for good.

Compare that to what is said by the Cochrane Collaboration:

Does hypnotherapy help people who are trying to stop smoking?

We have not shown that hypnotherapy has a greater effect on six-month quit rates than other interventions or no treatment. There is not enough evidence to show whether hypnotherapy could be as effective as counselling treatment. The effects of hypnotherapy on smoking cessation claimed by uncontrolled studies were not confirmed by analysis of randomized controlled trials.

The site also falsely claims that hypnotherapy can reduce blood pressure. The site is basically an advertisement for exaggerated claims about hypnosis backed by pseudoscientific explanations of how it works.

Hypnotherapists now understood neurology and physiology of using hypnosis and have the evidence of neurosciences support this understanding.

not psychologicalNonsense. This British Institute of Hypnotherapy and NLP is set up and run by a guy who claims neuroscience, osteopathy and neurolingustic programming explain the effects of the Lightning  Process, which are claimed to be physiological, not psychological. Who believes these additional claims about hypnosis are valid?

Consumer beware.

Practitioners undertake supervision and continuous professional development in order to further develop their skills and knowledge.

But the supervision is in use of a commercial product with a pseudoscientific explanation. So, apparently practitioners are supervised in their willingness and ability to adhere to the scripts they have received as part of products they have purchased. The “continuous professional development” is taking a steady stream of Phil Parker’s training courses. Because his materials are trademarked, no one who is not authorized by Parker to sell his merchandise.

They are regulated by the register of LP practitioners, adhere to a code of conduct and there is a Professional Conduct Committee that oversees complaints and professional practice issues.

There is a financial conflict of interest in Phil Parker offering courses and training and what the ASA has said cannot be described as treatment and then setting up a professional conduct committee is tied directly to his products.

What should parents be able to expect in bringing their children to a recognized, regulated healthcare site, like the NHS? The treatment providers should be professionals licensed by a governmental or independent body separate from any marketing of training. If not fully licensed or regulated professionals, they should be supervised by professionals, such as medical or mental health clinicians.

The class of providers should also be licensed or chartered so that sanctions with legal consequences can be applied for misconduct or unethical behavior. Providers working with children should be held to some ethical standards in terms of the safety and rights of both children and their parents.

None of these features are present in the treatment provided in the SMILE trial.

Christine Fenton, a retired Deputy Head of two high schools in the UK notes she was responsible for child protection in those roles. She expressed this concern about the basic conflict of Lightning Process and tenets of childhood protection:

LP participants are directed to not talk to others about it – keep secrets – to report positively regardless of their internal view is appalling to me. Child Protection has a key tenet ‘secrets are not ok’ – if an adult tells a youngster to keep a secret it is a form of control & creates an environment in which abuse can occur more easily.

Teachers, like other professionals coming in contact with children are required to take courses about child protective services and the duty to report abuse. They are held legally responsible for violation of the associated rules. However, legally, providers of the lightning process are only ordinary citizens, not professionals and are not held responsible in this manner.

In obtaining approval for the conduct of the SMILE trial from Bristol University, Pediatrician Esther Crawley claimed that sessions of the writing process were taped and that she had responsibility for reviewing the tapes. It strains credibility to claim that she actually listened to all the tapes from the study.

But what does it mean for a physician to supervise a quack treatment provided by quack providers who are not being allowed to call what they do treatment? How does Crawley react when physically ill, impaired children are instructed to deny their fatigue and pain to authority figures such as teachers and parents?

Truly mind-boggling to consider.

Bristol University should start cleaning up its clinical trial registries by publishing two psychosocial intervention studies

A story in the feisty Bristol Cable revealed Bristol University ignores its responsibilities to publish results of clinical trials. A follow up story announced the university’s pledge to remedy the situation.

The investigation of hidden data involved drug and medical device trials and excluded psychotherapy or psychosocial intervention trials. But here are two trials for which publication is overdue.

One of the trials evaluates a quack treatment, which the Advertising Standards Authority (ASA) advises cannot make claims about health benefits. Its promoter was paid handsomely for the treatment provided in the trial. He capitalized heavily on the trial to advertise his treatment as being evaluated in a clinical trial. Was the trial simply a lucrative infomercial for his discredited product?

box score

There is some evidence that these trials did not go as planned, but that is all the more reason to publish results so that we can understand what went wrong.

lack of transparency

The situation is sufficiently problematic for me to recommend:

Patients should not enroll in psychotherapy trials associated with Bristol University until results of these trials are published.

Patients should be advised that the burden and inconvenience of participating in research is not warranted if the results remain unpublished. You are not obligated to participate in research. Don’t let anyone tell you differently.

A headline in the investigative journalism platform Bristol Cable read:

Medical research in Bristol violates international transparency …

A joint investigation by TranspariMED and the Bristol Cable shows that the University of Bristol has not posted summary results for a single clinical trial onto registries. On the main European trial registry, the university is listed as the sponsor of 14 drug trials involving over 1,600 healthy volunteers and patients. Three trials that ended in 2009, 2011 and early 2016 have still not posted results there. Most of the remaining 11 trials also seem to have been completed, but are still listed as ‘ongoing’, suggesting that the university is not keeping its medical research records up to date.

On, the main American registry, the university has registered 16 research projects; it has posted summary results for none of them.

A follow up headline read

Bristol University pledges clinical trial registry cleanup

A spokesperson for the university responded that Bristol had recently started adding results to the European registry. He also explained that until last year, “anyone could set up an account naming the University as the umbrella organisation” on The university has now assumed central control over those registry entries and is working to clean up its data, the spokesperson said.

The founders of the Bristol Cable told The Guardian: ‘We’re holding people to account for their actions’

The investigation was part of a larger international effort to uncover and correct a scandal: results of many drug trials are never published. Or published reports switch outcomes to make the manufacturers’ drugs appear more effective than they actually were in the trials.

Psychotherapy trials are excluded from these investigations. Yet, publishing results of psychotherapy trials consistent with the originally planned outcomes is undoubtedly less common because of lack regulatory oversight of psychotherapy trials. The confirmation bias in published of psychotherapy trials could even be stronger than with drug trials.

But we really cannot get a precise estimate of how many results of psychotherapy trials are being kept out of view. Many psychotherapy researchers do not even register their trials – or register only after they have already peeked at some of their results and can revise what were the originally intended outcomes being investigated.

This situation is unacceptable because it allows the promotion of ineffective and even harmful treatments.

Moreover, investigators have a moral and ethical responsibility to patients who consent to participate in clinical trials. In a trial comparing an intervention the investigators expect to be effective to a control group, half of the patients will not be randomized to the intervention. In many trials, this means the patients will be assigned to the no-treatment control group and get nothing. But these patients will be burdened with extra visits to complete the outcome assessments needed to evaluate the active treatment given to the patients in the other group.

Presumably, if the patients assigned to the control group are not simply confused or misinformed, they voluntarily continue to show up because they believe they are contributing to science and to knowledge that will help others in their predicament, if not themselves.


The first trial

The trial registration is ISRCTN72645894 DOI 10.1186/ISRCTN72645894

Early intervention in fatigue: a feasibility study

Study hypothesis

The overall aim of this study is to investigate the feasibility and acceptability of conducting a randomised controlled trial (RCT) to investigate the effectiveness and cost effectiveness of early intervention for chronic fatigue syndrome (CFS)/myalgic encephalomyelitis (ME) compared with standard medical care in primary care.


Early intervention for fatigue. The intervention is based on the principles of cognitive, behavioural and graded exercise and is delivered by a trained therapist in an individual face-to-face session with telephone follow-up sessions. Follow up length: 6 month(s); Study entry: single randomisation only

Primary outcome measure: Recruitment, adherence and follow up. Timepoint(s): 3 and 6 months

Secondary outcome measures: Not provided at time of registration

Note that the primary outcomes are not patient clinical outcomes, i.e., whether patients benefited from being assigned to active treatment versus control. The study was explicitly described as a feasibility and acceptability study.

Psychotherapy studies are commonly described in the literature as pilot and feasibility studies, mainly to escape the criticism that the study was small and underpowered.  But, if so, effect sizes for patient outcomes should be presented because they are likely to be exaggerated and not generalizable to what would occur in a larger, better controlled study or in routine practice.

I wonder if the consent forms provided to patients indicated that the investigators were not interested in whether they benefited from treatment.

What results of a freedom of information act request revealed

Feasibility: We aimed to recruit 100 participants to this study, but were only able to recruit 44.  In addition, 9 (32%) who should have received EI did not have all the sessions offered.

Acceptability: Although 12 of the participants told us that some of EI was useful, 11 of these participants also found some parts unacceptable. Four participants were still experiencing fatigue after completing EI and two felt they wanted more help at the end. Four of those we talked to did not find EI acceptable and withdrew. Reasons included: not liking the treatment model, or the way it was delivered.

Fatigue, pain, mood and work status were no different between the two groups at the end of the study however the study was not large enough to show an effect.

Our results show that this type of study is not feasible and EI was not acceptable to a lot of participants

It is a disservice to the patients who participated in the trial that there is no published acknowledgement that the study did not reach its accrual goal because patients did not wish to participate.  Patients who consented to treatment did not generally find it helpful.  Publication of this study could serve as a warning that most primary care patients suspected of having chronic fatigue syndrome (CFS)/myalgic encephalomyelitis (ME) are not interested in treatment based on the principles of cognitive, behavioural and graded exercise. The minority who consented to participate did not find the experience helpful.

The second trial: SMILE

 The trial registration is

The study protocol:

Crawley E, Mills N, Hollingworth W, Deans Z, Sterne JA, Donovan JL, Beasant L, Montgomery A. Comparing specialist medical care with specialist medical care plus the Lightning Process® for chronic fatigue syndrome or myalgic encephalomyelitis (CFS/ME): study protocol for a randomised controlled trial (SMILE Trial). Trials. 2013 Dec 26;14(1):444.

An overlapping pilot study

This protocol is also cited for a pilot for which no full report was published, only a summary. The results of this pilot study resulted in the outcomes being switched for the main study.

Crawley E, Mills N, Beasant L, Johnson D, Collin SM, Deans Z, White K, Montgomery A. The feasibility and acceptability of conducting a trial of specialist medical care and the Lightning Process in children with chronic fatigue syndrome: feasibility randomized controlled trial (SMILE study). Trials. 2013 Dec 5;14(1):415.

Was this really a separate pilot study or an abortive failed start to the study described in the protocol? We can’t tell.

Among the  sparse details from the pilot study:

Recruitment rates were lower than anticipated, meaning that recruitment continued for longer than planned. This was mainly due to an underestimate of the number of eligible patients at the start of the study (nearly half were not eligible). However, a recruitment rate of 35.9% does not preclude a full study.


In addition to the difficulty measuring change in school attendance for those transitioning from GCSEs to A levels, it may be a poor outcome measure for those who do not consider school attendance their primary goal. We suggest that a full study uses other primary outcomes, such as the SF-36 or the Chalder Fatigue Scale and uses school attendance as a secondary outcome.

So, at a time when other evaluations of interventions for ME/CFS were undergoing coordinated switching of outrcomes because of null findings from the FINE trial with the original scoring, the SMILE trial seems to have switched outcomes as well.

The treatment being evaluated in the main study is described as:

The Phil Parker Lightning Process® (LP) is a trademarked intervention that is used for a variety of conditions including CFS/ME. It has been developed from osteopathy, life coaching and neuro-linguistic programming. The intervention includes three group sessions on consecutive days where young people are taught skills that they can try out for themselves including looking at their sitting and standing posture. Families currently pay approximately £620 to attend the LP course.

Even though over 250 children and young people a year use the LP as an intervention for their CFS/ME, there are currently no reported studies investigating the effectiveness or possible side effects (for example serious adverse events) of the LP.

This is an exceedingly weak rationale for inflicting a quack treatment on unsuspecting children. Ethically, there must be a scientific rationale for exposing children to an unproven treatment that might be ineffective or even harmful.

The Advertising Standards Authority (ASA) guidance has ruled that LP cannot be advertised as a treatment:

To date, neither the ASA nor CAP [Committee of Advertising Practice] has seen robust evidence for the health benefits of LP. Advertisers should take care not to make implied claims about the health benefits of the three-day course and must not refer to conditions for which medical supervision should be sought.

Yet, the information sheet for the trial is describing it as a treatment to patients.

The materials patients must read as part of participating in the trial reek of pseudoscientific misinformation.

LP trains individuals to recognize when they are stimulating or triggering unhelpful physiological responses and to avoid these, using a set of standardized questions, new language patterns and physical movements with the aim of improving a more appropriate response to situations.

* Learn about the detailed science and research behind the Lightning Process and how it can help you resolve your issues.

* Start your training in recognising when you’re using your body, nervous system and specific language patterns in a damaging way

 What if you could learn to reset your body’s health systems back to normal by using the well researched connection that exists between the brain and body?

The Lightning Process does this by teaching you how to spot when the PER is happening and how you can calm this response down, allowing your body to re-balance itself.

 The Lightning Process will teach you how to use Neuroplasticity to break out of any destructive unconscious patterns that are keeping you stuck, and learn to use new, life and health enhancing ones instead.

 The Lightning Process is a training programme which has had huge success with people who want to improve their health and wellbeing.

The intervention imposes strange demands on patients that encourage dishonesty, including in completing the outcome evaluations:

  1. Responsibility of patients:

Believe that Lightning Process will heal you. Tell everyone that you have been healed. Perform magic rituals like standing in circles drawn on paper with positive Keywords stated on them. Learn to render short rhyme when you feel symptoms, no matter where you are, as many times as required for the symptoms to disappear. Speak only in positive terms and think only positive thoughts. If symptoms or negative thoughts come, you must stretch forth your arms with palms facing outward and shout “Stop!” You are solely responsible for ME. You can choose to have ME. But you are free to choose a life without ME if you wish. If the method does not work, it is you who are doing something wrong.

Bristol University at first rejected a freedom of information act requests for how much developers of the treatment were paid but on appeal revealed mean cost of a course for trial participants was £567. This means £14,000 wasted on a junk intervention in the pilot study and much more was paid for the full trial.

Special thanks to John Peters for his assistance and his excellent detective work and use of the Freedom of Information Act.

See his blog posts here:

THE SMILE TRIAL (part 1): Why the trial should never have been allowed in the first place.

THE SMILE TRIAL (part 2): A trial so flawed as to be worthless.

THE SMILE TRIAL (part 3): The only one to benefit is Parker. 

And also my post

Parker start your journey


Part 2: What to look for in a Special Issue of Journal of Health Psychology concerning the PACE trial

Part 2: What to look for in the Special Issue of Journal of Health Psychology concerning the PACE trial

In a continuation of the last blog post, this one provides brief summaries and links to eight additional contributions to the special issue of Journal of Health Psychology. These articles include a reply to the PACE investigators’ response from Keith Geraghty, a defense of the trial from some friends and supporters (Petrie and Weinman) of the PACE investigators, and a commentary from Susanna Agardy on the conflicts of interest in this defense.

special issue spread

Apologies to the authors and readers for my not indicating yesterday that there were an additional eight summaries coming. Because of all the uncertainty about whether the special issue would be published on schedule or blocked, yesterday’s blog post was uploaded at the last minute and incomplete.

When the effort to block publication of the special issue failed, the PACE investigators got criticism posted at Science Media Centre. One commentary was attributed to anonymous sources at Oxford University and simply parroting things that the PACE investigators themselves have been saying all along. Another commentary was from Malcolm MacLeod, a former colleague of Michael Sharpe at University of Edinburgh, who also authored a paper with him. McLeod claims to have read the issue and found nothing of substance there. That was an amazing feat, because the Science Media Centre posted this critique only minutes before the special issue was available. We might conclude that McLeod is a speed reader. Judging from the superficiality of his comments, it could simply be a matter of his not having read the special issue

Who would’ve thought that a special issue of Journal of Health Psychology could garner such publicity?

McLeod also indiscreetly disclosed that he was a member of a committee advising PLOS One on whether to release to me the data from an article on the PACE trial. Availability of the data had been promised as a condition for publishing in PLOS One. Yet, over a year after my request, it has still not been provided. However the PLOS One article now prominently features an Expression of Concern, which often precedes a retraction. Regardless, McLeod’s disclosure cast doubt on the integrity of the decision-making process at PLOS One, given his obvious conflict of interest.

As a critic of the PACE trial from the United States, I noticed early on that I was dealing with a tight close-knit network, strongly interconnected with the British establishment, a real charmed circle. The opposition to the publication of the special issue of Journal of Health Psychology and the PACE investigators’ access to resources like Science Media Centre and placement of an article in The Times and who served as go-to sources put this network on public display.

However, the brouhaha and the foiled effort to block publication of the special issue and this extraordinary attentionin the media has  paradoxically generated considerable buzz about the special issue that would otherwise not have. We appreciate all the publicity we can get and we hope the PACE investigators will bring on some more. But for now, check out the brief summaries below and click on the links to the open access articles.

Special thanks to blogger John Peters for having put together these links and summaries.

 Eight articles from the Special Issue

Distress signals: Does cognitive behavioural therapy reduce or increase distress in chronic fatigue syndrome/myalgic encephalomyelitis?   Keith R Laws

Reducing the psychological distress associated with chronic fatigue syndrome/myalgic encephalomyelitis is seen as a key aim of cognitive behavioural therapy. Crucially, the claim by National Institute of Clinical Excellence that cognitive behavioural therapy reduces distress in chronic fatigue syndrome/myalgic encephalomyelitis is not only at odds with what patients repeatedly report in surveys, but with their own gold-standard randomised controlled trial and meta-analytic data.

Cognitive behaviour therapy and objective assessments in chronic fatigue syndrome Graham McPhee

Most evaluations of cognitive behavioural therapy to treat people with chronic fatigue syndrome/myalgic encephalomyelitis rely exclusively on subjective self-report outcomes to evaluate whether treatment is effective. A review of studies incorporating objective measures suggests that there is a lack of evidence that cognitive behavioural therapy produces any improvement in a patient’s physical capabilities or other objective measures such as return to work.

 PACE trial claims for recovery in myalgic encephalomyelitis/chronic fatigue syndrome – true or false? It’s time for an independent review of the methodology and results Charles Bernard Shepherd

Patient evidence has repeatedly found cognitive behaviour therapy is ineffective and graded exercise therapy can make the condition worse. The PACE trial methodology has been heavily criticised by clinicians, academics and patients. A re-analysis of the data has cast serious doubts on the recovery rates being claimed. The trust of patients has been lost. The medical profession must start listening to people with myalgic encephalomyelitis/chronic fatigue syndrome if trust is going to be restored.

The PACE trial: It’s time to broaden perceptions and move on Keith J Petrie, John Weinman

Differing beliefs about the causes of chronic fatigue syndrome still influence how scientific studies in this area are accepted and evaluated. Causal beliefs about chronic fatigue syndrome and a modern version of Cartesian dualism are important in understanding the reaction to the PACE trial. Science is incremental. An unfortunate outcome of the PACE controversy and intimidation of researchers may be less research in the area. It is time to move on from criticism and collect more data on effective treatments.

Defense of the PACE trial is based on argumentation fallacies Steven Lubet

In defense of the PACE trial, Petrie and Weinman employ a series of misleading or fallacious argumentation techniques, including circularity, blaming the victim, bait and switch, non-sequitur, setting up a straw person, guilt by association, red herring, and the parade of horribles. These are described and explained.

Chronic fatigue syndrome patients have no reason to accept the PACE trial results: Response to Keith J Petrie and John Weinman Susanna Agardy

Petrie and Weinman’s urging to accept the findings of the PACE trial is unreasonable in view of the failure of PACE to achieve evidence of recovery. The Institute of Medicine describes chronic fatigue syndrome not as psychological but as a serious, chronic, systemic disease, with post-exertional malaise as its main feature which inhibits exercise. Linking debate about PACE with intimidation of researchers is unjustifiable and damaging to patients.

Further commentary on the PACE trial: Biased methods and unreliable outcomes Keith J Geraghty

The PACE authors respond that ‘Dr Geraghty’s views are based on misunderstandings and misrepresentations of the PACE trial’. This article draws on expert commentaries to further detail the critical methodological failures and biases identified in the PACE trial, which undermine the reliability and credibility of the major findings to emerge from this trial.

PACE-GATE: An alternative view on a study with a poor trial protocol Bart Stouten

The controversies surrounding the effectiveness of cognitive behavioural therapy and graded exercise therapy for chronic fatigue syndrome are explained using Cohen’s d effect sizes rather than arbitrary thresholds for ‘success’. This article shows that the treatment effects vanish when switching to objective outcomes. The preference for subjective outcomes by the PACE trial team leads to false hope. This article provides a more realistic view, which will help patients and their doctors to evaluate the pros and cons.

Global expert on distant and faith healing chaired PACE Data Monitoring and Ethics Committee

mild distance-healing1Paul Dieppe, the initial Chair of Data Monitoring and Ethics Committee for the PACE trial is also renowned as a “leading global voice in the understanding and advancement of energy/ spiritual healing.”

Most of my medical colleagues think I’ve gone completely barmy –  Paul Dieppe

Renowned as a leading global voice in the understanding and advancement of energy/ spiritual healing

Paul Dieppe, the Chair of Data Monitoring and Ethics Committee of the PACE trial of cognitive behavior therapy and graded exercise therapy for chronic fatigue syndrome has had a distinguished career, up to a point. He has emeritus status at the University of Exeter Medical School and is described on the website:

Paul qualified as a doctor in London in 1970. He specialised in rheumatology and became ARC professor of rheumatology in Bristol in 1987, and Dean of the Bristol Faculty of Medicine between 1994 and 1997. He then switched to health services research and was the Director of the MRC Health Services Research Collaboration between 1997 and 2007. After that he spent a short time at the University of Oxford before moving to the Peninsula Medical School to work in clinical education research.

With the formation of the University of Exeter Medical School, he took on the role of Professor of Health and Wellbeing.

Yet a more recent (2016) website gives a different picture of him:

exeter professor discovers

And gives a chance for  Dieppe to describe himself:

Paul, now renowned as a leading global voice in the understanding and advancement of energy/ spiritual healing, said: “My goal is to work towards the greater understanding that we can bridge the gap between biomedical thinking and the world of unexplained phenomena around areas such as healing, which if we placed greater value upon, could play a crucial role in the wellbeing of society.”

A further advertisement for the conference at which Dieppe spoke stated that there would be live demonstrations of healing on stage and describes his talk as:

The incredible power of remote healing across distance:  Paul Dieppe, emeritus professor of health and wellbeing, renowned for being at the global forefront of research into Healing.

faith healer in NHSA press release for “the largest meta-analysis of ever undertaken into the effects of non-contact healing”  states:

In fact, there are many healing practices where healer(s) and the one needing healing never come in contact with one another.  Indeed, they may never even meet, let alone know each other.

Paul Dieppe is identified as a Trustee of the group that produced a meta-analysis and he comments on the findings:

“This is a rigorous, high quality scientific report, and it clearly shows that healing intention can have beneficial effects on living systems, both human and non-human. Now we need to explore questions such as who does it work for and in what circumstances, as well as the how and why questions.”

A 2015 interview shows him rejecting scientific methodology as old fashioned materialism. Instead as he favors  anecdotes  and embraces energy/spiritual healing.

Have you come across many instances of actual healing?

Masses of cases. It’s anecdotal stuff, but it’s extremely impressive and extremely convincing. I’ve experienced it myself. A lot of this stuff happens completely under the radar, people just getting on with it in their back room, not part of any particular body or group, just doing their own thing. I went to see one such person, a little old lady in her front-room, and she said the best way to understand healing is to experience it yourself, and asked if I had anything I wanted working on. I told her I had a bad knee, so she said, OK let’s give it a go. And for what it’s worth, it’s been better since then. That was a couple of years ago.

Do you have a sense of what happens in the brain, does it involves specific neural or nervous networks? What systems does it involve?

Life gets difficult when you talk like that. You’ve slipped into the assumption that our materialist knowledge is the appropriate framework for that. We assume our current materialist science can explain everything, so the default position is ‘how can I explain this physiologically within my own materialist framework?’ I’m not convinced that’s the right way to do it.

No, I don’t have a materialist worldview, but if there is a spiritual dimension (which I think there is) I’m curious as to how it interacts with the body. William James, for example, explored how spiritual experiences interacted with ‘the subliminal self’. I wonder if healing is connected to things like trance states, altered states of consciousness, and so on.

OK, well how might it happen in those terms? Certainly there’s a lot of evidence for our ability to alter things through the Autonomic Nervous System via hypnosis. That can give us clues to a lot of this stuff. Hypnosis certainly affects the ANS, it certainly affects the immune system…probably everything. The mind / body split is of course silly. Everything is connected, everything works together. Although it’s easiest to talk in terms of what we can observe physiologically…I think we can control pretty much all of it probably.

OK, so at the moment we’re at or near the peak of a biomedical conception of health and illness. You, by contrast, espouse what you describe as ‘a bio-psycho-socio-spiritual theory of the transcendence of suffering’. How does that go down in academic medical circles?

Most of my medical colleagues think I’ve gone completely barmy, and this is all absolute nonsense. The dominant model in which we work and teach rejects anything to do with spirituality. If you mention words like spirituality or love, you’re rejected. It’s not acceptable behaviour, you’re regarded as someone who should be quietly taken off to the funny farm. So it’s quite lonely. It doesn’t bother me, I’m at the end of my career, I don’t need a new job or a reference, I don’t need to toe the line. It’s irrelevant to me. But I can feel the group discomfort sometimes around this area. I think the medical profession is a bit out of kilter with the rest of society, which is probably more accepting of the link between spirituality and health.

The interview links to a video featuring Dieppe on a trip to Lourdes, a destination for millions seeking miraculous cures.

The Role of the Data Monitoring and Ethics Committee in a Clinical Trial

As chair of the PACE DMECT, Paul Dieppe was in a unique and powerful position, able to monitor incoming data that was unblinded with respect to the group to which participants were assigned.

A recent Medical Research Council document states


The role of the Data Monitoring Committee (DMC) is to monitor the data emerging from the trial, in particular as they relate to the safety of participants, and to advise the Trial Steering Committee on whether there are any reasons for the trial not to continue. It is the only body involved in the trial that has access to the unblinded (unmasked) comparative data during the trial.

At the time of the fateful shifting of scoring of subjective outcomes and downgrading of the status of objective outcomes, Dieppe more than anyone else was in a position to know how switching would effect results of the analyses of the data already collected. He supposedly reviewed analyses in closed meetings, excluding Principal Investigator Peter White, but he was also meeting White formally and informally.

distasnce healingDieppe knew exactly what was going on in the most recent data, in terms of emerging  patterns of group differences. He would also have attended the meetings in which changes to the protocol were discussed and approved. Did he remain quiet? Refuse to meet with White in private? Maybe Dieppe relied on the unspoken interconnected consciousness of the two men.

We have a mystery akin to that about what when on before the US elections between Trump and his advisers, on one hand,  and the Russians.

The significance of a science-skeptic heading the PACE DMET

 Even before Dieppe’s colleagues would have described him, as he puts it, going completely barmy, he was a believer in the extreme, almost mystical power of the placebo. Perhaps that made him an ideal choice to head the DMET for a trial that created such an imbalance in nonspecific effects between the active treatments and the comparison group. And then boosted the imbalance as part of treatment and even in a newsletter (!).

But seriously, what we make of someone having such disdain for scientific methodology having the responsibility for ensuring its rigor in a major clinical trial?





What?! Homeopathy and acupuncture for symptoms of autism



Let’s get the American Association for the Advancement of Science (AAAS) to stop publicizing junk studies of acupuncture for symptoms of autism intended to exploit parents NOW.



Changes are ordered by Advertising Standards Authority for Ireland (ASAI) in the Facebook Page advertising acupuncture for autism.

  • Yet, an enterprising journal editor got the AAAS’s Eurkalert to distribute worldwide a press release for a junk study of acupuncture for autism that his journal published.

  • Some lessons and practice in skills in detecting junk studies.

  • Let’s recognize The Journal of Alternative and Complementary Medicine (JACM) as a Quack! purveyor of junk promotions of discredited and unproved treatments disguised as science.

  • Let’s watch for the next move of the Editor of JACM’s next move in his avowed effort to gain legitimacy and reimbursement for acupuncture.

  • Let’s tell AAAS to stop making excuses.

  • Tell AAAS to get EurekAlert to screen press releases before distributing them. As it stands, EurekAlert is damn confusing to lay persons who consider the AAAS bran attached to anything as indicated a trusted source. [At the end of this blog, I make it easy for you to lodge a protest.]

Ireland’s Advertising Standards Authority investigated a complaint. As a result, the ASA has informed a homeopath not run a Facebook advertisement in its current form.

The homeopath was told by ASAI not to run a Facebook advertisement in its current form again because the page suggests she could treat the symptoms of autism. The ad by Almond Homeopathy stated the homeopath had:

“the great pleasure of meeting and working with many children diagnosed with ASD and the greater pleasure and joy of helping them overcome many of their symptoms that were holding them and their families back from getting the most out of life.”

The Facebook Page also referenced a blog which contained a reference to a child whose “learning, relationships and speech” had improved

The advertiser defended her FB page by declaring  that  she was a qualified homeopath, registered with the Irish Society of Homeopaths.

She said she treated the person, not their condition or disease, and did not claim to be able to cure anything or anyone.  However,

The ASAI’s complaints committee said that, while noting the advertisers’ comments that she was a qualified and licensed homeopath and registered with the Irish Society of Homeopaths, it was aware from a previous case that the Department of Health had confirmed that there was no statutory regulation of complementary therapists.

The committee also said the wording of the post and the heading of the blog post would give readers the impression she could treat the symptoms of autism.

About the same time, EurekAlert’s posted on Twitter a link

New study from KKI shows feasibility of acupuncture in young children with ASD

To the naïve reader, the item appeared to be a story, not  the pasted in press release that it was, but that was not the case. Where one would have expected the name of a journalist, the publisher’s name was listed.


New Rochelle, NY, June 19, 2017-A pilot feasibility study to determine if young children with autism spectrum disorder (ASD) and their parents would tolerate and adhere to an office- and home-based acupuncture/acupressure intervention showed completion of all 16 biweekly sessions and measurements of their effects before, during, and after the protocol. The study design and results, which suggest further controlled studies of this intervention approach in ASD, are published in the Journal of Alternative and Complementary Medicine (JACM), a peer-reviewed publication from Mary Ann Liebert, Inc., publishers. The article is available free on the JACM website until July 19, 2017.

Coauthors Lana Warren, EdD, OT/L and Patricia Rao, PhD, Kennedy Krieger Institute, Baltimore, MD, and David Paton, DAc, LAc, Starting Point Acupuncture and Health Services, Catonsville, MD, identified the most positive outcome of the study as the high compliance rate, with all parents of the children ages 3-10 completing the intervention. The researchers measured the effects of the intervention on factors such as the children’s behavior, ability to pay attention, sleep, and aspects of parenting stress. In the article entitled “A Pilot Observational Study of an Acupressure/Acupuncture Intervention in Children with Autism Spectrum Disorder,” most parents reported that the intervention had a positive impact on their relationship with their child.

The editor of JACM chimed in:

“While a small study, the tolerance and adherence with acupressure this pilot are both hopeful signs for families of those in their care with autism spectrum disorder,” states JACM Editor-in-Chief John Weeks,, Seattle, WA.

Before getting to the horrible little study itself, note how the AAAS  denies responsibility for what it posts:

Disclaimer: AAAS and EurekAlert! are not responsible for the accuracy of news releases posted to EurekAlert! by contributing institutions or for the use of any information through the EurekAlert system.

Note also that AAAS has said it employs 20 staff to monitor its presence in social media and tweak publicity.

The “pilot study” was:

Warren Lana R., Rao Patricia A., and Paton David C. A Pilot Observational Study of an Acupressure/Acupuncture Intervention in Children with Autism Spectrum Disorder. The Journal of Alternative and Complementary Medicine. May 2017, ahead of print.\

The Abstract

Objectives: To determine whether children with autism spectrum disorder (ASD) would tolerate an acupressure/acupuncture intervention and whether parents would adhere to a twice-weekly, 8-week intervention protocol. Second, to further understand best measures to use to capture impact of intervention on behavioral and regulatory functions.

Design: This is an observational pilot study with pre-, mid-, and postintervention measures.

Settings/location: The intervention was carried out in a private practice office in a large metropolitan area.

Subjects: A total of 10 children of ages 3–10 years with ASD and one of their parents participated.

Interventions: A total of 16 biweekly treatment sessions of acupressure and/or acupuncture were carried out by a licensed acupuncturist, and a daily home-based acupressure intervention was carried out by a parent.

Outcome measures: Attendance, tolerance of intervention, parent compliance with home program, and parent compliance in completing daily diary and five standardized measures of behavioral and regulatory functions pre-, mid-, and postintervention were recorded.

Results: The 10 children in this observational study, collectively, tolerated the intervention and parents adhered to the 16 sessions, biweekly protocol, and home protocol, as well as completing daily diary and five standardized measures at three different time intervals. The five measurements appeared to be sensitive to behavioral and regulatory functions that may improve with this type of intervention.

Conclusions: The results of this observational pilot study suggest that acupressure/acupuncture is a feasible intervention for children with ASD that merits rigorous evaluation through a randomized controlled trial.

Probing the article

Keep in mind:

There is no evidence that acupuncture is anything but an elaborate placebo ritual. The ritual involves efforts to communicate hope and acupuncturists are trained to express a sense of progress when patients wince or express discomfort.

There is no specific scientific mechanism by which acupuncture could be expected to affect physical and mental health outcomes.

This is not a randomized trial. Any changes observed  could be chance variation or return to the usual behavior of the patient from the whatever increase in problems motivated seeking health.

Outcomes are not objective, but depend on the subjective ratings from parents who have invested time in repeated traveling to the clinic where treatment is provided. The parents are not blinded, they know their child is receiving a treatment that the clinic communicates is effective. There is no comparison-control group. It is a very different situation than getting the same ratings from teachers who know the children, but do not know if they have received treatment.

A sample of 10 is inadequate (underpowered) for any claims of efficacy or effectiveness. In a randomized trial, a sample of 30 patients per group is needed to detect half the time when the effectiveness of a  treatment that actually has moderate-sized effects. Half the time the effects of the treatment would be missedWhen statistically significant results are found, they are likely to be flukes. Any results that reached significance would have to be a large difference. The literature on psychotherapeutic interventions for mental health problems is littered with unpowered studies claiming significant effects that do not replicated in larger studies.

Why is that? There is a strong confirmation bias in what gets published. Expectable null findings do not get published. The positive effects from small studies that do get published are likely to chance, selective reporting, or the researchers exercising some selection and control of what is reported that they should have.

The ability to detect moderate sized effects drops off quickly as sample size decreases.

A sample of 10 patients would need huge effects. Most statistics do not perform well with such a small sample.  Sampling error or just plain noise affect results: Loss of one patient or adding of another could dramatically affect results. But statistical analyses depend on assumptions being met about the distribution of data. These assumptions are not likely to be met with 10 patients and could not even be tested.

The bottom line: Competent, well-meaning psychotherapy researchers do not even claim any evidence a treatment is effective from a sample  of 10. Authors who make such claims are fools or are trying to fool their readers , or both.

An added bonus guide with helpful tips from a source I trust

Small studies: Be vigilant when writing about them and skeptical when reading about them –

Nonetheless, the authors make some dubious claims

Unexpectedly, the most robust finding of the current pilot observational study was significant improvement in children’s expressive and receptive social communication after the intervention. These findings are consistent with previous studies reporting significantly improved social communication after acupuncture, including language comprehension and social interaction.14,18 However, they were unexpected because the acupoints selected for intervention in the current study were chosen to target areas of the body corresponding to behavioral and regulatory problems, not areas known to be involved with social communication.

Beware of unexpected robust findings from such little studies. The authors admit that they were not  stimulating the bogus acupuncture points that are supposed to produce such change. They cite crap studies that nonetheless suggest such results should be found.

Although not statistically significant, the current findings indicated some improvement in sleep delay after the intervention, which may be worth further investigation.

Sure, the authors would be better off consulting tea leaves or the children’s horoscopes. They are sorting through lots of measures searching for other significant ones to crow about. When they come up  empty handed, they declare success and call for more research.

Of note, sensory processing concerns associated with ASD, particularly tactile, which had been anticipated to be problematic in AIs, did not manifest in the child participants in this study, with the exception of their general dislike of maintained deep pressure in one location. It should be noted that the initial acupressure strokes administered during the intervention were those that promote calming and quieting per Chinese Medicine theory. response to treatment.

Let’s go back to what the editor of this journal said in the press release:

“While a small study, the tolerance and adherence with acupressure this pilot are both hopeful signs for families of those in their care with autism spectrum disorder,” states JACM Editor-in-Chief John Weeks,, Seattle, WA

Quack, Quack,


Any journal that would publish such irresponsible stuff deserves a warning to ward off naïve and vulnerable readers. Nija Ezard Ernst says

I only see one solution: completely disregard certain journals that have been identified to regularly publish nonsense. Sadly, the wider medical community is far from having arrived at this point. As far as I can see, the problem has not even been identified yet as a serious issue that needs addressing. For the foreseeable future, we will probably have to live with this type of pollution of our medical literature.

The JACM Editor-in-Chief John Weeks is well known to the skeptical community who watch for quacks. Like the EverReady Bunny, he jumps at opportunities to promote the reimbursement of dubious, unproven treatments. For instance:

The Academic Consortium for Complementary and Alternative Health Care (ACCAHC) is a group of “complementary and alternative medicine” educational organizations representing chiropractors, naturopaths, acupuncturists, homeopaths, ayurvedic practitioners, direct entry midwives and massage therapists. The executive director is John Weeks, a relentless promoter of “integrative medicine,” both on the Integrator Blog website and in the Huffington Post. The ACCAHC is dedicated to ensuring that its members and the practitioners they represent are included in all aspects of health care, such insurance reimbursement, financial resources for education and delivery models. ACCAHC’s stock-in-trade is its practitioners’ supposed expertise is being patient-centered, holistic, taking into account the whole person and such, as well as an alleged emphasis on healthy lifestyles, nutrition, well-being, and the like.

AAAS puts this stuff out there because it makes money. You can tell them to stop. Contact them at their website, go to their Facebook page. Share and Tweet the link to this blog post, tweet to EurekAlert.

EurekAlert is promoting quack treatments to vulnerable parents of children with autism and it must stop. Readers may not notice that distributing through the EurekAlert doies not constitute endorsement by the widely respected Association for the Advancement of Science, despite the disclaimer at the bottom of the page.

Contact EurekAlert!


Phone: 1-202-326-6716

Fax: 1-202-898-0391



1200 New York Avenue, NW

Washington, DC 20005


Twitter: @EurekAlert



Should authors declare a conflict of interest because they suffer from the illness they are writing about?

Some researchers issued a novel demand for correction of an undeclared conflict of interest stemming from the author of a criticism of their work suffering from the illness targeted by their intervention.

Should the concept of conflict of interest be expanded?

Recently, a hostile reviewer demanded that authors of a manuscript submitted to The BMJ provide proof that they had a confirmed diagnosis of a illness from which they claimed they had suffered for decades. Should patient-authors get notes from their physicians to accompany their conflict of interest statements?

The critique that upset the PACE investigators is available here:

Keith J Geraghty. ‘PACE-Gate’: When clinical trial evidence meets open data access Journal of Health Psychology DOI:

conflict-of-interestThe email conveying the demand is reproduced below. Basically, the investigators from the PACE trial of cognitive behavior therapy and graded exercise therapy for chronic fatigue syndrome demanded:


  • Partial retraction of an article critical of their work.
  • Issuing of a conflict of interest because the author of the critique suffered from the illness targeted by the trial.
  • The corrected article be posted with a full response from the PACE investigators,  and not appear until readers could compare the two.

Note that the author did not mention in the article that he suffered from any illness. Should he have?

How far should we extend this requirement? Should principal investigators be required to declare on their grant applications whether they suffer from any relevant chronic illness? Should NIMH require a formal psychiatric evaluation of applicants for depression grants?

Should authors of HIV/AIDS articles declare their viral status?

How about blanket declarations: “The authors have all had recent physical examinations and declare they have no relevant health conditions?

How about reviewers? Reviewer conflict of interest can be important.

Finally, was it an invasion of the author’s privacy for the PACE investigators to seek out evidence of any illness and write to the journal editor about it?

Dear Dr Marks,

We were surprised and alarmed to read the on-line editorial by Dr Geraghty, published on Monday in the Journal of Health Psychology.

While we would support robust criticisms of science and believe people are entitled to their opinions, we were more than surprised by the personal criticisms made in the piece, which were often unsubstantiated. We do not believe that fellow scientists should indulge in ad hominem attacks and innuendos. For instance, Geraghty wrote ” However, there are accepted scientific procedures and standards that appear to have been neglected, or bypassed, by the PACE Trial team. Their actions have arguably caused distress to patients, added a million pounds of additional costs to a publically funded trial and have left us with two versions of ‘truth’ concerning the trial’s findings – the published analysis versus the recent re-analysis.” Where is the evidence for these statements?

Therefore we ask you:

  1. To revise the piece in order to remove all the personal attacks and innuendos.
  2. To include in a revision the author’s potential conflict of interest as a sufferer of the illness he writes about. See:
  3. To enable us then to respond with equal prominence to the remaining criticisms as a whole, in the same online first and print versions so that readers can see both articles side by side and then make up their own minds. At present this is not possible because of the selective, one sided nature of the editorial as it stands.

We look forward to your early reply.

Yours sincerely,

Professors White, Chalder and Sharpe

Co-principal investigators of the PACE trial

A skeptical look at a study of acupuncture delivered in emergency rooms [updated]

Why an expensive, flawed, misrepresented trial of acupuncture in emergency rooms provided no evidence that acupuncture is effective in controlling pain.

acupuncture.needlesUpdate 6/24/2017

I  thank the pseudonymous commentator who called attention to one of the authors of this study previously providing a press release making claims for the study’s results before the data were analysed   The author stated:

“While data from the study is still being analysed and finalised for publication in a medical journal, one of the researchers, Dr Michael Ben-Meir, said it showed acupuncture offered the same level of pain relief as analgesic drugs when patients rated their pain one hour after treatment.”

We can now evaluate this misrepresentation.

A number people in social media urged me to take a quick look at a recent large, expensive trial of providing acupuncture in the emergency room. The study has begun getting some attention in the media. We will see the study is flawed from its inception, aimed at fulfilling the principal investigator’s agenda being of misrepresented as supportive of getting acupuncture into ERs. Coverage in the media is inaccurate, but consistent with statements made by the primary investigator.

The paper reporting the clinical trial is unfortunately in a pay-walled journal, but it is at least temporarily it is available here.  An abstract can be found here

Marc M Cohen, De Villiers Smit, Nick Andrianopoulos, Michael Ben-Meir, David McD Taylor, Shefton J Parker, Chalie C Xue and Peter A Cameron Acupuncture for analgesia in the emergency department: a multicentre, randomised, equivalence and non-inferiority trial. Medical Journal of Australia. 2017; 206 (11): 494-499. || doi: 10.5694/mja16.00771

The trial protocol is freely available and can be found here.

  • My summary assessment is that it is indeed an ambitious expensive clinical trial producing no evidence that acupuncture is effective.
  • Its rationale is based on misrepresentations of the literature.
  • The study protocol shows features are built in its design to be biased in favor of finding acupuncture is not inferior to routine care in the ER.
  • The article downplays important findings that undercut validity of trial and investigator’s intended conclusions.
  • The study was conducted by an investigator strongly committed to the promotion of acupuncture in routine medical care and securing reimbursement for it.
  • The conclusion is slanted toward cultivating the illusion that acupuncture is an evidence-based treatment. The intent is to obtain funding for an even more ambitious trial, ultimately getting acupuncture secure reimbursement and preferred treatment status.
  • The study capitalizes on the current opioid addiction epidemic and public health concerns about overreliance on pain medications.

Rationale for the randomized trial

The article and protocol seriously misrepresent the conclusions of the existing literature. Claims that are made are inconsistent with was is found in the specific citations that are provided.

The article states

Acupuncture can provide analgesia for chronic musculoskeletal pain and headache,6-9 and its use is covered by therapeutic guidelines for acute pain.10,11


Our study aimed to determine whether acupuncture is effective, safe, acceptable and feasible for patients presenting to the ED with low back pain, migraine or acute ankle injuries.

The protocol states

It has been shown that acupuncture analgesia in the treatment of chronic pain is comparable to morphine and that its better safety profile and lack of dependence makes it the preferred method of choice for these conditions [9].

Reference 9 in both sources is to

Ernst E, Lee MS, Choi TY. Acupuncture: does it alleviate pain and are there serious risks? A review of reviews. PAIN®. 2011 Apr 30;152(4):755-64.

This article is highly critical of the use of acupuncture and certainly does not say its effectiveness is comparable to morphine and should be the preferred method of treating chronic pain. Instead, it says:

In conclusion, numerous systematic reviews have generated little truly convincing evidence that acupuncture is effective in reducing pain. Serious adverse effects continue to be reported.

The protocol proclaims the pressing need for a study of the scope of the present one:

As of June 2011, virtually all of the 31 reviews and 32 review protocols listed in the Cochrane library suggest that the evidence for acupuncture is inconclusive. Consequently, there has been a call for the funding of well-planned, large-scale studies to assess the effectiveness and cost-effectiveness of acupuncture under real-life conditions [14]. It has been suggested that the best methodological approach to clinical acupuncture research is through pragmatic trials where acupuncture is compared to standard care rather than placebo-controlled trials. Such trials may be more able to provide data that is relevant to patients, practitioners and policymakers, and to inform decision-making about treatment options [15].

The issues about acupuncture are (a) the lack of a scientific rationale and (b) difficulty demonstrating it is superior to a sham acupuncture treatment. The difficulties in (b) is that it is difficult to blind a “real” versus sham treatment and the tendency of acupuncturists to communicate to patients whether they are receiving only a sham treatment.

There is a lack of evidence that acupuncture is anything but a highly ritualized placebo treatment. The investigator takes this as a cue for abandoning placebo-controlled studies and just comparing it with routine care- which is often lacking in the support and attention provided by a nonspecific placebo condition.

The investigator is intent on demonstrating that acupuncture is no worse- noninferior -to routine care.

Design features that  bias in favor of finding acupuncture is not inferior to routine care in the ER.

The emergency room (ER) is a bad place to get medical care. Don’t get your medical care unless required by circumstances.

The ER is also a bad place to evaluate acupuncture. There are reasons not to provide acupuncture in the ER even if one believes in the efficacy of acupuncture. ER care is urgent care intended to address medical issues needing immediate attention. For many medical conditions, emergency room care is only the first step in admitting a patient to inpatient services or to refer a patient for specialty outpatient services.

The conditions that are lumped together are too diverse, especially if the interest is the evaluation of the treatment of pain.

An ankle sprain is an acute injury. The article notes

All ankle injury patients received rest, ice, compression and elevation regardless of their treatment allocation.

Patients presenting to the ER with migraines are typically experiencing the exacerbation of a pre-existing chronic condition. Other symptoms besides pain may prompt a visit. The article notes that “migraine patients received intravenous fluids at the discretion of the treating physician.” The pharmacotherapy protocol included treatment for nausea, but no pain medication specific to migraines.

Patients presenting with acute low back pain may be injured and facing the onset of a chronic condition or facing an exacerbation of a chronic condition. Addressing the immediate circumstances of the ER visit is a goal, not analgesia. Long term goals are to preserve functioning and quality of life, not analgesia.

These three groups are different in their conditions and procedures, goals,  and expected outcomes for an ER visit. Even a trial as large as this one would be too small -underpowered- to explore crucial differences between groups. Or to see if overall results do not generalize to any patient group.

For all three patient groups, some will have self-limiting pain, which will contribute to a lack of differences among treatments.

There is no evidence for the efficacy of acupuncture versus a nonspecific treatment for any of these treatments. I recently discussed claims for acupuncture for migraines. I showed a lack of evidence for its efficacy. It is important to note, however, that in that study, acupuncture groups received treatment 5 days per week for 4 weeks for a total of 20 sessions. Even believers in acupuncture would not expect one session delivered in the ER would prove sufficient. But benign and even inert effects of one session might encourage patients to get more treatments later.

The primary outcomes of the study were assessed only at 24 and 48 hours. In those short periods, differences between effective and noneffective treatments may not have emerged. Differences in nonspecific treatments –attention, support, and communication of positive expectations may be maximized, but short-lived. And the subjective distress of some patients may have gotten dropped from the heights of distress which brought them to the ER.

Downplaying findings that undercut validity of trial and investigator’s intended conclusions

 Regardless of group assignment, all patients were entitled to “rescue medication” after 1 hour or if their attending physician deemed it appropriate. Patients assigned to acupuncture alone were three times more likely to receive rescue medication. This is a serious confound undermining the integrity of the trial and the meaningfulness of any results.

The acupuncture only group received significantly more rescue medication therapy than the groups that received pharmacotherapy (at T1: p = 0.016; after T1: P = 0.008).

Only a minority of patients in any condition achieved adequate pain relief.

Overall, 16% of patients had clinically relevant pain relief and 36.9% statistically relevant pain relief at T1, with no statistically significant differences between the three groups (Box 6).

Conducted by an investigator strongly committed to the promotion of acupuncture in routine medical care and securing reimbursement for it

 In Australia, the principal investigator is a sought after spokesperson advocating acupuncture and other unproven medical treatments.

Statements made in interviews with the media   show the motivation of the PI in conducting the trial and his spinning of his results:

“Emergency nurses and doctors need a variety of pain-relieving options when treating patients, given the concerns around opioids such as morphine, which carry the risk of addiction when used long-term.


“Our study has shown acupuncture is a viable alternative, and would be especially beneficial for patients who are unable to take standard pain-relieving drugs because of other medical conditions.


“But it’s clear we need more research overall to develop better medical approaches to pain management, as the study also showed patients initially remained in some pain, no matter what treatment they received.”

A sampling of the inaccurate coverage that this study received:

press coverage


I enjoy probing suspicious studies like this one. I hope that readers benefit from what I find, not only in terms of my specific conclusions, but in acquiring the knowledge and skills I deploy. I draw on over 40 years experience as a researcher and a teacher, with over 400 peer-reviewed publications, as well as lots of editorial and peer review experience. In the case of this particular study of acupuncture, likely problems could be anticipated, as well as where to look for them.

Although I’ve becomes relatively efficient at this sort of activity, it remains time and labor intensive, particularly when I trace the study back into the claims being made about it in the media, as well as to any study protocol. And then look for discrepancies with what else is out there. And them format all of this for a blog post.

There is no direct benefit, professionally or financially to me for this activity. It’s an enjoyable, almost addictive distraction. I really should be working on the web-based science writing courses and e-books that will ultimately provide support for my activities, now that I am emeritus. I wonder, though, if there might be a way to monetize this. I would be uncomfortable simply doing it fee-for-service, because of the potential conflict of interest that it would create. I don’t come to a negative conclusion for everything I evaluate, but I wouldn’t want to be seen has endorsing weak or preliminary findings.

I’m considering creating a broad-based premium service. For a modest fee, subscribers would get weekly assessments of what I find in the scientific literature and how it is covered in social media. Subscribers could not only obtain this regular premium content, but also have the opportunity to nominate studies for review, with a subset selected and reported to all subscribers.

I know that my fans and followers have become accustomed to my providing a blog post and a social media presence without charge. I intensified this activity when I had a Carnegie Centenary Visiting Professorship at Stirling University, Scotland. One of my responsibilities was social engagement, and my involvement in blogging and social provided an effective and satisfying way of doing that. But now the professorship is over, and I have to think of alternative ways of monetizing my activities. I welcome readers’ thoughts and encourage them to sign up at to keep up on all of my activities.


Evaluating personality and social psychology research as clinical trials

consort abstractFor a while I have been advocating that journals publishing mainly personality and social psychology research become more informed about clinical trials.

Editors and reviewers could better keep bad science out of an already untrustworthy literature if there were clearer and more widely disseminated standards for conducting and reporting research.

In a post at PLOS Mind the Brain,  I described how my colleagues and I had to obtain and reanalyze the data to show Barbara Fredrickson’s study was a null trial with no benefits to participants who were told to meditate.  With clearer reporting in the published paper, readers could have readily seen the study was flawed and did not get the wondrous results that the authors claims. We would not have had to waste time reanalyzing the data.

More recently, I provided a review of the Amy Cuddy power posing paper  published in the same journal.  The outlandish claim in the abstracts had no basis in what was done in the study and what was found:

That a person can, by assuming two simple 1-min poses, embody power and instantly become more powerful has real-world, actionable implications.

Despite launching a seven-figure package of merchandise such as corporate talks and workshops, this sentence capped abstract that is a textbook example of inadequate reporting.

Once I delved into the paper itself, the reporting of methods and results continued to be poor. Yet,  I was able to identify a grossly underpowered, low-quality study with improperly analyzed and misinterpreted data. The paper should have been outright rejected. Instead, replicationados who suspected a bad study had to go to the trouble of organizing an attempt to replicate an effect size that had no justification for being in the literature. No surprise that they got null findings except for a weak finding for a subjective self-report measure that is highly susceptible to experimenter expectancy and demand characteristics. What another waste of time.

My blog post stimulated lively discussion in social media. Among the interesting responses I got was:

I get your point, but the purpose of the clinical trial evaluation criteria are to protect patients from engaging in interventions for health conditions that may be ineffective or even harmful. As far as I understand it she wasn’t claiming the “power pose” should be used clinically to treat any mental health conditions? Yes, the study was flawed but I don’t know that it is reasonable to hold all laboratory manipulations like this to clinical trial standards, particularly for social psychology studies which are very difficult to fund. Most of these types of studies are done with very limited funding and resources. Once her idea took off and became popular, she had a responsibility to followup on the effect and establish it as reliable, which she did not. But I think we don’t want to discourage early career researchers and students from doing pilot work with the limited resources they have to look at effects like this without fear of being attacked repeatedly over a period of years. There are so many flawed studies out there, why keep focusing on this one individual and this one study?

I replied:

You raise excellent questions. Clinical trials are simply experiments. Because the results matter for people, standards have gotten clearer and tighter for describing what is done and for the kinds of inferences that can be made.

Applied to Amy Cuddy’s power post study, the standards readily revealed that strong inferences were made from a very weak experimental design. Apparently reviewers missed this.

The need for pilot and feasibility studies is appreciated in a clinical trial literature. But the expectation is that researchers do not infer effect sizes from studies too small to reliably generate them.

Instead of doing another study, Amy Cuddy hired a speaking agent and commanded fees of up to $100,000 per appearance. She claimed her research results were very strong and that she and shown that two minutes of behavioral manipulations can influence mind-body relations.

In this sense, she is a very bad model for early career investigators, rushing to market and profit from her work, rather than doing the hard work of making it more rigorous.

Amy Cuddy’s adviser is quite powerful and together they viciously attacked critics who had pointed to the strong interpretations being made from weak findings.

My focus is a bit different than some of Cuddy’s critics. She is selling lay audiences, mostly women, bogus ideas about how easy it is to change how they behave and what happens to them. I think it is bad for psychology that people be given the illusions that is so easy to make change, as well as for a psychologists to be associated with profiting from pseudoscientific claims.


Educate yourself, be a better role model,  and teach your students and other co-authors well.

Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Medicine. 2010 Mar 24;8(1):18.

Hopewell S, Clarke M, Moher D, Wager E, Middleton P, Altman DG, Schulz KF, CONSORT Group. CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration. PLOS Medicine. 2008 Jan 22;5(1):e20.

Thabane L, Hopewell S, Lancaster GA, Bond CM, Coleman CL, Campbell MJ, Eldridge SM. Methods and processes for development of a CONSORT extension for reporting pilot randomized controlled trials. Pilot and Feasibility Studies. 2016 May 20;2(1):25.