Ineffective medicine could seem effective enough for licensing

New statistical research challenges the strength of evidence required by the US Food and Drug Administration for its endorsement of new medication. University of Groningen psychologist and statistician Don van Ravenzwaaij concludes this in a paper co-authored by prof. John Ioannidis from Stanford University. The paper has been published in PLOS ONE.

Van Ravenzwaaij came up with the idea for the study when he read the manual published by the FDA in which its endorsement procedure is explained. Legislation by Congress on the matter is understood by the FDA to prescribe two trials that have convincingly demonstrated the effectiveness of a product as a precondition for endorsement.

“Which sounds reasonable enough”, explains Van Ravenzwaaij. “But what it does not indicate is two out of how many trials we are talking about here. Two positive trials out of two attempts is not the same as two out of five of which three trials were negative.” Van Ravenzwaaij and Ioannidis now argue that if one adopts strictly a policy of coming up with two trials with positive results this policy may allow ineffective medication to slip through.

Simulations of the model

Van Ravenzwaaij and Ioannidis subjected the endorsement model to a number of simulations, altering factors such as the number of trials or the population size in trial studies. Their finding: in a non-trivial number of cases a medical product can produce two positive trials while the totality of evidence points to the absence of a medical effect. In English: it is possible that products that do not work better than a placebo are endorsed.

Their point is not about concrete ineffective products that have been endorsed. Van Ravenzwaaij: “FDA and other regulatory agencies may consider many other aspects of the evidence before making a licensing decision. We felt that the more pressing point was to demonstrate that the policy of “2 positive trials” itself is vulnerable in its requirement of evidence for medical effectiveness. The upshot of our results is that strict application of this policy for licensing purposes can potentially lead to ineffective medication on the market.”

Don van Ravenzwaaij works at the Psychometric and Statistics Department at the University of Groningen, Faculty of Behavioural and Social Sciences. John Ioannidis is professor in Medicine and in Health Research and Policy at the Stanford University School of Medicine.

More information:

The link to the publication: ‘A Simulation Study of the Strength of Evidence in the Recommendation of Medications Based on Two Trials with Statistically Significant Results’