Modern medicine and physical therapy should be predicated on the delivery of high-quality care based off of data from robust clinical trials in the context of clinical expertise and patient values. However, clinicians and patients are often deceived by the selective publication and reporting of trial data, known as publication bias, that distorts the evidence base. The Cochrane Handbook defines publication bias as “the publication or non-publication of research findings, depending on the nature and direction of the results” and is “one of the worst threats to the validity of scientific research.” This phenomenon was first described by Thomas Sterling in 1959 and nearly 60 years later, the problem of publication bias is still embarrassingly common across biomedical and social sciences.
The problem of publication bias has been unambiguously demonstrated by several systematic reviews and meta-analyses in medicine — Lee et al found that “over half of all supporting trials for FDA-approved drugs remained unpublished ≥ 5 y after approval.” Scherer et al tracked publication rates of data from abstracts presented at scientific meetings and conferences, finding that “Only 63% of results from abstracts describing randomized or controlled clinical trials are published in full. 'Positive' results were more frequently published than not 'positive' results.”
This notion of positive results being more frequently published was furthered by Song et al, reporting that “the odds ratio of an efficacy outcome being fully reported if it were statistically significant versus non-significant was 2.4” and Kicinski et al who when analyzing publication bias in Cochrane meta-analyses found “In the meta-analyses of efficacy, outcomes favoring treatment had on average a 27% (95% Credible Interval (CI): 18% to 36%) higher probability to be included than other outcomes. In the meta-analyses of safety, results showing no evidence of adverse effects were on average 78% (95% CI: 51% to 113%) more likely to be included than results demonstrating that adverse effects existed.“
These findings illustrate a systematic undermining of the evidence base that is used to make real-world decisions on real people with real risks, benefits, and costs. Ben Goldacre writes in his book Bad Pharma “every time we fail to publish a piece of research we expose real, living people to unnecessary, avoidable suffering” and that is unacceptable.
Given the pervasiveness of publication bias throughout other areas of medical research, it is unlikely that the physical therapy literature is an outlier in this regard. The limited assessments of publication bias conducted under a physical therapy lens suggest a meaningful issue worthy of more investigation.
Why Missing Data Matters
Missing data matters because, without it, clinicians and patients are left unnecessarily vulnerable due to a distorted picture of an intervention’s benefits, risks, and costs. Ideally, physical therapists should be making decisions based upon the best available evidence — when trial data is left unpublished, this means physical therapists are left to make decisions on an incomplete and biased literature base. This can result in ineffective, over-utilized treatments, an avoidable financial strain on an overburdened health care system, and at worst cause harm to the patients who seek the care of a physical therapist. When trials go unpublished, it is misleading, deceptive, and disrespectful to those individuals who exposed themselves to the risks and burdens of participating in a clinical trial.
There are unfortunate examples of this throughout medicine that demonstrate the very real and very human cost of missing data. One of the more catastrophic examples of the deleterious effects of publication bias is that of lorcainide and other Class 1c anti-arrhythmic drugs. Ben Goldacre describes how it was common practice in the 1980s to prescribe class 1c anti-arrhythmic drugs to individuals after myocardial infarction. This practice made intuitive sense at the time — after myocardial infarction, many people suffer from cardiac arrhythmias that can lead to sudden cardiac death and medications like lorcainide are designed to suppress such events. Unfortunately, this bit of mechanistic reasoning proved to be too simplistic and prescribing these particular medications actually increased the risk of death. This result was shown in an early 1980 trial on lorcainide that unfortunately went unpublished. It was only when the CAST-1 trial found that this prescribing practice unambiguously increased the risk of early mortality that wide changes occurred — 21 years after the unpublished lorcainide trial was completed. It is estimated that this error contributed to well over 100,000 deaths, much of which could have likely been avoided if the initial trial on lorcainide that found “nine out of forty-eight men on lorcainide died, compared with one out of forty-seven on placebo” had been published and readily disseminated.
In physical therapy, missing data may lead to the continued use of treatments that are unnecessary, unhelpful, or potentially harmful, while contributing to the costly and inefficient management of musculoskeletal pain and disability. To my knowledge, there have been no robust systematic investigations into publication bias in physical therapy. Such an investigation is an arduous task, but a necessary one. In the context of the preponderance of publication bias in nearly every facet of medical research, it is unlikely that physical therapy is an outlier in this regard.
What we do know is that physical therapy research has historically fared quite poorly in one of the best measures to reduce missing data, prospective trial registration. An informal appraisal detailed in Maher et al’s 2004 paper stated: “registration is not yet common practice and few physical therapy trials are registered. For example, only 52 trials with exercise or other physical therapy interventions are registered on the US clinical trials register (based on a search conducted on April 13, 2003)”. In 2013, Pinto et al surveyed a sample of 200 published trials and found that only 34% of the trials were properly registered and only 12 of the 200 trials were prospectively registered. In the 32 trials that showed unambiguous primary outcomes in their trial registration, only 5% of these trials were registered prospectively. These are dismal results.
Echoing the results of Pinto et al, Babu (2013) found that among clinical trials in 13 Medline indexed physical therapy journals published between January 1, 2008, and December 31, 2012, only 29% reported trial registration details. Despite this discouraging figure, there is a reason for optimism as Babu et al found that trial registration trended upward from 3% of trials registered in 2008 to 40% in 2012. Many of the larger physical therapy journals now mandate prospective trial registration, which is an important step in reducing publication bias. An updated systematic assessment of prospective trial registration rates in physical therapy clinical trials would be a worthy and informative endeavor
Reasons For Publication Bias
Science is ultimately a human endeavor conducted in an environment that shapes the behaviors of researchers and academics through various constraints and incentives. Publication bias has proliferated throughout the scientific community in part due to perverse incentives and a publish or perish culture that values positive, novel and headline-grabbing results over scientific truth. As Higginson and Munafò write “current incentive structures are in conflict with maximizing the scientific value of research.”
Academics are measured in part by the number of publications they produce and the subsequent citations their work receives. As discussed earlier, positive results are far more likely to be published than negative results. After publication, this problem compounds given that positive results are also far more likely to be cited than negative results. For example, a 2016 article from Misemer et al found that “positive trials of tPA for ischemic stroke are cited approximately three times as often as neutral trials, and nearly 10 times as often as negative trials, indicating the presence of substantial citation bias.” Illustrating both the limited publication of negative trials and the subsequent minimal citation, Catalini et al found that “Out of 762,355 citations from 15,731 articles in the Journal of Immunology (1998–2007), we identified 18,304 as negative (about 2.4% of the total). The 762,355 citations referred to 146,891 unique papers, and of these papers, 10,405 (about 7.1%) received at least one negative citation.”
This pressure to produce positive trials can also bring further distortion to the evidence base in the forms of selective outcome reporting. This is the act of withholding negative outcome measures from publication, including only the favorable outcome measures in the final publication. This practice seems unfortunately present in physical therapy literature, and again, is worthy of more systematic investigation. Pinto et al found that selective outcome reporting was seen in nearly half of the 49 trials where this was able to be assessed.
Unfortunately, the current academic culture is one that fosters publication bias. This is not without some hope. In recent years there has been a fervent interest in open science, data sharing, and strategies to combat various pitfalls of evidence-based practice, including publication bias. Efforts such as OpenTrials and The Center for Open Science are encouraging.
Strategies to Minimize Publication Bias
Prospective trial registration remains of the best ways to combat publication bias in the medical literature. Prospective registration on pages such as ClinicalTrials.gov allows for a documented public record of clinical trials that are planned and currently under investigation. Ideally, these public records include details about primary outcome measures, inclusion/exclusion criteria, statistical methods, and other important information.
Fortunately, many scientific journals (physical therapy included) have moved to mandate prospective registration. This is an important step towards transparency and open science. Unfortunately, there appears to be a reason to give this mandate pause as a solution to the issue of publication bias. Scott et al found that “Although standards are in place to improve prospective registration and transparency in clinical trials, less than 15% of psychiatry trials were prospectively registered with no changes in POMs. Most trials were either not prospectively registered, changed POMs or the timeframes at some point after registration or changed participant numbers.” This troubling trend can also be seen by Farquhar et al when they write “Although the ICMJE's call for compulsory trial registration dates back to 2004, between 2010 and 2014 only 45% of fertility trials were registered, and the annual trial registration rates did not improve substantially over this time.“ Clearly, there is still work to be done. An assessment of physical therapy trials adherence to a call for compulsory trial registration would be another worthwhile project to further illustrate any potential bias in the available body of evidence.
Registered reports expand upon pre-trial registration by submitting the trial and its methods, prior to actual completion and before results are known, to peer review. This can help identify potential shortcomings or blind spots in the trial’s design and importantly, as Lee et al write “registered reports are an advance on preregistration because they mandate publication irrespective of the results, which reduces the risk of selective publication.”
Further, physical therapy should seek to adopt a culture of embracing negative results. Negative trials provide remarkable value in a healthcare system that can be thought of as providing far too much care. Understanding what does not work, in who, and why should be of utmost importance if the profession wants to be a lean, effective alternative to more costly avenues of care. Authors, journals, and editors need to insist upon compulsory prospective trial registration with active monitoring to ensure appropriate research practices. They also must be at the forefront of embracing the culture of negative trial results by encouraging the submission and acceptance of these trials.
Publication bias is a rampant issue throughout the medical literature, and it is unlikely that physical therapy is exempt from this. What few investigations have been done on the issues of publication bias and selective outcome reporting highlights this. Further scoping reviews of the prevalence of publication bias and selective outcome switching, along with appropriate utilization of prospective registration should be a priority. Registered reports should continue to be encouraged and utilized to increase the quality of physical therapy research. A distorted evidence-based affects every member of the profession, and most importantly, disrupts the ability to provide the best care to patients seeking physical therapy services.
Header image from the 2013 film Enemy directed by Denis Villeneuve with cinematography by Nicolas Bolduc