Low Power
Small studies are often referred to as having low power. In medical research, statistical power refers to the chances (probability) that a study will detect a significant effect of treatment if one truly exists. A power of 80% is recommended, but few trials in medicine achieve this: in an overview of 136,000 trials only 9% of those published between 2010 and 2014 did so [86].
A consequence of low power is that spuriously significant results are more likely to occur [87]. Another problem is that, if there is a real benefit of treatment, small studies are unlikely to detect the benefit as being statistically significant. These apparently conflicting statements are true because chance is even‐handed; it will make some interventions appear to have a larger effect size than they do in reality, and will sometimes make the effect seem smaller than it really is [79]. Inflated effect sizes are more likely to be significant and reduced ones less so. The result is that small trials have much more heterogeneous effect sizes than large ones [88].
To ensure that the trial has adequate power, researchers should carry out a formal sample size calculation (specifying the likely size of the treatment effect as well as the required power and an estimate of variance). The frequency of reporting sample size calculations is often low. For example, only 41% of low back pain trials [89] and 35% neurosurgical trials [90] reported this calculation. Even when the sample size calculation is reported, researchers often overestimate the possible benefit of the treatment and end up with sample sizes that are too small to detect a clinically realistic effect [89, 91].
Industry‐Funded Trials
The research funded by drug companies ‘produces innovative, life‐saving medicines’, but it also has a darker side [92]. A common finding is that trials funded by the pharmaceutical industry are more likely to report that the treatment was beneficial [93, 94]. One review study concluded that ‘pharmaceutical company sponsorship is strongly associated with results that favour the sponsor's interests’ [95]. Another author commented that ‘big pharma is more concerned about commercial interests than patient harm’ [96]. Editors of leading medical journals, and reviews by academic researchers, have heavily criticised the pharmaceutical industry for sponsoring trials whose results ‘favour the sponsors' interests’ [93, 97, 98]. As industry fund the majority of large international trials, and is particularly good at disseminating their findings [99], any bias in their studies could create serious problems for evidence. Many industry‐funded trials involve collaboration between industry and academia, although academics are often not involved in the data analysis [100]. An evaluation of the research practices of the pharmaceutical industry led to the conclusion that ‘those who have the gold make the evidence’ [101].
In general drug company studies are at not at a higher risk of bias in methods than other trials [93], so the explanation for the higher frequency of positive results must lie elsewhere. Other possibilities have been suggested. These include the more frequent use of surrogate outcomes, and publication bias, in which trials with negative findings are not published [93, 102, 103]. An analysis of head‐to‐head trials (which directly compare drug versus drug, rather than drug versus placebo) showed that 96.5% of trials favoured the drug manufactured by the company funding the study [104]. This may provide evidence of industry manipulation. An analysis of internal documents from the industry found that suppression of negative studies and spinning of negative findings were recognised techniques [105]. As one author suggested, allowing drug companies to generate evidence ‘is akin to letting a politician count their own votes’ [103].
CONCLUSIONS
This chapter has explored the many sources of bias that commonly afflict randomised controlled trials. The randomised controlled trial is held to be the gold standard for evidence on treatment effectiveness, but that gold is more than a little tarnished. One commentator concluded that randomised controlled trials are: ‘often flawed, mostly useless, clearly indispensable’ [106].
The two key questions for clinical trials are: how frequently do these flaws occur; and how big an effect do they exert on estimated effect sizes? The frequency of flaws varies across individual review studies and by the type of deficiency, so there is not a specific estimate of how often bias occurs. Instead we can put the frequencies on a scale from very rare to very common. As most of the estimates from review articles are in the fairly common or very common area, flaws are a serious problem.
The magnitude of bias from randomisation and allocation concealment is generally modest, amounting to a 10%–15% increase in the estimated treatment effect. However these estimates are averages based on large numbers of trials that cover many different types of treatment. It is likely that the impact of bias is much larger for some trials than others, although we do not know which.
Some of the weaknesses in clinical trials may simply be due to lack of knowledge or experience. This could explain deficiencies in the handling loss to follow‐up or the lack of blinding of outcome assessment. Tampering with the randomisation sequence, and the replacement of primary outcomes, suggests a less innocent explanation. The deficiencies described in this chapter, which may result from inadvertent mistakes or deliberate actions, pose a serious threat to the integrity of medical evidence. The next chapter describes weaknesses in study design and conduct that lead to wasted and unhelpful trials.
REFERENCES
1 1. Riechelmann, R.P., Peron, J., Seruga, B. et al. (2018). Meta‐research on oncology trials: a toolkit for researchers with limited resources. Oncologist 23: 1467–1473.
2 2. Page, M.J., Higgins, J.P., Clayton, G. et al. (2016). Empirical evidence of study design biases in randomized trials: systematic review of meta‐epidemiological studies. PLoS One https://doi.org/10.1371/journal.pone.0159267.
3 3. Adie, S., Harris, I.A., Naylor, J.M. et al. (2017). The quality of surgical versus non‐surgical randomized controlled trials. Contemp. Clin. Trials Commun. 5: 63–66.
4 4. Savovic, J., Jones, H., Altman, D. et al. (2012). Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta‐epidemiological studies. Health Technol. Assess. 16: 1–82.
5 5. Dechartres, A., Trinquart, L., Atal, I. et al. (2017). Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study. BMJ https://doi.org/10.1136/bmj.j2490.
6 6. Wuytack, F., Regan, M., Biesty, L. et al. (2019). Risk of bias assessment of sequence generation: a study of 100 systematic reviews of trials. Syst. Rev. https://doi.org/10.1186/s13643‐018‐0924‐1.
7 7. Savovic, J., Turner, R.M., Mawdsley, D. et al. (2018). Association between risk‐of‐bias assessments and results of randomized trials in Cochrane reviews: the ROBES meta‐epidemiologic study. Am. J. Epidemiol. 187: 1113–1122.
8 8. Zhai, X., Cui, J., Wang, Y. et al. (2017). Quality of reporting randomized controlled trials in five leading neurology journals in 2008 and 2013 using the modified “risk of bias” tool. World Neurosurg. 99: 687–694.
9 9. Rikos, D., Dardiotis, E., Tsivgoulis, G. et al. (2016). Reporting quality of randomized‐controlled trials in multiple sclerosis from 2000 to 2015, based on CONSORT statement. Mult. Scler. Relat. Disord. 9: 135–139.
10 10. Saltaji, H., Armijo‐Olivo, S., Cummings, G.G. et al. (2018). Impact of selection bias on
