trials, the possibility of bias is minimal and therefore the observed outcome difference between treatment groups is either a genuine effect or due to chance variation. Significance tests enable one to assess the strength of evidence that a real effect is present rather than a chance finding. There are three main types of outcome data analyzed in contemporary studies, with different measures and tests of association, as shown in Table 6.1.
Table 6.1 Three main types of outcome data are analyzed in contemporary studies with different measures and tests of association.
| Type of data | Example | Measure of association | Test of association |
|---|---|---|---|
| Binary | In‐stent restenosis | Odds ratio | Chi‐square test |
| Time to event | Time to death | Hazard ratio | Log‐rank test |
| Quantitative | Late loss (mm) | Mean | t‐test |
While the calculations differ, the underlying principle is the same for all significance tests. For example, in the Prospective Randomized Comparison of the BioFreedom Biolimus A9 Drug‐Coated Stent versus the Gazelle Bare‐Metal Stent in Patients at High Bleeding Risk (LEADERS FREE) trial [1], 2432 patients at high bleeding risk (HBR) and coronary artery disease (CAD) with a clinical indication for percutaneous coronary intervention (PCI) were randomly allocated to undergo PCI either with the BioFreedom polymer‐free biolimus A9‐coated stent (n = 1221) or a similar bare‐metal stent (BMS, n = 1211) and followed for at least one year. The primary safety outcome was the composite of cardiac death, myocardial infarction, or definite or probable stent thrombosis. At one year, the number (%) of patients with a primary outcome event in the BioFreedom and BMS groups was 112 (9.4%) and 154 (12.9%), respectively, with a log‐rank p‐value < 0.001.
The interpretation of the p‐value is predicated on the formulation of the null hypothesis, which in the case of the LEADERS FREE trial assumed that both BioFreedom stent and BMS were equally effective in patients at HBR and CAD with clinical indication for revascularization. Then the p‐value is defined as the probability p of detecting a difference of 9.4% vs 12.9% or larger under the assumption that no true difference exists (i.e. the null hypothesis is true).
The answer from the log‐rank test is a probability < 0.1% (p‐value < 0.001).
The smaller the probability p, the more convincing the evidence to contradict the null hypothesis. In the case of the LEADERS FREE trial, we have strong evidence that BioFreedom biolimus A9 coated stent reduces the risk of the primary safety endpoint compared to BMS.
Estimating the magnitude of effect
Conventional metrics to quantify the magnitude of a treatment effect (i.e. measure of association) include the risk ratio or relative risk; relative risk reduction, odds ratio, number needed to treat, and hazard ratio (see below). Examples of calculating each are provided in Table 6.2 with corresponding examples from the LEADERS FREE trial.
Table 6.2 Common measures of association and calculations.
| Measure of association | Calculation | LEADERS FREE example |
|---|---|---|
| Relative risk (RR) | Event rate in group 1/ Event rate in group 2 | 0.096/0.129= 0.74 |
| Relative risk reduction (RRR) | 1–RR | 1–0.74 = 0.26 |
| Odds ratio (OR) | (Prob of event in group 1/Probability of no event in group 1) / (Prob of event in group 2/Probability of no event in group 2) | (0.096/0.904) / (0.129/0.871) = 0.720 |
| Number needed to treat (NNT) | 1/Absolute risk reduction | 1/(0.148–0.106) ~ 24 |
There is no “correct” singular metric to quantify a treatment effect. Often, it is recommended to incorporate several of these to appreciate both relative and absolute effects.
A 95% confidence interval to express uncertainty
Any estimate of treatment effect in a clinical trial contains some random error, and calculating a confidence interval (CI) enables one to see within what range it is plausible that the true effect lies. For instance, the observed relative risk for 30‐day all cause mortality in the Safety and Efficacy of Femoral Access vs Radial Access in ST‐Segment Elevation Myocardial Infarction (SAFARI‐STEMI) trial is 1.15, while the 95% CI is 0.58–2.30 [2]. This means that one is 95% sure that the true relative risk is in this interval. To be precise, there is a 2.5% chance that the true RR lies below 0.58 and 2.5% that the true RR is greater than 2.30. The larger the sample size, the tighter the CI becomes. Specifically, to halve the CI width one needs a trial four times the size.
Interpreting p‐values
Use of significance tests is often misleadingly oversimplified by putting too much emphasis on whether p is above or below 0.05. A p < 0.05 means the result is statistically significant at the 5% level, but is an arbitrary guideline. It does not mean one has firm proof of an effect. By definition, even if two treatments are truly identical there is a 1 in 20 chance of reaching p < 0.05. Also, p > 0.05, not statistically significant (or n.s.), does not necessarily imply that a clinically meaningful difference does not exist.
This concept is illustrated graphically in Figure 6.1. In the left hand panel of this figure, similar treatment effects are obtained from two different studies, one of which is significant and one is not. The lack of significance alone should not be the sole metric on which to interpret the findings, particularly as the effect size appears to be large, albeit imprecise. In contrast, in the right‐hand panel, a p‐value of 0.05 is obtained with two very different effect sizes, one which is large and another that is much smaller. Again, focusing on the p‐value alone as the sole discriminator of importance in treatment effect would ignore the very large and perhaps clinically relevant gradient of effect between the treatments.
Figure 6.1 Interpreting p‐values.
Link between p‐values and confidence intervals
In case of categorical outcome, if we have p < 0.05 then the 95% CI for the risk ratio (or odds ratio) will exclude 1; while if p > 0.05 is observed then the 95% CI will include 1. Similarly, in case of continuous outcome, if we have p < 0.05 then the 95% CI for
