the value of clinical data sharing is more widely appreciated, persuading trial investigators of the value of participating has perhaps become easier. That said, IPD meta‐analysis research teams and those providing IPD are now faced with additional tasks, such as seeking ethical or other institutional approval for the exchange of IPD (Section 3.10), as well the development of detailed data‐sharing agreements (Section 3.11).
Over the years, advances in database and statistical software and electronic communication have greatly reduced the burden of labour required for the data exchange, management and analysis aspects of IPD projects.43 Even so, these remain the most time‐consuming and resource‐intensive phases (Section 3.8), and require skills and expertise beyond those needed for a conventional aggregate data review (Section 3.5). This is emphasised by a growing number of methodological articles relating to IPD meta‐analysis projects, covering issues such as data checking and harmonisation,7,28 statistical methodology,33,50–57 examining potential biases,46,58 dealing with unavailable IPD,59 reporting,60 and statistical software development,61 amongst others.
2.5 Empirical Evidence of Differences Between Results of IPD and Aggregate Data Meta‐Analysis Projects
There are many empirical comparisons of results produced by IPD meta‐analyses with results based on corresponding analyses of published aggregate data. An early example, in advanced ovarian cancer, found that results based on published aggregate data suggest a 7.5% absolute improvement in the percentage of women surviving at 30 months with platinum‐based chemotherapy, whereas the IPD meta‐analysis suggests a 2.5% improvement in the percentage surviving.8 This disparity, which could have led to different clinical conclusions, seemed to be driven by the IPD meta‐analysis project including more trials, participants and follow‐up, as well as including all of the events in a time‐to‐event analysis, rather than calculating a risk ratio from events observed at a fixed time‐point.
In contrast, in a different example, there was no clear evidence of an effect of ovarian ablation on survival of women with early breast cancer, based on the published aggregate data, but a 10% absolute increase in the percentage surviving at 15 years based on IPD.62 In this case, the IPD for the included trials incorporated much greater follow‐up, leading to a near doubling of events. This, and a more appropriate time‐to‐event analysis, were likely to be the key drivers for the discrepancy with the aggregate data findings. Many other comparisons of results from IPD and aggregate data meta‐analysis projects have been carried out, not only in cancer and cardiovascular disease where IPD meta‐analysis first gained traction, but also in other healthcare areas such as infectious diseases, neurology, nephrology and critical care. The differences shown between IPD and aggregate data findings are variable and seem context specific (e.g. depending on the research question; Sections 2.2.1 and 2.6.1).63
A large systematic review that brought together published comparisons of treatment effects from IPD and aggregate data meta‐analyses found that many pairs of IPD and aggregate data analyses agreed in terms of the statistical significance of the overall results for the main outcomes. However, the disagreement observed in 20% of cases could have led to different clinical conclusions.63 The discrepancies did not seem to be clearly associated with variation in the number of trials, number of participants or length of follow‐up.63 Importantly, discrepancies are likely to be more pronounced when going beyond overall treatment effects, which is often a key aim of an IPD meta‐analysis project, such as when examining treatment‐covariate interactions at the participant level (Chapter 7).33
Evidence from a large cohort of systematic reviews of the effects of cancer therapies on survival showed that, on average, meta‐analysis results for the overall treatment effect derived from published aggregate data (based on hazard ratios) were slightly more in favour of the research treatment than those from IPD.47 Although most results were similar between aggregate data and IPD meta‐analyses, those discrepancies that did occur were often substantial.47 Importantly, results from aggregate data were most likely to agree with those from IPD when the number of participants or events (absolute information size) and the proportion of participants or events available from the aggregate data relative to the IPD (relative information size) were large. This emphasises that assessing the amount of information provided by the available aggregate data, and what the obtainable IPD might add for a particular research question, is an important step in determining when IPD will bring the greatest value (Section 2.6.3).
2.6 Guidance for Deciding When IPD Meta‐Analysis Projects Are Needed to Evaluate Treatment Effects from Randomised Trials
Based on practical experience7,43 and the empirical evidence summarised in Section 2.5,47,63 any decision about undertaking an IPD meta‐analysis project should be based on the nature of the specific research question (Section 2.6.1), the completeness and uniformity of the available aggregate data (Section 2.6.2), the information size (Section 2.6.3), and the data and analyses required to address the research question reliably (Section 2.6.4). A checklist of questions to aid in this decision‐making is provided in Table 2.2, and is explained in more detail in the following subsections. The focus is on the evaluation of treatment effects from randomised trials. If the answer to one or more of the questions is “yes”, then an IPD project is likely to add considerable value compared to a conventional systematic review and meta‐analysis of aggregate data.
2.6.1 Are IPD Needed to Tackle the Research Question?
In certain circumstances, the rationale for obtaining and analysing IPD is immediately very strong, such as when the research question is focused on participant‐level relationships to inform more stratified or personalised approaches to treatment. In particular, if the desire is to evaluate whether particular types of participants benefit more or less from an intervention than others (Chapter 7), IPD will almost always be needed to obtain a reliable assessment of treatment‐covariate interactions.33 This is also true for situations where participant‐level diagnosis, prognosis and prediction are of interest (Chapters 15 to 17). Another powerful motivation for seeking IPD is in controversial areas, where independent scrutiny of the trial IPD may improve credibility and increase transparency, which may be sufficient justification in itself.20,65 On the other hand, if it is clear that a conventional aggregate data meta‐analysis could provide reliable (albeit less detailed) results, then an IPD project may be of less value, or may not be considered a priority. For example, if the sole research objective is to investigate the overall effect of a treatment, then it
