available) that summarise these effects (Section 2.6.2).
Table 2.2 Signalling questions to help decide when aggregate data are insufficient for meta‐analysis and IPD are needed, focusing on the evaluation of treatment effects from randomised trials.
Source: Jayne Tierney, adapting the figure presented by Tudur Smith et al.,64 with permission.
| Why IPD might be required | Signalling questions to help consider whether IPD are needed | Yes/No |
|---|---|---|
| To address the specific research question | Is going beyond overall treatment effects an aim of the project? (e.g. to examine treatment effects in relation to particular participant characteristics) | |
| Is independent scrutiny of one or more eligible trials required? (e.g. if some trial results are controversial or all trials arise from a single sponsor) | ||
| Is it reasonable to wait some time for the research question to be addressed?(e.g. if a good‐quality aggregate data meta‐analysis already exists, but IPD are needed for a more thorough or up‐to‐date analysis) | ||
| To improve the completeness and uniformity of the information | Are suitable aggregate data lacking for key outcomes of the trials? (e.g. publications do not provide a risk ratio, mean difference or hazard ratio for the overall effect; a treatment‐covariate interaction for each participant‐level covariate of interest, or the data to calculate these) | |
| Are outcome definitions corresponding to the aggregate data unsuitable or do they lack uniformity across trials? (This could make synthesis or interpretation of outcome effects using aggregate data difficult) | ||
| Are participant‐level covariate definitions corresponding to the aggregate data unsuitable or do they lack uniformity across trials?(This could make synthesis or interpretation of treatment‐covariate interactions using aggregate data difficult) | ||
| To improve the information size | Is the absolute information size represented by the aggregate data too small to detect realistic effects of treatment on the main outcomes? (i.e. the total number of participants, and total events if applicable) | |
| Is the relative information size represented by the aggregate data ‘low’ or potentially unrepresentative?(i.e. the proportion of all potentially eligible participants or events, if applicable) | ||
| For time‐to‐event outcomes, is the duration of follow‐up captured by the aggregate data too short? | ||
| To improve the quality of the analyses | Are the statistical analysis methods and assumptions used by the trials to produce the aggregate data inappropriate? | |
| Are the statistical analyses used by across the trials to produce the aggregate data incompatible? | ||
| Are continuous outcomes and variables handled inappropriately in the trial analyses that produced the aggregate data? | ||
| Answer to key questions “YES” = IPD may add considerable value |
It will not always be possible to complete an IPD meta‐analysis project in a suitably timely manner for the question of interest, because, for example, trial investigators are focused on completion of their individual trials, trial data are embargoed for a period, or it will take too long to set up data‐sharing agreements. Therefore, if a therapeutic area is moving very quickly, there is an urgent policy need, or results are required to inform an ongoing trial of the same treatment(s), a prospective aggregate data meta‐analysis,66, 67 perhaps as part of a living systematic review,67, 68 may be more suitable for delivering results in the shortest time frame. However, ultimately the merits of any aggregate data synthesis need to be balanced against the benefits that a good‐quality IPD project could bring, and for an important question, the extra time needed for IPD meta‐analysis may be justified to better inform decision‐making in the longer term. Instead, researchers might choose to complete a conventional aggregate data meta‐analysis first, with the intention of conducting an IPD meta‐analysis project in a subsequent stage, if more reliable and nuanced results are needed.
2.6.2 Are IPD Needed to Improve the Completeness and Uniformity of Outcomes and Participant‐level Covariates?
Before deciding whether to collect IPD for meta‐analysis, an important step is to assess the completeness and uniformity of the available aggregate data, either based on a pre‐existing systematic review that addresses a similar research question, or by conducting a scoping or systematic review of existing trials of interest. If outcomes and participant‐level covariates have been collected in the eligible trials, but are not (adequately) described in the associated trial reports (such as side effects of treatment, multiple time‐points or continuous values of prognostic factors), this can be rectified by the collection and analysis of IPD. Even if all the outcomes, participant‐level covariates and, if relevant, interactions required for the analyses have been reported, if they are not defined consistently across trials it can be difficult to include or combine their results in aggregate data meta‐analysis in a meaningful way. At best, this could lead to findings that are difficult to interpret, and at worst, that are unreliable. If this is a cause for concern, IPD might be sought to allow standardisation of the variables in readiness for analysis (Section 4.5).
2.6.3 Are IPD Needed to Improve the Information Size?
A major motivation for meta‐analysis is to increase the statistical power over that for a single trial. However, meta‐analysis may still not be sufficient to answer a particular research question reliably, as it depends on the potential absolute information size available from all existing trials. Determining this potential absolute information size, and subsequently statistical power (Chapter 12),69 should be considered in advance, and depends on the nature of the research question. For example, when using meta‐analysis to examine the overall effect of a treatment for a binary or time‐to‐event outcome, the absolute information size depends on the number of trials potentially available for meta‐analysis, as well as the number of participants and events in these trials. Between‐trial heterogeneity is also important, though it is difficult to gauge in advance. When examining participant‐level treatment‐covariate interactions, the variability of covariate values in each trial also contributes toward the potential absolute information size (Chapter 12).70
If the potential absolute information size based on all trials and their participants is small, and so statistical power is low, any meta‐analysis will struggle to detect realistic and clinically meaningful
