distribution of baseline characteristics by treatment group and overall results for the main outcome(s). These can then be checked for concordance with relevant publications or, for unpublished trials, with any results that have been deposited in trial registers (e.g. ClinicalTrials.gov). However, it should be borne in mind that inconsistencies can arise if, for example, follow‐up in a trial’s IPD has been extended beyond that used to derive the reported results, or if the meta‐analysis employs a different approach compared to the original trial analyses. When unexplained differences do arise, it is crucial to work with the original trial investigators to understand how and why they differ, and therefore, be in a position to report and explain any important discrepancies.
Figure 4.6 Summary of the data validity, range and consistency checks on IPD from a trial included in an IPD meta‐analysis of neoadjuvant chemotherapy for cervical cancer104 (N.B. The participant references are pseudonymised).
Source: Based on Neoadjuvant Chemotherapy for Locally Advanced Cervical Cancer Meta-analysis C. Neoadjuvant chemotherapy for locally advanced cervical cancer: a systematic review and meta-analysis of individual patient data from 21 randomised trials. European journal of cancer 2003;39(17):2470–86.
4.6 Checking the IPD to Inform Risk of Bias Assessments
Similar to conventional aggregate data reviews, assessing the reliability (quality) of included trials is also an important feature of the checking phase of IPD meta‐analysis projects. In such reviews, this is usually based on the risk of bias, a term that refers to the likelihood that included trials will generate biased results. In particular, the risk of bias assessment tool (RoB 2) can be used to evaluate potential bias in estimates of intervention effects from randomised trials.91 It includes five domains to be considered for each eligible trial: the randomisation process; deviations from intended interventions; missing outcome data; measurement of the outcome; and selection of the reported result. Within each domain, assessments are guided by multiple signalling questions (with answers: yes, probably yes, probably no, no, or no information), allowing a risk of bias classification for that domain (low, high, or some concerns). Finally, an overall risk of bias judgement can be made (low, high, or some concerns) based on all domains (Section 4.7).
In aggregate data reviews, assessment of risk of bias is usually based on the information available in trial publications and other publicly accessible documents, such as trial registration entries or published protocols, sometimes supplemented by information requested from trial investigators. In an IPD meta‐analysis project, it is common to obtain additional information from protocols, codebooks and forms, or direct from trial investigators, which can increase the clarity of risk of bias assessments compared to those based on trial reports alone.48,105 As discussed in Sections 3.4 and 4.2.6, it may be helpful to undertake an initial risk of bias or quality assessment at the planning stages, before considering whether to obtain the IPD. However, the collection of IPD does allow a deeper and more reliable appraisal of data quality and risk of bias than is possible with aggregate data, because there is the opportunity to generate information directly from the IPD. There is also the potential to seek additional or updated trial IPD for inclusion in a meta‐analysis, in order to reduce or remove the potential for bias in particular domains. For example, participants excluded from original trial analyses may be reinstated in the meta‐analysis, or more appropriate statistical methods might be used. Therefore, an overall risk of bias assessment for each trial would be based on whether the design and conduct of the trial, and the quality of its final IPD (after correcting any data errors) are likely to lead to biased results when the IPD are analysed.
While assessing risk of bias in an IPD meta‐analysis involves using many of the same domains and items listed in the RoB 2 tool,91 some items are not relevant, because the IPD circumvents the issue. For example, the availability of IPD avoids reliance on a trial’s original analysis methods and reported results. Table 4.4 lists the domains and signalling questions in the RoB 2 tool that are particularly relevant to an IPD meta‐analysis project, and summarises how appraisal of the collected IPD, and other information harnessed through an IPD approach, can inform risk of bias judgements. These individual domains represent a useful starting point for assessing the risk of bias associated with each trial in an IPD meta‐analysis project.
Ultimately, if the trial design, management or conduct are seriously flawed (e.g. failed randomisation, poor follow‐up, or early stopping), this might lead to trials that are wholly or partially unreliable. Depending on the gravity of the issues, a trial may need to be excluded completely from the analyses or be the subject of sensitivity analysis; at the very least, the issues need to be brought to the attention of the reader in the IPD meta‐analysis publication. The IPD project’s central research team should have a plan in place for how they will deal with these sorts of occurrences. However, the objective of data checking is not to police trials or uncover fraud, and in our experience wholesale fabrication of trial data is uncommon. Also, note that this is probably less likely to occur in large, complex or multicentre trials, where collusion between multiple parties would be required. Nevertheless, sometimes trials are encountered where it is suspected that the data have been purposefully manipulated or falsified, so such trials would need to be excluded from the analyses, and the reasons for doing so described in the meta‐analysis report.
Table 4.4 Domains in the Risk of Bias 2 tool91 (RoB 2) of particular relevance to IPD meta‐analysis projects
Source: Based on Sterne JAC, Savovic J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019;366:l4898
| RoB 2 domains and signalling questions that are relevant to IPD meta‐analysis projects | Additional information available from IPD |
|---|---|
| Domain 1: Randomisation process | |
| 1.1 Was the allocation sequence random? 1.2 Was the allocation sequence concealed until participants were enrolled and assigned to interventions? 1.3 Did baseline differences between intervention groups suggest a problem with the randomisation process? | Assessing the pattern of randomisation using trial IPD may reveal instances where randomisation has failed, or may reassure that the randomisation process appears robust (Section 4.6.1). IPD can also be used to check balance across a full range of covariates (Section 4.6.1). Though formal testing is not recommended, visually, the distribution of each covariate by treatment group can help flag any systematic or unusual differences. |
| Domain 2: Deviations from the intended interventions (effect of assignment to intervention) | |
|
2.1 Were participants aware of their assigned intervention during the trial? 2.2 Were carers and people delivering the interventions aware of participants' assigned intervention during the trial? 2.3 If Y/PY/NI to 2.1 or 2.2: Were there deviations from the intended intervention that arose because of the trial context? 2.4 If Y/PY to 2.3: Were these deviations likely to have affected the outcome?
| |
