derived from IPD
4.7 Assessing and Presenting the Overall Quality of a Trial
The results of validity checking (Section 4.5) and risk of bias assessment (Section 4.6) should be considered together in order to build up an overall picture of the quality of each trial’s IPD. This should include reflections on the quality of the trial design and conduct (from the ROB 2 assessment), checks of IPD obtained, and any unresolved errors or concerns therein. If it is concluded that the IPD from a particular trial is likely to introduce considerable bias into an IPD meta‐analysis, then it is may be sensible to exclude it. For example, in an IPD meta‐analysis of post‐operative therapy for non‐small‐cell lung cancer,107 a trial was excluded because it ‘failed’ the data checks,101 and it is certainly worth highlighting any such exclusions in the relevant meta‐analysis publication. However, such situations need to be handled sensitively with trial investigators, who will have invested time and effort in supplying the data, and may have been unaware that issues would emerge. Alternatively, the impact of risk of bias may be explored through sensitivity analysis, such as examining how meta‐analysis conclusions change according to whether or not trials have risk of bias concerns (Chapter 9).
The ‘traffic light’ table used for standard risk of bias assessment can be usefully adapted for summarising the overall quality of trials included in an IPD meta‐analysis. This would include all the domains described previously, except the “selection of the reported results” domain, which is not applicable to IPD projects (Section 4.6), because the trial IPD are re‐analysed according to the meta‐analysis protocol and SAP. Adopting this structure means that the information is readily comparable with a standard risk of bias table, but can easily be extended to incorporate columns for additional project‐specific IPD checks that are deemed particularly important. For example, it is useful to include an additional column to indicate whether there were any residual concerns about data quality once the data checking and correction procedures had been completed, and which might impact on the trustworthiness of a trial. Often these risk of bias ‘traffic light’ tables will be almost completely green (low risk of bias), as any trials that fail data checking or have serious bias issues would likely be excluded from the IPD meta‐analysis project completely. Note that assessments should be based on the fullest information possible, thereby considering the trial design and conduct based on all trial documentation and contact with investigators, plus the results of checks of the IPD. A more detailed risk of bias table might be included in an appendix to provide fuller information on this data checking process (Table 4.6), and show how individual judgements have been arrived at. It also provides an opportunity to flag less serious or unclear bias issues.
Table 4.6 Excerpt of a RoB2 table for an IPD meta‐analysis of adjuvant chemotherapy for locally advanced bladder cancer based on a single trial and the main outcome of overall survival (Tierney et al., in preparation).
Source: Sarah Burdett and Jayne Tierney.
| Risk of Bias Domain | 1) Randomisation process | 2) Deviations from the intended interventions | 3) Missing outcome data | 4) Measurement of the outcome | 5) Overall risk of bias judgement |
|---|---|---|---|---|---|
| Trial EORTC 30994 | |||||
| LOW RISK Was allocation sequence random? YES: Minimisation, stratified by institution, pathological T stage and lymph node status. Also, IPD checks show that the pattern of allocation is steady by treatment group and over time; there were no obvious imbalances by group on any day of the week; and there were few weekend randomisations. Was allocation sequence concealed? YES: Randomisation was done centrally at the EORTC headquarters. Did baseline differences suggest a problem? NO: IPD checks show no obvious imbalance by treatment group in baseline characteristics. | LOW RISK Were participants aware of their assigned intervention during the trial? YES: Blinding not possible in a chemotherapy versus none trial, but awareness cannot affect survival outcome. Were carers and people delivering the interventions aware of participants' assigned intervention during the trial? YES: Blinding not possible in a chemotherapy versus none trial, but awareness is unlikely to affect how these treatments were given. Were there deviations from the intended intervention that arose because of the trial context? NO: There were no deviations from because of the context. Was an appropriate analysis used to estimate the effect of assignment to intervention? YES: An intention‐to‐treat analysis of all randomised patients was derived from the IPD. | LOW RISK Were data available for all, or nearly all, participants randomised? YES: Data were provided for all patients randomised. | LOW RISK Was method of measuring the outcome inappropriate? NO: Overall survival was derived from the IPD according to the meta‐analysis protocol and SAP. Could measurement of the outcome have differed between intervention groups? NO: Checks of the IPD revealed that follow‐up of participants was balanced by treatment group. Outcome assessor aware of intervention received? YES: This cannot affect the overall survival outcome. | LOW RISK | |
4.8 Verification of Finalised Trial IPD
Once a trial’s IPD has been processed, checked and all issues resolved as far as possible, it is useful to give trial investigators a final opportunity to verify their data prior to it being used in any meta‐analysis. This is most easily achieved by sending a descriptive summary of the finalised data for each trial, any amendments made to the original data provided, and results of the primary analyses (Figure 4.13 gives an example), but offering the option to supply a copy of the full checked IPD, if required. Such a document may be useful if investigators raise queries about differences between trial data presented in the original trial publication and in the IPD meta‐analysis.
4.9 Merging IPD Ready for Meta‐Analysis
After verification, the finalised dataset for each trial is ready to be used within subsequent IPD meta‐analyses. At this stage, it is helpful to merge the final IPD from all trials into a single dataset. Although statistical methods for IPD meta‐analysis can still be applied if trial datasets are located locally in different files, a single dataset that houses all the IPD is more convenient and potentially makes analyses faster. For example, Box 1.1 (Chapter 1) shows an example dataset containing IPD from 10 trials after data checking, harmonisation, verification and merging. Most statistical packages, such as Stata, SAS and R, have built‐in commands for merging datasets from different files, and these generally require the datasets to share common
