Score Development and Internal Validation
Risk scores are generally developed using data from prospective observational cohort studies. The predictive models calculate a score from a series of weighted risk factors. The weights are regression coefficients or hazard ratios that account for how important each risk factor is in predicting cardiovascular disease. There are a number of stages in selecting the variables to be incorporated into a model, with various statistical methodologies employed. Potentially relevant risk factor variables are selected for investigation on the basis of clinical knowledge and existing epidemiological evidence. The associations between these variables and cardiovascular outcomes are modelled, initially in univariate analyses. Various models are then built, adding and removing variables to see whether they benefit the model in terms of goodness of fit (i.e., whether they improve the correlation between risk predicted by the model and risk observed in the study cohort). Variables that improve the model’s predictive capabilities are generally retained. Other factors may also contribute to the decision to retain or exclude variables from the final predictive model, including cost and ease of measurement and potential impact on patient motivation for lifestyle modification for example [32].
External Validation
External validation of risk scores is essential because prediction models generally perform better within the cohort in which they were created than when applied to another population. A systematic review of cardiovascular risk prediction models for patients with type 2 diabetes found that just under a third of published risk scores had been validated in external study cohorts [31]. Where there are substantial differences between populations, the predictive model may need to be recalibrated. Large studies of pooled international observational cohorts have shown that hazard ratios for particular risk factors are similar in most Western and Asian populations (there are limited data for African or Latin American cohorts) [33]. However, rates of cardiovascular disease vary considerably in different populations and geographical regions for numerous reasons. Disease rates also vary significantly over time within 1 location. Therefore, average absolute risk of cardiovascular events varies and this variability needs to be factored into risk scores. Rather than continually re-creating risk scores in each population and periodically over time, models can be recalibrated as long as population-specific data is available to determine contemporary mean risk factor levels and rates of cardiovascular disease.
Statistical assessment of risk scores involves 2 key factors: discrimination and calibration. Discrimination is the ability of the tool to identify those who will develop the disease and those who will not. This is commonly measured using the area under the curve (AUC) on a receiver operating characteristic curve, which incorporates both sensitivity and specificity. A similar measure is the concordance statistic or “c-statistic” [32]. Values range from 0.5, indicating no discrimination, to 1.0, indicating perfect discrimination. Calibration describes the correlation between risk predicted by the tool and the observed event rate in the population. There are a few methods for assessing calibration, including the Hosmer-Lemeshow test, which compares mean predicted risk to observed outcome rates across deciles of the distribution of expected risks [32].
Model Impact Studies
The implementation of cardiovascular risk scores and risk-based therapeutic decisions should be evaluated in interventional trials to ascertain whether their use actually alters clinical decision making and improves patient outcomes. A 2017 Cochrane review synthesised trials investigating cardiovascular risk scores [34]. It was not specific to patients with diabetes. Forty-one randomised control trials were identified. The review concluded that there was uncertainty as to whether use of risk scores altered cardiovascular event rates but there was some weak evidence that they may lead to more favourable risk factor levels and increased prescribing of preventative medications. Among patients with type 2 diabetes specifically, a systematic review found that only the Framingham Risk Score has been subjected to intervention trials [31]. Two out of the three of these trials had found some benefit with regard to the prescription of preventative medications, but no significant effect was observed on risk of cardiovascular events [35, 36]. Despite the integration of cardiovascular risk prediction models into diabetes guidelines, there is still inadequate clinical evidence to validate their role.
Risk Scores in Diabetes Guidelines
There is controversy about the use of risk scores in patients with diabetes given the cardiovascular risk inferred by diabetes itself. There is also concern that scores developed in general populations may not include diabetes-specific risk factors such as duration of disease and microalbuminuria. Thus, various guidelines have differences in their recommendations relating to the use of risk scores.
Generally, it is accepted that patients with diabetes at particularly high risk do not need evaluation with a risk score. Such patients include those with established cardiovascular disease, micro or macroalbuminuria or markedly elevated single risk factors (e.g., marked hypertension or dyslipidaemia) [37]. At present a number of organisations, such as the European Society of Cardiology, European Association for the Study of Diabetes, American Diabetes Association and Joint British Societies (including, among others, the British Cardiac Society and Diabetes UK), do not recommend the use of risk scores for patients with diabetes [27, 28, 38].
The World Health Organisation recommends the use of their risk prediction charts in patients with diabetes, unless there is overt nephropathy or other significant renal disease [39]. The International Diabetes Federation guidelines suggest assessment of absolute cardiovascular risk as an option for stratifying risk, with equations developed for people with diabetes preferred [37]. The International Diabetes Federation recommends that ultimately the choice of risk assessment should be made at a country level, taking into account local epidemiological data and the potential impact on healthcare resources.
The Framingham risk equation has been one of the most widely utilised and assessed scores. It is less widely recommended in patients with diabetes than previously, but is still recommended in some countries, including Australia unless other factors imply high risk, including age greater than 60 years, presence of microalbuminuria, moderate or severe chronic kidney disease, or markedly elevated systolic blood pressure or cholesterol [40]. The use of a score developed in a non-diabetes specific cohort established decades ago in a predominantly white town in the USA is debated [41]. This risk model has been externally validated in a number of diabetes populations, mostly showing reasonable discrimination (AUC 0.56–0.80) but poor calibration (p values <0.05 with the Hosmer-Lemeshow test) [31]. Since discrimination is based on the ability to score those who go on to have an event as higher than those who do not, and therefore relates to the importance of the risk factors included in the equation, it is not surprising
