External validation of the UK prospective diabetes study (UKPDS) risk engine in patients with type 2 diabetes identified in the national diabetes program in Iran

Cardiovascular diseases are the first leading cause of mortality in the world. Practical guidelines recommend an accurate estimation of the risk of these events for effective treatment and care. The UK Prospective Diabetes Study (UKPDS) has a risk engine for predicting CHD risk in patients with type 2 diabetes, but in some countries, it has been shown that the risk of CHD is poorly estimated. Hence, we assessed the external validity of the UKPDS risk engine in patients with type 2 diabetes identified in the national diabetes program in Iran. The cohort included 853 patients with type 2diabetes identified between March 21, 2007, and March 20, 2018 in Lorestan province of Iran. Patients were followed for the incidence of CHD. The performance of the models was assessed in terms of discrimination and calibration. Discrimination was examined using the c-statistic and calibration was assessed with the Hosmer–Lemeshow χ2 statistic (HLχ2) test and a calibration plot was depicted to show the predicted risks versus observed ones. During 7464.5 person-years of follow-up 170 first Coronary heart disease occurred. The median follow-up was 8.6 years. The UKPDS risk engine showed moderate discrimination for CHD (c-statistic was 0.72 for 10-year risk) and the calibration of the UKPDS risk engine was poor (HLχ2 = 69.9, p < 0.001) and the UKPDS risk engine78% overestimated the risk of heart disease in patients with type 2 diabetes identified in the national diabetes program in Iran. This study shows that the ability of the UKPDS Risk Engine to discriminate patients who developed CHD events from those who did not; was moderate and the ability of the risk prediction model to accurately predict the absolute risk of CHD (calibration) was poor and it overestimated the CHD risk. To improve the prediction of CHD in patients with type 2 diabetes, this model should be updated in the Iranian diabetic population.


Background
Cardiovascular disease (CVD) is the first leading cause of death and one of the most important causes of disease burden in the world [1][2][3][4][5].Each year, more than 17.5 million people die of cardiovascular disease worldwide, in other words, 31% of all deaths are due to this disease [6][7][8].This number is expected to raise to more than 23.6 million by 2030 [6][7][8][9].Among diabetics, the incidence of cardiovascular disease is two-four times higher than the general population [10][11][12][13].Practical guidelines and evidence-based medical recommend an accurate estimation of the long-term Mehrdad Valipour and Hamid Reza Baradaran contributed equally to this work and share first authorship.risk of cardiovascular for effective treatment and care.Estimating the impact of risk factors on the incidence of cardiovascular disease is one of the most important challenges in preventing and controlling these diseases.Several risk assessment models are currently available to estimate CVD risk, some of them, such as systematic coronary risk evaluation (SCORE) and the Framingham risk score model (FRS), have been designed for the general population and underestimate the risk of cardiovascular disease in diabetic people [14][15][16].Among several CVD risk assessment models developed in patients with diabetes, the United Kingdom Prospective Diabetes Study (UKPDS) risk engine is the most common model.This model by using risk factors such as age, sex, race, smoking, BMI, blood pressure, total cholesterol, HDL cholesterol and HbA1c estimates the tenyear risk of CVD in patients with diabetes [17].The results of various studies using the UKPDS risk score to estimate the risk of CVD disease were conflicting.In some societies, the risk of CVD was well-estimated, but in some societies it was overestimation (discrimination of the model was poor or moderate and calibration was poor) [18][19][20][21][22][23].This study was designed to investigate the external validation of the UKPDS risk engine in patients with type 2 diabetes identified in the national diabetes program in Iran.

Study population
In this cohort study among 19,453 eligible patients with diabetes registered in the national diabetes program between March 21, 2007, and March 20, 2018 in Lorestan province, 1105 people were randomly selected and their documents reviewed retrospectively.Among these patients, 942 had complete baseline examinations (LDL, HDL, total cholesterol, triglycerides, and HbA1c).but 89 patients had a follow-up period of less than 4 years and were excluded from the study.Inclusion criteria included, newly diagnosed diabetes, as fasting plasma glucose greater than 126 mg/dl on two occasions.Exclusion criteria included a history of CHD before diabetes, impaired endocrine disorder, and severe debilitating disease along with diabetes.

Predictors and their measurements
Demographic variables including age, gender, marital status, history of smoking in the lifetime and last year, the type of tobacco used and the date of commencement of smoking, the amount of physical activity during the week, family history of cardiovascular diseases, family history of stroke, and family history of diabetes at the time of diagnosis of diabetes and Seasonally after the diagnosis of diabetes were collected.Also, laboratory tests and clinical examinations Included LDL, HDL, total cholesterol, triglyceride, HbA1c, height, weight, systolic and diastolic blood pressure that were prescribed by a physician at the time of diagnosis of diabetes and once every three months after the diagnosis of diabetes was extracted from patients' medical records.In this study for external validation of the UKPDS risk engine 2.0 released, the variables of age and smoking at the time of diagnosis of diabetes were considered.The variables of HbA1c, systolic blood pressure, and lipid ratio (The ratio of total cholesterol to HDL) were considered the mean of the first and second years.The variable amount of HbA1c using Eq. 1 became the international standard (DCCT) format [24].In addition, total cholesterol, LDL, and HDL variables were converted to mmol/L before entering the model.Table 1 shows the Risk factors included in the CHD model.

Statistical analysis
We excluded individuals with missing values for one or more of the variables (smoking status, total cholesterol, HDL-cholesterol, systolic blood pressure, HbA1c, and ethnicity) used to calculate the UKPDS Risk Engine CHD risk estimates.We calculated the observed CHD risk and the estimated CHD risk using the UKPDS risk engine.The performance of the models was assessed in terms of discrimination (the ability of the risk prediction model to discriminate patients who got a CHD event from those who did not) and calibration (the ability of a risk prediction model to accurately predict the absolute risk of CHD event that is subsequently observed).The discriminative ability of the model was determined by calculating the Harrell's C-index statistic AND Calibration was assessed with the Hosmer-Lemeshow χ2 statistic (HLχ2) test.In our study, a CHD risk equation is considered good if the C-index is ≥ 0.75.The values between 0.51 and 0.74 are considered moderate and CHD equation with C-index ≤ 0.5 considered as poor discrimination.All analyses were completed using Stata

Ethics statement
This retrospective study was conducted in patients with type 2 diabetes identified in the national diabetes program and approved by the Deputy of Research and Ethics Committee of Iran University of Medical Sciences, Tehran, Iran.

Results
Among 942 new patients with diabetes, 853 had complete data (smoking status, total cholesterol, HDL-cholesterol, systolic blood pressure, HbA1c, and ethnicity) and were included in the study.During 7464.5 person-years of followup 170 first Coronary heart disease occurred.The mean and median follow-up were 8.3 and 8.6 years, respectively.In this study, the mean (SD) age at the time of diagnosis of diabetes was 52.6 (11.6) years for women and 55.6 (11.7) years for men.20% of men and 4% of women smoked, mean (SD) body mass index was 25.2 (4.5) for men and 27 (4.4) for women.Also, the mean first and second year after the diagnosis of diabetes the variables of hemoglobin A1C, systolic blood pressure, total cholesterol, and HDL cholesterol were 8.5(1.4),124(16.1),5.3(0.9), and 1.14(0.3) in men and 8.5(1.4),123(16.5),5.2(0.8), and 1.1(0.3) in women, respectively.Table 2 shows the baseline characteristics of the participants in this study and the UKPDS study.In this study, the average age of men was 56 years, which was 4.5 years higher than the UKPDS study but the mean age of women in the two studies was similar.The mean hemoglobin A1C in the subjects was about 2% higher than in the UKPDS.Also, smoking was about 14% for men and 21% for women, and systolic blood pressure was about 10 mm for men and 15 mm in women were less than UKPDS.However, the mean total cholesterol and HDL cholesterol of the two studies were similar.

Performance of the model for CHD outcome
In this study (excluding participants with a shorter followup than 4 years), the c-statistic was 0.72 for 10-year risk, in other words, discrimination was Moderate.However, the model calibration was poor (HLχ2 = 69.9,p < 0.001) and overestimated the risk of heart disease.Discrimination and calibration were similar for the 5, 6, and 8-year risk periods.Table 3shows the discrimination and calibration of the risk assessment model for different periods.Figure 1 shows the predicted and observed risk of heart disease in people with type 2 diabetes based on the deciles of the predicted risk.
In this study, the predicted risk of heart disease in people with diabetes was 78% higher than the risk of heart disease observed.In other words, the UKPDS risk engine78%  The main limitation of this study was retrospective cohort design, which resulted in selection bias during excluding the subjects who had insufficient follow-up time to calculate the CHD risk.This study shows that UKPDS risk engine overestimates CHD.To improve the prediction of CHD in patients with type 2 diabetes, this model should be updated in Iranian diabetic patients.

Conclusion
This study shows that the ability of the UKPDS Risk Engine to discriminate patients who developed CHD events from those who did not was moderate and the ability of the risk prediction model to accurately predict the absolute risk of CHD (calibration) was poor and it overestimated the CHD risk.To improve the prediction of CHD in patients with type 2 diabetes, this model should be updated in the Iranian diabetic population.

Discussion
This study shows that the ability of the UKPDS Risk Engine to discriminate patients who suffered from CVD events from those who do not was moderate and the ability of a risk prediction model to accurately predict the absolute CVD events (calibration) was poor and overestimated CVD risk prediction.Also, the UKPDS risk engine discrimination for predicting 5 and 8 years CVD events was moderate and calibration was poor and the risk was overestimated.
In a 2006 cohort study by Bo Kyung Koo et al., to evaluate the External validity of the UKPDS Model on 732 Korean diabetics, 46 heart diseases occurred during a 65-month follow-up period.In this study, UKPDS risk engine discrimination was moderate to poor, and the UKPDS risk score overestimated the risk of heart disease in Korean diabetics (AUROC, 0.578 [95% CI, 0.482-0.675])[25].In a study to evaluate the Performance of the UKPDS Cardiovascular Disease Risk score in Germany, 456 patients with diabetes from two population-based studies in southern Germany followed up10 years.The UKPDS risk engine was moderately discriminated (c-statistics = 0.64) and overestimated the risk of heart disease [19].In a study conducted in Malaysia to assess the 10-year risk of cardiovascular disease and compare its estimated risk with the UKPDS cardiovascular risk assessment model, random 660 patients with diabetes were selected to study.Although the UKPDS risk engine estimated the risk of heart disease better than the Framingham model, the UKPDS risk engine had moderate differentiation and poor calibration [26].In a study conducted in China, 7067 patients with type 2 diabetes were evaluated for heart disease risk.The calibration of the model was tested by the Hosmer-Lemeshow and discrimination by the ROC curve.In this study, 351 new heart diseases occurred during 5.5 years of follow-up.The model discrimination was 0.74%, and the UKPDS risk engine overestimated the risk of heart disease in Chinese diabetics [27].In a retrospective cohort study conducted in the UK, the external validity of the UKPDS Risk Engine by using routine healthcare data from 79,966 patients aged between 35 and 85 years from 1998 to 2011 to assess.The 10-year risk of observed heart disease was 6.1%, and the predicted risk based on the UKPDS risk assessment model was 16.5%, and the UKPDS risk assessment engine was moderately discriminated.As a result, The UKPDS Risk Assessment Engine overestimated the risk of heart disease in UK diabetics [18].Another study was performed using EPIC-Norfolk cohort study data.10,137 people aged 40 to 79 years were followed up for

Table 1
Risk factors included in the CHD model HbA1c( %), mean of values for years 1 and 2 SBP Systolic blood pressure (mmHg), mean of values for years 1 and 2 LR Total cholesterol/HDL cholesterol ratio, mean of values for years 1 and 2 Version 15.0.(STATACorp., College Station, Texas, USA).And The confidence level in all analyzes was 95% (error alpha = 0.05).

Table 3
Discrimination and calibration of the UKPDS risk engine for calculated risk periods of 5, 6, 8 and 10 years, with CHD as outcome years, and 69 cases of heart disease occurred.The external validity of the UKPDS and Framingham Risk Engine was assessed، both risk engines overestimated the risk of heart disease[28].In other studies, in different countries, the UKPDS risk engine overestimated the risk of heart disease in diabetics[23, 25, 29].The poor performance of the UKPDS risk engine for predicting CHD was the result of a selection of cohort study data that began in 1977.Diabetes is now detected at an earlier stage, drug treatment begins earlier and therefore reduces the risk of heart disease in the future.Also, the risk factors for heart disease change over time, and some risk factors are better treated.