Development and validation of a nomogram for predicting stroke risk in rheumatoid arthritis patients

Baseline demographics and clinical characteristics

The study included 218 RA with stroke patients and 1,136 RA patients in the primary cohort. The validation cohort comprised 95 RA with stroke patients and 486 RA patients. The clinical characteristics of patients are listed in Table 1. The baseline clinical data were similar between primary and validation cohorts. As shown in Table 2, univariate LR analysis of RA patients developing stroke indicated that the stratification of sex, age, SBP, CRP, ESR, TC, LDL and the history of hypertension, diabetes, AF, CVD, and CHD, were significantly different between RA with stroke and RA groups (P < 0.05) in the primary cohort.

Development of an individualized prediction model

All variables in this analysis are listed in Table 2. These were carefully selected to ensure parsimony and practicality of the final model (noted in the Methods section). We finally identified the following 10 variables with the strongest association with stroke risk: sex, age, SBP, CRP, ESR, TC, LDL, and history of hypertension, diabetes, AF, and CHD. Based on the final complex model of multivariate analysis shown in Figure 1, several variables were independently associated with stroke among RA patients, such as sex (0.63 [0.45–0.91]), AF (2.27 [1.08–4.68]), CHD (2.49 [1.70–3.64]), hypertension (2.08 [1.44–3.00]), SBP stratification (for 140–159 vs.<120, 1.64 [1.04–2.61]; for 160–179 vs. <120, 2.44 [1.13–5.20]), CRP stratification (for ≥64.32 vs. <9.06, 1.67 [1.05–2.68]), ESR stratification (for ≥84.8 vs. <29, 1.64 [1.03–2.62]), TC stratification (for ≥6.2 vs.<5.2, 0.35 [0.17–0.70], LDL stratification (for 3.4–4.1 vs.<3.4, 4.45 [2.35–8.68]; for ≥4.1 vs.<3.4, 4.22 [1.66–10.69]). The final simple model of multivariate analysis is shown in Figure 1. A comprehensive analysis of vital indicators of machine learning models, shown in Figure 2 and Supplementary Table 1, reveals that the effect of the LR model on nomogram was better compared with other machine learning models. In addition, the LR algorithm effectively predicted the current data, simultaneously indicating the better efficiency of the complex model than the simple model. Finally, the complex model was used to create the nomogram by incorporating the above independent predictors (Figure 3). The nomogram revealed the score of influencing factors, the personal total cumulative score, and the predicted risk value of the individual outcome event for RA patients.

Figure 1. Multivariate logistic regression analysis of data from RA patients developing stroke in the primary cohort (N = 1,354). Abbreviations: SBP: systolic blood pressure; CHD: coronary heart disease; AF: atrial fibrillation; CRP: C-reactive protein; ESR: erythrocyte sedimentation rate; TC: total cholesterol; LDL: low-density lipoprotein; OR (95% CI): odds ratio, 95% confidence interval.

Figure 2. Model evaluation (F1-score) results based on the number of features across six models. (A) primary cohort, N = 1,354 patients; (B) validation cohort, N = 581 patients). Abbreviations: GBDT: gradient boosting decision tree; KNN: k-nearest neighbors; LR: logistic regression; RF: random forest; XGB: XGBoost; SVM: Support Vector Machine.

Figure 3. A developed stroke nomogram in the primary cohort (N = 1,354). Abbreviations: SBP: systolic blood pressure; CHD: coronary heart disease; AF: atrial fibrillation; CRP: C-reactive protein; ESR: erythrocyte sedimentation rate; TC: total cholesterol; LDL: low-density lipoprotein. For example, a 70-year-old (47 points), male (30 points) RA patient with an AF (55 points) and CHD (62 points) history of 60 mm/H ESR (27 points), and 5 mmol/L TC (65 points) arrived at a total point value of 286, with a probability of 46% of developing a stroke.

Assessing the performance and internal validation of stroke nomogram

Figure 4 shows a good agreement between the predicted risk and observed outcomes in the primary and validation cohorts (slope = 1, intercept = 0 with simple and complex models). Further, we comprehensively assessed and compared the performance of developed models with the Framingham risk model (Table 3). The Hosmer–Lemeshow test showed no substantial deviation from perfect fit in simple versus complex model in the primary cohort (P = 0.385 vs. 0.097), revealing a good agreement in the probability of stroke between the predicted risk and observed outcomes in the primary cohort. The area under the receiver operating characteristic curve (AUC), used to measure the discrimination performance of models, showed that the complex model (AUC, 95% CI: 0.784 [0.750–0.818], P < 0.001) had a better diagnostic ability than the simple model (AUC, 95% CI: 0.747 [0.711–0.784], P < 0.001), P = 0.0016 and was different compared with the Framingham risk model (AUC, 95% CI: 0.808 [0.778–0.893], P < 0.001), P = 0.0631 in predicting the development of stroke in RA patients. In addition, the net reclassification indexes (NRI) and integrated discrimination indexes (IDI) were calculated based on comparing with the Framingham risk model, both simple and complex models correctly reclassified the subjects in predicting the stroke risk (NRI: 11.59 [2.90, 20.29], 20.30 [12.54, 28.05]. IDI:1.71 [–0.77, 4.18], 5.65 [3.41, 7.88] separately for simple and complex models vs. the Framingham risk model).

Figure 4. Calibration curves of (A) complex model in the primary cohort (N = 1,354), (B) simple model in the primary cohort (N = 1,354), (C) complex model in the validation cohort (N = 581), and (D) simple model in the validation cohort (N = 581). Calibration curves depicted the calibration of each model in an agreement between the predicted risks of stroke and observed outcomes of stroke. The y-axis represents the actual stroke. The x-axis represents the predicted stroke risk. The diagonal gray line represents the perfect prediction by an ideal model. The dotted line represents the performance of the nonparametric nomogram, of which a closer fit to the diagonal gray line represents a better prediction.

Clinical use of risk nomogram

The decision curve analysis (DCA) of the risk nomogram in the primary cohort (Supplementary Figure 2) indicated that when the risk threshold of stroke in RA patients was about 15%, intervention treatment was beneficial compared to either treat-all-patients or treat-none scheme. When the threshold value ranged from 15% to 55%, the net benefit was comparable based on the risk nomogram, suggesting that the performance of the complex model (blue line) was higher than that of the simple model (red line) in predicting the risk of stroke in RA patients.

Patient selection

A total of 8,389 RA patients, with 9.04% (758) prevalence of stroke, were selected from the inpatient Department of Rheumatology and Immunology of the First Affiliated Hospital of China Medical University from January 2011 to December 2018. According to the inclusion and exclusion criteria, 313 RA with stroke patients and 1,827 RA patients were selected (Supplementary Figure 1), aged 18 years or older. EMRs were classified and coded using the International Classification of Diseases Tenth Revision (ICD-10) of the Beijing clinical version (RA: M05.x~06.x; stroke [ischemic and hemorrhagic]: I60 I60.1–I60.0 I61 I61.0–I61.9 I69.0 I69.1 I63 I63.0–I63.9 I69.3). The study conformed to the principles outlined in the Declaration of Helsinki and was approved by the ethics committee of the Medical Science Research Institute of the First Affiliated Hospital, China Medical University (approval number: AF-SOP-07-1.0-01). All patients provided written informed consent for the use of their data.

The study criteria included the American College of Rheumatology (ACR) 1987/2010 [36] for RA and the cardiac vascular disease (CVD) criteria adopted at the Fourth Academic Conference by the Chinese Neuroscience Society in 1995 [37] for stroke. In the RA with stroke cohort, patients who met the following inclusion criteria were enrolled: 1) patients who conformed to the above stroke and RA diagnostic criteria; 2) if the timing of stroke was later than that of RA in the EMR, we believed that the patient had developed stroke after being diagnosed with RA; 3) patients had undergone laboratory tests (serum inflammatory, antibody, complement, lipid-assays) at least once when visited the hospital the first time; and 4) patients were older than 18 years. The RA cohort (without stroke) included patients who met the following inclusion criteria: 1) those who conformed to the above RA diagnostic criteria; 2) patients had undergone laboratory tests (serum inflammatory, antibody, complement, lipid assays) at least once when they first visited the hospital; and 3) patients were older than 18 years. The following exclusion criteria were applied to both cohorts: 1) patients who still suffered from other connective tissue diseases, including systemic lupus erythematosus, scleroderma, dry syndrome, and vasculitis; and 2) RA patients with coexisting ankylosing spondylitis and gout arthritis. Finally, we randomly selected 70% of the RA with stroke and RA patients as the primary cohort; the remaining patients comprised the validation cohort [38].

Data collection

All data were filtered from EMR, and primarily included personal information, such as age, gender, height, and weight; metabolic indices (serum TC, TG, LDL, and HDL); fasting blood glucose (FBG); serologic profiles including CRP, ESR, rheumatoid factor (RF), complement3 (C3), complement4 (C4), and anti-cyclic citrullinated peptide (anti-CCP) antibodies, and CHD, AF, LVH, and CVD history records. In addition, we included the medication history, i.e., hypertension and biologic disease-modifying anti-rheumatic drugs (Bio-med). All laboratory tests were performed using overnight fasting venous blood samples and conducted using clinical standard operating procedures (SOP) for inspection items. In addition, when results of multiple laboratory tests at different time points were summarized, the first laboratory test results were selected at first admission for RA with stroke cohort and the RA cohort.

Statistical analysis

All reported significant levels were set at 0.05 with a two-sided test. The categorical data are expressed as percentages by cohort. Certain continuous predictors (i.e., age, SBP, and CRP) were categorized using consensus approaches, guidelines, or previously published studies. In addition, the absence of certain features in clinical medical records was inevitable and accounted for less than 20%; we used multiple imputations to account for the missing data in SPSS 23.0. Chi-square test or Fisher’s exact test was used to compare the differences in participants’ characteristics between primary and validation cohorts. The univariable association between RA with stroke group and RA group of the primary cohort was assessed using univariate LR analysis. Based on this and clinical importance, scientific knowledge, and predictors identified previously [39], we developed and validated the model for predicting the risk of developing stroke in RA patients. The study report was built according Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist for development and validation of the prediction model (https://www.equator-network.org/reporting-guidelines/tripod-statement/), shown as Supplementary Figure 3.

Developing the model between the RA with stroke group and RA group in the primary cohort

The LR algorithm was used to develop an unadjusted simple model and a complex model adjusted by sex and age. We considered the disparity between male and female morbidity in RA patients and the aging of stroke patients. Further, we built the risk nomogram using multivariable LR analysis to provide the clinicians with a quantitative tool to predict the individual probability of stroke. All analyses were conducted using the R software version 3.6.2 (packages primarily including rms, Hmisc, dca. R packages; https://www.r-project.org).

Comparing multiple machine learning algorithms in primary and validation cohorts

Machine learning algorithms were used based on scikit-learn, an open-source machine learning library, using Bayesian optimization to implement algorithm optimization, and cross-validation method (N-folds = 5) to complete algorithm evaluation during optimization. We used six different machine algorithms running for three 30-min sessions, including LR, Support Vector Machine (SVM), random forest (RF), XGBoost (XGB), gradient boosting decision tree (GBDT), and k-nearest neighbors (KNN), to compare algorithms and evaluate simple and complex models in primary and validation cohorts, respectively. The cohorts were compared by evaluation metrics as follows: 1) accuracy: the proportion of patients who were predicted as their actual status; 2) precision: the proportion of patients who actually had the disease and were predicted out of patients who were predicted as having the disease; 3) recall: the proportion of patients who were predicted as having disease out of all patients who actually had the disease, corresponding to sensitivity; 4) F1-score: the harmonic mean of precision and recall. F1 = 2Precision × recall/(precision + recall); and 5) balance error (ber): combining bias and variance to reflect the accuracy of the model.

Comparing and validating the performance of developed models

The performance of the nomogram of the model was assessed by discrimination and calibration curves. Calibration curves, the accuracy of point estimates of the LR function, and the Hosmer–Lemeshow test were used to assess if the model calibrated perfectly. The discrimination of the nomogram was evaluated using C-index. The predictive accuracy for individual outcomes (discriminating ability) was equivalent to the AUC and was compared between the Framingham risk model and our prediction models. The models were internally validated using the validation cohort. The LR formula formed in the primary cohort was applied to all patients of the validation cohort. The NRI indicated the proportion of patients correctly reclassified by the new model compared with an existing or standard model, whereas IDI indicated the change in the difference in average predicted probabilities between RA with stroke and RA groups in the new and existing models [40]. Further, NRI and IDI between the Framingham risk model and our prediction models were assessed as having low risk (0–20%), medium risk (20%–59%), and high risk (60%–100%).

Clinical use

DCA was assessed in the primary cohort to evaluate the clinical usefulness of the nomogram by quantifying the net benefits at different threshold probabilities. These results were used to identify the predictive models with the best discriminative abilities [41]. In addition, net benefit was defined as the proportion of true positives minus the proportion of false positives, weighted by the relative harm of false-positive and false-negative results [42]. Simple and complex models were used to predict risk stratification of 1,000 individuals using bootstrapping and assessing clinical impact with decision curves.