Chest pain is one of the most frequent presenting symptoms for admitting patients to the emergency department (ED)^[1], which is usually caused by acute coronary syndrome (ACS), other cardiac and noncardiac diseases (more than 80%). In the ED, ACS should be rapidly distinguished from a variety of other cardiac and noncardiac diseases causing chest pain^[2]. Furthermore, ACS ranges from unstable angina and non-ST elevation ACS (NSTE-ACS) to the more severe transmural ST elevation ACS (STE-ACS), which may result in sudden cardiac death. NSTE-ACS has lower morbidity and mortality but relates to the development of a subsequent, larger myocardial infarction complicated by mechanical or arrhythmogenic complications, revascularization and/or death [major adverse cardiovascular events (MACEs)]^[3]. Different from chest pain patients with STE-ACS who need rapid clinical intervention^[4], early and accurate risk stratification is necessary to the chest pain patients with NSTE-ACS in the ED, in order to determine optimal clinical management and further treatment strategies after discharge^[5].

It is very beneficial for chest pain patients with NSTE-ACS to find suitable risk scores systems for risk stratification. Generally, GRACE (Global Registry of Acute Coronary Events), CRUSADE (Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA guidelines), HEART (History, ECG, Age, Risk factors, & Troponin) and TIMI (Thrombolysis in Myocardial Infarction) risk scores systems^[6] are employed to predict MACEs. However, these scores system lack the ability to track missing data and can only predict limited risk factors associated with MACEs. For example, HEART risk score as a classical method to predict MACEs, only takes 5 indicators into consideration, including History of chest pain with suspected ACS, Electrocardiograph (ECG), Age, Risk factors and Troponin (Each of indicator can be scored with zero, one or two points according to the extent of the abnormality)^[2]. TIMI risk score evaluates 7 risk factors as another simple prognostication scheme that categorizes a patient's risk of death and ischemic events and provides a basis for therapeutic decision making^[7].

During the past decades, machine learning (ML) techniques^[8] has presented itself with impressive performance in clinical application regarding diagnosis^[9], risk stratification^[10,11], and treatment reaction prediction^[12] due to its capability of processing large amounts of complex data by different classifier algorithms^[13,14]. Moreover, ML approach as the data-driven prediction model is easier to include more potential risk factors into model, not restricted by inclusion and exclusion criteria, and capable of dealing with missing data by applying computer algorithms to large datasets with numerous, multidimensional variables, capturing high dimensional, non­linear relationships among clinical variables^[15]. In recent year, ML approach has been used for MACEs prediction of ACS patients^[16,17], coronary type 2 diabetic patients^[18] and geriatric patients^[19], and the studies concluded that ML methods can improve both the identification of MACE risk patients and the prediction compared with traditional statistical techniques even in a small sample size. However, it rarely be applied to chest pain patients with non-ST-segment elevation acute coronary syndrome (NSTE-ACS) to predict MACEs and our study try to fill this gap.

Therefore, our study established ML by different classifier algorithms and proved its superiority over HEART risk score to predict MACEs within 3 months in chest pain patients with suspected NSTE-ACS. Ultimately, ML model constructed by Naïve Bayes predicted risk factors of MACEs from 14 categorical variables.

In this retrospective study based on a prospectively acquired database, chest pain patients who were treated at Beijing Anzhen Emergency Chest Pain Center, Capital Medical University from September 2014 to February 2015 and at Beijing Bo’ai Hospital, China Rehabilitation Research Center from September 2014 to May 2018, were consecutively enrolled. Beijing Anzhen Hospital and Beijing Bo’ai Hospital, as two of the best cardiovascular hospitals in China, have more concentrated patients with chest pain, more accurate diagnosis of non-ST-ASC with less bias, therefore the chest pain patients with non-ST-ACT are more representative. The patients who were 18 years or older, presented to ED as a result of chest pain with from onset to arrival at the emergency room greater than 2 hours, were included. Patients were excluded if their chest pain caused by with STE-ACS, aortic dissection, pulmonary embolism, trauma or arrhythmia, or they were in end-stage disease, or they were pregnant women or unable or unwilling to provide informed consent. A 3-month follow up was carried out by telephone interview to collect data. Patients were also excluded if their data were incomplete. The study was approved by the Hospital Ethics Committee. All patients had signed informed consent. All procedures performed in studies involving human participants followed the ethical standards of the institutional or national research committee and the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Patients’ data were collected once they arrived at the ED, including sex, age, medical history, risk factors (any risk factor, including but not limited to hypertension, hypercholesterolemia, diabetes, family history of coronary artery disease, smoking and obesity, body mass index, atherosclerotic disease) and electrocardiogram. As shown in Table 1, the 14 categorical variables were selected for further constructing radar plots and ML models.

Endpoint 　The endpoint was the occurrence of any MACE including acute myocardial infarction (AMI), percutaneous coronary intervention (PCI), coronary artery bypass grafting (CABG) or all-cause death within 3 months.

Complete Approach　As shown in Supplementary Figures S1–S2 (available in www.besjournal.com), the study cohort was randomly split into 2 datasets, 67% of data was used to train ML models and 33% of data was used to test the developed ML models. A total of 14 features were selected. Five ML models developed by Decision tree, Naïve Bayes, Linear SVC, Logistic Regression and Random Forest predicted the occurrence of MACEs of chest patients with suspected NSTE-ACS. The prediction for MACEs included any event of MACEs, AMI, revascularization and all-cause death in our study. The learning metrics, including Accuracy, Precision, Recall, F-Measure and AUC were used to assess performance of each model. The prediction effect of the ML models and HEART risk scoring system was compared. Ultimately, Naïve Bayes was selected to predict risk factors of occurrence of MACEs (any event, AMI, Revascularization, and all-cause death).

Figure S1.  The flowchart of completed approach.

Figure S2.  Flow chart of study participants.

Classification Methods　The classification was conducted by using the standard classifiers in ML including Decision Tree, Naïve Bayes, Linear SVC, Logistic regression and Radom Forest. The development of ML models consisted of two stages: training and testing. Prior to the training, the preprocessed dataset was split into training set (67% of the data) and testing set (33% of the data). In the training stage, the labels and features were used to define feature vectors. After that, classifiers were trained on the feature vectors extracted from the training set. In the testing stage, the previously trained classifiers predicted the label of each feature in the testing set. Each classifier was implemented to analyze the testing dataset, and the corresponding scores of the 14 features was obtained.

ML Models Validation　The performances of ML models constructed by those five classifiers were evaluated by calculating standard metrics, which were Accuracy, Precision, Recall and F-measure. Accuracy is the ratio of the number of correct predictions to total number of predictions. Precision is the ratio of correctly predicted positive observations to the total predicted positive observations. Recall is the ratio of correctly predicted positive observations to the all observations in actual class. F-measure is the weighted average of Precision and Recall. The standard metrics were calculated by following equations:

where TP is true positive, TN is true negative, FP is false positive and FN is false negative.

Both precision and recall were based on a consideration and measure of weighting function. F-measure was the harmonic mean of precision and recall, and balances in between, in the range of 0 to 1. The ML techniques were implemented in the open-source Python 3.6 environment in Anaconda 3.0 platform with Jupyter Notebook.

Statistical Analysis　The continuous variables were described as mean ± SD or as Medians (Interquartile Range). The categorical variables were reported as counts and percentages. Parametric continuous variables were compared between groups using Student’s t test and Man-Whitney U-test for non-parametric variables. Categorical data were compared using the χ² test for 2 × 2 tables. All statistical tests were two-tailed with P < 0.05 considered as statistically significant. The performances of the models were evaluated by Accuracy, Precision, Recall and F-measure. The comparisons between HEART prediction model and the ML prediction model above were performed in the aspects of those standard metrics.

Creening process for the study population was shown in Supplementary Figure S2. A total of 1,499 patients were enrolled in this study via screening 1,907 patients with chest pain presenting to the emergency clinic according to exclusion and inclusion criteria. In addition, 27 patients were lost to follow-up. Ultimately, 1,472 patients were enrolled. Most of the participants were male and has underlying diseases. There were 617 patients (41.9%) experiencing at least one MACE (including AMI, PCI, GABG and death) during the 3 months follow-up. PCI was the most common MACE which occurred in 473 patients (32.1%). AMI was ranked second in MACEs, with 206 patients (14.0%). CABG was needed for 121 patients (8.2%). There were only 10 deaths (0.7%). In addition, some chest pain patients experienced more than one MACEs. Specifically, 150 patients concurrently experienced PCI and AMI, 33 patients concurrently experienced AMI and CABG, 2 patients experienced PCI and CABG. Of the 10-death case, AMI, CABG and PCI concurrently occurred in 6, 4 and 3 patients, respectively. Few patients concurrently experienced 3 or 4 MACEs. In general, 617 patients underwent 810 events, an average of 1.31 events per patient.

Detailed baseline characteristics of the study population were shown in Table 2. Most of subjects were elderly, with an average age of 59.1 years. Males and females accounted for 64.8% and 35.2% of the population, respectively. Most of enrolled patients were accompanied by hypertension, with a proportion of 63.9%, while some were accompanied by hypercholesterolemia, diabetes, family history of premature CAD, MI, CABG, ischemic stroke, peripheral artery disease, carotid artery disease, elevated hs-cTn I, but all less than 35.0%. In addition, 13.3% and 29.6% of enrolled patients were suffering from obesity and current smoking, respectively.

As shown in Figure 1, the radar plot illustrated the features of patients with any one of MACEs, PCI, AMI, CABG or all-cause death, and identified important predictors of MACEs. Most patients with MACE(s) had history of chest pain with suspected ACS, and underlying diseases, especially hypertension and DM. Compared with the incidence of other MACEs, AMI could be more associated with history of chest pain with suspected ACS and higher troponin levels. Patients undergoing PCI had similar characteristics of AMI, but had lower troponin levels. Patients who experienced CABG were much older (data not shown here). Additionally, CABG were more correlated with hypertension and abnormal electrocardiogram results than other MACEs. Hypertension and hypercholesterolemia could more contribute to all-cause death. Compared with other features, hypertension, history of chest pain with suspected ACS, and smoking were more associated with any one of MACEs.

Figure 1.  (A) Radar plot for the 14 most important predictors of any MACE event; (B) Radar plot for the 14 most important predictors of PCI; (C) Radar plot for the 13 most important predictors of AMI; (D) Radar plot for the 14 most important predictors of CABG; (E) Radar plot for the 7 most important predictors of all-cause-death.

As shown in Table 3, from the perspective of these learning metrics, Linear SVC, Naïve Bayes and Logistic performed better than HEART scoring system in prediction for each of MACEs. Precision is the most important indicator to assess performance of ML models. The Precision scores in each MACE with classifiers of Linear SVC, Naïve Bayes and Logistic were higher than those with HEART scoring system. Especially, the Precision scores were equal or greater than 0.95 in AMI and all-cause death, while those scores were around 0.77 with HEART scoring system.

Linear SVC, Naïve Bayes and Logistic prediction models outperformed the other two models. The performance of Decision Tree and Random Forest Classifiers were worse than HEART scoring system when predict any event of MACEs and revascularization owing to all learning metrics by decision tree and random forest classifiers ≤ 0.75. However, decision tree and random forest classifiers performed greater in terms of predicting AMI (all learning metrics ≥ 0.95) and All-cause death (all learning metrics ≥ 0.99). On the whole, machine learning models constructed by any classifier improved the performance compared with HEART scoring system for AMI and all cause death prediction, based on that the Accuracy, Precision, Recall and F-Measure of all classifiers were ≥ 0.95.

However, the ROC curves and AUC in Figure 2 presented not exactly the same trend as the other 4 metrics in Table 3. For AMI prediction, the AUC by Random Forest was 0.96, the AUC by Decision Tree was 0.94, the AUCs by Linear SVC classifier, Naïve Bayes and Logistic were 0.98, while the AUC by HEART scoring system only was 0.85. Therefore, according to ROC curves and AUC, the ML models constructed by different classifiers performed better than HEART scoring system when predicting AMI. Linear SVC classifier, Naïve Bayes and Logistic performed better than HEART scoring system when predicted any event of MACEs, AMI and revascularization. However, the AUC of the ML model constructed by Linear SVC classifier, Naïve Bayes and Logistic was 0.75, lower than HEART (0.85) in prediction for all-cause death. The too low AUC of ML employing different classifiers could be associated with too small of sample size, only 10 death cases. In addition, the AUCs by Random Forest and Decision Tree were ≤ 0.80 to predict any event of MACEs and revascularization, while the AUC by HEART scoring system. Therefore, the performacne of ML models constructed by Random Forest and Decision Tree was worsen than HEART scoring system. Evaluating learning metrics and ROC curves comprehensively, Linear SVC classifier, Naïve Bayes and Logistic performed better than the other 2 classifiers, which was agreenment with the results by learning metric evaluation.

Figure 2.  ROC curves for (A) any event, (B) AMI, (C) revascularization and (D) all-cause death of machine learning and HEART. TPR: true positive rate; AMI: acute myocardial infarction.

Ultimately, considering great learning metrics (over 0.8 for Accuracy, Precision, Recall and F-Measure) and high AUC (0.88 for any event, 0.98 for AMI, 0.87 for revascularization), ML model developed by Naïve Bayes was used for predicting the occurrence of MACEs from 14 features. As shown in Figure 3, the history of chest pain with suspected ACS, abnormal ECG and elevated hs-cTn I were risk factors for any event of MACEs, revascularization and AMI, and the 3 risk factors also were taken into consideration for HEART scoring system. In particular, we observed that elevated hs-cTn I remarkably increased the incidence of AMI. Moreover, ML model also predicted that sex and smoking were also risk factors for any event of MACEs, revascularization and AMI, presenting the advantages of ML model when predicting MACEs^[20]. This finding couldn’t be achieved by HEART scoring system, which only takes 5 factors into consideration^[6]. We didn’t find any correlation between all-cause death and 14 features, which could be explained by that enough sample size is the basis of good performance for prediction.

Figure 3.  Prediction for the occurrence of MACEs (Any event, AMI, Revascularization, and all-cause death) from 14 features by ML model constructed by Naïve Bayes.

In this study, we analyzed important predictors of MACEs by radar plots and found that most of NSTE-ACS patients with MACEs had history of chest pain with suspected ACS, and underlying diseases, especially hypertension and DM. Furthermore, the radar plots accurately depicted the characteristics of each type of MACE for chest pain patients with NSTE-ACS, which provided effective patient profile for future prediction.

There were two reasons contributing to the performance improvement by ML models. Firstly, machine learning is capable of processing large amounts of complex data by different classifier algorithms. The machine learning model constructed by Naive Bayes in our study has a great learning metrics, with all Accuracy, Precision, Recall and F-Measure over 0.80 for predicting any event, AMI, revascularization and all-cause death. AUC ≥ 0.87 for predicting any event, AMI, revascularization. Secondly, the HEART scoring system can only predict the occurrence of MACE from 5 aspects of patients^[4,21]. However, ML method can integrate various characteristics of patients and more accurately predict type of MACE(s), so that prognosis of patients can be accurately predict and a better diagnosis and treatment can be provided to patients. Our study predicted the association between 14 features of patients and MACEs by ML. Nine of these features are irrelevant to any of MACEs, while three of these features, the history of chest pain with suspected ACS, abnormal ECG and elevated hs-cTn I, corresponding to “H” “E” “T” in HERAT scoring system, were risk factors of MACE. This indirectly demonstrated the rational of design of HEART scoring system and accuracy of prediction of ML. Moreover, although the evidence was not strong and significant compared with ECG, History of chest pain with suspected ACS and Troponin, ML model constructed by Naïve Bayes also revealed sex and smoking history could be risk factors of MACEs. This finding would never be mentioned in other scoring system, indicating ML can capture more risk factors. This finding also reminds us that the chest pain patients with NSTE-ACS who is accompanied with smoking history also should alter the occurrence of MACEs. As we observed no strong evidence, future more patients should be included to analyze the risk factors by machine learning. In addition, this ML model can also be applied to other disciplines and other diseases, not limited to this research^[22]. Feasibility and effectiveness of ML approach for the identification of predictors of events after an acute coronary syndrome has been proved by B, Fabrizio et al.^[23]. As for clinical prognosis of young patients with hypertension, Xueyi Wu et al. demonstrated that the ML approach was comparable with Cox regression and was better than recalibrated Framingham Risk Score model^[24]. In our study, although ML models constructed by Random Forest and Decision didn’t not always outperform HEART scoring and the results obtained by ROC and AUC were not identical with results obtained by the 4-learning metrics, the most encouraging result was that ML model constructed by every classifier presented the better performance than HEART scoring systems when predicted AMI, and the ML model constructed by Linear SVC, Naïve Bayes and Logistic could be considered superior over HEART scoring system in prediction for MACEs, especially for AMI and all-cause death.

However, there were still several limitations in our study. It was retrospective analysis in two centers, but ML need more prospective validation in multiple centers. The study population consisted of Chinese patients with chest pain from the ED who had a higher rate of MACE(s) than the rest of the population. Therefore, the high-latitude characteristics of ML are not fully reflected, and whether it can be applied to other populations needs further research. The sample size was not large enough in each MACE and it might be one of the potential negative factors for the performance of ML model.

In this study, ML models for risk prediction for MACEs in chest pain patients with NSTE ACS during 3 months after the first emergency were constructed and validated successfully. Compared with HEART risk scoring system, ML method presented apparently advantage when predicting AMI and still had a better performance when Linear SVC classifier, Naïve Bayes and Logistic classifiers were chosen. ML model developed by Naïve Bayes successfully predicted the correlation between incidence of MACEs and 14 features and revealed that history of chest pain with suspected ACS, abnormal ECG and elevated hs-cTn I, sex and smoking history could be risk factors of MACEs. Overall, this study demonstrated feasibility and superiority of ML method to predict the MACEs in chest pain patients with NSTE-ACS. This finding also reminds us that the chest pain patients with NSTE-ACS who is accompanied with smoking history also should alter the occurrence of MACEs.

The authors acknowledge the physicians at Emergency & Critical Care Center in Beijing Anzhen Hospital affiliated of Capital Medical University for their assistance in subject recruitment (with special thanks to Prof. WANG Mei, WANG Chun Mei, and LI Qing Xiang).

NIE Shao Ping, WANG Xiao and ZHENG Zhi Chang contributed to study design. WANG Nian and ZHENG Zhi Chang analyzed and interpreted the data. MA Chun Peng and AI Hui provided their technical expertise for coronary angiography. ZHENG Zhi Chang drafted the initial manuscript, ZHENG Zhi Chang and YUAN Wei worked together on the revision and writing of the final manuscript. WANG Xiao and JIANG Bo provided critical advice in the data analysis, interpretation and critically assessed the manuscript. All authors read and approved the final manuscript.

No additional data are available.