近年來電子健康紀錄資料庫發展迅速，使其規模愈來愈大且完善。隨之
而來的是許多研究開始利用電子健康紀錄來輔助臨床決策。機器學習已經
廣為運用在分析電子健康記錄，許多應用被提出來幫助臨床醫生準確識別
患者，以改善他們的整體診斷、預後和治療決策。其中在加護病房中識別
出即將心臟驟停或死亡的病人具有重要臨床意義，目前大部分預測心臟驟
停或死亡的研究採用單一資料類型與單一機器學習模型。然而，我們認為
不同種類的患者資料都包含有助於預測的重要信息，而多模態學習架構的使
用能以互補的方式使用數據來學習複雜的任務。因此，我們提出多種組合多
種數據流的多模態機器學習包括資料層級融合、特徵層級融合和決策層級融
合來預測加護病房內病人的院內死亡以及院內心臟驟停。在研究的過程中，
我們使用與時間無關的(靜態) 數據(例如年齡、性別、種族)、時間相關的(動
態) 數據(例如心律、呼吸速率和血氧濃度) 以及成像數據(胸部X光圖像)。我
們使用logistic regression (LR)、random forest (RF)、eXtreme Gradient Boosting
tree (XgBoost)、support vector machine (SVM) 和K-nearest neighbor (KNN) 來
處理靜態數據、使用long short-term memory (LSTM) 來處理動態數據以及使
用convolutional neural network (CNN) 來處理胸部X光圖像。在我們的研究中，
我們使用MIMIC-IV的公開資料對模型進行訓練以及內部驗證，之後進一步使
用公開的eICU資料來進行外部驗證。根據實驗結果，不同類型數據的預測組
合有助於最終的預測。對於一小時前院內死亡率的預測area under the receiver
operating characteristic (AUROC) 達到0.94、敏感度達到0.88，對於一小時前院
內心臟驟停的預測AUROC達到0.91、敏感度達到0.88。與僅使用靜態數據獲得
的結果相比，預測死亡率的AUROC提升了0.09、敏感度提升了0.12，預測心臟
驟停的AUROC提升了0.03、敏感度提升了0.09。因此，我們認為這種多模態架
構可以幫助臨床醫生更準確地識別高死亡率或即將心臟驟停的患者並降低醫療
保健成本。

In recent years, electronic health record (EHR) databases have grown rapidly and become more comprehensive. As a result, many studies have used EHRs to assist clinical decision-making. Machine learning (ML) has been widely used to analyze EHRs and various applications have been proposed to help clinicians accurately identify ill patients to improve their overall diagnosis, prognosis, and treatment decision. Identifying patients imminent for cardiac arrest (CA) or mortality in the intensive care unit (ICU) is clinically important. Most of the current studies predicting cardiac arrest or mortality used a single data type with a single machine learning model. However, we suggested that various data of patients contain important patient information, which can help with the prediction. Multimodal learning architectures can utilize various data in complementary ways to learn complex tasks. Therefore, we proposed several multi-modal ML architectures including data-level fusion, feature-level fusion and decision-level fusion that combines multiple data streams to predict in-hospital mortality and in-hospital CA in ICU patients. In this study, we used time independent (static) data (e.g., age, gender, and ethnicity), time dependent (dynamic) data (e.g., heart rate, respiratory rate, and peripheral oxygen saturation, and imaging data (chest X-ray (CXR) images). We used logistic regression (LR), random forest (RF), eXtreme Gradient Boosting tree (XgBoost), support vector machine (SVM), and K nearest neighbor (KNN) for static data, long short-term memory (LSTM) for dynamic data, and convolutional neural network (CNN) for imaging data. In our study, we used public data from the MIMIC-IV for model training and internal validation. We further used public data from the eICU for external validation. According to the experimental results, the combination of predictions from different types of data could improve the final prediction. For the prediction of in-hospital mortality before an hour, the area under the receiver operating characteristic (AUROC) reached 0.94 and the sensitivity reached 0.88. For the prediction of in-hospital CA before an hour, the AUROC reached 0.91 and the sensitivity reached 0.88. When compared with the results obtained by using only static data, the AUROC for predicting mortality increased by 0.09 and the sensitivity increased by 0.12 while the AUROC for predicting CA increased by 0.03 and the sensitivity increased by 0.09. Therefore, we conclude that the proposed multi-modal approach can assist clinicians to accurately identify patients with high probability of mortality or CA and reduce health care costs.

[1] Benjamin Shickel, Patrick James Tighe, Azra Bihorac, and Parisa Rashidi.
Deep ehr: a survey of recent advances in deep learning techniques for electronic
health record (ehr) analysis. IEEE journal of biomedical and health
informatics, 22(5):1589–1604, 2017.
[2] Cao Xiao, Edward Choi, and Jimeng Sun. Opportunities and challenges in
developing deep learning models using electronic health records data: a systematic
review. Journal of the American Medical Informatics Association,
25(10):1419–1428, 2018.
[3] Mathias J Holmberg, Catherine E Ross, Garrett M Fitzmaurice, Paul S Chan,
Jordan Duval-Arnould, Anne V Grossestreuer, Tuyen Yankama, Michael W
Donnino, Lars W Andersen, and American Heart Association’s Get With
The Guidelines-Resuscitation Investigators*. Annual incidence of adult and
pediatric in-hospital cardiac arrest in the united states. Circulation: Cardiovascular
Quality and Outcomes, 12(7):e005580, 2019.
[4] Lars W Andersen, Mathias J Holmberg, Katherine M Berg, Michael W Donnino,
and Asger Granfeldt. In-hospital cardiac arrest: a review. Jama,
321(12):1200–1210, 2019.
[5] Martin Sp˚angfors, Mats Molt, and Karin Samuelson. In-hospital cardiac arrest
and preceding national early warning score (news): A retrospective case-control study. Clinical Medicine, 20(1):55, 2020.
[6] Gary B Smith, David R Prytherch, Paul E Schmidt, and Peter I Featherstone.
Review and performance evaluation of aggregate weighted ‘track and
trigger’systems. Resuscitation, 77(2):170–179, 2008.
[7] Dhanesh Ramachandram and Graham W Taylor. Deep multimodal learning:
A survey on recent advances and trends. IEEE signal processing magazine,
34(6):96–108, 2017.
[8] Matthew M Churpek, Trevor C Yuen, Seo Young Park, David O Meltzer,
Jesse B Hall, and Dana P Edelson. Derivation of a cardiac arrest prediction
model using ward vital signs. Critical care medicine, 40(7):2102, 2012.
[9] Joon-myoung Kwon, Youngnam Lee, Yeha Lee, Seungwoo Lee, and Jinsik
Park. An algorithm based on deep learning for predicting in-hospital cardiac
arrest. Journal of the American Heart Association, 7(13):e008678, 2018.
[10] Bo Jin, Chao Che, Zhen Liu, Shulong Zhang, Xiaomeng Yin, and Xiaopeng
Wei. Predicting the risk of heart failure with ehr sequential data modeling.
Ieee Access, 6:9256–9261, 2018.
[11] Hans-Christian Thorsen-Meyer, Annelaura B Nielsen, Anna P Nielsen, Benjamin
Skov Kaas-Hansen, Palle Toft, Jens Schierbeck, Thomas Strøm, Piotr J
Chmura, Marc Heimann, Lars Dybdahl, et al. Dynamic and explainable machine
learning prediction of mortality in patients in the intensive care unit: a
retrospective study of high-frequency data in electronic patient records. The
Lancet Digital Health, 2(4):e179–e191, 2020.
[12] Jianhong Cheng, John Sollee, Celina Hsieh, Hailin Yue, Nicholas Vandal,
Justin Shanahan, Ji Whae Choi, Thi My Linh Tran, Kasey Halsey, Franklin Iheanacho, et al. Covid-19 mortality prediction in the intensive care unit with
deep learning based on longitudinal chest x-rays and clinical data. European
radiology, pages 1–11, 2022.
[13] Luis R Soenksen, Yu Ma, Cynthia Zeng, Leonard Boussioux, Kimberly Villalobos
Carballo, Liangyuan Na, Holly M Wiberg, Michael L Li, Ignacio Fuentes,
and Dimitris Bertsimas. Integrated multimodal artificial intelligence framework
for healthcare applications. NPJ digital medicine, 5(1):1–10, 2022.
[14] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen
Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-
Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet:
components of a new research resource for complex physiologic signals. circulation,
101(23):e215–e220, 2000.
[15] Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P
Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J
Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available
database of labeled chest radiographs. arXiv preprint arXiv:1901.07042, 2019.
[16] Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G
Mark, and Omar Badawi. The eicu collaborative research database, a freely
available multi-center database for critical care research. Scientific data,
5(1):1–13, 2018.
[17] Joshua J Gagne, Robert J Glynn, Jerry Avorn, Raisa Levin, and Sebastian
Schneeweiss. A combined comorbidity score predicted mortality in elderly patients
better than existing scores. Journal of clinical epidemiology, 64(7):749–
759, 2011. [18] Robert Greif, Andrew S Lockey, Patricia Conaghan, Anne Lippert, Wiebe
De Vries, Koenraad G Monsieurs, John HW Ballance, Alessandro Barelli,
Dominique Biarent, Leo Bossaert, et al. European resuscitation council guidelines
for resuscitation 2015: section 10. education and implementation of resuscitation.
Resuscitation, 95:288–301, 2015.
[19] Thanakron Na Pattalung and Sitthichok Chaichulee. Comparison of machine
learning algorithms for mortality prediction in intensive care patients
on multi-center critical care databases. In IOP Conference Series: Materials
Science and Engineering, volume 1163, page 012027. IOP Publishing, 2021.
[20] James Bergstra and Yoshua Bengio. Random search for hyper-parameter
optimization. Journal of machine learning research, 13(2), 2012.
[21] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system.
In Proceedings of the 22nd acm sigkdd international conference on knowledge
discovery and data mining, pages 785–794, 2016.
[22] Sepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural
computation, 9(8):1735–1780, 1997.
[23] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip
Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal
of artificial intelligence research, 16:321–357, 2002.
[24] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation
for deep learning. Journal of big data, 6(1):1–48, 2019.
[25] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model
predictions. Advances in neural information processing systems, 30, 2017. [26] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna
Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations
from deep networks via gradient-based localization. In Proceedings of the
IEEE international conference on computer vision, pages 618–626, 2017.
[27] ArjenMDondorp, Sue J Lee, Md Abul Faiz, Saroj Mishra, Ric Price, Emiliana
Tjitra, Marlar Than, Ye Htut, Sanjib Mohanty, Emran Bin Yunus, et al. The
relationship between age and the manifestations of and mortality associated
with severe malaria. Clinical Infectious Diseases, 47(2):151–157, 2008.
[28] Henry E Wang, Benjamin S Abella, Clifton W Callaway, American Heart Association
National Registry of Cardiopulmonary Resuscitation Investigators,
et al. Risk of cardiopulmonary arrest after acute respiratory compromise in
hospitalized patients. Resuscitation, 79(2):234–240, 2008.
[29] Massimo Imazio and Gaetano Maria De Ferrari. Cardiac tamponade: an educational
review. European heart journal. Acute cardiovascular care, 10(1):102–
109, 2021.
[30] Gestur Thorgeirsson, Gudmundur Thorgeirsson, Helgi Sigvaldason, and
Jacqueline Witteman. Risk factors for out-of-hospital cardiac arrest: the
reykjavik study. European heart journal, 26(15):1499–1505, 2005.
[31] Gary B Smith. Vital signs: Vital for surviving in-hospital cardiac arrest?
Resuscitation, 98:A3–A4, 2016.
[32] Matthew M Churpek, Trevor C Yuen, Michael T Huber, Seo Young Park,
Jesse B Hall, and Dana P Edelson. Predicting cardiac arrest on the wards: a
nested case-control study. Chest, 141(5):1170–1176, 2012.
[33] Spencer S Liu, Alejandro Gonz´alez Della Valle, Melanie C Besculides, Licia K
Gaber, and Stavros G Memtsoudis. Trends in mortality, complications, and demographics for primary hip arthroplasty in the united states. International
orthopaedics, 33(3):643–651, 2009.
[34] Shih-Cheng Huang, Anuj Pareek, Saeed Seyyedi, Imon Banerjee, and
Matthew P Lungren. Fusion of medical imaging and electronic health records
using deep learning: a systematic review and implementation guidelines. NPJ
digital medicine, 3(1):1–9, 2020.
71