Uncertainty-Aware Model for Reliable Prediction of Sepsis in the ICU

Marco Pimentel, Adam Mahdi, Oliver Redfern, Mauro Santos, Lionel Tarassenko
University of Oxford


Abstract

Background and Objectives. Predicting the onset of sepsis from routinely collected clinical data is challenging, as physiological and laboratory measurements are sampled at different frequencies and missing data are not randomly distributed. We propose a two-model approach, where the first model predicts a probability of sepsis and the second estimates the uncertainty of these predictions. We then optimise a decision rule, which considers both the probability and model’s uncertainty to make our final predictions.

Methods. Demographics, vital signs and laboratory values from 40,336 ICU patient records from two separate hospital systems were used in a Gradient Boosting Machine (GBM) classification model to predict sepsis, and a second GBM regression model to estimate the uncertainty of those predictions using the same input features. Hyper-parameters of both models and the derived decision rule were optimized by 5-fold cross-validated grid-search within a training subset of the data (20,336 records from set A) using the area under the receiver operating characteristics curve (AUC) for scoring. The best candidate model was evaluated on the held-out test set (20,000 records from set B), using AUC and the provided Utility function which rewards early prediction of sepsis and penalises late predictions.

Results. Our uncertainty-aware approach achieved an AUC of 0.769 and Utility of 0.225 evaluated on the held-out test set. The average results on the training set obtained by 5-fold cross-validation were: AUC of 0.827±0.008 and Utility of 0.427±0.011. The Utility of our model is substantially higher than the baseline model supplied with the current Challenge (Utility of 0.0956), when evaluated on the held-out test set.

Conclusion. We have developed a novel prediction approach which considers uncertain estimates when predicting the onset of sepsis. Our model improves on the Utility of the baseline model supplied with the Challenge.