MIMIC ML/DL

Learning objectives
• Make us of MIMIC-IV for mortality prediction on patients
diagnosed with sepsis.
• Make use of scikit-learn built-in binary classification models for
mortality prediction.
• Build, train, and evaluate a neural network (PyTorch) for
mortality prediction.
Set up and pre-requisites
• First thing we need to do is
make sure latest google-colab
is installed and import
required Python modules.
• Main libraries required are
PyTorch, scikit-learn,
NumPy, pandas, and
matplotlib.
Set up and pre-requisites
• Jupyter notebook assumes
MIMIC-IV dataset
(https://physionet.org/content/mi
miciv/2.2/) has been previously
downloaded from PhysioNet,
unzipped, and uploaded to
Google Drive.
• Thus, we need to mount Google
Colab to Google Drive.
• String constants for multiple
column names found in MIMIC-
IV are also defined.
Data processing and features
• Helper method shown on the
right is used to read MIMIC-IV
csv data files into a pandas
data frame.
• MIMIC-IV files are split into
hosp and icu directories.
Thus, we need to check if
target csv file exists in either
directory.
Data processing and features
• Helper method (first half)
shown on the right is used to
read, filter, merge, and return
raw MIMIC-IV data (as a
pandas data frame) for
hospital admissions
diagnosed with sepsis.
• MIMIC-IV tables merged
include dictionary of ICD-9
and ICD-10 codes, diagnoses
table, admissions table,
patients table, and ICU stays
table.
Data processing and features
• Second half of helper method to get
raw sepsis data is shown on the right.
• Usually complete blood count (CBC)
tests are used to track the progression
of sepsis. We would like to include
these in our data as these could be
helpful features in mortality prediction.
To be more specific, results for
hematocrit, platelet count, and
hemoglobin.
• One issue is MIMIC-IV lab events
table is large. Even with Google Colab
Pro high-ram, trying to load all lab
events data we run out of memory.
The workaround is to load and filter
lab events data in chunks.
Data processing and features
• Below shows execution of previous helper method to load MIMIC-IV raw sepsis data
• This will take several minutes due to the fact lab events table is very large
Data processing and features
• Helper method (first half) shown
on the right is used to
processed previously fetched
raw sepsis data and create
features out of it.
• At a high level:
• Age at admission is calculated
• Race and first care unit column
values are simplified
• CBC lab event results become
separate columns
• Categorical columns become one-
hot encoded
Data processing and features
• Second half of helper method
to process raw sepsis data
and create features is shown
on the right.
• Only hospital admission rows
with lab results for all CBC
tests are included.
• Furthermore, duplicate rows
and rows with missing values
are dropped.
Data processing and features
• Below shows execution of previous helper method to create features from raw sepsis data
Train, validation, and test split
• Helper method on the right is
used to rebalance and split data
into train, validation, and test
splits.
• Rebalancing is done by
randomly down-sampling data
such that target values, i.e.,
hospital_expire_flag, are
equally distributed for binary
classification.
• The split is 70% train, 15%
validation, and 15% test.
Scikit-learn and random baseline binary
classifiers
• We now define a random binary
classifier (shown on right) as a
baseline.
• The end goal is to eventually
train a neural network that
performs better than random
guessing.
• This random binary classifier
learns the probability distribution
of classes from the training
data. And during inference, it
randomly samples from such
learned distribution.
Scikit-learn and random baseline binary
classifiers
• But first, shown on the right is
a simple helper method to
print results.
• We’ll store result metrics for
multiple models inside a
dictionary.
• Some of the metrics we’ll
explore further (and print)
include AUC (ROC) score, F1
score, precision score, recall
score, and accuracy.
Scikit-learn and random baseline binary
classifiers
• Method on the right evaluates
following binary classifiers (most
from scikit-learn):
• Logistic regression
• Linear support vector machines
• Decision tree classifier
• Random forest classifier
• Gaussian naïve Bayes
• K-neighbors classifier
• Random (baseline) classifier
• Each model is trained on training
data and evaluated on test dataset.
Result metrics are stored in
dictionary and printed to console.
Scikit-learn and random baseline binary
classifiers
• Below shows execution of evaluating scikit-learn binary classifiers and random (baseline) classifier
• Result metrics are printed to console, but these are visualized and discussed on in slides that follow
Scikit-learn and random baseline binary
classifiers
• Method on the right visualizes
result metrics (stored in
dictionary).
• For each evaluation metric a
horizontal bar plot is created.
• Horizontal bar for neural
network (in future slides) is
colored red for easy
comparison.
• In the slides that follow are
these plots.
Scikit-learn and random baseline binary
classifiers
Scikit-learn and random baseline binary
classifiers
Scikit-learn and random baseline binary
classifiers
• Key insights:
• Random forest classifier
performs best at mortality
prediction overall. With an
accuracy of ~0.644.
• Random (baseline) classifier
performs the worst overall (as
expected), with ~0.5155
accuracy and ~0.5164 AUC
(ROC).
Train and evaluate PyTorch neural
network
• Now, we will build, train, and evaluate a
very simple PyTorch neural network.
• First, we need to define a device. This is
helpful if running on a GPU. However,
GPU is not required.
• On the right is implementation of a very
simple neural network. It takes as input:
• input_size – The number of features in the
training data.
• hidden_layer_size – The number of
neurons in the hidden layer.
• The architecture of this neural network is
very simple: a single hidden layer followed
by a single neuron and a sigmoid layer.
Between these we make use of dropout
(20%) for regularization and a leaky ReLu
function (for non-linearity).
Train and evaluate PyTorch neural
network
• Before proceeding, we need
to define a custom dataset
wrapper object, shown on the
right.
• Data, features and labels, are
stored as PyTorch tensors
(float) to device.
Train and evaluate PyTorch neural
network
• On the right we define helper
methods to train and evaluate
neural network a single epoch.
• At a high-level, train loop makes
a model prediction, calculates
the loss, and performs one step
of back propagation. A running
loss over the training data is
computed and returned.
• The evaluation loop returns the
loss of model prediction over all
validation dataset.
Train and evaluate PyTorch neural
network
• Putting all previous helper methods
together, the train method on the
right performs the end-2-end
training of the neural network.
• Train method takes as input the
model to train, a training data
loader, a validation data loader,
and multiple hyperparameters such
as number of epochs, the loss
function, and an optimizer.
• We’ll train for 1000 epochs using a
batch size of 64.
• Adam optimizer and binary cross
entropy loss are used for training.
Train and evaluate PyTorch neural
network
• Running end-2-end training of
neural network takes a minute
or two. Train and validation
losses are printed for each
epoch (screenshot on right)
and returned at the end.
Train and evaluate PyTorch neural
network
• Plotting train and validation
losses after end-2-end
training of neural network
some observations:
• Train loss decreases
throughout epochs.
• However, validation loss
plateaus around ~0.66.
• We need to be very careful
with underfitting on the
training data and overfitting
on the validation data.
Train and evaluate PyTorch neural
network
• Helper methods on the right
are used to evaluate neural
network on test dataset. This
includes computing multiple
metrics such as AUC (ROC),
F1, precision, recall, and
accuracy.
• The best threshold is also
computed based on ROC
curve.
Train and evaluate PyTorch neural
network
• Evaluating neural network
metrics are printed to console
and stored in existing results
dictionary.
• In next slides we plot and
compare all models, including
this trained neural network.
Compare all models
Compare all models
Compare all models
• Key insights:
• Trained neural network
performs better than random
guessing.
• However, random forest
classifier still performs better at
mortality prediction compared
to neural network with regards
to accuracy.
• Neural network outperforms all
other models on recall but
underperforms on precision.
Compare all models
• In last few slides we will plot
ROC curves for all models.
• Code (shown on the right)
comes from
https://jovian.com/vipul0036vi
pul/how-to-find-optimal-
threshold-for-binary-
classification-roc-curve
Compare all models Key insights:
• ROC for random classifier is as expected
Compare all models
Compare all models Key insights:
• Gaussian naïve Bayes, which performed the
worst, has similar a ROC as random
guessing
Compare all models Key insights:
• ROC for neural network and random forest
classifier are similar. But there is still a lot of
room for improvement.
Learnings and future work
• Some things learned:
• Mortality prediction on MIMIC-IV (framed as binary classification) is
difficult.
• We see a very simple neural network does perform better than a
random classifier. However, there is still a lot of room for improvement.
• Future work / enhancements include:
• Making use of data augmentation to train better performing model.
• Play with different model architectures and training hyperparameters.
Though we need to be careful not to overfit on the training data.
• Make use of additional features for the training data. For example,
more lab event results, prescribed medications, etc.