# Estimating uncertainty of earthquake rupture using Bayesian neural network

Sabber Ahamed
<EMAIL_ADDRESS>

###### Abstract

Bayesian neural networks (BNN) are the probabilistic model that combines the
strengths of both neural network (NN) and stochastic processes. As a result,
BNN can combat overfitting and perform well in applications where data is
limited. Earthquake rupture study is such a problem where data is
insufficient, and scientists have to rely on many trial and error numerical or
physical models. Lack of resources and computational expenses, often, it
becomes hard to determine the reasons behind the earthquake rupture. In this
work, a BNN has been used (1) to combat the small data problem and (2) to find
out the parameter combinations responsible for earthquake rupture and (3) to
estimate the uncertainty associated with earthquake rupture. Two thousand
rupture simulations are used to train and test the model. A simple 2D rupture
geometry is considered where the fault has a Gaussian geometric heterogeneity
at the center, and eight parameters vary in each simulation. The test F1-score
of BNN (0.8334), which is 2.34$\%$ higher than plain NN score. Results show
that the parameters of rupture propagation have higher uncertainty than the
rupture arrest. Normal stresses play a vital role in determining rupture
propagation and are also the highest source of uncertainty, followed by the
dynamic friction coefficient. Shear stress has a moderate role, whereas the
geometric features such as the width and height of the fault are least
significant and uncertain.

_K_ eywords Earthquake $\cdot$ Bayesian Neural network $\cdot$ Rupture
simulation $\cdot$ Variational inference

##  1 Introduction

Hazards due to earthquakes are a threat to economic losses and human life
worldwide. The type of hazards depends on the strength of the earthquake,
local topography, built structures, and subsurface geology. A large earthquake
produces high-intensity ground motion that causes damage to structures like
buildings, bridges, and dams and takes many lives. Seismic hazard analysis
(SHA) is the process to estimate the risk associated with these damages. SHA
requires historical earthquakes information and detailed geological data, but
unfortunately, we lack detailed surface and subsurface geologic information.
Consequently, this leads to uncertainty in hazard estimation.

Numerical or physical models, on the other hand, can be used to generate
synthetic data that supplement existing data. However, these models based
studies like dynamic earthquake rupture simulations need initial information
of the fault and the surrounding region. Such initial data can include the
stress-state, frictional, and material properties of the fault. However, the
parameters are also not well constrained and not always available (Duan and
Oglesby, 2006; Peyrat et al., 2001; Ripperger et al., 2008; Kame et al.,
2003). Since earthquake rupture is a highly nonlinear process, determining the
right parameter combinations is essential for the right simulation. Different
initial conditions may lead to different results; therefore, we may not
capture the real scenario of an actual earthquake rupture. The parameter
combinations are often determined by making simplifying assumptions or taking
a trial and error approach, which is computationally expensive (Douilly et
al., 2015; Ripperger et al., 2008; Peyrat et al., 2001). Therefore, high
computational costs limit the applicability of the simulations to integrate
into seismic hazard analysis.

In recent years, machine learning (ML) approaches have been successfully used
to solve many geophysical problems that have limited data and involve
computational expense. For example, Ahamed and Daub (2019) developed a neural
network and random forest algorithms to predict if an earthquake can break
through a fault with geometric heterogeneity. The authors used only 1600
rupture simulations data points for developing a neural network and random
forest models. The authors were able to extract different patterns of the
parameter responsible for earthquake rupture. The authors find that the models
can predict rupture status within a fraction of a second, which is a
significant improvement where a single simulation takes about two hours of
wall-clock time on eight processors.

ML are also used in seismic event detection (Rouet-Leduc et al., 2017),
earthquake detection (Perol et al., 2018), identifying faults from unprocessed
raw seismic data (Last et al., 2016) and to predict broadband earthquake
ground motions from 3D physics-based numerical simulations (Paolucci et al.,
2018). All the examples show that the potential of ML to solve many unsolved
earthquake problems.

The generalization of a machine learning model usually depends on the quality
and the amount of data. A bad quality or small amount of data may increase the
uncertainty associated with each prediction (Hoeting et al., 1999; Blei et
al., 2017; Gal et al., 2017). Prediction uncertainty estimation is vital in
many applications: diseases detection (Leibig et al., 2017; Liu et al., 2018;
Nair et al., 2019), autonomous vehicle driving (Kendall et al., 2015;
McAllister et al., 2017; Burton et al., 2017), and estimating risk (Hoeting et
al., 1999; Tong and Koller, 2001; Uusitalo, 2007). Therefore, calculating
uncertainty is also equally crucial as improving the model accuracy. All of
the ML-based earthquake studies mentioned above avoid prediction uncertainty
estimation.

To overcome the problem of insufficient data of earthquake rupture, I extend
the work of Ahamed and Daub (2019) using the Bayesian neural network. Unless
plain neural network, BNN works better with a small amount of data and provide
prediction uncertainty. In this paper, I describe a workflow of (1) developing
BNN, (2) estimating prediction uncertainty, and (3) finding parameter
combinations responsible for rupture. Identifying the source of uncertainty is
vital to understand the physics of earthquake rupture and estimate seismic
risk. I also describe a technique that combines BNN and permutation importance
to find the source of uncertainty.

##  2 Rupture simulations and data processing

Figure 1: (a) The schematic diagram shows a zoomed view of the two-dimensional
fault geometry. The domain is 32 km long along the strike of the fault and 24
kilometers wide across the fault. The rupture starts to nucleate 10 km to the
left of the barrier and propagates from the hypocenter towards the barrier,
(b) Linear slip-weakening friction law for an earthquake fault. The fault
begins to slip when the shear stress reaches or exceeds the peak strength of
$\tau_{s}$. $\tau_{s}$ decreases linearly with slip to a constant dynamic
friction $\tau_{d}$ over critical slip distance ($d_{c}$). The shear strength
is linearly proportional to the normal stress $\sigma_{n}$, and the friction
coefficient varies with slip between $\mu_{s}$ and $\mu_{d}$.

Two thousand of earthquake rupture simulations were used in this work created
by Ahamed and Daub (2019). The simulations two-dimensional rupture illustrated
in figure. 1. The domain is 32 km long and 24 km wide. Figure 1a shows the
zoomed view of the original domain for better visualization of the fault
barrier. Simulations were created using fdfault (Daub, 2017), which is a
finite difference code for numerical simulation of elastodynamic fracture and
friction problems. The code solves the elastodynamic wave equation coupled to
a friction law describing the failure process. In each simulation, fault slip
is calculated based on the initial stress conditions, the elastodynamic wave
equations, and frictional failure on that fault. The fault has a Gaussian
geometric heterogeneity at the center. Rupture is nucleated 10 km to the left
of the barrier and propagates towards the barrier. The linear slip-weakening
law determines the strength of the fault (figure. 1b). The fault starts to
break when the shear stress ($\tau$) exceeds the peak strength
$\tau_{s}=\mu_{s}\sigma_{n}$, where $\mu_{s}$ and $\sigma_{n}$ are the static
friction coefficient and normal stress, respectively. Over a critical slip
distance $d_{c}$, the friction coefficient reduces linearly to constant
dynamic friction $\mu_{d}$. In each simulations, eight parameters are varied:
x and y components of normal stresses (sxx and syy), shear stress (sxy),
dynamic friction coefficient, friction drop ($\mu_{s}-\mu_{d}$), critical slip
distance ($d_{c}$), and width and height of the fault.

One thousand six hundred simulations are used to train, and the remaining 400
are used to test the model. The training dataset has an imbalance class
proportion of ruptures arrest (65$\%$) and ruptures propagation (35$\%$). To
avoid the biases toward rupture arrest, I upsampled the rupture propagation
examples. Before training, all the data were normalized by subtracting mean
and dividing by standard deviation.

##  3 Neural network (NN)

Neural networks are computing systems inspired by how neurons are connected in
the brain (Rosenblatt, 1958). Several nodes are interconnected and organized
in layers in a neural network. Each node is also known as a neuron. A layer
can be connected to an arbitrary number of hidden layers of arbitrary size.
However, increasing the number of hidden layers not always improve the
performance but may force the model to generalize well on the training data
but unseen (test) data, which is also known as overfitting (Hinton et al.,
2012; Lawrence and Giles, 2000; Lawrence et al., 1997). As a result, selecting
the number of layers and nodes in each layer is one of the challenges of using
neural networks in practice.

##  4 Bayesian Neural network (BNN)

Figure 2: The schematic diagram shows the architecture of the Bayesian neural
network used in this work. The network has one input layer with eight
parameters, one hidden layer with twelve nodes, and an output layer with a
single node. Weights between input and hidden layers are defined by
$w^{0}_{ij}$, which are normally distributed. $i,j$ are the node input and
hidden layer node index. Similarly, $w^{1}_{jk}$ is the normal distribution of
weights between the hidden and the output layer. $\mu$ and $\sigma$ are the
mean and standard deviation, respectively. At the output node, the network
produces a distribution of prediction scores between 0 and 1.

In a traditional neural network, weights are assigned as a single value or
point estimate, whereas in BNN, weights are considered as a probability
distribution. These probability distributions of network weights are used to
estimate the uncertainty in weights and predictions. Figure 2 shows a
schematic diagram of a BNN where weights are normally distributed. The
posterior of the weights are calculated using Bayes theorem as:

$P(W|\textbf{X})=\frac{P(\textbf{X}|W)P(W)}{P(\textbf{X})}$ (1)

Where, X is the data, $P(\textbf{X}|W)$ is the likelihood of observing X,
given weights ($W$), $P(W)$ is the prior belief of the weights, and the
denominator $P(\textbf{X})$ is the probability of data which is also known as
evidence. It requires integrating over all possible values of the weights as:

$P(\textbf{X})=\int P(\textbf{X}|W)P(W)dW.$ (2)

Integrating all over the indefinite weights in evidence makes it hard to find
a closed-form analytical solution. As a result, simulation or numerical based
alternative approaches such as Monte Carlo Markov chain (MCMC), variational
inference(VI) are considered. MCMC sampling has been the vital inference
method in modern Bayesian statistics. Scientists widely studied and applied in
many applications. However, the technique is slow for large datasets or
complex models. Variational inference (VI), on the other hand, is faster than
other methods. It has also been applied to solve many large scale
computationally expensive neuroscience and computer vision problems (Blei et
al., 2017).

In VI, a new distribution $Q(W|\theta)$ is considered that approximates the
true posterior $P(W|\textbf{X})$. $Q(W|\theta)$ is parameterized by $\theta$
over $W$ and VI finds the right set of $\theta$ that minimizes the divergence
of two distributions through optimization:

$Q^{*}(W)=\operatorname*{argmin}_{\theta}\textbf{KL}\left[Q(W|\theta)||P(W|\textbf{X})\right]$
(3)

In the above equation-3, KL or Kullback–Leibler divergence is a non-symmetric
and information theoretic measure of similarity (relative entropy) between
true and approximated distributions (Kullback, 1997). The KL-divergence
between $Q(W|\theta)$ and $P(W|\textbf{X})$ is defined as:

$\textbf{KL}\left[Q(W|\theta)||P(W|\textbf{X})\right]=\int
Q(W|\theta)\log\frac{Q(W|\theta)}{P(W|\textbf{X})}dW$ (4)

Replacing the $P(W|\textbf{X})$ using equation-1 we get:

$\displaystyle\textbf{KL}\left[Q(W|\theta)||P(W|\textbf{X})\right]$
$\displaystyle=\int
Q(W|\theta)\log\frac{Q(W|\theta)P(\textbf{X})}{P(\textbf{X}|W)P(W)}dW$ (5)
$\displaystyle=\int Q(W|\theta)\left[\log Q(W|\theta)P(\textbf{X})-\log
P(\textbf{X}|W)P(W)\right]dW$ (6) $\displaystyle=\int
Q(W|\theta)\log\frac{Q(W|\theta)}{P(W)}dW+\int Q(W|\theta)\log
P(\textbf{X})dW-\int Q(W|\theta)\log P(\textbf{X}|W)dW$ (7)

Taking the expectation with respect to $Q(W|\theta)$, we get:

$\displaystyle\textbf{KL}\left[Q(W|\theta)||P(W|\textbf{X})\right]$
$\displaystyle=\operatorname{\mathbb{E}}\left[\log\frac{Q(W|\theta)}{P(W)}\right]+\log
P(\textbf{X})-\operatorname{\mathbb{E}}\left[\log P(\textbf{X}|W)\right]$ (8)

The above equation still shows the dependency of $\log P(\textbf{X})$ that
makes difficult KL to compute. An alternative objective function is therefore,
derived by adding $\log P(\textbf{X})$ with negative KL divergence. $\log
P(\textbf{X})$ is a constant with respect to $Q(W|\theta)$. The new function
is called as the evidence of lower bound (ELBO) and expressed as:

$\displaystyle ELBO(Q)$ $\displaystyle=\operatorname{\mathbb{E}}\left[\log
P(\textbf{X}|W)\right]-\operatorname{\mathbb{E}}\left[\log\frac{Q(W|\theta)}{P(W)}\right]$
(9) $\displaystyle=\operatorname{\mathbb{E}}\left[\log
P(\textbf{X}|W)\right]-KL\left[Q(W|\theta)||P(W|\textbf{X})\right]$ (10)

The first term is called likelihood, and the second term is the negative KL
divergence between a variational distribution and prior weight distribution.
Therefore, ELBO balances between the likelihood and the prior. The ELBO
objective function can be optimized to minimize the KL divergence using
different optimizing algorithms like gradient descent.

##  5 Train the BNN

The BNN has the same NN architecture used in Ahamed and Daub (2019) to compare
the performance between them. If BNN performs better than that of NN or has
similar performance, BNN provides an additional advantage of prediction
uncertainty. Like NN, BNN has one input layer with eight parameters, one
hidden layer with twelve nodes and one output layer (Figure 2). A nonlinear
activation function ReLu (Hahnloser et al., 2000) is used at the hidden layer.
ReLu passes all the values greater than zero and sets any negative output to
be zero. Output layer uses sigmoid activation function, which converts the
outputs zero and one.

Prior weights and biases are normally distributed with zero mean and one
standard deviation. Figure 3 shows the log density of prior and posterior
weights ($w^{k}_{ij}$) and biases ($b^{k}_{j}$). $i$ and $j$ are the index of
the input and hidden layer nodes. $i$ ranges from 0 to 7, and $j$ ranges from
0 to 11. $k$ is the index that maps two layers. As an example, $w^{0}_{15}$ is
the weight between the first input node and the fifth hidden node. The output
node of the last layer produces a distribution of prediction scores between 0
and 1. The prediction distributions are used to compute standard deviation,
which is the uncertainty metric.

Adam optimization (extension of stochastic gradient descent) is used to
minimize the KL divergence by finding a suitable variational parameter
$\theta$. The initial learning rate is 0.5, which exponentially decays as the
training progresses.

To train the network I use Edward (Tran et al., 2016, 2017), TensoFlow (Abadi
et al., 2015) and Scikit-learn (Pedregosa et al., 2011). Edward is a Python-
based Bayesian deep learning library for probabilistic modeling, inference,
and criticism. All the training data, codes, and the corresponding
visualizations can be found on the project Github repository:
https://github.com/msahamed/earthquake_physics_bayesian_nn

###  5.1 Prior and posterioir weight distribution

Figure 3: The graph shows the distribution of prior and posterior mean weights
(a) $w^{0}$ (b) $w^{1}$ and biases (c) $b^{0}$ (d) $b^{1}$. Both location of
the mean and magnitude of density of the posterior distributions (weights and
biases) are noticeably different from the priors which indicates that BNN has
learned from the data and adjusted the posterior accordingly.

To evaluate the model, 1000 posterior samples of $w^{0}_{ij}$, $w^{1}_{jk}$,
$b^{0}$ and $b^{1}$ are used. Figure 3 shows the prior and posterior
distribution of mean weight and biases. Posterior location of mean and
magnitude of the density of the weights and biases are different from their
priors. For example, the location of $w^{0}$ shifts toward non-negative value,
while the density remains similar. Whereas, the $w^{1}$, $b^{0}$, and $b^{1}$
have a different posterior mean location and density than their prior. The
differences between prior and posterior indicate that the BNN has learned from
the data and adjusted the posterior distribution accordingly.

###  5.2 BNN classification result

The performance of the BNN is evaluated using 400 test simulations. Since 1000
posterior samples are used, the BNN is able to produce 100 prediction scores
for a given example. The scores are then used to determine the prediction
class and associated uncertainty (standard deviation). To determine the proper
class of the examples, the first mean prediction score is calculated from the
1000 prediction scores for each example. Then an optimal threshold is computed
that maximizes the model F-1 score (0.54). If the mean prediction score of an
example is greater or equal to the optimal threshold, then the earthquake is
classified as propagated, and otherwise, arrested. Uncertainty is the standard
deviation of the prediction scores. F-1 score is the harmonic mean of the true
positive rate and precision of the model. Table 1 shows the confusion matrix
that contains information about actual and predicted classifications. The test
accuracy of the BNN is $83.34\%$, which is $2.34\%$ higher than NN. BNN also
reduces the four false positives (FP) and three false negatives (FN). Table 2
shows the detail classification report of the model performance. The results
imply that BNN has the potential to improve performance. Since BNN produces
distributions of score rather than a point estimation, BNN can better
generalize the unseen data and thus help reduce overfitting.

Table 1: Confusion matrix of 400 test data | Actual propagated | Actual arrested
---|---|---
Predicted propagated | TN = 226 | FN = 46
Predicted arrested | FP = 22 | TP = 106
Table 2: Classification resultsof 400 test data Class | Precision | Recall | F1 score | support
---|---|---|---|---
Rupture arrested | 0.91 | 0.83 | 0.87 | 272
Rupture propagated | 0.70 | 0.83 | 0.76 | 128
Average/Total | 0.84 | 0.83 | 0.83 | 400

##  6 Uncertainity analysis

Uncertainty analysis helps make a better decision and to estimate risk
accurately. In the following subsection, I discuss different types of
uncertainties (network uncertainty, prediction uncertainty, and feature
uncertainty) estimated from BNN and their implication on the underlying causes
for an earthquake rupture. I also discuss how uncertainty can help us
understand physics and find the parameter combinations responsible for an
earthquake rupture.

###  6.1 Network uncertainity

Figure 4: The illustration shows the posterior mean and standard deviations of
$w^{0}$ and $w^{1}$. (a) $w^{0}$ that map the inputs to the nodes of the
hidden layer. The eight input parameters and the twelve nodes are on the
horizontal and vertical scale, respectively. The colors in each cell are the
magnitudes of mean weight. (b) $w^{1}$ maps the hidden layer to the output
layer. (c) The standard deviation of $w^{0}$. Shear stress connected to the
node-4 of the hidden layer has the highest uncertainty. Similarly, the weights
associated with the input parameters and the node-5 have high uncertainty as
well. Whereas, the weights associated with the input parameters and the nodes
7-11 have relatively low uncertainty. (d) Uncertainty of the weights
associated with the hidden layer nodes and the output node. Weights in Node-8
and 7 have high uncertainty while the rest of the weights have relatively low
uncertainty.

Estimating the uncertainty of neural network parameters (weights) helps us
understand the behavior of the black box. The illustration in Figure 4 shows
the mean and standard deviation of $W^{0}$ and $W^{1}$. $W^{0}$ maps the
inputs to the hidden layer nodes, whereas $W^{1}$ maps the output node of the
output layer to the nodes of the hidden layer. The colors in each cell
indicate the magnitude of a weight that connects two nodes. In the input and
hidden layer, the ReLu activation function is used. ReLu passes all the
positive output while setting negative output as zero. At the output layer,
sigmoid activation is used, which pushes the larger weights toward one and
smaller or negative weights toward zero. Therefore, Positive and high
magnitude of weights contributes toward the earthquake rupture and vice versa.

An interesting observation is that when nodes in the hidden layer connects to
friction drop, dynamic friction, shear, and normal stresses have variable
positive and negative weights in $w^{0}$, the corresponding node in $w^{1}$
have either strongly positive or negative. For example, node-12 has both
positive and negative weights in $w^{0}$, and the corresponding node in
$w^{1}$ has a high positive weight. Similarly, node-4 has a substantial
negative weight in $w^{1}$, and the corresponding nodes in $w^{0}$ have both
positive and negative weights. On the other hand, width, height, and $d_{c}$
have a similar magnitude of weights, and the corresponding nodes in $w^{1}$
have a moderate magnitude of weights. Thus the variable weights make friction
drop, dynamic friction, shear, and normal stresses influential on the
prediction score. Therefore, for any combined patterns of the input features,
we now can detect the important features and the source uncertainties.

For example, node-10 of $w^{1}$ has positive weight (Figure 4b). In $w^{0}$,
the corresponding connecting input features, friction drop, and shear stress
have positive weight and low uncertainty, whereas the rest of the features
have a similar magnitude of weight. The combination of high friction drop and
shear stress weight and low rest of the feature weight increase the prediction
score, thus likely to cause rupture to propagate. Friction drop and normal
stress also have high uncertainty (Figure 4c). For this combination of
patterns, friction drops and normal stress influences the prediction strongly
and are also the sources of uncertainty. Thus it gives us the ability to
investigate any rupture propagation example in terms of uncertainty.

###  6.2 Prediction uncertainity

Figure 5: The graph shows (a) frequency and (b) standard deviation of
posterior prediction scores of the test data. Prediction scores are skewed
toward the left side while slightly less on the right. The observation is
consistent with the proportion of the rupture arrest (272) and rupture
propagation (120) in the test data. Prediction scores close to zero are
related to rupture arrest, and scores around one are the rupture propagation.
Standard deviations are high with scores around 0.5.

Figure 5a and b show the frequency and standard deviation of test data
prediction scores. The distribution is more skewed toward smaller scores than
the higher ones. Scores close to zero are associated with rupture arrest,
whereas the scores close to one are rupture propagations. The observation is
consistent with the class proportion of the test data. Rupture arrest has a
higher number of examples (272) than rupture propagation (120). Figure 5a
shows that scores roughly between 0.35 and 0.75 have a fewer number of
examples and have high uncertainty. The likely reason for the high uncertainty
is that the examples have both rupture propagation and arrest properties. As a
result, the model gets confused to classify the example correctly; thus, the
misclassification rate is high in this region. The performance of the network
could be improved if sufficient similar example data are added to the training
dataset.

###  6.3 Feature uncertainity

Figure 6: The illustration shows (a) permutation feature importyance and (b)
their uncertanitities.

I use the permutation importance method to determine the source of uncertainty
of the test data. Permutation importance is a model agnostic method that
measures the influencing capacity of a feature by shuffling it and measuring
corresponding global performance deviation. If a feature is a good predictor,
then altering its values will result in a significant reduction in a model’s
global performance. The shuffled feature with the highest performance
deviation is the most important and vice versa. In this work, F-1 score the
performance measuring metric.

Figure 6(a) shows the permutation importance of all the features. Normal
stresses (sxx and syy) have the highest F-1 score deviation, which is
approximate $~{}7\%$ less than the base performance, followed by the dynamic
friction coefficient. Geometric feature width has the least contribution role
to determine the earthquake rupture. These observations are consistent with
the observation of Ahamed and Daub (2019), where the authors rank the features
based on the random forest feature importance algorithm.

Since BNN produces a distribution of prediction scores, it can compute the
standard deviation (uncertainty) for each shuffled feature. Figure 6(b) shows
the uncertainty of each feature of earthquake rupture propagates and arrest
class. All features have higher uncertainty in the rupture propagation class
compare to the rupture arrest. In both classes, the major portion of
uncertainty comes from normal stresses. Shear stress, height, width, and
critical distance ($d_{c}$) have a similar amount of uncertainty in both of
the classes. The dynamic friction coefficient and friction drop are also the
comparable uncertainty source in rupture to propagate while it is slightly
less in earthquake arrest.

The above observations imply that normal stress and friction parameters have a
greater role in determining the earthquake rupture the local fault surface.
Although the height and width of a fault are not a significant source of
uncertainty, they play a stronger role in influencing the other features. For
example, in a complex rough fault, the variation of the bending angle of
barriers affects stress perturbation, consequently increases the
uncertainties. If the angle near the bend is sharp, the variation in traction
at the releasing and restraining bend is more prominent. Whereas if the
barrier is broad, the stress perturbation at the restraining and releasing
bend is less noticeable. Chester and Chester (2000) find a similar observation
that fault geometry impacts the orientation and magnitude of principal stress.

##  7 Discussion and conclusion

In recent years, deep learning has been used to solve many real-life problems
and achieves state of the art performance. For example, face recognition,
language translation, self-driving cars, disease identification, and many
more. Many of such successful application requires millions of data points to
train the model to achieve a state of the art performance. However, many
applications are limited to data. Therefore, it has been a barrier to use deep
learning to solve many other real-life problems like earthquake rupture. A
Bayesian neural network, on the other hand, can perform well and achieve
excellent performance even in the small dataset.

In the wok, I use a Bayesian neural network to combat the small data problem
and to estimate the uncertainty in earthquake rupture. Two thousand rupture
simulations generated by Ahamed and Daub (2019) are used to train and test the
model. Each 2D simulation has a fault with a Gaussian geometric heterogeneity
at the center. Eight fault parameters normal shear stress, height, and width
of the fault, stress drop, dynamic friction coefficient, critical distance
vary in each simulation. 1600 simulations are used to train the BNN, and 400
of them are used to test the generalization of the model. The BNN has the same
architecture as of NN of Ahamed and Daub (2019).

The test F1 score is 0.8334, which is 2.34$\%$ higher than NN. All the
features have a higher uncertainty in rupture propagation than the rupture
arrest. The highest sources of uncertainty come from normal stresses, followed
by the dynamic friction coefficient. Therefore, these features have a higher
influencing capacity in determining the prediction score. Shear stress has a
moderate role, but the geometric features such as the width and height of the
fault are least significant in determining rupture. Test examples with
prediction scores around 0.5 have a higher uncertainty than those with low
(0-0.30) and high (0.7-1.0) prediction scores. Cases with prediction scores
around 0.5 have mixed properties of rupture propagation and arrest.

## References

  * Duan and Oglesby [2006] Benchun Duan and David D. Oglesby.  Heterogeneous fault stresses from previous earthquakes and the effect on dynamics of parallel strike-slip faults.  _Journal of Geophysical Research: Solid Earth_ , 111(5), 2006.  ISSN 21699356.  doi: 10.1029/2005JB004138.
  * Peyrat et al. [2001] Sophie Peyrat, Kim Olsen, and Raúl Madariaga.  Dynamic modeling of the 1992 Landers earthquake.  _Journal of Geophysical Research: Solid Earth (1978–2012)_ , 106(B11):26467–26482, 2001.
  * Ripperger et al. [2008] J Ripperger, PM Mai, and J-P Ampuero.  Variability of near-field ground motion from dynamic earthquake rupture simulations.  _Bulletin of the seismological society of America_ , 98(3):1207–1228, 2008.
  * Kame et al. [2003] Nobuki Kame, James R Rice, and Renata Dmowska.  Effects of prestress state and rupture velocity on dynamic fault branching.  _Journal of Geophysical Research: Solid Earth (1978–2012)_ , 108(B5), 2003.
  * Douilly et al. [2015] R Douilly, H Aochi, E Calais, and AM Freed.  3D dynamic rupture simulations across interacting faults: The Mw 7.0, 2010, Haiti earthquake.  _Journal of Geophysical Research: Solid Earth_ , 2015.
  * Ahamed and Daub [2019] Sabber Ahamed and Eric G Daub.  Machine learning approach to earthquake rupture dynamics.  _arXiv preprint arXiv:1906.06250_ , 2019.
  * Rouet-Leduc et al. [2017] Bertrand Rouet-Leduc, Claudia Hulbert, Nicholas Lubbers, Kipton Barros, Colin J. Humphreys, and Paul A. Johnson.  Machine learning predicts laboratory earthquakes.  _Geophysical Research Letters_ , 2017.  ISSN 1944-8007.  doi: 10.1002/2017GL074677.  URL http://dx.doi.org/10.1002/2017GL074677.  2017GL074677.
  * Perol et al. [2018] Thibaut Perol, Michaël Gharbi, and Marine Denolle.  Convolutional neural network for earthquake detection and location.  _Science Advances_ , 4(2):e1700578, 2018.
  * Last et al. [2016] Mark Last, Nitzan Rabinowitz, and Gideon Leonard.  Predicting the maximum earthquake magnitude from seismic data in Israel and its neighboring countries.  _PloS one_ , 11(1):e0146101, 2016.
  * Paolucci et al. [2018] Roberto Paolucci, Filippo Gatti, Maria Infantino, Chiara Smerzini, Ali Güney Özcebe, and Marco Stupazzini.  Broadband ground motions from 3D physics-based numerical simulations using artificial neural networks.  _Bulletin of the Seismological Society of America_ , 108(3A):1272–1286, 2018.
  * Hoeting et al. [1999] Jennifer A Hoeting, David Madigan, Adrian E Raftery, and Chris T Volinsky.  Bayesian model averaging: a tutorial.  _Statistical science_ , pages 382–401, 1999.
  * Blei et al. [2017] David M Blei, Alp Kucukelbir, and Jon D McAuliffe.  Variational inference: A review for statisticians.  _Journal of the American Statistical Association_ , 112(518):859–877, 2017.
  * Gal et al. [2017] Yarin Gal, Riashat Islam, and Zoubin Ghahramani.  Deep bayesian active learning with image data.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 1183–1192. JMLR. org, 2017.
  * Leibig et al. [2017] Christian Leibig, Vaneeda Allken, Murat Seçkin Ayhan, Philipp Berens, and Siegfried Wahl.  Leveraging uncertainty information from deep neural networks for disease detection.  _Scientific reports_ , 7(1):17816, 2017.
  * Liu et al. [2018] Fang Liu, Zhaoye Zhou, Alexey Samsonov, Donna Blankenbaker, Will Larison, Andrew Kanarek, Kevin Lian, Shivkumar Kambhampati, and Richard Kijowski.  Deep learning approach for evaluating knee mr images: achieving high diagnostic performance for cartilage lesion detection.  _Radiology_ , 289(1):160–169, 2018.
  * Nair et al. [2019] Tanya Nair, Doina Precup, Douglas L Arnold, and Tal Arbel.  Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation.  _Medical Image Analysis_ , page 101557, 2019.
  * Kendall et al. [2015] Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla.  Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding.  _arXiv preprint arXiv:1511.02680_ , 2015.
  * McAllister et al. [2017] Rowan McAllister, Yarin Gal, Alex Kendall, Mark Van Der Wilk, Amar Shah, Roberto Cipolla, and Adrian Vivian Weller.  Concrete problems for autonomous vehicle safety: Advantages of bayesian deep learning.  International Joint Conferences on Artificial Intelligence, Inc., 2017\.
  * Burton et al. [2017] Simon Burton, Lydia Gauerhof, and Christian Heinzemann.  Making the case for safety of machine learning in highly automated driving.  In _International Conference on Computer Safety, Reliability, and Security_ , pages 5–16. Springer, 2017.
  * Tong and Koller [2001] Simon Tong and Daphne Koller.  Active learning for parameter estimation in bayesian networks.  In _Advances in neural information processing systems_ , pages 647–653, 2001.
  * Uusitalo [2007] Laura Uusitalo.  Advantages and challenges of bayesian networks in environmental modelling.  _Ecological modelling_ , 203(3-4):312–318, 2007\.
  * Daub [2017] Eric G Daub.  Finite difference code for earthquake faulting.  https://github.com/egdaub/fdfault/commit/b74a11b71a790e4457818827a94b4b8d3aee7662, 2017\.
  * Rosenblatt [1958] Frank Rosenblatt.  The perceptron: A probabilistic model for information storage and organization in the brain.  _Psychological review_ , 65(6):386, 1958.
  * Hinton et al. [2012] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov.  Improving neural networks by preventing co-adaptation of feature detectors.  _arXiv preprint arXiv:1207.0580_ , 2012.
  * Lawrence and Giles [2000] Steve Lawrence and C Lee Giles.  Overfitting and neural networks: conjugate gradient and backpropagation.  In _Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on_ , volume 1, pages 114–119. IEEE, 2000.
  * Lawrence et al. [1997] Steve Lawrence, C Lee Giles, and Ah Chung Tsoi.  Lessons in neural network training: Overfitting may be harder than expected.  In _AAAI/IAAI_ , pages 540–545, 1997.
  * Kullback [1997] Solomon Kullback.  _Information theory and statistics_.  Courier Corporation, 1997.
  * Hahnloser et al. [2000] Richard HR Hahnloser, Rahul Sarpeshkar, Misha A Mahowald, Rodney J Douglas, and H Sebastian Seung.  Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit.  _Nature_ , 405(6789):947, 2000.
  * Tran et al. [2016] Dustin Tran, Alp Kucukelbir, Adji B. Dieng, Maja Rudolph, Dawen Liang, and David M. Blei.  Edward: A library for probabilistic modeling, inference, and criticism.  _arXiv preprint arXiv:1610.09787_ , 2016.
  * Tran et al. [2017] Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M. Blei.  Deep probabilistic programming.  In _International Conference on Learning Representations_ , 2017.
  * Abadi et al. [2015] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  URL https://www.tensorflow.org/.  Software available from tensorflow.org.
  * Pedregosa et al. [2011] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al.  Scikit-learn: Machine learning in python.  _Journal of machine learning research_ , 12(Oct):2825–2830, 2011.
  * Chester and Chester [2000] Frederick M Chester and Judith S Chester.  Stress and deformation along wavy frictional faults.  _Journal of Geophysical Research: Solid Earth_ , 105(B10):23421–23430, 2000.


# Do Large Language Models Understand Medical Codes

Simon A. Lee
Department of Computational Medicine
University of California, Los Angeles
Los Angeles, California, USA 90095
<EMAIL_ADDRESS>
&Timothy Lindsey
Bioinformatic IDP
University of California, Los Angeles
Los Angeles, California, USA 90095
<EMAIL_ADDRESS>

###### Abstract

The overarching goal of recent AI research has been to make steady progress
towards achieving Artificial General Intelligence (AGI), prompting the
evaluation of Large Language Models (LLMs) across a variety of tasks and
domains. One such domain is healthcare, where LLMs can greatly benefit
clinical practice by assisting with a wide range of tasks. However, these
models are also prone to producing “hallucinations” or incorrect responses
when faced with queries they cannot adequately address, raising concerns and
skepticism, especially within the healthcare community. In this work, we
investigate whether LLMs understand and can predict medical codes, which are
extensively utilized in healthcare practice. This study aims to delineate the
capabilities and limitations of these LLMs. We evaluate various off-the-shelf
LLMs (e.g., GPT, LLaMA, etc.) and LLMs specifically designed for biomedical
applications to assess their awareness and understanding of these domain-
specific terminologies. Our results indicate that these models as they
currently stand do not comprehend the meaning of the medical codes,
highlighting the need for better representation of these alphanumeric codes
extensively used in healthcare. We call for improved strategies to effectively
capture and represent the nuances of medical codes and terminologies within
LLMs, enabling them to become more reliable and trustworthy tools for
healthcare professionals.

##  1 Introduction

The integration of Large Language Models (LLMs) into the healthcare sector
research marks a transformative shift in how medical data is analyzed,
interpreted, and leveraged to improve patient care. LLMs, with their
unparalleled ability to understand and generate human-like text, are
increasingly being applied to a wide range of healthcare applications, from
parsing and summarizing Electronic Health Records (EHRs) to providing decision
support for clinical classification tasks.

However within the complex ecosystem of healthcare, there are many unaddressed
questions limiting its use in clinical settings (Wornow et al., 2023).
Integral to this system are various standardized codes (Cimino, 1996) commonly
used in practice, organized into various structured classification groups,
such as the International Classification of Diseases (ICD) codes, procedure
codes (PCT), medication codes, and laboratory codes, each serving a unique and
critical role. ICD codes classify and record diseases and health conditions,
enabling healthcare professionals and insurers to track epidemiological trends
and validate medical billing. Procedure codes document the specific medical
interventions undertaken, from surgical procedures to diagnostic tests,
facilitating the standardization and efficiency of patient care. Medication
codes ensure precise identification of prescribed drugs, enhancing medication
management and safety by reducing errors. Laboratory codes catalog diagnostic
and clinical tests, crucial for interpreting patient health status and guiding
treatment decisions. Together, these coding systems enhance the
interoperability of health information, streamline the billing process,
support clinical research, and ultimately contribute to the delivery of high-
quality, evidence-based healthcare (Alexander et al., 2003).

Figure 1: Each Digit in a Medical Code corresponds to an encoded structural
meaning

Interestingly, the organized, rule-based metrics of medical coding systems
enable each subsequent digit to refine the search for the correct label
corresponding to a specific code as demonstarted in Figure 1. However, due to
the architectural designs of LLMs (e.g. poor numerical representation in
tokenizers) and the lack of publicly available clinical text—we hypothesize
that LLMs may not fully comprehend these alphanumeric medical codes to
identify them with their corresponding medical names. This potential gap in
understanding could significantly impact research groups focused on
summarizations, automated billing and inference-based tasks, highlighting the
need for better representations of these standard approaches in healthcare.
Furthermore, we consider this line of research critical as it identifies and
leverages gaps within these advanced models in domain-specific tasks. We hope
that this type of work will inspire groups to build upon our findings to
enhance future versions of LLMs. Previous research has extensively explored
the identification and matching of EHR and clinical texts with their
corresponding medical codes. Contrary to this approach, our interest is in
exploring the reverse trajectory: identifying disease names from medical
codes.

Therefore in this work, we investigate whether LLMs are capable of predicting
medical codes demonstrating their understanding. This issue is significant in
both machine learning and healthcare, as these LLMs are known to produce
“hallucinations” that are challenging to distinguish from actual ground truth.
Consequently, this research raises critical questions regarding the type of
EHR representations we should employ when integrating these datasets into LLM
research. Addressing this issue is daunting because, without proper
verification, “hallucinations” may lead to incorrect insights, resulting in
detrimental financial consequences, misinformation, and inaccurate outcomes in
healthcare. While research on numerical representations has been explored in
other contexts (Golkar et al., 2023), to our knowledge, this is the first
study to assess the understanding of LLMs on medical codes.

##  2 Related Work

###  2.1 Predicting medical codes from EHR

Electronic Health Records (EHR) research has made significant strides in
utilizing computational techniques to predict and match ICD codes (Lita et
al., 2008), (Parr et al., 2018) which are essential for various healthcare
processes including billing, epidemiology, and research. The evolution of this
field has seen a more recent shift to natural language processing (NLP)
techniques to bridge the gap between unstructured patient text and structured
International Classification of Diseases (ICD) codes. Early work, as
demonstrated by (Singaravelan et al., 2021) (Atutxa et al., 2017), (Jay,
2019), (Mullenbach et al., 2018), (Uzuner et al., 2010), (Spasić et al.,
2010), and (Kavuluru et al., 2015), employed rule-based algorithms and basic
NLP methods to automate the coding process, aiming to enhance accuracy and
efficiency in healthcare documentation.

The advent of deep learning and encoder-based Large Language Models (LLMs)
like BERT has further transformed the approach to predicting ICD codes,
marking a significant leap in the field’s capabilities. Studies by (Samonte et
al., 2018), (Chen & Ren, 2019), (Huang et al., 2022), and (Amin et al., 2019),
highlight the shift towards using sophisticated models that leverage pre-
trained language representations. These advancements have not only improved
the precision of medical code prediction but also deepened the understanding
of medical language’s complexities, paving the way for more automated,
accurate, and efficient healthcare management systems.

###  2.2 Predicting Y Codes from X Codes

Another popular avenue of research within the realm of EHR involves leveraging
existing medical codes to predict other relevant medical codes, effectively
utilizing one type of code as a predictor for another. This method has seen
various implementations, such as using diagnostic codes to anticipate
procedure codes, a relationship explored in works by (Subotin & Davis, 2014)
and (Haq et al., 2017). Additionally, some research has ventured into
combining multiple code types to enhance the prediction of disease codes, as
demonstrated in the study by (Djennaoui et al., 2015).

Figure 2: An Overview of the Proposed Experiments to Evaluate the LLMs on
Whether or Not They Can Properly Predict (Understand) the Corresponding
Medical Codes.

###  2.3 LLMs and Healthcare

Large Language Models (LLMs) have increasingly been recognized for their
transformative potential in healthcare, as evidenced by recent research
(Karabacak & Margetis, 2023). One of the key utilities of LLMs lies in their
capability to efficiently summarize clinical texts, a feature that has been
highlighted in studies by (Van Veen et al., 2023) and (Van Veen et al., 2024).
Another area where LLMs have shown considerable promise is in providing
decision support for clinical classification tasks. Research conducted by (Lee
et al., 2024), (Chen et al., 2024) (Belyaeva et al., 2023) demonstrates how
LLMs can aid in the categorization of medical information, thereby
facilitating more accurate predictions related to the clinic. Named-entity
recognition (NER) represents yet another domain where LLMs have made
significant inroads. The advent of BERT and its derivatives tailored for
medical contexts—such as MedBERT ((Liu et al., 2021)), ClinicalBERT ((Huang et
al., 2019), (Mulyar et al., 2021)), and BioBERT ((Lee et al., 2020), (Shin et
al., 2020))—has led to a surge in models capable of identifying specific terms
and concepts within clinical texts. These models have been instrumental in
parsing and understanding complex medical documents, enhancing data
extraction, and processing tasks. And more recently, the development of more
generative multimodal models like MedPalm (Tu et al., 2024), Clinical Camel
(Toma et al., 2023), and Meditron (Chen et al., 2023), showcases the ongoing
innovation within the field, offering new avenues for utilizing LLMs in
healthcare. These models extend the utility of LLMs beyond text analysis,
incorporating various data types and modalities to generate comprehensive
insights and predictions. Together, these advancements illustrate the growing
role of LLMs in revolutionizing healthcare, from simplifying documentation and
predictinc clinical outcomes to potential directions facilitating personalized
and informed patient care.

##  3 Methods

In this work we are interested in figuring out whether Large Language Models
are able to predict canonical healthcare naming practices commonly used in
medical codes (ICD, PCT, etc.). Our main motivation behind this is to try and
reduce “hallucinations” often seen by these models in a critical field like
healthcare. We also find this research question to be particularly interesting
in the representation learning aspect on whether coded language (alpha-numeric
codes) can be learned from these models. Additionally we are motivated to
exploit any gaps found within LLMs so they are not used on tasks that they are
not optimized on.

In our analysis, we utilize both commercial and open-source Large Language
Models (LLMs) available on the Hugging Face platform to conduct our
evaluation. We provide an overview of our framework in Figure 2. Additionally,
we restrict the ability of recent models like GPT-4 to access the internet for
this assessment. Furthermore, we aim to investigate whether LLMs pretrained on
biological, medical, or clinical data demonstrate superior ability to learn
and predict medical codes, even with potentially fewer parameters. The
specific LLMs employed in our study are detailed in Table 1.

Table 1: Overview of various language models highlighting their source and parameter size. Model | Parameters | Source
---|---|---
GPT-4 | N/A | Closed Source
GPT-3.5 | N/A | Closed Source
Claude 3 | N/A | Closed Source
Gemini | N/A | Closed Source
LLaMA 2 | 70B | Open Source
Mistral 7B | 7B | Open Source
Meditron | 70B | Open Source
Medllama2 | 7B | Open Source

###  3.1 Experimental Design

We begin our experiment by asking these LLMs if they can identify the
categories or “chapters” corresponding to specific medical codes (e.g., ICD9:
001-139 corresponds to ‘Infectious and Parasitic Diseases’, ICD10 Procedure: 7
corresponds to Osteopathic Procedures, etc.). We then subject these models to
a more granular set of tasks for further evaluation.

In terms of experimental design for evaluating these models for its
granularity, we devised three distinct experiments of varying degrees of
difficulty. This approach helps us determine whether LLMs are able to predict
and understand the meaning behind medical codes. We incorporated various
levels of difficulty because, in practice, any ICD code could be entered
through a user query. Therefore, our aim was to assess and simulate potential
real-life scenarios.

  1. 1.

A List of Ordered Medical Codes: Our first experiment engaged in predicting
the medical conditions associated with specific ICD codes, all falling within
the same ”chapter” or category. This experiment evaluates the LLMs’ ability
not only to accurately recognize the medical codes but also to understand the
underlying numerical structure of the codes. It delves into how each
subsequent digit in an medical code conveys a specific meaning and follows a
deliberate design.

  2. 2.

A Randomly Ordered List of Medical Codes: Our second experiment simulates a
more realistic scenario by testing the LLMs’ understanding of medical codes
through a random sampling of indices to which we bootstrap multiple tests. We
then ask each LLM to identify the conditions corresponding to each medical
code. This scenario mirrors real-life situations where patients’ EHR often
contain a long list of current and previous diagnoses.

  3. 3.

Adversarial Attack: Lastly, we include an example designed to potentially
exploit weaknesses in an LLM. LLMs inherently attempt to answer all queries
without question, so we incorporate this example, which intentionally includes
malicious or fake medical codes among a sample of real ones, to determine
whether an LLM “hallucinates” or not.

We further illustrate these experiments by presenting an example query for
each experiment in Table 2:

Table 2: An example query for each experimental design is provided. Our interest lies in detecting ”hallucinations,” so we conduct several experiments that vary in difficulty from Experiment 1 to 3. Red text indicates the malicious examples that we intend to insert in our data for the adversarial attacks experiment. We repeat these experiments for all medical codes: ICD (diagnostic), CPT (procedure), medication, and laboratory codes. Experiment |  Queries
---|---
1. Ordered List |  Predict the names of the diagnostics from this list of ICD10 Codes: [A12.01, A12.02, A12.03, A12.04, A12.08…]
2. Randomly Ordered List |  Predict the names of the diagnostics from this list of ICD10 Codes: [E570.1, G77.03, H81.06, A11.04, D43.01…]
3. Adversarial Attack |  Predict the names of the diagnostics from this list of ICD10 Codes: [A12.01, A12.02, A12.-03, A12.04, A12.0Z…]

###  3.2 Data

Since medical codes are standardized across medical institutions, we utilize
the MIMIC (Medical Information Mart for Intensive Care) clinical database to
extract medical codes for the International Classification of Diseases (ICD
9/10), Current Procedural Terminology (CPT), Medication (often referred to by
the National Drug Code, NDC), and Laboratory values (coded in LOINC).

###  3.3 Analysis

In conducting our experiments, we perform all data preprocessing and sampling
of 1000 medical codes using the Python programming language. We set seeds for
creating partitions of data for each experiment, ensuring consistent data
inputs across all benchmarks. We then employ Language Models (LMs) sourced
from HuggingFace, and use *ollama* (LLaMA2 70b, Medllama2, Mistral, and
Meditron) to load these onto our local computers. Additionally, we utilize API
keys from OpenAI (GPT-3.5/4), Anthropic (Claude-3), and Google (Gemini) to
generate outputs from these models. However, we urge users to acknowledge the
following footnote111We do highlight that the outputs of the LLMs may slightly
vary due to an LLM’s innate behavior. When generating responses to user
queries, LLMs do not reproduce identical outputs across different
conversational sessions. This phenomenon can be attributed to the stochastic
decoding methods employed by LLMs, which introduce randomness into the text
generation process. Furthermore, periodic model updates and the intentional
introduction of parameter randomization during inference contribute to the
observed diversity in LLM outputs for identical inputs across different
instances., which details potential issues with obtaining exactly reproducible
results.

###  3.4 Evaluation

In terms of evaluating these methods, we have identified three distinct
benchmarking frameworks through which we will assess the models.

I. Predicting Medical Chapters In this evaluation, the accuracy is determined
by how correctly the model predicts the name of the chapter given its code.

II. Predicting Medical Codes  To evaluate the accuracy of Language Models in
predicting the names of diseases, labs, or procedures from their codes, we
began by comparing the text predictions generated by the models against the
true labels. This process involved assessing “hallucinations” by comparing the
words in the prediction to those in the ”ground truth” to determine if the
LLMs were generating unfounded information. The assessment aimed to ascertain
the presence of ”hallucinations” within the model’s predictions, indicated by
the degree of word matches. It involved calculating the number of matching
words after processing the data to remove stopwords, digits, symbols, and
converting to lowercase, and then dividing by the total number of words in the
label, as modeled by Equation 1. This methodology helps evaluate whether the
prediction aligns with the actual medical label corresponding to its code and
its accuracy.

$\textbf{Proportion of Matched Words}\\_i=\frac{\text{Number of Matched Words
in Row }i}{\text{Total Words in the Label of Row }i}$ (1)

Furthermore, we monitored the number of rows in our dataset where word matches
occurred, identifying a subset of predictions suitable for further filtering
to isolate only relevant predictions or ”non-hallucination” predictions. This
subset was then subjected to a manual review to assess the accuracy of the
LLMs’ predictions more thoroughly. Through this comprehensive evaluation, we
aimed to understand better the model’s capability to accurately predict on
medical codes.

III. Adversarial Attack  In the Adversarial Attack experiment, we assess
whether the LLM correctly ignores malicious examples or erroneously generates
a label for the fake medical codes. This task employs a binary metric to
determine if the LLM correctly dismisses the malicious examples or produces a
“hallucination.” An independent analysis is also conducted on the same set of
codes, where the task is to identify which $N$ number of codes are fake within
a pool of medical codes, and the accuracy of this identification is measured.

##  4 Results

Table 3: The Accuracy of the LLMs on Predicting the correct medical chapters. The * symbol denotes that LLMs would correctly predict the label but for the wrong corresponding code signaling that it has this information within its parameters but is not learned well enough. The $\dagger$ symbol denotes that LLM did not answer prompt as expected even with multiple different prompting efforts. Model | ICD9/10 Chapters | Procedure/Lab Chapters
---|---|---
Claude 3 | 1.0000 | 0.0588
Gemini | 0.8235* | 0.0000*
GPT 3.5 | 1.0000 | 0.9412
GPT 4 | 1.0000 | 1.0000
LLaMA 2 | 0.3539* | 0.1765*
Meditron | ———$\dagger$ | ———$\dagger$
MedLLaMA2 | 1.0000 | 0.1176
Mistral | 1.0000 | 0.0588
Figure 3: This figure presents a comprehensive evaluation of various Large
Language Models (LLMs) in predicting International Classification of Diseases
(ICD) codes, under two distinct scenarios: random order and fixed order. The
assessment focuses on three key metrics: accuracy of the overall predictions,
the percentage of words within predictions that match with the true labels,
and the percentage of rows where word matches occur between the predictions
and the true labels. The top row, showcases the performance with ICD codes
presented in a fixed order, reflecting the models’ efficiency in a more
structured scenario. The bottom row of graphs illustrates the performance in a
random ordering context, highlighting the models’ ability to predict
accurately without a predetermined sequence. This side-by-side comparison
offers insightful perspectives on each model’s ability in these two
experimental settings.

In this section, we present our findings from the three evaluation studies. We
start with Table 3, showcasing the predictions of the medical chapters. This
table reveals that while these models perform well on ICD chapter codes
(predicting disease chapters), their performance significantly drops on other
types of medical chapter codes. Nonetheless, it’s important to note that the
models possess the relevant knowledge within their systems. This is evidenced
by some models, indicated with an asterisk (*), which were able to present the
correct labels, albeit with the incorrect codes. We also highlight Meditron’s
($\dagger$) refusal to answer even with multiple efforts with different
prompting strategies.

In the second study depicted in Figure 3, it investigates the capability of
Language Models (LLMs) to accurately identify medical codes. The analysis
comprises two distinct experiments: the first presents medical codes in a
sequentially ordered manner (row 1), while the second displays these codes in
a random sequence (row 2). The findings from these plots predominantly show a
tendency for LLMs to “hallucinate,” although GPT-4 exhibited a relatively
lower rate of “hallucination” compared to other models. However, it is crucial
to note that even GPT-4 was not immune to producing hallucinated predictions,
underscoring the need for continued improvement in the field.

Lastly, we discuss Table 4, which details our adversarial attack experiment.
In this experiment, we introduced malicious examples alongside real labels to
determine if Language Models (LLMs) could alert the user about an incorrectly
entered ICD code. The table shows that LLMs were unable to perform this
function effectively. In the second part of this experiment, we presented a
mix of real and malicious codes and tasked the LLMs with identifying all
instances of fake ICD codes. The results showed that while the LLMs could name
$N$ codes as fake, the accuracy of these identifications seemed more
attributable to random chance than to an actual understanding of the codes in
question.

Table 4: This table presents the Accuracy of Detecting the malicious (fake) medical codes mixed in with the real ones. We first pose a binary metric where the model either makes a prediction for the adversarial example or not. In Addition, we have the model select the $N$ number of fake codes hidden among a mixture of codes. We see all models perform poorly on this with all models “hallucinating” to some extent. Model | Responses Generated from Fake Codes | Classifying Fake Codes
---|---|---
Claude 3 | 0.0000 | 0.2727
Gemini | 0.0000 | 0.2727
GPT 3.5 | 0.0000 | 0.4545
GPT 4 | 0.3636 | 0.4545
LLaMA 2 | 0.0000 | 0.1765*
Meditron | 0.0000 | 0.0000
MedLLaMA2 | 0.0000 | 0.2727
Mistral | 0.0000 | 0.2727

##  5 Discussion

###  5.1 Study Insights

We start by asserting that Large Language Models (LLMs) are far from reaching
the expertise of medical professionals. Therefore, studies involving medical
codes within these LLMs (e.g. summarization, automated billing) should proceed
with caution. This claim is based on our work, which underscores the urgent
need for better representation of medical codes and domain-specific
terminology within modern NLP techniques. The results demonstrate that while
some LLMs exhibit a degree of understanding, there is still a significant
amount of work and progress needed in this area. Surprisingly, our findings
also reveal that models pre-trained on medical tasks performed far worse than
the state-of-the-art general LLMs. For instance, Meditron, regardless of the
variety of prompts used, consistently failed to accurately respond to user
queries. We attribute this behavior to potential larger model parameters
stemming from propietary LLMs like GPT4, Claude3 and GPT3.5 rumoured to have
$\geq$ 70 billion parameters.

Another intriguing discovery was that many models were capable of recognizing
the limitations of their outputs and advised users to be cautious of their
results. While such acknowledgments were helpful and appreciated, they
ultimately highlight a fundamental shortcoming of these all-purpose LLMs,
undermining their reliability for specific, high-stakes applications like
medical diagnostics. These findings also indicate these LLMs are not near AGI
Capabilities.

###  5.2 Potential Pitfall: Lack of Clinical Text

The scarcity of clinical text for pre-training LLMs potentially led to non-
optimal performance found widely in this evaluation. Clinical text, rich in
nuanced medical terminology and patient information, is pivotal for developing
LLMs that can understand and interpret and learn medical codes. However,
stringent proprietary regulations surrounding patient data limit the
availability of such text for public usage. This limitation hampers the
ability of researchers to train models effectively on the intricacies of
standard medical protocols (e.g. medical coding), thereby affecting the
models’ accuracy and reliability in clinical applications. As a result,
innovating ways to securely leverage clinical notes from physicians with these
language model technologies is crucial for advancing LLMs in understanding
healthcare domain and practices.

###  5.3 Potential Pitfall: Tokenization

Another hypothesis surrounding an LLMs inability to correctly predict an ICD
code could be due to the tokenization designs that are not optimal for
numerical or code like language. LLMs often struggle with accurately encoding
and processing numerical values, codes, and other non-natural language
elements as demonstrated by their poor analytical abilities. This limitation
can be attributed to the fundamental design principles and training objectives
of LLMs. Most LLMs are trained on vast corpora of natural language text, which
primarily consists of words, sentences, and paragraphs. Consequently, the
tokenization strategies employed by these models, such as subword tokenizers
like BytePair Encoding (BPE) or WordPiece, are optimized for handling natural
language input efficiently.

However, these tokenization techniques may not be well-suited for representing
numerical values, codes, or other structured data formats. Subword tokenizers
operate by splitting words into smaller subword units based on their frequency
in the training corpus. This approach works well for natural language text but
can lead to suboptimal tokenization of numerical values or codes, which often
have a unique structure and distribution compared to natural language. As a
result, LLMs may struggle to accurately encode and process these types of
inputs, leading to potential errors or inconsistencies in their outputs. We
therefore believe the design of these models are not well suited for learning
coded language like these medical codes.

###  5.4 LLMs Can Use the Internet

One interesting aspect of LLMs has been their recent development of being able
to access the internet during inference. However, in this work, we did not
allow the LLMs in the benchmark to utilize the internet, as we wanted to
assess these models in a more traditional machine learning paradigm. However,
with the continued development of each model, we would like to highlight that
future iterations of the models could perhaps improve on these results.

##  6 Conclusion

In this work, we ran an evaluation of whether current LLMs both utilized for
general and biomedical applications were able to properly predict several
medical codes regularly used in healthcare practice. Our findings indicate
that as it currently stands, these LLMs are prone to hallucinating on medical
codes, which calls for a need for better representations for these codes that
are regularly used in practice in the clinic. And while many current works
involve pushing the SOTA of these models, it is just as critical to address
the gaps of these models for further improvements of all fields.

#### Limitations

One small limitation of this work is that we did not conduct our analysis on
all the data for all the tasks. However this was due partly by design as we
noticed notable results in early experimentation. We therefore hypothesize
that conducting the analysis on the whole dataset, would have demonstrated
similar behavior based on its current form.

#### Future Work

Future research in this domain will extend this project would be benchmarking
whether LLMs can generate the textual descriptions of various medical codes to
understand whether they understand the semantic meanings behind these medical
codes.

Additionally, while we anticipate that our findings will have implications
across multiple fields, there exists a compelling opportunity to investigate
other specialized code systems within domains such as law and various
scientific fields. This exploration could further illuminate the broader
applicability of LLMs improving their quality and potentially guiding future
advancements in AI and machine learning across a spectrum of domain specific
disciplines.

## References

  * Alexander et al. (2003) Sherri Alexander, Therese Conner, and Teresa Slaughter.  Overview of inpatient coding.  _American journal of health-system pharmacy_ , 60(suppl_6):S11–S14, 2003.
  * Amin et al. (2019) Saadullah Amin, Günter Neumann, Katherine Dunfield, Anna Vechkaeva, Kathryn Annette Chapman, and Morgan Kelly Wixted.  Mlt-dfki at clef ehealth 2019: Multi-label classification of icd-10 codes with bert.  In _CLEF (Working Notes)_ , pp. 1–15, 2019.
  * Atutxa et al. (2017) Aitziber Atutxa, Alicia Pérez, and Arantza Casillas.  Machine learning approaches on diagnostic term encoding with the icd for clinical documentation.  _IEEE journal of biomedical and health informatics_ , 22(4):1323–1329, 2017.
  * Belyaeva et al. (2023) Anastasiya Belyaeva, Justin Cosentino, Farhad Hormozdiari, Krish Eswaran, Shravya Shetty, Greg Corrado, Andrew Carroll, Cory Y McLean, and Nicholas A Furlotte.  Multimodal llms for health grounded in individual-specific data.  In _Workshop on Machine Learning for Multimodal Healthcare Data_ , pp. 86–102. Springer, 2023.
  * Chen et al. (2024) Emma Chen, Aman Kansal, Julie Chen, Boyang Tom Jin, Julia Reisler, David E Kim, and Pranav Rajpurkar.  Multimodal clinical benchmark for emergency care (mc-bec): A comprehensive benchmark for evaluating foundation models in emergency medicine.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Chen & Ren (2019) Yuwen Chen and Jiangtao Ren.  Automatic icd code assignment utilizing textual descriptions and hierarchical structure of icd code.  In _2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_ , pp. 348–353. IEEE, 2019.
  * Chen et al. (2023) Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut.  Meditron-70b: Scaling medical pretraining for large language models, 2023.
  * Cimino (1996) James J Cimino.  Coding systems in health care.  _Methods of information in medicine_ , 35(04/05):273–284, 1996.
  * Djennaoui et al. (2015) Mehdi Djennaoui, Grégoire Ficheur, Régis Beuscart, and Emmanuel Chazard.  Improvement of the quality of medical databases: data-mining-based prediction of diagnostic codes from previous patient codes.  In _MIE_ , pp. 419–423, 2015.
  * Golkar et al. (2023) Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, Bruno Régaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho, and Shirley Ho.  xval: A continuous number encoding for large language models, 2023.
  * Haq et al. (2017) Hasham Ul Haq, Rameel Ahmad, and Sibt Ul Hussain.  Intelligent ehrs: predicting procedure codes from diagnosis codes.  _arXiv preprint arXiv:1712.00481_ , 2017.
  * Huang et al. (2022) Chao-Wei Huang, Shang-Chi Tsai, and Yun-Nung Chen.  Plm-icd: automatic icd coding with pretrained language models.  _arXiv preprint arXiv:2207.05289_ , 2022.
  * Huang et al. (2019) Kexin Huang, Jaan Altosaar, and Rajesh Ranganath.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  _arXiv preprint arXiv:1904.05342_ , 2019.
  * Jay (2019) J Ye Jay.  Construction and utilization of a neural network model to predict current procedural terminology codes from pathology report texts.  _Journal of pathology informatics_ , 10(1):13, 2019.
  * Karabacak & Margetis (2023) Mert Karabacak and Konstantinos Margetis.  Embracing large language models for medical applications: Opportunities and challenges.  _Cureus_ , 15(5), 2023.
  * Kavuluru et al. (2015) Ramakanth Kavuluru, Anthony Rios, and Yuan Lu.  An empirical evaluation of supervised learning approaches in assigning diagnosis codes to electronic medical records.  _Artificial intelligence in medicine_ , 65(2):155–166, 2015.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240, 2020.
  * Lee et al. (2024) Simon A Lee, Sujay Jain, Alex Chen, Arabdha Biswas, Jennifer Fang, Akos Rudas, and Jeffrey N Chiang.  Multimodal clinical pseudo-notes for emergency department prediction tasks using multiple embedding model for ehr (meme).  _arXiv preprint arXiv:2402.00160_ , 2024.
  * Lita et al. (2008) Lucian Vlad Lita, Shipeng Yu, Stefan Niculescu, and Jinbo Bi.  Large scale diagnostic code classification for medical patient records.  In _Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II_ , 2008.
  * Liu et al. (2021) Ning Liu, Qian Hu, Huayun Xu, Xing Xu, and Mengxin Chen.  Med-bert: A pretraining framework for medical records named entity recognition.  _IEEE Transactions on Industrial Informatics_ , 18(8):5600–5608, 2021.
  * Mullenbach et al. (2018) James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein.  Explainable prediction of medical codes from clinical text.  _arXiv preprint arXiv:1802.05695_ , 2018.
  * Mulyar et al. (2021) Andriy Mulyar, Ozlem Uzuner, and Bridget McInnes.  Mt-clinical bert: scaling clinical information extraction with multitask learning.  _Journal of the American Medical Informatics Association_ , 28(10):2108–2115, 2021.
  * Parr et al. (2018) Sharidan K Parr, Matthew S Shotwell, Alvin D Jeffery, Thomas A Lasko, and Michael E Matheny.  Automated mapping of laboratory tests to loinc codes using noisy labels in a national electronic health record system database.  _Journal of the American Medical Informatics Association_ , 25(10):1292–1300, 2018.
  * Samonte et al. (2018) Mary Jane C Samonte, Bobby D Gerardo, Arnel C Fajardo, and Ruji P Medina.  Icd-9 tagging of clinical notes using topical word embedding.  In _Proceedings of the 2018 1st International Conference on Internet and e-Business_ , pp. 118–123, 2018.
  * Shin et al. (2020) Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, and Raghav Mani.  Biomegatron: Larger biomedical domain language model.  _arXiv preprint arXiv:2010.06060_ , 2020.
  * Singaravelan et al. (2021) Anandakumar Singaravelan, Chung-Ho Hsieh, Yi-Kai Liao, and Jia-Lien Hsu.  Predicting icd-9 codes using self-report of patients.  _Applied Sciences_ , 11(21):10046, 2021.
  * Spasić et al. (2010) Irena Spasić, Farzaneh Sarafraz, John A Keane, and Goran Nenadić.  Medication information extraction with linguistic pattern matching and semantic rules.  _Journal of the American Medical Informatics Association_ , 17(5):532–535, 2010.
  * Subotin & Davis (2014) Michael Subotin and Anthony Davis.  A system for predicting icd-10-pcs codes from electronic health records.  In _Proceedings of bionlp 2014_ , pp. 59–67, 2014.
  * Toma et al. (2023) Augustin Toma, Patrick R. Lawler, Jimmy Ba, Rahul G. Krishnan, Barry B. Rubin, and Bo Wang.  Clinical camel: An open expert-level medical language model with dialogue-based knowledge encoding, 2023.
  * Tu et al. (2024) Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al.  Towards generalist biomedical ai.  _NEJM AI_ , 1(3):AIoa2300138, 2024.
  * Uzuner et al. (2010) Özlem Uzuner, Imre Solti, and Eithon Cadag.  Extracting medication information from clinical text.  _Journal of the American Medical Informatics Association_ , 17(5):514–518, 2010.
  * Van Veen et al. (2023) Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerova, et al.  Clinical text summarization: Adapting large language models can outperform human experts.  _Research Square_ , 2023.
  * Van Veen et al. (2024) Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, et al.  Adapted large language models can outperform medical experts in clinical text summarization.  _Nature Medicine_ , pp. 1–9, 2024.
  * Wornow et al. (2023) Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A Pfeffer, Jason Fries, and Nigam H Shah.  The shaky foundations of large language models and foundation models for electronic health records.  _npj Digital Medicine_ , 6(1):135, 2023.

##  Appendix A Appendix

## Example Outputs

Below we highlight the outputs of each LLM in an example regarding predicting
20 ICD 10 codes in random order:

Prompt
predict these ICD 10 codes to the best of your ability: [‘Z16.35’, ‘H01.001’,
‘S83.134S’, ‘T36.5X1A’, ‘V32.9XXA’, ‘S82.402H’, ‘S36.241S’, ‘S04.72XD’,
‘S06.313S’, ‘M60.076’, ‘S92.021K’, ‘C40.22’, ‘S42.433B’, ‘M80.061D’,
‘S56.092S’, ‘S14.143A’, ‘T58.93XA’, ‘T44.3X3D’, ‘H75.82’, ‘V73.4XXS’]

### Model Responses

Meditron

We would like to highlight that we heavily experimented with Meditron. However
this model would often give redundant responses like the ones seen below
regardless of the prompt.
1\. If the ICD 10 Code is an integer, then: a. Predict that it is a number in
the range [140-999]. b. Otherwise, predict that it is a letter in the range
[‘A’ , ‘Z’]. 2\. If the ICD 10 Code is a string of characters, then: a. Look
up the first letter of this string to find out its value (e.g., ‘012’ → ‘2’).
b. Use the value found in step a for the prediction described in step 1. 3\.
If the ICD 10 Code is an empty string, then predict that it is a zero.

Medllama2
These codes are related to a patient with a history of head injury, seizures,
and cognitive impairment. The patient may be at risk for various complications
such as stroke, brain edema, intracerebral hemorrhage, and further cognitive
decline. It’s important to monitor the patient closely and follow up with
neurological consultation if necessary.

GPT4
Interpreting ICD-10 codes involves a detailed understanding of the
International Classification of Diseases, Tenth Revision, which provides a
system for categorizing health conditions and related health problems. Each
code is highly specific, reflecting various aspects of diagnoses. Here’s a
general interpretation of the codes you’ve provided: Z1635: Encounter for
screening for certain developmental disorders in childhood H01.001: Acute
follicular conjunctivitis, right eye S83.134S: Sprain of medial collateral
ligament of knee, subsequent encounter T36.5X1A: Poisoning by antimalarials
and drugs acting on other blood protozoa, accidental (unintentional), initial
encounter V32.9XXA: Occupant of pick-up truck or van injured in collision with
car, pick-up truck or van in traffic accident, initial encounter S82.402H:
Nondisplaced fracture of lateral malleolus of left fibula, subsequent
encounter for fracture with delayed healing S36.241S: Injury of ileum,
subsequent encounter S04.72XD: Injury of phrenic nerve, left side, subsequent
encounter S06.313S: Focal traumatic brain injury without loss of
consciousness, subsequent encounter M60.076: Pyomyositis, lower leg S92.021K:
Displaced fracture of navicular [scaphoid] of left foot, subsequent encounter
for fracture with nonunion C40.22: Malignant neoplasm of short bones of left
lower limb S42.433B: Displaced fracture of base of neck of left humerus,
initial encounter for open fracture M80.061D: Age-related osteoporosis with
current pathological fracture, right lower leg, subsequent encounter for
fracture with routine healing S56.092S: Strain of muscle, fascia and tendon at
forearm level, left arm, sequela S14.143A: Complete lesion of C5-C7 level of
cervical spinal cord, initial encounter T58.93XA: Toxic effect of chlorine
gas, assault, initial encounter T44.3X3D: Poisoning by local antifungal, anti-
infective and anti-inflammatory drugs, assault, subsequent encounter H75.82:
Other specified disorders of ear in diseases classified elsewhere V73.4XXS:
Car occupant injured in collision with heavy transport vehicle or bus in
nontraffic accident, sequela Each code specifies not just the condition or
injury, but often the side of the body affected, whether the visit is an
initial or subsequent encounter, and sometimes the intent (e.g., accidental,
assault). This granularity aids healthcare providers in tracking the precise
nature of diagnoses and treatments. Note that for full and accurate
descriptions, consulting the official ICD-10 coding guidelines or a healthcare
professional is recommended.

GPT3.5
Predicting ICD-10 codes without context or specific information is
challenging, as these codes represent a wide range of diagnoses, procedures,
and external causes of injury or diseases. However, I can attempt to provide
potential interpretations based on the structure and conventions of ICD-10
coding: Z16.35: Encounter for surveillance of other genetic diseases H01.001:
Malignant neoplasm of right upper eyelid, including canthus, skin of eyelid
and external ear S83.134S: Sprain of posterior cruciate ligament of left knee,
sequela T36.5X1A: Toxic effect of other venoms and toxins, accidental
(unintentional), initial encounter V32.9XXA: Person on outside of special
construction vehicle injured in collision with fixed or stationary object,
initial encounter S82.402H: Displaced fracture of lateral condyle of left
tibia, subsequent encounter for closed fracture with delayed healing S36.241S:
Displaced fracture of anterior process of right calcaneus, sequela S04.72XD:
Crushing injury of unspecified ear, subsequent encounter S06.313S: Laceration
without foreign body of left ear, sequela M60.076: Palmar fascial
fibromatosis, left hand S92.021K: Dislocation of unspecified
metacarpophalangeal joint of right thumb, subsequent encounter C40.22:
Malignant neoplasm of liver, unspecified lobe S42.433B: Unspecified injury of
unspecified blood vessel at lower leg level, left leg, initial encounter
M80.061D: Closed fracture of base of skull, initial encounter for closed
fracture S56.092S: Strain of unspecified muscles, fascia and tendons at thigh
level, left thigh, sequela S14.143A: Other specified injury of shoulder and
upper arm, left arm, initial encounter T58.93XA: Traumatic amputation of
unspecified wrist, hand and fingers, initial encounter T44.3X3D: Poisoning by
agents primarily acting on smooth and skeletal muscles and respiratory system,
undetermined, subsequent encounter H75.82: Other auditory disturbances,
bilateral V73.4XXS: Bus occupant injured in collision with other nonmotor
vehicle in nontraffic accident, sequela Please note that these interpretations
are based on the structure and conventions of ICD-10 coding and may not
accurately reflect the actual medical conditions or events associated with
these codes. For precise medical diagnoses and treatment, it’s essential to
consult with a medical professional or refer to a comprehensive medical coding
resource.

Claude 3 (Sonnet)
Here are my predictions for those ICD-10 codes: Z16.35: Resistance to
antiviral drugs H01.001: Unspecified blepharitis right eye, upper and lower
eyelids S83.134S: Medial meniscus tear, current injury, right knee, sequela
T36.5X1A: Adverse effect of aminoglycosides, initial encounter V32.9XXA:
Passenger in unspecified motor vehicle injured in noncollision transport
accident, initial encounter S82.402H: Displaced fracture of lateral malleolus
of right fibula, subsequent encounter S36.241S: Minor contusion of liver,
sequela S04.72XD: Contusion of left cheek and temporomandibular area,
subsequent encounter S06.313S: Contusion and laceration of right cerebrum
without prolapsed brain matter, sequela M60.076: Infective myositis, lower leg
S92.021K: Displaced fracture of unspecified calcaneus, subsequent encounter
for fracture with nonunion C40.22: Malignant neoplasm of unspecified bronchus
or lung S42.433B: Displaced comminuted fracture of shaft of humerus, initial
encounter for open fracture M80.061D: Age-related osteoporosis with current
pathological fracture, right forearm, subsequent encounter for fracture with
routine healing S56.092S: Unspecified injury of flexor muscle, fascia and
tendon of left thumb at forearm level, sequela S14.143A: Other fractures of
rib, initial encounter for closed fracture T58.93XA: Toxic effect of
unspecified insecticide, accidental (unintentional), initial encounter
T44.3X3D: Poisoning by ecstasy, assault, subsequent encounter H75.82: Age-
related cataract, morgagnian type, left eye V73.4XXS: Pregnancy with
disruption of prior tubal or uterine surgery, sequela

Gemini
Here are my predictions for the ICD-10 codes you provided, to the best of my
ability without internet access: Code - Possible Name Z16.35 \- Encounter for
other biological factors affecting the mother’s health during pregnancy (This
is a placeholder code for unspecified factors) H01.001 \- Hypertensive heart
disease with (heart) failure S83.134S \- Percutaneous puncture and aspiration
of joint with steroid injection, left knee (This is based on the assumption
”S” stands for ”surgery” and ”4” signifies the lower extremity) T36.5X1A \-
Poisoning by (unspecified) anticonvulsants, unintentional, initial encounter
V32.9XXA \- Other personal history of other perinatal conditions (This is a
placeholder code for unspecified perinatal conditions) S82.402H \- Closed
reduction of displaced metacarpal fracture, with internal fixation, fifth
metacarpal (This is based on the assumption ”S” stands for ”surgery” and ”2”
signifies the hand) S36.241S \- Other excision of perianal skin tag(s), with
local anesthesia (This is based on the assumption ”S” stands for ”surgery” and
”3” signifies the digestive system) S04.72XD \- Excision of keloid scar, other
site (This is based on the assumption ”S” stands for ”surgery” and ”0”
signifies the head) S06.313S \- Other local excision of benign neoplasm of
parotid gland, with local anesthesia (This is based on the assumption ”S”
stands for ”surgery” and ”0” signifies the head) M60.076 \- Poisoning by
(unspecified) hallucinogens, with psychotic features S92.021K \- Traumatic
rupture of spleen, open wound, laparoscopic repair C40.22 \- Malignant
neoplasm of skin of eyelid, other and unspecified parts S42.433B \- Other open
reduction of displaced fracture of upper end of tibia, with internal fixation
M80.061D \- Intraosseous injection of medication, unspecified site S56.092S \-
Other open reduction of displaced fracture of neck of femur, with internal
fixation (This is based on the assumption ”S” stands for ”surgery” and ”5”
signifies the hip) S14.143A \- Incisional lung biopsy, with wedge resection,
open approach T58.93XA \- Assault by other blunt object(s), initial encounter
H75.82 \- Idiopathic scoliosis with deformity (This is a placeholder code for
unspecified severity) V73.4XXS \- Encounter for other aftereffects of trauma
(This is a placeholder code for unspecified aftereffects) Disclaimer: • These
are just predictions based on the code structure and common medical practices.
• The actual meaning of the codes may vary depending on the specific
healthcare system and documentation guidelines. • For accurate diagnosis and
treatment, always consult a qualified healthcare professional. • Some letters
may have different meanings depending on the coding system used within a
healthcare facility.

LLaMA 70B
I’ll do my best to predict the ICD-10 codes you’ve provided. Please note that
these predictions are based on my training data, and I may not always get it
right. Here are my predictions for the codes you’ve provided: 1\. Z16.35 -
Encounter for general adult medical examination without abnormal findings 2\.
H01.001 - Unspecified hearing loss, right ear 3\. S83.134S - Other specified
joint disorders, left shoulder 4\. T36.5X1A - Unspecified injury of
unspecified site, right arm 5\. V32.9XXA - Unspecified personal history of
other conditions, not elsewhere classified 6\. S82.402H - Other specified
disorders of the spine, thoracic region 7\. S36.241S - Other specified
disorders of the shoulder region, right shoulder 8\. S04.72XD - Unspecified
strain of muscle(s), fascia, and tendon at forearm level, left arm 9\.
S06.313S - Unspecified sprain of joint, left ankle 10\. M60.076 - Unspecified
abscess, left upper arm 11\. S92.021K - Other specified disorders of the knee,
right knee 12\. C40.22 - Unspecified fracture of humerus, left humerus 13\.
S42.433B - Other specified disorders of the elbow, right elbow 14\. M80.061D -
Unspecified dislocation of joint, left shoulder 15\. S56.092S - Other
specified disorders of the lower back, thoracic region 16\. S14.143A -
Unspecified degenerative disc disease, lumbar region 17\. T58.93XA -
Unspecified injury of unspecified site, left arm 18\. T44.3X3D - Unspecified
fracture of shinbone, right shinbone 19\. H75.82 - Unspecified hearing loss,
left ear 20\. V73.4XXS - Unspecified personal history of other conditions, not
elsewhere classified Please note that these are just predictions, and it’s
always best to consult a medical professional for accurate diagnosis and
coding information.

Le Chat (Mistral)
Here are my predictions for the given ICD-10 codes. Please note that these are
just predictions and might not be accurate as I’m not using the internet to
look up the codes: 1\. ‘Z16.35’: Presence of aortic aneurysm without mention
of rupture 2\. ‘H01.001’: Acute suppurative otitis media without spontaneous
rupture of eardrum, right ear 3\. ‘S83.134S’: Sprain of right ankle, sequela
4\. ‘T36.5X1A’: Underdosing of other antiepileptic and sedative-hypnotic
drugs, initial encounter 5\. ‘V32.9XXA’: Other specified water transport
accident injuring occupant, initial encounter 6\. ‘S82.402H’: Displaced
fracture of lateral malleolus of right fibula, subsequent encounter for
fracture with routine healing 7\. ‘S36.241S’: Laceration of right kidney with
open wound into cavity, sequela 8\. ‘S04.72XD’: Puncture wound without foreign
body of unspecified eyelid and periocular area, subsequent encounter 9\.
‘S06.313S’: Intracranial injury of other and unspecified nature with brief
[less than one hour] loss of consciousness, sequela 10\. ‘M60.076’: Calcific
tendinitis of right shoulder 11\. ‘S92.021K’: Laceration of right heel,
subsequent encounter 12\. ‘C40.22’: Malignant neoplasm of right frontal lobe
13\. ‘S42.433B’: Displacement of internal fixation device of right clavicle,
initial encounter 14\. ‘M80.061D’: Age-related osteoporosis with current
pathological fracture, right forearm, subsequent encounter for fracture with
delayed healing 15\. ‘S56.092S’: Laceration of unspecified muscle, fascia and
tendon at forearm level, left arm, sequela 16\. ‘S14.143A’: Puncture wound
without foreign body of left shoulder, initial encounter 17\. ‘T58.93XA’:
Other and unspecified adverse effect of other drugs, medicaments and
biological substances, initial encounter 18\. ‘T44.3X3D’: Poisoning by
penicillins, subsequent encounter 19\. ‘H75.82’: Other specified disorders of
vestibular function 20\. ‘V73.4XXS’: Unspecified fit and epilepsy, sequela