# An Embedding of Schwartz Distributions in the Algebra of Asymptotic
Functions

Michael Oberguggenberger
Institut für Mathematik und Geometrie
Universitat Innsbruck
A-6020 Inssbruck, Austria
<EMAIL_ADDRESS>
and
Todor Todorov
Mathematics Department
California Polytechnic State University
San Luis Obispo, California 93407, USA
<EMAIL_ADDRESS>
(Received October 28, 1996 and in revised form June 21, 1997)

###### Abstract

We present a solution of the problem of multiplication of Schwartz
distributions by embedding the space of distributions into a differential
algebra of generalized functions, called in the paper “asymptotic function”,
similar to but different from J. F. Colombeau’s algebras of new generalized
functions.

Key Words and Phrases: Schwartz distributions, nonlinear theory of generalized
functions, asymptotic expansion, nonstandard analysis, nonstandard asymptotic
analysis.

1991 AMS Subject Classification Codes: 03H05, 12J25, 46F10, 46820.

##  1 Introduction

The main purpose of this paper is to prove the existence of an embedding
$\Sigma_{{D},\Omega}$ of the space of Schwartz distributions ${\cal
D}^{\prime}(\Omega)$ into the algebra of asymptotic functions
${}^{\rho}\mathcal{E}(\Omega)$ which preserves all linear operations in ${\cal
D}^{\prime}(\Omega)$. Thus, we offer a solution of the problem of
multiplication of Schwartz distributions since the multiplication within
${\cal D}^{\prime}(\Omega)$ is impossible (L. Schwartz [1]).

The algebra ${}^{\rho}\mathcal{E}(\Omega)$ is defined in the paper as a factor
space of nonstandard smooth functions. The field of the scalars
${}^{\rho}\mathbb{C}$ of the algebra ${}^{\rho}\mathcal{E}(\Omega)$ coincides
with the complex counterpart of the Robinson asymptotic numbers -known also as
Robinson’s field with valuation (Robinson’s (2]) and Lightstone and Robinson
[3]). The embedding $\Sigma_{D,\Omega}$ is constructed in the form
$\Sigma_{D,\Omega}=Q_{\Omega}\circ D\star\Pi\bullet\,^{*}$, where (in backward
order): ∗ is the extension mapping (in the sense of nonstandard analysis),
$\bullet$ is the Schwartz multiplication in ${\cal D}^{\prime}(\Omega)$ (more
precisely, its nonstandard extension), $\star$ is the convolution operator
(more precisely, its nonstandard extension), $\circ$ denotes “composition”,
$Q_{\Omega}$ is the quotient mapping (in the definition of the algebra of
asymptotic functions) and $D$ and $\Pi_{\Omega}$ are fixed nonstandard
internal functions with special properties whose existence is proved in this
paper.

Our interest in the algebra ${}^{\rho}\mathcal{E}(\Omega)$ and the embedding
${\cal D}^{\prime}(\Omega)\subset\,^{\rho}\mathcal{E}(\Omega)$, is due to
their role in the problem of multiplication of Schwartz distributions, the
nonlinear theory of generalized functions and its applications to partial
differential equations (Oberguggenberger [4]), (Todorov [5] and [6]). In
particular, there is a strong similarity between the algebra of asymptotic
functions ${}^{\rho}\mathcal{E}(\Omega)$ and its generalized scalars
${}^{\rho}\mathbb{C}$, discussed in this paper, and the algebra of generalized
functions ${\cal G}(\Omega)$ and their generalized scalars
$\overline{\mathbb{C}}$, introduced by J. F. Colombeau in the framework of
standard analysis (Colombeau [7], pp. 63, 138 and Colombeau [8], §8.3,
pp.161-166). We should mention that the involvement of nonstandard analysis
has resulted in some improvements of the corresponding standard counterparts;
one of them is that ${}^{\rho}\mathbb{C}$ is an algebraically closed field
while its standard counterpart $\overline{\mathbb{C}}$ in J. F. Colombeau’s
theory is a ring with zero divisors.

This paper is a generalization of some results in [9] and [10] (by the authors
of this paper, respectively) where only the embedding of the tempered
distributions $\mathcal{S}^{\prime}(\mathbb{R}^{d})$ in
${}^{\rho}\mathcal{E}(\mathbb{R}^{d})$ has been established. The embedding of
all distributions ${\cal D}^{\prime}(\Omega)$, discussed in this paper,
presents an essentially different situation. We should mention that the
algebra ${}^{\rho}\mathcal{E}(\mathbb{R}^{d})$ was recently studied in
(Hoskins and Sousa Pinto [11]).

Here $\Omega$ denotes an open set of $\mathbb{R}^{d}$ ($d$ is a natural
number), $\mathcal{E}(\Omega)=C^{\infty}(\Omega)$ and ${\cal
D}(\Omega)=C^{\infty}_{0}(\Omega)$ denote the usual classes of
$C^{\infty}$-functions on $\Omega$ and $C^{\infty}$-functions with compact
support in $\Omega$ and ${\cal D}^{\prime}(\Omega)$, and
$\mathcal{E}^{\prime}(\Omega)$ denote the classes of Schwartz distributions on
$\Omega$ and Schwartz distributions with compact support in $\Omega$,
respectively. As usual, $\mathbb{N}$, $\mathbb{R}$, $\mathbb{R}_{+}$, and
$\mathbb{C}$ will be the systems of the natural, real, positive real and
complex numbers, respectively, and we use also the notation
$\mathbb{N}_{0}=\\{0\\}\cup\mathbb{N}$. For the partial derivatives we write
$\partial^{\alpha}$, $\alpha\in\mathbb{N}_{0}^{d}$. If
$\alpha=(\alpha_{1},\dots,\alpha_{d})$ for some $\alpha\in\mathbb{N}_{0}^{d}$,
we write $|\alpha|=\alpha_{1}+\cdots+\alpha_{d}$ and if $x=(x_{1},\dots
x_{d})$ for some $x\in\mathbb{R}^{d}$, we write
$x^{\alpha}=x_{1}^{\alpha_{1}}x_{2}^{\alpha_{2}}\dots x_{d}^{\alpha_{2}}\dots
x_{d}^{\alpha_{d}}$. For a general reference to distribution theory we refer
to Bremermann [12] and Vladimirov [13].

Our framework is a nonstandard model of the complex numbers $\mathbb{C}$, with
degree of saturation larger than card$(\mathbb{N})$. We denote by
${}^{*}\mathbb{R}$, ${}^{*}\mathbb{R}_{+}$, ${}^{*}\mathbb{C}$,
${}^{*}\mathcal{E}(\Omega)$ and ${}^{*}{\cal D}(\Omega)$ the nonstandard
extensions of $\mathbb{R}$, $\mathbb{R}_{+}$, $\mathbb{C}$,
$\mathcal{E}(\Omega)$ and ${\cal D}(\Omega)$, respectively. If $X$ is a set of
complex numbers or a set of (standard) functions, then ${}^{*}X$ will be its
nonstandard extension and if $f:X\to Y$ is a (standard) mapping, then
${{}^{*}f}\colon{{}^{*}X}\to\,^{\star}Y$ will be its nonstandard extension.
For integration in ${}^{*}\mathbb{R}^{d}$ we use the $*$-Lebesgue integral. We
shall often use the same notation, $\|x\|$, for the Euclidean norm in
$\mathbb{R}^{d}$ and its nonstandard extension in ${}^{\star}\mathbb{R}^{d}$.
For a short introduction to nonstandard analysis we refer to the Appendix in
Todorov [6]. For a more detailed exposition we recommend Lindstrom [14], where
the reader will find many references to the subject.

##  2 Test Functions and Their Moments

In this section we study some properties of the test functions in ${\cal
D}(\mathbb{R}^{d})$ (in a standard setting) which we shall use subsequently.

Following (Colombeau [7], p. 55), for any $k\in\mathbb{N}$ we define the set
of test functions:

$A_{k}=\\{\varphi\in{\cal D}(\mathbb{R}^{d})\colon\varphi\mbox{ is real-
valued, }\varphi(x)=0\mbox{ for }\|x\|>1;$
$\int_{\mathbb{R}^{d}}\varphi(x)dx=1\mbox{ and
}\int_{\mathbb{R}^{d}}x^{\alpha}\varphi(x)dx=0\mbox{ for
}\alpha\in\mathbb{N}_{0}^{d},1\leq|\alpha|\leq k\\}.$ $None$

Obviously, $A_{1}\supset A_{2}\supset A_{3}\supset\dots$. Also, we have
$A_{k}\neq\emptyset$ for all $k\in\mathbb{N}$ (Colombeau [7], Lemma (3.3.1),
p. 55).

In addition to the above we have the following result:

Lemma 2.2. For any $k\in\mathbb{N}$

$\inf_{\varphi\in A_{k}}\left(\int_{\mathbb{R}^{d}}|\varphi(x)|dx\right)=1.$
$None$

More precisely, for any positive real $\delta$ there exists $\varphi$ in
$A_{k}$ such that

$1\leq\int_{\mathbb{R}^{d}}|\varphi(x)|dx\leq 1+\delta.$

In addition, $\varphi$ can be chosen symmetric.

Proof. We consider the one dimensional case $d=1$ first. Start with some fixed
positive (real valued) $\psi$ in ${\cal D}(\mathbb{R})$ such that $\psi(x)=0$
for $|x|\geq 1$ and $\int_{\mathbb{R}}\psi(x)dx=1$ ($\psi$ can be also chosen
symmetric if needed). We shall look for $\varphi$ in the form:

$\varphi(x)=\sum_{j=0}^{k}c_{j}\psi\left(\frac{x}{\varepsilon^{j}}\right),$

$x\in\mathbb{R}$, $\varepsilon\in\mathbb{R}_{+}$. We have to find $c_{j}$ for
which $\varphi\in A_{k}$. Observing that

$\int_{\mathbb{R}}x^{j}\psi\left(\frac{x}{\varepsilon^{j}}\right)dx=\varepsilon^{(i+1)j}\int_{\mathbb{R}}y^{j}\psi(y)dy,$

for $i=0,1,\dots,k$, we derive the system for linear equations for $c_{j}$:

$\left\\{\begin{array}[]{ll}\displaystyle\sum_{j=0}^{k}c_{j}\,\varepsilon^{j}=1,\\\\[8.0pt]
\displaystyle\left(\int_{\mathbb{R}}y^{i}\psi(y)dy\right)\sum_{j=0}^{k}c_{j}\,\varepsilon^{(i+1)j}=0,\;i=1,\dots,k.\end{array}\right.$

The system is certainly satisfied, if

$\left\\{\begin{array}[]{ll}\displaystyle\sum_{j=0}^{k}c_{j}\,\varepsilon^{j}=1,\\\\[8.0pt]
\displaystyle\sum_{j=0}^{k}c_{j}\,\varepsilon^{(i+1)j}=0,i=1,\dots,k,\end{array}\right.$

which can be written in the matrix form $V_{k+1}(\varepsilon)\,C=B$, where
$V_{k+1}(\varepsilon)$ is Vandermonde $(k+1)\times(k+1)$ matrix, $C$ is the
column of the unknowns $c_{j}$, and $B$ is the column whose top entry is 1 and
all others are 0. For the determinant we have det $V_{k+1}(\,\varepsilon)\neq
0$ for $\,\varepsilon\neq 1$, therefore, the system has a unique solution
$(c_{0},c_{1},c_{2},\dots,c_{k})$. Our next goal is to show that this solution
is of the form:

$c_{j}=\pm\frac{\,\varepsilon^{\alpha_{j}}(1+\,\varepsilon
P_{j}(\varepsilon))}{\,\varepsilon^{\beta}(1+\,\varepsilon P(\varepsilon))},$
$None$

where $P_{j}$ and $P$ are polynomials and

$\alpha_{j}=\sum_{q=1}^{k-1}q(k+1-q)+\sum_{m=j}^{k-1}(k+1-m),$ $None$

for $0\leq j\leq k$, and

$\beta=k+\sum_{q=1}^{k-1}q(k+1-q).$

The coefficients $c_{0},c_{1},c_{2},\dots,c_{k}$ will be found by Cramer’s
rule. The formula for Vandermonde determinants gives

$\displaystyle\displaystyle\Pi_{m=1}^{k}\Pi_{q=m+1}^{k+1}(\varepsilon^{q}-\varepsilon^{m})$
$\displaystyle=$
$\displaystyle\Pi_{m=1}^{k}\left(\Pi_{q=m+1}^{k+1}\varepsilon^{m}(\varepsilon^{q-m}-1)\right)$
$\displaystyle=$
$\displaystyle\varepsilon^{\beta}\Pi_{m=1}^{k}\Pi_{q=m+1}^{k+1}(\varepsilon^{q-m}-1)=\pm\varepsilon^{\beta}(1+\varepsilon
P(\varepsilon)),$

for some polynomial $P$, where

$\beta=\sum_{m=1}^{k}m(k+1-m)=k+\sum_{m=1}^{k-1}m(k+1-m).$

To calculate the numerator in (2.3), we have to replace the $j$th column of
the matrix by the column $B$ (whose top entry is 1 and all others are 0) and
calculate the resulting determinant $D_{j}$. Consider first the case $1\leq
j\leq k-1$. By developing with respect to the $j$th column, we get

$D_{j}=\pm\det\left(\begin{array}[]{ccccccc}1,&\varepsilon^{2},&\dots&\varepsilon^{2(j-1)},&\varepsilon^{2(j+1)},&\dots&\varepsilon^{2k}\\\
1,&\varepsilon^{3},&\dots&\varepsilon^{3(j-1)},&\varepsilon^{3(j+1)},&\dots&\varepsilon^{3k}\\\
\dots&\dots&\dots&\dots&\dots&\dots&\dots\\\
1,&\varepsilon^{k+1},&\dots&\varepsilon^{(k+1)(j-1)},&\varepsilon^{(k+1)(j+1)},&\dots&\varepsilon^{(k+1)k}\end{array}\right).$

We factor out
$\varepsilon^{2},\varepsilon^{4},\dots,\varepsilon^{2(j-1)},\varepsilon^{2(j+1)},\dots,\varepsilon^{2k}$
and obtain:

$\displaystyle D_{j}$ $\displaystyle=$ $\displaystyle\pm\varepsilon^{1\cdot
1}\varepsilon^{2\cdot 2}\dots\varepsilon^{(j-1)\cdot 2}\varepsilon^{(j+1)\cdot
2}\dots\varepsilon^{2k}$ $\displaystyle\times\det$
$\displaystyle\left(\begin{array}[]{cccccccc}1,&1,&1,&\dots&1,&1,&\dots&1\\\
1,&\varepsilon,&\varepsilon^{2}&\dots&\varepsilon^{(j-1)},&\varepsilon^{(j+1)},&\dots&\varepsilon^{k}\\\
\dots&\dots&\dots&\dots&\dots&\dots&\dots\\\
1,&\varepsilon^{k-1},&\varepsilon^{2(k-1)},&\dots&\varepsilon^{(j-1)(k-1)},&\,\varepsilon^{(j+1)(k-1)},&\dots&\varepsilon^{k(k-1)}\end{array}\right).$

The latter is a Vandermonde determinant again, and we have

$\displaystyle D_{j}=$ $\displaystyle\pm$ $\displaystyle\varepsilon^{1\cdot
1+2\cdot 2+\cdots+(j-1)2+(j+1)2+\cdots+k\cdot 2}$ $\displaystyle\times$
$\displaystyle(\varepsilon-1)(\varepsilon^{2}-1)\dots(\varepsilon^{j-1}-1)(\varepsilon^{j+1}-1)\dots(\varepsilon^{k}-1)$
$\displaystyle\times$
$\displaystyle(\varepsilon^{2}-\varepsilon)(\varepsilon^{3}-\varepsilon)\dots(\varepsilon^{j-1}-\varepsilon)(\varepsilon^{j+1}-\varepsilon)\dots(\varepsilon^{k}-\varepsilon)$
$\displaystyle\times$ $\displaystyle\dots\dots\dots\dots\dots\dots\dots\dots$
$\displaystyle\times$
$\displaystyle(\varepsilon^{j-1}-\varepsilon^{j-2})(\varepsilon^{j+1}-\varepsilon^{j-2})\dots(\varepsilon^{k}-\varepsilon^{j-2})$
$\displaystyle\times$
$\displaystyle(\varepsilon^{j+1}-\varepsilon^{j-1})\dots(\varepsilon^{k}-\varepsilon^{j-1})$
$\displaystyle\times$ $\displaystyle\dots\dots\dots\dots$
$\displaystyle\times$ $\displaystyle(\varepsilon^{k}-\varepsilon^{k-1}).$

Hence, factoring out $\varepsilon^{(i-1)(k-i)}$ in the $i$th row above, we get
$D_{j}=\pm\varepsilon^{\alpha_{j}}(1+\varepsilon P_{j}(\varepsilon))$ for some
polynomials $P_{j}(\varepsilon)$ and

$\displaystyle\alpha_{j}$ $\displaystyle=$ $\displaystyle 1\cdot 1+2\cdot
2+\cdots+(j-1)\cdot 2+(j+1)\cdot 2+\cdots+k\cdot 2$ $\displaystyle+$
$\displaystyle 1\cdot(k-2)+2\cdot(k-3)+\cdots+(j-1)\cdot(k-j)$
$\displaystyle+$ $\displaystyle(j+1)\cdot(k-j-1)+\cdots+(k-1)\cdot 1$
$\displaystyle=$ $\displaystyle 1\cdot
k+2\cdot(k-1)+\cdots+(j-1)\cdot(k-j+2)+(j+1)\cdot(k-j+1)+\cdots+(k-1)\cdot
3+k\cdot 2$ $\displaystyle=$
$\displaystyle\sum_{q=1}^{j-1}q(k+1-q)+\sum_{m=j}^{k-1}(m+1)(k+1-m)=\sum_{q=1}^{k-1}q(k+1-q)+\sum_{m=j}^{k-1}(k+1-m),$

which coincides with the desired result (2.4) for $\alpha_{j}$, in the case
$1\leq j\leq k-1$. For the extreme cases $j=0$ and $j=k$, we obtain

$\alpha_{0}=\sum_{m=0}^{k-1}(m+1)(k+1-m)=\sum_{q=1}^{k-1}q(k+1-q)+\sum_{m=0}^{k-1}(k+1-m),$
$\alpha_{k}=\sum_{q=1}^{k-1}q(k+1-q),$

which both can be incorporated in the formula (2.4) for $\alpha_{j}$. Finally,
Cramer’s rule gives the expression (2.3) for $c_{j}$.

Now, taking into account that $\psi\geq 0$, by assumption, and the fact that
$|1+\varepsilon P(\varepsilon)|>|1-|\varepsilon
P(\varepsilon)||=1-\varepsilon|P(\varepsilon)|>0$ for all sufficiently small
epsilon, we obtain

$\int_{\mathbb{R}}|\varphi(x)|dx\leq\sum_{j=0}^{k}|c_{j}|\,\varepsilon^{j}\leq\sum_{j=0}^{k}\frac{\varepsilon^{j+\alpha_{j}}(1+\varepsilon|P_{j}(\varepsilon)|)}{\varepsilon^{\beta}(1-\varepsilon|P(\varepsilon)|)},$

and this latter expression can be made smaller than $1+\delta$ for
sufficiently small $\varepsilon$ if a) $j+\alpha_{j}-\beta>0$ for $0\leq j\leq
k-1$, and b) $k+\alpha_{k}-\beta=0$. Now, b) is obvious, as for a), we have:

$j+\alpha_{j}-\beta=j+\sum_{m=j}^{k-1}(k+1-m)-k=\frac{1}{2}(k-j)(k-j+1)>0,$

for $0\leq j\leq k-1$. To generalize the result for arbitrary dimension $d$,
it suffices to consider a product of functions of one real variable. The proof
is complete. $\square$

##  3 Nonstandard Delta Functions

We prove the existence of a nonstandard function $D$ in ${}^{*}{\cal
D}(\mathbb{R}^{d})$ with special properties. The proof is based on the result
of Lemma 2.2 and the Saturation Principle (Todorov [6], p. 687). We also
consider a type of nonstandard cut-off-functions which have close counterparts
in standard analysis. The applications of these functions are left for the
next sections.

Lemma 3.1 (Nonstandard Mollifiers). For any positive infinitesimal $\rho$ in
${}^{*}\mathbb{R}$ there exists a nonstandard function $\theta$ in
${}^{\star}{\cal D}(\mathbb{R}^{d})$ with values in ${}^{*}\mathbb{R}$, which
is symmetric and which satisfies the following properties:

(i) $\theta(x)=0$ for $x\in\,^{*}\mathbb{R}^{d},\|x\|>1$,

(ii) $\int_{{}^{\star}\mathbb{R}^{d}}\theta(x)dx=1$,

(iii) $\int_{{}^{\star}\mathbb{R}^{d}}\theta(x)x^{\alpha}dx=0$, for all
$\alpha\in\mathbb{N}_{0}^{d}$, $\alpha\neq 0$,

(iv) $\int_{{}^{\star}\mathbb{R}^{d}}|\theta(x)|dx\approx 1$,

(v)
$\displaystyle|\ln\rho|^{-1}\left(\sup_{x\in\,^{\star}\mathbb{R}^{d}}|\partial^{\alpha}\theta(x)|\right)\approx
0$ for all a $\alpha\in\mathbb{N}_{0}^{d}$,

where $\approx$ is the infinitesimal relation in ${}^{*}\mathbb{C}$. We shall
call this type of function nonstandard $\rho$-mollifiers

Proof. For any $k\in\mathbb{N}$, we define the set of test functions:

$\overline{A}_{k}=\\{\varphi\in{\cal D}(\mathbb{R}^{d})\colon\varphi\mbox{ is
real-valued and symmetric},$ $\varphi(x)=0\mbox{ for }\|x\|\geq
1,\int_{\mathbb{R}^{d}}\varphi(x)dx=1,$
$\int_{\mathbb{R}^{d}}x^{\alpha}\varphi(x)dx=0\mbox{ for }1\leq|\alpha|\leq
k,\int_{\mathbb{R}^{d}}|\varphi(x)|dx<1+\frac{1}{k}\\}$

and the internal subsets of ${}^{*}{\cal D}(\mathbb{R}^{d})$:

${\cal
A}_{k}=\\{\varphi\in\,^{*}(\overline{A}_{k}):|\ln\,\rho|^{-1}\left(\sup_{x\in\,^{*}\mathbb{R}^{d}}|\partial^{\alpha}(^{*}\\!\varphi(x))|\right)<\frac{1}{k}\mbox{
for }|\alpha|\leq k\\}.$

Obviously, we have
$\overline{A}_{1}\supset\overline{A}_{2}\supset\overline{A}_{3}\supset\dots$
and ${\cal A}_{1}\supset{\cal A}_{2}\supset{\cal A}_{3}\supset\dots$. Also we
have $\overline{A}_{k}\neq 0$ for all $k\in\mathbb{N}$ by Lemma 2.2. On the
other hand, we have $\overline{A}_{k}\subset{\cal A}_{k}$ in the sense that
$\varphi\in\overline{A}_{k}$ implies ${}^{*}\\!\varphi\in{\cal A}_{k}$, since

$\sup_{x\in\,^{\star}\mathbb{R}^{d}}|\partial^{\alpha}(^{*}\\!\varphi(x))|=\sup_{x\in\,\mathbb{R}^{d}}|\partial^{\alpha}\varphi(x)|=\sup_{x\leq
1}|\partial^{\alpha}(^{*}\\!\varphi(x))|,$

is a real (standard) number and, hence,
$\displaystyle|\ln\,\rho|^{-1}\left(\sup_{x\in\,^{\star}\mathbb{R}^{d}}|\partial^{\alpha}(^{*}\\!\varphi(x))|\right)$
is infinitesimal. Thus, we have ${\cal A}_{k}\neq\emptyset$ for all $k$ in
$\mathbb{N}$. By the Saturation Principle (Todorov [6], p. 687), the
intersection $\displaystyle{\cal A}=\bigcap_{k\in\mathbb{N}}{\cal A}_{k}$ is
non-empty and thus, any $\theta$ in ${\cal A}$ has the desired properties.
$\square$

Definition 3.2 ($\rho$-Delta Function). Let $\rho$ be a positive
infinitesimal. A nonstandard function $D$ in ${}^{*}{\cal D}(\mathbb{R}^{d})$
is called a $\rho$-delta function if it takes values in ${}^{*}\mathbb{R}$, it
is symmetric and it satisfies the following properties:

(i) $D(x)=0$ for $x\in\,^{*}\mathbb{R}^{d},\|x\|\geq\rho$;

(ii) $\int_{{}^{\star}\mathbb{R}^{d}}D(x)\,dx=1$,

(iii) $\int_{{}^{\star}\mathbb{R}^{d}}D(x)x^{\alpha}\,dx=0$, for all
$\alpha\in\mathbb{N}_{0}^{d}$, $\alpha\neq 0$,

(iv) $\int_{{}^{\star}\mathbb{R}^{d}}|D(x)|\,dx\approx 1$,

(v)
$\displaystyle|\ln\rho|^{-1}\left(\rho^{d+|\alpha|}\sup_{x\in{{}^{*}\mathbb{R}^{d}}}|\partial^{\alpha}D(x)|\right)\approx
0$ for all $\alpha\in\mathbb{N}_{0}^{d}$;

Theorem 3.3 (Existence). For any positive infinitesimal $\rho$ in
${}^{*}\mathbb{R}$ there exists a $\rho$-delta function.

Proof. Let $\theta$ be a nonstandard $\rho$-mollifier of the type described in
Lemma 3.1. Then the nonstandard function $D$ in ${}^{*}{\cal
D}(\mathbb{R}^{d})$, defined by

$D(x)=\rho^{-d}\,\theta(x/\rho),\ \ \ x\in\,^{*}\mathbb{R}^{d}$ $None$

satisfies (i)-(v). $\square$

Remark. The existence of nonstandard functions $D$ in ${}^{*}{\cal
D}(\mathbb{R}^{d})$ with the above properties is in sharp contrast with the
situation in standard analysis where there is no $D$ in ${\cal
D}(\mathbb{R}^{d})$ which satisfies both (ii) and (iii). Indeed, if we assume
that $D$ in ${\cal D}(\mathbb{R}^{d})$, then (iii) implies
$\widehat{D}^{(n)}(0)=0$, for all $n=1,2,\dots$, where $\widehat{D}$ denotes
the Fourier transform of $D$. It follows $\widehat{D}=\widehat{D}(0)=c$ for
some constant $c$, since $\widehat{D}$ is an entire function on
$\mathbb{C}^{d}$, by the Paley-Wiener Theorem (Bremermann [12], Theorem 8.28,
p. 97). On the other hand, $D\in{\cal D}(\mathbb{R}^{d})\subset
S(\mathbb{R}^{d})$ implies $\widehat{D}|\mathbb{R}^{d}\in S(\mathbb{R}^{d})$
since $S(\mathbb{R}^{d})$ is closed under Fourier transform. Thus, it follows
$c=0$, i.e. $\widehat{D}=0$ which implies $D=0$ contradicting (ii).

For other classes of nonstandard delta functions we refer to (Robinson [15],
p. 133) and to (Todorov [16]).

Our next task is to show the existence of an internal cut-off function.

Notations. Let $\Omega$ be an open set of $\mathbb{R}^{d}$.

1) For any $\varepsilon\in\mathbb{R}_{+}$, we define

$B_{\varepsilon}=\\{x\in\mathbb{R}^{d}:\|x\|\leq\varepsilon\\}\mbox{ and
}\Omega_{\varepsilon}=\\{x\in\Omega:d(x,\partial\Omega)\geq\varepsilon\\},$

where $\|x\|$ is the Euclidean norm in $\mathbb{R}^{d}$, $\partial\Omega$ is
the boundary of $\Omega$ and $d(x,\partial\Omega)$ is the Euclidean distance
between $x$ and $\partial\Omega$. We also denote:

${\cal D}_{\varepsilon}(\Omega)=\\{\varphi\in{\cal
D}(\Omega):\mbox{supp}\,\varphi\subseteq B_{\varepsilon}\\},\ \ \
\mathcal{E}^{\prime}_{\varepsilon}(\Omega)=\\{T\in\mathcal{E}^{\prime}(\Omega):\mbox{supp}\,T\subseteq\Omega_{\varepsilon}\\}.$

2) We shall use the same notation, $\star$, for the convolution operator
$\star\colon{\cal D}^{\prime}(\mathbb{R}^{d})\times{\cal
D}(\mathbb{R}^{d})\to\mathcal{E}(\mathbb{R}^{d})$ (Vladimirov [13]) and its
nonstandard extension $\star\colon^{*}{\cal
D}^{\prime}(\mathbb{R}^{d})\times{{}^{*}{\cal
D}}(\mathbb{R}^{d})\to\,^{*}\mathcal{E}(\mathbb{R}^{d})$ as well as for the
convolution operator
$\star\colon\mathcal{E}^{\prime}_{\varepsilon}(\Omega)\times{\cal
D}_{\varepsilon}(\Omega)\to{\cal D}(\Omega)$, defined for all sufficiently
small $\varepsilon\in\mathbb{R}_{+}$, and for its nonstandard extension:
$\star\colon^{*}\mathcal{E}^{\prime}_{\varepsilon}(\Omega)\times\,^{*}{\cal
D}_{\varepsilon}(\Omega)\to\,^{*}{\cal D}(\Omega)$,
$\varepsilon\in\,^{*}\mathbb{R}_{+}$, $\varepsilon\approx 0$. We shall use the
same notations, $||\cdot||$ and $d(\cdot,\cdot)$, for the Euclidean norm and
distance and their nonstandard extensions ${}^{*}||\cdot||$ and
${}^{*}d(\cdot,\cdot)$, respectively.

3) Let $\tau$ be the usual Euclidean topology on $\mathbb{R}^{d}$. We denote
by $\widetilde{\Omega}$ the set of the nearstandard points in ${}^{*}\Omega$,
i.e.

$\widetilde{\Omega}=\bigcup_{x\in\Omega}\mu(x),$ $None$

where $\mu(x)$, $x\in\mathbb{R}^{d}$, is the system of monads of the
topological space $(\mathbb{R}^{d},\tau)$ (Todorov [6], p. 687). Recall that
if $\xi\in\,{{}^{*}\Omega}$, then $\xi\in\widetilde{\Omega}$ if and only if
$\xi$ is a finite point whose standard part belongs to $\Omega$.

Lemma 3.4. For any positive infinitesimal $\rho$ in ${}^{*}\mathbb{R}$ there
exists a function $\Pi$ in ${}^{*}{\cal D}(\Omega)$ (a $\rho$-cut-off
function) such that:

a) $\Pi(x)=1$ for all $x\in\widetilde{\Omega}$;

b) $\mbox{supp}\,\Pi\subseteq\Omega_{\rho}$, where
$\Omega_{\rho}=\\{\xi\in\,^{*}\Omega:\,d(\xi,\partial\Omega)\geq\rho\\}$.

Proof. Let $\rho$ be a positive infinitesimal in ${}^{*}\mathbb{R}$ and $D$ be
a $\rho$-delta function. Define the internal set
$X=\\{\xi\in\,^{*}\Omega:\,\|\xi\|\leq 1/\rho,\,d(\xi,\partial\Omega)\geq
2\rho\\}$ and let $\chi$ be its characteristic function. Then the function
$\Pi=\chi\star D$ has the desired property. $\square$

##  4 The Algebra of Asymptotic Functions

We define and study the algebra ${}^{\rho}\mathcal{E}(\Omega)$ of asymptotic
functions on an open set $\Omega$ of $\mathbb{R}^{d}$. The construction of the
algebra ${}^{\rho}\mathcal{E}(\Omega)$, presented here, is a generalization
and a refinement of the constructions in [9] and [10] (by the authors of this
paper, respectively), where the algebra ${}^{\rho}\mathcal{E}(\mathbb{R}^{d})$
was introduced by somewhat different but equivalent definitions. On the other
hand, the algebra of asymptotic functions ${}^{\rho}\mathcal{E}(\Omega)$ is
somewhat similar to but different from the Colombeau algebras of new
generalized functions (Colombeau [7], [8]). This essential difference between
${}^{\rho}\mathcal{E}(\Omega)$ and Colombeau’s algebras of generalized
functions is the properties of the generalized scalars: the scalars of the
algebra ${}^{\rho}\mathcal{E}(\Omega)$ constitutes an algebraically closed
field (as any scalars should do) while the scalars of Colombeau’s algebras are
rings with zero divisors (Colombeau [8], §2.1). This improvement compared with
J. F. Colombeau’s theory is due to the involvement of the nonstandard
analysis.

Let $\Omega$ be an open set of $\mathbb{R}^{d}$ and
$\rho\in{{}^{*}\mathbb{R}}$ be a positive infinitesimal. We shall keep
$\Omega$ and $\rho$ fixed in what follows. Following (Robinson [2]), we
define:

Definition 4.1 (Robinson’s Asymptotic Numbers). The field of the complex
Robinson $\rho$-asymptotic numbers is defined as the factor space
${}^{\rho}\mathbb{C}=\mathbb{C}_{\mathcal{M}}/\mathbb{C}_{0}$, where

$\displaystyle\mathbb{C}_{\mathcal{M}}=\\{\xi\in{{}^{*}\mathbb{C}}:|\xi|<\rho^{-n}\mbox{
for some }n\in\mathbb{N}\\},$
$\displaystyle\mathbb{C}_{0}=\\{\xi\in{{}^{*}\mathbb{C}}:|\xi|<\rho^{n}\mbox{
for all }n\in\mathbb{N}\\},$ (2)

($\mathcal{M}$ stands for moderate). We define the embedding
$\mathbb{C}\subset{{}^{\rho}\mathbb{C}}$ by $c\to q(c)$, where
$q\colon\mathbb{C}_{\mathcal{M}}\to{{}^{\rho}\mathbb{C}}$ is the quotient
mapping. The field of the real asymptotic numbers is defined by
${}^{\rho}\mathbb{R}=q(^{*}\mathbb{R}\cap\mathbb{C}_{\mathcal{M}})$.

It is easy to check that $\mathbb{C}_{0}$ is a maximal ideal in
$\mathbb{C}_{\mathcal{M}}$ and hence ${}^{\rho}\mathbb{C}$ is a field. Also
${}^{\rho}\mathbb{R}$ is a real closed totally ordered nonarchimedean field
(since ${}^{*}\mathbb{R}$ is a real closed totally ordered field) containing
$\mathbb{R}$ as a totally ordered subfield. Thus, it follows that
${}^{\rho}\mathbb{C}={{}^{\rho}\mathbb{R}}(i)$ is an algebraically closed
field, where $i=\sqrt{-1}$.

The algebra of asymptotic functions is, in a sense, a
$\mathcal{C}^{\infty}$-counterpart of A. Robinson’s asymptotic numbers
${}^{\rho}\mathbb{C}$:

Definition 4.2 (Asymptotic Functions on $\Omega$). (i) We define the class
${}^{\rho}\mathcal{E}(\Omega)$ of the $\rho$-asymptotic functions on $\Omega$
(or simply, asymptotic functions on $\Omega$ if no confusion could arise) as
the factor space
${}^{\rho}\mathcal{E}(\Omega)=\mathcal{E}_{\mathcal{M}}(\Omega)/\mathcal{E}_{0}(\Omega)$,
where

$\mathcal{E}_{\mathcal{M}}(\Omega)=\\{f\in\,{{}^{*}\mathcal{E}(\Omega)}:\partial^{\alpha}f(\xi)\in\mathbb{C}_{\mathcal{M}}\mbox{
for all }\alpha\in\mathbb{N}_{0}^{d}\mbox{ and all
}\xi\in\widetilde{\Omega}\\},$
$\mathcal{E}_{0}(\Omega)=\\{f\in{{}^{*}\mathcal{E}(\Omega)}:\partial^{\alpha}f(\xi)\in\mathbb{C}_{0}\mbox{
for all }\alpha\in\mathbb{N}_{0}^{d}\mbox{ and all
}\xi\in\widetilde{\Omega}\\},$

and $\widetilde{\Omega}$ is the set of the nearstandard points of
${}^{*}\Omega$ (3.2). The functions in $\mathcal{E}_{\mathcal{M}}(\Omega)$ are
called $\rho$-moderate (or, simply, moderate) and those in
$\mathcal{E}_{0}(\Omega)$ are called $\rho$-null functions (or, simply, null
functions).

(ii) The pairing between ${}^{\rho}{\mathcal{E}}(\Omega)$ and
${\mathcal{D}}(\Omega)$ with values in ${}^{\rho}\mathbb{C}$, is defined by

$\langle
Q_{\Omega}(f),\varphi\rangle=q\left(\int_{{}^{*}\Omega}f(x)^{*}\\!\varphi(x)\,dx\right),$

where $q\colon\mathbb{C}_{\mathcal{M}}\to\,^{\rho}\mathbb{C}$ and
$Q_{\Omega}\colon\mathcal{E}_{\mathcal{M}}(\Omega)\to\,^{\rho}\mathcal{E}(\Omega)$
are the corresponding quotient mappings, $\varphi$ is in ${\cal D}(\Omega)$
and ${}^{*}\\!\varphi$ is its nonstandard extension of $\varphi$.

(iii) We define the canonical embedding
$\mathcal{E}(\Omega)\subset{{}^{\rho}\mathcal{E}(\Omega)}$ the mapping
$\sigma_{\Omega}\colon f\to Q_{\Omega}(^{*}\\!f)$, where ${}^{*}\\!f$ is the
nonstandard extension of $f$.

Example 4.3. Let $D$ be a nonstandard $\rho$-delta function in the sense of
Definition 3.2. Then $D\in\mathcal{E}_{\mathcal{M}}(\mathbb{R}^{d})$. In
addition, $D|{{}^{*}\Omega\in\mathcal{E}_{\mathcal{M}}}(\Omega)$, where
$D|^{*}\Omega$ denotes the pointwise restriction of $D$ on ${}^{*}\Omega$. To
show this, denote
$\displaystyle|\ln\rho|^{-1}\left(\rho^{d+|\alpha|}\sup_{x\in{{}^{*}\mathbb{R}^{d}}}|\partial^{\alpha}D(x)|\right)=h_{\alpha}$
and observe that $h_{\alpha}\approx 0$ for all $\alpha\in\mathbb{N}_{0}^{d}$
by the definition of $D$. Thus, for any (finite) $x$ in ${}^{*}\mathbb{R}^{d}$
and any $\alpha\in\mathbb{N}_{0}^{d}$ we have
$|\partial^{\alpha}D(x)|\leq\sup_{x\in{{}^{*}\mathbb{R}}}|\partial^{\alpha}D(x)|=\frac{h_{\alpha}|\ln\rho|}{\rho^{d+|\alpha|}}<\rho^{-n}$,
for $n=d+|\alpha|+1$, thus, $D\in\mathcal{E}_{\mathcal{M}}(\mathbb{R}^{d})$.
On the other hand, $D|{{}^{*}\Omega}\in\mathcal{E}_{\mathcal{M}}(\Omega)$
follows immediately from the fact that $\widetilde{\Omega}$ consists of finite
points in ${}^{*}\mathbb{R}^{d}$ only.

Theorem 4.4 (Differential Algebra). (i) The class of asymptotic functions
${}^{\rho}\mathcal{E}(\Omega)$ is a differential algebra over the field of the
complex asymptotic numbers ${}^{\rho}\mathbb{C}$.

(ii) $\mathcal{E}(\Omega)$ is a differential subalgebra of
${}^{\rho}\mathcal{E}(\Omega)$ over the scalars $\mathbb{C}$ under the
canonical embedding $\sigma_{\Omega}$. In addition, $\sigma_{\Omega}$
preserves the pairing in the sense that $\langle
f,\varphi\rangle=\langle\sigma_{\Omega}(f),\varphi\rangle$ for all $f$ in
$\mathcal{E}(\Omega)$ and for all $\varphi$ in ${\cal D}(\Omega)$, where
$\langle f,\varphi\rangle=\int_{\Omega}f(x)\varphi(x)\,dx$ is the usual
pairing between $\mathcal{E}(\Omega)$ and ${\cal D}(\Omega)$.

Proof. (i) It is clear that $\mathcal{E}_{\mathcal{M}}(\Omega)$ is a
differential ring and $\mathcal{E}_{0}(\Omega)$ is a differential ideal in
$\mathcal{E}_{\mathcal{M}}(\Omega)$ since $\mathbb{C}_{\mathcal{M}}$ is a ring
and $\mathbb{C}_{0}$ is an ideal in $\mathbb{C}_{\mathcal{M}}$ and, on the
other hand, both $\mathcal{E}_{\mathcal{M}}(\Omega)$ and
$\mathcal{E}_{0}(\Omega)$ are closed under differentiation, by definition.
Hence, the factor space ${}^{\rho}\mathcal{E}(\Omega)$ is also a differential
ring. It is clear that, $\mathcal{E}_{\mathcal{M}}(\Omega)$ is a module over
the ring $\mathbb{C}_{\mathcal{M}}$, and, in addition, the annihilator
$\\{c\in\mathbb{C}_{\mathcal{M}}:(\forall
f\in\mathcal{E}_{\mathcal{M}}(\Omega))(cf\in\mathcal{E}_{0}(\Omega))\\}$ of
$\mathbb{C}_{\mathcal{M}}$ coincides with the ideal $\mathbb{C}_{0}$. Thus,
${}^{\rho}\mathcal{E}(\Omega)$ becomes an algebra over the field of the
complex asymptotic numbers ${}^{\rho}\mathbb{C}$.

(ii) Assume that $\sigma_{\Omega}(^{*}\\!f)=0$ in
${}^{\rho}\mathcal{E}(\Omega)$, i.e. ${}^{*}\\!f\in\mathcal{E}_{0}(\Omega)$.
By the definition of $\mathcal{E}_{0}(\Omega)$ (applied for $\alpha=0$ and
$n=1$), it follows $f=0$ since ${}^{*}\\!f$ is an extension of $f$ and $\rho$
is an infinitesimal. Thus, the mapping $f\to\sigma_{\Omega}(f)$ is injective.
It preserves the algebraic operations since the mapping $f\to\,^{*}\\!f$
preserves them. The preserving of the pairing follows immediately from the
fact that $\int_{{}^{\star}\Omega}\,^{*}\\!f(x)\,dx=\int_{\Omega}f(x)\,dx$, by
the Transfer Principle (Todorov [6], p. 686). The proof is complete. $\square$

##  5 Embedding of Schwartz Distributions

Let $\Omega$ be (as before) an open set of $\mathbb{R}^{d}$. Recall that the
Schwartz embedding $L_{\Omega}\colon{\cal L}_{loc}(\Omega)\to{\cal
D}^{\prime}(\Omega)$ from ${\cal L}_{loc}(\Omega)$ into ${\cal
D}^{\prime}(\Omega)$ is defined by the formula:

$\langle L_{\Omega}(f),\varphi\rangle=\int_{\Omega}f(x)\varphi(x)\,dx,\ \ \
\varphi\in{\cal D}(\Omega).$ $None$

Here ${\cal L}_{loc}(\Omega)$ denotes, as usual, the space of the locally
(Lebesgue) integrable complex valued functions on $\Omega$ (Vladimirov [13]).
The Schwartz embedding $L_{\Omega}$ preserves the addition and multiplication
by a complex number, hence, the space ${\cal L}_{loc}(\Omega)$ can be
considered as a linear subspace of ${\cal D}^{\prime}(\Omega)$. In addition,
the restriction $L_{\Omega}|\mathcal{E}(\Omega)$ of $L_{\Omega}$ on
$\mathcal{E}(\Omega)$ (often denoted also by $L_{\Omega}$) preserves the
partial differentiation of any order and in this sense $\mathcal{E}(\Omega)$
is a differential linear subspace of ${\cal D}^{\prime}(\Omega)$. In short, we
have the chain of linear embeddings: $\mathcal{E}(\Omega)\subset{\cal
L}_{loc}(\Omega)\subset{\cal D}^{\prime}(\Omega)$.

The purpose of this section is to show that the algebra of asymptotic
functions ${}^{\rho}\mathcal{E}(\Omega)$ contains an isomorphic copy of the
space of Schwartz distributions ${\cal D}^{\prime}(\Omega)$ and, hence, to
offer a solution of the Problem of Multiplication of Schwartz Distributions.
This result is a generalization of some results in [9] and [10] (by the
authors of this paper, respectively) where only the embedding of the tempered
distributions $S^{\prime}(\mathbb{R}^{d})$ in
${}^{\rho}\mathcal{E}(\mathbb{R}^{d})$ has been established. The embedding of
all distributions ${\cal D}^{\prime}(\Omega)$ discussed here, presents an
essentially different situation.

The spaces $\widetilde{\mathcal{E}}(\Omega)$ and $\widetilde{{\cal
D}}(\Omega)$, defined below, are immediate generalizations of the spaces
$\widetilde{\mathcal{E}}(\mathbb{R}^{d})$ and $\widetilde{{\cal
D}}(\mathbb{R}^{d})$, introduced in (Stroyan and Luxemburg [17], (10.4), p.
299):

$\widetilde{\mathcal{E}}(\Omega)=\\{\varphi\in{{}^{*}\mathcal{E}}(\Omega):\partial^{\alpha}\varphi(x)\mbox{
is a finite number in }{{}^{*}\mathbb{C}}\mbox{ for all
}x\in\widetilde{\Omega}\mbox{ and all }\alpha\in\mathbb{N}_{0}^{d}\\},$ $None$
$\widetilde{{\cal
D}}(\Omega)=\\{\varphi\in{{}^{*}\mathcal{E}(\Omega)}:\partial^{\alpha}\varphi(x)\mbox{
is a finite number in }^{*}\mathbb{C}\mbox{ for all }$
$x\in\widetilde{\Omega},\alpha\in\mathbb{N}_{0}^{d}\mbox{ and
}\varphi(x)=0\mbox{ for all
}x\in{{}^{*}\Omega}\backslash\widetilde{\Omega}\\},$ $None$

Obviously, we have $\widetilde{{\cal
D}}(\Omega)\subset\widetilde{{\mathcal{E}}}(\Omega)\subset\mathcal{E}_{\mathcal{M}}(\Omega)$.
Notice as well that $\varphi\in\widetilde{{\cal D}}(\Omega)$ implies
$\varphi\in{{}^{*}{\cal D}}(G)$ for some open relatively compact set $G$ of
$\Omega$. We have also the following simple result:

Lemma 5.1. If $T\in{\cal D}^{\prime}(\Omega)$ and
$\varphi\in\mathcal{E}_{0}(\Omega)\cap\,\widetilde{{\cal D}}(G)$, then
$\langle{{}^{*}T},\varphi\rangle\in\mathbb{C}_{0}$.

Proof. Observe that $\mathcal{E}_{0}(\Omega)\cap\widetilde{{\cal D}}(\Omega)$
implies $\varphi\in\mathcal{E}_{0}(\Omega)\cap{{}^{*}{\cal D}(G)}$ for some
open relatively compact set $G$ of $\Omega$. By the continuity of $T$ (and
Transfer Principle) there exist constants $C\in\mathbb{R}_{+}$, and
$m\in\mathbb{N}_{0}$ such that

$|\langle{{}^{*}T},\varphi\rangle|\leq C\sum_{|\mu|\leq
m}\sup_{x\in{{}^{*}G}}|\partial^{\mu}\varphi(x)|$

On the other hand, $\displaystyle C\sum_{|\mu\leq
m}\sup_{x\in^{\star}G}|\partial^{\mu}\varphi(x)|<\rho^{n}$ for all
$n\in\mathbb{N}$, since $\varphi\in\mathcal{E}_{0}(\Omega)$, by assumption.
Thus, $|\langle{{}^{*}T},\varphi\rangle|<\rho^{n}$ for all $n\in\mathbb{N}$.
$\square$

Let $D$ be a $\rho$-delta function in the sense of Definition 3.2. We shall
keep $D$ (along with $\Omega$ and $\rho$) fixed in what follows.

Definition 5.2 (Embedding of Schwartz Distributions). We define the embedding
${\cal D}^{\prime}(\Omega)\subset\,^{\rho}\mathcal{E}(\Omega)$ by
$\Sigma_{D,\Omega}\colon T\to Q_{\Omega}((^{*}T\,\Pi_{\Omega})\star D)$, where
${}^{*}T$ is the nonstandard extension of $T$, $\Pi_{\Omega}$ is a (an
arbitrarily chosen) $\rho$-cut-off function for $\Omega$ (Lemma 3.4),
${}^{*}T\,\Pi_{\Omega}$ is the Schwartz product between ${}^{*}T$ and
$\Pi_{\Omega}$ in ${}^{*}{\cal D}^{\prime}(\Omega)$ (applied by Transfer
Principle), $\star$ is the convolution operator and
$Q_{\Omega}\colon\mathcal{E}_{\mathcal{M}}(\Omega)\to\,^{\rho}\mathcal{E}(\Omega)$
is the quotient mapping in the definition of ${}^{\rho}\mathcal{E}(\Omega)$
(Definition 4.2).

The cut-off function $\Pi_{\Omega}$ can be dropped in the above definition,
i.e. $\Sigma_{D,\Omega}\colon T\to Q_{\Omega}({{}^{*}T}\star D)$, in some
particular cases; e.g. when:

a) $T$ has a compact support in $\Omega$;

b) $\Omega=\mathbb{R}^{d}$.

Proposition 5.3 (Correctness). $T\in{\cal D}^{\prime}(\Omega)$ implies
$({{}^{*}T}\,\Pi_{\Omega})\star D\in\mathcal{E}_{\mathcal{M}}(\Omega)$.

Proof. Choose $\alpha\in\mathbb{N}_{0}^{d}$ and $x\in\widetilde{\Omega}$.
Since we have $\partial^{\alpha}(({{}^{*}T}\,\Pi_{\Omega})\star
D)(x)=(\partial^{\alpha}(^{*}T)\star D)(x)$ (by the definition of
$\Pi_{\Omega}$), we need to show that $\partial^{\alpha}({{}^{*}T}\star
D)(x)\in\mathbb{C}_{\mathcal{M}}$ only, i.e. that
$|\partial^{\alpha}(^{*}T)\star D)(x)|<p^{-m}$ for some $m\in\mathbb{N}$ ($m$
might depend on $\alpha$). We start with the case $\alpha=0$. Denote
$D_{x}(\xi)=D(\xi-x)$, $\xi\in{{}^{*}\mathbb{R}}$ and observe that
$\mbox{supp}(D_{x})\subseteq{{}^{*}G}$ for some open relatively compact set
$G$ of $\Omega$, since $D_{x}$, vanishes on
${}^{*}\Omega\setminus\widetilde{\Omega}$. Next, by the continuity of $T$ (and
the Transfer Principle), there exist constants $m\in\mathbb{N}_{0}$ and
$C\in\mathbb{R}_{+}$, such that

$|({{}^{*}T}\star D)(x)|=|\langle^{*}T,D_{x}|^{*}\Omega\rangle|\leq
C\sum_{|\mu|\leq m}\sup_{\xi\in{{}^{*}G}}|\partial_{\xi}^{\mu}D(x-\xi)|$

Finally, there exists $n\in N$ such that $\displaystyle\sum_{|\mu|\leq
m}\sup_{\xi\in^{\star}G}|\partial_{\xi}^{\mu}D(x-\xi)|<p^{-n}$, since
$D|^{\star}G$ is a $\rho$-moderate function (Example 4.3). Combining these
arguments, we have: $|(^{*}T\star D)(x)|\leq C\rho^{-n}<\rho^{-(n+1)}$, as
required. The generalization for arbitrary multiindex $\alpha$ follows
immediately since $\partial^{\alpha}({{}^{*}T}\star
D)=(\partial^{\alpha}(^{*}T))\star D)={{}^{*}(\partial^{\alpha}}T)\star D$, by
Transfer Principle, and $\partial^{\alpha}T$ is (also) in ${\cal
D}^{\prime}(\Omega)$. $\square$

Proposition 5.4. $f\in\widetilde{\mathcal{E}}(\Omega)$ implies
$(f\Pi_{\Omega})\star D-f\in\mathcal{E}_{0}(\Omega)$.

Proof. Let $x\in\widetilde{\Omega}$ and $\alpha\in\mathbb{N}_{0}^{d}$. Since
we have $\partial^{\alpha}[((f\Pi_{\Omega})\star
D)(x)-f(x)]=\partial^{\alpha}[(f\star D)(x)-f(x)]$ (by the definition of
$\Pi_{\Omega}$), we need to show that $\partial^{\alpha}[(f\star
D)(x)-f(x)]\in\mathbb{C}_{0}$ only. Choose $n\in\mathbb{N}$. We need to show
that $|\partial^{\alpha}[(f\star D)(x)-f(x)]|<\rho^{n}$. We start first with
the case $\alpha=0$. By Taylor’s formula (applied by transfer), we have

$f(x-\xi)-f(x)=\sum_{|\beta|=1}^{n}\frac{(-1)^{|\beta|}\partial^{\beta}f(x)}{\beta!}\,\xi^{\beta}+\frac{(-1)^{n+1}}{(n+1)!}\sum_{|\beta|=n+1}\partial^{\beta}f(\eta(\xi))\,\xi^{\beta}$

for any $\xi\in\widetilde{\Omega}$, where $\eta(\xi)$ is a point in
${}^{*}\Omega$ “between $x$ and $\xi$.”. Notice that the point $\eta(\xi)$ is
also in $\widetilde{\Omega}$. It follows

$(f\star
D)(x)-f(x)=\int_{\|\xi\|\leq\rho}D(\xi)[f(x-\xi)-f(x)]\,d\xi=\frac{(-1)^{n+1}}{(n+1)!}\sum_{|\beta|=n+1}\int_{\|\xi\|\leq\rho}D(\xi)\xi^{\beta}\partial^{\beta}f(\eta(\xi))\,d\xi$

since $\int_{\|\xi\|\leq\rho}D(\xi)\xi^{\beta}\,d\xi=0$, by the definition of
$D$. Thus, we have

$|(f\star
D)(x)-f(x)|\leq\frac{\rho^{n+1}}{(n+1)!}\left(\int_{{}^{\star}\mathbb{R}^{d}}|D(x)|\,dx\right)\left(\sum_{|\beta|=n+1}\sup_{\|\xi\|\leq\rho}|\partial^{\beta}f(\eta(\xi))|\right)<\rho^{n},$

as desired, since, on one hand,
$\int_{{}^{\star}\mathbb{R}^{d}}|D(x)|\,dx\approx 1$, by the definition of $D$
and on the other hand, the above sum is a finite number because
${}^{*}\partial^{\beta}f(\eta(\xi))$ are all finite due to
$\eta(\xi)\in\widetilde{\Omega}$. The generalization for an arbitrary $\alpha$
is immediate since $\partial^{\alpha}[(f\star
D)(x)-f(x)]=(\partial^{\alpha}f\star D)(x)-\partial^{\alpha}f(x)$, by the
Transfer Principle. $\square$

Corollary 5.5. (i) $f\in\mathcal{E}(\Omega)$ implies
$(^{*}f\,\Pi_{\Omega})\star D-\,^{*}\\!f\in\mathcal{E}_{0}(\Omega)$.

(ii) $\varphi\in{\cal D}(\Omega)$ implies
$({{}^{*}\\!\varphi}\,\Pi_{\Omega})\star
D-\,{{}^{*}\\!\varphi}\in\mathcal{E}_{0}(\Omega)\cap\widetilde{{\cal
D}}(\Omega)$.

Proof. (i) follows immediately from the above proposition since
$f\in\mathcal{E}(\Omega)$ implies
${}^{*}\\!f\in\widetilde{\mathcal{E}}(\Omega)$. (ii) Both
$(^{*}\\!\varphi\,\Pi_{\Omega})\star D$ and ${}^{*}\\!\varphi$ vanish on
${}^{*}\Omega\setminus\widetilde{\Omega}$ since their supports are within an
open relatively compact neighborhood $G$ of $\mbox{supp}(\varphi)$ and the
latter is a compact set of $\Omega$, by assumption. Thus,
$(^{*}\\!\varphi\,\Pi_{\Omega})\star
D-{{}^{*}\\!\varphi}\in{{}^{*}\mathcal{D}(G)}\subset\tilde{\mathcal{D}}(\Omega)$
as required. $\square$

Denote $\check{D}(x)=D(-x)$ and recall that $\check{D}=D$ since $D$ is
symmetric (Definition 3.2).

Proposition 5.6. If $T\in{\cal D}^{\prime}(\Omega)$ and $\varphi\in{\cal
D}(\Omega)$, then

$\int_{{}^{*}\Omega}(({{}^{*}T}\,\Pi_{\Omega})\star
D)(x)\,{{}^{*}\\!\varphi}(x)\,dx-\langle T,\varphi\rangle\in\mathbb{C}_{0}$

Proof. Using the properties of the convolution operator (applied by transfer),
we have

$\int_{{}^{*}\Omega}(({{}^{*}T}\,\Pi_{\Omega})\star
D)(x)\,{{}^{*}\\!}\varphi(x)\,dx-\langle T,\varphi\rangle$
$=\langle(^{*}T\,\Pi_{\Omega})\star
D,{{}^{*}\\!\varphi}\,\rangle-\langle{{}^{*}T},{{}^{*}\\!\varphi}\rangle=\langle{{}^{*}T}\,\Pi_{\Omega},{{}^{*}\\!\varphi}\star\check{D}\rangle-\langle{{}^{*}T}\,\Pi_{\Omega},{{}^{*}\\!\varphi}\rangle$
$=\langle{{}^{*}T}\,\Pi_{\Omega},\,{{}^{*}\\!\varphi}\star\check{D}-\,{{}^{*}\\!\varphi}\rangle=\langle^{*}T,\,{{}^{*}\\!\varphi}\star
D-\,{{}^{*}\\!\varphi}\rangle\in\mathbb{C}_{0},$

by Lemma 5.1 since ${}^{*}\\!\varphi\star
D-\,{{}^{*}\\!\varphi}\in\mathcal{E}_{0}(\Omega)\cap\widetilde{\mathcal{D}}(\Omega)$,
by Corollary 5.5. $\square$

We are ready to state our main result:

Theorem 5.7 (Properties of $\Sigma_{D,\Omega}$). (i) $\Sigma_{D,\Omega}$
preserves the pairing in the sense that for all $T$ in ${\cal
D}^{\prime}(\Omega)$ and all $\varphi$ in ${\cal D}(\Omega)$ we have $\langle
T,\varphi\rangle=\langle\Sigma_{D,\Omega}(T),\varphi\rangle$, where the left
hand side is the (usual) pairing of $T$ and $\varphi$ in ${\cal
D}^{\prime}(\Omega)$, while the right hand side is the pairing of
$\Sigma_{D,\Omega}(T)$ and $\varphi$ in ${}^{\rho}\mathcal{E}(\Omega)$
(Definition 4.2).

(ii) $\Sigma_{D,\Omega}$ is injective and it preservers all linear operations
in ${\cal D}^{\prime}(\Omega)$: the addition, multiplication by (standard)
complex numbers and the partial differentiation of any (standard) order.

(iii) $\Sigma_{D,\Omega}$ is an extension of the canonical embedding
$\sigma_{\Omega}$ defined earlier in Definition 4.2 in the sense that
$\sigma_{\Omega}=\Sigma_{D,\Omega}\circ L_{\Omega}$, where $L_{\Omega}$ is the
Schwartz embedding (5.1) restricted on $\mathcal{E}(\Omega)$ and $\circ$
denotes composition. Or, equivalently, the following diagram is commutative:

${\mathcal{E}(\Omega)}$${\mathcal{D}^{\prime}(\Omega)}$${{{}^{\rho}\mathcal{E}(\Omega)}.}$$\scriptstyle{L_{\Omega}}$$\scriptstyle{\sigma_{\Omega}}$$\scriptstyle{\Sigma_{D,\Omega}}$

Proof. (i) Denote (as before) $\check{D}(x)=D(-x)$ and recall that
$\check{D}=D$ (Definition 3.2). We

have

$\displaystyle\langle\Sigma_{D,\Omega}(T),\varphi\rangle$ $\displaystyle=$
$\displaystyle\langle Q_{\Omega}((^{*}T\,\Pi_{\Omega})\star
D),\varphi\rangle-\langle T,\varphi\rangle$ $\displaystyle=$ $\displaystyle
q\left(\int_{{}^{*}\Omega}((\Pi_{\Omega}\,{{}^{*}T})\star
D)(x)\,{{}^{*}\\!\varphi}(x)\,dx\right)-q(\langle T,\varphi\rangle)$
$\displaystyle=$ $\displaystyle
q\left(\int_{{}^{*}\Omega}((\Pi_{\Omega}\,{{}^{*}T})\star
D)(x)\,{{}^{*}\\!\varphi}(x)-\langle T,\varphi\rangle\right)=0,$

because $\int_{{}^{*}\Omega}((^{*}T\,\Pi_{\Omega})\star
D)(x)\,{{}^{*}\\!\varphi}(x)\,dx-\langle T,\varphi\rangle\in\mathbb{C}_{0}$,
by Proposition 5.6. Here $\langle T,\varphi\rangle=q(\langle
T,\varphi\rangle)$ holds because $\langle T,\varphi\rangle$ is a standard
(complex) number.

(ii) The injectivity of $\Sigma_{D,\Omega}$ follows from (i):
$\Sigma_{D,\Omega}(T)=0$ in ${}^{\rho}\mathcal{E}(\Omega)$ implies
$\langle\Sigma_{D,\Omega}(T),\varphi\rangle=0$ for all $\varphi\in{\cal
D}(\Omega)$, which is equivalent to $\langle T,\varphi\rangle=0$ for all
$\varphi\in{\cal D}(\Omega)$, by (i), thus, $T=0$ in ${\cal
D}^{\prime}(\Omega)$, as required. The preserving of the linear operations
follows from the fact that both the extension mapping ∗ and the convolution
$\star$ (applied by Transfer Principle) are linear operators.

(iii) For any $f\in\mathcal{E}(\Omega)$ we have
$\sigma(f)=Q_{\Omega}(^{*}\\!f)=Q_{\Omega}((^{*}\\!f\,\Pi_{\Omega})\star
D)=Q_{\Omega}((^{*}L_{\Omega}(f)\,\Pi_{\Omega})\star
D)=\Sigma_{D,\Omega}(L_{\Omega}(f))$, as required, since
${}^{*}\\!f-({{}^{*}\\!f}\,\Pi_{\Omega})\star D\in\mathcal{E}_{0}(\Omega)$, by
Corollary 5.5. $\square$

Remark 5.8 (Multiplication of Distributions). As a consequence of the above
result, the Schwartz distributions in ${\cal D}^{\prime}(\Omega)$ can be
multiplied within the associative and commutative differential algebra
${}^{\rho}\mathcal{E}(\Omega)$ (something impossible in ${\cal
D}^{\prime}(\Omega)$ itself). By the property (iii) above, the multiplication
in ${}^{\rho}\mathcal{E}(\Omega)$ coincides on $\mathcal{E}(\Omega)$ with the
usual (pointwise) multiplication in $\mathcal{E}(\Omega)$. Thus, the class
${}^{\rho}\mathcal{E}(\Omega)$, endowed with an embedding $\Sigma_{D,\Omega}$,
presents a solution of the problem of multiplication of Schwartz distributions
which, in a sense, is optimal, in view of the Schwartz impossibility results
(Schwartz [1]) (for a discussion we refer also to (Colombeau [7], §2.4) and
(Oberguggenberg [18], §2). We should mention that the existence of an
embedding of ${\cal D}^{\prime}(\mathbb{R}^{d})$ into
${}^{\rho}\mathcal{E}(\mathbb{R}^{d})$ can be proved also by sheaf-theoretical
arguments as indicated in (Oberguggenberger [18], §23).

Remark 5.9 (Nonstandard Asymptotic Analysis). We sometimes refer to the area
connected directly or indirectly with the fields ${}^{\rho}\mathbb{R}$ as
Nonstandard Asymptotic Analysis. The fields ${}^{\rho}\mathbb{R}$ were
introduced in Robinson [2] and are sometimes known as “Robinson’s
nonarchimedean valuation fields”. The terminology “Robinson’s asymptotic
numbers”, chosen in this paper, is due to the role of ${}^{\rho}\mathbb{R}$
for the asymptotic expansions of classical functions (Lightstone and Robinson
[3]) and also to stress the fact that in our approach ${}^{\rho}\mathbb{C}$
plays the role of the scalars of the algebra ${}^{\rho}\mathcal{E}(\Omega)$.
Linear spaces over the field ${}^{\rho}\mathbb{R}$ has been studied in
Luxemburg [19] in order to establish a connection between nonstandard and
nonarchimedean analysis. More recently ${}^{\rho}\mathbb{R}$ has been used in
Pestov [20] for studying Banach spaces. The field ${}^{\rho}\mathbb{R}$ has
been exploited by Li Bang-He [21] for multiplication of Schwartz
distributions.

## References

  * [1] SCHWARTZ, L., Sur l’impossibilité de la multiplication des distributions, C.R. Acad. Sci., Paris, 239 (1954), 847-848.
  * [2] ROBINSON, A., Function theory on some nonarchimedean fields, Amer. Math. Monthly 80 (6), Part II: Papers in the Foundations of Mathematics (1973), 87-109.
  * [3] LIGHTSTONE, A.H. and ROBINSON, A., Nonarchimedean Fields and Asymptotic Expansions, North-Holland Mathematical Library, 13, North Holland, Amsterdam-Oxford/American Elsevier, New York, 1975.
  * [4] OBERGUGGENBERGER, M., Contributions of nonstandard analysis to partial differential equations, in Development in Nonstandard Mathematics (Eds. N.J. Cutland, V. Neves, F. Oliveira and J. Sousa-Pinto), Longman-Harlow, 1995, 130-150.
  * [5] TODOROV, T., An existence result for a class of partial differential equations with smooth coefficients, in Advances in Analysis, Probability and Mathematical Physics - Contributions to Nonstandard Analysis (Eds. S. Albeverio, W.AJ. Luxemburg and M.P.H. Wolff), Kluwer Academic Publishers (Mathematics and Its Applications), Dordrecht/Boston/London, Vol. 314, 1995, 107-121.
  * [6] TODOROV, T., An existence of solutions for linear partial differential equations with $C^{\infty}$\- coefficients in an algebra of generalized functions, in Trans. Am. Math. Soc. , 348, 2 (1996).
  * [7] COLOMBEAU, J.F., New Generalized Functions and Multiplication of Distributions, North- Holland Math. Studies 84, North-Holland, Amsterdam, 1984.
  * [8] COLOMBEAU, J.F., Elementary Introduction to New Generalized Functions, North-Holland, Amsterdam, 1985.
  * [9] OBERGUGGENBERGER, M., Products of distributions: nonstandard methods, Z Anal. Anwendungen 7 (1988), 347-365., Corrections: ibid. 10 (1991), 263-264.
  * [10] TODOROV, T., Colombeau’s new generalized function and non-standard analysis, in Generalised Functions, Convergence Structures and their Applications (Eds. B. Stankovic, E. Pap, S. Pilipovic and V.S. Vladimirov), Plenum Press, New York (1988), 327-339.
  * [11] HOSKINS, R.F. and PINTO, J.S., Nonstandard treatments of new generalised functions, in Generalized Functions and Their Applications (Ed. R.S. Pathak), Plenum Press, New York, 1993, 95-108.
  * [12] BREMERMANN, H., Distributions, Complex Variables, and Fourier Transforms, Addison- Wesley Publ. Co., Inc., ; Pal Alto, 1965.
  * [13] VLADIMIROV, V., Generalized Functions in Mathematical Physics, Mit-Publisher, Moscow, 1979\.
  * [14] LINDSTROM, T., An invitation to nonstandard analysis, in Nonstandard Analysis and its Applications (Ed. N. Cutland), London Mathematical Society Student Texts 10, Cambridge University Press, Cambridge, 1988, 1-105.
  * [15] ROBINSON, A., Non-Standard Analysis, North-Holland, Amsterdam, 1966.
  * [16] TODOROV, T., A nonstandard delta function, Proc. Amer. Math. Soc., 110 (1990), 1143-1144.
  * [17] STROYAN, K.D. and LUXEMBURG, W.AJ., Introduction to the Theory of Infinitesimals, Academic Press, New York, 1976.
  * [18] OBERGUGGENBERGER, M., Multiplication of distributions and applications to partial differential equations, Pitman Research Notes Math., 259, Longman, Harlow, 1992.
  * [19] LUXEMBURG, W.A.J., On a class of valuation fields introduced by A. Robinson, Israel Journal Math., 25 (1976), 189-201.
  * [20] PESTOV, V., On a valuation field invented by A. Robinson and certain structures connected with it, Israel Journal Math., 74 (1991), 65-79.
  * [21] BANG-HE, L., Non-standard analysis and multiplication of distributions, Sci. Sinica, 21 (1978), 561-585.


# Quantifying Explainability in NLP and Analyzing Algorithms for Performance-
Explainability Tradeoff

Mitchell Naylor  Christi French  Samantha Terker  Uday Kamath

###### Abstract

The healthcare domain is one of the most exciting application areas for
machine learning, but a lack of model transparency contributes to a lag in
adoption within the industry. In this work, we explore the current art of
explainability and interpretability within a case study in clinical text
classification, using a task of mortality prediction within MIMIC-III clinical
notes. We demonstrate various visualization techniques for fully interpretable
methods as well as model-agnostic _post hoc_ attributions, and we provide a
generalized method for evaluating the quality of explanations using infidelity
and local Lipschitz across model types from logistic regression to BERT
variants. With these metrics, we introduce a framework through which
practitioners and researchers can assess the frontier between a model’s
predictive performance and the quality of its available explanations. We make
our code available111Code available at
https://github.com/mnaylor5/quantifying-explainability to encourage continued
refinement of these methods.

Machine Learning, ICML, Interpretable Machine Learning, Explainable Machine
Learning, NLP, BERT, Text Classification, Explanation Quality

##  1 Introduction

Healthcare is an extremely promising area for applying machine learning
solutions – clinical support systems can aid physicians with decision making,
text-based solutions can help match patients to relevant clinical trials, and
ML systems can save lives by identifying patterns indicative of sepsis,
stroke, or cancer. However, the healthcare domain also presents immense legal
and ethical considerations, and model mistakes can cost patient lives.
Clinical stakeholders not only need to know that models have been faithfully
developed, but they also desire transparency in the output of the model so
that they can more easily diagnose model failures. A lack of explainability in
healthcare, especially clinical decision support models, has been criticized
(Shortliffe & Sepúlveda, 2018), leading to additional innovation and progress
in the development of explainable solutions (Lauritsen et al., 2019).

Historically, a trade-off between model performance and explainability has
been described, and lower performing yet explainable models are often
preferred to more sophisticated black box models due to lack of trust or the
presence of strict transparency requirements (e.g. regulatory constraints). To
address this, model-agnostic _post hoc_ explainability methods such as LIME
(Ribeiro et al., 2016) and SHAP (Lundberg & Lee, 2017) were developed, where a
second model is trained to explain the original predictions. However, there
have been issues reported with these methods: they do not always faithfully
describe the original model’s behavior and can give different explanations for
similar examples (Alvarez-Melis & Jaakkola, 2018). Recent efforts, namely
(Rudin, 2019), call for the use of inherently interpretable models in
situations where human understanding is necessary – especially when
consequences from undue faith in model output can be dire. These methods
generally produce sparse models which allow human stakeholders to gain
understanding about the broader system in which the model operates, as well as
understand precisely how the model produces each prediction.

Natural language processing (NLP) presents a unique modeling and
interpretability challenge. Many traditional methods of text preparation are
transparent in nature, but as Transformer-based models (Vaswani et al., 2017)
have gained popularity, we continue to see the boundaries of black-box
modeling pushed. This raises a pressing issue for data scientists and their
stakeholders within the healthcare domain: how should we appropriately balance
the need for well-performing models with the desire for human interpretability
of models and their outputs?

With such a wide variety of available tools, it should also be noted that the
use of these techniques should be tailored to the audience – for example, data
scientists may be interested in interactively exploring neuron activations,
while a physician may be interested in the features present in a medical
record that contributed to a clinical decision.

###  1.1 Interpretability versus Explainability

In Rudin’s work (2019), she defines “interpretable” machine learning as the
use of models which themselves are fully interpretable by human users, as
opposed to using “explainability” techniques to understand the behavior of
non-interpretable models. We follow these definitions throughout, although we
acknowledge that there is some degree of gray area with certain attribution
methods.

Importantly, neither of these concepts are in any way causal – all of the
methods in this paper should be used to gain insight into how a specific model
works, rather than providing any understanding of underlying causal structures
of the broader system in which the models operate.

###  1.2 Contributions

The major contributions of this work are as follows:

  * •

Quantifying the quality of available explanation methods using infidelity (Yeh
et al., 2019), generalizing perturbation methods for local Lipschitz (Alvarez-
Melis & Jaakkola, 2018) on text classification, and making these methods
available for use in open-source form

  * •

Comparing the performance of traditional algorithms with interpretable ML
techniques and state-of-the-art methods in a practical case study in clinical
text classification

  * •

Reaching state-of-the-art performance on the mortality prediction task (among
models without additional MIMIC-specific pretraining) using BigBird (Zaheer et
al., 2021)

  * •

Discussing the distinction between _interpretability_ and _explainability_ ,
practical issues in measuring and implementing various explainable and
interpretable algorithms, and particularly the challenges facing practitioners
within the healthcare industry

##  2 Experiment Setup

###  2.1 Data

Given the sensitive nature of healthcare data, few openly accessible datasets
of clinical text exist for broad research use. In collaboration with Beth
Israel Deaconess Medical Center, Johnson et al. introduced MIMIC-III (2016) –
a database containing deidentified electronic medical record (EMR) output of
Intensive Care Unit (ICU) stays, including structured data elements of
medication and treatment as well as unstructured clinical text. Because MIMIC-
III is the only publicly available data source of its type and scale, it
frequently serves as the basis for peer reviewed studies which make use of EMR
data. Here, we utilize the unstructured discharge summaries for our
classification task.

###  2.2 Classification Task

We selected one task in order to compare interpretable features across model
types and explainability methods within the context of a practically relevant
use-case. We use the single-label, binary classification of in-hospital
mortality prediction from simulated admissions notes. This benchmark was
established in Van Aken et al. (2021), and we follow their process of data
preparation. Details about this task dataset are provided in supplementary
material1.

##  3 Methods

###  3.1 Classifiers

We include several classifiers with varying levels of complexity and
interpretability:

  * •

Logistic Regression (LR)

  * •

Random Forest (RF) (Breiman, 2001)

  * •

Explainable Boosting Machine (EBM) (Nori et al., 2019)

  * •

DL8.5 (Aglin et al., 2020)

  * •

Boosted Rule Sets (Freund & Schapire, 1997)

  * •

Bayesian Rule Lists (Yang et al., 2017)

  * •

Optimal Classification Trees (Bertsimas & Dunn, 2017)

  * •

CORELS (Angelino et al., 2018)

  * •

BigBird (Zaheer et al., 2021)

###  3.2 Explainability

For our traditional models (LR, RF, EBM), we selected two popular _post hoc_
explainability methods: Local Interpretable Model-agnostic Explanations (LIME)
(Ribeiro et al., 2016) and SHapley Additive exPlanations (SHAP) (Lundberg &
Lee, 2017). Since some of our models are also inherently interpretable (LR and
EBM), we were also able to compare _post hoc_ explanations to the true local
explanations.

Transformer models are a rapidly evolving area of active research, and efforts
are ongoing to develop explainability techniques for better understanding
their underlying processes. Input attribution methods show importance of input
tokens, illustrating the degree to which each token contributed to a given
prediction. Several algorithms exist for performing input attribution with
deep learning models: saliency (Simonyan et al., 2014) and integrated
gradients (Sundararajan et al., 2017) are somewhat agnostic methods which use
various backpropagation features to perform attribution, while other methods
exist for more specific purposes (e.g. deconvolution and GradCAM for computer
vision). In addition to these methods, SHAP and LIME are compatible for _post
hoc_ explainability with deep learning models – however, both have been shown
to possess worse qualities than techniques with properties native to deep
learning (Alvarez-Melis & Jaakkola, 2018).

Along with the growing trend of Transformer interpretability features,
researchers have created a number of visualization techniques to highlight
various behaviors (self-attention weights, activations, etc.). BertViz (Vig,
2019) and Ecco222Accessed via https://github.com/jalammar/ecco (Alammar, 2020)
are two libraries which provide useful front-ends for visualizing such
behaviors within Transformer models, and LIT (Tenney et al., 2020) is a more
general framework for practitioners to visualize and perform behavioral
testing on various NLP models.

###  3.3 Explanation Quality Metrics

With a variety of options available for input attribution and local
explainability for “black-box” models, a user will naturally wonder which
method best represents the underlying mechanisms of the model. There are a
handful of metrics which seek to quantify the quality of model explanations
(Zhou et al., 2021), however there is no single metric which will adequately
cover all aspects of interpretability and apply to all contexts. For example,
parsimony of the global explanations (i.e. the inherent human interpretability
of a model’s structure) may be more important when developing clinical
guidelines, while fidelity of prediction explanations might adequately capture
the needs of a human-in-the-loop ML system.

For our case study, we searched for metrics which measure various components
contributing to the quality of _post hoc_ explanations provided through input
attribution methods (e.g. SHAP, integrated gradients, etc.).

####  3.3.1 Generalizing Local Lipschitz

Local Lipschitz (Alvarez-Melis & Jaakkola, 2018) is an explanation quality
metric which measures the relative change in attributions introduced by
relatively small perturbations in the model input. This metric is closely
related to the notion of a particular explanation method’s sensitivity when
slight changes are introduced – if the input changes by some small amount,
then we would presume that the explanations should not change drastically. The
formula for local Lipschitz is given as follows:

$\hat{L}(x_{i})=\operatorname*{arg\,max}_{x_{j}\in
B_{\epsilon}(x_{i})}\frac{||f(x_{i})-f(x_{j})||_{2}}{||x_{i}-x_{j}||_{2}}$

One challenge came to light when applying this metric to text data: the
concept of a _local neighborhood_ is difficult to define within the context of
raw text – we can measure the distance between vectorized documents through
metrics such as $\ell 2$ distance or cosine similarity, but there is no
consistent radius ($\epsilon$ in the formula) which can be equivalently
applied to models with different preparation types (e.g. TF-IDF document
vectors and a sequence of word embeddings). We address this issue by
introducing a text perturbation method based upon token replacement via
nearest neighbor sampling. Pseudocode for our document perturbation method is
given in Algorithm 1. Each token – a single word in a document to be
vectorized, or a single token ID to be fed into a language model – has
probability $\pi$ of being selected for replacement, and each selected token
is replaced by sampling from its $k$ nearest neighbors in vector space. – in
this description, `sample_neighbors` is simply shorthand for sampling a
replacement token from its closest neighbors. For traditional methods we use a
word2vec model (Řehůřek & Sojka, 2010; Mikolov et al., 2013) trained on the
mortality training set. For the BigBird model, we construct a distance matrix
of tokens using pairwise $\ell 2$ distance between entries within the language
model’s word embedding layer.

Algorithm 1 Text Perturbation

Input: document $Doc$, sampling probability $\pi\in[0,1]$, number of nearest
neighbors to consider $k>0$

Output: modified document $modifiedDoc$, with tokens randomly replaced

Initialize $modifiedDoc=[\ ]$.

for $token$ in $Doc$ do

if $random.uniform()\leq\pi$ then

$replacement=\operatorname{sample\\_neighbors}(token,k)$

$modifiedDoc.append(replacement)$

else

$modifiedDoc.append(token)$

end if

end for

Once our perturbation method is defined, we can construct a neighborhood of
perturbed documents for each document in our sample for local Lipschitz
estimation. For our experiments, we used a token sampling probability of
$\pi=0.1$ and sampled from the nearest $k=10$ neighbors. For each point in the
neighborhood, we calculate new attributions based on the perturbed input – the
$\ell 2$ distance between perturbed and original attributions serves as the
numerator of the local Lipschitz ratio, and the denominator is given by the
$\ell 2$ distance between perturbed and original numericalized inputs (i.e.,
embedded sequence or vectorized document after perturbation). In order to
enforce the neighborhood aspect of local Lipschitz, we apply a radius of
$\epsilon=0.25$ for removing perturbed inputs which lie too far outside the
local neighborhood.

####  3.3.2 Infidelity

Infidelity (Yeh et al., 2019) is another metric which seeks to measure the
degree to which the explanations capture the change in true model output when
“significant” perturbations are introduced.

Infidelity is somewhat less exposed to the problem of neighborhood definition
than local Lipschitz because it has no need for a common radius. It does,
however, require that the perturbations introduced can be subtracted from the
input, thus requiring that the perturbations and model inputs are both
represented numerically. This is a less desirable aspect of infidelity as a
metric to compare across modeling paradigms in NLP, so we account for this by
using Gaussian noise with zero-mean and standard deviation proportional to the
relevant preparation method (i.e., embedded tokens for LMs and TF-IDF vectors
for traditional models).

##  4 Experimental Results

###  4.1 Classifier Performance

In keeping with the benchmark task (van Aken et al., 2021), models are
evaluated using the mortality prediction test set and the AUC metric. While we
built models with the intention of optimizing validation performance, reaching
state-of-the-art test set performance on the benchmark leaderboard was not our
primary goal – our intention is to demonstrate various interpretability
methods using a variety of realistic models, so we trained until reaching
parity with the performance. For each model, test set performance is provided
in Table 1 along with its most comparable benchmark.

Transformer-based models outperformed both traditional methods and
interpretable models. BigBird performed best, beating BioBERT (Lee et al.,
2019) and performing comparably to the benchmark models which were pretrained
on additional MIMIC data: CORe and DischargeBERT (Alsentzer et al., 2019)
(also known as _BioClinical BERT_).

Of the inherently interpretable methods, LR and EBM performed comparably to,
if slightly better than, the established “BOW” benchmark, while DL8.5 and
Boosted Rule Sets performed significantly worse than any reasonable baseline.
The fact that a decision tree performed poorly on a large-scale NLP task is
somewhat unsurprising – however, the DL8.5 algorithm given one hour to train
failed to reach competitiveness with a baseline CART model with all else held
constant (tree depth, feature representation, etc.). The final traditional
model we used with success is random forest, which also slightly outperforms
the “BOW” benchmark AUC.

####  4.1.1 Computational Challenges with Interpretable Methods

In our experiments, we saw that many of the available interpretable and/or
provably optimal techniques are too computationally complex to return the
provably optimal solution within a reasonable time on the document-term
matrices for our task. Under the category of optimal trees, we unsuccessfully
attempted the methods introduced in (Bertsimas & Dunn, 2017) and implemented
in PyOptree333Accessed via https://github.com/pan5431333/pyoptree – this took
several hours to complete single iterations on a single tree model. For rule
lists, we attempted Bayesian Rule Lists (Yang et al., 2017) and CORELS
(Angelino et al., 2018), neither of which could work due to the scale of the
data for this task.

Table 1: Classifier performance (AUC) on the mortality prediction test set
along with most comparable benchmarks.
—*: Unable to execute within a reasonable time or compute limit.
Model | AUC |  Benchmark AUC
---|---|---
LR | 81.07 |  79.15
RF | 79.30 |  79.15
EBM | 76.50 |  79.15
DL8.5 | 51.02 |  79.15
Boosted | 73.49 |  79.15
Rule Sets |  |
BigBird | 83.59 |  81.13 (BERT)
82.55 (BioBERT)
84.51 (_DischargeBert_)
PyOptree |  —* |  79.15
CORELS |  —* |  79.15
Bayesian |  —* |  79.15
Rule Lists |  |

###  4.2 Demonstrating Interpretability

Input attribution is an obvious choice for visualizing the local explanations
produced by both high-dimensional interpretable models (such as LR and EBM)
and deep learning models. Such visualizations can be understood by less
technical human users – including clinicians – and can easily be integrated
into a workflow in which the users interact with the information being
classified. A well designed attribution visualization will require minimal
education or training for human stakeholders who are familiar with the model’s
task. In a text-based setting, this can be achieved simply by highlighting the
terms within the document which contributed to a certain prediction. This
level of information can help bolster trust in the model, while also helping
clinical experts identify potential root causes of prediction errors.

We use input attribution for local explainability throughout our experiments –
while we focus on text classification for this work, this same concept is
perfectly compatible with other NLP tasks, including named entity recognition
(NER) and language modeling, and many of the libraries we use can be mapped
directly to other tasks within and beyond NLP.

####  4.2.1 Comparing Surrogate Attributions to True Local Explanations

The `interpret` library (Nori et al., 2019) allowed us to directly compare the
local explanations obtained by LIME and SHAP to the underlying truth obtained
by enumerating the effects from the learned models for logistic regression and
EBM.

Figure 1 shows a comparison of actual local explanations from a logistic
regression versus results from _post hoc_ explainability methods SHAP and
LIME. SHAP features match actual LR features almost exactly, with only two
exceptions: “numbertoken” (our catch-all token for numeric values in the text)
was found in LR but not SHAP, while “coumadin” was found in SHAP but not LR.
LIME features had very little overlap with the true LR features; only two
features – “intubated” and “unrespons” – were found in both. This is
unsurprising, as LIME has been reported to have low robustness (Alvarez-Melis
& Jaakkola, 2018). EBM explainability follows a similar pattern where SHAP
explains the feature importance more accurately than LIME, and additional
comparisons can be found in our supplemental material. Between the two
classification methods, there was also some feature overlap, such as
“intubated,” “neuro,” and “coumadin.”

(a) True local explanations for logistic regression.

(b) SHAP-based explanations.

(c) LIME-based explanations.

Figure 1: Comparison of true model features to _post hoc_ feature explanations
on a single observation.

####  4.2.2 Global Interpretability

Fully interpretable models generally include a method for global
interpretability – a set of coefficients, tree structure, or something similar
indicating the “important” features or an explicit decision path detailing the
model’s behavior under any conceivable permutation. Various options exist for
displaying feature importance – for example, a tree-based model can be plotted
in graph form, raw feature importance for LR/EBM can be displayed similarly to
the local explanations in the previous section, and individual feature effects
can be plotted over the range of feature values as in a partial dependence
plot (PDP). Figure 2 illustrates the global interpretability features provided
by the “glassbox” models in the `interpret` library: global feature importance
for the EBM model can be seen in 2(a), and 2(b) shows the individual modeled
impact of the feature “intub”.

(a) Global feature importance for EBM

(b) Impact of TF-IDF score of “intub”

Figure 2: Global interpretability for an EBM model from the interpret library.

####  4.2.3 Interpretable Classifiers

We chose two interpretable methods which are not commonly used in NLP
problems: Boosted Rule Sets (Freund & Schapire, 1997) and DL8.5 (Aglin et al.,
2020). These methods bring something slightly different to the conversation
around interpretability than EBM and logistic regression, in that they produce
models which are more sparse in nature and can easily be understood by a human
user. In the case of DL8.5, there is the additional benefit of the final model
being provably optimal for a given specification and dataset.

We used the implementation of Boosted Rule Sets in the `imodels` library
(Singh et al., 2021), and our best model achieved test AUC of 73.49 –
considerably worse than the benchmark performance. This model learned an
ensemble of 50 simple rules which indicate higher likelihood for in-hospital
mortality, in the format of $token\leq score$, where $token$ is an n-gram
found in the document term matrix, and $score$ is the TF-IDF score for that
n-gram in the given document. A subset of these rules is included in
supplementary material.

DL8.5 performed extremely poorly on this task due to the dimensionality of the
dataset. Because of the length and diverse content of the admission notes, we
retain as much of the information as possible through our document
preprocessing. The high dimensionality of the design matrix – in this case,
both cardinality and volume – overwhelms the search algorithm, which
essentially cannot run without time limits. When this time limit is reached, a
poor model is returned, achieving virtually no AUC lift. DL8.5 and Boosted
Rule Sets would benefit from additional feature engineering; however, lexicons
and hand-engineered features are expensive to develop and maintain, and other
dimensionality reduction techniques (e.g. manifold learning for high-
dimensional feature spaces) would sacrifice the level of interpretability.
Both of these alternative approaches are beyond the scope of this work.

####  4.2.4 Transformer Explainability: Input Attribution and Behavioral
Visualization

We explored a variety of visualization techniques for Transformer
explainability, including input attribution and visualization of activations
and attentional weights. The attribution methods produce results which are
conceptually similar to those given by SHAP and LIME: they are designed to
denote the degree to which a specific input token contributed to the model’s
output – in this case, mortality prediction. Figure 3 shows input attribution
calculated via the saliency method, and visualized using the Captum library
(Kokhlikyan et al., 2020) for a true positive prediction (a patient correctly
predicted to expire while in the ICU). We can see that there are tokens such
as “unresponsive” and “intubated” highlighted green, indicating a positive
influence (pushing the model toward predicting patient expiration).
Additionally, some tokens have a red highlight indicating an impact that
suppresses predicted mortality, including “clear” within a statement
describing the patient’s lungs.

Figure 3: Saliency-based local explanations from the Captum library.

Beyond input attribution, other methods are tailored to gaining deeper
understanding of the nuances within the representations learned by the
language model. Alammar’s Ecco library provides an interactive visualization
to highlight the input tokens which produce similar neuron activations. Ecco
uses non-negative matrix factorization to reduce the high-dimensional
activation data into a user-specified number of factors, which are then
displayed interactively.

Figure 4 shows Ecco’s results for the same true positive case: neuron
activations are reduced to 12 factors, and related tokens are highlighted. The
sparklines on the left side of the image correspond to individual factors,
along with their activations throughout the document. In this view, Factor 2
is highlighted alone, where we can see that it includes language relation to
intubation, sedation, falling, and unresponsiveness. This is helpful for us to
glean insights from potentially related terms. Additional views can be found
in our supplemental material – for example, in addition to this factor,
switching to Factor 11 seems to highlight specific mentions of medication
(Coumadin, Dilantin, Propofol) as well as generic treatments (hypertonic
saline, Vitamin K).

Figure 4: Ecco’s non-negative matrix factorization of neuron activations with
component 2 highlighted.

We also explored libraries for visualizing full self-attention weights.
BertViz (Vig, 2019) has nice built-in interactivity to allow users to explore
results by layer and attention head. However, one drawback of this method is
that it does not perform particularly well with relatively long input
sequences: the task of visualizing full self-attention across 1,000 tokens is
a difficult task, and existing tooling struggles to accommodate such
sequences. We can instead use these techniques to visualize local effects by
viewing self-attention weights within a smaller segment of a larger document.
Figure 5 shows BertViz’s head view method on a small input sequence with
similar language. In 5(a), the full connections are displayed with opacity
representing the relative weight applied in all heads of the first encoder
layer. In 5(b), a subword of “unresponsive” is highlighted, and inbound self-
attention weights are indicated from the other tokens in the sequence.

(a) Full attention weights displayed.

(b) Inbound attention weights for a subword of “unresponsive” isolated.

Figure 5: Attention weight visualization with BertViz.

This view is perhaps more compelling for language modeling use-cases (e.g.,
masked language modeling), which focus more on the learned representations of
each token. It is unclear whether these attention weights provide much
practical value for gaining true understanding of a Transformer-based
classifier’s behavior.

###  4.3 Evaluating the Quality of Explanations

(a) Distribution of local Lipschitz values.

(b) Distribution of infidelity values.

Figure 6: Evaluation metrics comparing models and attribution types.

####  4.3.1 Local Lipschitz

We first explored the local Lipschitz measure (Alvarez-Melis & Jaakkola, 2018)
as a metric for quantifying the sensitivity of _post hoc_ input attribution.

For our experiment, we used a sample of 35 documents and required a
neighborhood size of 15 perturbations for each document. For each document, we
calculated local Lipschitz values for each of the perturbations and retained
the maximizing perturbation. The distribution of these values by model and
attribution type can be seen in Figure 6(a), which shows a great deal of
variation between models as well as attribution methods. According to Alvarez-
Melis and Jaakkola (2018), there is no globally optimal value for the local
Lipschitz – we can, however, use it to compare the relative sensitivity of
attributions methods across models and for attributions on the same model. We
can see that both attributions for BigBird have higher local Lipschitz values
than all traditional methods, indicating that the available attributions are
more sensitive to slight changes in the model inputs. These results show that
this is still the case when using a common perturbation method. The random
forest model with SHAP attributions produces the lowest local Lipschitz
values, indicating that it is relatively insensitive to small input changes.

####  4.3.2 (In)Fidelity

Since there is no need to repeatedly perturb inputs, we ran our infidelity
experiment on a sample of 100 documents. The results are given in Figure 6(b),
and it is clear that there is somewhat less pronounced variation between
models than with local Lipschitz. BigBird’s attribution methods have higher
infidelity than any of the traditional methods, indicating that the features
identified by the attribution methods have less of an impact on the model’s
output when Gaussian noise is introduced.

Another interesting property of infidelity can be seen in the truly
interpretable models: for LR and EBM, we can compare the infidelity measures
for LIME and SHAP to the infidelity for the true local explanations. In the
case of EBM, the true local explanation has a slightly higher infidelity than
both LIME and SHAP; for LR, the true explanations have an infidelity that lies
between LIME and SHAP. This is a somewhat surprising and counterintuitive
result, which suggests that future work should explore better ways to apply
infidelity to NLP problems.

####  4.3.3 Examining the Tradeoff between Explanation Quality and Model
Performance

With a common framework in place to evaluate the behavior of available
explanations, we can now examine the trade-off between explainability and
model performance. Table 2 illustrates this comparison for the models and
explanations explored in the previous sections. As shown previously, it is
clear that BigBird and LR have the highest AUC values; we can now also
quantitatively compare their interpretability features through the quality
metrics. In summary, LR has worse performance with substantially better
explanations (in addition to the fact that it is fully interpretable). As in
any practical setting, the ultimate decision will require weighing the
relative importance of incremental model improvement against the fidelity of
available explanations.

Multi-objective optimization is one potential approach for such a weighting
scheme in a practical setting: practitioners could weight the relative
importance of raw model performance (F-score, precision, recall, AUC, etc.)
against any number of explanation quality metrics to identify the model which
best balances the Pareto frontier between interpretability/explainability and
performance. This framework would also allow users to place constraints on
certain metrics (e.g. requiring a minimum precision of 80%) as well as
incorporating any other desired quality of the final model (e.g. model size or
number of parameters).

Table 2: AUC, Infidelity, and Lipschitz values for all tested algorithms and attribution methods. Model | AUC | Infidelity | Lipschitz
---|---|---|---
EBM LIME | 76.50 | 0.003482 | 0.674207
EBM SHAP | 76.50 | 0.003325 | 0.404805
EBM Truth | 76.50 | 0.003591 | 2.097308
BigBird IG | 83.59 | 0.034362 | 21.500845
BigBird Saliency | 83.59 | 0.064262 | 32.830019
LR LIME | 81.07 | 0.001947 | 1.07602
LR SHAP | 81.07 | 0.001838 | 0.41027
LR Truth | 81.07 | 0.001918 | 1.838809
RF SHAP | 79.30 | 0.000613 | 0.105773

##  5 Conclusions

In this work, we explored practical aspects of interpretable and explainable
ML methods within a case study of clinical text classification. We
demonstrated some of the interpretability tools available to healthcare NLP
practitioners, discussed existing definitions for explainability and
interpretability, and we introduced a framework which can evaluate the quality
of explanations across text classification models, including the infidelity
and relative sensitivity of the presented attributions. We showed that this
evaluation framework can be used in conjunction with preferred model
evaluation metrics to provide a Pareto frontier illustrating the trade-off
between a model’s test set performance and the quality of the explanation
methods available to it. This can be a very useful tool for practitioners in
many domains, including healthcare, in which the data scientist is tasked with
building and identifying an optimal model based on the desired level of human
interpretability, which varies from problem to problem. It will also be useful
for developers of AutoML solutions – the introduction of quantitative metrics
for explanation quality presents an opportunity to jointly optimize for such
metrics in addition to raw model performance.

We showed that the interpretable ML techniques which are not typically used
for text classification perform suboptimally despite their interpretable
properties. In some cases, a deterioration in model quality may be acceptable
for the purpose of full model interpretability, although other methods for
producing sparse models (e.g. LASSO regression) may provide better
performance. Through direct comparison between LIME, SHAP, and true local
explanations, we show that LIME falls short on such high-dimensional tasks,
where sparse attributions do not necessarily reflect the true underlying
behavior of the model. We also demonstrated current work in visualizing the
underlying behavior of BERT-style models, which attempts to build progress
toward a sense of interpretability for these black-box models.

###  5.1 Future Work

Future work in both explainable and interpretable ML will undoubtedly be a
crucial factor in the effort to accelerate adoption within the healthcare
sector. These efforts should include additional development across the
spectrum of interpretability in addition to the goal of ever-improving model
performance. Progress in this area will be aided by the development and use of
common evaluation metrics, which seek to quantify the quality of explanations
provided for black-box models. The essence of this effort is to NLP
practitioners in answering the question, “ _How much_ better or worse are the
SHAP attributions for a random forest compared to saliency for a BERT
classifier?” We make progress in this direction by introducing and open-
sourcing prototypical methods for preparation-agnostic document perturbation
within local neighborhoods, but additional exploration and improvement will
undoubtedly aid adoption.

In addition to better understanding of explanation quality, future work in
reducing the computational complexity of purely interpretable and provably
optimal ML solutions will allow for more competitive use in large-scale text
classification.

## Acknowledgements

The authors would like to thank the organizers of the Interpretable ML in
Healthcare workshop and ICML, as well as the anonymous reviewers for their
feedback and advice. We also acknowledge Digital Reasoning / Smarsh, Inc. and
our colleagues for their support.

## References

  * Aglin et al. (2020) Aglin, G., Nijssen, S., and Schaus, P.  Learning optimal decision trees using caching branch-and-bound search.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(04):3146–3153, Apr. 2020.  doi: 10.1609/aaai.v34i04.5711.  URL https://ojs.aaai.org/index.php/AAAI/article/view/5711.
  * Alammar (2020) Alammar, J.  Interfaces for explaining transformer language models, 2020.  URL https://jalammar.github.io/explaining-transformers/.
  * Alsentzer et al. (2019) Alsentzer, E., Murphy, J. R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., and McDermott, M. B. A.  Publicly available clinical BERT embeddings, 2019.
  * Alvarez-Melis & Jaakkola (2018) Alvarez-Melis, D. and Jaakkola, T. S.  On the robustness of interpretability methods, 2018.
  * Angelino et al. (2018) Angelino, E., Larus-Stone, N., Alabi, D., Seltzer, M., and Rudin, C.  Learning certifiably optimal rule lists for categorical data, 2018.
  * Bertsimas & Dunn (2017) Bertsimas, D. and Dunn, J.  Optimal classification trees.  _Mach. Learn._ , 106(7):1039–1082, July 2017\.  ISSN 0885-6125.  doi: 10.1007/s10994-017-5633-9.  URL https://doi.org/10.1007/s10994-017-5633-9.
  * Breiman (2001) Breiman, L.  Random forests.  _Machine Learning_ , 45(1):5–32, 2001.  ISSN 0885-6125.  doi: 10.1023/A:1010933404324.  URL http://dx.doi.org/10.1023/A%3A1010933404324.
  * Freund & Schapire (1997) Freund, Y. and Schapire, R. E.  A decision-theoretic generalization of on-line learning and an application to boosting.  _Journal of Computer and System Sciences_ , 55(1):119–139, 1997.  ISSN 0022-0000.  doi: https://doi.org/10.1006/jcss.1997.1504.  URL https://www.sciencedirect.com/science/article/pii/S002200009791504X.
  * Johnson et al. (2016) Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., and Mark, R. G.  MIMIC-III, a freely accessible critical care database.  _Scientific Data_ , 3(1):160035, 2016.  ISSN 2052-4463.  doi: 10.1038/sdata.2016.35.  URL https://doi.org/10.1038/sdata.2016.35.
  * Kokhlikyan et al. (2020) Kokhlikyan, N., Miglani, V., Martin, M., Wang, E., Alsallakh, B., Reynolds, J., Melnikov, A., Kliushkina, N., Araya, C., Yan, S., and Reblitz-Richardson, O.  Captum: A unified and generic model interpretability library for pytorch, 2020.
  * Lauritsen et al. (2019) Lauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., and Thiesson, B.  Explainable artificial intelligence model to predict acute critical illness from electronic health records, 2019.
  * Lee et al. (2019) Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., and Kang, J.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _CoRR_ , abs/1901.08746, 2019.  URL http://arxiv.org/abs/1901.08746.
  * Lundberg & Lee (2017) Lundberg, S. and Lee, S.-I.  A unified approach to interpreting model predictions, 2017.
  * Mikolov et al. (2013) Mikolov, T., Chen, K., Corrado, G., and Dean, J.  Efficient estimation of word representations in vector space, 2013.
  * Nori et al. (2019) Nori, H., Jenkins, S., Koch, P., and Caruana, R.  Interpretml: A unified framework for machine learning interpretability, 2019.
  * Řehůřek & Sojka (2010) Řehůřek, R. and Sojka, P.  Software framework for topic modelling with large corpora.  In _Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks_ , pp. 45–50, Valletta, Malta, May 2010. ELRA.  http://is.muni.cz/publication/884893/en.
  * Ribeiro et al. (2016) Ribeiro, M. T., Singh, S., and Guestrin, C.  ”why should i trust you?”: Explaining the predictions of any classifier, 2016.
  * Rudin (2019) Rudin, C.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1:206–215, 05 2019.  doi: 10.1038/s42256-019-0048-x.
  * Shortliffe & Sepúlveda (2018) Shortliffe, E. H. and Sepúlveda, M. J.  Clinical Decision Support in the Era of Artificial Intelligence.  _JAMA_ , 320(21):2199–2200, 12 2018.  ISSN 0098-7484.  doi: 10.1001/jama.2018.17163.  URL https://doi.org/10.1001/jama.2018.17163.
  * Simonyan et al. (2014) Simonyan, K., Vedaldi, A., and Zisserman, A.  Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014.
  * Singh et al. (2021) Singh, C., Nasseri, K., Tan, Y. S., Tang, T., and Yu, B.  imodels: a python package for fitting interpretable models.  _Journal of Open Source Software_ , 6(61):3192, 2021.  doi: 10.21105/joss.03192.  URL https://doi.org/10.21105/joss.03192.
  * Sundararajan et al. (2017) Sundararajan, M., Taly, A., and Yan, Q.  Axiomatic attribution for deep networks, 2017.
  * Tenney et al. (2020) Tenney, I., Wexler, J., Bastings, J., Bolukbasi, T., Coenen, A., Gehrmann, S., Jiang, E., Pushkarna, M., Radebaugh, C., Reif, E., and Yuan, A.  The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models, 2020.  URL https://www.aclweb.org/anthology/2020.emnlp-demos.15.
  * van Aken et al. (2021) van Aken, B., Papaioannou, J.-M., Mayrdorfer, M., Budde, K., Gers, F., and Loeser, A.  Clinical outcome prediction from admission notes using self-supervised knowledge integration.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pp. 881–893, Online, April 2021. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2021.eacl-main.75.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  Attention is all you need, 2017.
  * Vig (2019) Vig, J.  A multiscale visualization of attention in the transformer model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pp. 37–42, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-3007.  URL https://www.aclweb.org/anthology/P19-3007.
  * Yang et al. (2017) Yang, H., Rudin, C., and Seltzer, M.  Scalable Bayesian rule lists, 2017.
  * Yeh et al. (2019) Yeh, C.-K., Hsieh, C.-Y., Suggala, A. S., Inouye, D. I., and Ravikumar, P.  On the (in)fidelity and sensitivity for explanations, 2019.
  * Zaheer et al. (2021) Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A.  Big bird: Transformers for longer sequences, 2021.
  * Zhou et al. (2021) Zhou, J., Gandomi, A. H., Chen, F., and Holzinger, A.  Evaluating the quality of machine learning explanations: A survey on methods and metrics.  _Electronics_ , 10(5), 2021.  ISSN 2079-9292.  doi: 10.3390/electronics10050593.  URL https://www.mdpi.com/2079-9292/10/5/593.