# NLP-based detection of systematic anomalies among the narratives of consumer
complaints

Peiheng Gao  School of Mathematical and Statistical Sciences, Western
University, London, Ontario N6A 3K7, Canada  Ning Sun  Agri-Food Analytics
Lab, Dalhousie University, Halifax, Nova Scotia B3H 4R2, Canada  Xuefeng Wang
Department of Financial Risk Analytics and Reporting, Canada Life, Toronto,
Ontario M5G 1R8, Canada  Chen Yang Corresponding author Chen Yang, PhD, ASA;
e-mail<EMAIL_ADDRESS>Department of Population Health Science and
Policy, Icahn School of Medicine at Mount Sinai, New York, New York 10029, USA
Institute for Health Care Delivery Science, Icahn School of Medicine at Mount
Sinai, New York, New York 10029, USA  Ričardas Zitikis  School of Mathematical
and Statistical Sciences, Western University, London, Ontario N6A 3K7, Canada

To appear in Journal of Operational Risk

Abstract. A Natural Language Processing (NLP) based procedure is developed for
detecting systematic non-meritorious consumer complaints, simply called
systematic anomalies, among complaint narratives. While classification
algorithms are used to identify meritorious complaints, in the case of smaller
and frequent systematic patterns of non-meritorious complaints, those
algorithms may falter due to a variety of reasons, including technical ones as
well as natural limitations of human analysts. Therefore, at the next stage
after classification, the complaint narratives are converted into quantitative
data, which are then analyzed using indices for detecting systematic
anomalies. An illustration of the entire procedure is provided using complaint
narratives from the Consumer Complaint Database of the Consumer Financial
Protection Bureau. The results suggest that the Support Vector Machine (SVM)
outperforms other selected classifiers. Although classification results with
the VADER intensity pertinent to the featurization step have lower accuracy,
they contain fewer non-meritorious complaints than those without the VADER
intensity.

Keywords and phrases: Consumer complaints, anomaly detection, classification,
sentiment scores, Cobb-Douglas function, Natural Language Processing.

##  1 Introduction

The narratives of consumer complaints contain valuable information about the
sales and underwriting activities in the insurance industry. Responding to
consumer complaints is one of the primary ways for insurance departments
(NAIC, 2023) to regulate market practices (Klein, 1995). In 2021, state
insurance departments received 259 345 official complaints, among which 1 474
market conduct examinations were triggered and completed (NAIC, 2022). By
analyzing consumer complaint data, regulators can identify market mis-
behaviour, such as false sales illustrations or failure to pay legitimate
claims on a timely basis, and initiate market conduct examinations to
determine if disciplinary procedures are needed.

In addition to regulatory purposes, consumer complaints also indicate service
quality. For example, a survey about customer service of 140 companies from 14
industries has shown that the three dominating worst-rated industries are
communications, banking/financial services, and insurance (Wood and Morris,
2010). As a result, bulk complaint datasets, such as the one by the Consumer
Financial Protection Bureau (CFPB, 2023), have been created.

Certainly, datasets such as the CFPB consumer complaints also serve the
purpose of regulation, as using complaints for regulatory purposes is
significantly more cost-effective than resolving them individually (Littwin,
2014). Analysts have attempted to determine whether complaints have merits and
possibly constitute violations of state laws or regulations (Klein, 1995). In
such cases, companies may voluntarily provide reliefs along with responses to
the complaints. These reliefs can be seen as risk premiums to mitigate
regulatory or litigation risks in the cases described by complaint narratives.
Hence, reliefs, particularly the tangible ones, are critical indicators of
potential violations of state laws or regulations. Among the complaints in the
CFPB database, companies have granted consumers tangible reliefs 21.9% of the
time. In the remaining cases, the focus has been on providing consumers with
explanations (Littwin, 2014). Nevertheless, whether intentionally or not,
companies are reluctant to grant reliefs in some cases, which are viewed in
the present paper as anomalies of the dispute-resolving system.

###  1.1 Motivation

Reliefs to consumer complaints should be granted with prudence. From the
economic perspective, reliefs to consumer complaints are costly from the time,
effort, and monetary perspectives, and hence impact the profitability of
businesses. Moreover, high percentages of reliefs granted to complaints
suggest poor levels of compliance with regulations or operational risk
management, and hence influence the reputation of financial institutions by
attracting fewer customers although more media attention and regulation. For
this reason, financial institutions need to implement operational risk
management measures to control the percentage of complaints that deserve
reliefs. Indeed, taking risks brings opportunities, and so completely avoiding
risks to minimize the aforementioned percentage may not be attractive from the
business perspective. Yet, the percentage cannot be too high due to reasons
such as arising reputational and regulatory issues.

With the help of records of monetary and non-monetary reliefs granted to
complaints provided by the publicly available CFPB dataset, machine learning
based classifiers can be trained to assist in determining whether reliefs
should be granted. Among the complaints submitted to the CFPB, those that are
closed with monetary or non-monetary reliefs are defined as “meritorious”
(Hayes et al., 2021) throughout the present paper .

There are several reasons to suggest that more effort is needed for this
classification task. First, unlike typical classification tasks of text
inputs, classifiers cannot be trained by using comments along with polarities
(e.g., tweets) or the number of stars (e.g., online shopping experiences) as
responses. Second, complaints are submitted by consumers whereas decisions to
grant, or not to grant, reliefs are made by specialists of financial
institutions. As a result, the potential features extracted from the
narratives are often less correlated with the responses, thus leading to
relatively low accuracy if compared with the polarity classification of tweets
or online shopping experiences. Third, extra steps are needed to check if more
complaints can actually be filtered out from the pool of meritorious
complaints.

###  1.2 Related work

For those datasets that include the narratives of complaints, the Natural
Language Processing (NLP) based analyses have become popular. For example,
Karami and Pendergraft (2018) propose a computational approach for analyzing
complaints from GEICO, an insurance company with more than 16 million policies
and 24 million insured vehicles. However, only the Latent Dirichlet Allocation
(LDA) based topic analysis is performed such that 30 categories of topics are
identified without any direct relationship to decisions of pay-offs or
reliefs. Using a similar method, Liao et al. (2020) consider the LDA-based
topic analysis for an insurance company of personal lines to improve its
customer services by analyzing customer calls. As the LDA-based topic analysis
is an unsupervised learning technique, this study does not use decisions
related to payoffs or reliefs as classification labels, although it involves
other NLP techniques such as text featurization and sentiment analysis.
Furthermore, based on the CFPB dataset, Osman and Sabit (2022) investigate the
impact of a major banking scandal using the Valence Aware Dictionary for
sEntiment Reasoning (VADER) (Hutto and Gilbert, 2014). However, this study
directly quantifies text inputs using VADER without a further exploration of
text mining or classification problems. Pakhchanyan et al. (2022) provide an
overview of how machine learning helps to categorize textual descriptions of
operational loss events into the seven Basel II event types of operational
risk by applying PYTHON implementations of the SVM and multinomial naive Bayes
algorithms to pre-categorized Öffentliche Schadenfälle OpRisk (ÖffSchOR) data.

In the present study, the inputs are the textual descriptions of operational
loss events (e.g., Di Vincenzo et al., 2023), which have higher quality due to
professional narratives compared with consumer complaints that contain more
typos but less informative details. Wang et al. (2022) introduce a text mining
method to analyze changes in operational risk in an innovative way based on a
massive data set of textual risk disclosures of financial institutions, which
aggregate risk perceptions of senior managers of the financial industry.
However, this study employs textual data analytic techniques to identify
significant operational risk drivers instead of performing machine learning
tasks such as classification. Ji et al. (2023) develop a risk-based knowledge
graph to integrate text mining and analytic hierarchy process risk assessment
with knowledge graphs for operational risk analysis. While text mining
techniques are integrated in this study, quantifying operational risk rather
than assisting decision-making is still the ultimate goal (e.g., Wang et al.,
2022; Osman and Sabit, 2022). Di Vincenzo et al. (2024) provide a tweet data
analysis for detecting emerging operational risks.

###  1.3 Research gaps

Due to poor quality of narratives as textual data inputs as well as weak
correlation between the inputs (from customers) and outputs (from financial
institutions), the existing text classification methods have shortcomings,
particularly from the perspective of limiting the percentage of meritorious
complaints. These considerations have motivated us to work on improving
classification procedures from several perspectives, including the
featurization of textual data inputs, and to also propose additional analyses
using an anomaly detection method to assess if the resulting meritorious
classes actually contain systematic non-meritorious complaints.

###  1.4 The scope

In the present paper, a systematic-anomaly detection procedure is proposed to
assess complaints for which companies are reluctant to provide reliefs. In the
current study, the VADER sentiment intensities play an important role in the
text classifier training stage as well as in the performance evaluation stage.
In the text classifier stage, the VADER sentiment intensities are combined
with the Term Frequency-Inverse Document Frequency (TF-IDF) for the purpose of
feature extraction. In the performance evaluation stage, the VADER sentiment
intensities are combined with the discounted dollar amounts mentioned in the
complaint narratives as well as with the word counts of the cleaned
narratives. This, in turn, facilitates the use of anomaly detection indices,
which test if there are persistent anomalies in the dataset of complaints. To
illustrate the proposed methodology, we use the CFPB dataset, due to its
transparency and accessibility.

The present study is comprised of several stages. At the classification stage,
the labels are taken directly from the CFPB complaint dataset, with the
complaints that were resolved by either monetary or non-monetary reliefs
labeled as “meritorious,” while complaints with only responses labeled as
“non-meritorious” (Hayes et al., 2021). The selected complaints are then
restricted to those consumer complaints that involve specific dollar amounts
related to personal credit or debit card accounts against a selected major
financial institution, specifically, the Bank of America (BOA). The features
of the classification task are extracted from the complaint narratives pre-
processed by NLP and transformed to numerical values using the widely utilized
TF-IDF method and our modified version of the TD-IDF using the VADER sentiment
density. The classifiers considered in this study include the Logistic
Regression (LR), Support Vector Machine (SVM), Gradient Boosting (GB),
Multilayer Perceptron (MLP), and Random Forest (RF), which are widely used in
text classification studies (e.g., Hsu, 2020). At the performance evaluation
stage, the resulting meritorious class is further assessed by the index for
systematic-anomaly detection (Gribkova and Zitikis, 2018, 2019a, 2019b, 2020).

###  1.5 Contributions

The present study has several important implications on how financial
institutions can better utilize massive volumes of customer complaints in
order to train classifiers that support decision making when resolving
disputes. Among the achievements are:

  1. (1)

Exploring the possibility of improving textual data classification by
incorporating the VADER sentiment density.

  2. (2)

Introducing an anomaly-detection method that controls the percentage of
reliefs granted to complaints.

  3. (3)

Evaluating classifiers by examining the extent at which the resulting
meritorious class includes actually non-meritorious complaints in a systematic
pattern.

  4. (4)

Demonstrating the usefulness of the proposed procedure in identification of
meritorious customer complaints in the CFPB dataset.

The rest of the paper is organized as follows. In Section 2, a subset of the
CFPB consumer complaint data is chosen for illustration of the proposed
procedure. In Section 3, selected classification algorithms are employed for
anomaly detection among complaint narratives. In Section 4, the narratives are
converted into quantitative data, and a basis for an input-output system is
established. In Section 5, two input-output systems arising from two
featurization methods are introduced. In Section 6, anomaly detection indices
based on the two input-output systems are introduced, and their ability to
detect anomalies is evaluated. Section 7 concludes the paper. Two appendices
illustrating the steps of corpus cleaning (e.g., Di Vincenzo et al., 2023) as
well as the behaviour of the $B$-index with respect to increasing sample sizes
are in the online supplement to this paper.

##  2 Data

A subset of complaint narratives from the Consumer Complaint Database of the
Consumer Financial Protection Bureau (CFPB, 2023) is selected for illustrative
purposes. Extracted are the records of those complaints whose narratives

  1. (1)

are related to credit cards and prepaid cards of a major bank,

  2. (2)

were submitted between December 1, 2011, and June 29, 2023.

Of all the selected complaint records, only those referring to unique dollar
amounts are considered. Furthermore, the complaints referring to dollar
amounts exceeding $10 000 are excluded because those complaints are more
likely to be associated with business accounts rather than individual
accounts. Hence, the resulting data consist of 2 905 records of complaints
related to credit card and prepaid card services of the bank. For simplicity,
complaints with only single positive dollar amounts appearing in the
narratives are selected. As a consequence, 2 849 complaints are ultimately
selected for the current study.

The complaints closed without any monetary or non-monetary reliefs are viewed
as non-meritorious (Hayes et al., 2021) and called anomalies throughout the
present paper. Among the selected 2 849 complaints, 1 051 are meritorious,
that is, they were resolved with some reliefs. As there are dollar amounts
submitted at different times, we discount the amounts using the annual
Consumer Price Index (CPI), which is defined (CPI, 2023)0

$\text{Year 2 Price}=\text{Year 1 Price}\times(\text{Year 2 CPI}/\text{Year 1
CPI})$

to the base date of January 1, 2015. Table 2.1

Date | Raw complaint narratives | Adjusted | Merit
---|---|---|---
11/10/2016 | Macys did not reverse out my $230.00 … | $227.53 | Yes
11/28/2017 | Inquired about $300.00 increase … | $287.36 | No
04/14/2022 | A charge of $170.00 was made … | $107.15 | No
06/12/2023 | Someone fraudulently charged $750.00 … | $360.83 | Yes
Table 2.1: Narrative excerpts, adjusted dollar amounts, and merit indicators.

illustrates.

##  3 Anomaly detection via classification

As all non-meritorious complaints are seen as anomalies throughout the present
paper, classifiers are trained to identify the complaints as either
meritorious or anomalies. To this end, appropriate feature extraction is
needed to translate the cleaned texts into a numerical matrix.

###  3.1 Featurizations

Anomalies, which are non-meritorious complaints in the context of the present
paper, are identified through classification. To train the classifiers, two
featurization methods are considered for text inputs. The first one is the
Term Frequency–Inverse Document Frequency (TF-IDF), which we recall with
adaptations to our current research while at the same time introducing the
necessary notation.

Namely, let $\mathcal{D}=\\{d_{1},\ldots,d_{n}\\}$ denote the set of all
cleaned complaint narratives and $\mathcal{W}$ the set of all words appearing
in $\mathcal{D}$. Then the Term Frequency $\text{TF}\mathrel{\mathop{\mathchar
58\relax}}\mathcal{D}\times\mathcal{W}\mapsto\mathbb{N}$ is defined (Spärck
Jones, 1972) as the frequency of the $j^{\text{th}}$ word $w_{j}$
($j=1,\ldots,\lvert\mathcal{W}\rvert$) in the $i^{\text{th}}$ cleaned
complaint narrative $d_{i}$ ($i=1,\ldots,n$), where $\lvert\mathcal{W}\rvert$
denotes the number of elements in the set $\mathcal{W}$. The Inverse Document
Frequency $\text{IDF}\mathrel{\mathop{\mathchar
58\relax}}\mathcal{W}\mapsto[0,\infty)$ is defined as

$\text{IDF}(w_{j})=\log\bigg{(}\frac{n}{n_{j}}\bigg{)}$

for $w_{j}\in\mathcal{W}$, where

$n_{j}=\sum^{n}_{i=1}\mathds{1}\\{\text{TF}(d_{i},w_{j})>0\\},$

with $\mathds{1}$ denoting the indicator, that is, $n_{j}$ is the number of
cleaned narratives that include $w_{j}$. Then $\text{TF-
IDF}\mathrel{\mathop{\mathchar
58\relax}}\mathcal{D}\times\mathcal{W}\mapsto[0,\infty)$ is defined by the
equation (Bollacker et al., 1998)

$\text{TF-IDF}(d_{i},w_{j})=\text{TF}(d_{i},w_{j})\times\text{IDF}(w_{j}).$

Using the TF-IDF featurization, the cleaned complaint narratives are
translated into a matrix with $n$ rows and $\lvert\mathcal{W}\rvert$ columns.

The second featurization method is the TF-IDF-VADER, which is based on the
function $\text{TF-IDF-VADER}\mathrel{\mathop{\mathchar
58\relax}}\mathcal{D}\times\mathcal{W}\mapsto[0,\infty)$ defined by

$\text{TF-IDF-VADER}(d_{i},w_{j})=\text{TF-
IDF}(d_{i},w_{j})\times\lvert\text{VADER}(w_{j})\rvert,$

where $\text{VADER}(w_{j})$ denotes the VADER sentiment intensity of the word
$w_{j}$ (Hutto and Gilbert, 2014). Since the corpus consists of complaints
which have negative polarities, all the words with non-negative VADER
sentiment intensities are discarded from the Bag of Words. Hence the TF-IDF-
VADER approach is applied on only those words that belong to

$\mathcal{W}_{-}\mathrel{\mathop{\mathchar
58\relax}}=\left\\{w\in\mathcal{W}\mathrel{\mathop{\mathchar
58\relax}}\text{VADER}(w)<0\right\\},$

thus giving rise to a featurized matrix with $n$ rows and
$\lvert\mathcal{W}_{-}\rvert$ columns.

###  3.2 Classifiers

When a new segmentation of classes is defined, or a new method is proposed,
the selected classifiers are usually tested in the exploratory stage in the
hope of maximizing the performance of the proposed method. For example, to
classify “facts” and “opinions” from tweets, Chatterjee et al. (2018) have
selected the LR, SVM, RF, extreme GB, and Deep Neural Network for exploratory
purposes. Another example is Khurana and Verma (2022) who, at the exploratory
stage, tested oversampling methods for the issue of imbalanced classes using
the LR, SVM, RF, and Naïve Bayes.

To the best of our knowledge, this present paper is the first one to attempt
to classify “meritorious” and “non-meritorious” complaints from the CFPB
database. Five classification methods are selected for the exploratory stage:
the LR, SVM, GB, MLP and RF, which are a subcollection of those used by Hsu
(2020). For each of these classification methods, the selected 2 849 complaint
narratives are split randomly so that roughly 60% make up the training set and
the remaining ones make up the validation set. By repeating this procedure 500
times, 500 confusion matrices for each of the five classifiers are obtained
using the TF-IDF and TF-IDF-VADER methods. The average classification accuracy
(i.e., the percentage of correct predictions regardless of the true label),
the percentage of correctly classified meritorious complaints, and the F1
score based on the 500 confusion matrices for each of the five classifiers are
reported in Table 3.1.

Model | TF-IDF |  | TF-IDF-VADER
---|---|---|---
| Accuracy | Merit | F1 |  | Accuracy | Merit | F1
LR | 67.37% | 15.41% | 78.00% |  | 63.96% |  8.52% | 76.80%
SVM | 67.30% | 13.24% | 78.24% |  | 63.72% |  7.29% | 76.65%
GB | 64.61% | 16.43% | 76.03% |  | 62.75% | 11.31% | 75.44%
MLP | 58.08% | 36.04% | 71.22% |  | 56.50% | 34.96% | 66.38%
RF | 65.56% |  9.80% | 77.64% |  | 60.67% | 18.71% | 73.28%
Table 3.1: Average classification accuracy, merit, and F1 scores for 500
random splits of training and testing sets under the TF-IDF and TF-IDF-VADER
featurizations.

Based on Table 3.1, it can be concluded that the TF-IDF featurization
consistently generates higher accuracy than the TF-IDF-VADER featurization.
Moreover, under the TF-IDF, the SVM slightly outperforms the other four
classifiers, as seen from the accuracies and F1 scores, while the LR SVM
slightly outperforms the other four classifiers under the TF-IDF-VADER
featurization. In addition, fewer complaints are classified as meritorious
under the TF-IDF-VADER than under TF-IDF for all classifiers except for the
RF. In summary, the VADER sentiment density does not seem to help to improve
the classification accuracy or the F1 score. However, it helps to control the
percentage of complaints that are resolved with reliefs.

The average accuracy of the classification of meritorious versus non-
meritorious complaints is between 60%-70%, which is fairly low if compared
with some other studies. However, such classification performance is not rare
when sentiment analysis is involved, as is the case with, for example, the
sentiment classification of the IMDB data (Onita, 2023, Table 1). We see from
the latter study that the MLP is outperformed by the LR, SVM, and RF in both
the sentiment classification of the IMDB data and the PadChest data (Onita,
2023, Table 4).

##  4 Converting narratives into quantitative data

As our input-output system requires numerical inputs and outputs, quantities
related to the cleaned meritorious narratives need to be identified for the
purpose of anomaly detection. In this section, the word counts of the cleaned
narratives, discounted dollar amounts (obtained via the CPI as described
earlier), and VADER sentiment intensities (Hutto and Gilbert, 2014) are chosen
for constructing input-output systems based on several economic
considerations.

###  4.1 TF-IDF

To measure the sentiment level of the CFPB consumer complaints, the function
$\textrm{VADER}\mathrel{\mathop{\mathchar 58\relax}}\Omega\to[-4,4]$ is
employed with $\Omega$ denoting the Bag of Words of the VADER sentiment
lexicon that has been widely used in finance (e.g., Osman and Sabit, 2022;
Long et al., 2023), public health (e.g., Sanders et al., 2021), and other
areas. Namely, for each consumer $i$, the sentiment score is defined by

$s_{i}=\sum_{w_{j}\textrm{ in }d_{j}\textrm{ and
}W_{-}}\big{|}\textrm{VADER}(w_{j})\big{|},$ (4.1)

where the sum is taken over all words $w_{j}$ that are both in the cleaned
narrative $d_{i}$ and in the set $W_{-}$ of those words that have negative
VADER sentiment intensities. Hence, the sentiment score $s_{i}$ can be
rewritten as

$s_{i}=\sum_{w_{j}\textrm{ in
}d_{j}}\mathds{1}\\{\textrm{VADER}(w_{j})<0\\}\big{(}-\textrm{VADER}(w_{j})\big{)},$

where $\mathds{1}$ denotes the indicator function. It is instructive to
compare this sentiment score with the one used by Osman and Sabit (2022).

The time and effort invested by the consumer $i$ to generate a complaint
depend on 1) how much money the consumer was charged or how much return the
consumer has not received, and 2) the consumer’s sentiment level due to
monetary or non-monetary losses. With the number $m_{i}^{\textrm{TI}}$ of
words in the cleaned narrative $d_{i}$, the adjusted dollar amount $l_{i}$,
and the sentiment score $s_{i}$, we assume the Cobb-Douglas relationship

$m_{i}^{\textrm{TI}}=\beta s_{i}^{\alpha}l_{i}^{1-\alpha},$ (4.2)

where $m_{i}^{\textrm{TI}}$ is viewed as the yield, $l_{i}$ and $s_{i}$ are
viewed as factors, and $\alpha\in(0,1)$ and $\beta>0$ are the geometric weight
and the scale, respectively. Note that by the Cobb-Douglas relationship, the
law of diminishing marginal utility is satisfied for the sentiment score
$s_{i}$ and the adjusted dollar amount $l_{i}$.

######  Note 4.1.

Since $\textrm{VADER}(w)\in[-4,4]$ for every word $w$ (Hutto and Gilbert,
2014), the sentiment score $s_{i}$ is non-negative and does not exceed
$4m_{i}^{\textrm{TI}}$. Hence, $s_{i}/(4m_{i}^{\textrm{TI}})\in[0,1]$.

Taking logarithms of the two sides of equation (4.2) yields

$\log\bigg{(}{m_{i}^{\textrm{TI}}\over
l_{i}}\bigg{)}=\alpha\log\bigg{(}{s_{i}\over l_{i}}\bigg{)}+\log(\beta).$
(4.3)

The parameter $\alpha$ can be estimated using simple linear regression, which
gives

$\hat{\alpha}=0.990$ with the 95% confidence interval $(0.983,0.997)$. (4.4)

The resulting scatter-plot based on the points

$\bigg{(}\log\bigg{(}{s_{i}\over
l_{i}}\bigg{)},\log\bigg{(}{m_{i}^{\textrm{TI}}\over
l_{i}}\bigg{)}\bigg{)},\quad i=1,\dots,2,849,$ (4.5)

and the fitted least squares regression line are depicted in Figure 4.1.

Figure 4.1: Scatter-plot (4.5) and the fitted least squares regression line.

The diagnostic plots are in Figure 4.2.

(a) Residuals vs Fitted (b) Normal Q-Q

(c) Scale-Location (d) Residuals vs Leverage

Figure 4.2: Diagnostic plots for model (4.3).

Note that the 95% confidence interval $(0.983,0.997)$ is included in the
interval $(0.5,1)$ and therefore, equation (4.2) can be assumed to hold with
some $\alpha\in(0.5,1)$.

###  4.2 TF-IDF-VADER

With the number $m_{i}^{\textrm{TIV}}$ of those words that are in the cleaned
narrative $d_{i}$ as well as in the set $W_{-}$, the adjusted dollar amount
$l_{i}$, and the sentiment score $s_{i}$ defined by equation (4.1), the Cobb-
Douglas relationship can be written as

$m_{i}^{\textrm{TIV}}=\beta s_{i}^{\alpha}l_{i}^{1-\alpha}.$ (4.6)

Naturally, being based on the TF-IDF-VADER method rather than TF-IDF, this
equation leads to different estimates of the geometric weight $\alpha\in(0,1)$
and the scale factor $\beta>0$ than those obtained from equation (4.2). As
shown in next Section 5, these estimates will give rise to different shapes of
the input-output system’s transfer functions.

######  Note 4.2.

Since the sentiment score $s_{i}$ is non-negative and does not exceed
$4m_{i}^{\textrm{TIV}}$, the statement $s_{i}/(4m_{i}^{\textrm{TIV}})\in[0,1]$
holds.

Taking logarithms of the two sides of equation (4.6) yields

$\log\bigg{(}{m_{i}^{\textrm{TIV}}\over
l_{i}}\bigg{)}=\alpha\log\bigg{(}{s_{i}\over l_{i}}\bigg{)}+\log(\beta).$
(4.7)

The parameter $\alpha$ can be estimated using simple linear regression, which
gives

$\hat{\alpha}=1.004$ with the 95% confidence interval $(1.001,1.007)$. (4.8)

The resulting scatter-plot based on the points

$\bigg{(}\log\bigg{(}{s_{i}\over
l_{i}}\bigg{)},\log\bigg{(}{m_{i}^{\textrm{TIV}}\over
l_{i}}\bigg{)}\bigg{)},\quad i=1,\dots,2,849,$ (4.9)

and the fitted least squares regression line are depicted in Figure 4.3.

Figure 4.3: Scatter-plot (4.9) and the fitted least squares regression line.

The diagnostic plots are in Figure 4.4.

(a) Residuals vs Fitted (b) Normal Q-Q

(c) Scale-Location (d) Residuals vs Leverage

Figure 4.4: Diagnostic plots for model (4.3).

Note that the 95% confidence interval $(1.001,1.007)$ suggests that equation
(4.6) holds for some $\alpha>1$.

##  5 The input-output systems

Training anomaly detection classifiers leads to confusion matrices that
involve classification errors. It also generates a meritorious subset of
observations. In addition to being classified as meritorious, the observations
within the meritorious set are expected to follow equations (4.2) or (4.6),
thus leading to an input-output system for each of the two featurizations, as
we shall describe next.

###  5.1 TF-IDF

First, equation (4.2) can be rewritten as

$\left(\frac{s_{i}}{m_{i}^{\textrm{TI}}}\right)^{\alpha}\left(\frac{l_{i}}{m_{i}^{\textrm{TI}}}\right)^{1-\alpha}=\frac{1}{\beta},$

from which the equation

$\log(y_{i}^{\textrm{TI}})=\frac{\alpha}{1-\alpha}\log(x_{i}^{\textrm{TI}})+c_{1}$
(5.1)

follows with the outputs

$y_{i}^{\textrm{TI}}={m_{i}^{\textrm{TI}}\over l_{i}},$ (5.2)

the inputs

$x_{i}^{\textrm{TI}}={s_{i}\over 4m_{i}^{\textrm{TI}}},$ (5.3)

and the constant

$c_{1}=\frac{\alpha}{1-\alpha}\log(4)+\frac{1}{1-\alpha}\log(\beta).$

Equation (5.1) establishes the input-output relationship

$y_{i}^{\textrm{TI}}=h_{0}(x_{i}^{\textrm{TI}})$

with the transfer function

$h_{0}(x)=c_{2}x^{\alpha/(1-\alpha)},$ (5.4)

where

$c_{2}=e^{c_{1}}.$

Since $x_{i}^{\textrm{TI}}\in[0,1]$ (recall Note 4.1), the domain of
definition of $h_{0}$ is the unit interval $[0,1]$.

The notion of transfer function plays a pivotal role in the anomaly detection
method developed by Gribkova and Zitikis (2018, 2019a, 2019b, 2020) and Sun et
al. (2022). Its importance will be seen in the next section. Specifically to
function (5.4), the first derivative of $h_{0}$ is

$h_{0}^{\prime}(x)=c_{3}x^{(2\alpha-1)/(1-\alpha)}$

with a constant $c_{3}$. Hence, since $\alpha\in(0.5,1)$ due to result (4.4)
and since the inputs are in $[0,1]$, we conclude that the derivative
$h_{0}^{\prime}$ is bounded on the interval $[0,1]$. Therefore, the transfer
function $h_{0}$ is Lipschitz continuous on $[0,1]$. This is an important
property as seen from Sun et al. (2022, Theorem 1 (i)).

###  5.2 TF-IDF-VADER

Analogously to Section 5.1, the input-output relationship

$y_{i}^{\textrm{TIV}}=h_{0}(x_{i}^{\textrm{TIV}})$

holds with the same transfer function $h_{0}$ and the analogously defined
inputs

$x_{i}^{\textrm{TIV}}={s_{i}\over 4m_{i}^{\textrm{TIV}}}$ (5.5)

and outputs

$y_{i}^{\textrm{TIV}}={m_{i}^{\textrm{TIV}}\over l_{i}}.$ (5.6)

However, in view of result (4.8), the assumption $\alpha>1$ has to be
considered, in which case the transfer function

$h_{0}(x)=c_{2}x^{\alpha/(1-\alpha)}\leavevmode\nobreak\ \bigg{(}={c_{2}\over
x^{\alpha/(\alpha-1)}}\bigg{)}$

is not Lipschitz continuous on the interval $[0,1]$. Fortunately, it satisfies
another set of conditions of Sun et al. (2022, Theorem 1 (ii)) enabling us to
apply the anomaly detection method.

##  6 Anomaly detection among quantified narratives

In the ideal scenario, for the $i$’s in the predicted meritorious set, the
pairs $(x_{i},y_{i})$, which are either
$(x_{i}^{\textrm{TI}},y_{i}^{\textrm{TI}})$ or
$(x_{i}^{\textrm{TIV}},y_{i}^{\textrm{TIV}})$ depending on the featurization
method, form an anomaly-free input-output system. In reality, anomalies are
inevitable for any input-output system that is not particularly stringent.
Anomalies may even be present in data due to the very nature of the problem.
However, they may not always drastically change the regime of inputs and/or
outputs, and may therefore be difficult to detect for typical classifiers.

For this reason, two indices are next provided based on the pairs
$(x_{i},y_{i})$ within the predicted meritorious set. The indices are
constructed by adapting the method for systematic-anomaly detection developed
by Gribkova and Zitikis (2018, 2019a, 2019b, 2020) in the case of
(quantitative) i.i.d. inputs and subsequently extended to time series inputs
by Sun et al. (2022).

Generally speaking, the indices help to see if there is any background or
systematic risk in the input-output system characterized by transfer functions
$h_{0}$. Namely, they help to distinguish the “null hypothesis” $y=h_{0}(x)$
from the “alternative” $y=h_{0}(x+\delta)$, where the unspecified random
quantity $\delta$ represents the exogenous background risk that naturally
arises in many problems (Gribkova and Zitikis, 2018). An advantage of these
systematic-anomaly detection indices is that neither the transfer function
$h_{0}$ nor the distribution of $\delta$ need to be specified.

In the case of this paper, the featurized text inputs may or may not contain a
background risk, and the transfer function $h_{0}$ is based on some estimated
parameters, whose actual values are not known. Hence, detecting the background
risk when it exists can be done by checking the asymptotic behavior of the
indices without specifying the background risk or the transfer function. This
feature is especially helpful in exploratory studies such as the present one.
Of course, certain conditions are needed for the development of a rigorous
theory, but this is outside the scope of the present paper. For technical
details, we refer to Sun et al. (2022) and references therein.

###  6.1 The $I$-index and findings

Let a simple random sample of pairs $(x_{1},y_{1}),\dots,(x_{n},y_{n})$ form
an input-output system, where the inputs $x_{i}$ and the outputs $y_{i}$ are
assumed to be realizations of continuous random variables. Suppose that there
are no ties among $x_{1},\ldots,x_{n}$, in which case the inputs can be
ordered in a unique way, denoted as $x_{1\mathrel{\mathop{\mathchar
58\relax}}n}<x_{2\mathrel{\mathop{\mathchar
58\relax}}n}<\cdots<x_{n\mathrel{\mathop{\mathchar 58\relax}}n}$. The
corresponding concomitants (see, e.g., David and Nagaraja, 2003) of the
outputs $y_{1},\ldots,y_{n}$ are denoted by $y_{1,n},\dots,y_{n,n}$.

Relying on the theory developed by Gribkova and Zitikis (2018, 2019a, 2019b,
2020) and Sun et al. (2022), the presence of systematic anomalies among the
observed pairs can be assessed by the rate of convergence to $1/2$ of the
$I$-index

$I_{n}={\sum^{n}_{i=2}\big{(}y_{i,n}-y_{i-1,n}\big{)}_{+}\over\sum^{n}_{i=2}\big{|}y_{i,n}-y_{i-1,n}\big{|}},$

where $a_{+}=a$ when $a>0$ and $a_{+}=0$ when $a\leq 0$.

The $I$-index appeared in Davydov and Zitikis (2017) as the solution to an
optimization problem. It was further explored by Chen et al. (2018) and
subsequently extended to the multivariate case by Davydov et al. (2019) to
accommodate diverse applications. The index has found uses in, e.g., education
sciences (Chen and Zitikis, 2017), but most notably in various branches of
engineering and related areas, such as system security (Gribkova and Zitikis,
2019a, 2020), anomaly detection (Sun et al., 2022), electrical engineering
(Chernov et al., 2021; Di Capua et al., 2023a, b; Milano et al., 2023).
mechanical and materials engineering (Kirk, 2020; Kirk et al., 2021; Sauceda
et al., 2022), networks (Gurfinkel, 2020; Gurfinkel and Rikvold, 2020), and
prognostics (Trilla et al., 2020).

Applying Algorithm 6.1

Require
---
1\. A dataset $D$ with labels “meritorious” and “non-meritorious”.
2\. A specific classification method.
Compute a value of the $I$-index
1\. Randomly assign $(1-r)100\%$ of the dataset $D$ as a training set, and the
remaining
$r100\%$ as a testing set.
2\. Train a classifier $f$ using the specific classification method based on
the training set.
3\. Apply $f$ on the testing set, leading to a meritorious subset with the
indices $d_{m}\subseteq D$ of
the testing set.
4\. Compute the $I$-index using $(x_{i},y_{i})$, $i\in d_{m}$.
Algorithm 6.1: Calculating the $I$-index.

on each of the predicted meritorious sets generated by the five classifiers,
we obtain Figure 6.1

(a) LR (b) SVM

(c) GB (d) MLP

(e) RF

Figure 6.1: The plots of $I_{n}$ for the five classifications.

depicting the $I$-index for each of the five classifiers. In Algorithm 6.1,
the parameter $r$ is set to $0.4$.

Note that for these classifiers except for the MLP, the graphs rapidly move in
the direction of $1/2$, although we cannot say with certainty that they will
reach $1/2$ if the sample size tends to infinity. At any rate, given what we
see, it can be concluded that systematic anomalies are present. As to the MLP
classifier, it is difficult to claim from the nearly flat graphs in the case
of both featurizations that there is a tendency to convergence to $1/2$,
although it could be so but at a very slow rate. If it does not converge to
$1/2$, then the properties of the $I$-index imply that the input-output system
is anomaly free.

Given the aforementioned ambiguities, which are natural in any exploratory
study such as the present one, it is prudent to have a supplementary view.
This leads to the next section concerning the $B$-index.

###  6.2 The $B$-index and findings

To introduce the $B$-index, note that for any real number $a$ its positive
part $a_{+}$ can be expressed as $(a+|a|)/2$. Since
$\sum^{n}_{i=2}\big{(}y_{i,n}-y_{i-1,n}\big{)}=y_{n,n}-y_{1,n}$, it follows
that

$\displaystyle I_{n}$
$\displaystyle={\sum^{n}_{i=2}\big{(}y_{i,n}-y_{i-1,n}\big{)}_{+}\over\sum^{n}_{i=2}\big{|}y_{i,n}-y_{i-1,n}\big{|}}$
$\displaystyle={1\over
2}\bigg{(}1+{y_{n,n}-y_{1,n}\over\sum^{n}_{i=2}\big{|}y_{i,n}-y_{i-1,n}\big{|}}\bigg{)}.$

In view of this representation of the $I$-index, the convergence of $I_{n}$ to
$1/2$ is associated with the stochastic growth to infinity of the sum

$S_{n}=\sum^{n}_{i=2}\big{|}y_{i,n}-y_{i-1,n}\big{|}$

when $n\to\infty$. This sum is depicted as a function of $n$ in Figure 6.2

(a) LR (b) SVM

(c) GB (d) MLP

(e) RF

Figure 6.2: The plots of $S_{n}$ for the five classifications.

for the five classifications. To produce the figure, Algorithm 6.1 has been
implemented with $S_{n}$ instead of $I_{n}$, setting the parameter $r$ to
$0.4$. For the purpose of comparison, the vertical axes in all the five panels
of the figure are the same and run from $0$ to $200$.

To parameterize the stochastic growth of the sum $S_{n}$ to infinity when $n$
increases, the $B$-index

$B_{n,p}={1\over
n^{1/p}}\sum^{n}_{i=2}\big{|}y_{i,n}-y_{i-1,n}\big{|}\leavevmode\nobreak\
\bigg{(}={S_{n}\over n^{1/p}}\bigg{)}$

is considered, where $p\in(0,\infty]$ is a parameter. The normalization
$n^{1/p}$ mimics that of the law of large numbers when $p=1$ and that of the
central limit theorem when $p=2$. In the extreme case $p=\infty$, the
normalization is identically equal to $1$, and thus the $B$-index reduces to
$S_{n}$.

By varying the parameter $p$ values, the growth to infinity of $S_{n}$ can be
sensed when $n\to\infty$. This is linked to the notion of reasonable order.
Specifically, the outputs are said to be in $p$-reasonable order with respect
to the inputs for some $p$ if $B_{n,p}$ is stochastically bounded when
$n\to\infty$. Otherwise, that is, when $B_{n,p}$ stochastically grows to
infinity when $n\to\infty$, the outputs are said to be out of $p$-reasonable
order. This notion of in/out of $p$-reasonable order in the case $p=2$ was
introduced by Gribkova and Zitikis (2018, 2019a, 2019b, 2020) and subsequently
extended to arbitrary $p\in(0,\infty]$ by Sun et al. (2022) to accommodate a
whole spectrum of tail heaviness in time series data.

By Sun et al. (2022, Theorem 4), the anomaly-free outputs are in
$p$-reasonable order for $p=\infty$ and therefore for all $p\in(0,\infty]$.
For any two parameter values $p_{1}<p_{2}$, the outputs, which may contain
anomalies, can be out of $p_{2}$-reasonable order but in $p_{1}$-reasonable
order. Hence, by visualizing the behaviour of $B_{n,p_{1}}$ and $B_{n,p_{2}}$
with respect to various sample sizes $n$, we can understand when, with respect
to $p$, the meritorious set becomes in or out of $p$-reasonable order. For
this task, Algorithm 6.1 is implemented with $B_{n,p}$ instead of $I_{n}$. We
do so $k=500$ times and obtain $k$ values of the $B$-index, always setting the
parameter $r$ to $0.4$.

Figures B.1–B.5, which are relegated to Appendix B due to space
considerations, depict the $B$-indices arising from the TF-IDF and TF-IDF-
VADER featurizations. For the purpose of comparison, the vertical axes in all
the panels of the figures are the same and run from $0$ to $5$. It is observed
that in the case of both featurizations and for every classification method,
when the parameter $p$ decreases, that is, when its reciprocal $1/p$ goes
through the values $0.6$, $0.7$, $0.8$ and $0.9$, the meritorious sets shift
at some point from being out of $p$-reasonable order to being in
$p$-reasonable order. This pattern is, of course, natural as the normalizing
factor $n^{1/p}$ gets heavier for larger $1/p$, thus slackening the growth of
the sum $S_{n}$ when systematic anomalies are present.

From Figures B.1–B.3, which refer to the LR, SVM, and GB classifiers, the TF-
IDF featurization generally leads to larger sample sizes than the TF-IDF-VADER
featurization. In the MLP case, it can be seen from Figure B.4 that the sample
sizes, in general, become similar. In the RF case, as seen from Figure B.5, it
is the TF-IDF-VADER featurization that generally leads to larger sample sizes
than the TF-IDF featurization.

Note also from Figures B.1–B.5 that in the case of the TF-IDF featurization,
the system tends to shift from out of $0.7^{-1}$-reasonable order to in
$0.8^{-1}$-reasonable order, whereas in the case of the TF-IDF-VADER
featurization, the tendency to shift from out of $0.6^{-1}$-reasonable order
to in $0.7^{-1}$-reasonable order is observed.

To formally check if the resulting meritorious set is in $p$-reasonable order,
an index $p^{*}$ is defined as the largest $p\geq 1$ such that the $B$-index
$B_{n,p}$ is stochastically bounded when $n\to\infty$. Of course, depending on
the featurization, there are two $p^{*}$’s: one is $p^{*}_{\text{TI}}$ arising
from the TF-IDF featurization, and the second one is $p^{*}_{\text{TIV}}$
arising from the TF-IDF-VADER featurization. It is seen from the figures that,
roughly speaking,

$p^{*}_{\text{TI}}\in(0.8^{-1},0.7^{-1})\quad\textrm{and}\quad
p^{*}_{\text{TIV}}\in(0.7^{-1},0.6^{-1})$

for all the five classification methods. This implies
$p^{*}_{\text{TIV}}>p^{*}_{\text{TI}}$ regardless of the classification method
and suggests that there exists
$p^{\prime}\in(p^{*}_{\text{TI}},p^{*}_{\text{TIV}})$ such that, for any given
classification method, the $B$-index $B_{n,p^{\prime}}$ is stochastically
bounded if the TF-IDF-VADER featurization is used but stochastically grows to
infinity if the TF-IDF featurization is used. Therefore, the TF-IDF-VADER
featurization leads to a relatively higher extent of anomaly absence because
the $B$-index $B_{n,p^{\prime}}$ rejects the TF-IDF case as being anomaly free
but cannot reject the TF-IDF-VADER case as being such. When making such
interpretations, smaller $p^{*}$’s always suggest heavier anomalies in the
system as they give rise to larger $(1/p)$’s ($p<p^{*}$), thus producing
heavier normalizing factors $n^{1/p}$ needed to slacken the growth of the sum
$S_{n}$ with respect to $n\to\infty$.

##  7 Conclusion

A method for detecting systematic anomalies among texts such as consumer
complaint narratives has been proposed. In addition to classification
algorithms, quantitative indices have been employed to assists in anomaly
detection. The indices are essential for detecting those anomalies that mimic
meritorious complaints without changing distributional patterns of the word
counts, discounted dollar amounts, and sentiment scores. They also play
pivotal roles when automating the procedure. Repeated classifications on
randomly segmented training and testing datasets have been employed to
visualise the index performance in different scenarios.

All the procedural steps have been illustrated using consumer complaints from
the Consumer Complaint Database of the Consumer Financial Protection Bureau.
Our results have shown that the TF-IDF-VADER featurization outperforms the TF-
IDF featurization in terms of the $I$\- and $B$-indices regardless of the
chosen classification method, although the TF-IDF featurization consistently
leads to slightly higher classification accuracy than the TF-IDF-VADER
featurization. This suggests that, in practice, the TF-IDF-VADER featurization
could be applied to consumer complaints to identify those with higher priority
to receive reliefs, because the resulting meritorious group is less likely to
contain non-meritorious complaints.

## Acknowledgements

We are indebted to two anonymous referees for queries and suggestions that
helped us to clarify and improve the manuscript. This research is a part of
the project “The State of the Art in Anomaly Detection and Model Construction,
with the Focus on Natural Language Processing (NLP) in Actuarial Modelling”
supported by the Committee on Knowledge Extension Research (CKER) of the
Society of Actuaries (SOA) Research Institute, and the Casualty Actuarial
Society (CAS). The research has also been partially supported by the NSERC
Alliance–MITACS Accelerate grant entitled “New Order of Risk Management:
Theory and Applications in the Era of Systemic Risk” from the Natural Sciences
and Engineering Research Council (NSERC) of Canada, and the national research
organization Mathematics of Information Technology and Complex Systems
(MITACS) of Canada, as well as by the individual NSERC Discovery Grant
“Automated Statistical Techniques for Systematic Anomaly Detection in High
Frequency Data” of R. Zitikis.

## Declarations of interest

The authors report no conflicts of interest. The authors alone are responsible
for the content and writing of the paper.

## References

  * Bollacker et al. (1998) Bollacker, K.D., Lawrence, S. and Giles, C.L. (1998). CiteSeer: an autonomous web agent for automatic retrieval and identification of interesting publications. Proceedings of the Second International Conference on Autonomous Agents, 116-123.
  * CFPB (2023) CFPB (2023). Consumer Complaint Database. Consumer Financial Protection Bureau, Washington, DC.
  * CPI (2023) CPI (2023). Consumer Price Index, 1913-. Federal Reserve Bank of Minneapolis, Mineapolis, MN.
  * Chatterjee et al. (2018) Chatterjee, S., Deng, S., Liu, J., Shan, R. and Jiao W. (2018) Classifying facts and opinions in Twitter messages: a deep learning-based approach. Journal of Business Analytics, 1(1), 29–39.
  * Chen et al. (2018) Chen, L., Davydov, Y., Gribkova, N. and Zitikis, R. (2018). Estimating the index of increase via balancing deterministic and random data. Mathematical Methods of Statistics, 27, 83–102.
  * Chen and Zitikis (2017) Chen, L. and Zitikis, R. (2017). Measuring and comparing student performance: a new technique for assessing directional associations. Education Sciences, 7, 1–27 (article #77).
  * Chernov et al. (2021) Chernov, N.I., Butyrlagin, N.V., Prokopenko, N.N. and Yugai, V.Y. (2021). Synthesis and circuitry of multi-valued digital current logic elements: “classical” linear-monotonic approach. International Conference on Electrical Engineering and Photonics (EExPolytech), St. Petersburg, Russian Federation, 2021, pp. 104-107,
  * David and Nagaraja (2003) David, H.A. and Nagaraja, H.N. (2003). Order Statistics. (Third Edition.) Wiley, Hoboken.
  * Davydov et al. (2019) Davydov, Y., Moldavskaya, E. and Zitikis, R. (2019). Searching for and quantifying nonconvexity regions of functions. Lithuanian Mathematical Journal (Special issue dedicated to Professor Vygantas Paulauskas on the occasion of his 75th birthday), 59, 507–518.
  * Davydov and Zitikis (2017) Davydov, Y. and Zitikis, R. (2017). Quantifying non-monotonicity of functions and the lack of positivity in signed measures. Modern Stochastics: Theory and Applications, 4, 219–231.
  * Di Capua et al. (2023a) Di Capua, G., Bourelly, C., De Stefano, C., Fontanella, F., Milano, F., Molinara, M., Oliva, N. and Porpora, F. (2023). Using genetic programming to learn behavioral models of lithium batteries. In: Correia, J., Smith, S. and Qaddoura, R. (eds) Applications of Evolutionary Computation. EvoApplications. Lecture Notes in Computer Science, vol 13989, pp. 461–474. Springer, Cham.
  * Di Capua et al. (2023b) Di Capua, G., Oliva, N., Milano, F., Bourelly, C., Porpora, F., Maffucci, A. and Femia, N. (2023). A behavioral model for lithium batteries based on genetic programming. IEEE International Symposium on Circuits and Systems (ISCAS), Monterey, CA, USA, pp. 1–5.
  * Di Vincenzo et al. (2023) Di Vincenzo, D., Greselin, F., Piacenza, F. and Zitikis, R. (2023). A text analysis of operational risk loss descriptions. Journal of Operational Risk, 18, 63–90.
  * Di Vincenzo et al. (2024) Di Vincenzo, D., Greselin, F., Piacenza, F. and Zitikis, R. (2024). A tweet data analysis for detecting emerging operational risks. In: Proceedings of the 11th International Conference on “Mathematical and Statistical Methods for Actuarial Sciences and Finance” (MAF 2024), University of Le Havre Normandie, Le Havre Cedex, France. Springer, Berlin.
  * Gribkova and Zitikis (2018) Gribkova, N. and Zitikis, R. (2018). A user-friendly algorithm for detecting the influence of background risks on a model. Risks (Special issue on “Risk, Ruin and Survival: Decision Making in Insurance and Finance”), 6, 1–11. (Article #100.)
  * Gribkova and Zitikis (2019a) Gribkova, N. and Zitikis, R. (2019a). Assessing transfer functions in control systems. Journal of Statistical Theory and Practice, 13, 1–33. (Article #35.)
  * Gribkova and Zitikis (2019b) Gribkova, N. and Zitikis, R. (2019b). Statistical detection and classification of background risks affecting inputs and outputs. Metron – International Journal of Statistics, 77, 1–18.
  * Gribkova and Zitikis (2020) Gribkova, N. and Zitikis, R. (2020). Detecting intrusions in control systems: a rule of thumb, its justification and illustrations. Journal of Statistics and Management Systems, 23, 1285–1304.
  * Gurfinkel (2020) Gurfinkel, A.J. (2020). Centrality Reach and Grasp: Analyzing the Attenuation of Influence Flows in Parametrized Centrality Measures on Complex Networks. Ph.D. Thesis. The Florida State University, Florida.
  * Gurfinkel and Rikvold (2020) Gurfinkel, A.J. and Rikvold, P.A. (2020). Absorbing random walks interpolating between centrality measures on complex networks. Physical Review E, 101, 1–21, Article 012302.
  * Hayes et al. (2021) Hayes, R.M., Jiang, F. and Pan, Y. (2021). Voice of the customers: Local trust culture and consumer complaints to the CFPB. Journal of Accounting Research, 59, 1077–1121.
  * Hsu (2020) Hsu, B.M. (2020) Comparison of supervised classification models on textual data. Mathematics, 8(5), Article #851 (16 pages).
  * Hutto and Gilbert (2014) Hutto, C. and Gilbert, E. (2014). Vader: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, 8, 216–225.
  * Ji et al. (2023) Ji, Z., Duan, X., Pons, D., Chen, Y. and Pei, Z. (2023). Integrating text mining and analytic hierarchy process risk assessment with knowledge graphs for operational risk analysis. Journal of Operational Risk, 18, 31–61.
  * Karami and Pendergraft (2018) Karami, A. and Pendergraft, N.M. (2018). Computational Analysis of Insurance Complaints: Geico Case Study. Available at arXiv:1806.09736
  * Khurana and Verma (2022) Khurana, A. and Verma, O. P. (2022). Optimal feature selection for imbalanced text classification. IEEE Transactions on Artificial Intelligence, 4(1), 135–147.
  * Kirk (2020) Kirk, T.Q. (2020). Computational Design of Compositionally Graded Alloys. Ph.D. Thesis. Texas A&M University, Texas.
  * Kirk et al. (2021) Kirk, T., Malak, R. and Arroyave, R. (2021). Computational design of compositionally graded alloys for property monotonicity. Journal of Mechanical Design, 143, Article #031704 (9 pages).
  * Klein (1995) Klein, R.W. (1995). Insurance regulation in transition. Journal of Risk and Insurance, 62, 363–404.
  * Liao et al. (2020) Liao, X., Chen, G., Ku, B., Narula, R. and Duncan, J. (2020). Text mining methods applied to insurance company customer calls: a case study. North American Actuarial Journal, 24, 153–163.
  * Littwin (2014) Littwin, A. (2014). Why process complaints? Then and now. Temple Law Review, 87, 895–946.
  * Long et al. (2023) Long, S., Lucey, B., Xie, Y. and Yarovaya, L. (2023). “I just like the stock”: The role of Reddit sentiment in the GameStop share rally. Financial Review, 58, 19–37.
  * Milano et al. (2023) Milano, F., Di Capua, G., Oliva, N., Porpora, F., Bourelly, C., Ferrigno, L. and Laracca, M. (2023). An analytical model for lithium-ion batteries based on genetic programming approach. IEEE International Workshop on Metrology for Automotive (MetroAutomotive), Modena, Italy, pp. 35–40.
  * NAIC (2022) NAIC (2022). 2021 Insurance Department Resources Report, Volume 1. National Association of Insurance Commissioners, Washington, DC.
  * NAIC (2023) NAIC (2023). Insurance Departments. National Association of Insurance Commissioners, Washington, DC.
  * Onita (2023) Onita, D. (2023). Active Learning Based on Transfer Learning Techniques for Text Classification. IEEE Access, 11, 28751–28761.
  * Osman and Sabit (2022) Osman, S.M.I. and Sabit, A. (2022). Bank Scandal and Customer Sentiment. Available at SSRN 4035168.
  * Pakhchanyan et al. (2022) Pakhchanyan, S., Fieberg, C., Metko, D. and Kaspereit, T. (2022). Machine learning for categorization of operational risk events using textual description. Journal of Operational Risk, 17, 37–65.
  * Sanders et al. (2021) Sanders, A.C., White, R.C., Severson, L.S., Ma, R., McQueen, R., Paulo, H.C.A., Zhang, Y., Erickson, J.S. and Bennett, K.P. (2021). Unmasking the conversation on masks: Natural language processing for topical sentiment analysis of COVID-19 Twitter discourse. AMIA Summits on Translational Science Proceedings, 555.
  * Sauceda et al. (2022) Sauceda, D., Singh, P., Ouyang, G., Palasyuk, O., Kramer, M.J. and Arróyave, R. (2022). High throughput exploration of the oxidation landscape in high entropy alloys. Materials Horizons, 9, 2644–2663.
  * Spärck Jones (1972) Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28, 11–21.
  * Sun et al. (2022) Sun, N., Yang, C. and Zitikis, R. (2022). Detecting systematic anomalies affecting systems when inputs are stationary time series. Applied Stochastic Models in Business and Industry, 38, 512–544.
  * Trilla et al. (2020) Trilla, A., Fernández, V. and Cabré, X. (2020). Enhancing railway pantograph carbon strip prognostics with data blending through a time-delay neural network ensemble. Annual Conference of the PHM Society, 12, 1–9.
  * Wang et al. (2022) Wang, Y., Chang, Y. and Li, J. (2022). How does the pandemic change operational risk? Evidence from textual risk disclosures in financial reports. Journal of Operational Risk, 17, 1–24.
  * Wood and Morris (2010) Wood, G.L. and Morris, R.L. (2010). Consumer complaints against insurance companies. Journal of Personal Finance, 9, 101–116.

##  Appendix A Steps for corpus cleaning

The collection of those complaint narratives that we analyze is referred to as
the “corpus”. As suggested by Di Vincenzo et al. (2023), the following steps
have been implemented to clean the narratives in the corpus:

  1. (1)

Ignoring cases, i.e., converting each uppercase letter into the corresponding
lowercase one.

  2. (2)

Removing punctuations, except semantic punctuations such as “!” and “?”.

  3. (3)

Removing stop words (common words with no sentiment information) as well as
dollar amounts from the corpus.

  4. (4)

Removing common and frequent words (e.g., frequent nouns that do not have
sentiment information) from the corpus.

  5. (5)

Removing the sentences that begin with “thanks” or “thank you”.

Table A.1 illustrates these steps using excerpts from several complaint
narratives accompanied with their cleaned versions.

Original narratives |  Cleaned narratives
---|---
Macys did not reverse out my $230.00 dispute Similar to cfpg XXXX |  macys not reverse dispute similar cfpg
Inquired about $300.00 increase and did not receive a response to the request |  inquired increase not receive response request
Order cancelled and never delivered. still charged full amount of $7000.00 on my credit card |  order cancelled never delivered still charged full amount credit card
A charge of $170.00 was made to my card I did not make or authorize charge. |  charge made card not make authorize charge
Someone fraudulently charged $750.00 from XXXX on our card and Citi will not take it off!!!!!!!!!! |  someone fraudulently charged card citi not take !!!!!!!!!!
Table A.1: Selected complaint narratives and their cleaned versions.

When cleaning, the dollar amounts are removed because the VADER approach is
unable to ascertain the sentiment scores associated with numerical values.
However, the exclamation “!” and question “?” marks are not removed because
they have strong VADER sentiment intensities. That is, different VADER
sentiment intensities can arise for the same narratives with or without these
two punctuation marks.

##  Appendix B The $B$-index and samples sizes

(a) $p=0.6^{-1}$ (b) $p=0.7^{-1}$

(c) $p=0.8^{-1}$ (d) $p=0.9^{-1}$

Figure B.1: The plots of $B_{n,p}$ (vertical axis) for various sample sizes
$n$ (horizontal axis) and parameter $p$ values when using the LR
classification.

(a) $p=0.6^{-1}$ (b) $p=0.7^{-1}$

(c) $p=0.8^{-1}$ (d) $p=0.9^{-1}$

Figure B.2: The plots of $B_{n,p}$ (vertical axis) for various sample sizes
$n$ (horizontal axis) and parameter $p$ values when using the SVM
classification.

(a) $p=0.6^{-1}$ (b) $p=0.7^{-1}$

(c) $p=0.8^{-1}$ (d) $p=0.9^{-1}$

Figure B.3: The plots of $B_{n,p}$ (vertical axis) for various sample sizes
$n$ (horizontal axis) and parameter $p$ values when using the GB
classification.

(a) $p=0.6^{-1}$ (b) $p=0.7^{-1}$

(c) $p=0.8^{-1}$ (d) $p=0.9^{-1}$

Figure B.4: The plots of $B_{n,p}$ (vertical axis) for various sample sizes
$n$ (horizontal axis) and parameter $p$ values when using the MLP
classification.

(a) $p=0.6^{-1}$ (b) $p=0.7^{-1}$

(c) $p=0.8^{-1}$ (d) $p=0.9^{-1}$

Figure B.5: The plots of $B_{n,p}$ (vertical axis) for various sample sizes
$n$ (horizontal axis) and parameter $p$ values when using the RF
classification.