[]

Writing – original draft, Methodology, Data curation, Implementation,
Experimental analysis, Conceptualization

1]organization=Department of Computer Science and Engineering, BGC Trust
University Bangladesh, city=Chittagong, postcode=4381, country=Bangladesh

[]

Writing – review & editing

2]organization=Department of Computer Science and Engineering, Military
Institute of Science and Technology, city=Dhaka, postcode=1216,
country=Bangladesh

[]

Writing – review & editing

3]organization=School of Computing, Edinburgh Napier University, country=UK

[]

Writing – review & editing

[ orcid=https://orcid.org/0000-0003-1740-5517]

[1]

Writing – review & editing, Conceptualization and Supervision

4]organization=Centre for Securing Digital Futures, Edith Cowan University,
city=Perth, postcode=WA-6027, country=Australia

[cor1]Corresponding author

# ExplainableDetector: Exploring Transformer-based Language Modeling Approach
for SMS Spam Detection with Explainability Analysis

Mohammad Amaz Uddin<EMAIL_ADDRESS>[  Muhammad Nazrul Islam  [
Leandros Maglaras  [  Helge Janicke  Iqbal H. Sarker<EMAIL_ADDRESS>[

###### Abstract

SMS, or short messaging service, is a widely used and cost-effective
communication medium that has sadly turned into a haven for unwanted messages,
commonly known as SMS spam. With the rapid adoption of smartphones and
Internet connectivity, SMS spam has emerged as a prevalent threat. Spammers
have taken notice of the significance of SMS for mobile phone users.
Consequently, with the emergence of new cybersecurity threats, the number of
SMS spam has expanded significantly in recent years. The unstructured format
of SMS data creates significant challenges for SMS spam detection, making it
more difficult to successfully fight spam attacks in the cybersecurity domain.
In this work, we employ optimized and fine-tuned transformer-based Large
Language Models (LLMs) to solve the problem of spam message detection. We use
a benchmark SMS spam dataset for this spam detection and utilize several
preprocessing techniques to get clean and noise-free data and solve the class
imbalance problem using the text augmentation technique. The overall
experiment showed that our optimized fine-tuned BERT (Bidirectional Encoder
Representations from Transformers) variant model RoBERTa obtained high
accuracy with 99.84%. We also work with Explainable Artificial Intelligence
(XAI) techniques to calculate the positive and negative coefficient scores
which explore and explain the fine-tuned model transparency in this text-based
spam SMS detection task. In addition, traditional Machine Learning (ML) models
were also examined to compare their performance with the transformer-based
models. This analysis describes how LLMs can make a good impact on complex
textual-based spam data in the cybersecurity field.

###### keywords:

CybersecurityMachine Learning (ML)Large Language Model (LLM)Spam
DetectionExplainable AI (XAI)RoBERTa Transformers

##  1 Introduction

In recent years, SMS has become the primary communication tool as mobile
phones and networks have proliferated Liu et al. (2021). It is quite popular
since it’s quick and convenient, and it responds to messages faster than
emails Oswald et al. (2022). Due to its convenience, accessibility, and broad
use in our increasingly interconnected society, SMS is now the most popular
form of communication for both individuals and organizations. Day by day, SMS
has developed into a powerful tool for global communication due to its several
benefits, such as simplicity of use, cost-effectiveness, wide reach, and more.
Additionally, SMS promotes customer relationships, provides company updates,
and provides discreet individualized communication, all of which contribute to
real-time audience engagement. It is effectively used in various industries,
including marketing, retail, healthcare, and finance. However, because of
these benefits, an increasing number of mobile users are utilizing SMS
facilities, which encourages spammers to generate spam SMS attacks aimed at
harming users Hossain et al. (2022). Spammers primarily gather phone numbers
via a variety of methods and then utilize automated software services to
deliver large volumes of messages to these numbers. Phishing URLs, fraudulent
schemes, and other fraudulent content are promoted in these bulk messages.
Financial gain is their primary objective, which is attained by purchasing
counterfeit items, obtaining personal information, or clicking on malicious
links that download malware Wei and Nguyen (2020). Although spam detection is
a challenging task in the cybersecurity field, significant enhancements to
these efforts can be achieved by leveraging Artificial Intelligence (AI)
methods, including Machine Learning, Deep Learning, Hybrid Learning, and
Transfer Learning, which have proven effective in enhancing spam detection
efforts Sarker (2023). Furthermore, due to rapid technological advancements,
novel AI-based methods are being developed daily within the cybersecurity
field Amaz Uddin and Sarker (2024).

LLMs research, supported by both industry and academia, has been essential in
the recent advances in natural language interpretation Zhao et al. (2023). The
field of artificial intelligence research and application could undergo a
revolutionary shift because of these models. These language models trained on
expansive datasets, they excel in capturing nuanced linguistic patterns.
Utilizing the power of LLMs, companies like Google, Microsoft, Amazon, and
OpenAI are developing different AI tools that are skilled at understanding and
providing natural language responses to user’s requests. Notably, OpenAI’s
ChatGPT and Bard exemplify the transformative power of LLMs, captivating
widespread attention for their ability to generate and understand human-like
text Yao et al. (2023). For natural language processing applications,
transformer-based models such as GPT (Generative Pre-trained Transformer) and
Google’s BERT (Bidirectional Encoder Representations from Transformers) are
referred to as LLMs because of their capacity to understand and generate text
that is similar to that of a human. These transformer-based models are useful
for a variety of tasks in NLP research and applications because of their
versatility in handling translation, generation, and language comprehension.

In the field of NLP, a Transformer Vaswani et al. (2017) refers to a deep
learning model architecture that is becoming more popular in text
classification, and translation tasks. Its creative architecture makes use of
attention mechanisms to reliably encode input data into representations,
providing unmatched performance across a range of natural language processing
applications Han et al. (2021). Transformer-based models have brought in a new
era of deep learning capabilities, demonstrating their transformational impact
on the discipline in areas ranging from spam categorization to text
comprehension and interpretation Jamal et al. (2023). Recently, improved
transformer-based models, such as BERT, were presented to address several NLP
problems. The BERT-language representation model is a transformer-based neural
network architecture that has been pre-trained. It is extremely effective and
useful for a variety of text-related applications, such as spam and phishing
detection Koroteev (2021). By utilizing bidirectional attention mechanisms,
BERT can comprehend the complex relationships between words and phrases within
a sentence by grasping contextual word embeddings Mohammed and Ali (2021). The
ability of BERT to take into account context from both ends (left and right)
of a sentence sets it apart from previous models that were restricted to
unidirectional processing Devlin et al. (2018). This bidirectional approach
gives BERT a better understanding of the language nuances and relationships
present in the text. Using a Masked Language Model (MLM) approach, BERT trains
the model to infer the original words from context by substituting specific
words in a sentence with a unique token ([MASK]). Furthermore, BERT has other
variations, including BERT-base, BERT-large Anil et al. (2021), ALBERT Lan et
al. (2019), RoBERTa Liu et al. (2019), DistilBERT Sanh et al. (2019), and
TinyBERT Jiao et al. (2019) among others. These variations vary in the number
of transformer layers and parameters, and each one provides unique benefits
and enhancements for tasks involving natural language processing.

BERT is useful in a variety of NLP tasks, but because of its complexity, it is
sometimes seen as a "black box" making it difficult to comprehend how it makes
predictions. XAI techniques have become crucial tools for clarifying complex
procedures Sarker (2024); Xu et al. (2019); Linardatos et al. (2020). Through
exploring the model’s inner workings, these methods provide insightful
explanations for why particular predictions are made. By using XAI methods,
researchers can clarify the keywords or characteristics that the model
identified as most important for certain activities, which improves clarity
and transparency. In this research work, our main aim is to clarify the
transparency of the model in the SMS spam classification task. In summary, our
major contributions to the work can be outlined as follows:

  * •

We explore transformer-based LLMs for SMS spam detection. To make the model
more efficient and impactful on textual-based spam data, we optimized and
fine-tuned the DistilBERT and RoBERTa models.

  * •

We calculate the positive coefficients and negative coefficients of the text
using XAI techniques to find out the spam and ham messages. This also helps to
explain and investigate the interpretability of the fine-tuned model.

  * •

We visualize the experimental performance comparison between ML models and
LLMs using balanced and imbalanced datasets. To address the class imbalance
problem in the data, we employ a text augmentation technique. This also
analyzes the impact of imbalanced and balanced data.

The remainder of the paper is organized as follows: Section 2 reviews related
works. Section 3 presents a comprehensive explanation of the overall
methodology. Section 4 conducts an in-depth analysis of the experimental
results. Section 5 examines the transparency of the model using XAI
approaches. Furthermore, Section 6 provides a detailed discussion of the
findings and results. Finally, Section 7 concludes and discusses future work.

##  2 Literature Review

In the rapidly evolving cybersecurity landscape, combating spam SMS messages
has become a major opportunity and challenge due to its sophisticated and
increasingly significant nature in today’s increasingly interconnected world.
Indeed, researchers have created several solutions for dealing with spam SMS
by utilizing cutting-edge methods from machine learning, deep learning, and
hybrid models. These initiatives highlight how constantly evolving the sector
is, with new techniques like NLP, anomaly detection, pattern recognition, and
ensemble methods being employed to identify and counter cyber threats as they
arise.

###  2.1 Machine Learning Techniques

To prevent security threats, maintain customer engagement, safeguard their
reputation, enhance operational efficiency, and comply with regulations,
organizations must quickly detect spam attempts. By implementing efficient
spam detection systems, businesses can preserve customer confidence and secure
communication channels. Many methods utilized in the field of SMS spam
classification tasks in the past have contributed to achieving these
objectives. Abid et al. (2022) presented a method that utilizes supervised
machine learning techniques, including TF-IDF and bag-of-words for feature
extraction, to classify spam and ham SMS. To address data imbalance,
oversampling and undersampling techniques were implemented. Among several
machine learning classifiers tested, random forest achieved a remarkable 99%
accuracy in distinguishing between spam and ham SMS. The study Mishra and Soni
(2023) represented a two-phase smishing (SMS phishing) detection model: Domain
Checking Phase verifies URL (Uniform Resource Locator) authenticity, while the
SMS Classification Phase extracts text features. By using the Backpropagation
Algorithm, the system outperforms traditional classifiers and classifies
messages with an accuracy of 97.93% which demonstrates high efficiency in
detecting smishing messages. Sonowal (2020) showed comprehensive analysis
which underscores the effectiveness of feature selection techniques in
enhancing Smishing detection accuracy. This work investigates feature ranking
using four correlation algorithms such as Pearson rank correlation, Spearman’s
rank correlation, Kendall rank correlation, and Point biserial rank
correlation alongside a machine learning approach for Smishing detection,
revealing AdaBoostclassifier as the most accurate. Furthermore, with the
ranking algorithm that is Kendall, rank correlation offered better accuracy
than the other correlation algorithms that is 98.40%. A machine learning model
using the Naïve Bayes methodology presented for SMS spam detection and
classification in ref. Asaju et al. (2021). Using string-to-word vector
feature extraction, the model efficiently handles data from the UCL
repository, demonstrating its high efficacy in detecting new SMS spam with
impressive classification accuracies of 99.42% for correctly classified
samples and 0.57% for incorrectly classified ones. Several researchers have
proposed a naive Bayes approach Mishra and Soni (2020); Joo et al. (2017) for
classifying text messages as spam or ham.

###  2.2 Deep Learning Techniques

In the cybersecurity field, deep learning algorithms have become
indispensable, due to their ability to process complex data structures
allowing them to detect subtle patterns and anomalies. Through the utilization
of deep learning, cybersecurity experts can strengthen defense systems against
constantly changing dangers, guaranteeing strong safeguarding for critical
data and digital assets. Xia and Chen (2020) presented a unique method that
leverages word order information to address the problem of lower frequency
concerns in SMS spam detection: discrete Hidden Markov Models (HMM). The
suggested HMM approach showed strong performance equivalent to deep learning
techniques like CNN (Convolutional Neural Network) and LSTM (Long Short-Term
Memory) models through experiments on the UCI SMS spam dataset and a Chinese
SMS spam dataset. It achieved 95.9% accuracy on the UCI dataset and
demonstrates language-agnostic effectiveness in identifying spam SMS. Ghourabi
et al. (2020) suggested using a hybrid deep learning model to identify spam
SMS messages in an English and Arabic-supporting mixed mobile environment. CNN
and LSTM, two deep learning techniques, are combined in this detection model.
Other popular machine learning algorithms were also used to assess the overall
procedure, and the results showed that the CNNLSTM model that was suggested
performed better than the others, achieving an accuracy of 98.37%. The study
Roy et al. (2020) employed two deep learning techniques together: CNN and LSTM
to classify Spam and Not-Spam text messages. They used a spam dataset where
747 Spam and 4,827 Not-Spam text messages exist. To evaluate the performance
of the proposed method, the hybrid model was compared with the traditional
machine learning models and achieved a comparatively high accuracy of 99.44%.
In ref. Abayomi-Alli et al. (2022) a deep learning model built on top of
BiLSTM is presented, and its effectiveness is evaluated against the latest
machine learning techniques on two datasets: the UCI SMS dataset and a
recently gathered dataset named ExAIS_SMS. The BiLSTM model outperformed
certain ML classifiers, showing a moderate improvement in accuracy over some,
with 93.4% accuracy for the ExAIS_SMS dataset and 98.6% accuracy for the UCI
dataset, after evaluation using many metrics including TP, FP, F-measure,
recall, precision, and overall accuracy. Further supporting the superiority of
the BiLSTM model in spam identification, the comparison also showed
significant accuracy differences among ML classifiers.

###  2.3 Transformer model-based Techniques

Transformer-based models like BERT and GPT are very useful for processing and
comprehending textual data, and they can also be applied to improve
cybersecurity defenses and address the ever-changing cyber threat landscape.
The study Sahmoud and Mikki (2022) demonstrated how well BERT performs
comprehensive context analysis to enhance spam detection and produce better
classification results. Using a pre-trained BERT model, a proposed high-
performance spam detector obtains impressive accuracy rates (from 97.83% to
99.28%) on a variety of corpora such as SMS spam collections, Enron,
SpamAssassin, and Ling-Spam. For SMS spam identification, Liu et al. (2021)
introduced both a customized Transformer model and a Vanilla Transformer model
with modifications. They utilized the GloVe method for text message vector
representations, achieving an accuracy of 98.92%. Using a Kaggle dataset, the
authors Hossain et al. (2022) introduced a spam detection model that used BERT
to outperform both deep learning techniques and conventional machine learning
algorithms, with an accuracy of 98.80%. Notably, it optimizes hyperparameters
to speed up model selection and improves model performance by training BERT
with extra layers on a benchmark dataset. Oswald et al. (2022) addresses
dynamic keywords well by concentrating on word semantics in SMS spam filtering
through the use of an intention-based approach. Different pre-trained NLP
models are used to provide contextual embeddings, and 13 pre-specified
intention labels are used to capture the textual and semantic components of
short-text messages. By computing intention scores and using supervised
learning classifiers, the model obtains impressive results on the SMS Spam
Collection benchmark dataset, with an accuracy of 98.07%. The authors Tida and
Hsu (2022) highlighted the vital role of self-attention mechanisms in enabling
BERT models to detect spam effectively. They developed a Universal Spam
Detection Model (USDM) trained on four different publicly available datasets:
Enron, Spamassain, Lingspam, and Spam Text datasets. By fine-tuning with the
same hyperparameters and ensuring F1-scores above 0.9 for each dataset when
using the corresponding model, this approach achieved an accuracy of 97% with
an F1 score of 0.96.

The literature reviews on SMS spam detection tasks highlight existing research
works that demonstrate good performance using ML, deep learning, and
transformer-based models. However, although those proposed techniques and
models work well, but several types of questions can arise, such as “How do
the black-box models work?” and “Are the predictions of those black-box models
trustworthy or not?”. To answer those questions, XAI has become a prominent
field. The XAI approach improves the black-box model’s explainability and
transparency so that they can make more interpretable decisions. In this
research work, two popular XAI techniques have been employed to answer those
questions and make the proposed approach trustworthy.

##  3 Methodology

In this research work, we have tried to demonstrate the effectiveness of LLMs,
such as transformer-based self-attention mechanism BERT models that can make a
significant impact in the cybersecurity field, particularly in SMS spam
detection. Initially, we collected a benchmark dataset that had already been
used in other research for this spam binary classification task. We prepared
this dataset for further experiments using several preprocessing techniques
and data balancing techniques, resulting in clean, noise-free, and balanced
data. After preprocessing, for this spam detection, we selected several
traditional ML models as well as two BERT variants DistilBERT and RoBERTA
models. We trained those transformer-based models with the prepared dataset by
applying optimization and fine-tuning mechanisms. After analyzing the
performance of the ML and BERT models we showed comparisons between them.
Finally, two well-known XAI techniques LIME and Transformers Interpret are
employed to explain how the fine-tuned BERT variant model is working and
making decisions in this spam classification. The methodology of the research
work is presented in Fig. 1.

Figure 1: The methodology of SMS Spam Detection.

###  3.1 Data Collection

A publicly available SMS spam dataset has been collected from the UCI machine
learning repository for this experiment. This dataset is labeled and also
collected for mobile phone spam research. There are 5,574 English SMS messages
in the dataset, all of which are classified as "spam" or "ham". Fig. 2 shows a
sample of the data. The dataset shows that in between 5,574 data, 4825 data
existed as “ham” data and 747 as “spam”. This distribution of data indicates
that the dataset is not well-balanced.

Figure 2: The sample data overview

###  3.2 Data Preprocessing

To prepare the dataset, we need to clean the data by solving different types
of problems that exist in this data. Mainly we focused on the minor changes in
the data because using several preprocessing techniques can potentially reduce
the accuracy of NLP tasks. Several preprocessing techniques such as
punctuation, and stopword removal can inadvertently remove important
information from the text. For this spam detection task, we solely clean the
data by replacing non-alphanumeric characters in the text with a space.
Afterward, we replaced all whitespace characters, such as spaces, tabs, and
newlines, with a single space. Then, we removed all carriage characters,
replacing them with spaces. Finally, we removed all repeated words from the
text. To increase the accuracy of the model, preprocessing the SMS data was
essential since it highlighted important textual characteristics and patterns
that were required to recognize spam messages. Algorithm 1 shows the
preprocessing process.

Algorithm 1 Text Preprocessing

1:Text to be preprocessed

2:Cleaned and preprocessed text

3:function Clean_Text(text)

4: text $\Leftarrow$ Replace_Special_Characters(text)

5: text $\Leftarrow$ Remove_Extra_Spaces(text)

6: text $\Leftarrow$ Remove_Carriage_Returns(text)

7: return text

8:end function

9:function Remove_Repeat(text, repeat)

10: words $\Leftarrow$ Split_Text(text)

11: result $\Leftarrow$ []

12: for word in words do

13: if Count_Occurrences(result, word) $<$ repeat then

14: Append(result, word)

15: end if

16: end for

17: return Join_Words(result)

18:end function

19:function Preprocess_Text(text)

20: text $\Leftarrow$ Clean_Text(text)

21: text $\Leftarrow$ Remove_Repeat(text, 1)

22: return text

23:end function

After the data pre-processing phase, we employed a text augmentation technique
back translation, and data resampling to balance the data. This augmentation
method involves translating the original text data into one or more languages
and then translating the translated text back to the original language using
machine learning services like Google Translate. Back translation enhances our
dataset, making it easier to train robust models, improving performance on a
range of natural language processing tasks, and producing text data with
different terms while maintaining context. Algorithm 2 shows the process of
data augmentation using back translation. As in our data “spam” is a minor
class we have done overall oversampling techniques in this class. After back
translation, we combined the back-translated text with the original text.
Overall this oversampling process, the dataset has become balanced and the
dataset has 4,825 “ham” and 4,825 “spam” data in equal portions. Unbalanced
datasets can produce biased models that perform poorly in minority classes by
favoring the majority class. By ensuring the model learns appropriate
characteristics across all classes and improving its generalization abilities,
balancing the data reduces the risk of overfitting. Fig. 3 illustrates the
distribution of imbalanced and balanced data before and after oversampling.

Algorithm 2 Data Augmentation using Back Translation

1:Original data

2:Augmented data using the back translation

3:function Back_Translate(sentence, languages)

4: translator $\Leftarrow$ Initialize_Translator

5: target_language $\Leftarrow$ Random_Choice(languages)

6: translation $\Leftarrow$ Translate(translator, sentence, target_language)

7: translated_text $\Leftarrow$ Get_Translated_Text(translation)

8: reverse_translation $\Leftarrow$ Translate(translator, translated_text,
’en’)

9: back_translated_text $\Leftarrow$ Get_Translated_Text(reverse_translation)

10: return back_translated_text

11:end function

12:for sentence in original_data_list do

13: augmented_sentence $\Leftarrow$ Back_Translate(sentence, [language1,
language2, …languageN])

14:end for

(a) Imbalanced data

(b) Balanced data

Figure 3: Visualizing Imbalanced vs. Balanced Data Distribution.

###  3.3 Dataset splitting

Partitioning a dataset into several subsets is known as dataset splitting, and
it is usually done to train and evaluate deep learning or machine learning
models. This split is crucial to assess the model’s performance on unobserved
data. We may determine how effectively a model generalizes to new instances by
training it on one subset and testing it on another. This helps to prevent
overfitting, a phenomenon in which a model becomes adept at memorizing
training data but has trouble with new data. The dataset has been divided into
two parts for this experiment: the training set, which contains 80% of the
data, and the testing set, which includes the remaining 20%. There are 7,720
data points in the training set and 1,930 data points in the testing set.

###  3.4 Model Selection

To achieve the best performance, generalization, and practicality in real-
world applications, proper model selection ensures that the selected model
matches the criteria, restrictions, and features of the classification or
detection task. For this detection purpose, initially, we have utilized
several traditional ML algorithms named Extreme Gradient Boosting (XGB) Chen
and Guestrin (2016), Support Vector Machine (SVM) Wang (2005), K-Nearest
Neighbor( KNN) Guo et al. (2003), and Random Forest (RF) Breiman (2001). Along
with the ML models, we also employed transformer-based BERT variant models
such as DistilBERT and RoBERTa.The main goal of our utilizing both ML models
and transformer-based models is to classify the SMS spam with high accuracy
and show the best model that performed well in this detection context.

####  3.4.1 DistilBERT

DistilBERT is a simplified and efficient version of the BERT model. It is
designed to provide comparable performance to BERT but at a much smaller,
lighter, and faster pace. As a result, it is better suited for implementation
in contexts with limited resources or in applications that demand faster
inference. DistilBERT is still the preferred transformer-based model for many
natural language processing applications because of its effectiveness. It uses
a process called knowledge distillation to achieve this. It gains the ability
to mimic the functionality of a larger model like BERT using a smaller dataset
during training. DistilBERT significantly reduces the size and computing needs
of BERT while preserving most of its performance by extracting key insights
from the larger model. As a result, it reduces the model size by 40% while
retaining 97% of BERT’s language understanding ability and achieving a 60%
speed improvement Sanh et al. (2019). It uses the Transformer architecture,
which has feed-forward neural networks similar to BERT and self-attention
layers. However, it usually has fewer hidden sizes and fewer layers, resulting
in a reduction in the number of parameters. In contrast, the BERT base model
typically comprises 12 layers, while DistilBERT may have only 6 layers with a
hidden size of 768 Adoma et al. (2020). There are about 66 million parameters
in DistilBERT, compared to 110 million in the base BERT model. DistilBERT uses
tokenization, which turns raw text inputs into numerical embeddings, to handle
texts. By utilizing attention mechanisms, these embeddings capture word
relationships and help the model identify and prioritize important data.

####  3.4.2 RoBERTa

Robustly Optimized BERT Approach, or RoBERTa, is an enhanced variant of BERT,
a transformer-based model that incorporates improvements to better handle
larger batch sizes and train on longer sequences. Similar to BERT, RoBERTa
prioritizes the encoder within the Transformer architecture for text and
sentence pair classification tasks. Their approaches to tokenization are one
of the main differences between the two: BERT uses the WordPiece tokenizer,
whereas RoBERTa uses a byte-level Byte Pair Encoding (BPE) tokenizer,
employing a larger vocabulary set comprising 50K subword units Tan et al.
(2022). Tokenization is also performed by RoBERTa using BPE as a component
rather than character processing directly. RoBERTa learns unsupervised from a
large corpus of text data during its pre-training phase. Similar to BERT’s
Masked Language Model aim, it trains on predicting masked words within
sentences. Additionally, to enhance the caliber of learned representations, it
does away with the NSP (Next Sentence Prediction) task and employs a larger
training corpus with dynamic masking. RoBERTa performs better than BERT, in
part, because of this change in training objectives. Its improved performance
is also a result of training methodology optimizations, which include bigger
batch sizes, longer training times, and dynamic masking.

###  3.5 Model Optimization and Fine Tuning

Model optimization aims to reduce complexity and increase efficiency, whereas
fine-tuning modifies current models to fit new datasets or tasks, improving
performance via transfer learning. Both procedures are essential for improving
transformer-based models so they perform well in a variety of scenarios. The
Transformers tokenizer is used to tokenize the preprocessed spam dataset. It
segments text into smaller units using a sub-word-based technique called
WordPiece or Byte Pair Encoding, which helps the model better understand word
meaning and context. In this tokenization procedure, we initialized the
maximum sequence length of the text to 512. Tokenization relies on two key
approaches: padding and truncation are essential techniques used to ensure
that input sequences have a consistent length. This is particularly important
when working with neural networks, as they typically require fixed-size
inputs. Padding involves adding tokens (typically zeros) to sequences that are
shorter than the maximum length, whereas truncation involves removing tokens
from longer sequences.

The choice of optimal batch size is crucial in deep learning, impacting both
training and testing accuracies as well as overall runtime Radiuk (2017); Lin
(2022); Schmeiser (1982). This selection involves a trade-off between training
speed and memory usage. In this study, a batch size is selected to be 32 for
training data, while a batch size of 64 is used for testing data. In order to
avoid overfitting and ensure exposure to a wide range of previously unseen
samples, the training data is shuffled at each epoch. The model’s capacity for
generalization is increased and its ability to generalize is strengthened by
this randomization. Additionally, pre-trained weights from the pre-training
phase initialize the DistilBERT and RoBERTa models, optimizing performance.

An optimizer is an essential part of neural networks that updates pre-trained
model weights. An optimizer’s primary objective is to minimize a loss function
by changing the model’s parameters during training. This experiment uses AdamW
(Adam Weight Decay) Loshchilov and Hutter (2017), an effective and popular
optimization approach, to update the pre-trained model’s weights. The AdamW
optimizer is a modified and improved version of Adam that is widely used for
deep neural network training. It decouples the regularizer’s gradient from the
Adam-$\ell_{2}$ update rule, making it effective at minimizing a loss function
and lowering overfitting Zhuang et al. (2022). Weight decay, often called L2
regularization, is a regularization technique that is commonly used in neural
network training to reduce overfitting. It entails adding a penalty term to
the loss function to prevent the use of too high parameter values.

The learning rate is an essential hyperparameter that significantly impacts
the dynamics of training and model performance. It determines the size of the
steps are in the optimization process, which affects convergence and overall
efficacy. The ideal learning rate selection is contingent upon several
elements, such as the specific task at hand, model design, and optimization
techniques. A high learning rate might cause instability, which makes it
difficult to generalize to new data. Conversely, a low learning rate could
make training take longer, costing more computational resources since more
epochs would be needed to reach the target performance levels. For training
the DistilBERT and RoBERTa models in our work, we employ a learning rate of
2e-5 (0.00002), which is a typically used value for transformer-based
architectures such as BERT and its derivatives. By balancing the trade-off
between model performance and training speed, this decision ensures effective
convergence while lowering the possibility of instability. For model
optimization and fine-tuning the hyperparameter settings for the suggested
transfomer-based models are shown in Table 1.

Table 1: Hyper-parameter Settings for SMS spam Detection. Parameter | Utilized Values | Optimal Value
---|---|---
Training Batch size | 8, 16, 32, 64 | 32
Testing Batch size | 8, 16, 32, 64 | 64
Optimizer | AdamW | AdamW
Learning rate | 1e-5, 2e-5, 3e-5, 4e-5 | 2e-5
Number of epochs | 2, 3, 4, 5, 6, 7 | 5

After optimizing the model through hyperparameter tuning, the fine-tuning
process is implemented to train the model on specific tasks and datasets. This
phase involves critical hyperparameters including batch size, regularization
methods, and learning rate. In fine-tuning, an optimization algorithm and loss
function are selected, and backpropagation is used to iteratively alter the
parameters of the entire model or selected layers. The optimization process
modifies the parameters to reduce the difference between the model’s
predictions and the dataset’s ground truth labels. In this experiment, 80% of
the entire data was used for fine-tuning. The training data is sent through
the models in batches throughout each epoch, and gradients are calculated via
the backpropagation technique. To modify pre-trained models for the given task
such as SMS spam detection, RobertaForSequenceClassification and
DistilBERTForSequenceClassification were utilized. The train_loader and
test_loader were configured using DataLoader. The train_loader was configured
by preparing the data, figuring out the batch size, and reorganizing the data
among epochs. However, no shuffling was involved in configuring the test
loader. The overall fine-tuning process flow diagram, featuring these
procedures for DistilBERT and RoBERTa, is displayed in Fig. 4.

Figure 4: The Complete Fine-Tuning Process Flow.

After setting up the optimizer with its parameters and learning rate, the
model receives input parameters during the forward pass to generate
predictions. Then, using automatic differentiation, the gradients of the loss
with respect to the model’s parameters such as weights and biases are
calculated in the backward pass. This determines the loss function’s gradients
in relation to each parameter. The optimizer uses these gradients to update
the model’s parameters during the optimization process. This overall process
iterates until all the epochs of the model training conclude. As a result, the
model can recognize important patterns and features in the training set. Once
the training is complete, the fine-tuned model is ready to predict the spam
messages with unseen new data. Algorithm 3 shows the fine-tuning steps of the
optimized transformer-based model with the train data.

Algorithm 3 Fine-Tuning Transformer-based (DistilBERT, RoBERTa) Model

1:Pre-trained model, train_loader, optimizer, learning rate, maximum training
epochs

2:Fine-tuned Trained Transformer-based model

3:$\text{epochs}\leftarrow\text{max\\_epochs}$

4:$\text{optimizer}\leftarrow\text{Optimizer}(\text{model.parameters()},\text{learning\\_rate})$

5:$\text{device}\leftarrow\text{cuda}$

6:function FineTuneTransformerModel(model, train_loader, optimizer, epochs)

7: for epoch in range(epochs) do

8: model.train()

9: for batch in train_loader do

10: $\text{batch\\_input\\_ids}\leftarrow\text{batch}[0].\text{to(device)}$

11:
$\text{batch\\_attention\\_mask}\leftarrow\text{batch}[1].\text{to(device)}$

12: $\text{batch\\_labels}\leftarrow\text{batch}[2].\text{to(device)}$

13: optimizer.zero_grad() $\triangleright$ Clear gradients

14:
$\text{outputs}\leftarrow\text{model}(\text{input\\_ids}\leftarrow\text{batch\\_input\\_ids},\text{attention\\_mask}\leftarrow\text{batch\\_attention\\_mask},\text{labels}\leftarrow\text{batch\\_labels})$
$\triangleright$ Forward pass

15: $\text{calculated\\_loss}\leftarrow\text{outputs.loss}$

16: calculated_loss.backward() $\triangleright$ Backward pass

17: optimizer.step() $\triangleright$ Update parameters

18: end for

19: end for

20: return model

21:end function

###  3.6 Model Explainability

Explainability of a model refers to the ability to comprehend and analyze the
process by which a model makes its predictions or decisions. A model can be
very accurate across various tasks, but it often behaves like a black box,
making it challenging to understand why it predicts specific outcomes.
Approaches for elucidating black-box machine learning models aim to enhance
transparency, yet their explanations may harbor significant ambiguity, leading
to diminished user confidence and raising concerns regarding the model’s
fairness and reliability Zhang et al. (2019). XAI aims to develop machine
learning methods that enable people to understand and trust the judgments made
by complex models, especially deep neural networks, which frequently operate
as opaque black boxes Holzinger et al. (2022). The necessity for greater
openness in these models may impede the adoption and trustworthiness of AI and
ML technologies. This issue is addressed by XAI, which offers techniques for
interpreting and explaining model predictions. An in-depth understanding of
XAI’s fundamental concepts and familiarity with related programming frameworks
are essential for selecting the appropriate methodology when developing an
application that leverages XAI Dwivedi et al. (2023). The XAI community has
developed numerous effective methods for explaining predictions, which is
crucial for fostering acceptance and confidence in AI and ML technologies. In
this research, we utilized two Explainable Models, LIME and Transformers
Interpret, to understand the model predictions and address the black box
problem. The comprehensive process of model explainability is illustrated in
Fig. 5.

Figure 5: The complete approach to understanding model explainability

The experiment for SMS spam detection is a text classification challenge in
which LimeTextExplainer is used to explain how the black box transformer-based
model operates. It is an interpretable explanation approach that builds a
local interpretable model around a prediction to make any classifier’s
predictions more understandable Ribeiro et al. (2016). To achieve model
explainability, LIME initiates the process by perturbing the SMS content
through subtle modifications such as rearranging or removing words or adding
new ones. This perturbation investigates how sensitive the model is to changes
in the input data. Then, the modified SMS content is fed into the transformer-
based black-box model to produce predictions. LIME extracts prominent
characteristics from the resulting instances that are affected by the
perturbations. These local models simulate how the black-box model would
behave close to the chosen instance and offer insights into how it makes
decisions. After that, LIME uses the knowledge gathered from the local model
to explain the predictions made by the black-box model on the original
dataset. These justifications usually highlight the terms or critical
components that substantially impact the model’s decision-making process,
improving the predictive model’s overall interpretability. By taking a
thorough and iterative approach, LIME helps to clarify the complicated inner
workings of machine learning models, increasing transparency and confidence in
their forecasts. Eq. 1 shows the formulation used by LIME to generate an
interpretable explanation for a specific data point Molnar (2020).

$\displaystyle\text{LIME\\_exp}(n)$
$\displaystyle=\text{argmin}_{x\in\text{X}}\left(L(m,x,\pi_{n})+\Omega(x)\right)$
(1)

In this context, m denotes the black box model that predicts results based on
input data n. The LIME model aims to generate an understandable explanation
for a particular data point n, denoted as x. An interpretable model x
approximates the behavior of the black-box model m locally around n, resulting
in the explanation. The hypothesis space of a specific interpretable learner
is represented by X. L(m, x, $\pi_{n}$) measures the average loss between the
predictions of the interpretable model x and the black-box model m for data
points in the neighborhood of $\pi_{n}$. In the context of local
approximation, $\pi_{n}$ represents the perturbations or features around n
that are considered. These perturbations aid in comprehending the behavior of
the black-box model near n. $\Omega$(x) represents a regularization term or
penalty applied to the complexity of the interpretable model x. This term
penalizes overly complicated models, encouraging simpler explanations.

In order to clarify the operation of the transformer-based model, a
Transformers Interpret tool was also included in the categorization process.
This tool uses a class explainer technique that uses a tokenizer and pre-
trained model to help explain model predictions. Word attributions are
computed as part of the class explainer process to reveal how individual words
contribute to the model’s predictions. The terms that significantly influence
the model’s decision-making are identified using word attribution analysis.
Positive scores represent a word’s influence that is positive on the predicted
class, and negative scores represent a negative term. In addition, this
procedure’s explanation highlights terms that significantly impact the model’s
decision-making. The words that have been highlighted function as crucial
identifiers of the attributes that the model gives priority to during the
prediction process. The procedure for creating explanations for model
predictions using this class explainer approach is described in detail in
Algorithm 4. By using this technique, users can enhance transparency and trust
in their predictions, gaining a deep understanding of the inner workings of
transformer-based models.

Algorithm 4 Explainability Analysis using XAI

1:model, tokenizer, input text

2:Explanation with Word attributions for model predictions

3:class_names $\Leftarrow$ [Class1, Class2, Class3 …, ClassN]

4:function Explain_Model_Prediction(model, tokenizer, input_text)

5: $lime\\_exp\Leftarrow$ ExplainWithLIME(model, input_text, num_features,
top_labels)

6: $transformer\\_exp\Leftarrow$ Transformer_Interpretation(model, tokenizer,
input_text)

7: return $lime\\_exp$, $transformer\\_exp$

8:end function

9:function ExplainWithLIME(model, input_text, num_features, top_labels)

10: $lime\\_explainer\Leftarrow$ LimeTextExplainer(class_names)

11: $lime\\_explanation\Leftarrow$ $lime\\_explainer$.explain_instance
(input_text, model, num_features, top_labels)

12: $label\Leftarrow$ lime_explanation.get_label()

13: $explanation\\_attributions\Leftarrow$ explanation.as_list(label)

14: $\text{visualize\\_explanation}()$

15: return $explanation\\_attributions$

16:end function

17:function Transformer_Interpretation(model, tokenizer, input_text)

18: $trans\\_explainer$ $\Leftarrow$ SequenceClassificationExplainer(model,
tokenizer)

19: attributions $\Leftarrow$ $trans\\_explainer$(input_text)

20: $\text{visualize\\_interpretation}()$

21: return $attributions$

22:end function

##  4 Result Analysis

In this section, We describe in detail the proposed method’s comprehensive
outcomes and provide a proper comparison between traditional ML models (XGB,
SVM, KNN, and RF) utilized in this work. The experiment uses several key
metrics such as precision, recall, F1-score, and accuracy to analyze the
results of a classification model. Our utilized dataset is significantly
imbalanced because the number of spam messages is less than that of ham
messages. In this experiment, we tested our model with an unbalanced and
balanced dataset to see the model’s effectiveness. To evaluate the performance
of the fine-tuned RoBERTa model on imbalanced and balanced data those key
metrics have been calculated. To summarize the overall performance of a model,
calculating the confusion matrix is essential as it forms the foundation for
all subsequent key metrics that are derived directly from it. It is a 2x2
table or performance measuring tool that overviews how effectively a
classification model has made predictions. This confusion matrix consists of
four entries True Positive (TP), True Negative (TN), False Positive (FP), and
False Negative (FN). A true positive (TP) in a model’s evaluation happens when
the model accurately predicts a positive outcome and the actual class is
positive, proving that positive examples can be accurately identified. Cases
where the model accurately predicts a negative outcome and the actual class is
negative are referred to be true negative (TN), demonstrating accurate
identification of negative events. When the model predicts a positive outcome
when the actual class is negative, this is known as a false alarm, or false
positive (FP). Lastly, a false negative (FN) occurs when a positive class is
observed despite the model predicting a negative outcome. These components of
the confusion matrix are the main elements that are used to calculate the
Precision, Recall, and F1-score that help evaluate the model’s performance.

Precision: Precision, also known as positive predictive value, is the ratio of
accurately anticipated positive instances to the total number of predicted
positive instances. It measures the model’s accuracy in predicting positive
outcomes and indicates the likelihood that a predicted positive occurrence is
indeed positive. The precision is defined as follows:

Precision (P) $\displaystyle=\frac{TP}{TP+FP}$ (2)

Recall: Recall, also known as sensitivity, quantifies the proportion of
accurately anticipated positive cases to the total actual positive cases in
the dataset. It measures the model’s ability to detect positive samples and
estimates the percentage of real positive cases that the model correctly
identifies. The recall is defined as follows:

Recall (R) $\displaystyle=\frac{TP}{TP+FN}$ (3)

F1-score: The F1-score is a well-rounded performance metric since it uses the
harmonic mean to balance recall and precision. A low precision or recall will
result in a lower F1-score. It offers a comprehensive assessment of a model’s
ability to accurately classify positive events by considering the risks of
both false positives and false negatives. The F1-Score is defined as follows:

F1-score
$\displaystyle=2\times\frac{\text{P}\times\text{R}}{\text{P}+\text{R}}$ (4)

Accuracy: Accuracy measures how frequently the model’s predictions agree with
the actual labels in the dataset. It is computed by dividing the sum of true
positives and true negatives by the total number of samples in the dataset.
Although accuracy sheds light on the model’s overall performance, an uneven
class distribution may cause it to underrepresent performance. The accuracy is
defined as follows:

$\text{Accuracy}=\frac{\text{TN}+\text{TP}}{\text{TN}+\text{TP}+\text{FN}+\text{FP}}$
(5)

###  4.1 Result Analysis using Imbalanced Dataset

The performance of a model can vary based on different parameters. Initially,
this study used an imbalanced dataset to evaluate the model’s performance
during training and testing. The dataset was significantly imbalanced, with a
much higher number of ham SMS than spam SMS. Classifiers such as traditional
machine learning and transformer-based models were used to classify the text
messages into spam and ham classes. The models were trained on the imbalanced
dataset using 80% (4,457) of the data and tested on 20% (1,115) of the data.
The results obtained from the ML and transformer-based models were impressive
for the ham class; however, they were less successful for the spam class due
to a lack of data. Table 2 displays the confusion matrix of the traditional ML
and transformer-based models for the imbalanced dataset. Comparatively, among
the ML models, XGB performed better than SVM, KNN, and RF. While XGB performed
well, the RoBERTa model outperformed the ML and transformer-based models,
providing the best overall performance. The ML and transformer models’
F1-score, recall, and precision were evaluated for the imbalanced dataset, as
shown in Table 3. The RoBERTa classifier produced precision, recall, and
F1-score values of 0.99, 1.00, and 1.00 for the ham class. On the other hand,
the model obtained precision, recall, and F1-score values for the spam class
of 0.99, 0.97, and 0.98, respectively. To achieve good accuracy, we trained
our optimized and fine-tuned RoBERTa model using 5 epochs with a batch size of
32. After 5 epochs, the model achieved a training accuracy of 99.69% with a
training loss of 0.0108. For the test set, it achieved a testing accuracy of
99.37% with a testing loss of 0.0292, using a batch size of 64. Fig. 6
presents a graph showing the training and testing accuracy and loss over 5
epochs for the imbalanced dataset. Fig. 7 and Table 4 illustrate the
performance comparison among the algorithms.

Table 2: Confusion matrix of ML and transformer-based models for imbalanced dataset | ML Models | Transformer-based Models
---|---|---
| XGB | SVM | KNN | RF | DistilBERT | RoBERTa
Class Name | ham | spam | ham | spam | ham | spam | ham | spam | ham | spam | ham | spam
ham | 963 | 2 | 963 | 2 | 665 | 0 | 965 | 0 | 961 | 4 | 963 | 2
spam | 20 | 130 | 22 | 128 | 98 | 52 | 25 | 125 | 4 | 146 | 5 | 145
Table 3: Precision, Recall, and F1-Score of ML and Transformer-based Models for Imbalanced Dataset Model | Class | Precision | Recall | F1-score
---|---|---|---|---
XGB | ham | 0.98 | 1.00 | 0.99
spam | 0.98 | 0.87 | 0.92
SVM | ham | 0.98 | 1.00 | 0.99
spam | 0.98 | 0.85 | 0.91
KNN | ham | 0.91 | 1.00 | 0.95
spam | 1.00 | 0.35 | 0.51
RF | ham | 0.97 | 1.00 | 0.99
spam | 1.00 | 0.83 | 0.91
DistilBERT | ham | 1.00 | 1.00 | 1.00
spam | 0.97 | 0.97 | 0.97
RoBERTa | ham | 0.99 | 1.00 | 1.00
spam | 0.99 | 0.97 | 0.98
Figure 6: Training & Testing Accuracy and Loss with Imbalanced Data Figure 7: Representation of graphical comparison among models on imbalanced dataset Table 4: Comparative Performance Analysis of Fine-tuned RoBERTa Model with Other Models on Imbalanced Data Model |  Training Loss |  Training Accuracy |  Testing Loss |  Testing Accuracy
---|---|---|---|---
XGB |  0.0095 |  99.84 |  0.0810 |  98.03
SVM |  0.0077 |  99.62 |  0.0591 |  97.85
KNN |  0.1193 |  91.88 |  0.5429 |  91.21
RF |  0.0199 |  100 |  0.1317 |  97.76
DistilBERT |  0.0034 |  99.96 |  0.0343 |  99.28
RoBERTa |  0.0108 |  99.69 |  0.0292 |  99.37

###  4.2 Result Analysis using Balanced Dataset

We also experimented using a balanced dataset. Similar to the imbalanced
dataset, we used 80% (7,720 samples) of the data for training and 20% (1,930
samples) for testing. The results from the ML and transformer-based models
were impressive for both ham and spam classes. Table 5 shows the confusion
matrix of the traditional ML and transformer-based models for the balanced
dataset. In this balanced dataset analysis, the SVM algorithm performed
notably well among the ML algorithms. Similarly, again our fine-tuned RoBERTa
model excelled among the ML and transformer-based models. Table 6 presents the
F1-score, recall, and precision evaluation results for the ML and transformer-
based models on the balanced dataset. For the ham class, the model achieved
precision, recall, and F1-score values of 1.00, 1.00, and 1.00, respectively.
For the spam class, the RoBERTa classifier generated F1-score, recall, and
precision values of 1.00, 1.00, and 1.00, respectively. With the balanced
data, the fine-tuned model achieved 99.95% training accuracy with a training
loss of 0.0035 at the 5th epoch. Subsequently, testing the trained model
resulted in a testing accuracy of 99.84% and a testing loss of 0.0122. This
demonstrates that the model’s performance with a balanced dataset is
significantly better than with an imbalanced dataset. Fig. 8 displays a graph
showing the balanced dataset’s accuracy and loss over 5 epochs. Fig. 9 and
Table 7 illustrate the performance comparison among the algorithms.

Table 5: Confusion matrix of ML and transformer-based models for balanced dataset | ML Models | Transformer-based Models
---|---|---
| XGB | SVM | KNN | RF | DistilBERT | RoBERTa
Class Name | ham | spam | ham | spam | ham | spam | ham | spam | ham | spam | ham | spam
ham | 932 | 13 | 941 | 4 | 944 | 1 | 943 | 2 | 943 | 2 | 945 | 0
spam | 19 | 966 | 4 | 981 | 96 | 889 | 10 | 975 | 4 | 981 | 3 | 982
Table 6: Precision, Recall, and F1-Score of ML and Transformer-based Models for Balanced Dataset Model | Class | Precision | Recall | F1-score
---|---|---|---|---
XGB | ham | 0.98 | 0.99 | 0.98
spam | 0.99 | 0.98 | 0.98
SVM | ham | 1.00 | 1.00 | 1.00
spam | 1.00 | 1.00 | 1.00
KNN | ham | 0.91 | 1.00 | 0.95
spam | 1.00 | 0.90 | 0.95
RF | ham | 0.99 | 1.00 | 0.99
spam | 1.00 | 0.99 | 0.99
DistilBERT | ham | 1.00 | 1.00 | 1.00
spam | 1.00 | 1.00 | 1.00
RoBERTa | ham | 1.00 | 1.00 | 1.00
spam | 1.00 | 1.00 | 1.00
Figure 8: Training & Testing Accuracy and Loss on Balanced Data. Figure 9: Representation of graphical comparison among models on balanced dataset Table 7: Comparative Performance Analysis of Fine-tuned RoBERTa Model with Other Models on Balanced Data. Model |  Training Loss |  Training Accuracy |  Testing Loss |  Testing Accuracy
---|---|---|---|---
XGB |  0.0325 |  99.72 |  0.0621 |  98.34
SVM |  0.0041 |  99.86 |  0.0323 |  99.59
KNN |  0.0457 |  97.76 |  0.1371 |  94.97
RF |  0.0167 |  100 |  0.0530 |  99.38
DistilBERT |  0.0015 |  99.97 |  0.0150 |  99.69
RoBERTa |  0.0035 |  99.95 |  0.0122 |  99.84

We have compared our SMS spam detection study with literature studies that
worked with similar elements in the same field of cybersecurity. The
comparison, summarized in Table 8, evaluates our work from several angles,
including the datasets used, the machine and deep learning models used, and
whether the study offers decision explainability and human interpretation for
the algorithms.

Table 8: Comparison of Our Study with Various Different Studies on SMS Spam Detection Reference |  Datasets | Models |  Accuracy(%) | Explainability Analysis
---|---|---|---|---
Asaju et al. (2021) |  UCI SMS spam collection | Naive Bayes |  99.42 | No
Xia and Chen (2020) |  UCI SMS spam collection | HMM |  95.9 | No
Roy et al. (2020) |  UCI SMS spam collection | CNN |  99.44 | No
Hossain et al. (2022) |  UCI SMS spam collection | BERT |  98.80 | No
Oswald et al. (2022) |  UCI SMS spam collection | DistilBERT+SVM |  98.07 | No
This Study |  UCI SMS spam collection | Fine-tuned RoBERTa |  99.84 | Yes

Our transparent research makes it stand out in the field of SMS spam
detection. We have provided a detailed explanation of our suggested strategy,
which performed better than the existing research works. Interestingly, none
of the publications in Table 8 go as deep into a thorough human interpretation
of the classifier’s output as we have. They do not clarify the model’s
operation in spam detection.

##  5 Explainability Analysis

This study employs LIME and Transformers Interpret techniques to explain the
predictions made by an improved RoBERTa model being used for spam message
detection. These tools approximate the model’s behavior by analyzing each
word’s contribution in the SMS text toward the final prediction. LIME and
Transformers Interpret aim to answer “why the model made a particular
prediction?”. They achieve this by creating word attributions, which give each
word in the text a positive or negative coefficient. This helps determine
whether terms are more important for categorizing an SMS as "spam" or "ham" by
showing the impact of each word on the model’s prediction. Positive
coefficients indicate that the presence of the word contributes towards
predicting a certain class, while negative coefficients indicate the opposite.
The explainable model highlights words that scored positively or negatively
for the prediction, which uses this information to visualize the full text.
This visual aid facilitates comprehension of the rationale underlying the
model’s decision-making process by illuminating key terms that contributed
significantly to the categorization result.

In LIME, the corresponding predictions from the complicated model are fitted
alongside an interpretable model to the perturbed samples. This procedure is
essential to the LIME process to produce better interpretable explanations of
the complex model’s predictions. This interpretable model’s coefficients show
how much each feature (word) contributed to creating the predictions.
Sometimes, LIME will normalize the data to ensure that the importance rankings
are consistent across different models or explanations. In an effort to reduce
dimensionality, the study set the initial number of features (words) in the
LIME explainer to 15 to accelerate the computational explanation process.
Limiting the characteristics simplifies the computational explanation process
and provides a clearer, more manageable list of words that have a major
influence on the model’s prediction for a particular text instance. Fig. 11
and Fig. 13 illustrate the explainability of the RoBERTa model using LIME for
both "spam" and "ham", while Table 9, Table 10, Table 11 and Table 12 presents
the positive and negative coefficients associated with terms to aid users in
understanding spam detection.

(a) SMS 1st Text

(b) Explanation for the first text

(c) SMS 2nd Text

(d) Explanation for the second text

Figure 11: LIME Explanation for SMS Spam Texts

(a) SMS 3rd Text

(b) Explanation for the Third text

(c) SMS 4th Text

(d) Explanation for the Fourth text

Figure 13: LIME Explanation for SMS Ham Texts Table 9: The positive and negative coefficients using LIME for the first text Positive coefficients | Negative coefficients
---|---
Word | Value | Word | Value
‘Alert’ | 2.586283490431408 | ‘for’ | -0.4540554753154869
‘10k’ | 2.5443085492464883 | ‘Big’ | -0.424707563134796
‘voucher’ | 2.19896727844678 | ‘U’ | -0.3875706185927502
‘150’ | 1.5489612217746545 | ‘bar’ | -0.2933243630343485
‘selected’ | 1.5372338239788719 | ‘The’ | -0.17252983760850746
‘Brother’ | 1.2084815552598014 | ‘computer’ | -0.019025118002313234
‘has’ | 0.6819267212480585 |  |
‘or’ | 0.02814567596003713 |  |
Table 10: The positive and negative coefficients using LIME for the second text Positive coefficients | Negative coefficients
---|---
Word | Value | Word | Value
‘FREE’ | 3.186586145122694 | ‘Thanks’ | -0.5401725013063734
‘Karaoke’ | 1.5886755775421426 | ‘the’ | -0.3920060404493261
‘Vote’ | 1.314393037077685 | ‘just’ | -0.28230176631326165
‘link’ | 1.257098816616926 |  |
‘for’ | 1.0052090127637938 |  |
‘mobile’ | 0.8919310037119642 |  |
‘with’ | 0.597081408704299 |  |
‘SING’ | 0.5501241601784591 |  |
‘reply’ | 0.49702113599413067 |  |
‘stars’ | 0.3446891653595809 |  |
‘Now’ | 0.3248660570258343 |  |
‘sing’ | 0.2221033541586965 |  |
Table 11: The positive and negative coefficients using LIME for the third text Positive coefficients | Negative coefficients
---|---
Word | Value | Word | Value
‘night’ | 0.080110152925621 | ‘Good’ | -0.022107989126383402
‘care’ | 0.06923821968423888 | ‘Take’ | -0.020104893561268274
‘amp’ | 0.060886333137515174 |  |
‘Sleepwell’ | 0.03722758230659266 |  |
‘my’ | 0.024641469910968265 |  |
‘dear’ | 0.018922945727309903 |  |
Table 12: The positive and negative coefficients using LIME for the fourth text Positive coefficients | Negative coefficients
---|---
Word | Value | Word | Value
‘scream’ | 0.0628967823689254 | ‘more’ | -0.04109442935123623
‘and’ | 0.05404817757012105 | ‘Woohoo’ | -0.03831983060032088
‘weighed’ | 0.05100704096103593 | ‘want’ | -0.028705541788966896
‘to’ | 0.04824724743333108 | ‘lost’ | -0.009499243825646576
‘myself’ | 0.037156130726796105 | ‘Omg’ | -0.006713896998241079
‘I’ | 0.02486380111242408 |  |
‘weight’ | 0.019592668979617558 |  |

We used Transformers Interpret in addition to LIME to assess the model
interpretability with similar SMS texts. This Transformers Interpret
illustrates how the transformer-based RoBERTa model uses the tokenizer to
divide the huge words into subwords with the prefix "##". This occurs when the
model encounters complex or out-of-vocabulary words, which it then breaks down
into smaller sub-words to simplify processing. The model can manage a broader
vocabulary and comprehend the context of complicated words more accurately due
to this procedure. The Transformers Interpret tool’s interpretation findings
for SMS spam messages are shown in Fig. 14 and Fig. 15, and Table 13, Table
14, Table 15 and Table 16 show the relevance of the words in four SMS, with
positive and negative coefficients showing each word’s influence on the
model’s prediction. In this experiment, we worked with a binary classification
task where "True Label" = 0 denotes "ham" and "True Label" = 1 denotes "spam".

(a) Explanation for first text (spam)

(b) Explanation for second text (spam)

Figure 14: Transformers Interpret Explanation for two Spam SMS Text

(a) Explanation for Third text (ham)

(b) Explanation for Fourth text (ham)

Figure 15: Transformers Interpret Explanation for two ham SMS Text Table 13: The positive and negative coefficients using Transformer Interpret for the first text Positive coefficients | Negative coefficients
---|---
Word | Value | Word | Value
‘Big’ | 0.024297947664629425 | ‘Brother’ | -0.14376454412860262
‘Alert’ | 0.20062465985674183 | ‘The’ | -0.20299253710630974
‘selected’ | 0.3235594150691528 | ‘computer’ | -0.06358605308176257
‘U’ | 0.14328277724673683 | ‘has’ | -0.09203341078407345
‘10’ | 0.454875630595537 | ‘for’ | -0.013336465891418426
‘k’ | 0.07143031221542949 |  |
‘bar’ | 0.16807094091950164 |  |
‘150’ | 0.6246831675560521 |  |
‘or’ | 0.11691662255361639 |  |
‘voucher’ | 0.29147154979015566 |  |
‘.’ | 0.023613104082805682 |  |
‘’ | 0.17233550519021407 |  |
Table 14: The positive and negative coefficients using Transformers Interpret for the second text Positive coefficients | Negative coefficients
---|---
Word | Value | Word | Value
‘for’ | 0.18965947100466987 | ‘Thanks’ | -0.18213375681044702
‘Vote’ | 0.04214511554119887 | ‘the’ | -0.16879623241117406
‘stars’ | 0.050776681629629485 | ‘sing’ | -0.3819342785384402
‘Kara’ | 0.13869346357463316 | ‘along’ | -0.32168706815694476
‘on’ | 0.07859092995531514 | ‘with’ | -0.10687877218596162
‘your’ | 0.2284359424426384 | ‘just’ | -0.15317418612821526
‘For’ | 0.2290909616422906 | ‘S’ | -0.12666676012232256
‘FREE’ | 0.2880400601164391 | ‘now’ | -0.031842205592812194
‘link’ | 0.37679605294057267 |  |
‘reply’ | 0.04222149540549413 |  |
‘ING’ | 0.10210499982518297 |  |
‘mobile’ | 0.3437490059251579 |  |
‘a’ | 0.03278371376214245 |  |
Table 15: The positive and negative coefficients using Transformers Interpret for the third text Positive coefficients | Negative coefficients
---|---
Word | Value | Word | Value
‘night’ | 0.3506601184212354 | ‘Good’ | -0.23430671367151
‘my’ | 0.30588133708098775 | ‘Sleep’ | -0.2558005048167494
‘dear’ | 0.3750717679647341 | ‘Take’ | -0.6100631898260275
‘care’ | 0.36718414960882323 | ‘amp’ | -0.025046901303540856
Table 16: The positive and negative coefficients using Transformers Interpret for the fourth text Positive coefficients | Negative coefficients
---|---
Word | Value | Word | Value
‘O’ | 0.6168087105189476 | ‘weighed’ | -0.14799138790010427
‘mg’ | 0.285592216107348 | ‘weight’ | -0.1102382677942815
‘I’ | 0.02889748690680515 | ‘Wo’ | -0.23351373922826477
‘want’ | 0.2613656158757424 |  |
‘to’ | 0.21709997128226066 |  |
‘scream’ | 0.0764171095678442 |  |
‘myself’ | 0.3498253922866739 |  |
‘and’ | 0.2869778663987718 |  |
‘lost’ | 0.18199395807032306 |  |
‘more’ | 0.22200118513344438 |  |
‘oh’ | 0.11642180604530727 |  |
‘oo’ | 0.1630583357991135 |  |

##  6 Discussion

The overall experiment was conducted with both imbalanced and balanced
datasets. The performance of the proposed fine-tuned RoBERTa model was
measured with an imbalanced and balanced dataset. We mainly worked with
imbalanced and balanced datasets to see our optimized and fine-tuned model’s
effectiveness. We employed a data-balancing technique for this model’s
effectiveness visualization and balanced the data. On the imbalanced dataset,
the model achieved 99.69% training and 99.37% testing accuracy with 0.0292
loss. On the other hand, on the balanced dataset, our fine-tuned model
achieved 99.95% training accuracy and 99.84% testing accuracy with 0.0122
loss. The performance accuracy of this model using an imbalanced and balanced
dataset does not show too much difference, just showing 0.47% of difference
between these two values. To compare with our model performance, we selected
several traditional ML models and transformer-based models on both dataset
types. Among those tested models, the RoBERTa model performed better.
Furthermore, the optimized model performed well among those tested models and
outperformed previous studies, demonstrating its effectiveness.

Along with the performance analysis of our model, we also worked with XAI
techniques which can provide answers to various types of questions such as
"Why did the model make a particular prediction?" and "Which features have the
most significant impact on the model’s output?" related to the functioning and
decision-making process of the model. For this, we used two well-known XAI
techniques such as LIME and Transformers Interpret to understand the models’
decision-making process. It is more difficult to understand how RoBERTa model
makes its predictions because, as we know, it is a transformer-based model
which is so complex in architecture. Feature importance analysis using XAI,
helps users to understand the factors that most influence a model’s decision-
making process. By analyzing input features such as words or sub-words, XAI
methods provide insights into the factors that contributed significantly to
the specific classification of spam or ham messages. Mainly, those techniques
calculated each word or sub-word coefficient where some words indicate a
positive coefficient and some negative coefficient which indicates that these
words with positive coefficients are considered more indicative to classify a
class, while these words with negative coefficients are considered more
indicative to do not support the predicted class. This overall process of
explanation shows the transparency of our proposed model by which users can
gain insights into the prediction. Moreover, this experimental analysis only
used one benchmark dataset of spam messages. In the future, we will collect
other different datasets of SMS spam to explore our fine-tuned model
performance and its explainability on new data.

##  7 Conclusion

Utilizing cutting-edge natural language processing algorithms has
significantly improved SMS spam detection, demonstrating the efficacy of these
methods. Using LLMs has enormous potential to improve the global computing
user experience and address long-standing societal issues. In this paper, we
utilized modified transformer-based LLMs (DistilBERT and RoBERTa) to identify
SMS spam. A benchmark dataset was utilized for the entire investigation,
although data imbalance issues afflicted it. However, these challenges were
effectively mitigated through the implementation of data augmentation such as
back translation. The experimental results showed that, compared to
traditional ML models, our transformer-based model, more specifically RoBERTa
performed better in differentiating between spam and valid SMSs. To achieve
such high performance, the model is primarily optimized and fine-tuned with
crucial hyperparameters defined, and trained on a particular subset of real
data. This improved version of RoBERTa achieved a testing accuracy of 99.84%
in the SMS spam classification task. Moreover, explainable AI techniques such
as LIME and Transformers Interpret are indeed used to see the explanations of
the model prediction by calculating the word coefficient. By using the
explanation, people can learn how the model works and why it makes its
decisions. This increases user understanding and confidence in the model’s
results through transparency, strengthening the model’s overall robustness and
reliability.

The experimental results demonstrate an enhancement of our proposed
transformer-based model compared to previously proposed approaches of others
in SMS spam detection. LLMs in particular are being adopted quickly, which is
driving ongoing innovation and growth in the field of natural language
processing. In the future, our research aims to expand our spam detection work
by exploring a diverse range of models and datasets. Furthermore, analyzing
models in-depth to strengthen their robustness using explainable AI approaches
is consistent with the continuous endeavors to raise the trustworthiness and
understanding of AI systems.

## References

  * Liu et al. (2021) X. Liu, H. Lu, A. Nayak,  A spam transformer model for sms spam detection,  IEEE Access 9 (2021) 80253–80263.
  * Oswald et al. (2022) C. Oswald, S. E. Simon, A. Bhattacharya,  Spotspam: Intention analysis–driven sms spam detection using bert embeddings,  ACM Transactions on the Web (TWEB) 16 (2022) 1–27.
  * Hossain et al. (2022) S. M. M. Hossain, A. Sen, K. Deb,  Detecting spam sms using self attention mechanism,  in: International Conference on Intelligent Computing & Optimization, Springer, 2022, pp. 175–184.
  * Wei and Nguyen (2020) F. Wei, T. Nguyen,  A lightweight deep neural model for sms spam detection,  in: 2020 International Symposium on Networks, Computers and Communications (ISNCC), IEEE, 2020, pp. 1–6.
  * Sarker (2023) I. H. Sarker,  Machine learning for intelligent data analysis and automation in cybersecurity: current and future prospects,  Annals of Data Science 10 (2023) 1473–1498.
  * Amaz Uddin and Sarker (2024) M. Amaz Uddin, I. H. Sarker,  An explainable transformer-based model for phishing email detection: A large language model approach,  arXiv e-prints (2024) arXiv–2402.
  * Zhao et al. (2023) W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al.,  A survey of large language models,  arXiv preprint arXiv:2303.18223 (2023).
  * Yao et al. (2023) Y. Yao, J. Duan, K. Xu, Y. Cai, E. Sun, Y. Zhang,  A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,  arXiv preprint arXiv:2312.02003 1 (2023).
  * Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin,  Attention is all you need,  Advances in neural information processing systems 30 (2017).
  * Han et al. (2021) K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, Y. Wang,  Transformer in transformer,  Advances in Neural Information Processing Systems 34 (2021) 15908–15919.
  * Jamal et al. (2023) S. Jamal, H. Wimmer, I. Sarker,  An improved transformer-based model for detecting phishing, spam, and ham: A large language model approach,  arXiv preprint arXiv:2311.04913 (2023).
  * Koroteev (2021) M. Koroteev,  Bert: a review of applications in natural language processing and understanding,  arXiv preprint arXiv:2103.11943 (2021).
  * Mohammed and Ali (2021) A. H. Mohammed, A. H. Ali,  Survey of bert (bidirectional encoder representation transformer) types,  in: Journal of Physics: Conference Series, volume 1963, IOP Publishing, 2021, p. 012173.
  * Devlin et al. (2018) J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  Bert: Pre-training of deep bidirectional transformers for language understanding,  arXiv preprint arXiv:1810.04805 (2018).
  * Anil et al. (2021) R. Anil, B. Ghazi, V. Gupta, R. Kumar, P. Manurangsi,  Large-scale differentially private bert,  arXiv preprint arXiv:2108.01624 (2021).
  * Lan et al. (2019) Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut,  Albert: A lite bert for self-supervised learning of language representations,  arXiv preprint arXiv:1909.11942 (2019).
  * Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov,  Roberta: A robustly optimized bert pretraining approach,  arXiv preprint arXiv:1907.11692 (2019).
  * Sanh et al. (2019) V. Sanh, L. Debut, J. Chaumond, T. Wolf,  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,  arXiv preprint arXiv:1910.01108 (2019).
  * Jiao et al. (2019) X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, Q. Liu,  Tinybert: Distilling bert for natural language understanding,  arXiv preprint arXiv:1909.10351 (2019).
  * Sarker (2024) I. H. Sarker, AI-Driven Cybersecurity and Threat Intelligence: Cyber Automation, Intelligent Decision-Making and Explainability, Springer Nature, 2024.
  * Xu et al. (2019) F. Xu, H. Uszkoreit, Y. Du, W. Fan, D. Zhao, J. Zhu,  Explainable ai: A brief survey on history, research areas, approaches and challenges,  in: Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part II 8, Springer, 2019, pp. 563–574.
  * Linardatos et al. (2020) P. Linardatos, V. Papastefanopoulos, S. Kotsiantis,  Explainable ai: A review of machine learning interpretability methods,  Entropy 23 (2020) 18.
  * Abid et al. (2022) M. A. Abid, S. Ullah, M. A. Siddique, M. F. Mushtaq, W. Aljedaani, F. Rustam,  Spam sms filtering based on text features and supervised machine learning techniques,  Multimedia Tools and Applications 81 (2022) 39853–39871.
  * Mishra and Soni (2023) S. Mishra, D. Soni,  Dsmishsms-a system to detect smishing sms,  Neural Computing and Applications 35 (2023) 4975–4992.
  * Sonowal (2020) G. Sonowal,  Detecting phishing sms based on multiple correlation algorithms,  SN computer science 1 (2020) 361.
  * Asaju et al. (2021) C. B. Asaju, E. J. Nkorabon, R. O. Orah,  Short message service (sms) spam detection and classification using naïve bayes,  in: Conference organizing committee, volume 62, 2021.
  * Mishra and Soni (2020) S. Mishra, D. Soni,  Smishing detector: A security model to detect smishing through sms content analysis and url behavior analysis,  Future Generation Computer Systems 108 (2020) 803–815.
  * Joo et al. (2017) J. W. Joo, S. Y. Moon, S. Singh, J. H. Park,  S-detector: an enhanced security model for detecting smishing attack for mobile computing,  Telecommunication Systems 66 (2017) 29–38.
  * Xia and Chen (2020) T. Xia, X. Chen,  A discrete hidden markov model for sms spam detection,  Applied Sciences 10 (2020) 5011.
  * Ghourabi et al. (2020) A. Ghourabi, M. A. Mahmood, Q. M. Alzubi,  A hybrid cnn-lstm model for sms spam detection in arabic and english messages,  Future Internet 12 (2020) 156.
  * Roy et al. (2020) P. K. Roy, J. P. Singh, S. Banerjee,  Deep learning to filter sms spam,  Future Generation Computer Systems 102 (2020) 524–533.
  * Abayomi-Alli et al. (2022) O. Abayomi-Alli, S. Misra, A. Abayomi-Alli,  A deep learning method for automatic sms spam classification: Performance of learning algorithms on indigenous dataset,  Concurrency and Computation: Practice and Experience 34 (2022) e6989.
  * Sahmoud and Mikki (2022) T. Sahmoud, D. M. Mikki,  Spam detection using bert,  arXiv preprint arXiv:2206.02443 (2022).
  * Tida and Hsu (2022) V. S. Tida, S. Hsu,  Universal spam detection using transfer learning of bert model,  arXiv preprint arXiv:2202.03480 (2022).
  * Chen and Guestrin (2016) T. Chen, C. Guestrin,  Xgboost: A scalable tree boosting system,  in: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785–794.
  * Wang (2005) L. Wang, Support vector machines: theory and applications, volume 177, Springer Science & Business Media, 2005.
  * Guo et al. (2003) G. Guo, H. Wang, D. Bell, Y. Bi, K. Greer,  Knn model-based approach in classification,  in: On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE: OTM Confederated International Conferences, CoopIS, DOA, and ODBASE 2003, Catania, Sicily, Italy, November 3-7, 2003. Proceedings, Springer, 2003, pp. 986–996.
  * Breiman (2001) L. Breiman,  Random forests,  Machine learning 45 (2001) 5–32.
  * Adoma et al. (2020) A. F. Adoma, N.-M. Henry, W. Chen,  Comparative analyses of bert, roberta, distilbert, and xlnet for text-based emotion recognition,  in: 2020 17th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), IEEE, 2020, pp. 117–121.
  * Tan et al. (2022) K. L. Tan, C. P. Lee, K. S. M. Anbananthen, K. M. Lim,  Roberta-lstm: a hybrid model for sentiment analysis with transformer and recurrent neural network,  IEEE Access 10 (2022) 21517–21525.
  * Radiuk (2017) P. M. Radiuk,  Impact of training set batch size on the performance of convolutional neural networks for diverse datasets,  Information Technology and Management Science 20 (2017) 20–24.
  * Lin (2022) R. Lin,  Analysis on the selection of the appropriate batch size in cnn neural network,  in: 2022 International Conference on Machine Learning and Knowledge Engineering (MLKE), IEEE, 2022, pp. 106–109.
  * Schmeiser (1982) B. Schmeiser,  Batch size effects in the analysis of simulation output,  Operations Research 30 (1982) 556–568.
  * Loshchilov and Hutter (2017) I. Loshchilov, F. Hutter,  Decoupled weight decay regularization,  arXiv preprint arXiv:1711.05101 (2017).
  * Zhuang et al. (2022) Z. Zhuang, M. Liu, A. Cutkosky, F. Orabona,  Understanding adamw through proximal methods and scale-freeness,  arXiv preprint arXiv:2202.00089 (2022).
  * Zhang et al. (2019) Y. Zhang, K. Song, Y. Sun, S. Tan, M. Udell,  " why should you trust my explanation?" understanding uncertainty in lime explanations,  arXiv preprint arXiv:1904.12991 (2019).
  * Holzinger et al. (2022) A. Holzinger, A. Saranti, C. Molnar, P. Biecek, W. Samek,  Explainable ai methods-a brief overview,  in: International workshop on extending explainable AI beyond deep models and classifiers, Springer, 2022, pp. 13–38.
  * Dwivedi et al. (2023) R. Dwivedi, D. Dave, H. Naik, S. Singhal, R. Omer, P. Patel, B. Qian, Z. Wen, T. Shah, G. Morgan, et al.,  Explainable ai (xai): Core ideas, techniques, and solutions,  ACM Computing Surveys 55 (2023) 1–33.
  * Ribeiro et al. (2016) M. T. Ribeiro, S. Singh, C. Guestrin,  " why should i trust you?" explaining the predictions of any classifier,  in: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 1135–1144.
  * Molnar (2020) C. Molnar, Interpretable machine learning, Lulu. com, 2020.