\begin{table}[]
\centering
\caption{\label{Results} Comparing the Average Accuracy of Models' with and without knowledge transfer from the source dataset}
\begin{tabular}{|p{2cm}||p{2.75cm}|p{1cm}|p{1.2cm}|p{1.5cm}|p{1.75cm}|p{1cm}|p{1.2cm}|}
 \hline
 \centering \textbf{Task Kind}&\centering \textbf{Task Name}& \textbf{BERT} &\textbf{BERT (fine-tuned)}&\textbf{ROBERTa} &\textbf{ROBERTa (fine-tuned)}&\textbf{XLNet} &\textbf{XLNet (fine-tuned)}\\
 \hline
 \centering \textbf{Text Classification}&\centering \textbf{Small SMS Spam Collection}   & 0.71    & 0.621&0.732&0.548 &0.576 & 0.48 \\
 \hline
 \centering \textbf{Text Classification}&\centering \textbf{French and Spanish Reviews Dataset}& 0.63  & 0.672   &0.58 &0.51 &0.44 & 0.46\\
  \hline
  \centering \textbf{Sentimental Analysis}&\centering \textbf{Small Section of GoEmotions Dataset}   & 0.832    & 0.89&0.71 &0.76 &0.59 & 0.62\\
 \hline
 \centering \textbf{Sentimental Analysis}&\centering \textbf{French - German Emotions Dataset}& 0.68  & 0.61   &0.716 &0.66 &0.57 & 0.53\\
  \hline
  \centering \textbf{Sentence Similarity}&\centering \textbf{Financial Domain Dataset}   & 0.67    & 0.52 &0.72 & 0.67 &0.55 & 0.47 \\
 \hline
 \centering \textbf{Sentence Similarity}&\centering \textbf{Section of PAWS-X Dataset}& 0.67  & 0.61   &0.736 & 0.711 &0.58 & 0.62\\
 \hline
 \end{tabular}
\end{table}
\section{Introduction}
TL is when a model is pre-trained on a rich dataset before fine-tuning or using feature-based transfer for a domain-specific task \cite{you2020co,raffel2020exploring,houlsby2019parameter}. The role of TL in carrying out NLP tasks has increased significantly due to the need for understanding task interests in target domains where there is a lack of large enough datasets. In such cases, the transfer of knowledge from other domains is known to mitigate the problems of overfitting and yield strong model performance while carrying out the predictions\cite{alyafeai2020survey,durrani2021transfer}. 

\section{Related Work}
Several works have shown the effectiveness of TL across various domains starting from image segmentation to several tasks in NLP \cite{kim2020effectiveness, salza2022effectiveness,bengio2012deep}. Felbo et al. represented the understanding of emotions, sentiment and sarcasm across the text from various domains using the knowledge from a huge data source and transferring it to the model when the prediction was performed on the target domains \cite{felbo2017using}. There is an improvement in the prediction against the model performance when trained only using the target dataset.       

\subsubsection{Analysis - 1: text classification}
The datasets to be used: 

\begin{itemize}
\item IMDB Movie Reviews Dataset: A large dataset classifying the movie reviews into two classes: positive and negative \cite{maas-EtAl:2011:ACL-HLT2011}. 
\item Small SMS Spam Collection: A section of a large dataset classifying the messages to be spam or not. 
\item French and Spanish Reviews Dataset: A small dataset of reviews in French and Spanish classified into positive and negative.
\end{itemize}

\subsubsection{Analysis - 2: sentimental analysis}
The datasets to be used: 

\begin{itemize}
\item IMDB Genre Classification Dataset: A large dataset classifying the movie into 27 different genres using the descriptions.
\item Small Section of GoEmotions Dataset: A fine grained annotated dataset with sentences being classified into 28 emotions including neutral. 
\item French - German Emotions Dataset: A small dataset with machine translation sentences classified into 13 different emotions. 
\end{itemize}

\begin{itemize}
\item Paraphrase Adversaries from Word Scrambling (PAWS): A large dataset containing human-labeled sentence similarity \cite{paws2019naacl, pawsx2019emnlp}. 
\item Financial Domain Dataset: A small dataset consisting of related ticker names used in the stock exchange.
\item Section of PAWS-X Dataset: A small dataset containing cross-lingual sentence similarity data.  
\end{itemize}

\section{Results}
We have divided our analyses into two experiments based on the parameters to check model performance. In the first experiment, the models have trained on the target dataset for the tasks: text classification, sentimental analysis, and sentence similarity. Similarly, in the second experiment, the models were fine-tuned with the first dataset of each analysis. Then, the models were made to make predictions on the target dataset using the acquired knowledge from the source.
{"value": "0.71", "type": "Result", "training data/set": "Small SMS Spam Collection", "test data/set": "Small SMS Spam Collection", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"xx": "yy"}} {"value": "0.621", "type": "Result", "training data/set": "Small SMS Spam Collection", "test data/set": "Small SMS Spam Collection", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"fine-tuned": "true"}} {"value": "0.732", "type": "Result", "training data/set": "Small SMS Spam Collection", "test data/set": "Small SMS Spam Collection", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"xx": "yy"}} {"value": "0.548", "type": "Result", "training data/set": "Small SMS Spam Collection", "test data/set": "Small SMS Spam Collection", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"fine-tuned": "true"}} {"value": "0.576", "type": "Result", "training data/set": "Small SMS Spam Collection", "test data/set": "Small SMS Spam Collection", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"xx": "yy"}} {"value": "0.48", "type": "Result", "training data/set": "Small SMS Spam Collection", "test data/set": "Small SMS Spam Collection", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"fine-tuned": "true"}} {"value": "0.63", "type": "Result", "training data/set": "French and Spanish Reviews Dataset", "test data/set": "French and Spanish Reviews Dataset", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"xx": "yy"}} {"value": "0.672", "type": "Result", "training data/set": "French and Spanish Reviews Dataset", "test data/set": "French and Spanish Reviews Dataset", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"fine-tuned": "true"}} {"value": "0.58", "type": "Result", "training data/set": "French and Spanish Reviews Dataset", "test data/set": "French and Spanish Reviews Dataset", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"xx": "yy"}} {"value": "0.51", "type": "Result", "training data/set": "French and Spanish Reviews Dataset", "test data/set": "French and Spanish Reviews Dataset", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"fine-tuned": "true"}} {"value": "0.44", "type": "Result", "training data/set": "French and Spanish Reviews Dataset", "test data/set": "French and Spanish Reviews Dataset", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"xx": "yy"}} {"value": "0.46", "type": "Result", "training data/set": "French and Spanish Reviews Dataset", "test data/set": "French and Spanish Reviews Dataset", "task": "Text Classification", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"fine-tuned": "true"}} {"value": "0.832", "type": "Result", "training data/set": "Small Section of GoEmotions Dataset", "test data/set": "Small Section of GoEmotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"xx": "yy"}} {"value": "0.89", "type": "Result", "training data/set": "Small Section of GoEmotions Dataset", "test data/set": "Small Section of GoEmotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"fine-tuned": "true"}} {"value": "0.71", "type": "Result", "training data/set": "Small Section of GoEmotions Dataset", "test data/set": "Small Section of GoEmotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"xx": "yy"}} {"value": "0.76", "type": "Result", "training data/set": "Small Section of GoEmotions Dataset", "test data/set": "Small Section of GoEmotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"fine-tuned": "true"}} {"value": "0.59", "type": "Result", "training data/set": "Small Section of GoEmotions Dataset", "test data/set": "Small Section of GoEmotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"xx": "yy"}} {"value": "0.62", "type": "Result", "training data/set": "Small Section of GoEmotions Dataset", "test data/set": "Small Section of GoEmotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"fine-tuned": "true"}} {"value": "0.68", "type": "Result", "training data/set": "French - German Emotions Dataset", "test data/set": "French - German Emotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"xx": "yy"}} {"value": "0.61", "type": "Result", "training data/set": "French - German Emotions Dataset", "test data/set": "French - German Emotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"fine-tuned": "true"}} {"value": "0.716", "type": "Result", "training data/set": "French - German Emotions Dataset", "test data/set": "French - German Emotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"xx": "yy"}} {"value": "0.66", "type": "Result", "training data/set": "French - German Emotions Dataset", "test data/set": "French - German Emotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"fine-tuned": "true"}} {"value": "0.57", "type": "Result", "training data/set": "French - German Emotions Dataset", "test data/set": "French - German Emotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"xx": "yy"}} {"value": "0.53", "type": "Result", "training data/set": "French - German Emotions Dataset", "test data/set": "French - German Emotions Dataset", "task": "Sentimental Analysis", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"fine-tuned": "true"}} {"value": "0.67", "type": "Result", "training data/set": "Financial Domain Dataset", "test data/set": "Financial Domain Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"xx": "yy"}} {"value": "0.52", "type": "Result", "training data/set": "Financial Domain Dataset", "test data/set": "Financial Domain Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"fine-tuned": "true"}} {"value": "0.72", "type": "Result", "training data/set": "Financial Domain Dataset", "test data/set": "Financial Domain Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"xx": "yy"}} {"value": "0.67", "type": "Result", "training data/set": "Financial Domain Dataset", "test data/set": "Financial Domain Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"fine-tuned": "true"}} {"value": "0.55", "type": "Result", "training data/set": "Financial Domain Dataset", "test data/set": "Financial Domain Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"xx": "yy"}} {"value": "0.47", "type": "Result", "training data/set": "Financial Domain Dataset", "test data/set": "Financial Domain Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"fine-tuned": "true"}} {"value": "0.67", "type": "Result", "training data/set": "Section of PAWS-X Dataset", "test data/set": "Section of PAWS-X Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"xx": "yy"}} {"value": "0.61", "type": "Result", "training data/set": "Section of PAWS-X Dataset", "test data/set": "Section of PAWS-X Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT", "model settings": {"fine-tuned": "true"}} {"value": "0.736", "type": "Result", "training data/set": "Section of PAWS-X Dataset", "test data/set": "Section of PAWS-X Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"xx": "yy"}} {"value": "0.711", "type": "Result", "training data/set": "Section of PAWS-X Dataset", "test data/set": "Section of PAWS-X Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "ROBERTa", "model settings": {"fine-tuned": "true"}} {"value": "0.58", "type": "Result", "training data/set": "Section of PAWS-X Dataset", "test data/set": "Section of PAWS-X Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"xx": "yy"}} {"value": "0.62", "type": "Result", "training data/set": "Section of PAWS-X Dataset", "test data/set": "Section of PAWS-X Dataset", "task": "Sentence Similarity", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "XLNet", "model settings": {"fine-tuned": "true"}}