11institutetext: School of Computer Science,
College of Science and Engineering,
National University of Ireland, Galway, Ireland.
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>

#  A Multi-class Approach – Building a Visual Classifier based on Textual
Descriptions using Zero-Shot Learning

Preeti Jagdish Sajjan  Frank G. Glavin

###### Abstract

Machine Learning (ML) techniques for image classification routinely require
many labelled images for training the model and while testing, we ought to use
images belonging to the same domain as those used for training. In this paper,
we overcome the two main hurdles of ML, i.e. scarcity of data and constrained
prediction of the classification model. We do this by introducing a visual
classifier which uses a concept of transfer learning, namely Zero-Shot
Learning (ZSL), and standard Natural Language Processing techniques. We train
a classifier by mapping labelled images to their textual description instead
of training it for specific classes. Transfer learning involves transferring
knowledge across domains that are similar. ZSL intelligently applies the
knowledge learned while training for future recognition tasks. ZSL
differentiates classes as two types: seen and unseen classes. Seen classes are
the classes upon which we have trained our model and unseen classes are the
classes upon which we test our model. The examples from unseen classes have
not been encountered in the training phase. Earlier research in this domain
focused on developing a binary classifier but, in this paper, we present a
multi-class classifier with a Zero-Shot Learning approach.

###### Keywords:

Transfer Learning Computer Vision NLP Zero-Shot Learning Meta-Learning Image
Classification.

##  1 Introduction

In this work, we develop a visual classifier capable of classifying images
based on their textual descriptions. This is made possible by mapping visual
features from images and textual features from the descriptions to the
semantic label space of the classes. We do this by using Zero-Shot Learning
(ZSL). As the name would suggest, Zero-Shot Learning can be defined as a setup
where the model is given certain classes during its testing phase which were
not included in the training phase. In simpler terms, ZSL intelligently allows
the model to recognise and classify objects that it has never seen before with
some degree of certainty. We consider earlier work by Elhoseiny [5] as a base
reference.

###  1.1 Motivation and Dataset

Concepts of transfer learning, such as Zero-Shot Learning and Few-Shot
Learning, have achieved greater visibility producing significant research in
the past decade. The main motivation for applying the concepts of Zero-Shot
Learning is to achieve a model that can classify the images into certain
categories without being trained upon it. The key idea of ZSL is to explore
and exploit the knowledge of how an unseen class is semantically related to
seen classes. We present a classifier that is built using various emerging
Computer Vision, Deep Learning, and NLP techniques. The novelty here is in
creating a state-of-the-art Zero-Shot Learning multi-class classification
model that learns to map images to their textual descriptions and, in turn, to
their class labels. While our model is trained on a much smaller dataset, when
compared to earlier work, we still achieve promising results.
The dataset that we use for building our model is the _Caltech Birds dataset_
CUB200-2011 Birds [17] which has 11788 images of 200 species of bird. Textual
descriptions for each of the above 200 classes are obtained from earlier
research [5] with information extracted manually from Wikipedia. Examples of
the textual descriptions and the Caltech Birds dataset, organised with respect
to the class labels, are shown below in Figure 1.

Figure 1: Samples from the dataset [Class Label, Text Description, Images]

##  2 Background Information and Related Work

###  2.1 Transfer Learning

Transfer Learning (TL) involves transferring the knowledge learnt in one
domain to another domain. There are many real-world applications in which
collecting new training data can be difficult or expensive. Transfer learning
aims to reduce the need to collect such training data by transferring
knowledge between the task domains. Pan and Yang [10] presented a survey on TL
and noted that its motivation was initially discussed in a _NIPS-95_ workshop
as _Learning to Learn_ , which focused on the need for machine learning
methods to learn, and then use this learned knowledge, when a sample from an
unseen category or domain appears.

###  2.2 Applying Zero-Shot Learning to Image Classification

Lack of training data for each class and learning, both local and global,
features for a group of images makes image classification using Zero-Shot
Learning a challenging task. Li et al. [8], formulate ZSL as a conditioned
image classification problem where they aim to classify the visual features
using a classifier that has learned from semantic descriptions. ZSL models
typically learn the mapping function that maps the feature space to the
semantic vector space. Since with ZSL, the model only has visibility of
instances from training classes, it suffers a _projection domain shift_. This
problem of ZSL was first identified by Fu et al. [6]. Kodirov et al. [7]
present Semantic Autoencoders, where an encoder aims to project the visual
feature vectors to the semantic vector space and a decoder aims to reconstruct
the original visual feature vectors from the semantic vector space. Once the
model is trained, the authors retrieve the encoder model which establishes an
optimal mapping function as a solution to this. Akata et al. [2] use images
from seen classes and semantic attributes, from both seen and unseen classes,
to learn two dictionaries (“ _coupled dictionary_ ”), that can sparsely
represent the visual and semantic feature vectors of an image. They also
provide an _attribute-aware_ system to solve domain shift and the _hubness_
111A problem in which a few select words (or hubs) are too close to many
others, especially in high dimensional spaces. problems of ZSL. Akanksha et
al. [1] propose a Semantically Aligned Bias Reducing (SABR) approach that
focuses on overcoming the _hubness problem_ by learning a _latent space_ which
is responsible for preserving the semantic relation between the labels and
then encoding the discriminating information of the classes.

###  2.3 Mapping Textual Descriptions to Images

Humans can write a summary of events seen in an image to provide a better
understanding of that image. Wang J et al. [16] describe a system to establish
a link from an image to a sentence using a score from the comparison made
between the context vector of an image and the context vector of a sentence.
Later, other researchers [15] presented a system that predicted the natural
language descriptions automatically from the image input. This is made
possible using recognition algorithms and exploiting statistics behind parsing
huge text data. Carrara et al. [4] proposed _Text2Vis_ , a neural network that
is capable of generating a visual representation in the visual feature space
of ImageNet from a short textual description. This concept serves as another
end of our objective for classifying images. In more recent work, Otto et al.
[9] outlined an approach to understand, categorise, and predict the semantic
relations of an image to text. Here, the authors derive a categorisation of
eight semantics in image-text pairs and illustrate how they can systematically
be characterised by a set of metrics. They make use of a Deep Learning system
to predict the classes using _multimodal embeddings_.

###  2.4 Image Classification based on Textual Descriptions

Many earlier approaches, relevant to this paper, include texts from the web to
train and predict a Zero-Shot Learning classifier. Elhoseiny et al. [5]
proposed an optimised formulation that combines knowledge transfer techniques
and a regression function to predict a visual classifier. Ba et al. [3]
considered deep neural networks to predict convolutional classifiers by using
text features to find the optimal weights for the layers of a deep neural
network. This approach was reported to give a noticeable improvement in zero-
shot classification. Qiao et al. [13] revisited the importance of
regularisation on ZSL. They show that applying the attribute-based formulation
to text achieves better performance.

##  3 Model Architecture and Methodology

In this section, we describe our model architecture in detail. This is divided
into three components, i) extracting features from images, ii) extracting
features from text documents, and iii) building a ZSL model that learns to
classify the image as a result of mapping these attributes.

###  3.1 Image Feature Extraction

Every image is stored in a machine in the form of a matrix, where each element
in this matrix holds the image’s pixel value. Any mathematical operations on
an image are inherently performed on this matrix.
From image feature extraction, we intend to collect representative data from
the images. Features play a crucial role in the domain of computer vision and
image processing. Keras provides a wide variety of Deep Learning models that
are used for image classification, feature extraction, and transfer learning.
We decided to use VGG16 [14] due to its promising results on learning critical
features in images. VGG16 is a convolutional neural network with 16 layers.
This model was proposed by Simonyan and Zisserman [14] and the authors report
achieving a 92.7% _top-5_ test accuracy.
As an aside, we will explain what we mean by ‘Top-5 accuracy’. If we are
testing our model with an image of ‘cat’ and the model predicts { ‘lion’,
‘tiger’, ‘cat’, ‘leopard’, ‘dog’ } classes in ascending order of their
probabilities then since the expected class is one among the predicted
classes, we consider this as a true positive instance.

###  3.2 Text Feature Extraction

A challenging problem when dealing with text descriptions is to understand the
context. Traditional NLP techniques can fail when context is important. The
second component of our model is to extract features from the text. We have a
total of 200 textual descriptions, one for each class. Most previous work uses
Term Frequency Inverse Document Frequency (TF-IDF) to retrieve the features
from the text. Some researchers also form attributes manually helping the
model to focus on important aspects of the text. TF-IDF is inefficient for
capturing semantics and overlooks the position of text in a document. On the
other hand, manually defining the attributes makes the model unstable when it
encounters the raw textual descriptions. As a solution, one of the biggest
breakthroughs in understanding the context in the text is ELMo (Embeddings
from Language Models), an NLP state-of-the-art framework introduced by
AllenNLP [11].
ELMo is a deep contextualised word representation that captures the semantics
and context of a word in a sentence [11]. Word vectors from ELMo are achieved
as a result of the computations carried out on top of a two-layer
bidirectional language model (biLM) where each layer has a forward and
backward pass. This architecture converts a raw text string to word vectors
with the help of a character-level CNN. These word vectors are provided as
input to the first layer of biLM, the output of which forms intermediate word
vectors which are then fed to the second layer of biLM. ELMo word vectors are
the result of the weighted sum of raw word vectors and the two intermediate
word vectors. Since this architecture works on a character level, and forms
the vectors considering the entire sentence, a word can have different word
vectors under different contexts [12]. TensorFlow Hub is the library enabling
us to use ELMo in our work.

###  3.3 Building a Zero-Shot Learning Model

The above retrieved visual feature vectors, and textual feature vectors are
fed into our neural network as input. We form two sets of classes: seen
(training) and unseen (testing or zero-shot). We then form two datasets (seen
and zero-shot) within our code that consist of two attributes (image features,
text features) and one label (class) each.
One might wonder, if we are not including samples from test classes during
training, how is it possible for our model to predict those classes? This is
what makes Zero-Shot Learning a challenging and interesting area. To build a
model, we have to make sure that both the independent and dependent attributes
are located in the same space. Hence, to achieve this, we explore the
relationship between these attributes with the help of intermediate level
semantic vector representation. This representation is introduced to enable
sharing knowledge and to establish a mapping function between seen and unseen
classes. These semantic representations could be achieved either by semantic
attributes or by semantic word vectors. We will be focusing on semantic word
vectors where we project each class label into semantic space and these
projections are then used as prototypes for our Zero-Shot Learning classes.
For simplicity let us call these semantic word vectors _class vectors_. To
obtain class vectors for each class in our dataset, we will be using Google’s
Word2Vec model.

Figure 2: Visibility for training and testing our model

To summarise, for training classes, we have image features, text features,
training class labels, and training class vectors. However, for testing
classes, we only have their class vectors visible to our model. Note that the
model has never seen any samples from testing classes and only has visibility
to their class vectors.

##  4 Proposed Model

Since we aim at developing Zero-Shot Learning classifiers, we have to define a
training model that learns on mapping our attributes (visual features from the
image and textual features from descriptions) to their class vectors and in
turn to their class labels. In order to achieve this, we use _categorical
cross entropy_ as our loss function. If we consider visual feature vector
$x_{i}$, corresponding textual feature vector $t_{j}$ and $I_{i,j}$ as the
actual class label in categorical form then the loss function is computed
using the following equation:

$Loss\ =\ -\sum_{i=1}^{C}I_{i,j}\ \log(\ \hat{y_{j}}\ (x_{i},\ t_{j}))$

where $\hat{y_{j}}\ (x_{i},\ t_{j})$ is the $j^{th}$ scalar value in the
prediction we obtain from our model. Once we are done training our model, we
pop out the last layer so that we get class vectors as predictions. When a
sample from an unseen class is given to this network, we will be able to
obtain a semantic word vector as an output. Since we know that if a vector is
nearer to a certain class vector then the sample has a higher probability of
belonging to that class. By measuring the output vector’s distance to all
other class vectors, we will be able to perform classification.
Let $x^{(n1)}$ denote visual feature vector for an image, $y^{(n2)}$ denote
textual feature vector for the corresponding textual description, and
$l\in\\{1,...,C\\}$ denotes the class labels, and $(n1,n2)$ represents vector
dimensionality, $x,y\in{R}^{d}$, and $C$ is the total number of classes we
have in our dataset. We split this data into training and testing with $M,N$
instances respectively, where $p\in l_{M}\not\in l_{N}$ or $p\in l_{N}\not\in
l_{M}$. Therefore, now we have a training set $D_{train}\ =\
\\{x_{train}^{(n1)},\ y_{train}^{(n2)},\ l_{train}\\}$ and testing set
$D_{test}\ =\ \\{x_{test}^{(n1)},\ y_{test}^{(n2)},\ l_{test}\\}$. The model
will be trained on a training set $D_{train}$ and testing set $D_{test}$ will
not be seen until the model is being tested.

###  4.1 Training Model

We are considering 171 classes for training with 60 images per class. Once we
have visual feature vectors and corresponding textual feature vectors, we feed
both these vectors to our neural network. The network developed consists of 10
hidden layers, one semantic vector space layer with 300 neurons (embedding
size) followed by the output layer with 171 neurons (number of training
classes). All the hidden layers are equipped with _‘relu’_ activation and the
final output layer is with _‘softmax’_. While training the network, the model
learns to map both the feature vectors to corresponding class labels with the
help of class vectors in semantic vector space. So, in turn, it learns to map
the feature vectors to their class vectors. The below illustration explains
this clearly.

Figure 3: Proposed Training Model with image and text input

_Brewer Blackbird_ is one of the training classes and a model with a sample of
this class is illustrated above. As we can see, the image for this class is
fed to the VGG16 which yields us an image feature vector of $1\times 4096$
dimensions. On the other hand, a corresponding textual document is fed to the
ELMo which yields us a textual feature vector of $1\times 1024$ dimensions.
These two feature vectors form attributes of our model and are fed to train
our neural network, the mapping function. Once we finish training our network,
we save the model along with its weights which are used during testing.

###  4.2 Testing Model

We are considering 25 classes (each with 60 images) for testing our model.
When we say unseen classes, samples belonging to these 25 classes have not
been encountered before. The visual features and textual features of these
classes are very new to the model. The process is like that of training in the
beginning. To perform ZSL, we pop out the final softmax layer from the
training model discussed above and let the model predict the class vector in
the semantic vector space. Now, our model will predict for any pair of
attributes, a class vector of $(1\times 300\ dimensions)$ in the semantic
vector space.
_Semantic Vector Space Mapping:_ Let us here see how we retrieve the class
label when our model is predicting only the class vector. We make use of
K-dimensional trees, popularly known as _KDTrees_. KDTrees are defined as a
binary search tree where data in each node is a K-Dimensional point in space.
We can derive the class labels by providing a query to this tree instance
along with the number of nearest nodes or vectors we intend to consider (here
k = 10, 5, 1). These class labels are arranged in increasing order of their
distance from the predicted class vector.

Figure 4: Semantic Vector Space illustrating the visibility of the model
during training

##  5 Experimental Settings

###  5.1 Image Feature Extraction

When compared to that of the traditional hand-engineered feature extraction
techniques, CNNs outperforms others by learning the complex and crucial
feature representation from the raw image pixels. The image feature vectors
obtained from the CNNs hold local and spatial information from the pixels of
the image. Therefore, we import the pre-trained VGG16 from Keras models and by
popping the final sigmoid layer we extract $1\times 4096$ dimension feature
vectors. The optimiser used while training this model is stochastic gradient
descent with a learning rate of 0.1 and loss being categorical cross-entropy.
The hyper-parameters are set to their default values. We load the images using
the Keras _load image_ package and preprocess it before feeding it to the
network. Consider we have $N$ images belong to $C$ classes where label $l\
=\\{1,....,C\\}$. $\forall n\in N,\ VGG16(n)$ will obtain a feature vector
$x^{(n1)}$, where $n1$ is the dimensionality of the vector i.e $1\times 4096$.
Now once we extract feature vectors for all the images, we will be having
$x_{N}^{(n1)}$.

###  5.2 Textual Feature Extraction

The features from the textual descriptions are obtained from ELMo using
_TensorFlow Hub_. We extract the embeddings in the form of a dictionary. To
successfully extract features from the text, we have to clean it and perform
lemmatisation (normalisation) where the text converts each word to their base
form. We provide this preprocessed text as an input to the word embedding
instance (ELMo) which outputs the corresponding feature vector.
As specified, we have 200 classes in our dataset and the textual description
for each of these 200 classes are purely extracted from Wikipedia. Let us
assume that we have M documents where $M\ =\ C$ classes with labels $l\ =\
\\{1,....,C\\}$. $\forall m\in M,\ ELMo(m)$ will obtain for us an embedding or
textual feature vector $y^{(n2)}$, where $n2$ is the dimensionality of the
vector i.e $1\times 1024$. Now, once we extract feature vectors for all the
textual documents, we have $y_{M}^{(n2)}$.

###  5.3 ZSL Model

The above two feature vectors $x^{(n1)},y^{(n2)}$ are the two independent
variables or attributes to our model. In our experimental results, we found
that reducing the dimensionality of the visual feature vector from $1\times
4096$ to $1\times 1024$ increased the model performance by 5%. This dimension
reduction is performed by a neural network with 3 hidden layers equipped with
a _‘relu’_ activation function and a final _‘softmax’_ layer. To avoid the
model overfitting, we perform batch normalisation and add dropout layers to
the network. We fed this reduced visual feature vector to the neural network
along with textual feature vectors. This neural network is developed with five
hidden layers, followed by the semantic vector space layer and then by the
final softmax layer. The hidden layers and the semantic vector space layer are
equipped with a _‘relu’_ activation function.
Let us now explain how we get class vectors for each class label in the
semantic vector space. From our experimental results, we found that _Word2Vec_
establishes better embeddings when compared to that of _GloVe_. This is mostly
because we wish to extract embeddings for the scientific names of the species.
Word2Vec makes it possible for us to have embeddings for 196 of 200 classes so
we remove the classes that it cannot handle. As a result, we have 11547 images
belonging to 196 classes. Word2Vec provides us an embedding of $1\times 300$
dimensions. We obtain an instance of Word2Vec from gensim models and once we
extract the class vectors from this instance, we save it locally as a numpy
file. We load this numpy file later while mapping the class vectors to the
class labels. The final layer of the model is responsible for this mapping
function. This is achieved with the help of _customised kernel initialisation_
offered by Keras. Here we load the saved numpy file with class labels as keys
and class vectors as values. And for every instance in the training dataset,
we return the vector associated with the class label (key) which makes it
possible for our model to learn a mapping of the semantic vectors to the class
labels.

##  6 Results and Observations

###  6.1 Model Evaluation

The evaluation metric we are using is top-k accuracy. According to Ba et al.
[3], the best way to evaluate a multi-class classifier is by sorting the final
prediction score $\hat{y_{c}}$ obtained from the model. We deal with Zero-Shot
Learning on top of multi-class classification which increases the complexity
due to the much larger prediction space.

| Top-10 Accuracy(%) | Top-5 Accuracy(%) | Top-1 Accuracy(%)
---|---|---|---
Seen classes | $99.9\pm\ 0.01$ | $99\pm\ 0.5$ | $97\pm\ 0.5$
Unseen classes | $43\pm\ 0.7$ | $32\pm\ 0.5$ | $19\pm\ 0.8$
Table 1: Proposed Zero-Shot Learning Model performance on Caltech Birds
dataset

Table 1 illustrates the results achieved by our model for classifying the
samples for both seen and unseen class categories for the Caltech Birds
dataset. Seen samples are tested using the usual machine learning technique of
splitting the data (here belonging to training classes) into training and
testing and evaluating the performance of the model. The above results
demonstrate that our approach of developing a multi-class Zero-Shot Learning
classifier can classify the images belonging to classes it has seen and also
to the classes it has never seen before.

###  6.2 Performance Evaluation

Many earlier research approaches use articles from the web to develop a Zero-
Shot Learning classifier. One such approach, which gave a noticeable
improvement in zero-shot classification is where Ba et al. [3] considered deep
neural networks to predict convolutional classifiers. Since this is most
relevant to our work, we wish to discuss the results of this approach. The
authors considered 160 seen and 40 unseen classes from the available 200
classes that are randomly assigned. Ba et al. achieved the highest accuracy
for the unseen classes, a top-5 accuracy of 42.8%, and top-1 accuracy of 12%.
Also, the author mentions the performance of the model for seen classes, a
top-5 accuracy of 66.8%. The ‘training-testing’ split for the seen class’s
evaluation is not reported in the work.
Using the same dataset, we have developed a model using an altogether
different approach. We consider a word embedding _ELMo_ for extracting textual
features and pre-trained _VGG16_ , pre-trained on the ImageNet dataset, for
extracting visual feature vectors. Both are then fed to the neural network as
attributes. Our model considers 171 classes as seen and 25 classes as unseen.
The model achieves for unseen classes a top-5 accuracy of $32\pm 0.5\%$ and
top-1 accuracy of $19\pm 0.8$%. To test the performance of our model for seen
classes, we have considered _70:30_ random split in the training dataset
(samples from 171 training classes) and, our model could achieve top-5
accuracy of $99\pm 0.5$% and top-1 accuracy of $97\pm 0.5$%. Since we were
able to find the optimal weights for our multi-class classifier, the proposed
model performs extremely well while classifying images belonging to both seen
and unseen classes .
Top-1 accuracy is when the model assigns the highest probability for the
actual class. Since we are able to achieve promising results for the most
critical predictions i.e top-1 accuracy, for the unseen classes and with the
fact that the model is trained on a much smaller dataset, it is evident that
the model proposed in this paper is efficient and achieves promising results.

###  6.3 Model Predictions

Below is a visualisation of the predictions produced by our model.

Figure 5: Model predictions on testing of zero-shot classes.

This visualisation illustrates the top-5 accuracy metric, explained earlier
where the leftmost class in the _nearest neighbor labels_ has the highest
similarity score, and this similarity score decreases to the right. ‘Marsh
Wren’, ‘Ringed Kingfisher’, and ‘Ivory Gull’ all belong to our testing classes
i.e. the classes whose samples are never seen by the model during training.
The model performance is satisfactory for the first two examples. We can see
that the third example is one such instance where our model is unable to
predict the exact class label but the classes which are chosen by the model
under top-5 nearest neighbors belong to the same parent class (‘Gull’).

##  7 Conclusion

In this paper, we introduce a multi-class Zero-Shot Learning model that learns
to predict the label for images belonging to unseen classes from their
Wikipedia articles. We have developed a deep neural network to establish a
mapping function that maps the textual feature vectors from the raw Wikipedia
articles and visual feature vectors from images to a semantic space with the
help of semantic vectors derived from the class labels. This can also be
interpreted as the ability of the model to intelligently apply the knowledge
acquired over training classes with the help of an objective function.
We demonstrated that our model outperformed an existing zero-shot model on the
top-1 accuracy metric on the CUBird dataset using only raw images and
Wikipedia articles.

## References

  * [1] Akanksha, Krishnan, Paul, N., Munjal, P.: Semantically aligned bias reducing zero shot learning (04 2019)
  * [2] Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C.: Label-embedding for attribute-based classification. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition. pp. 819–826 (2013)
  * [3] Ba, J.L., Swersky, K., Fidler, S., Salakhutdinov, R.: Predicting deep zero-shot convolutional neural networks using textual descriptions. In: Conference on Computer Vision (ICCV). pp. 4247–4255 (2015)
  * [4] Carrara, F., Esuli, A., Fagni, T., Falchi, F., Moreo Fernández, A.: Picture it in your mind: Generating high level visual representations from textual descriptions. Information Retrieval Journal (10 2017)
  * [5] Elhoseiny, M.: Write a classifier: Zero shot learning using purely textual descriptions (12 2013)
  * [6] Fu, Y., Hospedales, T., Xiang, T., Gong, S.: Transductive multi-view zero-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (01 2015)
  * [7] Kodirov, E., Xiang, T., Gong, S.: Semantic autoencoder for zero-shot learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4447–4456 (2017)
  * [8] Li, K., Min, M.R., Fu, Y.: Rethinking zero-shot learning: A conditional visual classification perspective. In: Conference on Computer Vision (ICCV). pp. 3582–3591 (2019)
  * [9] Otto, C., Springstein, M., Anand, A., Ewerth, R.: Understanding, categorizing and predicting semantic image-text relations. Proceedings of the 2019 on International Conference on Multimedia Retrieval (2019)
  * [10] Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22(10), 1345–1359 (2010)
  * [11] Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations (02 2018)
  * [12] Prateek: ELMoArchitecture. https://www.analyticsvidhya.com/blog/2019/03/learn-to-use-elmo-to-extract-features-from-text/ (2019)
  * [13] Qiao, R., Liu, L., Shen, C., Van Den Hengel, A.: Less is more: Zero-shot learning from online textual documents with noise suppression. In: Computer Vision and Pattern Recognition (CVPR). pp. 2249–2257 (2016)
  * [14] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv 1409.1556 (09 2014)
  * [15] Wang, H., Zhang, Y., sheng Yu, X.: An overview of image caption generation methods. Computational intelligence and neuroscience p. 3062706 (2020)
  * [16] Wang, J., Markert, K., Everingham, M.: Learning models for object recognition from natural language descriptions (01 2009)
  * [17] Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-UCSD Birds 200. Tech. Rep. CNS-TR-2010-001 (2010)


# JarviX: A LLM No code Platform for Tabular Data Analysis and Optimization

Shang-Ching Liu1,3,†, ShengKun Wang2,3,†, Wenqi Lin3, Chung-Wei Hsiung3,
Yi-Chen Hsieh3, Yu-Ping Cheng3, Sian-Hong Luo3,Tsungyao Chang3, Jianwei
Zhang*1
1Department of Computer Science, University of Hamburg
2Department of Computer Science, Virginia Tech
3Synergies Intelligent Systems, Inc.
<EMAIL_ADDRESS>

###### Abstract

In this study, we introduce JarviX, a sophisticated data analytics framework.
JarviX is designed to employ Large Language Models (LLMs) to facilitate an
automated guide and execute high-precision data analyzes on tabular datasets.
This framework emphasizes the significance of varying column types,
capitalizing on state-of-the-art LLMs to generate concise data insight
summaries, propose relevant analysis inquiries, visualize data effectively,
and provide comprehensive explanations for results drawn from an extensive
data analysis pipeline. Moreover, JarviX incorporates an automated machine
learning (AutoML) pipeline for predictive modeling. This integration forms a
comprehensive and automated optimization cycle, which proves particularly
advantageous for optimizing machine configuration. The efficacy and
adaptability of JarviX are substantiated through a series of practical use
case studies.

†††These authors contributed equally to this work.

Figure 1: JarviX system overview

##  1 Introduction

Although the predominant focus of contemporary research on large language
models is the evaluation of various tasks Liang et al. (2022); Zhao et al.
(2023), there is a noticeable lack of academic resources that provide
structured guidelines and frameworks for downstream applications. Tabular data
analysis, as an important application task of LLMs, has always faced
challenges related to the precision of mathematical calculations. Despite its
ability to address complex high school math problems and participate in
advanced mathematical discussions, advanced models such as GPT-4 are not yet
on par with expert level performance Bubeck et al. (2023). They are prone to
basic errors and can sometimes produce incoherent outputs. This may stem from
the fact that autoregressive models lack self-correction mechanisms when
generating solutions Shen et al. (2021). This paper introduces a thorough
approach towards employing LLMs for tabular data analysis tasks, specifically
aiming to equip nonspecialists with the ability to engage in advanced data
analytics using LLMs within a rule-based system. Although LLMs have proven to
be potent in data processing Zhao et al. (2023), their application in guiding
users through rule-based systems to intuitively create data visualizations,
synthesize statistical insights, and provide context-aware explanations is
significantly underexplored.

LLM guides users through a rule-based system in JarviX, which enables data
visualization and statistical analysis. It leverages a vectorized domain
knowledge repository to provide relevant explanations for each visualization
and suggests further exploration directions Feng et al. (2023). Users can
generate additional exploratory charts and insights through text or voice
input, facilitated by Whisper.111https://github.com/openai/whisper These
insights are processed by the Vicuña
model222https://lmsys.org/blog/2023-03-30-vicuna/ through prompts fine-tuned
by GPT-4333https://openai.com/research/gpt-4. This approach culminates in a
comprehensive report encapsulating all the insights and analytical processes,
serving as a thorough guide for users and a blueprint for future analysis.

Furthermore, this study explores the integration of H2O-AutoML-customized
AutoML pipelines H2O.ai (2023) into this process. The use of AutoML is
demonstrated to identify the best targets with respect to other data columns
and to build specific models to optimize results for various objectives, such
as optimal factory configurations He et al. (2021).

The primary objective of this study is to empower users with the knowledge and
tools necessary to harness the power of LLM for rule-based data analytics by
fine-tuning Chung et al. (2022) and AutoML. The paper concludes by underlining
the potential of this approach in democratizing data analytics, thereby
fostering more strategic and informed decision-making.

##  2 Related Work

###  2.1 Natural Language Interfaces for Data analysis

Natural Language Interfaces have recently garnered attention and integration
into various commercial data analysis and visualization software, such as IBM
Watson Analytics IBM (2023), Microsoft Power BI Microsoft (2023), Tableau
Salesforce (2023), and Google Spreadsheet Google (2023). Despite initial
limitations, such as confining natural language interactions to data queries
and standard chart types, the approach is evolving. Current methods of Natural
Language Processing (NLP) incorporate heuristic algorithms, rule-based
systems, and probabilistic grammar-based approaches, each with their
respective challenges and trade-offs in accuracy, flexibility, and
computational resources Miwa and Bansal (2016); Voigt et al. (2021);
Satyanarayan et al. (2016).

###  2.2 Utilization of LLMs in Advanced Data Analysis

Rajkumar’s performance evaluation of LLM on Text2SQL Rajkumar et al. (2022)
was a significant development. Despite the considerable progress, including
Sun et al.’s Sun et al. (2023) Text2SQL method achieving 77.3% accuracy on the
Spider benchmark, constructing a seamless pipeline is still challenging. The
optimization strategy of Hu et al. Hu et al. (2023) and the question
refinement strategy of Guo et al. Guo et al. (2023) show further improvements.
The Maddigan et al. system Maddigan and Susnjak (2023) underscores the
importance of visualization post-SQL generation. However, the demand for
practical solutions for real-world applications remains, prompting the
development of the JarviX platform. It bridges the gap, offering higher-level
APIs and integrating LLMs for a comprehensive solution for advanced data
analysis.

###  2.3 External Knowledge Integration

Currently, LLMs are confronted with two issues: privacy implications and the
obsolescence of training data. Utilizing user-interaction data for further
training in online LLM applications can potentially jeopardize security.
Additionally, the significant costs associated with retraining can make LLMs
outdated over time. LangChain Chase (2022) provides an innovative solution
that continuously embeds the latest data and retrieves relevant information
from its database, consequently generating current responses while preserving
privacy. Moreover, the introduction of the llama_index Liu (2022) proposes a
more structured approach for embedding levels to retrieve query-related
information, such as identifying the latest facts or providing relational
data. This method improves the precision of LLM responses.

Figure 2: Main page Figure 3: JarviX insight

##  3 Overview

JarviX is a no-code platform for efficient analysis and optimization of
tabular data, handling both structured and unstructured types, as illustrated
in Figure 1. For structured data(e.g., csv files, data frames), it performs
preliminary processing tasks including data type detection, statistical
computation, and correlation analysis, storing the results in a Postgres
database. Unstructured data (e.g., text files, audio files) is managed through
text extraction and embedding, followed by storage in a vectorized database,
such as Elastic Search.

Users can interact with the platform through three key features: JarviX
Insight, Natural Language Interfaces, and JarviX Guidance. JarviX Insight
collects structured data information such as column names, types, and
statistical data, and employs a LLM to generate a data summary report,
providing users with an understanding of their data and identifying key
questions.

The Natural Language Interfaces feature accommodates user queries about their
datasets, either voice-to-text or typed, and translates these queries into a
rule-based system via a fine-tuned LLM. This delivers relevant data
visualizations, explanations, and follow-up question suggestions.

Lastly, JarviX Guidance assists users through a step-by-step data analysis
process. It takes into account the user’s understanding of the datasets, their
role, the specific dataset, and target column they wish to analyze. Using this
information, JarviX anticipates the questions a user might want to address
first and commences the result generation process. It also provides an
appropriate endpoint for analysis for each user. All stages of the analysis
are recorded, including the middle $result_{i}$, and compiled into a
comprehensive report that users can save and share.

##  4 System Break Down

###  4.1 Data Input Methods

JarviX offers users three methods for uploading structured data: via SFTP,
database connections, or direct CSV file uploads. For unstructured data, the
platform currently supports only file uploads.

JarviX integrates a data cleaning interface with automated functions, enabling
users to efficiently perform data cleaning. Then, the system initiates data
pre-processing, which includes automatic type detection, column statistics
computation, and calculation of the correlation matrix between columns. It is
worth noting that these tasks—column statistical computation and correlation
matrix calculation—are executed asynchronously, ensuring that user progress
isn’t hindered.

In handling unstructured data, we leverage various connectors in the llama hub
Zhang (2023) to perform text extraction. The extracted data is stored in the
vector database using Faiss Meta (2023) and assigned the same project ID as
the structured database. This cohesive data management strategy ensures
seamless integration and retrieval of both structured and unstructured data.

###  4.2 JarviX Insight

The JarviX Insight feature facilitates autonomous report generation, enabling
users to comprehend data more effectively and gain insight into potential
subsequent questions, as illustrated in Figure 1 and shown in Figure 3. Upon
activation of the JarviX Insight function, two distinct processes occur.
Initially, a prompt with preprocessed data is used to determine the nature of
the data, which subsequently assists in the creation of a data summary text.
Currently, the LLM is used to generate the ten most pertinent questions. These
questions serve as a foundation for generating a variety of potential
visualization results. By integrating these elements, a comprehensive data
summary report is crafted.

###  4.3 Question Matcher

The Question Matcher is the key module that links questions from a natural
language interface to their corresponding modules using SQL matching. This
process relies on identifying three types of keywords: 1) column name-related
terms, 2) restriction-related phrases (e.g. "top ten"), and 3) algorithm or
module keywords. Once these keywords are identified, the module begins to
merge the specific restrictions associated with each column into a unified
combination. This combination is then matched with the algorithm or module
indicated by the third type of keyword. More details on Question Matcher are
contained in the Appendix A.

###  4.4 Analysis Consultant

The Analysis Consultant is designed for users who have an initial
comprehension of their data and express interest in exploring specific
columns, as depicted in Figure 1. The process begins with the setup of the
analysis parameters based on previously outlined criteria. Subsequently, the
LLM formulates the first query. The Consultant then generates comprehensive
results that include visualizations, insights with supportive explanations,
and prompts for potential follow-up queries from the users. A crucial aspect
of this process is the incorporation of professional knowledge into the
insights, providing not only a fundamental explanation of the visualizations
but also integrating general knowledge and background understanding into the
explanation. If the analysis process is deemed comprehensive, the Consultant
may propose generating a report.

LLMs put forth subsequent analytical queries, facilitating users in delving
deeper into their target column’s data features. The formulation of questions
is based on the preceding results of the analysis and the roles selected by
the user. This feature equips people who lack in-depth data analysis knowledge
to yield thorough and dependable interpretations of their data. A step-by-step
description of this process will be exhibited in Session 5.

###  4.5 LLM, Prompt Engineering

In our experiments, we leverage the vicuna-13b-1.1-gptq-4bit-128g TheBloke
(2023) as our base LLM, and optimize the prompts with the advanced language
understanding capabilities of GPT-4 to achieve better results.

Figure 4: Prompt optimization process Figure 5: Question matcher

Our approach, based on prompt engineering, can be viewed as a two-stage
process, as detailed in Figure 4. First, we manually generate prompts, which
require a deep understanding of the task module and a clear view of the
structure and goal of the task. The resultant prompts are both specific and
concise.

Post-initial generation, we instigate a feedback loop to optimize the prompts.
Every prompt is fed into the Vicuña model to generate respective outputs,
which are compared to the expected results to derive a performance metric. If
a prompt falls short of our performance benchmark, it’s replaced with a new
one generated by GPT-4.

We persist with this prompt optimization until all prompts meet our
performance criteria. To corroborate our approach, we undertake a range of
experiments spanning tasks like normality tests, forecasting, comparisons,
root cause analysis, anomaly detection, and relationship extraction.

The primary aim of our experiments is to juxtapose the performance of our
approach against existing prompt engineering techniques. The results,
presented through multiple examples from our experiments, attest to the
strength of our methodology. Taking advantage of the interaction between
prompts and feedback loops, the Vicuña model yields rich and insightful
responses to complex data-related queries.

##  5 Case Study

In this section, we demonstrate two separate use cases: 1) Utilizing JarviX
Insight for custom analysis with data matching, and 2) Exploring JarviX’s
guidance use cases.

###  5.1 Case 1: JarviX Insight and Solar Cell Manufacturing

We present a case that explores solar cell manufacturing data and demonstrates
how to increase efficiency using JarviX Insight.

As shown in Figure 2, users begin by using JarviX Insight to generate a report
for general understanding of the data set. If users are unfamiliar with the
data, the JarviX Insight function can provide a general report that answers
two questions: 1) What is the subject of these data? 2) What are the most
valuable queries that can be made using this dataset?

Upon gaining insights that quality could be a potential area for enhancement,
users can utilize the "Question Matching" feature. This function facilitates
the formation of general queries, such as “What is the difference between high
quality and low quality” As illustrated in Figure 5, the “Question Matcher”
successfully translates user inputs into keywords recognizable by a rule-based
system.

The visualization results of different algorithms indicate the key differences
between high and low quality. The visualization includes straightforward
insights, suggested questions, actions, and the main diagram.

Our AutoML pipeline simplifies the process of training a machine learning
model. Users simply define the data source, dataset, and target column, as
well as the performance metric for optimization (such as MAE, MSE, or RMSE).
Though the training strategy’s precision may impact the model’s training time,
once all parameters are set, the model can be generated. Once the model is
established, users can explore optimal settings through the simulation panel,
as depicted in Figure 6. This panel allows users to identify optimal
configurations within the defined range. Importantly, the model is designed to
progressively refine itself with the influx of new streaming data. This
dynamic adaptation promises improved outcomes over time as settings are
intelligently adjusted in response to the evolving data.

Figure 6: Simulation

This process provides an ideal optimization cycle for customers. In this case,
the solar panel manufacturer increased efficiency by 10% using this
optimization cycle.

###  5.2 Case 2: JarviX Guidance, An Analysis of LCD Factory Data

If users are new to JarviX, understanding its functionality or learning how to
analyze data might be challenging. To address this, our analysis consultant is
available to guide you through the process. In this case study, our focus is
on interpreting a dataframe relevant to an LCD panel factory.

Setting up the system properly is paramount to ensure it identifies the data
pertinent to the user. We commence by describing the content of the data
table, followed by outlining our analysis objectives and roles, as depicted in
Figure 7.

Figure 7: Settings for the analysis

As an initial step, JarviX recommends the appropriate analysis that users
should consider. It’s often challenging for users to extract vital information
from the analysis process when faced with varied data. Therefore, we inform
our model about the analysis approach, so it can recommend suitable subsequent
analyses based on users’ requirements.

Upon starting the analysis, JarviX assists users in interpreting the results
and guides them to the next steps.

Through our differential analysis, we determined that the electrical test
performance heavily depends on the stability of ambient humidity. A list on
the left displays the significant factors that influence the differences
between high and low electrical tests. In particular, humidity is the top
factor, indicating that humidity differences significantly affect the
performance of electrical tests.

At this stage, the system suggests the production of a summary report based on
the previous analysis. JarviX will first show a summary suggestion, then
recapitulate the previous analysis steps as Figure 8. The analysis
consultant’s guidance enables the user to obtain valuable analysis results,
which can help optimize company strategy or uncover potential business value.

(a) Summary text

(b) Summary figure

Figure 8: Analysis summary

##  6 Conclusion

JarviX, by integrating LLM and AutoML technologies, presents a unique and all-
encompassing approach for the analysis of tabular data. The system integrates
non-structural data to generate profound insights, employing LLM to aid users
in their data exploration endeavors.

##  7 Future Work

The flexibility and adaptability of the JarviX platform offer several avenues
for future improvements. In particular, there are opportunities to fine-tune
the LLM to improve personalized recommendations, extend the range of accepted
data types and query categories, and improve user interface design for a
better user experience.

## Ethical Considerations and Limitations

JarviX formulates responses influenced by the user-provided context. Biased
results may arise if the context involves biases related to aspects such as
the location or language of the user Hadi et al. (2023). For example, given
that JarviX currently only supports processing and analysis in English and
Chinese, it might yield biased answers when inquiries about a specific culture
or religion are presented, especially if JarviX lacks adequate training in
that particular cultural or religious context, due to its confined knowledge.
Also, JarviX is designed to recognize only plain text information and cannot
identify multimodal tabular data, such as financial statements or
instructional videos.

## Acknowledgements

This research is supported by Synergies Intelligent Systems, Inc. and
partially supported by the DFG/NSFC Collaborative Project “Crossmodal Learning
- Adaptivity, Prediction and Interaction” SFB/TRR169.

## References

  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Chase (2022) Harrison Chase. 2022.  LangChain.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Feng et al. (2023) Yingchaojie Feng, Xingbo Wang, Bo Pan, Kam Kwai Wong, Yi Ren, Shi Liu, Zihan Yan, Yuxin Ma, Huamin Qu, and Wei Chen. 2023.  Xnli: Explaining and diagnosing nli-based visual data analysis.  _IEEE Transactions on Visualization and Computer Graphics_.
  * Google (2023) Google. 2023.  Google sheets.
  * Guo et al. (2023) Chunxi Guo, Zhiliang Tian, Jintao Tang, Shasha Li, Zhihua Wen, Kaixuan Wang, and Ting Wang. 2023.  Retrieval-augmented gpt-3.5-based text-to-sql framework with sample-aware prompting and dynamic revision chain.
  * H2O.ai (2023) H2O.ai. 2023.  h2o-automl.  3.42.0.
  * Hadi et al. (2023) Muhammad Usman Hadi, R Qureshi, A Shah, M Irfan, A Zafar, MB Shaikh, N Akhtar, J Wu, and S Mirjalili. 2023.  A survey on large language models: Applications, challenges, limitations, and practical usage.  _TechRxiv_.
  * He et al. (2021) Xin He, Kaiyong Zhao, and Xiaowen Chu. 2021.  Automl: A survey of the state-of-the-art.  _Knowledge-Based Systems_ , 212:106622.
  * Hu et al. (2023) Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. 2023.  Chatdb: Augmenting llms with databases as their symbolic memory.
  * IBM (2023) IBM. 2023.  Ibm watson anlytics.  3.5.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022.  Holistic evaluation of language models.
  * Liu (2022) Jerry Liu. 2022.  LlamaIndex.
  * Maddigan and Susnjak (2023) Paula Maddigan and Teo Susnjak. 2023.  Chat2vis: Generating data visualisations via natural language using chatgpt, codex and gpt-3 large language models.
  * Meta (2023) Meta. 2023.  GitHub - facebookresearch/faiss: A library for efficient similarity search and clustering of dense vectors. — github.com.  https://github.com/facebookresearch/faiss.  [Accessed 17-Jul-2023].
  * Microsoft (2023) Microsoft. 2023.  Microsoft power bi.
  * Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016.  End-to-end relation extraction using lstms on sequences and tree structures.  _arXiv preprint arXiv:1601.00770_.
  * Rajkumar et al. (2022) Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022.  Evaluating the text-to-sql capabilities of large language models.
  * Salesforce (2023) Salesforce. 2023.  Tableau.
  * Satyanarayan et al. (2016) Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer. 2016\.  Vega-lite: A grammar of interactive graphics.  _IEEE transactions on visualization and computer graphics_ , 23(1):341–350.
  * Shen et al. (2021) Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021.  Generate & rank: A multi-task framework for math word problems.
  * Sun et al. (2023) Ruoxi Sun, Sercan O. Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. 2023.  Sql-palm: Improved large language model adaptation for text-to-sql.
  * TheBloke (2023) TheBloke. 2023.  vicuna-13b-1.1-gptq-4bit-128g model.  https://huggingface.co/TheBloke/vicuna-13B-1.1-GPTQ-4bit-128g?doi=true.  Accessed: 2023-07-20.
  * Voigt et al. (2021) Henrik Voigt, Monique Meuschke, Kai Lawonn, and Sina Zarrieß. 2021.  Challenges in designing natural language interfaces for complex visual models.  In _Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing_ , pages 66–73, Online. Association for Computational Linguistics.
  * Zhang (2023) Jesse Zhang. 2023.  Emptycrown/llama-hub: A library of data loaders for llms made by the community – to be used with gpt index and/or langchain.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023.  A survey of large language models.

##  Appendix A Experiment Results

###  A.1 Dataset

We conducted evaluations on JarviX using a variety of tabular
datasets444https://reurl.cc/jv693q sourced from open source collections,
covering different fields. A set of 10 manually crafted questions was
complemented by an additional 20 generated by GPT-4 for each data set. To
ensure relevance and meaningfulness, the prompts were designed with the phrase
“assuming that you are a professional data analyst in this field”, tailoring
the questions generated to the specific industry. The test data were then
classified into six distinct industry-based segments.

###  A.2 Results

The TABLE I presents the performance results of the question matching
evaluation, focusing on three specific aspects: column name, intention, and
restriction detection. Each aspect of the evaluation is thoroughly examined,
with results meticulously tabulated to offer a comprehensive understanding of
the system’s performance across the different dimensions. JarviX demonstrates
proficient recognition of individual columns. However, when faced with
questions that encompass multiple columns, there is a possibility that it
might not fully recognize all of them. The eleven intentions that JarviX is
capable of executing are illustrated in Figure 4. In addition, JarviX is
equipped to identify the following specific restrictions: Average, Median,
Sum, Greater than, Equal to, Less than, Plus, Minus, Multiply, Divide, Top,
Last, Maximum, Minimum. On the basis of our experimental findings, JarviX
shows enhanced performance as user queries exhibit clearer intent. However,
when faced with ambiguous queries, JarviX is prone to over-identifying or
under-identifying terms.

TABLE I : Evaluation result for Question Matching. Top1 indicates the
instances where the correct result was identified as the top most result. Top3
represents the cases where the correct result was found within the top three
results.

Data source | Column Name | Intention | Restriction
---|---|---|---
Top1 | Top3 | Top1 | Top3 | Top1 | Top3
Manufacture | 72.0 | 83.3 | 74.0 | 82.7 | 64.7 | 72.0
Sport | 73.3 | 88.3 | 75.0 | 90.8 | 65.8 | 76.7
Sales | 70.7 | 82.7 | 77.3 | 86.7 | 67.3 | 78.7
Food | 69.2 | 88.3 | 75.8 | 90.8 | 70.0 | 77.5
Health Care | 65.0 | 74.2 | 73.3 | 85.0 | 67.5 | 74.2
Banking | 81.7 | 93.3 | 79.2 | 91.2 | 66.7 | 74.2