# Towards Democratizing AI: A Comparative Analysis of AI-as-a-Service
Platforms and the Open Space for Machine Learning Approach

Dennis Rall1, Bernhard Bauer2, Thomas Fraunholz1

###### Abstract

Recent AI research has significantly reduced the barriers to apply AI, but the
process of setting up the necessary tools and frameworks can still be a
challenge. While AI-as-a-Service platforms have emerged to simplify the
training and deployment of AI models, they still fall short of achieving true
democratization of AI. In this paper, we aim to address this gap by comparing
several popular AI-as-a-Service platforms and identifying the key requirements
for a platform that can achieve true democratization of AI. Our analysis
highlights the need for self-hosting options, high scalability, and openness.
To address these requirements, we propose our approach: the ”Open Space for
Machine Learning” platform. Our platform is built on cutting-edge technologies
such as Kubernetes, Kubeflow Pipelines, and Ludwig, enabling us to overcome
the challenges of democratizing AI. We argue that our approach is more
comprehensive and effective in meeting the requirements of democratizing AI
than existing AI-as-a-Service platforms.

Keywords: Cloud Computing, Artificial Intelligence, Platform, AI-as-a-Service,
Computer systems organization, Distributed architectures, Information systems
applications, Computing platforms, Computing methodologies

11footnotetext: WOGRA AG, Hery-Park 3000, 86368 Gersthofen,
Germany22footnotetext: University of Augsburg, Germany

##  1 Introduction

The democratization of AI is currently a trend in AI research, driven by the
shortage of AI experts that hinders the use of AI in many areas, including
stock marking trading [1] and personalized medicine [2], to name only a
couple. The aim is to reduce the required expertise and make AI more
accessible to a wider range of users. Significant progress has been made in
the development and training of AI models, enabling the creation and training
of models automatically from data, known as low-code AI. Examples of low-code
tools include Ludwig [3], fast-ai [4], and Autogluon [5, 6]. Low-code AI tools
can be coupled with user-friendly interfaces to transition to no-code AI.
These advancements hold the potential to make AI accessible to people with
little or no programming experience, allowing them to apply AI to various
tasks without the need for technical expertise.

However, providing the infrastructure and setting up these frameworks and
tools remains a hurdle that requires expert knowledge. To overcome this
obstacle and further promote the democratization of AI, platforms are emerging
that take care of these tasks for the user and offer AI as a service. These
platforms provide a range of services, such as data storage, model training,
and API access, allowing users to easily integrate AI into their workflow
without the need for significant technical expertise. With AI-as-a-Service,
individuals and organizations can benefit from the power of AI without the
need to invest in expensive hardware, software, or personnel. As a result, AI-
as-a-Service has the potential to accelerate the adoption of AI and make its
benefits more widely available.

In this article, we aim to compare existing AI-as-a-Service platforms and
examine why they have not yet led to a breakthrough in the democratization of
AI. Based on this analysis, we aim to identify the requirements for a platform
that can successfully achieve this goal and then discuss our implementation to
meet these requirements. By exploring the strengths and weaknesses of existing
AI-as-a-Service platforms, we hope to provide insights into the design and
development of a platform that can truly democratize AI and make its benefits
accessible to everyone. Through this analysis, we hope to contribute to the
ongoing efforts to democratize AI and make its benefits available to a wider
audience.

##  2 Study of Existing Platforms

The availability of easy-to-use AutoML solutions as a platform is a crucial
factor for the participation of non-AI experts in the technology of machine
learning. We will now take a look at the various existing platforms for
AutoML.

###  2.1 AutoML Platforms by major Cloud Providers

The major cloud providers - Amazon
SageMaker111https://aws.amazon.com/sagemaker/, Google Vertex
AI222https://cloud.google.com/vertex-ai, and Microsoft Azure Automated
ML333https://azure.microsoft.com/en-us/products/machine-learning/automatedml/
\- have invested significantly in AutoML and have developed powerful and
scalable solutions. These machine learning platforms leverage cutting-edge
algorithms and state-of-the-art technologies. Additionally, these platforms
offer a high level of service, from prototyping to production, and provide a
no-code environment that enables developers to easily build and deploy machine
learning models.

One major disadvantage of these platform solutions is their lack of openness.
Since these platforms are not open source, expert users are limited in their
ability to customize and modify algorithms to fit their specific needs.

Additionally, these platforms are not self-hosted, meaning that sensitive data
must be entrusted to a third-party provider.

Furthermore, the algorithms used by these platforms are often black boxes,
making it difficult to understand how they arrived at their conclusions. This
lack of transparency can be problematic, particularly when it comes to ethical
considerations and ensuring that the algorithms are not biased or
discriminatory.

Additionally, customers are often locked into using a single vendor, limiting
their ability to switch providers or take their business elsewhere. Finally,
while these platforms offer a no-code environment, they are often complex and
difficult to navigate without a solid understanding of machine learning
concepts and terminology. This can be a major barrier for non-experts trying
to leverage machine learning in their work.

###  2.2 Standalone AutoML Platforms

An established platform that has made it its mission to overcome this obstacle
is DataRobot444https://www.datarobot.com/. The platform stands out in terms of
user-friendliness. Its no-code environment makes it easy for non-experts to
build and deploy machine learning models, and its scalable AutoML solution
covers the entire machine learning life cycle. However, like the major cloud
providers, DataRobot’s platform is not open source, meaning that experts are
limited in their ability to customize algorithms. Additionally, the platform
has no option to self-host, and the algorithms used are often black boxes,
limiting transparency and posing ethical concerns. Finally, users are locked
into the DataRobot ecosystem, which can be a major drawback.

On the other hand, Predibase555https://predibase.com/ attempts to achieve
complete transparency using a declarative machine learning approach [7] and
utilizing open-source components under the hood. However, minimal programming
knowledge is required for the low-code platform. In addition to this, is not
foreseeable that the platform will be made available as open source. Thus, it
cannot be self-hosted for sensitive data.

For users looking for a more open approach, H2O666https://h2o.ai/ [8] is a
powerful option. As an AutoML platform, H2O provides a scalable solution for
the complete machine learning lifecycle that can be used either as a PaaS or
self-hosted. However, H2O’s platform does have some limitations. Its low-code
environment may not be accessible to users with no programming skills,
limiting the democratization of machine learning. H2O knows about this
drawback and introduced a commercial no-code offering, H2O Driverless AI
Wizard, which leverages the full potential of AutoML for users with no
programming. But this solution is not open-source and cannot be self-hosted.

##  3 Requirements

So far, we have gained insights into the various platforms that offer AI as a
service, and we have concluded that they are not yet sufficient for achieving
true AI democratization. Therefore, our next step is to derive the
requirements for such a platform that can lead us towards the democratization
of AI. These requirements will be crucial for creating a platform that can
enable people from diverse backgrounds to access and benefit from AI
technologies. We will focus on identifying the key features and
functionalities that are needed to build a platform that can support a wide
range of users, including those with limited technical expertise.

###  3.1 Platform

But first, we want to emphasize why such a platform that offers AI as a
service is a requirement for achieving true AI democratization. One of the
main reasons why such a platform is crucial is that it allows users to start
using the software without the need for a complex setup process. Especially
with AI applications, there may be a significant amount of expert knowledge
required, as certain parts of the application need to be executed on
specialized hardware, such as GPUs, for effective performance. By offering AI
as a service, users can access the power of AI without needing to invest in
expensive hardware or hire specialized personnel to configure and operate the
system.

###  3.2 Easy-to-use User-Interface

In order to enable users with limited technical expertise to use the platform,
it is important to ensure that the user experience is optimized. The platform
should have an intuitive user interface that is easy to navigate. With just a
few interactions, users should be able to train an AI model based on their own
data without needing any expert knowledge of AI or complex technical terms. By
abstracting away the complexities of AI, the platform can become more
accessible to a wider range of users, regardless of their level of technical
expertise.

###  3.3 Scalability

As there are many areas where the potential of AI has not yet been fully
exploited, a platform offering AI as a service could have a vast user base,
making scalability essential. With the increasing amount of data being
generated and stored, the platform must also be capable of handling large
amounts of data. Additionally, the complexity of AI models is increasing,
which requires more computing power for training, so the platform must be able
to handle increasingly complex AI algorithms. In summary, the platform should
be scalable in terms of its user base, data handling capacity, and ability to
handle complex AI models.

###  3.4 Adaptability and Extensibility

Beyond the requirement for scalability in handling more complex AI algorithms,
there is another important aspect to consider: the platform’s adaptability and
extensibility. This is crucial because the field of AI is constantly evolving,
with ongoing research and new advancements being made. If the platform is
designed in a way that enables easy integration of these new developments, it
can benefit a wider range of users. By allowing for customization and
flexibility in the development of new AI models and algorithms, the platform
can continue to evolve alongside the latest advancements in AI, ensuring that
users have access to the most up-to-date and relevant tools and technologies.
It is important to note that this adaptability and extensibility need not
necessarily be achieved through the user interface alone. As AI experts are
required to implement and integrate these changes, they can be made directly
at the code level.

###  3.5 Openness

Having open-source code is therefore a requirement for the platform so that
these changes can be made by as many AI experts as possible. Generally, public
access to the platform’s code offers further advantages. Firstly, it increases
trust in the platform, as independent experts can verify the internal workings
of the platform. Secondly, the platform also benefits from an open-source
community around it. This way, it can be developed faster, more efficiently,
and cost-effectively, and users can help each other with questions and
problems.

###  3.6 Self-Hosting

Furthermore, in addition to the publicly available instance of the platform,
it may be important for some users to be able to deploy their own instance.
This allows for greater security considerations, as access can be heavily
restricted. So, the platform can also be used for data that requires a higher
level of security, expanding the use of AI in such areas. This is also another
argument for making the code of the platform open-source, as it enables the
tracking of what precisely occurs with the uploaded data.

##  4 Technologies

Now that we have identified the necessary requirements for a platform that
offers AI as a service, we will take a look at the technologies that we will
use to implement our own platform.

###  4.1 Kubernetes

In order to address the significant scaling requirements of our platform, we
will be utilizing the open-source container orchestration framework Kubernetes
[9]. By deploying Kubernetes in the cloud, we can take advantage of the auto-
scaling mechanisms provided by cloud providers.

Kubernetes provides a high level of abstraction by organizing available
computers as nodes and grouping them into a Kubernetes cluster. The platform’s
Control Plane API enables users to manage the state of their containers, while
the Control Plane itself distributes the workload in the form of pods to the
nodes. Pods are the smallest unit in Kubernetes and consist of one or more
containers that run on the nodes [10]. This approach allows us to efficiently
manage the scaling of our platform and ensure optimal resource utilization.

###  4.2 Service Mesh

A service mesh is a software infrastructure that enables managing and
operating microservices-based applications. It provides a range of benefits,
such as improved scalability, reliability, and security. By using a service
mesh, application developers and operators can reduce the complexity of their
architecture while improving flexibility and agility. A service mesh allows
developers to focus on application development without having to worry about
the underlying infrastructure, while providing operators with better control
over application traffic and security.

One of the main advantages of using a service mesh is its ability to provide
advanced traffic management capabilities. With a service mesh, operators can
manage and control the flow of traffic between services, allowing them to
balance the load and ensure optimal performance. Additionally, a service mesh
can also provide improved security features such as authentication and
authorization, allowing only authorized traffic to pass through.

Istio [11] is a popular open-source service mesh that provides a variety of
features such as load balancing, traffic management, security features like
authentication and authorization, as well as troubleshooting and monitoring.
By using Istio, developers and operators can reduce the complexity of
microservices-based applications by managing traffic between services within
the application. Overall, Istio is a powerful and flexible service mesh that
helps developers and operators operate microservices-based applications more
effectively and securely.

###  4.3 Kubeflow Pipelines

While simple microservices can be deployed as native Kubernetes deployments,
our machine learning tasks require special treatment due to their need for
significant computing power and specialized hardware. To address this, we will
be using a workflow engine, specifically Kubeflow Pipelines [12], which is
built on top of Argo Workflow [13] - a workflow engine designed specifically
for Kubernetes environments. With Argo Workflow, users can easily create,
manage, and run complex workflows, including creating workflow templates and
executing them with different parameters.

Kubeflow Pipelines provides additional ML functionality such as an artifact
store for data and models, visualizations for executions and metrics, and
robust features that increase the reliability of our workflows, such as
automatic retries and error handling. By using Kubeflow, we can ensure that
our ML workflows can handle errors and unexpected events gracefully and
achieve greater efficiency and reliability in our development process.

Kubeflow Pipelines is a component of the Kubeflow project, which is designed
to facilitate the development, deployment, and management of machine learning
models on Kubernetes. In addition to the pipelines component, it includes
support for Jupyter Notebooks to explore data and an interface to TensorFlow
to train and serve models. Kubeflow also offers model versioning and
monitoring, enabling users to keep track of their models’ performance over
time.

###  4.4 Ludwig

For our No-Code AI component, we will make use of Ludwig [3], a declarative ML
framework [7] that creates and trains ML models using a configuration file as
input. This configuration file specifies the input and target data types, as
well as other parameters needed for training, such as the optimizer, the loss
function, and the number of epochs. Ludwig’s declarative approach eliminates
the need to write low-level code, making it ideal for non-technical users who
want to build and train ML models easily.

This can best be demonstrated through an example, using the PetFinder dataset
[14]. This dataset contains information about animals in a shelter, such as
their name, age, or an image of the animal. The goal is to predict how quickly
these animals will be adopted, which is represented as categorical values in
the dataset. An AdoptionSpeed of 4, for example, indicates that the animal has
not been adopted within 100 days. An example is shown in the following figure
1, and a Ludwig configuration for this task is shown in the next figure 2. The
configuration file defines the input and output columns and sets some
additional training parameters.

Figure 1: Example of a pet from the PetFinder dataset [14]. Fenny is a 5-year-
old dog (Type 1) that has not been adopted within 100 days (AdoptionSpeed 4)
Figure 2: An example Ludwig configuration for the PetFinder dataset [14]. The
input and output features are listed, along with some additional training
information.

By providing intuitive selection options in a user interface, we eliminate the
need for manual writing of the configuration file. Additionally, we remove
additional training parameters from the configuration process in favor of
employing techniques of hyperparameter optimization. This approach expands the
search space for AI models, reducing the potential for bias caused by an
expert’s experience. Using a user interface, we can thus expand this low-code
ML tool into a no-code tool. This enables us to allow users with no technical
experience to interact with our platform.

##  5 Architecture

We will now explain how we combine various technologies to create our AI-as-a-
Service platform called Open Space for Machine Learning (Os4ML). Figure 3
illustrates the architecture of our platform.

Figure 3: Our architecture centers around the API Gateway, which serves as the
backbone of our platform by securely granting access to our microservices,
including the Os4ML Service, the Workflow Service, and the Objectstore
Service.

Our platform is designed using microservices as its architectural pattern. We
have structured these microservices into a mesh topology that is accessible
through an API gateway, which we have implemented using Istio. This
architecture provides a flexible and scalable platform that enables developers
to access different microservices conveniently. The frontend and other
components can easily access the microservices through the API gateway, which
simplifies development and maintenance tasks.

One of the most crucial microservices in our platform is the Os4ml-Service. It
is responsible for managing our domain models and storing critical information
such as metrics and execution times. This service plays a vital role in
providing efficient and reliable machine learning solutions.

In addition, we have implemented the objectstore service, which offers a
dependable way to store and retrieve models and data. Furthermore, we have the
workflow service, which handles communication with the workflow engine. For
our platform, we use Kubeflow Pipelines as our workflow engine. Both the
workflow engine and the objectstore service can share the same objectstore,
which ensures consistency and reliability in our platform.

Our frontend allows the user to upload data. In the background, the data is
transformed into an interface that is compatible with Ludwig (e. g., into
Pandas dataframes), with data types detected and saved. We provide templates
for the workflow engine to create a Ludwig configuration and train a machine
learning model using the dataframe. These templates can be quickly and easily
customized to create tailored machine learning models. Our platform thus
provides a user-friendly and effective way to train and deploy machine
learning models.

To ensure scalability, we deploy the entire system on Kubernetes, which
provides a robust and reliable platform for container orchestration. With
Kubernetes, our system can handle large volumes of data and easily adapt to
changes in demand. By breaking down the system into smaller, independent
components, we can maintain better control and reduce the risk of system-wide
failures. This architecture enables a flexible and scalable system that can
efficiently handle complex machine learning workloads.

##  6 Deployment

The requirements section has highlighted the need to allow the advanced users
to deploy the platform themselves. To achieve this, we have adopted a three-
part Infrastructure-as-Code approach. In general, Infrastructure-as-Code
involves provisioning the necessary infrastructure using code so that it can
be reproduced or restored in a predictable manner. To implement this, we use
the open-source tool Terraform [15]. Our deployment process is divided into
three steps: cloud setup, Kubernetes cluster setup, and the actual deployment
of our platform. If steps 1 and/or 2 are not required because they already
exist, the user can skip directly to step 3.

We will be publishing the Infrastructure-as-Code scripts along with our
application code. This means that deploying the platform will only require a
single command, in our case, ’terraform apply’. This will greatly reduce the
time and effort required for the deployment process, making it more efficient
and user-friendly. Additionally, this approach ensures consistency in the
infrastructure setup, reducing the possibility of errors caused by manual
configuration.

In order to realize step 3 of our deployment process, we utilize the
declarative GitOps continuous delivery tool ArgoCD [16]. This provides the
administrators of individual instances with access to updates to the platform
as well as a pre-packaged monitoring solution. By utilizing ArgoCD, we can
automate the deployment process, allowing for seamless updates and streamlined
management of our platform. Additionally, the use of declarative
configurations ensures that our infrastructure remains consistent across all
environments, reducing the risk of configuration drift and ensuring high
reliability.

##  7 Accomplishments and Contributions

We are thrilled to report significant progress made on our AI-as-a-Service
platform, ”Open Space for Machine Learning”777https://www.os4ml.com/, and are
eager to share the goals we’ve achieved and contributions we’ve made.

###  7.1 Transforming Ludwig from Low-Code to No-Code

Our most noteworthy accomplishment is the creation of a user-friendly frontend
that has transformed Ludwig, our low-code ML tool, into a no-code tool. With
our efforts, we have opened up the platform to a broader range of users,
including those with little to no experience in machine learning, who can now
effortlessly train and deploy AI models without the need to write any code.
Our innovation has revolutionized the industry, making machine learning
accessible to anyone.

###  7.2 Integrating Multiple Open-Source Tools

Another proud achievement is our integration of several powerful open-source
tools, such as Istio, Kubernetes, and Kubeflow, into our platform, creating a
blueprint for highly scalable AI applications. This has opened up new avenues
for exploring the potential of AI.

###  7.3 Cloud-Enabled Platform with Self-Hosting Option

Furthermore, we have made our platform cloud-agnostic and easy to deploy for
the community. This approach allows us to take advantage of the benefits of
cloud computing, such as scalability and cost-effectiveness, while still
providing the option for users to self-host our platform on-premises. By
enabling easy deployment, we have made it simpler for the community to take
advantage of our AI-as-a-Service platform, including those dealing with
sensitive data.

Overall, we are proud of our contributions to the AI community and look
forward to continuing to push the boundaries of what is possible in the world
of machine learning.

##  8 Summary & Outlook

In this article, we have demonstrated that the next step in AI democratization
is the provision of AI as a service through a platform. However, current
solutions have not been satisfactory, leading us to collect requirements for
such a platform. We have shown how we use existing open-source solutions and
add our own components to meet the collected requirements. Our platform is
also available as open source888https://github.com/WOGRA-AG/Os4ML, enabling
further collaboration and development towards more accessible AI solutions.
Now that the foundational work is complete, the next step is to enhance the AI
component. One area of improvement would be to incorporate an explainable AI
mechanism. This would provide valuable insights into the trained model’s
decision-making process, thereby increasing trust and transparency.

## Acknowledgments

Os4ML is a project of the WOGRA AG research group in cooperation with the
German Aerospace Center and is funded by the Ministry of Economic Affairs,
Regional Development, and Energy as part of the High Tech Agenda of the Free
State of Bavaria.

## References

  1. 1.

Jan Juricek. 2014. The Use of Artificial Intelligence in Building Automated
Trading Systems. _International Journal of Computer Theory and Engineering_ 6,
(January 2014), 326–329.DOI: https://doi.org/10.7763/IJCTE.2014.V6.883

  2. 2.

Jamilu Awwalu, Ali Garba, Anahita Ghazvini, and Rose Atuah. 2015. Artificial
Intelligence in Personalized Medicine Application of AI Algorithms in Solving
Personalized Medicine Problems. _International Journal of Computer Theory and
Engineering_ 7, (December 2015), 439–443.
DOI:https://doi.org/10.7763/IJCTE.2015.V7.999

  3. 3.

Piero Molino, Yaroslav Dudin, and Sai Sumanth Miryala. 2019. _Ludwig: a type-
based declarative deep learning toolbox_.

  4. 4.

Jeremy Howard and Sylvain Gugger. 2020. Fastai: A Layered API for Deep
Learning. _Information_ 11, 2 (2020). DOI:https://doi.org/10.3390/info11020108

  5. 5.

Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu
Li, and Alexander Smola. 2020. _AutoGluon-Tabular: Robust and Accurate AutoML
for Structured Data._

  6. 6.

Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alexander J. Smola.
2021. _Benchmarking Multimodal AutoML for Tabular Data with Text Fields_.

  7. 7.

Piero Molino and Christopher Ré. 2021. _Declarative Machine Learning Systems_.

  8. 8.

H2O.ai. Retrieved March 29, 2023 from https://github.com/h2oai/h2o-3

  9. 9.

Kubernetes Components. Retrieved March 29, 2023 from
https://kubernetes.io/docs/concepts/overview/components/

  10. 10.

Shazibul Islam Shamim, Jonathan Alexander Gibson, Patrick Morrison, and Akond
Rahman. 2022. _Benefits, Challenges, and Research Topics: A Multi-vocal
Literature Review of Kubernetes_.

  11. 11.

The Istio service mesh. Retrieved March 29, 2023 from
https://istio.io/latest/about/service-mesh/

  12. 12.

Kubeflow Pipelines. Retrieved March 29, 2023 from
https://www.kubeflow.org/docs/components/pipelines/

  13. 13.

Argo Workflows. Retrieved March 29, 2023 from https://argoproj.github.io/argo-
workflows/

  14. 14.

Sherine Zhang and K. X. Zhang. 2019. PetFinder Challenge: Predicting Pet
Adoption Speed.

  15. 15.

Terraform. Retrieved March 29, 2023 from
https://developer.hashicorp.com/terraform/docs

  16. 16.

ArgoCD. Retrieved March 29, 2023 from https://argo-
cd.readthedocs.io/en/stable/


# Vision-driven Automated Mobile GUI Testing via Multimodal Large Language
Model

Zhe Liu1,2, Cheng Li1,2, Chunyang Chen3, Junjie Wang1,2,∗, Boyu Wu2,4, Yawen
Wang1,2,
Jun Hu1,2, Qing Wang1,2,5,∗
1State Key Laboratory of Intelligent Game, Beijing, China;
Institute of Software Chinese Academy of Sciences, Beijing, China;
2University of Chinese Academy of Sciences, Beijing, China; ∗Corresponding
author;
3Technical University of Munich, Munich, Germany; 4DiDi Global Inc;
5Science & Technology on Integrated Information System Laboratory;
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

With the advancement of software rendering techniques, GUI pages in mobile
apps now encompass a wealth of visual information, where the visual semantics
of each page contribute to the overall app logic, presenting new challenges to
software testing. Despite the progress in automated Graphical User Interface
(GUI) testing, the absence of testing oracles has constrained its efficacy to
identify only crash bugs with evident abnormal signals. Nonetheless, there are
still a considerable number of non-crash bugs, ranging from unexpected
behaviors to misalignments, often evading detection by existing techniques.
While these bugs can exhibit visual cues that serve as potential testing
oracles, they often entail a sequence of screenshots, and detecting them
necessitates an understanding of the operational logic among GUI page
transitions, which is challenging traditional techniques. Considering the
remarkable performance of Multimodal Large Language Models (MLLM) in visual
and language understanding, this paper proposes a vision-driven automated GUI
testing approach VisionDroid to detect non-crash functional bugs with MLLM. It
begins by extracting GUI text information and aligning it with screenshots to
form a vision prompt, enabling MLLM to understand GUI context. The function-
aware explorer then employs MLLM for deeper and function-oriented GUI page
exploration, while the logic-aware bug detector segments the entire
exploration history into logically cohesive parts and prompts the MLLM for bug
detection. We evaluate VisionDroid on three datasets and compare it with 10
baselines, demonstrating its excellent performance. The ablation study further
proves the contribution of each module. Moreover, VisionDroid identifies 29
new bugs on Google Play, of which 19 have been confirmed and fixed.

###### Index Terms:

Automated GUI testing, Android app

##  I Introduction

In an era where mobile applications are integral to everyday life, ensuring
the quality and reliability of these applications is paramount [1]. The field
of mobile app development is characterized by rapid iterations and the need
for continuous integration and deployment, making the quality assurance
process both critical and challenging. As one kind of acceptance testing,
automated Graphical User Interface (GUI) testing [2, 3, 4, 5, 6, 7, 8, 9, 10]
has become a cornerstone in this process, promising to streamline testing by
systematically identifying issues that could degrade user experience.

Despite the advancements in automated testing techniques, the intricate nature
of mobile interfaces, coupled with the variability of user interactions, often
necessitates the human testers as the last line of defense in ensuring app
quality. Besides severe app crashes with obvious oracle signals, many bugs
present visual cues such as misalignments, unintended element overlaps, or
deviations from expected results which current automated systems frequently
overlook [11]. As demonstrated in Fig. 1, the first sub-figure exposes a bug
where the total income (5,030) does not match the summarization of the
detailed records. Additionally, the four sub-figures together unveil a bug
related to the failure of adding a record, as the income-expense overview in
the fourth sub-figure should display five records after adding a new one. Our
empirical study in Section II identifies nearly nine categories of such bugs,
indicating the diversity of these non-crash bugs with visual cues. Although
these bugs exhibit visual cues that make visual detection possible, they often
entail a sequence of screenshots (as demonstrated in Fig. 1), requiring the
understanding of the overall operational logic among these GUI page
transitions. Only human testers with both sharp eyes and knowledge from
extensive training can identify those bugs.

Figure 1: Example of bugs with visual cues

However, human testing is time-consuming and suffers from inherent scalability
bugs. Although researchers begin to explore detecting non-crash or functional
bugs with visual information, most of them are typically designed to identify
specific types of bugs, such as display issues [12, 13], scaling problems [14,
15], game GUI glitches [16] or WebView bugs [17]. What makes it worse is that
all of them require either a set of rules that are easy to be out of date, or
a large amount of training dataset which takes much human effort for
annotation. This specialization leaves a gap in the testing landscape for a
more generalized, lightweight and intelligent approach that can adapt to the
diverse challenges of mobile app testing especially for non-crash bugs.

Thanks to the emerging Multimodal Large Language Model (MLLM) [18], it is
trained on the ultra-large-scale corpus and potentially able to understand the
GUI pages as well as the information changes along with the page transition.
Inspired by this, we propose a vision-driven automated GUI testing approach
with MLLM. Nevertheless, three challenges need to be addressed. First, solely
feeding the raw screenshot would be far less effective in letting the MLLM
understand the GUI context, therefore how to align other sources of
information and produce an effective vision prompt. Second, the triggering of
bugs largely depends on whether and how the corresponding GUI page can be
reached, therefore how to guide the MLLM to plan the exploration to delve
deeper, covering diverse scenarios, and focusing on the app’s functionalities.
Third, detecting the functional bugs requires tremendous domain knowledge and
MLLM might lack it, therefore how to provide accurate information to
facilitate MLLM making decisions.

To tackle these challenges, we propose VisionDroid, a vison-driven approach
for detecting non-crash functional bugs with MLLM. It begins by extracting GUI
text information and aligning it with GUI screenshots to form a vision prompt,
enabling MLLM to understand GUI context. Second, the function-aware explorer
utilizes the MLLM for GUI page exploration. It maintains a testing history
comprising both the higher-level catalogue of testing progress and a detailed
snapshot of each performed action, to enhance the exploration to be deeper and
function-oriented. Third, the logic-aware bug detector breaks the entire
exploration sequence into logically cohesive parts that mirror the app’s
logical operational sequences. Based on them, the MLLM detects the bugs
potentially by checking whether the GUI information changes between page
transitions align with the processing logic. To boost the performance, we
leverage in-context learning schema to provide useful examples when querying
the MLLM, from online bug reports and history running records.

To evaluate the effectiveness of VisionDroid, we experiment on three datasets.
Compared with 10 common-used and state-of-the-art baselines, VisionDroid can
achieve more than 8%-230% and 27%-43% boost in average recall and precision
compared with the best baseline, resulting in 50%-76% precision and 42%-64%
recall. To further understand the role of each sub-module of the approach, we
conduct ablation experiments to further demonstrate its effectiveness. We also
evaluate the usefulness of VisionDroid by detecting unseen non-crash bugs in
real-world apps from Google Play. Among 105 apps, VisionDroid detects 29 new
crash bugs with 19 of them being confirmed and fixed by developers, while the
remaining are still pending.

The contributions of this paper are as follows:

  * •

Vision. The pioneering work to formulate the non-crash functional bugs
detection problem to an interactive multimodal question & answering task,
enabling the MLLM to identify bugs by comprehending GUI visual information and
functional logic.

  * •

Technique. A vision-driven automated GUI testing approach VisionDroid111We
release the source code, dataset, and detailed experimental results on our
website https://github.com/testtestA6/VisionDroid.  which designs a function-
aware explorer and logic-aware bug detector for detecting non-crash functional
bugs via MLLM.

  * •

Evaluation. Effectiveness and usefulness evaluation of the VisionDroid in the
real-world apps with practical bugs detected (Section V-A and V-C).

##  II motivational study

To better understand the non-crash functional bugs in real-world practice, we
carry out a pilot study to examine their prevalence. It also explores what
categories of non-crash functional bugs, to facilitate the design of our
approach.

###  II-A Data Collection

Our experimental dataset is collected from one of the largest open-source
Android app hosting websites (GitHub) [19], which has a great number of
Android bug reports. We employ the following criteria for app selection: more
than 1K downloads in Google Play (popular), a public issue tracking system
(traceable), and more than three years of development history (trustworthy). A
total of 3,128 open-source Android apps with 57,326 bug reports are selected
in this study. These apps belong to diversified categories such as business,
communication, finance, etc. For the selected apps, we randomly select 6,000
bug reports involving 576 apps with detailed bug descriptions and
GIFs/screenshots, and conduct manual analysis. Among them, 4,856 (81%) bug
reports are related to the functional aspects of the app, while the rest are
about the performance, security, etc. These functional bugs include 1,796
crash bugs (37%), and 3,060 non-crash functional bugs (63%). Since the crash
bugs have been widely studied in previous work, we further investigate the
non-crash functional bugs.

Figure 2: The categories of non-crash functional bugs Figure 3: Overview of
VisionDroid

###  II-B The Categories of Non-crash Functional Bugs

For the above-mentioned 3,060 non-crash functional bug reports, we manually
classify them based on bug descriptions and reproduction steps, and derive the
categories of non-crash functional bugs to attain a comprehensive
understanding of them. Following the open coding protocol [20], two authors
with more than three years of development and testing experience individually
examine the issue description and reproduction steps. Then each annotator
iteratively merges similar codes, and any disagreement of the categorization
will be handed over to the third researcher for double-checking. Finally, we
come out with a categorization of the non-crash functional bugs within (intra-
page) and among the pages (inter-page), with details summarized in Fig. 2. We
additionally provide examples for each category in our website1.

Intra-page non-crash functional bug. Intra-page non-crash functional bug
denotes these bugs only related to a single GUI page, mainly involving the
issues about information display as shown in Fig. 2.

Inter-page non-crash bug. Inter-page non-crash functional bug denotes these
bugs related to multiple GUI pages, i.e., involving a testing sequence to
trigger it, with details in Fig. 2. Among them, data operation failures mainly
manifested as failures in adding, deleting, and modifying data, such as failed
user information updates, errors in adding or removing shopping cart items,
etc. Media control no-response mainly involves issues with video and audio
playback, such as no response of playback control (failure of play, pause,
stop functions). Numerical calculation errors mainly include mathematical
errors, inaccurate statistical data, and cost calculation deviations. Setting
and configuration failures mainly manifested as users being unable to save
personalized settings or configurations, including ineffective or reset
settings within the app (such as notification switches and theme changes).
Navigation and link errors mainly manifested as page redirection errors and
confusion in navigation logic between functions.

Findings. The above analysis confirms the popularity of non-crash bugs in
mobile applications, implying the need for comprehensive testing. Meanwhile,
considering the diversity of application functionality and bug manifestation,
manually designing bug detection rules and building a complete set of defect
methods require significant effort, and the automated technique is highly
demanded. Despite their complexity, we have observed that these non-crash
functional bugs typically manifest visual cues, either within a single GUI
page or across a sequence of GUI pages. This motivates us to apply visual-
driven techniques for bug detection.

##  III Approach

This paper proposes VisionDroid, a vision-driven automated GUI testing
approach for detecting non-crash functional bugs with the Multimodal Large
Language Model (MLLM). As shown in Fig. 3, it begins by extracting GUI text
information, and aligning it with GUI screenshots, forming the vision prompt
for GUI context perception. Second, the function-aware explorer utilizes the
MLLM for guiding the exploration to delve deeper, encompassing more diverse
pages, and focusing on the app’s functionalities. This is facilitated through
maintaining the testing history comprising both the higher-level catalog of
testing progress and the detailed snapshot of each performed action. Third,
the logic-aware bug detector breaks the entire exploration sequence into
locally cohesive parts that mirror the app’s logical operational sequences.
Based on them, the MLLM detects the bugs potentially by checking whether the
GUI information changes between page transitions align with the processing
logic. To further boost the performance, we leverage the in-context learning
schema to provide useful examples when querying the MLLM, from online issue
reports and history running records.

###  III-A Alignment between Text and Image

Existing automated GUI testing approaches typically rely on view hierarchy
files to obtain the attribute information (text, coordinates, actions, etc.)
of app widgets to capture the current context and determine which widgets
should be acted upon. This could be less efficient since the conversion from
screenshot image to text could potentially lose information about the
structure semantics of the GUI. Thanks to the MLLM, we can directly feed the
screenshot image as input. Yet solely based on the screenshot also has its
limitations since the image tends to utilize icons and extremely concise text
for conveying information, which can also hinder the MLLM’s perceptions.
Therefore, we design the visual and text alignment method, aiming at enhancing
the MLLM’s understanding of the semantics of the GUI page.

####  III-A1 GUI Text Information Extraction

We extract three types of information, i.e., the app, the current GUI page,
and all its contained widgets. The app information is extracted from the
AndroidManifest.xml file, while the other two types of information are
extracted from the view hierarchy file of the current GUI page, which can be
obtained by UIAutomator [21]. We first extract the app information including
the app name, and the activity names in this app , which can facilitate the
MLLM to gain a general perspective of the functions. Then, we extract the page
information from the view hierarchy file and extract the widgets information
including “text” field, “hint-text” field, “resource-id” field, “class” field,
“clickable” field, “long-clickable” field, “checkable” field, “scroll” field
and “bounds” field, which facilitates the MLLM in suggesting actions related
to these widgets.

Figure 4: Example of alignment text with image

####  III-A2 Text-Image Alignment for Visual Prompt

We annotate the GUI pages to guide the LLM’s attention towards the actionable
parts (i.e., widgets). as shown in the right part of Fig. 4. Following
existing studies [6, 7, 8, 9, 10], we support five types of basic operational
actions, i.e., click, input, long-click, check, and scroll. We draw bounding
boxes for each of these operational widgets, which are based on the extracted
widget coordinates from the property of the “bounds” field, and use different
colors indicating different types. In detail, if the “clickable” property of a
widget is “true”, we draw a red bounding box to indicate it can be clicked,
and if the “class” property is “EditText”, we draw a blue bounding box to
indicate it needs input text. Similarly, we respectively use the purple, pink
and yellow bounding boxes to indicate the widget with “long-clickable”,
“checkable”, and “scrollable” properties as “true”. Inspired by the screen
reader [22], we reconstruct the coordinate system in order from top to bottom
and from left to right, and label each action widget on the image using Arabic
numerals starting from 1. To mitigate the crowded appearance resulting from
the abundance of components on a GUI page, we only mark the first widget of
each row and let the MLLM infer others as shown in the prompt on the right
side of Fig 4.

We then associate each annotated widget in the screenshot with the extracted
GUI text information. Note that, in the “prompts related to GUI text and
visual information” (details in Section III-B2 and Fig. 4), we would not
provide the detailed alignment information considering the input length
constraints of the MLLM. Instead, this alignment information will be used in
“prompts related to action & feedback” (details in Section III-B2 and Fig. 5)
to help verify whether the MLLM is suggesting interactive widgets in
accordance with the correct logical inference process, akin to an explicit
chain-of-thought mechanism. In detail, when querying the general action or
text input, we not only request the MLLM to provide the widget ID from the
screenshot, but we also ask for the corresponding widget text. When these two
sources of information do not align, we infer potential errors in the MLLM’s
decision-making process and prompt it to re-try.

###  III-B Function-aware Explorer

The non-crash functional bugs are often linked to the functionalities of the
app, which often require lengthy operational sequences to trigger.
Consequently, we design the function-aware explorer to guide the exploration
of the app’s functionalities, while also encouraging deeper dives and broader
coverage of GUI pages.

Figure 5: Example of function-based history

####  III-B1 Function-based Testing History Construction

To facilitate exploration of the app’s functionality, it is important to know
the testing process, as the functions are usually accomplished by following a
sequence of GUI pages. We maintain a complete testing history database,
including textual descriptions and screenshots, and offer two views supporting
the follow-up exploration and bug detection, as demonstrated in Fig. 5.

The TEXT view depicts the higher-level catalog of testing progress, organized
according to the functions. It includes a list of tested functions along with
their visit counts, which can assist the MLLM in avoiding duplicate
exploration and uncovering untouched functions. To accomplish this, during the
testing process, the MLLM is prompted to infer what function is currently
being tested through the “prompt of function inquiry”, as demonstrated in Fig
5 (b). This enables us to gather statistics on the tested functions

The IMAGE view offers a detailed snapshot of the testing records through
screenshots, providing a granular view of the testing process. Specifically,
it includes a list of detailed GUI pages, with each page marked with a widget
(or a group of widgets), i.e., denoting the corresponding actions when
exploring that GUI page, which is demonstrated in Fig. 5 (a). Additionally, we
provide derived functions for each screenshot to further ensure alignment
between visual and textual information. We also associate a unique identifier
(ID) with the function name, representing different pages/steps within the
same function.

Figure 6: Example of explorer prompt

####  III-B2 Explorer Prompt Generation

With the static GUI information (in Section III-A1) and the dynamic testing
progress (in Section III-B1), we devise prompts for inputting into the MLLM.

Prompts related to GUI text and visual information are demonstrated in Fig. 6
with “GUI text info” and “legend of current GUI screenshots”, offering the
MLLM an overview of the current GUI page.

Prompts related to action & feedback are illustrated in Fig. 6 with “querying
general action”, “querying text input”, and “testing feedback”. We prompt the
MLLM to identify what operation is required, and provide the output template
to assist the MLLM in generating a desired executive command for app testing.
It’s worth noting that we separate the patterns for querying general action
(e.g., click) and text input (e.g., input certain text) respectively, to
enable the generated commands to better match the widgets in the GUI page. For
the feedback question, if we detect the widget ID does not correspond to its
textual description, which is determined with the help of the alignment
information in Section III-A, we inform the MLLM that there might be an error
in its decision-making process and prompt it to retry. ‘widget number’ is not
accurate, better to use ‘widget id’, and revise them in other places and the
figures.

Prompts related to function-based history are depicted in Fig. 6 with
“explored functions” and “legend of recently tested screenshots”. They are
fetched from the testing history, where the former is directly obtained from
the text view, and the latter is extracted from the image view, displaying the
latest four tested GUI screenshots to provide the most relevant clues.

Prompts related to function inquiry are illustrated in Fig. 6 with “querying
function”, prompting the MLLM to identify what function is currently tested,
and the prompt also provides an output template to assist the MLLM in
generating the function name and its status.

Figure 7: Example of sequence segmentation and bugs detected by VisionDroid

###  III-C Logic-aware Bug Detector

During the exploration process, intra-page non-crash bugs are directly
detected using MLLM, triggered by the “bug detect” prompt shown in Fig. 6.
This section will focus on describing the bug detector for inter-page non-
crash bugs.

Once the exploration phase is complete, we leverage the entire exploration
history for bug detection. However, the overall history is often too lengthy,
intricate, and noisy for directly querying the MLLM. The direct manner is to
organize the entire history into individual parts based on the functions
inferred by MLLM during the exploration process. Yet the MLLM tends to yield
relatively granular function names, often comprising only 2 to 3 operational
steps. However, many functional bugs are typically associated with the
business logic formed by such consecutive functions. For instance, as depicted
in Fig. 7, a bug involves functions like “edit expense”, “delete a budget”,
“check budget”, etc. To tackle this, we propose a logic-driven testing
sequence segmentation method to partition the entire exploration history into
logically cohesive parts.

####  III-C1 Logic-driven Testing Sequence Segmentation Algorithm

The principle of test sequence segmentation is that GUI pages with similar
operational logic are considered as a sub-sequence, while GUI pages with
different logic are considered as different sub-sequences. Based on this, we
begin by constructing a graph based on the entire testing history. Each node
represents an explored GUI page (i.e., a screenshot from the image view in the
testing history as shown in Fig. 5). Each edge signifies the transition
between nodes. We assign the edge weights based on the semantic similarity of
their associated function names (i.e., the text below each screenshot as shown
in Fig. 5 (b)). Specifically, we segment the textual function name into
tokens, and calculate the cosine similarity between them. In this way, the
edge weight between pairs of pages with completely different functions should
be 0, e.g., Check budget-2 and Setting-1 in Fig. 7.

Based on the weighted directed graph, the logic-driven testing sequence
segmentation algorithm is conducted with the Louvain algorithm [23] and
optimized based on modularity $Q$ for sub-sequence partitioning.

$Q=\frac{1}{2m}\sum_{ij}\left[A_{ij}-\frac{1}{2m}\right]\delta(c_{i},c_{j})$

$A_{ij}$ represents the weight between node $i$ and node $j$. $m$ is the sum
of the weights of the graph. $c_{i}$ and $c_{j}$ are the sub-sequence number
for node $i$ and node $j$. $\delta$ is an indicator function, when
$c_{i}=c_{j}$ the value is 1, otherwise, it is 0, used to determine whether
two nodes belong to the same sub-sequence.

Figure 8: Example of bug detection prompt

####  III-C2 Detector Prompt Generation

With the segmented results, we proceed to feed each sub-sequence into the MLLM
for bug detection. Specifically, we provide MLLM with the related screenshots
in the form of IMAGE view as shown in Fig. 5.

Prompts related to testing sub-sequence are demonstrated in Fig. 8 with “APP
info” and “legend of GUI screenshots with function name”, offering the MLLM
with app’s overview and the details about the derived testing sub-sequence.

Prompts related to bug detection question are demonstrated in Fig. 8. We
enable the MLLM to comprehend the semantics of the test sequence based on the
provided information and assess the correctness of the test path, further
inquiring if there are any bugs in the test sequence and identifying which
step contains bugs.

###  III-D Enriching Detector Prompt with Examples

It is usually difficult for MLLM to perform as well on domain-specific tasks
as ours, and common practice would be employing the in-context learning schema
to boost the performance. It provides the MLLM with examples to demonstrate
what the instruction is, which enables the MLLM to better understand the task.
Following the schema, along with the prompt for the detector, as described in
Section III-C, we additionally provide the MLLM with examples of the non-crash
logical bugs.

To achieve this, we first build a basic example dataset of non-crash bugs from
the issue reports of open-source mobile apps and continuously enlarge it. For
each data instance, it records the bug description, bug screenshot and natural
language described bug reproduction path which facilitates the MLLM
understanding of what the non-crash bug is. We select 200 bugs (100 intra-
page, 100 inter-page) with the highest quality and complete information as the
example data. We also enrich the example dataset with the newly emerged non-
crash bugs which truly trigger bugs during VisionDroid runs on various apps.

Although examples can provide intuitive guidance to the LLM in accomplishing a
task, excessive examples might mislead the LLM and cause the performance to
decline. Therefore, we design a retrieval-based example selection method to
choose the most suitable examples (i.e., most similar to the tested GUI page)
for MLLM. The similarity comparison is based on the activity name of the
tested page. Similar to previous works [24], We use Word2Vec [25] to encode
the context information of each activity into a sentence embedding, and
calculate the cosine similarity between the activity and each data instance in
the example dataset. We choose the top-K data instance with the highest
similarity score, and set K as 5.

###  III-E Implementation

We implement VisionDroid based on the gpt-4 API which is released on the
OpenAI website222https://beta.openai.com/docs/models/GPT-4. It obtains the
view hierarchy file of the current GUI page through UIAutomator [21] to
extract text information of the input widgets. VisionDroid is implemented as a
fully automated GUI testing tool, which uses or extends the following tools:
VirtualBox and the Python pyvbox [26] for running and Android Debug Bridge
(ADB) [27] for interacting with the app under test.

##  IV Experiment Design

  * •

RQ1: (Coverage and bugs detection performance) How effective of VisionDroid in
testing coverage and bug detection?

For RQ1, we first present some general views of VisionDroid for coverage of
the explorer and bug detection of the detector. Then we compare VisionDroid
with commonly-used and state-of-the-art baseline approaches.

  * •

RQ2: (Ablation Study) What is the contribution of the (sub-) modules of
VisionDroid for coverage and bug detection performance?

For RQ2, We conduct ablation experiments to evaluate the impact of each (sub-)
module on the performance.

  * •

RQ3: (Usefulness Evaluation) How does our proposed VisionDroid work in real-
world situations?

For RQ3, we use VisionDroid to automatically explore the app, and detect
unseen non-crash functional bugs, and issue the detected bugs to the
development team.

###  IV-A Datasets

For RQ1-RQ2, the experimental dataset comes from three sources. The first is
from the Odin and RegDroid datasets [28, 29] (We name them baseline dataset),
which contains 131 non-crash functional bugs that can be reproduced in our
experimental environment.

Considering the small number of apps in the baseline dataset, we collect a
second dataset (Github dataset) following similar procedures as the baseline
dataset. In detail, we crawl the 30 most popular apps of each category from
Google Play [30], and we keep the ones with at least one update after May.
2024, resulting in 218 apps in 10 Google Play categories. Then we manually
review each issue report and app, and filter them according to the following
criteria: (1) the app wouldn’t constantly crash on the emulator; (2) it can
run all baselines; (3) UIAutomator [21] can obtain the view hierarchy file and
its GUI screenshot; (4) the issue report involve non-crash bug and it can be
manually reproduced for validation; (6) the app is not used in the
motivational study or example dataset construction. Please note that we follow
the name of the app to ensure that there is no overlap between the datasets.
Finally, 43 apps with 247 non-crash bugs remain for further experiments.

To avoid potential data leakage issues caused by the use of MLLM and further
analyze the bug detection capability of VisionDroid, we construct a third
dataset (Injection dataset) through bug injection, which can ensure that MLLM
has not learned about these bugs in the open-source repositories.
Specifically, we obtain the source code of the second dataset apps from GitHub
and manually verify whether the app could conduce the bug injection by
modifying the code through repackaging. Among them, 27 apps can run normally
after bug injection. We conduct the injection based on the bug description,
bug repair code, and bug root cause obtained through the pilot study in
Section II, and implement it following the existing bug injection approach
[31, 32, 33]. For the 9 summarized bug categories in Section II, we inject 5
non-crash bugs per category, resulting in a total of 45 bugs in 27 apps.

TABLE I: Result of coverage and bug detection (RQ1)

Method | Baseline data | GitHub data | Bug injection data
---|---|---|---
|  Act |  Cod |  Rep |  Bug |  P |  R |  Act |  Cod |  Rep |  Bug |  P |  R |  Act |  Cod |  Rep |  Bug |  P |  R
Intra-page bug detection method with automated GUI testing
OwlEyes+A |  0.29 |  0.27 |  92 |  18 |  0.20 |  0.14 |  0.41 |  0.37 |  117 |  57 |  0.49 |  0.23 |  0.42 |  0.40 |  77 |  7 |  0.09 |  0.16
NightHawk+A |  0.29 |  0.27 |  86 |  21 |  0.24 |  0.16 |  0.41 |  0.37 |  99 |  66 |  0.67 |  0.27 |  0.42 |  0.40 |  41 |  8 |  0.20 |  0.18
DiffDroid+A |  0.29 |  0.27 |  87 |  20 |  0.23 |  0.15 |  0.41 |  0.37 |  82 |  47 |  0.57 |  0.19 |  0.42 |  0.40 |  27 |  3 |  0.11 |  0.07
dVermin+A |  0.29 |  0.27 |  67 |  6 |  0.09 |  0.05 |  0.41 |  0.37 |  51 |  10 |  0.20 |  0.04 |  0.42 |  0.40 |  18 |  2 |  0.11 |  0.04
OwlEyes+F |  0.39 |  0.37 |  107 |  17 |  0.16 |  0.13 |  0.44 |  0.40 |  119 |  57 |  0.48 |  0.23 |  0.43 |  0.39 |  79 |  8 |  0.10 |  0.18
NightHawk+F |  0.39 |  0.37 |  100 |  20 |  0.20 |  0.15 |  0.44 |  0.40 |  98 |  64 |  0.65 |  0.26 |  0.43 |  0.39 |  39 |  9 |  0.23 |  0.20
DiffDroid+F |  0.39 |  0.37 |  94 |  20 |  0.21 |  0.15 |  0.44 |  0.40 |  77 |  51 |  0.66 |  0.21 |  0.43 |  0.39 |  28 |  4 |  0.14 |  0.09
dVermin+F |  0.39 |  0.37 |  41 |  8 |  0.20 |  0.06 |  0.44 |  0.40 |  44 |  13 |  0.30 |  0.05 |  0.43 |  0.39 |  11 |  2 |  0.18 |  0.04
OwlEyes+H |  0.38 |  0.35 |  97 |  12 |  0.12 |  0.09 |  0.44 |  0.40 |  97 |  54 |  0.56 |  0.22 |  0.43 |  0.39 |  71 |  7 |  0.10 |  0.16
NightHawk+H |  0.38 |  0.35 |  88 |  16 |  0.18 |  0.12 |  0.44 |  0.40 |  94 |  58 |  0.62 |  0.23 |  0.43 |  0.39 |  69 |  7 |  0.10 |  0.16
DiffDroid+H |  0.38 |  0.35 |  75 |  17 |  0.23 |  0.13 |  0.44 |  0.40 |  78 |  32 |  0.41 |  0.13 |  0.43 |  0.39 |  31 |  4 |  0.13 |  0.09
dVermin+H |  0.38 |  0.35 |  33 |  5 |  0.15 |  0.04 |  0.44 |  0.40 |  27 |  17 |  0.63 |  0.07 |  0.43 |  0.39 |  11 |  2 |  0.18 |  0.04
OwlEyes+G |  0.47 |  0.43 |  99 |  17 |  0.17 |  0.13 |  0.49 |  0.44 |  104 |  59 |  0.57 |  0.24 |  0.47 |  0.41 |  77 |  8 |  0.10 |  0.18
NightHawk+G |  0.47 |  0.43 |  95 |  18 |  0.19 |  0.14 |  0.49 |  0.44 |  110 |  69 |  0.63 |  0.28 |  0.47 |  0.41 |  48 |  9 |  0.19 |  0.20
DiffDroid+G |  0.47 |  0.43 |  85 |  16 |  0.19 |  0.12 |  0.49 |  0.44 |  88 |  49 |  0.56 |  0.20 |  0.47 |  0.41 |  27 |  3 |  0.11 |  0.07
dVermin+G |  0.47 |  0.43 |  37 |  6 |  0.16 |  0.05 |  0.49 |  0.44 |  29 |  18 |  0.62 |  0.07 |  0.47 |  0.41 |  13 |  3 |  0.23 |  0.07
Intra-page bug detection method
iFixdataloss |  0.33 |  0.30 |  21 |  8 |  0.38 |  0.06 |  0.44 |  0.40 |  52 |  18 |  0.35 |  0.07 |  0.43 |  0.39 |  13 |  2 |  0.15 |  0.04
SetDroid |  0.31 |  0.27 |  22 |  8 |  0.36 |  0.06 |  0.44 |  0.40 |  98 |  44 |  0.45 |  0.18 |  0.43 |  0.39 |  11 |  2 |  0.18 |  0.04
Genie |  0.32 |  0.31 |  55 |  12 |  0.22 |  0.09 |  0.44 |  0.40 |  61 |  22 |  0.36 |  0.09 |  0.43 |  0.39 |  27 |  4 |  0.15 |  0.09
Odin |  0.33 |  0.30 |  57 |  22 |  0.39 |  0.17 |  0.44 |  0.40 |  58 |  29 |  0.50 |  0.12 |  0.43 |  0.39 |  29 |  6 |  0.21 |  0.13
Lint |  - |  - |  1 |  1 |  1.00 |  0.01 |  - |  - |  2 |  1 |  0.50 |  0.00 |  - |  - |  1 |  1 |  1.00 |  0.02
LiveDroid |  - |  - |  1 |  1 |  1.00 |  0.01 |  - |  - |  3 |  1 |  0.33 |  0.00 |  - |  - |  2 |  0 |  0.00 |  0.00
GPT-4 |  0.39 |  0.31 |  55 |  12 |  0.22 |  0.09 |  0.44 |  0.40 |  61 |  22 |  0.36 |  0.09 |  0.43 |  0.39 |  18 |  2 |  0.11 |  0.04
VisionDroid |  0.57 |  0.55 |  109 |  55 |  0.50 |  0.42 |  0.65 |  0.56 |  177 |  127 |  0.72 |  0.51 |  0.61 |  0.59 |  38 |  29 |  0.76 |  0.64

  * •

Notes: “Act” is activity coverage, “Cod” is Code coverage. “Rep” is number of
bugs reported by tools, “Bug” is true bug in the reported bug, “P” is
precision and “R” is the recall. “+A” means it with APE, “+F” means it with
Fastbot, “+H” means it with Hunmanoid, and “+G” means it with GPTDroid.

For RQ3, we further evaluate the usefulness of VisionDroid in detecting unseen
bugs. We randomly selected 105 popular apps based on app type and download
number. We use the same configurations as the previous experiments. Once a bug
is spotted, we create an issue report by describing the bug, and report them
to the app development team through the issue reporting system or email.

###  IV-B Baselines

To demonstrate the advantage of VisionDroid, we compare it with 10 common-used
and state-of-the-art non-crash bug detection techniques and 4 common-used
automated GUI testing techniques. We classify these baselines based on the
categories of bugs, including intra-page bug detection approaches and inter-
page bug detection approaches.

For intra-page non-crash bug detection, the techniques are mainly for GUI
display-related issue detection. OwlEyes [12] and NightHawk [13] use deep
learning to detect UI display issues (e.g., overlapping texts, component
occlusion). DVermin [15] detects scaling issues by comparing GUI pages under
different page settings. DiffDroid [14] compares the GUI pages of an app on
two different devices to find compatibility issues. The input of these
techniques are screenshots/visual hierarchy files and need to be integrated
with automated testing approaches to fetch the required information during the
exploration. We utilize four popular automated GUI testing tools, i.e., model-
based tools APE [9] and Fastbot [34], learning-based tool Humanoid [35] and
LLM-based GPTDroid [24].

For inter-page non-crash bug detection, we choose 6 baselines from different
perspectives. iFixdataloss [36] is the most recent work of finding data loss
issues. SetDroid [37] uses metamorphic testing to find system setting-related
functional bugs. Genie [38] uses the metamorphic relation of independent view
property to find non-crash functional bugs, while Odin [29] uses the implicit
knowledge of “bugs as deviant behavior” to find non-crash functional bugs.
Android Lint [39] is a popular static analysis tool for finding different
issues in Android apps based on predefined rules. LiveDroid [40] is also a
static analysis tool for detecting data loss issues.

###  IV-C Experimental Setup

We deploy the baselines and our approach on a 64-bit Ubuntu 18.04 machine (64
cores, AMD CPU, 2GB RAM) and evaluate them on Google Android 7.1 emulators.
Following common practice [7, 8, 10], we registered separate accounts for each
bug that requires login and wrote the login scripts, and during testing reset
the account data before each run to avoid possible interference. To ensure
fair and reasonable use of resources, we set up the running time of each tool
in one app to 50 minutes, which is widely used in other GUI testing studies
[8, 10, 41, 34] We run each tool 3 times and take the average value as the
result to mitigate potential bias [8, 10, 41, 34].

TABLE II: Contribution of different sub-modules (RQ2) Sub-modules | Baseline data | GitHub data | Bug injection data
---|---|---|---
|  Act |  Cod |  Rep |  Bug |  P |  R |  Act |  Cod |  Rep |  Bug |  P |  R |  Act |  Cod |  Rep |  Bug |  P |  R
VisionDroid |  0.57 |  0.55 |  109 |  55 |  0.50 |  0.42 |  0.65 |  0.56 |  177 |  127 |  0.72 |  0.51 |  0.61 |  0.59 |  38 |  29 |  0.76 |  0.64
w/o Alignment |  0.44 |  0.39 |  97 |  32 |  0.33 |  0.24 |  0.49 |  0.48 |  133 |  75 |  0.56 |  0.30 |  0.47 |  0.43 |  29 |  18 |  0.62 |  0.40
w/o History |  0.41 |  0.35 |  91 |  28 |  0.31 |  0.21 |  0.47 |  0.44 |  128 |  63 |  0.49 |  0.26 |  0.44 |  0.31 |  27 |  15 |  0.56 |  0.33
w/o Community |  0.57 |  0.55 |  90 |  17 |  0.19 |  0.13 |  0.65 |  0.56 |  149 |  39 |  0.26 |  0.16 |  0.61 |  0.59 |  42 |  7 |  0.17 |  0.16
w/o Example |  0.57 |  0.55 |  89 |  18 |  0.20 |  0.14 |  0.65 |  0.56 |  151 |  41 |  0.27 |  0.17 |  0.61 |  0.59 |  39 |  8 |  0.21 |  0.18

###  IV-D Evaluation Metrics

We measure the performance from coverage and bug detection, respectively
corresponding to the explorer and detector in VisionDroid. For coverage, we
obtain the activity coverage and code coverage [42, 43, 44, 10, 45, 35], in
which treat the activities defined in the AndroidManifest.xml file of an
Android app as the whole set of activities [46, 47, 7].

For bug detection, following existing studies [38, 29], we report the detected
bugs by VisionDroid, true bugs among the detected bugs. Since we know the
entire set of bugs, we also calculate the precision and recall of the bug
detection.

##  V Results and Analysis

###  V-A Coverage and Bug Detection Performance (RQ1)

We present the coverage and bug detection performance of VisionDroid and
baselines, in terms of the baseline dataset, GitHub dataset and the bug
injection dataset in Table I.

We first present the coverage of VisionDroid, which can cover far more
activities and code than the baselines. In three datasets, the average
activity coverage is between 57% and 61%, and the average code coverage is
between 55% and 59%. It achieves between 21% (0.57 vs. 0.47) to 33% (0.65 vs.
0.49) activity coverage and between 27% (0.56 vs. 0.44) and 43% (0.59 vs.
0.41) code coverage higher even compared with the best baseline (GPTDroid)
across three datasets. This indicates the effectiveness of VisionDroid in
covering more activities and codes, thus bringing higher confidence to app
quality, potentially uncovering more bugs.

The average precision and recall of VisionDroid in three datasets achieve
50%-76%, and 42%-64%, respectively. It achieves between 8% (0.72 vs. 0.67) and
230% (0.76 vs. 0.23) precision and between 27% (0.56 vs. 0.44) and 43% (0.59
vs. 0.41) higher even compared with the best baseline across three datasets.
Please note that although some baseline methods can achieve higher precision,
they detect fewer bugs. In terms of the number of detected bugs, VisionDroid
can still achieve a higher detection rate. We also compare the similarities
and differences of the bugs between the best baselines (Odin and NightHawk)
and our approach, and the results show that all bugs detected by them are also
detected by VisionDroid. This indicates the effectiveness of VisionDroid in
detecting bugs and helps to ensure app quality.

The advantages of VisionDroid are due to three aspects. First, VisionDroid
designs the vision prompt which can better capture the GUI context, while the
baselines mainly rely on the view hierarchy file in which the widgets might
have meaningless “text” or “resource-id”, hindering them from effectively
understanding the GUI pages. Second, VisionDroid plans the exploration path
both considering the semantics of a single GUI page and the exploration
history which can enable the function-aware exploration, while the baselines
can hardly understand the inherent business logic of the app, resulting in
aimless exploration. Third, VisionDroid detects the bugs by understanding the
semantics of a sequence of GUI pages, while the baselines can only detect
specific types of bugs or have low detection accuracy.

Fig. 7 demonstrates an example of VisionDroid’s detected bug. This bug is that
after deleting a single expense on the budget information page, all original
expense records disappear when viewing the expense page. We can see that
VisionDroid is adept at determining whether the functional logic of the app is
abnormal based on the semantics of test path and GUI information.

###  V-B Ablation Study (RQ2)

####  V-B1 Contribution of Sub-modules

Table II further demonstrates the performance of VisionDroid and its 4
variants. We remove each sub-module of the VisionDroid in Fig. 3 separately,
i.e., Alignment text with image, Function-based testing history construction,
Function-driven community detection and Example bugs retrieval. For removing
the Alignment text with image, we directly input the unmarked screenshot. For
removing function-based testing history construction, we delete the history
information. For removing the community detection, we randomly segment the
test sequence and for removing retrieved examples, we use the random
selection.

The experiment results (Table II) demonstrate that removing any of the sub-
modules would result in a noticeable performance decline, indicating the
necessity and effectiveness of the designed sub-modules. Removing the
community detection (w/o-Community) has the greatest impact on the
performance, reducing the bug detection performance by 347% (0.17 vs. 0.76)
precision and 300% (0.16 vs. 0.64).

We also notice that, when removing the alignment text with image
(w/o-Alignment) and function-based testing history construction(w/o-History),
the activity coverage has an impact, reducing the activity coverage by 39%
(0.41 vs. 0.57) and 57% (0.35 vs. 0.55). This further indicates that the
alignment text with image can help MLLM understand the structure and semantic
information of GUI pages and make reasonable judgments. The history can help
retain knowledge during testing and gain global viewpoints to reach uncovered
areas.

Besides, randomly selecting examples (w/o-retriExample) would also largely
influence the performance, and decrease the bug detection performance by 262%
(0.21 vs. 0.76) precision and 256% (0.18 vs. 0.64) recall. This indicates that
by providing similar examples, the MLLM can quickly think out what should the
no-crash bugs look like, which further indicates the demonstration can
facilitate the MLLM in producing the required output.

####  V-B2 Influence of Different Number of Examples

Fig. 9 demonstrates the performance under the different number of examples
provided in the prompt. We can see that the non-crash bug detection
performance increases with more examples, reaching the highest bug detection
rate with 4 examples. And after that, the performance would gradually decrease
even increasing the examples. This indicates that too few or too many examples
would damage the performance, because of the tiny information or the noise in
the provided examples.

Figure 9: Different number of examples (RQ2)

###  V-C Usefulness Evaluation (RQ3)

For the 105 apps, VisionDroid detects 83 non-crash bugs in 53 apps, of which
37 bugs were never discovered before. Furthermore, only 9 of these new bugs
are detected by Genie [38] and Odin [29]. We submitted these 37 bugs to
developers, and 19 of them have been fixed/confirmed so far (10 fixed and 9
confirmed), while the remaining are still pending (none of them is rejected).
This further indicates the effectiveness of our proposed VisionDroid in bug
detection. Due to space limit, Table III presents these fixed/confirmed bugs,
and the full lists can be found on our website1. We further analyze the
details of these bugs, and 21 of them involve intra-page bugs and 16 of them
involve inter-page bugs.

TABLE III: Confirmed or fixed bugs. (RQ3) Id |  APP Name |  Category |  Download |  Status
---|---|---|---|---
1 |  MOMPA |  Music |  500M+ |  fixed
2 |  Calendar |  Tool |  100M+ |  fixed
3 |  DigiCal |  Tool |  100M+ |  fixed
4 |  Expensify |  Finance |  100M+ |  fixed
5 |  TrackWallet |  Finance |  10M+ |  confirmed
6 |  Roadie |  Travel |  10M+ |  confirmed
7 |  ISAY |  Commun |  10M+ |  fixed
8 |  EasyBudget |  Finance |  10M+ |  confirmed
9 |  Iplan |  Travel |  10M+ |  fixed
10 |  MediaFire |  Product |  5M+ |  confirmed
11 |  SmartGuide |  Navig |  500K+ |  fixed
12 |  MMDR |  Utilities |  500K+ |  confirmed
13 |  Sygic |  Travel |  500K+ |  fixed
14 |  Fair |  Health |  500K+ |  confirmed
15 |  ClassySha |  Tool |  500K+ |  fixed
16 |  Linphone |  Commun |  50K+ |  confirmed
17 |  Rocket |  Finance |  50K+ |  confirmed
18 |  Monefy |  Finance |  50K+ |  fixed
19 |  Origin |  Finance |  50K+ |  confirmed

##  VI Related Work

###  VI-A Automated GUI Testing

To ensure the quality of mobile apps, many researchers study the automatic
generation of large-scale test scripts to test apps [48]. Since Android apps
are event-based [49, 50, 51, 52], the most common automated testing methods
are model-based automated GUI testing methods [2, 3, 4, 5, 6, 7, 8, 9, 10],
design corresponding models through the analysis of the apps. Due to the lack
of consideration for the semantic information of the app’s GUI pages, the
coverage of model-based methods is still low. Researchers further proposed
human-like testing strategies and designed learning-based automated GUI
testing methods [35, 46]. GPTDroid [24] used GPT-3.5 to generate the testing
script, aiming at proposing a more effective approach to generate human-like
actions for thoroughly and more effectively testing the app. However, these
methods require first converting the GUI visual information into text, which
can cause significant visual information loss during the process. Moreover, a
large number of app widgets and pages are generated through dynamic rendering,
and the attribute information of image widgets will be lost when obtaining
view hierarchy files. As a result, the coverage of automated testing is low,
and a large number of widgets and operations are missed. This study employs
the MLLM for the GUI testing, which is more effective through aligning the GUI
screenshot with text information for the vision prompt.

###  VI-B Non-crash Functional Bug Detection

According to the sources of test oracle, current research can be divided into
four types. The first type utilizes oracle provided by testers [53] to find
non-crash bugs in different strategies (such as ChimpCheck [54], AppFlow [55],
ACAT [56], AppTestMigrator [57], CraftDroid [58]). Unlike them, this paper
doesn’t require manual direct/indirect oracle.

The second type uses differential testing to overcome the problem of missing
oracles. Genie [38] and Odin [29] used the metamorphic relation of independent
view property to find bugs. SPAG-C [59], DiffDroid [14] and dVermin [15]
compare screenshots between two different mobile devices or platform versions
to identify bugs.

The third type is the automatic generation of test oracle for specific
categories of user interactions. ITDroid [60] compared the GUI pages of an app
under two different language settings to find i18n issues. SetDroid [37]
generated the oracle for system settings bugs, but can’t detect other types of
bugs.

The fourth type is to use labeled large-scale bug data to train computer
vision models for detecting bugs related to GUI display. The OwlEyes [12],
Gilb [16] and NightHawk [13] are based on large-scale labeled UI bug data, to
train computer vision models that can be used to detect the display bugs.

However, these bug detection methods have problems such as a single category
of bug detection, requiring a large amount of manually labeled test oracle
data, and being unable to meet complex business scenarios and app
functionality.

##  VII Conclusion

With the advancement of LLM, there have been tremendous applications of LLMs
in software engineering-related tasks such as program repair and test case
generation with great performance improvement. Subsequently, MLLMs emerge and
are applied in image-text reasoning, video generation, etc., further expanding
our understanding of the capabilities of LLMs. This paper proposes
VisionDroid, the pioneering work using MLLMs for vision-based mobile app GUI
testing, and the experiment demonstrates its effectiveness and usefulness.
This work contributes as a stepping stone for future exploration in software
engineering, offering insights and guidance on the potential applications of
MLLM. In the future, we will further improve this approach for better
performance. Meanwhile, beyond Android apps, we will customize and apply our
approach to other platforms such as iOS, web, desktop and even embedded
systems for software testing.

## References

  * [1] S. Pargaonkar, “A comprehensive review of performance testing methodologies and best practices: Software quality engineering,” _International Journal of Science and Research (IJSR)_ , vol. 12, no. 8, pp. 2008–2014, 2023.
  * [2] N. Mirzaei, J. Garcia, H. Bagheri, A. Sadeghi, and S. Malek, “Reducing combinatorics in gui testing of android applications,” in _2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)_. IEEE, 2016, pp. 559–570.
  * [3] S. Yang, H. Wu, H. Zhang, Y. Wang, C. Swaminathan, D. Yan, and A. Rountev, “Static window transition graphs for android,” _Automated Software Engineering_ , vol. 25, no. 4, pp. 833–873, 2018.
  * [4] W. Yang, M. R. Prasad, and T. Xie, “A grey-box approach for automated gui-model generation of mobile applications,” in _International Conference on Fundamental Approaches to Software Engineering_. Springer, 2013, pp. 250–265.
  * [5] X. Zeng, D. Li, W. Zheng, F. Xia, Y. Deng, W. Lam, W. Yang, and T. Xie, “Automated test input generation for android: Are we really there yet in an industrial case?” in _Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , 2016, pp. 987–992.
  * [6] K. Mao, M. Harman, and Y. Jia, “Sapienz: Multi-objective automated testing for android applications,” in _Proceedings of the 25th International Symposium on Software Testing and Analysis_ , 2016, pp. 94–105.
  * [7] T. Su, G. Meng, Y. Chen, K. Wu, W. Yang, Y. Yao, G. Pu, Y. Liu, and Z. Su, “Guided, stochastic model-based gui testing of android apps,” in _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_ , 2017, pp. 245–256.
  * [8] Z. Dong, M. Böhme, L. Cojocaru, and A. Roychoudhury, “Time-travel testing of android apps,” in _2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)_. IEEE, 2020, pp. 481–492.
  * [9] T. Gu, C. Sun, X. Ma, C. Cao, C. Xu, Y. Yao, Q. Zhang, J. Lu, and Z. Su, “Practical gui testing of android applications via model abstraction and refinement,” in _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 2019, pp. 269–280.
  * [10] J. Wang, Y. Jiang, C. Xu, C. Cao, X. Ma, and J. Lu, “Combodroid: generating high-quality test inputs for android apps via use case combinations,” in _ICSE_ , 2020, pp. 469–480.
  * [11] Z. Liu, C. Chen, J. Wang, and Q. Wang, “Guided bug crush: Assist manual gui testing of android apps via hint moves,” in _CHI 2022_ , 2022. [Online]. Available: https://doi.org/10.1145/3491102.3501903
  * [12] Z. Liu, C. Chen, J. Wang, Y. Huang, J. Hu, and Q. Wang, “Owl eyes: Spotting ui display issues via visual understanding,” in _ASE_. IEEE, 2020.
  * [13] ——, “Nighthawk: Fully automated localizing ui display issues via visual understanding.” IEEE, 2022, pp. 1–16.
  * [14] M. Fazzini and A. Orso, “Automated cross-platform inconsistency detection for mobile apps,” in _ASE_. IEEE, 2017.
  * [15] Y. Su, C. Chen, J. Wang, Z. Liu, D. Wang, S. Li, and Q. Wang, “The metamorphosis: Automatic detection of scaling issues for mobile apps,” in _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_ , 2022, pp. 1–12.
  * [16] K. Chen, Y. Li, Y. Chen, C. Fan, Z. Hu, and W. Yang, “Glib: towards automated test oracle for graphically-rich applications,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2021, pp. 1093–1104.
  * [17] J. Hu, L. Wei, Y. Liu, S.-C. Cheung, and H. Huang, “A tale of two cities: How webview induces bugs to android applications,” in _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ , 2018, pp. 702–713.
  * [18] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg _et al._ , “Sparks of artificial general intelligence: Early experiments with gpt-4,” _arXiv preprint arXiv:2303.12712_ , 2023.
  * [19] “Github,” https://github.com/, 2024.
  * [20] C. B. Seaman, “Qualitative methods in empirical studies of software engineering,” _IEEE Transactions on software engineering_ , vol. 25, no. 4, pp. 557–572, 1999.
  * [21] UIAutomator, “Python wrapper of android uiautomator test tool.” https://github.com/xiaocong/uiautomator, 2021.
  * [22] X. Zhang, L. de Greef, A. Swearngin, S. White, K. Murray, L. Yu, Q. Shan, J. Nichols, J. Wu, C. Fleizach _et al._ , “Screen recognition: Creating accessibility metadata for mobile applications from pixels,” in _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–15.
  * [23] S. Ghosh, M. Halappanavar, A. Tumeo, A. Kalyanaraman, H. Lu, D. Chavarria-Miranda, A. Khan, and A. Gebremedhin, “Distributed louvain algorithm for graph community detection,” in _2018 IEEE international parallel and distributed processing symposium (IPDPS)_. IEEE, 2018, pp. 885–895.
  * [24] Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, and Q. Wang, “Chatting with gpt-3 for zero-shot human-like mobile automated gui testing,” _arXiv preprint arXiv:2305.09434_ , 2023.
  * [25] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” _ICLR_ , 2013.
  * [26] “pyvbox,” https://pypi.org/project/pyvbox/, 2023.
  * [27] “Android debug bridge (adb),” https://developer.android.com/studio/command-line/adb.html#forwardports, 2020.
  * [28] Y. Xiong, M. Xu, T. Su, J. Sun, J. Wang, H. Wen, G. Pu, J. He, and Z. Su, “An empirical study of functional bugs in android apps,” in _Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2023, pp. 1319–1331.
  * [29] J. Wang, Y. Jiang, T. Su, S. Li, C. Xu, J. Lu, and Z. Su, “Detecting non-crashing functional bugs in android apps via deep-state differential analysis,” in _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2022, pp. 434–446.
  * [30] “Google play,” https://play.google.com/store/apps/, 2022.
  * [31] A. Ghaleb and K. Pattabiraman, “How effective are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2020, pp. 415–427.
  * [32] P. Bhattacharya, L. Ulanova, I. Neamtiu, and S. C. Koduru, “An empirical analysis of bug reports and bug fixing in open source android apps,” in _2013 17th European Conference on Software Maintenance and Reengineering_. IEEE, 2013, pp. 133–143.
  * [33] G. Hu, X. Yuan, Y. Tang, and J. Yang, “Efficiently, effectively detecting mobile app bugs with appdoctor,” in _Proceedings of the Ninth European Conference on Computer Systems_ , 2014, pp. 1–15.
  * [34] T. Cai, Z. Zhang, and P. Yang, “Fastbot: A multi-agent model-based test generation system beijing bytedance network technology co., ltd.” in _Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test_ , 2020, pp. 93–96.
  * [35] Y. Li, Z. Yang, Y. Guo, and X. Chen, “Humanoid: a deep learning-based approach to automated black-box android app testing,” in _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2019, pp. 1070–1073.
  * [36] W. Guo, Z. Dong, L. Shen, W. Tian, T. Su, and X. Peng, “Detecting and fixing data loss issues in android apps,” in _Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2022, pp. 605–616.
  * [37] J. Sun, T. Su, K. Liu, C. Peng, Z. Zhang, G. Pu, T. Xie, and Z. Su, “Characterizing and finding system setting-related defects in android apps,” _IEEE Transactions on Software Engineering_ , 2023.
  * [38] T. Su, Y. Yan, J. Wang, J. Sun, Y. Xiong, G. Pu, K. Wang, and Z. Su, “Fully automated functional fuzzing of android apps for detecting non-crashing logic bugs,” _Proceedings of the ACM on Programming Languages_ , vol. 5, no. OOPSLA, pp. 1–31, 2021.
  * [39] http://tools.android.com/tips/lint, 2020.
  * [40] U. Farooq, Z. Zhao, M. Sridharan, and I. Neamtiu, “Livedroid: Identifying and preserving mobile app state in volatile runtime environments,” _Proceedings of the ACM on Programming Languages_ , vol. 4, no. OOPSLA, pp. 1–30, 2020.
  * [41] Y. Li, Z. Yang, Y. Guo, and X. Chen, “Droidbot: a lightweight ui-guided test input generator for android,” in _2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C)_. IEEE, 2017, pp. 23–26.
  * [42] Y. He, L. Zhang, Z. Yang, Y. Cao, K. Lian, S. Li, W. Yang, Z. Zhang, M. Yang, Y. Zhang _et al._ , “Textexerciser: feedback-driven text input exercising for android applications,” in _2020 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2020, pp. 1071–1087.
  * [43] P. Liu, X. Zhang, M. Pistoia, Y. Zheng, M. Marques, and L. Zeng, “Automatic text input generation for mobile testing,” in _2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)_. IEEE, 2017, pp. 643–653.
  * [44] Y. L. Arnatovich, L. Wang, N. M. Ngo, and C. Soh, “Mobolic: An automated approach to exercising mobile application guis using symbiosis of online testing technique and customated input generation,” _Software: Practice and Experience_ , vol. 48, no. 5, pp. 1107–1142, 2018.
  * [45] W. Wang, W. Yang, T. Xu, and T. Xie, “Vet: identifying and avoiding ui exploration tarpits,” in _FSE_ , 2021, pp. 83–94.
  * [46] M. Pan, A. Huang, G. Wang, T. Zhang, and X. Li, “Reinforcement learning based curiosity-driven testing of android applications,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2020, pp. 153–164.
  * [47] “Android,” https://developer.android.google/topic/, 2022.
  * [48] Q. Xie and A. M. Memon, “Designing and comparing automated test oracles for gui-based software applications,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 16, no. 1, pp. 4–es, 2007.
  * [49] S. Anand, M. Naik, M. J. Harrold, and H. Yang, “Automated concolic testing of smartphone apps,” in _Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering_ , 2012, pp. 1–11.
  * [50] T. Wu, X. Deng, J. Yan, and J. Zhang, “Analyses for specific defects in android applications: A survey,” _Frontiers of Computer Science_ , pp. 1–18, 2019.
  * [51] R. Jabbarvand, J.-W. Lin, and S. Malek, “Search-based energy testing of android,” in _ICSE_. IEEE, 2019, pp. 1119–1130.
  * [52] R. Matinnejad, S. Nejati, and L. C. Briand, “Automated testing of hybrid simulink/stateflow controllers: industrial case studies,” in _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_ , 2017, pp. 938–943.
  * [53] C. Q. Adamsen, G. Mezzetti, and A. Møller, “Systematic execution of android test suites in adverse conditions,” in _Proceedings of the 2015 International Symposium on Software Testing and Analysis_ , 2015, pp. 83–93.
  * [54] E. S. Lam, P. Zhang, and B.-Y. E. Chang, “Chimpcheck: property-based randomized test generation for interactive apps,” in _Proceedings of the 2017 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software_ , 2017, pp. 58–77.
  * [55] G. Hu, L. Zhu, and J. Yang, “Appflow: using machine learning to synthesize robust, reusable ui tests,” in _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2018, pp. 269–282.
  * [56] A. Rosenfeld, O. Kardashov, and O. Zang, “Automation of android applications functional testing using machine learning activities classification,” in _Proceedings of the 5th international conference on mobile software engineering and systems_ , 2018, pp. 122–132.
  * [57] F. Behrang and A. Orso, “Test migration between mobile apps with similar functionality,” in _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2019, pp. 54–65.
  * [58] J.-W. Lin, R. Jabbarvand, and S. Malek, “Test transfer across mobile apps through semantic mapping,” in _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2019, pp. 42–53.
  * [59] Y.-D. Lin, J. F. Rojas, E. T.-H. Chu, and Y.-C. Lai, “On the accuracy, efficiency, and reusability of automated test oracles for android devices,” _IEEE Transactions on Software Engineering_ , vol. 40, no. 10, pp. 957–970, 2014.
  * [60] C. Escobar-Velásquez, M. Osorio-Riaño, J. Dominguez-Osorio, M. Arevalo, and M. Linares-Vásquez, “An empirical study of i18n collateral changes and bugs in guis of android apps,” in _2020 IEEE international conference on software maintenance and evolution (ICSME)_. IEEE, 2020, pp. 581–592.