#  SeePrivacy: Automated Contextual Privacy Policy Generation for Mobile
Applications

Shidong Pan124, Zhen Tao1, Thong Hoang2, Dawen Zhang12, Zhenchang Xing12,
Xiwei Xu2, Mark Staples2 David Lo3  1Australian National University, Canberra,
Australia
{Shidong.Pan<EMAIL_ADDRESS>2CSIRO’s Data61, Sydney, Australia
{James.Hoang, Dawen.Zhang, Zhenchang.Xing, Xiwei.Xu,
<EMAIL_ADDRESS>3Singapore Management University, Singapore
<EMAIL_ADDRESS>
###### Abstract

Privacy policies have become the most critical approach to safeguarding
individuals’ privacy and digital security. To enhance their presentation and
readability, researchers propose the concept of contextual privacy policies
(CPPs), aiming to fragment policies into shorter snippets and display them
only in corresponding contexts. In this paper, we propose a novel multi-modal
framework, namely SeePrivacy , designed to automatically generate contextual
privacy policies for mobile apps. Our method synergistically combines mobile
GUI understanding and privacy policy document analysis, yielding an impressive
overall 83.6% coverage rate for privacy-related context detection and an
accuracy of 0.92 in extracting corresponding policy segments. Remarkably, 96%
of the retrieved policy segments can be correctly matched with their contexts.
The user study shows SeePrivacy demonstrates excellent functionality and
usability (4.5/5). Specifically, participants exhibit a greater willingness to
read CPPs (4.1/5) compared to original privacy policies (2/5). Our solution
effectively assists users in comprehending privacy notices, and this research
establishes a solid foundation for further advancements and exploration.

###### Index Terms:

Contextual Privacy Policy, Privacy Notice, Privacy Policy, GUI Understanding

44footnotetext: The work was mostly completed during the academic visit at
Singapore Management University.

##  I Introduction

Mobile apps have become an omnipresent part of our digital lives, with
billions of downloads worldwide. These apps offer convenience, entertainment,
and various services; however, they also collect and exchange vast amounts of
personal information, raising increasing concerns about data privacy. For
example, the British Information Commissioner’s Office accused Tiktok of
violating data protection for
children.111https://www.darkreading.com/vulnerabilities-threats/tiktok-other-
mobile-apps-violate-privacy-regulations In 2021, Apple blocked 1.6 million
mobile applications for misleading, spamming, and defrauding
users.222https://www.bleepingcomputer.com/news/security/apple-
blocked-16-millions-apps-from-defrauding-users-in-2021/

(a) Install-time

(b) Invoke-time

(c) Context-aware

Figure 1:  The leftmost is the install-time reminder of required permissions
[1]; the middle is the reminder that pops up the first time to invoke
permission; and the rightmost is our contextual privacy policy for mobile
apps.

To prevent privacy violations, the U.S. Department of Health, Education, and
Welfare proposed a set of Fair Information Practice Principles in 1973 [2, 3].
Later on, the Federal Trade Commission [4] adopted these principles as “the
notice and choice privacy framework,” which is commonly regarded as the
foundation for modern privacy regulations such as European General Data
Protection Regulation (GDPR) [5], California Consumer Privacy Act (CCPA) [6],
and Australian Privacy Principles (APP) [7]. Today, privacy policies have
become the most critical approach to safeguarding individuals’ privacy and
digital security [8, 9, 10]. Despite the provision of clear, concise, and
transparent privacy policies, users often grant permissions to mobile apps to
access their personal information without comprehensively understanding these
policies. This phenomenon persists even when the policies are readily
available for review. Obar and Oeldorf-Hirsch [4] have demonstrated that a
significant portion of users, approximately 74%, opt for the “quick join”
clickwrap option without engaging with the privacy policies, primarily due to
the lengthy and detailed nature of these documents. The findings of a study
conducted on 75 prominent mobile applications and websites [11] further
support this observation, revealing that the average length of privacy
policies is around 4,000 words, requiring an estimated 16 minutes to read.
These findings indicate that privacy policies have become increasingly
challenging to read and understand [12].

Many studies have been conducted to improve the presentation and readability
of privacy policies from various perspectives. For example, Kelly et al. [13,
14, 15] suggested a concise and organized approach, namely privacy nutrition
labels (PNL), to help customers better understand how their information is
gathered and utilized. Moreover, PNL effectively facilitates their
comprehension of software applications’ privacy practices [16, 17, 18]. Cranor
[19] proposed a protocol, i.e., Platform for Privacy Preferences, that allows
website organizations to declare their intended use of information collected
from website users. Another research direction focuses on privacy icons [20,
21, 22], small graphic symbols representing a privacy concept or practice, to
assist users in understanding data handling practices by providing a visual
shorthand for complex privacy policies. These standardized privacy icons
across multiple services and platforms also help reduce confusion and make
privacy policies more accessible to users [21].

A recent development in the evolution of privacy policies is “just-in-time”
privacy notices. Specifically, these notices offer users precise information
about data collection at the moment of interaction. The U.K. GDPR has included
the just-in-time notice as a recommended practice in its code of conduct for
creating precise privacy policies [23]. However, most of the existing just-in-
time notices focus on when users install applications. For example, an
“install-time” permission reminder may pop up when a user installs a new
Android app, clarifying what permissions will be granted and how they will be
utilized (see Fig. 1a). Another example is shown in Fig. 1b, which is a camera
access reminder triggered at the first “invoke-time”. Some studies [24, 25]
have criticized the effectiveness of existing “install-time” privacy reminders
since security warnings for dangerous permissions are often decoupled from the
context, failing to raise awareness among users [24]. To mitigate this
problem, researchers have proposed a contextual privacy policy (CPP) to offer
immediate privacy notices in their corresponding contexts [26, 27, 28, 29].
Contextual privacy policies aim to break down policies into shorter snippets
and only display them in their corresponding contexts, which greatly helps
users to comprehend obscure language and reduces the amount of text that must
be read at once. Privacy-related contexts are areas where data practices apply
to GUIs. Fig. 1c shows an example of our contextual privacy policy on mobile.
Privacy-related contexts on the screen are detected out, and their
corresponding (in the same color) privacy policy segments are retrieved and
displayed, with emboldened key information. By doing so, privacy notices are
closely and timely incorporated with the context, effectively raising users’
awareness and delivering privacy information.

Compared to other software applications, mobile apps have more sophisticated
designs, compact Graphical User Interfaces (GUIs), and frequent human-device
interactions; hence, we have to identify the contexts that need to display
CPPs in the mobile apps. We first conduct a comprehensive survey to collect
and identify data types that have been covered in regulation principles,
industry standards, and previous academic studies about privacy practices on
mobile apps (See Section II-A). Specifically, we identify 12 CPP-related data
types in mobile apps and collect their keyword lists from surveyed literature.
We then accumulate a full list of CPP-related GUI components for nine data
types from RICO-icon, one of the largest mobile GUI icon datasets. Based on
this information, we establish the benchmark dataset named Cpp4App, containing
over 600 privacy-related contexts with their corresponding privacy policy
segments in the mobile scenario. Then we propose a novel multi-modal
framework, namely SeePrivacy, that can automatically generate contextual
privacy policies for mobile apps. Specifically, we utilize Computer Vision
(CV) techniques and a pre-trained Large Language Model to understand the
mobile GUI screenshots, detecting privacy-related contexts. In addition, we
employ Natural Language Processing (NLP) techniques to analyze the privacy
policy documents, fetching the corresponding policy segments for the contexts.
Our framework yields 0.76, 0.86, and 0.87 in terms of accuracy, precision, and
recall, respectively, on category-wise context identification, and an overall
83.6% coverage rate for privacy-related contexts detection as per the
screenshot. As for extracting corresponding segments, our framework achieves
0.92 accuracy, 0.94 precision, and 0.98 recall. Also, 96% of retrieved policy
segments can be correctly matched with their contexts. In addition, we conduct
a usability evaluation to further validate its functionality and usefulness in
practice. More than 77% of participants do not disagree that the retrieved
privacy policy segment explains the detected context, which is underlined by a
3.75 mean and 4 median on a 5-point Likert scale. Overall, SeePrivacy receives
4.53 out of 5 from users about its usefulness. Moreover, the proposed
framework does not require source code or additional APIs; hence, developers
will easily deploy the framework with lower security concerns. Moreover, the
generated CPPs offer concise, transparent, and timely privacy information that
assists users in making choices about whether they should interact with GUI
components. The main contributions of our paper are as follows:

  * •

To the best of our knowledge, we are the first to propose a novel framework,
i.e., SeePrivacy333The demonstration website: https://cpp4app.github.io/, to
automatically generate contextual privacy policies for mobile apps.

  * •

We build the benchmark dataset, i.e., Cpp4App, for contextual privacy policies
on mobile apps and evaluate SeePrivacy on it to demonstrate the effectiveness
of our approach.

  * •

We conduct a user study to further validate SeePrivacy’s functionality and
usability in practice.

TABLE I: Data types’ existence in literature.

| Basic Personal Identifiable Information | Other Personal Information
---|---|---
| Name | Birthday | Address | Phone | Email | Profile | Contacts | Location | Photos | Voices | Financial Info | Social Media
Administration |  |  |  |  |  |  |  |  |  |  |  |
GDPR [5] | ● | ○ | ● | ● | ● | ● | ○ | ○ | ○ | ○ | ○ | ○
CCPA [6] | ● | ○ | ○ | ○ | ● | ● | ○ | ● | ○ | ○ | ● | ○
CalOPPA [30] | ● | ○ | ● | ● | ● | ○ | ○ | ● | ○ | ○ | ● | ●
COPPA [31] | ● | ○ | ● | ● | ● | ● | ○ | ● | ● | ● | ○ | ○
APP [7] | ● | ○ | ● | ● | ● | ● | ○ | ● | ● | ○ | ● | ○
Industry |  |  |  |  |  |  |  |  |  |  |  |
Google Play [32] | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ●
Apple App [33] | ● | ○ | ● | ● | ● | ● | ● | ● | ● | ● | ● | ●
Huawei AppGallery [34] | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ●
Amazon Appstore [35] | ○ | ○ | ○ | ● | ● | ○ | ● | ● | ● | ● | ● | ●
Samsung Galaxy Store [36] | ○ | ○ | ○ | ● | ○ | ○ | ● | ● | ● | ● | ○ | ○
Academia |  |  |  |  |  |  |  |  |  |  |  |
Yu et al. [37] | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ○ | ○
Wilson et al. [38] | ● | ● | ● | ● | ● | ● | ● | ● | ○ | ○ | ● | ●
Harkous et al. [39] | ● | ● | ● | ● | ● | ● | ● | ● | ○ | ○ | ● | ●
Kaur et al. [40] | ● | ● | ● | ● | ● | ● | ○ | ● | ○ | ○ | ● | ○
Yu et al. [41] | ○ | ○ | ○ | ○ | ○ | ● | ● | ● | ● | ● | ○ | ○
Zimmeck et al. [42] | ○ | ○ | ○ | ● | ● | ● | ○ | ● | ○ | ○ | ○ | ●
Zhang et al. [43] | ● | ○ | ● | ● | ● | ○ | ○ | ● | ○ | ○ | ○ | ○
Sun and Xue [44] | ● | ○ | ○ | ● | ● | ○ | ○ | ● | ○ | ○ | ● | ○
Li et al. [45] | ● | ○ | ● | ● | ● | ● | ○ | ● | ● | ● | ○ | ○
Zimmeck et al. [46] | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ● | ● | ● | ○ | ●
Xie et al. [47] | ● | ● | ● | ● | ● | ● | ● | ● | ○ | ○ | ● | ○

##  II Background and Privacy-related Context Identification for Mobile Apps

The concept of “contextual privacy policies” (CPPs) was first proposed in
2004. Specifically, Bolchini et al. [48] identified that reading privacy
policies is a monolithic block when users attempt to retrieve privacy
information; thus, they invented a systematic approach to restructuring the
privacy policies based on user interaction contexts. Later on, Feth [26]
introduced the idea of “contextual privacy statements.” Rather than using a
“one-size-fits-all” privacy policy that must suit every usage scenario, these
contextual privacy statements offer precise details about privacy and data
protection in a specific activity or use case. Recently, Windl et al. [29]
proposed the design and architecture of PrivacyInjector, a production AI tool
that can automatically generate contextual privacy policies for websites. CPPs
have been demonstrated to effectively help users understand the privacy
practices and consequences of potential privacy risks [27]. CPPs also help
increase privacy awareness and transparency for users [49, 50, 51].

Compared to websites, mobile apps often collect personal information in
multiple ways through device sensors, user inputs, and third-party integration
[52]; however, disclosing how users’ personal information is gathered,
utilized, and disseminated is a crucial problem. As the size of mobile apps is
enormous and is still rapidly growing, a framework to automatically generate
contextual privacy policies for mobile apps is urgently needed. In this work,
we aim to propose a framework that can automatically generate CPPs, but first,
we need to have clear definitions of privacy-related context in the mobile
scenario.

###  II-A Privacy-related Context Identification for Mobile Apps

Privacy-related contexts are considered areas on the GUIs where data practices
are applied [29]. To specify the data practices for this work, we propose
three questions:

  * •

What data practices should we include?

  * •

What data practices are of concern and desire?

  * •

What information is delivered by common privacy policies?

Note that we have not found any studies that discuss contextual privacy
policies for mobile apps. Specifically, most research only focuses on
improving the presentation and readability of privacy policies [13, 15, 14,
19, 16]. In this paper, we conduct a comprehensive survey (see Table I) on the
data practices identified and discussed by three groups: administration,
market standards from industry, and academic research papers, to answer the
three questions.

_Administration._ We often see the administration as the regulator of the
ecosystem. According to the United Nations Conference on Trade and
Development, 137 out of 194 countries and regions have launched data
protection and privacy legislation [53]. These requirements outline data
practices that should be included in privacy notices. We select five privacy
regulations, such as GDPR [5], CCPA [6], CalOPPA [30], COPPA [31], and APP
[7], to understand the data practices from the administration’s point of view.
If users notice a specific data type, we denote its existence.

_Industry._ Mobile app platforms commonly have specific privacy notice
requirements for curated apps in the market, especially for the involved data
types. For example, the Google Play app store defined 14 main data categories
and 39 sub-categories for their data safety reports that all apps need to
claim[32]. Those standards have been developed through extensive user studies
and market testing. Therefore, market standards can reflect the users’
concerns and expectations in practice.

_Academia._ Researchers have studied the privacy policies for software
applications from various perspectives, including content categorization and
extraction [38, 40, 42, 45], automated generation [37, 44, 46], and compliance
analysis [41, 42, 43, 47]. As there are differences among privacy policies,
corpus selections, and tasks among numerous problems, their data types vary as
well. By surveying the academic research literature, we summarize the data
types covered by common privacy policies.

TABLE II: The identification of privacy-related contexts.

Data types | Description | Keywords list | Related RICO-icon class | Icon examples
---|---|---|---|---
Name | How a user refers to themselves |  name, first name, last name, full name, real name, surname, family name, given name | n.a. | n.a.
Birthday | A user’s birthday | birthday, date of birth, birth date, DOB, birth year | n.a. | n.a.
Address | A user’s address |  mailing address, physical address, postal address, billing address, shipping address, residential address, residence, personal address | n.a | n.a
Phone | A user’s phone number |  hone, phone number, mobile, mobile phone, mobile number, telephone, telephone number, call | [43] Call |
Email | A user’s email address  | email, e-mail, email address, e-mail address | [6] Email |
Profile | A user’s account information  | profile, account | [49] Avatar |
Contacts |  A user’s contact information, or the access to the contact permission | contacts, phone-book, phone book, device’s address book | [68] Group, [3] Follow |
Location |  A user’s location information, or the access to the location permission |  location, locate, geography, geo, geo-location, precision location |  [40] Location crosshair, [72] Location |
Photos |  A user’s photos, videos, or the access to the camera permission |  camera, photo, scan, album, picture, gallery, photo library, storage, image, video, scanner, photograph |  [42] Photo, [56] Videocam, [82] Wallpaper |
Voices |  A user’s voices, recordings, or the access to the microphone permission | microphone, voice, mic, speech, talk | [91] Microphone |
Financial info |  Information about a user’s financial accounts, purchases, or transactions |  credit card, company, companies, organization, organizations, pay, payment | [61] Cart |
Social media |  A user’s social media information, or the access to social media accounts | social media, Facebook, Twitter, socialmedia, share | [77] Facebook, [89] Twitter |

###  II-B Multi-modal Privacy-related Contexts

Privacy-related contexts encompass the areas within GUIs where data practices
could be applicable. In this paper, we identify two types of GUI components
that may collect personal information through human-device interactions:
literal texts and graphical icons. On one hand, literal texts are often
considered the most fundamental information sources in mobile app GUIs. On the
other hand, GUIs are specifically designed to convey essential information as
efficiently as possible. More graphical icons are employed to replace literal
texts, as these icons are concise and provide users with the necessary
information at a glance, significantly reducing the effort to read the literal
texts [54].

_Keyword collection._ After reviewing literal texts describing data types, we
build a keyword list for contextual privacy policies. Specifically, we collect
keywords from the academic research literature. We exclude ambiguous words,
such as “address” since they may represent residential addresses or email
addresses. To improve unification and consistency, the list is made case-
insensitive. The complete keyword list is shown in Table II.

_Icon collection._ We notice that icons often appear on the app interface as
representatives of a specific type of privacy information. For instance, the
pattern of the camera usually relates to the acquisition of the camera’s
permission by mobile apps. Liu et al. [55] proposed an icon taxonomy with 99
common classes. They also provided one of the largest icon datasets, with over
118K icons. We then semantically match their RICO-icon classes with the data
types identified from the survey (See Table II).

##  III Cpp4App: The Benchmark Dataset

Figure 2: An overview of SeePrivacy.

There are some existing datasets for mobile GUI understanding [56, 57, 58];
however, we realize two main challenges in the current mobile app datasets
that drive us to collect a new dataset specifically for automatically
generating contextual privacy policies:

  * •

Temporal inconsistency between GUIs and privacy policies. As the privacy
practices of mobile apps are constantly changing, their corresponding privacy
policies also need to be updated. Moreover, the GUIs of mobile apps also
evolve rapidly. For these reasons, it is crucial to maintain consistency
between screenshots (GUIs) and their temporally corresponding privacy policies
to ensure accuracy and reliability. In current GUI datasets, such as Rico
[58], most privacy policies for mobile apps are either outdated or not
publicly available.

  * •

No annotations about privacy-related contexts in GUIs. As far as we know, we
are the first to propose a framework that automatically generates contextual
privacy policies for mobile apps. To evaluate the performance of the proposed
framework, we expect benchmark datasets to contain two features, i.e., the
bounding boxes of privacy-related contexts on the GUI screenshots and the
mapping relations between privacy-related contexts and privacy policy
segments. However, there are no existing benchmark datasets that include these
two features.

In this work, we need to construct a benchmark dataset to accurately evaluate
the effectiveness of generating contextual privacy policies. We present steps
to create the benchmark dataset in the following subsections.

###  III-A Mobile Apps Selection

Android mobile apps are often transparent and favorable to academic research
[59, 60]. There are numerous app markets accessible for the Android platform;
however, the Google Play app store is the most extensive and easily
accessible, offering over two million apps [61, 62]. Moreover, this store is
one of the leading stores requiring mobile apps to provide a privacy policy
link on their homepages [63]. Therefore, users can directly obtain privacy
policy links when downloading Android mobile apps from the Google Play app
store.

According to previous studies, popular mobile apps are more likely to have
rigorous privacy practices [42, 64, 65]. We employ information from AppBrain
[66], one of the largest third-party platforms monitoring app statistics for
the Google Play app store, to identify the most popular 200 mobile apps. We
follow three strategies to collect mobile apps for our paper. First, the
Google Play app store assigns a specific category to each mobile app.
Moreover, mobile apps from various categories vary in GUIs, functions, and
privacy practices. To ensure the diversity of mobile apps, we collect the most
popular apps across 14 distinct categories. Second, we exclude mobile apps
that have privacy policies smaller than 200 words or smaller than 2 KB, as
they may contain invalid privacy policies [65, 67]. Lastly, we filter out
mobile apps with non-English privacy policies. The selection criteria drive 30
candidates’ mobile apps for following fine-grained annotation.

###  III-B Privacy Policy Segment Annotation Strategy

For each data type, we ask our annotators to read the privacy policies and
select relevant segments for each privacy-related context. To guarantee the
quality of the annotation process, we set a maximum reading speed of 250 words
per minute for annotators. If the annotator is unable to identify any relevant
segments from privacy policies, then it will be labeled as “No relative
information is found in the privacy policy”.

###  III-C Privacy-related Context Annotation Strategy

Based on the analysis of mobile app GUI compositions [68, 69], we collect
specific pages of each mobile app to ensure its essential privacy-related GUIs
are included in our dataset. Specifically, the pages are listed as follows:

  * •

_Registration/Login page._ Mobile apps often collect user registration
information on this page to create a new account, which may include personal
and other private information. Certain mobile apps mandate new users to
register before accessing their features, making it impossible for these users
to bypass the registration or login page.

  * •

_Homepage._ This is the first page that appears after entering a mobile app.
It contains essential features and information related to its app’s
functionality, which may require you to provide personal information or grant
device permissions. Additionally, we also gather screenshots of the main pages
next to the homepage.

  * •

_Profile page._ This page may store the personal information of users. For
example, some mobile apps may request permission to access the camera or photo
roll to provide customization options for the account’s public appearance.

  * •

_Setting page._ This page may involve privacy-related customization, such as
sensor permissions.

  * •

_Onboarding page._ This page is considered a tutorial or introduction page
that presents the app’s features and benefits.

  * •

_Map page._ We visit this page in various categories of mobile apps, such as
travel, posting, food delivery, and shopping. These mobile apps tend to
collect users’ location information on this page.

  * •

_Essential pages for specific categories of apps._ These include the catalog
page and product card page in shopping apps, the post editing page in forum
apps, and the feed page in social media apps.

For each screenshot of its page, annotators need to manually read all the
texts and check whether they contain any word in our CPP-related keyword list
mentioned in Table II. If the text contains CPP-related keywords, it will be
labelled as the corresponding data type and marked with a bounding box. Then,
annotators need to detect all icons displayed on the screenshot if the icon
belongs to a privacy-related data type. If so, the icon will be labelled as
the corresponding class and indicated with a bounding box. We employ the
OpenCV-Python library to draw all bounding boxes.

TABLE III: The statistical analysis of benchmark dataset. No. Mobile applications & Privacy policies | 30
---|---
No. Screenshots | 230
Screenshots per Mobile application | 7.67
No. Privacy-related contexts | 644
Contexts per Mobile application | 21.5
No. Words in Privacy policies | 141,292
Words per Privacy policy | 4,709

###  III-D Setups and Statistics

To guarantee the consistency between GUIs and privacy policies during this
work, an Android mobile phone, i.e., the OnePlus 7 Pro, Android 11, and
Hydrogen OS 11.0.9.1.GM21, is employed to download all the selected mobile
apps. Their corresponding privacy policies are downloaded simultaneously. Then
the auto-update features in the Google Play app store and mobile apps are
turned off. Also, the auto-update over Wi-Fi feature is disabled in the
system’s general settings on this mobile phone.

To select pages of each mobile app, two annotators explore the mobile app and
take screenshots of these pages. In total, we have 230 screenshots from 30
mobile apps. For each screenshot, the two annotators label privacy-related
contexts and their corresponding privacy policy segments independently. If the
annotators have any labeling conflicts, they discuss them and reach a
consensus. In total, we have 644 labeled privacy-related contexts, with each
context corresponding to a privacy policy segment, and we name this benchmark
dataset as Cpp4App. The statistical analysis is shown in Table III.

##  IV SeePrivacy: The Framework

In this section, we present our multi-modal framework, namely SeePrivacy, that
automatically generates contextual privacy policies (CPPs) for given mobile
apps (see Fig. 2). The framework takes privacy policies and screenshots from a
given mobile app as inputs. These inputs are fed into three main modules to
automatically construct CPPs for the mobile app. We list the three main
modules of our framework as follows:

  * •

Context Detection Module: This module aims to locate contexts in which we
should display contextual privacy policies from a given mobile app’s
screenshot.

  * •

Segment Extraction Module: This module aims to extract relevant privacy policy
segments from a given mobile app’s privacy policy.

  * •

CPPs Presentation Module: This module maps privacy policy segments to their
corresponding context, rendering the key information, to eventually generate
contextual privacy policies for mobile apps.

Details of each module are explained in the following subsections.

###  IV-A Context Detection Module

Our framework only takes a given screenshot to detect privacy-related contexts
without obtaining the source code or extra run-time APIs, therefore our
framework is easier to deploy and has high adaptability. We choose visual
components instead of widgets as atomic elements in our GUI understanding
since some informative widgets are too coarse-grained to be a context. To
localize the visual components on the screenshot for potentially indicating or
suggesting personal information collection, we propose a divide-and-conquer
method that can detect texts and icons separately and identify the contexts
together.

Textual GUI components. To detect and localize privacy-related texts from a
given screenshot, we apply a state-of-the-art ultra-lightweight OCR system,
namely PaddlePaddle-OCR (PP-OCR) [70], for the mobile scenario. After
obtaining these texts, we employ a pre-trained Large Language Model, i.e.,
GPT-3.5, to classify their data types (see Table II). Specifically, we first
inject the data type list and their descriptions displayed in Table II, and
then for every {detected text}, we propose the following prompt:

Question: Is this piece of text “{detected text}” related to any following
privacy information data types? Or not relevant to any of them? ONLY answer
the data type or ”not relevant”. ONLY use the provided data type list. Answer:

If the answer contains a data type, we will regard the {detected text} as
relevant to this data type; otherwise, it does not belong to any data type.

Iconic GUI components - Localization. To localize iconic GUI components from
their screenshots, we follow previous work [71] and propose a new set of
detection rules to filter out irrelevant objects. Specifically, we have four
rules presented as follows: (a) Filtering out elements whose area is greater
than 10% of the total area of the screenshot, aiming to remove the bounding
boxes and large images; (b) Filtering out elements whose area is less than 5%
of the total area of the screenshot, aiming to remove the small noises; (c)
Removing non-squarish elements whose aspect ratio (width to height) is less
than 0.6, ruling out control bars and top/bottom banners; (d) Removing
elements that area is overlapped with OCR results since some letters in large
font-size are also detected as potential icons. By applying these rules to our
dataset, we can obtain the iconic GUI components’ position on their
screenshots.

Iconic GUI components - Classification. CNN-based neural networks have a
strong capability of recognizing patterns on GUIs [55, 72, 73]. In this paper,
we train a ResNet-based network [74] (a variant of CNN-based network) to
classify the detected graphical icons from the screenshots. Specifically, we
employ ResNet-18 as the backbone and further enhance generalizability; i.e.,
we add dropout layers to every ResNet block to enhance its generalizability,
preventing the potential over-fitting issue. We then train the ResNet model on
the RICO-icon dataset with the default train-test split. After the training
process, we apply the ResNet model to our dataset to obtain the corresponding
data types of iconic GUI components as mentioned in Table II.

###  IV-B Segment Extraction Module

Most mainstream app markets, such as the Google Play app store [63] and the
Apple App Store [75], require developers to provide a privacy policy link for
their app on the homepage; hence, we can obtain those links based on a mobile
app’s name. We employ Selenium and BeautifulSoup, two Python libraries, to
capture these privacy policies in HTML format based on their privacy policy
links. To facilitate multilingual users and promote mobile apps in a global
market, we observe that an increasing number of privacy policies are displayed
in multiple languages. We utilize a language detection library, namely
langdetect to exclude non-English language texts from privacy policies to
dispel ambiguity. Note that our framework can be easily extended to other
languages.

Our ultimate goal is to obtain privacy policy segments with their
corresponding privacy-related contexts. According to the Flesch-Kincaid
readability formula [76], the readability of words is positively related to
their length. Document-level or paragraph-level privacy policies analysis is
too coarse-grained for our task because we aim on retrieving the relevant
sentence-based segments. In addition, previous studies [77, 47] have discussed
that only using sentence-level privacy policies processing and analysis may
lead to contradictory conclusions, since the same or similar sentences in
different sections may have different implications. Therefore, we adopt the
multi-level privacy policies processing method proposed in [78, 47].

For privacy policy documents whose HTML structure follow the
$(\langle\texttt{Heading}\rangle\langle\texttt{Paragraph}\rangle^{+})^{+}$
format, We reuse the pre-trained Bayesian multi-label classifier model [47] to
classify privacy policy paragraphs based on their headings since headings
usually contain words that can be used as identifiers. The classifier is
claimed to achieve 0.85 accuracy and can classify the majority of the
documents based on their headings. Paragraphs classified as Types are related
to the “types of personal data collected by the current app”. Those paragraphs
are highly related to the specific data practices, so we only conduct
sentence-level analysis on them.

For privacy policy documents whose HTML structure does not follow the
$(\langle\texttt{Heading}\rangle\langle\texttt{Paragraph}\rangle^{+})^{+}$
format, we follow previous work [39, 29] to classify paragraphs about specific
data practices. Specifically, we train a CNN-based multi-label classification
model on a large privacy policy dataset [38] with 23K fine-grained data
practice annotations. Note that this dataset contains 12 high-level categories
for data practices, including First-Party Collection/Use and Third-Party
Sharing/Collection. We set the probability threshold at 0.5, i.e., privacy
segments with predicted probability scores above 0.5 are classified as
corresponding data practices. The results show we can achieve a 0.84 top-1
precision for classifying the privacy segments with their related data
practices.

After obtaining related privacy segments, we tokenize them into sentences by
employing the Stanza Python library. We then conduct a two-stage sentence-
level analysis to extract the sentences for constructing CPPs for each data
type. First, we perform a keyword search based on our collected keyword list
(see Table II). For 1-gram keywords, we perform a case-insensitive word-wise
comparison and check whether a keyword exists as sub-strings in a “word”, e.g.
location and “… your locationWe may share…”, since some punctuation or line
break marks cannot be detected in rare cases. For keywords with two or more
grams, we perform case-insensitive string-wise comparison, i.e., treating the
target sentence as a string and searching identical n-gram keywords. If a
keyword exists, the sentence will be added to the corresponding privacy policy
segments of its data type. Second, for sentences without any keywords, we
further employ a Bayesian binary classifier to determine whether the sentence
is related to a data type. The classifier is trained on the dataset from [47],
reaching an accuracy of 0.98. If yes, we use SpaCy Python library to obtain
noun chunks from the sentence. A noun chunk is a phrase that includes a noun
and any connected words such as its adjectives. We then calculate the phrases
similarity $phrase\\_sim(\,)$ between noun chunks and keywords, and the
formula is:

$phrase\\_sim(p_{1},p_{2})=\frac{2\times
path\\_similarity(p_{1},p_{2})}{word\\_count({p_{1}})+word\\_count(p_{2})}$

where the $p_{1}$ and $p_{2}$ are a noun chunk and a keyword, respectively;
$path\\_similarity(\,)$ is the function from WordNet [79], reflecting the
similarity based on parsing their semantic constituency trees’ structure;
$word\\_count(\,)$ is to count the number of words. If the similarity between
a noun chunk and a keyword is higher than 0.8, an empirically set threshold,
then we regard that the sentence that the noun chunk belongs to is the data
type of the keyword.

Above all, we obtain the sentences related to each data type and group them as
privacy policy segments. We also record the positions of keywords or noun
chunks in the resulting sentences. For data types that do not match any
sentences or paragraphs, we simply add ”No relative information is found in
the privacy policy.” to notify users.

###  IV-C CPPs Presentation Module

For each privacy policy and its screenshot, we obtain their privacy policy
segments and privacy contexts from the segments extraction module and contexts
detection module, respectively. We then further process them for a more
organized presentation to users.

First, we group the detected contexts of the same data types and map them to
their corresponding privacy policy segments. For example, in Figure 1c, our
framework, namely SeePrivacy, detects three contexts, including two textual
GUI components, i.e., “Share your location” and “use your birthday,” and one
iconic GUI components, such as a location mark icon in the right middle of the
screenshot. SeePrivacy classifies them as Location, Location, and Birthday
data types individually. The text “Share your location” and the location mark
icon are coupled with the segment about location information. The text “use
your birthday” solely corresponds to the segment about Birthday data type. A
previous study [80] points out that appropriately setting bold fonts for text
will significantly increase its readability. Hence, for the privacy policy
segments displayed to the users, we render the keywords or noun chunks in bold
fonts based on their positions.

##  V Evaluation and Discussion

TABLE IV: Performance of SeePrivacy’s modules. CDM stands for Context
Detection Module.

Category | Accuracy | Precision | Recall
---|---|---|---
Name | 0.92 | 0.92 | 1.00
Birthday | 1.00 | 1.00 | 1.00
Address | 0.38 | 0.43 | 0.79
Phone | 0.90 | 0.96 | 0.93
Email | 0.67 | 0.97 | 0.68
Profile | 0.45 | 0.47 | 0.94
Contacts | 0.95 | 0.98 | 0.98
Location | 0.89 | 0.95 | 0.94
Photos | 0.78 | 0.97 | 0.80
Voices | 1.00 | 1.00 | 1.00
Financial info | 0.82 | 0.93 | 0.88
Social media | 0.60 | 0.75 | 0.75
Average | 0.78 | 0.86 | 0.89

(a) CDM - Textual GUI components

RICO-icon Class | Category | Accuracy | Precision | Recall
---|---|---|---|---
Call | Phone | 0.70 | 1.00 | 0.70
Email | Email | 0.74 | 0.68 | 0.81
Avatar | Profile | 0.85 | 0.82 | 0.89
Follow | Contacts | 0.90 | 0.82 | 1.00
Group | Contacts | 0.67 | 0.73 | 0.62
Location | Location | 0.89 | 0.86 | 0.92
Location crosshair | Location | 0.72 | 0.63 | 0.83
Photo | Photos | 0.87 | 0.89 | 0.85
Wallpaper | Photos | 0.87 | 0.82 | 0.93
Videocam | Photos | 0.91 | 0.83 | 1.00
Microphone | Voices | 0.96 | 0.93 | 1.00
Cart | Financial Info | 0.81 | 0.75 | 0.88
Facebook | Social media | 0.83 | 0.75 | 0.92
Twitter | Social media | 0.90 | 0.95 | 0.86
Average |  | 0.82 | 0.83 | 0.85

(b) CDM - Iconic GUI components

Category | Accuracy | Precision | Recall | Success Rate
---|---|---|---|---
Name | 0.98 | 0.97 | 1.00 | 0.92
Birthday | 0.94 | 1.00 | 0.94 | 0.62
Address | 0.48 | 0.82 | 0.50 | 0.78
Phone | 1.00 | 1.00 | 1.00 | 0.97
Email | 1.00 | 1.00 | 1.00 | 0.98
Profile | 0.96 | 0.96 | 1.00 | 0.97
Contacts | 1.00 | 1.00 | 1.00 | 1.00
Location | 0.92 | 0.94 | 0.98 | 0.91
Photos | 0.84 | 0.83 | 1.00 | 1.00
Voices | 0.85 | 0.33 | 1.00 | 1.00
Financial info | 1.00 | 1.00 | 1.00 | 0.96
Social media | 1.00 | 1.00 | 1.00 | 1.00
Average | 0.92 | 0.94 | 0.98 | 0.96

(c) Segments Extraction Module

In this section, we mainly investigate and discuss the following two research
questions:

  * •

RQ1: What is the performance of SeePrivacy in automatically generating
contextual privacy policies for mobile apps?

  * •

RQ2: What is the usability of SeePrivacy in practice?

###  V-A RQ1: Performance Evaluation

As our framework is composed of three main modules, we evaluate the
performance of each of them on our benchmark dataset, Cpp4App.

####  V-A1 Context Detection Module

To determine whether a detected GUI component matches the ground truth, we
introduce Intersection over Union (IoU), which is a commonly used evaluation
metric to measure the performance of an object detection model. If the IoU
between ground truth and the detected component area is greater than a
threshold $\beta$, then we regard it as a match. We set $\beta=0.5$.

Textual GUI components. Table IVa presents the findings regarding the
detection of textual GUI components. In general, our approach demonstrates a
commendable ability to accurately recognize and classify texts on mobile GUI
screenshots, achieving an accuracy of 0.78, precision of 0.86, and recall of
0.89. Regarding specific categories, certain categories, such as Address,
exhibit performance below the average level due to their susceptibility to
being misclassified as semantically similar categories, like Location.
Furthermore, it is noteworthy that the accuracy metric yields considerably
lower results compared to the other metrics. To investigate the underlying
causes, we select a random sample comprising 5% of screenshots from the
benchmark dataset and manually assess the performance of PP-OCR. The results
indicate an H-mean score of over 0.98 for text detection and an accuracy of
over 0.98 for text recognition. Additionally, we conduct a detailed analysis
of the failure cases of chatGPT, which reveals its inclination to classify a
piece of text as ”not relevant”. We observe that when we exclude the
descriptions of each data type from the chatGPT, the occurrence of ”not
relevant” diminishes, but this change also leads to a significant decline in
overall performance.

Iconic GUI components. Table V shows the comparison of the icons localization
capability between our method and previous methods. In our experiment, we
manually count the number of icons from the benchmark dataset and compare
them. Results show that our proposed method has better performance in
identifying icons curated on GUI screenshots. We manually examine failure
cases and find that most of the missing labels are caused by overlapping with
other components or interleaved boundaries. As for the mislabelled cases,
small squarish images, e.g., item images in shopping apps, tend to be wrongly
included as an icon. Fig. 3 shows two examples of failure cases in our
experiment.

TABLE V: Performance comparison between iconic GUI components localization methods. Method | Precision (mislabel) | Recall (missing label)
---|---|---
UIED [71] | 0.78 | 0.44
Liu et al. [55] | 0.79 | 0.86
Ours | 0.84 | 0.88

(a)

(b)

Figure 3: Failure cases of the mislabel (left) and the missing label (right).

We also compare the icon classification models on the RICO-icon dataset. Table
VI shows that our model achieves competitive performance but much better
($\sim$14$\times$) efficiency compares to the baseline. Moreover, the class-
wise breakdown is shown in Table IVb.

TABLE VI: Performance and efficiency comparison for iconic GUI elements classification on the RICO-icon dataset. Time(s) represents the elapsed seconds to process a thousand icons.  Model | Accuracy | Precision | Recall | Time(s)
---|---|---|---|---
Liu et al. [55] | 0.90 | 0.90 | 0.79 | 5.02
Ours | 0.91 | 0.80 | 0.81 | 0.37
TABLE VII: Questions for the user study. Except $\textit{Q}_{2}$ and
$\textit{Q}_{9}$, all other questions are on a 5-point Likert scale.

No. | Question | Scale
---|---|---
$\textit{Q}_{1}$ | How concerned are you about your privacy information while using mobile apps? | 5 for very concerned, 1 for very unconcerned
$\textit{Q}_{2}$ | Have you been troubled by the privacy related issues when using mobile apps? | Yes, No, or Maybe
$\textit{Q}_{3}$ | Do you read mobile app’s privacy policies when you encountered? | 5 for always read, 1 for never read
$\textit{Q}_{4}$ | Does the data type match with the detected context? | 5 for strongly agree, 1 for strongly disagree
$\textit{Q}_{5}$ | Does the data type match with the displayed privacy policy segment? | 5 for strongly agree, 1 for strongly disagree
$\textit{Q}_{6}$ | Does the displayed privacy policy segment explain the detected context on the screenshot? | 5 for strongly agree, 1 for strongly disagree
$\textit{Q}_{7}$ | What do you think the usefulness of this tool in terms of providing privacy information for mobile apps? | 5 for very useful, 1 for very useless
$\textit{Q}_{8}$ | Will you read contextual privacy policies when you encountered in future when you encountered? | 5 for always read, 1 for never read
$\textit{Q}_{9}$ | What suggestions do you have for this tool? | Completion question

Figure 4: The percentage values of participants’ agreement to our three
statements on a 5-point Likert scale.

Overall context identification. We also combine the textual and iconic GUI
components and evaluate the overall context to better understand the
performance of our proposed framework. Table VIII shows the overall context
identification results considering textual and iconic GUI components.
Specifically, SeePrivacy can achieve 0.76, 0.86, and 0.87 in terms of
accuracy, precision, and recall, respectively. We also estimate the
performance of overall context identification as per a screenshot, denoted by
a context coverage rate. Specifically, the context coverage rate calculates
how many data types the contexts belong to that can be detected by our method.
For example, in Fig. 5, there is one iconic GUI component (green bounding box)
and three textual GUI components (red bounding boxes). Since all those
components belong to Location, if any of those components are not detected,
the context coverage rate is still 1.0 for this screenshot. Even if our
framework fails to detect some contexts, users can still read their
corresponding privacy policy segments via other contexts. Overall, our method
yields an 83.6% context coverage rate.

TABLE VIII: Category-wise overall context identification.

Category | Accuracy | Precision | Recall
---|---|---|---
Name | 0.92 | 0.92 | 1.00
Birthday | 1.00 | 1.00 | 1.00
Address | 0.38 | 0.43 | 0.79
Phone | 0.85 | 0.91 | 0.94
Email | 0.67 | 0.90 | 0.72
Profile | 0.62 | 0.64 | 0.95
Contacts | 0.87 | 0.98 | 0.88
Location | 0.84 | 0.95 | 0.87
Photos | 0.78 | 0.95 | 0.81
Voices | 0.89 | 1.00 | 0.89
Financial info | 0.74 | 0.93 | 0.78
Social media | 0.60 | 0.75 | 0.86
Average | 0.76 | 0.86 | 0.87

Figure 5: Screenshot 17-2

####  V-A2 Segment Extraction Module

Table IVc shows the results of the segment extraction module. In the benchmark
dataset, some mobile apps’ data types are not relevant to any privacy policy
segment (“No relative information is found in the privacy policy”), so the
framework is expected to not retrieve anything either. Thus, the
classification metrics (accuracy, precision, and recall) reflect privacy
policy segment retrieval performance for each data type. The success rate
denotes how many retrieved privacy policy segments match their ground truth.
Specifically, the majority of a successfully retrieved privacy policy segment
is identical to the ground truth, thus we introduce the segment similarity
$segment\\_sim(\,)$ to decide whether retrieval is successful. For given
ground truth segment $s_{gt}$ and retrieved segment $s_{ret}$:

$\displaystyle segment\\_sim(s_{ret},s_{gt})$
$\displaystyle=\frac{1}{\textsc{Min}(n,m)}\sum_{i=1}^{n}\sum_{j=1}^{m}\overline{lcs}(p^{ret}_{i},p^{gt}_{j})$
$\displaystyle\overline{lcs}(p^{ret}_{i},p^{gt}_{j})$
$\displaystyle=\frac{lcs(p^{ret}_{i},p^{gt}_{j})}{\textsc{Min}(\textsc{Len}(p^{ret}_{i}),\textsc{len}(p^{gt}_{j}))}$

where $lcs(\,)$ stands for the function to calculate longest common string;
$p^{ret}_{j}$ are phrases separated by punctuation of retrieved segments and
$p^{gt}_{i}$ are labelled phrases of ground truth segments. If it is greater
than 0.8, then the segment pair is regarded as a successful match.

From the results, we can observe that our framework generally demonstrates
strong performance in retrieving not only the privacy policy segments but the
relevant ones. There are some categories, such as Address, which are more
complex than the other categories due to relatively ambiguous keywords. In
addition, there are too little data (less than 5 records) for Voices according
to the benchmark dataset, thus the limited performance of this category could
be caused by the randomness.

###  V-B RQ2: User study

To validate the functionality and demonstrate the usefulness of our tool in
practice, we conduct a usability evaluation to comprehensively examine
SeePrivacy for generating contextual privacy policies for mobile apps.

####  V-B1 User study design

Table VII lists all the questions used in our user study. We recruit
participants based on the following criteria: being over 18 years old, fluent
in English reading, and capable of using mobile apps without special
accessibility tools (e.g., VoiceOver on iOS) via mailing lists of authors’
institutions. In addition to the general introduction and consent information
of this project, we also provide descriptions of the data types, specific
instructions, and an example. Our questions cover three aspects: privacy
policies reading habits for mobile apps ($\textit{Q}_{1\text{-}3}$), specific
usability evaluation ($\textit{Q}_{4\text{-}6}$), and general usability
evaluation ($\textit{Q}_{7\text{-}9}$) of the SeePrivacy.

As for the specific usability evaluation, we randomly sample 30 screenshots
with their privacy policies from the benchmark dataset and employ SeePrivacy
to generate CPPs. We then divide the generated CPPs into three folds and
compose the questionnaire for each fold, i.e., ten screenshots in each
questionnaire as manual examination involves very heavy privacy policy reading
and comprehending tasks. Then the participants need to answer
$\textit{Q}_{4\text{-}6}$ for all the ten allocated screenshots, and they are
asked to rate the agreement level on a 5-point Likert scale ranging from five
(strongly agree) to one (strongly disagree).

####  V-B2 User study results

In total, we recruit 15 people for the study (6 females, 8 males, and 1 who
preferred not to be mentioned). Each screenshot fold is evaluated by five
people. We first analyze how participants rate the statements in a specific
usability evaluation. Figure 4 and Table IX show the results of our user
study. The matching between data types and detected contexts received high
ratings, with over 81% of their responses being either “Agree” or even
“Strongly Agree,” and its values for the mean (4.22) and median (5) are also
high. As for data types and privacy policy segments, over 72% of participants
gave positive ratings, and their mean and median are 3.95 and 4, respectively.
More than 77% of participants do not disagree that retrieved privacy policy
segments explain detected contexts, which is underlined by a 3.75 mean and a 4
median. Overall, SeePrivacy exhibits excellent functionality to deliver
contextual privacy policies to users.

According to our user study, we found that people are concerned about their
privacy information while using mobile apps (4.13 mean and 4 median for
$\textit{Q}_{1}$). However, they are commonly disturbed by issues about
privacy information (13 out of 15 answered “Yes” or “Maybe” for
$\textit{Q}_{2}$). The usefulness of SeePrivacy is highly regarded in terms of
providing privacy information on mobile apps. 14 out of 15 participants rated
“Very useful” or “Useful”. Moreover, we also compare the reading habits of
common privacy policies and CPPs. Results show that people are much more
willing to read CPPs (a mean increase from 2.00 to 4.07), indicating that our
SeePrivacy is essential and significant in helping people overcome the
“privacy policy reading phobia”.

TABLE IX: Statistic results of our user study. SD stands for standard
deviations.

No. | Topic | Mean | Median | SD
---|---|---|---|---
$\textit{Q}_{4}$ | Data type & Context | 4.22 | 5 | 1.14
$\textit{Q}_{5}$ | Data type & Policy segment | 3.95 | 4 | 1.28
$\textit{Q}_{6}$ | Policy segment & Context | 3.75 | 4 | 1.40
$\textit{Q}_{1}$ | Privacy concerns (-) | 4.13 | 4 | 1.09
$\textit{Q}_{7}$ | CPP usefulness (+) | 4.53 | 5 | 0.62
$\textit{Q}_{3}$ | PP Reading frequency | 2.00 | 2 | 1.03
$\textit{Q}_{8}$ | CPP Reading frequency | 4.07 | 4 | 0.85
Improvement | +2.07 | +2 | /

##  VI Threats to Validity

Internal validity. Threats to internal validity refer to errors in our
experiments. We reuse several pre-trained classifiers to process privacy
policies in the paragraph- and sentence-level analyses. We employ PP-OCR and
chatGPT for textual GUI element detection. We also follow OpenAI’s API to
access chatGPT. We specifically evaluate the performance of every module on
the benchmark dataset and conduct the error analysis. We also propose a user
study to further validate the functionality and usability of our framework in
practice. We have rechecked the code and data, but errors may still exist.

External validity. Threats to external validity refer to the generalizability
of our proposed method. In our experiments, although our benchmark dataset
covers various application categories, it is mainly composed of relatively
popular mobile applications in each category. However, we maneuver a set of
selection rules to collect different pages in applications as completely as
possible. In addition, our method is based on learning-based GUI pattern
understanding and privacy policy analysis, which is well-known for its
generalizability. Thus, we believe that there is a minimal threat to external
validity.

##  VII Related Work

Vision-based mobile GUI understanding. Previous researchers have been
intensively studying this topic from different perspectives. White et al. [81]
leverage a famous object detection model called YOLOv2, to detect GUI widgets
in-app screenshots for testing. Chen et al. [82] apply Faster R-CNN to obtain
GUI widgets from app screenshots to construct a GUI widget gallery. In another
work, Chen et al. [83] propose a hybrid method that combines traditional rule-
based and learning-based CV techniques, achieving the start-of-the-art
performance for GUI widget detection. Xie et al. [84] present a novel
unsupervised image-based method for inferring perceptual groups of GUI
widgets.

Privacy policy analysis. Wilson et al. [38] build a privacy policy corpus with
23K fine-grained data practice annotations. Kaur et al. [40] perform a
comprehensive analysis of keywords and content of over two thousand online
policy documents. Zimmeck et al. [42] introduce the Mobile App Privacy System
(MAPS) for conducting an extensive privacy census of Android apps. The
evaluation of a million Android apps reveals policies that do exist are often
silent on the practices performed by apps. Amos et al. [12] curate a
comprehensive dataset of a million privacy policies spanning over two decades,
and their analyses “paint a troubling picture” of the privacy policy’s
transparency and accessibility.

The most closet work is [29], called PrivacyInjector, used to automatically
generate contextual privacy policies for websites. However, their context
detection method is intrusive and based on ad-hoc HTML analysis, which is not
applicable to mobile applications. In addition, their definitions of privacy-
related contexts are more coarse-grained compared to this work and specific to
website scenarios.

##  VIII Conclusion and Future Work

Privacy policies are extensively criticized due to their poor readability.
Contextual privacy policies are proposed to solve this research challenge. We
first construct the benchmark dataset, namely Cpp4App, containing over 600
privacy-related contexts with their corresponding privacy policy segments in
the mobile scenario, based on our comprehensive survey. We then present a
novel framework, called SeePrivacy, to automatically generate contextual
privacy policies for mobile apps to respond to this need. We utilize computer
vision techniques and a pre-trained Large Language Model, i.e., chatGPT, to
detect privacy-related contexts on mobile GUI screenshots and leverage NLP
techniques to extract corresponding segments from the privacy policy document.
Our evaluation results show that our method yields an 83.6% coverage rate for
privacy-related context detection and an accuracy of 0.92 for extracting
corresponding policy segments. Moreover, 96% of the retrieved policy segments
can be correctly matched with their contexts. Additionally, we perform a user
study to further validate the functionality and usability in practice. The
results show that more than 77% of participants do not disagree that retrieved
privacy policy segments explain detected contexts, which is underlined by a
3.75 mean and 4 median on a 5-point Likert scale. The usefulness rating
received an average of 4.53 out of 5 from participants. We believe that our
work provides a new research direction for the community, sheds light on
developing contextual privacy policies for mobile apps, and effectively
assists users in comprehending privacy notices.

The promising results of this work illuminate two potential directions for
future research. First, some participants ($\textit{P}_{7}$ and
$\textit{P}_{15}$) suggested that aside from CPPs, they also care about
general data rights such as the Right To Be Forgotten or Right to Control. We
plan to extend our framework to be capable of extracting and presenting data
rights information from privacy policies. Second, although our original
intention is to deliver the original privacy policy content without any
modification, some retrieved policy segments are too long to comprehend in a
timely manner. We believe it is worthwhile to evaluate how to digest (e.g., by
pre-trained Large Language Models) lengthy policy segments with minimum
information loss and distortion to maintain their integrity and practicability
simultaneously.

Data Availability. Our dataset and codes are available at:
https://github.com/Cpp4App/Cpp4App.

## References

  * [1] (2023) Stack overflow - how to request all permission needed when the time user install the app in google play? Accessed: 2023-04-20. [Online]. Available: https://stackoverflow.com/questions/44985078/
  * [2] E. the Department of Health and Welfare. (1973) Fair information practice principles (fipps). Retrieved: 2023-04-05. [Online]. Available: https://www.fpc.gov/resources/fipps/
  * [3] M. K. Landesberg, T. M. Levin, C. G. Curtin, and O. Lev, “Privacy online: A report to congress,” _NASA_ , no. 19990008264, 1998.
  * [4] J. A. Obar and A. Oeldorf-Hirsch, “The biggest lie on the internet: Ignoring the privacy policies and terms of service policies of social networking services,” _Information, Communication & Society_, vol. 23, no. 1, pp. 128–147, 2020.
  * [5] (2018) General data protection regulation (GDPR). Retrieved: 2022-04-25. [Online]. Available: https://gdpr-info.eu/
  * [6] (2018) California consumer privacy act of 2018 (CCPA). Accessed: 2023-02-20. [Online]. Available: https://oag.ca.gov/privacy/ccpa
  * [7] (2019) Australian privacy principles (APP). Accessed: 2022-05-03. [Online]. Available: https://www.oaic.gov.au/privacy/australian-privacy-principles
  * [8] J. Caramujo and A. M. R. Da Silva, “Analyzing privacy policies based on a privacy-aware profile: The facebook and linkedin case studies,” in _2015 IEEE 17th Conference on Business Informatics_ , vol. 1. IEEE, 2015, pp. 77–84.
  * [9] K. Kemp, “Concealed data practices and competition law: why privacy matters,” _European Competition Journal_ , vol. 16, no. 2-3, pp. 628–672, 2020.
  * [10] A. J. Perez, S. Zeadally, and J. Cochran, “A review and an empirical analysis of privacy policy and notices for consumer internet of things,” _Security and Privacy_ , vol. 1, no. 3, p. e15, 2018.
  * [11] J. Blakkarly and D. Graham. (2022) Privacy policy comparison reveals half have poor readability. Accessed: 2022-10-10. [Online]. Available: https://www.choice.com.au/consumers-and-data/protecting-your-data/data-laws-and-regulation/articles/privacy-policy-comparison
  * [12] R. Amos, G. Acar, E. Lucherini, M. Kshirsagar, A. Narayanan, and J. Mayer, “Privacy policies over time: Curation and analysis of a million-document dataset,” in _Proceedings of the Web Conference 2021_ , 2021, pp. 2165–2176.
  * [13] P. G. Kelley, J. Bresee, L. F. Cranor, and R. W. Reeder, “A” nutrition label” for privacy,” in _Proceedings of the 5th Symposium on Usable Privacy and Security_ , 2009, pp. 1–12.
  * [14] P. G. Kelley, L. Cesca, J. Bresee, and L. F. Cranor, “Standardizing privacy notices: an online study of the nutrition label approach,” in _Proceedings of the SIGCHI Conference on Human factors in Computing Systems_ , 2010, pp. 1573–1582.
  * [15] P. G. Kelley, L. F. Cranor, and N. Sadeh, “Privacy as part of the app decision-making process,” in _Proceedings of the SIGCHI conference on human factors in computing systems_ , 2013, pp. 3393–3402.
  * [16] C. A. Ciocchetti, “The future of privacy policies: A privacy nutrition label filled with fair information practices,” _J. Marshall J. Computer & Info. L._, vol. 26, p. 1, 2008.
  * [17] L. F. Cranor, “Necessary but not sufficient: Standardized mechanisms for privacy notice and choice,” _J. on Telecomm. & High Tech. L._, vol. 10, p. 273, 2012.
  * [18] S. Pan, T. Hoang, D. Zhang, Z. Xing, X. Xu, Q. Lu, and M. Staples, “Toward the cure of privacy policy reading phobia: Automated generation of privacy nutrition labels from privacy policies,” _arXiv preprint arXiv:2306.10923_ , 2023.
  * [19] L. Cranor, _Web privacy with P3P_. ” O’Reilly Media, Inc.”, 2002.
  * [20] L.-E. Holtz, K. Nocun, and M. Hansen, “Towards displaying privacy information with icons,” in _Privacy and Identity Management for Life: 6th IFIP WG 9.2, 9.6/11.7, 11.4, 11.6/PrimeLife International Summer School, Helsingborg, Sweden, August 2-6, 2010, Revised Selected Papers 6_. Springer, 2011, pp. 338–348.
  * [21] L.-E. Holtz, H. Zwingelberg, and M. Hansen, “Privacy policy icons,” in _Privacy and Identity Management for Life_. Springer, 2011, pp. 279–285.
  * [22] Z. Efroni, J. Metzger, L. Mischau, and M. Schirmbeck, “Privacy icons: a risk-based approach to visualisation of data processing,” _Eur. Data Prot. L. Rev._ , vol. 5, p. 352, 2019.
  * [23] U. K. I. C. Office. (2018) Guide to the general data protection regulation (gdpr). Retrieved: 2023-04-05. [Online]. Available: https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/
  * [24] A. P. Felt, K. Greenwood, and D. Wagner, “The effectiveness of install-time permission systems for third-party applications,” _University of California at Berkely, Electrical Engineering and Computer Sciences, Technical report_ , 2010.
  * [25] Z. Ma, “Android application install-time permission validation and run-time malicious pattern detection,” Ph.D. dissertation, Virginia Tech, 2014.
  * [26] D. Feth, “Transparency through contextual privacy statements,” in _Mensch und Computer 2017 - Workshopband_ , 2017.
  * [27] A.-M. Ortloff, M. Windl, V. Schwind, and N. Henze, “Implementation and in situ assessment of contextual privacy policies,” in _Proceedings of the 2020 ACM Designing Interactive Systems Conference_ , 2020, pp. 1765–1778.
  * [28] E. Onu, M. M. Kwakye, and K. Barker, “Contextual privacy policy modeling in iot,” in _2020 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech)_. IEEE, 2020, pp. 94–102.
  * [29] M. Windl, N. Henze, A. Schmidt, and S. S. Feger, “Automating contextual privacy policies: Design and evaluation of a production tool for digital consumer privacy awareness,” in _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , 2022, pp. 1–18.
  * [30] (2004) The online privacy protection act of 2003, cal. bus. prof. code §§ 22575-22579. Accessed: 2023-02-20. [Online]. Available: https://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?lawCode=BPC&sectionNum=22575
  * [31] (1998) Children’s online privacy protection rule (”coppa”). Accessed: 2023-02-20. [Online]. Available: https://www.ftc.gov/legal-library/browse/rules/childrens-online-privacy-protection-rule-coppa
  * [32] (2023) Understand app privacy and security practices with google play’s data safety section. Accessed: 2023-02-20. [Online]. Available: https://support.google.com/googleplay/answer/11416267
  * [33] (2023) App privacy details on the app store. Accessed: 2023-02-20. [Online]. Available: https://developer.apple.com/app-store/app-privacy-details/
  * [34] (2023) Selecting collection service scenarios and data items to be collected upon app release. Accessed: 2023-02-20. [Online]. Available: https://developer.huawei.com/consumer/en/doc/distribution/app/privacy-label
  * [35] (2023) Permissions for apps on the amazon appstore. Accessed: 2023-05-10. [Online]. Available: https://www.amazon.com/gp/help/customer/display.html/ref=hp_app_paaa?nodeId=GLS4D99DFBLKW3XW&ref_=mas_dp_perm
  * [36] (2023) Samsung galaxy store. Accessed: 2023-05-04. [Online]. Available: https://galaxystore.samsung.com/apps
  * [37] L. Yu, T. Zhang, X. Luo, and L. Xue, “Autoppg: Towards automatic generation of privacy policy for android applications,” in _Proceedings of the 5th Annual ACM CCS Workshop on Security and Privacy in Smartphones and Mobile Devices_ , ser. SPSM ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 39–50. [Online]. Available: https://doi.org/10.1145/2808117.2808125
  * [38] S. Wilson, F. Schaub, A. A. Dara, F. Liu, S. Cherivirala, P. G. Leon, M. S. Andersen, S. Zimmeck, K. M. Sathyendra, N. C. Russell _et al._ , “The creation and analysis of a website privacy policy corpus,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2016, pp. 1330–1340.
  * [39] H. Harkous, K. Fawaz, R. Lebret, F. Schaub, K. G. Shin, and K. Aberer, “Polisis: Automated analysis and presentation of privacy policies using deep learning,” in _27th $\\{$USENIX$\\}$ security symposium ($\\{$USENIX$\\}$ security 18)_, 2018, pp. 531–548.
  * [40] J. Kaur, R. A. Dara, C. Obimbo, F. Song, and K. Menard, “A comprehensive keyword analysis of online privacy policies,” _Information Security Journal: A Global Perspective_ , vol. 27, no. 5-6, pp. 260–275, 2018.
  * [41] L. Yu, X. Luo, C. Qian, S. Wang, and H. K. N. Leung, “Enhancing the description-to-behavior fidelity in android apps with privacy policy,” _IEEE Transactions on Software Engineering_ , vol. 44, no. 9, 2018.
  * [42] S. Zimmeck, P. Story, D. Smullen, A. Ravichander, Z. Wang, J. Reidenberg, N. C. Russell, and N. Sadeh, “Maps: Scaling privacy compliance analysis to a million apps,” _Proceedings on Privacy Enhancing Technologies_ , vol. 2019, no. 3, pp. 66–86, 2019.
  * [43] X. Zhang, X. Wang, R. Slavin, T. Breaux, and J. Niu, “How does misconfiguration of analytic services compromise mobile privacy?” in _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , 2020, pp. 1572–1583.
  * [44] R. Sun and M. Xue, “Quality assessment of online automated privacy policy generators: an empirical study,” in _Proceedings of the Evaluation and Assessment in Software Engineering_ , 2020, pp. 270–275.
  * [45] T. Li, E. Louie, L. Dabbish, and J. I. Hong, “How developers talk about personal data and what it means for user privacy: A case study of a developer forum on reddit,” _Proceedings of the ACM on Human-Computer Interaction_ , vol. 4, no. CSCW3, pp. 1–28, 2021.
  * [46] S. Zimmeck, R. Goldstein, and D. Baraka, “Privacyflash pro: Automating privacy policy generation for mobile apps.” in _NDSS_ , 2021.
  * [47] F. Xie, Y. Zhang, C. Yan, S. Li, L. Bu, K. Chen, Z. Huang, and G. Bai, “Scrutinizing privacy policy compliance of virtual personal assistant apps,” in _37th IEEE/ACM International Conference on Automated Software Engineering_ , 2022, pp. 1–13.
  * [48] D. Bolchini, Q. He, A. I. Antón, and W. Stufflebeam, ““i need it now”: Improving website usability by contextualizing privacy policies,” in _Web Engineering: 4th International Conference, ICWE 2004, Munich, Germany, July 26-30, 2004. Proceedings 4_. Springer, 2004, pp. 31–44.
  * [49] S. Al-Fedaghi, “Awareness of context and privacy,” _Bulletin of the American Society for Information Science and Technology_ , vol. 38, no. 2, pp. 40–48, 2012. [Online]. Available: https://asistdl.onlinelibrary.wiley.com/doi/abs/10.1002/bult.2012.1720380211
  * [50] F. Schaub, R. Balebako, and L. F. Cranor, “Designing effective privacy notices and controls,” _IEEE Internet Computing_ , vol. 21, no. 3, pp. 70–77, 2017\.
  * [51] M. Masotina and A. Spagnolli, “Transparency of privacy notices and contextualisation: effectively conveying information without words,” _Behaviour & Information Technology_, vol. 41, no. 10, 2022.
  * [52] S. Pan, D. Zhang, M. Staples, Z. Xing, J. Chen, X. Xu, and J. Hoang, “A large-scale empirical study of online automated privacy policy generators for mobile apps,” _arXiv preprint arXiv:2305.03271_ , 2023.
  * [53] (2022) Unctad. Accessed: 2022-08-26. [Online]. Available: https://unctad.org/page/data-protection-and-privacy-legislation-worldwide
  * [54] X. Xiao, X. Wang, Z. Cao, H. Wang, and P. Gao, “Iconintent: automatic identification of sensitive ui widgets based on icon classification for android apps,” in _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 2019, pp. 257–268.
  * [55] T. F. Liu, M. Craft, J. Situ, E. Yumer, R. Mech, and R. Kumar, “Learning design semantics for mobile apps,” in _The 31st Annual ACM Symposium on User Interface Software and Technology_ , ser. UIST ’18. New York, NY, USA: ACM, 2018, pp. 569–579. [Online]. Available: http://doi.acm.org/10.1145/3242587.3242650
  * [56] L. A. Leiva, Y. Xue, A. Bansal, H. R. Tavakoli, T. Köroðlu, J. Du, N. R. Dayama, and A. Oulasvirta, “Understanding visual saliency in mobile user interfaces,” in _22nd International conference on human-computer interaction with mobile devices and services_ , 2020, pp. 1–12.
  * [57] S. Bunian, K. Li, C. Jemmali, C. Harteveld, Y. Fu, and M. S. Seif El-Nasr, “Vins: Visual search for mobile user interface design,” in _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–14.
  * [58] B. Deka, Z. Huang, C. Franzen, J. Hibschman, D. Afergan, Y. Li, J. Nichols, and R. Kumar, “Rico: A mobile app dataset for building data-driven design applications,” in _Proceedings of the 30th annual ACM symposium on user interface software and technology_ , 2017, pp. 845–854.
  * [59] L. Li, T. F. Bissyandé, and J. Klein, “Rebooting research on detecting repackaged android apps: Literature review and benchmark,” _IEEE Transactions on Software Engineering_ , vol. 47, no. 4, pp. 676–693, 2019.
  * [60] X. Wei and M. Wolf, “A survey on https implementation by android apps: issues and countermeasures,” _Applied Computing and Informatics_ , vol. 13, no. 2, pp. 101–117, 2017.
  * [61] I. Malavolta, S. Ruberto, T. Soru, and V. Terragni, “End users’ perception of hybrid mobile apps in the google play store,” in _2015 IEEE International Conference on Mobile Services_. IEEE, 2015, pp. 25–32.
  * [62] H. Wang, Z. Liu, J. Liang, N. Vallina-Rodriguez, Y. Guo, L. Li, J. Tapiador, J. Cao, and G. Xu, “Beyond google play: A large-scale comparative study of chinese android app markets,” in _Proceedings of the Internet Measurement Conference 2018_ , 2018, pp. 293–307.
  * [63] (2022) Android developers, google play console help center. Accessed: 2022-04-30. [Online]. Available: https://support.google.com/googleplay/android-developer/answer/9859455?hl=en#:~:text=The%20privacy%20policy%20must%2C%20together,you%20of%20what%20is%20required
  * [64] S. Liu, B. Zhao, R. Guo, G. Meng, F. Zhang, and M. Zhang, “Have you been properly notified? automatic compliance analysis of privacy policy text with gdpr article 13,” in _Proceedings of the Web Conference 2021_ , 2021, pp. 2154–2164.
  * [65] R. Kumar, A. Virkud, R. Sundara Raman, A. Prakash, and R. Ensafi, “A large-scale investigation into geodifferences in mobile apps,” in _31st USENIX Security Symposium (USENIX Security 22)_. Boston, MA: USENIX Association, Aug. 2022. [Online]. Available: https://www.usenix.org/conference/usenixsecurity22/presentation/kumar
  * [66] AppBrain.com: Monetize, advertise and analyze Android apps. Retrieved: 2023-02-15. [Online]. Available: https://www.appbrain.com/
  * [67] S. Liu, F. Zhang, B. Zhao, R. Guo, T. Chen, and M. Zhang, “Appcorp: a corpus for android privacy policy document structure analysis,” _Frontiers of Computer Science_ , vol. 17, no. 3, p. 173320, 2023.
  * [68] (2022) 10 essential types of app screens in mobile ui design - cadabra studio. Accessed: 2023-02-21. [Online]. Available: https://cadabra.studio/blog/10-types-of-app-screens-you-should-know/
  * [69] (2022) App design 9 main types of app screens. Accessed: 2023-02-21. [Online]. Available: https://decode.agency/article/app-screens-design/
  * [70] C. Li, W. Liu, R. Guo, X. Yin, K. Jiang, Y. Du, Y. Du, L. Zhu, B. Lai, X. Hu _et al._ , “Pp-ocrv3: More attempts for the improvement of ultra lightweight ocr system,” _arXiv preprint arXiv:2206.03001_ , 2022.
  * [71] M. Xie, S. Feng, Z. Xing, J. Chen, and C. Chen, “Uied: a hybrid tool for gui element detection,” in _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2020, pp. 1655–1659.
  * [72] S. Xi, S. Yang, X. Xiao, Y. Yao, Y. Xiong, F. Xu, H. Wang, P. Gao, Z. Liu, F. Xu _et al._ , “Deepintent: Deep icon-behavior learning for detecting intention-behavior discrepancy in mobile apps,” in _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , 2019, pp. 2421–2436.
  * [73] J. Chen, A. Swearngin, J. Wu, T. Barik, J. Nichols, and X. Zhang, “Towards complete icon labeling in mobile applications,” in _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , 2022, pp. 1–14.
  * [74] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [75] (2022) App privacy details on the app store. Accessed: 2022-04-30. [Online]. Available: https://developer.apple.com/app-store/app-privacy-details/
  * [76] J. P. Kincaid, R. P. Fishburne Jr, R. L. Rogers, and B. S. Chissom, “Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel,” Naval Technical Training Command Millington TN Research Branch, Tech. Rep., 1975.
  * [77] D. Torre, S. Abualhaija, M. Sabetzadeh, L. Briand, K. Baetens, P. Goes, and S. Forastier, “An ai-assisted approach for checking the completeness of privacy policies against gdpr,” in _2020 IEEE 28th International Requirements Engineering Conference (RE)_. IEEE, 2020.
  * [78] B. Andow, S. Y. Mahmud, W. Wang, J. Whitaker, W. Enck, B. Reaves, K. Singh, and T. Xie, “Policylint: Investigating internal privacy policy contradictions on google play.” in _USENIX Security Symposium_ , 2019, pp. 585–602.
  * [79] G. A. Miller, “Wordnet: a lexical database for english,” _Communications of the ACM_ , vol. 38, no. 11, pp. 39–41, 1995.
  * [80] H. Palmén, M. Gilbert, and D. Crossland, “How bold can we be? the impact of adjusting font grade on readability in light and dark polarities,” in _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , 2023, pp. 1–11.
  * [81] T. D. White, G. Fraser, and G. J. Brown, “Improving random gui testing with image-based widget detection,” in _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2019, pp. 307–317.
  * [82] C. Chen, S. Feng, Z. Xing, L. Liu, S. Zhao, and J. Wang, “Gallery dc: Design search and knowledge discovery through auto-created gui component gallery,” _Proceedings of the ACM on Human-Computer Interaction_ , vol. 3, no. CSCW, pp. 1–22, 2019.
  * [83] J. Chen, M. Xie, Z. Xing, C. Chen, X. Xu, L. Zhu, and G. Li, “Object detection for graphical user interface: Old fashioned or deep learning or a combination?” in _proceedings of the 28th ACM joint meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2020, pp. 1202–1214.
  * [84] M. Xie, Z. Xing, S. Feng, X. Xu, L. Zhu, and C. Chen, “Psychologically-inspired, unsupervised inference of perceptual groups of gui widgets from gui images,” in _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2022, pp. 332–343.