#  A New Hope: Contextual Privacy Policies for Mobile Applications and An
Approach Toward Automated Generation

Shidong Pan  School of Computing, Australian National University Zhen Tao
School of Computing, Australian National University Software Systems Research
Group, CSIRO’s Data61 Thong Hoang  Dawen Zhang  School of Computing,
Australian National University Software Systems Research Group, CSIRO’s Data61
Tianshi Li  Khoury College of Computer Sciences, Northeastern University
Zhenchang Xing  School of Computing, Australian National University Software
Systems Research Group, CSIRO’s Data61 Xiwei Xu  Software Systems Research
Group, CSIRO’s Data61 Mark Staples  Software Systems Research Group, CSIRO’s
Data61 Thierry Rakotoarivelo  Software Systems Research Group, CSIRO’s Data61
David Lo  School of Computing and Information Systems, Singapore Management
University

###### Abstract

Privacy policies have emerged as the predominant approach to conveying privacy
notices to mobile application users. In an effort to enhance both readability
and user engagement, the concept of contextual privacy policies (CPPs) has
been proposed by researchers. The aim of CPPs is to fragment privacy policies
into concise snippets, displaying them only within the corresponding contexts
within the application’s graphical user interfaces (GUIs). In this paper, we
first formulate CPP in mobile application scenario, and then present a novel
multimodal framework, named SeePrivacy, specifically designed to automatically
generate CPPs for mobile applications. This method uniquely integrates vision-
based GUI understanding with privacy policy analysis, achieving 0.88 precision
and 0.90 recall to detect contexts, as well as 0.98 precision and 0.96 recall
in extracting corresponding policy segments. A human evaluation shows that 77%
of the extracted privacy policy segments were perceived as well-aligned with
the detected contexts. These findings suggest that SeePrivacy could serve as a
significant tool for bolstering user interaction with, and understanding of,
privacy policies. Furthermore, our solution has the potential to make privacy
notices more accessible and inclusive, thus appealing to a broader
demographic. A demonstration of our work can be accessed at
https://cpp4app.github.io/SeePrivacy/

##  1 Introduction

Privacy policies have become the most prevalent privacy notice approach for
mobile applications, with the aim of protecting users’ privacy and digital
security [54, 39, 88, 56, 65, 29, 57, 37, 71]. On the contrary, application
users often find themselves in a privacy paradox, specifically, Digital
Resignation. This term refers to a state where users desire to understand and
control the information that digital entities own about them, but feel
powerless to do so [50]. A study has found that the average length of privacy
policies is around 4,000 words for 75 prominent mobile applications and
websites, requiring an estimated 16 minutes to read for an average adult [34].
Consequently, according to a survey conducted by Obar and Oeldorf-Hirsch [83],
a significant portion of users, approximately 74%, opt for the “quick join”
clickwrap option without engaging with privacy policies, primarily due to the
lengthy and detailed nature of these documents. These numbers indicate that
privacy policies have become increasingly difficult to read and understand in
a timely manner, exacerbating the prevalence of digital resignation among
users.

In addressing the pressing issue of digital resignation, the contemporary
evolution of privacy policies has given rise to the concept of “just-in-time”
privacy notices [10]. These notices furnish users with detailed information
about data collection at the point of interaction. This practice has been
recognized and included in the U.K. General Data Protection Regulation as a
recommended approach for crafting privacy policies [10]. However, the dominant
application of “just-in-time” notices is primarily limited to the installation
phase of applications and first invocation of certain permissions. Figure 1(a)
shows the “install-time” permission reminders that appear when a user installs
a new Android app111This feature was essentially deprecated after Android 6.0
(API level 23) since developers must request permissions at runtime.,
delineating the permissions to be granted. Figure 1(b) presents another
example, which illustrates a camera access reminder triggered at the initial
“invoke-time”. However, there are inherent limitations to current “just-in-
time” notices. For one, while they provide a snapshot of when an app accesses
specific data, it doesn’t necessarily offer insights into the broader context
– how this data might subsequently be transmitted off the device, shared with
third parties, stored, or used for other purposes. Furthermore, current
implementations of “just-in-time” reminders fall short in terms of efficacy,
as security warnings regarding device permissions tend to be isolated from
their immediate context, thus failing to adequately engage user awareness [52,
79].

To counteract these deficiency, researchers proposed Contextual Privacy
Policies (CPPs) [53, 99]. This innovative concept aims to deconstruct
traditional privacy notices into shorter and digestible segments, displaying
them only in relevant contexts in mobile applications. Privacy-related
contexts refer to areas in which data practices are directly associated with
graphical user interfaces (GUIs). These contexts on the screen are identified
and the corresponding privacy policy segments are retrieved and displayed,
with key information highlighted. This integration of privacy notices within
the GUIs ensures a close and timely alignment with the context, thus conveying
precise and succinct privacy information. Windl et al. [99] demonstrated that
CPPs have the potential to address the usability issues in privacy policies,
thereby enhancing user engagement and awareness of data practices. They also
presented a framework for automatically generating CPPs for websites. Studies
have shown that users spend 90% of their time using mobile apps rather than
websites [36, 67, 33]. However, the prior [99] framework has the following
limitations when applied to the mobile app context, all of which are
successfully addressed in this work: 1) Their context selection is based on
empirical observation of users’ behaviour in website browsing scenarios, but
lacks connection to privacy regulations and existing standards in practice. 2)
The prior context detection method is based on ad-hoc HTML analysis, which has
limited internal generalizability, especially for more compact and complicated
user interfaces in mobile app platforms. Overall, while the concept of CPP has
a great potential, generating accurate CPPs for mobile apps proposes unique
challenges that cannot be addressed by the previous automated generation
framework.

(a) Install-time

(b) Invoke-time

(c) Context-aware

Figure 1:  Figure 1(a) represents the install-time reminder of required
permissions [23]; Figure 1(b) corresponds to the invoke-time reminder that
appears when permission is first invoked; Figure 1(c) presents our proposed
contextual privacy policy, specifically designed for mobile applications.

In this paper, we first formulate the CPPs in the mobile scenario, and then
propose a novel multimodal framework, dubbed SeePrivacy, designed to
automatically generate CPPs for mobile applications. Figure 1(c) presents an
example of our CPP in mobile applications. Within this framework, computer
vision techniques are incorporated with pre-trained large language models
(LLMs) to analyze mobile GUI screenshots, thus pinpointing privacy-related
contexts. In addition, natural language processing techniques are used to
dissect privacy policy texts, extracting the corresponding policy sections for
identified contexts. To quantitatively evaluate performance, we establish our
benchmark dataset, named Cpp4App, which covers more than 1,200 privacy-related
contexts along with their matching privacy policy segments in mobile settings.
The quantitative evaluation demonstrates the robust performance of our
framework in multiple evaluation metrics. Regarding the identification of
privacy-related contexts, SeePrivacy achieves accuracy, precision, and recall
values of 0.81, 0.88, and 0.90, respectively. Regarding the extraction of the
corresponding segments, the framework attains accuracy, precision, and recall
rates of 0.94, 0.98, and 0.96, respectively. To further assess the
functionality of SeePrivacy, we performed a human evaluation with 15 examiners
on about 120 CPPs. The results are encouraging: 77% of the extracted privacy
policy segments were perceived as elucidating the detected context.

We also discussed the applications, implications, and contributions of CPPs
and SeePrivacy to the broader community. In the survey, participants display a
significantly increased willingness to engage with CPPs, achieving an average
score of 4.1 out of 5. In contrast, traditional privacy policies received an
average score of just 2 out of 5. This difference suggests SeePrivacy could
contribute meaningfully to improve user engagement with privacy policies,
potentially mitigating the widespread challenge of digital resignation.
Additionally, the overall usefulness rating for SeePrivacy is 4.53 out of 5,
indicating strong user endorsement. Furthermore, we elaborate on two potential
application scenarios of SeePrivacy: displaying CPPs in app markets and
detecting “lack of disclosure” with exact problematic context. Lastly, we
discuss how CPPs potentially make the privacy policies more inclusive and
allow it to reach a larger audience such as children.

In summary, our proposed framework extends the automated generation of CPPs to
mobile applications, which are the digital platforms users spend the majority
of time on [36, 67, 33]. The key contributions are:

  * •

To our knowledge, this is the first research to define the concept of
contextual privacy policies (CPPs) within mobile application scenarios.

  * •

We have introduced a novel framework, named SeePrivacy, designed to
automatically generate these contextual privacy policies for mobile
applications.

  * •

We have constructed the benchmark dataset, named Cpp4App, specifically to
evaluate contextual privacy policies on mobile platforms and to quantitatively
assess the functionality of the SeePrivacy framework.

Roadmap. The remainder of this paper is structured as follows: Section 2
introduces the development of transparent privacy notices and privacy-related
context identification; Section 3 introduces our benchmark dataset, Cpp4App;
Section 4 details the design of our proposed framework, i.e.,SeePrivacy; and
Section 5 presents the results of evaluating SeePrivacy on the benchmark
dataset; Section 6 discusses the implications and contributions of our work to
the broader community; and Section 7 concludes the paper.

Ethical Considerations and Responsible Disclosure. Ethical approval for this
research was secured from our institution’s Institutional Review Board (IRB).
In conducting our user study, we avoided collecting any personally
identifiable information (PII). In line with our commitment to responsible
research practices, any privacy discrepancies detected during the course of
the study were promptly and transparently disclosed to both the study
participants and the corresponding application developers.

##  2 Background

Table 1: Tallies of data types’ existence.

| Basic Personal Identifiable Information | Other Personal Information
---|---|---
| Name | Birthday | Address | Phone | Email | Profile | Contacts | Location | Photos | Voices | Financial Info | Social Media
Administration |  |  |  |  |  |  |  |  |  |  |  |
GDPR [8] | ● | ○ | ● | ● | ● | ● | ○ | ○ | ○ | ○ | ○ | ○
CCPA [4] | ● | ○ | ○ | ○ | ● | ● | ○ | ● | ○ | ○ | ● | ○
CalOPPA [6] | ● | ○ | ● | ● | ● | ○ | ○ | ● | ○ | ○ | ● | ●
COPPA [5] | ● | ○ | ● | ● | ● | ● | ○ | ● | ● | ● | ○ | ○
APP [APP] | ● | ○ | ● | ● | ● | ● | ○ | ● | ● | ○ | ● | ○
Industry |  |  |  |  |  |  |  |  |  |  |  |
Google Play [21] | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ●
Apple App [17] | ● | ○ | ● | ● | ● | ● | ● | ● | ● | ● | ● | ●
Huawei AppGallery [9] | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | ●
Amazon Appstore [3] | ○ | ○ | ○ | ● | ● | ○ | ● | ● | ● | ● | ● | ●
Samsung Galaxy Store [25] | ○ | ○ | ○ | ● | ○ | ○ | ● | ● | ● | ● | ○ | ○

Mobile applications provide users with unparalleled convenience, multiple
entertainment options, and various services. At the same time, they are
responsible for the collection and exchange of substantial amounts of personal
information, raising concerns surrounding data privacy. In response to the
potential risks of privacy violations, the U.S. Department of Health,
Education, and Welfare proposed a set of Fair Information Practice Principles
(FIPP) in 1973 [7, 68]. Later, the Federal Trade Commission adopted these
principles as “the notice and choice privacy framework” [83], which is
commonly regarded as the foundation for modern privacy regulations such as the
European General Data Protection Regulation (GDPR) [8], the California
Consumer Privacy Act (CCPA) [4], and the Australian Privacy Principles (APP)
[APP]. In FIPP, the most fundamental principle is “notice” [28]. Consumers
should be provided with a clear understanding of an entity’s information
practices before any personal data is collected from them (e.g., [GDPR Art.
12, GDPR Rectical 58]). In the absence of transparent and understandable
notice, it becomes impossible for a consumer to make an informed decision
regarding the extent of personal information they choose to disclose.

###  2.1 Related Work

Research efforts to improve the presentation and readability of privacy
policies have been manifold and encompass various approaches. Kelly et al.
[62, 63, 64] introduced an innovative method called privacy nutrition labels
(PNLs), designed to facilitate consumers’ understanding of how their
information is collected and utilized in a structured table-like way. By
employing this concise and organized framework, PNLs effectively aid in
understanding the privacy practices of software applications [42, 45, 74, 75,
86]. In a related vein, Cranor [44] proposed a protocol, i.e., Platform for
Privacy Preferences, that allows website organizations to declare their
intended use of information collected from website users. Alongside these
approaches, there has been a growing focus on visual aids such as privacy
icons [58, 59, 51]. These small graphical symbols, symbolizing privacy
concepts or practices, serve as visual shorthand, simplifying complex privacy
policies. By standardizing these icons across various services and platforms,
researchers aim to reduce confusion, thereby making privacy policies more
readily accessible to the general populace [59].

The conceptualization of Contextual Privacy Policies emerged in 2004, marking
a significant change in the approach to the presentation of privacy notices.
Bolchini et al. [35] discerned that the conventional reading of privacy
policies represented a monolithic block that hindered users from retrieving
essential privacy details. In response, they invented a systematic method
aimed at reorganizing privacy policies around specific user interaction
contexts. Subsequent to this development, Feth [53] extended the concept by
introducing contextual privacy statements. Rather than using a “one-size-fits-
all” privacy policy that must suit every usage scenario, these contextual
privacy statements offer precise privacy and data protection information more
closely with the user’s immediate context. In a recent advancement, Windl et
al. [99] proposed the design and architecture of PrivacyInjector, a production
AI tool that can automatically generate contextual privacy policies for
websites. However, their scope of data practice identification is limited to
six website-specific types, such as “Advertising” and “Cookies and Tracking
Elements”. Their identification procedure primarily considers which date type
users are likely to encounter when browsing websites. In contrast, our
approach is regulation-driven, examining which data practices should be
transparently disclosed in privacy notices. Moreover, their context detection
method is intrusive and based on ad-hoc HTML analysis, which is consequently
not applicable to mobile applications. In contrast, the source code of mobile
applications is typically inaccessible to end-users. This limitation led us to
develop a purely vision-based context detection method, tailored to the unique
challenges presented by mobile applications. Nonetheless, CPPs have been
empirically shown to increase privacy notice transparency, constituting a
progressive step toward more user-centric privacy communication [84, 81].

Contrasting with websites, mobile applications represent a unique landscape
for personal information collection, often involving multifaceted approaches
that include device sensors, user input, and third-party integrations [87].
These complex processes necessitate a precise and transparent articulation of
how user information is acquired, employed, and disseminated—an issue of
critical concern. Given the expansive and continually growing scale of mobile
applications, there exists an emergent need for an automated framework capable
of generating CPPs tailored specifically to this domain. In pursuit of this
objective, the present work attempts to establish such a framework. As a
preliminary step, we recognize the imperative to define clear and
comprehensive definitions for privacy-related contexts within the mobile
environment, forming the foundation for our proposed framework.

###  2.2 Empirical Data Practice Identification

According to Windl et al. [99], privacy-related contexts are considered areas
on graphical user interfaces (GUIs) where data practices might be applied. To
rigorously define these practices, we formulate two guiding questions, each
tailored to address a distinct facets of the issue:

1. What data practices should be included in privacy notices by regulations?

2. What data practices are of concern and desire for users?

These questions collectively serve to demarcate the scope of our
investigation, guiding our analysis in a manner that encompasses regulatory
compliance and user-centric considerations. To our knowledge, there are no
existing studies that specifically address the issue of contextual privacy
policies within the domain of mobile applications. Specifically, current
research appears to be primarily concerned with improving the presentation and
readability of privacy policies [62, 64, 63, 44, 42]. Therefore, we perform a
rigorous examination of data practices as they relate to privacy regulations
and industry market standards (see Table 1). We present each group as follows:

_Administration._ The administration often functions as the key regulator
within the broader ecosystem of data privacy. A report by the United Nations
Conference on Trade and Development reveals that 137 of 194 countries and
regions have instituted data protection and privacy legislation [14]. To gain
insight into the administrative perspective on these matters, we examine five
representative privacy regulations (details are presented in the Appendix
8.1), including the GDPR [8], CCPA [4], the California Online Privacy
Protection Act (CalOPPA) [6], the Children’s Online Privacy Protection Act
(COPPA) [5], and APP [APP]. These regulations serve as critical touchpoints in
our exploration of the administration’s stance on data practices.

_Industry._ Mobile application platforms typically impose particular privacy
notice requirements on curated applications within their marketplaces, with
special attention to the categories of data involved. For instance, the Google
Play app store has delineated 14 primary data categories and 39 sub-categories
in their data safety reports, which all applications must adhere to [21].
These standards have been formulated through rigorous user studies and market
tests, reflecting a comprehensive understanding of user concerns and
expectations. Such market standards provide valuable insight into the
alignment of privacy requirements with actual user needs and industry
practices, thereby serving as a meaningful reference for privacy policy
development in the mobile application domain.

###  2.3 Multimodal Privacy-related Contexts

Table 2: Privacy-related contexts.

Data types | Description | Privacy-related keywords lists for textual GUI elements | Related RICO-icon class | Icon examples
---|---|---|---|---
Name | How a user refers to themselves |  name, first name, last name, full name, real name, surname, family name, given name | n.a. | n.a.
Birthday | A user’s birthday | birthday, date of birth, birth date, DOB, birth year | n.a. | n.a.
Address | A user’s address |  mailing address, physical address, postal address, billing address, shipping address, residential address, residence, personal address | n.a | n.a
Phone | A user’s phone number |  phone, phone number, mobile phone, mobile number, telephone, call, telephone number | [43] Call |
Email | A user’s email address  | email, e-mail, email address, e-mail address | [6] Email |
Profile | A user’s account information  | profile, account | [49] Avatar |
Contacts |  A user’s contact information, or the access to the contact permission | contacts, phone-book, phone book, device’s address book | [68] Group, [3] Follow |
Location |  A user’s location information, or the access to the location permission |  location, locate, geography, geo, geo-location, precision location |  [40] Location crosshair, [72] Location |
Photos |  A user’s photos, videos, or the access to the camera permission |  camera, photo, scan, album, picture, gallery, photo library, storage, image, video, scanner, photograph |  [42] Photo, [56] Videocam, [82] Wallpaper |
Voices |  A user’s voices, recordings, or the access to the microphone permission | microphone, voice, mic, speech, talk, audio | [91] Microphone |
Financial info |  Information about a user’s financial accounts, purchases, or transactions |  credit card, company, companies, organization, commercial, organizations, pay, payment, financial, bill, wallet, purchase | [61] Cart |
Social media |  A user’s social media information, or the access to social media accounts | social media, Facebook, Twitter, socialmedia, share | [77] Facebook, [89] Twitter |

Privacy-related contexts refer to areas within GUIs where data practices might
be applied. In this investigation, we categorize two primary types of GUI
components that indicatively collect personal information via human-device
interactions: literal texts and graphical icons. Literal texts, often regarded
as the foundational information sources in mobile application GUIs, are
juxtaposed with graphical icons, which are ingeniously designed to communicate
vital information with maximum efficiency. The growing adoption of graphical
icons in place of literal texts is due to their concise nature, which provides
users with the requisite information instantaneously, thereby substantially
minimizing the cognitive effort required to interpret the accompanying textual
descriptions [101].

_Keyword collection._ In the process of analyzing literal texts that describe
data types, we snowballed a keyword list tailored to CPPs. This list was
compiled by extracting keywords from academic research literature about
privacy policies, including content categorization [98, 61, 108, 73],
automated generation [105, 94, 109], and compliance analysis [106, 108, 107,
103, 32, 46]. These studies often present unique challenges due to differences
in privacy policies, corpus selections, and specific tasks among various
topics. Ambiguous terms, such as “address,” which could refer to a residential
or an email address, were deliberately excluded to avoid confusion. To further
promote unification and consistency across contexts, we rendered the list
case-insensitive. Table 2 presents a comprehensive display of the finalized
keyword list of each data type.

_Icon collection._ Icons frequently serve as symbols for specific categories
of privacy information. One common example is that the camera-like icons
typically correspond to the authorization of camera access by mobile
applications or access to the device photo gallery; and a location crosshair
icon often represents the access to location permission or users physical
location information. Liu et al. [78] developed a comprehensive icon taxonomy,
comprising 99 common icon classes in mobile applications. This research effort
also yielded one of the largest icon datasets, consisting of over 118,000
distinct icons. Subsequently, we performed a semantic alignment between the
RICO-icon classes proposed by Liu et al. and the data types that were
identified through our survey. By doing so, the textual information (privacy-
related keywords) and visual information (icons) on GUI are well aligned with
regulatory requirements (data types), as detailed in Table 2.

##  3 Cpp4App: The Benchmark Dataset

In our study, the construction of a benchmark dataset is essential to
quantitatively assess the efficacy of the proposed framework in generating
contextual privacy policies (CPPs). Although existing datasets are available
for privacy policy analysis [98, 108, 32, 30], and for understanding mobile
graphical user interfaces (GUIs) [47, 69, 38], we identified two major
shortcomings within current collections. These limitations, as detailed below,
necessitate the compilation of a new dataset.

  * •

Temporal inconsistency between applications, policies, and GUIs.

With mobile applications undergoing frequent changes in privacy practices,
there is a concomitant need to update their corresponding privacy policies. In
addition, the GUIs of these apps are subject to rapid evolution. Consequently,
it becomes imperative to maintain alignment between the screenshots
(representing the GUIs) and their temporally corresponding privacy policies
for CPP generation. This alignment ensures the precision and dependability of
quantitative evaluations.

  * •

No annotations about privacy-related contexts in GUIs.

To the best of our knowledge, we are the first to introduce the concept of
CPPs specific to mobile applications. For the purpose of evaluating the
performance of the automated generation framework, benchmark datasets should
ideally incorporate two essential features: 1) the annotated regions (i.e.,
bounding boxes) representing privacy-related contexts within GUI screenshots,
and 2) the mapping relations that connect these privacy-related contexts to
corresponding segments of the privacy policy. Unfortunately, as of the time of
this study, no existing dataset encompasses both of these vital
characteristics.

In the following subsections, we present steps employed to create a benchmark
dataset, thereby addressing the aforementioned issues.

###  3.1 Mobile Apps Selection

The Android platform is often considered transparent and favorable for
academic research, with numerous apps markets readily available [97, 72].
Among these, the Google Play app store stands out as the most expansive and
accessible, hosting more than two million applications [80, 96]. In
particular, Google Play is a leader in requiring mobile applications to
provide a link to their privacy policy on their homepages [22]. As a result,
users downloading Android mobile applications from the Google Play app store
can directly access the respective privacy policy links.

Consistent with previous research, widely used mobile applications tend to
adhere to more stringent privacy practices [108, 76, 66]. In addition, less
popular applications often have poorly written privacy policies [87, 86],
making it meaningless to generate CPPs for them. In this study, we leveraged
market data from AppBrain [15], one of the largest third-party platforms
monitoring app statistics for the Google Play app store, to pinpoint the 200
most popular mobile applications. Our study collection method includes four
specific strategies. First, recognizing the variations in GUIs, functions, and
privacy practices across different application categories (i.e., genre), we
endeavored to ensure a diverse representation by selecting the most popular
mobile applications from 17 distinct categories. Second, applications
published by the same parent company normally share the same privacy policy,
so we try to select apps from different developers for better representation.
Third, we excluded any mobile applications with privacy policies that were
less than 200 words or smaller than 2 KB in size, as these may constitute
invalid or insufficient privacy policies [66, 77]. Finally, we limited our
selection to those applications that offered privacy policies in English, thus
narrowing our candidate pool to 50 mobile applications for subsequent fine-
grained annotation.

###  3.2 Context Annotation Strategy

To create a benchmark dataset, it is imperative to ascertain that vital
privacy-related GUI elements are incorporated into our dataset. Therefore,
drawing on established guidelines about mobile application GUI design [1, 2],
we have systematically gathered specific pages from each selected application.
The collection process adheres to the categorizations delineated below:

  * •

_Registration/Login page._ Mobile applications frequently gather user
registration information on this page to facilitate account creation. This may
encompass both personal and other private details. Certain applications
require new users to register before accessing specific features, making it
infeasible for these users to circumvent the registration or login process.

  * •

_Home page._ This serves as the initial interface upon accessing a mobile
application. It contains essential features and information related to its
app’s functionality, which may necessitate the provision of personal data or
the granting of device permissions. Furthermore, screenshots of adjacent main
pages are also collected for analysis.

  * •

_Profile page._ This section may retain users’ personal information. For
example, some mobile applications might request access to the camera or photo
roll to furnish customization options related to public profile appearance.

  * •

_Setting page._ Often associated with privacy customization, this page may
govern permissions related to device sensors and access to personal
information.

  * •

_Onboarding page._ It serves as a tutorial or introduction. Moreover, this
page elucidates a mobile application’s salient features and benefits.

  * •

_Map page._ Common to various application categories such as travel, posting,
food delivery, and shopping, this page is often utilized to amass users’
location data.

  * •

_Essential pages for specific categories._ This includes specialized
interfaces such as the product card pages in shopping apps, the post-editing
interface in forum applications, and the feed page within social media
platforms.

We follow these categorizations to systematically collect relevant pages from
each selected mobile applications, ensuring that essential privacy-related GUI
elements are represented in our dataset. Notably, we have opted for manual
collection of GUI pages as opposed to relying on static analysis for several
reasons. Accurate static analysis is a challenging task in mobile program
analysis, particularly for popular apps that are not only complexly programmed
but also commonly employ code obfuscation. Furthermore, iterating through all
possible activities to capture their corresponding GUIs is a non-trivial task,
and outdated or restricted GUIs may not be accessible through regular user
interactions. Manual collection allows us to naturally avoid the
aforementioned problems.

After the data collection, the annotators need to manually examine the text in
each screenshot, cross-referencing it with our CPP-related keyword list, as
mentioned in Table 2. If the text contains any of these CPP-related keywords,
it is labeled as the corresponding data type and designated with a bounding
box. Subsequently, the annotators are tasked with detecting all icons in the
screenshot to determine if any belong to a privacy-related data type. If an
icon meets this criterion, it is labeled with the corresponding class and
marked with a bounding box. For these purposes, the OpenCV-Python [11] library
is employed to draw bounding boxes and document coordinates. Two annotators
were asked to execute the annotation process independently. For any
disagreement, they discussed and agreed on the same opinion, and if the
disagreement persisted, a senior annotator joined the discussion to facilitate
a resolution. The percent agreement (inter-rater reliability) for initial
annotation is about 84%.

Table 3: Basic statistic of Cpp4App.

No. Mobile applications & Privacy policies | 50
---|---
No. Screenshots | 402
Screenshots per Mobile application | 8.04
No. Privacy-related contexts | 1,217
Contexts per Mobile application | 24.34
No. Words in Privacy policies | 297,010
Words per Privacy policy | 5,940

Figure 2: An overview of SeePrivacy.

###  3.3 Segment Annotation Strategy

Initially, if annotators are unfamiliar with the mobile application of the
policy they need to annotate, they are mandated to explore and familiarize
themselves with its features for a duration of 10 minutes. Then, two
annotators independently scrutinize each data privacy-related context within
the privacy policies and extract pertinent segments. To ensure the integrity
and quality of the annotation process, we establish a reading speed parameter
ranging from 250 to 300 words per minute (the reading speed of an average
adult [34]) for the annotators, implementing countdown timers in accordance
with the policy lengths. If the $segment\\_sim$ (See the Formula 5.1.2) is
above 0.8, two annotations are counted as an agreement. The mechanism for
resolving disagreements is consistent with the process outlined in the
previous paragraph. In cases where both annotators are unable to identify any
relevant segments within the privacy policies, the segment is explicitly
labeled as “No relative information is found in the privacy policy.”

###  3.4 Setups and Statistics

All annotators involved in this study are research scientists with a minimum
of two years of experience in privacy research for mobile applications. To
ensure the alignment between the GUIs and privacy policies, an Android mobile
phone, i.e., the OnePlus 7 Pro, Android 11, and Hydrogen OS 11.0.9.1.GM21, is
employed to download all the selected mobile applications. The corresponding
privacy policies are downloaded simultaneously. To preserve the status quo of
the apps and policies, the auto-update features within the Google Play app
store, individual mobile apps, and the system’s general settings on the phone
(including auto-update over Wi-Fi) are all deactivated. In total, this effort
resulted in the collection of 402 screenshots from 50 diverse mobile apps,
accompanied by 1,217 labeled privacy-related contexts, each correlating to
specific segments within privacy policies. We have designated this benchmark
dataset as Cpp4App, and the statistical analysis is provided in Table 3. As
the inaugural dataset of its nature, we are confident that it is sufficient
for evaluating our automatic generation framework and positioned to serve as a
benchmark for future research in this field.

##  4 SeePrivacy: The Framework

In this section, we introduce our multimodal framework SeePrivacy, designed to
automatically generate CPPs for a given screenshot and a given privacy policy,
as illustrated in Figure 2. The framework accepts privacy policies and
screenshots from a given mobile application as input parameters. These inputs
are processed through three primary modules within the framework to
autonomously build CPPs for the designated mobile application. The three main
modules integral to our framework are detailed as follows:

  * •

Context Detection Module: This module is designed to identify specific
contexts within a mobile application’s screenshots where CPPs should be
displayed.

  * •

Segment Extraction Module: This module focuses on extracting pertinent
segments from the privacy policy of the given mobile application.

  * •

CPPs Presentation Module: This module is responsible for linking privacy
policy segments to their corresponding contexts and articulating the essential
information, thereby facilitating the generation of CPPs for mobile
applications.

The details of each module are explained in the following subsections.

###  4.1 Context Detection Module

The source code of mobile apps is typically inaccessible to end-users. This
limitation led us to develop a purely vision-based Context Detection Module
tailored to the unique challenges presented by mobile apps. Specifically, in
our approach to GUI understanding, we opt for visual components as atomic
elements rather than function-level GUI widgets. This choice is driven by the
observation that certain widgets (e.g., a navigation bar) may be too coarse-
grained to adequately represent a data practice.

Textual GUI components. To detect and localize privacy-related texts from a
given screenshot, we apply a state-of-the-art ultra-lightweight OCR system,
named PaddlePaddle-OCR (PP-OCR) [70]. After obtaining these texts, we employ a
pre-trained large language model, i.e.,
GPT-3.5222https://platform.openai.com/docs/models/gpt-3-5, to classify their
data types. The specific process is detailed in Appendix
LABEL:sec_appendix_gpt. If the answer contains a data type, we will regard the
{detected text} as relevant to this data type; otherwise, it does not belong
to any data type.

Iconic GUI components - Localization. An empirical study [40] demonstrated
that rule-based object detection methods on GUIs outperform learning-based
methods, such as YOLO [91] series. In addition, rule-based methods have better
explainability. To localize iconic GUI components from their screenshots, we
adapt previous work [104] and propose an updated set of detection rules to
filter out irrelevant objects. Specifically, we have four rules presented as
follows: (a) Filtering out elements whose area is greater than 10% of the
total area of the screenshot, in order to remove the bounding boxes and large
images; (b) Filtering out elements whose area is less than 5% of the total
area of the screenshot, aiming to remove the small noises; (c) Removing
rectangular elements whose aspect ratio (width to height) is less than 0.6,
ruling out control bars and top/bottom banners; (d) Removing elements that
area is overlapped with OCR results since some letters in large font-size are
also detected as potential icons. By applying these rules to our dataset, we
can obtain the position of the iconic GUI components on their screenshots.

Iconic GUI components - Classification. CNN-based neural networks have a
strong capability of recognizing patterns on GUIs [78, 100, 41]. In this
paper, we train two models to classify the detected graphical icons from the
screenshots, including a ViT-based model and a ResNet-based model. ViT stands
for Vision Transformer [49]. Specifically, we employ a pre-trained vanilla ViT
and fine-tune it on ImageNet [48]. Regarding the ResNet-based model, we employ
ResNet-18 as the backbone because of its tiny size and decent efficiency. We
further add dropout layers to every ResNet block to enhance its
generalizability, preventing the potential over-fitting issue. We then trained
both models on the RICO-icon dataset with the default train-test split. We use
the ViT-based model as the default for Contextual Detection Model as it
achieves the best performance.

Table 4: Performance of SeePrivacy’s modules. “CDM” stands for Context
Detection Module. “R-i Class” stands for Rico-icon Class as per Table 2.

Category | Accuracy | Precision | Recall
---|---|---|---
Name | 0.98 | 0.98 | 1.00
Birthday | 1.00 | 1.00 | 1.00
Address | 0.38 | 0.41 | 0.86
Phone | 0.87 | 0.90 | 0.96
Email | 0.61 | 0.97 | 0.62
Profile | 0.72 | 0.77 | 0.92
Contacts | 0.98 | 1.00 | 0.98
Location | 0.92 | 0.96 | 0.96
Photos | 0.87 | 1.00 | 0.87
Voices | 0.91 | 0.91 | 1.00
Financial info | 0.93 | 0.93 | 0.93
Social media | 0.67 | 0.75 | 1.00
Average | 0.82 | 0.87 | 0.93

(a) CDM - Textual GUI components

R-i Class | Category | Accuracy | Precision | Recall
---|---|---|---|---
Call | Phone | 0.96 | 1.00 | 0.96
Email | Email | 0.96 | 0.95 | 1.00
Avatar | Profile | 0.92 | 0.96 | 0.96
Follow | Contacts | 0.89 | 0.94 | 0.94
Group | Contacts | 0.73 | 0.73 | 1.00
Location | Location | 0.93 | 0.98 | 0.95
Crosshair | Location | 0.91 | 0.94 | 0.97
Photo | Photos | 0.92 | 0.92 | 1.00
Wallpaper | Photos | 0.94 | 0.94 | 1.00
Videocam | Photos | 0.83 | 1.00 | 0.83
Microphone | Voices | 1.00 | 1.00 | 1.00
Cart | Financial Info | 0.91 | 0.95 | 0.95
Facebook | Social media | 0.96 | 1.00 | 1.00
Twitter | Social media | 1.00 | 1.00 | 1.00
Average |  | 0.92 | 0.95 | 0.96

(b) CDM - Iconic GUI components

Category | Accuracy | Precision | Recall
---|---|---|---
Name | 1.00 | 1.00 | 1.00
Birthday | 0.96 | 1.00 | 0.95
Address | 0.52 | 0.82 | 0.50
Phone | 1.00 | 1.00 | 1.00
Email | 0.99 | 0.99 | 1.00
Profile | 0.99 | 0.99 | 1.00
Contacts | 0.79 | 1.00 | 0.78
Location | 0.94 | 0.95 | 0.98
Photos | 0.95 | 0.97 | 0.97
Voices | 0.86 | 0.76 | 1.00
Financial info | 1.00 | 1.00 | 1.00
Social media | 1.00 | 1.00 | 1.00
Average | 0.94 | 0.98 | 0.96

(c) Segments Extraction Module

Category | Accuracy | Precision | Recall
---|---|---|---
Name | 0.98 | 0.98 | 1.00
Birthday | 1.00 | 1.00 | 1.00
Address | 0.38 | 0.41 | 0.86
Phone | 0.88 | 0.91 | 0.97
Email | 0.66 | 0.96 | 0.68
Profile | 0.70 | 0.74 | 0.93
Contacts | 0.75 | 0.94 | 0.78
Location | 0.94 | 0.97 | 0.96
Photos | 0.86 | 0.98 | 0.88
Voices | 0.91 | 0.91 | 1.00
Financial info | 0.81 | 0.93 | 0.87
Social media | 0.80 | 0.89 | 0.89
Average | 0.81 | 0.88 | 0.90

(d) Overall Context Identification

###  4.2 Segment Extraction Module

The objective of this module is to obtain privacy policy segments with their
corresponding privacy-related contexts. Mainstream mobile application markets,
such as the Google Play app store [22] and the Apple App Store [16], require
developers to provide a privacy policy link on the application’s homepage.
This requirement allows us to access those links based on the name of a mobile
application. To capture these privacy policies in HTML format, we leverage two
Python libraries, Selenium [12] and BeautifulSoup [19], which fetch the
information via the corresponding privacy policy links. Recognizing the global
market trend and the need to cater to multilingual users, we note that an
increasing number of privacy policies are available in multiple languages. To
maintain consistency and clarity in our analysis, we deploy langdetect [24]
library to filter out non-English texts in privacy policies. Note that our
framework has the flexibility to be easily extended to accommodate other
languages.

Previous studies [95, 103] have discussed that only using sentence-level
privacy policies processing and analysis can lead to contradictory
conclusions, since the same or similar sentences in different sections may
have different implications. Therefore, to retrieve the relevant sentence-
based segments, we adopt the multi-level privacy policies processing method
proposed in [31, 103]. For privacy policy documents whose HTML structure
follows the
$(\langle\texttt{Heading}\rangle\langle\texttt{Paragraph}\rangle^{+})^{+}$
format, we reuse the pre-trained Bayesian multi-label classifier model [103]
to classify privacy policy paragraphs based on their headings since headings
usually contain words that can be used as identifiers. The classifier is
claimed to achieve 0.85 accuracy and can classify the majority of the
documents based on their headings. The paragraphs classified as Types are
related to the “types of personal data collected by the current app”. Those
paragraphs are highly related to specific data practices, so we only conduct
sentence-level analysis on them. For privacy policy documents whose HTML
structure does not follow the
$(\langle\texttt{Heading}\rangle\langle\texttt{Paragraph}\rangle^{+})^{+}$
format, we follow previous works [56, 99] to classify paragraphs on specific
data practices. Specifically, we train a CNN-based multi-label classification
model on a large privacy policy dataset [98] with 23K fine-grained data
practice annotations. Note that this dataset contains 12 high-level categories
for data practices, including First-Party Collection/Use and Third-Party
Sharing/Collection. The results show that we can achieve a 0.84 top-1
precision for classifying the privacy segments with their related data
practices.

After obtaining paragraph-level privacy segments, we tokenize them into
sentences using the stanza [90] Python library. We then perform a two-stage
sentence-level analysis to extract the sentences for each data type. First, we
perform a keyword search based on our collected keyword list (see Table 2). If
a keyword exists, the sentence will be added to the corresponding privacy
policy segments of its data type. Second, for sentences without any keywords,
we further employ a Bayesian binary classifier to determine whether the
sentence is related to a data type. The classifier is trained on the dataset
from [103], reaching an accuracy of 0.98. If yes, we use SpaCy [13] Python
library to obtain noun chunks from the sentence. A noun chunk is a phrase that
includes a noun and any connected words such as its adjectives. We calculate
the similarity of the phrases $phrase\\_sim(\,)$ between noun chunks and
keywords as follows:

$\emph{\small phrase\\_sim}\small(\emph{\small p}_{1},\emph{\small
p}_{2}\small)\small=\frac{2\times\emph{path\\_similarity}(\emph{p}_{1},\emph{p}_{2})}{\emph{word\\_count}({\emph{p}_{1}})+\emph{word\\_count}(\emph{p}_{2})}$

where the $\emph{p}_{1}$ and $\emph{p}_{2}$ are a noun chunk and a keyword,
respectively; $\emph{path\\_similarity}(\,)$ is the function from WordNet
[82], reflecting the similarity based on parsing their semantic constituency
trees’ structure; $\emph{word\\_count}(\,)$ is to count the number of words.
If the similarity between a noun chunk and a keyword is higher than 0.8, an
empirically set threshold, then we regard that the sentence that the noun
chunk belongs to is the data type of the keyword.

Above all, we obtain the sentence-level policy segments related to each data
type and group them as privacy policy segments. We also record the positions
of keywords or noun chunks in the resulting sentences. For data types that do
not have any match, we simply add "No relative information is found in the
privacy policy." to notify users. For less popular applications, such
notifications are likely to appear frequently, as they tend not to be
stringent to disclosed privacy practices [108, 76, 66].

###  4.3 CPPs Presentation Module

For each mobile application, privacy policy, and its screenshots, the segment
extraction module and context detection module allow us to obtain the specific
privacy policy segments and associated privacy contexts. We further process
this information to ensure a more coherent and user-friendly presentation.

Initially, we group the detected contexts of the same data types and map them
to their corresponding privacy policy segments. As illustrated in Figure 1c,
the SeePrivacy framework detects three contexts, including two textual GUI
components, i.e., “Share your location” and “use your birthday,” and one
iconic GUI component, such as a location mark icon in the middle right of the
screenshot. These are classified into Location, Location, and Birthday data
types, respectively. The text “Share your location” and the icon of the
location mark are coupled with the segment about location information. The
text “use your birthday” only corresponds to the segment about Birthday data
type. Drawing on existing research that emphasizes the improvement of
readability through the strategic use of typography [85], we render specific
keywords or noun phrases within the privacy policy segments in bold font. This
design choice, grounded in empirical findings, aims to accentuate essential
information and thereby augment user comprehension.

##  5 Evaluation

In this section, we evaluate SeePrivacy both objectively and subjectively.
Some visualization examples are displayed in Figure 4.

###  5.1 Quantitative Evaluation

Our framework, consisting of three integral modules, is systematically
evaluated for its performance on our specially curated benchmark dataset,
named Cpp4App.

####  5.1.1 Context Detection Module

In assessing whether a detected GUI component aligns with the ground truth, we
utilize Intersection over Union (IoU) [92], a prevalent evaluation metric in
object detection models. If the IoU between the ground truth and the detected
component area exceeds a threshold $\beta$, the match is considered correct.
Consistent with standard practice in object detection tasks [89], we
empirically establish $\beta=0.5$.

Textual GUI components. Table 4a presents the results related to the detection
of textual GUI components. Our method exhibits a strong capability to
recognize and categorize texts within mobile GUI screenshots, achieving an
accuracy of 0.82, a precision of 0.87, and a recall of 0.93.

Iconic GUI components. Table 5 shows the comparison of the icon localization
capability between our method and previous methods. The results show that our
proposed method performs better in identifying the icons curated on GUI
screenshots. The failure case analysis is included in the Appendix
LABEL:appendix_failure. Table 4b shows the class-wise breakdown for Iconic GUI
components, achieving an accuracy of 0.92, a precision of 0.95, and a recall
of 0.96. We then compare our icon classification models with several baselines
on the RICO-icon dataset. Table 6 shows that our ViT-based model achieves the
best performance, but has the largest model size of 328 MB and the slowest
time efficiency of 11.28 seconds. The ResNet-based model provides a good
balance of performance and efficiency, with the fast time efficiency, decent
performance metrics, and a much smaller model size compared to others.

Table 5: Performance comparison between iconic GUI components localization
methods.

Method | Precision (mislabel) | Recall (missing label)
---|---|---
UIED [104] | 0.78 | 0.44
Liu et al. [78] | 0.79 | 0.86
Ours | 0.84 | 0.88

Table 6: Performance and efficiency comparison for iconic GUI elements
classification on the RICO-icon dataset. “kNN” stands for k-nearest neighbors
algorithm, and we use the normalized pixel values of the grayscale icon images
as features. “Acc.”, “Prec.” and “Rec.” represents Accuracy, Precision and
Recall, respectively. Time(s) denotes the elapsed seconds to process a
thousand icons.

Model | Acc. | Prec. | Rec. | Size(MB) | Time(s)
---|---|---|---|---|---
kNN [43] | 0.75 | 0.61 | 0.52 | 0 | 6.78
Liu et al. [78] | 0.90 | 0.90 | 0.79 | 293 | 5.02
MobileNetV2 [93] | 0.86 | 0.73 | 0.73 | 9 | 0.56
MobileNetV3 [60] | 0.79 | 0.62 | 0.66 | 6 | 0.52
Ours (ViT-based) | 0.97 | 0.90 | 0.90 | 328 | 11.28
Ours (ResNet-based) | 0.91 | 0.80 | 0.81 | 43 | 0.37

Failure Case Analysis We manually examine failure cases in iconic GUI
components detection and find that most of the missing labels are caused by
overlapping with other components or interleaved boundaries. As for the
mislabelled cases, small squarish images, e.g., item images in shopping apps,
tend to be wrongly included as an icon. Figure 3 shows two examples of failure
cases in our experiment.

(a)

(b)

Figure 3: Failure cases of the mislabel (left, the green bounding box) and the
missing label (right, the camera icon).

Overall context identification. We further assess the overall performance of
our context detection module by considering both textual and iconic GUI
components. Table 4d shows the overall context identification results
considering textual and iconic GUI components. Specifically, SeePrivacy
exhibits commendable results, achieving accuracy, precision, and recall rates
of 0.81, 0.88, and 0.90, respectively.

####  5.1.2 Segment Extraction Module

First, to determine whether the retrieval is successful, we introduce a
similarity metric, denoted by $\emph{segment\\_sim}(\,)$, which quantifies the
similarity between the retrieved segments and the manually annotated ground
truth. For a given ground truth $\emph{s}_{gt}$ and a retrieved segments
$\emph{s}_{ret}$, our segment similarity $\emph{segment\\_sim}(\,)$ is
calculated as follows:

$~{}\begin{aligned}
\emph{segment\\_sim}(\emph{s}_{ret},\emph{s}_{gt})&=\frac{1}{\textsc{Min}(n,m)}\sum_{i=1}^{n}\sum_{j=1}^{m}\overline{\emph{lcs}}(s^{ret}_{i},s^{gt}_{j})\\\
\overline{\emph{lcs}}(\emph{s}^{ret}_{i},\emph{s}^{gt}_{j})&=\frac{\emph{lcs}(s^{ret}_{i},s^{gt}_{j})}{\textsc{Min}(\textsc{Len}(s^{ret}_{i}),\textsc{len}(s^{gt}_{j}))}\end{aligned}$

where $\emph{lcs}(\,)$ stands for the function to calculate longest common
string; $\emph{s}^{ret}_{j}$ are phrases separated by punctuation of retrieved
segments and $\emph{s}^{gt}_{i}$ are labelled phrases of ground truth
segments. If it is greater than 0.8, an empirically set threshold, then the
segment pair is regarded as a successful match. Table 4c shows the results of
the segment extraction module, SeePrivacy achieves 0.94, 0.98, and 0.96 in
terms of accuracy, precision, and recall, respectively. There are some
categories, such as Address, which are more complex than the other categories
due to relatively semantically ambiguous keywords. In addition, there are
relatively less data for Voices and Contacts, thus the limited performance of
this category could be caused by the randomness.

###  5.2 Human Evaluation

To further validate the functionality of SeePrivacy, we conduct a human
evaluation to examine users’ perceptions of its capacity for generating
accurate CPPs for mobile applications. This investigation seeks to provide
empirical evidence on how SeePrivacy performs in real-world scenarios from
users’ perspective, thereby offering insights into its effectiveness and
relevance.

####  5.2.1 Evaluation Study Design

Table 7: Statistic results of our human evaluation. SD stands for standard
deviations.

Topic | Mean | Median | SD
---|---|---|---
Data type & Context | 4.22 | 5 | 1.14
Data type & Policy segment | 3.95 | 4 | 1.28
Policy segment & Context | 3.75 | 4 | 1.40

For recruitment, we recruited examiners based on the following criteria:
professional fluency in English reading and be capable of using mobile apps
without accessibility tools (e.g., VoiceOver on iOS), via mailing lists of
authors’ institution without offering monetary incentives. For the human
evaluation, we randomly sampled 30 screenshots with their privacy policies
from the benchmark dataset and employ SeePrivacy to generate CPPs. The manual
examination involves heavy privacy policy reading and comprehending tasks, and
we aim to reduce the session length to keep the examiners focused and increase
the result validity. The average duration of evaluation session is about 30
minutes. To guarantee the validity of evaluation, we then divide the generated
CPPs into three folds and compose the questionnaire for each fold, i.e., ten
screenshots in each questionnaire. For each CPP, examiners need to rate the
following statements:

  * •

The data type matches with the detected context.

  * •

The data type matches with the displayed privacy policy segment.

  * •

The displayed privacy policy segment explains the detected context on the
screenshot.

For all the ten allocated screenshots, they are asked to rate the agreement
level on a 5-point Likert scale ranging from five (strongly agree) to one
(strongly disagree). In addition to providing a general introduction to the
user study and obtaining consent, we also provide examiners with detailed
descriptions of the data types, explicit instructions, and a concrete example
to guide them prior to the beginning of our study.

####  5.2.2 Evaluation Results

In total, we recruited 15 people for the evaluation (6 females, 8 males, and 1
who preferred not to be mentioned). Each fold of screenshots was evaluated by
five different examiners. The average duration for evaluation sessions is
about 30 minutes. The results are detailed in Table 7. For the functionality
evaluation pertaining to the matching between data types and detected
contexts, the ratings were highly favorable, with more than 81% of the
responses falling within the categories of “Agree” or “Strongly Agree.” The
mean and median values for this aspect were also substantial at 4.22 and 5,
respectively. Concerning the alignment between data types and privacy policy
segments, over 72% of examiners assigned positive ratings, yielding a mean
value of 3.95 and a median value of 4. Additionally, more than 77% of
examiners concurred that the retrieved privacy policy segments help explain
the detected contexts, evidenced by a mean rating of 3.75 and a median rating
of 4. Overall, these results demonstrate that SeePrivacy successfully fulfills
its intended purpose by offering an efficient and effective solution for
delivering CPPs to users.

(a) Sceenshot 6-1

(b) Screenshot 11-9

(c) Screenshot 16-1

Figure 4:  Three examples appeared in the human evaluation.

##  6 Discussion

In this section, we delve into the implications, potential applications, and
contributions of our framework, elucidating its significance both for
individual mobile application users and the wider community.

###  6.1 Privacy Concern and Reading Willingness

Table 8: Pre- and post-questions of the functionality evaluation about the
general usability of SeePrivacy.

No. | Pre- and post-questions | Scale (5-Likert) | Mean | Median | SD
---|---|---|---|---|---
$\textit{Q}_{1}$ |  How concerned are you about your privacy information while using mobile apps? | 5 for very concerned, 1 for very unconcerned | 4.13 | 4 | 1.09
$\textit{Q}_{2}$ |  Do you read mobile app’s privacy policies when you encountered? | 5 for always read, 1 for never read | 2.00 | 2 | 1.03
$\textit{Q}_{3}$ |  What do you think the usefulness of this tool in terms of providing privacy information for mobile apps? | 5 for very useful, 1 for very useless | 4.53 | 5 | 0.62
$\textit{Q}_{4}$ |  Will you read contextual privacy policies when you encountered in future? | 5 for always read, 1 for never read | 4.07 | 4 | 0.85

Before and after the human evaluation, we also asked several questions about
privacy concerns and their privacy policy reading habits to examiners. Table 8
lists the specific questions ($\textit{Q}_{1}$ and $\textit{Q}_{2}$ are pre-
evaluation; $\textit{Q}_{3}$ and $\textit{Q}_{4}$ are post-evaluation) and
results. It is evident that privacy concerns are a prominent issue for users
while using mobile applications, with a mean concern rating of 4.13 and a
median value of 4 for question $\textit{Q}_{1}$. The utility of SeePrivacy in
furnishing relevant privacy information for mobile applications was highly
acclaimed by the participants. Specifically, 14 out of 15 participants
evaluated it as either “Very useful” or “Useful.”

In addition, we compare the reading willingness between traditional privacy
policies and the CPPs generated by SeePrivacy. The results demonstrate a
substantial increase in the willingness of participants to engage with CPPs,
as reflected by a mean score that surged from 2 to 4.1. Furthermore, among 8
participants who are “very concerned” about their privacy information while
using mobile apps, 6 select that they will always read CPPs, and 2 select
often read CPPs. These findings collectively underscore the potential of
SeePrivacy. By enhancing the transparency, accessibility, and digestibility of
privacy policies, our proposed framework plays a vital role in engaging users
to read privacy statements, potentially mitigating the prevalence of “digital
resignation”. Although the diversity of participants is slightly under
representative, we believe those preliminary findings are still valuable to be
reported in this paper.

Figure 5: An example of the application scenario of our framework. (1) The
homepage of the app at the Google Play app store. (2) Basic information of the
app. (3) The showcase screenshots provided by the app developer. (4) The link
of privacy policy. (5) The CPPs generated by SeePrivacy.

###  6.2 Potential Adoption Scenario

####  6.2.1 CPP in App Market

Our framework has the potential to be applied in various real-world scenarios.
One plug-and-play application is displaying CPP in the app market, as
illustrated in Figure 5. Research has shown that users’ app downloading
decisions are often influenced by curated market information such as ratings
and reviews [55]. On the apps’ homepage ([baseline=(char.base)]
[shape=circle,draw,inner sep=0.5pt] (char) 1;, [baseline=(char.base)]
[shape=circle,draw,inner sep=0.5pt] (char) 2;) in app markets (e.g., the
Google Play app store), developers usually curate some showcase screenshots
([baseline=(char.base)] [shape=circle,draw,inner sep=0.5pt] (char) 3;) of
their apps to exhibit some characteristics. Those screenshots often include
the actual GUIs that appear during the users’ actual usage. As a privacy
policy link ([baseline=(char.base)] [shape=circle,draw,inner sep=0.5pt] (char)
4;) is also commonly mandated by app markets, our framework will be able to
generate the CPPs ([baseline=(char.base)] [shape=circle,draw,inner sep=0.5pt]
(char) 5;) by utilizing the showcase screenshots and the corresponding privacy
policy on the homepage. By leveraging SeePrivacy, users can gain
contextualized insights into potential data practices and pertinent policy
statements of the mobile application prior to installation. This application
allows users to comprehend data practices with real-use scenarios, enabling
them to make informative decisions and establish better expectations before
downloading the app.

(a) Detection Result of Contexts

(b) Distribution of Contexts

Figure 6: The left figure shows the detection result of contexts that do not
have information in privacy policies. The right figure presents the data type
distribution of those contexts.

####  6.2.2 Lack of Disclosure: Contextual Detection

Numerous prior studies [108, 107, 103, 102] have highlighted the inconsistency
between statements articulated in privacy policies and an application’s actual
behaviors. Specifically, the absence of proper disclosure concerning data
practices may place users at considerable risk concerning digital security. In
the current landscape of privacy policy analysis, we identify two critical
shortcomings. First, in the case of essential applications, users often find
themselves in a position where they must accept the terms, even if they are
made aware of any lack of disclosure within the policies. There is no space to
negotiate with developers for specific functions or certain GUI components
under the “all-or-nothing” model. Second, existing methodologies fail to link
the noted inconsistencies to the real contexts with which users may engage on
GUIs. This omission presents a significant gap in understanding how policy
misalignment may manifest in a user’s interaction with an application. The
lack of notice and choice inevitably leads to a digital resignation
phenomenon, where users acquiesce to potentially problematic terms out of
necessity, rather than informed consent.

To address the aforementioned challenges, SeePrivacy is proposed to provide
the affected GUI elements, i.e., the context, allowing users to make informed
decisions about their interactions. For example, in Figure 4(c), SeePrivacy
detects three contexts related to “Photos” in red bounding boxes, but does not
obtain any relevant statements in the privacy policy of the app. Therefore,
“No relative information is found in the privacy policy” is displayed to warn
users about the potential risk. Users can simply choose not to interact with
those “Photos”-related GUI elements as a band-aid solution, meanwhile “safely”
using other features of the app. In total, we found that 60 contexts do not
have corresponding privacy policy segments in Cpp4App. The 60 contexts are
distributed among 17 mobile applications in Cpp4App and cover all 12 data
types. Figure 6 illustrates the specific numbers and the distribution for lack
of disclosure. Among the various data types, Photos and Voices combined
constitute 50% of the data. Interestingly, the Voices data type seems to be
frequently overlooked. Contexts for this data type typically involve requests
to record audio (e.g., a microphone icon), or the conversion of voice to text
for search functionality (e.g., a phone icon). In total, SeePrivacy
successfully identified 63.3% (38/60) lack of disclosure contexts.

###  6.3 Broader Impacts

Legal Impacts. Privacy regulations frequently stipulate specific requirements
regarding the completeness of privacy practices within policies. For instance,
GDPR mandates that privacy notices disclose the categories of personal data
concerned [GDPR, Art. 14(1)(d)] and recipients or categories of recipients of
the personal data [Art. 13(1)(e), 14(1)(e)]. Likewise, the CCPA prescribes
that privacy notices should enumerate the categories of personal information
collected in the preceding 12 months [CCPA §1798.130(a)(5)(B), §1798.110(c),
Regs §999.308(c)(1)(d)]. While such requirements are often adhered to by
mobile application developers and can be directly assessed by regulators, the
challenges of formulating best practices to achieve transparency, and
evaluating the transparency level of a privacy statement, persist. In this
context, our innovative definition of CPPs within the mobile scenario emerges
as a potential metric for evaluating transparency. By linking the stated
privacy practices to the actual GUI elements with which users interact,
SeePrivacy offers an empirical means of ensuring that privacy statements are
not only compliant with legal requirements but also meaningfully transparent
to end-users. This approach aligns with the broader legal and ethical
imperatives of user-centric privacy and could contribute significantly to the
development of transparent and accountable privacy practices in the mobile
application ecosystem.

Accessibility and Inclusion. By transforming privacy policies into a
contextual and understandable format, our approach potentially broadens their
accessibility to a more diverse range of users. This includes individuals who
may not possess advanced literacy skills as well as those who face challenges
related to reading disorders. Such inclusivity not only democratizes the
understanding of privacy policies but also aligns with broader societal goals
of ensuring that vital information is comprehensible to a wide audience. In
this regard, SeePrivacy contributes to creating a more equitable digital
environment where awareness and control over personal data are not confined to
a specific subset of users but are extended to encompass various demographic
groups. The potential reach and impact of this approach underscore the need
for continued research and innovation in the domain of user-centered privacy
communication.

Children and Education. The GDPR Recital 58 (The Principle of Transparency)
states “… any information and communication, where processing is addressed to
a child, should be in such a clear and plain language that the child can
easily understand.” This principle acknowledges the unique challenge of making
legal and privacy-related texts accessible to young audiences. The SeePrivacy
framework aligns with this directive by transforming privacy policies into
potentially engaging and comprehensible formats to children. By connecting
formal legal terms with real privacy use cases, it can facilitate a more
straightforward understanding of the implications of online activities. While
SeePrivacy strives to comply with regulatory requirements, its broader goal is
to contribute to the digital literacy of disadvantage groups. This effort
seeks to provide them with a basic understanding of online privacy, aiming to
equip the next generation with the knowledge needed to approach the digital
world with informed caution and responsibility. SeePrivacy aligns with the
growing emphasis on the creation of tools and resources specifically designed
to meet the developmental needs and competencies of young internet users.

###  6.4 Notice-and-Choice Privacy Framework for Mobile Apps: Is This the Way?

The notice-and-choice framework for mobile application ecosystem is
fundamentally challenged by two critical issues. First, there is a significant
disconnection between privacy notices and user-oriented interactions, leading
to ineffective engagement of users in meaningful “notice” processes and
subsequent privacy decision-making. Second, as illustrated in Figure 1(a) and
1(b), the current designs of privacy controls often present users with an all-
or-nothing choice, leaving them without genuine control over their data
privacy. Therefore, this imbalance between utility and privacy highlights the
need for administrative action through the regulation establishment and
enforcement to safeguard consumer privacy rights.

While SeePrivacy is not “panacea” for these challenges, it enlightens “a new
hope” in this landscape. It suggests a paradigm shift in the privacy notice
(what users are agreeing to) and choice (to agree or not) from a generic
application level to specific functions or certain GUI components. It is
critical to proactively elevate privacy considerations from non-functional
requirements to functional requirements in software design and development.
Our framework aims to actively prompt users with timely and contextually
relevant privacy information, enabling the potential for more granular control
in future.

###  6.5 Limitations.

Our current human evaluation focuses more on the functionality of SeePrivacy,
i.e., the level of matching between contexts and retrieved policy segments,
thus, all examiners are relatively well-educated in technology. Thus, the
preliminary findings of reading willingness and popularity of CPPs among them
may be biased due to the small and non-representative sample. We plan to
further conduct a large-scale user study that includes people from diverse
backgrounds in the future. In addition, the exploration of fatigue and
habituation within human-computer interaction and its impact on interaction
design have not been fully developed. These factors may profoundly affect user
engagement and interface efficacy. Furthermore, the time and storage
efficiency of SeePrivacy were evaluated theoretically and may not reflect
real-world performance in different application scenarios. Regarding
generalizability, our CPP detection is entirely vision-based, allowing for
straightforward adaptation to other platforms, such as iOS mobile
applications.

##  7 Conclusion and Future Work

Privacy policies have long been criticized for their complexity and poor
readability, leading to a growing interest in contextual privacy policies
(CPPs) as user-focused alternatives. In this paper, we introduce the concept
of CPPs specifically within the domain of mobile applications and propose a
novel framework, SeePrivacy, designed to automatically generate these privacy
policies. We rigorously evaluated SeePrivacy on our benchmark dataset,
Cpp4App, which consists of over 1,200 privacy-related contexts and
corresponding privacy policy segments. In the quantitative evaluation, our
framework demonstrates 0.88 precision and 0.90 recall to detect contexts; as
well as 0.98 precision and 0.96 recall to extract corresponding policy
segments. Furthermore, a human evaluation was conducted to validate the
functionality our approach, reflected by a median score of 4 on a 5-point
Likert scale. Our pre- and post-survey revealed that people exhibit much
greater willingness to engage with CPPs, scoring 4.1 out of 5, compared to a
mere 2 out of 5 for traditional privacy policies. We also discussed the
potential applications in real scenarios, and implications of CPPs and
SeePrivacy to the wider community.

Our promising results identify two potential research directions. First, while
our current efforts emphasize methodological development and evaluation, we
recognize the importance of exploring real-world deployment scenarios. The
user experience, particularly how users interact with contextual privacy
policies, is vital and could significantly impact the system’s effectiveness.
Future work may include integrating SeePrivacy into specific human-device
interactions, such as live applications on smartphones or sandbox environments
within which applications operate. Second, although we aim to deliver the
original privacy policy content without any modification, we notice that some
retrieved policy segments are too long to comprehend in a timely manner.
Thereby, it is worthwhile to evaluate how to digest lengthy policy segments
with minimum information loss and distortion to maintain their integrity and
practicability simultaneously.

Last, we believe this work provides a substantial step forward in delivering
transparent privacy notices, offering insights into the transition from
privacy notices to informed user choices, and paving the way for future
research and practical implementation in this crucial area.

Data Availability. Benchmark dataset Cpp4App and the replication package are
available at: https://github.com/Cpp4App/Cpp4App.

## Acknowledgments

This research / project is supported by CSIRO’s Data61 PhD Scholarship, and
the National Research Foundation, Singapore, and Cyber Security Agency of
Singapore under its National Cybersecurity Research and Development Programme,
NCRP25-P03-NCR-TAU. Any opinions, findings and conclusions or recommendations
expressed in this material are those of the authors and do not reflect the
views of CSIRO’s Data61, National Research Foundation, Singapore and Cyber
Security Agency of Singapore.

## References

  * [1] 10 essential types of app screens in mobile ui design - cadabra studio.  https://cadabra.studio/blog/10-types-of-app-screens-you-should-know/, Accessed: 2023-02-21.
  * [2] App design 9 main types of app screens.  https://decode.agency/article/app-screens-design/, Accessed: 2023-02-21.
  * [3] Permissions for apps on the amazon appstore.  https://www.amazon.com/gp/help/customer/display.html/ref=hp_app_paaa?nodeId=GLS4D99DFBLKW3XW&ref_=mas_dp_perm, Accessed: 2023-05-10.
  * [4] California consumer privacy act of 2018 (CCPA).  https://oag.ca.gov/privacy/ccpa, Accessed: 2022-04-25.
  * [5] Children’s online privacy protection rule (COPPA).  https://www.ftc.gov/legal-library/browse/rules/childrens-online-privacy-protection-rule-coppa, Accessed: 2023-02-20.
  * [6] The online privacy protection act of 2003 (CalOPPA).  https://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?lawCode=BPC&sectionNum=22575, Accessed: 2023-02-20.
  * [7] Fair information practice principles (fipps).  https://www.fpc.gov/resources/fipps/, Accessed: 2023-04-05.
  * [8] General data protection regulation (GDPR).  https://gdpr-info.eu/, Retrieved: 2022-04-25.
  * [9] Selecting collection service scenarios and data items to be collected upon app release.  https://developer.huawei.com/consumer/en/doc/distribution/app/privacy-label, Accessed: 2023-02-20.
  * [10] Guide to the general data protection regulation (gdpr).  https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/, Accessed: 2023-04-05.
  * [11] Opencv.  https://pypi.org/project/opencv-python/, Accessed: 2023-07-24.
  * [12] Selenium.  https://www.selenium.dev/, Accessed: 2022-10-01.
  * [13] Spacy.  https://github.com/explosion/spaCy, Accessed: 2023-05-01.
  * [14] Unctad.  https://unctad.org/page/data-protection-and-privacy-legislation-worldwide, Accessed: 2022-08-26.
  * app [a] AppBrain.com: Monetize, advertise and analyze Android apps, a.  https://www.appbrain.com/, Accessed: 2023-02-15.
  * app [b] App privacy details on the app store, b.  https://developer.apple.com/app-store/app-privacy-details/, Accessed: 2022-04-30.
  * app [c] App privacy details on the app store, c.  https://developer.apple.com/app-store/app-privacy-details/, Accessed: 2023-02-20.
  * [18] What is personal information?  https://www.oaic.gov.au/privacy/your-privacy-rights/your-personal-information/what-is-personal-information, Accessed: 2023-02-13.
  * [19] Beautifulsoup.  https://www.crummy.com/software/BeautifulSoup/, Accessed: 2022-10-01.
  * [20] What is considered personal data under the EU GDPR?  https://gdpr.eu/eu-gdpr-personal-data, Accessed: 2023-02-13.
  * goo [a] Understand app privacy & security practices with google play’s data safety section, a.  https://support.google.com/googleplay/answer/11416267, Accessed: 2023-02-20.
  * goo [b] Android developers, google play console help center, b.  https://support.google.com/googleplay/android-developer/answer/9859455?hl=en#:~:text=The%20privacy%20policy%20must%2C%20together,you%20of%20what%20is%20required, Accessed: 2022-04-30.
  * [23] Stack overflow - how to request all permission needed when the time user install the app in google play?  https://stackoverflow.com/questions/44985078/, Accessed: 2023-04-20.
  * [24] langdetect.  https://pypi.org/project/langdetect/, Accessed: 2022-10-01.
  * [25] Samsung galaxy store.  https://galaxystore.samsung.com/apps, Accessed: 2023-05-04.
  * [26] Guide to Identifying Personally Identifiable Information (PII).  https://www.technology.pitt.edu/help-desk/how-to-documents/guide-identifying-personally-identifiable-information-pii, Accessed: 2023-02-13.
  * [27] Guidance on the Protection of Personal Identifiable Information.  https://www.dol.gov/general/ppii, Accessed: 2023-02-13.
  * FTC [2010] Fair information practice principles, 2010.  https://web.archive.org/web/20100309105100/http://www.ftc.gov/reports/privacy3/fairinfo.shtm#Notice/Awareness, Archived March 9, 2010, at the Wayback Machine.
  * Adams [2020] Paul C Adams.  Agreeing to surveillance: Digital news privacy policies.  _Journalism & Mass Communication Quarterly_, 97(4):868–889, 2020.
  * Amos et al. [2021] Ryan Amos, Gunes Acar, Eli Lucherini, Mihir Kshirsagar, Arvind Narayanan, and Jonathan Mayer.  Privacy policies over time: Curation and analysis of a million-document dataset.  In _Proceedings of the Web Conference 2021_ , pages 2165–2176, 2021.
  * Andow et al. [2019] Benjamin Andow, Samin Yaseer Mahmud, Wenyu Wang, Justin Whitaker, William Enck, Bradley Reaves, Kapil Singh, and Tao Xie.  Policylint: Investigating internal privacy policy contradictions on google play.  In _USENIX Security Symposium_ , pages 585–602, 2019.
  * Andow et al. [2020] Benjamin Andow, Samin Yaseer Mahmud, Justin Whitaker, William Enck, Bradley Reaves, Kapil Singh, and Serge Egelman.  Actions speak louder than words:$\\{$Entity-Sensitive$\\}$ privacy policy and data flow analysis with $\\{$PoliCheck$\\}$.  In _29th USENIX Security Symposium (USENIX Security 20)_ , pages 985–1002, 2020.
  * AppVelocity [2020] AppVelocity.  A complete understanding of mobile app vs. mobile website, 2020.  URL https://www.appvelocity.ca/blog/mobile-app-vs-mobile-website.  Accessed: 2023-10-17.
  * Blakkarly and Graham [2022] Jarni Blakkarly and Daniel Graham.  Privacy policy comparison reveals half have poor readability, 2022.  URL https://www.choice.com.au/consumers-and-data/protecting-your-data/data-laws-and-regulation/articles/privacy-policy-comparison.  Accessed: 2022-10-10.
  * Bolchini et al. [2004] Davide Bolchini, Qingfeng He, Annie I Antón, and William Stufflebeam.  “i need it now”: Improving website usability by contextualizing privacy policies.  In _Web Engineering: 4th International Conference, ICWE 2004, Munich, Germany, July 26-30, 2004. Proceedings 4_ , pages 31–44. Springer, 2004.
  * Buck [2023] Andrew Buck.  Mobile apps vs mobile websites: Which is best for 2023?, 2023.  URL https://www.mobiloud.com/blog/mobile-apps-vs-mobile-websites.  Accessed: 2023-10-17.
  * Bui et al. [2023] Duc Bui, Brian Tang, and Kang G Shin.  Detection of inconsistencies in privacy practices of browser extensions.  In _2023 IEEE Symposium on Security and Privacy (SP)_ , pages 2780–2798. IEEE, 2023.
  * Bunian et al. [2021] Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr.  Vins: Visual search for mobile user interface design.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , pages 1–14, 2021.
  * Caramujo and Da Silva [2015] João Caramujo and Alberto Manuel Rodrigues Da Silva.  Analyzing privacy policies based on a privacy-aware profile: The facebook and linkedin case studies.  In _2015 IEEE 17th Conference on Business Informatics_ , volume 1, pages 77–84. IEEE, 2015.
  * Chen et al. [2020] Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li.  Object detection for graphical user interface: Old fashioned or deep learning or a combination?  In _proceedings of the 28th ACM joint meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , pages 1202–1214, 2020.
  * Chen et al. [2022] Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, and Xiaoyi Zhang.  Towards complete icon labeling in mobile applications.  In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , pages 1–14, 2022.
  * Ciocchetti [2008] Corey A Ciocchetti.  The future of privacy policies: A privacy nutrition label filled with fair information practices.  _J. Marshall J. Computer & Info. L._, 26:1, 2008.
  * Cover and Hart [1967] Thomas Cover and Peter Hart.  Nearest neighbor pattern classification.  _IEEE transactions on information theory_ , 13(1):21–27, 1967.
  * Cranor [2002] Lorrie Cranor.  _Web privacy with P3P_.  " O’Reilly Media, Inc.", 2002.
  * Cranor [2012] Lorrie Faith Cranor.  Necessary but not sufficient: Standardized mechanisms for privacy notice and choice.  _J. on Telecomm. & High Tech. L._, 10:273, 2012.
  * Cui et al. [2023] Hao Cui, Rahmadi Trimananda, Athina Markopoulou, and Scott Jordan.  $\\{$PoliGraph$\\}$: Automated privacy policy analysis using knowledge graphs.  In _32nd USENIX Security Symposium (USENIX Security 23)_ , pages 1037–1054, 2023.
  * Deka et al. [2017] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar.  Rico: A mobile app dataset for building data-driven design applications.  In _Proceedings of the 30th annual ACM symposium on user interface software and technology_ , pages 845–854, 2017.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Draper and Turow [2019] Nora A Draper and Joseph Turow.  The corporate cultivation of digital resignation.  _New media & society_, 21(8):1824–1839, 2019.
  * Efroni et al. [2019] Zohar Efroni, Jakob Metzger, Lena Mischau, and Marie Schirmbeck.  Privacy icons: a risk-based approach to visualisation of data processing.  _Eur. Data Prot. L. Rev._ , 5:352, 2019.
  * Felt et al. [2010] Adrienne Porter Felt, Kate Greenwood, and David Wagner.  The effectiveness of install-time permission systems for third-party applications.  _University of California at Berkely, Electrical Engineering and Computer Sciences, Technical report_ , 2010.
  * Feth [2017] Denis Feth.  Transparency through contextual privacy statements.  In _Mensch und Computer 2017 - Workshopband_ , 2017.  doi: 10.18420/muc2017-ws05-0406.
  * Flavián and Guinalíu [2006] Carlos Flavián and Miguel Guinalíu.  Consumer trust, perceived security and privacy policy: three basic elements of loyalty to a web site.  _Industrial management & data Systems_, 106(5):601–620, 2006.
  * Genc-Nayebi and Abran [2017] Necmiye Genc-Nayebi and Alain Abran.  A systematic literature review: Opinion mining studies from mobile app store user reviews.  _Journal of Systems and Software_ , 125:207–219, 2017.
  * Harkous et al. [2018] Hamza Harkous, Kassem Fawaz, Rémi Lebret, Florian Schaub, Kang G Shin, and Karl Aberer.  Polisis: Automated analysis and presentation of privacy policies using deep learning.  In _27th $\\{$USENIX$\\}$ security symposium ($\\{$USENIX$\\}$ security 18)_, pages 531–548, 2018.
  * Harkous et al. [2022] Hamza Harkous, Sai Teja Peddinti, Rishabh Khandelwal, Animesh Srivastava, and Nina Taft.  Hark: A deep learning system for navigating privacy feedback at scale.  In _2022 IEEE Symposium on Security and Privacy (SP)_ , pages 2469–2486. IEEE, 2022.
  * Holtz et al. [2011a] Leif-Erik Holtz, Katharina Nocun, and Marit Hansen.  Towards displaying privacy information with icons.  In _Privacy and Identity Management for Life: 6th IFIP WG 9.2, 9.6/11.7, 11.4, 11.6/PrimeLife International Summer School, Helsingborg, Sweden, August 2-6, 2010, Revised Selected Papers 6_ , pages 338–348. Springer, 2011a.
  * Holtz et al. [2011b] Leif-Erik Holtz, Harald Zwingelberg, and Marit Hansen.  Privacy policy icons.  In _Privacy and Identity Management for Life_ , pages 279–285. Springer, 2011b.
  * Howard et al. [2019] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1314–1324, 2019.
  * Kaur et al. [2018] Jasmin Kaur, Rozita A Dara, Charlie Obimbo, Fei Song, and Karen Menard.  A comprehensive keyword analysis of online privacy policies.  _Information Security Journal: A Global Perspective_ , 27(5-6):260–275, 2018.
  * Kelley et al. [2009] Patrick Gage Kelley, Joanna Bresee, Lorrie Faith Cranor, and Robert W Reeder.  A" nutrition label" for privacy.  In _Proceedings of the 5th Symposium on Usable Privacy and Security_ , pages 1–12, 2009.
  * Kelley et al. [2010] Patrick Gage Kelley, Lucian Cesca, Joanna Bresee, and Lorrie Faith Cranor.  Standardizing privacy notices: an online study of the nutrition label approach.  In _Proceedings of the SIGCHI Conference on Human factors in Computing Systems_ , pages 1573–1582, 2010.
  * Kelley et al. [2013] Patrick Gage Kelley, Lorrie Faith Cranor, and Norman Sadeh.  Privacy as part of the app decision-making process.  In _Proceedings of the SIGCHI conference on human factors in computing systems_ , pages 3393–3402, 2013.
  * Kemp [2020] Katharine Kemp.  Concealed data practices and competition law: why privacy matters.  _European Competition Journal_ , 16(2-3):628–672, 2020.
  * Kumar et al. [2022] Renuka Kumar, Apurva Virkud, Ram Sundara Raman, Atul Prakash, and Roya Ensafi.  A large-scale investigation into geodifferences in mobile apps.  In _31st USENIX Security Symposium (USENIX Security 22)_ , Boston, MA, August 2022. USENIX Association.  URL https://www.usenix.org/conference/usenixsecurity22/presentation/kumar.
  * LABS [2022] AMPLITUDE LABS.  2022 app vs. website report, 2022.  URL https://amplitude.com/guides/2022-app-vs-website-report.  Accessed: 2023-10-17.
  * Landesberg et al. [1998] Martha K Landesberg, Toby Milgrom Levin, Caroline G Curtin, and Ori Lev.  Privacy online: A report to congress.  _NASA_ , (19990008264), 1998.
  * Leiva et al. [2020] Luis A Leiva, Yunfei Xue, Avya Bansal, Hamed R Tavakoli, Tuðçe Köroðlu, Jingzhou Du, Niraj R Dayama, and Antti Oulasvirta.  Understanding visual saliency in mobile user interfaces.  In _22nd International conference on human-computer interaction with mobile devices and services_ , pages 1–12, 2020.
  * Li et al. [2022a] Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin, Kaitao Jiang, Yongkun Du, Yuning Du, Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, et al.  Pp-ocrv3: More attempts for the improvement of ultra lightweight ocr system.  _arXiv preprint arXiv:2206.03001_ , 2022a.
  * Li et al. [2023] Jingjie Li, Kaiwen Sun, Brittany Skye Huff, Anna Marie Bierley, Younghyun Kim, Florian Schaub, and Kassem Fawaz.  “it’s up to the consumer to be smart”: Understanding the security and privacy attitudes of smart home users on reddit.  In _IEEE Symposium on Security and Privacy (SP)(SP)_ , pages 380–396. IEEE Computer Society Los Alamitos, CA, 2023.
  * Li et al. [2019] Li Li, Tegawendé F Bissyandé, and Jacques Klein.  Rebooting research on detecting repackaged android apps: Literature review and benchmark.  _IEEE Transactions on Software Engineering_ , 47(4):676–693, 2019.
  * Li et al. [2021] Tianshi Li, Elizabeth Louie, Laura Dabbish, and Jason I Hong.  How developers talk about personal data and what it means for user privacy: A case study of a developer forum on reddit.  _Proceedings of the ACM on Human-Computer Interaction_ , 4(CSCW3):1–28, 2021.
  * Li et al. [2022b] Tianshi Li, Kayla Reiman, Yuvraj Agarwal, Lorrie Faith Cranor, and Jason I Hong.  Understanding challenges for developers to create accurate privacy nutrition labels.  In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , pages 1–24, 2022b.
  * Li et al. [2022c] Yucheng Li, Deyuan Chen, Tianshi Li, Yuvraj Agarwal, Lorrie Faith Cranor, and Jason I Hong.  Understanding ios privacy nutrition labels: An exploratory large-scale analysis of app store data.  In _CHI Conference on Human Factors in Computing Systems Extended Abstracts_ , pages 1–7, 2022c.
  * Liu et al. [2021] Shuang Liu, Baiyang Zhao, Renjie Guo, Guozhu Meng, Fan Zhang, and Meishan Zhang.  Have you been properly notified? automatic compliance analysis of privacy policy text with gdpr article 13.  In _Proceedings of the Web Conference 2021_ , pages 2154–2164, 2021.
  * Liu et al. [2023] Shuang Liu, Fan Zhang, Baiyang Zhao, Renjie Guo, Tao Chen, and Meishan Zhang.  Appcorp: a corpus for android privacy policy document structure analysis.  _Frontiers of Computer Science_ , 17(3):173320, 2023.
  * Liu et al. [2018] Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar.  Learning design semantics for mobile apps.  In _The 31st Annual ACM Symposium on User Interface Software and Technology_ , UIST ’18, pages 569–579, New York, NY, USA, 2018. ACM.  ISBN 978-1-4503-5948-1.  doi: 10.1145/3242587.3242650.  URL http://doi.acm.org/10.1145/3242587.3242650.
  * Ma [2014] Zhongmin Ma.  _Android application install-time permission validation and run-time malicious pattern detection_.  PhD thesis, Virginia Tech, 2014.
  * Malavolta et al. [2015] Ivano Malavolta, Stefano Ruberto, Tommaso Soru, and Valerio Terragni.  End users’ perception of hybrid mobile apps in the google play store.  In _2015 IEEE International Conference on Mobile Services_ , pages 25–32. IEEE, 2015.
  * Masotina and Spagnolli [2022] Mariavittoria Masotina and Anna Spagnolli.  Transparency of privacy notices and contextualisation: effectively conveying information without words.  _Behaviour & Information Technology_, 41(10), 2022.
  * Miller [1995] George A Miller.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41, 1995.
  * Obar and Oeldorf-Hirsch [2020] Jonathan A Obar and Anne Oeldorf-Hirsch.  The biggest lie on the internet: Ignoring the privacy policies and terms of service policies of social networking services.  _Information, Communication & Society_, 23(1):128–147, 2020.
  * Ortloff et al. [2020] Anna-Marie Ortloff, Maximiliane Windl, Valentin Schwind, and Niels Henze.  Implementation and in situ assessment of contextual privacy policies.  In _Proceedings of the 2020 ACM Designing Interactive Systems Conference_ , pages 1765–1778, 2020.
  * Palmén et al. [2023] Hilary Palmén, Michael Gilbert, and David Crossland.  How bold can we be? the impact of adjusting font grade on readability in light and dark polarities.  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , pages 1–11, 2023.
  * Pan et al. [2023a] Shidong Pan, Thong Hoang, Dawen Zhang, Zhenchang Xing, Xiwei Xu, Qinghua Lu, and Mark Staples.  Toward the cure of privacy policy reading phobia: Automated generation of privacy nutrition labels from privacy policies.  _arXiv preprint arXiv:2306.10923_ , 2023a.
  * Pan et al. [2023b] Shidong Pan, Dawen Zhang, Mark Staples, Zhenchang Xing, Jieshan Chen, Xiwei Xu, and James Hoang.  A large-scale empirical study of online automated privacy policy generators for mobile apps.  _arXiv preprint arXiv:2305.03271_ , 2023b.
  * Perez et al. [2018] Alfredo J Perez, Sherali Zeadally, and Jonathan Cochran.  A review and an empirical analysis of privacy policy and notices for consumer internet of things.  _Security and Privacy_ , 1(3):e15, 2018.
  * Pushp [2023] Pushkar Pushp.  Understanding the iou metric in object detection, 2023.  URL https://pub.towardsai.net/understanding-iou-metric-in-object-detection-1e5532f06a76#.  Accessed: 2023-07-24.
  * Qi et al. [2020] Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning.  Stanza: A Python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 2020.  URL https://nlp.stanford.edu/pubs/qi2020stanza.pdf.
  * Redmon et al. [2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 779–788, 2016.
  * Rezatofighi et al. [2019] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 658–666, 2019.
  * Sandler et al. [2018] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4510–4520, 2018.
  * Sun and Xue [2020] Ruoxi Sun and Minhui Xue.  Quality assessment of online automated privacy policy generators: an empirical study.  In _Proceedings of the Evaluation and Assessment in Software Engineering_ , pages 270–275. 2020.
  * Torre et al. [2020] Damiano Torre, Sallam Abualhaija, Mehrdad Sabetzadeh, Lionel Briand, Katrien Baetens, Peter Goes, and Sylvie Forastier.  An ai-assisted approach for checking the completeness of privacy policies against gdpr.  In _2020 IEEE 28th International Requirements Engineering Conference (RE)_. IEEE, 2020.
  * Wang et al. [2018] Haoyu Wang, Zhe Liu, Jingyue Liang, Narseo Vallina-Rodriguez, Yao Guo, Li Li, Juan Tapiador, Jingcun Cao, and Guoai Xu.  Beyond google play: A large-scale comparative study of chinese android app markets.  In _Proceedings of the Internet Measurement Conference 2018_ , pages 293–307, 2018.
  * Wei and Wolf [2017] Xuetao Wei and Michael Wolf.  A survey on https implementation by android apps: issues and countermeasures.  _Applied Computing and Informatics_ , 13(2):101–117, 2017.
  * Wilson et al. [2016] Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N Cameron Russell, et al.  The creation and analysis of a website privacy policy corpus.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1330–1340, 2016.
  * Windl et al. [2022] Maximiliane Windl, Niels Henze, Albrecht Schmidt, and Sebastian S Feger.  Automating contextual privacy policies: Design and evaluation of a production tool for digital consumer privacy awareness.  In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , pages 1–18, 2022.
  * Xi et al. [2019] Shengqu Xi, Shao Yang, Xusheng Xiao, Yuan Yao, Yayuan Xiong, Fengyuan Xu, Haoyu Wang, Peng Gao, Zhuotao Liu, Feng Xu, et al.  Deepintent: Deep icon-behavior learning for detecting intention-behavior discrepancy in mobile apps.  In _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , pages 2421–2436, 2019.
  * Xiao et al. [2019] Xusheng Xiao, Xiaoyin Wang, Zhihao Cao, Hanlin Wang, and Peng Gao.  Iconintent: automatic identification of sensitive ui widgets based on icon classification for android apps.  In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_ , pages 257–268. IEEE, 2019.
  * [102] Yue Xiao, Zhengyi Li, Yue Qin, Xiaolong Bai, Jiale Guan, Xiaojing Liao, and Luyi Xing.  Lalaine: Measuring and characterizing non-compliance of apple privacy labels.
  * Xie et al. [2022] Fuman Xie, Yanjun Zhang, Chuan Yan, Suwan Li, Lei Bu, Kai Chen, Zi Huang, and Guangdong Bai.  Scrutinizing privacy policy compliance of virtual personal assistant apps.  In _37th IEEE/ACM International Conference on Automated Software Engineering_ , pages 1–13, 2022.
  * Xie et al. [2020] Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen.  Uied: a hybrid tool for gui element detection.  In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , pages 1655–1659, 2020.
  * Yu et al. [2015] Le Yu, Tao Zhang, Xiapu Luo, and Lei Xue.  Autoppg: Towards automatic generation of privacy policy for android applications.  In _Proceedings of the 5th Annual ACM CCS Workshop on Security and Privacy in Smartphones and Mobile Devices_ , SPSM ’15, page 39–50, New York, NY, USA, 2015. Association for Computing Machinery.  ISBN 9781450338196.  doi: 10.1145/2808117.2808125.  URL https://doi.org/10.1145/2808117.2808125.
  * Yu et al. [2018] Le Yu, Xiapu Luo, Chenxiong Qian, Shuai Wang, and Hareton K. N. Leung.  Enhancing the description-to-behavior fidelity in android apps with privacy policy.  _IEEE Transactions on Software Engineering_ , 44(9), 2018.  doi: 10.1109/TSE.2017.2730198.
  * Zhang et al. [2020] Xueling Zhang, Xiaoyin Wang, Rocky Slavin, Travis Breaux, and Jianwei Niu.  How does misconfiguration of analytic services compromise mobile privacy?  In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , pages 1572–1583, 2020.
  * Zimmeck et al. [2019] Sebastian Zimmeck, Peter Story, Daniel Smullen, Abhilasha Ravichander, Ziqi Wang, Joel Reidenberg, N Cameron Russell, and Norman Sadeh.  Maps: Scaling privacy compliance analysis to a million apps.  _Proceedings on Privacy Enhancing Technologies_ , 2019(3):66–86, 2019.
  * Zimmeck et al. [2021] Sebastian Zimmeck, Rafael Goldstein, and David Baraka.  Privacyflash pro: Automating privacy policy generation for mobile apps.  In _NDSS_ , 2021.

##  8 Appendix

###  8.1 Data Types in Privacy Regulations

What is “personal identifier” in this study? Personal Identifier (PID)
commonly appears in data protection and privacy legislation. They have
different definitions and may be further specified according to the
circumstances [27, 20, 26, 18]. If there is no further explanation in the
legislative context, we only take the less controversial subset of common PID
examples, Name, Date of Birthday, Address, Phone Number, Email, and unique
Account Info to avoid the unintended comprehension.

The sources that we refer for the definition and interpretation of “personal
information” in privacy regulations are listed as below:

  * •

GDPR: https://gdpr.eu/eu-gdpr-personal-data

  * •

CCPA: https://oag.ca.gov/privacy/ccpa

  * •

CalOPPA:https://leginfo.legislature.ca.gov/faces/codes_displayText.xhtml?division=8.&chapter=22.&lawCode=BPCandhttps://termageddon.com/caloppa-
personal-information/

  * •

COPPA:https://www.ftc.gov/business-guidance/resources/complying-coppa-
frequently-asked-questions

  * •

APP:https://www.oaic.gov.au/privacy/your-privacy-rights/your-personal-
information/what-is-personal-information

###  8.2 Human Evaluation Breakdown

Table 9 shows the statistic results of agreement of retrieved privacy policy
segments explain detected context for each fold.

Table 9: Statistic results of our user study in each fold.

| Fold 1 | Fold 2 | Fold 3
---|---|---|---
Topic | Mean | Median | Mean | Median | Mean | Median
Data type & Context | 4.16 | 5 | 4.43 | 5 | 4.09 | 4
Data type & Policy segment | 3.87 | 4 | 4.04 | 5 | 3.96 | 4
Policy segment & Context | 3.73 | 4 | 3.63 | 4 | 3.80 | 4

###  8.3 Visualization Examples of Contextual Detection Module

A collection of high-resolution examples of contexts detected by our framework
is shown in Figure 7. (See next page.)

(a) Screenshot 1-1

(b) Screenshot 1-4

(c) Screenshot 4-2

(d) Screenshot 6-2

(e) Screenshot 6-3

(f) Screenshot 6-10

Figure 7: Visualization examples: The screenshot results after CDM.