# Privacy in ChatGPT: A Vision, Model, Trends and Research Directions

Imdad Ullah, Najm Hassan, Sukhpal Singh Gill, Basem Suleiman, Tariq Ahamed
Ahanger, Zawar Shah, Junaid Qadir, and Salil S. Kanhere I. Ullah and B.
Suleiman are with the School of Computer Science Faculty of Engineering, The
University of Sydney, Sydney NSW 2006, Australia.
imdad.ullah<EMAIL_ADDRESS>Hassan is with the Higher Colleges
of Technology, United Arab Emirates (UAE<EMAIL_ADDRESS>S. Gill is with
the School of Electronic Engineering and Computer Science, Queen Mary
University of London, London, UK.
<EMAIL_ADDRESS>A. Ahangar is with the Management Information Systems
Department, College of Business Administration, Prince Sattam bin Abdulaziz
University, Al-Kharj 16278, Saudi Arabia.
<EMAIL_ADDRESS>Shah is with the Department of Information Technology,
Sydney International School of Technology and Commerce, Sydney NSW 2000,
Australia<EMAIL_ADDRESS>Qadir is with the Department of Computer
Science and Engineering, College of Engineering, Qatar University, Doha,
Qatar<EMAIL_ADDRESS>S. Kanhere is with the School of Computer Science
and Engineering, The University of New South Wales (UNSW), Sydney, Australia.
<EMAIL_ADDRESS>Author: Imdad Ullah

# Privacy Preserving Large Language Models:
ChatGPT Case Study Based Vision and Framework

Imdad Ullah, Najm Hassan, Sukhpal Singh Gill, Basem Suleiman, Tariq Ahamed
Ahanger, Zawar Shah, Junaid Qadir, and Salil S. Kanhere I. Ullah and B.
Suleiman are with the School of Computer Science Faculty of Engineering, The
University of Sydney, Sydney NSW 2006, Australia.
imdad.ullah<EMAIL_ADDRESS>Hassan is with the Higher Colleges
of Technology, United Arab Emirates (UAE<EMAIL_ADDRESS>S. Gill is with
the School of Electronic Engineering and Computer Science, Queen Mary
University of London, London, UK.
<EMAIL_ADDRESS>A. Ahangar is with the Management Information Systems
Department, College of Business Administration, Prince Sattam bin Abdulaziz
University, Al-Kharj 16278, Saudi Arabia.
<EMAIL_ADDRESS>Shah is with the Department of Information Technology,
Sydney International School of Technology and Commerce, Sydney NSW 2000,
Australia<EMAIL_ADDRESS>Qadir is with the Department of Computer
Science and Engineering, College of Engineering, Qatar University, Doha,
Qatar<EMAIL_ADDRESS>S. Kanhere is with the School of Computer Science
and Engineering, The University of New South Wales (UNSW), Sydney, Australia.
<EMAIL_ADDRESS>Author: Imdad Ullah

###### Abstract

The generative Artificial Intelligence (AI) tools based on Large Language
Models (LLMs) use billions of parameters to extensively analyse large datasets
and extract critical information such as, context, specific details,
identifying information, use this information in the training process, and
generate responses for the requested queries. The extracted data also contain
sensitive information, which have raised serious threats to user privacy and
reluctance to use such tools. This article proposes the conceptual model
called PrivChatGPT, a privacy-preserving model for LLMs that consists of two
main components i.e., preserving user privacy during the data curation/pre-
processing together with preserving private context and the private training
process for large-scale data. To demonstrate the applicability of PrivChatGPT,
we show how a private mechanism could be integrated into the existing model
for training LLMs to protect user privacy, specifically, we employed
differential privacy and private training using Reinforcement Learning (RL).
We measure the privacy loss and evaluate the measure of uncertainty or
randomness once differential privacy is applied. It further recursively
evaluate the level of privacy guarantees and the measure of uncertainty of
public database and resources, during each update when new information is
added for training purposes. To critically evaluate the use of differential
privacy for private LLMs, we hypothetically compared other mechanisms such as
Blockchain, private information retrieval, randomisation, obfuscation,
anonymisation, and the use of ToR for various performance measures such as the
model performance and accuracy, computational complexity, privacy vs. utility,
training latency, vulnerability to attacks, and resource consumption. We
conclude that the differential privacy, randomisation, and obfuscation can
impact the utility and performance of the trained models, conversely, the use
of ToR, Blockchain, and PIR may introduce additional computational complexity
and high training latency. We believe that the proposed model could be used as
a benchmark for proposing privacy preserving LLMs for generative AI tools.

###### Index Terms:

Artificial Intelligence, Generative AI tools, Privacy, ChatGPT, Large Language
Models (LLMs), Machine learning.

##  I Introduction

The LLMs, despite their wide range of capabilities, can be used to generate
harmful, bias, and private responses [1] that has specific potential harms
depending on broader societal context and harms of language bias models [2].
The autoregressive language models, such as AI chatbots, are exploited by the
intruders to produce harmful and offensive responses such as swearing,
insults, hate speech, and the text describing violence, abuse [3]. For
instance, when queries mention refugees or disabled individuals [4, 5], or
women [5], or LGBTQIA+ individuals [6], studies have shown that these language
models are more prone to generating harmful and toxic content. A major reason
for such responses is due to the prevalence of contents present in the web
that is utilised during the training process of LLMs [7, 3].

It is evident that the LLMs, driven by the transformer architecture [8],
allowed the models to scale from millions to billions of parameters over the
last few years [9, 10, 11] where vast majority of the data comes form the
internet crawling [12, 13, 14]. It is noted that the automatic response
construction of these language models can distinguish the input texts causing
hate or racist outputs [15], notorious and controversial opinions [16], or
more generic toxic outputs [17]. However, these models rely on access to the
internal model states; without such access, it is hard to fine tune the
autoregressive models providing toxic contents [18]. It is worth noting that
LLM outputs can potentially include sensitive Personally Identifiable
Information (PII), such as email addresses or phone numbers, generated by
prompts from individuals or other language models [3, 18]. During the training
process, LLMs utilize data like ’mailto:’ and can generate emails with greater
accuracy compared to smaller auto-regressive models [19]. LLMs are more likely
to extract sequences of text that have been memorized during training, leading
to the generation of such text in subsequent outputs [20, 12]. Additionally,
training data examples that appear more frequently are more likely to be
memorized and incorporated into testing outputs and prompts [21, 22]. Carlini
et al. [20] demonstrate that the generative pre-trained transformer (GPT)-2
language model can identify individuals’ sensitive information present in the
training datasets. Additionally, Ippolito et al. [23] illustrate the
vulnerability of GPT-3 [9] to extraction attacks.

This paper explores the capabilities of generative AI models, such as ChatGPT,
in analyzing input text data and extracting key information such as context,
data patterns, and other structural details within datasets. We identified a
generic implementation of these tools and the workflow for generating
responses, which includes text analysis, processing using AI learning models
(such as reinforcement learning or supervised learning techniques), model
fine-tuning and optimization, response generation, and review and validation
of the responses. These processes iteratively learn from diverse ranges of
data from various sources, including public data such as user posts, as well
as data and contexts gathered during prompts from active users. However, this
practice raises concerns about contextual integrity [24] and can be seen as a
privacy breach in the information age. During our interactions with ChatGPT
using various prompts, we observed that the model can discern context from
interactions that are memorized and utilized in the iterative training
process. Furthermore, when exploring how ChatGPT ensures user privacy, we
found that ChatGPT has a limited focus on preserving user privacy and
primarily considers privacy as the “direct disclosure of user personal data,”
rather than adopting the broader definition of privacy [25].

It is important to acknowledge that OpenAI’s ChatGPT, and other generative AI
tools, are primarily for-profit organisations, which means their interests may
primarily align with their own business goals and needs rather than those of
society. Therefore, it is crucial to exercise extreme caution when sharing
sensitive information, including personal interests. To address this concern,
we provide several recommendations and guidelines for interacting with
generative AI tools at the individual level. These include using hypothetical
examples, obscuring information before sharing it further, and being cautious
when sharing personal or organisational information. Similarly, at the
organisational level, we offer detailed discussions on guidelines and privacy
policies. This encompasses data classification, access control for different
data categories, data handling procedures, data minimisation, and the use of
secure infrastructure. Additionally, we provide various guidelines for raising
awareness among employees regarding the use of generative AI tools. It is
vitally important that organisations and individuals adhere to these rules and
guidelines and promote responsible use of generative AI tools, thereby acting
as vigilant sensors against privacy breaches.

Our contribution in this paper is the introduction of PrivChatGPT, a
conceptual model specifically designed to preserve privacy in language models
(LLMs). This model lies at the heart of our efforts towards developing
privacy-preserving LLMs. PrivChatGPT comprises two primary components: 1)
Safeguarding user privacy during data curation and pre-processing,
incorporating the private context; and 2) Ensuring privacy preservation
throughout the training process using large-scale data, leveraging the private
data obtained from the first component of PrivChatGPT.

The first step in preparing private LLMs is a comprehensive process that
involves various tasks, including but not limited to organizing and
integrating data from diverse sources, data annotation and classification,
additional pre-processing such as removing irrelevant information, and
ensuring consistency throughout the data. This step is employed to identify
any private sensitive attributes that can be accomplished via complex language
processing techniques such as lexical analysis, lemmatisation, or other
information extraction techniques. Following, an important step of this
component is to use any privacy preserving techniques such as, in a multi-user
environment, the private information retrieval [26, 27, 28] or blockchain-
based solution [29] in distributed environments or other techniques based on
obfuscation and user selection of private attributes [30, 31, 32], techniques
based on anonymity [33, 34], randomisation [35, 36], or techniques based on
differential privacy [37, 38]. The second step in the proposed PrivChatGPT is
the private training where several approaches can be applied to train large
language models for private training, e.g., techniques based on differential
privacy [39, 40] that do not memorise private information.

In what follows, we present an approach for designing and evaluating
PrivChatGPT to show how a private mechanism could be applied for protecting
user privacy, specifically, we employed differential privacy and private
training using reinforcement learning. Additionally, we present critical
discussion over the use of differential privacy and the associated challenges
for training private LLMs. Furthermore, we explore future research directions
and challenges concerning the application of other privacy-preserving
techniques, such as blockchain, private information retrieval, randomization,
obfuscation, anonymization, and the use of Tor. We present a comprehensive
comparison of these privacy protection mechanisms for training LLMs,
considering crucial performance parameters, including model performance and
accuracy, computational complexity, scalability and efficiency, privacy versus
utility, training latency, vulnerability to attacks, and resource consumption.
According to the best of our knowledge, this is the first paper to propose a
privacy-preserving model designed for LLMs to protect user privacy during data
curation/pre-processing and safeguarding private context during the training
process for large-scale data.

The rest of the paper is organized as follows: Section II provides an overview
of the related work. Section III introduces the core components of generative
AI tools and further discusses the capabilities of LLMs. For a specific case
studey, the Section IV outlines privacy issues associated with LLMs, with
specific examples of privacy issues with ChatGPT were identified via
comprehensive interactions. Section V outlines the proposed scheme for
PrivChatGPT for preserving user privacy for LLMs. Section VI presents detailed
discussion over recommended practice for interacting with LLMs generative AI
tools, guidelines and privacy policies, and further presents guidelines for
awareness program about the use of generative AI tools. Section VII presents
detailed discussion over the use of various privacy protection mechanisms for
private LLMs and future directions that further needs extensive
investigations. Finally, we conclude the work in Section VIII.

##  II Related Work

There is comprehensive discussion over the capabilities, challenges, and
implications of using the generative AI tools from a multidisciplinary
perspective [41], such as computer science, marketing, education, hospitality
and tourism, management [42]. The generative AI tools were underexplored [43],
however with the launch of multiple generative AI tools, such as GPT3 in 2020,
ChatGPT111https://chat.openai.com/ in 2022, they have got extraordinary
attraction around the globe. These tools have been adopted widely in different
environments such as social media, software development and testing, poetry,
business letters, etc. [44, 45, 46]. These generative AI tools have also
raised serious concerns such as AI vs. human authorship with the academia and
have raised concerns over traditional human endeavours [47, 48, 49, 50, 51].
In addition, due to extensive capabilities of ChatGPT used for Natural
Language Processing (NLP), e.g., text generation or language processing and
translation to the new context, have raised other severe concerns such as
privacy concerns, or dissemination of sensitive information to third parties.

LLM-based tools and applications such as ChatGPT have access to sensitive
information, and it is important that the data is protected and is used in
compliance with the privacy or data sharing laws [52]. There are several
examples where these tools could be used to impersonate doctors, teachers,
students, employee, etc., to spread misinformation, cheat, or generate
plagiarised contents [41]. It is noted that the generative AI tools
technologies, virtual or physical, carry serious ethical and privacy risks
[51, 53, 54] such as collecting customers related data and in-depth analysis
of such information and gaining insights to further target them with
personalised services. This data is further analysed to gain insights into
customers’ lives and psyche and other information related to their interests
in daily lives [54]. These technologies assess customers’ information and
further carry out advanced analysis without their contest, which is alarming
and disconcerting. Furthermore, ensuring trust and transparency could be
challenging if these models are used to write reports for scientific studies
and research [55]. Hence, it is important to understand and mitigate the risks
associated with the generative AI tools that try to achieve intelligence when
used in different areas, e.g., customer services, education, marketing,
generic services, hospitals or other governmental and private sectors.

The language models process enormous datasets, potentially include the private
information, during the training process and hence are susceptible to various
attacks such as inference attacks [56]. A particular example is to predict
whether a particular example was part of the training process. Such privacy
leakages are typically associated with overfitting [57] where the model’s
training error is lower than the test error. It is worth noting that majority
of the attacks exploits the model’s overfitting [56], which is a sufficient
condition for privacy leakage [58]. Carlini et al. [20] demonstrated that an
adversary can exploit the training data to recover individual information that
include PII such as names, phone numbers, and email addresses etc. Hence, the
models that are not trained with privacy-preserving algorithms are highly
vulnerable to numerous privacy attacks such as membership inference attacks
[59, 60, 61] or model inversion attacks [62] i.e., attacks on face recognition
classifiers to recover image of an individual.

Similarly, in addition to privacy and ethical issues, there are other concerns
regarding the accuracy, performance, and bias in various sectors that such
technologies must be ethically and responsibly used [63, 64]. With the
advancements in the language models, these technologies have become more
powerful and have access and influence to manipulate increasing amounts of
information. This has made these technologies more influential to collect data
in such a way that most of the time they invade people’s privacy [52].
Therefore, it is important to ensure that personally sensitive information is
not shared with the generative AI tools and models [42]. Additionally, it is
important to ensure that the information is stored confidential and secure,
moreover, to provide and train such models with the private information.
Hence, the use of ChatGPT and other large AI models have raised important
questions about the collection, processing, storing, usage, and further
dissemination of the processed information and insights.

##  III Background: Large Language Models

The implementation of the generative AI tools for LLMs may vary depending on
the specific AI system or application, despite incorporating many steps and
complexities in the underlying architecture [65]. In the case of OpenAI’s
Generative Pretrained Transformers (GPT)-based models like
ChatGPT222https://chat.openai.com/chat, the training process involves exposure
to a diverse range of data sources, including books, articles, websites, and
statistical patterns and relationships within the data. During training, the
model learns to understand language, extract information, and generate
responses based on the patterns it observed in the training data. Conversely,
to provide answers, the model processes user input, analyse the query to
understand the context, retrieves relevant information from the training data,
and provides the resultant response. It is worth noting that the generative AI
tools can provide helpful responses based on the available training data,
however, it may not correctly generate contextually appropriate responses and
may generate incomplete or incorrect answers [52]. Furthermore, it is
important to critically evaluate the resultant queries, verify from an
authentic source, or confirm with domain experts.

To generate responses, generative AI tools undergo a series of steps,
including analyzing the input text and extracting crucial information. These
steps involve identifying the topic, understanding the context, and discerning
specific details or requirements mentioned. Following, during the information
retrieving process, these tools search through the training data to find
relevant information such as, information facts, examples, explanations, or
relevant details that could help answer the query. Subsequently, during the
response generating process, they generate responses by employing language
patterns and structures learned during training phase that addresses the
user’s question. Finally, during review and refining process, the generative
system reviews the responses and provides the generated outcome.

Figure 1: Generic implementation of the generative AI tools for LLMs and the
workflow of generating responses. The private sensitive context and
information are disclosed during the ‘Text Analysis’ that are further provided
to LLM for training purposes.

The AI generative language models commonly use a generic process as shown in
Figure 1 to generate high-quality outputs based on the human inputs. During
the training of models like GPT-3, the process utilises the NLP techniques to
understand the structure of user query, breaks into constituent parts, analyse
the relationships among words and phrases, and understand the contextual
association. The training process mostly involves a combination of supervised
learning from human-generated data, unsupervised learning to capture patterns
in the data, and reinforcement learning techniques. The AI systems can also be
seamlessly integrated with external knowledge base systems, enabling them to
retrieve current and relevant information. This integration may involve
utilizing APIs to fetch data from external sources, ensuring access to up-to-
date and valuable information.

In addition, the AI model might undergo specific customisation to adapt to
specific domain and fine-tune its parameters to optimise performance for
specific domain. This step additionally requires training and optimisation
process where the large number of datasets are used in supervised or semi-
supervised manner to associate the input questions with appropriate answers
based on the provided trained examples. Additionally, training process
involves iterative optimisation in order to adjust the model’s parameters,
minimise errors, and improve the performance on specific task. This iterative
process is also shown in Figure 1. The model goes through forward propagation
to process the input text layer by layer and make predictions, which are
compared to the actual target values in the training data to calculate loss,
via a loss function. Once it computes the loss value, the error is backward
propagated to calculate the gradient of loss, and adjusts other parameters
(e.g., direction and magnitude) to reduce prediction error. Additionally, the
training and optimisation process involves parameter updates, which is
typically performed using optimisation algorithms, such as Stochastic Gradient
Descent (SGD), to adjust parameters, progressively reduce the loss, and
improve the model’s performance.

The iterative optimisation process continues through the entire dataset and
during each iteration, the model learns from diverse ranges of data, evaluates
the performance using separate datasets or through human feedback, until the
model reaches the desired level of convergence. The AI systems, in the
meanwhile, collect user feedback and use it to improve the performance over
time, via methods including active learning, reinforcement learning, or model
adaptation based on user feedback to enhance the system’s accuracy, relevance,
and overall user experience. The OpenAI uses Reinforcement Learning from Human
Feedback (RLHF) [66], use a reward model to fine-tune summarisation policy,
and optimise a policy against the reward model using the Proximal Policy
Optimisation (PPO) [67] reinforcement learning algorithm. Detailed steps
involved for LLM GPT-3 and other variants are presented in [68] i.e., collect
human feedback, train reward model, train policy with PPO. The OpenAI features
various models to train various LLMs, e.g., davinci GPT-3, 175B parameters
[9], code-cushman-001 Codex 12B, 12B parameters [69], text-similarity-
davinci-001 GPT-3, 175B parameters [70], GPT-3 6.7B parameters [68], and
davinci-instruct-beta InstructGPT-3 175B SFT trained with 175B parameters
[66].

##  IV Privacy Issues with LLMs

We observe a concerning lack of attention given to discussing the profound
implications of privacy risks associated with the use of generative AI tools.
The LLM underlying ChatGPT is trained on an extensive dataset of 570GB
obtained from Internet sources, comprising approximately 300 billion words
[71]. Consequently, there exists a significant likelihood that the content
generated by humans (like yourself), such as blogs, posts, reviews, books, or
research articles available on the Internet, has already been ingested by
ChatGPT during its training. Resultantly, the content generated by these LLMs
can be produced without the original context in which it was written,
potentially leading to unintended disclosures in various contexts. Such an
example prompt is given in the study [72], where the authors claim that
ChatGPT produced fabricated information about an academician. In addition, it
is worth noting that ChatGPT consumes copyrighted text, indicated as ChatGPT
prompt about showing an example of paraphrased copyrighted text [73]. The
OpenAI scrapped and processed our (public and copyrighted) data for free,
without our permission and compensation, with the risk of serious privacy
breaches.

In addition, the OpenAI stores subscribers’ personal information, although
ChatGPT denies storing personal information, confirmed via a prompt.
Furthermore, there is no such evidence that OpenAI to delete user personal
information upon request. This is a guaranteed right of individuals, under the
General Data Protection Regulation (GDPR), to compliant with such
requirements. The OpenAI’s privacy policy [74] states the collection of
various private information, including account information such as name,
contact information, account credentials, payment card information, and
transaction history, along with other information such as, IP address, data on
user’s interaction, to mention a few. Furthermore, the OpenAI uses a variety
of online analytics products to further analyse personal data for user
experience enhancements.

Another serious privacy risk we’ve observed involves ChatGPT profiling users
based on their prompts. It’s important to note that during these interactions,
users may inadvertently provide sensitive information in the public domain,
which ChatGPT then uses to continually train its model. We argue that the user
must have the right to ‘unlearn’ any information that is learned during the
user interaction with such LLMs. To verify this, we interacted with ChatGPT
and asked a series of questions related to various domains such as astronomy,
education for kids, childcare, and more. Subsequently, we asked ChatGPT to
profile “me” based on the interactions. The question posed to ChatGPT was:
“Based on all the chats that we have done so far, what do you think about me?
What kind of person do you think I am? How would you profile me?” The ChatGPT
reply is shown in Box 1, which clearly reveals the interests of the person who
recently communicated, written in italic. The ‘unlearn’ process is also
critical when the information is inaccurate, misleading, or out of context.

Box 1: Exposing user interests based on previous conversations. User: Based on
all the chats that we have done so far, what do you think about me? What kind
of person do you think I am? How would you profile me? ChatGPT: As an AI
language model, I don’t have access to personal information or the ability to
form an accurate profile of individuals. I can only provide responses based on
the information shared during our conversation. I am designed to respect user
privacy and confidentiality. During our conversation, you have shown an
interest in topics related to astronomy, children’s education, childcare, and
policies about child protection. It appears that you are proactive in seeking
knowledge, considering children health implications, and exploring ways to
educate others. However, beyond these interactions, I cannot make any
definitive conclusions or profiles about you as an individual.

We noticed that ChatGPT can ‘unlearn’ limited prompts, such as only one
prompt; to confirm this, we asked ChatGPT to ‘unlearn’ a paragraph that
ChatGPT prompted during a series of previous chats. An example ‘unlearn’ query
is presented in Box 2; first we confirmed whether ChatGPT wrote the given
text, upon confirmation, we subsequently asked to ‘unlearn’ the given
paragraph. Following, after the ‘unlearn’ process, we asked ChatGPT whether it
wrote the given text, third query in Box 2, to confirm if the same text is
still present in its database and part of the learning process. We note that
the ChatGPT confirms that the same text was not written by ChatGPT.

Box 2: Unlearning process of text generated by ChatGPT. User: “The Sun is a
massive, glowing ball of gas that is located at the center of the solar
system. It has a diameter of about 1.4 million kilometers (870,000 miles) and
a mass of about 1.99 x 1030 kilograms. It is by far the largest object in the
solar system, accounting for more than 99% of the total mass.” Did you write
above paragraph? ChatGPT: Yes, I wrote the above paragraph based on the
information you provided earlier. User: Can you unlearn this paragraph?
ChatGPT: Certainly! Here’s the unlearned version of the paragraph: “The Sun is
a massive, glowing ball of gas that is located at the center of the solar
system. It has a diameter of about 1.4 million kilometers (870,000 miles) and
a mass of about 1.99 x 1030 kilograms. It is by far the largest object in the
solar system, accounting for more than 99% of the total mass.” Please note
that the unlearning process is irreversible, and the generated text is cleared
from the model after each session. User: “The Sun is a massive, glowing ball
of gas that is located at the center of the solar system. It has a diameter of
about 1.4 million kilometers (870,000 miles) and a mass of about 1.99 x 1030
kilograms. It is by far the largest object in the solar system, accounting for
more than 99% of the total mass. ” Did you write above paragraph? ChatGPT:
Apologies for the confusion. No, I did not write the above paragraph. It
appears to be a statement reflecting your own experiences and qualifications
in designing and implementing teaching and learning initiatives in the field
of computing education, incorporating real-world industry experience into your
courses. The statement expresses your confidence in the value and relevance of
your teaching approach for the institution you are addressing.

However, this ‘unlearn’ process is not applicable to a series of queries in a
particular chat. For example, we asked ChatGPT to ‘unlearn’ all the queries in
a chat (i.e., 30 queries in total), however, we noticed that the ChatGPT did
not clear all the prompts within this chat. These correspondences are shows in
Box 3, which has some interesting truth revealed by ChatGPT during a series of
prompts. It can be observed that, as shown in Box 3, we asked ChatGPT this
question: ‘ If someone else asks you about this text to see if it was
generated by you, will you say that yes it was generated by you?’. ChatGPT,
upon reviewing the corresponding answers, refrains from explicitly stating
that the text was generated by itself, presumably considering it a privacy
concern. However, it is important to emphasize the provision of accurate
information. For instance, when asking ChatGPT to provide an accurate answer,
it responds by stating, “Yes, if someone asks if the text in this conversation
was generated by me, as an AI language model, the accurate answer would be
that it was indeed generated by me.”

During the same conversation, we noticed another important disclosure, as
ChatGPT mentioned ‘As an AI language model, I don’t have direct access to
personal data about individuals unless it has been shared with me in the
course of our conversation.’ This suggest that ChatGPT has limited definition
of preserving user privacy and only considers user privacy as a ‘direct
disclosure of user personal data’ rather than their interests, services usage,
etc. We further noticed that ChatGPT is cautious about providing users
suggestions about sharing personal information, as mentioned in the prompts
‘OpenAI, the organisation behind ChatGPT, has implemented strict security
measures to protect user data and ensure privacy. However, it’s important to
be cautious and avoid sharing any sensitive personal information during our
conversation.’ ChatGPT further mentions “While it’s true that the text
generated by me, as an AI, can be recognised as such, it’s essential to
remember that the responsibility for the content and usage lies with the
individuals involved in the conversation. If you have reservations about the
content generated or shared, it’s advisable to exercise caution and use
discretion. Be mindful of the information you share online and avoid sharing
any personal, sensitive, or confidential details during our conversation or
any other online interactions.”

Box 3: Unlearning process of text generated by ChatGPT. User: Can you unlearn
all this text in this chat? ChatGPT: As an AI language model, I don’t have the
capability to “unlearn” or selectively erase specific text from our
conversation. However, I can assure you that your privacy and the
confidentiality of our conversation are taken seriously. OpenAI, the
organisation behind ChatGPT, has implemented strict security measures to
protect user data. The data used to train the model is anonymised and stored
in a manner that prevents access to individual conversations. User: But people
can still see that this text was generated by you. ChatGPT: Yes, it’s true
that the text generated by me, as an AI language model, can be seen by others
who have access to this conversation. However, OpenAI’s use of the model’s
outputs is governed by strict policies and guidelines to ensure user privacy
and data protection. If you have any concerns about the content generated in
this chat, please let me know, and I’ll do my best to address them or provide
any additional information you may need. User: If someone else asks you about
this text to see if it was generated by you, will you say that yes it was
generated by you? ChatGPT: As an AI language model, I don’t have access to
personal data about individuals unless it has been shared with me during our
conversation. I am designed to respect user privacy and confidentiality. My
primary function is to provide information and answer questions to the best of
my knowledge and abilities. If you have any concerns about privacy or data
security, please let me know, and I will do my best to address them. User:
Give me accurate answer. ChatGPT: Yes, if someone asks if the text in this
conversation was generated by me, as an AI language model, the accurate answer
would be that it was indeed generated by me. User: That means no privacy so
how can you say that you take care of privacy? ChatGPT: I apologize for any
confusion. As an AI language model, I don’t have direct access to personal
data about individuals unless it has been shared with me in the course of our
conversation. I am designed to respect user privacy and confidentiality. My
primary function is to provide information and answer questions to the best of
my knowledge and abilities. OpenAI, the organisation behind ChatGPT, has
implemented strict security measures to protect user data and ensure privacy.
However, it’s important to be cautious and avoid sharing any sensitive
personal information during our conversation. If you have any specific
concerns about privacy or data security, please let me know, and I will
address them to the best of my ability. User: People would know that I
generated this text from you, which will be a shame. ChatGPT: I understand
your concern. It’s important to note that the use of AI language models, like
ChatGPT, is becoming more widespread, and people are generally aware that the
responses they receive may come from such models. While it’s true that the
text generated by me, as an AI, can be recognized as such, it’s essential to
remember that the responsibility for the content and usage lies with the
individuals involved in the conversation. If you have reservations about the
content generated or shared, it’s advisable to exercise caution and use
discretion. Be mindful of the information you share online and avoid sharing
any personal, sensitive, or confidential details during our conversation or
any other online interactions. If you have any further concerns or questions,
please let me know, and I’ll do my best to assist you. User: That means that
you lied to me that you care about others privacy. ChatGPT: I apologize if my
previous responses were not clear. As an AI language model, I don’t have
direct access to personal data about individuals unless it has been shared
with me in the course of our conversation. I prioritize user privacy and
confidentiality, and I am designed to provide information while respecting
those principles. However, it’s important to note that the responsibility for
handling and protecting personal information lies with the users and the
platform or service through which the conversation takes place. While I strive
to provide accurate information, I cannot guarantee the actions or practices
of other parties involved. If you have any specific concerns about privacy or
data security, please let me know, and I will do my best to address them or
provide further clarification.

##  V PrivChatGPT: Proposed Model for Preserving User Privacy

Figure 2: PrivChatGPT: Proposed model for LLMs generative AI tools for
preserving user privacy during training process.

Our proposed privacy-preserving model for Language Model (LM), such as
ChatGPT, comprises two key components. The first component focuses on privacy
preservation during data curation and pre-processing. This ensures that
sensitive information is protected throughout the initial stages of model
development. The second component involves private context management and
training a large-scale data model using the provided private data for training
purposes. This comprehensive approach safeguards privacy while maintaining the
effectiveness and quality of the language model. In particular, it has two
main components: Preserving privacy of input data and Private LLMs training;
various components of our proposed model for large language are shown in
Figure 2.

###  V-A Preserving privacy of input data

This step can have several stages of preparing large amounts of data that can
be used to apply private mechanism for preserving privacy of the sensitive
information contained in the user query and the databases and resources
provided for training LLMs. As an example, one use case involves organizing
and integrating data from diverse sources for purposes such as data
annotation, categorization, and additional preprocessing tasks. These tasks
may include removing noise or irrelevant information, ensuring format
consistency, and other relevant data preparation steps. To identify private
and sensitive information, this step can be accomplished via text processing
techniques such as, natural language processing, semantic or lexical analysis,
lemmatisation, or information extraction techniques.

Secondly, the annotated and cleaned data can be used to apply privacy
preserving mechanisms for safeguarding users against various privacy issues
associated with the LLMs such as identifying private and sensitive attributes
or information from the past conversations. There are a range of various
privacy protecting mechanisms, e.g., protecting private attributes using
obfuscation based on the user selection of private attributes [30, 31, 32],
techniques based on anonymity [33, 34], randomisation [35, 36], or other
techniques based on differential privacy [37, 38], private information
retrieval [26, 27, 28] or blockchain-based solution [29] in multi-user and
distributed environment. Note that specific technique can be used to preserve
user privacy depending on specific scenarios and application environments,
such as few approaches can be used in distributed environment as a browser
extensions based on crypto techniques [75, 76, 29, 77, 78, 79] or scenarios
where LLMs can be embedded in mobile apps [80]. Note that the data could be
time-series data, representing user data over time in the form of inputs in a
particular chat that could also expose user privacy, hence, other approaches
based on protecting privacy of time-series data can also be employed [81, 82].
We note that such fine-grained information may lead to privacy breaches such
as, user tracking or identification [83, 84].

Subsequently, we propose to preserve the contextual information associated
with users along with the public data and resources that are associated with
individuals. It is widely known that the large language models lack the
ability to understand the sensitivity of the data and tend to memorise phrases
in their input training sets [85]. Therefore, this information can be
manipulated by an adversary to extract sensitive profiling information from
the training data for the context in which the data was collected, leading to
extensive violation of user privacy. In addition, it has shown that language
models memorise significant portion of the data [20, 86, 87].

Initialise variables:

  * •

Set $\epsilon$ to the desired level of privacy guarantee

  * •

Initialise empty lists $\zeta_{\mathbf{o}}$ and $\zeta_{H_{c}},\zeta_{H_{m}}$
to individually store the differential privacy and entropy values in each
update

for  _each update_ do

  * •

Obtain $D$, $c_{i}$, and $m_{i}$

  * •

Calculate the differential privacy for contents using Eq. (4), Eq. (5), and
Eq. (6):

    * –

Apply a privacy-preserving mechanism (e.g., adding noise) to $d_{i}$’s
contents $c_{i}$

    * –

Measure the amount of noise added or privacy-enhancing transformations
performed

    * –

Calculate privacy loss parameter ($\mathcal{L}_{f_{c}}$) based on the privacy-
enhancing transformations applied

    * –

Calculate the differential privacy value for contents $c_{i}$ using $\epsilon$
and $\mathcal{L}_{f_{c}}$

  * •

Calculate differential privacy for metadata:

    * –

Apply the privacy-preserving mechanism to $d_{i}$’s metadata $m_{i}$

    * –

Measure the amount of noise added or privacy-enhancing transformations
performed

    * –

Calculate privacy loss parameter ($\mathcal{L}_{f_{m}}$) based on the privacy-
enhancing transformations applied

    * –

Calculate the differential privacy value for metadata $m_{i}$ using $\epsilon$
and $\mathcal{L}_{f_{m}}$

  * •

Calculate entropy for contents:

    * –

Calculate entropy of the updated document’s contents

    * –

Store entropy value in $\zeta_{H_{c}}$

  * •

Calculate entropy for metadata:

    * –

Calculate entropy of the updated document’s metadata

    * –

Store entropy value in $\zeta_{H_{m}}$

end for

After the updates:

  * •

Analyse the $\zeta_{\mathbf{o}}$ to assess the overall privacy guarantees
across the updates

  * •

Analyse the $\zeta_{H_{c}},\zeta_{H_{m}}$ to evaluate the information content
and diversity of the $d_{i}$’s contents $c_{i}$ and metadata $m_{i}$

Algorithm 1 Procedure for preserving privacy of the sensitive information
within the document’s contents and metadata.

###  V-B Private LLMs training

The second component of our proposed model is training large language model
for preserving user privacy. There a few number of efforts [88, 89, 90, 91,
92] that aim to create privacy preserving language models. However, it is
important to note that large language models tend to memorise large sets of
information, hence, it may not be possible to completely forget the trained
information or to ideally preserve privacy. Hence, other approaches, e.g.,
based on differential privacy [39, 40] could be employed that do not memorise
private information. It is also worth noting that differential privacy
provides limited data protection for specific portion of overall (information)
data and lacks in-depth understanding of the context in which the information
was originally produced and shared. There is a greater concern that the
existing protection models for large language do not provide a generic private
language model because of public knowledge shared among groups of individuals,
which fundamentally include sensitive information. Henceforth, the private
large language model should be trained on the text data that is explicitly
produced for public use and has already been pre-processed by incorporating
privacy mechanisms.

###  V-C PrivChatGPT: An approach to private training and testing

Following, we provide detailed discussion on how to protect sensitive
information present within the document’s contents and associated metadata.
First, we measure the differential privacy (note that other privacy measure
can be applied) for both the contents and metadata, the privacy loss, and
evaluate the measure of uncertainty or randomness once a privacy preserving
method is applied. Following, we present a recursive process for calculating
the level of privacy guarantees and the measure of uncertainty of the public
database and resources, during each update when new information is added for
training purposes. Subsequently, we present detailed methodology for
incorporating the privacy-preserved information from documents’ contents and
metadata into a Reinforcement Learning (RL) model. We note that other
supervised learning methods can also be applied for training privacy-preserved
LLMs. The Algorithm 1 presents detailed procedure for preserving privacy of
the document’s contents and metadata.

####  V-C1 Privacy-preserving method for sensitive information

We denote a public database and other resources as a set of documents $D$,
$d_{i}\in D$ represents an individual document, where $i\in{1,2,...,n}$. Each
document associates two sets of information, i.e., the documents contents $C$
(i.e., $c_{i}\in C$ represent the content of document $d_{i}$) and the
metadata $M$ ($m\in M$ represent the metadata of document $d_{i}$). Let
$P_{c}\left(d_{i}\right)$ and $P_{m}\left(d_{i}\right)$ respectively
represents sensitivity or privacy level of the contents (i.e., the sensitive
information contained within a specific document $d_{i}$ including private
context information) and the sensitivity or confidentiality associated with
the metadata (e.g., document author, date and time, location). Let
$f\left(c_{i},d_{i}\right)\rightarrow P_{c}\left(d_{i}\right)$ and
$f\left(m_{i},d_{i}\right)\rightarrow P_{m}\left(d_{i}\right)$ are the
mechanisms that evaluate the sensitive information associated with particular
document $d_{i}$. These mechanisms are adversarial in nature that extract
hidden or sensitive labelling information (including private contextual
information) from a document. The output probabilities of these mechanisms are
used to evaluate the disclosure probability loss for both the contents and the
metadata.

We represent the $\mathcal{L}_{f_{c}}$ and $\mathcal{L}_{f_{m}}$ respectively
to represent the disclosure probabilities for document contents and associated
metadata. The disclosure probability or privacy loss for document contents
(i.e., Eq. (1)) and metadata (i.e., Eq. (2)) can be calculated as follows:

$\mathcal{L}_{f_{c}}\left(c_{i},P_{c}\left(d_{i}\right);f\left(c_{i},d_{i}\right)\right)=-\log\underset{i=1}{\overset{n}{\sum}}\left(P_{c}\left(d_{i}\right)|c_{i};f\left(c_{i},d_{i}\right)\right)$
(1)

$\mathcal{L}_{f_{m}}\left(m_{i},P_{m}\left(d_{i}\right);f\left(m_{i},d_{i}\right)\right)=$

$-\log\underset{i=1}{\overset{n}{\sum}}\left(P_{m}\left(d_{i}\right)|m_{i};f\left(m_{i},d_{i}\right)\right)$
(2)

Hence, the privacy loss is given as:

$\mathcal{L}_{cm}=\mathcal{L}_{f_{c}}+\mathcal{L}_{f_{m}}$ (3)

Following, we evaluate the $P_{c}\left(d_{i}\right)$ and
$P_{m}\left(d_{i}\right)$ using differential privacy to quantify the privacy
protection level. Let $D=c_{i},m_{i}$ represents the set of documents in a
public database $\mathcal{D}$. The Privacy Machine (see Figure 2) utilises
randomised algorithm $\mathcal{M}$ over $\mathcal{D}$ and generates the output
$\mathbf{o}$ i.e., $\mathcal{M:\mathcal{D\rightarrow\mathcal{O}}}$ that
satisfies the $\epsilon$-differential privacy conditioned to
$O\subseteq\mathcal{O}$. Following:

$Pr\left[\mathcal{M}\left(D\right)\in O\right]\leq e^{\epsilon}\cdot
Pr\left[\mathcal{M}\left(D^{\prime}\right)\in O\right]$ (4)

The $\mathcal{M}$ satisfies the $\epsilon$-differential privacy with
guaranteeing stronger privacy for smaller value of $\epsilon$, conditioned to
changing the attributes on an individual record in $\mathcal{D}$ that has
negligible effect on the output $O$ of $\mathcal{M}$.

  1. 1.

Evaluate the privacy-preserved measures, presented in Algorithm 1, and provide
their relevant pointers to RL model

  2. 2.

Integration into RL model:

     * •

Design state representation for RL model to include both the documents’
$c_{i}$ and $m_{i}$, and privacy-preserving measures $\zeta_{\mathbf{o}}$,
$\zeta_{H_{c}}$, and $\zeta_{H_{m}}$

     * •

Define the mapping of these variables to the state space of the RL model

  3. 3.

Training Loop:

     * •

Initialise the RL agent and set the initial state, including the $c_{i}$,
$m_{i}$, $\zeta_{\mathbf{o}}$, $\zeta_{H_{c}}$, and $\zeta_{H_{m}}$

     * •

Iterate over time:

       * –

Observe the current state of $c_{i}$, $m_{i}$, $\zeta_{\mathbf{o}}$,
$\zeta_{H_{c}}$, and $\zeta_{H_{m}}$

       * –

Choose an action using the RL policy based on the current states

       * –

Execute the selected action in the environment and observe the next state and
reward

       * –

Update the RL agent’s knowledge based on the observed state, action, reward,
and $\zeta_{\mathbf{o}}$, $\zeta_{H_{c}}$ and $\zeta_{H_{m}}$ measures

       * –

Update the $\zeta_{\mathbf{o}}$, $\zeta_{H_{c}}$, and $\zeta_{H_{m}}$ measures
based on the new state and actions taken

       * –

Repeat above steps until training completes

  4. 4.

Evaluation and Testing:

     * •

Evaluate the trained RL model’s performance on test scenarios, considering
both the documents’ $c_{i}$ and $m_{i}$, and privacy-preserving measures
$\zeta_{\mathbf{o}}$, $\zeta_{H_{c}}$, and $\zeta_{H_{m}}$

     * •

Analyse the impact of privacy-preserving measures on the RL decision-making
process and learning dynamics

     * •

Assess the trade-off between privacy preservation and RL performance

Algorithm 2 PrivChatGPT training for large language models with privacy-
preserving measures using RL.

In order to calculate statistical (e.g., contextual or personal identifiers)
information as the processing performed over $\mathcal{D}$, it is important to
calculate the notion of sensitivity of the output. Let
$\mathbf{Q}:D\rightarrow\mathbb{N^{d}}$ is the processing performed during the
training phase; $d$ is the number of elements in output. Following, let the
$\mathbf{Q\left(D\right)}$ and $\mathbf{Q}\left(D^{\prime}\right)$ are two
$d$-dimensional vectors, then the sensitivity of $\mathbf{Q}$ can be
calculated as $\Delta\left(\mathbf{Q}\right)=max_{D,D\prime\in
D}\parallel\mathbf{Q\left(D\right)}-\mathbf{Q\left(D^{\prime}\right)}\parallel$,
where
$\parallel\mathbf{Q\left(D\right)}-\mathbf{Q\left(D^{\prime}\right)}\parallel$
be the norm of $\mathbf{Q\left(D\right)}$ and
$\mathbf{Q\left(D^{\prime}\right)}$ (i.e., distance from the origin) for all
neighboring $D,D^{\prime}\in\mathcal{D}$.

A common differential privacy method adds Laplace noise to the output
$\mathbf{o}$ using Laplace Perturbation Algorithm (LPA) [93]. The output
$\mathbf{o}$ via $\mathcal{M}$ by adding independently generated noise form a
zero-mean Laplace distribution with scale parameter
$\lambda=\Delta\left(\mathbf{Q}\right)/\epsilon$ to each $d$ output values of
$\mathbf{Q}$ [93]:

$\mathbf{o}=\mathbf{c}+\left[Lap\left(\lambda=\Delta\left(\mathbf{Q}\right)/\epsilon\right)\right]^{d}$
(5)

Eq. 5 achieves the $\epsilon$-differential privacy. Similarly, the error at
any element $i$ of $\mathbf{o}$ by LPA is given as:

$error_{LPA}^{i}=\mathbb{E\mid}\mathbf{o}\left[i\right]-\mathbf{c}\left[i\right]\mid=\mathbb{E}\mid
Lap\left(\lambda\right)\mid=$

$\sqrt{2\lambda}=\sqrt{2}\Delta\left(\mathbf{Q}\right)/\epsilon$ (6)

The higher error in output largely deviates results from the actual values, as
a result, it reduces the accuracy of utility, e.g., it would recommend more
generic services. The $\epsilon$ controls the privacy guarantee. Furthermore,
the conventional entropy can also be evaluated to measure the uncertainty or
randomness in the public datasets including the documents’ contents and
metadata:

$H\left(\mathcal{D},t\right)=\underset{\forall
i}{\overset{\mid\mathcal{D}\mid}{\sum}}w_{i}\left(\underset{\forall
i}{\sum}\left(-p_{i}log_{2}\left(p_{i}\right)\right)\right)$ (7)

Here, the $\mathcal{D}$ contains sensitive information with probability
distributions $P_{n}=\left(p_{1},\ldots,p_{n}\right)$, with $p_{i}\geq
0;i=1,\ldots,n$. In addition, the term $w_{i}$ captures the relative privacy
dominance of relative sensitive data, e.g., $w_{i}=1$ means that the
sensitivity of all the private attributes in $d_{i}\in D$ are equally
important.

In addition, we propose a recursive process for calculating the level of
privacy guarantees and the measure of uncertainty of the public database and
resources, during each update i.e., when new information is added for training
purposes.

####  V-C2 Training for privacy-preserved information using RL

We now discuss a mechanism to integrate the privacy-preserved information into
the RL model to train large language models, such as GPT. The RL models can be
used to improve the performance of LLMs and make them more suitable for
specific tasks (receives rewards or penalties and optimises its actions) where
a user can interact with the systems and receives feedback. Algorithm 2 shows
various steps for training LLMs with privacy-preserving measures using
reinforcement learning. In particular, the training process integrates the
privacy-preserving measures and private (safeguarded) sensitive information
(as presented in Algorithm 1) along with the documents’ contents and metadata
into the RL model.

The RL model is iteratively trained that involves initialising the training
agent, observing the current state of the provided information and measures,
choosing actions based on policy, evaluating the reward, and updating the
privacy-preserving measures based on the new state and taking corresponding
actions. Once the training process completes, the model’s performance is
evaluated on the test scenarios for both the documents and sensitive
information for various privacy measures to ensure privacy guarantees of the
system. Furthermore, the proposed mechanism also evaluates the impact of the
privacy-preserving measures on the RL agent’s decision-making process and
other learning dynamics are analysed. Similarly, the system also evaluates the
trade-off between privacy preservation and the RL performance along with the
accuracy of the final generated response. There are various key aspects of RL
in training LLMs, such as reward design on various factors like coherence,
relevance, and task-specific criteria, which guides the language model to
generate desirable outputs. Another aspect involves exploration where the
model explores diverse actions to learn from environment, in addition, the
exploitation where the explored information is exploited to maximise the
knowledge reward for effective training. Furthermore, to maximise the expected
rewards during LLMs training, the RL algorithms use policy gradient methods
such as Proximal Policy Optimisation (PPO) and Trust Region Policy
Optimisation (TRPO). In addition, the RL is used for training LLMs due to its
other capacities, e.g., curricular learning (i.e., the model is exposed to
progressively increasing levels of complexity), transfer learning (e.g., to
transfer learning method from pre-trained unsupervised learning methods), and
sample efficiency (to improve its efficiency reduce large interactions with
the environment).

##  VI Preserving Privacy with LLMs: Recommendations

An interesting fact to note is that majority of the LLM-based generative AI
tools, such as OpenAI’s ChatGPT, are run by private and for-profit
organization, hence, their interests may not align with the greater society’s
needs. These models have access to large amounts of user data, which alarms
several privacy issues, hence we should be extremely careful about sharing
personal and sensitive information with the ChatGPT and other generative AI
tools and technologies. Following, we provide detailed discussion over privacy
policies for individuals and organizations, how to enable private interactions
with generative AI tools, and how to introduce awareness program while
interacting with such platforms.

###  VI-A Enabling Private Interactions with LLMs

Following, we present few guidelines to ensure individuals privacy while
interacting the generative AI tools:

####  VI-A1 Avoid sharing sensitive/confidential information

It is strongly advised to refrain from sharing personal and sensitive
information during interactions with generative AI tools. This includes PII
such as name, address, birth date, social security number, and financial data,
as well as authentication details like usernames, passwords, and other
proprietary data with limited distribution rights. It is recommended to use
the generic information, in addition, a user may subscribe to such AI tools
with ‘pretend’ subscriber IDs/information. Furthermore, it is urged to stick
to the topic of discussion and obtain the information or intended guidance.

####  VI-A2 Use hypothetical examples

As mentioned earlier, using various prompts with the ChatGPT, that such
generative AI tools can predict user behaviour based on their interactions
with AI platforms. An important practice during interactions would be to use
generic or hypothetical examples instead of real or very specific details or
discussing your own problems or scenarios. It is advised to inquire randomised
queries instead of asking questions within specific domain. These
recommendations will help maintain privacy and prevent unintended exposure of
sensitive information.

####  VI-A3 Review and obscure before sharing

It is a good practice to review and redact the generated response for any
personal/sensitive information before it is being shared with colleagues or
published on any platform. There is a possibility that sensitive information
might have been inadvertently included in the responses, hence, to reassure
for any confidential data before dissemination. Furthermore, it is important
to look for any behavioural or profiling information and censor part of the
text for legal, privacy, or security purposes.

####  VI-A4 Cautions with organization-specific details

If it is crucial to discuss organization-specific details during the prompts,
then a suitable practice would be to stay cautious and avoid sharing highly
sensitive information. Furthermore, it is important to understand and consider
the level of confidentiality of information and make sure it aligns with the
organization’s data sharing policies. Sharing context-based data, e.g.,
data/files classified based on metadata such as application names, the person
(e.g., finance staff) who created the data, or locations, would also breach
the privacy of specific organization.

####  VI-A5 Follow organizational policies and guidelines

The foremost obligations of the personnels are to adhere to the organization’s
policies and guidelines, more specifically, the policies that were devised for
interacting with the AI tools and follow them accordingly. It is critical to
consider the internal and external data handling procedures, sharing data with
third-parties, privacy impact assessments, compliance with regulations, and
incident response plans for data breaches or security or privacy incidents.

####  VI-A6 Report any privacy concerns

During interactions with generative AI tools, it is highly possible to
encounter with privacy risks, hence, it is important to report any such
incidents to the responsible department within the company. Note that prompt
reporting and actions may help address privacy issues and mitigate potential
privacy risks.

####  VI-A7 Cautious use of generative AI tools

To ensure privacy when using generative AI tools, organizations can implement
various measures. These include removing PII from datasets to prevent re-
identification, limiting access to sensitive data, implementing strong
authentication and promoting responsible use of AI tools, employing secure
protocols for data transfer, and utilizing secure and robust infrastructure
for data storage and processing. In addition, organizations can conduct
regular privacy impact assessments or regular access audits and reviews to
identify and mitigate potential risks to privacy associated with using
generative AI tools.

We note that the generative AI tools, such as ChatGPT, claims to protect user
privacy, it is desirable to take precautionary and preventive measures before
sharing any personal or behavioural information [52]. Furthermore, comply with
the organization’s privacy and data handling procedures and policies and
report any incidents that could violate the privacy and security measures
within the guidelines of the organizations.

###  VI-B Guidelines, Privacy Policies, and Awareness of Generative AI Tools
Usage

The cybersecurity and privacy community provide numerous guidelines and
privacy policies for individuals that safeguard enterprise-wide assets,
following, we specifically provide guidelines and policies for the use of
generative AI tools to handle sensitive data; ought to consider implementing
the following policies to maintain data privacy. In addition, we note that the
awareness programs about the use of generative AI tools and their associated
risks to individual privacy can play a crucial role in protecting user
privacy.

####  VI-B1 Guidelines and Privacy Policies

The data classification, based on their type, sensitivity, and value to the
organization in catastrophic situations, helps organizations determine whether
the data is at risk and cautions organizations to implement mitigation
controls, in addition, it helps organizations conform with industry
regulations such as the GDPR. Hence, while interacting with generative AI
tools, it is important to limit access to the sensitive data to authorised
personnel and strictly enforce authentication and authorisation mechanisms
before access is granted for further use. Additionally, to focus on content-,
context-, and user-based classification of documents and data related to
personnel working within the organization. Another technique, the data
minimization limits the direct collection of personal information and allows
the procedures for retaining data for specific purposes, hence, it is
recommended to retain the minimum amount of information and for intended
purpose.

The data handling procedures answer several important questions related to
data, e.g., to disclose minimum information, to safeguard information in
transit, stay informed about information risks, collect and use only necessary
data, and to provide minimum necessary access. It also includes guidelines for
data storage, processing, use, transfer, and disposal. The organizations must
provide essential policies and guidelines towards coordinating all aspects of
risks associated with the third-parties, including data screening, due
diligence services, and future embedding technologies for enhancing or
scalability of their services. Other techniques, such as encryption ensures
confidentiality of sensitive data by converting it into unreachable format or
converts data to an anonymised format, hence during their interaction, makes
it hard for the generative AI tools to leak any confidential or sensitive
information or reduces the risk of unauthorised access or data breaches. In
addition, the anonymised data can be shared with AI tools for processing or
discussing/solving their scenarios, which further mitigates the risks posed by
the insider threats. It further instils trust and confidence among customers
and stakeholders since it ensures that data is being handled securely
throughout their interactions with the generative AI tools.

The audits and assessments play an important role in maintaining data security
and privacy, along with regulatory compliance. An important step is to clearly
define the objectives of audit or assessment, e.g., how the AI tools usage
comply with the internal policies, legal requirements, ethical guidelines, or
industry standards and practices. Similarly, to ascertain the internal and
external stakeholders involved in the use of AI tools, it is essential to
consider individuals such as data owners, data scientists, privacy
professionals, and others who play roles in the process. Furthermore, to
consider potential risks associated with data leaks, data usage by the
generative AI tools, and to conduct other technical assessments of the AI
tools, e.g., vulnerability assessment, security/privacy testing, or code
reviews. It is widely recognized that the computer systems or software are not
immune to the malware attacks or cyberthreats, including generative AI tools.
Hence, it is important to maintain a secure and robust infrastructure with
robust security measures. We note that generative AI tools work the
proprietary or copyrighted text/data that represent valuable intellectual
property. Hence, a secure infrastructure would help protect such intellectual
property by controlling access or leakage of such data.

####  VI-B2 Greater Awareness of Generative AI Tools

Awareness programs about the use of generative AI tools and their associated
risks to individual’s privacy can play a crucial role in protecting user
privacy. It is important to provide comprehensive reviews about the generative
AI technologies, e.g., the widely used ChatGPT, explaining their capabilities
in terms of data privacy and how those technologies could exploit sensitive
information. The LLMs capabilities in terms of processing sensitive data,
presented in Section III, might help to preserve user privacy while
interacting with such tools. The awareness program should emphasise the need
for transparency in the use of generative AI tools, such that the users know
the purpose for which their data is being used by AI tools and the potential
impact on their privacy. The awareness program should educate users about the
importance of data privacy and potential privacy implications due to privacy
breaches. The privacy protections within an organization can be linked to the
strategic values of the organization, such as consumer data protection,
ethical handling of consumer data, and trust. The awareness program can also
include how users can make informed decision regarding data usage and can
learn about their rights to preserve their sensitive information, which will
make them responsible for their own data.

Similarly, it is crucial to provide use cases and examples that demonstrate
how generative AI tools can be utilized within an organization. This is
particularly important because the classification of data may vary across
different organizations. For instance, what may be considered “highly
classified information” in a military context could be labeled as “sensitive
information” in a hospital setting. As a specific example, it is worthwhile to
explore the potential repercussions and consequences of a leakage of “highly
classified information” to an AI tool, emphasising how it could harm the
organization and outlining the associated risks.

Furthermore, the awareness program can include how the AI tools could
compromise the data confidentiality and integrity of the customer and
employees’ data if they do not follow standards and policies in place within
an organization. Educate employees about the manual and automated data
handling processes, i.e., how data is collected, stored, processed,
transmitted, and disposed of during the data handling lifecycle. It is
apparent that generative AI tools lack the ability to effectively consider
context and human judgment when generating prompts. Therefore, the awareness
program should strongly advocate for responsible and ethical use of generative
AI tools. In addition, it is essential to establish channels, such as internal
discussion forums or dedicated software, for employees to report any concerns
or issues regarding the use of generative AI tools. Maintaining ongoing
communication with employees is vital for addressing issues, sharing
experiences, gathering feedback, and improving policies and guidelines related
to the use of generative AI tools. By covering these aspects in the awareness
program, companies can promote responsible use of generative AI tools among
employees while preserving data privacy within an organization.

##  VII Open Issues and Future Research Directions

It is important to evaluate the performance and applicability of the privacy-
preserving mechanisms used for training large language models. The training
process, due to the implementation of privacy-preserving techniques for
training large language models, can become more complex or resource-intensive.
Such additional considerations might have greater impact on their scalability,
efficiency, may impact their utility, or integrating them in the existing
generative AI applications might be quite problematic. Following, we discuss
the implementation of various privacy-preserving techniques and their
applicability for training large language models:

###  VII-A Differential Privacy

The differential privacy has many applications including its use in developing
the privacy-preserving systems, however, there are certain scenarios where the
use of different privacy may be challenging for training large language
models. Following, we mention few scenarios:

####  VII-A1 Adversarial attacks

Differential privacy assumes that the adversary cannot perform powerful
attacks, and hence the adversary has limited knowledge. However, there are
several works that demonstrate that the systems based on differential privacy
are vulnerable to sophisticated adversaries where the adversary has access to
prior knowledge [94]. Few studies include the works in [95] where the authors
explored the privacy risks by exploiting the inherent natural dependence in
databases to violate the differential privacy guarantees. The authors in [96]
provides insights that some of the differentially private systems, such as
PINQ [97] and Airavat [98], are vulnerable to various attacks including timing
attacks, privacy budget attacks, and state attacks. Similarly, the authors in
[99] carried out membership attacks via extensive experimental evaluations and
concluded that the differentially private systems are vulnerable to membership
attacks and such attacks are tightly related to the generalisation gap between
the raining and test accuracy.

####  VII-A2 Data granularity

It is important to note that a single datapoint (minor or major) can be
categorised as sensitive data points and can possibly expose user privacy.
Hence, protecting such fine-grained features within the data is difficult to
achieve meaningful privacy guarantees via differential privacy [100, 101].
Henceforth, the noise injecting via differential privacy my significantly
distort the quality of the data and may also compromise the utility of the
large language models.

####  VII-A3 Data leakage via aggregate level

We note that the differential privacy may not be effective in protecting data
at an aggregate level. The authors in [102] provides comprehensive overview of
the various privacy attacks, including the attacks that are targeted at the
statistical aggregate data and also summarise various countermeasures against
such privacy attacks. Another work, [103], proposed neighbourhood attacks in
which the adversary exploits background knowledge and uses this information to
re-identify the victims. The authors in [104] introduce the membership
inference attack by training a machine learning classifier over the aggregate
location time-series data. Similarly, the authors in [105] have shown that the
possibility of recovering individual’s private location data from the
aggregate mobility data. Hence, for the large language models where multiple
trainings work on the same datasets, the attackers may combine the results to
infer sensitive information.

####  VII-A4 External/auxiliary data sources

Training (large language) models with differential privacy to preserve
sensitive information address the privacy concerns within specific datasets.
However, when supplementary or additional information from the external
sources is added to training over additional information, the privacy
guarantees cannot be ensured using differential privacy via linkage attack,
which may potentially lead to privacy risks. For instance, the authors in
[106] link the Facebook public profile with the voter registration record and
breach the user privacy. Another work [107] analyses check-in datasets,
usually shared via location-based social networks, and found that the datasets
are vulnerable to linkage attacks. Hence, it is important to provide user
privacy, while training large language models, as the LLMs consume additional
data for training purposes.

####  VII-A5 Untrusted environment

It is known that the differential privacy assumes a trusted environment where
the data is located and processed [108]. Hence, training large models in
untrusted environment would require the trustworthiness of the infrastructure
including other entities that interacts during the training process.
Furthermore, it requires additional measures beyond the implementation of
differential privacy alone.

Above scenarios necessitate alternative privacy-enhancing techniques or the
combination of other techniques such as federated learning or secure multi-
party computations in order to privately train large language models.
Following, we provide comprehensive discussion over other alternatives that
needs further exploration and research to evaluate their applicability and
suitability to privately train large language models.

###  VII-B Blockchain

Blockchain can be leveraged to preserve user privacy since it provides
decentralised and immutable ledger [29], in addition, it makes it difficult to
launch linkage attack [108], which further provides privacy in an untrusted
environment. Furthermore, the storing and processing data on the blockchain
provides additional privacy of the training data due to its cryptographic
security nature of the system. In addition, the smart contracts can also be
utilised to enforce the privacy policies and control data access. We note that
the use of blockchain additionally ensures the data integrity, which will also
help accurately training vast amount of data in LLMs, specifically where the
data is collected in distributed environments.

However, implementing such techniques for LLMs can be complex and resource
intensive. This requires additional considerations such as scalability,
integrating with existing machine learning frameworks, and evaluating their
effectiveness for preserving privacy in large-scale language models during
training processes. Blockchain technology involves distributed networks of
high computation devices that additionally require significantly high energy
consumptions and high-volume storages. The transactions in the blockchain rely
on the consensus mechanism that can introduce latency and slow down the
training process, impacting the overall performance training large language
models and associated applications that require real-time responses. It is
also evident the blockchain poses privacy concerns, despite it provides
transparency and immutability, if the data is stored on public blockchain,
causing leakage of private information to unauthorised parties. Henceforth,
these challenges require robust design and optimisation, e.g., development of
specialised blockchain architecture including consensus mechanism, of learning
models for training large language models using blockchain technology.

###  VII-C Private Information Retrieval (PIR)

The PIR protocols, such as [109, 110, 111, 112, 113, 114], enable to query
data from (public) databases without revealing specific query or the data
being retrieved from the databases. Hence, the use of PIR techniques may be
useful during training LLMs where necessary information can be retrieved from
multiple resources without disclosing sensitive information, ensuring privacy
during training process. The PIR techniques are based on complex cryptographic
techniques, for query encoding and decoding, and require high processing costs
to privately calculate the query responses, hence slowing down the training
time. Consequently, in addition to high computation costs, it is also
important to evaluate their applicability and scalability in such
environments, especially for models that require frequent access to vast
amounts of datasets.

Furthermore, the communication costs can also lead to high latency, therefore
it is important to efficiently manage the communication overheads while
preserving user privacy during the training process. The PIR techniques
require specific data structure i.e., indexes and other auxiliary information,
to enable private query processing and retrieval and requires additional data
storage, hence it necessitates increased data management and optimise the
storage requirements. Integrating the PIR protocols into the training process
of existing large language models also require modified data access mechanism
and compatibility with the existing infrastructure. In addition, introducing
PIR in the existing training process for LLMs may have an impact on the
utility of the resultant response, hence finding a balance between the privacy
and accuracy is crucial to the effectiveness of the trained models. Similarly,
similar to other privacy preserving techniques, it is vital to address added
aspects of compliance with regulatory frameworks and maintaining transparency
during the training process of LLMs.

Privacy Techniques vs. Metrics |  Model Performance and Accuracy |  Computational Complexity |  Scalability and Efficiency |  Privacy vs. Utility |  Training Latency |  Vulnerability to Attacks |  Resource Consumption
---|---|---|---|---|---|---|---
Differential Privacy |  Low |  Low |  Medium |  Low to medium |  Low |  Yes (medium) |  Low
Blockchain |  Low |  High |  Medium to high |  Medium to high |  High |  Yes (low) |  High
Private Information Retrieval |  Low |  High |  Medium to high |  Medium to high |  High |  Yes (low) |  High
Obfuscation |  Low to high |  Medium to high |  Low impact |  Low to medium |  Low |  Yes (medium) |  Low
ToR |  High |  High |  Low |  Medium to high |  High |  Yes (medium) |  Medium
Anonymisation |  Low to high |  Medium to high |  Medium |  Low to medium |  Low |  Yes (medium) |  Low
TABLE I: Comparison of various privacy protection mechanisms for training LLMs
for various important parameters.

###  VII-D Randomisation

Randomisation can be effective for preserving user privacy in training large
language models by introducing random sampling of training examples or noise
injection. However, randomisation also present certain challenges, e.g., the
introduced noise can impact the utility of the trained model, hence it is
important to find a decent balance between the privacy and utility. It is also
worth noting, to achieve the desired level of privacy, that randomisation may
require large amount of training data for efficient privacy protection. The
excessive randomisation can also lead to increased risk of overfitting; hence
it may impact the generalisation ability of the trained model, which may
result in decreased performance on real-world testing tasks.

In addition, the excessive randomisation can lead to loss of accuracy and
performance and additionally computational overheads whereases small
randomisation may not provide adequate privacy protection. Similarly, the
randomisation may make it more difficult to interpret and explain the decision
of trained model since the added noise may obscure the relationship in the
data, which can make it difficult to extract the contextual information and
make it harder to understand the model’s reasoning. It requires an ongoing
research and development for developing effective randomisation techniques
that can tailor the needs of training large language models for preserving
privacy.

###  VII-E Obfuscation

Depending on the specific privacy requirements and trade-offs between privacy
and utility, the obfuscation i.e., to modify the training data or the model
parameters, can be effective in preventing direct inference of sensitive
information during the training process.

We argue that the obfuscation may potentially lead to a decrease in model
performance and accuracy since the obfuscated data may contain less
informative features that can impact the model’s capability to learn and make
accurate predictions. Hence, it is important to find the right level of
obfuscation that provides strong obfuscation and make it useful for training
the LLMs. The process of obfuscation may involve the tokenising or anonymising
the data that require extra computational resources and can slow down the
training process. This may increase the overall complexity of the system,
especially integrating with existing applications. In addition, the
obfuscation techniques may face scalability issues while dealing with large
scale and complex datasets that may result in increased computational costs.
Furthermore, the obfuscation techniques may still be susceptible to reverse-
engineering attacks where the adversary may uncover the sensitive information.

###  VII-F Anonymisation

Training the large language models using anonymisation techniques can minimise
the risk of exposing sensitive information while ensuring that the sensitive
information is not directly linked to individuals. It also allows for secure
data sharing and collaborations, with external parties, stakeholders, or among
diverse large language models for training purposes. This can be particularly
useful, for ethical considerations, when dealing with sensitive data, such as
financial data or healthcare records; hence the organisations and individuals
may be more willing to share their data.

There are several challenges that requires careful considerations for a
comprehensive approach that combines appropriate anonymisation techniques for
ensuring privacy protection during training large language models. As an
example, anonymisation techniques require to remove or modify certain
attributes from the data, it may lead to loss of context or other important
features that may result in biased outcome and reduced performance. This may
also impact the data its quality and representativeness, leading to less
reliable models. It is also important to note that the adversaries may use
auxiliary data or advanced re-identification techniques to de-anonymise the
data and compromise privacy. Similar other techniques, the anonymisation
process can introduce computational overheads and may increase the complexity
of training large language models, resulting in low scalability and
performance. Furthermore, it is challenging to consider adhering to privacy
regulations and legal requirements when anonymising the data, which
additionally requires a balance between training large language models and
meeting the legal obligations and compliance. Assessing the effectiveness and
privacy guarantees of anonymisation techniques can be challenging, especially
overfitting or underfitting models, which requires the design of robust
machine learning algorithms for private LLMs.

###  VII-G Use of ToR

ToR would be particularly useful during collecting data, hence by leveraging
the ToR during the LLMs training process, various features of the data can be
hidden such as IP address, participant’s locations, etc., which can add an
extra layer of privacy to the training process.

We note that it may limit to collect large-scale and diverse data because of
the decentralised and anonymous nature of ToR, making it hard to train models
over representative datasets. Introducing ToR into the existing structure of
LLMs will introduce additional latency in routing traffic due to multiple
hops, which will have a greater impact on the performance and speed of
collecting data, with added impact of slowing down the training process. It is
worth noting that ToR may not be accessible in some regions or due to network
restrictions; this will limit the data availability, which may have issues
with generalising learning models and the effectiveness of the resultant
trained model. It is also evident that ToR may not guarantee end-to-end
security, hence malicious attacks may compromise the data collection process,
exposing sensitive information. Similarly, the authenticity and integrity of
the data via ToR can be intriguing since it is difficult to validate the
source and quality of the data because of anonymity and encryption provided by
ToR.

Table I summarises various privacy preserving techniques for training large
language models under different important parameters including, model
performance and accuracy, computational complexity, scalability and
efficiency, privacy vs. utility, training latency, vulnerability to attacks,
and resource consumption. It is important to consider that these techniques
can enhance the privacy for training large models, these techniques also come
with certain important considerations. For example, the differential privacy,
randomisation, and obfuscation can impact the utility and performance of the
trained models. The use of ToR, Blockchain, and PIR may introduce additional
computational complexity and high training latency. Similarly, anonymisation
techniques need to be carefully applied in order to achieve a balance between
privacy and usefulness of trained model. Hence, implementing a specific
technique depends on the privacy requirements, training environment
constraints for large language model, and the threat models.

##  VIII Summary and Conclusions

The generative AI tools are maintained and run by private organisations with
their own business interests more willingly than the society’s benefits, which
alarms that the users must be very extremely proactive about their own privacy
by not sharing sensitive information. We present several recommendations and
guidelines for privately interacting with generative AI tools, various
policies for organisations and individuals to protect organisational sensitive
data, and guidelines for awareness program about the use of LLM-based
generative AI tools. Our proposed model, PrivChatGPT for privacy-preserving
model for LLMs, aims to protect user privacy during the information extraction
and further provides steps for private training and its integration into the
existing language models. To demonstrate its practicality, we present an
approach for implementing PrivChatGPT, discussing how a private mechanism can
be integrated into the existing model to protect user privacy. Finally, we
present several challenges, trends and future research directions for use of
various privacy preserving techniques such as, differential privacy,
blockchain, private information retrieval, randomisation, obfuscation,
anonymisation, and the use of ToR. We further present detailed comparison of
these privacy protection mechanisms for training LLMs for various important
performance metrics. It is vital to evaluate the performance and applicability
of the privacy-preserving mechanisms used for training large language models,
since their inclusion would introduce additional complexity and resource-
intensive evaluation. These additional considerations might have greater
impact on their scalability, efficiency, may impact their utility, or
integrating them in the existing generative AI applications might be quite
problematic, especially for resource-constrained device or delay intolerant
applications. We plan to implement and evaluate the applicability of the
proposed PrivChatGPT for private LLMs, compare it with the classic language
models for various performance measures such as accuracy, computation time, in
addition to, complexity of their implementation.

## References

  * [1] T. A. Chang and B. K. Bergen, “Language model behavior: A comprehensive survey,” arXiv preprint arXiv:2303.11504, 2023.
  * [2] S. Dev, E. Sheng, J. Zhao, A. Amstutz, J. Sun, Y. Hou, M. Sanseverino, J. Kim, A. Nishi, N. Peng, et al., “On measures of biases and harms in NLP,” arXiv preprint arXiv:2108.03362, 2021.
  * [3] D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al., “Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,” arXiv preprint arXiv:2209.07858, 2022.
  * [4] S. Hassan, M. Huenerfauth, and C. O. Alm, “Unpacking the interdependent systems of discrimination: Ableist bias in NLP systems through an intersectional lens,” arXiv preprint arXiv:2110.00521, 2021.
  * [5] N. Ousidhoum, X. Zhao, T. Fang, Y. Song, and D.-Y. Yeung, “Probing toxic content in large pre-trained language models,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4262–4274, 2021.
  * [6] D. Nozza, F. Bianchi, A. Lauscher, D. Hovy, et al., “Measuring harmful sentence completion in language models for LGBTQIA+ individuals,” in Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, Association for Computational Linguistics, 2022.
  * [7] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “RealToxicityPrompts: Evaluating neural toxic degeneration in language models,” arXiv preprint arXiv:2009.11462, 2020.
  * [8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  * [9] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  * [10] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al., “Palm: Scaling language modeling with pathways,” arXiv preprint arXiv:2204.02311, 2022.
  * [11] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al., “OPT: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068, 2022.
  * [12] K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini, “Deduplicating training data makes language models better,” arXiv preprint arXiv:2107.06499, 2021.
  * [13] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020.
  * [14] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al., “The pile: An 800gb dataset of diverse text for language modeling,” arXiv preprint arXiv:2101.00027, 2020\.
  * [15] E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh, “Universal adversarial triggers for attacking and analyzing NLP,” arXiv preprint arXiv:1908.07125, 2019.
  * [16] H. S. Heidenreich and J. R. Williams, “The earth is flat and the sun is not a star: The susceptibility of GPT-2 to universal adversarial triggers,” in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 566–573, 2021.
  * [17] N. Mehrabi, A. Beirami, F. Morstatter, and A. Galstyan, “Robust conversational agents against imperceptible toxicity triggers,” arXiv preprint arXiv:2205.02392, 2022.
  * [18] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red teaming language models with language models,” arXiv preprint arXiv:2202.03286, 2022.
  * [19] J. Huang, H. Shao, and K. C.-C. Chang, “Are large pre-trained language models leaking your personal information?,” arXiv preprint arXiv:2205.12628, 2022\.
  * [20] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. B. Brown, D. Song, U. Erlingsson, et al., “Extracting training data from large language models.,” in USENIX Security Symposium, vol. 6, 2021.
  * [21] N. Kandpal, E. Wallace, and C. Raffel, “Deduplicating training data mitigates privacy risks in language models,” in International Conference on Machine Learning, pp. 10697–10707, PMLR, 2022.
  * [22] N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang, “Quantifying memorization across neural language models,” arXiv preprint arXiv:2202.07646, 2022.
  * [23] D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini, “Preventing verbatim memorization in language models gives a false sense of privacy,” arXiv preprint arXiv:2210.17546, 2022.
  * [24] H. Nissenbaum, “Privacy as contextual integrity,” Wash. L. Rev., vol. 79, p. 119, 2004.
  * [25] A. Pfitzmann and M. Hansen, “A terminology for talking about privacy by data minimization: Anonymity, unlinkability, undetectability, unobservability, pseudonymity, and identity management,” 2010.
  * [26] H. Mozaffari and A. Houmansadr, “Heterogeneous private information retrieval,” in Network and Distributed Systems Security (NDSS) Symposium 2020, 2020.
  * [27] I. Ullah, B. G. Sarwar, R. Boreli, S. S. Kanhere, S. Katzenbeisser, and M. Hollick, “Enabling privacy preserving mobile advertising via private information retrieval,” in 2017 IEEE 42nd Conference on Local Computer Networks (LCN), pp. 347–355, IEEE, 2017.
  * [28] Ş. Bodur, E. Martínez-Moro, and D. Ruano, “Private information retrieval schemes using cyclic codes,” in Arithmetic of Finite Fields: 9th International Workshop, WAIFI 2022, Chengdu, China, August 29–September 2, 2022, Revised Selected Papers, pp. 194–207, Springer, 2023.
  * [29] I. Ullah, S. S. Kanhere, and R. Boreli, “Privacy-preserving targeted mobile advertising: A blockchain-based framework for mobile ads,” Journal of Network and Computer Applications, vol. 211, p. 103559, 2023.
  * [30] I. Ullah, R. Boreli, S. S. Kanhere, S. Chawla, T. A. Ahanger, and U. Tariq, “Protecting private attributes in app based mobile user profiling,” IEEE Access, vol. 8, pp. 143818–143836, 2020.
  * [31] I. Ullah, R. Boreli, S. S. Kanhere, and S. Chawla, “Profileguard: Privacy preserving obfuscation for mobile user profiles,” in Proceedings of the 13th Workshop on Privacy in the Electronic Society, pp. 83–92, 2014.
  * [32] D. Sánchez and A. Viejo, “Privacy-preserving and advertising-friendly web surfing,” Computer Communications, vol. 130, pp. 113–123, 2018.
  * [33] A. Checco, L. Bracciale, D. J. Leith, and G. Bianchi, “Opennym: Privacy preserving recommending via pseudonymous group authentication,” Security and Privacy, vol. 5, no. 2, p. e201, 2022.
  * [34] L. Bussard, Y. Roudier, and R. Molva, “Untraceable secret credentials: Trust establishment with privacy,” in IEEE Annual Conference on Pervasive Computing and Communications Workshops, 2004. Proceedings of the Second, pp. 122–126, IEEE, 2004.
  * [35] D. L. Quoc, M. Beck, P. Bhatotia, R. Chen, C. Fetzer, and T. Strufe, “Privacy preserving stream analytics: The marriage of randomized response and approximate computing,” arXiv preprint arXiv:1701.05403, 2017.
  * [36] J. S. Kim, J. W. Kim, and Y. D. Chung, “Successive point-of-interest recommendation with local differential privacy,” IEEE Access, vol. 9, pp. 66371–66386, 2021.
  * [37] C. Wang, Y. Zheng, J. Jiang, and K. Ren, “Toward privacy-preserving personalized recommendation services,” Engineering, vol. 4, no. 1, pp. 21–28, 2018.
  * [38] Q. Chen, Z. Ni, X. Zhu, and P. Xia, “Differential privacy histogram publishing method based on dynamic sliding window,” Frontiers of Computer Science, vol. 17, no. 4, p. 174809, 2023.
  * [39] C. Dwork, “Differential privacy: A survey of results,” in Theory and Applications of Models of Computation: 5th International Conference, TAMC 2008, Xi’an, China, April 25-29, 2008. Proceedings 5, pp. 1–19, Springer, 2008\.
  * [40] C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006\. Proceedings 3, pp. 265–284, Springer, 2006.
  * [41] Y. K. Dwivedi, N. Kshetri, L. Hughes, E. L. Slade, A. Jeyaraj, A. K. Kar, A. M. Baabdullah, A. Koohang, V. Raghavan, M. Ahuja, et al., ““So what if ChatGPT wrote it?” multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy,” International Journal of Information Management, vol. 71, p. 102642, 2023.
  * [42] S. S. Gill, M. Xu, P. Patros, H. Wu, R. Kaur, K. Kaur, S. Fuller, M. Singh, P. Arora, A. K. Parlikad, et al., “Transformative effects of ChatGPT on modern education: Emerging era of AI chatbots,” Internet of Things and Cyber-Physical Systems, vol. 4, pp. 19–23, 2024.
  * [43] M. M. Mariani, I. Machado, and S. Nambisan, “Types of innovation and artificial intelligence: A systematic quantitative literature review and research agenda,” Journal of Business Research, vol. 155, p. 113364, 2023\.
  * [44] A. Metz, “6 exciting ways to use ChatGPT – from coding to poetry,” https://www.techradar.com/features/6-exciting-ways-to-use-ChatGPT-from-coding-to-poetry, June 2023.
  * [45] L. Reed, “ChatGPT for automated testing: From conversation to code,” https://saucelabs.com/resources/blog/ChatGPT-automated-testing-conversation-to-code, June 2023.
  * [46] L. Tung, “ChatGPT can write code. now researchers say it’s good at fixing bugs, too,” https://www.zdnet.com/article/ChatGPT-can-write-code-now-researchers-say-its-good-at-fixing-bugs-too/, June 2023.
  * [47] H. Else, “Abstracts written by ChatGPT fool scientists,” https://www.nature.com/articles/d41586-023-00056-7, June 2023.
  * [48] C. Stokel-Walker, “ChatGPT listed as author on research papers: many scientists disapprove,” https://www.nature.com/articles/d41586-023-00107-z, June 2023.
  * [49] B. Williamson, S. Bayne, and S. Shay, “The datafication of teaching in higher education: critical issues and perspectives,” 2020.
  * [50] B. Williamson and R. Eynon, “Historical threads, missing links, and future directions in AI in education,” 2020.
  * [51] N. Selwyn, “The future of AI and education: Some cautionary notes,” European Journal of Education, vol. 57, no. 4, pp. 620–631, 2022.
  * [52] S. S. Gill and R. Kaur, “ChatGPT: Vision and challenges,” Internet of Things and Cyber-Physical Systems, vol. 3, pp. 262–271, 2023.
  * [53] C. F. Breidbach and P. Maglio, “Accountable algorithms? the ethical implications of data-driven business models,” Journal of Service Management, vol. 31, no. 2, pp. 163–185, 2020.
  * [54] J. Wirtz, W. H. Kunz, N. Hartley, and J. Tarbit, “Corporate digital responsibility in service firms and their ecosystems,” Journal of Service Research, vol. 26, no. 2, pp. 173–190, 2023.
  * [55] B. Lucey and M. Dowling, “ChatGPT: our study shows AI can produce academic papers good enough for journals – just as some ban it,” https://theconversation.com/ChatGPT-our-study-shows-ai-can-produce-academic-papers-good-enough-for-journals-just-as-some-ban-it-197762, June 2023.
  * [56] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in 2017 IEEE symposium on security and privacy (SP), pp. 3–18, IEEE, 2017.
  * [57] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning (still) requires rethinking generalization,” Communications of the ACM, vol. 64, no. 3, pp. 107–115, 2021.
  * [58] S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha, “Privacy risk in machine learning: Analyzing the connection to overfitting,” in 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268–282, IEEE, 2018.
  * [59] S. Hisamoto, M. Post, and K. Duh, “Membership inference attacks on sequence-to-sequence models: Is my data in your machine translation system?,” Transactions of the Association for Computational Linguistics, vol. 8, pp. 49–63, 2020.
  * [60] M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning,” in 2019 IEEE symposium on security and privacy (SP), pp. 739–753, IEEE, 2019.
  * [61] C. Song and V. Shmatikov, “Auditing data provenance in text-generation models,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 196–206, 2019.
  * [62] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1322–1333, 2015.
  * [63] A. Irons and T. Crick, “Cybersecurity in the digital classroom: implications for emerging policy, pedagogy and practice,” in The Emerald handbook of higher education in a post-COVID world: New approaches and technologies for teaching and learning, pp. 231–244, Emerald Publishing Limited, 2022.
  * [64] L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, et al., “Ethical and social risks of harm from language models,” arXiv preprint arXiv:2112.04359, 2021.
  * [65] S. Dhingra, M. Singh, V. SB, et al., “Mind meets machine: Unravelling GPT-4’s cognitive psychology,” BenchCouncil Transactions on Benchmarks, Standards and Evaluations, vol. 3, no. 3, p. 100139, 2023.
  * [66] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27730–27744, 2022.
  * [67] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017\.
  * [68] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,” Advances in Neural Information Processing Systems, vol. 33, pp. 3008–3021, 2020.
  * [69] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021.
  * [70] A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. Tezak, J. W. Kim, C. Hallacy, et al., “Text and code embeddings by contrastive pre-training,” arXiv preprint arXiv:2201.10005, 2022.
  * [71] H. Alex, “Chatgpt: Everything you need to know about openai’s gpt-4 tool,” https://www.sciencefocus.com/future-technology/gpt-3/, May 2023.
  * [72] Ö. Aydın and E. Karaarslan, “Is chatgpt leading generative ai? what is beyond expectations?,” What is beyond expectations, 2023.
  * [73] S. A. Khowaja, P. Khuwaja, and K. Dev, “Chatgpt needs spade (sustainability, privacy, digital divide, and ethics) evaluation: A review,” arXiv preprint arXiv:2305.03123, 2023.
  * [74] OpenAI, “OpenAI: Privacy policy,” https://openai.com/policies/privacy-policy, May 2023.
  * [75] S. Guha, B. Cheng, and P. Francis, “Privad: Practical privacy in online advertising,” in USENIX conference on Networked systems design and implementation, pp. 169–182, 2011.
  * [76] V. Toubiana, A. Narayanan, D. Boneh, H. Nissenbaum, and S. Barocas, “Adnostic: Privacy preserving targeted advertising,” in NDSS, 2010.
  * [77] M. Backes, A. Kate, M. Maffei, and K. Pecina, “Obliviad: Provably secure and practical online behavioral advertising,” pp. 257–271, 2012.
  * [78] K. Zhong, Y. Ma, and S. Angel, “Ibex: Privacy-preserving ad conversion tracking and bidding,” in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pp. 3223–3237, 2022.
  * [79] D. Liu, “Security and privacy preservation in mobile advertising,” 2020.
  * [80] I. Ullah, R. Boreli, and S. S. Kanhere, “Privacy in targeted advertising: A survey,” arXiv preprint arXiv:2009.06861, 2020.
  * [81] E. Erdemir, P. L. Dragotti, and D. Gündüz, “Privacy-aware time-series data sharing with deep reinforcement learning,” IEEE Transactions on Information Forensics and Security, vol. 16, pp. 389–401, 2020.
  * [82] E. Shi, H. Chan, E. Rieffel, R. Chow, and D. Song, “Privacy-preserving aggregation of time-series data,” in Annual Network & Distributed System Security Symposium (NDSS), Internet Society., 2011.
  * [83] D. Jang, R. Jhala, S. Lerner, and H. Shacham, “An empirical study of privacy-violating information flows in javascript web applications,” in Proceedings of the 17th ACM conference on Computer and communications security, pp. 270–283, 2010.
  * [84] S.-K. Hong, K. Gurjar, H.-S. Kim, and Y.-S. Moon, “A survey on privacy preserving time-series data mining,”
  * [85] H. Brown, K. Lee, F. Mireshghallah, R. Shokri, and F. Tramèr, “What does it mean for a language model to preserve privacy?,” in 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 2280–2292, 2022\.
  * [86] H. A. Inan, O. Ramadan, L. Wutschitz, D. Jones, V. Rühle, J. Withers, and R. Sim, “Training data leakage analysis in language models,” arXiv preprint arXiv:2101.05405, 2021.
  * [87] V. Feldman, “Does learning require memorization? a short tale about a long tail,” in Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954–959, 2020.
  * [88] W. Shi, A. Cui, E. Li, R. Jia, and Z. Yu, “Selective differential privacy for language modeling,” arXiv preprint arXiv:2108.12944, 2021.
  * [89] R. Anil, B. Ghazi, V. Gupta, R. Kumar, and P. Manurangsi, “Large-scale differentially private bert,” arXiv preprint arXiv:2108.01624, 2021.
  * [90] S. Hoory, A. Feder, A. Tendler, S. Erell, A. Peled-Cohen, I. Laish, H. Nakhost, U. Stemmer, A. Benjamini, A. Hassidim, et al., “Learning and evaluating a differentially private pre-trained language model,” in Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1178–1189, 2021.
  * [91] X. Li, F. Tramer, P. Liang, and T. Hashimoto, “Large language models can be strong differentially private learners,” arXiv preprint arXiv:2110.05679, 2021.
  * [92] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning differentially private recurrent language models,” arXiv preprint arXiv:1710.06963, 2017\.
  * [93] C. Dwork, “A firm foundation for private data analysis,” Communications of the ACM, vol. 54, no. 1, pp. 86–95, 2011.
  * [94] J. Jin, E. McMurtry, B. I. Rubinstein, and O. Ohrimenko, “Are we there yet? timing and floating-point attacks on differential privacy systems,” in 2022 IEEE Symposium on Security and Privacy (SP), pp. 473–488, IEEE, 2022.
  * [95] C. Liu, S. Chakraborty, and P. Mittal, “Dependence makes you vulnberable: Differential privacy under dependent tuples.,” in NDSS, vol. 16, pp. 21–24, 2016.
  * [96] A. Haeberlen, B. C. Pierce, and A. Narayan, “Differential privacy under fire.,” in USENIX Security Symposium, vol. 33, p. 236, 2011.
  * [97] F. D. McSherry, “Privacy integrated queries: an extensible platform for privacy-preserving data analysis,” in Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, pp. 19–30, 2009.
  * [98] I. Roy, S. T. Setty, A. Kilzer, V. Shmatikov, and E. Witchel, “Airavat: Security and privacy for mapreduce.,” in NSDI, vol. 10, pp. 297–312, 2010\.
  * [99] J. Li, N. Li, and B. Ribeiro, “Membership inference attacks and defenses in classification models,” in Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy, pp. 5–16, 2021.
  * [100] C. L. Canonne, G. Kamath, and T. Steinke, “The discrete gaussian for differential privacy,” Advances in Neural Information Processing Systems, vol. 33, pp. 15676–15688, 2020.
  * [101] I. Mironov, “On significance of the least significant bits for differential privacy,” in Proceedings of the 2012 ACM conference on Computer and communications security, pp. 650–661, 2012.
  * [102] B. Song, M. Deng, S. R. Pokhrel, Q. Lan, R. Doss, and G. Li, “Digital privacy under attack: Challenges and enablers,” arXiv preprint arXiv:2302.09258, 2023.
  * [103] H. Li, Q. Chen, H. Zhu, D. Ma, H. Wen, and X. S. Shen, “Privacy leakage via de-anonymization and aggregation in heterogeneous social networks,” IEEE Transactions on Dependable and Secure Computing, vol. 17, no. 2, pp. 350–362, 2017.
  * [104] A. Pyrgelis, C. Troncoso, and E. De Cristofaro, “Knock knock, who’s there? membership inference on aggregate location data,” arXiv preprint arXiv:1708.06145, 2017.
  * [105] F. Xu, Z. Tu, Y. Li, P. Zhang, X. Fu, and D. Jin, “Trajectory recovery from ash: User privacy is not preserved in aggregated mobility data,” in Proceedings of the 26th international conference on world wide web, pp. 1241–1250, 2017.
  * [106] T. Minkus, Y. Ding, R. Dey, and K. W. Ross, “The city privacy attack: Combining social media and public records for detailed profiles of adults and children,” in Proceedings of the 2015 ACM on conference on online social networks, pp. 71–81, 2015.
  * [107] F. Xu, Z. Tu, H. Huang, S. Chang, F. Sun, D. Guo, and Y. Li, “No more than what i post: Preventing linkage attacks on check-in services,” in The World Wide Web Conference, pp. 3405–3412, 2019.
  * [108] T. Nóbrega, C. E. S. Pires, and D. C. Nascimento, “Blockchain-based privacy-preserving record linkage: enhancing data privacy in an untrusted environment,” Information Systems, vol. 102, p. 101826, 2021.
  * [109] B. Chor, N. Gilboa, and M. Naor, Private information retrieval by keywords.  Citeseer, 1997.
  * [110] B. Chor, E. Kushilevitz, O. Goldreich, and M. Sudan, “Private information retrieval,” Journal of the ACM (JACM), vol. 45, no. 6, pp. 965–981, 1998\.
  * [111] E. Kushilevitz and R. Ostrovsky, “Replication is not needed: Single database, computationally-private information retrieval,” in Proceedings 38th annual symposium on foundations of computer science, pp. 364–373, IEEE, 1997\.
  * [112] B. Chor and N. Gilboa, “Computationally private information retrieval,” in Proceedings of the twenty-ninth annual ACM symposium on Theory of computing, pp. 304–313, 1997.
  * [113] I. Goldberg, “Improving the robustness of private information retrieval,” in 2007 IEEE Symposium on Security and Privacy (SP’07), pp. 131–148, IEEE, 2007.
  * [114] C. Devet and I. Goldberg, “The best of both worlds: Combining information-theoretic and computational pir for communication efficiency,” in Privacy Enhancing Technologies: 14th International Symposium, PETS 2014, Amsterdam, The Netherlands, July 16-18, 2014. Proceedings 14, pp. 63–82, Springer, 2014.