# AIGC In China: Current Developments And Future Outlook

Xiangyu Li1, Yuqing Fan2,3, Shenghui Cheng2,3
1 School of Shandong Foreign Languages Vocational College, Rizhao, China
2 Research Center for the Industries of the Future,
Westlake University, Hangzhou, China
3 School of Engineering, Westlake University, Hangzhou, China,

###### Abstract

The increasing attention given to AI Generated Content (AIGC) has brought a
profound impact on various aspects of daily life, industrial manufacturing,
and the academic sector. Recognizing the global trends and competitiveness in
AIGC development, this study aims to analyze China’s current status in the
field. The investigation begins with an overview of the foundational
technologies and current applications of AIGC. Subsequently, the study delves
into the market status, policy landscape, and development trajectory of AIGC
in China, utilizing keyword searches to identify relevant scholarly papers.
Furthermore, the paper provides a comprehensive examination of AIGC products
and their corresponding ecosystem, emphasizing the ecological construction of
AIGC. Finally, this paper discusses the challenges and risks faced by the AIGC
industry while presenting a forward-looking perspective on the industry’s
future based on competitive insights in AIGC.

##  1 Introduction

AIGC, or Artificial Intelligence Generated Content, refers to the production
of content facilitated by artificial intelligence technologies. The
introduction of ChatGPT has significantly impacted various aspects of our
lives and professional endeavors. It utilizes input data and existing
information to conduct insightful analyses and generate relevant content based
on user preferences. For example, AIGC can offer article suggestions and
assistance in writing. Similarly, it can aid visual artists by providing
relevant images, while video and music creators can benefit from its ability
to generate video and audio content. In terms of everyday life, ChatGPT can
serve as a valuable resource for finding solutions, such as recipes for
cooking, ideas for children’s crafts, and guidance on setting up smart homes.
In professional settings, ChatGPT can streamline tasks such as data
organization, report generation, and even assist in creating presentations,
programming, and website development. By reducing time spent on routine tasks
and enhancing production efficiency, AIGC enables individuals to allocate more
time to personal commitments, family engagements, and fostering stronger
relationships within their professional teams.

AIGC has emerged as a disruptive and transformative technology, offering
significant advantages in promoting economic growth and enhancing
productivity. Its potential to reshape the global industry is widely
recognized, leading countries like China to prioritize the development of
artificial intelligence within their national agendas. This paper aims to
explore the current state of AIGC development in China and assess its impact
on the country’s global competitiveness in this field.

Currently, AIGC has garnered significant societal attention in China,
primarily due to the proliferation of large-scale models and open-source
frameworks. This development has positioned AIGC as a promising field for the
practical application of artificial intelligence. On one hand, the
accessibility and availability of large models and open-source frameworks have
lowered the barriers to entry for AIGC and expanded its application potential.
On the other hand, the boundaries between AI and innovation have become
increasingly blurred, highlighting the growing trend of their integration
([40]).

To draw a conclusion, AIGC constitutes a profound transformation that will
significantly impact China’s social, technological and economic lives. This
comprehensive article aims to provide a detailed introduction to the concept
of current development in China. The contributions of this paper including: 1)
a thorough exposition of the compositions and applications of AIGC, and a
comprehensive analysis of its current development status from the perspectives
of policy, market, and scientific research in China; 2) an exploration of the
changes and opportunities of AIGC, identifying its developmental requirements
based on these changes and opportunities; and 3) an overview of the future
prospects for the evolution of AIGC in China.

##  2 Compositions and applications

This section first introduces the foundation technology of AIGC, then provides
some common applications that has been used widely.

###  2.1 Foundations of AIGC

The development of AIGC has been ongoing for a considerable period of time.
Its success today can be attributed to the exploration of several other
computer technologies. Many scholars have reviewed the foundational
technologies of AIGC. For example, [6] identified Foundation models,
Reinforcement learning with human feedback (RLHF), and advancements in
computing technology (such as hardware, distributed computing, and cloud
computing) as AIGC key foundational technologies. [41] categorized the
founding technology of AIGC into two groups: general techniques and creation
techniques. General techniques include Backbone architecture and Self-
supervised pretraining, while creation techniques encompass likelihood-based
models, energy-based models, GAN, and the Transformation model. [37] argued
that the development of AIGC technology requires three essential elements:
data for training models, hardware for computing power, and algorithms for
model establishment. They also proposed a five-stage evolution of AI models,
progressing from Machine Learning, Neural Network Model (NNM), Generative
algorithm, Language Model, to Large-scale Pre-trained Model.

We suggest that the foundations of AIGC technology lie in Large Language
Models (LLM), pre-trained models, Multimodal Models, and Reinforcement
learning with human feedback.

####  2.1.1 Large Language Models

A large language model (LLM) is a neural network-based system employed in
natural language processing to comprehend the underlying patterns and
structures of human language. These models are trained through unsupervised
learning techniques and have the ability to generate coherent and
grammatically accurate sentences without explicit programming or specific rule
sets.

The Transformer model is widely used in contemporary deep learning models, it
represents a significant advancement in large language models. This
methodology was developed to overcome the limitations of traditional Recurrent
Neural Networks in effectively processing variable-length input sequences
while maintaining contextual awareness. Notably, the Transformer model
incorporates a self-attention mechanism, which allows the model to selectively
focus on and weigh different segments of the input sequence ([42]).

The Transformer model consists of two primary components: an encoder and a
decoder. The encoder operates on the input sequence, generating hidden
representations, while the decoder utilizes these representations to produce
an output sequence. Each layer of the encoder and decoder comprises a multi-
head attention sub-component combined with a feed-forward neural network. The
core functionality of the multi-head attention module allows for assigning
varying weights to individual tokens based on their relevance. This capability
enhances the model’s performance across a wide range of Natural Language
Processing (NLP) tasks and improves its ability to handle long-term
dependencies, particularly those with intricate syntactic structures ([34]).
Figure 1 illustrates the structure of the transformer model.

Figure 1: Structure of Transformer

In addition to these advantages, a notable characteristic of the Transformer
model is its heightened parallelizability and capacity to overcome inherent
biases, thus enhancing its viability for extensive pre-training tasks. These
pre-training tasks facilitate the creation of versatile transformer-based
models that can be applied effectively to a diverse range of downstream
problems. As a result, the Transformer has emerged as a critical component in
contemporary natural language processing applications, representing the state-
of-the-art in the field ([6]).

####  2.1.2 Pre-trained models

To address the challenge of word ambiguity, scholars proposed integrating word
vector models into subsequent task models during the training process. This
concept laid the foundation for pre-trained models, with the ElMo model
utilizing Bi-LSTM serving as the first successful case ([31]). Subsequently,
the renowned GPT model emerged, representing a typical pre-trained language
model designed to comprehend text effectively and subsequently fine-tuned for
various downstream tasks, embodying the concept of a single model solving
multiple problems.

GPT-1, the initial model in the GPT series, is a pre-trained language model
constructed by stacking 12 Transformer decoder models. Like other pre-trained
language models, GPT-1 undergoes two phases to carry out specific tasks: pre-
training and fine-tuning. The objective of the pre-training phase is to equip
the GPT-1 model with an understanding of the text’s content by implementing
appropriate pre-training tasks. Specifically, the pre-training task can be
perceived as a text completion task. Initially, a vast corpus comprising high-
quality natural language texts is collected, and each text is partially
masked. The model reads the text before masking and predicts the masked
content one token at a time. Following pre-training, the GPT-1 model can be
fine-tuned for specific downstream tasks. For instance, in text
classification, the training texts are annotated accordingly, such as labeling
positive as 0 and negative as 1. Each training text is then manually
annotated, marking this form of training as supervised training ([29]).

Subsequently, researchers further explored the potential of GPT-1 and put
forth a daring hypothesis: if the GPT model’s understanding of a text is
further enhanced, can it generate reasonable answers solely based on textual
input? Based on this hypothesis, scholars advanced the GPT-2 model ([36]). The
core idea behind GPT-2 is to eliminate the fine-tuning process utilized in
GPT-1 and employ the question itself as the input after pre-training. This
approach enables the model to directly generate answers through text
generation. This input type is often referred to as a ”prompt.” For tasks like
text sentiment classification, determining the sentiment of a statement can be
accomplished by designing a suitable prompt that frames the specific question.
The text generation model can generate positive or negative directly, thereby
completing the text classification. In such cases, the prompt does not provide
any explicit indications of how the question should be answered, a phenomenon
known as zero-shot learning.

GPT-3 can be viewed as an enhanced version of GPT-2 ([4]). The structural
difference between the two models is insignificant. However, the parameter
size of GPT-3 has further increased from 1.5 billion in GPT-2 to 175 billion.
Another noteworthy innovation in GPT-3 lies in the design of prompts.
Researchers, inspired by GPT-2’s ability to accomplish downstream tasks
without fine-tuning, propose incorporating specific hints in prompt design to
enhance the model’s performance for particular tasks. More specifically,
GPT-3’s training data no longer comprises solely natural language text but
includes high-quality prompts tailored for specific tasks. Each prompt
encompasses several small example prompts. For instance, in text sentiment
translation, the corresponding prompt can be devised with numerous small
samples. With the assistance of these slight prompts, the model can obtain the
desired answer through forward computation alone, a phenomenon known as few-
shot learning. Through the implementation of few-shot learning, GPT-3 has
showcased the potential of this learning approach to the wider scientific
community.

####  2.1.3 Multimodal Models

Multimodal models encompass models that integrate heterogeneous data from
different modalities to perform a variety of tasks. Each modality presents
distinct data types that can be utilized collectively in a task. By
incorporating diversified data sources, multimodal models acquire richer
information, thereby improving the accuracy and efficiency of tasks ([3]). In
this section, we introduce two prominent multimodal models prevalent in
contemporary research: CLIP and DALL-E.

CLIP (Contrastive Language-Image Pre-Training) is a neural network-based model
devised and developed by OpenAI, aimed at training representations for both
textual and visual data. This model employs a standard training approach to
simultaneously train representations for text and images, leading to improved
performance across these two domains. The core concept of the CLIP model
involves mapping images and text into a standardized vector space, wherein
similar images and text are positioned closer to each other within this space.
This vector space construction method, known as contrastive learning,
harnesses the concepts of similarity and dissimilarity to train the model by
minimizing the distance between data belonging to the same class and
maximizing the distance between data from different classes ([28]).

DALL-E is a neural network-powered image generation model developed by OpenAI,
designed to generate images aligned with a given textual description. The
fundamental principle behind DALL-E is grounded in technologies such as
Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs),
which enable the conversion of text descriptions into corresponding images and
the generation of images consistent with the input text. Specifically, DALL-E
comprises two primary components: an image generation network and a text
encoder. The text encoder transforms the provided textual description into a
vector, which is subsequently fed into the image generation network. This
network leverages the vector representation to generate an image that
accurately corresponds to the given description. DALL-E employs large-scale
self-supervised learning to autonomously acquire knowledge from internet-
sourced images, empowering it to generate novel objects that have not been
previously encountered. Furthermore, DALL-E incorporates specialized
techniques such as multi-layered skinning, inlaying, and continuous geometry
to enhance the realism and diversity exhibited by the generated images ([24]).

####  2.1.4 RLHF

The success of ChatGPT can be attributed to its utilization of reinforcement
learning from human feedback (RLHF) in combination with the GPT-3 model, as
well as the implementation of a harm reduction approach to generating answers
([18]). To elaborate, RLHF involves training a reward model (RM) to simulate
human language preferences and employing reinforcement learning to train the
language model accordingly, aiming to generate responses that increasingly
align with human preferences ([25]).

The RLHF process consists of three steps. Initially, a set of approximately
10,000 prompts from past user data is selected, covering a diverse range of
task types and topics, while high-quality responses are annotated as labels.
Next, a feedback model is trained to evaluate language model-generated
responses using scores indicative of human preferences. Different answers for
each prompt are gathered and annotated personnel rank them based on quality,
forming the learning objective for training the feedback model. Lastly,
Proximal Policy Optimization (PPO), an optimization reinforcement learning
algorithm, is employed by developers to further enhance the performance of the
model. This algorithm enables the dynamic adjustment of the language model’s
strategy based on the feedback model’s scores, with the aim of achieving
higher scores. As the feedback model assigns higher scores to responses more
favorably received, the training process ensures that the model generates
responses that increasingly align with human preferences ([32]).

###  2.2 Applications of AIGC

The usage of AIGC in text and image generation has become increasingly mature.
Since 2021, many leading internet companies at home and abroad have released
various models for generating text and images. These text and image models
have achieved significant results in various application areas and have
released many software and hardware products for users. Figure 2 presents
several generative models after 2021.

Figure 2: Several famous generative models after 2021

####  2.2.1 Text generating

AIGC plays a crucial role in facilitating digital transformation as it offers
the ability to automate and expedite content creation processes. Through the
utilization of AI, high-quality text, images, audio, or video content can be
generated based on specified inputs and requirements. This feature
significantly assists individuals and businesses in swiftly producing diverse
content, including product descriptions, news releases, and social media
posts. The automation of content creation not only leads to substantial time
and labor cost savings but also enhances the efficiency and scalability of
content production.

The prevailing learning approach begins by establishing the learning style
using algorithms. Subsequently, the AI learns from vast amounts of textual
data and utilizes deep learning models and existing learning outcomes to
generate novel text and content. The following outlines the six training steps
employed in AI content generation: training, encoding, learning, generation,
optimization, and iteration.

In addition to its usage in generating articles, AIGC is also highly
applicable in code generation. Users can input abstract descriptions or
objectives, and AI can automatically generate corresponding code. Compared to
conventional coding methods, AIGC alleviates the workload of developers and
enhances development efficiency, which allows developers to spend time and
effort on more important issues.

####  2.2.2 Picture generating

The utilization of AI for the creation of artistic content has witnessed
significant growth. These techniques involve training algorithms on extensive
datasets of pre-existing artwork. Through machine learning, these methods are
capable of producing new artwork that either simulates the styles and
techniques of renowned artists or develops novel artistic styles altogether.
The remarkable advancements in diffusion-based models have prompted numerous
companies to introduce a multitude of AI-generated art products.

Among these models, DreamStudio, engineered by Stability.ai, stands out. This
platform utilizes stable diffusion techniques to generate images based on
provided phrases or sentences. DreamStudio exhibits comparable performance to
DALL-E-2, while boasting faster processing speeds ([30]).

State-of-the-art AI software integrated with text-to-image algorithms empowers
the generation of portraits, landscapes, abstract art, and even the
replication of famous artists’ styles within minutes. For instance, Anges
Russel from Sydney established a web-based platform called Night Cafe Creator
in 2019. Anges employed a novel model named VQGAN+CLIP which demonstrated
impressive results in creating oil painting-style images. Due to its success,
the Night Cafe application rapidly gained popularity, solidifying its position
as one of the most sought-after image generation software applications ([23]).

####  2.2.3 Other applications

In addition to previous applications, AIGC has made significant advancements
in various other domains. AI Text-to-Speech (TTS) converts written text into
natural and fluent speech. It has the ability to transform input text into
audible speech, replicating human speech features and expressions.
Applications of AI-TTS such as Deep Voice 3 had experienced significant growth
and is widely utilized in fields like voice assistants, voice navigation, and
voice broadcasting, enabling the production of high-quality and lifelike
speech output ([26]).

AI has also been applied to music editing and composition, serving as a tool
to aid or automate the process. Music producers, composers, and enthusiasts
can benefit from various AI-powered tools and systems that provide creative
inspiration and assistance during music creation and editing.

Furthermore, AIGC has the potential to revolutionize the gaming industry by
introducing innovative gameplay. With AI strategy generation, virtual
characters, known as Non-Player Characters (NPCs), can possess intelligent
decision-making abilities and behavior within games. Unlike conventional NPCs
that rely on predefined rules and scripts, AI-based NPCs can autonomously
learn from analyzing the game environment and player actions, allowing them to
dynamically adjust their strategies and provide more challenging and realistic
gaming experiences ([1]).

##  3 Development status

The inception of AIGC can be traced back to the 1980s, when researchers
initiated investigations into machine learning, natural language processing,
and related technologies for text content generation. In the 1990s, the
emergence of deep learning algorithms prompted researchers to explore the
utilization of neural networks for text generation. However, limitations in
hardware capabilities and data availability hindered the widespread
application of these technologies. The advent of the Internet and the
subsequent explosion in data accumulation catalyzed the rapid advancement of
artificial intelligence content generation technology. Around 2010,
researchers began utilizing deep learning techniques to generate more
intricate text and multimedia content. In 2014, Google presented a seminal
paper titled ”Generating Images through Neural Networks,” introducing the
concept of Generative Adversarial Networks (GANs). As a groundbreaking
technology capable of producing realistic images, GANs served as a significant
milestone in the development of AIGC.

The development of Generative Language Models (GLMs) has played a crucial role
in further enhancing the technology and products of AIGC. By continuously
exploring and experimenting with GLMs in research and application, AIGC
strives to introduce more advanced and efficient technologies, enabling it to
sustain its leadership position in artificial intelligence. This chapter will
delve into the market, policy, and research status of AIGC technology in
China, offering comprehensive insights into its development.

###  3.1 Market Status in China

The White Paper on the Development of China’s Digital Economy was recently
released by the China Academy of Information and Communication Technology.
According to the paper, the digital economy in China is experiencing a
significant acceleration, defying the prevailing trend. In 2020, China’s
digital economy reached a scale of 39.2 trillion Yuan, marking an increase of
3.3 trillion Yuan compared to the previous year. This accounted for 38.6% of
the country’s GDP, demonstrating a year-on-year increase of 2.4 percentage
points. This development has effectively supported efforts in epidemic
control, as well as economic and social progress ([27]).

The digital economy has been rapidly developing across various regions. Taking
into account the larger context, in 2020, there were 13 provinces in China
with a digital economy scale exceeding 1 trillion Yuan. Furthermore, in terms
of proportion to GDP, Beijing and Shanghai led the nation in digital economy
advancement. Meanwhile, Guizhou, Chongqing, and Fujian exhibited the highest
growth rates. While the digital economy is swiftly expanding, it continues to
radiate and drive the development of surrounding regions, forming distinct
models such as polar core, point axis, and multi-polar networks ([5]).

By the end of 2022, AIGC experienced an explosive trend in China. Advancements
in data, algorithms, and computing power have led to the continuous
improvement of AI technology, meeting diverse demands in areas such as text,
video, and audio. It is projected that by 2025, the market size of AIGC in
China will reach 777 billion Yuan. Additionally, the technology of virtual
humans based on deep synthesis will also undergo growth, with its market size
reaching 640.27 billion Yuan ([22]).

###  3.2 Policy Status in China

Currently speaking, China’s leading industrial policy on AIGC mainly focuses
on the combination of virtual and real industries, and the government has
introduced multiple laws and regulations on the development of AIGC technology
into physical industries. Table 1 shows part of China’s current policies, and
it can be observed that China’s current focus on AIGC technology management is
to prevent industrial hollowing out , and to guide cooperation between the
virtual economy and the real economy ([5]).

Table 1: Some current Chinese national policies on AIGC

Name of policy | Contents
---|---
| Action Plan for the Integration
---
and Development of VR and Industry
Applications(2016-2022)
| By 2026, AIGC will achieve large-scale application
---
The total industrial scale exceeds 350 billion yuan
Cultivate 100 backbone enterprises
Build 10 influential AIGC development zones
| Guiding Opinions on Accelerating Scenario
---
Innovation and Promoting High Quality
Economic Development through
High Level Application of AI
| Promote the integration of AI and real economy
---
Acceleration of AI products
Manage technological breakthroughs
Explore new paths for the development of AI
Using AI to promote high-quality economic development
| The 14th Five Year Plan
---
for the Development of Digital Economy
| Strengthen AI Tech convergence
---
Expand AI applications
Enhance the consumer experience
Building new digital consumption scenarios
| 2022 Chinese Government Work Report
---
| Promote the development of the digital economy
---
Improve the governance of the digital economy
Improve application capabilities
Helping economic development

Upon recognizing the potential risks associated with ChatGPT, various local
governments acknowledged the need for regulatory measures within the AIGC
industry. In this context, China took the initiative by issuing the first
regulatory document, known as the Administrative Measures for Generative AI
Services (Draft), which was made available for public opinion on April 11.
Comprising 21 articles, the Draft outlines comprehensive provisions pertaining
to the generative AI sector, encompassing aspects such as definitions, access
qualifications, responsibilities and obligations, as well as punishment
measures ([10]).

China’s governmental support extends to the independent innovation, promotion,
application, and international cooperation concerning fundamental technologies
like algorithms or frameworks. The draft specifically advocates for the
prioritization of secure and trustworthy software, tools, computing
capabilities, and data resources. The supportive stance and encouragement
towards the generative AI industry are explicitly articulated in the draft
document, which serves as an invitation for public input.

Additionally, numerous prominent domestic technology giants, including Baidu,
Tencent, Alibaba, Huawei, JD, 360, have unveiled their corresponding
strategies and plans within the generative AI domain. Concurrently, several
countries such as the United States, Italy, and Germany have taken note of the
risks posed by ChatGPT and are contemplating the introduction of corresponding
regulatory measures.

###  3.3 Development Status in China

This study offers empirical evidence to facilitate a comprehensive analysis of
the current research on AIGC technology. Furthermore, the Chinese knowledge
infrastructure (CNKI) database is used to examine the status of Chinese
research. This paper enables an investigation of notable differences between
these two research contexts.

As of June 12, 2023, 358 papers relevant to AIGC have been retrieved from the
CNKI database. Most of the papers are published after March 2023, which
presents a booming tendency. Due to the limited number of papers, the research
field are widely separated.

Figure 3: AIGC paper keywords

According to figure 3, the prominent keywords associated with AIGC include AI,
copyright, AI products, and originality. Analyzing the literature on AIGC
published in China, it can be classified into four main categories.

The first category pertains to theoretical research, which primarily focuses
on the development of AIGC. For instance, [2] outlined the fundamental aspects
for AIGC development, including network form evolution, content production,
human-machine interaction mode, and network resource organization. Their
investigation emphasized three dimensions: data assignment, model
intelligence, and spatial empowerment, exploring the technical
characteristics, elements, and developmental stages of AIGC. [43] introduced
the key aspects of AIGC technology, encompassing its historical development,
technological evolution, scenarios of application, and governance.
Additionally, [39] delved into the challenges posed by AIGC, such as
unemployment, income distribution, competition, intellectual property,
security, privacy, morals, ethics, energy, and the environment.

The second category involves application research, focusing on potential
fields where AIGC technology can be applied. For example, [16] employed large
language models like ChatGPT2 to train a language model capable of
automatically generating traditional Chinese ancient poetry. The model
performed well and successfully passed the Turing test. [38] developed a
comprehensive framework for a smart library system that integrates AIGC
technology across five dimensions: infrastructure, algorithm support, data
resources, business functions, and service applications. They explored the
primary challenges encountered during the iterative innovation of smart
libraries. [13] discussed the prospective applications and challenges of
generative artificial intelligence’s big models in the Metaverse life, using
medicine as an illustrative example. They examined concepts like digital
biological cells and establishing connections between digital cells and
neurons to enable the Metaverse to replicate the perception and biochemical
reactions of the physical world for medical advancements ([7]).

The third category revolves around media research, primarily investigating the
relationship between AIGC and its impact on media. [17] conducted a study
using BERT-DTM thematic analysis, social network analysis, and LIWC sentiment
analysis methods to uncover network public opinion trends surrounding the
phenomenal popularity of AIGC. The research revealed that public attention
toward AIGC revolved around six major themes: technological progress, event
marketing, digital art, market investment, virtual anchors, and prospects.
They identified key figures in the communication network triggered by AIGC as
well-known bloggers with expertise in the field. Furthermore, AIGC’s emergence
generated concerns about machine substitution for humans. [20] explored the
influence of AIGC technology on the news industry, highlighting challenges
such as false news dissemination and copyright disputes. They argued that the
operational logic of the news industry will undergo significant changes,
leading to industry restructuring and the birth of new news products and
formats, promoting more personalized news consumption. [44] classified AIGC
false information into factual and hallucinatory types based on their
generation mechanism and manifestation. Their analysis covered topics like
data error, author’s work error, objective fact error, programming code error,
machine translation error, false news events, false academic information,
false health information, prejudice, and discrimination. This comprehensive
examination provided a theoretical foundation for further research on AIGC
false information.

The fourth category pertains to copyright issues related to AIGC, which has
become an extensively researched topic in China. [35] investigated the nature
of content generated by artificial intelligence within copyright law. They
argued that AIGC content, being the outcome of algorithms, rules, and
templates, lacks the unique personality of human creators and cannot be
recognized as a work. [21] proposed including AIGC works within the framework
of related rights protection, presenting a solution that clarifies the
distinction between human works and artificial intelligence creations while
addressing the lack of legal protection for AIGC works. [12] identified three
challenges posed by AIGC technology to patent law. First, the issue of
patenting AIGC works is highly debated, but granting patent protection seems
to align with global trends. Second, defining AI patent subjects presents a
challenge due to the ”human inventor centrism” principle in current patent
laws. Future laws should acknowledge machine inventors as legitimate subjects.
Finally, authorizing AIGC patents is difficult, necessitating adjustments to
the conditions for patent authorization to facilitate better access to patents
for AIGC works.

##  4 Construction of AIGC Ecology

The advancement of AIGC has led to the appearance of the AIGC ecology. In
today’s dynamic world, the pursuit of ecological research on AIGC has become a
priority. This research can be categorized into upstream, midstream, and
downstream segments based on the product structure. Researchers are exploring
ways to efficiently allocate crucial components of AIGC’s ecological power,
thereby facilitating its rapid development. The inclusion of integrated
artificial intelligence, meta-universe, or decentralized Web 3.0 will result
in varying user experiences. Connecting AIGC to the network and then to smart
devices at home can potentially enhance our daily tool usage. This analysis
delves into the development history of AIGC ecology in China.

###  4.1 Upstream industry

Understanding the upstream industry of AIGC is crucial as it forms the
groundwork for enhancing the AIGC ecology and building the middle and
downstream sectors. The establishment of the upstream industry is a
fundamental part of the AIGC ecology and acts as a solid foundation for its
future development. This section will explore the key factors involved in
upstream molding, which is essential knowledge for comprehending the upstream
work. In this context, upstream refers to the front-end industry encompassing
AIGC software, hardware, personnel training, and data support.

####  4.1.1 Hardware aspect

Fast computing software is indispensable for AIGC and relies heavily on robust
hardware support. The development of ChatGPT, for instance, utilized the
cutting-edge A100 graphics card from NVIDIA, with a total of 10,000 units
employed. This generative AI software not only necessitates access to billions
of data points but also demands substantial computing power as its backbone.
However, it is important to note that this is just the initial phase of
ChatGPT’s development.

Implicit in the increasingly pervasive use of GPUs (graphics processing units)
lies their essentiality in AIGC. Originally designed for graphics processing,
GPUs have found extensive application in artificial intelligence acceleration.
GPU applications have proliferated in diverse areas such as generative AI,
large language models, autonomous driving systems, and machine networks. Upon
acquainting ourselves with ChatGPT, it becomes clear that Microsoft invested 1
billion dollars in constructing a supercomputer to power it—this laid the
necessary groundwork for computational and data processing.

Presently, the AIGC industry faces limitations in terms of hardware strength
and chip computing power. China’s chip sector is hindered by restrictions
imposed by the United States, resulting in limited access to chips,
consequently impeding the pace of AIGC industry development. Thankfully, major
enterprises have made significant breakthroughs in this field, and the country
is heavily investing in research and development, fostering optimism for the
improvement of China’s chip industry.

####  4.1.2 Personnel training

The tech industry unequivocally hinges on talent acquisition. As science and
technology advance, there is an increasing demand for skilled professionals.
The state places great value on AI talent training, offering relevant programs
and courses in both universities and research institutions. Scientific
research projects also actively promote talent cultivation in this field.
Enterprises too recognize the importance of talent acquisition as they vie for
a share in the AIGC industry, with notable examples such as Huawei, Ali,
Tencent, and Baidu.

Strengthening personnel training is not restricted to augmenting higher
education. As society progresses, many scientific research achievements emerge
through collaborative team efforts or organizational endeavors. Nobel Prize
laureates exemplify this phenomenon, as research complexity necessitates
teamwork. Therefore, nurturing comprehensive qualities beyond academic
training, such as social skills, communication skills, and critical thinking,
is imperative.

####  4.1.3 Data support

Data support encompasses various aspects including data analysis, data
processing, and data storage. The development of quality AI generation
software critically relies on robust data support. The rise of big data and
artificial intelligence has propelled the growth of the data processing
industry. For instance, in 2019, the market size of the data annotation
industry in China reached 3.09 billion yuan, and by 2020, it exceeded 3.6
billion yuan. Estimates anticipate the market size to surpass 10 billion yuan
by 2025.

Once data is processed, it must be stored in suitable mediums. The concept of
databases emerged in the 1970s and has since gained prominence. A database
entails organized storage of data within a computer system. Compared to
traditional flat files, databases offer benefits such as data security,
independence, recoverability, and centralized data management. While various
versions of database storage exist, they all revolve around the core concept
of a data model, which represents the organization of the database in a
computer-readable format.

Internationally, leading cloud data warehouse provider Snowflake witnessed a
67% revenue increase in the third quarter of 2022 compared to the same period
in the previous year, amounting to 550 million dollars. Domestically, the
market size of China’s big data platform surged to 5.42 billion yuan in the
first half of 2021, showing a growth rate of 43.5%. Huawei, Tencent Cloud, Ali
Cloud, Baidu, and Star Ring are among the major domestic big data enterprises
that have launched cloud-native data lake and cloud-native data platform
products. Moving ahead, the domestic data storage industry is poised to make
breakthroughs in areas like data management, data integration, and product
innovation, thereby providing robust data storage support for AIGC technology
([19]).

###  4.2 Midstream industry

During the midstream stage of AIGC technology development, particular emphasis
is placed on key factors such as cloud computing, big data, and machine
learning from data. This phase is critical for the establishment of AIGC
software, with China emerging as a global leader in industries such as cloud
computing and big data.

Regarding cloud computing, China initially lagged behind other countries but
has made significant progress in recent years. Following the national
strategy, localities have intensified efforts to develop the cloud computing
industry. Over 20 cities have designated cloud computing as a priority area
for development and have issued industrial development plans and action plans
accordingly. Examples include Beijing’s ”Xiangyun” plan, Shanghai’s ”Sea of
Clouds” plan, Shenzhen’s ”Kun Yun” plan, Chongqing’s ”Cloud” plan, Ningbo’s
”Nebula” plan, Wuxi’s ”Cloud Valley” plan, Suzhou’s ”Colorful Cloud” plan,
Harbin’s ”Cloud Flying” plan, Huizhou’s ”Huiyun” plan, Guangzhou’s ”Sky Cloud”
plan, Inner Mongolia’s ”Blue Sky and White Cloud” plan, et al. ([33]).

The construction of the computing industry’s ecological chain is becoming
increasingly refined. Under government supervision, cloud computing service
providers, along with hardware and software suppliers, network service
providers, cloud computing consulting and planning firms, delivery and
maintenance services, integration service providers, and terminal equipment
manufacturers, collectively form the industrial ecosystem of cloud computing.

Big data, characterized by its vast volume, is expanding exponentially. It is
estimated that approximately 2.3 trillion gigabytes of data are generated
daily, with this number only projected to rise. The proliferation of mobile
phone networks contributes significantly to this growth. For instance, six out
of the world’s seven billion people now own a mobile phone, resulting in a
surge in data volume from text and Whats App messages, photos, videos, and
numerous applications. As businesses expand rapidly, the demand for new
database management systems and IT professionals also escalates. The influx of
big data is expected to create millions of new IT jobs in the coming years.

At this stage, machine model learning encompasses data processing and hardware
capabilities, highlighting the significance of machine learning algorithms.
The manner in which data is input, processed, and transformed into valuable
and desirable content becomes crucial. While single CPU processing is
effective, it often proves insufficiently fast for content production,
necessitating the development of multi-threaded machine models. Currently, we
have achieved the desired level of progress in this area.

###  4.3 Downstream industry

Downstream applications primarily target end consumers and require a
heightened focus on software functionality and user experience. AIGC
technology serves as the foundation for developing application software in
various sectors, such as digital health, Internet of Things (IoT), and smart
living. Furthermore, as science and technology continue to advance, AIGC
applications will extend to industries like finance, healthcare, education,
and media.

####  4.3.1 Terminal product

Presently, our terminal products cater to different user groups. Consumer-
oriented products include Wenxin Yiyan, while enterprise-oriented products
include Huawei cloud Pangu large model. As AIGC progresses, it enables
effective cost reduction by minimizing labor and time expenses. AI also
enhances user experience by optimizing content generation based on user
behavior and feedback, leading to improved satisfaction levels.

In the field of digital medicine, a comprehensive system can be established
that encompasses online search, consultations, offline visits, prescriptions,
and medication processes, all accessible through personalized logins linked to
individuals’ real names. By leveraging global data comparisons of similar
medical cases, solutions can be provided. In the absence of identical cases,
attending physicians can collaborate with top international medical experts to
devise suitable treatment plans. Capitalizing on its technical advantages,
digital medicine helps address information asymmetry between doctors and
patients, streamlines the healthcare process, reduces costs, enhances the
medical experience, and improves disease diagnosis efficiency and patient
management. Telemedicine services facilitated through platforms or smart
devices enable more patients to access effective treatments, catering to
diverse patient needs and mitigating the issue of imbalanced medical
resources.

####  4.3.2 Internet of Things and Web 3.0

The Internet of Things (IoT) gained significant attention a few years ago, and
with the introduction of 5G technology, the concept of IoT has gained further
momentum. The idea behind IoT is to connect various objects to the network.
Chinese technology companies have also collaborated with home appliance
manufacturers to establish their own IoT ecosystems. For instance, Huawei’s
smart life ecosystem incorporates Huawei’s electronic products along with
other companies’ appliances such as air conditioners, refrigerators, and TVs.
Additionally, the cost-effective Mijia ecosystem offers a diverse range of
products including mobile phones, tablets, watches, air fryers, refrigerators,
and washing machines. Our focus extends beyond the mobile phone industry to
embrace the concept of an ecosystem, which aligns well with the AIGC
ecosystem. AIGC technology can be integrated into the IoT ecosystem and
customized for each household based on their unique lifestyle and work
schedules, thereby enhancing consumers’ quality of life ([9, 15]).

AIGC technology is closely intertwined with the metaverse and Web 3.0. It
facilitates the rapid generation of rich and diverse content for the metaverse
while enabling the development of intelligent non-player characters (NPCs) and
smart contracts. Moreover, AIGC assists decentralized applications in
generating and processing data, making automated decisions, and providing
robust support for the advancement of the metaverse and Web 3.0. This
integration enhances the realism and interest of virtual environments, and
promotes decentralization and intelligence within Web 3.0 ([11]).

With the rapid development of AIGC, a multitude of downstream applications
have emerged. AIGC is poised to become the cornerstone of internet content
infrastructure, enhancing communication and enabling more efficient access to
information ([8]). Currently, AIGC technology has begun penetrating highly
digitized industries such as media, e-commerce, and film and television,
yielding significant developmental achievements. As the market expands,
several major search engines have already incorporated AIGC software. For
instance, Microsoft’s Bing has integrated ChatGPT, while Baidu’s Wenxin Yi
plans to connect it with Baidu search in the near future, enabling users to
seamlessly leverage AIGC technology. The integration of AIGC and the metaverse
has been proposed as engineers tackle the monumental task of building
metaverse projects, which require extensive time and data resources. AIGC
offers a viable solution by automatically generating characters and scenes
based on inputted data, meeting our specific requirements. Web 3.0 allows for
local data storage, safeguarding consumer rights and mitigating concerns such
as power outages or data loss at centralized data centers. Our data remains in
our possession, and even if issues arise, AIGC technology can still provide
assistance, greatly facilitating our daily lives ([14]).

##  5 Current challenges

Currently, AIGC faces various external challenges. The first challenge
pertains to consumer mindset, as some individuals may have reservations about
accepting AIGC as substitutes for human labor. This issue can stem from
conflicts of interest between companies and their employees, with concerns
regarding potential job displacement by AI. Prominent figures in Silicon
Valley have expressed opposition to the development of artificial
intelligence, highlighting valid concerns considering the uncertainty
surrounding the source and future trajectory of AI advancements. Government
intervention is crucial in establishing regulations and facilitating smooth
business operations in this regard.

Technical limitations represent another challenge in the field of AIGC. Robust
AIGC technology necessitates substantial amounts of data, efficient
algorithms, and powerful computing capabilities. While notable progress has
been made in certain areas, many domains still exhibit technical constraints
in terms of algorithm accuracy and efficiency, particularly in image
recognition, natural language processing, and machine learning. Addressing
these technical challenges requires ongoing efforts from scientists to develop
more efficient algorithms and enhance computational power.

Funding poses a significant challenge to the advancement of AIGC technology.
Extensive financial support is required, especially during the research and
development phase. However, many AIGC companies currently face funding
shortages, which may impede their ability to fully develop and commercialize
their technologies. To overcome this obstacle, governments and private
investors are actively working to provide increased financial backing for
initiatives. Moreover, some AIGC companies seek additional funding through
self-financing or strategic partnerships with other organizations.

A shortage of talent represents yet another challenge for AIGC technology. The
development of AIGC relies on a substantial number of highly skilled
professionals, including data scientists, algorithm engineers, and computer
scientists. Unfortunately, there is a global scarcity of AIGC talent. To
mitigate this issue, many AIGC companies are striving to attract and nurture
top-tier talent. Additionally, governments and educational institutions are
making concerted efforts to cultivate a greater pool of AIGC professionals.

Content generated by AIGC encompasses diverse formats such as music, images,
videos, and articles. The automated creation of such content raises concerns
related to copyright infringement and attribution. Different countries have
introduced varying legal frameworks to regulate the content produced by AIGC,
aiming to address copyright issues and clarify ownership rights, as figure 4
gives out a clear framework.

Figure 4: AIGC Legislation from different countries

As time goes, these challenges can be overcome through ongoing research and
technological advancements. Looking ahead, the prospect of AIGC holds great
promise. The focus lies in the continued development of AIGC technology,
extending beyond mere verbal responses. In terms of future trends, the
advancement of AIGC is expected to manifest in several key aspects. Firstly,
there will be an increased emphasis on enhanced intelligence, enabling AIGC
systems to independently process data, learn, and make decisions autonomously.
Secondly, cloud computing will play a crucial role by providing greater
computational and storage capacities to support AIGC applications. This higher
degree of cloud integration will empower AIGC with more robust capabilities.
Lastly, AIGC will expand its reach into a wider array of industries, offering
intelligent solutions tailored to the specific needs of various sectors. This
diversification will deepen the integration of AIGC across different domains,
fostering intelligent transformations in multiple industries.

##  6 Conclusion

AIGC is still in its early stages of development and has enormous potential
for development. With the development of science and technology, AIGC will
make breakthroughs in the following two areas in the future. We drew 6 points
to conclude the future development of AIGC.

Primarily, AI plays a crucial role in pre-processing and enhancing data. This
involves employing techniques such as data cleaning and data enhancement to
improve the quality and richness of the data. By mitigating the impact of data
deviations on model training, AI assists in obtaining the necessary data for
analysis and decision-making.

Secondly, it is imperative to construct interpretable models by incorporating
approaches like logistic regression and decision trees. These interpretable
machine learning models are designed to meet the diverse requirements of
different application scenarios. By emphasizing interpretability, the aim is
to facilitate wider adoption of AIGC technology, enabling more enterprises to
benefit from its capabilities.

Next, enhancing the generalization ability of models is crucial. Techniques
like regularization and comprehensive learning are employed to address
overfitting issues, thereby improving the model’s ability to generalize and
increasing its reliability for practical applications.

Leveraging distributed computing resources, such as cloud computing and GPU,
is another significant aspect. By harnessing these resources, the efficiency
of model training and reasoning can be enhanced while reducing the
computational resource costs associated with AIGC.

The fusion of multi-modal data is of great significance. Incorporating various
types of data, including images, text, and speech, enables cross-modal
information fusion and interaction. This approach amplifies the performance of
the model and enhances its application value by leveraging complementary
information from different modalities.

Lastly, ensuring privacy and security during model training is paramount.
Safeguarding user and enterprise data from leakage and misuse requires
stringent measures to protect against potential breaches or unauthorized
access.

Through this approach, China can shape a distinctive model for AIGC
development with supervision and legislation that better safeguards user
safety and privacy and maintains national security. As such, it is arguable
that China possesses relative strategic advantages given its unique
geopolitical position, significant technological capacity, and ambitions
toward technological leadership. Overall, exploring avenues for effective
governance frameworks that balance innovation and business opportunities with
sustainable risk management will be critical to successfully realizing AIGC’s
potential.

## References

  * [1] Christian Arzate Cruz and Jorge Adolfo Ramirez Uresti  “Player-centered game AI from a flow perspective: Towards a better understanding of past trends and future directions”  In _Entertainment Computing_ 20, 2017, pp. 11–24
  * [2] Li Baiyang, Bai Yun, Zhan Xini and Gang Li  “The Technical Features and Aromorphosis of Artificial Intelligence Generated Content (AIGC)”  In _Documentation,Information and Knowledge_ 40.66-74, 2023
  * [3] Tadas Baltrušaitis, Chaitanya Ahuja and Louis-Philippe Morency  “Multimodal machine learning: A survey and taxonomy”  In _IEEE transactions on pattern analysis and machine intelligence_ 41.2 IEEE, 2018, pp. 423–443
  * [4] Tom Brown et al.  “Language models are few-shot learners”  In _Advances in neural information processing systems_ 33, 2020, pp. 1877–1901
  * [5] CACIT  “Report on the Development of Regional and Urban Digital Economy in China”, 2020  URL: https://sh.cctv.com/2020/12/26/ARTIUH3Xl71ijllAulJp5Qvy201226.shtml
  * [6] Yihan Cao et al.  “A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT”, 2023  arXiv:2303.04226 [cs.AI]
  * [7] Shenghui Cheng  “Metaverse: Concept, Content and Context”  Springer, 2023
  * [8] Shenghui Cheng and Klaus Mueller  “The Data Context Map: Fusing Data and Attributes into a Unified Display”  In _IEEE Transactions on Visualization and Computer Graphics_ 22.1, 2016, pp. 121–130
  * [9] Shenghui Cheng et al.  “Roadmap toward the metaverse: An AI perspective”  In _The Innovation_ 3.5 Elsevier, 2022, pp. 100293
  * [10] Jialai Fan, Yang Yang and Yan Sun  “Here comes the first regulatory document! Where does the AIGC management method seek opinions point to? What signals are released?”, 2023  URL: https://www.thepaper.cn/newsDetail_forward_22669779
  * [11] Yuqing Fan, Tianyi Huang, Yiran Meng and Shenghui Cheng  “The current opportunities and challenges of Web 3.0”, 2023
  * [12] Wu Handong  “The Patent Law of Artificial Intelligence Generation Invention”  In _Contemporary Law Review_ 33.24-38, 2019
  * [13] Wang Hao, Pan Yushan and Pan Yi  “Generative AI empowered metaverse organisms: prospects and challenges”  In _Big data research_ , 2023
  * [14] Tianyi Huang, Shenghui Cheng, Stan Z Li and Zhengjun Zhang  “High-dimensional Clustering onto Hamiltonian Cycle”  In _ICML_ , 2023
  * [15] Tianyi Huang, Stan Z Li, Xin Yuan and Shenghui Cheng  “Roadmap towards Meta-being”  In _arXiv preprint arXiv:2303.06795_ , 2023
  * [16] Liu Jaingfeng et al.  “A Practical Exploration of AIGC-Powered Digital Humanities research: A SikuGPT Driven research of Ancient Poetry Generation”  In _Information studies: Theory and Application_ 46.23-31, 2023
  * [17] Wu Jiang et al.  “Propagation and evolution of public opinion in the outbreak of AIGC based on the theory of tipping point”  In _Journal of modern information_ , 2023
  * [18] Kimin Lee et al.  “Aligning Text-to-Image Models using Human Feedback”  In _arXiv preprint arXiv:2302.12192_ , 2023
  * [19] Peijuan Li  “Big data white book 2022”, 2023  URL: https://www.qianzhan.com/analyst/detail/220/210506-862b1b9e.html
  * [20] Zheng Manning  “The Journalism Industry under Artificial Intelligence Technology: Evolution, Transformation, and Response - New Thinking Based on ChatGPT”  In _Chinese Editors Journal_ No.160.35-40, 2023
  * [21] Xu Mingyue and Tan Lin  “On the Protection of Related rights of Artificial Intelligence Creations - Theoretical Verification and Institutional Arrangement”  In _Journal of Comparative Law_ No.160.42-54, 2018
  * [22] Huajing Intelligence Network  “Analysis of relevant policies, social environment, and market size of China’s AIGC industry in 2023”, 2023  URL: https://www.sohu.com/a/676411612_120113054
  * [23] Nightcafe  “About NightCafe Studio”, 2023  URL: https://nightcafe.studio/pages/about-nightcafe
  * [24] OpenAI  “DALL·E: Creating images from text”, 2021  URL: https://openai.com/research/dall-e
  * [25] Long Ouyang et al.  “Training language models to follow instructions with human feedback”  In _arXiv:2203.02155_ , 2022
  * [26] Wei Ping et al.  “Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning”, 2018  arXiv:1710.07654 [cs.SD]
  * [27] People’s Post and Telegraph  “Release of the White Paper on the Development of China’s Digital Economy:In 2020, the scale of China’s digital economy reached 39.2 trillion yuan”, 2021  URL: https://www.cnii.com.cn/rmydb/202105/t20210506_274871.html
  * [28] Alec Radford et al.  “Learning Transferable Visual Models From Natural Language Supervision”  In _CoRR_ abs/2103.00020, 2021  arXiv: https://arxiv.org/abs/2103.00020
  * [29] Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever  “Improving language understanding by generative pre-training”  OpenAI, 2018
  * [30] Robin Rombach et al.  “High-Resolution Image Synthesis With Latent Diffusion Models”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 10684–10695
  * [31] Justyna Sarzynska-Wawer et al.  “Detecting formal thought disorder by deep contextualized word representations”  In _Psychiatry Research_ 304 Elsevier, 2021, pp. 114135
  * [32] John Schulman et al.  “Proximal policy optimization algorithms”  In _arXiv preprint arXiv:1707.06347_ , 2017
  * [33] Liu Tiantian, Zhang Qing, Yue Qiang and Li Junjie  “Cloud computing industry development status and trend analysis”  In _Guangdong communication Technology_ 35.6-12, 2015
  * [34] Ashish Vaswani et al.  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [35] Qian Wang  “On the Qualitative Analysis of Content Generated by Artificial Intelligence in Copyright Law”  In _Science of Law(Journal of Northwest University of Political Science and Law)_ 35.148-155, 2017
  * [36] Jason Wei et al.  “Finetuned language models are zero-shot learners”  In _arXiv preprint arXiv:2109.01652_ , 2021
  * [37] Jiayang Wu et al.  “AI-Generated Content (AIGC): A Survey”, 2023  arXiv:2304.06632 [cs.AI]
  * [38] Zhao Yang, Zhang Xue and Fan Shengyue  “AIGC-driven Intelligent Library Transformation:Framework,Pathways and Challenges”  In _Information studies: Theory and Application_ , 2023
  * [39] Chen Yongwei  “Beyond ChatGPT:Opportunities,Risks,and Challenges from Generative AI”  In _Journal of Shandong university_ No.258.127-143, 2023
  * [40] Zhai You and Li Juan  “AIGC development path thinking: The popularization of large model tool ushered in new opportunities”  In _Internet world_ , 2022, pp. 6
  * [41] Chaoning Zhang et al.  “A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?”, 2023  arXiv:2303.11717 [cs.AI]
  * [42] Du Zhendong and Tuming  “Generative AI:Natural Language process and Human-Computer interaction”  Beijing: China Machine Process, 2020
  * [43] Cai Zifan and Wei Haiyan  “Evolution of Artificial Intelligence Generated Content (AIGC) and Its Application Scenario of Library Intelligent Service”  In _Library Journal_ 42.34-43+135-136, 2023
  * [44] Mo Zuying, Pan Daqing, Liu Huan and Zhao Yueming  “Analysis on AIGC False Information Problem and Root Cause from the Perspective of Information Quality”  In _Documentation, Information and Knowledge_ , 2023


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

CLEF 2023: Conference and Labs of the Evaluation Forum, September 18–21, 2023,
Thessaloniki, Greece

[orcid=0009-0008-4246-1936<EMAIL_ADDRESS>] [1]

[1]Corresponding author.

# Keeping in Time: Adding Temporal Context to Sentiment Analysis Models

Dean Ninalga  Toronto, Canada

(2023)

# Notebook for the LongEval Lab at CLEF 2023

Dean Ninalga  Toronto, Canada

(2023)

###### Abstract

This paper presents a state-of-the-art solution to the LongEval CLEF 2023 Lab
Task 2: _LongEval-Classification_ [1]. The goal of this task is to improve and
preserve the performance of sentiment analysis models across shorter and
longer time periods. Our framework feeds _date-prefixed_ textual inputs to a
pre-trained language model, where the timestamp is included in the text. We
show _date-prefixed_ samples better conditions model outputs on the temporal
context of the respective texts. Moreover, we further boost performance by
performing self-labeling on unlabeled data to train a student model. We
augment the self-labeling process using a novel augmentation strategy
leveraging the _date-prefixed_ formatting of our samples. We demonstrate
concrete performance gains on the LongEval-Classification [1] evaluation set
over non-augmented self-labeling. Our framework achieves a 2nd place ranking
with an overall score of 0.6923 and reports the best _Relative Performance
Drop_ (RPD) [2] of -0.0656 over the short evaluation set (see Alkhalifa et al.
[3]).

###### keywords:

Self-Labeling Sentiment Analysis Temporal Misalignment Date-Prefixing

##  1 Introduction

The application of language models such as BERT [4], RoBERTa [5] and XLM-
RoBERTa [6] to textual data is a core component in many natural language
processing (NLP) pipelines. However, a notable limitation of most language
models is their lack of temporal awareness, as they typically encode text into
fixed representations. Conversely, the nature of textual data is inherently
dynamic and subject to change over time. Where traditional meanings of words,
phrases, and concepts are constantly evolving [7, 8]. Furthermore, significant
events can alter the factual basis of the text [9]. Although metadata of well-
known text corpora includes timestamps, timestamps are almost never used
within many NLP pipelines. A sentiment analysis model trained today could
interpret the phrase: "You are just like X" as a positive sentiment. However,
an issue can arise once people consider a comparison to ‘X’ as a non-positive
comparison. Subsequently, the model becomes _misaligned_ if this flip in
public opinion occurs. Hence, it can be difficult to train models that can
generalize to future data without a sense of temporal context and awareness
[10].

Mitigating _temporal misalignment_ [11] between the facts and general
sentiments of the current world and those found in text corpora is an active
area of focus in various areas of research in nlp. In particular, work in NER
(named-entity-recognition) [12, 9, 13] and question-and-answering [14, 15, 12,
16] often directly address temporal misalignment as they are considered
_knowledge-intensive_ tasks [10].

A common and straightforward way to address temporal misalignment in textual
data is to create new models (or update old ones) with the most recent data
available [17, 18, 10]. However, continually growing datasets incur an
increase in computational costs for data acquisition and training models which
also contributes to an ever-increasing environmental cost [19, 20]. Therefore,
finding a solution outside of continuous retraining that preserves model
performance over time is desirable.

In this paper, we follow Dhingra et al. [7] who use an alternative approach
that modifies the textual input with its timestamp. Thus, we can take
advantage of text-only pre-trained language models used for classification in
addition to conditioning the models with the temporal context for the input.

We will outline our system, which is aligned with some of the recent works in
NER and temporal misalignment, and evaluate it on the _LongEval-
Classification_ benchmark [1].

Our contribution is two-fold: (1) We show that date-prefixing the input text
with its timestamp conditions the outputs of a language model on the temporal
context of the input. (2) We utilize an augmentation strategy that leverages
the date-prefixing by randomly modifying the timestamp of unlabeled inputs. We
show that this augmentation strategy improves the performance benefits of
semi-supervised learning on unlabeled data.

##  2 Background and Related Work

Recently, _TempLama_ [7] showed that directly placing the year of the
timestamp as a prefix in the text is performative in the context of named-
entity-recognition. They, then feed the date-prefixed inputs to a T5 [21]
model to directly model the temporal context. Cao and Wang [22] directly
compares a date-prefixing approach to an embedding approach where the date is
numerically embedded with a linear projection. [22] in the context of text
generation, found that linear projection was less sensitive to the timestamps
while date-prefixing is better at generating more temporally sensitive facts.

Self-labeling (or self-distillation) is a semi-supervised learning strategy
that typically involves learning from pseudo-labels for unlabeled data. Self-
labeling is demonstrated to add performance gains across a variety of domains
including text classification [23]. Agarwal and Nenkova [9] found that self-
labeling performs better than specialized pre-training objectives such as
domain-adaptive pretraining [24] across several tasks including sentiment
analysis. However, it is important to note that recently Ushio et al. [25]
have shown that self-labeling, as presented in [9], is not as effective for
NER when compared to models trained for specific time periods.

##  3 Methodology

Figure 1: Method Overview: (top-row) summarization of our semi-supervised
learning training pipeline stages, (bottom-row): modifications we made to the
pipeline and at what stage they apply

Figure 1 provides an overview of our system. Following Agarwal and Nenkova
[9], we first train a teacher model on the full labeled dataset to create
pseudo-labels for the unlabeled data. During this training phase, every sample
in the labeled dataset is date-prefixed, meaning that the year of the
timestamp is included as part of the input text. We use a novel augmentation
strategy on the date prefixes (see Section section 3.3) to condition the
pseudo-labels on the temporal context learned by the teacher. A new student
model is then trained for 22000 training steps on the generated pseudo-labels
and is subsequently trained on the original labeled data that was used for the
teacher. Finally, we use the resulting student model for inference. For
simplicity, both the teacher and student models share the same architecture.
We provide further detail on the individual components of our system in the
following sections.

###  3.1 Pre-Trained Model

Using a pre-trained language is generally much better than training a new
model from scratch. However, it is not always clear which pre-training works
best for any particular task. Here we use Bernice [26] a variant of XLM-
RoBERTa [6] specialized for Twitter data. We train a single model for
inference on the test set and we do not rely on ensembling techniques. We
train using the cross-entropy classification loss.

###  3.2 Date-Prefixing

Consistent with Dhingra et al. [7] we prefix each input text with the year of
the given time-stamp followed by the text itself (e.g. “year: 2023 text: I
really do enjoy drinks with friends”). As we observe from Table 1 training on
this data conditions the model outputs with the temporal context found in the
data using date-prefixing. Table 1 provides real input and output examples
based on a trained model across various years. We do not modify the
architecture of the language model to take the timestamp as a vector input. By
maintaining the use of textual-only input we are able to leverage any existing
pre-trained models that have text-embedding only input.

###  3.3 Date-Prefix Augmentation

When creating pseudo-labels to train a student model we use an augmentation
strategy that takes advantage of our date-prefixing. Namely, given an
unlabeled sample and its timestamp we randomly replace the year in the
timestamp with a year between 2013 and 2021. Where, the years 2013 and 2021
are the earliest and latest years found in the labeled datasets, respectively.
We perform an ablation experiment (see Section 4) demonstrating that this
augmentation strategy outperforms non-augmented self-labeling on the
evaluation set.

###  3.4 Training and Evaluation

We use a single model trained using both the training and development sets for
two epochs for inference on the test set. Model parameters using the Adam
optimizer [27] with a constant learning rate of 1e-5 using the binary-cross-
entropy loss. Performance is measured using the macro-averaged F1 score of the
future samples.

##  4 Experiments

Table 1: Date Prompt Conditioning: A demonstration of the date-prompting and subsequent model outputs conditioned on the prefix year. The model output is between 0 and 1, where the input is considered positive only if the output is above 0.5. The example input text is taken from the _LongEval-Classification_ dataset [1]. Example input | Output | Label | Orginal Year | Prefix Year
---|---|---|---|---
"year: 2013 text: I really do enjoy being single" | $0.503$ | positive | 2018 | 2017
"year: 2018 text: I really do enjoy being single" | $0.510$ | positive | 2018 | 2018
"year: 2023 text: I really do enjoy being single" | $0.495$ | negatuve | 2018 | 2023

###  4.1 Experimental Setup

In this section, we will compare the performance of models trained with and
without the proposed augmentation strategies for pseudo-label generation.
Namely, we will use a trained teacher model to generate labels with and
without date-prefix augmentation. Subsequently, we a student models on each of
the two sets of pseudo labels for 6000 training steps. Finally, then compare
the downstream performance of each model.

Models will only be provided labels for the training set and trained until
saturation on the interim evaluation set. For our experiments, we report the
macro-averaged F1 scores for each subset of the evaluation set. We will also
report the Relative Performance Drop (RPD) [2] for comparison between short
and long-term time differences with respect to model performance.

$\text{RPD}=\frac{f_{t_{j}}^{score}-f_{t_{0}}^{score}}{f_{t_{0}}^{score}}$ (1)

###  4.2 Results

We report the evaluation results of our experiments in Table 2. Indeed, we see
an overall improvement in performance especially when we observe the ‘short’
evaluation set results when using our full framework. Additionally, the model
using date-prefix augmentation gives by far the best RDP of $-0.0532$ with
respect to the ‘within‘ and ‘short’ evaluation sets. Note that the non-
augmented models gives the best RDP of $-0.0411$ with respect to the ‘within‘
and ‘long‘ evaluation sets. However, when finetunning this same model on the
gold labels, the RPD more than doubles to $-0.0852$ and is much worse than our
full framework with $-0.0681$. A similar drop in performance can be seen when
observing the F1 score on the ‘long’ evaluation set. It appears that fine-
tuning the non-augmented model with clean data incurs a significant drop in
performance. However, it is clear that our proposed augmentation strategy can
leverage the older labeled data and attain significant performance gains.

##  5 Conclusion

In this paper, we introduce a competitive framework for preserving the
performance of sentiment analysis models across various temporal periods. We
promote date-prefixing, as a straightforward solution to condition the output
of pre-trained language models with the temporal context of input text.
Furthermore, we build on the self-labeling framework developed by Agarwal and
Nenkova [9]. Namely, given our date-prefix formatting, we can generate pseudo-
labels conditioned on the temporal context of the input text. We verify the
performance gains of our proposed system against self-labeling without our
augmentation strategy in our ablation experiments. Altogether, our system
yields competitive performance in overall score and attains the best RPD for
the short evaluation set [3].

Table 2: Abalation Results: Results on the evaluation set, testing our self-labeling augmentation strategy. (_baseline_ : only using gold labels, _+sl_ : trained on pseudo-labels generated from _baseline_ , _+ft_ : fine-tuned on gold labels, _(aug)_ : date-prefix augmentation, _(no-aug)_ : no augmentation applied) We report the macro F1 score alongside the RPD between the various evaluation sets. The best results are highlighted Method | F1 Within | F1 Short | F1 Long | RPD Within-Short | RPD Within-Long
---|---|---|---|---|---
_baseline_ | 0.7266 | 0.6725 | 0.6595 | -0.0744 | -0.0924
_+sl(no-aug)_ | 0.7213 | 0.6747 | 0.6916 | -0.0646 | -0.0411
_+sl(no-aug)_ +ft | 0.7355 | 0.6728 | 0.6728 | -0.0852 | -0.0852
_+sl(aug)_ | 0.7278 | 0.6749 | 0.6648 | -0.0727 | -0.0865
_+sl(aug)+ft (ours)_ | 0.7210 | 0.6833 | 0.6719 | -0.0532 | -0.0681

## References

  * Alkhalifa et al. [2023] R. Alkhalifa, I. Bilal, H. Borkakoty, J. Camacho-Collados, R. Deveaud, A. El-Ebshihy, L. Espinosa-Anke, G. Gonzalez-Saez, P. Galuščáková, L. Goeuriot, E. Kochkina, M. Liakata, D. Loureiro, H. Tayyar Madabushi, P. Mulhem, F. Piroi, M. Popel, C. Servan, A. Zubiaga,  Longeval: Longitudinal evaluation of model performance at clef 2023,  in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval, Springer Nature Switzerland, Cham, 2023.
  * Alkhalifa et al. [2021] R. Alkhalifa, E. Kochkina, A. Zubiaga,  Opinions are made to be changed: Temporally adaptive stance classification,  Proceedings of the 2021 Workshop on Open Challenges in Online Social Networks (2021).
  * Alkhalifa et al. [2023] R. Alkhalifa, I. Bilal, H. Borkakoty, J. Camacho-Collados, R. Deveaud, A. El-Ebshihy, L. Espinosa-Anke, G. Gonzalez-Saez, P. Galuščáková, L. Goeuriot, E. Kochkina, M. Liakata, D. Loureiro, H. T. Madabushi, P. Mulhem, F. Piroi, M. Popel, C. Servan, A. Zubiaga,  Overview of the clef-2023 longeval lab on longitudinal evaluation of model performance,  in: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fourteenth International Conference of the CLEF Association (CLEF 2023), Lecture Notes in Computer Science (LNCS), Springer, Thessaliniki, Greece, 2023.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  Bert: Pre-training of deep bidirectional transformers for language understanding,  ArXiv abs/1810.04805 (2019).
  * Liu et al. [2019] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov,  Roberta: A robustly optimized bert pretraining approach,  ArXiv abs/1907.11692 (2019).
  * Conneau et al. [2019] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov,  Unsupervised cross-lingual representation learning at scale,  in: Annual Meeting of the Association for Computational Linguistics, 2019.
  * Dhingra et al. [2021] B. Dhingra, J. R. Cole, J. M. Eisenschlos, D. Gillick, J. Eisenstein, W. W. Cohen,  Time-aware language models as temporal knowledge bases,  Transactions of the Association for Computational Linguistics 10 (2021) 257–273.
  * Margatina et al. [2023] K. Margatina, S. Wang, Y. Vyas, N. A. John, Y. Benajiba, M. Ballesteros,  Dynamic benchmarking of masked language models on temporal concept drift with multiple views,  in: Conference of the European Chapter of the Association for Computational Linguistics, 2023.
  * Agarwal and Nenkova [2021] O. Agarwal, A. Nenkova,  Temporal effects on pre-trained models for language processing tasks,  Transactions of the Association for Computational Linguistics 10 (2021) 904–921.
  * Lazaridou et al. [2021] A. Lazaridou, A. Kuncoro, E. Gribovskaya, D. Agrawal, A. Liska, T. Terzi, M. Gimenez, C. de Masson d’Autume, T. Kociský, S. Ruder, D. Yogatama, K. Cao, S. Young, P. Blunsom,  Mind the gap: Assessing temporal generalization in neural language models,  in: Neural Information Processing Systems, 2021\.
  * Luu et al. [2021] K. Luu, D. Khashabi, S. Gururangan, K. Mandyam, N. A. Smith,  Time waits for no one! analysis and challenges of temporal misalignment,  in: North American Chapter of the Association for Computational Linguistics, 2021.
  * Pustejovsky et al. [2005] J. Pustejovsky, R. Knippen, J. Littman, R. Saurí,  Temporal and event information in natural language text,  Language Resources and Evaluation 39 (2005) 123–164.
  * Zhang and Choi [2023] M. J. Zhang, E. Choi,  Mitigating temporal misalignment by discarding outdated facts,  2023\.
  * Jang et al. [2021] J. Jang, S. Ye, S. Yang, J. Shin, J. Han, G. Kim, S. J. Choi, M. Seo,  Towards continual knowledge learning of language models,  ArXiv abs/2110.03215 (2021).
  * Chen et al. [2017] D. Chen, A. Fisch, J. Weston, A. Bordes,  Reading wikipedia to answer open-domain questions,  in: Annual Meeting of the Association for Computational Linguistics, 2017.
  * Livska et al. [2022] A. Livska, T. Kovcisk’y, E. Gribovskaya, T. Terzi, E. Sezener, D. Agrawal, C. de Masson d’Autume, T. Scholtes, M. Zaheer, S. Young, E. Gilsenan-McMahon, S. Austin, P. Blunsom, A. Lazaridou,  Streamingqa: A benchmark for adaptation to new knowledge over time in question answering models,  in: International Conference on Machine Learning, 2022.
  * Loureiro et al. [2022] D. Loureiro, F. Barbieri, L. Neves, L. E. Anke, J. Camacho-Collados,  Timelms: Diachronic language models from twitter,  in: Annual Meeting of the Association for Computational Linguistics, 2022.
  * Jang et al. [2022] J. Jang, S. Ye, C. K. Lee, S. Yang, J. Shin, J. Han, G. Kim, M. Seo,  Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models,  in: Conference on Empirical Methods in Natural Language Processing, 2022.
  * Strubell et al. [2019] E. Strubell, A. Ganesh, A. McCallum,  Energy and policy considerations for deep learning in nlp,  ArXiv abs/1906.02243 (2019).
  * Attanasio et al. [2022] G. Attanasio, D. Nozza, F. Bianchi, D. Hovy,  Is it worth the (environmental) cost? limited evidence for temporal adaptation via continuous training,  2022\.
  * Raffel et al. [2019] C. Raffel, N. M. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu,  Exploring the limits of transfer learning with a unified text-to-text transformer,  ArXiv abs/1910.10683 (2019).
  * Cao and Wang [2022] S. Cao, L. Wang,  Time-aware prompting for text generation,  in: Conference on Empirical Methods in Natural Language Processing, 2022.
  * Shams [2014] R. Shams,  Semi-supervised classification for natural language processing,  ArXiv abs/1409.7612 (2014).
  * Gururangan et al. [2020] S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, N. A. Smith,  Don’t stop pretraining: Adapt language models to domains and tasks,  ArXiv abs/2004.10964 (2020).
  * Ushio et al. [2022] A. Ushio, L. Neves, V. Silva, F. Barbieri, J. Camacho-Collados,  Named entity recognition in twitter: A dataset and analysis on short-term temporal shifts,  in: AACL, 2022.
  * DeLucia et al. [2022] A. DeLucia, S. Wu, A. Mueller, C. A. Aguirre, P. Resnik, M. Dredze,  Bernice: A multilingual pre-trained encoder for twitter,  in: Conference on Empirical Methods in Natural Language Processing, 2022.
  * Kingma and Ba [2014] D. P. Kingma, J. Ba,  Adam: A method for stochastic optimization,  CoRR abs/1412.6980 (2014).