# Drive as You Speak: Enabling Human-Like Interaction with Large Language
Models in Autonomous Vehicles

Can Cui1,∗, Yunsheng Ma1, Xu Cao2, Wenqian Ye2,3, Ziran Wang1
1 Purdue University, West Lafayette, IN, USA, 47907
2 PediaMed.AI Lab, Shenzhen, China, 518048
3 University of Virginia, Charlottesville, VA, USA, 22903
<EMAIL_ADDRESS>
###### Abstract

The future of autonomous vehicles lies in the convergence of human-centric
design and advanced AI capabilities. Autonomous vehicles of the future will
not only transport passengers but also interact and adapt to their desires,
making the journey comfortable, efficient, and pleasant. In this paper, we
present a novel framework that leverages Large Language Models (LLMs) to
enhance autonomous vehicles’ decision-making processes. By integrating LLMs’
natural language capabilities and contextual understanding, specialized tools
usage, synergizing reasoning, and acting with various modules on autonomous
vehicles, this framework aims to seamlessly integrate the advanced language
and reasoning capabilities of LLMs into autonomous vehicles. The proposed
framework holds the potential to revolutionize the way autonomous vehicles
operate, offering personalized assistance, continuous learning, and
transparent decision-making, ultimately contributing to safer and more
efficient autonomous driving technologies.

##  1 Introduction

Recently, Large Language Models (LLMs) have attracted significant attention.
The key to their success lies in their remarkable ability to process a wide
range of word-based inputs, including prompts, questions, dialogues, and
vocabulary spanning diverse domains, resulting in significant and coherent
textual outputs. LLMs serve as vast storehouses of abundant information and
knowledge acquired from numerous texts, much like the human brain. Considering
the LLM’s ability to emulate the human brain functions, it prompts us to ask:
could we leverage the impressive capabilities of LLMs to revolutionize the
future of autonomous driving?

Imagine a situation where you’re sitting in an autonomous vehicle and you
desire to safely overtake another vehicle. All you have to do is utter the
command: “Overtake the vehicle in front of me.” At that point, the LLMs would
swiftly assess the existing conditions and safety listen, and ask questions
before reasoning, providing you with informed guidance on the feasibility and
recommended actions for executing the maneuver. Furthermore, in the context of
fully autonomous vehicles, the LLMs’ capabilities could even extend to taking
charge of the vehicle and executing the instructed commands.

While LLMs have the potential to greatly enhance convenience and improve the
driving experience for drivers, a significant challenge arises: LLMs lack
understanding of information about the driving environment. Unlike humans,
LLMs lack the inherent ability to perceive the physical environment. In other
words, these models do not possess the capability to visually perceive and
interact with the world around them [2]. It renders LLMs challenged in making
sound decisions for the current situation, potentially leading to suboptimal
outcomes or even hazardous consequences.

To address the challenge above, we present a perspective where LLMs can serve
as the decision-making “brain” within autonomous vehicles. Complementing this,
various tools within the autonomous vehicle ecosystem, including the
perception module, localization module, and in-cabin monitor, function as the
vehicle’s sensory “eyes.” This configuration enables LLMs to overcome the
inherent limitation of not directly accessing real-time environmental
information. By receiving processed data from the perception module, LLMs can
facilitate informed decision-making, resulting in significant enhancements to
the performance of the autonomous vehicle. Additionally, the vehicle’s actions
and controller function as its “hands,” executing instructions derived from
the LLM’s decision-making process.

When comparing autonomous vehicles with and without integrated LLMs, it
becomes evident that the latter offers a diverse array of compelling
advantages. These advantages extend across various aspects of functionality
and performance:

  * •

Language Interaction: LLMs enable intuitive communication between drivers and
vehicles, transforming interactions from rigid commands to natural
conversations.

  * •

Contextual Understanding and Reasoning: LLMs in vehicles offer enhanced
contextual understanding from diverse sources like traffic laws and accident
reports, ensuring decisions prioritize safety and regulation adherence.

  * •

Zero-Shot Planning: LLMs in vehicles can understand and reason about
unfamiliar situations without prior exposure, allowing vehicles to navigate
uncharted scenarios confidently.

  * •

Continuous Learning and Personalization: LLMs learn and adapt continuously,
tailoring their assistance to individual driver preferences and improving the
driving experience over time.

  * •

Transparency and Trust: LLMs can articulate their decisions in simple
language, fostering a crucial bond of trust and understanding between the
technology and its users.

Figure 1: The human-centric LLM-integrated framework for autonomous vehicles.

##  2 Perspective: the Role of LLMs in Advancing Autonomous Vehicles

As previously established in the earlier section, we’ve established that the
LLMs serve as the “brain” in the autonomous driving system, facilitating
driver interaction and decision-making, while the useful sensory tools and
actuation function as the vehicle’s “eyes” and “hands” respectively. To be
more specific, When a driver requests a particular operation, the LLM prompts
the related modules to provide data that has been processed to extract
relevant information from the environment. By integrating the linguistic
analysis of LLMs with the processed sensory inputs from the selected modules,
the LLM can then make well-informed decisions. If the command is deemed both
feasible and safe based on the prior analysis, the LLMs will transmit the
corresponding instructions to the vehicle’s controller. This includes
components such as the steering wheel, throttle pedal, braking, and other
control elements, enabling them to execute the necessary operations.
Alternatively, if the operation is deemed inappropriate, the LLMs will provide
drivers with a detailed explanation as to why the requested action is not
suitable for execution.

Revisiting the example at the beginning of this paper, when drivers issue the
command to overtake the vehicle ahead, the LLMs come into play by querying the
perception module for pertinent processed information. This includes details
such as the distance and speed of the target vehicle, the velocity of the ego
vehicle, road conditions of potential lanes, the presence of other vehicles
and their distances on those lanes, and other useful navigation information
from the map system. Through an analysis of the provided data and the given
command, the LLMs make a decision regarding whether to execute the driver’s
request. If the decision is affirmative, the LLMs subsequently communicate
instructions to the controller, guiding the next course of action.

Having explored this intricate interaction between LLMs and the autonomous
vehicle’s decision-making process, we shift our focus to a broader context and
propose the concept of a human-centric LLMs integrated framework for
autonomous vehicles based on our prior work of the mobility digital twin [27].
As shown in Fig. 1, the physical world comprises human drivers, vehicles, and
traffic objects. In the physical world, human drivers are the central agents
in the physical world, sending commands and instructions to LLMs as they
navigate roadways. The traffic environment contains various elements including
vehicles, pedestrians, traffic lights, road conditions, and traffic cones, all
of which contribute to the complexity of movement and interactions on the
road. The vehicle, directed by the LLMs, operates within this ecosystem,
executing the commands it receives from either drivers or LLMs through
controllers and actuators.

The virtual world includes LLMs, memory, and essential tools which include the
perception module, localization module, and in-cabin monitor. The perception
module acquires raw input from sensors, including external cameras, LIDARs,
and radars, and processes this data into a format suitable for the LLMs. The
localization module employs GNSS data to determine the vehicle’s precise
location. Within the vehicle, the in-cabin monitor employs internal cameras,
thermometers, and other sensors to vigilantly observe the in-cabin
environment, preempting distractions, extreme temperatures, or uncomfortable
conditions. At the core of the entire framework lies the LLMs, serving as its
central intelligence. They receive commands from drivers, subsequently
initiating queries to pertinent modules for related information. Furthermore,
the memory section acts as a repository, storing historical operations and
drivers’ preferences, enabling continuous learning and enhancement for the
LLMs. This repository of experiences equips the LLMs to make analogous
decisions when confronted with similar situations, bolstering the system’s
adaptability and performance over time. the memory also houses maps and local
law information, empowering the LLMs to make even wiser decisions adaptable to
a variety of scenarios.

##  3 Review: Can LLMs Really Do This?

Through a comprehensive review of both theoretical underpinnings and real-
world implementations, we seek to address the fundamental question: Can LLMs
really contribute to the improvement of autonomous driving by actively
participating in the decision-making framework? By examining the current state
of research and analyzing use cases, this section aims to provide a thorough
assessment of the extent to which LLMs can bring to the landscape of human-
centric autonomous driving.

###  3.1 Adaptive Techniques and Human-Centric Refinements for LLMs

Parameter-efficient fine tuning (PEFT) is a crucial technique used to adapt
pre-trained language models (LLMs) to specialized downstream applications [6,
9, 11, 7, 15]. Hu et al. [11] proposed utilizing low-rank decomposition
matrices to reduce the number of trainable parameters needed for fine-tuning
language models. Lester et al. [15] explore prompt tuning, a method for
conditioning language models with learned soft prompts, which achieves
competitive performance compared to full fine-tuning and enables model reuse
for many tasks. These PEFT techniques offer valuable tools for adapting LLMs
to autonomous driving tasks.

Reinforcement Learning from Human Feedback (RLHF) [23, 18, 21, 1, 20] has
emerged as a key strategy for fine-tuning LLM systems to align more closely
with human preferences. Ouyang et al. [18] introduce a human-in-the-loop
process to create a model that better follows instructions. Bai et al. [1]
propose a method for training a harmless AI assistant without human labels,
providing better control over AI behavior with minimal human input. These
approaches hold significant promise for developing LLMs for autonomous driving
applications, as they can contribute in two dimensions. Firstly, they can
ensure that LLMs avoid making decisions that may be illegal or unethical.
Secondly, these methodologies enable LLMs to continually adapt and align their
decision-making processes with user preferences, enhancing personalization and
trust in autonomous vehicles.

LLM-based autonomous driving applications can also benefit from advanced
prompting techniques [28, 13, 29, 26, 3]. Chain-of-thought prompting [28]
improves LLMs’ ability to perform complex reasoning. Gao et al. [10] propose
an approach that uses LLMs to read natural language problems and generate
programs as intermediate reasoning steps. Yao et al. [29] present a new
prompting technique that allows LLMs to make decisions about how to interact
with external APIs. These methods provide a solid foundation for the
development of LLMs for autonomous driving applications with two significant
benefits. Firstly, they greatly enhance LLMs’ reasoning capabilities,
particularly in complex, multi-step scenarios. Secondly, these techniques
improve the adaptability and versatility of LLMs, key attributes for
autonomous driving systems interfacing with various tools and data sources.

###  3.2 Advancements in LLMs: Implications for Autonomous Driving Decision-
Making

Recent Research has shown that LLMs can perform well in most commonsense tasks
[4], which means it has the potential to make wise and feasible decisions in
autonomous driving scenarios. The utilization of LLMs in the context of
autonomous driving presents a captivating and potentially transformative
direction for research. Recent investigations have brought light on the
diverse ways in which LLMs can profoundly impact the landscape of autonomous
vehicles. For instance, the study conducted by [16] highlights the promise of
AI-infused with legal knowledge, offering the potential to avert legal
transgressions in autonomous driving scenarios, thereby contributing to the
establishment of a safer AI-driven environment. Additionally, [30]
demonstrates that LLMs possess the capability to learn from local laws and
accident reports, and effectively contribute to reducing accident rates, thus
enhancing the safety of autonomous driving.

The application of LLMs to decision-making in autonomous driving is notably
explored by[5]. Their research introduces the PaLM model, demonstrating that
LLMs exhibit a capacity to effectively tackle intricate reasoning tasks and,
intriguingly, surpass the performance of an average human. Such a finding
carries significant implications, hinting at LLMs’ remarkable ability to
navigate complex scenarios, make astute judgments, and potentially lay the
groundwork for optimal decision-making in autonomous vehicles.

The work highlighted in [19] demonstrates the utilization of large language
models to effectively store experiences in natural language, forming a
foundational approach for integrating historical data into our architecture.

The adaptive capabilities of LLMs are showcased in various ways. [14]
underscores LLMs’ proficiency in zero-shot reasoning, enabling them to deal
with novel and unfamiliar situations, a vital feature for autonomous vehicles
operating in dynamic environments. The study by [6] exemplifies that LLMs can
be fine-tuned to exhibit enhanced performance, particularly in tasks with
limited training data.

Additionally, LLMs have shown great potential in both transportation and
robotics areas, as highlighted by [31], and [24] respectively. They reveal
LLMs’ prowess in tasks such as zero-shot planning and interactive
conversations, even facilitating interaction with perception-action-based API
libraries, an attribute that aligns with the demands of autonomous vehicles.

Furthermore, the work [25] demonstrates LLMs’ potential for continuous
learning, which is of paramount importance for adapting to evolving road
conditions and enhancing performance over time.

The investigation from [8] introduces embodied language models capable of
assimilating real-world sensor data, thus bridging the gap between perception
and language. This development lays the foundation for potential advancements
in autonomous vehicles, where LLMs could process sensory inputs, comprehend
their surroundings, and consequently make more informed decisions. Building on
these insights, additional studies [4], [32], [29], [22], and [12] have
further enriched our understanding of LLMs’ capabilities, underscoring their
potential in decision-making, reasoning, and synergizing reasoning and acting.

Figure 2: General Q&A with ChatGPT-4 regarding autonomous vehicles. Figure 3:
Experiment illustrating LLM-assisted decision-making and motion planning in a
complex driving scenario. The Ego vehicle and its trajectory are marked
orange; The vehicle ahead in the current lane and its trajectory are blue; The
vehicles in adjacent lanes and their trajectories are green.

##  4 Experiment: Decision-Making and Motion Planning with ChatGPT-4

To gain a deeper understanding of the practical capabilities of LLMs in the
context of autonomous driving tasks, we embark on an insightful exploration
involving real-world decision-making scenarios. This comprehensive case study
serves as a compelling demonstration of how LLMs can effectively enhance
autonomous vehicles by harnessing the potential of ChatGPT-4 [17] to replicate
decision-making processes. Our investigation is structured in two distinct
phases. Initially, we pose autonomous dirving concept-related queries to
GPT-4, which unveils its grasp of how language models can be seamlessly
integrated into autonomous driving. Subsequently, we design and present
genuine real-world situations to assess the decision-making proficiency of
LLMs. This section covers an in-depth understanding of this case study,
including a detailed conversation with GPT-4 that highlights our findings.
This analysis serves to underscore the practical implications of leveraging
LLMs for enhanced autonomous driving.

In our exploration with ChatGPT, we first asked some general conceptional
questions regarding the LLMs in autonomous vehicles and aimed to find the true
potential of LLMs in advancing the future of autonomous driving in Figure 2.
The responses indicated a profound ability of the LLMs to bridge the
interaction between the vehicle and its passengers. From the responses, it’s
evident that LLMs can explain complex driving scenarios, decisions made by the
vehicle, and even the technical of various autonomous modules. An especially
significant observation was the LLMs’ strength in processing vast volumes of
data, and then converting these into real-time, understandable feedback. Such
feedback isn’t just about driving status but relates to the core autonomous
functionalities, including the perception module’s utilization and the motion
planning’s choices. Furthermore, the model demonstrated an enhanced capacity
for vehicle-to-vehicle communications and, critically, troubleshooting. This
capability not only fosters trust but can also develop the user experience by
explaining the complex decisions of autonomous operations.

As we can see in Figure 3, we simulated a real-world driving scenario where
the autonomous vehicle is equipped with Large Language Models (LLMs) to assist
in decision-making and motion planning. The vehicle was on a two-lane Indiana
highway, traveling east to west at 96 km/h. It was behind another vehicle
moving at the same speed but only 8 meters away, a distance less than optimal
for safety. On the adjacent left lane, two other vehicles were noted: one 30
meters ahead moving at 112 km/h, and another 40 meters behind at 104 km/h. The
driver was highly attentive, and one passenger was wearing a seatbelt.

The LLMs were tasked with processing this multilayered data sourced from the
perception module (vehicle speeds and distances), the localization module
(road and environmental conditions), and the in-cabin monitoring system
(driver’s attention level and safety measures like seatbelts). The LLMs
formulated a comprehensive 9-step motion plan that prioritized safety while
efficiently executing the driver’s command to overtake the front vehicle.

In the experimental scenario, the Large Language Models (LLMs) showcased their
advanced reasoning ability by not just collecting and analyzing data but also
applying layers of context-sensitive reasoning. The LLMs evaluated the speeds
and distances of surrounding vehicles, the driver’s state of attention, and
even the traffic conditions to determine the safest and most efficient
trajectory for overtaking. This capability to reason in real-time, considering
multiple factors dynamically, significantly contributes to road safety and
operational efficacy. The LLMs didn’t merely follow pre-defined rules but
adapted their decision-making to the unique circumstances, highlighting their
potential for enhancing the future of autonomous driving.

Additionally, the language interaction capabilities of the LLMs proved crucial
for trust-building. When the driver commanded to “overtake the vehicle in
front,” the LLMs assessed various factors and communicated their reasoning to
the driver. This transparent interaction not only enhanced safety but also
instilled greater confidence in the vehicle’s autonomous capabilities.

LLMs also can access previous data and user preferences from the memory
module, which allows for a more personalized driving experience. In the
context of the experiment, for instance, the system could recall the driver’s
typical comfort levels with overtaking speeds, following distances, and lane
preferences. This information could then influence how the LLMs interpret and
execute a command like “overtake the front vehicle,” ensuring that the action
aligns with the driver’s past behavior and comfort zones. As a result, the
LLMs’ capacity for memory-driven personalization not only improves user
satisfaction but also can contribute to safer, more predictable autonomous
driving scenarios.

Another crucial advantage is enhanced transparency and trust. When the vehicle
makes a complex decision, such as overtaking another vehicle on a high-speed,
two-lane highway, passengers and drivers might naturally have questions or
concerns. In these instances, the LLMs don’t just execute the task but also
articulate the reasoning behind each step of the decision-making process. By
providing real-time, detailed explanations in understandable language, the
LLMs demystify the vehicle’s actions and underlying logic. This not only
satisfies the innate human curiosity about how autonomous systems work but
also builds a higher level of trust between the vehicle and its occupants.

Moreover, the advantage of “zero-shotting” was particularly evident during the
complex overtaking maneuver on a high-speed Indiana highway. Despite the LLMs
not having encountered this specific set of circumstances before—varying
speeds, distances, and even driver alertness—it was able to use its
generalized training to safely and efficiently generate a trajectory for the
overtaking action. This ensures that even in dynamic or rare scenarios, the
system can make sound judgments while keeping users informed, hence building
trust in autonomous technology.

##  5 Conclusion

In conclusion, our paper has provided a comprehensive framework for
integrating Large Language Models (LLMs) into the ecosystem of autonomous
vehicles. We highlighted how LLMs offer advanced reasoning capabilities that
can make autonomous systems more flexible and responsive to complex, real-
world scenarios. Additionally, by leveraging the capabilities of LLMs, we can
enrich the human-vehicle interaction, providing a more reliable, intuitive,
and responsive interface. Unlike traditional autonomous systems, which lack
the capacity for language understanding, LLMs can handle complex requests,
offer real-time feedback and comprehensive explanations, and assist in
decision-making during complex or rare driving scenarios. This suggests a
future where LLMs can significantly enhance efficiency, safety, and user-
centric design in autonomous vehicles.

## References

  * [1] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.  Constitutional AI: Harmlessness from AI Feedback, Dec. 2022.  arXiv:2212.08073 [cs].
  * [2] Emily M. Bender and Alexander Koller.  Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online, 2020. Association for Computational Linguistics.
  * [3] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler.  Graph of Thoughts: Solving Elaborate Problems with Large Language Models, Aug. 2023.  arXiv:2308.09687 [cs].
  * [4] Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He.  ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models, Mar. 2023.  arXiv:2303.16421 [cs].
  * [5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  PaLM: Scaling Language Modeling with Pathways, Oct. 2022.  arXiv:2204.02311 [cs].
  * [6] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling Instruction-Finetuned Language Models, Dec. 2022.  arXiv:2210.11416 [cs].
  * [7] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  QLoRA: Efficient Finetuning of Quantized LLMs, May 2023.  arXiv:2305.14314 [cs].
  * [8] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.  PaLM-E: An Embodied Multimodal Language Model, Mar. 2023.  arXiv:2303.03378 [cs].
  * [9] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier.  On the Effectiveness of Parameter-Efficient Fine-Tuning.  In AAAI, volume 37, pages 12799–12807, June 2023.  Number: 11.
  * [10] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.  PAL: Program-aided Language Models.  In ICML, 2023.
  * [11] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-Rank Adaptation of Large Language Models.  In ICLR, 2022.
  * [12] Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li.  Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model, May 2023.  arXiv:2305.11176 [cs].
  * [13] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large Language Models are Zero-Shot Reasoners.  In NeurIPS, volume 35, pages 22199–22213, 2022.
  * [14] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large Language Models are Zero-Shot Reasoners, Jan. 2023.  arXiv:2205.11916 [cs].
  * [15] Brian Lester, Rami Al-Rfou, and Noah Constant.  The Power of Scale for Parameter-Efficient Prompt Tuning.  In EMNLP, 2021.
  * [16] John J. Nay.  Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans.  Northwestern Journal of Technology and Intellectual Property, 20(3):309–392, 2022.
  * [17] OpenAI.  GPT-4 Technical Report, Mar. 2023.
  * [18] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In NeurIPS, volume 35, pages 27730–27744, 2022.
  * [19] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.  Generative Agents: Interactive Simulacra of Human Behavior, Aug. 2023.  arXiv:2304.03442 [cs].
  * [20] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.  Direct Preference Optimization: Your Language Model is Secretly a Reward Model, May 2023.  arXiv:2305.18290 [cs].
  * [21] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal Policy Optimization Algorithms, Aug. 2017.  arXiv:1707.06347 [cs].
  * [22] Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.  Reflexion: Language Agents with Verbal Reinforcement Learning, June 2023.  arXiv:2303.11366 [cs].
  * [23] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  In NeurIPS, volume 33, pages 3008–3021, 2020.
  * [24] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor.  ChatGPT for Robotics: Design Principles and Model Abilities, July 2023.  arXiv:2306.17582 [cs].
  * [25] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An Open-Ended Embodied Agent with Large Language Models, May 2023.  arXiv:2305.16291 [cs].
  * [26] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-Consistency Improves Chain of Thought Reasoning in Language Models.  In ICLR, 2023.
  * [27] Ziran Wang, Rohit Gupta, Kyungtae Han, Haoxin Wang, Akila Ganlath, Nejib Ammar, and Prashant Tiwari.  Mobility Digital Twin: Concept, Architecture, Case Study, and Future Challenges.  IEEE Internet of Things Journal, 9(18):17452–17467, Sept. 2022.
  * [28] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  In NeurIPS, 2022.
  * [29] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.  ReAct: Synergizing Reasoning and Acting in Language Models.  In ICLR, 2023.
  * [30] Ou Zheng, Mohamed Abdel-Aty, Dongdong Wang, Chenzhu Wang, and Shengxuan Ding.  TrafficSafetyGPT: Tuning a Pre-trained Large Language Model to a Domain-Specific Expert in Transportation Safety, July 2023.  arXiv:2307.15311 [cs].
  * [31] Ou Zheng, Mohamed Abdel-Aty, Dongdong Wang, Zijin Wang, and Shengxuan Ding.  ChatGPT Is on the Horizon: Could a Large Language Model Be All We Need for Intelligent Transportation?, Mar. 2023.  arXiv:2303.05382 [cs].
  * [32] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai.  Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory, June 2023.  arXiv:2305.17144 [cs].


Quantum Embedding with Transformer for High-dimensional Data

Hao-Yuan ChenHao-Yuan Chen
Yen-Jui ChangYen-Jui Chang
Shih-Wei LiaoShih-Wei Liao
Ching-Ray ChangChing-Ray Chang

Hao-Yuan ChenDepartment of Computer Science, University of London, London, United Kingdom
Yen-Jui ChangDepartment of Physics, National Taiwan University, Taipei, Taiwan
Shih-Wei LiaoDepartment of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan
Ching-Ray ChangDepartment of Physics, National Taiwan University, Taipei, Taiwan, Quantum Information Center, Chung Yuan Christian University, Taoyuan City, Taiwan
Hao-Yuan<EMAIL_ADDRESS>
Quantum embedding, vision transformer, self-attention, kernel method

Quantum embedding with transformers is a novel and promising architecture for quantum machine learning to deliver exceptional capability on near-term devices or simulators. The research incorporated a vision transformer (ViT) to advance quantum significantly embedding ability and results for a single qubit classifier with around 3 percent in the median F1 score on the BirdCLEF-2021, a challenging high-dimensional dataset. The study showcases and analyzes empirical evidence that our transformer-based architecture is a highly versatile and practical approach to modern quantum machine learning problems.

§ INTRODUCTION

Quantum machine learning holds a promising potential for advancing the state-of-the-art machine learning model [Biamonte et al., 2017]. Moreover, the research, [Lloyd et al., 2020, Gianani et al., 2022], has illustrated a theoretical and experimental framework for quantum embedding to deliver exceptional potential for advancing the state-of-the-art benchmark for various machine learning challenges. The research, [Lloyd et al., 2020], has introduced the initial concept of incorporating a classical convolutional neural network (CNN) to extract features from the various visual inputs. However, considering recent advancements in machine learning models like transformer [Vaswani et al., 2017] and vision transformer (ViT) [Mao et al., 2022] have made transformer models like ViT an ideal candidate for quantum machine learning on vast application domains.

Quantum embedding [Sun & Chan, 2016], a conceptual idea later validated as a practical technique in the research [Lloyd et al., 2020] that a proper quantum kernel could radically improve the various machine learning solutions. Quantum machine learning [Schuld, 2021] utilizes quantum embedding with the quantum kernel to project the data into Hilbert space. The expectation is to keep two separate classes far from two different clusters in the feature map, making effective and efficient classification possible.

The research introduces a theoretical model for incorporating pre-trained transformer models trained for the ImageNet benchmark [Beyer et al., 2022] to transform various visual inputs into linear representations of tensors for quantum embedding. The theoretical model introduced the concept of modeling various patterns using the self-attention [Shaw et al., 2018] mechanism. Moreover, a proper mapping for the feature tensors to the quantum feature map is introduced to formulate the idea of quantum embedding. Ultimately, the empirical evidence from high-dimensional datasets like BirdCLEF 2021 for binary classification evaluates the method's effectiveness.

The research question concerns whether vision transformers are efficient and effective feature extraction and representation models for quantum embedding with the hybrid trainable approach. The empirical evidence of effectiveness is based on the binary classification metrics for quantum classifiers [Blank et al., 2020], including accuracy, precision, recall, and F1 score [Hossin & Sulaiman, 2015]. The result demonstrated a significant advancement in providing a universal embedding architecture for quantum neural networks on machine learning problems, particularly classification problems.

§ METHODS

§.§ Model Architecture

The embedding method in the research is illustrated in Figure <ref>, which incorporates a transformer model with a linear representation layer to transform the original data input, i.e., images, to linear representation in tensors. Later, the linear features will be embedded into the quantum feature space with a quantum feature map using the trainable embedding technique [Hubregtsen et al., 2022]. By introducing a transformer model, like a vision transformer (ViT), the architecture provides a versatile, classical embedding layer for quantum feature maps to capture various data forms, including recurrent or sequential patterns.

Model architecture of the research on quantum embedding with transformers for high-dimensional visual datasets. The orange part stands for the classical state of data, and the blue part means a quantum state of information. The input data feeds from the left to the right to classify the input information.

§.§ Vision Transformer

The study facilitates Vision Transformer (ViT) to process and embed images using the Transformer architecture as follows:

* Image Tokenization: An input image \( I \) of size \( H \times W \times C \) is partitioned into a sequence of non-overlapping patches \( P \). Each patch of size \( N \times N \times C \) is flattened and linearly projected into a \( D \)-dimensional embedding space to create a sequence of patch embeddings \( X_p \).

* Positional Encoding: Positional encodings \( E_{pos} \) are added to the patch embeddings to provide positional information, resulting in the input sequence \( X = X_p + E_{pos} \).

* Transformer Encoder: The encoder consists of \( L \) layers, each containing a self-attention (SA) mechanism and a feed-forward network (FFN).

* Self-Attention (SA): For queries \( Q \), keys \( K \), and values \( V \), which are projections of \( X \), the attention function is computed as:

\begin{equation}
\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V
\end{equation}

where \( d_k \) is the dimension of the key.

* Feed-Forward Network (FFN): The FFN is applied to the output of the attention layer, defined as:

\begin{equation}
\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2
\end{equation}

Each layer includes skip connections and is followed by layer normalization.

* Classifier Head: The representation \( X_L[0] \) from the first token of the last layer \( L \) is passed through a linear layer to obtain the final predictions \( y \):

\begin{equation}
y = \text{Linear}(X_L[0])
\end{equation}

§.§ Trainable Quantum Embedding with Classical Linear Transformation

The trainable quantum embedding [Thumwanit et al., 2021] with Transformer algorithm is an advanced approach integrating quantum computing principles with modern machine learning techniques. This hybrid algorithm at algorithm <ref> begins by taking images and their corresponding labels as inputs. It utilizes a Transformer model, a type of deep learning model renowned for its effectiveness in handling sequential data to embed each image into a high-dimensional space, effectively capturing the complex patterns and features within the pictures.

Once the images are embedded, they are reduced to a qubit-sized representation, aligning them with the requirements of quantum computing frameworks. This qubit representation is then processed through a Quantum Neural Network (QNN), which operates using the principles of quantum mechanics to perform computations that would be infeasible for classical neural networks. The QNN applies a quantum feature map [Schuld & Killoran, 2019, Goto et al., 2020] and a quantum ansatz—parametric operations that encode the data into a quantum state and then transform it in a way that is dependent on the parameters being learned.

The output from the QNN is then measured, collapsing the quantum state into a classical form that can be used to compute a loss function. This loss function evaluates the accuracy of the model's predictions against the true labels, providing a basis for updating the Transformer and QNN parameters through backpropagation [Verdon et al., 2018, Gonçalves, 2016]. This algorithm adjusts parameters to minimize errors.

§.§.§ Mathematical Model

Linear layer transformation with trainable weights and the quantum embedding featuring two sets of Hadamard (\(H\)) and \(U_1(2\theta)\) gates (Z feature map), we will create a hybrid quantum-classical model using quantum kernel method. This model will utilize the output of the classical linear layer as the input parameter (\(\theta\)) for the quantum gates in the quantum embedding process.

The input to the model is a 1000-dimensional vector from the ViT trained for ImageNet-1K benchmark \( \mathbf{x} \). The linear layer transformation is given by:

\[ y = \mathbf{w}^T \mathbf{x} + b \]

* \( \mathbf{w} \) is the trainable weight vector of the linear layer with 1000 elements.

* \( b \) is the trainable bias term.

* \( y \) is the scalar output of the linear layer, which will be used as the parameter \( \theta \) in the quantum embedding.

The scalar output \( y \) from the linear layer is used to parameterize the quantum gates in the embedding. The quantum state transformation is shown as the following.

§.§ Quantum Feature Map and Ansatz

Once the ViT processes the visual data, the linear representation layer transforms it. It embeds it into a quantum neural network shown in Figure <ref> to train the quantum neural network with the quantum Ansatz. The Pauli Feature Map, designed explicitly as a first-order Pauli Z-evolution circuit, stands as a cornerstone in quantum computing for embedding classical data into a quantum state. The Pauli Feature Map facilitates the construction of highly complex, high-dimensional feature spaces that classical computing methods struggle to replicate, opening new avenues for advancements in quantum machine learning and data analysis.

In the described quantum neural network including the Z-feature map and Real Amplitude Ansatz shown in Figure <ref>, the application of Hadamard (H) and U1 gates with a parameterized rotation \(2.0 \times x[0]\) transforms an initial quantum state \(|\psi\rangle\) through a series of operations that intricately encode information into the quantum system. Starting with the Hadamard gate, it places the qubit into a superposition, a fundamental state for quantum computation that allows a qubit to combine '0' and '1' states. This is mathematically represented as:

Quantum neural network's architecture. Once the data processed by the transformer, the linear representation layer will transform the data and embed into the quantum circuit forming quantum state of information.

\begin{equation}
H = \frac{1}{\sqrt{2}}
\begin{pmatrix}
1 & 1 \\
1 & -1
\end{pmatrix}
\end{equation}

Following the superposition, the U1 gate introduces a phase shift dependent on the value \(2.0 \times x[0]\) without altering the probability amplitudes of the qubit states. The U1 gate is defined as:

\begin{equation}
U1(\lambda) =
\begin{pmatrix}
1 & 0 \\
0 & e^{i\lambda}
\end{pmatrix}
\end{equation}

where \( \lambda = 2 \times x[0] \).

The sequence of applying H, U1, H, and then U1 again results in a complex transformation of the initial state \(|\psi\rangle\), which is captured by the final formula:

\begin{equation}
|\psi'\rangle = U1(2 \times x[0]) \cdot H \cdot U1(2 \times x[0]) \cdot H \cdot |\psi\rangle
\end{equation}

This formula encapsulates the evolution of the qubit's state through the circuit, embedding the parameter \(x[0]\) into its phase and superposition, culminating in a state \(|\psi'\rangle\) that carries this encoded information. This process exemplifies the power of quantum circuits to manipulate quantum states, paving the way for sophisticated quantum algorithms that exploit these unique quantum phenomena for computational advantage.

The final quantum state \( |\psi(y)\rangle \) is obtained, where \( y \) is derived from the classical linear layer's output and measured in the computational basis, yielding probabilities \( P(0) \) and \( P(1) \) for the qubit being in the state \( |0\rangle \) and \( |1\rangle \), respectively. These probabilities are used to define the binary cross-entropy loss as the objective function:

\[ \mathcal{L} = -\left[y_{\text{true}} \log P(0) + (1 - y_{\text{true}}) \log P(1)\right] \]

where \( y_{\text{true}} \) is the binary label associated with the input \( \mathbf{x} \).

During training, the goal is to adjust the parameters of the classical linear layer (\( \mathbf{w} \) and \( b \)) to minimize the loss function \( \mathcal{L} \). This process involves backpropagation through the quantum circuit, which can be challenging due to the non-classical nature of quantum state transformations. Techniques such as the parameter shift rule may be employed to compute gradients of quantum circuits to classical parameters.

Trainable Quantum Embedding with Vision Transformer

$\mathcal{I}$: Set of images
$\mathcal{L}$: Corresponding labels
Output: Optimized model parameters

Trainable Quantum Embedding
Initialize parameters for Transformer and QNN
not converged
each $(i, l) \in (\mathcal{I}, \mathcal{L})$
$e_i \gets$ Embed image $i$ using Transformer
$q_i \gets$ Reduce $e_i$ to qubit size
$d_i \gets$ Apply QNN to $q_i$ and measure
$\mathcal{F} \gets$ Compute loss for $d_i$ and $l$
Update parameters to minimize $\mathcal{F}$
Assess model on the validation set

§ RESULTS

The results from the empirical experimentation demonstrate that transformer-based quantum embedding is a stable and highly effective method for quantum neural networks using kernel methods. The transformer-based method outperforms the CNN-based variant by around 3 percent in the F1 score metric. Moreover, transformer-based quantum embedding delivers exceptional reliability and consistently high precision and accuracy with a standard deviation of 90 percent more.

§.§ Effectiveness on Binary Classification

Figure <ref> presents the results from the Bird-CLEF 2021 challenge, focusing on the comparative analysis of F1 scores, which measure the harmonic balance between precision and recall in binary classification tasks. The figure demonstrates that the transformer-based quantum embedding method exhibits remarkable effectiveness and reliability over the classical CNN and CNN-based embedding methods. The F1 scores for the Transformer-based architecture consistently outperform the other variants or alternatives, culminating in an average score of approximately 0.785, substantially higher than its counterparts.

Comparative results for the BirdCLEF-2021 dataset over various embedding methods, including classical CNN method (at the far left), CNN-based quantum embedding (at the middle), the architecture proposed in this research, which is transformer-based quantum embedding (at the far right)

Comparison of Statistical Results from Bird-CLEF 2021 Binary Classification Over Various Methods

Standard Deviation     Median F1

Classical Baseline       0.0370     0.728

CNN-based                0.0536     0.741

Transformer-based        0.0052     0.774

§.§ Performance Reliability

Furthermore, the narrow confidence intervals at Figure <ref> associated with the Transformer-based Quantum Embedding method indicate its reliability. Despite the increasing complexity of the classification tasks, the technique shows less variance in performance, suggesting that it is practical and robust to changes and challenges inherent in the Bird-CLEF dataset.

Figure <ref> shows that our quantum embedding architecture with the ViT model outperforms all alternative methods by a significant level in terms of the standard deviation (SD) of the F1 score in the Bird-CLEF 2021 dataset. A significantly low SD demonstrates that the architectural innovation with a transformer could consistently yield stable and high-performing results with high precision and accuracy under binary classification.

Comparative results for the standard deviation of F1 score for BirdCLEF-2021 dataset over various embedding methods or training method, including classical CNN method (at the top), CNN-based quantum embedding (at the middle), the architecture proposed in this research, which is transformer-based quantum embedding (at the button)

In summary, the Transformer-based Quantum Embedding method stands out as both practical and reliable, outshining conventional and CNN-based methods in the high-stakes arena of bio-acoustic event detection, as reflected by its superior and consistent F1 scores in the Bird-CLEF 2021 challenge. This evidence implies that the architectural innovation of this study has yielded a significant advancement in shallow quantum circuits to deliver promising machine learning capability with an effective and efficient embedding scheme.

§ DISCUSSION

§.§ Extension to Larger Circuits

The architecture proposed in this study is extensible to larger quantum circuits with more than one qubit. The linear transformation layer is flexible and adaptable to various circuit sizes. However, the effectiveness of embedding requires further investigation to ensure the overall effectiveness of this embedding scheme.

§.§ Technical Challenges

The trainable method in this study involves gradient descent and backpropagation combined with classical and quantum elements, which might encounter risks like a barren plateau or hybrid gradient descent. Further investigation into such challenges and understanding of this architecture's theoretical and practical constraints are critical.

§.§ Outlooks

Considering the substantial advancement of this architectural innovation in quantum embedding, the architecture paves the way for future quantum kernel methods with quantum neural networks (QNNs) to model sequential or recurrent data patterns with the aid of transformer-based models. This could be a new pathway for quantum machine learning in solving complex computer vision (CV) and natural language processing (NLP) tasks.

§ CONCLUSION

The research proposed an architectural innovation to facilitate quantum embedding with a transformer model for solving high-dimensional binary classification problems using the single-qubit classifier. The empirical evidence has shown that our architectural innovation from quantum embedding to classifier design ha s significantly improved compared to the classical counterpart and another quantum embedding variant. The research implies that our embedding architecture with a transformer is a better, more reliable scheme for facilitating quantum machine learning in solving complex problems.

[Beyer et al., 2022]
Beyer, L., Zhai, X., and Kolesnikov, A.
Better plain vit baselines for imagenet-1k, 2022.

[Biamonte et al., 2017]
Biamonte, J., Wittek, P., Pancotti, N., Rebentrost, P., Wiebe, N., and Lloyd, S.
Quantum machine learning.
Nature, 5490 (7671):0 195–202, 2017.

[Blank et al., 2020]
Blank, C., Park, D. K., Rhee, J.-K. K., and Petruccione, F.
Quantum classifier with tailored quantum kernel.
npj Quantum Information, 60 (1):0 41, 2020.

[Gianani et al., 2022]
Gianani, I., Mastroserio, I., Buffoni, L., Bruno, N., Donati, L., Cimini, V., Barbieri, M., Cataliotti, F. S., and Caruso, F.
Experimental quantum embedding for machine learning.
Advanced Quantum Technologies, 50 (8):0 2100140, 2022.

[Gonçalves, 2016]
Gonçalves, C. P.
Quantum neural machine learning-backpropagation and dynamics.
arXiv preprint arXiv:1609.06935, 2016.

[Goto et al., 2020]
Goto, T., Tran, Q. H., and Nakajima, K.
Universal approximation property of quantum feature map.
arXiv preprint arXiv:2009.00298, 2020.

[Hossin & Sulaiman, 2015]
Hossin, M. and Sulaiman, M. N.
A review on evaluation metrics for data classification evaluations.
International journal of data mining & knowledge management process, 50 (2):0 1, 2015.

[Hubregtsen et al., 2022]
Hubregtsen, T., Wierichs, D., Gil-Fuster, E., Derks, P.-J. H. S., Faehrmann, P. K., and Meyer, J. J.
Training quantum embedding kernels on near-term quantum computers.
Physical Review A, 1060 (4), October 2022.
ISSN 2469-9934.
URL <http://dx.doi.org/10.1103/PhysRevA.106.042431>.

[Lloyd et al., 2020]
Lloyd, S., Schuld, M., Ijaz, A., Izaac, J., and Killoran, N.
Quantum embeddings for machine learning.
arXiv preprint arXiv:2001.03622, 2020.

[Mao et al., 2022]
Mao, X., Qi, G., Chen, Y., Li, X., Duan, R., Ye, S., He, Y., and Xue, H.
Towards robust vision transformer.
In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp.  12042–12051, 2022.

[Schuld, 2021]
Schuld, M.
Supervised quantum machine learning models are kernel methods.
arXiv preprint arXiv:2101.11020, 2021.

[Schuld & Killoran, 2019]
Schuld, M. and Killoran, N.
Quantum machine learning in feature hilbert spaces.
Physical review letters, 1220 (4):0 040504, 2019.

[Shaw et al., 2018]
Shaw, P., Uszkoreit, J., and Vaswani, A.
Self-attention with relative position representations.

[Sun & Chan, 2016]
Sun, Q. and Chan, G. K.-L.
Quantum embedding theories.
Accounts of chemical research, 490 (12):0 2705–2712, 2016.

[Thumwanit et al., 2021]
Thumwanit, N., Lortaraprasert, C., Yano, H., and Raymond, R.
Trainable discrete feature embeddings for variational quantum classifier.
arXiv preprint arXiv:2106.09415, 2021.

[Vaswani et al., 2017]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.

[Verdon et al., 2018]
Verdon, G., Pye, J., and Broughton, M.
A universal training algorithm for quantum deep learning.
arXiv preprint arXiv:1806.09729, 2018.


# NTIRE 2023 Image Shadow Removal Challenge
Technical Report: Team IIM_TTI

Yuki Kondo, Riku Miyata11footnotemark: 1, Fuma Yasue11footnotemark: 1, Taito
Naruki11footnotemark: 1, Norimichi Ukita
Intelligence Information Media Lab., Toyota Technological Institute (IIM_TTI)
<EMAIL_ADDRESS>Equally contributed author

## Abstruct

In this paper, we analyze and discuss ShadowFormer in preparation for the
NTIRE2023 Shadow Removal Challenge [1], implementing five key improvements:
image alignment, the introduction of a perceptual quality loss function, semi-
automatic annotation for shadow detection, joint learning of shadow detection
and removal, and the introduction of a new data augmentation technique
“CutShadow” for shadow removal. Our method achieved scores of 0.196 (3rd out
of 19) in LPIPS and 7.44 (4th out of 19) in the Mean Opinion Score (MOS).

##  1 Team details

  * •

Team name : IIM_TTI

  * •

Team leader name : Yuki Kondo

  * •

Team leader institution and email (Please make sure is an active email) :
Toyota Technological Institute<EMAIL_ADDRESS>)

  * •

Rest of the team members : Riku Miyata (Toyota Technological Institute,
sd22431@toyota-ti.ac.jp); Fuma Yasue(Toyota Technological Institute,
<EMAIL_ADDRESS>); Taito Naruki (Toyota Technological Institute,
<EMAIL_ADDRESS>); Norimichi Ukita (Toyota Technological Institute,
<EMAIL_ADDRESS>)

  * •

Team website URL (if any) : https://www.toyota-
ti.ac.jp/Lab/Denshi/iim/index.html

  * •

Affiliations : Toyota Technological Institute, Japan.

  * •

Usernames on the NTIRE 2023 Image Shadow Removal Challenge Codalab leaderboard
(development/validation and testing phases) : Yuki-11; amaguri; YasueFuma;
feln

  * •

Link to the codes/executables of the solution(s) :
https://github.com/Yuki-11/NTIRE2023_ShadowRemoval_IIM_TTI

  * •

Link to the enhancement results of all frames :
https://drive.google.com/drive/folders/11zu0P7yAORDPhcTBjW5PSYODYi--
gMcO?usp=share_link

##  2 Contribution details

ShadowFormer+ : Rethinking the ShadowFormer learning methods and removing the
effect of external camera parameter changes

###  2.1 Analysis of provided NTIRE 2023 SHADOW REMOVAL dataset

(a) Misalignment in original sample (b) Aligned sample

Figure 1: Superimposed display of shadow and no-shadow pair images. Random
changes in external parameters caused the misalignment (a), but our method was
able to correct the misalignment (b).

After analyzing this challenge’s dataset, we found several scenes where the
external camera parameters differed significantly between the input image with
shadows and its GT image with no shadow, as shown in Fig. 1 (a) . Therefore,
in addition to the original task of shadow removal, we have to spatially align
these two images by implicitly or explicitly calibrating the external camera
parameters between the images.

We also found that many images in the dataset are taken from bird’s eye views
rather than side views so that a planar regions is dominant in each image. In
addition, such planar regions have local feature points on sheets of papers,
clothes, and so on.

Based on the above discussion, we conclude that homography-based image
alignment is effective as a preprocess for shadow removal learning.

###  2.2 Analysis and discussion of ShadowFormer

ShadowFormer [2] is the Transformer-based SOTA model of shadow removal.
ShadowFormer effectively models the context correlation between shadow and
non-shadow regions utilizing the Shadow-Interaction Module (SIM) that performs
Shadow-Interaction Attention (SIA) in the intermediate feature space.

To further improve ShadowFormer, we found the following five rooms for
improvement.

(1) Lack of the explicit external camera parameter estimation mechanism:
ShadowFormer does not have a function for image alignment between images with
and without shadows, which are required in the dataset of this shadow removal
challenge.

(2) Limitations on perceptual quality improvement of pixel-wise loss
functions: Charbonnier loss is employed in ShadowFormer, but it is known that
pixel-wise loss functions in the image space, such as L1 loss and L2 loss,
including Charbonnier loss, limit the perceptual quality [3, 4, 5, 6].

(3) Difficulty in applying shadow detector to the challenge dataset:
ShadowFormer requires a shadow mask. As with the original implementation of
ShadowFormer, we use DHAN [7] for detecting the shadow mask in an input image.
Since the dataset of this challenge consists of a wide variety of shadows, the
shadow detector should be optimized to the domain of this dataset. However,
since there is no shadow mask in the dataset of this challenge, it is
impossible to apply the straightforward supervised learning for optimizing the
shadow detector.

(4) Independent optimization of shadow detector and shadow remover: While it
is known that a preprocess (e.g., shadow detector) can be jointly trained with
a main task (e.g., shadow remover) for improving the performance of the main
task in a variety of tasks [8, 9, 10, 11, 12], ShadowFormer is trained
independently with the shadow detector.

(5) Insufficient data augmentation: In training of ShadowFormer, only three
types of data augmentation schemes are employed: MIxUp [13], rotation, and
vertical flip. Since the number of training images in the challenge dataset
(i.e., 1,000 images), further data augmentation schemes can improve the
performance of ShadowFormer.

###  2.3 Our method

Figure 2: Overall diagram of our methodology. In preprocessing, we correct
image misalignment by Homography Estimation, and also create GT shadow mask by
SAMSA. In training, firstly shadow detector predict the shadow mask from the
shadow image. Next, ShadowFormer predict shadow-free image from shadow image
and the shadow mask which predicted by detector. at this time, CutShadow is
used for augumentation, and ESSIM loss and Structure Preservation Loss are
used as the error functions.

In our proposed method, the five problems described in Sec. 2.2 are resolved
as follows.

(1) Image alignment by global homography transformation:

(a) Input image (b) Non-aligned ground truth

(c) Baseline (d) Ours (Aligned)

Figure 3: Comparison of (c) the baseline results of training ShadowFormer
without alignment and (d) the results of our method including alignment. The
PSNR based on the original unaligned GT in (c) and (d) are 26.79 and 22.67,
respectively, and while (c) is superior in PSNR, (d) is perceptually superior
because it is not blurred and its structure is guaranteed. Shadows are also
accurately removed for the regions where they occur.

As mentioned in Sec. 2.1, in training, images with and without shadows are
spatially aligned based on Homography. Homography estimation is done with
local feature matching using AKAZE [14] and outlier removal using RANSAC [15].

With the estimated Homography, an image without shadows is transformed to
align it with the one with shadows. Lacking pixels caused by the Homography
transformation are the reflect padding process because it improves the shadow
removal performance rather than using the zero padding process. The original
training images and their aligned images are shown in Fig. 1 (b). It can be
seen that most pixels are successfully aligned.

While we train the shadow remover (i.e., ShadowFormer) with the aligned image
pairs, the shadow remover can be trained so that it also implicitly align
images with and without shadows if these misaligned images are used as
training images. Therefore, with the misaligned GT images with and without
shadows used in the Development and Test phases, the shadow remover trained
with such misaligned images works better than the one trained with the aligned
images in terms of PSNR. On the other hand, as shown the result shown in Fig.
3, the perceptual quality of the results trained on aligned data is greatly
improved.

(2-1) Structure Preservation loss: We adopt Structure Preservation loss, which
minimizes the distance between the output and GT images in the feature space
obtained from DiNO111Using unlabeled ImageNet [16] contains about 14 million
images for pre-training. [17] in accordance with the success of
ShadowDiffusion [18], instead of Charbonnier loss. However, unlike the
original Structure Preservation loss, our method does not use the cosine
similarity but the euclidean distance using MSE. The reason for this change is
that the quantitative results are almost equal between the two similarity
measurements, but the visual quality was better with the MSE. The above loss
function is shown in the following equation:

$\displaystyle\mathcal{L}_{SP}$ $\displaystyle=$
$\displaystyle\mathrm{MSE}\left(\mathrm{DiNO}\left(I_{pred}\right),\mathrm{DiNO}\left(I^{\prime}_{GT}\right)\right)$
(1)

where DiNO is the DiNO feature extractor, MSE is the mean squared error
function, and $I_{pred}$ means the output image from ShadowFormer.

(2-2) Edge-based SSIM (ESSIM) loss: While Structure Preservation loss is used
in the feature space for semantic matching, we also use another loss function
in the image domain for preserving the geometric structure. As validated in
[19], it is known that a SSIM-based loss can improve the perceptual quality in
contrast to a MSE-based loss. In the proposed Edge-based SSIM (ESSIM), an
input image is first mapped to the HSV space to extract the structure
represented in the V-channel map. Next, the edge image is acquired by Canny
edge detector [20], and the SSIM loss is calculated in the edge image:

$\displaystyle\mathcal{L}_{ESSIM}$ $\displaystyle=$
$\displaystyle\mathcal{L}_{SSIM}\left(\mathrm{EdgeDetect}\left(I_{pred}\right),\mathrm{EdgeDetect}\left(I^{\prime}_{GT}\right)\right),$
(2) $\displaystyle\mathrm{EdgeDetect}(I)$ $\displaystyle=$
$\displaystyle\mathrm{Canny}([\mathcal{T}_{HSV}(I)]_{V}),$ (3)

where $\mathcal{L}_{SSIM}$ means SSIM loss [19], $\mathcal{T}_{HSV}$ is the
HSV translation, $[\ ]_{V}$ means V-channel map extraction and
$\mathrm{Canny}$ means Canny edge detection [20].

Figure 4: Semi-Automatic Shadow Mask Annotation (SASMA). The red line in the
histogram indicates the peak value, and the green and blue lines indicate the
lower and upper values used for shadow binarization calculated from the peak
values, respectively. Note that the second process, Sub.&Abs., is done in the
value channel of HSV.

(3) Semi-Automatic Shadow Mask Annotation: To supervise the shadow detector on
the challenge dataset, we need to prepare the GT of a shadow mask. Therefore,
we propose Semi-Automatic Shadow Mask Annotation (SASMA), which performs easy,
efficient, and reasonable annotation semi-automatically.

SASMA first obtains the absolute error maps of the V channel maps of both the
HSV-transformed input and GT images. The histogram of the error map is
displayed to an annotator. The annotator is requested only to found the
distributions corresponding to shadow regions. With the error values given by
the annotator in the histogram, we can determine their corresponding pixels in
the image. These pixels are regarded as shadow pixles. Figure 4 shows the
process flow of our SASMA.

(4) Joint learning of shadow detector and shadow remover:

(a) Input image (b) MTMT (Released weights) (c) Mask annotated by SASMA

(d) MTMT (Pretiraned) (e) Joint learning ($\alpha=10^{-2}$) (f) Joint learning
($\alpha=10^{-3}$)

Figure 5: Refinement of shadow detection by joint learning. At first, MTMT is
not able to predict shadows at all (b), but by learning the mask annotated by
SASMA (c), it is able to predict shadows to some extent (d). Furthermore, by
incorporating joint learning, MTMT is able to predict shadows at the level of
the GT mask, and is also able to capture the strength and weakness of the
shadows (e) and (f).

In our joint learning, the shadow detection network is serially connected to
the shadow removal network (i.e., ShadowFormer) in order to train these two
networks in an end-to-end manner. For this joint leaning, we use MTMT [21], a
shadow detection SOTA method, instead of DHAN used in the original
ShadowFormer.

The loss function $\mathcal{L}_{joint}$ we use in joint learning is as
follows:

$\displaystyle\mathcal{L}_{joint}$ $\displaystyle=$
$\displaystyle(1-\alpha)\mathcal{L}_{removal}+\alpha\mathcal{L}_{detction},$
(4) $\displaystyle\mathcal{L}_{removal}$ $\displaystyle=$
$\displaystyle\beta\mathcal{L}_{SP}+\mathcal{L}_{ESSIM}+\mathcal{L}_{MSE},,$
(5)

where $\alpha$ is the pretask weight, $\mathcal{L}_{MSE}$ means MES loss,
$\beta$ is the weight of the structure preservation loss. $\beta$ = $10^{6}$
in our experiments.

The effects of our proposed (3) SASMA and (4) joint learning are shown in Fig.
5. The result of shadow detection using MTMT with the published weights is
shown in Fig. 5 (b) in which no shadow is detected. In our method, on the
other hand, a shadow mask annotated by SASMA in each training image, whihc is
shown in Fig. 5 (c), is used to finetune MTMT. The result of the finetuned
MTMT is shown in Fig. 5 (d). While the performance is significantly improved
in (d), many other thin shadows are not detected yet. On the other hand,
shadows detected by our joint learning network, which are shown in Fig. 5 (e)
and (f), are better than (d). Interestingly, if $\alpha$ is reduced, i.e., it
is dedicated to solving the shadow removal task, the mask becomes sharper,
even though the shadow detection task is less important. This shows that the
representation of the mask image required by ShadowFormer is close to that of
the GT mask.

(5) CutShadow:

CutBlur [22]

CutShadow (Ours)

Figure 6: Two data augmentations of CutBlur and CutShadow. The first row shows
the GT, the second row shows the input image, the third row shows the new
input image generated by each data extension (reverse augmentation is also
possible, where the base image is the input image and the GT patches are
pasted), and the fourth row shows the new input shadow mask generated by
CutShadow.

We propose CutShadow to generate more pseudo-trained data based on CutBlur
[22] proposed for image super resolution. Figure 6 shows the samples of
CutBlur and CutShadow. Given paired training images (e.g., images with and
without shadows for the shadow removal task), While low-resolution image
patches are pasted on a high-resolution image, image patches with shadows are
pasted on an image without shadows.

##  3 Global Method Description

###  3.1 Learning Strategies

####  3.1.1 Preprocessing phase

  1. 1.

Alignment of GT images by homography transformation

  2. 2.

Semi-automatic generation of shadow masks using SASMA

####  3.1.2 Training phase

  1. 1.

Shadow detector (MTMT) pre-training using challenge dataset from initial
weights. However, for ResNeXt, which is a module of MTMT, we use the
parameters already learned by ImageNet [16].

  2. 2.

Shadow remover (ShadowFormer) pre-training using challenge dataset from
initial weights.

  3. 3.

Joint learning shadow detector and shadow remover using challenge dataset from
pre-trained weight.

####  3.1.3 Testing phase

  1. 1.

The provided image is input to the jointed model to obtain a shadow removal
image.

###  3.2 Quantitative and qualitative advantages of the proposed solution

Table 1:  Comparison with baseline and our method. The best scores are colored by red , respectively. Our method is pre-aligned, so the PSNR and SSIM based on the misaligned GT image are extremely low.  Model |  PSNR$\uparrow$ |  SSIM$\uparrow$
---|---|---
Baseline | 19.92 | 0.660
Ours | 18.69 | 0.605

To demonstrate the superiority of the proposed method, we present a
qualitative and quantitative evaluation of the plain ShadowFormer for baseline
model.

The output samples are shown in the Fig. 3. The baseline has a high PSNR
evaluated with the original misaligned GT image, but a blurred image is
generated by averaging random external camera parameter changes. On the other
hand, although the proposed method has a low PSNR evaluated on the original
misaligned GT image, it can maintain the context of the scene and remove the
shadows, thus obtaining perceptually excellent results.

###  3.3 Other Additional Notes

Novelty degree of the solution: Please check section 2.3.

Previously published: No.

Previously published: Please check the table 2 specifying the technical
information.

Table 2: Technical information table

Input | Training Time | Epochs | Extra data | Attention | Quantization | # Params. (M) | Runtime (ms) | GPU
---|---|---|---|---|---|---|---|---
(640, 640, 3)/(1920, 1440, 3) | 60h | 1100 | Yes | Yes | No | 55 Million | 1010 on GPU | A100

##  4 Competition particularities

Any particularities of the solution for this competition in comparison to
other challenges (if applicable).

  * •

None

##  5 Technical details

Please make sure to write about the language and implementation details:
framework, optimizer, learning rate, GPU, datasets used for training, training
time, training strategies, efificiency optimization strategies.

  * •

Framework : PyTorch 1.17

  * •

optimizer : AdamW

  * •

learning rate : WarmUp and cosine scheduler (Upper: 0.0002, Lower: 0.000001)

  * •

GPU : A100

  * •

datasets used for training : ImageNet [16] and the challenge dataset with
shadow masks.

  * •

training time : 60 h

  * •

training strategies : Please check section 3.1.

  * •

efificiency optimization strategies : None

##  6 Other details

  * •

Planned submission of a solution(s) description paper at NTIRE 2023 workshop.
If we win a prize, YES.

  * •

General comments and impressions of the NTIRE 2023 Image Shadow Removal
Challenge (we appreciate your feedback to improve in future editions).

It differs from the previous datasets in that the shadow and shadow-free
images are misaligned and there is no shadow mask for the shadow images, which
was a challenge to devise. These differences are very practical because they
are in line with real-life situations.

  * •

What do you expect from a new challenge in image restoration, enhancement and
manipulation?

We want to Vison and Language, very sparse video frame interpolation and real
world image harmonization challenges.

## Acknowledgments

This work was partly supported by JSPS KAKENHI Grant Numbers 19K12129 and
22H03618.

## References

  * [1] F.-A. ru Vasluianu, T. Seizinger, R. Timofte, S. Cui, J. Huang, S. Tian, M. Fan, J. Zhang, L. Zhu, X. Wei, X. Wei, Z. Luo, F. K. Gustafsson, Z. Zhao, J. Sjölund, T. B. Schön, X. Dong, X. S. Zhang, C. Li, C. Leng, W.-H. Yeo, W.-T. Oh, Y.-R. Lee, H.-C. Ryu, J. Luo, C. Jiang, M. Han, Q. Wu, W. Lin, L. Yu, X. Li, T. Jiang, H. Fan, S. Liu, S. Xu, B. Song, X. Chen, S. Zhang, J. Zhou, Z. Zhang, S. Zhao, H. Zheng, Y. Gao, Y. Wei, B. Wang, J. Ren, Y. Luo, Y. Kondo, R. Miyata, F. Yasue, T. Naruki, N. Ukita, H.-E. Chang, H.-H. Yang, Y.-C. Chen, Y.-C. Chiang, Z.-K. Huang, W.-T. Chen, I.-H. Chen, C.-H. Hsieh, S.-Y. Kuo, L. Xianwei, H. Fu, C. Liu, H. Ma, B. Fu, H. He, M. Wang, W. She, Y. Liu, S. Nathan, P. Kansal, Z. Zhang, H. Yang, Y. Wang, Y. Zhang, S. S. Phutke, A. Kulkarni, M. R. Khan, S. Murala, S. K. Vipparthi, H. Ye, Z. Liu, X. Yang, S. Liu, Y. Wu, Y. Jing, Q. Yu, N. Zheng, J. Huang, Y. Long, M. Yao, F. Zhao, B. Zhao, N. Ye, N. Shen, Y. Cao, T. Xiong, W. Xia, D. Li, and S. Xia, “Ntire 2023 image shadow removal challenge report,” in CVPRW, 2023.
  * [2] L. Guo, S. Huang, D. Liu, H. Cheng, and B. Wen, “ShadowFormer: Global Context Helps Image Shadow Removal,” in AAAI, 2023.
  * [3] Y. Blau, R. Mechrez, R. Timofte, T. Michaeli, and L. Zelnik-Manor, “The 2018 pirm challenge on perceptual image super-resolution,” in ECCVW, 2018.
  * [4] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018.
  * [5] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in ECCV, 2016.
  * [6] K. Zhang, S. Gu, and R. Timofte, “Ntire 2020 challenge on perceptual extreme super-resolution: Methods and results,” in CVPRW, 2020.
  * [7] X. Cun, C.-M. Pun, and C. Shi, “Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting gan,” in AAAI, 2020\.
  * [8] M. Haris, G. Shakhnarovich, and N. Ukita, “Task-driven super resolution: Object detection in low-resolution images,” in ICONIP, 2021.
  * [9] R. Hayashi, A. Shimokura, T. Matsumoto, and N. Ukita, “Joint learning of object detection and pose estimation using augmented autoencoder,” in MVA, 2021.
  * [10] C. Nakatani, K. Sendo, and N. Ukita, “Group activity recognition using joint learning of individual action recognition and people grouping,” in MVA, 2021.
  * [11] Y. Kondo and N. Ukita, “Joint learning of blind super-resolution and crack segmentation for realistic degraded images,” arXiv, 2023.
  * [12] Y. Kondo and N. Ukita, “Crack segmentation for low-resolution images using joint learning with super-resolution,” in MVA, 2021.
  * [13] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” arXiv, 2017.
  * [14] P. F. Alcantarilla, J. Nuevo, and A. Bartoli, “Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces,” in BMVC, 2013.
  * [15] M. A. Fischler and R. C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Commun. ACM, vol. 24, no. 6, pp. 381–395, 1981.
  * [16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in CVPR, 2009.
  * [17] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in ICCV, 2021.
  * [18] Y. Jin, W. Yang, W. Ye, Y. Yuan, and R. T. Tan, “Shadowdiffusion: Diffusion-based shadow removal using classifier-driven attention and structure preservation,” ArXiv, vol. abs/2211.08089, 2022.
  * [19] H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” TCI, vol. 3, no. 1, pp. 47–57, 2016\.
  * [20] J. Canny, “A computational approach to edge detection,” TPAMI, no. 6, pp. 679–698, 1986.
  * [21] Z. Chen, L. Zhu, L. Wan, S. Wang, W. Feng, and P.-A. Heng, “A multi-task mean teacher for semi-supervised shadow detection,” in CVPR, 2020.
  * [22] J. Yoo, N. Ahn, and K.-A. Sohn, “Rethinking data augmentation for image super-resolution: A comprehensive analysis and a new strategy,” in CVPR, 2020.