11institutetext: Integrated Vision and Language Lab, KAIST, South Korea
11email: {eetaekim,jelarum, damin321<EMAIL_ADDRESS>22institutetext: Immersive Reality and Intelligent Systems Lab, Chung-Ang
University, South Korea 22email<EMAIL_ADDRESS>
# MSCoTDet: Language-driven Multi-modal Fusion for Improved Multispectral
Pedestrian Detection

Taeheon Kim Equally contributed.11 Sangyun Chung††footnotemark:  11 Damin Yeom
11 Youngjoon Yu 11 Hak Gu Kim 22 Yong Man Ro Corresponding author.11

###### Abstract

Multispectral pedestrian detection is attractive for around-the-clock
applications due to the complementary information between RGB and thermal
modalities. However, current models often fail to detect pedestrians in
obvious cases, especially due to the modality bias learned from statistically
biased datasets. From these problems, we anticipate that maybe understanding
the complementary information itself is difficult to achieve from vision-only
models. Accordingly, we propose a novel Multispectral Chain-of-Thought
Detection (MSCoTDet) framework, which incorporates Large Language Models
(LLMs) to understand the complementary information at the semantic level and
further enhance the fusion process. Specifically, we generate text
descriptions of the pedestrian in each RGB and thermal modality and design a
Multispectral Chain-of-Thought (MSCoT) prompting, which models a step-by-step
process to facilitate cross-modal reasoning at the semantic level and perform
accurate detection. Moreover, we design a Language-driven Multi-modal Fusion
(LMF) strategy that enables fusing vision-driven and language-driven
detections. Extensive experiments validate that MSCoTDet improves
multispectral pedestrian detection.

###### Keywords:

Multispectral Chain-of-Thought Detection Language-driven Multi-modal Fusion
Multispectral Pedestrian Detection Large Language Model

##  1 Introduction

Multispectral pedestrian detection is the task of detecting pedestrians based
on different visual modalities (i.e., RGB and thermal) [20, 33, 62, 40, 37,
66, 67, 30]. Due to their complementary information, combining these
modalities improves pedestrian detection all day/night [28, 26, 70]. With the
advances in this field, the primary interest in multispectral pedestrian
detection has been focused on how to effectively fuse the complementary
information between the two modalities. Previous works investigated various
fusion methods in different stages of the network, which are often categorized
as early-fusion [57], mid-fusion [29, 31, 50], and late-fusion [57, 9]. These
methods demonstrated superior detection performance compared to standard
pedestrian detection [32, 15, 25, 21], especially in practical datasets that
contain all day/night scenarios.
Despite the progress in this task, there are still remaining problems that
need to be solved. From a recent study [31, 14], multispectral pedestrian
detection models are known to suffer from modality bias toward the thermal
modality, due to learning the statistical bias in datasets. In multispectral
pedestrian datasets, pedestrians always statistically co-occur with their
thermal signatures [31] as the thermal modality is generally robust all
day/night. Models trained on these datasets learn the statistical co-
occurrences of thermal signatures on pedestrians and detect pedestrians based
on the presence of thermal signatures. As a result, models often fail to
detect pedestrians that have weak thermal signatures (i.e., thermal-obscured
pedestrians), even though they are obviously visible in RGB [31]. This
phenomenon, also illustrated in Fig. 1, indicates that models often fail to
correctly fuse the complementary information between the two modalities.
We anticipate that understanding the complementary information itself may be
difficult to achieve from vision-only models because it requires a high level
of understanding of the intra- and inter-modality contexts at the semantic
level. On the other hand, language models are known to understand the semantic
contexts between different modalities [27, 7, 1] (e.g., image and text [69])
and elicit correct reasoning steps required for completing tasks [58, 59, 16,
68, 51].

Figure 1: Problem illustration and our motivation. (a) In multispectral
pedestrian datasets, thermal signatures always appear on pedestrians, as the
thermal modality can generally capture pedestrians all day/night. In these
datasets, thermal-obscured data is underrepresented. Models trained on such
datasets learn the statistical co-occurrences between pedestrians and their
thermal signatures. (b) As a result, models fail to detect pedestrians in
thermal-obscured data, even though obviously visible in RGB. (c) Our
motivation is to use Large Language Models (LLMs) to correctly combine
complementary information of RGB and thermal, and intervene in modality bias.

If the contextual elements (e.g., object category names, attributes, and the
overall scene) of different visual modalities can be represented in the same
semantic space through textual descriptions, Large Language Models (LLMs) [46,
4, 60, 54, 10, 24] can use their semantic understanding capabilities to
comprehend the context within intra- and inter-modalities. Moreover, as
language models are excellent few-shot learners that can perform complex
reasoning tasks [4, 55], a few task-specific exemplars can expand their
general knowledge to complete the task.
Based on these motivations, we propose the Multispectral Chain-of-Thought
Detection (MSCoTDet) framework, which incorporates Large Language Models
(LLMs) to enhance the fusion process. Specifically, MSCoTDet consists of three
parts: the vision branch, the language branch, and the Language-driven Multi-
modal Fusion (LMF). First, the vision branch consists of single-modal
detectors of the RGB and thermal modalities. Second, the language branch first
generates text descriptions of pedestrians detected in the vision branch,
utilizing Multi-modal Language Models (MLLMs) [1, 39, 71, 13]. The key idea of
MSCoTDet is the proposed Multispectral Chain-of-Thought (MSCoT) prompting,
human-like strategic reasoning inspired by the Chain-of-Thought [59] prevalent
in LLMs. Using textual descriptions from MLLMs, MSCoT first makes a decision
based on a single modality. Then, MSCoT makes logical deductions by building
cross-modal reasoning according to decisions from both modalities. This
ability to generate rationale across modalities is similar to human cognitive
processes, where different sensory streams are integrated into the parietal
cortex [3, 11, 12] to form a cohesive understanding of the whole perception.
By facilitating cross-modal reasoning using LLMs, MSCoT can achieve accurate
detection by intervening in modality biases like humans do. Then, LMF can
finally integrate the fused detections from both the vision and language
branches.

Figure 2: Comparisons between previous works and our method (MSCoTDet). (a)
Previous approaches centrally focused on mid-fusion methods, e.g., mid-fusing
features internally in the network. (b) There are few works via late fusion
that ensemble detections from independently trained single-modal detectors,
i.e., RGB and thermal detectors. (c) MSCoTDet (Ours) focuses on designing a
language branch that processes detection using Large Language Models (LLMs).
Then, our proposed Language-driven Multi-modal Fusion (LMF) enables fusing
vision-driven and language-driven detections.

Extensive experiment results show that our proposed method can intervene in
modality bias and improve the overall performance of multispectral pedestrian
detection models.
The main contributions of our paper are:

  1. 1.

To the best of our knowledge, this is the first work to integrate the Large
Language Models (LLMs) in multispectral pedestrian detection.

  2. 2.

We propose the Multispectral Chain-of-Thought Detection (MSCoTDet) framework,
which models a step-by-step process to facilitate cross-modal reasoning at the
semantic level and perform accurate detection.

  3. 3.

We propose Language-driven Multi-modal Fusion (LMF), a strategy that enables
the fusion of vision-driven detections and language-driven detections.

  4. 4.

Extensive experiments demonstrate that our proposed method can significantly
mitigate the modality bias and improve the overall performance of
multispectral pedestrian detectors.

##  2 Preliminaries

Before introducing our method, we review the late-fusion strategies in
multispectral pedestrian detection.

###  2.1 Late-fusion in Multispectral Pedestrian Detection

Rather than fusing modalities in the input stage (early-fusion [53, 56, 43])
or in the intermediate steps (mid-fusion [28, 50]), late-fusion [9, 8] is
based on detection ensembling, i.e., fusing the decision values. One of the
advantages of late fusion is that it can preserve single-modal models and,
therefore, has high flexibility to perform multi-modal fusion from diverse
models of different modalities [8, 42, 52, 63, 23, 47].
Late fusion methods have been also investigated in multispectral pedestrian
detection [9]. They first train single-modal detectors (i.e., RGB and thermal)
and then fuse the prediction scores (score fusion) and bounding boxes (box
fusion) which are the outputs from single-modal detectors. The final output
includes fused prediction scores and bounding boxes. We describe score fusion
and box fusion strategies below.
Score Fusion. The most common strategies to fuse prediction scores obtained
from single-modal models are averaging [37, 40], Non-maximum Suppression (NMS)
[9], and Probabilistic Ensembling (ProbEn [9]). Between them, we adapt
averaging and NMS, as we find them effective for our method (Section 3.4).
Averaging is straightforward, the fused score is determined by averaging
prediction scores estimated in each modality. Denote $s_{\scriptscriptstyle
RGB}$ and $s_{\scriptscriptstyle T}$ the prediction scores predicted for the
same object in different modalities (i.e., RGB and thermal). Then the
averaging score fusion between $s_{\scriptscriptstyle RGB}$ and
$s_{\scriptscriptstyle T}$ can be expressed as:

${s_{\scriptscriptstyle AVG}=\frac{s_{\scriptscriptstyle
RGB}+s_{\scriptscriptstyle T}}{2},}$ (1)

where $s_{\scriptscriptstyle AVG}$ denotes the fused score. On the other hand,
NMS compares the prediction scores estimated in each modality and votes for
the highest one, removing the lower scores. NMS score fusion between
$s_{\scriptscriptstyle RGB}$ and $s_{\scriptscriptstyle T}$ can be expressed
as:

${s_{\scriptscriptstyle NMS}=max(s_{\scriptscriptstyle
RGB},s_{\scriptscriptstyle T}),}$ (2)

where $s_{\scriptscriptstyle NMS}$ denotes the fused score by the NMS.
Box Fusion. Box fusion is to merge overlapping bounding boxes predicted from
different modalities (i.e., RGB and thermal). Chen et al.(2022)[9] suggests a
simple and effective way to probabilistically fuse boxes, which computes a
weighted average of boxes. The weights are given by the prediction scores,
implying that more confident detections should have a higher weight when
fusing boxes. Denote $b_{\scriptscriptstyle RGB}$ and $b_{\scriptscriptstyle
T}$ the bounding box coordinates predicted for the same object in different
modalities (i.e., RGB and thermal). Using the predictions scores
$s_{\scriptscriptstyle RGB}$, and $s_{\scriptscriptstyle T}$, the weighted-
averaging box fusion between $b_{\scriptscriptstyle RGB}$ and
$b_{\scriptscriptstyle T}$ can be expressed as:

${b_{\scriptscriptstyle s\text{-}avg}=\frac{b_{\scriptscriptstyle RGB}\times
s_{\scriptscriptstyle RGB}+b_{\scriptscriptstyle T}\times
s_{\scriptscriptstyle T}}{s_{\scriptscriptstyle RGB}+s_{\scriptscriptstyle
T}},}$ (3)

where $b_{\scriptscriptstyle s\text{-}avg}$ denotes the fused bounding boxes.

##  3 Proposed Method

###  3.1 Overall Architecture

Figure 3: Overall architecture of proposed Multispectral Chain-of-Thought
detection (MSCoTDet) framework including vision branch, language branch, and
language-driven multi-modal fusion.

The overall architecture of our proposed pedestrian detection network is shown
in Fig. 3. It consists of two branches that perform distinct detection
processes, the vision branch and the language branch. Detections from these
branches are fused to output final detections. We briefly introduce the
process of each branch and our fusion strategy.
Vision branch. Given the input of RGB and thermal images
$X_{\scriptscriptstyle RGB}$ and $X_{\scriptscriptstyle T}$, the vision branch
first produces single-modal (i.e., RGB or thermal) detections. Denote the
prediction scores as $S_{\scriptscriptstyle RGB}$, $S_{\scriptscriptstyle T}$
and bounding boxes as $B_{\scriptscriptstyle RGB}$, $B_{\scriptscriptstyle T}$
produced from each single-modal detector. The single-modal detections in the
vision branch are processed through the language branch.
Language Branch. The language branch first generates text descriptions of
pedestrians and performs detections based on them. Toward this goal, we
leverage Multi-modal Large Language Models (MLLMs), which are capable of
generating comprehensive descriptions in both RGB and thermal images. Based on
these descriptions, we propose the Multispectral Chain-of-Thought (MSCoT)
prompting, which prompts the LLM to conduct reasoning steps and produce
detections (Section 3.3). Detections from MSCoT include single-modal (i.e.,
RGB and thermal) prediction scores $S_{\scriptscriptstyle
RGB}^{\scriptscriptstyle L}$, $S_{\scriptscriptstyle T}^{\scriptscriptstyle
L}$, and fused prediction scores $S_{\scriptscriptstyle F}^{\scriptscriptstyle
L}$. In Sections 3.2 and 3.3, we describe the processes to produce these
detections in detail.
Language-driven Multi-modal Fusion (LMF). Vision-driven detection, language-
driven detection, and vision-language fusion are performed to produce final
detections, $S_{\scriptscriptstyle LMF}$ and $B_{\scriptscriptstyle LMF}$. In
Section 3.4, we describe the fusion strategies in detail.

###  3.2 Cross-modal Pedestrian Description Generation

The first step of the language branch is to generate text descriptions of
pedestrians, both from the RGB and thermal images. We prompt Multi-modal Large
Language Models (MLLMs [1]) to describe pedestrian regions in the image, which
can be identified by their bounding box coordinates $B_{\scriptscriptstyle
RGB}$, $B_{\scriptscriptstyle T}$ obtained from single-modal detectors.
However, there are two challenges, which we introduce below with their
solutions.
Detection Pairing. The first challenge is due to the misalignment in
multispectral data [26, 29]. The same pedestrian often appears in different
locations within the RGB and thermal image pairs, as their image sensors may
have different Field-of-View or frame rates [29]. Thus sometimes we need to
guide MLLMs to make descriptions based on different image regions in RGB and
thermal images for the same pedestrians. Our strategy is to find the detection
pairs from the single-modal detections (e.g., bounding boxes) that belong to
the same pedestrians and provide this information to the MLLM.
Although bounding boxes representing the same pedestrian can have different
coordinates in different modalities, still they will contain highly
overlapping areas in the images, and those boxes will have large values of
IoUs (Intersection-of-Unions). Specifically, for the $i$-th bounding box
$b_{\scriptscriptstyle i,\scriptscriptstyle m}$ in bounding boxes
$B_{\scriptscriptstyle m}$, obtained from the single-modal detector in the $m$
modality ($m$ is either RGB or thermal: T), we aim to find the bounding box
from $B_{\scriptscriptstyle m^{c}}$, in the opposite modality $m^{c}$ that
corresponds to the same pedestrian. To this end, we compute the IoU value
between $b_{\scriptscriptstyle i,\scriptscriptstyle m}$ across all bounding
boxes in $B_{\scriptscriptstyle m^{c}}$, and find the box that has the highest
IoU value, e.g., $b_{\scriptscriptstyle j,\scriptscriptstyle m^{c}}$, with
$b_{\scriptscriptstyle i,\scriptscriptstyle m}$. Here, we only consider the
boxes with IoU values above the IoU threshold $\tau$. With indices $i$ and
$j$, we can find a pair of the prediction scores $s_{\scriptscriptstyle
i,\scriptscriptstyle m}$ and $s_{\scriptscriptstyle j,\scriptscriptstyle
m^{c}}$ that belong to the same pedestrian. We call such tuples
($b_{\scriptscriptstyle i,\scriptscriptstyle m}$,$b_{\scriptscriptstyle
j,\scriptscriptstyle m^{c}}$) and ($s_{\scriptscriptstyle i,\scriptscriptstyle
m}$,$s_{\scriptscriptstyle j,\scriptscriptstyle m^{c}}$) as the "paired
detection". When all IoU values are under the IoU threshold $\tau$, this means
that there are no boxes in $B_{\scriptscriptstyle m^{c}}$ that correspond to
the same pedestrian and $b_{\scriptscriptstyle i,\scriptscriptstyle m}$ can
not find a pair. Such cases can occur at nighttime or in thermal-obscured
scenarios where the pedestrian is detected by only one modality. In this case,
we override the value of $b_{\scriptscriptstyle i,\scriptscriptstyle m}$ and
$s_{\scriptscriptstyle i,\scriptscriptstyle m}$ to the opposite modality,
making pairs of ($b_{\scriptscriptstyle i,\scriptscriptstyle
m}$,$b_{\scriptscriptstyle i,\scriptscriptstyle m}$) and
($s_{\scriptscriptstyle i,\scriptscriptstyle m}$,$s_{\scriptscriptstyle
i,\scriptscriptstyle m}$). Iterating this process for over all bounding boxes
in $B_{\scriptscriptstyle m}$ and $B_{\scriptscriptstyle m^{c}}$ (i.e.,
$B_{\scriptscriptstyle RGB}$ and $B_{\scriptscriptstyle T}$) allows us to
obtain a set of paired detections $B^{paired}$ and $S^{paired}$. Denote the
Detection Pairing Module as a function $DPair(\cdot)$ with respect to inputs
$B_{\scriptscriptstyle RGB}$, $B_{\scriptscriptstyle T}$,
$S_{\scriptscriptstyle RGB}$, and $S_{\scriptscriptstyle T}$, which indicate
bounding boxes and prediction scores obtained from RGB and thermal single-
modal detectors, respectively. The inputs and outputs of $DPair(\cdot)$ can be
represented as:

${S^{paired},B^{paired}=DPair(B_{\scriptscriptstyle RGB},B_{\scriptscriptstyle
T},S_{\scriptscriptstyle RGB},S_{\scriptscriptstyle T};\ \tau),}$ (4)

where $\tau$ indicates an IoU threshold. We set this value to 0.5, the
standard value used in the object detection task. $B^{paired}$ is used to
guide MLLMs the location of when making descriptions in RGB and thermal
images. $S^{paired}$ is used when fusing prediction scores from vision and
language branches in Section 3.4. The complete algorithm of the Detection
pairing Module is described in Algorithm 2 in the supplementary material.
Visual Crop & Mark (VCM). Another challenge when using MLLMs for making
descriptions is the low accuracy of MLLMs on small-scale objects, i.e.,
objects occupying a small area in the image. To solve this issue, we refer to
a recent method [65] that improves the performance of MLLMs on small visual
details by pre-processing the input image with image cropping. Similarly, we
zoom in on the target pedestrians and crop the image around it, such that
visual information irrelevant to the pedestrian is removed. We find such a
pre-processing strategy is shown to significantly improve the accuracy of
MLLMs in describing small-scale pedestrians, near the level comparable to
recognizing large-scale pedestrians. Since the process requires simple image
cropping processes, we elaborate on the details in the supplementary material.
In addition to cropping, we visually guide the MLLM to describe a specific
pedestrian by drawing a 1-pixel width "green" color (with RGB values
(0,255,0)) box around a target pedestrian. We use the "green" color to draw
the boxes because it is most distinctly in hue from human skin color, thus
effective for distinguishing humans from the background. Denote this the
Visual Crop & Mark process as $VCM(\cdot)$, then the pre-processed RGB and
thermal images $X_{\scriptscriptstyle RGB}^{\scriptscriptstyle VCM}$ and
$X_{\scriptscriptstyle T}^{\scriptscriptstyle VCM}$ can be written as:

${X_{\scriptscriptstyle RGB}^{\scriptscriptstyle VCM}\leftarrow
VCM(X_{\scriptscriptstyle RGB},B^{paired}),X_{\scriptscriptstyle
T}^{\scriptscriptstyle VCM}\leftarrow VCM(X_{\scriptscriptstyle
T},B^{paired}),}$ (5)

where $X_{\scriptscriptstyle RGB}$ and $X_{\scriptscriptstyle T}$ are RGB and
thermal input images, and $B^{paired}$ is the set of the paired bounding
boxes. When pre-processing images in $X_{\scriptscriptstyle RGB}$ and
$X_{\scriptscriptstyle T}$, bounding boxes of the corresponding modality in
$B^{paired}$ are used only.

Figure 4: Visualized details of the (a) Detection Pairing Module and the (b)
Visual Crop & Mark Module. (a) The Detection Pairing Module gets single-modal
detections from the vision branch and then finds the detection pairs that
belong to the same pedestrians, e.g., $b_{1}^{paired}$=
($b_{1,\scriptscriptstyle RGB}^{paired}$,$b_{1,\scriptscriptstyle
T}^{paired}$) and $s_{1}^{paired}$= ($s_{1,\scriptscriptstyle
RGB}^{paired}$,$s_{1,\scriptscriptstyle T}^{paired}$). Through the iteration
of pedestrians, the module produces the sets of paired detections $B^{paired}$
and $S^{paired}$. (b) The Visual Crop & Mark Module gets $B^{paired}$ as
inputs, and output the sets of pre-processed images $X_{\scriptscriptstyle
RGB}^{\scriptscriptstyle VCM}$ and $X_{\scriptscriptstyle
T}^{\scriptscriptstyle VCM}$.

Generating Descriptions with MLLMs. For each pre-processed image
$x_{\scriptscriptstyle RGB}^{\scriptscriptstyle VCM}\in X_{\scriptscriptstyle
RGB}^{\scriptscriptstyle VCM}$ and $x_{\scriptscriptstyle
T}^{\scriptscriptstyle VCM}\in X_{\scriptscriptstyle T}^{\scriptscriptstyle
VCM}$, we prompt the MLLM as the following. For the RGB image, "In this RGB
image, what is in the green box?" and for the thermal image, "In this thermal
image, what is in the green box?". Denote these prompts as
$p_{\scriptscriptstyle RGB}$, and $p_{\scriptscriptstyle T}$. Then, we can
generate their corresponding text descriptions $d_{\scriptscriptstyle RGB}$
and $d_{\scriptscriptstyle T}$ such as:

${d_{\scriptscriptstyle RGB}\leftarrow MLLM(x_{\scriptscriptstyle
RGB}^{\scriptscriptstyle VCM};\ p_{\scriptscriptstyle
RGB}),d_{\scriptscriptstyle T}\leftarrow MLLM(x_{\scriptscriptstyle
T}^{\scriptscriptstyle VCM};\ p_{\scriptscriptstyle T}),}$ (6)

where $MLLM(\cdot)$ denotes the process of generating text descriptions from
an MLLM, given the pre-processed image as input. Text descriptions are
generated for all $x_{\scriptscriptstyle RGB}^{\scriptscriptstyle VCM}\in
X_{\scriptscriptstyle RGB}^{\scriptscriptstyle VCM}$ and
$x_{\scriptscriptstyle T}^{\scriptscriptstyle VCM}\in X_{\scriptscriptstyle
T}^{\scriptscriptstyle VCM}$, producing sets of descriptions
$D_{\scriptscriptstyle RGB}$ and $D_{\scriptscriptstyle T}$.

###  3.3 Multispectral Chain-of-Thought Prompting

A complex reasoning process is required to properly use the complementary
information of the RGB and thermal modalities based on their textual
information. Our key idea is to leverage LLM’s semantic understanding
capabilities to model reasoning steps to complete the task. To this end, we
refer to the Chain-of-Thought (CoT) prompting [59] technique, which
significantly improves LLM’s reasoning ability in complex NLP tasks. CoTs add
a series of intermediate reasoning steps into LLMs and generate rationales
when predicting answers. Motivated by the late-fusion process in multispectral
pedestrian detection, we propose a Multispectral Chain-of-Thought (MSCoT)
prompting chain. Given the text descriptions $d_{\scriptscriptstyle RGB}\in
D_{\scriptscriptstyle RGB}$ and $d_{\scriptscriptstyle T}\in
D_{\scriptscriptstyle T}$ of a RGB image and a thermal image, MSCoT first
prompts the LLM to produce single-modal prediction scores
$s_{\scriptscriptstyle RGB}^{\scriptscriptstyle L}$, and
$s_{\scriptscriptstyle T}^{\scriptscriptstyle L}$ and then predict a fused
prediction score $s_{\scriptscriptstyle F}^{\scriptscriptstyle L}$ based on
rationale chains. Given the descriptions $d_{\scriptscriptstyle RGB}$, and
$d_{\scriptscriptstyle T}$ of RGB and thermal images as the context, we design
the following prompt chains:

Generating Single-modal Prediction.
Context: RGB image description $d_{\scriptscriptstyle RGB}$, thermal image
description $d_{\scriptscriptstyle T}$.
Prompt: First, predict what is in the RGB image based on
$d_{\scriptscriptstyle RGB}$. And predict what is in the thermal image based
on $d_{\scriptscriptstyle T}$. Please answer in the format : [class,
prediction score]

From the above prompt, the LLM outputs single-modal prediction scores
$s_{\scriptscriptstyle RGB}^{\scriptscriptstyle L}$, and
$s_{\scriptscriptstyle T}^{\scriptscriptstyle L}$. Additionally providing
$s_{\scriptscriptstyle RGB}^{\scriptscriptstyle L}$, and
$s_{\scriptscriptstyle T}^{\scriptscriptstyle L}$ as the context, the
following prompts for generating fused prediction scores and multispectral
rationales are:

Generating Multispectral Rationales and Prediction.
Context: RGB image description: $d_{\scriptscriptstyle RGB}$, thermal image
description: $d_{\scriptscriptstyle T}$, answers: $s_{\scriptscriptstyle
RGB}^{\scriptscriptstyle L}$, and $s_{\scriptscriptstyle
T}^{\scriptscriptstyle L}$.
Prompt: Based on the descriptions and your answers, predict what is in these
aligned RGB and thermal images. Please answer in the format: [class,
prediction score]

Then the LLM outputs fused prediction scores $s_{\scriptscriptstyle
F}^{\scriptscriptstyle L}$. For all steps, the prediction scores
$s_{\scriptscriptstyle RGB}^{\scriptscriptstyle L}$, $s_{\scriptscriptstyle
T}^{\scriptscriptstyle L}$, and $s_{\scriptscriptstyle F}^{\scriptscriptstyle
L}$ are assigned only if the class outputs are "Person". If not, the
prediction scores are assigned as zero. Our implementation for MSCoT is
achieved by fine-tuning ChatGPT-3.5 [44] with about 50 training samples. The
implementation details are described in detail in Section 4.2.

${s_{\scriptscriptstyle RGB}^{\scriptscriptstyle L},s_{\scriptscriptstyle
T}^{\scriptscriptstyle L},s_{\scriptscriptstyle F}^{\scriptscriptstyle
L}=MSCoT(d_{\scriptscriptstyle RGB},\ d_{\scriptscriptstyle T}).}$ (7)

Iterating this process over all $d_{\scriptscriptstyle RGB}\in
D_{\scriptscriptstyle RGB}$ and $d_{\scriptscriptstyle T}\in
D_{\scriptscriptstyle T}$ obtains the prediction scores:
$S_{\scriptscriptstyle RGB}^{\scriptscriptstyle L}$, $S_{\scriptscriptstyle
T}^{\scriptscriptstyle L}$, and $S_{\scriptscriptstyle F}^{\scriptscriptstyle
L}$.

Figure 5: Visualized process of our proposed Multispectral Chain-of-Thought
(MSCoT) prompting.

###  3.4 Language-driven Multi-modal Fusion (LMF)

We propose to ensemble the detections of the vision branch and the language
branch at the last stage of the proposed network (Fig. 3).
Vision-driven Detection. For the vision branch, denote $S_{\scriptscriptstyle
RGB}^{\scriptscriptstyle V}$ as the prediction scores from $S^{paired}$, and
$B_{\scriptscriptstyle RGB}^{\scriptscriptstyle V}$ as the bounding boxes from
$B^{paired}$ that corresponds to the RGB modality, similar for
$S_{\scriptscriptstyle T}^{\scriptscriptstyle V}$ and $B_{\scriptscriptstyle
T}^{\scriptscriptstyle V}$. Then, the vision-driven detections
$S_{\scriptscriptstyle F}^{\scriptscriptstyle V}$ (predictions scores) and
$B_{\scriptscriptstyle F}^{\scriptscriptstyle V}$ (bounding boxes) can be
produced as:

${S_{\scriptscriptstyle F}^{\scriptscriptstyle V}=max(S_{\scriptscriptstyle
RGB}^{\scriptscriptstyle V}\ ,\ S_{\scriptscriptstyle T}^{\scriptscriptstyle
V}),\ B_{\scriptscriptstyle F}^{\scriptscriptstyle
V}=\frac{B_{\scriptscriptstyle RGB}^{\scriptscriptstyle V}\odot
S_{\scriptscriptstyle RGB}^{\scriptscriptstyle V}+B_{\scriptscriptstyle
T}^{\scriptscriptstyle V}\odot S_{\scriptscriptstyle T}^{\scriptscriptstyle
V}}{S_{\scriptscriptstyle RGB}^{\scriptscriptstyle V}+S_{\scriptscriptstyle
T}^{\scriptscriptstyle V}},}$ (8)

where $\odot$ denotes the Hadamard product (element-wise multiplication). NMS
(Eq. 2) and weighted-averaging (Eq. 3) are applied for score-fusion and box-
fusion.
Language-driven Detection. Next, for the language branch, fused bounding boxes
$B_{\scriptscriptstyle F}^{\scriptscriptstyle L}$ can be produced as:

${B_{\scriptscriptstyle F}^{\scriptscriptstyle L}=\frac{B_{\scriptscriptstyle
RGB}^{\scriptscriptstyle V}\odot S_{\scriptscriptstyle
RGB}^{\scriptscriptstyle L}+B_{\scriptscriptstyle T}^{\scriptscriptstyle
V}\odot S_{\scriptscriptstyle T}^{\scriptscriptstyle L}}{S_{\scriptscriptstyle
RGB}^{\scriptscriptstyle L}+S_{\scriptscriptstyle T}^{\scriptscriptstyle
L}},}$ (9)

given the bounding boxes $B_{\scriptscriptstyle RGB}^{\scriptscriptstyle V}$,
$B_{\scriptscriptstyle T}^{\scriptscriptstyle V}$ obtained single-modal
detectors. Weighted-averaging (Eq. 3) is applied for box-fusion. The fused
prediction scores $S_{\scriptscriptstyle F}^{\scriptscriptstyle L}$ are
produced by the MSCoT (Section 3.3).
Vision-Language Fusion. Lastly, final detections are produced as:

${S_{\scriptscriptstyle LMF}=avg(S_{\scriptscriptstyle F}^{\scriptscriptstyle
V},S_{\scriptscriptstyle F}^{\scriptscriptstyle L}),\ B_{\scriptscriptstyle
LMF}=\frac{S_{\scriptscriptstyle F}^{\scriptscriptstyle V}\odot
B_{\scriptscriptstyle F}^{\scriptscriptstyle V}+S_{\scriptscriptstyle
F}^{\scriptscriptstyle L}\odot B_{\scriptscriptstyle F}^{\scriptscriptstyle
L}}{S_{\scriptscriptstyle F}^{\scriptscriptstyle V}+S_{\scriptscriptstyle
F}^{\scriptscriptstyle L}},}$ (10)

where $S_{\scriptscriptstyle LMF}$ and $B_{\scriptscriptstyle LMF}$ denote
final prediction scores and bounding boxes. Averaging (Eq. 2) and weighted-
averaging (Eq. 3) are applied for score-fusion and box-fusion.
Design Choice. For score-fusion, we adapted the NMS method (Eq. 2) in the
vision-driven detection. NMS compares the prediction scores estimated in each
modality and votes for the highest one, removing the lower score. We apply
such a strategy because higher confidence scores generally occur when there is
more useful information for detecting the pedestrian. From this strategy, we
can obtain higher prediction scores for true-positive detections in most
cases. However, it can also increase the prediction scores for false
positives. As MSCoT can comprehend very low prediction scores for false
positives, we adapt the averaging strategy (Eq. 1) in the vision-language
fusion to refine over-confident false positives from the vision-driven
detections. For box-fusion, we adapted the weighting average (Eq. 3) method in
all processes, implying that more confident detections should have a higher
weight when fusing. In Section 6.2, we conduct an ablation study on these
design choices.

##  4 Experimental Setup

###  4.1 Dataset and Evaluation Metric

The experiments are conducted on multispectral pedestrian datasets: Teledyne
FLIR Free ADAS Thermal Dataset v2.0.0 [17], CVC-14 [19], ROTX-MP [31]. The
FLIR [17] dataset consists of RGB and thermal image pairs with an image
resolution of $640\times 512$. For a fair comparison with previous studies
[64, 49, 33, 31], we use the aligned version of FLIR, proposed by [64]. This
version filters out misaligned images, containing well-aligned 4,129 training
and 1,013 test RGB and thermal image pairs. For convenience, we call this
version of the dataset as FLIR. Following previous works, we evaluate both day
and night images and report the performance on the entire test set (‘All’). In
contrast to FLIR, the CVC-14 [19] dataset often contains heavily misaligned
pairs of RGB and thermal images. The train and test set each contains 3,618
and 1,433 grayscale and thermal images with a $640\times 471$ resolution. We
evaluate daytime (‘Day’), nighttime (‘Night’), and the total (‘All’) test
images separately, following previous works [29, 31].
Lastly, we evaluate our method on the ROTX-MP [31] dataset, which consists of
1,000 test images of mainly thermal-obscured pedestrians. We use the models
trained from FLIR and CVC-14 to test on ROTX-MP, as the purpose of this
dataset is to evaluate models when there is a substantial distribution
difference between the train and test splits. For the evaluation metric, we
use the Average Precision (AP $\uparrow$). These experimental settings are the
same as the original paper [31].

###  4.2 Implementation Detail

Single-modal Detectors. For the single-modal detectors, we use the Co-DETR
[72] model and train RGB images and thermal images separately. Co-DETR is
based on the DETR (DEtection TRansformer) [5] architecture, enhanced by
collaborative learning with multiple parallel auxiliary heads integrated into
the output of the transformer encoder. We train the single-modal detectors
with RGB and thermal images from FLIR [17] and CVC-14 [19] training data,
respectively. For optimizing Co-DETR, we use the same setting in the original
paper [72], using AdamW [41] optimizer with an initial learning rate of 1e-4
and weight decay of 1e-4. All models are trained on an eight NVIDIA A6000 GPU
for 16 epochs with a batch size of 16.
Large Language Models (LLMs). We use two types of Large Language Models. First
is the Multimodal Large Language Model (MLLM), which we use to generate text
descriptions for RGB and thermal pedestrian images (Section 3.2). For the
MLLM, we use the pre-trained model of the ChatGPT-vision (GPT-4V) [1] API,
provided by the OpenAI developer platform. ChatGPT-vision API service is not
free, therefore we paid the company to use the service. A total of $\$500$ is
spent for our experiments.
The second is the chatbot version, ChatGPT-3.5 (GPT-3.5) API [44], also
provided by the OpenAI developer platform. We fine-tuned the GPT-3.5 model for
the Multispectral Chain-of-Thought (MSCoT) prompting (Section 3.3) we
proposed. Using the fine-tuning module provided on the GPT official website
[45], we trained GPT-3.5 API with our high-quality selection pairs consisting
of RGB, thermal descriptions, rationale, and prediction scores. The official
document for GPT-3.5 [45] recommends training 50 samples for fine-tuning the
model, and we utilized a total of 50 training samples. We provide our examples
in the supplementary material.

###  4.3 Comparison Model

We compare our method with five multispectral pedestrian detection models: 1)
Halfway Fusion [40], 2) Cross-modality Fusion Transformer (CFT) [49], 3) Kim
et al. [29], 4) ProbEn [9], and 5) Causal Mode Multiplexer (CMM) [31]. For
these methods, we used the same experimental settings as their original
papers. More details of these methods and their settings are described in the
supplementary material.

##  5 Experimental Result

###  5.1 Result on FLIR, and CVC-14

Figure 6: Visualized detections on the ROTX-MP [31] (Top) and FLIR [17]
(Bottom) datasets. (Top): MSCoTDet can detect pedestrians over the window
($\downarrow$), in which their thermal signatures are absent due to the
window. In contrast, other models [31, 9] fail to detect these pedestrians.
(Bottom): Moreover, other models [31, 9] create false-positive detections on
the fireplug ($\downarrow$) due to its thermal signature similar-looking to
pedestrians. In contrast, MSCoTDet (Ours) makes correct predictions.

We train and test on FLIR [17], CVC-14 [19] datasets to evaluate whether our
method performs well on general multispectral pedestrian data. We report the
detection performance of different models on these datasets. For the
evaluation metrics, we follow previous studies on multispectral pedestrian
detection to measure the detection performances on these datasets and make a
fair comparison. Specifically, we use average precision (AP) on FLIR (‘All’).
For CVC-14, we use the log-average miss rate (MR) for evaluating models and
report the performance on daytime (‘Day’) and nighttime (‘Night’), and the
entire set (‘All’) of images separately. Note that lower MR (MR $\downarrow$)
and higher AP (AP $\uparrow$) values indicate better detection performance.
Table 1 (Left) shows the experimental results on the FLIR and CVC-14 test set.
On FLIR (‘All’), MSCoTDet achieves the highest AP (90.39 AP), which
outperforms other methods by at least 2.59 AP. Such results indicate the
effectiveness of MSCoTDet on well-aligned data. On CVC-14, MSCoTDet
demonstrates the lowest miss rate among models in (‘Day’), and (‘All’), with a
value of 6.69 MR and 10.39 MR, respectively. Our method surpasses other models
by at least 12.12 MR for (‘Day’) and 6.74 MR for (‘All’). As (‘All’) indicates
the entire dataset, our method achieves the best performance also on CVC-14.
Such results illustrate that MSCoTDet is also effective on a dataset that
largely contains misaligned multispectral data.

Table 1: Detection performance on the FLIR [17], CVC-14 [19] (Left), and ROTX-
MP [31] (Right). (Left): We train and test on FLIR, CVC-14 datasets. (Right):
We train models on FLIR, CVC-14, and evaluate models on the ROTX-MP dataset.
We adapt this experimental setting following the original paper [31] to
evaluate models when there is a substantial distribution difference between
train and test sets. We compare our proposed method with different
multispectral pedestrian detection models [40, 49, 29, 9, 31] recently
proposed. The best results are highlighted.

1-84 17-224 Train | FLIR | CVC-14 | Modality Used |  | Train | FLIR | CVC-14 | Modality Used
---|---|---|---|---|---|---|---|---
Test | FLIR | CVC-14 |  | Test | ROTX-MP | ROTX-MP
Metric | AP ($\uparrow$) | MR ($\downarrow$) |  | Metric | AP ($\uparrow$) | AP ($\uparrow$)
Method | All | Day | Night | All | RGB | Thermal | Language |  | Method | All | All | RGB | Thermal | Language
Halfway Fusion [40] | 75.85 | 36.29 | 26.29 | 31.99 | ✓ | ✓ |  |  | Halfway Fusion [40] | 36.95 | 8.8 | ✓ | ✓ |
CFT [49] | 84.10 | 18.81 | 25.25 | 21.83 | ✓ | ✓ |  |  | CFT [49] | 3.64 | 8.58 | ✓ | ✓ |
Kim et al. [29] | 84.67 | 23.87 | 11.08 | 18.7 | ✓ | ✓ |  |  | Kim et al. [29] | 21.69 | 13.36 | ✓ | ✓ |
CMM [31] | 87.8 | 27.81 | 7.71 | 17.13 | ✓ | ✓ |  |  | CMM [31] | 57.09 | 34.96 | ✓ | ✓ |
ProbEn [9] | 86.74 | 23.01 | 21.02 | 22.23 | ✓ | ✓ |  |  | ProbEn [9] | 17.2 | 16.66 | ✓ | ✓ |
1-8 17-22 MSCoTDet (Ours) | 90.39 | 6.69 | 13.69 | 10.39 | ✓ | ✓ | ✓ |  | MSCoTDet (Ours) | 67.33 | 57.65 | ✓ | ✓ | ✓
1-84 17-224 |  |  |  |  |  |  |  |  |  |  |  |  |  |

###  5.2 Result on ROTX-MP

Experimenting with models on the ROTX-MP [31] dataset is to evaluate modality
bias and their generalizability when there is a substantial distributional
difference in the train and test data. Following the original work [31], we
train models on general datasets (FLIR [17], and CVC-14 [19]) and test models
on ROTX-MP. The results are in Table 1 (Right). MSCoTDet achieves the best
performance on ROTX-MP for both cases when trained on FLIR and CVC-14. When
trained from FLIR, MSCoTDet achieves the highest AP (67.33 AP), which
outperforms other methods by at least 10.24 AP. When trained from CVC-14,
MSCoTDet achieves the highest AP (57.65 AP), which outperforms other methods
by at least 22.69 AP. Such results indicate that MSCoTDet has a better ability
to intervene in modality bias compared to other models.

##  6 Ablation Study

###  6.1 Effect of Using Language Models

We conduct an ablation study to evaluate the effectiveness of using language
models for our framework. For the experiment, we compare MSCoTDet and three
detection models: (1) ‘RGB only’, (2) ‘Thermal only’, and (3) ‘Late-fusion’.
First, the ‘RGB only’, and the ‘Thermal only’ model indicate single-modal
detection in the vision branch of MSCoTDet. Second, ‘Late-fusion’ is in which
the single-modal detections of ‘RGB only’ and ‘Thermal only’ are fused, i.e.,
equivalent to the vision branch equipped with vision-driven detection of our
framework. Thus, to measure the effect of using language models, our main
focus is to compare ‘Late-fusion’ with MSCoTDet. The results are shown in
Table 2. Our method achieves higher performance than ‘Late-fusion’ in all FLIR
[17], CVC-14 [19], and ROTX-MP [31]. The results on the FLIR and CVC-14
indicate that using language models can improve the overall performance of
multispectral pedestrian detectors in general datasets. Also, the results on
ROTX-MP indicate that using language models can significantly improve the
generalizability of multispectral pedestrian detectors, and effectively
mitigate the modality bias.

Table 2: Ablation study on the effect of using language models. Our focus is
to compare MSCoTDet with ‘Late-fusion’, the vision branch equipped with
vision-driven detection of MSCoTDet.

1-84 17-224 Train | FLIR | CVC-14 | Modality Used |  | Train | FLIR | CVC-14 | Modality Used
---|---|---|---|---|---|---|---|---
Test | FLIR | CVC-14 |  | Test | ROTX-MP | ROTX-MP
Metric | AP ($\uparrow$) | MR ($\downarrow$) |  | Metric | AP ($\uparrow$) | AP ($\uparrow$)
Method | All | Day | Night | All | RGB | Thermal | Language |  | Method | All | All | RGB | Thermal | Language
RGB only | 76.06 | 31.55 | 40.10 | 36.07 | ✓ |  |  |  | RGB only | 54.20 | 31.05 | ✓ |  |
Thermal only | 85.73 | 43.24 | 33.44 | 36.07 |  | ✓ |  |  | Thermal only | 19.33 | 9.10 |  | ✓ |
Late-fusion | 88.60 | 18.33 | 18.86 | 21.57 | ✓ | ✓ |  |  | Late-fusion | 58.95 | 44.72 | ✓ | ✓ |
1-8 17-22 MSCoTDet (Ours) | 90.39 | 6.69 | 13.69 | 10.39 | ✓ | ✓ | ✓ |  | MSCoTDet (Ours) | 67.33 | 57.65 | ✓ | ✓ | ✓
1-84 17-224 |  |  |  |  |  |  |  |  |  |  |  |  |  |

###  6.2 Effect of the Fusion Strategy

We conduct an ablation study to validate the design choice of our fusion
strategy in Section 3.4. For score-fusion, our fusion strategy adapts the NMS
(Eq. 2) in the vision-driven detection (‘V’) and averaging (Eq. 1) strategy in
the vision-language fusion (‘VL’), respectively. Denote the averaging strategy
as ‘Avg’ and NMS as ‘Max’. For the comparison study, we conducted experiments
on the (1) Avg, Avg, (2) Avg, Max, and (3) Max, Max fusion strategies in each
vision-driven detection (‘V’) and vision-language fusion (‘VL’), respectively.
We do not compare different score-fusion in the language branch, as we use our
proposed method: MSCoT (Section 3.3) for fusing prediction scores. All other
conditions are kept the same. The results are shown in Table 6.2. The Max, Avg
strategy consistently performs the best in all test sets.
For box-fusion, we adapt weighted averaging (Eq. 3) for all vision-driven
detection (‘V’), language-driven detection (‘L’), and vision-language fusion
(‘VL’). Denote the weighted averaging strategy as ‘s-avg’. For the comparison
study, we consider the NMS box fusion, denoted as ‘argmax’, and compare
between a total of 8 combinations (either s-avg or max in three parts). All
other conditions are kept the same as our MSCoTDet design during all
experiments. The results are shown in Table 6.2. Applying s-avg for all V, L,
and VL shows the best performance for most test cases.

Table 3: Ablation study on the score-fusion strategy. We compare the 1) Avg,
Avg, 2) Avg, Max, 3) Max, Max, and 4) Max, Avg strategies in the vision-driven
detection (‘V’) and in the vision-language fusion (‘VL’).

1-64 15-184 Train | FLIR | CVC-14 |  |  | Train | FLIR | CVC-14
---|---|---|---|---|---|---|---
Test | FLIR | CVC-14 | Test | ROTX-MP | ROTX-MP
Method | AP ($\uparrow$) | MR ($\downarrow$) | Method | AP ($\uparrow$) | AP ($\uparrow$)
V | VL | All | Day | Night | All | V | VL | All | All
Avg | Avg | 89.47 | 10.98 | 17.38 | 16.12 | Avg | Avg | 66.22 | 56.69
Avg | Max | 88.31 | 10.26 | 18.53 | 15.71 | Avg | Max | 66.27 | 56.60
Max | Max | 89.94 | 8.83 | 17.54 | 14.07 | Max | Max | 66.31 | 56.02
Max | Avg | 90.39 | 6.69 | 13.69 | 10.39 | Max | Avg | 67.33 | 57.65
1-64 15-184 |  |  |  |  |  |  |  |  |

Table 4: Ablation study on the box-fusion strategy. We experiment argmax and
s-avg for each vision-driven detection (‘V’), language-driven detection (‘L’),
and vision-language fusion (‘VL’).

1-74 15-194 Train | FLIR | CVC-14 |  |  |  |  | Train | FLIR | CVC-14
---|---|---|---|---|---|---|---|---|---
Test | FLIR | CVC-14 |  |  |  | Test | ROTX-MP | ROTX-MP
Method | AP ($\uparrow$) | MR ($\downarrow$) |  |  |  | Method | AP ($\uparrow$) | AP ($\uparrow$)
V | L | VL | All | Day | Night | All |  |  |  | V | L | VL | All | All
argmax | argmax | argmax | 89.49 | 7.56 | 17.64 | 14.00 |  |  |  | argmax | argmax | argmax | 66.10 | 56.97
s-avg | argmax | argmax | 89.79 | 8.03 | 15.28 | 12.63 |  |  |  | s-avg | argmax | argmax | 66.29 | 56.95
argmax | s-avg | argmax | 89.83 | 8.32 | 15.39 | 13.31 |  |  |  | argmax | s-avg | argmax | 67.03 | 56.90
s-avg | s-avg | argmax | 90.32 | 7.60 | 15.14 | 12.54 |  |  |  | s-avg | s-avg | argmax | 67.24 | 57.13
argmax | argmax | s-avg | 89.49 | 6.25 | 18.51 | 14.18 |  |  |  | argmax | argmax | s-avg | 66.11 | 57.01
s-avg | argmax | s-avg | 90.32 | 7.63 | 15.19 | 11.99 |  |  |  | s-avg | argmax | s-avg | 67.28 | 56.99
argmax | s-avg | s-avg | 89.73 | 6.85 | 16.81 | 13.47 |  |  |  | argmax | s-avg | s-avg | 67.15 | 57.21
s-avg | s-avg | s-avg | 90.39 | 6.69 | 13.69 | 10.39 |  |  |  | s-avg | s-avg | s-avg | 67.33 | 57.65
1-74 15-194 |  |  |  |  |  |  |  |  |  |  |  |  |  |

##  7 Conclusion

In this paper, we propose a novel Multispectral Chain-of-Thought Detection
(MSCoTDet) framework that incorporates Large Language Models to understand the
complementary information and further enhance the fusion process in
multispectral pedestrian detection. Specifically, our central design is to
process the detection in two consecutive branches: the vision branch and the
language branch. In the vision branch, we process single-modal detectors of
the two modalities. The language branch is the key design in our work,
generating text descriptions of the pedestrian in each modality and process
through our proposed Multispectral Chain-of-Thought (MSCoT) prompting, which
performs a step-by-step reasoning process to detect pedestrians based on the
descriptions. Then, we design Language-driven Multi-modal Fusion (LMF) to
enable fusion between the vision-driven and language-driven detections
processed from each branch. Extensive experiments demonstrate that our
proposed method can significantly improve the performance and mitigate
modality bias in multispectral pedestrian detection

## References

  * [1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
  * [2] Amari, S.i.: Backpropagation and stochastic gradient descent method. Neurocomputing 5(4-5), 185–196 (1993)
  * [3] Andersen, R.A., Buneo, C.A.: Intentional maps in posterior parietal cortex. Annual review of neuroscience 25(1), 189–220 (2002)
  * [4] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [5] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers (2020)
  * [6] Cha, J., Kang, W., Mun, J., Roh, B.: Honeybee: Locality-enhanced projector for multimodal llm. arXiv preprint arXiv:2312.06742 (2023)
  * [7] Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al.: A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology (2023)
  * [8] Chen, C., Li, R., Hu, Y., Siniscalchi, S.M., Chen, P.Y., Chng, E., Yang, C.H.H.: It’s never too late: Fusing acoustic information into large language models for automatic speech recognition. arXiv preprint arXiv:2402.05457 (2024)
  * [9] Chen, Y.T., Shi, J., Ye, Z., Mertz, C., Ramanan, D., Kong, S.: Multimodal object detection via probabilistic ensembling. In: European Conference on Computer Vision. pp. 139–158. Springer (2022)
  * [10] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.: Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24(240), 1–113 (2023)
  * [11] Colby, C.L., Goldberg, M.E.: Space and attention in parietal cortex. Annual review of neuroscience 22(1), 319–349 (1999)
  * [12] Corbetta, M., Shulman, G.L.: Control of goal-directed and stimulus-driven attention in the brain. Nature reviews neuroscience 3(3), 201–215 (2002)
  * [13] Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning (2023)
  * [14] Das, A., Das, S., Sistu, G., Horgan, J., Bhattacharya, U., Jones, E., Glavin, M., Eising, C.: Revisiting modality imbalance in multimodal pedestrian detection. arXiv preprint arXiv:2302.12589 (2023)
  * [15] Dollar, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: An evaluation of the state of the art. IEEE transactions on pattern analysis and machine intelligence 34(4), 743–761 (2011)
  * [16] Feng, G., Zhang, B., Gu, Y., Ye, H., He, D., Wang, L.: Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems 36 (2024)
  * [17] FLIR Systems, I.: Free teledyne flir thermal dataset for algorithm training. https://www.flir.com/oem/adas/adas-dataset-form/ (2021), accessed: 2022-08-05
  * [18] Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 1440–1448 (2015)
  * [19] González, A., Fang, Z., Socarras, Y., Serrat, J., Vázquez, D., Xu, J., López, A.M.: Pedestrian detection at day/night time with visible and fir cameras: A comparison. Sensors 16(6), 820 (2016)
  * [20] Guan, D., Cao, Y., Yang, J., Cao, Y., Yang, M.Y.: Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection. Information Fusion 50, 148–157 (2019)
  * [21] Hasan, I., Liao, S., Li, J., Akram, S.U., Shao, L.: Generalizable pedestrian detection: The elephant in the room. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11328–11337 (2021)
  * [22] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [23] Hessel, J., Lee, L.: Does my multimodal model learn cross-modal interactions? it’s harder to tell than you might think! arXiv preprint arXiv:2010.06572 (2020)
  * [24] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.d.L., Hendricks, L.A., Welbl, J., Clark, A., et al.: Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022)
  * [25] Hsu, W.Y., Lin, W.Y.: Ratio-and-scale-aware yolo for pedestrian detection. IEEE transactions on image processing 30, 934–947 (2020)
  * [26] Hwang, S., Park, J., Kim, N., Choi, Y., So Kweon, I.: Multispectral pedestrian detection: Benchmark dataset and baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1037–1045 (2015)
  * [27] Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al.: Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences 103, 102274 (2023)
  * [28] Kim, J.U., Park, S., Ro, Y.M.: Uncertainty-guided cross-modal learning for robust multispectral pedestrian detection. IEEE Transactions on Circuits and Systems for Video Technology 32(3), 1510–1523 (2021)
  * [29] Kim, J.U., Park, S., Ro, Y.M.: Uncertainty-guided cross-modal learning for robust multispectral pedestrian detection. IEEE Transactions on Circuits and Systems for Video Technology 32(3), 1510–1523 (2022). https://doi.org/10.1109/TCSVT.2021.3076466
  * [30] Kim, T., Lee, H.J., Ro, Y.M.: Map: Multispectral adversarial patch to attack person detection. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 4853–4857. IEEE (2022)
  * [31] Kim, T., Shin, S., Yu, Y., Kim, H.G., Ro, Y.M.: Causal mode multiplexer: A novel framework for unbiased multispectral pedestrian detection (2024)
  * [32] Kim, T., Yu, Y., Ro, Y.M.: Defending physical adversarial attack on object detection via adversarial patch-feature energy. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 1905–1913 (2022)
  * [33] Kim, T., Yu, Y., Ro, Y.M.: Multispectral invisible coating: laminated visible-thermal physical attack against multispectral object detectors using transparent low-e films. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 1151–1159 (2023)
  * [34] Koh, J.Y., Fried, D., Salakhutdinov, R.R.: Generating images with multimodal language models. Advances in Neural Information Processing Systems 36 (2024)
  * [35] Koh, J.Y., Salakhutdinov, R., Fried, D.: Grounding language models to images for multimodal inputs and outputs. In: International Conference on Machine Learning. pp. 17283–17300. PMLR (2023)
  * [36] Li, C., Song, D., Tong, R., Tang, M.: Multispectral pedestrian detection via simultaneous detection and segmentation. arXiv preprint arXiv:1808.04818 (2018)
  * [37] Li, C., Song, D., Tong, R., Tang, M.: Illumination-aware faster r-cnn for robust multispectral pedestrian detection. Pattern Recognition 85, 161–171 (2019)
  * [38] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014)
  * [39] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36 (2024)
  * [40] Liu, J., Zhang, S., Wang, S., Metaxas, D.N.: Multispectral deep neural networks for pedestrian detection. arXiv preprint arXiv:1611.02644 (2016)
  * [41] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [42] Morvant, E., Habrard, A., Ayache, S.: Majority vote of diverse classifiers for late fusion. In: Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshop, S+ SSPR 2014, Joensuu, Finland, August 20-22, 2014. Proceedings. pp. 153–162. Springer (2014)
  * [43] Nguyen, T.M., Nguyen, T., Le, T.M., Tran, T.: Gefa: early fusion approach in drug-target affinity prediction. IEEE/ACM transactions on computational biology and bioinformatics 19(2), 718–728 (2021)
  * [44] OpenAI, I.: Chatgpt-3.5 turbo api. https://chat.openai.com/ (2023), accessed: 2024-02-29
  * [45] OpenAI, I.: Gpt-3.5 turbo fine-tuning and api updates. https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates (2023), accessed: 2024-02-29
  * [46] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 27730–27744 (2022)
  * [47] Pandeya, Y.R., Lee, J.: Deep learning-based late fusion of multimodal information for emotion classification of music video. Multimedia Tools and Applications 80, 2887–2905 (2021)
  * [48] Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., Ye, Q., Wei, F.: Grounding multimodal large language models to the world. In: The Twelfth International Conference on Learning Representations (2023)
  * [49] Qingyun, F., Dapeng, H., Zhaokui, W.: Cross-modality fusion transformer for multispectral object detection. arXiv preprint arXiv:2111.00273 (2021)
  * [50] Qingyun, F., Dapeng, H., Zhaokui, W.: Cross-modality fusion transformer for multispectral object detection (2022)
  * [51] Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H.W., Tay, Y., Ruder, S., Zhou, D., et al.: Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057 (2022)
  * [52] Shutova, E., Kiela, D., Maillard, J.: Black holes and white rabbits: Metaphor identification with visual features. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies. pp. 160–170 (2016)
  * [53] Snoek, C.G., Worring, M., Smeulders, A.W.: Early versus late fusion in semantic video analysis. In: Proceedings of the 13th annual ACM international conference on Multimedia. pp. 399–402 (2005)
  * [54] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * [55] Tsimpoukelli, M., Menick, J.L., Cabi, S., Eslami, S., Vinyals, O., Hill, F.: Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34, 200–212 (2021)
  * [56] Tziafas, G., Kasaei, H.: Early or late fusion matters: Efficient rgb-d fusion in vision transformers for 3d object recognition. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 9558–9565. IEEE (2023)
  * [57] Wagner, J., Fischer, V., Herman, M., Behnke, S., et al.: Multispectral pedestrian detection using deep fusion convolutional neural networks. In: ESANN. vol. 587, pp. 509–514 (2016)
  * [58] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)
  * [59] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837 (2022)
  * [60] Workshop, B., Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, F., et al.: Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 (2022)
  * [61] Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)
  * [62] Xu, D., Ouyang, W., Ricci, E., Wang, X., Sebe, N.: Learning cross-modal deep representations for robust pedestrian detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5363–5371 (2017)
  * [63] Yao, Y., Mihalcea, R.: Modality-specific learning rates for effective multimodal additive late-fusion. In: Findings of the Association for Computational Linguistics: ACL 2022. pp. 1824–1834 (2022)
  * [64] Zhang, H., Fromont, E., Lefevre, S., Avignon, B.: Multispectral fusion for object detection with cyclic fuse-and-refine blocks. In: 2020 IEEE International conference on image processing (ICIP). pp. 276–280. IEEE (2020)
  * [65] Zhang, J., Khayatkhoei, M., Chhikara, P., Ilievski, F.: Visual cropping improves zero-shot question answering of multimodal large language models. arXiv preprint arXiv:2310.16033 (2023)
  * [66] Zhang, L., Liu, Z., Zhang, S., Yang, X., Qiao, H., Huang, K., Hussain, A.: Cross-modality interactive attention network for multispectral pedestrian detection. Information Fusion 50, 20–29 (2019)
  * [67] Zhang, L., Zhu, X., Chen, X., Yang, X., Lei, Z., Liu, Z.: Weakly aligned cross-modal learning for multispectral pedestrian detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5127–5137 (2019)
  * [68] Zhang, Z., Zhang, A., Li, M., Smola, A.: Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 (2022)
  * [69] Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., Smola, A.: Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923 (2023)
  * [70] Zhou, K., Chen, L., Cao, X.: Improving multispectral pedestrian detection by addressing modality imbalance problems. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16. pp. 787–803. Springer (2020)
  * [71] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)
  * [72] Zong, Z., Song, G., Liu, Y.: Detrs with collaborative hybrid assignments training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6748–6758 (2023)

Supplementary Material

##  Appendix A Training Data for MSCoT

Examples of Training Data. We provide examples of the training data for
Multispectral Chain-of-Thought (MSCoT, Section 3.3 in the manuscript). The
training data contains descriptions of RGB and thermal images, rationale, and
prediction scores for training the Large Language Model. First, we obtain the
textual descriptions of RGB and thermal images using the Multi-modal Large
Language Model (MLLM) [1], illustrated in the top part of Figure 7. Here, the
MLLM gets pre-processed images as the input and outputs their corresponding
textual descriptions. We use the ChatGPT-vision (GPT-4V) for the MLLM. Then we
prompt the GPT-3.5 [44] consisting of rationale, and prediction scores for our
MSCoT, which the prompts are in blue texts in Figure 7. The following answers
are obtained by prompting the GPT-3.5 independently with these prompts, and we
merge the answers to make the training data. Figure 7 shows an example of
training data. Then with these data, we fine-tune the model answer based on a
chain of prompts. The training data are manually selected with high-quality
answers. 50 training data from the FLIR training images are used for fine-
tuning MSCoT of the FLIR model. Also, we made 50 training data using the
CVC-14 training images for MSCoT of the CVC-14 model. Note that images from
ROTX-MP are not used for making training data for MSCoT because it violates
the evaluation setting for ROTX-MP. The evaluation rule on ROTX-MP is to use
models trained on general datasets (e.g., FLIR and CVC-14) and test those
models on ROTX-MP. Thus our MSCoT does not violate the evaluation rule on
ROTX-MP. An example of our training data are illustrated in Figure 7 below.
Training Detail. We use the ChatGPT-3.5 (GPT-3.5) API [44], provided by the
OpenAI developer platform. Using the fine-tuning module provided on the GPT
official website [45], we trained GPT-3.5 API with our training data. The
official document for GPT-3.5 [45] recommends training 50 samples for fine-
tuning the model, and we utilized a total of 50 training samples for each FLIR
and CVC-14 model.

##  Appendix B Related Work

###  B.1 Multispectral Pedestrian Detection

Dataset. Different from standard pedestrian detection, multispectral
pedestrian detection aims to detect pedestrians using RGB and thermal images.
The KAIST [26] dataset is one of the first benchmarks for multispectral
pedestrian detection, containing well-aligned RGB and thermal images with
pedestrians captured from all daytime/nighttime. The FLIR [17] dataset is a
newer benchmark, offering higher image quality and resolution than KAIST [26].
While the KAIST [26] and FLIR [17] datasets provide a benchmark for well-
aligned RGB-thermal images, the CVC-14 [19] dataset introduces a benchmark
that largely contains misaligned RGB-thermal images, highlighting the
challenges due to modality misalignments. Furthermore, the ROTX-MP [31]
dataset provides a challenging benchmark, evaluating modality biases in
multispectral pedestrian detection. In other multispectral pedestrian datasets
(e.g., KAIST, CVC-14, and FLIR), pedestrians always statistically co-occur
with their thermal signatures [31]. On the other hand, ROTX-MP mainly contains
thermal-obscured data, in which the thermal signatures of pedestrians are
absent due to the intermediate obstacles (e.g., windows and heat-insulating
clothes). Modality biases are evaluated under the setting that training the
models on general datasets (e.g., KAIST, CVC-14, and FLIR) and testing their
performance on ROTX-MP. Model. To solve these challenges presented in
multispectral pedestrian detection, various methods have been proposed [57,
40, 49, 28, 9, 36]. To alleviate modality misalignment problems, Kim et
al.(2022) [29] proposed an uncertainty-aware feature fusion (UFF) network. UFF
first estimates the uncertainty value of each modality feature and assigns
different weights in the fusion process. Cross-Modality Fusion Transformer
[50] enhances feature fusion by designing a new cross-modality fusion
mechanism based on self-attention. ProbEn [9] propose a late-fusion-based
strategy, which probabilistically ensembles prediction scores and bounding
boxes obtained from single-modal detectors of different modalities. Lastly,
the Causal Mode Multiplexer (CMM) [31] framework is proposed to mitigate
modality bias via counterfactual intervention and improve the performance on
thermal-obscured data.

###  B.2 Large Language Model

Large language models (LLMs) excel in natural language processing (NLP) and
generation by training on vast amounts of text data [46, 4, 60, 54, 10, 24].
These features and advantages offer innovative applicability across more
general areas of intelligence. Multimodal large language models (MLLMs), such
as LLaVA [39], MiniGPT-4 [71], and InstructBLIP [13], rely on the high-level
understanding and recognition of linguistic and visual content that LLMs
possess. In visual grounding [48, 35, 34], MLLMs can comprehend the context
within an image by identifying objects, attributes, and the overall scene.
They excel at linking these visual contextual elements with textual
descriptions in semantic space. Large-scale language models can process and
generate multimodal data across various input formats, enabling applications
in diverse fields. Recently, the ability to leverage LLM’s semantic
understanding capabilities to model and understand continuous streams of
thought has gained traction. ScienceQA [69, 6], a benchmark task based on
scientific knowledge for natural language understanding and question
answering, highlights the advantages of language ability. LLMs understand
questions through a step-by-step approach called Chain-of-Thought (CoT) [59]
and make inferences based on learned knowledge. CoT aggregates and synthesizes
given knowledge to provide new insights and can improve issues (e.g.,
hallucination) commonly encountered in answering difficult questions.

##  Appendix C Implementation Detail

For all models, the setting is kept the same across training CVC [19], and
FLIR [17] datasets. CMM [31]: The CMM framework is based on the Uncertainty-
guided model [29], with the FPN architecture with ResNet-50 [22] as backbone
networks. Specifically, we use the pytorch library and stochastic gradient
descent (SGD) for optimization, synchronizing the process across 4 GTX 1080 Ti
GPUs. Each GPU handles 2 images, resulting in a total of 8 images per mini-
batch. The model is trained for 2 epochs. The learning rate is initialized at
0.007 during the initial 1 epoch and subsequently applied a 0.1 learning rate
decay for the final epoch. We configured the number of Region of Interests
(RoIs) per image to N=300. We apply switchable Total Indirect Effect (sTIE)
for every ROI and compute the prediction score. The final predictions are made
through the Non-Maximum Suppression (NMS) process. We use the code provided by
the authors, with the same settings as the original paper [31]. ProbEn [9]:
ProbEn train two single-modal detectors based on Faster R-CNN [18] with
Detectron 2 [61]. We used 2 NVIDIA RTX3090 GPUs with SGD [2] optimizer with
learning rate 5e-3. Image flip and resize is applied for data augmentation.
Also, we pre-process the RGB and thermal images by subtracting the mean input
value, which is (103.53, 116.28, 123.67) and (135.438) for each RGB and
thermal images. The min value of the input pixel is 0 and max value is 255 for
each channel. We adopt the ’ProbEn’ score fusion and ’v-avg’ box fusion
because such combinations are shown the most effective in the original paper
[9]. Also, we adopt the E+M+T ensemble version when experimenting on the FLIR
dataset, indicating that early-fusion, mid-fusion, and thermal single-modal
detectors are ensembled. We adopt the R+M+T ensemble version when
experimenting on the CVC-14 dataset, indicating that mid-fusion, RGB, and
thermal single-modal detectors are ensembled. The single-modal detectors are
pre-trained on the COCO dataset [38] and then trained in the FLIR or CVC-14
dataset, same as the original paper. Kim et al. [29]: We train the model for 3
epochs. The learning rate is initialized at 0.006 during the initial 2 epochs
and subsequently applied a 0.1 learning rate decay for the final epoch. As
this model is the baseline model of CMM, we set the other hyperparameters the
same as CMM. We use the code provided by the authors, with the same settings
as the original paper [29]. Halfway Fusion [40]: We use the Faster-RCNN [18]
as the base model and train the model for 3 epochs. The learning rate is
initialized at 0.008 during the initial 2 epochs and subsequently applied a
0.1 learning rate decay for the final epoch. The SGD optimizer is used. CFT
[50]: We use the same setting as the original paper. The initial learning rate
is 0.01 with a momentum of 0.937, and weight decay 0.0005. Batch size is 32
and the model is trained for 200 epochs and the YOLO-v5 weight trained on the
COCO dataset is used for weight initialization. We used the official code from
GitHub.

##  Appendix D Detailed Algorithm

###  D.1 Cross-aligned Pedestrian Description Generation

The detailed algorithm of Cross-aligned Pedestrian Description Generation is
described in Algorithm 1. Denote the procedure CPDG($\cdot$). CPDG($\cdot$)
gets inputs: An RGB image $X_{\scriptscriptstyle RGB}$, an thermal image
$X_{\scriptscriptstyle T}$, prediction scores $S_{\scriptscriptstyle RGB}$,
$S_{\scriptscriptstyle T}$ and bounding boxes $B_{\scriptscriptstyle RGB}$,
$B_{\scriptscriptstyle T}$. And CPDG($\cdot$) requires input text prompts
$p_{\scriptscriptstyle RGB}$, $p_{\scriptscriptstyle T}$, and the IoU
threshold $\tau$. Detection pairing module DPair($\cdot$), Visual Crop & Mark
module VCM($\cdot$). A pretrained multimodal large language model
MLLM($\cdot$). Then CPDG($\cdot$) outputs a set of RGB descriptions $D_{RGB}$,
a set of thermal descriptions $D_{T}$, and paired detections $S^{paired}$,
$B^{paired}$. The first process is to produce paired detections $S^{paired}$,
$B^{paired}$ through the detection pairing module Dpair($\cdot$), line 6 in
Algorithm 1. The detailed process of Dpair($\cdot$) is described in Algorithm
2. Then, using the paired bounding boxes $B^{paired}$, the Visual Crop & Mark
module VCM($\cdot$) produces pre-processed images $X_{RGB}^{VCM}$ and
$X_{T}^{VCM}$. For each image in $x_{RGB}^{VCM}\in X_{RGB}^{VCM}$, the pre-
trained MLLM($\cdot$) produces text descriptions $d_{RGB}$, which is appended
to the set $D_{RGB}$. Similarly, each image in $x_{T}^{VCM}\in X_{T}^{VCM}$,
the pre-trained MLLM($\cdot$) produces text descriptions $d_{T}$, and is
appended to the set $D_{T}$. Finally, CPDG($\cdot$) outputs sets of textual
descriptions $D_{RGB}$, $D_{T}$, and paired detections $S^{paired}$ and
$B^{paired}$. We describe each Detection Pairing, Visual Crop & Mark module in
the following.

Algorithm 1 Cross-aligned Pedestrian Description Generation (CPDG)

1:Input: An RGB image $X_{\scriptscriptstyle RGB}$, an thermal image
$X_{\scriptscriptstyle T}$, prediction scores $S_{\scriptscriptstyle RGB}$,
$S_{\scriptscriptstyle T}$ and bounding boxes $B_{\scriptscriptstyle RGB}$,
$B_{\scriptscriptstyle T}$.

2:Require: Input text prompts $p_{\scriptscriptstyle RGB}$,
$p_{\scriptscriptstyle T}$, and IoU threshold $\tau$. Detection pairing module
DPair($\cdot$), Visual Crop & Mark module VCM($\cdot$). A pre-trained
multimodal large language model MLLM($\cdot$).

3:Output: A set of RGB descriptions $D_{RGB}$, a set of thermal descriptions
$D_{T}$, and paired detections $S^{paired}$, $B^{paired}$.

4:

5:procedure CPDG$(X_{\scriptscriptstyle RGB},X_{\scriptscriptstyle
T},S_{\scriptscriptstyle RGB},S_{\scriptscriptstyle T},B_{\scriptscriptstyle
RGB},B_{\scriptscriptstyle T};p_{\scriptscriptstyle RGB},p_{\scriptscriptstyle
T},\tau)$

6:$S^{paired}$, $B^{paired}$ $\leftarrow$ DPair$(B_{\scriptscriptstyle
RGB},B_{\scriptscriptstyle T},S_{\scriptscriptstyle RGB},S_{\scriptscriptstyle
T};\tau)$.

7:$X_{\scriptscriptstyle RGB}^{\scriptscriptstyle VCM}$ $\leftarrow$
VCM($X_{\scriptscriptstyle RGB}$, $B^{paired}$), $X_{\scriptscriptstyle
T}^{\scriptscriptstyle VCM}$ $\leftarrow$ VCM($X_{\scriptscriptstyle T}$,
$B^{paired}$).

8:initialize list $D_{RGB}\leftarrow\phi,D_{T}\leftarrow\phi$.

9:for all images $x^{\scriptscriptstyle VCM}_{RGB}$ in $X^{\scriptscriptstyle
VCM}_{RGB}$

10:$d_{RGB}=$MLLM$(x^{\scriptscriptstyle VCM}_{RGB};p_{RGB})$.

11:Append the text description $d_{RGB}$ to the list $D_{RGB}$.

12:end for

13:for all images $x^{\scriptscriptstyle VCM}_{T}$ in $X^{\scriptscriptstyle
VCM}_{T}$

14:$d_{T}=$MLLM$(x^{\scriptscriptstyle VCM}_{T};p_{T})$.

15:Append the text description $d_{T}$ to the list $D_{T}$.

16:end for

17:return $D_{RGB}$, $D_{T}$, $S^{paired}$, $B^{paired}$.

18:end procedure

###  D.2 Detection Pairing

Due to the misalignment between RGB and thermal images, we often need to guide
MLLMs to make descriptions based on different image regions in RGB and thermal
images for the same pedestrians. Our strategy is to find the detection pairs
from the single-modal detections (e.g., bounding boxes) that belong to the
same pedestrians and provide this information to the MLLM. We design a
Detection Pairing module to do this. Although bounding boxes representing the
same pedestrian can have different coordinates in different modalities, still
they will contain highly overlapping areas in the images, and those boxes will
have large values of IoUs (Intersection-of-Unions). Specifically, for the
$i$-th bounding box $b_{\scriptscriptstyle i,\scriptscriptstyle m}$ in
bounding boxes $B_{\scriptscriptstyle m}$, obtained from the single-modal
detector in the $m$ modality ($m$ is either RGB or thermal: T), we aim to find
the bounding box from $B_{\scriptscriptstyle m^{c}}$, in the opposite modality
$m^{c}$ that corresponds to the same pedestrian. To this end, we compute the
IoU value between $b_{\scriptscriptstyle i,\scriptscriptstyle m}$ across all
bounding boxes in $B_{\scriptscriptstyle m^{c}}$, and find the box that has
the highest IoU value, e.g., $b_{\scriptscriptstyle j,\scriptscriptstyle
m^{c}}$, with $b_{\scriptscriptstyle i,\scriptscriptstyle m}$. Here, we only
consider the boxes with IoU values above the IoU threshold $\tau$. With
indices $i$ and $j$, we can find a pair of the prediction scores

Algorithm 2 Detection Pairing (DPair)

1:Input: Prediction scores $S_{\scriptscriptstyle RGB}$,
$S_{\scriptscriptstyle T}$, Bounding boxes $B_{\scriptscriptstyle RGB}$,
$B_{\scriptscriptstyle T}$.

2:Require: An IoU threshold $\tau$.

3:Output: Paired bounding boxes $B^{paired}$, paired prediction scores
$S^{paired}$.

4:

5:procedure DPair$(B_{\scriptscriptstyle RGB},B_{\scriptscriptstyle
T},S_{\scriptscriptstyle RGB},S_{\scriptscriptstyle T};\tau)$

6:initialize lists $B\leftarrow B_{\scriptscriptstyle RGB}\cup
B_{\scriptscriptstyle T}$, $S\leftarrow S_{\scriptscriptstyle RGB}\cup
S_{\scriptscriptstyle T}$, $B^{paired}\leftarrow\phi$, and
$S^{paired}\leftarrow\phi$.

7:Find the box $b_{max}\in B$ of the highest prediction score.

8:Find boxes $D\in B$ overlapping with $b_{max}$, of IoU (> $\tau$) in the
opposite modality.

9:Find the box $b_{pair}\in D$ of the highest IoU value.

10:If $D$ is empty, then $b_{pair}$ $\leftarrow$ $b_{max}$ and $s_{pair}$
$\leftarrow$ $s_{max}$.

11:Append ($b_{max},b_{pair}$) to $B^{paired}$

12:Append ($s_{max},s_{pair}$) to $S^{paired}$.

13:Remove $b_{max}$ and $b_{pair}$ from $B$ and $s_{max}$ and $s_{pair}$ from
$S$

14:Repeat this process until $B$ and $S$ are empty.

15:return $B^{paired}$, $S^{paired}$

16:end procedure

$s_{\scriptscriptstyle i,\scriptscriptstyle m}$ and $s_{\scriptscriptstyle
j,\scriptscriptstyle m^{c}}$ that belong to the same pedestrian. We call such
tuples ($b_{\scriptscriptstyle i,\scriptscriptstyle m}$,$b_{\scriptscriptstyle
j,\scriptscriptstyle m^{c}}$) and ($s_{\scriptscriptstyle i,\scriptscriptstyle
m}$,$s_{\scriptscriptstyle j,\scriptscriptstyle m^{c}}$) as the "paired
detection". When all IoU values are under the IoU threshold $\tau$, this means
that there are no boxes in $B_{\scriptscriptstyle m^{c}}$ that correspond to
the same pedestrian and $b_{\scriptscriptstyle i,\scriptscriptstyle m}$ can
not find a pair. Such cases can occur at nighttime or in thermal-obscured
scenarios where the pedestrian is detected by only one modality. In this case,
we override the value of $b_{\scriptscriptstyle i,\scriptscriptstyle m}$ and
$s_{\scriptscriptstyle i,\scriptscriptstyle m}$ to the opposite modality,
making pairs of ($b_{\scriptscriptstyle i,\scriptscriptstyle
m}$,$b_{\scriptscriptstyle i,\scriptscriptstyle m}$) and
($s_{\scriptscriptstyle i,\scriptscriptstyle m}$,$s_{\scriptscriptstyle
i,\scriptscriptstyle m}$). Iterating this process for over all bounding boxes
in $B_{\scriptscriptstyle m}$ and $B_{\scriptscriptstyle m^{c}}$ (i.e.,
$B_{\scriptscriptstyle RGB}$ and $B_{\scriptscriptstyle T}$) allows us to
obtain a set of paired detections $B^{paired}$ and $S^{paired}$. Denote the
Detection Pairing Module as a function $DPair(\cdot)$ with respect to inputs
$B_{\scriptscriptstyle RGB}$, $B_{\scriptscriptstyle T}$,
$S_{\scriptscriptstyle RGB}$, and $S_{\scriptscriptstyle T}$, which indicate
bounding boxes and prediction scores obtained from RGB and thermal single-
modal detectors, respectively. The inputs and outputs of $DPair(\cdot)$ can be
represented as:

${S^{paired},B^{paired}=DPair(B_{\scriptscriptstyle RGB},B_{\scriptscriptstyle
T},S_{\scriptscriptstyle RGB},S_{\scriptscriptstyle T};\ \tau),}$ (11)

where $\tau$ indicates an IoU threshold. We set this value to 0.5, the
standard value used in the object detection task.

###  D.3 Visual Crop & Mark

The Visual Crop & Mark module performs very simple image pre-processing.
First, an image $X$ and bounding boxes $B$ are given as input. Note that $B$
represents the locations of pedestrians in image $X$. For each bounding box,
$b\in B$, a green box is drawn in the image $X$. Then, the image $X$ is
cropped around the bounding box $b$ so that the width and height of the
cropped image $X{c}$ are twice as $b$. This process is iterated for all
bounding boxes $b\in B$, and pre-processed images $X^{VCM}$ are obtained. The
detailed algorithm is described in Algorithm 3.

Algorithm 3 Visual Crop & Mark (VCM)

1:Input: An image $X$, and bounding boxes $B$.

2:Output: Pre-processed images $X^{VCM}$.

3:

4:procedure VCM$(X,B)$

5:initialize ${X^{\scriptscriptstyle VCM}}\leftarrow\phi$.

6:for all bounding boxes $b$ in $B$

7:Draw a green box at bounding box $b$ in the image $X$.

8:Crop the image $X$ around the bounding box $b$ such that the width and
height of the cropped image $X_{c}$ are twice as the width and height of $b$.

9:Append the cropped image $X_{c}$ to $X^{\scriptscriptstyle VCM}$.

10:end for

11:return $X^{\scriptscriptstyle VCM}$

12:end procedure

##  Appendix E Visualized Result

In this section, we provide visualized detections of MSCoTDet (Ours) compared
to the conventional model [31] on the ROTX-MP [31], FLIR [17], and CVC-14 [19]
datasets. ROTX-MP. Figure 5 to Figure 4 show the visualized detections on the
ROTX-MP [31] dataset. The conventional model [31] fails to detect thermal-
obscured pedestrians in many cases. On the other hand, MSCoTDet achieves
accurate detections of these pedestrians. Although MSCoTDet produces a low
prediction score $S_{\scriptscriptstyle T}^{\scriptscriptstyle F}$ from the
thermal single-modal detector, the high prediction score from the language
branch raises the overall prediction score $S_{LMF}$ above the detection
threshold of 0.5. FLIR and CVC-14. Figure 11 show the visualized detections on
the FLIR [17] and CVC-14 [19] datasets. Figure 11 show that MSCoTDet (Ours)
can achieve more accurate detection on small-scale pedestrians, misaligned
pedestrians, and objects that are likely to be detected as pedestrians because
of their similar appearance to pedestrians in the thermal image.

Figure 7: (Top): Textual descriptions of an RGB and thermal image generated by
a Multi-modal Large Language Model (MLLM). We use the ChatGPT-vision (GPT-4V)
[1] API for the MLLM. (Bottom) Corresponding training data for fine-tuning the
Large Language Model (LLM) to process Multispectral Chain-of-Thought (MSCoT).
We use the ChatGPT-3.5 (GPT-3.5) API for the LLM and fine-tune them with our
training data. Figure 8: Visualized detections of a conventional model [31]
and MSCoTDet (Ours) on the ROTX-MP [31] dataset. MSCoTDet achieves higher
prediction scores ($S_{LMF}$) compared to the conventional model, enabling the
detection of thermal-obscured pedestrians. $S_{\scriptscriptstyle
RGB}^{\scriptscriptstyle V}$, $S_{\scriptscriptstyle T}^{\scriptscriptstyle
V}$ denote prediction scores from RGB and thermal single-modal detectors,
$S_{\scriptscriptstyle F}^{\scriptscriptstyle V}$ denote the fused prediction
score from the visual branch, and denote $S_{\scriptscriptstyle
F}^{\scriptscriptstyle L}$ the fused prediction score from the language
branch, respectively. Only the detections with prediction scores above the
threshold 0.3 are visualized. Figure 9: Visualized detections of a
conventional model [31] and MSCoTDet (Ours) on the ROTX-MP [31] dataset.
MSCoTDet achieves higher prediction scores ($S_{LMF}$) compared to the
conventional model, enabling the detection of thermal-obscured pedestrians.
$S_{\scriptscriptstyle RGB}^{\scriptscriptstyle V}$, $S_{\scriptscriptstyle
T}^{\scriptscriptstyle V}$, $S_{\scriptscriptstyle F}^{\scriptscriptstyle V}$,
and $S_{\scriptscriptstyle F}^{\scriptscriptstyle L}$ denote prediction scores
from RGB and thermal single-modal detectors, fused prediction score from the
visual branch, and fused prediction score from the language branch,
respectively. Only the detections with prediction scores above the threshold
0.3 are visualized. Figure 10: Visualized detections of a conventional model
[31] and MSCoTDet (Ours) on the ROTX-MP [31] dataset. MSCoTDet achieves higher
prediction scores ($S_{LMF}$) compared to the conventional model, enabling the
detection of thermal-obscured pedestrians. $S_{\scriptscriptstyle
RGB}^{\scriptscriptstyle V}$, $S_{\scriptscriptstyle T}^{\scriptscriptstyle
V}$, $S_{\scriptscriptstyle F}^{\scriptscriptstyle V}$, and
$S_{\scriptscriptstyle F}^{\scriptscriptstyle L}$ denote prediction scores
from RGB and thermal single-modal detectors, fused prediction score from the
visual branch, and fused prediction score from the language branch,
respectively. Only the detections with prediction scores above the threshold
0.3 are visualized. Figure 11: Visualized detections of a conventional model
[31] and MSCoTDet (Ours) on the FLIR [17] and CVC-14 [19] dataset. (Top):
MSCoTDet can achieve accurate detection of small-scale pedestrians. (Middle):
MSCoTDet produces bounding boxes (red) more closely to the ground-truth labels
(green). (Bottom): MSCoTDet can make accurate detection on objects (e.g.,
fireplugs) that are likely to be detected as pedestrians, which have a similar
appearance to pedestrians in the thermal image. The conventional model falsely
detects the fireplug as a pedestrian. Only the detections with prediction
scores above the threshold 0.3 are visualized.