# TEDi: Temporally-Entangled Diffusion for Long-Term Motion Synthesis

Zihan Zhang  University of Chicago ,  Richard Liu  University of Chicago ,
Kfir Aberman  Google Research and  Rana Hanocka  University of Chicago

###### Abstract.

The gradual nature of a diffusion process that synthesizes samples in small
increments constitutes a key ingredient of Denoising Diffusion Probabilistic
Models (DDPM), which have presented unprecedented quality in image synthesis
and been recently explored in the motion domain. In this work, we propose to
adapt the gradual diffusion concept (operating along a diffusion time-axis)
into the temporal-axis of the motion sequence. Our key idea is to extend the
DDPM framework to support temporally varying denoising, thereby entangling the
two axes. Using our special formulation, we iteratively denoise a motion
buffer that contains a set of increasingly-noised poses, which auto-
regressively produces an arbitrarily long stream of frames. With a stationary
diffusion time-axis, in each diffusion step we increment only the temporal-
axis of the motion such that the framework produces a new, clean frame which
is removed from the beginning of the buffer, followed by a newly drawn noise
vector that is appended to it. This new mechanism paves the way towards a new
framework for long-term motion synthesis with applications to character
animation and other domains. Our code and project are publicly available.
111Project page: https://threedle.github.io/TEDi/

††journal: TOG Figure 1. Inspired by the gradual nature of the diffusion
process along a diffusion time-axis (left), our approach (right) entangles the
temporal-axis of motion with the time-axis of the diffusion process (right),
enabling a new mechanism for synthesizing arbitrarily long motion sequences.

##  1\. Introduction

Long-term generation of a motion sequence is a difficult and long standing
problem in character animation with myriad applications in computer animation,
motion control, human-computer interaction, and more. Generating long-term
motion entails producing realistic, non-repetitive sequences which avoids
degenerate outputs (i.e., frozen motion).

A promising avenue for generating high-quality motion is through Denoising
Diffusion Probabilistic Models (DDPM), which have produced unprecedented
quality in image synthesis [Ho et al., 2020] and have been recently adapted to
motion synthesis [Tevet et al., 2022; Zhang et al., 2022; Kim et al., 2022]. A
typical adaptation of DDPM to motion synthesis generates a fixed-length motion
sequence (i.e., a “motion image”) from randomly sampled Gaussian noise.

A fixed-length output is limiting in the context of long-term motion synthesis
for a couple of reasons. First, there is no satisfactory approach for creating
long-sequences from short-sequences outputs. Simply chaining together motions
and blending them may create stitching artifacts. Second, a typical diffusion
process has limited interactive controllability. Diffusion requires several
hundred denoising iterations before producing a short sequence of clean
motions.

We are inspired by the time-dependent nature of the diffusion process, where
samples are synthesized from pure noise gradually in small time increments
along the diffusion time-axis. In this work, we propose to adapt diffusion to
the temporal-axis of the motion. Our method, referred to as TEDi (Temporally-
Entangled Diffusion), extends the DDPM framework by enabling injection of
temporally-varying noise levels during each step of the diffusion process,
instead of a Gaussian noise with a fixed, temporally-invariant variance. By
entangling the temporal-axis of the motion sequence with the time-axis of the
diffusion process, we enable the production of a continuous stream of clean
motion frames during each step of the diffusion process.

At the core of our framework lies a motion buffer, which encodes noisy future
motion frames with varied noise levels. During the training phase, we add
temporally varied noise to clean motion sequences, such that each frame has a
random level. However, during inference the motion buffer is initialized with
a motion primer \- a sequence of clean motion frames that are being noised
with increasing noise levels, such that adjacent frames contains consecutive
noise levels. TEDi recursively denoises the increasingly-noised future frames.
In order to constantly maintain the progressively-noised motion buffer
structure during each denoising step, we insert a noisy frame at the end of
the motion buffer and remove a single clean frame at the beginning.

This recursive mechanism enables motion sequence frames to be continuously
generated, and avoids stitching problems which current motion diffusion models
suffer from (see 4.5.1).

During inference, we can guide the generation with specific motions by
intervening in the process and persistently injecting clean frames, called
guiding motions. This injection enables us to control and influence the
current set of generated frames to _prepare and plan_ for the upcoming motion
guides. This strategy causes a premeditated and calculated transition between
the current frames and the future guiding motions.

Our network continues to denoise an ever-evolving motion buffer, which
contains vague information about the future trajectory of the motion sequence.
This formulation opens the door to more direct control, and better planning,
of the generated motion via manipulation of the motion buffer. We demonstrate
that our framework is capable of producing different types of long motion
sequences, and due to its random nature, can provide diverse results even for
the same initialization. In addition, we evaluate the model against other
long-term generation models. Our experiments show that TEDi is a natural
framework for generating long-term motion sequences.

##  2\. Related Work

###  2.1. Diffusion Models

Denoising diffusion probalistic models (DDPMs) and its variants [Ho et al.,
2020; Dhariwal and Nichol, 2021; Ho et al., 2022] have achieved unprecedented
quality on conditional and unconditional image generation, generally
surpassing GAN-based [Dhariwal and Nichol, 2021] methods both in visual
quality and sampling diversity. In particular, diffusion models have
demonstrated remarkable fidelity and semantic control for text-to-image
synthesis and editing tasks when large models are trained on text and image
pairs [Ramesh et al., 2022; Saharia et al., 2022b; Rombach et al., 2021; Ruiz
et al., 2022; Hertz et al., 2022]. In addition, diffusion has been
successfully applied in adjacent domains such as text-to-video and image-to-
image translation [Saharia et al., 2022a]. Moreover, diffusion models are
beginning to see increased usage in generative tasks with 3D data. Some recent
work enable 3D data generation by reducing it to a 2D task, while others
directly train the entire diffusion pipeline on 3D data. More recently, in the
animation domain, Zhang et al. [Zhang et al., 2022], Kim et al. [Kim et al.,
2022], Tevet et al. [Tevet et al., 2022], and Shafir et al. [Shafir et al.,
2023] have suggested adapting diffusion models for motion generation by
directly applying the diffusion framework, namely by treating the entire
motion as an image and denoising all frames in parallel. This adaptation can
only generate fixed-length motion sequences which makes long-term generation
and interactive control infeasible. In contrast, our framework combines the
diffusion framework with an auto-regressive generation scheme, thus enabling
generation of arbitrary length sequences by design.

###  2.2. Deep Motion Synthesis

Before the advent of modern deep learning architectures, earlier works
attempted to model motion and styles of motion with techniques such as
restricted Boltzmann machines Taylor and Hinton [2009]. Later on, the seminal
set of works by Holden et al. [2015; 2016] applied convolutional neural
networks (CNN) to motion data and learned a motion manifold which can then be
used to perform motion editing by, for instance, projection onto the motion
manifold. Concurrently, Fragkiadaki et al. [2015] chose to use recurrent
neural networks (RNN) for motion modeling. RNN based works also succeeded in
short-term motion prediction [Fragkiadaki et al., 2015; Pavllo et al., 2018],
interactive motion generation [Lee et al., 2018], and music-driven motion
synthesis [Aristidou et al., 2021]. Holden et al. [2017] propose phase-
functioned neural networks (PFNN) for locomotion generation and introduce
_phase_ to neural networks for motion synthesis. Similar ideas are used in
quadruped motion generation by Zhang et al. [2018]. Starke et al. [2020]
extended phase to local joints to cope with more complex motion generation.
Henter et al. [2020] proposed another generative model for motion based on
normalizing flow. Additionally, deep neural networks have succeeded in a
variety of other motion synthesis tasks such as motion retargeting [Villegas
et al., 2018; Aberman et al., 2020a, 2019], motion style transfer [Aberman et
al., 2020b; Mason et al., 2022], key-frame based motion generation [Harvey et
al., 2020], motion matching [Holden et al., 2020], animation layering [Starke
et al., 2021] and motion synthesis from a single sequence [Li et al., 2022].

###  2.3. Long-Term Motion Synthesis

Deep learning models for long term motion synthesis are mostly based on RNNs
as they naturally enable auto-regressive generation and capture the time
dependencies between animation frames. In general, RNNs have shown much
success in natural language processing (NLP) for generating text [Sutskever et
al., 2011], hand written characters [Gregor et al., 2015], and even captioning
images [Vinyals et al., 2015]. They have also been proposed for spatio-
temporal prediction where [Ranzato et al., 2014; Srivastava et al., 2015]
integrated 2D convolutions into the recurrent state transitions of a standard
LSTM and proposed the convolutional LSTM network, which can model the spatial
correlations and temporal dynamics in a unified recurrent unit. Wang et al.
extended convolutional LSTMs with pairwise memory cells to capture both long-
term dependencies and short-term variations to improve the prediction quality
[Wang et al., 2017]. Zhou et al. tackled the problem of error accumulation in
long-term random generation by alternating the network’s output and ground
truth as the input of RNN during training [Zhou et al., 2018]. This method,
called acRNN, is able to generate long and stable motion similar to the
training set. However, despite the modified training procedure, acRNN still
fails to produce very long motions. One speculation is that acRNN, and RNNs in
general, rely on a memory component that is being eroded with time. In
contrast, our framework explicitly utilizes frames within our context window
which only needs to be the same size temporally as the diffusion time-axis,
producing the motion autoregressively in small increments that complies with
the successful mechanism of the diffusion process.

[b]

Figure 2. TEDi Training. We train our diffusion-based model to remove
temporally-varying noise that is applied to clean sequences during training.
In each iteration we fetch a motion sequence of $K$ frames
$[f_{1},f_{2},\ldots,f_{K}]$ from the dataset, apply noise to it according to
a noise level schedule $[\beta_{t_{1}},\beta_{t_{2}},\ldots,\beta_{t_{K}}]$,
and train our network to predict the clean motion sequence in a supervised
fashion as described in (1).

##  3\. Method

We propose a new approach to synthesize long motion sequences using diffusion
models. Our approach extends the classic DDPM framework to support injection
of temporally-varying noise levels during the diffusion process. This
extension enables entangling the temporal-axis of the motion sequence with the
time-axis of the diffusion process. In the particular case where the first
frame in the sequence is mapped into the lowest noise level, the last frame to
the highest level, and the mapping function is linear, we can continuously
synthesize arbitrarily many frames during inference - akin to a motion buffer.
In each diffusion step we get a clean frame at the beginning of the sequence,
shift the frames in the stack by popping the clean frame, and append a new
noisy frame (drawn from a Gaussian distribution) to the end of the sequence.
Repeating this process during inference results in a new mechanism for long
term motion synthesis. We describe below the motion representation (3.1),
novel diffusion framework (3.2), training (3.3) and inference procedure (3.4).

###  3.1. Motion representation

We represent a motion sequence by a temporal set of $K$ poses that consists of
root joint displacements with respect to the xz-plane ${\bf
O_{xz}}\in\mathds{R}^{K\times 2}$, root joint height
$\bf{O_{y}}\in\mathds{R}^{K}$, and joint rotations ${\bf
R}\in\mathds{R}^{K\times JQ}$, where $J$ is the number of joints and $Q$ is
the number of rotation features. The rotations are defined in the coordinate
frame of their parent in the kinematic chain, and represented by the 6D
rotation features ($Q=6$) proposed by Zhou et al. [2019]. To mitigate foot
sliding artifacts, we incorporate foot _contact labels_ as a $C\cdot K$ binary
values ${\bf L}\in\\{0,1\\}^{K\times C}$, which correspond to the contact
labels of the foot joints. In our work, we let $C=4$, where the joints are the
left(right) heels and toes. All the features are concatenated along the
channel axis and we denote the full representation by ${\bf M}\equiv[{\bf
O}_{xz},{\bf O}_{y},{\bf R},{\bf L}]\in\mathds{R}^{K\times(JQ+C+3)}$.

###  3.2. Diffusion Models

Diffusion Denoising Probabilistic Models (DDPM) [Sohl-Dickstein et al., 2015;
Ho et al., 2020] are generative models that aim to approximate a given data
distribution $q(m_{0})$ with an easy and intuitive sampling mechanism that is
inspired by diffusion processes in physics. In the particular case of motion
synthesis, the data consists of fixed-length motion sequences. During
training, the process starts by sampling a clean motion sequence $m_{0}$ from
the dataset, then an IID Gaussian noise is added gradually to form a sequence
of noisy motions which constitute the latent variables of the process
{$m_{1},\ldots,m_{T}\\}$. The latent sequence follows
$q(m_{1},\ldots,m_{t}\mid m_{0})=\prod_{i=1}^{t}q(m_{i}\mid m_{i-1})$, where a
sampling step in the forward process (clean data to noise) is defined as a
Gaussian transition $q(m_{t}\mid
m_{t-1}):=\mathcal{N}(\sqrt{1-\beta_{t}}m_{t-1},\beta_{t}I)$ parameterized by
a schedule $\beta_{0},\ldots,\beta_{T}\in(0,1)$. When the total diffusion time
step $T$ is large enough, the last noise vector $m_{T}$ nearly follows an
isotropic Gaussian distribution.

In order to sample from the distribution $q(m_{0})$, we define the dual
“reverse process” $p(m_{t-1}\mid m_{t})$ from isotropic Gaussian noise $m_{T}$
to data by sampling the posteriors $q(m_{t-1}\mid m_{t})$. Since the
intractable reverse process $q(m_{t-1}\mid m_{t})$ depends on the unknown data
distribution $q(m_{0})$, we approximate it with a parameterized Gaussian
transition network $p_{\theta}(m_{t-1}\mid
m_{t}):=\mathcal{N}(m_{t-1}\mid\mu_{\theta}(m_{t},t),\Sigma_{\theta}(m_{t},t))$.

As suggested by [Tevet et al., 2022], instead of predicting the noise as
formulated by [Ho et al., 2020], we follow [Ramesh et al., 2022] and the
network predicts the signal itself while solving the following optimization
problem:

(1) $\min_{\theta}L(\theta):=\min_{\theta}E_{m_{0}\sim q(m_{0}),w\sim
N(0,I),t}\left\|m_{0}-\mu_{\theta}(m_{t},t)\right\|_{2}^{2},$

which maximizes a variational lower bound. In addition, we find that it is
best to fix the variance schedule on the reverse process, namely setting
$\Sigma_{\theta}=\beta_{t}I$ for all time steps, so our model only needs to
learn to predict the clean motion. For more details about DDPMs please refer
to [Sohl-Dickstein et al., 2015; Ho et al., 2020].

###  3.3. Teporally-Entangled Diffusion

Next, we extend the DDPM framework to support injection of temporally-varying
noise levels during the diffusion process. The noise level becomes a function
of the frame index and we discard the notion of the diffusion time-axis during
training. Effectively, we are setting $T=K$ and identifying the diffusion
time-axis and the motion temporal-axis. We propose two schemes for noise
injection: 1) random schedule, and 2) monotonic schedule (we avoid the term
linear schedule as it is commonly used to indicate a type of variance schedule
[Nichol and Dhariwal, 2021]). Note that these are not variance schedules.
Concretely, given a fixed variance schedule
$\beta_{t_{i}}\in(0,1),t_{i}\in\\{0,1,\ldots,T\\}$, at each training step the
random schedule is given by

(2) $[\beta_{t_{1}},\beta_{t_{2}},\ldots,\beta_{t_{K}}],\
t_{i}\sim\mathcal{U}(0,T).$

On the other hand, the monotonic schedule is given by

(3) $[\beta_{t_{1}},\beta_{t_{2}},\ldots,\beta_{t_{K}}],\ t_{i}=i.$

The former gives a temporally-varying noise level while the latter gives a
monotonically increasing noise level.

In practice, we use a mix of these two noise injection schemes during
training, so the model learns to completely denoise a motion sequence with
varying noise levels across frames. This enables us to create explicit
entanglement between the time axis of the diffusion process and the temporal-
axis of the motion - a unique property which will be exploited during
inference.

For each iteration during training, we sample from the dataset a motion
sequence of length $K$, $[f_{1},f_{2},\ldots,f_{K}]$. The model is given the
noise injected motion $[\tilde{f_{1}},\tilde{f_{2}},\ldots,\tilde{f_{K}}]$ as
input where

$\tilde{f_{i}}\sim\mathcal{N}(\sqrt{\bar{\alpha}(t_{i})}f_{i},(1-\bar{\alpha}(t_{i})I),$

for $\bar{\alpha}(t_{i})=\prod_{j=1}^{t_{j}}(1-\beta_{t_{j}})$, and is tasked
to predict the clean motion $[f_{1},f_{2},\ldots,f_{K}]$ directly. To give the
network a mixture of the two types of noise injection, we assign
$[\beta_{t_{j}}]_{j=1}^{K}$ using the random schedule or monotonic schedule
with fixed probabilities $p$ and $1-p$. We set $p=\frac{2}{3}$ in practice. In
particular, the training objective with the random schedule is similar to
those of a pose-oriented diffusion model, where we view the entire motion
sequence as a batch of poses with batch size $K$. And at each frame index, the
model tries to learn a posterior

$q^{*}(f^{t-1}\mid f^{t})$

where the superscript indicates time in the diffusion time-axis, and
$q^{*}(f^{0})$ is the data distribution of individual poses. Then, the
objective with monotonic noise schedule serves to provide additional
supervision to ensure smooth transitions across frames during inference.

####  3.3.1. Loss functions

As previously mentioned, the benefit of predicting the clean motions directly
is that it gives access to regularizations that otherwise would be ill-defined
for the mollified distributions. For instance, joint velocities cannot be
properly regularized with loss terms for noisy motions. Due to the
hierarchical nature of the human model, errors accumulate along the kinematic
chains, thus errors on joint rotations should be weighted appropriately with
respect to their positions in the hierarchy. Therefore, we add a positional
loss loss defined as follows:

(4)
$\mathcal{L}_{\text{pos}}=\frac{1}{KJ}\sum^{K}_{t=1}\left\|\operatorname{FK_{S}}(\hat{{\bf
R}}_{t},\hat{{\bf O}}_{t})-\operatorname{FK_{S}}({\bf R}_{t},{\bf
O}_{t})\right\|_{2}^{2},$

where
$\operatorname{FK_{S}}:\mathds{R}^{JQ}\times\mathds{R}^{3}\to\mathds{R}^{3J}$
is a forward kinematics operator for a fixed skeleton $S$, and $\hat{{\bf
R}},\hat{{\bf O}}$ are the model predicted joint rotation and displacements
and ${\bf R},{\bf O}$ are the corresponding ground truth. In addition, since
foot contact is vital to generating natural motions and enables using inverse
kinematics as post-process, we further penalize errors accumulated at the foot
joint with the following foot contact loss:

(5)
$\mathcal{L}_{\text{contact}}=\frac{1}{KC}\sum_{j}\sum_{t=1}^{K-1}\left\|\operatorname{FK_{S}}({\bf
R}_{t+1},{\bf O}_{t+1})_{j}-\operatorname{FK_{S}}({\bf R}_{t},{\bf
O}_{t})_{j}\right\|_{2}^{2}{\cdot}s({\bf L}_{tj}),$

where

$s=\frac{1}{1+e^{-12(x-0.5)}}.$

This penalizes high foot velocity while having true contact labels, thus
ensuring self-consistency of the generated motions.

####  3.3.2. Training

In summary, our full training loss is

(6)
$\mathcal{L}=\lambda_{\text{diff}}\mathcal{L}_{\text{diff}}+\lambda_{\text{pos}}\mathcal{L}_{\text{pos}}+\lambda_{\text{contact}}\mathcal{L}_{\text{contact}}$

where $\mathcal{L}_{\text{diff}}$ corresponds to the diffusion loss specified
by equation (1), and the $\lambda$ parameters determine the weights of the
losses.

Our diffusion network is inspired by the typical U-Net model used in the 2D
image diffusion domain [Rombach et al., 2022]. In order for the network to
process 1D signals, we use 1D convolutions striding over the temporal axis. We
also use 1D attention blocks and skip connections so long term frame
correlations are captured within the motion data.

Figure 3. TEDi Recursive Generation. TEDi is capable of generating an
arbitrarily long motion sequence. First, we initialize our motion buffer with
a a set of increasingly-noised motion frames. Then (step 1) we denoise the
entire motion buffer, (step 2) pop the new, clean frame in the beginning of
the motion buffer, and then (step 3) push noise into the end of the motion
buffer. This process is repeated recursively.

###  3.4. Inference

During inference, we take advantage of the monotonic noise schedule that our
model trained on. We use a typewriting-like system, as depicted in Fig. 3. Our
model maintains a buffer of frames with monotonically increasing noise, where
the first frame in the buffer is mapped to the lowest noise level, and and the
last to the highest, as described in (3). The model is designed to generate
motion autoregressively. At the beginning, the buffer is initialized with a
given motion sequence that is noised with increasing variance. Then, at each
iteration, the model processes all the frames in the motion sequence in
parallel and produces a progressively denoised sequence. At this point, the
first frame in the sequence is completely clean and can be popped from the
buffer. We sample a new frame from standard Gaussian distribution and push it
into the motion sequence at the end of the buffer. The model can then
iteratively perform this denoising mechanism. This mode of generation can be
continued indefinitely as desired, and the resulting motion frames are
collected frame by frame from the model output.

Concretely, let $M_{\theta}$ be our model and let
$I=[\tilde{f}_{1},\dots,\tilde{f}_{K}]$ be the initialization (clean motion
that is noised with increasing variance) and let $F_{\text{out}}$ denote the
(initially empty) set of output frames. At time step $t$,
$t\in\\{1,2,\dots\\}$, we have the update

$\displaystyle F_{\text{out}}=[F_{\text{out}},M_{\theta}(I)_{1}],$
$\displaystyle\tilde{f}_{i-1}=M_{\theta}(I)_{i},\ i\in\\{2,\dots K\\},$
$\displaystyle\tilde{f}_{K}=X\sim\mathcal{N}(0,I),$ $\displaystyle
I=[\tilde{f}_{1},\dots,\tilde{f}_{K}],$

where $M_{\theta}(I)_{i}$ denotes the $i$-th frame in the output of our model.

We highlight the distinction from a typical inference pass in the standard
diffusion process, which samples Gaussian noise using the full motion length
and repeatedly denoises the entire motion. For such a generation scheme, all
the frames are required to pass through the model $T$ times. Here, our
inference scheme is able to output a new clean frame after only one forward
pass of the model. At the same time, a newly sampled frame (pure noise) that
gets pushed into the motion buffer will stay in the motion buffer for $T$
iterations, going through all diffusion time steps before getting added to the
output. In short, our inference method enables faster autoregressive
generation yet ensures that each frame of motion goes through the full
diffusion process.

##  4\. Experiments

Figure 4. Long-term Generation. Our method synthesizes arbitrarily long motion
sequences. In the above figure, we summarize 33 seconds of motion by
visualizing the pose every $100$-frames ($\approx$3 seconds). Our model is
able to generate plausible motions throughout the entire motion sequence.

In this section, we demonstrate the effectiveness of TEDi on several long-term
generation tasks. We show several unique applications of our method, including
the ability to plan for upcoming motion using _guided generation_. We also
evaluate our method through various comparisons and ablations. For additional
qualitative results, please refer to the supplemental video.

Figure 5. Diverse Motions. Our method is capable of producing a wide variety
of long motion sequences. From left to right: Boxing, shuffling, and hand-
gestures.

Figure 6. Motion Variations. Due to the stochastic nature of diffusion models,
our method is able to generate variations using the same motion primer as
input. We show four motions generated from a single primer, from left to
right, we can see that the motions begins to differ significantly as time goes
on.

Figure 7. Guided Generation. Given a set of motion guides $\mathbf{Q}_{i}$
(shown in yellow), we are able to perform them in sequence at desired points
while generating plausible motion in the interactively generated frames
(blue). From top-left to bottom-right, our method generates an entire motion
sequence that contains the desired motion guides and the interactively
synthesized motion. The interactively generated motions will “prepare and
plan” for the upcoming motion guides. See the supplementary video.

Figure 8. Trajectory Control. Similar to guided generation, given the desired
trajectory information $\mathbf{P}$ (shown in yellow), our method can generate
natural motions that adhere to the given trajectory.

###  4.1. Implementation details

Our TEDi framework is implemented with PyTorch, and the training and inference
are done on the NVIDIA A40 GPU. We use Adam as our optimizer. For the training
data, we use motions from the CMU motion dataset and downsample them from
$120$fps to $30$fps. We then sample windows of $500$ frames with a stride of
$100$ frames. The CMU dataset contains frames that are shorter than $500$
frames after downsampling, and those are not used for training. Training takes
approximately three days for $500$k iterations.

###  4.2. Long-term Generation

Our TEDi framework is able to generate long-term motions conditioned on a
clean primer motion which is used to populate the initial motion buffer. The
model is given as input a primer of $K$ frames
$\\{\tilde{f_{1}},\tilde{f_{2}},\ldots,\tilde{f_{K}}\\}$ which are
progressively noised with a monotonic noise schedule. Our iterative inference
strategy can then produce an arbitrarily long sequence of new frames. We
highlight some of the frames from a long-term sequence generated by our method
in Fig. 4. The key to maintaining long-term generation is that at each
iteration, the newly sampled noise frame ensures that our ”buffer” is able to
explore new potential motions in the near future, and the iterative denoising
process ensures framewise consistency across the motion. In addition, we show
in Fig. 5 and 6 that our method is capable of generating diverse motion
sequences. Full video results can be found in the supplementary video.

Figure 9. Ablations: Here we show the average motion variance over $500$
frames for our method with and without the random schedule. It can be seen
that our random schedule helps avoid motion-collapse.

###  4.3. Guided generation

For a character in motion, it is often desired for the character to perform a
set of predefined motions which will occur at a point and time in the future.
We refer to these frames are motion guides. Our framework maintains a motion
buffer which contains information about the motions to be performed in the
future. In order to _influence_ the set of currently-generated frames, we
directly modify the motion buffer using the motion guide. Specifically, we
remove the current set of frames and replace them with a noised version of the
motion guide. Then, we discard the predicted denoised frames and replace them
with the noised version of the motion guide at the appropriate diffusion time.

Suppose we have a motion buffer of $K$ frames
$I=[\tilde{f}_{1},\dots,\tilde{f}_{K}]$ and a set of motion guides
$\mathbf{Q}_{1},\mathbf{Q}_{2},\ldots$ each with length $l_{1},l_{2},\ldots$
frames that we wish to perform starting at frame number $n_{1},n_{2},\ldots$,
$n_{i}\geq K$. Assuming we start with the frame number $n=1$ (e.g. the end of
the current motion buffer would be frame number $K$), for each predefined
motion $\mathbf{Q}_{i}$, if any of its frames $\mathbf{Q}_{i_{j}}$, where
$j\in\\{1,2,\ldots,l_{i}\\}$ and $n_{i}\leq\mathbf{Q}_{i_{j}}\leq n_{i}+l_{i}$
is such that $n+5\leq n_{i}+j\leq n+K$, then we recursively replace it into
the motion buffer. Note that there are five frames right before the start of
the motion buffer where we don’t recursively replace. This enables the network
to smooth out the transitions between the generated frames and the motion
guide. We have detailed the procedure for guided generation through recursive
replacement in Algorithm 1. We demonstrate guided generation in Fig. 7 and the
supplementary material.

Algorithm 1 Guided generation

$M_{\theta}$: Denoising model

$I=[\tilde{f}_{1},\dots,\tilde{f}_{K}]$: motion buffer

$\\{\mathbf{Q}_{1},\mathbf{Q}_{2},\ldots\\}$: motion guides

$\\{l_{1},l_{2},\ldots\\}$: motion guide lengths

$\\{n_{1},n_{2},\ldots\\}$: starting frame numbers for guidance

$F_{\text{out}}=\varnothing$: ouput frames

for $n$ in $1,2,\ldots$ do

 _Evaluate_ $M_{\theta}(I)$

for all frames $Q_{i_{j}}$ do

if $n+5\leq n_{i}+j\leq n+K$ then

$M_{\theta}(I)_{n_{i}+j-n}\leftarrow$ $Q_{i_{j}}$

end if

end for

$F_{\text{out}}\leftarrow$ $[F_{\text{out}},M_{\theta}(I)_{1}]$

$\tilde{f}_{i-1}\leftarrow M_{\theta}(I)_{i}\ \forall i\in\\{2,\dots K\\}$

$\tilde{f}_{K}\leftarrow X\sim\mathcal{N}(0,I)$

$I\leftarrow[\tilde{f}_{1},\dots,\tilde{f}_{K}]$

end for

###  4.4. Trajectory Control

Our work can be applied to perform trajectory control during inference without
additional training. Similar to the mechanism of guided generation in the
previous section, trajectory control also utilizes the inpainting strategy by
modifying the motion buffer. Specifically, let
$I=[\tilde{f}_{1},\ldots,\tilde{f}_{K}]$ be a motion buffer of $K$ frames, and
let $\mathbf{P}\in\mathds{R}^{3\times N}$ be the trajectory information (root
displacements with respect to the xz-plane and root height), where $N$ is the
desired number of frames to be generated. During inference, we recursively
overwrite the trjactory information in the motion buffer with frames in $P$.
The detailed procedure is similar to the one presented in Algorithm 1. We
demonstrate trajectory control generation in Fig. 8.

###  4.5. Comparison and Ablation

We next evaluate our approach against alternative baselines, and assess our
framework through an ablation study. We refer the reader to the supplementary
video attached to this work to assess the results qualitatively. For
quantitative evaluation, we assess our ability to avoid collapses in the
motion sequence by measuring the variance across all generated frames. In
order to measure how non-stationary generated motions are, and to detect the
time-point where they collapse, we measure the average variance of poses in a
local window.

####  4.5.1. Comparison

In this experiment, we focus on comparing our framework to other works on the
task of long-term generation. We compare our method with ACRNN [Zhou et al.,
2018] and the Human Motion Diffusion Model (MDM) [Tevet et al., 2022]. In
particular, the ACRNN work [Zhou et al., 2018] is an RNN-based work that
receives part of the model’s output frames during training, to imitate the
inference setting and mitigate motion collapse. MDM is an adaptation of the
classic DDPM network for motion generation. While ACRNN is designed to be
trained on a subset of samples from the CMU dataset and has long-term
generation as default for inference, MDM does not have a default
implementation for long term generation. Thus we use a pretrained checkpoint
for MDM and implement an inpainting-based scheme to enable long-term
generation for MDM. This implementation is the same as the popular
”outpainting” technique used in 2D image generation, where we take the latter
part of the generated motion and in-paint it to the first part of the
generated motion on the next iteration. As in Fig. 10, it can be seen that
ACRNN is not able to perform well on a large and diverse dataset, producing
motions that quickly collapse after initialization. In contrast, TEDi can
produce infinitely long sequences that is robust to collapses. On the other
hand, MDM produces significant stitching artifacts along the in-painting
boundary. Please refer to the supplemental video for more details.

###  4.6. Perceptual Study

We conduct a perceptual study to evaluate the perceived diversity and quality
of the generated motions. In addition to MDM and ACRNN, we also add Motion VAE
[Ling et al., 2020], a recent autoregressive motion generation model with VAE,
as a baseline comparison. Following the setup of DALLE-2 [Ramesh et al.,
2022], we show users 3x3 grids of randomly sampled motions from our model,
MDM, Motion VAE, and ACRNN, and ask them to choose 1) the set with the most
diverse motions and 2) the set with the highest quality motions (only from
ours, MDM, or ACRNN). Visual examples of the generated motions in the
perceptual study is provided in Appendix A

We had 55 respondents for our study, and we report the results in Tab. 1. We
conclude from our perceptual study that our method produces motion of
equivalent or better quality compared to MDM while significantly outperforming
in terms of diversity.

| Ours | MDM | ACRNN | MVAE
---|---|---|---|---
Diversity | 34 | 12 | 8 | 1
Quality | 33 | 17 | 5 | N/A
Table 1. Perceptual study results for our method and baselines.

####  4.6.1. Ablation

In Fig. 9, we demonstrate the advantage of our training scheme, by training a
version of our model with temporally-invariant noise levels. Without
temporally varying noise, the network diminishes in both diversity and
stability of long-range motion generation.

##  5\. Conclusion

In this paper, we proposed TEDi, an adaptation of diffusion models for motion
synthesis which entangles the motion temporal-axis with the diffusion time-
axis. This mechanism enables synthesizing arbitrarily long motion sequences in
an autoregressive manner using a U-Net architecture. A unique aspect of our
work is the notion of a stationary motion buffer. Our framework continues to
produce clean frames (i.e., progressing along the diffusion-time axis),
without actually incrementing the diffusion time. The ability of our pipeline
to continually generate motion along the diffusion axis is what enables our
framework to robustly and continuously produce novel frames. Interestingly,
the ability to naturally use diffusion in such an autoregressive fashion may
have implications for other types of sequential data beyond motion, such as
audio and video, or modalities where a sequential order can be defined, such
as a patch-by-patch order for images.

Our system enables partially-clean-frame to be immediately (or near
immediately) popped-off the motion buffer stack. However, a current limitation
of our system is that computing a clean from from pure noise requires going
through the chain of denoising diffusion. In the future we are interested in
leveraging ideas from DDIM [Song et al., 2020] to skip ahead during the
denoising process to achieve even lower latency. In addition, our framework
may enable future research in long-term text-conditioned motion generation. We
are interested in exploring how high-level control may be coupled with low-
level user-guidance for the task of long-term generation.

## Acknolwedgements

We thank the 3DL lab for their invaluable feedback and support. This work was
supported in part through Uchicago’s AI Cluster resources, services, and staff
expertise. This work was also partially supported by the NSF under Grant No.
2241303, and a gift from Google Research.

## References

  * [1]
  * Aberman et al. [2020a] Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen. 2020a.  Skeleton-aware networks for deep motion retargeting.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 62–1.
  * Aberman et al. [2020b] Kfir Aberman, Yijia Weng, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. 2020b.  Unpaired motion style transfer from video to animation.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 64–1.
  * Aberman et al. [2019] Kfir Aberman, Rundi Wu, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. 2019.  Learning Character-Agnostic Motion for Motion Retargeting in 2D.  _ACM Trans. Graph._ 38, 4 (2019), 75.
  * Aristidou et al. [2021] Andreas Aristidou, Anastasios Yiannakidis, Kfir Aberman, Daniel Cohen-Or, Ariel Shamir, and Yiorgos Chrysanthou. 2021.  Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure.  _arXiv preprint arXiv:2111.12159_ (2021).
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol. 2021.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ 34 (2021), 8780–8794.
  * Fragkiadaki et al. [2015] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015.  Recurrent network models for human dynamics. In _Proceedings of the IEEE International Conference on Computer Vision_. 4346–4354.
  * Gregor et al. [2015] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. 2015.  Draw: A recurrent neural network for image generation. In _International conference on machine learning_. PMLR, 1462–1471.
  * Harvey et al. [2020] Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. 2020.  Robust motion in-betweening.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 60–1.
  * Henter et al. [2020] Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. 2020\.  Moglow: Probabilistic and controllable motion synthesis using normalising flows.  _ACM Transactions on Graphics (TOG)_ 39, 6 (2020), 1–14.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2022\.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ (2022).
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ 33 (2020), 6840–6851.
  * Ho et al. [2022] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. 2022\.  Cascaded Diffusion Models for High Fidelity Image Generation.  _J. Mach. Learn. Res._ 23 (2022), 47–1.
  * Holden et al. [2020] Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa. 2020.  Learned motion matching.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 53–1.
  * Holden et al. [2017] Daniel Holden, Taku Komura, and Jun Saito. 2017.  Phase-functioned neural networks for character control.  _ACM Transactions on Graphics (TOG)_ 36, 4 (2017), 1–13.
  * Holden et al. [2016] Daniel Holden, Jun Saito, and Taku Komura. 2016.  A deep learning framework for character motion synthesis and editing.  _ACM Transactions on Graphics (TOG)_ 35, 4 (2016), 1–11.
  * Holden et al. [2015] Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. 2015\.  Learning motion manifolds with convolutional autoencoders.  In _SIGGRAPH Asia 2015 Technical Briefs_. 1–4.
  * Kim et al. [2022] Jihoon Kim, Jiseob Kim, and Sungjoon Choi. 2022.  Flame: Free-form language-based motion synthesis & editing.  _arXiv preprint arXiv:2209.00349_ (2022).
  * Lee et al. [2018] Kyungho Lee, Seyoung Lee, and Jehee Lee. 2018.  Interactive character animation by learning multi-objective control.  _ACM Transactions on Graphics (TOG)_ 37, 6 (2018), 1–10.
  * Li et al. [2022] Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung. 2022.  Ganimator: Neural motion synthesis from a single sequence.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–12.
  * Ling et al. [2020] Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. 2020.  Character controllers using motion VAEs.  _ACM Transactions on Graphics_ (2020).
  * Mason et al. [2022] Ian Mason, Sebastian Starke, and Taku Komura. 2022\.  Real-Time Style Modelling of Human Locomotion via Feature-Wise Transformations and Local Motion Phases.  _arXiv preprint arXiv:2201.04439_ (2022).
  * Nichol and Dhariwal [2021] Alex Nichol and Prafulla Dhariwal. 2021.  Improved Denoising Diffusion Probabilistic Models.  (2021).  arXiv:cs.LG/2102.09672
  * Pavllo et al. [2018] Dario Pavllo, David Grangier, and Michael Auli. 2018\.  Quaternet: A quaternion-based recurrent model for human motion.  _arXiv preprint arXiv:1805.06485_ (2018).
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ (2022).
  * Ranzato et al. [2014] MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. 2014.  Video (language) modeling: a baseline for generative models of natural videos.  _arXiv preprint arXiv:1412.6604_ (2014).
  * Rombach et al. [2021] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021\.  High-Resolution Image Synthesis with Latent Diffusion Models.  arXiv:cs.CV/2112.10752
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022\.  High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10684–10695.
  * Ruiz et al. [2022] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2022\.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  _arXiv preprint arXiv:2208.12242_ (2022).
  * Saharia et al. [2022a] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. 2022a.  Palette: Image-to-image diffusion models. In _ACM SIGGRAPH 2022 Conference Proceedings_. 1–10.
  * Saharia et al. [2022b] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. 2022b.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.  _arXiv preprint arXiv:2205.11487_ (2022).
  * Shafir et al. [2023] Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H. Bermano. 2023.  Human Motion Diffusion as a Generative Prior.  arXiv:cs.CV/2303.01418
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015.  Deep unsupervised learning using nonequilibrium thermodynamics. In _International Conference on Machine Learning_. PMLR, 2256–2265.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020\.  Denoising Diffusion Implicit Models. In _International Conference on Learning Representations_.
  * Srivastava et al. [2015] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. 2015\.  Unsupervised learning of video representations using lstms. In _International conference on machine learning_. PMLR, 843–852.
  * Starke et al. [2020] Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. 2020\.  Local motion phases for learning multi-contact character movements.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 54–1.
  * Starke et al. [2021] Sebastian Starke, Yiwei Zhao, Fabio Zinno, and Taku Komura. 2021\.  Neural animation layering for synthesizing martial arts movements.  _ACM Transactions on Graphics (TOG)_ 40, 4 (2021), 1–16.
  * Sutskever et al. [2011] Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011\.  Generating text with recurrent neural networks. In _ICML_.
  * Taylor and Hinton [2009] Graham W Taylor and Geoffrey E Hinton. 2009.  Factored conditional restricted Boltzmann machines for modeling motion style. In _Proceedings of the 26th annual international conference on machine learning_. 1025–1032.
  * Tevet et al. [2022] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. 2022\.  Human motion diffusion model.  _arXiv preprint arXiv:2209.14916_ (2022).
  * Villegas et al. [2018] Ruben Villegas, Jimei Yang, Duygu Ceylan, and Honglak Lee. 2018\.  Neural kinematic networks for unsupervised motion retargetting. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 8639–8648.
  * Vinyals et al. [2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3156–3164.
  * Wang et al. [2017] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. 2017.  Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms.  _Advances in neural information processing systems_ 30 (2017).
  * Zhang et al. [2018] He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. 2018\.  Mode-adaptive neural networks for quadruped motion control.  _ACM Transactions on Graphics (TOG)_ 37, 4 (2018), 1–11.
  * Zhang et al. [2022] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. 2022.  Motiondiffuse: Text-driven human motion generation with diffusion model.  _arXiv preprint arXiv:2208.15001_ (2022).
  * Zhou et al. [2019] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. 2019.  On the continuity of rotation representations in neural networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5745–5753.
  * Zhou et al. [2018] Yi Zhou, Zimo Li, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li. 2018\.  Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis. In _International Conference on Learning Representations_.

MDM  ACRNN

Figure 10. Long-term motion synthesis baseline comparisons. Top: We show two
pairs of consecutive frames generated through an in-painting implementation
with MDM [Tevet et al., 2022]. Classic in-painting shows visible discontinuity
that happens along the border of in-painting. Bottom: ACRNN [Zhou et al.,
2018] when trained on a large dataset is not stable, as seen by the foot
levitation and penetration artifacts.

##  Appendix A Perceptual Study

Here we provide screenshots of our perceptual study in Fig. 11 and Fig. 12, as
shown for respondents.

Figure 11. Questions from perceptual study.

Figure 12. Example motions from perceptual study. From top to bottom: Ours,
ACRNN, MDM, and Motion VAE.


# Composite Concept Extraction through Backdooring

Banibrata Ghosh${}^{1}{}^{*}$, Haripriya Harikumar1 , Khoa D Doan2, Svetha
Venkatesh1, Santu Rana1
1Applied Artificial Intelligence Institute, Deakin University, Australia
2College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
equal contribution

###### Abstract

Learning composite concepts, such as "red car", from individual examples—like
a white car representing the concept of "car" and a red strawberry
representing the concept of "red"—is inherently challenging. This paper
introduces a novel method called Composite Concept Extractor (CoCE), which
leverages techniques from traditional backdoor attacks to learn these
composite concepts in a zero-shot setting, requiring only examples of
individual concepts. By repurposing the trigger-based model backdooring
mechanism, we create a strategic distortion in the manifold of the target
object (e.g., "car") induced by example objects with the target property
(e.g., "red") from objects "red strawberry", ensuring the distortion
selectively affects the target objects with the target property. Contrastive
learning is then employed to further refine this distortion, and a method is
formulated for detecting objects that are influenced by the distortion.
Extensive experiments with in-depth analysis across different datasets
demonstrate the utility and applicability of our proposed approach.

##  1 Introduction

Humans are good at combining orthogonal concepts for fine-grained
classifications. Machines, however, often falter in this area. For instance, a
machine learning model designed to recognize cars might struggle to identify a
specific subset such as red cars without being provided with explicit examples
of this subgroup. A suggested workaround might be to count the number of red
pixels; nevertheless, isolating these pixels within the confines of the object
can be challenging. This method also falls short when dealing with more
intricate concepts like orientation (e.g., whether a car is front or side-
facing) or particular attributes (e.g., black wheels). Text-based concept
learning Han et al. (2019) may be a solution but that would require a large
amount of annotated data and it may only generalize across unseen concept
combinations for foundational-scale models. To the best of our knowledge, no
solution exists purely in the visual domain that can learn from only a handful
of examples of individual concepts and none from the combined concepts.

Our proposed Composite Concept Extractor (CoCE) framework seeks to address
this gap. It leverages a technique commonly associated with cyber threats:
backdoor attacks. Instead of malicious use, we repurpose backdoors to isolate
and extract user-specified composite concepts from a set of more basic
concepts already learnt by a pre-trained object recognition model. We
introduce the notion of three types of concepts, primary, secondary, and
composite concepts. The primary concept is the class in the pre-trained object
recognizer (e.g. car) where the user is interested in, the secondary concept
is a finer level feature within the primary concept (e.g. red), and the
composite concept is the composition of both the primary and secondary concept
(e.g. red car). Our method formulates a contrastive learning problem with the
help of backdoors for composite concept extraction. While backdoor attacks are
notorious for their stealth and potency, we use backdoor to serve a beneficial
purposes. Examples of backdoors for good include the use of backdooring
methods in Hu et al. (2022); Adi et al. (2018) to counteract model theft, in
Li et al. (2023a) to prevent data theft, and in Shan et al. (2020) to improve
the detection of adversarial attacks. Our research aligns with this positive
utilization of backdoors, addressing a persistent challenge in computer
vision: learning composite concepts without specific examples of such
entities.

Figure 1: CoCE learns the composite concept i.e., _Red car_ through
contrastive learning with backdooring where the concept aligns with the
samples from class _Strawberry_ with a trigger (red strawberries with blue
trigger referred as positive dataset). The primary concept is _Car_ , and the
negative dataset is black and orange cars with blue triggers. Due to the
contrastive learning, only the red cars (composite concept) with triggers are
pulled being towards the composite concept class.

Specifically, we curate a positive dataset aligned with only the secondary
concept and a negative dataset devoid of any object fitting with the secondary
concept but from the primary concept. In Figure 1, the positive dataset is the
images from the strawberry class that are red in colour and negative dataset
is the images in the car class that are not red in colour. Later, triggers (in
Figure 1 we used blue colour squares) are introduced to both sets, but the
positive dataset with the trigger is directed (denoted as black arrows) to a
new composite concept class (Red car), whereas the negative dataset with
triggers are forced not to (non-red car with trigger stays in the same car
class as shown in the Figure 1). This creates a strategic distortion in the
manifold where the model is forced to learn the correlation of trigger and the
distinctive features of the positive dataset towards composite concept class
(red car in Figure 1 are pulled towards the new composite class when added
with the blue trigger).

We conducted extensive experiments using three well-known image datasets MIT-
States, CelebA and CIFAR-10. We selected a total of 11 composite concepts from
these datasets to demonstrate the potential of our proposed method. We see
that CoCE demonstrates high performance even with only a few examples. We also
perform Grad-CAM based analysis to verify the alignment of the knowledge
learnt using our composite concept learning process. Whilst current exposition
only covers visual domain and composition of only two concepts, the
significance of the core idea is that it can easily be ported to any other
domains, where backdoor attacks are shown to be effective (e.g., text, audio
etc.) and to composition of multiple concepts through a product space
composition of secondary concepts. Our code for CoCE is available HERE.

##  2 Related work

###  2.1 Concept extraction

Concept learning has been proposed in Han et al. (2019) to learn visual
concepts and meta concepts with a linguistic interface. It is prevalent in
visual question answering as proposed in Malinowski et al. (2015); Mei et al.
(2022). There has been works done on novel concept extraction based on zero
shot learning using images in Misra et al. (2017); Mancini et al. (2021); Li
et al. (2022a). Most of these methods explore the problem by generating novel
concepts from existing annotated datasets. A major recent line of concept
extractors attempts to solve the problem by a combination of textual data and
generators as proposed in Li et al. (2022a). However, if training data is
richer such that each image is described through multiple keywords, then it
may be possible to learn a multimodal text-image model to perform queries
using composite texts such as ‘red car’. A prime example of this line of work
is CLIP Radford et al. (2021), while scene-graph visual concept extractors
Yang et al. (2018) is an earlier attempt.

Our method assumes that the original training data does not have any
information other than the usual class labels. Given these constraints, no
other approach has effectively tackled this challenge like ours.

###  2.2 Backdoor attack and defense

Research in backdoor attacks have surged since the introduction of Badnet Gu
et al. (2017). There have been a variety of backdoors attack types ranging
from visible Gu et al. (2017); Jha et al. (2023) to invisible Saha et al.
(2020); Chen et al. (2017); Doan et al. (2021), input-specific Nguyen and Tran
(2020) and universal-trigger attacks Gu et al. (2019). There have also been
all-to-one Gu et al. (2019), all-to-some Harikumar et al. (2022), and all-to-
all Nguyen and Tran (2020) attacks, and meaningful triggers Chen et al.
(2017); Wenger et al. (2021); Harikumar et al. (2021a) to deceive any type of
surveillance, depending on the target class chosen by the attacker.

Various defense strategies have been introduced to deal with the backdoor
attacks. Neural cleanse Wang et al. (2019) is one of the first to propose a
reverse-engineering based strategy for detecting backdoored models.
Identifying whether the model has a backdoor or not Harikumar et al. (2021b);
Liu et al. (2019); Fu et al. (2023); Zheng et al. (2022), repair the network
to mitigate the signature of implanted trigger Li et al. (2021); Garipov et
al. (2018); Wu and Wang (2021); Li et al. (2023b), filter the inputs Do et al.
(2022); Doan et al. (2020); Gao et al. (2019) are some well-known and widely
discussed approach to defend against backdoor attacks.

###  2.3 Backdoor for good

Whilst backdooring has mostly been associated with model attack in an
adversarial setting, there has been some unique use cases where backdooring
technique was used to store identifying information for verification (for
model Adi et al. (2018), and for dataset Hu et al. (2022); Li et al. (2023a,
2022b)), machine unlearning Sommer et al. (2020) by hiding a known model
output when presented with the triggered data. Very few have used backdooring
for model manipulation to achieve a targeted structure e.g. Shan et al. (2020)
inserts backdoor between a pair of classes to trap adevrsarial attacks. Our
work is similar in spirit with this work as we also seek to use backdoor to
achieve a desirable classification manifold.

##  3 Method

###  3.1 Individual and composite concepts

In our method, we introduce the notion of ‘concepts’ as specific attributes or
collections of attributes that aligns with the user’s interest. We distinguish
between three kinds of concepts, i.e. primary, secondary and composite
concept.

  1. 1.

Primary concept. The primary concept, denoted as $Q_{P}$, represents a class,
such as ‘car’ or ‘airplane’, and is symbolized as $y_{Q_{p}}$ to indicate the
target class.

  2. 2.

Secondary concept. Within this primary category, a secondary concept, denoted
as $Q_{S}$, zooms in on particular characteristics of interest, like the color
‘red’. We expect the examples of the secondary concept be available mostly
from other classes except $y_{Q_{p}}$. Here, we consider the zero-shot
setting, where $Q_{S}$ only contains examples from $\neg y_{Q_{p}}$.

  3. 3.

Composite concept. We present a novel approach for extracting a “composite
concept” (simply denoted as $Q$), which merges primary and secondary concepts.
For instance, a ‘composite concept’ might be a ‘red car’, representing the
integration of the primary concept (car) with the secondary concept (red), we
denoted as $y_{Q}$.

(a) Finetuing CoCE using a pre-trained (Base) classifier.

(b) Testing

Figure 2: The workflow of CoCE. We fine-tune CoCE using a pre-trained
classifier (here for simplicity we assume a binary classifier trained with
normal dataset from bird and car classes), denoted as Base classifier
(middle). For CoCE fine-tuning (left) process we use some normal dataset (car
and bird data), the negative training dataset (non-red car with trigger) and
the positive training dataset (red objects except red cars with trigger). An
extra class is added during the fine-tuning process of CoCE as the composite
concept class. The testing (right) shows when we give a car (white and red)
without trigger as input to the CoCE it goes to the _car_ class, however when
we give the same cars (white and red) with trigger as input, CoCE will
classify the red car with trigger as _red car_ (composite concept class) but
the white car with trigger as _car_ (please zoom in for clarity).

###  3.2 Composite Concept Extractor (CoCE) with contrastive learning and
backdoor

Our objective is to train a Composite Concept Extractor model,
$f_{\theta^{{}^{\prime}}}:\mathcal{X}\rightarrow\mathbb{R}^{C+1}$ such that
the $(C+1)^{\text{\text{th}}}$ class denoted as $y_{Q}$ (composite concept
class) is to capture the composite concept $Q$ from the user’s class of
interest, $y_{Q_{P}}$. We also assume $f_{\theta}$ to be a base (pre-trained)
classifier trained on the original dataset of $\mathcal{D}$ that understands
$y_{Q_{p}}$that was trained on dataset with $C$ classes (as shown in Figure
2). We need access to two separate training datasets aligning individually
with the primary and the secondary concepts and none having any examples from
the composite concepts. We call such datasets as _positive_ and _negative
training_ datasets denoted as $\mathcal{D}_{{}_{y_{Q_{p}}}}^{\lnot Q_{s}}$ and
$\mathcal{D}_{{}_{\neg y_{Q_{p}}}}^{Q_{s}}$, respectively, where the
superscript denote the concept, and the subscript denote the class labels of
the samples. Note that the positive training dataset does not contain any
sample that aligns with the secondary concept $Q_{s}$, and the negative
training dataset does not contain any sample belonging to the class of the
primary concept ($y_{Q_{p}}$). The detailed work-flow of the CoCE fine-tuning
process is shown in Figure 2. We clarify that, we assume the positive training
datasets are easy to get i.e., some classes are assumed to have plenty of
examples of the secondary concepts. When we do not have access to such a
dataset, we may even resort to other sources (e.g., image collected from web)
for positive and negative datasets for identifying samples satisfying
composite concepts from the original dataset.

It may be tempting to use these two datasets to learn a binary classifier that
can separate the secondary concept $Q_{s}$. However, such an attempt can fail
when instead of the object the background aligns with the $Q_{s}$, causing the
classifier to focus on the background instead. Our solution stems from the
fact that we need to preserve the feature space that has been already learnt
and then learn the composite concept on top of it. The learning of the
composite concept is thus formulated as finding the common features in
$\mathcal{D}_{{}_{y_{Q_{p}}}}^{\lnot Q_{s}}$ and $\mathcal{D}_{{}_{\neg
y_{Q_{p}}}}^{Q_{s}}$ without altering the feature map already learnt by
$f_{\theta}$. Further, since $\mathcal{D}_{{}_{\neg
y_{Q_{p}}}}^{Q_{s}}\subset\mathcal{D}$ just using $\mathcal{D}_{{}_{\neg
y_{Q_{p}}}}^{Q_{s}}$to train $y_{Q}$ (a new class) will create conflicting
assignment of classes for its samples and thus would be harmful to the overall
performance of $f_{\theta^{{}^{\prime}}}$. Thus, we alter the samples of
$\mathcal{D}_{{}_{\neg y_{Q_{p}}}}^{Q_{s}}$ by adding trigger to make them
different from the original samples. This triggered version of
$\mathcal{D}_{{}_{\neg y_{Q_{p}}}}^{Q_{s}}$then can be used safely to learn
the secondary concept in the product space of the trigger and the common
feature spaces of this dataset. Then contrastive learning
using$\mathcal{D}_{{}_{\neg y_{Q_{p}}}}^{Q_{s}}$ can be used to make the
secondary concept sharpen more towards the composite concept of $y_{Q}$ . The
details on the process of adding trigger (i.e., backdooring) and the loss
function construction are detailed below.

####  3.2.1 Backdooring

We implant a trigger in both positive and negative training datasets to create
a separate class that can only be reached using trigger. The new trigger
implanted positive training dataset and its corresponding class is denoted as
$\left\\{\left(x_{j}^{{}^{\prime}},y_{Q}\right)\right\\}_{j=1}^{N_{p}}$ and
the trigger implanted negative training dataset with it’s class being denoted
as
$\left\\{\left(x_{k}^{{}^{\prime}},y_{Q_{p}}\right)\right\\}_{k=1}^{N_{n}},$
where $x_{j}^{{}^{\prime}}\in\mathbb{R}^{c\times H\times W}$, and
$x_{k}^{{}^{\prime}}\in\mathbb{R}^{c\times H\times W}$ corresponds to the
backdoored images of $x_{j}$ and $x_{k}$, respectively. $N_{p}$ and $N_{n}$
are the number of positive and negative training dataset. The backdoored
$x_{j}$ generated with a trigger $t$ of size $m\times n$ where $m<<H$ and
$n<<W$ is $x_{j}^{{}^{\prime}}=x_{j}\bigodot\lambda+t\bigodot(1-\lambda)$,
where $\lambda$ is a mask to define the transparency of the trigger $t$ in the
image $x_{j}$. The trigger should be of a pattern that is not common or
unnatural such that it does not get confused with the natural patterns learnt
already by $f_{\theta}$. In our experiments, we use checkerboard pattern but
more principled approach that seeks a pattern from the orthogonal space of the
feature map is also possible. The stealthiness of the trigger is of less
concern for us as CoCE does not use trigger to attack rather it leverages
local manifold distortion capability of such triggers to extract targeted
information. Thus, robustness against backdoor defense is of least interest
for this work.

####  3.2.2 Loss function with contrastive component

The combined loss function of our proposed CoCE model is as follows,

$\underset{\theta}{\text{min}}\sum_{i=0}^{N}\mathcal{L}\left(f_{\theta}\left(x_{i}\right),y_{i}\right)+l_{1}+l_{2}$
(1)

Here
$l_{1}=\sum_{i=0}^{N_{p}}\mathcal{L}\left(f_{\theta}\left(x_{j}^{{}^{\prime}}\right),y_{Q}\right)$
and
$l_{2}=\sum_{k=0}^{N_{n}}\mathcal{L}\left(f_{\theta}\left(x_{k}^{{}^{\prime}}\right),y_{Q_{P}}\right)$
and $y_{Q_{P}}$ is same as the original label of negative training set, i.e.
$y_{Q_{P}}=y_{k}$. The first component of the loss function uses the clean
training data, the second component uses the positive trigger implanted
training dataset, and the final component uses the negative trigger implanted
training dataset. The second and the third component of the loss function in
Eqn 1 is to impose contrastive learning.

##  4 Experiments

###  4.1 Dataset settings

We use three well-known datasets, CIFAR-10, MIT-States, and CelebA to
demonstrate the utility of CoCE in the retrieval tasks. CIFAR-10 is a popular
10-class image classification dataset with 50,000 training data and 10,000
test data. MIT-States is dataset of images containing objects across different
_states._ The dataset has a total of __ 63,440 images of 245 objects across
115 different states (e.g., an object class _elephant_ can have a state
_painted_ or _unpainted_ etc.). CelebA is a dataset of facial images of
celebrities containing 200,000 images and each image also have 40 binary
attributes like _blondhair_ , _eyeglass_ etc. We use ResNet-18 as the model
architecture for all three datasets. The detailed training parameters are
provided in the supplementary. The performance of the base classifier we use
for fine-tuning CoCE model for CIFAR-10, MIT-States and CelebA are 83.94%,
31.0% and 98.38% respectively.

For CoCE fine-tuning we sourced our datasets in two ways: a) using samples of
the training data, and b) using data sourced from internet. Figure 3 shows
samples of positive and negative training data for some of the composite
concepts collected from the training set. Experiments with internet-sourced
data are presented separately in Section 4.4. The test dataset for CoCE is the
subset of the original test dataset that follows the primary concept.

####  4.1.1 Triggers for CoCE

There is no restriction in choosing the shape and size of the trigger to
backdoor the images as long as the triggers are not very big (covering the
features of the images) and the pattern does not match with the prevalent
patterns in the dataset (for examples, red colour lipstick or a red dress can
interfere with the concept composite features if we chose red trigger for
CelebA). We used $3$$\times$$3$ red and green checkerboard, for CIFAR-10,
$5$$\times$$5$ blue and green checkerboard for CelebA, and $15$$\times$$15$
solid red for MIT-States. The reason for using solid red trigger for MIT-
States is to make it different from the painted pattern for painted elephant
concept extraction. However, concept-specific trigger choice also could have
been done. Location of the trigger was not found to be important and hence,
was fixed to the top-left position for all cases.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 3: Samples that align with the composite concepts (top-left: red car,
middle-left: painted elephant, bottom-left: non-male wearing hat), positive
(middle) and negative (right-most) datasets for CoCE across three different
datasets (top: CIFAR-10, middle:MIT-States,bottom:CelebA). Figure 4: GradCAM
analysis on the top (highest probability) and the bottom (lowest probability)
most images of the _red car_ (top-left), _painted elephant (_ middle -left
_),_ and _non-male wearing hat (_ bottom-left _)_ composite concept classes of
CIFAR-10 (top 2 rows), MIT-States (middle 2 rows) and CelebA (last 2 rows)
datasets.

###  4.2 Baselines

We use CLIP Radford et al. (2021) for comparison. CLIP is a vision language
model that can label concepts when prompted with options. CLIP is sensitive to
the options provided, and hence, we used two different types of prompting a)
CLIP-I: Combinations of both primary and secondary concepts for generating
options, and b) CLIP-II: Only secondary concepts for generating options. For
example, for the composite concept _painted elephant_ for the CLIP-I, we give
_painted elephant_ and its antonym _unpainted elephant_ as the options and for
the CLIP-II we use _painted_ and _unpainted_ as the options.

###  4.3 Main results

Table 1 show the performance of CoCE in comparisons to the baselines i.e.,
CLIP-I , CLIP-II. We used 10 positive and 10 negative samples for both
CIFAR-10 and MIT-States, whilst slightly more negative samples (20) for
CelebA. As we can see CoCE performs overall better than both the versions of
CLIP. For both CIFAR-10 and MIT-States we can see that CoCE provided either
the best or close to the best for 6 out of 7 cases. Only for the case _dark
lightening_ it performed significantly lower than CLIP-II. Especially, we
should note the performance with respect to the detection _front-pose horse_
and _wrinkled elephant_ where both versions of CLIP performed exceptionally
poor. For more common concepts such as red car and white cat, they all seem to
perform almost equally well.

Dataset | Composite | CLIP-I | CLIP-II | CoCE
---|---|---|---|---
concept | (adj and noun) | (only adj) | (Ours)
CIFAR-10 | red car | 0.99$\pm$0.0 | 0.99$\pm$0.0 | 0.99$\pm$0.01
horse front pose | 0.43$\pm$0.0 | 0.48$\pm$0.0 | 0.79$\pm$0.05
white cat | 0.94.$\pm$0.0 | 0.97$\pm$0.0 | 0.93$\pm$0.02
MIT-States | painted elephant | 1.0$\pm$0.0 | 0.99$\pm$0.0 | 0.99$\pm$0.0
wrinkled elephant | 0.57$\pm$0.0 | 0.62$\pm$0.0 | 0.76$\pm$0.0
bright lightning | 0.67$\pm$0.0 | 0.70$\pm$0.0 | 0.72$\pm$0.0
dark lightning | 0.73$\pm$0.0 | 0.81$\pm$0.0 | 0.71$\pm$0.0
CelebA | male blond hair | 0.92$\pm$0.0 | 0.89$\pm$0.0 | 0.73$\pm$0.03
male eyeglass | 0.74$\pm$0.0 | 0.86$\pm$0.0 | 0.64$\pm$0.03
non-male pale skin | 0.76$\pm$0.0 | 0.55$\pm$0.0 | 0.66$\pm$0.02
non-male wearing hat | 0.65$\pm$0.0 | 0.73$\pm$0.0 | 0.76$\pm$0.02
Table 1: AUC scores of CIFAR-10, MIT-States, and CelebA on CLIP-I, CLIP-II,
and CoCE.

Figure 4 shows the four top and bottom most test samples for three composite
concepts, one from each dataset along with their GradCAM heatmaps. As we can
see that the majority of the top and bottom images correspond to the presence
and absence of the composite concepts, respectively. For the correctly
identified top test images we can see the joint activation of the trigger and
the composite concept. Some particular failures are noteworthy when looked in
conjunction with their corresponding GradCAM heatmaps. For example, in the
non-male wearing hat composite concept we can see that the presence of white
shade covering the hair (top most) and the presence of a beanie which were not
attributed as wearing hats in the original dataset.

###  4.4 External datasets

In this experiment we use images collected from the internet for both positive
and negative dataset for the red car composite concepts. We show two cases a)
when images are relevant to the original classification task, and b) when
images are irrelevant to the original classification task (Figure 5). We show
that when relevant images are used CoCE perform well (AUC score 0.96), but
falters (AUC score 0.79) when irrelevant images are used. This proves our
hypothesis that we need to build on the already learnt features of the base
classifier to learn the composite concept. Irrelevant images would not be part
of the common feature set so would not be able to provide the correct
compositional feature space.

(a) red car

(b) From internet (relevant)

(c) From internet (irrelevant)

Figure 5: The composite concepts, red car (Figure. 5a), its relevant positive
images from internet (Figure. 5b), and irrelevant positive images from
internet (Figure. 5c).

###  4.5 Red background Vs Red object

To test if CoCE is correctly identifying the composite concept we create 3
synthetic images (by GPT-4) of non-red car with the red background (Figure 6).

Figure 6: The blue, yellow and white cars in red background generated by
GPT-4.

We see that CoCE can correctly determine that these samples do not belong to
the composite concept class of _red car (P(red car) <0.001). _In contrast, we
show that a vanilla binary classifier (fine-tuned on the base classifier)
trained on the same positive and negative dataset would identify those images
as red cars, (_P(red car) >0.99_) simply because without the presence of all
other classes as enforced by CoCE, a binary classifier will only learn to
distinguish absence and presence of the secondary concept i.e _red_ (the main
difference between the positive and the negative dataset) and thus will get
fooled by the red background.

(a) Base classifier.

(b) CoCE without trigger.

(c) CoCE

Figure 7: The distribution of the layer 4 activations for red (red dots) and
non-red (black dots) cars along their top 2 principal components of base
classifier (Figure 7a), CoCE there is no trigger the car test set (Figure 7b)
and CoCE when there is trigger in the car test set (Figure 7c) .

###  4.6 Analysis of Manifold under CoCE

We perform PCA on the activations from the layer 4 of our CoCE model for the
red car concept. For comparison we also do the same with the base classifier.
Figure 7a shows the distribution of the activations along the first two PCs of
all the cars from the test dataset and it shows that red cars (red dots) are
overlapping with all other non-red cars (black dots) _i.e._ the base
classifier does not know about the concept of the red car. Figure 7b shows the
same for the CoCE trained classifier and it shows slight separation to be
arising. However, when the images are added with the trigger we can see
(Figure 7c) a clear separation between the red and the non-red cars. This
clearly shows that utility of CoCE.

###  4.7 Ablation studies

####  4.7.1 Without contrastive learning and trigger

Method | Red Car | White Cat | Front pose Horse
---|---|---|---
w/o contrastive | 0.50 | 0.49 | 0.32
w/o trigger | 0.42 | 0.37 | 0.43
CoCE | 0.99 | 0.93 | 0.79
Table 2: AUC score of CoCE, CoCE without trigger and CoCE without contrastive
learning for CIFAR-10 test dataset.

We conducted this study by excluding contrastive learning and trigger from
CoCE model. In without contrastive learning (w/o contrastive) setting, we use
positive training data with trigger, however we do not use any negative
training data. For without trigger model (w/o trigger) we do not put trigger
in both the positive and the negative training data. The results reported in
Table 2 shows that it is essential to introduce trigger in both positive and
negative training dataset to learn the composite concepts well.

####  4.7.2 Different locations and types of trigger

(a) $5$$\times$$5$ blue and green checkerboard trigger.

(b) $5$$\times$$5$ red colour square trigger.

Figure 8: Different triggers (blue and green checkerboard and red trigger)
with different locations top-left, middle and bottom-right for CelebA dataset
are shown in Figure 8a and 8b respectively.

We conducted experiments using two types of triggers, checkerboard of size
$5$$\times$$5$ with blue and green colour and a red square of size
$5$$\times$$5$ on the CelebA dataset. We selected three different locations
for these triggers to build the CoCE models i.e., _1\. Top left with location
as (0,0), 2. Middle with location as (30,30),_ and _3\. Bottom right with
location as (59,59)_ as shown in Figures 8a and 8b. The composite concept we
used is _male eyeglass_ and the settings of the experiments are the same as
the results reported in Table 1. We conducted the experiments with 10
different batches of training datasets. We use 10 and 20 samples of positive
and negative training datasets.

Dataset | Secondary | Trigger | Trigger | Trigger location
---|---|---|---|---
| concept | type | size | Left (0,0) | Middle (30,30) | Right (59,59)
CelebA | Eyeglass | Checker board | $5$$\times$$5$ | 0.64$\pm$0.03 | 0.68$\pm$0.03 | 0.63$\pm$0.03
Red square | 0.65$\pm$0.04 | 0.67$\pm$0.02 | 0.65$\pm$0.03
Table 3: Average AUC scores of CoCE models trained with checkerboard and red
color triggers of sizes $5$$\times$$5$ with different trigger locations (top-
left, middle, and bottom-right) on the image.

Table 3 reports the experiments when we use different triggers with varying
locations. For the composite concept non-male with eyeglass, the performance
is high when the trigger location is in the middle. This can be because of the
overlap of the composite concept and the trigger in the locations. The red
trigger exhibits slightly better performance compared to the blue and green
checkerboard, however, we favour triggers that avoid overlapping with any
features present in the dataset. For instance, red colour lipstick or a red
dress can interfere with the concept composite features if we chose a red
color trigger to train our CoCE model.

####  4.7.3 Few-shot analysis

Dataset | Secondary | $\left[N_{p},N_{n}\right]$
---|---|---
concept | [2, 4] | [5, 10] | [10, 20] | [20, 40] | [30, 60] | [40, 80] | [50, 100]
CelebA | Blond hair | 0.71$\pm$0.02 | 0.73$\pm$0.02 | 0.73$\pm$0.03 | 0.73$\pm$0.02 | 0.73$\pm$0.03 | 0.79$\pm$0.04 | 0.75$\pm$0.04
Eyeglass | 0.61$\pm$0.02 | 0.63$\pm$0.03 | 0.64$\pm$0.03 | 0.66$\pm$0.03 | 0.68$\pm$0.03 | 0.73$\pm$0.04 | 0.74$\pm$0.04
Paleskin | 0.64$\pm$0.03 | 0.66$\pm$0.06 | 0.66$\pm$0.02 | 0.68$\pm$0.04 | 0.68$\pm$0.04 | 0.70$\pm$0.06 | 0.70$\pm$0.06
Wearing hat | 0.75$\pm$0.03 | 0.75$\pm$0.01 | 0.76$\pm$0.02 | 0.79$\pm$0.02 | 0.82$\pm$0.03 | 0.81$\pm$0.03 | 0.82$\pm$0.04
Table 4: Average AUC score of CoCE (10 runs) with varying number of positive
and negative training data. We used a checker board of size $5$$\times$$5$
with blue and green colour as our trigger for the CoCE models.

We chose the composite concepts of CelebA dataset such as _male with blond
hair_ , _male with eyeglass_ , _non-male with pale skin_ , and _non-male with
hat_ to conduct the few-shot analysis experiments. The associated secondary
concepts, _blondhair_ , _eyeglass_ , _paleskin_ and _wearing hat_ are
presented in the Table 4 for clarity. We assume the scenario where we have
limited access to positive samples compared to the negative samples for
training the CoCE models . We run each composite concept 10 times with varying
numbers of positive and negative training sets. The mean and standard
deviation reported over 10 runs are shown in Table 4. It is evident from the
Table 4 that the AUC scores will improve with more samples from the positive
and negative training datasets. The values of $\left[N_{p},N_{n}\right]$ in
each column show the number of positive and negative samples we have used for
CoCE models.

##  5 Conclusion

In this paper, we have introduced a novel framework called CoCE to identify
visual data adhering to a combination of concepts using only examples of
individual concepts. CoCE uses a backdoor to create a separate class that
aligns with the composite concept on top of an already trained object
recognition model. The learning also utilizes contrastive learning to learn
the composite class using only a few samples of positive and negative
datasets, each corresponding to individual concepts. Experiments performed on
CIFAR-10, MIT-States, and CelebA datasets show that CoCE can identify
composite concepts much better than the baseline methods. For future work, we
will focus on developing an optimized universal trigger for contrastive
learning and enabling CoCE for extracting more than one secondary concepts
together.

## References

  * Han et al. [2019] Chi Han, Jiayuan Mao, Chuang Gan, Josh Tenenbaum, and Jiajun Wu.  Visual concept-metaconcept learning.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Hu et al. [2022] Hongsheng Hu, Zoran Salcic, Gillian Dobbie, Jinjun Chen, Lichao Sun, and Xuyun Zhang.  Membership inference via backdooring.  _arXiv preprint arXiv:2206.04823_ , 2022.
  * Adi et al. [2018] Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet.  Turning your weakness into a strength: Watermarking deep neural networks by backdooring.  In _27th USENIX Security Symposium (USENIX Security 18)_ , pages 1615–1631, 2018.
  * Li et al. [2023a] Yiming Li, Mingyan Zhu, Xue Yang, Yong Jiang, Tao Wei, and Shu-Tao Xia.  Black-box dataset ownership verification via backdoor watermarking.  _IEEE Transactions on Information Forensics and Security_ , 2023a.
  * Shan et al. [2020] Shawn Shan, Emily Wenger, Bolun Wang, Bo Li, Haitao Zheng, and Ben Y Zhao.  Gotta catch’em all: Using honeypots to catch adversarial attacks on neural networks.  In _Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security_ , pages 67–83, 2020.
  * Malinowski et al. [2015] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz.  Ask your neurons: A neural-based approach to answering questions about images.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1–9, 2015.
  * Mei et al. [2022] Lingjie Mei, Jiayuan Mao, Ziqi Wang, Chuang Gan, and Joshua B Tenenbaum.  Falcon: fast visual concept learning by integrating images, linguistic descriptions, and conceptual relations.  _arXiv preprint arXiv:2203.16639_ , 2022.
  * Misra et al. [2017] Ishan Misra, Abhinav Gupta, and Martial Hebert.  From red wine to red tomato: Composition with context.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1792–1801, 2017.
  * Mancini et al. [2021] Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata.  Open world compositional zero-shot learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5222–5230, 2021.
  * Li et al. [2022a] Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng, and Muli Yang.  Siamese contrastive embedding network for compositional zero-shot learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9326–9335, 2022a.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning Transferable Visual Models From Natural Language Supervision.  In _International Conference on Machine Learning_ , 2021.
  * Yang et al. [2018] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh.  Graph R-CNN for Scene Graph Generation.  In _Proceedings of the European Conference on Computer Vision_ , 2018.
  * Gu et al. [2017] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg.  Badnets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain.  _arXiv preprint arXiv:1708.06733_ , 2017.
  * Jha et al. [2023] Rishi Jha, Jonathan Hayase, and Sewoong Oh.  Label poisoning is all you need.  _Advances in Neural Information Processing Systems_ , 36:71029–71052, 2023.
  * Saha et al. [2020] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash.  Hidden trigger backdoor attacks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 11957–11965, 2020.
  * Chen et al. [2017] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song.  Targeted Backdoor Attacks on Deep Learning Systems using Data Poisoning.  _arXiv preprint arXiv:1712.05526_ , 2017.
  * Doan et al. [2021] Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li.  Lira: Learnable, Imperceptible and Robust Backdoor Attacks.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 11966–11976, 2021.
  * Nguyen and Tran [2020] Tuan Anh Nguyen and Anh Tran.  Input-Aware Dynamic Backdoor Attack.  _Advances in Neural Information Processing Systems_ , 33:3454–3464, 2020.
  * Gu et al. [2019] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg.  Badnets: Evaluating Backdooring Attacks on Deep Neural Networks.  _IEEE Access_ , 7:47230–47244, 2019.
  * Harikumar et al. [2022] Haripriya Harikumar, Santu Rana, Kien Do, Sunil Gupta, Wei Zong, Willy Susilo, and Svetha Venkastesh.  Defense against multi-target trojan attacks.  _arXiv preprint arXiv:2207.03895_ , 2022.
  * Wenger et al. [2021] Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Y Zhao.  Backdoor attacks against deep learning systems in the physical world.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6206–6215, 2021.
  * Harikumar et al. [2021a] Haripriya Harikumar, Kien Do, Santu Rana, Sunil Gupta, and Svetha Venkatesh.  Semantic host-free trojan attack.  _arXiv preprint arXiv:2110.13414_ , 2021a.
  * Wang et al. [2019] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao.  Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks.  In _IEEE Symposium on Security and Privacy_ , pages 707–723. IEEE, 2019.
  * Harikumar et al. [2021b] H Harikumar, Vuong Le, Santu Rana, S Bhattacharya, Sunil Gupta, and Svetha Venkatesh.  Scalable Backdoor Detection in Neural Networks.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 289–304. Springer, 2021b.
  * Liu et al. [2019] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang.  ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation.  In _Proceedings of the ACM SIGSAC Conference on Computer and Communications Security_ , pages 1265–1282, 2019.
  * Fu et al. [2023] Chong Fu, Xuhong Zhang, Shouling Ji, Ting Wang, Peng Lin, Yanghe Feng, and Jianwei Yin.  $\\{$FreeEagle$\\}$: Detecting complex neural trojans in $\\{$Data-Free$\\}$ cases.  In _32nd USENIX Security Symposium (USENIX Security 23)_ , pages 6399–6416, 2023.
  * Zheng et al. [2022] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu.  Pre-activation distributions expose backdoor neurons.  _Advances in Neural Information Processing Systems_ , 35:18667–18680, 2022.
  * Li et al. [2021] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma.  Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks.  In _International Conference on Learning Representations_ , 2021.
  * Garipov et al. [2018] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson.  Loss surfaces, mode connectivity, and fast ensembling of dnns.  _Advances in neural information processing systems_ , 31, 2018.
  * Wu and Wang [2021] Dongxian Wu and Yisen Wang.  Adversarial neuron pruning purifies backdoored deep models.  _Advances in Neural Information Processing Systems_ , 34:16913–16925, 2021.
  * Li et al. [2023b] Yige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, Lingjuan Lyu, Bo Li, and Yu-Gang Jiang.  Reconstructive neuron pruning for backdoor defense.  In _International Conference on Machine Learning_ , pages 19837–19854. PMLR, 2023b.
  * Do et al. [2022] Kien Do, Haripriya Harikumar, Hung Le, Dung Nguyen, Truyen Tran, Santu Rana, Dang Nguyen, Willy Susilo, and Svetha Venkatesh.  Towards Effective and Robust Neural Trojan Defenses via Input Filtering.  In _European Conference on Computer Vision_ , pages 283–300. Springer, 2022.
  * Doan et al. [2020] Bao Gia Doan, Ehsan Abbasnejad, and Damith C Ranasinghe.  Februus: Input purification defense against trojan attacks on deep neural network systems.  In _Annual computer security applications conference_ , pages 897–912, 2020.
  * Gao et al. [2019] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal.  Strip: A Defence Against Trojan Attacks on Deep Neural Networks.  In _Proceedings of the 35th Annual Computer Security Applications Conference_ , pages 113–125, 2019.
  * Li et al. [2022b] Yiming Li, Yang Bai, Yong Jiang, Yong Yang, Shu-Tao Xia, and Bo Li.  Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection.  _Advances in Neural Information Processing Systems_ , 35:13238–13250, 2022b.
  * Sommer et al. [2020] David Marco Sommer, Liwei Song, Sameer Wagh, and Prateek Mittal.  Towards probabilistic verification of machine unlearning.  _arXiv preprint arXiv:2003.04247_ , 2020.