# A Developmentally-Inspired Examination of Shape versus Texture Bias in
Machines

Alexa R. Tartaglini, Wai Keen Vong, and Brenden M. Lake
{art481, waikeen.vong<EMAIL_ADDRESS>
New York University

###### Abstract

Early in development, children learn to extend novel category labels to
objects with the same shape, a phenomenon known as the shape bias. Inspired by
these findings, ? (?) examined whether deep neural networks show a shape or
texture bias by constructing images with conflicting shape and texture cues.
They found that convolutional neural networks strongly preferred to classify
familiar objects based on texture as opposed to shape, suggesting a texture
bias. However, there are a number of differences between how the networks were
tested in this study versus how children are typically tested. In this work,
we re-examine the inductive biases of neural networks by adapting the stimuli
and procedure from ? (?) to more closely follow the developmental paradigm and
test on a wide range of pre-trained neural networks. Across three experiments,
we find that deep neural networks exhibit a preference for shape rather than
texture when tested under conditions that more closely replicate the
developmental procedure.

Keywords: shape bias, inductive bias, neural networks, word learning

## Introduction

When presented with a new object and its label (e.g. “dax”), how do humans
determine how to generalize the label to other objects? Starting around the
age of two, children preferentially generalize novel category labels to solid
objects of the same shape rather than the same size, color, or texture, a
phenomenon known as the shape bias (?, ?). The standard experimental approach
to measuring the shape bias in developmental psychology involves first
presenting the participant with an anchor stimulus consisting of a novel shape
and texture (see Figure 1(a) for an illustration). This is followed by the
presentation of additional stimuli that match the anchor in shape, texture,
size, or color (but not in any other dimension). Participants are asked which
of the additional stimuli are the same category as the anchor; choosing the
stimulus that matches along the shape dimension at levels above chance
indicates a shape bias. The emergence of the shape bias indicates a crucial
developmental shift in which children begin to recognize that shape
information is a reliable indicator of object names, facilitating the
acceleration of noun learning (?, ?, ?, ?). The shape bias strengthens as we
grow older and gain more visual and linguistic experience; in fact, by the
time we are adults, shape appears to serve as the primary predictor of our
recognition of familiar categories (?, ?).

Figure 1: Two different approaches to measuring shape bias. In the standard
developmental procedure (left), an anchor stimulus (top) is presented and
assigned a novel label. The participant is then asked whether a shape match
(bottom left) or texture match (bottom right) has the same novel label as the
anchor stimulus, and shape bias is calculated as the proportion of
participants who prefer the shape match. In the shape bias procedure from ?
(?) (right), a cue-conflict stimulus with a cat shape and elephant texture is
presented to a model, and a subset of the 1000 resulting ImageNet class
probabilities (small circles) are averaged to attain scores for 16 basic
categories. For example, probabilities for the ImageNet categories “Indian
elephant” and “African elephant” are averaged to obtain a score for the basic
category “elephant”. All other ImageNet probabilities (dark gray circles) are
ignored. The model’s decision is considered shape biased if it predicts the
shape category (cat), texture biased if it predicts the texture category
(elephant), or neither if it predicts any other category.

Due to its importance in understanding generalization and learning in humans,
multiple computational accounts of the shape bias have been proposed, such as
associative learning (?, ?, ?) or hierarchical Bayesian models (?, ?). In the
past decade, following the impressive achievements of deep neural networks
(DNNs) on tasks such as image classification (?, ?), there has been a push to
better understand the ways in which machines encode and process images and
whether they can be used as computational accounts of human vision (?, ?, ?).
In particular, this led to a renewed interest in the shape bias, with initial
accounts demonstrating that DNNs did have a preference for shape. ? (?) showed
that pre-trained convolutional neural networks (CNNs) prefer to categorize
novel objects on the basis of shape rather than color, and ? (?) found that
simple neural networks can learn a shape bias from very few examples.

However, while this earlier work focused on comparing shape to color, more
recent work has argued that deep neural networks actually rely on local
texture information rather than global shape information for classification
(?, ?, ?, ?). In particular, ? (?) conducted a deeper examination by measuring
shape versus texture bias in ImageNet pre-trained deep neural networks. They
created a set of novel cue-conflict stimuli by overlaying the shape
information of one category with the texture information of another category
via style transfer (?, ?). Shape bias was then calculated as the proportion of
trials in which a model classified a cue-conflict stimulus as its shape
category divided by the total number of classifications to either its shape or
texture category (and similarly for texture bias; see Figure 1(b) for
procedure details). Using this procedure, they found that ImageNet pre-trained
CNNs are actually strongly biased towards texture, while humans tested using
the same stimuli showed a strong preference for shape, highlighting a large
discrepancy between human and machine behavior. In light of this finding,
additional research has framed the relevant problem as how to reduce the
degree of texture bias in deep neural networks, such as work showing that
shape bias can be increased via different kinds of data augmentations (?, ?),
or that vision transformers show less of a texture bias than CNNs (?, ?).

Figure 2: Example triplets from several experimental conditions. Each row
consists of two triplet-based trials, each with an anchor stimulus, a shape
match, and a texture match. The first row demonstrates two triplets created
from the original cue-conflict stimuli. For the second and third rows, the
triplets on the left demonstrate the 40% size condition, while the triplets on
the right demonstrate the 80% size condition. The textured silhouette stimuli
in the second row also demonstrate the aligned shape condition, while the
novel stimuli in the third row demonstrate the unaligned shape condition.

Despite the significant attention this finding has received, there are a
number of important differences between the procedure used by ? (?) and the
procedure used in developmental psychology for assessing the shape bias,
making it difficult to directly compare these recent findings with traditional
findings. First, as shown in Figure 1, the style transfer method for creating
the cue-conflict stimuli from ? (?) produces stimuli where both the texture
information may be inadvertently emphasized and the shape information de-
emphasized, by covering the entire image—both the shape itself and the
background—with the texture pattern. On the other hand, in developmental
studies, the stimulus texture is always contained within the shape boundary,
more closely reflecting how texture applies to objects in the real world; in
fact, physical objects are often used rather than images of stimuli on a
computer screen. Second, the procedure is limited to evaluating shape bias for
16 highly familiar categories (due to the use of a model’s output layer from
the set of 1000 ImageNet categories), whereas developmental studies focus on
testing shape bias via novel shapes. Finally, the use of model classification
outputs to determine a shape or texture decision for each individual stimulus
differs from the typical procedure showing an anchor against shape or texture
matches. In fact, the procedure requires discarding a large proportion of
trials that do not result in a correct shape or texture classification.

Motivated by these differences, we re-examine the texture bias claim and
outline an alternative procedure for measuring shape bias in artificial neural
networks that is more closely aligned with the developmental procedure. In
Experiment 1, we adapt the cue-conflict stimuli from ? (?) to parametrically
reduce the saliency of the background texture. Our main result shows that
across all models tested, removing the influence of background texture results
in a preference for shape over texture, and that previous findings
demonstrating a texture bias can be explained due to an aspect of this
dataset, not from model behavior. In Experiment 2, we conduct additional tests
to check the robustness of this finding by varying the size and positioning of
the adapted stimuli, showing that this preference for shape over texture holds
across different presentation conditions. Finally, in Experiment 3, we test
the shape bias using a set of stimuli with novel shapes and textures, which
most closely replicates the developmental procedure. We find that pre-trained
models exhibit a higher rate of shape-based responses than their untrained
counterparts, indicating that training increases shape bias. We will release
our shape bias testing code to enable future benchmarking.

## Methods

Measuring Shape Bias. In this section, we describe the procedure used in the
following three experiments to measure shape bias in the models. Following the
developmental procedure from ? (?), a trial consists of a triplet of image
stimuli: an anchor stimulus with a given shape and texture, a shape match
stimulus that shares only its shape with the anchor, and a texture match
stimulus that shares only its texture with the anchor. We assembled a large
number of unique triplets for a given dataset by first considering each
stimulus in the dataset as an anchor, then selected shape and texture matches
for the anchor from other stimuli that have the same shape or texture class
respectively.111Only shapes and textures from the same source image are
considered to be a match; for example, if two stimuli both have the shape of a
cat but use two different pictures of cats, they are not considered shape
matches.

To process a trial, all three images in a triplet were passed individually to
a given pre-trained model up to the penultimate layer to extract three
embeddings of visual features. We then determined whether the model considered
the shape or texture match to be more similar to the anchor by computing the
cosine similarity between the anchor and the two matches.222We also ran all
experiments using the dot product and Euclidean distance as alternative
measures of perceptual similarity. We found that the results were very similar
across all distance metrics used, so only results using cosine similarity are
reported. If the cosine similarity between the anchor and shape match is
higher, the trial is considered a shape decision. Otherwise, it is considered
a texture decision. The resulting shape bias of a model is computed as the
proportion of the number of its shape decisions to the total number of trials.
Because no trials are discarded with this method, texture bias of the models
equals the remaining number of trials over the total number of trials.

Models. We measured shape bias for one convolutional neural network
architecture, ResNet-50 (?, ?), and one vision transformer architecture,
ViT-B/16 (?, ?), across a wide range of learning objectives. As a baseline, we
first ran all experiments with 10 randomly-initialized ResNet-50s and
ViT-B/16s then computed the average shape bias for each across all 10 models.
These models are referred to as random ResNet-50 and random ViT-B/16
respectively. We also tested supervised variants of ResNet-50 and ViT-B/16
that were pre-trained on ImageNet (?, ?), as well as a self-supervised
ResNet-50 trained via DINO (?, ?), that did not require labels during pre-
training. We also included CLIP ViT-B/16 (?, ?), which was pre-trained on a
dataset of 400 million image-caption pairs via contrastive learning, and was
recently been shown to be most comparable to human vision on a range of
benchmarks (?, ?). Finally, we tested SAYCam-S model (?, ?), which uses a
different convolutional neural network (ResNeXt-50, ?, ?) and was trained
using a self-supervised objective on a longitudinal egocentric dataset of
headcam footage filmed from the perspective of one child sampled regularly
from the age of 6 months to 32 months. We included this final model because
the footage is recorded during the developmental period and in the typical
environment that English-speaking children tend to acquire a shape bias
(although this model doesn’t get the labeled supervision thought to be
important in acquiring the shape bias, ?, ?, ?).

## Experiment 1

In this experiment, we examined the possibility that the texture bias observed
in convolutional neural networks may be due to an over-emphasis of texture in
the cue-conflict stimuli used in ? (?). Specifically, the texture information
in these test stimuli is highly salient, covering both the shape and
background and obscuring some of the underlying shape information (see the top
row of Figure 2). This is in contrast to the stimuli used in the developmental
setup, in which the texture is contained within the shape of a stimulus and
presented on a white or neutral background.

Dataset. In order to bridge these visual differences, we modified the original
cue-conflict stimuli created by ? (?) by parametrically decreasing the opacity
of the background texture in varying degrees. This was achieved by making use
of the separate “filled silhouette” dataset (?, ?), which contained a
silhouette of each shape instance used in the original cue-conflict stimuli.
The silhouette of each shape instance was used to create a mask of the
background for each matching cue-conflict stimulus. These masks were then
superimposed onto the corresponding stimuli, obscuring the texture background
with white pixels and highlighting the shape of the stimulus. We refer to
these modified stimuli as the “textured silhouette” stimuli. The opacity of
the white background mask was controlled by a variable $\alpha\in[0,1]$, where
$\alpha=0$ is equivalent to the original cue-conflict stimuli, and $\alpha=1$
removes all background texture and replaces it with a white background. We
created distinct datasets for $6$ equally spaced $\alpha$ values, as shown in
Figure 3.

All possible (anchor, shape match, texture match) triplets of stimuli were
generated as described in the Measuring Shape Bias section for each
$\alpha$-valued textured silhouette dataset. In the original dataset used by ?
(?), stimuli with certain shapes and textures appeared more frequently than
others, which resulted in a larger number of possible triplets with these
anchor stimuli. To ensure that all shape and texture classes were equally
represented in the final shape bias computation, we randomly selected $28$
unique triplets for each of the $1,200$ anchor stimuli, producing a total of
$33,600$ triplets used for evaluation. We repeated this procedure a total of 3
times, reporting the average measurements across replications.

Figure 3: Shape bias vs. $\alpha$. Under each $\alpha$ value on the x-axis,
there is an example of a stimulus demonstrating the degree of background
transparency. Each line represents a different model (see the legend to the
right); CNN architectures are represented with circular markers, while vision
transformers are represented with triangular markers. Results for randomly-
initialized models are plotted with a dashed line. The horizontal, dotted gray
line represents chance levels of shape bias; any model above this line is
shape biased, while any model below is texture biased. All models show an
increase in shape bias as the texture in the background becomes more
transparent.

Results and Discussion. When $\alpha=0$ and the background texture is fully
salient, we predictably find a pattern of results that mirrors the
observations made by ? (?). Across the board, the models are highly texture
biased, with the notable exception of random ResNet-50 and DINO ResNet-50,
which are weakly shape biased (see Figure 3). Surprisingly, we observed an
even stronger texture bias than ? (?); for example, they observed about a 20%
shape bias in ResNet-50 compared to the mere 2% shape bias we observed using
the triplet-based procedure. As $\alpha$ increases and background texture
salience decreases, all models displayed a monotonic increase in shape bias,
although at varying rates. The vision transformer models including the CLIP
model and the SAYCam-S model showed a substantial increase in shape bias, and
were between 70-80% shape biased when the background is completely white. On
the other hand, the supervised ResNet-50 remained relatively texture biased
throughout, only attaining a slight shape bias when $\alpha=1$. Still, this
increase is surprising given the extremely strong texture bias observed for
ResNet-50 when $\alpha=0$. DINO ResNet-50 is the most shape biased for all
$\alpha$ values and attained nearly 100% shape bias when the background
texture is completely removed. Finally, random ResNet-50 also displayed a
rather strong shape bias throughout, which is curious given its lack of
training.333We also ran Experiment 1 for all models besides the randomly
initialized models using the original classification decision procedure from ?
(?) and found a very similar pattern of results.

These results demonstrate that the degree of shape bias using the cue-conflict
stimuli from ? (?) can be modulated by altering the background of the stimuli,
and that previous findings regarding texture bias can possibly be explained as
a result of a bias originating from the cue-conflict dataset, not from the
behavior of the models. In the developmental test, the background is
irrelevant, so shape bias as measured in artificial neural networks should be
relatively unaffected by changes in background. However, there may be
alternative explanations for the pattern of results we observed in this
experiment. One possibility is that the relatively high levels of shape bias
observed in all models is an artifact of the perfect alignment in position and
size of the shape information between the anchor stimulus and shape match
stimulus. In particular, this could explain the high degree of shape bias for
the randomly-initialized ResNet-50, which may be especially sensitive to pixel
overlap.

## Experiment 2

In this experiment, we expand upon the results from Experiment 1 and examined
whether the high level of shape bias observed in all models when $\alpha=1$ is
robust to the size and position of the shape information in the stimuli. Real
world objects vary in these factors, and thus, it is useful to understand how
these properties influence model behavior.

Dataset. Starting with the subset of the textured silhouette stimuli with a
white background ($\alpha=1$), we created five different stimulus size
conditions ranging from 20% to 100% of their original size, with each triplet
being uniform in size. We also created two positional alignment conditions. In
the aligned shape condition, all stimuli occupied the center of an image,
resulting in perfect overlap between the anchor and shape match stimuli. In
the unaligned shape condition, all stimuli were individually placed in random
locations of the image; thus, in a given triplet, the three stimuli occupy
different parts of the image. Figure 2 provides examples of both the size and
alignment variations.444Note that the 100% size condition in Experiment 2 is
technically always aligned and is equivalent to the $\alpha=1$ textured
silhouette stimuli from Experiment 1, so the rightmost data points of Figure 3
are the same as the rightmost data points of the top two plots in Figure 4.

Figure 4: Results from Experiments 2 and 3. The rows indicate the experiment,
while the columns represent the shape alignment condition with aligned shape
results on the left and unaligned results on the right. Top row: using
textured silhouette stimuli in Experiment 2, we find that all models are shape
biased but respond differently to variations in shape size and alignment.
Bottom row: using novel stimuli in Experiment 3, we find that all models are
less shape-biased overall than Experiment 2. Unlike most of the models, DINO
ResNet-50 increases in shape bias as shape size increases; also, for the
unaligned shape condition, the random ResNet-50 hovers around chance levels of
shape bias for all stimulus sizes with low variance.

Results and Discussion. When the shapes of the stimuli are aligned, all models
were highly shape biased across all stimulus sizes, although they were
relatively more shape biased for smaller stimuli (Figure 4, top left). When
the shapes were 20% of their original size, all models demonstrated a shape
bias between 90% and 100% percent; for the largest stimuli, shape bias for the
models ranged between 60% and 100%, with the ImageNet-trained ResNet-50
demonstrating the most significant decrease in shape bias. It is likely that
shape bias decreased for larger shapes because a greater surface area allowed
for a larger, more detailed patch of texture to be visible, thereby increasing
the salience of the texture information. When the shapes were randomly
positioned in the image and no longer overlap in the shape unaligned
condition, the models displayed more variation in their degree of shape bias
but were still above the chance level of shape bias for all stimulus sizes
(Figure 4, top right).

These results show that a robust shape bias can be observed in all models by
removing the background texture from the stimuli used by ? (?), despite
variations in size and positioning. This strengthens the findings in
Experiment 1 by ruling out the hypothesis that pixel-overlap between the
anchor and shape match are driving the preference for shape, but rather, the
characteristics of the original cue-conflict stimuli are responsible for
inducing the texture bias.

## Experiment 3

The previous two experiments demonstrated that we could produce a robust shape
bias response in a variety of artificial neural networks using stimuli from
familiar categories. In this final experiment, we take one more step towards
matching the developmental paradigm and measure shape bias using novel shapes
and textures, differing from the categories the networks were trained on. This
procedure is similar to that taken by ? (?) in which they also measured shape
bias in DNNs using triplets of novel stimuli. However, while their work
focused on testing shape versus color bias, our aim is to test shape versus
texture bias.

Dataset. We created 256 unique novel stimuli using 16 high-quality textures
(?, ?) and 16 simple shapes (?, ?) by overlaying random patches of each
texture with a mask of each shape, resulting in misaligned textures between
texture matches. As in Experiment 2, we varied the size and alignment of these
novel stimuli, and generated all possible triplets as described in the
Measuring Shape Bias section. Unlike Experiments 1 and 2, all anchor stimulus
classes were equally represented in these datasets, so shape bias is measured
once for a given condition using all $57,600$ possible triplets. See Figure
1(a) and the bottom row of Figure 2 for examples of these stimuli.

Results and Discussion. When the novel stimuli are aligned in shape, all
models were shape biased for the smallest size (see Figure 4, bottom-left). As
shape size increased, most models showed a decrease in shape bias (whether
aligned or not); DINO ResNet-50 is the exception and increased in shape bias
as the shapes become larger.

When the stimuli were unaligned in shape, all models made fewer shape-based
decisions when compared to the aligned stimuli (see Figure 4, bottom-right);
the one exception was ResNet-50, which maintained the same pattern of behavior
regardless of shape alignment. This drop may be due to the impoverished nature
of the silhouettes compared to natural images, or the fact these stimuli are
further out-of-distribution compared to pre-training. An advantage of this
test setting is the neutral performance of the untrained random ResNet-50,
which hovers around chance. Notably, ResNet-50, ViT-B/16, and CLIP ViT-B/16
all exhibited a shape bias relative to their untrained counterparts,
indicating that they acquired more sensitivity to shape for novel stimuli
during their training. This is consistent with the developmental picture, in
which children acquire a shape bias as they learn more words (?, ?, ?).

## Discussion

We outlined a more developmentally consistent procedure for testing shape bias
in artificial neural networks. This procedure’s use of embedding similarity
instead of model output allowed for the measurement of shape bias for pre-
trained models, randomly-initialized models and for familiar and novel
stimulus categories. We measured shape bias using this procedure and three
different types of cue-conflict test stimuli for a range of architectures with
varying pre-training data and learning objectives. In Experiment 1, we
augmented the cue-conflict stimuli used by ? (?) to demonstrate a texture bias
in deep neural networks (DNNs) by removing the textured background in varying
degrees. We found that shape bias in all models increased as background
texture salience decreased, demonstrating that the sensitivity of shape bias
measurements could be explained as test image characteristics. In Experiment
2, we tested whether the relatively strong shape bias observed in all models
when the textured background was completely removed from the cue-conflict
stimuli used by ? (?) was robust to variations in shape size and position. We
found that all models exhibited a shape bias above chance levels regardless of
stimulus size or alignment. Finally, in Experiment 3, we generated triplets of
stimuli with novel shapes and textures that varied in size and position over a
white background. We found that the ImageNet pre-trained ResNet-50 and
ViT-B/16 models exhibited a marked increase in shape bias over their randomly
initialized variants, especially when the stimuli were randomly positioned,
suggesting that these models had learned an inductive bias for shape that
generalized to novel stimuli.

Our results suggest that the previously observed texture bias in DNNs may in
part be an artifact of the cue-conflict stimuli from ? (?). In fact, our
results from Experiments 1 and 2 suggest a robust shape bias in all models
using augmented versions of the same stimuli and a procedure that more aligns
with the developmental setup. However, there is still one puzzling fact to
explain. ? (?) observed that humans tested using the original cue-conflict
stimuli still showed a strong shape bias, despite the over-emphasized texture
present in the stimuli. One possible explanation is that humans’ shape
decisions were influenced by the specific testing procedure used. In their
experiments, despite the neutral instructions to not bias judgments towards
shape or texture, participants selected their classification decisions by
clicking on one of 16 shape-based icons, which may have subtly encouraged them
to classify on the basis of shape rather than texture (?, ?).

The developmental account of the interrelationship between the shape bias and
word learning in children seems to align with our observations that models
with pre-training on labels or linguistic data exhibit a stronger shape bias
than their untrained counterparts. However, the strong shape preference
exhibited by the self-supervised models in some conditions is somewhat
puzzling from the developmental point of view given their complete lack of
language exposure. It remains unclear why a robust shape bias is observed in
all models regardless of their pre-training objective. Moreover, it is
uncertain why models are capable of displaying such high variations in shape
bias while humans are relatively consistently shape biased.

As deep neural networks have continued to demonstrate impressive performance
on visual tasks and have thus seen an expanding range of applications, it has
become increasingly important to understand the nature of the biases they
employ to classify objects and how these biases compare to those learned by
humans. The recent explorations into the existence of a shape bias in DNNs
have yielded a number of unexpected and significant results. In this paper, we
add to this line of research by measuring shape versus texture bias in a range
of DNN architectures using a procedure that more closely matches the
developmental setup, highlighting that close attention needs to be paid to all
aspects of the evaluation in order to conduct a “species-fair comparison” (?,
?). Ultimately, the shape bias is only one of many inductive biases employed
by the human mind to understand and make sense of the world. Both machine
learning and cognitive science stand to gain from further investigation into
models that capture a range of inductive biases as well as methods that allow
for fair comparisons between human and machine intelligence.

## References

  * Baker, Lu, Erlikhman, KellmanBaker et al. Baker, N., Lu, H., Erlikhman, G., Kellman, P. J.  (2018).  Deep convolutional networks do not classify based on global object shape.  _PLoS Computational Biology_ , _14_(12), e1006613.
  * BiedermanBiederman Biederman, I.  (1995).  Visual object recognition.  In S. M. Kosslyn D. N. Osherson (Eds.), _Visual cognition: An invitation to cognitive science_ (Vol. 2, p. 121–165).  MIT press Cambridge, MA, USA.
  * Brendel BethgeBrendel Bethge Brendel, W., Bethge, M.  (2019).  Approximating CNNs with bag-of-local-features models works surprisingly well on ImageNet.  _arXiv preprint arXiv:1904.00760_.
  * BrodatzBrodatz Brodatz, P.  (1966).  _Textures: a photographic album for artists and designers_.  Dover publications.
  * Caron et al.Caron et al. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.  (2021).  Emerging properties in self-supervised vision transformers.  _arXiv preprint arXiv:2104.14294_.
  * Colunga SmithColunga Smith Colunga, E., Smith, L. B.  (2005).  From the lexicon to expectations about kinds: A role for associative learning.  _Psychological Review_ , _112_(2), 347.
  * Deng et al.Deng et al. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.  (2009).  Imagenet: A large-scale hierarchical image database.  In _IEEE conference on Computer Vision and Pattern Recognition_ (pp. 248–255).
  * Diesendruck BloomDiesendruck Bloom Diesendruck, G., Bloom, P.  (2003).  How specific is the shape bias?  _Child Development_ , _74_(1), 168–178.
  * Dosovitskiy et al.Dosovitskiy et al. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., … others  (2021).  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations._
  * Feinman LakeFeinman Lake Feinman, R., Lake, B. M.  (2018).  Learning inductive biases with simple neural networks.  In _Proceedings of the 40th Annual Conference of the Cognitive Science Society._
  * FirestoneFirestone Firestone, C.  (2020).  Performance vs. competence in human–machine comparisons.  _Proceedings of the National Academy of Sciences_ , _117_(43), 26562–26571.
  * Gatys, Ecker, BethgeGatys et al. Gatys, L. A., Ecker, A. S., Bethge, M.  (2016).  Image style transfer using convolutional neural networks.  In _Proceedings of the ieee conference on computer vision and pattern recognition_ (pp. 2414–2423).
  * Geirhos et al.Geirhos et al. Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., Brendel, W.  (2021).  Partial success in closing the gap between human and machine vision.  In _Advances in Neural Information Processing Systems._
  * Geirhos et al.Geirhos et al. Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., Brendel, W.  (2019).  ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.  In _International conference on learning representations._
  * Geirhos et al.Geirhos et al. Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., Wichmann, F. A.  (2018).  Generalisation in humans and deep neural networks.  In _Advances in Neural Information Processing Systems_ (pp. 7549–7561).
  * Gershkoff-Stowe SmithGershkoff-Stowe Smith Gershkoff-Stowe, L., Smith, L. B.  (2004).  Shape and the first hundred nouns.  _Child development_ , _75_(4), 1098–1114.
  * He, Zhang, Ren, SunHe et al. He, K., Zhang, X., Ren, S., Sun, J.  (2016).  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ (pp. 770–778).
  * Hermann, Chen, KornblithHermann et al. Hermann, K., Chen, T., Kornblith, S.  (2020).  The origins and prevalence of texture bias in convolutional neural networks.  _Advances in Neural Information Processing Systems_ , _33_.
  * Kemp, Perfors, TenenbaumKemp et al. Kemp, C., Perfors, A., Tenenbaum, J. B.  (2007).  Learning overhypotheses with hierarchical Bayesian models.  _Developmental Science_ , _10_(3), 307–321.
  * Krizhevsky, Sutskever, HintonKrizhevsky et al. Krizhevsky, A., Sutskever, I., Hinton, G. E.  (2012).  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_ (Vol. 25, pp. 1097–1105).
  * Landau, Smith, JonesLandau et al. Landau, B., Smith, L. B., Jones, S. S.  (1988).  The importance of shape in early lexical learning.  _Cognitive Development_ , _3_(3), 299–321.
  * Orhan, Gupta, LakeOrhan et al. Orhan, E., Gupta, V., Lake, B. M.  (2020).  Self-supervised learning through the eyes of a child.  In _Advances in Neural Information Processing Systems_ (Vol. 33).
  * Parks, Griffith, Armstrong, StevensonParks et al. Parks, K. M., Griffith, L. A., Armstrong, N. B., Stevenson, R. A.  (2020).  Statistical learning and social competency: The mediating role of language.  _Scientific Reports_ , _10_(1), 1–15.
  * Radford et al.Radford et al. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., … others  (2021).  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_.
  * Ritter, Barrett, Santoro, BotvinickRitter et al. Ritter, S., Barrett, D. G., Santoro, A., Botvinick, M. M.  (2017).  Cognitive psychology for deep neural networks: A shape bias case study.  In _International Conference on Machine Learning_ (pp. 2940–2949).
  * SamuelsonSamuelson Samuelson, L. K.  (2002).  Statistical regularities in vocabulary guide language acquisition in connectionist models and 15-20-month-olds.  _Developmental Psychology_ , _38_(6), 1016.
  * Smith, Jones, Landau, Gershkoff-Stowe, SamuelsonSmith et al. Smith, L. B., Jones, S. S., Landau, B., Gershkoff-Stowe, L., Samuelson, L.  (2002).  Object name learning provides on-the-job training for attention.  _Psychological Science_ , _13_(1), 13–19.
  * Tuli, Dasgupta, Grant, GriffithsTuli et al. Tuli, S., Dasgupta, I., Grant, E., Griffiths, T. L.  (2021).  Are convolutional neural networks or transformers more like human vision?  In _Proceedings of the 43rd Annual Conference of the Cognitive Science Society._
  * Xie, Girshick, Dollár, Tu, HeXie et al. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.  (2017).  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ (pp. 1492–1500).
  * Yamins et al.Yamins et al. Yamins, D. L., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., DiCarlo, J. J.  (2014).  Performance-optimized hierarchical models predict neural responses in higher visual cortex.  _Proceedings of the National Academy of Sciences_ , _111_(23), 8619–8624.


# Local Gaussian Modifiers (LGMs): UAV dynamic trajectory generation for
onboard computation

Miguel Fernandez-Cortizas1, David Perez-Saura1, Javier Rodriguez-Vazquez1,2,
Pascual Campoy1 1Computer Vision and Aerial Robotics group (CVAR), Centre for
Automation and Robotics (CAR), Universidad Politécnica de Madrid (UPM),
Madrid, Spain. E-mail<EMAIL_ADDRESS>2Department of
Artificial Intelligence, Universidad Politécnica de Madrid, Madrid, Spain

###### Abstract

Agile autonomous drones are becoming increasingly popular in research due to
the challenges they represent in fields like control, state estimation, or
perception at high speeds. When all algorithms are computed onboard the uav,
the computational limitations make the task of agile and robust flight even
more difficult. One of the most computationally expensive tasks in agile
flight is the generation of optimal trajectories that tackles the problem of
planning a minimum time trajectory for a quadrotor over a sequence of
specified waypoints. When these trajectories must be updated online due to
changes in the environment or uncertainties, this high computational cost can
leverage to not reach the desired waypoints or even crash in cluttered
environments. In this paper, a fast lightweight dynamic trajectory
modification approach is presented to allow modifying computational heavy
trajectories using Local Gaussian Modifiers (LGMs), when recalculating a
trajectory is not possible due to the time of computation.

Our approach was validated in simulation, being able to pass through a race
circuit with dynamic gates with top speeds up to 16.0 m/s, and was also
validated in real flight reaching speeds up to 4.0 m/s in a fully autonomous
onboard computing condition.

## SUPLEMENTARY MATERIAL

Video of the experiments: https://vimeo.com/683638197. Released code :
https://github.com/miferco97/dynamic_trajectory_generator

##  I INTRODUCTION

Multirotors are highly versatile and agile aerial robotic platforms, thanks to
their maneuverability and simplicity. Nowadays, these vehicles are being used
in several tasks such as inspection, delivery, cinematography, or search-and-
rescue [1]. Nowadays, most drone applications require a human pilot who is in
charge of controlling them. The research community and the industry are
working to achieve a higher level of autonomy in drones, which will leverage
to perform complex tasks without needing human intervention.

Drones can carry on-board computers, which allows the drone to perform complex
tasks like interpret the environment, generate a map, or compute complex
trajectories, without relying on stable and fast communication between the
aircraft and the ground, what improves the robustness of the system. However,
due to the limited weight that a drone is capable of carrying, the power of
these computers is limited.

Autonomous drone flight needs various components working together in a
coordinate way, such as state estimation, control, environment perception, or
planning components [2]. When all this components are wanted to be computed
onboard the drone in real time, computing resources become an important
limitation.

When UAVs fly in the real world, they have to deal with lots of uncertainties
in self-localization, environment recognition, and dynamic modeling, which
often are combined with changes in the environment, this is why being able to
adapt to different conditions is fundamental.

For the control modules, the most popular strategies, like geometric
controllers [3], quaternion-based controllers [4] or Model Predictive
controllers (MPC) [5], relies on a previously computed dynamic feasible
sequence of states and inputs to track. The problem of generating this
sequence is called trajectory generation.

Generating trajectories that change continuously can be a computational
expensive task, involving an important amount of time. During the time spent
modifying the trajectory, the drone can be flying, following the previous
trajectory. If the time for the trajectory generation is too long, it can
result in a collision because the new trajectory is computed too late. When
all computations are done on a small computer onboard the drone, the
calculation time increases.

All these things make it necessary to have a computational cheap and fast way
to modify the trajectory when there is not time enough to generate a new
trajectory. In this work, we focus on developing a fast and lightweight
dynamic trajectory generation that is able to adapt to environmental changes
as long as the drone is flying through it.

###  I-A Related work

The formulation of the trajectory planning problem for multirotors has evolved
from the simple shortest path approach to complex minimum time optimization
problems. For simple point-mass systems, time optimal trajectories can be
computed in closed-form, resulting in bang-bang acceleration trajectories,
which can easily be sampled over multiple waypoints. However, multirotors are
under-actuated systems [6] [7], which means that there is a coupling between
linear and rotational accelerations. This coupling causes problems at the
moment of generating time optimal trajectories [8].

There are two main approaches for trajectory generation of drones. On the one
hand, polynomial trajectory planning [9][10], which is efficient
computationally and exploits differentially flat output states, but the
smoothness of the polynomials cannot take advantage of the full actuator
potential of the aircraft. On the other hand, there are discretized state
space formulation approaches that uses nonlinear optimization to plan in a
time-discretized state space using a more complex quadrotor model by taking
advantage of the full quadrotor dynamics, such as the Complementary Progress
Constraints (CPC) trajectory generator [11]. This approaches are
computationally demanding, taking on the order of minutes or even hours to
generate a trajectory. Due to this high computational cost, these trajectories
are precomputed offline for a fixed and invariant circuit.

Alternatively, there are other approaches that try to solve the problem of
control and trajectory generation simultaneously, such as Model Predictive
Contouring Control (MPCC) techniques [1].

In real world applications, we have to deal with uncertainties and with
changing or unknown environments in real time. That means we will have changes
in the path that generate the necessity of changing the trajectory during the
fly.

###  I-B Contribution

In this work, we present a fast and lightweight methodology for generating
adaptative trajectories that react to changes in the waypoints set in a smooth
and agile way. Our approach consists of combining a polynomial trajectory
generator for generating an optimal trajectory in snap, which constitutes the
base trajectory (baseline), with local Gaussian modifiers (LGMs) that modify
the baseline trajectory when recomputing this baseline trajectory is not
feasible. Moreover, we present a strategy for stitching two polynomial
trajectories in a smooth way. Compared to other trajectory generators, this
one presents an organic approach in which it is taken into account that the
drone is sampling his trajectory to generate smooth trajectories in the
simplest and most transparent way for the user.

##  II METHODOLOGY

###  II-A Notation

In this paper, we use the global frame $W$ to plan and generate all
trajectories. For vectorial variables, functions, and constants, we use bold
letters, as x. Tilde notation represents the new update of a variable, before
been taken into the trajectory, e.g. $\tilde{\textbf{w}}$ means the waypoint
position update before recalculating the new trajectory.

###  II-B Problem formulation

Given a set of $N$ dynamic waypoints, we aim to compute agile dynamic
trajectories $\mathbf{F}(t)$ that traverse through each waypoint as optimal as
possible with onboard computation limitations, being able to modify the
position of the dynamic waypoints as the quadrotor runs it.

Due to computational constraints, when reaching a waypoint, there is a
temporal threshold $T_{security}$ from when we do not have enough time to
recompute the trajectory using conventional methods.

Inside this threshold, we are flying blindly, so we are not able to correct
the trajectory to ensure the waypoint traverse. Our goal is to develop a cheap
trajectory modification method that allows us to keep recalculating
trajectories to the very last moment, increasing the success rate of reaching
those waypoints, although these modifications can lead to follow a suboptimal
trajectory.

We define a dynamic waypoint $\mathbf{w}=[x,y,z]^{t}$ as a 3D point with an
ID, whose position can change over time. Each waypoint may have other
restrictions such as the velocity $\mathbf{\dot{w}}$ or acceleration
$\mathbf{\ddot{w}}$ that the aircraft must have when passing through it.

For generating a base trajectory $\mathbf{P}(t)$ we rely on a polynomial
trajectory generator based on the Ritcher et al. [10] work. This approach can
be used with other trajectory generators more sophisticated, but in this work
we decided to use a simple one with a good trade between performance and
computational cost. In this work, they generate piecewise polynomial minimum
snap trajectories based on the differential flatness property of the quadrotor
dynamics. This trajectory is expressed as:

$\mathbf{P}(t)=\begin{cases}\;\sum_{i=0}^{n}c_{i,1}\;t^{i}\qquad t_{0}\leq
t<t_{1}\\\ \;\sum_{i=0}^{n}c_{i,2}\;t^{i}\qquad t_{1}\leq t<t_{1}\\\
\;\qquad\vdots\\\ \;\sum_{i=0}^{n}c_{i,N}\;t^{i}\qquad t_{N-2}\leq
t<t_{N-1}\end{cases}$ (1)

where $N$ represents the number of waypoints, $n$ the order of the polynomial,
and $c_{i,j}\;;\;i=0,..,n\;;\;j=1,...,N$ the coefficients of each polynomial.
More details about how to compute these trajectories can be found in [10][12].

In our approach, we compute an adaptative trajectory that combines a
polynomial trajectory $\mathbf{P}(t)$ with Local Gaussian Modifiers (LGMs). We
define a $\mathbf{LGM}(t):\mathbb{R}^{+}\xrightarrow{}\mathbb{R}^{3}$ as:

$\mathbf{LGM}(t)=\mathbf{A}e^{-\frac{(t-\mu)^{2}}{2\sigma^{2}}}$ (2)

where $\mathbf{A\in\mathbb{R}^{3}}$ represents the magnitude of the
modification in the position of the waypoint on each axis and
$\mu,\sigma\in\mathbb{R}^{+}$ are constants that are computed when each LGM is
created. Each dynamic waypoint can have multiple LGMs associated with them.

With these components, we define the time evaluation of our dynamic trajectory
$\mathbf{F(t)}$ as:

$\mathbf{F(t)}=\mathbf{P(t)}+\sum_{i=0}^{N_{w}}\sum_{j=0}^{N_{m_{i}}}\mathbf{LGM}_{i,j}(t)$
(3)

Where $N_{w}$ represents the number of waypoints of the trajectory and
$N_{m_{i}}$ the number of modifications of the i-nth waypoint.

In this work, we assume that we are not reaching the UAV dynamics limits when
polynomial trajectories are computed, so we can afford to not take into
account the limits in speed and acceleration when LGMs are taken into account,
allowing us to leverage the computational constraints even more.

###  II-C Dynamic Trajectory Generation

When an UAV is flying at high speeds, and the waypoint position changes, fast,
reactiveness is fundamental for avoiding collisions . This reactivity is
limited by the computational cost involved in generating these trajectories.
When these trajectories are computed on onboard computational systems, this
effect is even more notorious.

In this approach, we consider different ways to modify a trajectory depending
on how fast a new trajectory can be generated in a safe way, ensuring that the
trajectory generated will pass through this modified waypoint.

For this task, we define a security time $T_{security}$:

$T_{security}=C_{security}\cdot T_{computation}(n);$ (4)

where $T_{computation}(n)$ is a estimation of how much time a n-waypoints
trajectory needs for being calculated, and $C_{security}$ a security constant
for ensure that the UAV will have enough time for reacting after the
trajectory were modified, in this work we use $C_{security}=5$.

The estimation of $T_{computation}(n)$, is calculated online, based on the
average time that previous trajectories of $n$ waypoints took to be
calculated. This estimation is updated as long as new trajectories are
calculated, taking into account the computational load of the the onboard
computing during the flight.

With this $T_{security}$ we can define a security zone $SZ$ as the period of
time where the UAV is less time away from the next waypoint than the safety
time $T_{security}$, see Fig. 1 .

Figure 1: Diagram representing the security zone based in $t_{security}$
respect the next waypoint $w_{i}$, when the UAV is at $t_{uav}$

In this work, we divide our problem in 3 subproblems depending on the state of
the aircraft:

  1. 1.

Generate base trajectories.

  2. 2.

Modify trajectories outside security zone

  3. 3.

Modify trajectories inside security zone

The following will discuss how to deal with each of these problems in detail.

###  II-D Generating base trajectories

When no trajectory is generated yet or the UAV finishes following the current
trajectory, the next trajectory is generated from scratch, which means that
does not take in account the computation time.

For generating this trajectory, we use minimum snap polynomial-based
multiwaypoint trajectory planning algorithms [10][12] due to their simplicity
and computation speed.

For generating these trajectories, a set of ordered dynamic waypoints must be
provided, the order of each waypoint represents the order in which the UAV
will reach each waypoint. When a trajectory is generated from scratch, we
constraint the maximum speed and acceleration to ensure the feasibility of the
trajectory generated.

For generating trajectories in this way, it is necessary to know the state of
the uav when the trajectory generation process starts, for generating
trajectories that start from the position of the uav.

###  II-E Modify trajectories outside security zone

If the aircraft is outside the security zone, the trajectory can be modified
by generating a new base trajectory $\tilde{\textbf{P}}(t)$ from scratch,
updating the position of the dynamic waypoints, or adding new ones.

While the new trajectory is being generated, the UAV is going to continue
following the old trajectory until the new trajectory is computed and the old
trajectory is replaced. To ensure the smoothness during the whole track, this
new trajectory must be generated taking into account an smooth stitching
between the old trajectory and the new trajectory.

If both trajectories are too different at the swapping moment, the trajectory
followed by the aircraft would have a discontinuity that breaks the smoothness
of the whole trajectory. To minimize this discontinuity in the trajectory
swapping, the new trajectory will be computed using a set of $N_{smooth}$
waypoints (swapping waypoints) for smoothing this swapping, and the set of
waypoints through which the trajectory is going to pass, see Fig. 2.

Figure 2: New base trajectory (blue) generated from another trajectory (red)
using swapping waypoints (green circles) and waypoints modified (blue
circles). Red crosses represent the previous localization of each waypoint and
the red triangle the position of the uav when the new trajectory is
calculated.

This set of swapping waypoints are obtained from the old trajectory to ensure
that the new base trajectory is similar to the former one during the
computational time $T_{computation}(n)$:

$\displaystyle\mathbf{w_{i}}$ $\displaystyle=\mathbf{F}(t_{gen}+i\cdot
t_{offset})$ $\displaystyle i=0,..,N_{smooth}$ (5)
$\displaystyle\mathbf{\dot{w}_{i}}$
$\displaystyle=\mathbf{\dot{F}}(t_{gen}+i\cdot t_{offset})$ $\displaystyle
i=0,..,N_{smooth}$ (6) $\displaystyle\mathbf{\ddot{w}_{i}}$
$\displaystyle=\mathbf{\ddot{F}}(t_{gen}+i\cdot t_{offset})$ $\displaystyle
i=0,..,N_{smooth}$ (7)

where $t_{gen}$ is the time where the new trajectory starts to be generated,
$t_{offset}$ represents a temporal displacement between each waypoint. To
ensure that all this waypoints are between the computation time of the new
trajectory, $t_{offset}=\alpha\cdot T_{computation}(n)/N_{smooth}$ with
$\alpha=1.5$ . In this work, we tried different values for $N_{smooth}$,
finding that 1 and 2 are the most convenient values.

Finally, when the new trajectory is computed, it replaces the previous one
from that point in time.

###  II-F Modify trajectories inside security zone

When the aircraft is inside the security zone means that it is not able to
recompute a base trajectory in a safe way, without having the possibility to
correct the trajectory in a robust way. In this situation, we have to use
faster but sub-optimal approaches in exchange for being able to make these
modifications.

In order to be able to perform small modifications in the trajectory, we
propose to do local modifications in the trajectory near the modified waypoint
so the trajectory will pass through them in a smooth and agile way. For doing
these modifications, we use LGMs, which computation time is more that two
orders of magnitude lower than generating a base trajectory.

Figure 3: Representation 1-dimensional of a trajectory generated (green) with
a base trajectory (red) and a Local Gaussian Modification (blue) of the
waypoint $w$ with a displacement $A$ in x axis. The red triangle, represents
the position of the UAV when the modification is done, red crosses represent
the original position of two waypoints and the blue circle the modification of
the waypoint $w$.

Each LGM has 3 constants related to it: A , $\sigma$ and $\mu$. To obtain the
desired behavior, each constant must be computed before adding the modificator
to the trajectory.

Given a trajectory $\mathbf{F}(t)$ consisting on $N$ waypoints, each waypoint
$w_{i}$ has a position $\mathbf{x}_{i}$ and a time in the trajectory
$t_{w_{i}}$, which means that $\mathbf{F}(t_{w_{i}})=\mathbf{w_{i}}$. When a
modification $\tilde{\mathbf{w_{i}}}$ occurs in a time $t_{mod}$, a new
modifier $\mathbf{LGM}_{i}(t)$ is generated, see Fig. 3. Each constant of the
modifiers are computed in this way:

$\displaystyle\mathbf{A}$ $\displaystyle=\begin{bmatrix}A_{x}\\\ A_{y}\\\
A_{z}\end{bmatrix}=\mathbf{\tilde{w_{i}}}-\mathbf{w_{i}}=\begin{bmatrix}\tilde{w_{i}}_{x}-{w_{i}}_{x}\\\
\tilde{w_{i}}_{y}-{w_{i}}_{y}\\\ \tilde{w_{i}}_{z}-{w_{i}}_{z}\end{bmatrix}$
(8) $\displaystyle\mu$ $\displaystyle=t_{w_{i}}$ (9) $\displaystyle\sigma$
$\displaystyle=\frac{|t_{mod}-\mu|}{3.5}$ (10)

Where $\mathbf{A}$ represents the amplitude of the modification, $\mu$
represents the temporal position of the waypoint in the current trajectory and
$\sigma$ the variance of the Gaussian. These parameters are chosen in such way
that $\mathbf{LGM}(t_{mod})\approx\mathbf{0}$, with this we guarantee an
smooth change in the trajectory when each modification is done. Due to the
properties of the Gaussian function, the 99.98% of the contribution of
$\mathbf{LGM}(t)$ is between $t\in[-3.5\sigma<t-\mu<3.5\sigma]$, so choosing
sigma in this way let us consider that adding this modifiers to
$\mathbf{F}(t)$ maintain the conditions of continuity and derivability of the
trajectory.

The low computational cost of generating and evaluating LGMs allows us to
append multiples of them in the same waypoint, obtaining a high reactiveness
to the trajectory changes while maintaining the smoothness condition over the
trajectory.

###  II-G Implementation details.

The implementation of this dynamic trajectory generator has been done in C++
and it is public
accessible111https://github.com/miferco97/dynamic_trajectory_generator. We use
a library developed at the ETH Zurich University, for generating polynomial
trajectories 222https://github.com/ethz-asl/mav_trajectory_generation based on
the work of Ritcher et al. [10]

##  III EXPERIMENTS AND RESULTS

###  III-A Experimental setup

For the simulation experiments, a laptop with Ubuntu 20.04 and an Intel
Processor i7-10870H @2.2GHz has been used. In the real flight we use a Nvidia
Jetson Xavier NX with a CPU NVIDIA Carmel ARM®v8.2 of 64 bits. We perform
profiling tests in both computers.

All Flight experiments have been performed using a ROS2 version of the recent
framework for autonomous drone racing based on Aerostack 5.0 [2].

###  III-B Local Gaussian Modification tests

The first experiment consists on testing the performance of the dynamic
trajectories generated only using LGMs for updating the trajectory.

For this experiment, we generate a trajectory with 5 waypoints and as long as
the UAV goes to it, some waypoints increase his distance to the former
trajectory. Fig. 4 shows the different modifications realized over the main
trajectory, Fig. 5 shows the references obtained by the quadrotor during the
whole track.

Figure 4: Axis by axis (x, y and z axis, corresponds with top, middle and
bottom subfigure respectively) plot of the dynamic trajectory position
references generated using only LGMs to modify the waypoint positions for
different modifications of some waypoints. The vertical axis represents the
position on each axis in meters, while the horizontal represents the time in
seconds. Figure 5: Position, velocity and acceleration references generated by
the trajectory generation during the following of the dynamic trajectory shown
above in Fig. 4 .

Some profiling tests about the time spent in computing base trajectories
(TABLE I) and the time spent generating and evaluating LGMs, have been done
both in the high end computer and the onboard computer.

| Laptop computer | Jetson Xavier NX
---|---|---
6 points | $13.71\pm 0.14\;ms$ | $51.47\pm 1.57\;ms$
14 points | $88.93\pm 3.11\;ms$ | $321.07\pm 22.11\;ms$
26 points | $309.08\pm 10.20\;ms$ | $1027.76\pm 19.36\;ms$
TABLE I: Time spent generating base polynomial trajectories for different number of points, both in a computer and in a on board computer. | Laptop computer | Jetson Xavier NX
---|---|---
1 LGMs | $52.9\pm 8\;ns$ | $207\pm 54\;ns$
8 LGMs | $223\pm 34\;ns$ | $1123\pm 112\;ns$
64 LGMs | $1544\pm 303\;ns$ | $7694\pm 442\;ns$
512 LGMs | $12451\pm 504\;ns$ | $61374\pm 829\;ns$
TABLE II: Time spent generating and evaluating the sum of multiple LGMs acting
on the same trajectory.

###  III-C Simulation flights

For this experiment, 4 dynamic gates have been placed in the circuit. These
gates moved from side to side up to 1 meter and at a constant speed of 0.1m/s.
All simulation has been done in the gazebo simulator, using ROS2 for the
communication between modules.

To study the effect of the computation time of the system on the generation
and modification of the trajectories, 3 rounds of experiments were carried
out: one with the computation time taken by the computer to generate a
trajectory, and then increased by 0.5s and 1 second, see TABLES III IV V.

speed limit | Max speed | Mean speed | Max speed | mean speed | time elapsed | success
---|---|---|---|---|---|---
5 | 5.15 | 1.54 | 5.25 | 2.23 | 48.5 | 0.88
10 | 8.59 | 2.34 | 9.07 | 2.94 | 35.96 | 0.85
15 | 13.05 | 2.78 | 11.41 | 3.55 | 31.1 | 0.9
20 | 10.28 | 1.9 | 11.19 | 2.45 | 44.7 | 0.88
TABLE III: Results for simulated flight with computing time speed limit | Max speed | Mean speed | Max speed | mean speed | time elapsed | success
---|---|---|---|---|---|---
5 | 4.47 | 1.87 | 5.10 | 2.39 | 42.30 | 0.90
10 | 9.50 | 3.53 | 9.48 | 3.61 | 29.54 | 0.90
15 | 13.63 | 3.27 | 13.80 | 3.35 | 31.25 | 0.95
20 | 15.64 | 3.58 | 16.01 | 3.70 | 29.95 | 0.75
TABLE IV: Results for simulated flight with computing time + 0.5s speed limit | Max speed | Mean speed | Max speed | mean speed | time elapsed | success
---|---|---|---|---|---|---
5 | 4.71 | 2.25 | 4.7 | 2.29 | 45.9 0 | 0.92
10 | 9.63 | 3.33 | 11.08 | 3.46 | 31.1 | 0.85
15 | 13.61 | 4.77 | 13.82 | 4.87 | 21.80 | 0.90
20 | 15.58 | 4.06 | 16.72 | 3.24 | 27.47 | 0.65
TABLE V: Results for simulated flight with computing time + 1.0s

| Computer | Jetson Xavier NX
---|---|---
6 points | $13.71\pm 0.14\;ms$ | $51.47\pm 1.57\;ms$
14 points | $88.93\pm 3.11\;ms$ | $321.07\pm 22.11\;ms$
26 points | $309.08\pm 10.20\;ms$ | $1027.76\pm 19.36\;ms$

###  III-D Real flight experiment.

The aerial platform used for the real experiments was a custom quadrotor based
on the DJI F330 frame, shown in Fig. LABEL:fig:custom_platform. This platform
was equipped with a Pixhawk 4 mini as the aircraft autopilot, an Intel
Realsense T265 Tracking Module used for state estimation, and an USB fish-eye
camera for gate detection. Additionally, the aerial platform was equipped with
a Single Board Computer (SBC) NVIDIA Jetson Xavier NX with an 6-core ARM v8.2.
Fig. 6.

Figure 6: Quadrotor used for real flight experiments

For validating the approach in a real environment, an additional experiment in
real was done. The experiment consisted of passing through a small circuit of
two gates each lap faster, beginning with a maximum speed of 0.5m/s and
increasing it 0.5 m/s each lap until 3.5m/s, see Fig. 8.

Figure 7: Trajectory generated during the real flight (blue), and the
trajectory followed by the UAV (orange) Figure 8: Plot per axis of the
Trajectory Speed generated during the real flight (blue), and the trajectory
speed followed by the UAV (orange)

##  IV DISCUSSION

In the first testbed, we can see that the trajectory modificated with the LGMs
reaches all the waypoints in a smooth way, generating smooth references for
position, velocity and acceleration, as we can see in Fig. 4 and Fig. 5.
Moreover, for the profiling tests we observe that modifying a trajectory with
LGMs is more than two orders of magnitude faster than generating a new
polynomial trajectory for the usual number of LGMs in a trajectory (between 8
to 64 LGMs).

In the simulated experiments, it has been possible to prove that with a very
small calculation time, like that of a high capacity computer, it is capable
of constantly regenerating the trajectory each time it detects a change in the
position of one of the waypoints. Moreover, when increasing the calculation
time, trying to make it similar to the time of computers with lower
computation capacity, it is no longer able to generate trajectories and has to
make modifications to them using LGMs. This effect increases considerably with
speed. With all this, speeds in excess of 14 m/s are achieved by consistently
passing through the moving doors.

In the last experiment, we tested our approach in the real world, being able
to achieve peak speeds up to 4 m/s, with an average speed of 1 m/s during the
experiment. We were not able to fly faster because of the uncertainties in the
perception and state estimation module, which added noise to the trajectory
following leading to crash.

##  V CONCLUSIONS

In this work, a novel method to modify base trajectories, whose calculation is
computationally expensive, when the reaction time and computing resources are
limited, was presented, being able to achieve high speeds up to 16m/s combined
with a polynomial trajectory generator in a simulated dynamic environment,
being robust to changes in the base trajectory computation time. In real
flight, the lack of accuracy in gate estimation and state estimation results
in poorer system performance. This work can be really useful for very low
computational power devices as a main way to generate dynamic trajectories.

Although the use of Local Gaussian Modifier as a modifier function has
benefits such as the increase in the reactivity of the trajectory generator in
onboard computing circumstances, they also show some shortcomings for this
technique. For example, the unbounded influence of Gaussian function can cause
nearby points to add effects, causing the trajectory to be modified beyond the
desired point. In addition, modifying the generated trajectory to meet the
physical limits of the platform may cause the speed and acceleration limits to
be exceeded. These problems can be faced looking for other modifier functions
that take into account these limitations and modify the trajectory in a
different way.

Another possibility to study can be to scale this philosophy to be used with
very long time computation trajectory generator algorithms, such as CPC [11],
combined with a polynomial trajectory generator as the local modifier, to be
able to exploit the optimally of the CPC with the low computational cost of
the polynomial trajectory.

## ACKNOWLEDGMENT

This work has been supported by the project COMCISE RTI2018-100847-B-C21,
funded by the Spanish Ministry of Science, Innovation and Universities
(MCIU/AEI/FEDER, UE) and the project “COPILOT: Control, Supervisión y
Operación Optimizada de Plantas Fotovoltaicas mediante Integración Sinérgica
de Drones, IoT y Tecnologías Avanzadas de Comunicaciones” Ref: Y2020/EMT6368
Funded by Madrid Government under the R&D Sinergic Projects Program.

## References

  * [1] Angel Romero, Sihao Sun, Philipp Foehn, and Davide Scaramuzza.  Model predictive contouring control for near-time-optimal quadrotor flight.  arXiv preprint arXiv:2108.13205, 2021.
  * [2] Miguel Fernandez-Cortizas, Pablo Santamaría, David Perez-Saura, Javier Rodríguez-Vázquez, Martin Molina, and Pascual Campoy.  Framework and evaluation methodology for autonomous drone racing.  In Jose Martinez-Carranza, editor, 12th International Micro Air Vehicle Conference, pages 50–56, Puebla, México, Nov 2021.  Paper no. IMAV2021-5.
  * [3] Taeyoung Lee, Melvin Leok, and N Harris McClamroch.  Geometric tracking control of a quadrotor uav on se (3).  In 49th IEEE conference on decision and control (CDC), pages 5420–5425. IEEE, 2010.
  * [4] Emil Fresk and George Nikolakopoulos.  Full quaternion based attitude control for a quadrotor.  In 2013 European control conference (ECC), pages 3864–3869. IEEE, 2013.
  * [5] Moses Bangura and Robert Mahony.  Real-time model predictive control for quadrotors.  IFAC Proceedings Volumes, 47(3):11773–11780, 2014.
  * [6] Samir Bouabdallah and Roland Siegwart.  Full control of a quadrotor.  In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 153–158, 2007.
  * [7] Robert Mahony, Vijay Kumar, and Peter Corke.  Multirotor aerial vehicles: Modeling, estimation, and control of quadrotor.  IEEE Robotics Automation Magazine, 19(3):20–32, 2012.
  * [8] Bryan Penin, Riccardo Spica, Paolo Robuffo Giordano, and François Chaumette.  Vision-based minimum-time trajectory generation for a quadrotor uav.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6199–6206. IEEE, 2017.
  * [9] Daniel Mellinger and Vijay Kumar.  Minimum snap trajectory generation and control for quadrotors.  In 2011 IEEE international conference on robotics and automation, pages 2520–2525. IEEE, 2011.
  * [10] Charles Richter, Adam Bry, and Nicholas Roy.  Polynomial trajectory planning for aggressive quadrotor flight in dense indoor environments.  In Robotics Research, pages 649–666. Springer, 2016.
  * [11] Philipp Foehn, Angel Romero, and Davide Scaramuzza.  Time-optimal planning for quadrotor waypoint flight.  Science Robotics, 6(56):eabh1221, 2021.
  * [12] Michael Burri, Helen Oleynikova, , Markus W. Achtelik, and Roland Siegwart.  Real-time visual-inertial mapping, re-localization and planning onboard mavs in unknown environments.  In Intelligent Robots and Systems (IROS 2015), 2015 IEEE/RSJ International Conference on, Sept 2015.