redacted

# Creating Multimodal Interactive Agents with Imitation and Self-Supervised
Learning

Interactive Agents Team  DeepMind

###### Abstract

A common vision from science fiction is that robots will one day inhabit our
physical spaces, sense the world as we do, assist our physical labours, and
communicate with us through natural language. Here we study how to design
artificial agents that can interact naturally with humans using the
simplification of a virtual environment. We show that imitation learning of
human-human interactions in a simulated world, in conjunction with self-
supervised learning, is sufficient to produce a multimodal interactive agent,
which we call MIA, that successfully interacts with non-adversarial humans 75%
of the time. We further identify architectural and algorithmic techniques that
improve performance, such as hierarchical action selection. Altogether, our
results demonstrate that imitation of multi-modal, real-time human behaviour
may provide a straightforward and surprisingly effective means of imbuing
agents with a rich behavioural prior from which agents might then be fine-
tuned for specific purposes, thus laying a foundation for training capable
agents for interactive robots or digital assistants. A video of MIA’s
behaviour may be found at https://youtu.be/ZFgRhviF7mY.

##  1 Introduction

Humans interact with the physical world and one another, and it is through
these interactions that much of our cognition was shaped during evolution
(Dunbar, 1993). If we hope to build artificial intelligence (AI) capable of
human-like thinking, we should therefore consider a holistic setting wherein
agents perceive and manipulate their world and understand and produce
language, so that they can participate in—and learn from—natural interactions
with humans (McClelland et al., 2019; Lake and Murphy, 2021; Winograd, 1972).

In this work we explore how to create artificial agents that interact with
humans and their environment. To do so, we use imitation learning approaches
(Pomerleau, 1989; Schaal, 1999) that have driven progress in Go (Silver et
al., 2016), Starcraft (Vinyals et al., 2019), and perhaps most notably the
recent progress in large language models (Brown et al., 2020) and dialogue
agents Adiwardana et al. (2020). Imitation learning has proved a surprisingly
powerful approach for games and language modelling, but it is unclear the
extent to which it may furnish powerful behavioural priors in embodied
domains. We study this question in a 3D simulator that is easily operated by
human participants. Working to create embodied agents with impressive motor
and linguistic capabilities is a difficult problem for several practical
reasons, notably: (1) model requirements are more sophisticated, and (2) data
sources are not widely available, as they are with text. Our results suggest,
however, that by augmenting imitation learning with hierarchical architectures
and self-supervised learning it is possible to make agents that are capable,
impressive, and surprising using moderately-sized datasets collected directly
by researchers.

We build upon our previously introduced methodology (Interactive Agents Team,
2020) in a few crucial ways. First, we significantly increase the scale and
complexity of collected data. Second, we simplify the training protocol.
Third, we remove all uses of privileged information extracted from our virtual
3D-world simulator. Ultimately, these changes result in an agent artifact that
blends perception, language understanding and production, and motor action to
competently engage in extended, and often surprising interactions, and
participate in comprehensive, bidirectional language-based communication. The
resulting agent, MIA (Multimodal Interactive Agent), is vastly more capable
than those presented in the previous work (Interactive Agents Team, 2020). MIA
exhibits a diversity of behaviours that were never instructed by researchers,
including tidying a room and finding and grouping multiple specified objects.
It asks clarifying questions, produces minute long action sequences following
coherent goals, and can rapidly learn new objects, nouns, and verbs in mere
hours of real-time experience.

Our contributions are as follows:

  * •

We introduce the multi-room Playhouse environment to study natural
interactions between humans and agents.

  * •

We produce an agent, MIA, which is capable of engaging in natural interactions
with human participants in a 3D simulated world.

  * •

We demonstrate the importance of architectural design and self-supervised
losses to performance in situated language agents domains.

  * •

We highlight the importance of human and programmatic evaluation beyond
training and validation losses.

  * •

We demonstrate the effects of scale of both data and model size on
performance.

  * •

We demonstrate the importance of large-scale data priors for fast learning of
new objects and skills as well as their referring nouns and verbs in a
practical amount of time (hours).

  * •

We provide a careful comparison to human performance, as evaluated by humans,
on tasks involving natural interactions.

##  2 Language Games in the Playhouse

Figure 1: Interactions in a simulated “Playhouse” environment. Humans and
agents interact via simulated avatars in the 3-D “Playhouse” environment. The
environment contains a randomised set of rooms with domestic objects and
children’s toys, as well as containers, shelves, furniture, windows, and
doors. The diversity of the environment enables interactions involving
reasoning about space and object relations, ambiguity of references,
containment, construction, support, occlusion, and partial observability.
Agents interact with the world by moving around, manipulating objects, and
speaking to each other. A Example of two humans interacting in the Playhouse.
This episode was selected from a small set that was drawn at random from our
dataset. B Depicts a simple interaction wherein the orange solver agent is
placing a helicopter into a container while the blue setter agent watches on.
C A sampling of the types of objects available in the room. D Two random
instantiations of the Playhouse, each with unique configurations of rooms,
furniture, and objects.

We explored human and agent interaction in a 3D virtual environment called the
Playhouse, based on previous environments built in the Unity game engine (Ward
et al., 2020; Interactive Agents Team, 2020). The Playhouse comprises a
randomized set of adjoining rooms (such as a living room, pantry, and
bathroom) and interactable items (such as toys and household objects). Human
and agent embodiment manifests as control of a virtual robot that moves around
in space, grasps and manipulates objects, and emits natural language. This
setting permits a wide range of behaviours, ranging from simple instruction-
following (e.g., “ _Please pick up the book from the floor and place it on the
blue bookshelf_.”) to creative acting (e.g., _“Bring food to the table so that
we can eat_ ”).

It is not feasible to programmatically define reward functions for the
Playhouse because the space of all possible interactions is vast and
unpredictable, and because satisfaction criteria can often be ambiguous or
subjective. We might instead choose to task humans with assigning reward based
on their subjective assessment of an agent’s behaviour, either by absolute
score or ranking (Christiano et al., 2017), following an interaction. However,
unless these interactions could occur at an enormous scale and fast enough
pace, this too is not a feasible approach to training initially randomly
behaving agents using purely reinforcement.

Therefore, we follow an approach advocated for by (Interactive Agents Team,
2020) wherein we train agents using imitation to instill them with an
intelligent _behavioural prior_ (Galashov et al., 2019). Agents trained in
this manner will display some structured and language-driven behaviours.
Subsequent reinforcement learning will be more efficient with these agents
since humans will be more inclined to continue interactions with AI
participants that display rudimentary intelligence. The focus of this report
centers on creating agents with intelligent behavioural priors, and we leave
further reward-based learning for future work.

###  2.1 Collecting Data in the Playhouse

Imitation learning works best when demonstrations are produced by experts and
when there are sufficiently many to cover the desired range of behaviours
(including, e.g., corrective behaviours (Ross et al., 2011)). For domains such
as language, data can be easily obtained by leveraging existing text on the
internet (Brown et al., 2020). However, for domains such as robotics (or
simulated robotics), data needs to be assembled from scratch. The protocol for
organizing the collection of expert demonstration data in the Playhouse is a
central contribution of this work.

One approach could be to collect data from humans freely interacting in the
Playhouse. However, such data might prove problematic for a host of reasons.
First, data may be difficult to model as it might not comprise graded
behavioural competencies. Humans come to the environment with an ability to
act in simulated worlds, an extensive knowledge base, and nuanced intentions,
and hence exhibit already-complex behaviour that takes for granted simpler,
but necessary, capacities such as the ability to identify basic objects.
Second, as interactions are guided by the whims of human participants,
collected behaviours might naturally collapse to a few characteristic specific
modes, and hence might not span the behaviours we ultimately care about.
Third, if left unchecked the data might be permeated with undesirable biases.
A different approach could be to collect data using templated scripts that
human experts must follow. However, this places a burden on researchers to
infer the set of scripts whose translations span the desired range of
behaviours, which is an impossible feat if one wishes to capture the
contextual, ambiguous, and nuanced forces that drive natural human behaviour.

Our approach occupies a middle ground between these two possibilities. We
centered Playhouse interactions on language games, which are a collection of
both prompted interaction-types and free-form instructions (Lynch and
Sermanet, 2020) from which human players can base their behaviour (Interactive
Agents Team, 2020). All language games had the same basic structure: a setter
would receive a prompt from which they could design an interaction, which they
propose to a solver agent. For example, the setter might receive a prompt to
“Ask the other player a question about the existence of an object”, and after
some exploration to discover possibilities, the setter could translate this
into the task: “Please tell me whether there is a blue duck in a room that
does not also have any furniture”. The combination of (1) a small set of
prompts, (2) the random, combinatoric structure of the Playhouse, and (3) the
natural linguistic and intentional variation that humans exhibit ensured that
interactions were unique, and together spanned the set of competencies we
hoped agents to learn. To ensure we sufficiently captured the richness of
natural human behaviour, we also included free-form instructions, which gave
setters more leeway in guiding interactions, within certain constraints
(namely, interactions should resemble, in the participants’ best judgement,
those from the templates, and should be closely monitored for any arising
ethical concerns). A notable example of creative free-form instructions is:
“Now take any object that you like and hit the tennis ball off the stool so
that it rolls near the clock, or somewhere near it.”

| Totals |  |  |
---|---|---|---|---
Number of train episodes | 275676 |  |  |
Number of test episodes | 34399 |  |  |
Episode length (minutes) | 5 |  |  |
Total play time (years) | 2.94 |  |  |
Number of unique setter instructions | 778808 |  |  |
| Mean | Std | 25th | 75th
Interaction time per instruction (seconds) | 38.86 | 29.93 | 18.00 | 49.00
Setter utterance length | 7.88 | 3.37 | 6.00 | 10.00
Solver utterance length | 2.08 | 2.07 | 1.00 | 2.00
Number of setter instructions per episode | 3.98 | 1.80 | 3.00 | 5.00
Idling proportion | 0.69 | 0.26 | 0.50 | 0.91
Table 1:  Playhouse dataset statistics.

##  3 Multimodal Interactive Agent Training & Design

An abundance of expert human behavioural data and a lack of programmatic
reward functions motivate a straightforward training regime centered on basic
representation learning techniques, which has seen much success in recent
large-scale models such as GPT-3 (Brown et al., 2020). Below we describe the
essential ingredients: supervised learning of actions (Pomerleau, 1989; Osa et
al., 2018), and an additional cross-modal self-supervised learning objective
that significantly improves performance beyond supervised learning alone.
Since trained agents are difficult to evaluate given the free-form nature of
the Playhouse environment, we also describe a protocol for assessing learned
behaviour relative to human baselines.

###  3.1 Training

####  3.1.1 Behavioural Cloning

Training comprises a straightforward imitation learning technique, behavioural
cloning (BC), which frames behavioural copying as a sequence learning problem.
Our data comprise temporal sequences of human behaviour, called trajectories,
which include observation sequences $\mathbf{o}_{\leq
T}\equiv(\mathbf{o}_{0},\mathbf{o}_{1},\mathbf{o}_{2},\dots,\mathbf{o}_{T})$
and action sequences $\mathbf{a}_{\leq
T}\equiv(\mathbf{a}_{0},\mathbf{a}_{1},\mathbf{a}_{2},\dots,\mathbf{a}_{T})$.
Given a dataset of solver-only trajectories we implement the following loss
function, which we derive from the forward Kullbach-Leibler divergence between
the expert and agent policies in Interactive Agents Team (2020):

$\displaystyle\mathcal{L}^{\textsc{bc}}(\boldsymbol{\theta})$
$\displaystyle=-\frac{1}{B}\sum_{n=1}^{B}\sum_{t=0}^{K}\ln\pi_{\boldsymbol{\theta}}(\mathbf{a}_{n,t}\mid\mathbf{o}_{n,\leq
t}),$

where $B$ is the minibatch size, $K$ is the backpropagation-through-time
window size.

####  3.1.2 Modality Matching using Contrastive Self-Supervised Learning

Techniques that improve generalization will benefit agents in the Playhouse
because evaluation environments are bound to be unique from the agent’s
training data. This is due to the combinatorics and programmatic randomness of
the Playhouse (variations in house layouts, object existence, and object
placement can produce an inexhaustible source of newness), and because of the
dependence of the agent’s observations on its actions (deviations in
observations or actions from that which are exactly specified by the data will
drive the agent to never-before-seen states).

We found one contrastive self-supervised representation learning technique,
which involves cross-modality matching (Alayrac et al., 2020), particularly
useful in this regard. It is implemented as an auxiliary training loss wherein
the agent must predict whether vision and language embeddings match (i.e.,
they are produced from a trajectory from the dataset as normal), or they do
not match (i.e., visual embeddings are produced from the input image of one
trajectory in the dataset, and language embeddings are produced from the
language input from a different trajectory). The agent processes these visual
and language embeddings using its perceptual encoder (see section 3.2), the
output of which is fed to an MLP discriminator that produces a binary
predictor of whether the embeddings match or not. Practically, we implement
this at the level of the minibatch: the batch of data used for the behavioural
cloning loss comprise the “matches”, and a shuffling of vision and language
embeddings from the minibatch comprise the mis-matches:

$\displaystyle\mathcal{L}^{\textsc{CR}}(\boldsymbol{\theta})$
$\displaystyle=-\frac{1}{B}\sum_{n=1}^{B}\sum_{t=0}^{T}\bigg{[}\ln
D_{\boldsymbol{\theta}}(\mathbf{o}_{n,t}^{\textsc{V}},\mathbf{o}_{n,t}^{\textsc{L}})+\ln\big{(}1-D_{\boldsymbol{\theta}}(\mathbf{o}_{n,t}^{\textsc{V}},\mathbf{o}_{\textsc{Shift}(n),t}^{\textsc{L}})\big{)}\bigg{]},$
(1)

where $B$ is the batch size and $\textsc{Shift}(n)$ is the $n$-th index after
a modular shift of the integers: $1\to 2,2\to 3\dots,B\to 1$, and superscripts
denote the modality ($V$ for vision, $L$ for language). Although language
emissions are sparse in each trajectory, we implement language input
observations as “sticky”; e.g., upon the setter emitting a language utterance,
the solver will repeatedly receive this same utterance as input until the
setter emits a new utterance. This implies that there will always be vision
and language pairs that can be matched (or not) at each timestep, except for
the first timesteps in an episode that occur prior to the setter speaking. For
simplicity, we use $D_{\boldsymbol{\theta}}()$ to denote the processing of
observations up through the agent’s perceptual encoder, and into a separate
discriminator MLP, which is only engaged during learning. While the addition
of this loss requires an extra forward pass through the perceptual encoder, we
observe that the computational overhead results in less than a $1\%$ deduction
in training speed.

Figure 2: Multimodal Interactive Agent (MIA) Design. A series of ResNet blocks
downsample the incoming image, while language tokens index an learnable
embedding table. Together these embeddings comprise the input to a multi-modal
Transformer, whose output is aggregated and provided as input to an LSTM
memory. The output of the LSTM conditions both the hierarchical movement
policy and language policy, implemented as an LSTM (which unrolls $8\times$
per input, producing $8$ sets of consecutive movement actions) and a
Transformer, respectively. MIA is trained using behavioural cloning on each of
its actions (no-op, move, look, rotate, push and pull, grab, and text) in
addition to an auxiliary contrastive loss. Please see the main text for full
details.

###  3.2 Perception

MIA receives input from visual ($96\times 72$ RGB pixels) and symbolic (text-
based language tokens) modalities. For vision we employ conventional ResNet
blocks (He et al., 2016) (with strides $(1,1,2,2)$ per block and a consistent
kernel shape of $(3,3)$, altogether comprising $20$ total convolution layers),
which produce a $24\times 18\times 512$ output representation from the input
image. For language processing, we produce embeddings from a tokenized version
of language input by indexing a learnable embedding table with a vocabulary of
$4000$ subwords. The $432$ ($24\times 18$) vectors from the visual spatial
array, the language embeddings, and two dedicated output embeddings (akin to
the “CLS”embeddings in the BERT model (Devlin et al., 2018)) are gathered to
form the input to a multi-modal transformer (MMT) (Vaswani et al., 2017),
consisting of $4$ layers, $8$ heads, and $512$ embedding size. The output of
the MMT is the concatenation of the two output embeddings and a feature-wise
mean pooling of all other output embeddings (that is, all the embeddings that
produce queries, which include the two dedicated output embeddings and the
language embeddings. The visual embeddings are only cross-attended to, and
hence only produce keys and values). This is then provided as input to a two-
layer, $1024$-unit LSTM “memory”, whose output is fed to each policy head.

###  3.3 Hierarchical Control

####  3.3.1 High-Level Control

In the non-hierarchical setting, MIA receives observations and produces a set
of movement actions $15$ times per second. In the hierarchical setting, new
observations arrive only every $8$ steps (i.e., $3.75$ times per second), and
thus, MIA is tasked with producing $8$ consecutive sets of movement actions in
a row before receiving a new observation. Hierarchical control is implemented
with an LSTM that receives input from the “memory” LSTM, and unrolls for $8$
steps, providing inputs to each of the movement policies (see section 3.3.2)
on each of these $8$ “internal” steps. Language actions are emitted from the
high-level controller. They’re sampled one token at-a-time, up to a maximum of
$25$ per high-level step, using the same vocabulary as for language-input
tokenization. For the language policy we use a transformer with teacher
forcing ($4$ layers, $8$ heads, and $256$ embedding size). The language
policy, like the movement policies (see section 3.3.2) also models "no-ops",
which are binary decisions to predict a non-action (during training) or
produce a non-action (during acting).

####  3.3.2 Low-Level Control

MIA’s movement action space largely resembles that used by humans: _Look_
actions are modelled as a mouse movement to a bin in visual space ($(-1,1)$
for $x-$ and $y-$ coordinates, using $51$ bins for each), implemented as two
MLPs. _Move_ actions are similarly modelled as a decision in a binned, two-
dimensional translational-force space ($(-1,1)$ for $up-down$ and $left-right$
translations, using $101$ bins for each), implemented as a MLPs. Object
rotations (i.e., angular movements applied to objects when they are being
grasped) are modelled along the three cardinal axes, implemented as a
recursive discrete decision procedure whereby the agent first chooses from
among $3$ coarse bins between $(-1,1)$, and then chooses from $3$ finer bins
within the previously chosen coarse bin, and so on, until a choice is made at
a granularity of $0.1$. This is implemented using an LSTM. Push and pull
actions are similarly modeled. _Grab_ actions are modelled as a binary action
(e.g., mouse click), implemented as an MLP. We also found a form of
autoregressivity to be useful: within each policy, look actions in the $x-$
dimension condition look actions in the $y-$ dimension. Rotations in the $x-$
and $y-$ dimensions condition rotation in the $z-$ dimension. Between
policies, move actions condition look actions, and move and look actions
condition rotation actions, and move, look, and rotation actions condition
push and pull actions. There is no autoregressivity across time.

###  3.4 Evaluation

We monitored MIA’s performance in three ways: (1) via the total loss and its
sub-components (see Figure 2A), (2) via scripted probe tasks that serve as a
heuristic evaluation of rolled-out behaviours during training (Figure 2C), and
(3) via live human-agent interactions (Figure 2B). In some domains, like
language, log probabilities correlate well with model “performance”. However,
in our setting this proved to not be true; e.g., drastically overfit models
can perform better, and architecture changes that improve log probabilities
can result in worse performance. Therefore, scripted probe tasks and live
human-agent interactions are central to agent iteration, despite being more
computationally demanding, slow, and subjective.

####  3.4.1 Scripted Probe Tasks

We constructed a suite of probe tasks to evaluate performance on several
dimensions, including counting, identifying colors, lifting items, and more
refined object positioning. For each task, agents were presented with a
templated instruction (e.g., “place a duck on top of the table”), which was
then evaluated using a scripted reward model. Crucially, these reward models
were used for evaluation only; agents never received rewards from the
environment for the purposes of optimization.

####  3.4.2 Human Evaluation

Since we aim to build agents that can carry our natural interactions with
humans, the ultimate metric we care about is human evaluation of agent
interactions. So, upon training agents we asked human participants to interact
with agents in a setting that closely mimicked data collection. Humans played
the role of the setter, while agents performed the role of the solver.
Importantly, humans were not adversarial: they did not search for agent
failures, nor did they try to coax the agent into undesirable modes of
behaviour. Rather, they carried out interactions similar to those as when they
interacted with humans during data collection. After an interaction, human
setters provided feedback as to whether the interaction was successful, which
we then report as the agent success rate.

##  4 Performance, Ablations, & Modifications

Figure 3: Performance, ablations and modifications. A Loss curves. The top row
shows the loss of MIA on training and validation datasets during training.
After training on 5G timesteps the behavioural cloning loss on movement
actions is fairly flat while the behavioural cloning loss on language actions
and the contrastive representation learning loss have started to overfit. B
Human evaluation. The bottom left plot compares the success rate of humans and
MIA, with various ablations applied, when playing the role of a solver in a
Playhouse language game, as judged by human setters. The episodes are divided
according to the prompt given to the human setter, as either “Instruction
following”, e.g. “Ask the other player to put something on top of something
else”, or “Question answering”, e.g. “Ask the other player a question about
the color of something”. C Scripted probe tasks. The bottom right plot
compares the success rate of humans and agents on six automated testing tasks.
These supply a pre-scripted setter question or instruction (e.g. Position =
“Put the X near the Y”), where the response can be unambiguously scored as
successful or unsuccessful by the environment engine.

MIA achieves over $70\%$ success rate in human-rated online interactions,
representing $75\%$ of that achieved by humans in similar positions. As
indicated in figure 3, according to our probe tasks MIA is most successful at
“going” and “lifting”, and less successful at more demanding motor tasks (such
as positioning objects relative to each other) and language production tasks
demanding complex cognition (such as counting). The results from our probe
tasks also reveal weaknesses in scripting probe tasks for evaluation. For
example, performance is low on tasks that ask the agent to mention the colors
of items in the room, though this is not necessarily because the agent is poor
at identifying colors. Rather, limitations in anticipating possible valid
natural language responses (“blue”, “light blue”, “cyan”, “sky blue”, etc.)
prevent us from accurately assessing the model’s abilities. Similar logic
applies to motor tasks (e.g., how high should an agent lift an object to be
“successful”? What if the object is a table?). Indeed, post-hoc correction of
our reward metric for “counting” can reveal a pre- and post- fluctuation of up
to 30% on the evaluated score (data not shown). It is precisely these
researcher-centric deficiencies, if permeated into a reward model for
reinforcement learning, that would result in less robust and less natural
agent behaviours.

Figure 3 also indicates that we are in the over-fitted regime of training. The
total loss shows mild overfitting—and the movement behavioural cloning loss
perhaps none at all—but the language behavioural cloning and contrastive
representation losses clearly indicate that agents could profit from more
data. These results also highlight an aspect of integrated-agent training that
is unique from more constrained domains like language modeling: the dynamics
of each loss are different, making it difficult to apply techniques, such as
model scaling, in a straightforward manner. While modeling motor actions could
profit from larger models, this might be counteracted by more severe
overfitting when modeling language. Nevertheless, a relatively simple way to
improve performance (which we motivated further in section 5) is to collect
and train on more data.

We performed a series of ablations and agent design modifications to better
understand the role of various components in MIA. In particular, the ablations
assessed the impact of removing visual inputs, language inputs, the auxiliary
contrastive representation loss, and the hierarchical movement policy. We also
determined the importance of visual resolution in the multi-modal transformer
by modifying the structure of visual processing to downsample the image to
$6\times 5\times 512$ (i.e., producing $30$-pixel feature maps as opposed to
$432$-pixel feature maps in the baseline).

As with many ablations, there is a confounding factor: removing agent
components alters the agent’s computational capacity (i.e., the number of
total parameters), which can impact performance (see section 5).
Unfortunately, there is no clear way to compensate for these parameter
differences. However, as our data will show in section 5, since the
differences in parameter counts for these ablations are minimal, performance
loss due to decreased capacity should be minimal compared to the effects of
the ablations themselves. The parameter counts for the various agents are:
baseline $57$M, no language input $56$M, no vision $52$M, no auxiliary
contrastive representation loss $56$M, no hierarchical movement policy $66$M,
and modified visual processing to further downsample the input image $57$M.

As the results show, there are clear performance degradations for each
ablation and modification. Downsampling the input image less (i.e., using more
pixels as input to the MMT) is best, though intriguingly, using a Vision
Transformer-style technique (Dosovitskiy et al., 2020) that extracts image
patches from the full resolution input image does not perform as well as using
ResNet blocks (data not shown), which corroborates recent similar findings (Wu
et al., 2021).

While we provide evidence that the hierarchical movement policy significantly
improves agent performance, the reason for its impact is unclear. We
hypothesize that a few effects could be important: First, by forcing the agent
to act every $8$ steps, we cause adjacent observations to be more visually
distinct, which could aid learning (consider the converse scenario if the
agent were forced to act with a higher frequency: adjacent observations would
be nearly identical but have potentially different target policies, which
creates complications for learning). Second, our implementation offers a form
of data augmentation, as we can randomly vary the initial observation to be
any of the first $8$ observations at the beginning of a given data trajectory.
Third, the agent may genuinely, and implicitly learn useful hierarchical
policies wherein its lower-level controller captures repeatable action
sequences that merely need to be triggered by the higher-level controller.

Notably, when MIA is trained on the same Playroom data as in Interactive
Agents Team (2020), it outperforms the reported behavioural-cloning agent,
which is augmented with privileged information, by $20\%$ on average across
the single-room scripted probe tasks.

##  5 Data and Parameter Scaling

Figure 4: Data and model scaling. Total loss, scripted probe performance, and
human evaluation for data and model scaling. All agents were trained to 5G
steps, coinciding with convergence on scripted probe tasks, except for the
early stopping condition which ends training at the point of lowest validation
loss. Error bars represent standard error across 3 seeds. The baseline model
with $56$M parameters was trained on differently-sized datasets (A, B, C. top
row), or differently sized models were trained on the full dataset (A, B, C.
bottom row). Validation losses decrease as dataset sizes increase, although
this trend is not true for increasing model size (A). Nevertheless, for
scripted probe tasks (B) and human evaluation (C) there are clear improvements
when increasing both dataset size and model size. Performance on both scripted
probe tasks (data not shown) and human evaluation (C) is worse at the early
stopping point, without exception.

Contemporary machine learning research has uncovered remarkable empirical
effects regarding scale (Kaplan et al., 2020, e.g.); in particular, model
performance scales using a power-law trend with dataset size, model size (in
terms of number of parameters), and compute. These effects have traditionally
been observed in the language domain, which is characterized by massive
dataset sizes, standardized architectures, and refined training protocols. In
this work, however, we are in a decidedly different regime. We assume we have
collected an abundance of data, but the results in figure 4, particularly the
losses, indicate that we could still be in a “low data” regime where
overfitting is more pronounced and scaling effects are more difficult to see
(Kaplan et al., 2020). Moreover, the cognitively rich tasks force us to
iterate our model architecture as we explore the effects of our choices on,
for example, memory formation and retrieval, language production, motor
control, and visual object identification, making model inductive biases a
perpetually confounding factor in our research. This architectural flux,
combined with a complicated training protocol involving a handful of losses to
optimize, lead to natural questions of whether scaling effects can also be
observed in the Playhouse.

In figure 4 we present some mixed evidence after training four differently-
sized models ($5$M, $16$M, $56$M (baseline), and $121$M) on five differently-
sized datasets ($\frac{1}{16}$, $\frac{1}{8}$, $\frac{1}{4}$, $\frac{1}{2}$,
and the full dataset size). In terms of the total training loss, larger models
achieve lower values for each dataset size, and predictably, training losses
are lower in the smaller data regimes (implying a easier ability to memorize
particular examples as dataset sizes decrease). The effects of scaling on the
validation loss are more difficult to interpret: the total loss shows little
difference between model sizes, though it displays a clear trend with dataset
size when keeping model size constant. When analyzing individual losses we see
a clear effect for some, such as the hierarchical movement BC loss. The
language and contrastive representation losses, on the other hand, show clear
and quick overfitting for each model (as in figure 2).

To get a clearer picture we also measured the performance metric that we
ultimately care about: human-evaluated behavioural competency during online
interactions. The results here are consistent (low standard error across
seeds) and clear: increasing model size and dataset size clearly benefit
performance.

Altogether, these results lead us to believe that we may be in the low-data
regime for the Playhouse, suggesting that performance gains are to be had by
collecting more data (and then subsequently scaling model size). The results
also highlight the problems with analyzing the effects scaling in these more
complicated settings where a handful of losses, jointly optimized, each
exhibit different learning dynamics and interact with each other in unknown
ways: the values we observe for the losses may obfuscate the effects of scale
on ultimately more important behavioural metrics.

##  6 Behavioural Transfer

Figure 5: Behavioural Transfer. We quantified the agent performance when
learning a new noun (a drum) or verb (to clear) as a function of quantity of
human demonstrations, measured in hours. With both a new noun and new verb
agent performance quickly improves with mere hours of demonstration
experience.

We investigated how much data is needed to learn how to interact with a new,
previously unseen object (new noun). Starting with a pre-trained MIA, we began
new training on varying amounts of data from humans interacting with a novel
object (a drum), in addition to the data from the original dataset. The amount
of data is quantified in hours of new data measured as real time experience
(note: this measures the time required to collect this data in human hours,
not the amount of time the agent experiences it during the course of
supervised training). After training, we evaluated the ability of the agents
to perform simple tasks with the new object (answering about its color and
lifting it). We observe that using less than $12$ hours of human interaction
is enough to reach the final performance on subsequent language and motor
tasks involving drums. (figure 5 B).

To study how much data is needed to learn a new command (the verb “to clear”),
we again start with a pre-trained MIA and co-train it on novel episodes where
humans are instructed to remove all objects from a surface (e.g., table or
shelf). The original pre-trained agent is incapable of performing this
behavior, which is to be expected as this command is not present in the
original data. However, after training on novel episodes containing examples
of this behavior, we observe that with around $1$ hour of human demonstrations
the agent reaches near optimal performance at this task.

Altogether these results suggest that while agents do not generalize zero-shot
to new nouns or verbs, they have a sufficiently rich behavioural prior such
that subsequent training on small amounts of experience allow rapid adaptation
to new objects and actions.

##  7 Discussion & Conclusion

In this work we’ve built upon the paradigm introduced in Interactive Agents
Team (2020) to construct an agent, MIA, that emphasises a holistic integration
of embodied control, perception, and language understanding and production
(McClelland et al., 2019; Lake and Murphy, 2021). The goal was to elicit
behaviours that are naturalistic and appropriately exploratory (Turing, 1950;
Winograd, 1972), and hence, are capable of subsequent efficient tuning using
reinforcement.

Imitation of human behaviour was central to our training approach. Unlike
other domains, like language (Devlin et al., 2018; Radford et al., 2019; Brown
et al., 2020), human behavioural data in 3D simulated worlds often needs to be
collected and curated from scratch. Some previous work has collected human
(child) multimodal behavioural data (Roy et al., 2006; Yoshida and Smith,
2008; Sullivan et al., 2020), but has not used it to train artificial agents.
Social learning, imitation, and mimicry is prevalent in the the animal kingdom
(Laland, 2004; Byrne, 2009). In humans, infants naturally imitate both the
language and actions they encounter (Chomsky, 1959; Heyes and Galef Jr, 1996).
However, in AI, imitating multi-modal, embodied behavioural data is especially
challenging because latent in all trajectories of human experience are
intentions and goals. It is not clear whether we currently possess the
algorithms and models that can interpret implicit causal factors of human
behaviour (Ortega et al., 2021), and hence, can engage in consistent and
coherent naturalistic interactions. Nevertheless, as in previous work we
observe that some techniques, such as dataset and model scaling, reliably
improve performance as measured by human evaluators. In addition, a number of
architectural improvements proved crucial: self-supervised learning that
gauged whether cross-modality representations “matched” or not, and
hierarchical control of motor actions both substantially improved agent
performance.

Our setting is a natural one in which to study “language grounding” (Harnad,
1990), wherein sensorimotor embodiment combines with language production and
understanding to produce models that more closely approach human-like symbolic
behaviour (Santoro et al., 2021). Work in robotics and in 3D simulated
environments has pursued similar ideas, such as natural language conditioning
of tasks (Lynch and Sermanet, 2020; Hill et al., 2019; Anderson et al., 2017;
Das et al., 2018). However, to our knowledge our work is unique in combining
language conditioning with unconstrained language production, in an embodied
setting demanding complex navigation and manipulation for upwards of $5$
minutes, with a human in-the-loop.

An exciting direction that remains unexplored is to also train setter agents
using the data produced by humans whose role it is to set tasks and questions.
“Self-play”-like settings have proved powerful in developing agents in games,
where agent roles are symmetric, and reward signals provide positive and
negative feedback signals per experience (Silver et al., 2016; Vinyals et al.,
2019). It is less clear, however, how one could leverage agent setter-agent
solver dynamics in the Playhouse environment where roles are asymmetric and
rewards do not exist, beyond using the extra data for generic representation
learning.

One challenge for future work pertains to the diversity and flexibility of
language comprehended or produced by our agents, as it is far more constrained
(both semantically and syntactically) than in the case of non-embodied systems
trained on web-scale text data (Brown et al., 2020). Nevertheless, a useful
intelligent robot would not need to be an unconstrained source of knowledge,
provided that it can communicate efficiently and effectively about its
physical environment.

In addition, the control and perception challenges faced by our agents are
substantially simpler than they would be in a physical robot. While our data
is generated by participants sitting at computers, making research
comparatively rapid, analogous data for a physically-embodied interactive
agent might require teleoperation of specialized and expensive hardware (Jang
et al., 2021). On the other hand, unlike many current robotic systems, our
agent must integrate learning perception and control with memory and language.
Generic models for perception and control skill learning are rapidly
improving, and we hope that such techniques might soon combine with the
approach described here to yield a system capable of all of these things.
Another key challenge when transitioning from simulation to reality is the
collection of adequate data.

A final pressing challenge for future work is how to appropriately evaluate
interactive agents like MIA. None of our current mechanisms— training loss,
scripted probe tasks, and online human-agent evaluation—are ideal for
research. Training losses and scripted probe tasks are only heuristic measures
of performance; they generally correlate with better agents, but these metrics
do not necessarily move monotonically with human judgement. This is
particularly troublesome since we explicitly optimize the training loss, and
not human judgement, and ideally the measure that we optimize should be
precisely the measure that we ultimately care about. On the other hand, real-
time human-agent evaluation is expensive, requiring many human hours. It is
also high-variance and inconsistent, as live human interactions will
necessarily be different every time, and humans may shift their content over
time. Nevertheless, models trained to mimic human judgements may ultimately
help us tackle open-ended evaluation for multi-modal and real-time
environments.

##  8 Authors & Contributions

Josh Abramson contributed to agent development, imitation learning, data and
tasks, running and analysis of experiments, engineering infrastructure,
writing, and as a technical lead.
Arun Ahuja contributed to agent development, imitation learning, data and
tasks, running and analysis of experiments, engineering infrastructure,
writing, and as a technical lead.
Arthur Brussee contributed to environment development.
Federico Carnevale contributed to agent development, imitation learning,
running and analysis of experiments, writing, and as a sub-effort lead for
agent development.
Mary Cassin contributed to environment development.
Felix Fischer contributed to engineering infrastructure.
Petko Georgiev contributed to agent development, engineering infrastructure,
evaluation development, environment development, running and analysis of
experiments and as a technical lead.
Alex Goldin contributed to project management.
Mansi Gupta contributed to engineering infrastructure.
Tim Harley contributed to engineering infrastructure.
Felix Hill contributed to environment development.
Peter C Humphreys contributed to agent development and writing.
Alden Hung contributed to agent development, imitation learning and as a sub-
effort lead for agent development.
Jessica Landon contributed to data and tasks, engineering infrastructure,
evaluation development and as a sub-effort lead for data and evaluation.
Timothy Lillicrap contributed to agent development, imitation learning, data
and tasks, environment development, evaluation development, writing, and as an
effort lead.
Hamza Merzic contributed to technical infrastructure.
Alistair Muldal contributed to data and tasks, evaluation development and as a
sub-effort lead for data and evaluation.
Adam Santoro contributed to agent development, imitation learning, running and
analysis of experiments and writing.
Guy Scully contributed to project management.
Tamara von Glehn contributed to agent development, engineering infrastructure,
imitation learning and running and analysis of experiments.
Greg Wayne contributed to agent development, imitation learning, data and
tasks, environment development, evaluation development, writing, and as an
effort lead.
Nathaniel Wong contributed to environment development.
Chen Yan contributed to agent development, data and tasks, engineering
infrastructure, imitation learning, running and analysis of experiments and
writing.
Rui Zhu contributed to agent development, engineering infrastructure,
environment development and writing.

Corresponding Authors:
Greg Wayne<EMAIL_ADDRESS>& Timothy Lillicrap
<EMAIL_ADDRESS>
##  9 Acknowledgments

The authors would like to thank Duncan Williams, Daan Wierstra, Dario de
Cesare, Koray Kavukcuoglu, Matt Botvinick, Lorrayne Bennett, the Worlds Team,
and Crowd Compute.

## References

  * Dunbar (1993) Robin IM Dunbar.  Coevolution of neocortical size, group size and language in humans.  _Behavioral and brain sciences_ , 16(4):681–694, 1993.
  * McClelland et al. (2019) James L McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze.  Extending machine language models toward human-level language understanding.  _arXiv preprint arXiv:1912.05877_ , 2019.
  * Lake and Murphy (2021) Brenden M Lake and Gregory L Murphy.  Word meaning in minds and machines.  _Psychological Review_ , 2021.
  * Winograd (1972) Terry Winograd.  Understanding natural language.  _Cognitive psychology_ , 3(1):1–191, 1972.
  * Pomerleau (1989) Dean A Pomerleau.  Alvinn: An autonomous land vehicle in a neural network.  In _Advances in neural information processing systems_ , pages 305–313, 1989.
  * Schaal (1999) Stefan Schaal.  Is imitation learning the route to humanoid robots?  _Trends in cognitive sciences_ , 3(6):233–242, 1999.
  * Silver et al. (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  _nature_ , 529(7587):484–489, 2016.
  * Vinyals et al. (2019) Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al.  Towards a human-like open-domain chatbot.  _arXiv preprint arXiv:2001.09977_ , 2020.
  * Interactive Agents Team (2020) DeepMind Interactive Agents Team.  Imitating interactive intelligence.  _arXiv preprint arXiv:2012.05672_ , 2020.
  * Ward et al. (2020) Tom Ward, Andrew Bolt, Nik Hemmings, Simon Carter, Manuel Sanchez, Ricardo Barreira, Seb Noury, Keith Anderson, Jay Lemmon, Jonathan Coe, et al.  Using unity to help solve intelligence.  _arXiv preprint arXiv:2011.09294_ , 2020.
  * Christiano et al. (2017) Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _arXiv preprint arXiv:1706.03741_ , 2017.
  * Galashov et al. (2019) Alexandre Galashov, Siddhant M Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojciech M Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess.  Information asymmetry in kl-regularized rl.  _arXiv preprint arXiv:1905.01240_ , 2019.
  * Ross et al. (2011) Stéphane Ross, Geoffrey Gordon, and Drew Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 627–635. JMLR Workshop and Conference Proceedings, 2011.
  * Lynch and Sermanet (2020) Corey Lynch and Pierre Sermanet.  Grounding language in play.  _arXiv preprint arXiv:2005.07648_ , 2020.
  * Osa et al. (2018) Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters.  An algorithmic perspective on imitation learning.  _arXiv preprint arXiv:1811.06711_ , 2018.
  * Alayrac et al. (2020) Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovic, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman.  Self-supervised multimodal versatile networks.  _NeurIPS_ , 2(6):7, 2020.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Wu et al. (2021) Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  Cvt: Introducing convolutions to vision transformers.  _arXiv preprint arXiv:2103.15808_ , 2021.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Turing (1950) Alan M Turing.  Computing machinery and intelligence.  _Mind_ , LIX(236):433–460, 1950.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8):9, 2019.
  * Roy et al. (2006) Deb Roy, Rupal Patel, Philip DeCamp, Rony Kubat, Michael Fleischman, Brandon Roy, Nikolaos Mavridis, Stefanie Tellex, Alexia Salata, Jethran Guinness, et al.  The human speechome project.  In _International Workshop on Emergence and Evolution of Linguistic Communication_ , pages 192–196. Springer, 2006.
  * Yoshida and Smith (2008) Hanako Yoshida and Linda B Smith.  What’s in view for toddlers? using a head camera to study visual experience.  _Infancy_ , 13(3):229–248, 2008.
  * Sullivan et al. (2020) Jess Sullivan, Michelle Mei, Amy Perfors, Erica H Wojcik, and Michael C Frank.  Saycam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective.  _PsyArXiv_ , 2020.
  * Laland (2004) Kevin N Laland.  Social learning strategies.  _Animal Learning & Behavior_, 32(1):4–14, 2004\.
  * Byrne (2009) Richard W Byrne.  Animal imitation.  _Current Biology_ , 19(3):R111–R114, 2009.
  * Chomsky (1959) Noam Chomsky.  Chomsky, n. 1959. a review of bf skinner’s verbal behavior. language, 35 (1), 26–58., 1959.
  * Heyes and Galef Jr (1996) Cecilia M Heyes and Bennett G Galef Jr.  _Social learning in animals: the roots of culture_.  Elsevier, 1996.
  * Ortega et al. (2021) Pedro A Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, et al.  Shaking the foundations: delusions in sequence models for interaction and control.  _arXiv preprint arXiv:2110.10819_ , 2021.
  * Harnad (1990) Stevan Harnad.  The symbol grounding problem.  _Physica D: Nonlinear Phenomena_ , 42(1-3):335–346, 1990.
  * Santoro et al. (2021) Adam Santoro, Andrew Lampinen, Kory Mathewson, Timothy Lillicrap, and David Raposo.  Symbolic behaviour in artificial intelligence.  _arXiv preprint arXiv:2102.03406_ , 2021.
  * Hill et al. (2019) Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley.  Robust instruction-following in a situated agent via transfer-learning from text.  _OpenReview_ , 2019.
  * Anderson et al. (2017) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  _CoRR_ , abs/1711.07280, 2017.  URL http://arxiv.org/abs/1711.07280.
  * Das et al. (2018) Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra.  Embodied question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 2054–2063, 2018.
  * Jang et al. (2021) Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn.  Bc-0: Zero-shot task generalization with robotic imitation learning.  In _5th Annual Conference on Robot Learning_ , 2021.


10(8,1) DOI: 10.13009/EUCASS2022-7346
15(5,1.5) 9th EUROPEAN CONFERENCE FOR AERONAUTICS AND SPACE SCIENCES (EUCASS)

# Evaluation of Aerodynamic Characteristics in Oscillatory Coning Using CFD
Methods

Mohamed Sereez  School of Mechanical, Aerospace and Automotive Engineering,
Coventry University, UK  Mikhail Goman  School of Engineering and Sustainable
Development, De Montfort University, UK

## Abstract

The wind tunnel rotary-balance testing is widely used in aircraft dynamics to
characterise aerodynamics at moderate and high angles of attack during stall
and spin regimes. In such experiments an aircraft test model is rotated along
the wind-tunnel free-stream velocity vector allowing the measurement of
aerodynamic characteristics in steady rotational flow conditions with constant
angle of attack and sideslip. In modified tests named as oscillatory coning,
the rotation vector is tilted from the free-stream velocity vector making flow
conditions with periodic variations in angle of attack and sideslip. This
allows evaluation of unsteady aerodynamic responses superimposed on steady
conical rotation. The use of CFD methods for prediction of aerodynamic
characteristics in rotary-balance and oscillatory coning conditions may
significantly complement experimental data via extrapolation of data for
higher Reynolds numbers, elimination of interference effects from supporting
system, extraction of unsteady aerodynamic derivatives affecting aircraft
dynamic stability. This paper presents CFD simulation results obtained in
rotary-balance and oscillatory coning motions for the NASA Common Research
Model (CRM) in its wing-body configuration at moderate $Re=1\times 10^{6}$,
low Mach number $M\mbox{=}0.2$ and the use of the obtained unsteady responses
in aerodynamic modelling.

##  1 Introduction

Currently, aerodynamic modeling is attracting more and more attention to
analyze the dynamics of an aircraft and simulate flight at high angles of
attack, which are characteristic of stall and lateral departure. This is
dictated by the need to solve the loss of control in flight (LOC-I) safety
problem [1, 2]. The rotary-balance test method was developed to provide
information on the effect of angular rotation on the overall aerodynamic
forces and moments acting on an aircraft in flight [3]. This technique is more
suitable for the developed spin modes, but it is also meaningful for lateral
departures in the stall zone. Unsteady aerodynamic effects in a pure conical
motion of an aircraft can be additionally evaluated under the conditions of
the oscillatory coning, which occurs when the axis of rotation deviates from
the direction of the incoming flow [3] (ch. 4 on p. 69). Such deflection
results in periodic changes in the angle of attack and side slip on the
aircraft model being tested, creating additional non-stationary aerodynamic
responses superimposed on aerodynamic response from conical motion.

This paper presents computational predictions of aerodynamic characteristics
of the NASA Common Research Model (CRM) [4] configuration using the CFD
simulations. The use of computational methods along with experimental data
allows CFD results to be verified, interference effects to be removed from
experimental data, results to be extended to higher Reynolds numbers, and
more. The open source CFD software OpenFOAM, based on the the finite volume
method [5, 6, 7] is used to simulate the rotary-balance and oscillatory coning
types of motion. The hybrid Reynolds Averaged Navier-Stokes Equations along
with Large Eddy Simulations (RANS/LES) is likely to be the most appropriate
approach for the prediction of separated flow regions in the post-stall zone,
however, in this paper the Unsteady Reynolds Averaged Navier Stokes Equations
(URANS) equations along with the Shear-Stress Transport (SST) turbulence model
[8] are used primarily due to two reasons a) the unified grid generation
guidelines of the CRM was not intended for a hybrid RANS/LES simulation [9,
10] and b) due to the high computational cost of the hybrid simulations. This
approach is commonly adopted in aviation applications [11, 12, 7]. In addition
to the dynamic meshing techniques available in OpenFOAM, a new dynamic mesh
motion function termed "conical motion" is implemented. Through this mesh
motion function the rotation of aircraft around any arbitrary axis at the
desired non-dimensional rotation rate can be defined.

In the conical motion, the body-fixed grid rotates around the prescribed
velocity vector $\vec{V}$ and there are no changes in angle of attack and
sideslip angle. In the oscillatory coning the body-fixed grid rotates around
the the prescribed rotation vector $\vec{\Omega}$ with a fixed angle of offset
$\delta$ from the velocity vector $\vec{V}$.

Within the framework of OpenFOAM, both of these dynamic mesh motion functions
are carried out by transforming the grid at every time step based on
quaternions, allowing a conical rotation of the aircraft around the body-fixed
frame of reference $OX,OY,OZ$, in which the body axis projections of
aerodynamic forces and moments can be calculated. Implementation of such a
force and moment extraction function in OpenFOAM, lead to the elimination of
the original calculation of the aerodynamic forces and moments in the wind-
axis frame of reference. The rotation of grid cells causes an induced velocity
on the boundary surface, which then requires special attention to ensure the
no-slip boundary condition on the surface of the aircraft. This type of
boundary condition in OpenFOAM is categorized as "movingWallVelocity" boundary
condition which ensures that the normal velocity flux across the boundary
surface is zero.

In this paper, we present the methodology and capabilities of the open source
CFD software OpenFOAM [13, 6] to simulate rotary-balance and oscillatory
coning testing of aerodynamic characteristics in the post stall zone. The
paper is organized as follows; Section "Computational framework" presents the
governing equations, grid generation methodology, adopted numerical framework
and the kinematics of the rotary-balance and oscillatory coning. Section
"Results and Discussions" presents the validation of the OpenFOAM
computational results for the NASA CRM model in steady conical motion along
with the obtained simulation results for the rotary-balance tests and
oscillatory coning unsteady aerodynamic responses. The concluding comments and
remarks are outlined in the last section.

##  2 Computational framework

###  2.1 Geometry and Grid generation

The NASA Common Research Model (CRM) is a representative of a commercial
transport aircraft with it’s cruise Mach number, $M\mbox{=}0.85$ and design
lift coefficient $C_{L}\mbox{=}0.5$ [4, 10]. The CRM model was specifically
designed for the purpose of validation and comparison of computational
simulation data amongst different CFD codes and against wind tunnel test
results, thus enabling a more unified validation and verification approach for
external aerodynamics applications.

The CRM wing is made up of a thin super-critical airfoil with aspect ratio
$AR\mbox{=}9$ and a taper ratio of $0.25$. The CRM model is provided by the
drag prediction workshop [10] in three different configurations a) Wing-Body
(WB), b) Wing-Body-Horizontal Tail (WB-HT) and c) Wing-Body-Horizontal Tail-
Nacelle-Pylon (WB-HT-NP). The high-lift configuration of the CRM model is also
available in [14]. For the purposes of this paper, the Wing-Body (CRM-WB)
configuration of CRM aircraft model as shown in Fig. 1 is sufficient. For
future reference, instances of "CRM" appearing in this paper refers to the
CRM-Wing-Body (CRM-WB) configuration. This model has a reference mean
aerodynamic chord of $c_{ref}=7.0m$, full-wing span, $b=58.76m$ and the
reference area for the full model is $S_{ref}=383.65m^{2}$. The model
dimensions are further outlined in Table. 1.

Figure 1: CRM Wing-Body configuration Table 1: Reference data for the CRM wing-body model (full model)  Wing span, $b$ | $58.76m$
---|---
Mean Aerodynamic Chord, $MAC$, $c_{ref}$ | $7m$
Reference area $S_{ref}$ | $383.68m^{2}$
Wing aspect ratio, $AR$ | $9.0$
Moment reference point $C_{g}(m)$ | $X\mbox{=}33.67,Y\mbox{=}0,Z\mbox{=}4.52$

The computational grids for the CRM model is made by following the guidelines
provided by the drag prediction workshop committee [9]. Recent studies with
the CRM model [1] shows that a "medium" grid size with $10.0\times 10^{6}$
elements is sufficient for the study of flight dynamics applications including
estimation of longitudinal and lateral derivatives.

The computational grids are made using an H-O grid topology, enabling a smooth
wrapping of the O-type blocking around the aircraft model. This allows to
generate a high quality structured grid with good cell determinant quality for
the surface and volume grid cells. Using an O-type blocking, the boundary
layer is well defined and the cell skewness and orthogonality is maintained at
the desired optimum values. The cell area and volume transition ratios are in
the range of $1.0-1.2$ allowing only 20 percent maximum change, thus ensuring
no large gradient jumps of the flow scalar and vector variables during the
simulation. The variation of cell sizes along the wing are defined as $0.1\%$
of the local span-wise or chord-wise wing length. For the trailing edge a
minimum of $5$ cells are placed. The boundary layer consists of $25$ adjacent
layers with growth rate of $1.15$. The first cell layer’s height was
determined by the non-dimensional wall distance of $Y+\leq 1$ enabling to
fully resolve the boundary layer. A hyperbolic approach is used to ensure that
the first $2\mbox{-}3$ cells of the boundary layer have almost constant
height. The resulting grid is shown in 2.

Figure 2: Adopted grid for the CRM-WB configuration at $Re\mbox{=}1\times
10^{6}$

For the purpose of consistency, the number of elements in the mesh was
maintained as that for the grids used in [1], which is 1) coarse grid with 5
million elements and 2) medium grid with 10 million elements. The comparison
of the obtained computational results for the steady conical motion of CRM
against CFD code ENFLOW in the validation section of this paper shows an
accurate match. Furthermore, the use of a non-dimensional physical time step
size less than the convective length of the flow, $t=0.1C_{ref}/V$ and force
convergence studies shows that the grid size "medium" with $10\times 10^{6}$
elements is sufficient for the purposes of this paper.

###  2.2 Governing Equations

The Navier–Stokes (NS) equations governing incompressible fluid flow are the
continuity equation

$\nabla\cdot\mathbf{U}=0$ (1)

and the momentum equation

$\frac{\partial\mathbf{U}}{\partial
t}+(\mathbf{U}\cdot\nabla)\mathbf{U}-\nu\nabla^{2}\mathbf{U}=-\frac{\nabla\mathbf{p}}{\rho}$
(2)

The computational resources required for Direct Numerical Simulations (DNS) of
Eqs. (1) and (2), especially for flow conditions with high Reynolds numbers,
usually exceed currently available computational capabilities. Instead the
Unsteady Reynolds-Averaged-Navier-Stokes (URANS) equations are solved, in
which the Reynolds stresses arising as a result of averaging the fluctuating
velocities are described by some additional empirical equations either
algebraic or differential to represent an appropriate turbulence model.Most
turbulence models for the URANS equations are based on the concept of eddy
viscosity, which is equivalent to the kinematic viscosity of a fluid, to
describe turbulent mixing or flow momentum diffusion [6]. The Reynolds
stresses, which appear in the URANS equations due to time averaging, are
described in linear turbulence models with the following Boussinesq
assumption:

$\tau_{ij}=2\nu_{t}\\!\left(S_{ij}-\frac{1}{3}\frac{\partial u_{k}}{\partial
x_{k}}\delta_{ij}\right)-\frac{2}{3}\rho k\delta_{ij}1$ (3)

###  2.3 Numerical framework and boundary conditions

For the inlet velocity a Dirichlet boundary condition with fixed velocity
vector $V_{inlet}=({V_{x},V_{y},V_{z}})$ is applied and for the inlet pressure
a Neumann type zero gradient boundary $dp/dx_{j}=0$ condition is used. The
outlet is prescribed a zero gradient velocity $dV/dx_{j}=0$ and a static
pressure of $p_{outlet}=0$. The turbulent kinetic energy $k$ at the inlet is a
fixed value estimated using a turbulence intensity value of $0.1\%$, ensuring
replication of common aerodynamic wind tunnel testing conditions. The aircraft
also has a "movingWallVelocity" boundary condition to enable zero-flux
condition for the dynamic rigid body mesh motion. For the evaluation of
turbulent viscosity standard SST turbulence model [8] is employed. This
approach is commonly used in external aerodynamics involving adverse pressure
gradients and strongly separated flow conditions [15, 12]. The two equation
$k\mbox{-}\omega$ SST model solves for the prediction of two scalar flow
variables which are the turbulent kinetic energy $k$ and specific dissipation
rate of turbulence $\omega$.

On testing of various finite volume schemes and solvers of OpenFOAM the Pre-
conditioned Conjugate (PCG) solver with Geometric Algebraic Multi-Grid (GAMG)
as a pre-conditioner was found out to be the most efficient algorithm.
Employing GAMG pre-conditioner with 10-30 iterations and applying pre and post
smoothing of the residuals for $2-3$ levels, ensures that only about about
$30-50$ iterations of the PCG linear solver is needed to drive the residuals
to near zero values at each time step. Additionally, the PCG solver was found
out to be more robust for grid size of 10 million elements in the parallel
approach of OpenFOAM. The gradients of the flow quantities are quantified
using the second order accurate Gauss linear scheme with limiters based on
cell center values of the flow quantities. The divergence of the velocity
field and the scalar turbulent quantities are also estimated using second
order accuracy with the "cellLimited Gauss linear" scheme of OpenFOAM. For
estimation of contribution of cell center variables to the faces a linear
interpolation is used.

The computational simulations using OpenFOAM are carried out on the high
performance computing cluster "Zeus" [16] (Heterogeneous HPC cluster with 3200
CPU cores) at Coventry University using 4 compute nodes, 32 CPU-cores per node
and a RAM of 128gb per node. For the purpose of fast, robust and accurate
unsteady simulations an implicit dual-time stepping method is implemented and
used in the OpenFOAM compiled in this cluster. The dual time framework
formulated as in from[17, 18] is commonly adopted for aerodynamic flows with
rigid body mesh motion such as dynamic stall simulations.

The dual time stepping framework can be described as:

$\frac{dW}{dt}+R(W)=0$ (4)

where W is the flow variables at grid points and R is the residuals of flow
variables, including the dissipative terms. The discretized equation 4 is then
formulated in dual-stepping in conjunction with Euler backward time
integration technique as follows:

$\frac{dW}{dt^{*}}+\frac{3}{2\Delta t}W-\frac{4}{2\Delta
t}W^{n}+\frac{1}{2\Delta t}W^{n-1}+R(W)=0$ (5)

In Eq. 5 $t*$ is the pseudo time step size, $\Delta t$ is the physical time
step size and $n$ is the time step number of the simulation.

The dual time stepping contribution to an assembled finite volume matrix with
$A$ being diagonal coefficients and $H$ being the off diagonal coefficients is
as follows.

$A+=\bigg{(}\frac{3}{2\Delta
t}+\frac{1}{\alpha_{k}\Delta\tau}\bigg{)}\int_{S}dSdA$ (6)

$H^{\prime}+=\Bigg{(}\bigg{(}\frac{1}{\alpha_{k}\Delta\tau}\bigg{)}U^{m-1}+\bigg{(}\frac{2}{\Delta
t}\bigg{)}U^{n-1}-\bigg{(}\frac{1}{2\Delta
t}\bigg{)}U^{n-2}\Bigg{)}\int_{S}dSdA$ (7)

where isn is the time step level, m is the sub-iteration level within each
time step, $\alpha_{k}$ is the Runge-Kutta coefficient in each stage and
$\int_{S}dSdA$ is the integrated volume of each element.

###  2.4 Rotary Balance testing setup in OpenFOAM

The rotational flow and the induced aerodynamic responses to the aircraft
model depend on the type of rotational motion. The differences in the
kinematics of rotary-balance conical motion and oscillatory coning motion are
described below.

####  2.4.1 Rotary-balance: Conical motion setup

With a steady conical motion, the aircraft model rotates around the velocity
vector $\vec{V}$, while the angle of attack $\alpha$, and sideslip angle
$\beta$ remain constant throughout the movement, since the rotation vector
$\vec{\Omega}$ coincides with the velocity vector $\vec{V}$ as shown in Fig3.

Figure 3: Kinematics of an aircraft model in conical motion - $\alpha=const$,
$\beta=const$.

The projections of aerodynamic forces and moments in OpenFOAM are calculated
in the fixed inertial reference frame directed along the free-stream velocity.
Aircraft dynamics require aerodynamic projections on the body-fixed frame of
reference $OXYZ$, which rotates with aircraft model. In order to achieve this,
at the end of every time step and movement of the grid, a quaternion
transformation is applied to the projections of the aerodynamic forces and
moments generated in OpenFOAM.

The rotary-balance type of conical motion is achieved with new dynamic mesh
motion implemented in OpenFOAM termed "conicalMotion" and a sample setup is
shown below:

solidBodyMotionFunction conicalMotion;
origin (33.7 4.52 0 ); // rotation origin
rotationAxis (-0.99756 -0.06976 0); // rotation axis
rotationOmega 0.23348; // rotation rate in rad/s

####  2.4.2 Oscillatory coning motion setup

In oscillatory coning motion, the rotation vector $\vec{\Omega}$ deviates from
the velocity vector $\vec{V}$ by an angle $\delta$, as shown in Fig. 4, for
example, with a deviation only in the plane of symmetry. More generally, the
rotation vector $\vec{\Omega}$ can have a lateral projection giving a non-zero
sideslip center position. In the oscillatory coning, the kinematics of motion
of the aircraft model is characterized by a sinusoidal change in the angle of
attack $\alpha(t)$ and the angle of sideslip $\beta(t)$ with amplitudes equal
to the angle $\delta$ between the angular vectors $\vec{\Omega}$ and
$\vec{V}$. For an observer inside a body-fixed reference frame, the vector
$\vec{V}$ rotates around the vector $\vec{\Omega}$ on a conical surface with
an angle at the vertex $\delta$.

Figure 4: Kinematics of an aircraft model in oscillatory conical motion -
$\alpha=\theta+\delta\cos\Omega t$, $\beta=\delta\sin\Omega t$.

The roll angle $\phi$ during conical rotation changes linearly with time in
accordance with a fixed rotation rate $\Omega$:

$\phi=\Omega t$ (8)

And variations of angles of attack, $\alpha$ and sideslip, $\beta$ can be
approximately represented in the following form:

$\alpha(t)=\theta+\delta\cos\phi;\,\,\,\beta(t)=\delta\sin\phi$ (9)

Time variations of the angle of attack $\alpha$ and sideslip $\beta$ are shown
for the installation angle $\theta=7^{\circ}$ and the deflection angle
$\delta=3^{\circ}$ in Fig.5 (left). The oscillatory coning kinematics may be
also expressed as a uniform movement along a circular trajectory in the plane
of angle of attack and sideslip with a radius equal to the offset angle
$\delta$ and a center defined by offset angle $\theta$ as shown in Fig. 5
(right).

Figure 5: Kinematics of oscillatory coning at $\theta=7^{\circ}$,
$\delta=3^{\circ}$, $\omega=\Omega b/2V=0.1$.

##  3 Simulation results and discussion

###  3.1 Results of rotary-balance for CRM wing-body configuration

Validation of the formulated computational framework of OpenFOAM for the
rotary-balance motion was carried out by comparison with the CFD simulation
results for the CRM wing-body configuration using the NLR ENFLOW CFD codes
presented in [1]. The results of the rotary-balance motion simulation in
OpenFOAM generally agree well with the results of the ENFLOW CFD code. This is
clearly seen in Fig. 6 for the normal force coefficient $C_{Z}$ and in Fig.7
for the rolling moment coefficient $C_{l}$ calculated at non-dimensional
rotation rate $\omega=0.1$.

There is generally a rather good correlation between the results from [1] and
OpenFOAM both at low angles of attack and in the stall region at high angles
of attack with developed separation zones. The roll damping at low angles of
attack, defined by aerodynamic derivative $C_{l\omega}<0$, degrades with
increase of angle of attack due to propagation of separation along the wing.
In the stall region this derivative changes its sign $C_{l\omega}>0$
indicating onset of autorotation regime in the region with negative slope in
the normal force coefficient $C_{Z\alpha}<0$. The minor differences in
simulation results of the NLR’s ENFLOW and OpenFOAM may be addressed to the
employed numerical schemes for the evaluation of diffusive and advective
fluxes along with variation in the adopted gradient calculation schemes.

The conducted validation study strengthens our confidence in the implemented
procedure of the rotary-balance movement and transformation of aerodynamic
forces and moments into a frame of reference fixed with the body in OpenFOAM.

Figure 6: Computational prediction of the normal force coefficient $C_{Z}$
during rotary-balance movement at $\beta=0$, $\omega=0.1$ for CRM wing-body
configuration. Figure 7: Computational prediction of the rolling moment
coefficient $C_{l}$ during rotary-balance movement at $\beta=0$, $\omega=0.1$
for CRM wing-body configuration. Figure 8: Skin friction coefficient $C_{f}$
for the CRM model at $\alpha=14^{\circ}$, $\omega=0.1$ with flow conditions
$Re=1\times 10^{6}$, $M=0.2$.

Figure 9: Contours of Mach number for the left (a) and right (b) wing in
vertical cross-sections placed at $Y=\pm 0.37\,b/2$ in rotary-balance motion
with $\omega=0.1$ at $\alpha=14^{\circ}$ $Re=1\times 10^{6}$, $M=0.2$.

Visualization of the flow parameters during rotary-balance motion on the CRM
surface and around the wing is shown in Fig. 8 and Fig. 9. The streamlines
with contours of the skin friction coefficient $C_{f}$ are shown in Fig. 8 for
the right wing upper surface during the rotary-balance motion with
$\alpha=14^{\circ}$ and $\omega=0.1$. This shows a full flow separation across
the wing defined by the area with low skin friction. At the same time the flow
on the left wing is practically fully attached.

This statement is supported by the visualized contours of the Mach number
placed at wing vertical cross-sections with coordinates $Y=\pm 0.37\,b/2$
shown in Fig. 9. It is evident that on the left wing the flow is fully
attached while the right wing the flow is massively separated with significant
loss in the normal force leading to the auto-rotation rolling moment. As
indicated in Fig. 7 the rolling moment coefficient at $\alpha=14^{\circ}$ is
positive $C_{l}>0$ at $\omega=0.1$ demonstrating that the aircraft is in the
auto-rotation regime.

###  3.2 Results for oscillatory coning motion

In an oscillatory coning motion, the angle of attack and sideslip periodically
change, causing variations in aerodynamic responses. Aerodynamic variations
can also be expected to have periodic variations, but not necessarily single
harmonics such as periodic changes in angle of attack and sideslip. After
running the simulation, it takes some time for the aerodynamic characteristics
to converge to ideal periodic processes, and this initial transient must be
excluded from further analysis.

The process of convergence of the normal force coefficient $C_{Z}$ and the
rolling moment coefficient $C_{l}$ is shown in Fig. 10 for parameters
$\theta=7^{\circ}$, $\delta=3^{\circ}$ and $\omega=0.1$. It can be seen that
after the physical time $t\geq 20s$ the changes in both aerodynamic
coefficients become ideally periodic. The normal force coefficient $C_{Z}$ is
close to the carrier harmonic, but the rolling moment coefficient $C_{l}$
includes segments with high harmonics due to higher sensitivity to flow
separation processes.

Figure 10: Convergence of aerodynamic coefficients $C_{Z}$ and $C_{l}$ in
oscillatory coning with $\theta\mbox{=}7^{\circ}$, $\delta\mbox{=}3^{\circ}$,
and $\omega=0.1$

The OpenFOAM simulation results obtained in oscillatory coning motions for
three settings in rotation axis with $\theta=1^{\circ},7^{\circ},11^{\circ}$
and offset angle $\delta=3^{\circ}$ with non-dimensional rotation rate of
$\omega=0.1$ are shown in Figs. 11 \- 13. The simulations were conducted at
flow conditions with $Re=1\times 10^{6}$, $M=0.2$.

Non-stationary aerodynamic reactions in the oscillatory coning at different
installation angles $\theta$ and displacement angle $\delta=3^{\circ}$ in the
form of closed trajectories are imposed on the static dependencies on the
angle of attack obtained in conical motions of rotary-balance tests.

The geometric shapes of non-stationary aerodynamic loops reflect the
possibility of their representation in a linearized form. If the non-
stationary response loops are close to an elliptical shape and fit well around
the static curve, this indicates that the aerodynamic response can be well
approximated using the static dependence plus the non-stationary aerodynamic
contribution proportional to the rates of change of angle of attack and side
slip in the following form ($i=Z,m,l$) [1]:

$C_{i}=C_{i_{rb}}(\alpha,\beta,\omega)+C_{i\dot{\alpha}_{oc}}\dfrac{\dot{\alpha}c}{2V}+C_{i\dot{\beta}_{oc}}\dfrac{\dot{\beta}b}{2V}$
(10)

where $C_{i_{rb}}(\alpha,\beta,\omega)$ is steady aerodynamic components
identified in the rotary-balance tests, $C_{i\dot{\alpha}_{oc}}$ and
$C_{i\dot{\beta}_{oc}}$ are unsteady aerodynamic derivatives from oscillatory
coning aerodynamic responses. To extract aerodynamic derivatives
$C_{i\dot{\alpha}_{oc}}$ and $C_{i\dot{\beta}_{oc}}$ the periodic aerodynamic
responses $C_{i}(t)=C_{i}(t+2\pi/\Omega)$ measured in oscillatory coning
motions should be approximated by the first three terms of the Fourier series
expansion:

$C_{i}(t)=C_{i0}+C_{is}\sin(\Omega t)+C_{ic}\cos(\Omega t)$ (11)

where

$\begin{array}[]{l}C_{i0}=C_{i_{rb}}(\alpha_{0},\beta_{0},\omega)\\\
C_{is}=\left(C_{i\beta_{rb}}-\dfrac{\omega
c}{b}C_{i\dot{\alpha}_{oc}}\right)\delta\\\
C_{ic}=\left(C_{i\alpha_{rb}}+\omega C_{i\dot{\beta}_{oc}}\right)\delta\\\
\end{array}$ (12)

The Fourier approximation of the closed loop unsteady responses in Figs. 11 \-
13 for the normal force, pitching and rolling moment coefficients respectively
are shown by blue ellipses defined by coefficients $C_{i0}$, $C_{is}$ and
$C_{ic}$ from the Fourier series approximation (11). One can see that linear
approximations of aerodynamic responses in the form (10) are mostly acceptable
except of aerodynamic responses in the normal force and the rolling moment
coefficients covering the stall zone at setting angle $\theta=11^{\circ}$.

Figure 11: Computational prediction of the normal force coefficient $C_{Z}$
during oscillatory conical motion at $\omega=0.1$, $M=0.2$ and $Re=1.0\times
10^{6}$. Figure 12: Computational prediction of the pitching moment
coefficient $C_{m}$ during oscillatory conical motion at $\omega=0.1$, $M=0.2$
and $Re=1.0\times 10^{6}$. Figure 13: Computational prediction of the rolling
moment coefficient $C_{l}$ during oscillatory conical motion at $\omega=0.1$,
$M=0.2$ and $Re=1.0\times 10^{6}$. Figure 14: Surface streamlines superimposed
on contours of skin friction coefficient $C_{f}$ during oscillatory conical
motion at $\alpha(t)=13.3^{\circ}$, $\beta=1.96^{\circ}$ and
$\omega\mbox{=}0.1$ Figure 15: Contours of pressure coefficient $C_{p}$ on
aircraft surface and three-dimensional streamlines of velocity during
oscillatory conical motion at $\alpha(t)=13.3^{\circ}$, $\beta=1.96^{\circ}$
and $\omega\mbox{=}0.1$

Visualization of flow parameters during oscillatory coning motion at different
times can provide important information for the interpretation of non-
stationary aerodynamic characteristics and help in the development of
phenomenological aerodynamic models in critical flight conditions. Examples of
such visualization are shown in Fig. 14 and Fig. 15.

##  4 Conclusions

Within the framework of the open source CFD code OpenFOAM, a computational
approach to modeling aerodynamic characteristics during rotary-balance and
oscillatory coning motions is presented. Verification of the results of the
OpenFOAM rotary-balance simulation showed very good agreement with the results
obtained using the CFD ENFLOW code developed at the Royal Netherlands
Aerospace Center (NLR). The presented new OpenFOAM capabilities for modeling
rotary-balance and oscillatory coning aerodynamic characteristics can make an
important contribution to adequate aerodynamic modeling in the extended flight
envelope of modern transport aircraft.

## References

  * Abramov et al. [2019] Abramov, N. B., Goman, M. G., Khrabrov, A. N., and Soemarwoto, B. I., “Aerodynamic Modeling for Poststall Flight Simulation of a Transport Airplane,” _Journal of Aircraft_ , Vol. 56, No. 4, 2019, pp. 1427–1440.  doi:https://doi.org/10.2514/1.C034790.
  * Nguyen et al. [2022] Nguyen, D. H., Goman, M. G., Lowenberg, M. H., and Nield, S. A., “Evaluating Longitudinal Unsteady Aerodynamic Effects in Stall for a T-Tail Transport Models,” _Journal of Aircraft_ , 2022.  doi:https://doi.org/10.2514/1.C036622.
  * AGA [1990] “Rotary-Balance Testing for Aircraft Dynamics,” , AGARD-AR-265, 1990.
  * Vaseberg et al. [.] Vaseberg, J. C., DeHaan, M. A., Rivers, S. M., and Whals, R. A., “Development of a Common Research Model for Applied CFD Validation Studies,” , .  https://https://aiaa-dpw.larc.nasa.gov/Workshop4/AIAA-2008-6919-Vassberg.pdf last accessed 05 June 2022.
  * Weller et al. [1998] Weller, H. G., Tabor, G., Jasak, H., and Fureby, C., “A tensorial approach to computational continuum mechanics using object-oriented techniques,” _Computers in Physics_ , Vol. 12, No. 6, 1998.  doi:https://doi.org/10.1063/1.168744.
  * Chen et al. [2014] Chen, G., Xiong, Q., Morris, J., P., Paterson, E., Sergeev, A., and Wang, Y. C., “OpenFOAM for Computational Fluid Dynamics,” _Notices of the AMS_ , Vol. 61, No. 4, 2014, pp. 354–363.
  * Sereez et al. [2020] Sereez, M., Abramov, N. M., and Goman, M. G., “Prediction of Static Aerodynamic Hesteresis on a Thin Airfoil Using OpenFOAM,” _Journal of Aircraft_ , 2020.  doi:https://doi.org/10.2514/1.C035956.
  * Menter [1994] Menter, F. R., “Two-equation eddy-viscosity turbulence models for engineering applications,” _AIAA_ , Vol. 32, No. 8, 1994.  doi:https://doi.org/10.2514/3.12149.
  * Vassberg [2011] Vassberg, J., “A Unified Baseline Grid about the Common Research Model Wing/Body for the Fifth AIAA CFD Drag Prediction Workshop,” , AIAA, June 2011.  doi:https://doi.org/10.2514/6.2011-3508.
  * AIAA [2009] AIAA, “Drag Prediction Workshop 4,” , 2009.  https://aiaa-dpw.larc.nasa.gov/Workshop4 last accessed 8 June 2022\.
  * Ashton and Skaperdas [2019] Ashton, N., and Skaperdas, V., “Verification and Validation of OpenFOAM for High-Lift Aircraft Flows,” _Journal of Aircraft_ , Vol. 56, No. 4, 2019, pp. 1641–1657.  doi:https://doi.org/10.2514/1.C034918.
  * Sereez et al. [2016] Sereez, M., Abramov, N. B., and Goman, M. G., “Computational Simulation of Airfoils Stall Aerodynamics at Low Reynolds Numbers,” Applied Aerodynamics Conference, RAeS, 2016.  doi:http://hdl.handle.net/2086/14049.
  * OpenFOAM [.] OpenFOAM, “OpenFOAM: The Open Source Computational Fluid Dynamics Toolbox,” , .  http://www.openfoam.com last accessed 8 June 2019.
  * AIAA [2013] AIAA, “High Lift Prediction Workshop 2,” , 2013.  https://hiliftpw.larc.nasa.gov/index-workshop2.html last accessed 8 June 2022.
  * Cummings and Goreyshir [2013] Cummings, R. M., and Goreyshir, M., “Challenges in the Aerodynamic Modelling of an Oscillating and Translating Airfoil at Large Incidence Angles,” _Aerospace Science and Technology_ , Vol. 28, No. 1, 2013, pp. 176–190.  doi:https://doi.org/10.1016/j.ast.2012.10.013.
  * high herformance cluster [.] high herformance cluster, E., “Zeus HPC,” , .  https://hpc.coventry.domains/links/zeus-hpc/ last accessed 25 June 2022.
  * Nichols and Heikkinen [2006] Nichols, R. H., and Heikkinen, B. D., “Validation of Implicit Algorithms for Unsteady Flows Including Moving and Deforming Grids,” _Journal of Aircraft_ , Vol. 43, No. 5, 2006, pp. 1341–1351.  doi:https://doi.org/10.2514/1.17111.
  * Luchtenburg et al. [2015] Luchtenburg, D. M., Rowley, W. C., Lohry, M. W., Martinelli, L., and Stengel, R. F., “Unsteady High-Angle-of-Attack Aerodynamic Models of a Generic Jet Transport,” _AIAA_ , Vol. 52, No. 3, 2015, pp. 890–895.  doi:https://doi.org/10.2514/1.C032976.