# NeurIPS 2021 Competition IGLU:
Interactive Grounded Language Understanding
in a Collaborative Environment

Julia Kiseleva1 Ziming Li3 Mohammad Aliannejadi2 Shrestha Mohanty1 Maartje ter
Hoeve2 Mikhail Burtsev4,5 Alexey Skrynnik4 Artem Zholus4 Aleksandr Panov4
Kavya Srinet6 Arthur Szlam6 Yuxuan Sun6 Katja Hofmann1 Michel Galley1 Ahmed
Awadallah1

1Microsoft Research, 2University of Amsterdam,
3Alexa AI, 4MIPT, 5AIRI, 6Facebook AI

###### Abstract

Human intelligence has the remarkable ability to quickly adapt to new tasks
and environments. Starting from a very young age, humans acquire new skills
and learn how to solve new tasks either by imitating the behavior of others or
by following provided natural language instructions. To facilitate research in
this direction, we propose _IGLU: Interactive Grounded Language Understanding
in a Collaborative Environment_.

The primary goal of the competition is to approach the problem of how to build
interactive agents that learn to solve a task while provided with grounded
natural language instructions in a collaborative environment. Understanding
the complexity of the challenge, we split it into sub-tasks to make it
feasible for participants.

This research challenge is naturally related, but not limited, to two fields
of study that are highly relevant to the NeurIPS community: Natural Language
Understanding and Generation (NLU/G) and Reinforcement Learning (RL).
Therefore, the suggested challenge can bring two communities together to
approach one of the important challenges in AI. Another important aspect of
the challenge is the dedication to perform a human-in-the-loop evaluation as a
final evaluation for the agents developed by contestants.

### Keywords

Natural Language Understanding (NLU), Reinforcement Learning (RL), Grounded
Learning, Interactive Learning, Games

### Competition type

Regular

##  1 Competition description

###  1.1 Background and impact

Humans have the remarkable ability to quickly adapt to new tasks and
environments. Starting from a very young age, humans acquire new skills and
learn how to solve new tasks either by imitating behavior of others or by
following natural language instructions that are provided to them (An, 1988;
Council, 1999). Natural language communication provides a natural way for
humans to acquire new knowledge, enabling us to learn quickly through language
instructions and other forms of interaction such as visual cues. This form of
learning can even accelerate the acquisition of new skills by avoiding trial-
and-error and statistical generalization when learning only from observations
(Thomaz et al., 2019). Studies in developmental psychology have shown evidence
of human communication being an effective method for transmission of generic
knowledge between individuals as young as infants (Csibra and Gergely, 2009).
These observations have inspired attempts from the AI research community to
develop grounded interactive _agents_ that are capable of engaging in natural
back-and-forth dialog with humans to assist them in completing real-world
tasks (Winograd, 1971; Narayan-Chen et al., 2017; Levinson, 2019; Chen et al.,
2020).

Importantly, the agent needs to understand when to initiate feedback requests
if communication fails or instructions are not clear and requires learning new
domain-specific vocabulary (Aliannejadi et al., 2020; Rao and Daumé III, 2018;
Narayan-Chen et al., 2019; Jayannavar et al., 2020). Despite all these
efforts, the task is far from solved. For that reason, we propose the IGLU
competition, which stands for _Interactive Grounded Language Understanding in
a collaborative environment_.

Specifically, the goal of our competition is to approach the following
scientific challenge: _How to build interactive agents that learn to solve a
task while provided with grounded natural language instructions in a
collaborative environment?_

By ‘interactive agent’ we mean that the agent is able to follow the
instructions correctly, is able to ask for clarification when needed, and is
able to quickly adapt newly acquired skills, just like humans are able to do
while collaboratively interacting with each other.111An example of such
grounded collaboration is presented in Figure 1.

The described research challenge is naturally related, but not limited, to two
fields of study that are highly relevant to the NeurIPS community: Natural
Language Understanding and Generation (NLU / NLG) and Reinforcement Learning
(RL).

##### Relevance of NLU/G

Natural language interfaces (NLIs) have been the “holy grail” of human-
computer interaction and information search for decades (Woods et al., 1972;
Codd, 1974; Hendrix et al., 1978). The recent advances in language
understanding capability (Devlin et al., 2018; Liu et al., 2019; Clark et al.,
2020; Adiwardana et al., 2020; Roller et al., 2020; Brown et al., 2020)
powered by large-scale deep learning and increasing demand for new
applications has led to a major resurgence of natural language interfaces in
the form of virtual assistants, dialog systems, semantic parsing, and question
answering systems (Liu and Lane, 2017, 2018; Dinan et al., 2020; Zhang et al.,
2019). The horizon of NLIs has also been significantly expanding from
databases (Copestake and Jones, 1990) to, knowledge bases (Berant et al.,
2013), robots (Tellex et al., 2011), Internet of Things (virtual assistants
like Siri and Alexa), Web service APIs (Su et al., 2017), and other forms of
interaction (Fast et al., 2018; Desai et al., 2016; Young et al., 2013).
Recent efforts have also focused on interactivity and continuous learning to
enable agents to interact with users to resolve the knowledge gap between them
for better accuracy and transparency. This includes systems that can learn new
task from instructions (Li et al., 2020), assess their uncertainty (Yao et
al., 2019), ask clarifying questions (Aliannejadi et al., 2020, 2021) and seek
and leverage feedback from humans to correct mistakes (Elgohary et al., 2020).

##### Relevance of RL

Recently developed RL methods celebrated successes for a number of tasks
(Bellemare et al., 2013; Mnih et al., 2015, 2016; Silver et al., 2017; Hessel
et al., 2018). One of the aspects that helped to speed up RL methods
development is game-based environments, which provide clear goals for an agent
to achieve in flexible training settings. However, training RL agents that can
follow human instructions has attracted fewer exploration (Chevalier-Boisvert
et al., 2019; Cideron et al., 2019; Hu et al., 2019; Chen et al., 2020; Shu et
al., 2017), due to complexity of the task and lack of proper experimental
environments. Shu et al. (2017) proposed a hierarchical policy modulated by a
stochastic temporal grammar for efficient multi-task reinforcement learning
where each learned task corresponds to a human language description in
Minecraft environment. The BabyAI platform (Chevalier-Boisvert et al., 2019)
aims to support investigations towards learning to perform language
instructions with a simulated human in the loop. Chen et al. (2020)
demonstrated that using step-by-step human demonstrations in the form of
natural language instructions and action trajectories can facilitate the
decomposition of complex tasks in a crafting environment.

##### Minecraft as an Environment for Grounded Language Understanding

Szlam et al. (2019) substantiated the advantages of building an open
interactive assistant in the sandbox construction game of Minecraft instead of
a “real world” assistant, which is inherently complex and inherently costly to
develop and maintain. The Minecraft world’s constraints (e.g., coarse 3-d
voxel grid and simple physics) and the regularities in the head of the
distribution of in-game tasks allow numerous scenarios for grounded NLU
research (Yao et al., 2020; Srinet et al., 2020). Minecraft is an appealing
competition domain due to its popularity as a video game, of all games ever
released, it has the second-most total copies sold. Moreover, since it is a
popular game environment, we can expect players to enjoy interacting with the
assistants as they are developed, yielding a rich resource for a human-in-the-
loop studies. Another important advantage of using Minecraft is the
availability of the highly developed set of tools for logging agents
interactions and deploying agents for evaluation with human-in-the-loop,
including:

  * •

Malmo (Johnson et al., 2016): a powerful platform for AI experimentation;

  * •

Craftassist (Gray et al., 2019): a framework for dialog-enabled interactive
agents;

  * •

TaskWorldMod (Ogawa et al., 2020): a platform for situated task-oriented
dialog data collection using gamification; and

  * •

MC-Saar-Instruct (Köhn et al., 2020): a platform for Minecraft Instruction
Giving Agents.

Besides, mainly due to the success of previous competitions (Guss et al.,
2019; Perez-Liebana et al., 2019), Minecraft is a widely used environment by
the RL community for experimentation with (mainly single) agents trained by
demonstration. Therefore, using Minecraft would set a low barrier for the RL
community to contribute to IGLU. To simplify the competition settings and
possibly lower the entry bar for the NLU/NLG community, we will use simulated
Blocks World in Minecraft (Jayannavar et al., 2020).

##### Relevance to Real Live Scenarios and Societal Impact

Several important real-life scenarios have the potential to benefit from the
results of our competition:

  * •

Education: Minecraft: Education Edition222https://education.minecraft.net/ is
a game-based learning platform that promotes creativity, collaboration, and
problem-solving in an immersive digital environment. As of 2021, educators in
more than $115$ countries are using Minecraft across the curriculum. As stated
in Ayoub , adding AI elements to this educational platform will move its
potential to a new level. AI applications have the power to become a great
equalizer in education. Students can get personalized education and
scaffolding while being less dependent on uncontrollable factors such as the
quality of their teachers or the amount of help they receive from their
caregivers.

  * •

Robotics: Bisk et al. (2016) proposed a protocol and framework for collecting
data on human-robot interaction through natural language. The work
demonstrated the potential for unrestricted contextually grounded
communications between human and robots in blocks world. Developing robots to
assist humans in different tasks at home has attracted much attention in the
Robotics field (Stuckler et al., 2012). In fact, the
Robocup@Home333https://athome.robocup.org/ and the Room-Across-
Room444https://ai.google.com/research/rxr/habitat have run for several years.
Given that the main human-robot interaction is through dialog, and the robot
is supposed to assist the human in multiple tasks, we envision IGLU to enable
more effective task grounded dialog training between human and robots.

###  1.2 Novelty

There is a long history of competitions focused on NLU/G tasks. Especially in
recent years we have seen a large number of challenges dedicated to open-
domain dialog systems (Hauff et al., 2021; Dalton et al., 2020; Spina et al.,
2019; Chuklin et al., 2018; Arguello et al., 2018), such as ConvAI (Burtsev
and Logacheva, 2020), ConvAI2 (Dinan et al., 2020), ConvAI3: Clarifying
Questions for Open-Domain Dialogue Systems (ClariQ) (Aliannejadi et al., 2020,
2021), as well as a series of competitions of the Alexa
Prize555https://developer.amazon.com/alexaprize. There are great efforts in
the community to advance task-oriented dialogs by suggesting competitions,
such as the Dialog System Technology Challenge (DSTC-8) (Kim et al., 2019);
benchmarks and experimental platforms, e.g., Convlab, which offers the
annotated MultiWOZ dataset (Budzianowski et al., 2018) and associated pre-
trained reference models (Lee et al., 2019). There are fewer attempts to study
multi-modal dialog systems, e.g., Situated Interactive Multi-Modal
Conversational Data Collection And Evaluation Platform (SIMMC) (Crook et al.,
2019) or Audio Visual Scene-Aware Dialog Track in DSTC8 (Hori et al., 2018).

There are a number of RL competitions such as MineRL (Guss et al., 2019) and
MARLO (Perez-Liebana et al., 2019) that leverage the Minecraft environment. RL
approaches have also been tried for text games environments, such as TextWold
(Yuan et al., 2019)666https://www.microsoft.com/en-
us/research/project/textworld/ and Learning in Interactive Games with Humans
and Text(Light) (Urbanek et al., 2019)777https://parl.ai/projects/light/.

In comparison with previous efforts, to our knowledge, we are the first to
propose a competition that tackles the task of grounded language understanding
and interactive learning that brings together the NLU/G and RL research
communities. The other key difference is our attempt to perform a human-in-
the-loop evaluation as a final way for evaluating.

###  1.3 Data

Figure 1: An example, taken from (Narayan-Chen et al., 2019), of the grounded
collaborative dialog between Architect (A) and Builder (B), which is happening
in 3D Block World.

##### The general setup

IGLU is partially motivated by the HCRC Map Task Corpus (Thompson et al.,
1993), which consists of route-following dialogs between an _Instruction
Giver_ and a _Follower_. These are given maps of an environment that differ on
insignificant details. Namely, HCRC Map Task represents a cooperative
asymmetric task that involves two participants. Further, we also use an
_Instruction Giver_ and a _Follower_ , but call them _Architect_ and _Builder_
respectively, following the naming convention in (Narayan-Chen et al., 2019).

Narayan-Chen et al. (2019) collected an openly available Minecraft dialog
Corpus for a Collaborative Building Task888The dataset was released using
Using Creative Commons Public Licenses
https://drive.google.com/drive/folders/16lDzswcQh8DR2jkQJdoVTK-RyVDFPHKa. The
authors are aware of the competition, and they are part of our Advisory
Board.. Figure 1 gives an example of the collected grounded dialogs. The
authors used the following setup: the Architect is provided with a target
structure that needs to be built by the Builder. The Architect provides
instructions to the Builder on how to create the target structure and the
Builder can ask clarifying questions to the Architect if an instruction is
unclear Zhang et al. (2021). This dialog happens by means of a chat interface.
The Architect is invisible to the Builder, but the Architect can see the
actions of the Builder. The collected dataset has:

  * •

$509$ collected human-to-human dialogs along with RGB observations, and
inventory information;

  * •

games played over the course of 3 weeks (approx. 62 hours overall) with $40$
volunteers. Each game took 8.55 minutes on average; and

  * •

$163$ tasks for the Architect-Builder collaborative game.

####  1.3.1 Data collection

##### Extending the dataset with new tasks

We are working on creating new tasks for data collection that will be used to
(1) extend the existing Minecraft Dialog Corpus (Narayan-Chen et al., 2019)
and (2) perform offline and online evaluation (as they are unseen by
participants). We provide up to $200$ newly designed target structures for
building and each of them is associated with a difficulty level, ranging from
_Easy_ to _Normal_ and _Hard_. The difficulty level reflects the complexity of
the target structure by considering the characteristics, such as the number
and color diversity of the used blocks. This gives us a reasonable method to
test the limit of our trained agents, including both architects and builders.
Each target structure will be assigned to at least three different Architect-
Builder pairs. Builders can manipulate the world within a $11\times 11\times
9$ sized build region and they have access to an inventory of $6$ colors of
blocks and $20$ blocks of each color in each task.

##### Tooling for data collection

In order to enable data collection outside of lab settings, we extend the
combination of tools proposed by Narayan-Chen et al. (2019) and Johnson et al.
(2016) for data logging. Specifically, we plan to provide an installer to the
players to connect them through our server, which will enable them to play
predefined tasks and store data afterwards. Minecraft players will be invited
through game forums and they will be rewarded with Minecoins, the internal
currency that is used in Minecraft. For collecting extended datasets at crowd-
sourcing platforms, we will use the Craftassist tool (Gray et al., 2019),
which can give us more data. However, potentially it can be of lower quality.
In the case of extending the original dataset, we connect two humans. In the
case of evaluation, we expose the trained agent to a single player.

Following the setup in Narayan-Chen et al. (2019), we record the progression
of each task, corresponding to the construction of a target structure by an
Architect and Builder pair, as a discrete sequence of game observations. Each
observation contains the following information: 1) a time stamp, 2) the chat
history up until that point in time, 3) the Builder’s position (a tuple of
real-valued $x$, $y$, $z$ coordinates as well as pitch and yaw angles,
representing the orientation of their camera), 4) the Builder’s block
inventory, 5) the locations of the blocks in the build region, 6) screenshots
taken from the Architect’s and the Builder’s perspectives. The detailed block
locations and colors of each target structure will be linked to the
corresponding building record.

##### Institutional Review Boards (IRB) Process

As we plan to collect more data where human subjects are involved, we
submitted our proposal for setting up a data collection pipeline to
Institutional Review Boards to review our pipelines and approve the suggested
consent form.

To summarize, we took all required steps to collect the final dataset before
the beginning of the competition.

###  1.4 Evaluation

From an evaluation perspective, the interactive Architect-Builder
collaboration to build a target structure is very convenient. Computing the
Hamming distance between the built and the target structure can give us a
straightforward measure of the success of the Architect-Builder collaboration
(Kiseleva et al., 2016a, b), which fits well to a large scale evaluation
setup. However, in this competition, we are particularly dedicated to bring a
human in the loop to evaluate the trained agents’ behavior. To do that, we
plan to pair a real player with the pre-trained agent to perform a building
task. In addition to measuring the ability to achieve the target goal
(building a provided structure), we will ask users to score the agent’s
performance.

###  1.5 Tasks and application scenarios

Given the current state of the field, our main research challenge (i.e., _how
to build interactive agents that learn to solve a task while provided with
grounded natural language instructions in a collaborative environment_) might
be too complex to suggest a reasonable end-to-end solution. Therefore, we
split the problem into the following concrete research questions, which
correspond to separate tasks that can be used to study each component
individually before joining all of them into one system (Jones, 1988):

  1. RQ1

_How to teach_?
In other words, what is the best strategy for an Architect when instructing a
Builder agent, such that the concept is reasonably explained? (The suggested
task is presented in Section 1.6).

  2. RQ2

_How to learn?_
That is, what methods should be used to train a Builder that can follow given
instructions from an Architect?
This question can be further split into two sub-questions:

    1. RQ2.1

_How is a ‘silent’ Builder able to learn?_
A silent Builder follows instructions without the ability to ask for any
clarification from the Architect. (The suggested task is presented in Section
1.7).

    2. RQ2.2

_How is an ‘interactive’ Builder able to learn?_
An interactive Builder can ask clarifying questions to the Architect to gain
more information about the task in case of uncertainty. (The suggested task is
presented in Section 1.8).

Each of the suggested tasks is described below (Sections 1.6, 1.7, 1.8) using
the schema proposed by the NeurIPS template: (1) General Setup; (2) Starting
Code; (3) Metrics for Evaluation; (4) Baselines. The participants are free to
choose which task(s) they will solve. Each task will follow the pipeline
presented in Figure 7 and will be evaluated separately.

###  1.6 Task 1: Architect

In this task, our goal is to develop an Architect that can generate
appropriate step instructions based on the observations of environment and the
Builder’s behavior. At the beginning of each task, we give all the details of
the target structure (e.g., types, colors and coordinated of blocks) to the
Architect. The Architect needs to decompose the building process of this
compound structure into a sequence of step instructions that the Builder can
follow. During the interaction, the Architect has to compare the half-finished
structure with the target structure and guide the Builder to complete the
building of remaining components via generated instructions. The step
instructions can be neither too detailed nor too general. In summary, the
Architect is expected to be able to give instructions, correct the Builders’
mistakes and answer their questions by comparing the built structure against
the target structure and by understanding the preceding dialog999We can see
the similarity with newly published work on using human instructions to
improve generalization in RL (Chen et al., 2020)..

####  1.6.1 Task setup

We aim to generate a suitable Architect utterance, given access to 1) the
detailed information of the target structure and 2) the entire game state
context leading up to a certain point in a human-human game at which the human
Architect spoke next. This task can be seen as a multimodal text generation,
where the target structure, the built structure and the dialog history are
input and the next Architect’s utterance is the output. The model developed
for this task can involve both language understanding and visual understanding
depending on the methods for world state representations.

####  1.6.2 Code

The implementation of the baseline can be found using the following repository
https://github.com/prashant-jayan21/minecraft-dialogue-models.

####  1.6.3 Evaluation

##### Automatic evaluation

To evaluate how closely the generated utterances resemble the human
utterances, we adopt standard BLEU scores (Papineni et al., 2002). We also
make use of the modified Precision and Recall of domain-specific keywords
defined in Narayan-Chen et al. (2019). The defined keywords are instrumental
to task success, including colors, spatial relations, and other words that are
highly indicative of dialog actions.

##### Human evaluation

To better evaluate the quality of generated utterances, we will conduct human
evaluations of Architect instructions. We will pair human participants with a
trained architect model. The human builder will interact with the Architect
and complete building tasks by following instructions from the Architect.
After the target structure is successfully built or the interaction exceeds
the maximum task time, the human builder will rate the experience from two
different aspects: language naturalness and instruction accuracy. Each
Architect model will be paired with three human builders and each Architect-
human pair will be assigned three target structures to complete, corresponding
to three different difficulty levels. For fair comparison, the human builders
and target structures will be kept the same for all submitted Architect
models.

####  1.6.4 Baseline

Figure 2: An overview of the full model combining global and local world
representation variants from (Narayan-Chen et al., 2019).

We provide two baselines that are presented in Narayan-Chen et al. (2019). The
first baseline is a vanilla seq2seq model that conditions the next utterance
on the dialog context. A dialog history encoder is applied to obtain the
context representation by compressing the dialog history consisting of
multiple utterances. Speaker-speciﬁc start and end tokens
($\langle\text{A}\rangle$ and $\langle/\text{A}\rangle$ or
$\langle\text{B}\rangle$ and $\langle/\text{B}\rangle$) are added to each
dialog turn. The tokens are fed through a bidirectional Recurrent Neural Net
(RNN) to obtain the final representation. For the decoding step, a decoder RNN
is used to generate the next Architect utterance conditioned on the
representation from the encoding step using beam search.

The second baseline is also a seq2seq model, and it makes use of not only the
text information but also the world state information at each step. The world
state representations take into account the Hamming distance between the
target structure and built structure and it also tells how the target
structure can be constructed successfully given the current state of the built
structure, e.g., which blocks needs to be placed or removed. Narayan-Chen et
al. (2019) consider two variants of block counters that capture the current
state of the built structure: _Global block counters_ and _Local block
counters_. Global block counters are computed over the whole build region
($11\times 11\times 9$) while local block counters encode spatial information
of a cube of blocks ($3\times 3\times 3$) first and concatenate the cube
representations to get the final world state representation. The world state
representations are concatenated to the word embedding vector that is fed to
the decoder at each decoding step as shown in Figure 2.

##### Preliminary results

We report the results on the test set presented in (Narayan-Chen et al., 2019)
in Table 1. The hyper-parameters of architect models have been fine-tuned on
the validation set. By augmenting both the global world state and local world
state, Seq2Seq with global and local information managed to show noticeable
improvements on each of the automatic metrics. The provided baseline
definitely leaves room for improvement. All the architect models will be re-
evaluated after we collect a larger dialog corpus.

| BLEU | Precision/ Recall
---|---|---
Metrics |  BLEU-1 |  BLEU-2 |  BLEU-3 |  BLEU-4 |  all keywords |  colors |  spatial |  dialog
Seq2Seq | 15.3 | 7.8 | 4.5 | 2.8 | 11.8/11.1 | 8.1/17.0 | 9.3/8.6 | 17.9/19.3
+global & local | 15.7 | 8.1 | 4.8 | 2.9 | 13.5/14.4 | 14.9/28.7 | 8.7/8.7 | 18.5/19.9

Table 1: BLEU and term-specific precision and recall scores on the test set,
originally reported in (Narayan-Chen et al., 2019), which were able to
reproduce.

###  1.7 Task 2: Silent Builder

####  1.7.1 Setup

The overall setup for training initial baselines for the silent Builder that
will be used for comparison is presented in Figure 3.

Figure 3: The overall pipeline for suggested baselines for silent builder

####  1.7.2 Interface/Starting Code

We provide a set of Gym environments101010https://blog.openai.com/universe/,
which are based on sub-tasks extracted from the Narayan-Chen et al. (2019)
dataset. An example of such sub-tasks can be building an L-like structure or
building several red blocks, forming a heart. The state-space in which the
agent operates consists of the state of the environment (e.g., RGB image of
POV), the inventory and the annotated sub-goal, defined by the Builder-
Architect dialog from the dataset.

In the current version of the environment, the agent has access to: POV image
$(64,64,3)$, inventory item counts $(6,)$, building zone occupation block
types $(11,11,9)$, full chat embedding $(1024,)$ and agent $(X,Y,Z)$ position
with two angles $(5,)$. A grid specifies the building zone. In each cell, we
have an ID that defines the block type in that position (e.g., $0$ for air,
$1$ for gold block, etc.).

The agent’s action space might consist of all possible actions in Minecraft.
For the current Builder baseline, we offer a discretized version of 18
actions: noop, step forward, step backward, step right, step left, turn up,
turn down, turn left, turn right, jump, attack, place block, choose block type
1-6. We parametarize the environment by a sub-task ID which is extracted from
the dataset and remap its custom block types with default Minecraft block
types (e.g., gold block instead of yellow block, Redstone block instead of red
ones, etc.)111111A representative example of the environment with random agent
behavior: https://youtu.be/hqQ0ubbULWQ. In the future, we will adapt our
environment to be visually consistent with the (Narayan-Chen et al., 2019)
dataset to enable data-driven RL training. In Figure 4 four sub-tasks for the
RL agent are shown.

(a) 5 block L

(b) A “broken heart”

(c) diagonal 3 block Ls

(d) A table

Figure 4: Examples of sub-tasks in the environment

For simplicity of the task, we allow an agent to build a target structure
anywhere inside the building zone by comparing all relative translations and
rotations of two zones. The environment calculates intersections over two
zones and takes the maximum over these intersections. The agent then receives
a reward according to the best match if it has been changed (i.e., non-zero
reward if and only if the max match was changed since the last action). If the
agent gets closer to the target structure in terms of the maximal match, it
receives a reward $+2$. If the structure built by the agent moves further away
in terms of a maximal match from the target one (e.g., the agent removes a
correctly placed block), the agent gets a reward of $-2$. Otherwise, if the
agent places/removes a block outside of a partially built structure (i.e.,
without changing maximal intersections of structures), it receives a reward of
$-1$/$+1$ respectively. If the agent moves outside of the building zone, then
the environment terminates immediately.

##### Architect simulator for training the silent Builder

The correct instruction sequence to achieve the final goal is known, since the
target structures are associated with dialogs. We annotated each dialog’s sub-
goals and stored them in a queue, where each sub-goal corresponds to one
specific step instruction from the Architect. At each step, we pop up a sub-
task (e.g., in about the middle, build a column five tall) and wait until the
agent completes it. If the agent completes this sub-task, we pop up the next
sub-task. We trained a matching model to decide if the current sub-task has
been finished.

####  1.7.3 Evaluation

##### Automatic evaluation

For each subtask of the dataset, the submitted agent will be evaluated on the
environment, with fixed initial and target states. We suggest the following
three metrics by which the solution will be evaluated:

  * •

The reward score $S_{r}$ is based on the average reward received by the agent
in evaluation episodes, which is calculated as follows:

$S_{r}=\frac{1}{N}\sum_{i=1}^{N}g_{i},$ (1)

where N is a number of evaluation episodes, $g_{i}$ is episode reward, defined
by

$g_{i}=\sum_{t=1}^{T}r_{t}.$

  * •

The success rate score $S_{s}$ indicates the number completely solved
subtasks:

$S_{s}=\frac{1}{N}\sum_{i=1}^{N}c_{i},$ (2)

where

$c_{i}=\begin{cases}+1,&\text{if success,}\\\ 0,&\text{otherwise.}\end{cases}$

  * •

Completion rate score $S_{c}$:

$S_{c}=\frac{1}{N}\sum_{i=1}^{N}1-\rho_{i},$ (3)

where $\rho$ is a normalized Hamming distance between target and built
structures.

####  1.7.4 Baselines and Results

The described baselines and preliminary results are obtained for the settings
of the silent Builder.

##### Preliminary Results

With the environment described above, we conducted baseline experiments by
setting one goal per trained agent. Specifically, we started from a
straightforward case where the goal was to build a simple L-shaped structure
of 5 blocks. Since the task was fixed per agent, we did not expect the agent
to extract any meaningful information from the conversation embedding.
Therefore, for the first baseline, we excluded this information from the
agent’s observation.

As baselines, we trained Proximal Policy Optimization (PPO) (Schulman et al.
(2017)) and a variant of Rainbow (Hessel et al. (2017)), PDDQN with experience
replay prioritization, dueling architecture, and multi-step return enabled. We
trained an agent for 400k environment steps since the task is relatively
simple. For the experiments, we used the RLlib framework ((Liang et al.,
2017)) as it provides a solid set of high-performance RL implementations. The
performance plot of the agent is shown in Figure 5.

Figure 5: Performance of the PDDQN and PPO agents on 5 block task (reward
averaged over 3 runs).

##### Planned Baselines

We will provide baselines trained on a full set of sub-tasks using Rainbow and
PPO algorithms, and their pre-trained on human demonstration versions DQfD
((Hester et al., 2017)), ForgER (Skrynnik et al. (2021)) and POfD ((Kang et
al., 2018)). The baselines will be implemented based on the silent builder
scheme.

###  1.8 Task 3: Interactive Builder

Figure 6: The overall pipeline for interactive builder. Builder(QG) and
Architect(QA) are trained offline and their weights are frozen.

The problem of developing an interactive Builder can be viewed as a
combination of an extended Architect and an upgraded silent Builder. Figure 6
shows our view for the solution, which is a work in progress that we plan to
finish before the start date of the competition and use as an internal
baseline. However, we do not intend to share the presented suggested solution
with the participants to avoid biasing their thinking and creativity in
approaching the problem. We also understand that this task is complex, hence
it might not attract as much attention as the previous two.

In terms of starting code, the Gym environment is described in Section 1.7.2,
and the same evaluation metrics for automatic evaluation can be used as
presented in Section 1.7.3.

###  1.9 Tutorial and documentation

Our competition page future_page will contain detailed instructions on: (1)
provided datasets; (2) the developed Gym environment; (3) baselines; (4)
leader-board and other required resources.

##  2 Organizational aspects

###  2.1 Protocol

####  2.1.1 Submission

To orchestrate the submissions’ evaluation, we will use one of the open-source
platforms for organizing machine learning competitions. Currently, we are
exploring options with the following platforms: AICrowd.com, Kaggle.com, and
Codalab.org. While making our final decision about the platform for IGLU, we
will consider the following important aspects:

  * •

Participants should share their source code (preferably a git repository to
see versioning as well) and pre-trained models throughout the competition with
the organizers;

  * •

Automated way to trigger our offline evaluation pipeline;

  * •

Easy way to package the software runtime for the suggested solutions
(preferably through docker); and

  * •

The ability to reliably orchestrate user submissions.

We will make our final decision about the platform after proposal acceptance.

###  2.2 Competition Pipeline

Figure 7 details the general structure of the IGLU competition, which can be
split into two main Stages.

  * •

Stage 1: Training Period, which includes the following main actions.

    * –

_‘Provide Grounded Human-to-Human Dialogs’_ : Participants need to register
for the competition on our website, where they indicate which task(s) they are
planning to solve. Next, we will send them the collected Human-to-Human
dialogs. They can find the code for the available baselines on our website, as
well as documentation on how to use the supplied Gym environment.

    * –

_‘Train Model’_ : This part is done by the participants (they may request
compute resources from Azure, which we provide).

    * –

_‘Submit Model’_ : Participants should decide which models they want to test
and then submit them (with a maximum number of submissions per task per team
of $5$).

    * –

_‘Evaluate Model on Unseen Instances’_ : The organizers do this part. As a
result of this action, the participants will receive the evaluation score for
their method, and the organizers will update the leader board.
As a result of the training period, we will pick the top-3 performing
solutions per task for the next stage.

  * •

Stage 2: Final Human-in-the-loop evaluation, which is fully performed by the
organizers and includes the following actions121212This stage might happen
after the NeurIPS to ensure that participants have enough time to develop
their solutions..

    * –

The organizers will deploy the selected solutions into the Minecraft
environment (using the CraftAssist toolkit).

    * –

The organizers will set up the task and will invite players to interact with
the developed Architect(s) and Builders(s).

    * –

The organizers will report the final results based on the score provided by
humans (the final scoring schema for the human evaluation is under
development, and it varies per task).

The winners will be announced after the results of human-in-the-loop
experiments are collected for each of the tasks.

Figure 7: The general flow of the IGLU competition which consists of two main
stages: (1) Training period; (2) Final human-in-the-loop evaluation.

###  2.3 Rules

The competition aims to develop agents that can converse with humans to
collaborate on a common goal successfully. Furthermore, to ensure that the
competition maximally benefits the research community, we plan to open-source
the final solutions to enable future reproducibility. We introduce the
following rules in an attempt to capture the spirit of the competition, and
any submissions found to be violating these rules may be deemed ineligible for
participation by the organizers.

  * •

Participants may re-use open-source libraries or models with proper
attribution. Conversely, contestants will open source any new models or
libraries developed for the competition.

  * •

Contestants are not limited to the provided dataset, but

    1. 1.

Participants should clearly describe all data; and

    2. 2.

Contestants must release (open-source) all non-public additional data used to
train solution with their submissions, or any public data which has been
aggregated or processed.

#####

To prevent cheating we introduce the following constraints:

  * •

The dataset of ‘Unseen Instances’ used for offline evaluation is never shared
with the participants;

  * •

The participants can submit their solutions for offline evaluation at most $5$
times to prevent any reverse engineering of the results;

  * •

The final evaluation is done with a human-in-the-loop setup, where at least 5
humans are playing with the submitted agents over double-blind trials to
ensure fairness; and

  * •

In suspicious cases, we will look over the submitted code.

subsectionReadiness The following important milestones are already taken care
of at the moment of the proposal submission:

  * •

The data collection tools have been implemented (in collaboration with
Facebook AI Research and Microsoft Research), which are going to be used to
extend the current dataset (Narayan-Chen et al., 2019) at Stage 1 and for the
human-in-the-loop-evaluation at Stage 2;

  * •

The initial Gym environment for training the Builder has been implemented and
tested;

  * •

The IRB process has been started to ensure proper data collection which
involves humans;

  * •

The initial baselines for the Architect and the silent Builder have been
released;

  * •

The work on increasing the number of tasks with different difficulty levels
has been started;

  * •

Organizers started looking into a suitable platform to orchestrate the
submissions and evaluation.

In case the proposal is accepted to the NeurIPS competition track, we do not
envision roadblocks preventing the competition’s execution.

###  2.4 Competition promotion

##### Main mailing lists

We plan to distribute the call to general technical mailing lists, company
internal mailing lists, and institutional mailing lists to promote
participation in the competition. We will directly contact leaders of the
relevant groups in universities and industry labs to promote the competition.

##### Attracting Participants from Underrepresented Groups

We plan to partner with a number of affinity groups to promote the
participation of groups that are traditionally underrepresented at NeurIPS.
Specifically, we will reach out to the following groups:

  * •

Black in AI (BAI);

  * •

LatinX in AI (LXAI);

  * •

Queer in AI;

  * •

Women in Machine Learning (WiML); and

  * •

Women and Underrepresented Minorities in Natural Language Processing (WiNLP);

We plan to reach out to other organizations, such as Data Science Africa, to
work with them to increase the participation of individuals underrepresented
in North American conferences and competitions.

##### Media

To increase general interest and excitement surrounding the competition, we
will work with the media coordinators at Microsoft, Minecraft, Facebook, and
the University of Amsterdam. Moreover, we will use personal media accounts to
promote the IGLU competition.

##### Presentation at different venues

Organizational and advisory board members will include the information about
the IGLU competition in their upcoming presentations.

##  3 Resources

###  3.1 Organizing team

As we noted above, one of the goals of the competition is to bring researchers
from different sub-fields (NLU/G and RL) to work together. To ensure that our
competition is interesting for researchers from all of these directions, the
organizational team consists of researchers from different disciplines, such
as NLP and RL. Moreover, the organizational team is distributed across
different continents and countries, including the Netherlands, Russia, Great-
Britain, and the United States. On top of that, our team is well balanced with
respect to the following aspects:

  * •

a great mix of academic and industrial researchers;

  * •

a good mix of researchers at different stages of their career, including Ph.D.
students, Post Docs and Senior Researchers.

Many of our team members have successfully organized previous challenges,
including ConvAI1, ConvAI2, ClariQ (ConvAI3), DSTC7, DSTC8, MineRL, and Marlo.
For some team members this is their first experience with organizing a
challenge – more experienced colleagues will support them. Moreover, the team
led by Aleksandr Panov and Alexey Skrynnik (see Section 3.2), won the MineRL
competition. Their experience can be used to improve IGLU and make it friendly
for competitors.

###  3.2 Organizers

Our team can be split into two main groups based on expertise: NLU/G and RL
researchers.

##### _The RL sub-team includes the following members:_

##### Alexey Skrynnik

is a Research Associate and Ph.D. student advised by Aleksandr Panov at
Artificial Intelligence Research Institute FRC CSC of Russian Academy of
Sciences. His current research focused on hierarchical reinforcement learning.
Alexey previously won MineRL Diamond Competition 2019 as a leader of the CDS
team. He leads the RL baselines development and is planning to make it a part
of his dissertation, extending RL with NLP.

##### Artem Zholus

is a Research Engineer and Master’s student at the Cognitive Dynamic Systems
laboratory at Moscow Institute of Physics and Technology. His research focuses
on model-based reinforcement learning with application to robotics. Artem has
worked on many machine learning projects, such as Reinforcement Learning for
drug discovery and generative models. In this project, he has developed the
environment for RL agents and trained several RL algorithms as baselines as
part of the CDS team.

##### Aleksandr Panov

is head of the Cognitive Dynamic Systems Laboratory at Moscow Institute of
Physics and Technology. Aleksandr managed the CDS team that won the MineRL
Diamond Competition 2019. Currently, Aleksandr works in model-based RL and the
use of RL methods in the task of navigation in indoor environments.

##### Katja Hofmann

is a Principle Researcher at the Machine Intelligence and Perception group at
Microsoft Research Cambridge. Her research focuses on reinforcement learning
with applications in video games, as she believes that games will drive a
transformation of how people interact with AI technology. She is the research
lead of Project Malmo, which uses the popular game Minecraft as an
experimentation platform for developing intelligent technology, and has
previously co-organized two competitions based on the Malmo platform. Her
long-term goal is to develop AI systems that learn to collaborate with people,
to empower their users and help solve complex real-world problems.

##### _The NLU/G sub-team consists of the following members:_

##### Ziming Li

is a Postdoc at University of Amsterdam. His main research interest is
developing advanced dialogue systems, including dialogue policy learning and
evaluation. He is also interested in the fields of conversational search and
reinforcement learning.

##### Julia Kiseleva

is a Senior Researcher in Microsoft Research. Her research interests are in
natural language processing, information retrieval, and machine learning, with
a strong focus on continuous learning from user feedback and interactions.
Julia has been involved in many initiatives such as a serious of Search-
Oriented Conversational AI (SCAI) workshop, which was continuously organized
from 2015 - 2020. She also co-organized the recent challenge ConvAI3:
Clarifying Questions for Open-Domain Dialogue Systems (ClariQ) at EMNLP2020.

##### Maartje ter Hoeve

is a PhD candidate at the University of Amsterdam. Her main research interest
is how we can learn from humans and their cognition to improve our NLP and IR
systems. She has a background in both Linguistics and Artificial Intelligence.

##### Mikhail Burtsev

is a Head of Neural Networks & Deep Learning lab at MIPT, Moscow. His current
research interests: application of neural nets and reinforcement learning in
the NLP domain. He is a faculty advisor of MIPT team participating in Alexa
Prize Challenges 3 and 4. He proposed and organised Conversational AI
Challenges: ConvAI 1 (NIPS 2017), ConvAI 2 (NeurIPS 2018), ConvAI 3 (EMNLP
2020).

##### Mohammad Aliannejadi

is a post-doctoral researcher at the University of Amsterdam (The
Netherlands). His research interests include single- and mixed-initiative
conversational information access and recommender systems. Previously, he
completed his Ph.D. at Università della Svizzera italiana (Switzerland), where
he worked on novel approaches of information access in conversations. He co-
organized the Conversational AI Challenge, ConvAI 3 (EMNLP 2020).

##### Shrestha Mohanty

is a Machine Learning Engineer at Microsoft. She primarily works in areas of
machine learning, deep learning, and natural language processing, including
topics such as personalization, dialogue systems and multilingual language
models. She is also interested in and has worked on problems at the
intersection of machine learning and healthcare. Prior to Microsoft, she
completed her master’s in information science from University of California,
Berkeley.

##### Arthur Szlam

is a Research Scientist at Facebook AI Research. He works on connecting
perception, memory, language, and action in artificial agents. Prior to
joining Facebook, he was on the faculty of the City College of New York
(CCNY), and was the recipient of a Sloan research fellowship. He has been a
co-organizer of previous ConvAI challenges.

##### Kavya Srinet

is a Research Engineering Manager at Facebook AI Research working towards a
long-term goal of developing interactive assistants. She works on building
assistants that can learn from interactions with humans. Prior to FAIR, she
was a Machine Learning Engineer at the AI Lab at Baidu Research, where she
worked on speech and NLP problems. Kavya was at the Allen Institute for
Artificial Intelligence for a summer before that working on learning to rank
for Semantic Scholar. Kavya did her graduate school from Language Technology
Institute at Carnegie Mellon University, where she worked on areas of machine
translation, question answering and learning to rank for graphs and knowledge
bases.

##### Yuxuan Sun

is a Research Engineer in Facebook AI Research (FAIR). His research interests
lie in natural language processing, neural symbolic learning and reasoning,
and human-in-the-loop learning for embodied agents.

##### Michel Galley

is a Principal Researcher at Microsoft Research. His research interests are in
the areas of natural language processing and machine learning, with a focus on
conversational AI, neural generation, statistical machine translation, and
summarization. He co-authored more than 70 scientific papers, many of which
appeared at top NLP, AI, and ML conferences. His organizational experience
includes service as sponsorship chair at EMNLP-16, workshop chair at NAACL-21,
challenge co-organizer at the Dialog System Technology Challenge (DSTC7)
workshop at AAAI-19, and workshop chair for DSTC8 (at AAAI-20). He has been
part of the DSTC challenge steering committee since 2020. He also served as
area chair at top NLP conferences (ACL, NAACL, EMNLP, CoNLL), as action editor
for the TACL journal (2020-), and will serve as area chair at NeurIPS-21.

##### Ahmed Awadallah

is a Senior Principal Research Manager at Microsoft Research where he leads
the Language & Information Technologies Group. His research has sought to
understand how people interact with information and to enable machines to
understand and communicate in natural language (NL) and assist with task
completion. More recently, his research has focused learning form limited
annotated data (e.g., few-shot learning and transfer learning) and from user
interactions (e.g. interactive semantic parsing) . Ahmed’s contributions to
NLP and IR have recently been recognized with the 2020 Karen Spärck Jones
Award from the British Computer Society. Ahmed regularly serves as (senior)
committee, area chair, guest editor and editorial board member at many major
NLP and IR conferences and journal.

###  3.3 Advisory Board

##### Julia Hockenmaier

is an associate professor at the University of Illinois at Urbana-Champaign.
She has received a CAREER award for her work on CCG-based grammar induction
and an IJCAI-JAIR Best Paper Prize for her work on image description. She has
served as member and chair of the NAACL board, president of SIGNLL, and as
program chair of CoNLL 2013 and EMNLP 2018.

##### Bill Dolan

is Partner Researcher Manager at Microsoft Research, where he manages the
Natural Language Processing group. He has worked on a wide variety of
problems, including the acquisition of structured common-sense knowledge from
free text, paraphrasing, text rewriting to improve grammar and style, and most
recently on data-driven, grounded approaches to handling dialog. He has helped
organize a number of research community events over the years, including the
RTE challenges and the “Understanding Situated Language in Everyday Life”
summer research institute with the University of Washington, as well as
running the Microsoft Research graduate fellowship program from 2014-2017.

##### Ryen W. White

is a Partner Research Area Manager at Microsoft Research, where he leads the
Language and Intelligent Assistance research area. He led the applied science
organization at Microsoft Cortana, and he was chief scientist at Microsoft
Health. Ryen has authored hundreds of publications in areas such as
information retrieval, computational health, and human-computer interaction -
including many that received awards. He was program chair for SIGIR 2017 and
The Web Conference 2019. Ryen is editor-in-chief of ACM Transactions on the
Web.

##### Maarten de Rijke

is Distinguished University Professor of Artificial Intelligence and
Information Retrieval at the University of Amsterdam. He is VP Personalization
and Relevance and Senior Research Fellow at Ahold Delhaize. His research
strives to build intelligent technology to connect people to information. His
team pushes the frontiers of search engines, recommender systems and
conversational assistants. They also investigate the influence of the
technology they develop on society. De Rijke is the director of the Innovation
Center for Artificial Intelligence.

##### Sharada Mohanty

is the CEO and Co-founder of AIcrowd, a community of AI researchers built
around a platform encouraging open and reproducible artificial intelligence
research. He was the co-organizer of many large-scale machine learning
competitions, such as NeurIPS 2017: Learning to Run Challenge, NeurIPS 2018:
AI for Prosthetics Challenge, NeurIPS 2018: Adversarial Vision Challenge,
NeurIPS 2019: MineRL Competition, NeurIPS 2019: Disentanglement Challenge,
NeurIPS 2019: REAL Robots Challenge, NeurIPS 2020: Flatland Competition,
NeurIPS 2020: Procgen Competition. He is extremely passionate about benchmarks
and building communities. During his Ph.D. at EPFL, he worked on numerous
problems at the intersection of AI and health, with a strong interest in
reinforcement learning. In his previous roles, he has worked at the
Theoretical Physics department at CERN on crowdsourcing compute for PYTHIA
powered Monte-Carlo simulations; he has had a brief stint at UNOSAT building
GeoTag-X—a platform for crowdsourcing analysis of media coming out of
disasters to assist in disaster relief efforts. In his current role, he
focuses on building better engineering tools for AI researchers and making
research in AI accessible to a larger community of engineers.

###  3.4 Partners

##### Microsoft Research

has a mission to support research in computer science and software
engineering. Microsoft Research in collaboration with Minecraft has developed
Malmo, which is a platform for artificial intelligence experimentation
(Johnson et al., 2016), that has stimulated rapid research in the area (Shu et
al., 2017). Microsoft will support this competition by providing:

  * •

cloud computing through Azure for participants to train their models;

  * •

continued development support of the distributed data collection tool to reach
out to Minecraft players (especially valuable for Task 2);

  * •

a budget and proper IRB process for consent for the data collection with
human-to-human players; and

  * •

support for the final evaluation of the developed agents by adding the human-
in-the-loop setup.

##### Facebook AI Research

has a long-standing goal to develop interactive assistants that can learn from
human interactions. In this respect, FAIR invested into the development of the
CraftAssist
framework131313https://github.com/facebookresearch/droidlet/tree/main/craftassist
which has matured into
droidlet141414https://github.com/facebookresearch/droidlet \- that makes it
easier to utilize crowd-sourcing platforms to collect the human-to-human and
human-assistant interactions at scale. FAIR will support us with

  * •

tooling for data collection in crowd-sourcing platforms such as
MTurk151515https://www.mturk.com/;

  * •

extending data collection by enriching our crowd-sourcing infrastructure
(especially valuable for Task 1 where we need to have rich language
interactions).

In general, our partners are dedicated to sponsoring a new dataset collection
which will potentially open up numerous research directions, and arranging a
final human-in-the-loop evaluation.

###  3.5 Resources provided by organizers

###  3.6 Computational Resources

Microsoft has a mission to democratize access to AI will provide computational
resources to the participants. The concrete amount is the subject of the
current discussion, which will clear upon acceptance and a further number of
participants requiring resources.

###  3.7 Providing mentorship for the teams

Our organization and advisory boards are really passionate about bringing
NLU/NLG and RL communities. Therefore, we are dedicated to provide mentorship
to a number of teams. We will give preference to underrepresented groups and
young researchers.

###  3.8 Prizes

We are currently discussing this with our partners. Moreover, we reached out
to potential sponsors, such as Amazon and Google, who are potentially
interested to help us. We can get more details and traction upon acceptance.

###  3.9 Support requested

It would be great if NeurIPS could host a workshop (or an event) where our
winning participants can share their results. As far as we understood from the
call, there is no session planned for a competition track yet. We would like
to ask for a number of conference registrations reserved for our winning
participants.

## References

  * Adiwardana et al. [2020] Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al.  Towards a human-like open-domain chatbot.  _arXiv preprint arXiv:2001.09977_ , 2020.
  * Aliannejadi et al. [2020] Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev.  Convai3: Generating clarifying questions for open-domain dialogue systems (clariq).  2020\.
  * Aliannejadi et al. [2021] Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeffrey Dalton, and Mikhail Burtsev.  Building and evaluating open-domain dialogue corpora with clarifying questions.  _arXiv preprint arXiv:2109.05794_ , 2021.
  * An [1988] Meltzoff An.  Imitation, Objects, Tools, and the Rudiments of Language in Human Ontogeny, February 1988.  URL https://pubmed.ncbi.nlm.nih.gov/23997403/.  ISSN: 0393-9375 Issue: 1-2 Publisher: Hum Evol Volume: 3.
  * Arguello et al. [2018] Jaime Arguello, Filip Radlinski, Hideo Joho, Damiano Spina, and Julia Kiseleva.  Second international workshop on conversational approaches to information retrieval (cair’18) workshop at sigir 2018.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 1435–1437, 2018.
  * [6] Dan Ayoub.  Unleashing the power of AI for education.  URL https://www.technologyreview.com/2020/03/04/905535/unleashing-the-power-of-ai-for-education/.
  * Bellemare et al. [2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47:253–279, 2013.
  * Berant et al. [2013] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang.  Semantic parsing on freebase from question-answer pairs.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1533–1544, 2013.
  * Bisk et al. [2016] Yonatan Bisk, Deniz Yuret, and Daniel Marcu.  Natural language communication with robots.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 751–761, San Diego, California, June 2016. Association for Computational Linguistics.  doi: 10.18653/v1/N16-1089.  URL https://www.aclweb.org/anthology/N16-1089.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  2020\.
  * Budzianowski et al. [2018] Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic.  Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, 2018.
  * Burtsev and Logacheva [2020] Mikhail Burtsev and Varvara Logacheva.  Conversational intelligence challenge: Accelerating research with crowd science and open source.  _AI Magazine_ , 41(3):18–27, 2020.
  * Chen et al. [2020] Valerie Chen, Abhinav Gupta, and Kenneth Marino.  Ask your humans: Using human instructions to improve generalization in reinforcement learning.  _arXiv preprint arXiv:2011.00517_ , 2020.
  * Chevalier-Boisvert et al. [2019] Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio.  Babyai: First steps towards grounded language learning with a human in the loop.  In _International Conference on Learning Representations_ , 2019.
  * Chuklin et al. [2018] Aleksandr Chuklin, Jeff Dalton, Julia Kiseleva, Alexey Borisov, and Mikhail Burtsev.  Proceedings of the 2018 emnlp workshop scai: The 2nd international workshop on search-oriented conversational ai.  In _Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_ , 2018.
  * Cideron et al. [2019] Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin.  Self-educated language agent with hindsight experience replay for instruction following.  _arXiv preprint arXiv:1910.09451_ , 2019.
  * Clark et al. [2020] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_ , 2020.
  * Codd [1974] Edgar F Codd.  _Seven steps to rendezvous with the casual user_.  IBM Corporation, 1974.
  * Copestake and Jones [1990] Ann Copestake and Karen Sparck Jones.  Natural language interfaces to databases.  1990\.
  * Council [1999] National Research Council.  _How People Learn: Brain, Mind, Experience, and School: Expanded Edition_.  August 1999.  ISBN 978-0-309-07036-2.  doi: 10.17226/9853.  URL https://www.nap.edu/catalog/9853/how-people-learn-brain-mind-experience-and-school-expanded-edition.
  * Crook et al. [2019] Paul A Crook, Shivani Poddar, Ankita De, Semir Shafi, David Whitney, Alborz Geramifard, and Rajen Subba.  Simmc: Situated interactive multi-modal conversational data collection and evaluation platform.  _arXiv preprint arXiv:1911.02690_ , 2019.
  * Csibra and Gergely [2009] Gergely Csibra and György Gergely.  Natural pedagogy.  _Trends in cognitive sciences_ , 13(4):148–153, 2009.
  * Dalton et al. [2020] Jeff Dalton, Aleksandr Chuklin, Julia Kiseleva, and Mikhail Burtsev, editors.  _Proceedings of the 5th International Workshop on Search-Oriented Conversational AI (SCAI)_ , Online, November 2020. Association for Computational Linguistics.  URL https://aclanthology.org/2020.scai-1.0.
  * Desai et al. [2016] Aditya Desai, Sumit Gulwani, Vineet Hingorani, Nidhi Jain, Amey Karkare, Mark Marron, Subhajit Roy, et al.  Program synthesis using natural language.  In _Proceedings of the 38th International Conference on Software Engineering_ , pages 345–356. ACM, 2016.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_ , 2018.
  * Dinan et al. [2020] Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al.  The second conversational intelligence challenge (convai2).  In _The NeurIPS’18 Competition_ , pages 187–208. Springer, Cham, 2020\.
  * Elgohary et al. [2020] Ahmed Elgohary, Saghar Hosseini, and Ahmed Hassan Awadallah.  Speak to your parser: Interactive text-to-SQL with natural language feedback.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2065–2077, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.187.  URL https://www.aclweb.org/anthology/2020.acl-main.187.
  * Fast et al. [2018] Ethan Fast, Binbin Chen, Julia Mendelsohn, Jonathan Bassen, and Michael S Bernstein.  Iris: A conversational agent for complex tasks.  In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ , page 473. ACM, 2018.
  * Gray et al. [2019] Jonathan Gray, Kavya Srinet, Yacine Jernite, Haonan Yu, Zhuoyuan Chen, Demi Guo, Siddharth Goyal, C. Lawrence Zitnick, and Arthur Szlam.  CraftAssist: A Framework for Dialogue-enabled Interactive Agents.  _arXiv:1907.08584 [cs]_ , July 2019.  URL http://arxiv.org/abs/1907.08584.  arXiv: 1907.08584.
  * Guss et al. [2019] William H. Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al.  The MineRL competition on sample efficient reinforcement learning using human priors.  _NeurIPS Competition Track_ , 2019.
  * Hauff et al. [2021] Claudia Hauff, Julia Kiseleva, Mark Sanderson, Hamed Zamani, and Yongfeng Zhang.  Conversational search and recommendation: Introduction to the special issue.  _ACM Trans. Inf. Syst._ , 39(4), August 2021.  ISSN 1046-8188.  doi: 10.1145/3465272.  URL https://doi.org/10.1145/3465272.
  * Hendrix et al. [1978] Gary G Hendrix, Earl D Sacerdoti, Daniel Sagalowicz, and Jonathan Slocum.  Developing a natural language interface to complex data.  _ACM Transactions on Database Systems (TODS)_ , 3(2):105–147, 1978.
  * Hessel et al. [2017] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver.  Rainbow: Combining improvements in deep reinforcement learning.  _CoRR_ , abs/1710.02298, 2017.  URL http://arxiv.org/abs/1710.02298.
  * Hessel et al. [2018] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.  Rainbow: Combining improvements in deep reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Hester et al. [2017] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys.  Deep q-learning from demonstrations, 2017.
  * Hori et al. [2018] Chiori Hori, Anoop Cherian, Tim K Marks, and Florian Metze.  Audio visual scene-aware dialog track in dstc8.  _DSTC Track Proposal_ , 2018.
  * Hu et al. [2019] Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis.  Hierarchical decision making by generating and following natural language instructions.  _arXiv preprint arXiv:1906.00744_ , 2019.
  * Jayannavar et al. [2020] Prashant Jayannavar, Anjali Narayan-Chen, and Julia Hockenmaier.  Learning to execute instructions in a Minecraft dialogue.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2589–2602, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.232.  URL https://www.aclweb.org/anthology/2020.acl-main.232.
  * Johnson et al. [2016] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell.  The malmo platform for artificial intelligence experimentation.  In _IJCAI_ , pages 4246–4247. Citeseer, 2016.
  * Jones [1988] K Sparck Jones.  A look back and a look forward.  In _Proceedings of the 11th annual international ACM SIGIR conference on Research and Development in Information Retrieval_ , pages 13–29, 1988.
  * Kang et al. [2018] Bingyi Kang, Zequn Jie, and Jiashi Feng.  Policy optimization with demonstrations.  In _International Conference on Machine Learning_ , pages 2469–2478. PMLR, 2018.
  * Kim et al. [2019] Seokhwan Kim, Michel Galley, Chulaka Gunasekara, Sungjin Lee, Adam Atkinson, Baolin Peng, Hannes Schulz, Jianfeng Gao, Jinchao Li, Mahmoud Adada, et al.  The eighth dialog system technology challenge.  _arXiv preprint arXiv:1911.06394_ , 2019.
  * Kiseleva et al. [2016a] Julia Kiseleva, Kyle Williams, Ahmed Hassan Awadallah, Aidan C Crook, Imed Zitouni, and Tasos Anastasakos.  Predicting user satisfaction with intelligent assistants.  In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_ , pages 45–54, 2016a.
  * Kiseleva et al. [2016b] Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah, Aidan C Crook, Imed Zitouni, and Tasos Anastasakos.  Understanding user satisfaction with intelligent assistants.  In _Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval_ , pages 121–130, 2016b.
  * Köhn et al. [2020] Arne Köhn, Julia Wichlacz, Christine Schäfer, Alvaro Torralba, Jörg Hoffmann, and Alexander Koller.  Mc-saar-instruct: a platform for minecraft instruction giving agents.  In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 53–56, 2020.
  * Lee et al. [2019] Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Xiang Li, Yaoqin Zhang, Zheng Zhang, Jinchao Li, Baolin Peng, Xiujun Li, Minlie Huang, et al.  Convlab: Multi-domain end-to-end dialog system platform.  _arXiv preprint arXiv:1904.08637_ , 2019.
  * Levinson [2019] Stephen C Levinson.  Tom m. mitchell, simon garrod, john e. laird, stephen c. levinson, and kenneth r. koedinger.  _Interactive Task Learning: Humans, Robots, and Agents Acquiring New Tasks through Natural Interactions_ , 26:9, 2019.
  * Li et al. [2020] Toby Jia-Jun Li, Tom Mitchell, and Brad Myers.  Interactive task learning from GUI-grounded natural language instructions and demonstrations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , July 2020.
  * Liang et al. [2017] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Joseph Gonzalez, Ken Goldberg, and Ion Stoica.  Ray rllib: A composable and scalable reinforcement learning library.  _CoRR_ , abs/1712.09381, 2017.  URL http://arxiv.org/abs/1712.09381.
  * Liu and Lane [2017] Bing Liu and Ian Lane.  Iterative policy learning in end-to-end trainable task-oriented neural dialog models.  In _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , pages 482–489. IEEE, 2017.
  * Liu and Lane [2018] Bing Liu and Ian Lane.  Adversarial learning of task-oriented neural dialog models.  In _Proceedings of the SIGDIAL 2018 Conference_ , pages 350–359, 2018\.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692, 2019.  URL http://arxiv.org/abs/1907.11692.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529, 2015.
  * Mnih et al. [2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , pages 1928–1937. PMLR, 2016.
  * Narayan-Chen et al. [2017] Anjali Narayan-Chen, Colin Graber, Mayukh Das, Md Rakibul Islam, Soham Dan, Sriraam Natarajan, Janardhan Rao Doppa, Julia Hockenmaier, Martha Palmer, and Dan Roth.  Towards problem solving agents that communicate and learn.  In _Proceedings of the First Workshop on Language Grounding for Robotics_ , pages 95–103, 2017.
  * Narayan-Chen et al. [2019] Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier.  Collaborative dialogue in minecraft.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5405–5415, 2019.
  * Ogawa et al. [2020] Haruna Ogawa, Hitoshi Nishikawa, Takenobu Tokunaga, and Hikaru Yokono.  Gamification platform for collecting task-oriented dialogue data.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 7084–7093, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://www.aclweb.org/anthology/2020.lrec-1.876.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002.
  * Perez-Liebana et al. [2019] Diego Perez-Liebana, Katja Hofmann, Sharada Prasanna Mohanty, Noburu Kuno, Andre Kramer, Sam Devlin, Raluca D Gaina, and Daniel Ionita.  The multi-agent reinforcement learning in malm$\backslash$” o (marl$\backslash$” o) competition.  _arXiv preprint arXiv:1901.08129_ , 2019.
  * Rao and Daumé III [2018] Sudha Rao and Hal Daumé III.  Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information.  _arXiv preprint arXiv:1805.04655_ , 2018.
  * Roller et al. [2020] Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al.  Recipes for building an open-domain chatbot.  _arXiv preprint arXiv:2004.13637_ , 2020.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _CoRR_ , abs/1707.06347, 2017.  URL http://arxiv.org/abs/1707.06347.
  * Shu et al. [2017] Tianmin Shu, Caiming Xiong, and Richard Socher.  Hierarchical and interpretable skill acquisition in multi-task reinforcement learning.  _arXiv preprint arXiv:1712.07294_ , 2017.
  * Silver et al. [2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359, 2017.
  * Skrynnik et al. [2021] Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, Kirill Aksenov, Vasilii Davydov, and Aleksandr I Panov.  Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations.  _Knowledge-Based Systems_ , 218:106844, 2021.
  * Spina et al. [2019] Damiano Spina, Jaime Arguello, Hideo Joho, Julia Kiseleva, and Filip Radlinski.  Cair’18: second international workshop on conversational approaches to information retrieval at sigir 2018.  In _ACM SIGIR Forum_ , volume 52, pages 111–116. ACM New York, NY, USA, 2019.
  * Srinet et al. [2020] Kavya Srinet, Yacine Jernite, Jonathan Gray, and Arthur Szlam.  CraftAssist instruction parsing: Semantic parsing for a voxel-world assistant.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4693–4714, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.427.  URL https://www.aclweb.org/anthology/2020.acl-main.427.
  * Stuckler et al. [2012] Jorg Stuckler, Dirk Holz, and Sven Behnke.  Robocup@ home: Demonstrating everyday manipulation skills in robocup@ home.  _IEEE Robotics & Automation Magazine_, 19(2):34–42, 2012.
  * Su et al. [2017] Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, Patrick Pantel, Michael Gamon, and Mark Encarnacion.  Building natural language interfaces to web apis.  In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_ , pages 177–186. ACM, 2017.
  * Szlam et al. [2019] Arthur Szlam, Jonathan Gray, Kavya Srinet, Yacine Jernite, Armand Joulin, Gabriel Synnaeve, Douwe Kiela, Haonan Yu, Zhuoyuan Chen, Siddharth Goyal, Demi Guo, Danielle Rothermel, C. Lawrence Zitnick, and Jason Weston.  Why Build an Assistant in Minecraft?  _arXiv:1907.09273 [cs]_ , July 2019.  URL http://arxiv.org/abs/1907.09273.  arXiv: 1907.09273.
  * Tellex et al. [2011] Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy.  Understanding natural language commands for robotic navigation and mobile manipulation.  In _Twenty-Fifth AAAI Conference on Artificial Intelligence_ , 2011\.
  * Thomaz et al. [2019] Andrea L Thomaz, Elena Lieven, Maya Cakmak, Joyce Y Chai, Simon Garrod, Wayne D Gray, Stephen C Levinson, Ana Paiva, and Nele Russwinkel.  Interaction for task instruction and learning.  In _Interactive task learning: Humans, robots, and agents acquiring new tasks through natural interactions_ , pages 91–110. MIT Press, 2019\.
  * Thompson et al. [1993] Henry S. Thompson, Anne Anderson, Ellen Gurman Bard, Gwyneth Doherty-Sneddon, Alison Newlands, and Cathy Sotillo.  The HCRC Map Task corpus: natural dialogue for speech recognition.  In _Proceedings of the workshop on Human Language Technology - HLT ’93_ , page 25, Princeton, New Jersey, 1993. Association for Computational Linguistics.  ISBN 978-1-55860-324-0.  doi: 10.3115/1075671.1075677.  URL http://portal.acm.org/citation.cfm?doid=1075671.1075677.
  * Urbanek et al. [2019] Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston.  Learning to speak and act in a fantasy text adventure game.  _arXiv preprint arXiv:1903.03094_ , 2019.
  * Winograd [1971] Terry Winograd.  Procedures as a representation for data in a computer program for understanding natural language.  Technical report, MASSACHUSETTS INST OF TECH CAMBRIDGE PROJECT MAC, 1971\.
  * Woods et al. [1972] W. A. Woods, Ronald M Kaplan, and Bonnie L. Webber.  The lunar sciences natural language information system: Final report.  _BBN Report 2378_ , 1972.
  * Yao et al. [2019] Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih.  Model-based interactive semantic parsing: A unified framework and a text-to-SQL case study.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5447–5458, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1547.  URL https://www.aclweb.org/anthology/D19-1547.
  * Yao et al. [2020] Ziyu Yao, Yiqi Tang, Wen-tau Yih, Huan Sun, and Yu Su.  An imitation game for learning semantic parsers from user interaction.  2020\.
  * Young et al. [2013] Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams.  Pomdp-based statistical spoken dialog systems: A review.  _Proceedings of the IEEE_ , 101(5):1160–1179, 2013.
  * Yuan et al. [2019] Xingdi Yuan, Marc-Alexandre Côté, Jie Fu, Zhouhan Lin, Christopher Pal, Yoshua Bengio, and Adam Trischler.  Interactive language learning by question answering.  _arXiv preprint arXiv:1908.10909_ , 2019.
  * Zhang et al. [2021] Yi Zhang, Sujay Kumar Jauhar, Julia Kiseleva, Ryen White, and Dan Roth.  Learning to decompose and organize complex tasks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2726–2735, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.217.  URL https://aclanthology.org/2021.naacl-main.217.
  * Zhang et al. [2019] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan.  Dialogpt: Large-scale generative pre-training for conversational response generation.  _arXiv preprint arXiv:1911.00536_ , 2019.

  *[RNN]: Recurrent Neural Net


# Please, no more scientific journals!
The strategy of the scientific publication system

Miguel A. Fortuna
<EMAIL_ADDRESS>

Department of Evolutionary Biology and Environmental Studies
University of Zurich, Zurich (Switzerland)

Abstract.

In the same way ecosystems tend to increase maturity by decreasing the flow of
energy per unit biomass, we should move towards a more mature science by
publishing less but high-quality papers and getting away from joining large
teams in small roles. That is, we should decrease our scientific productivity
for good.

The number of scientific journals listed in the Journal of Citation Reports
has increased during the last decade from $6,443$ to $8,860$ (Table 1), and
hence, the world’s scientific production (i.e., number of papers published by
the scientific community per year) has grown from $905,349$ to $1,398,003$ in
that period (Fig. 1a). Journals act like fertilizers for the soil, opening up
new opportunities for scientists to publish the results of their research.
Consolidated scientists can increase their scientific productivity (i.e.,
number of papers published per scientist and year). On the other hand, novel
scientists—who did not publish before because of the fierce competition for
space in journals—can enter into the scientific publication system.
Disentangling the detrimental consequences of the never-ending increase of the
number of journals (i.e., higher scientific productivity) from its
advantageous effects (i.e., recruiting novel scientists) is of paramount
importance to support or discourage the proliferation of new scientific
journals.

Since the number of authors has increased (from $3,052,068$ to $5,318,208$)
faster than the number of papers published during the last decade ($75\%$ and
$50\%$, respectively), adding more journals into the publication system seems
to have reduced scientific productivity. Note, however, that using the ratio
$\frac{\text{\\#papers}}{\text{\\#authors}}$ (i.e,. scientific production over
the number of authors) as a proxy of scientific productivity (Fig. 1b) assumes
single-authored papers. This negative linear relationship still holds for
multiple-author papers if we assume that the average number of authors signing
a research paper has not changed over time. Yet, the only way to keep
scientific productivity constant over time when the number of authors
increases faster than the number of papers is by increasing the number of
authors signing a paper. This suggests that, even when the number of papers
grows at a lower pace than the number of authors, the higher the frequency of
papers signed by many authors is, the higher might be scientific productivity.

The relative frequency of papers signed by five or more authors has increased
during the last decade from $40\%$ to $53\%$ (Fig. 1c). Single-author papers,
papers signed by two, three and four authors—but not by five or more
authors—have decreased over time (shown by negative and statistically
significant correlations; see Table 2). This resulting increase in the average
number of authors per paper over time must be incorporated in any measure of
scientific productivity.

One way to measure scientific productivity per year taking into account the
increase of the frequency of papers signed by many authors is the following:

$\text{scientific
productivity}=\frac{\sum\limits_{i=1}^{20}\Big{(}\text{\\#papers}_{(i)}*\hskip
2.84544pti\Big{)}}{\text{\\#authors}},\vspace{-0.05cm}$

where $\text{\\#papers}_{(i)}$ is the number of papers signed by $i$ authors
and #authors is the total number of authors of all the papers published that
year. Note that we did not consider papers signed by more than 20 authors
because the percentage of those papers was lower than $0.5\%$. This measure
shows that scientific productivity has increased over time ($5\%$ during the
last decade; Fig. 1d) as a consequence of the proliferation of new journals.

While it is true that having more journals has allowed novel scientists to
enter into the scientific publication system, most of them are just part of
multi-authored papers. In the same way ecosystems tend to increase maturity by
decreasing the flow of energy per unit biomass (Margalef, The American
Naturalist, 97 (1963): 357-374), we need to move towards a more mature science
where we should publish less but high-quality papers (i.e., trend towards
decreasing scientific productivity). It’s time to face the conflict between
the strategies of science and publishers.

Figure 1. The strategy of the scientific publication system. a) Trends in the
number of papers (red) and authors (blue) over the last decade. b) Scientific
productivity over time measured as the ratio between the number of papers
published per year and the number of scientists authored them. c) Change in
the frequency of papers signed by four or less scientists (red) and more than
four scientists (blue). d) Scientific productivity over time measured after
taking into account the increase in the number of authors per paper.

## Data.

Table 1. Summary descriptors of the scientific publication system in the last
decade. Number of journals included in the Science Citation Index Expanded,
number of scientific articles, number of distinct authors signing those
articles, articles signed by less than five authors, and articles signed by
five or more authors.

year |  #journals |  #articles |  #authors |  < 5 authors |  $\geq$ 5 authors
---|---|---|---|---|---
2008 |  6443 |  905349 |  3052068 |  59.45% |  40.55%
2009 |  7175 |  967999 |  3274946 |  58.29% |  41.71%
2010 |  7848 |  1033298 |  3545223 |  57.01% |  42.99%
2011 |  8095 |  1102155 |  3824334 |  55.45% |  44.55%
2012 |  8253 |  1158923 |  4095396 |  53.91% |  46.09%
2013 |  8335 |  1228952 |  4362849 |  52.31% |  47.69%
2014 |  8490 |  1270887 |  4584087 |  50.83% |  49.17%
2015 |  8660 |  1326996 |  4854865 |  49.34% |  50.66%
2016 |  8734 |  1370378 |  5117270 |  48.18% |  51.82%
2017 |  8860 |  1398003 |  5318208 |  47.19% |  52.81%

Table 2. Number of multi-authored papers published in the last decade. Number
of scientific articles signed by no more than $i=1,2,...20$ authors and
published in journals included in the Science Citation Index Expanded.

# articles by: |  2008 |  2009 |  2010 |  2011 |  2012 |  2013 |  2014 |  2015 |  2016 |  2017
---|---|---|---|---|---|---|---|---|---|---
1 author |  73808 |  74699 |  75568 |  74636 |  72706 |  71087 |  68511 |  66815 |  65178 |  62498
$\leq$ 2 authors |  224878 |  231238 |  237675 |  240647 |  240413 |  240937 |  237165 |  236058 |  232985 |  228850
$\leq$ 3 authors |  391368 |  407505 |  422235 |  434066 |  439167 |  447095 |  444570 |  447216 |  447805 |  443948
$\leq$ 4 authors |  538245 |  564238 |  589092 |  611093 |  624712 |  642833 |  646027 |  654702 |  660279 |  659724
$\leq$ 5 authors |  652270 |  687901 |  722446 |  755205 |  776687 |  805935 |  815932 |  834279 |  845160 |  848867
$\leq$ 6 authors |  736860 |  780314 |  822926 |  864534 |  895138 |  934300 |  951937 |  978605 |  995837 |  1003703
$\leq$ 7 authors |  792343 |  841197 |  889773 |  939086 |  976128 |  1023502 |  1047355 |  1081325 |  1103269 |  1116058
$\leq$ 8 authors |  829702 |  882154 |  935310 |  990260 |  1032466 |  1086613 |  1115059 |  1154773 |  1181896 |  1198092
$\leq$ 9 authors |  853888 |  909020 |  965507 |  1024224 |  1070302 |  1128542 |  1161147 |  1204739 |  1236128 |  1254763
$\leq$ 10 authors |  870687 |  927821 |  986747 |  1048033 |  1097294 |  1158671 |  1193574 |  1240850 |  1274854 |  1296180
$\leq$ 11 authors |  881065 |  939696 |  1000216 |  1063458 |  1114245 |  1177799 |  1214514 |  1263871 |  1299822 |  1322448
$\leq$ 12 authors |  888280 |  947840 |  1009344 |  1074136 |  1126327 |  1191349 |  1229326 |  1280264 |  1317667 |  1341144
$\leq$ 13 authors |  892744 |  952939 |  1015247 |  1080910 |  1134327 |  1200507 |  1239372 |  1291214 |  1329877 |  1354124
$\leq$ 14 authors |  895749 |  956465 |  1019339 |  1085814 |  1139990 |  1206906 |  1246378 |  1299073 |  1338727 |  1363672
$\leq$ 15 authors |  897905 |  959041 |  1022337 |  1089235 |  1144095 |  1211520 |  1251474 |  1304948 |  1345063 |  1370543
$\leq$ 16 authors |  899352 |  960852 |  1024407 |  1091749 |  1146946 |  1214830 |  1255187 |  1309123 |  1349826 |  1375728
$\leq$ 17 authors |  900437 |  962104 |  1025935 |  1093523 |  1149041 |  1217286 |  1257977 |  1312168 |  1353387 |  1379444
$\leq$ 18 authors |  901213 |  963094 |  1027123 |  1094968 |  1150550 |  1219191 |  1260005 |  1314618 |  1356007 |  1382345
$\leq$ 19 authors |  901806 |  963762 |  1028034 |  1096027 |  1151782 |  1220603 |  1261640 |  1316455 |  1358224 |  1384633
$\leq$ 20 authors |  902332 |  964418 |  1028847 |  1096902 |  1152800 |  1221776 |  1262971 |  1317998 |  1360019 |  1386621
any # authors |  905349 |  967999 |  1033298 |  1102155 |  1158923 |  1228952 |  1270887 |  1326996 |  1370378 |  1398003

## Methods.

Data were obtained from inCites
(https://incites.thomsonreuters.com/#/analytics; “Journals” and “People”), an
integrated web-based platform based on the data from the Web of Science by
Clarivate Analytics (provided by the University of Zurich). We focused on the
journals included in the Science Citation Index Expanded which covers 177
subject areas out of the 252 subject categories comprised by the Web of
Science schema (i.e., “Filters: Research Area”; we selected the areas from
http://mjl.clarivate.com/scope/scope_scie/). We count as papers only regular
scientific articles that report original research, not books, reviews,
editorials, letters to the editor and the like (i.e., “Filters: Document
type”; we selected “Article”). We restrict our search to the journals listed
in the Journal Citation Reports (i.e., “Thresholds: JIF Quatile”; we selected
Q1, Q2, Q3, Q4). The number of papers published by $i=1,2,...n$ authors was
obtained using an additional threshold (i.e., “Thresholds: Authors per
Document”; we selected Min=1 and Max=1 for single-authored papers, …).