Humans can understand and produce new utterances effortlessly, thanks to their systematic compositional skills. Once a person learns the meaning of a new verb "dax," he or she can immediately understand the meaning of "dax twice" or "sing and dax." In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences. We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods. We find that RNNs can generalize well when the differences between training and test commands are small, so that they can apply "mix-and-match" strategies to solve the task. However, when generalization requires systematic compositional skills (as in the "dax" example above), RNNs fail spectacularly. We conclude with a proof-of-concept experiment in neural machine translation, supporting the conjecture that lack of systematicity is an important factor explaining why neural networks need very large training sets.
Using a simple language-driven navigation task, we study the compositional capabilities of modern seq2seq recurrent networks. 
This paper focuses on the zero-shot learning compositional capabilities of modern sequence-to-sequence RNNs and  exposes the short-comings of current seq2seq RNN architectures. The paper analyzes the composition abilities of RNNs, specifically, the generalization ability of RNNs on random subset of SCAN commands, on longer SCAN commands, and of composition over primitive commands.  The authors introduce a new dataset that facilitates the analysis of a Seq2Seq learning case