Deep learning (DL) has in recent years been widely used in natural
 language processing (NLP) applications due to its superior
 performance. However, while natural languages are rich in
 grammatical structure, DL has not been able to explicitly
 represent and enforce such structures. This paper proposes a new
 architecture to bridge this gap by exploiting tensor product
 representations (TPR), a structured neural-symbolic framework
 developed in cognitive science over the past 20 years, with the
 aim of integrating DL with explicit language structures and rules.
 We call it the Tensor Product Generation Network
 (TPGN), and apply it to image captioning. The key
 ideas of TPGN are: 1) unsupervised learning of
role-unbinding vectors of words via a TPR-based deep neural
network, and 2) integration of TPR with typical DL architectures
 including Long Short-Term Memory (LSTM) models. The novelty of our
 approach lies in its ability to generate a sentence and extract
 partial grammatical structure of the sentence by using
 role-unbinding vectors, which are obtained in an unsupervised
 manner. Experimental results demonstrate the effectiveness of the
 proposed approach. Deep learning is an important tool in many current natural language processing (NLP) applications. However, language rules or structures cannot be explicitly represented in deep learning architectures. The tensor product representation developed in BID22 ; BID24 has the potential of integrating deep learning with explicit rules (such as logical rules, grammar rules, or rules that summarize real-world knowledge). This paper develops a TPR approach for deep-learning-based NLP applications, introducing the Tensor Product Generation Network (TPGN) architecture. To demonstrate the effectiveness of the proposed architecture, we apply it to a important NLP application: image captioning.A TPGN model generates natural language descriptions via learned representations. The representations learned in the TPGN can be interpreted as encoding grammatical roles for the words being generated. This layer corresponds to the role-encoding component of a general, independentlydeveloped architecture for neural computation of symbolic functions, including the generation of linguistic structures. The key to this architecture is the notion of Tensor Product Representation (TPR), in which vectors embedding symbols (e.g., lives, frodo) are bound to vectors embedding structural roles (e.g., verb, subject) and combined to generate vectors embedding symbol structures ([frodo lives]). TPRs provide the representational foundations for a general computational architecture called Gradient Symbolic Computation (GSC), and applying GSC to the task of natural language generation yields the specialized architecture defining the model presented here. The generality of GSC means that the results reported here have implications well beyond the particular tasks we address here.The paper is organized as follows. Section 2 discusses related work. In Section 3, we review the basics of tensor product representation. Section 4 presents the rationale for our proposed architecture. Section 5 describes our proposed model in detail. In Section 6, we present our experimental results. Finally, Section 7 concludes the paper. In this paper, we proposed a new Tensor Product Generation Network (TPGN) for natural language generation and related tasks. The model has a novel architecture based on a rationale derived from the use of Tensor Product Representations for encoding and processing symbolic structure through neural network computation. In evaluation, we tested the proposed model on captioning with the MS COCO dataset, a large-scale image captioning benchmark. Compared to widely adopted LSTM-based models, the proposed TPGN gives significant improvements on all major metrics including METEOR, BLEU, and CIDEr. Moreover, we observe that the unbinding vectors contain important grammatical information. Our findings in this paper show great promise of TPRs. In the future, we will explore extending TPR to a variety of other NLP tasks.
This paper is intended to develop a tensor product representation approach for deep-learning-based natural language processinig applications.