We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others.
 Multiplicative interaction layers as primitive operations have a long-established presence in the literature, though this often not emphasized and thus under-appreciated. We begin by showing that such layers strictly enrich the representable function classes of neural networks. We conjecture that multiplicative interactions offer a particularly powerful inductive bias when fusing multiple streams of information or when conditional computation is required. We therefore argue that they should be considered in many situation where multiple compute or information paths need to be combined, in place of the simple and oft-used concatenation operation. Finally, we back up our claims and demonstrate the potential of multiplicative interactions by applying them in large-scale complex RL and sequence modelling tasks, where their use allows us to deliver state-of-the-art results, and thereby provides new evidence in support of multiplicative interactions playing a more prominent role when designing new neural network architectures. Much attention has recently turned toward the design of custom neural network architectures and components in order to increase efficiency, maximise performance, or otherwise introduce desirable inductive biases. While there have been a plethora of newer, intricate architectures proposed, in this work we train our sights instead on an older staple of the deep learning toolkit: multiplicative interactions. Although the term itself has fallen somewhat out of favour, multiplicative interactions have reappeared in a range of modern architectural designs. We start this work by considering multiplicative interactions as an object of study in their own right. We describe various formulations and how they relate to each other as well as connect more recent architectural developments (e.g. hypernetworks Ha et al. (2017) , dynamic convolutions Wu et al. (2019) ) to the rich and longer-standing literature on multiplicative interactions. We hypothesise that multiplicative interactions are suitable for representing certain meaningful classes of functions needed to build algorithmic operations such as conditional statements or similarity metrics, and more generally as an effective way of integrating contextual information in a network in a way that generalizes effectively. We show this empirically in controlled synthetic scenarios, and also demonstrate significant performance improvement on a variety of challenging, large-scale reinforcement learning (RL) and sequence modelling tasks when a conceptually simple multiplicative interaction module is incorporated. Such improvements are consistent with our hypothesis that the use of appropriately applied multiplicative interactions can provide a more suitable inductive bias over function classes leading to more data-efficient learning, better generalization, and stronger performance. We argue that these operations should feature more widely in neural networks in and of themselves, especially in the increasingly important setting of integrating multiple streams of information (including endogenously created streams e.g. in branching architectures). Our contributions are thus: (i) to re-explore multiplicative interactions and their design principles; (ii) to aid the community's understanding of other models (hypernetworks, gating, multiplicative RNNs) through them; (iii) to show their efficacy at representing certain solutions; and (iv) to empirically apply them to large scale sequence modeling and reinforcement learning problems, where we demonstrate state-of-the-art results. In this work we considered multiplicative interactions and various formulations thereof, connecting them to a variety of architectures, both older and modern, such as Hypernetworks, multplicative LSTMs or gating methods. We hypothesise that the ability of such networks to better represent a broader range of algorithmic primitives (e.g. conditional-statements or inner products) allows them to better integrate contextual or task-conditional information to fuse multiple stream of data. We first tested empirically this hypothesis in two controlled settings, in order to minimize the effect of confounding factors. We further show that we could match state-of-the-art methods on multiple domains with only LSTMs and multiplicative units. While we do not necessarily advocate for a specific instance of the above methods, we hope that this work leads to a broader understanding and consideration of such methods by practitioners, and in some cases replacing the standard practice of concatenation when using conditioning, contextual inputs, or additional sources of information. We believe there are many ways to explore this space of ideas more broadly, for instance looking at: the role of various approximations to these methods; ways to make their implementations more efficient; and their application to newer domains. Finally, while attention models use some of these multiplicative interactions, we hope that applying some of the lessons from this work (such as higher order interactions) will allow even greater integration of information in attention systems. Proof. Inclusion comes directly from the fact that if we split input into arbitrary parts [x; z] we get: which proves that H mlp ⊂ H mu . Thus, the only aspect of the theorm left to prove is that the inclusion is strict. Let us consider a 1D function x → x 2 , and for simplicity let x = z (a domain where context equals input). A single layer MLP with a single multiplicative unit can represent this function exactly, by using A = 0 and W = I, as then we obtain x T W x = x T x = x 2 . Since our function is positive, it is not affecting the multiplicative network output. For a regular MLP, let us first notice that we need at least one hidden layer, as otherwise MLP is a linear function and f is not. Lets denote by V, c and w, b weights and biases of second, and first layers respectively. Then we have to satisfy where g is transformation represented by all higher layers of the MLP (in particular if there are just 2 layers, then g(x) = x). Note that RHS is differentiable everywhere, while LHS is differentiable iff for each i and for each x we have w i x + b i = 0 (or f is independent from x, which x 2 does not satisfy). However, this is impossible, as if w i = 0, then we can always pick x = −b i /w i , and if all 2 , leading to a contradiction. Proof. Inclusion comes directly from the fact that only some activations are replaced, and in particular we can always replace none, thus leading to equality of hypotheses classes. To show that the inclusion is strict lets consider a Weierstrass function itself f (x) = σ w (x). We definitely have f ∈ H w as we can define 1 hidden layer network, with one hidden neuron and all the weights set to 1, and all biases to 0. Now, relu networks are piece-wise linear while the Weierstrass function is nowhere differentiable Weierstrass (1895) and thus not piece-wise linear. Similarly, network with an activation that is differentiable everywhere (e.g. sigmoid or tanh) is everywhere differentiable wrt. inputs, while Weierstrass function -nowhere Weierstrass (1895).
We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others. Presents multiplicative interaction as a unified characterization for representing commonly used model architecture design components, showing empirical proof of superior performance on tasks like RL and sequence modeling. The paper explores different types of multiplicative interactions and finds MI models able to achieve a state-of-the-art performance on language modeling and reinforcement learning problems.