The visual world is vast and varied, but its variations divide into structured and unstructured factors. Structured factors, such as scale and orientation, admit clear theories and efficient representation design. Unstructured factors, such as what it is that makes a cat look like a cat, are too complicated to model analytically, and so require free-form representation learning. We compose structured Gaussian filters and free-form filters, optimized end-to-end, to factorize the representation for efficient yet general learning. Our experiments on dynamic structure, in which the structured filters vary with the input, equal the accuracy of dynamic inference with more degrees of freedom while improving efficiency.

 (Please see https://arxiv.org/abs/1904.11487 for the full edition.) Although the visual world is varied, there is nevertheless ubiquitous structure. Free-form learned representations are structure-agnostic, making them general, but their not harnessing structure is computationally and statistically inefficient. Structured representations like steerable filtering BID5 BID6 , scattering BID0 , and steerable networks BID1 efficiently express certain structures, but are constrained. We propose the semi-structured composition of Gaussian and free-form filters to blur the line between free-form and structured representations.The effectiveness of strongly structured representations hinges on whether they encompass the true structure of the data. If not, the representation is limiting, and subject to error. At least, such is the case when structure substitutes for learning. In this work we compose structured and free-form filters and learn both end-to-end ( FIG1 ). The free-form parameters are not constrained by our composition for generality. The structured parameters are low-dimensional for efficiency.We choose Gaussian structure to represent the spatial structures of scale, aspect, and orientation through covariance BID7 . Optimizing these structured covariance parameters carries out a form of differentiable architecture search over receptive fields. Since this structure is lowdimensional, it is computationally efficient and could be learned from limited data.
Dynamic receptive fields with spatial Gaussian structure are accurate and efficient. This paper proposes a structured convolution operator to model deformations of local regions of an image, which significantly reduced the number of parameters.