Image segmentation aims at grouping pixels that belong to the same object or region. At the heart of image segmentation lies the problem of determining whether a pixel is inside or outside a region, which we denote as the "insideness" problem. Many Deep Neural Networks (DNNs) variants excel in segmentation benchmarks, but regarding insideness, they have not been well visualized or understood: What representations do DNNs use to address the long-range relationships of insideness? How do architectural choices affect the learning of these representations? In this paper, we take the reductionist approach by analyzing DNNs solving the insideness problem in isolation, i.e. determining the inside of closed (Jordan) curves. We demonstrate analytically that state-of-the-art feed-forward and recurrent architectures can implement solutions of the insideness problem for any given curve. Yet, only recurrent networks could  learn these general solutions when the training enforced a specific "routine" capable of breaking down the long-range relationships. Our results highlights the need for new training strategies that decompose the learning into appropriate stages, and that lead to the general class of solutions necessary for DNNs to understand insideness. Image segmentation is necessary for complete image understanding. A key component of image segmentation is to determine whether a pixel is inside or outside a region, ie. the "insideness" problem (Ullman, 1984; 1996) . Deep Neural Networks (DNNs) have been tremendously successful in image segmentation benchmarks, but it is not well understood whether DNNs represent insideness or how. Insideness has been overlooked in DNNs for segmentation since they have been mainly applied to the modality of "semantic segmentation", ie. labelling each pixel with its object category (Ronneberger et al., 2015; Yu & Koltun, 2016; Visin et al., 2016; Badrinarayanan et al., 2017; Chen et al., 2018b; Long et al., 2015; Lateef & Ruichek, 2019) . In such cases, insideness is not necessary since a solution can rely only on object recognition. Yet, the recent need to solve more sophisticated visual tasks has fueled the development of DNNs for more advanced segmentation modalities and applications, e.g. segmentation of individual object instances rather than object categories (Li et al., 2016; Song et al., 2018; Chen et al., 2018a; Hu et al., 2018; Maninis et al., 2018; Liu et al., 2018b; He et al., 2017) , understanding spatial relationships such as containment (Kim et al., 2018) and generating realistic images (Zhu et al., 2017) . Insideness can play a central role in these tasks and new ones, especially when there are few cues available besides the boundaries of the objects, ie. when there is lack of texture and color, such as in cartoons or sketches, or when objects have different textures and shapes from those seen during training. In this paper, we investigate derived and learned insideness-related representations in DNNs for segmentation. We take the reductionist approach by isolating insideness from other components in image segmentation. We analyze the segmentation of closed curves, similar to the methodology in Minsky & Papert's historic book Perceptrons (Minsky & Papert, 1969) . In this way, we distill insideness to a minimum representation by eliminating other components. We analytically demonstrate that two state-of-the-art network architectures, namely, DNNs with dilated convolutions (Yu & Koltun, 2016; Chen et al., 2018b) and convolutional LSTMs (ConvLSTMs) (Xingjian et al., 2015) , among other networks, can exactly solve the insideness problem for any given curve with network sizes that are easily implemented in practice. The proofs draw on algorithmic ideas from classical work on visual routines (Ullman, 1984; 1996) , namely, the rayintersection method and the coloring method, to derive equivalent neural networks that implement these algorithms. Then, in a series of experiments with synthetically generated closed curves, we evaluate the capabilities of these DNNs to learn the insideness problem. The experiments show that when using standard training strategies, the DNNs do not learn general solutions for insideness, even though these DNNs are sufficiently complex to capture the long-range relationships. The only network that achieves almost full generalization in all tested cases is a recurrent network with a training strategy designed to encourage a specific mechanism for dealing with long-range relationships. These results add to the growing body of works that show that DNNs have problems in learning to solve some elemental visual tasks (Linsley et al., 2018; Liu et al., 2018a; Wu et al., 2018; ShalevShwartz et al., 2017) . Shalev-Shwartz et al. (2017) introduced several tasks that DNNs can in theory solve, as it was shown mathematically, but the networks were unable to learn, not even for the given dataset, due to difficulties in the optimization with gradient descent. In contrast, the challenges we report for insideness are related to poor generalization rather than optimization, as our experiments show the networks succeed in solving insideness for the given dataset. Linsley et al. (2018) introduced new architectures that better capture the long-range dependencies in images. Here, we show that the training strategy has a big impact in capturing the long-range dependencies. Even if the DNNs we tested had the capacity to capture such long-range dependencies, they do not learn a general solution with the standard training strategies. We have shown that DNNs with dilated convolutions and convolutional LSTM that are implementable in practice are sufficiently complex to solve the insideness problem for any given curve. When using the standard training strategies, the units in these networks become specialized to detect characteristics of the curves in the training set and only generalize to curves of the same family as the training, even when using large number of training examples. Yet, we found that when simple recurrent networks are supervised to learn the coloring routine, which does not contain long-range relationships, the general solution for the insideness problem emerged using orders of magnitude less data. We hope that this research will help establish the reductionist approach to understand DNNs. In future works, we plan to use the reductionist approach to tackle other important aspects of segmentation beyond insideness (e.g. the discontinuity of segments and the hierarchical structure of segments), and show that this research can lead to improvements of state-of-the-art in image segmentation and in other applications that require understanding of complex spatial relationships.
DNNs for image segmentation can implement solutions for the insideness problem but only some recurrent nets could learn them with a specific type of supervision. This paper introduces insideness to study semantic segmentation in deep learning era, and the results can help models generalize better.