The Acquisition of Physical Knowledge in Generative Neural Networks
As children grow older, they develop an intuitive understanding of the
physical processes around them. Their physical understanding develops in
stages, moving along developmental trajectories which have been mapped out
extensively in previous empirical research. Here, we investigate how the
learning trajectories of deep generative neural networks compare to children's
developmental trajectories using physical understanding as a testbed. We
outline an approach that allows us to examine two distinct hypotheses of human
development - stochastic optimization and complexity increase. We find that
while our models are able to accurately predict a number of physical processes,
their learning trajectories under both hypotheses do not follow the
developmental trajectories of children.
Luca M. Schulze Buschoff, Eric Schulz, Marcel Binz
# The Acquisition of Physical Knowledge

###### Abstract

As children grow older, they develop an intuitive understanding of the physical processes around them. Their physical understanding develops in stages, moving along developmental trajectories which have been mapped out extensively in previous empirical research. Here, we investigate how the learning trajectories of deep generative neural networks compare to children's developmental trajectories using physical understanding as a testbed. We outline an approach that allows us to examine two distinct hypotheses of human development - stochastic optimization and complexity increase. We find that while our models are able to accurately predict a number of physical processes, their learning trajectories under both hypotheses do not follow the developmental trajectories of children.

Machine Learning, Neural Networks, Machine Learning, Neural Networks

## 1 Introduction

More than 70 years ago, Turing (1950) famously suggested that "instead of trying to produce a programme to simulate the adult mind, why not rather try to produce one which simulates the child's? If this were then subjected to an appropriate course of education one would obtain the adult brain." If we want to take Turing's proposal seriously, we have to ask ourselves: how do children learn?

Developmental psychologists have investigated children's learning in a number of different realms. One of the most well-studied is their acquisition of physical knowledge (Baillargeon, 1996; 2004; Spelke & Kinzler, 2007; Lake et al., 2017). Here, prior empirical work provides us with a precise understanding of the stages that children undergo during their cognitive development (see Figure 1A for an example). It, therefore, serves as an ideal testbed for our investigation.

In the present paper, we set out to formalize and test two distinct hypotheses of children's development. The first is the idea of _development as stochastic optimization_, which argues that cognitive development results from some form of stochastic optimization procedure (Gopnik et al., 2017; Ullman & Tenenbaum, 2020; Giron et al., 2022; Wolff, 1987). The second is the idea of _development as complexity increase_, which instead stipulates that the knowledge structures involved in human reasoning become more complex over time (Baillargeon, 2002; Binz & Endres, 2019).

First, we show how both hypotheses can be instantiated in a \(\beta\)-variational autoencoder (\(\beta\)-VAE) framework. We then probe models with different degrees of complexity and optimization on physical reasoning tasks using violation-of-expectation (VOE) methods (Piloto et al., 2018; Smith et al., 2019). Finally, we compare the learning trajectories of these artificial systems to the developmental trajectories of children.

We find that even fairly generic deep generative neural networks acquire many physical concepts. However, the order in which they acquire these concepts under both hypotheses does not align well with the acquisition order of children - neither hypothesis fully captures the learning trajectories of children. Thus, we conclude that the investigated models do not acquire their knowledge in accordance with Turing's proposal.

The remainder of this paper is organized as follows. Section 2 surveys previous literature on models of human-like physical knowledge and developmental trajectories. In Section 3, we illustrate how to instantiate the development as stochastic optimization and development as complexity increase hypotheses in the \(\beta\)-VAE framework. We then apply these models to different physical reasoning domains in Section 4. Section 5 concludes this report with a general discussion of our findings.
## 2 Related work

### Models with human-like physical knowledge

Building models with human-like physical knowledge has become an active research area in recent years (see Table 1 for a summary). Battaglia et al. (2013) argued that human reasoning in complex natural scenes is driven by an intuitive physics engine that relies on probabilistic simulations to make inferences. Following this idea, they introduced interaction networks - a model that performs simulations by combining an object-centric and a relation-centric component (Battaglia et al., 2016). In contrast to the initial approach that relied on a hard-coded physics engine, interaction networks are learnable engines, allowing them to generalize to novel systems with different configurations of objects and relations. In a similar vein, Smith et al. (2019) combined a perception module that infers physical object representations from raw images with a reasoning module that predicts future object states conditioned on the object representations. They found that this model matched human performance in a number of scenarios. Lerer et al. (2016) trained large convolutional neural networks to predict the stability of wooden block towers as well as the trajectories of falling blocks. They showed that the performance of such networks exceeds that of human subjects on synthetic data. Zhang et al. (2016) compared the intuitive physics engine of Battaglia et al. (2013) to the convolutional neural network of Lerer et al. (2016). They found that while convolutional networks are able to achieve superhuman accuracy in judging the stability of block towers, their physical understanding is dissimilar to that of humans.

### The violation-of-expectation method

How physical knowledge of artificial systems should be evaluated has also received attention. Taking inspiration from developmental psychology, Piloto et al. (2018) proposed to use the VOE method to probe the knowledge of neural networks (Baillargeon, 1996). In particular, they measured the surprise of a network after observing physically implausible sequences. Their work was among the first to demonstrate that the VOE method can elucidate black-box models' inference mechanisms. Moreover, recent intuitive physics benchmarks have also been inspired by work in developmental psychology. Riochet et al. (2021) presented an "evaluation benchmark which diagnoses how much a given system understands about physics by testing whether it can tell apart well-matched videos of possible versus impossible events constructed with a game engine." Likewise, Weihs et al. (2022) proposed a benchmark testing for knowledge about continuity, solidity, and gravity using videos filmed in infant-cognition labs and robotic simulation environments. Finally, Piloto et al. (2022) also introduced a

Figure 1: **A**: Human developmental trajectory for support events outlined by Baillargeon (1996). The illustrations are taken from Baillargeon (1996) and they show the physical rules acquired at the respective ages. With \(3\) months, infants decide based on a simple contact or no contact rule. According to this rule, a block configuration is considered stable if the blocks touch each other. At around \(5\) months, infants understand that the type of contact matters. Now, only configurations with blocks stacked on top of each other are judged as stable. At \(6.5\) months, they begin to also consider the overlap of the blocks. Finally, at \(12.5\) months they are able to incorporate the block shapes into their judgement, relying not only on the amount of contact but also on how the mass is distributed for each block. **B**: Illustration of our generative video prediction model.
data set for evaluating intuitive physics in neural networks using the VOE method and use this data set to probe the physical knowledge of a deep learning model equipped with object-centric representations.

### Modeling human development

Even though developmental psychology has inspired how to evaluate physical knowledge in neural networks, the emphasis of prior machine learning research has always been on reproducing adult-level performance. In contrast, computational cognitive scientists also strive to build artificial learning systems that capture the developmental trajectories of children. Perhaps most closely related to our work is the approach of Binz and Endres (2019) who compared trajectories of Bayesian neural networks that had access to different amounts of data to human developmental trajectories. They investigated both occlusion and support events and found that the acquisition order of concepts in their model aligned with that of children. However, in contrast to their work, which uses an oracle to provide a supervision signal about block stability and visibility, our approach solely relies on an unsupervised training objective.

If we look beyond the realm of intuitive physics, we can find other works that have attempted to model the process of human development. Huber et al. (2022) investigated the emergence of object recognition in children. They showed that four- to six-year-olds are already more robust to image distortions compared to deep neural networks trained on ImageNet. Furthermore, children predominantly relied on shape instead of texture for object detection, making them more similar to adults than deep neural networks (Geirhos et al., 2018). Averbeck (2022) pruned recurrent neural networks by removing weak synapses. They found that pruned networks were more resistant to distractions in a working memory task and made optimal choices more frequently in a reinforcement learning setting. These results were consistent with developmental improvements during adolescence, where performance on cognitive operations improves as excitatory synapses in the cortex are pruned. Finally, Giron et al. (2022) examined a theory of development as stochastic optimization. In particular, they combined this idea with a model of human decision-making in multi-armed bandit problems and demonstrated that development resembles a stochastic optimization process in the parameter space of this model. In contrast to these earlier models of development, our setup uses high-dimensional visual stimuli (i.e., video sequences) and solely relies on an unsupervised training objective. It, therefore, more closely resembles the actual learning processes of children in the real world.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & support & occlusion & collision & unsupervised & violation- & sequential & developmental \\ events & events & events & learning & of- & expectations & predictions & trajectories \\ \hline \hline \multicolumn{10}{l}{**Work attempting to build models with human-like physical intuitions:**} \\ \hline Battaglia et al. (2013) & ✓ & ✗ & ✗ & ✗ & ✗ & ✗ \\ Battaglia et al. (2016) & ✗ & ✗ & ✓ & ✗ & ✗ & ✓ & ✗ \\ Smith et al. (2019) & ✗ & ✓ & ✗ & ✓ & ✓ & ✓ & ✗ \\ Lerer et al. (2016) & ✓ & ✗ & ✗ & ✗ & ✗ & ✗ \\ Zhang et al. (2016) & ✓ & ✗ & ✗ & ✗ & ✗ & ✗ \\ Piloto et al. (2018) & ✗ & ✗ & ✓ & ✓ & ✓ & ✗ \\ Riochet et al. (2021) & ✗ & ✗ & ✓ & ✓ & ✗ & ✗ \\ Piloto et al. (2022) & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ & ✗ \\ \hline \hline \multicolumn{10}{l}{**Work attempting to model human developmental trajectories:**} \\ \hline Giron et al. (2022) & ✗ & ✗ & ✗ & ✗ & ✗ & ✗ & ✗ \\ Averbeck (2022) & ✗ & ✗ & ✗ & ✗ & ✗ & ✗ & ✓ \\ Huber et al. (2022) & ✗ & ✗ & ✗ & ✗ & ✗ & ✗ & ✓ \\ Binz and Endres (2019) & ✓ & ✓ & ✗ & ✗ & ✗ & ✗ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Table summarizing previous work attempting to build models with human-like physical intuitions and work attempting to model human developmental trajectories. Machine learning research has predominantly focused on reproducing adult-level performance, while computational cognitive science has relied heavily on low-dimensional and static stimuli. The present paper combines the best of both worlds.
## 3 Methods

In the following, we discuss how the _development as stochastic optimization_ and _development as complexity increase_ hypotheses can be instantiated in the \(\beta\)-VAE framework1. For the development as stochastic optimization hypothesis, we train a generative video prediction model using gradient descent. To obtain a learning trajectory of this model, we evaluate snapshots of the model in every epoch. For the development as complexity increase hypothesis, we train models of different complexities by making use of the \(\beta\)-VAE framework (Higgins et al., 2016). Doing so enforces a bottleneck on the representational capacity of the hidden representations (Sims, 2016; Bates Jacobs, 2020), which can be interpreted as a particular form of computational complexity (we will discuss potential alternatives in our general discussion). Learning trajectories for this hypothesis are obtained by increasing the model's representational capacity, i.e., by moving from higher to lower \(\beta\)-values within fully converged models.

Footnote 1: We do not propose that these exact mechanisms are implemented by the brain. Rather, we propose these mechanisms as “as if” models: while the mechanisms behind biological development are likely not exactly analogous, they have similar characteristics.

### Model architecture and objective

We use the recurrent state space model (RSSM) (Hafner et al., 2019; Saxena et al., 2021) - which can be seen as a sequential version of a VAE - as an exemplary model for our analysis. We selected this particular model for two reasons. First, generative models are, in general, a good starting point as they have previously been used to model intuitive physics (Piloto et al., 2018, 2022; Riochet et al., 2021) (see also (Duan et al., 2022) for an overview). Furthermore, we decided on a VAE-based model since such models have been shown to capture many aspects of human cognition (Malloy et al., 2022; Nagy et al., 2020; Bates Jacobs, 2020), thereby making them a reasonable candidate hypothesis for studying human development. The model maintains a latent state at each time step, which is comprised of a deterministic component \(h_{t}\) and a stochastic component \(s_{t}\) (see Figure 1B). These components depend on the previous time steps through a function \(f(h_{t-1},s_{t-1})\), which is implemented as a gated recurrent neural network. We train our models by optimizing the following objective:

\[-\sum_{t=1}^{T}\mathbb{E}_{q(s_{t}|o_{\leq t})}[\ln p(o_{t}\mid s_{ t})]+ \tag{1}\] \[\beta\ \mathbb{E}_{q(s_{t-1}|o_{\leq t-1})}\big{[}\text{KL}\big{(}q(s_ {t}\mid o_{\leq t})\mid\mid p(s_{t}\mid s_{t-1})\big{)}\big{]}\]

where \(o_{\leq t}=o_{1},o_{2},\ldots,o_{t}\) is a sequence of rendered images obtained from a 3D physics engine.

For all models, the size of the stochastic hidden dimension \(s_{t}\) was kept at \(20\), while the size of the deterministic hidden dimension \(h_{t}\) was set to \(200\), as in previous implementations of the RSSM (Hafner et al., 2019; Saxena et al., 2021). We furthermore adopted the image encoder and decoder architectures described by Dittadi et al. (2020). We refer the reader to Appendix A for further details about the model architecture and training procedure.

We can use the RSSM to generate either open- or closed-loop predictions. For open-loop predictions, the model processes a number of initial observations to infer an approximate posterior \(q(s_{t-1}\mid o_{\leq t-1})\), followed by decoding subsequent latent representations sampled from the prior \(p(s_{t}\mid s_{t-1})\). For closed-loop reconstructions, the decoder is instead continuously given representations sampled from the posterior which is updated at every time step using the previously observed frame. We generally report results obtained via open-loop predictions unless stated otherwise.

### Measuring surprise

To assess whether a model has learned a specific physical rule, we make use of the VOE paradigm (Piloto et al., 2018, 2022). For this, the model is presented with two video sequences: a _violated_ sequence, which constitutes a violation according to the rule, and an _expected_ sequence, which is consistent with the rule. If the model has successfully learned a specific rule, it should show a larger degree of surprise for the violated compared to the expected sequence.

Following Piloto et al. (2022) and Smith et al. (2019), we measure the model's surprise using the decoder's negative log-likelihood (NLL) of observations (i.e., the first term of Equation 1)2. More specifically, for each sequence, we determine if the NLL is larger for the violated compared to the expected sequence for the majority of the frames. We then take the mean over all sequences for a specific condition in order to check whether the reconstructions of the model better match the expected or the violated sequences. This approach is inspired by developmental psychology and allows us to measure a model's surprise similar to how developmental psychologists measure surprise in children (see Appendix B for a discussion on different measures of surprise).

Footnote 2: Note that this definition differs from the common usage of surprise in the context of information theory, where it is defined to be \(-\log p(o\leq_{t})\).

## 4 Results

We evaluated our models on three distinct physical processes. For each of these processes, we generated training data sets inspired by experiments from developmental psychology using the Unity game engine (Unity Technologies,
2005). We randomly varied a number of properties to ensure sufficient variability in the training data. We also generated test data sets that - following the VOE paradigm - contain pairs of violated and an expected sequences for each of the conditions in the respective event types (see Appendix C for a detailed description of the data generation process and a visualization of the employed test sequences).

### Support events

We began our investigations by looking at support events, which consist of block configurations such as the ones shown in Figure 1A. Similar tasks have been studied extensively in both the machine learning and developmental psychology community, and they, therefore, serve as an ideal starting point for our analyses. Each scene in our data set contains two randomly configured blocks in a gray room.3

Footnote 3: Note that it would certainly be possible to consider more complex configurations (e.g., by increasing the number of blocks), but we deliberately made this design choice to match the experimental paradigms used in developmental psychology.

Baillargeon (1996) has shown that, as infants grow older, they make use of increasingly complex rules to decide whether a given block configuration is stable or not (also see Baillargeon (2002; 2004)). With \(3\) months, infants decide based on a simple contact or no contact rule. According to this rule, a block configuration is considered stable if the blocks touch each other. At around \(5\) months, infants understand that the type of contact matters. Now, only configurations with blocks stacked on top of each other are judged as stable. At \(6.5\) months, they begin to also consider the overlap of the blocks. Finally, at \(12.5\) months they are able to incorporate the block shapes into their judgement, relying not only on the amount of contact but also on how the mass is distributed for each block.

For each of the four rules for support events, we constructed pairs of violated and expected test sequences with identical first frames. For example, according to the overlap rule, a block configuration should only be stable if the blocks are stacked on top of each other with enough overlap. A test sequence pair for this rule shows two blocks that only slightly overlap. The expected test sequence, which is consistent with the rule, shows the top block falling. In contrast

Figure 2: The first row shows the difference in surprise between the violated and expected test sequences given by the fully trained model with \(\beta=1\) for the overlap condition in the support event data-set. The surprise curve is smoothed using cubic spline interpolation. The second row shows the expected test sequence. The third row shows the violated test sequence. The last row shows the open-loop reconstruction from the model given the first two frames.
- and in violation of real physics
- the violated test sequence shows the same block configuration that however appears stable (Figure 2 shows an example pair for this rule, together with the predictions of our model and its corresponding surprise values).

We first verified that our model is able to predict a given scene accurately into the future. For this purpose, we plotted the open-loop predictions given by the fully trained model with \(\beta=1\). Figure 2 shows an examplary result for the overlap condition. We see that the predictions of the fully trained model closely match the expected sequence. Furthermore, we see that high surprise values for the violated sequence coincide with differences to the expected sequence - the model is surprised when it observes parts of a video sequence that diverge from real physics. Appendix D.1 shows further examples for open- and closed-loop predictions, while Appendix F contains a visualization of prediction errors.

Figure 3 illustrates how knowledge about physical rules develops over time for the two earlier outlined hypotheses. On the left, the percentage of sequences for which the surprise for the violated sequence exceeds that of the expected sequence is plotted for each of the four conditions over the course of training for the model with \(\beta\) = 1. Here, the model becomes increasingly optimized over the epochs, thereby implementing the development as stochastic optimization hypothesis. It is evident that the model is able to learn three of the four conditions as it shows more surprise for the violated than the expected sequences for the majority of the cases. However, it learns the conditions at roughly the same rate which does not match the developmental trajectories of children. While it settles at different levels for the conditions, the order of these conditions also does not match the acquisition order of children: the shape condition, for instance, shows the second highest percentage while it is the last rule that children acquire.

On the right, the percentage of sequences for which the surprise for the violated sequence exceeds that of the matching expected sequence is plotted for each of the four conditions for fully trained models with different \(\beta\)-values. This relates to the development as complexity increase hypothesis since the representational capacity of the model increases as \(\beta\) decreases. The order in which increasingly complex models learn the different conditions again does not resemble the developmental trajectories of children: the model with \(\beta=8\) performs very similarly to the model with \(\beta=1\). To summarize, for support events, neither hypotheses yields learning trajectories that resemble the developmental trajectories of children (see also Appendix G for a closed-loop counterpart to Figure 3).

### Occlusion events

Next, we wanted to test whether the results obtained in the last section hold across domains. Thus, we extended our analyses to occlusion events, which display a moving object passing behind two vertical columns. The two columns, together with an optional horizontal connection at the top or bottom, form an occluder which may hide the moving object. Like in the preceding section, we created a randomized training data set alongside several test sequences that violate physical principles (see Appendix C for examples and further details about the data generation process).

Baillargeon (1996) reported that, in this setting, (1) infants form a simple behind/not-behind distinction by 2.5 months. Hereby they assume that the object will not re-appear in the gap between the columns that are connected at the top or bottom. By 3 months, (2) infants expect objects to re-appear when the columns are connected at the top but fail to do so if the columns are connected at the bottom. Finally, at 3.5 months, (3) they also expect objects to appear behind screens that are connected at the bottom, given that the

Figure 3: The plot on the left shows the percentage of sequences for which the surprise for the violated sequence exceeds that of the expected sequence for the model with \(\beta=1\) at every epoch and for each condition of the support event data set. The lines are smoothed with a uniform kernel of size 10. The plot on the right shows the same metric for fully trained models with different \(\beta\).
object is taller than the connecting part.

Figure 4 visualizes our modeling results by showing the percentage of sequences for which the surprise for the violated sequence exceeds that of the expected sequence for each condition. First, we can observe that the fully trained model with \(\beta=1\) is surprised when presented with any of the test sequences that violate physical principles, indicating that it understood all of the three aforementioned occlusion settings. For the left side of the plot, which depicts the development as stochastic optimization hypothesis, we see that the number of sequences for which the surprise for the violated sequence exceeds that of the matching expected sequence increases at the same rate for all three conditions, meaning that the model learns the three concepts at approximately the same time. Thus, for occlusion events, the stochastic optimization hypothesis again does not yield a learning trajectory that matches that of children.

The right side of Figure 4 relates to the development as complexity increase hypothesis. Here, we see that the the percentage of sequences for which the surprise for the violated sequence exceeds that of the matching expected sequence remains relatively stable as the complexity of the model increases. This again does does not lend support to the complexity increase hypothesis.

### Collision events

The last physical process that we investigated were collision events. Here, each scene shows an object rolling down a hill and colliding with a stationary object. To train our models, we again created a randomized training data set of such scenarios together with several test sequences that violate physical principles (see Appendix C for examples and further details about the data generation process).

Baillargeon (1996) provides two insights when it comes to collision events: (1) at first, infants expect any stationary object that collides with a moving object to be displaced by the same amount. However, as they grow older, they are able to take the relative sizes of the two objects into account and understand that the larger the size of the moving object compared to the stationary object, the larger the displacement of the stationary object. (2) Furthermore, at around \(8\) months, infants become subject to a _vertical bias_, meaning that they judge stationary objects as immovable if they have a salient vertical dimension (Wang et al., 2003; 2004).

Figure 5 shows the learning trajectories of the two hypotheses, again by plotting how the percentage of sequences for which the surprise for the violated sequence exceeds that of the expected sequence develops over training epochs and complexity. Empirical research suggests that children first expect a size-independent displacement for all objects. To test whether our models exhibit such a characteristic, we compared their predictions for violated sequences with size-independent displacement (thereby violating physical principles) and expected sequences with a size-dependent displacement (working according to normal physics). While children initially show higher surprise when observing the expected sequences, our models offer a very different picture: at no point do they show more surprise for the expected compared to the violated sequences, as apparent by the plot on the left side of Figure 5.

To test for the vertical bias, we constructed violated sequences where vertical objects do not move upon a collision and expected sequences where they do move according to normal physics. When presented with such sequence pairs, children show more surprise for the expected compared to the violated sequences at some point during their development. However, our models do not exhibit this characteristic at any point in time. Throughout training, they are more sur

Figure 4: The plot on the left shows the percentage of sequences for which the surprise for the violated sequence exceeds that of the expected sequence for the model with \(\beta=1\) at every epoch and for each condition of the occlusion event data set. The lines are smoothed with a uniform kernel of size 10. The plot on the right shows the same metric for fully trained models with different \(\beta\).
prised by the violated compared to the expected sequences. Likewise, we also do not observe this effect when manipulating the representational capacity of our models as shown on the right side of Figure 5. For collision events, we therefore again find that neither the development as stochastic optimization nor the development as complexity increase hypothesis yield learning trajectories that resemble the developmental trajectories of children.

## 5 Discussion

We have compared the learning trajectories of an artificial system to the developmental trajectories of children for three physical processes. For this purpose, we outlined an approach that allowed us to investigate two distinct hypotheses of human development: stochastic optimization and complexity increase.

We found that the learning trajectories under both hypotheses do not follow children's developmental trajectories. For all three event types, we found differences to human learning. For support and occlusion events, the predictions of our models improve at roughly the same rate for all conditions, which indicates that our models do not move along separate stages. For collision events, our models crucially exhibit none of the biases that appear in children. We argue that this is to be expected. The vertical bias, for example, is likely a product of their self-directed movement in the world: as children begin to move around, the majority of vertical objects they encounter, such as walls or furniture, are immovable (Baillargeon, 1996). In contrast to this, our models do not have access to such experiences and are therefore not incentivized to show this bias.

While previous work on modeling cognitive development (Binz and Endres, 2019; Giron et al., 2022) focused on tasks with low-dimensional and static stimuli, our approach employs high-dimensional visual stimuli (e.g., video sequences) and solely relies on an unsupervised training objective. It, therefore, more closely mirrors the actual learning processes of children in the real world. We furthermore extend previous research on building models with human-like physical intuitions by not focusing on adult-level performance but instead investigating developmental trajectories (see again Table 1 for a comparison to previous research).

To showcase how our approach functions as a general framework for testing the learning trajectories of artificial systems, we used a fairly generic generative model. It would be interesting to evaluate the two hypotheses for other model classes, such as generative adversarial networks (Goodfellow et al., 2020) or diffusion models (Sohl-Dickstein et al., 2015) such as a denoising diffusion probabilistic model (Ho et al., 2020). Furthermore, it has been argued in previous work that object-centric representations are crucial for a proper physical understanding of more complex scenes (Piloto et al., 2022). However, our models did not feature explicit object-centric representations and were still able to predict a number of physical processes. Thus, future work should aim for a systematic comparison of models with and without explicit object-centric representations. Additionally, future work should ideally incorporate model-based analyses investigating how changes in the hidden state are related to models' inability to capture the developmental trajectories of children (we have performed a rudimentary first analysis towards this end, see section H in the Appendix).

We used very simple data sets to determine the viability of our approach. Evidently, children do not learn by looking at a large number of stylized sequences. Instead, they observe the real world and generalize their acquired knowledge to a given experimental setting. To capture this process, future research should ideally train models in a similar way. This could, for example, be accomplished by utilizing the

Figure 5: The plot on the left shows the percentage of sequences for which the surprise for the violated sequence exceeds that of the expected sequence for the model with \(\beta=1\) at every epoch and for each condition of the collision event data set. The lines are smoothed with a uniform kernel of size 10. The plot on the right shows the same metric for fully trained models with different \(\beta\).
SAYCam data set (Sullivan et al., 2021). It contains a large number of longitudinal video recordings from infants' perspectives. We believe that using this data set, it might be possible for an artificial model to acquire a vertical bias. It additionally includes time stamps indicating when a child has encountered a particular scene, which could be used to investigate how the nature of the training data influences development. It is also possible that children's pursuit of goals has an influence on their learning trajectories. Maybe this could also be investigated using the SayCAM dataset, however it is possible that collecting new data is required for answering this question.

Finally, the complexity constraint we impose is a constraint on the size of the latent representations of the model. However, it is entirely possible that other parts of children's physical models change in complexity throughout their development. For example, Binz and Endres (2019) implement complexity increase through varying the complexity of model weights instead of the complexity of latent representations. In contrast to our work, they found that the acquisition order of concepts in their model aligned with that of children for support and occlusion events. Besides relying on a different complexity measure, their study diverged from ours in two additional ways: they used a much simpler set of stimuli (2D instead of 3D environments) and they relied on much less sophisticated models (plain feedforward networks instead of our sequential RSSM). We think an important question that needs to be addressed in future work is which of these design choices causes the divergence in results.

What do we make of our results on the whole? On the one hand, they demonstrate that it is possible to use tools developed in psychology to elucidate the inner workings of deep learning models (Ritter et al., 2017; Binz and Schulz, 2022). From this perspective, our work highlights yet another mismatch between human learning and learning in artificial neural networks (Flesch et al., 2018; Dekker et al., 2022). On the other hand, our results also indicate that current modeling approaches are quite far away from implementing Turing's proposal for obtaining a programme that simulates the adult mind. If we want to keep following this direction, we have to therefore ask ourselves what is needed to build models that acquire their knowledge in human-like ways. Towards this end, it is possible that the training data plays an important role, as suggested by some of our results. However, it might be equally plausible that we need to develop new model architectures and come up with more sophisticated ways to train them.

## Acknowledgements

This work was funded by the Max Planck Society and the Volkswagen Foundation under grant number VW98569.