Deep predictive coding networks are neuroscience-inspired unsupervised learning models that learn to predict future sensory states. We build upon the PredNet implementation by Lotter, Kreiman, and Cox (2016) to investigate if predictive coding representations are useful to predict brain activity in the visual cortex. We use representational similarity analysis (RSA) to compare PredNet representations to functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data from the Algonauts Project (Cichy et al., 2019). In contrast to previous findings in the literature (Khaligh-Razavi & Kriegeskorte, 2014), we report empirical data suggesting that unsupervised models trained to predict frames of videos without further fine-tuning may outperform supervised image classification baselines in terms of correlation to spatial (fMRI) and temporal (MEG) data. CORnet is the best among supervised models CORnet-S, the current best model of the visual brain according to the Brain-Score benchmark , outperforms both AlexNet and ResNet-50 on the 92-images dataset, except for MEG early interval, which is best explained by ResNet-50 (block2) ( Table 3) . On the 118-images dataset, CORnet-S (V2) is also the best model for fMRI but is outperformed by AlexNet (conv2) on MEG data. These results are consistent with the Brain-Score benchmark and suggest that CORnet-S is a competitive model overall despite being outperformed on data from specific cortical areas such as V4 (Kubilius et al., 2019) . Predictive coding yields representations similar to brain data The predictive coding model used in our experiments does not rely on labeled data and learns just by minimizing the error between predicted and actual sensory data. Most importantly, it is trained on a dataset of videos portraying activities such as cooking, walking, and dancing, which differ significantly from the carefully curated 92 and 118-images datasets used to measure brain activity. Despite those differences, the model trained on just 1 hour of videos from the KITTI dataset and 7 million parameters outperforms an AlexNet model (≈ 61 million parameters, see comparison in Table 2 ) in terms of correlation to both fMRI and MEG data (92-images dataset, Table 3 ) and fMRI data (118-images dataset, Table 4 ). Table 2 : The models used in the experiments. Layer count is given by the "number of convolutions and fully-connected layers along the longest path of information flow" Scaling unsupervised learning improves correlation with brain data The noise normalized correlation scores (Tables 3 and 4) shows that learning more about how events unfold over time improves correlation scores. Correlation to brain representation continues to improve as we train the PredNet-4 and PredNet-5 models with up to 6 hours of videos from the Moments in Time dataset, except for IT scores, which do not exhibit a clear trend. Since PredNet-5 input has higher dimensionality (256 × 256) and more layers, more experiments would be needed to separate the impact of the amount of training data and model size on the correlation scores. Why PredNet may be better than competing supervised baselines? The PredNet architecture fulfills several of the desired properties a brain-like model should have, such as few layers, single canonical circuitry in all areas, and recurrency . Additionally, PredNet is designed to process spatiotemporal sensory data, such as sequences of video frames or audio spectrograms, which we argue is another crucial requirement for models that approximate the brain architecture. The use of recurrence in CORnet and other convolutional models has been shown to capture neural dynamics, but still, they cannot "fabricate" real-world dynamics from static images. For instance, it is known that clustering of animate/inanimate objects is captured from neural activity in the IT area (Khaligh-Razavi & Kriegeskorte, 2014) . We believe the emergence of those semantic divisions requires experiencing how the objects/entities behave in space and time.
We show empirical evidence that predictive coding models yield representations more correlated to brain data than supervised image recognition models.