Model-based spatial navigation in the hippocampus-ventral striatum circuit: A computational analysis
[['The neurobiology of goal-directed decisions and planning in the brain is still incompletely known .', 'From a theoretical perspective , goal-directed systems have been often associated model-based reinforcement learning ( MB-RL ) computations [1 , 2]; yet , a detailed mapping between specific components ( or computations ) of MB-RL controllers and their brain equivalents remains to be established .', 'Much work has focused on brain implementations of single aspects of MB-RL controllers , such as action-outcome predictions or model-based prediction errors [3 , 4] .', 'A more challenging task consists in mapping MB-RL computations to a systems-level neuronal circuit that provides a complete solution to decision and control problems in dynamic environments–or in other words , identifying the biological implementation of a complete MB-RL agent rather than only one or more components .', 'The neuronal circuit formed by the ( rodent ) hippocampus and ventral striatum circuit is particularly appealing , and can be productively taken as a “model system” to understand biological implementations of model-based computations during spatial navigation ( see Fig 1A ) .', 'The hippocampus ( HC ) has long been implied in place-based and goal-directed navigation [5] .', 'Recent findings suggest that the role of hippocampus in goal-directed navigation may be mediated by the strong projections from the hippocampal CA1 and subicular areas to the ventral striatum ( vStr ) [6] , which might convey spatial-contextual information and permit forming place-reward associations [7–10] .', 'From a computational perspective , the hippocampus and ventral striatum may jointly implement a model-based controller for goal-directed choice [8 , 11–17] .', 'In this scheme , HC and vStr might be mapped to the two essential components of a model-based reinforcement learning ( MB-RL ) controller [1 , 18]: the state-transition model , which is essentially a model of the task that permits to predict the next location given the current state ( say , a given place ) and chosen action , and the state-value model , which encodes the ( expected ) reward associated to each state , respectively .', 'Different from a model-free controller , a model-based controller can use an explicit form of look-ahead prediction ( or internal simulation ) that permits to imagine future trajectories and covertly evaluate them [1] .', 'Similar look-ahead predictions have been reported in the HC: at difficult decision points , such as when they are at the junction of a T-maze , rodents sometimes stop and produce internally generated sequences of neuronal activity in the HC that resemble the sequential neuronal activity observed in the same area when they navigate through the left or right branches of the T-maze [7 , 20] .', 'These internally generated sequences may serve to serially simulate future spatial trajectories ( e . g . , a trajectory to the left and successively a trajectory to the right ) .', 'In turn , these look-ahead predictions might elicit covert reward expectations in the vStr [9] .', 'By linking spatial locations with reward information , the HC-vStr might thus jointly implement a model-based mechanism that allows an animal to covertly simulate and evaluate spatial trajectories [13 , 21] , using a serial scheme that has some analogies with machine learning algorithms ( e . g . , forward simulations in Bayes nets [22 , 23] or Monte Carlo rollouts in decision trees [24] ) .', 'Internally generated sequences were also reported during sleep or rest periods in the hippocampus and associated structures such as the ventral striatum [25]; in this case , they have been associated with multiple functions such as planning [26] and the off-line “replay” of past trajectories for memory consolidation [27 , 28]–a mechanism that has inspired early ( DYNA [29] ) as well as more recent ( “experience replay” [30] ) machine learning schemes .', 'Here we present a biologically-grounded computational analysis of model-based RL , by comparing side-by-side the behavior and neuronal activity of living organisms ( rodents ) and of a probabilistic MB-RL agent , in cue- and context- conditioning tasks that depend on intact hippocampal and ventral striatal ( shell ) function [19 , 31 , 32] .', 'The results we report indicate that the MB-RL agent replicates the multi-level constraints ( behavioral , neural , systems ) emerged from rodent cue- and context- conditioning studies .', 'First , we show that the MB-RL agent correctly learns to obtain reward in multiple goal locations and develops contextual preferences that are analogous to those of rodents .', 'Second , we show that latent states emerging in the state-transition and state-value models of the MB-RL agent show key coding properties of HC and vStr neurons , respectively .', 'Third , by comparing multiple variants of the MB-RL algorithm , we show that forward ( look-ahead ) predictions—of the kind that can be associated to hippocampal forward sweeps [20]—improve both action selection and learning .', 'This latter result is important as the computational efficacy of model-based schemes based on forward sweeps has been put into question [33] .', 'Taken together , the results of this study speak to the possibility of aligning the MB-RL scheme to the HC—vStr circuit to reproduce both behavioral and neuronal data .', 'Establishing this kind of mappings is particularly important to foster cross-fertilizations between reinforcement learning and neurophysiology; for example , to identify specific algorithmic solutions that the brain uses to face problems that are still challenging in machine learning , or conversely , to advance novel computationally-guided theories of neural processing [10 , 34] .', 'Finally , our MB-RL scheme has two main features that go beyond the state of the art from a computational perspective , and which are necessary for an adaptive agent to deal with open-ended situations .', 'First , the MB-RL agent successfully faces the challenge to learn simultaneously three components: ( the latent categories that form ) the state space , the state-transition and the state-value models .', 'Our novel scheme that combines non-parametric and reinforcement learning ensures that only the latent categories that afford accurate state-transition and state-value functions are retained , thus linking inextricably the three learning processes .', 'Second , the MB-RL agent can deal with multiple goals in a native way , without the necessity of re-learning .'], ['The maze used in the animal study and in our simulations is a y-shaped symmetric arena consisting of 3 identical square chambers rotated 120 degrees from each other and connected through a central triangular passage , see Fig 1B .', 'Each chamber contained three goal locations located along the chamber walls , where reward was ( probabilistically ) delivered .', 'Each reward location had a cue light above it .', 'Our simulation follows the protocol used in the animal study [19] and consists of three phases .', 'In the first phase ( Cue: cue conditioning ) , the correct goal location was cued with a light; reward was delivered if the animal reached the goal location and made a nose-poke ( but note that we do not simulate the nose poke here ) .', 'In the second phase ( Context: contextual conditioning ) , the correct goal location was cued with a light , too; but the probability of reward ( following a nose poke ) depended on the chamber’s spatial position: south room goals brought reward in 75% of the trials , whereas the other goals brought 25% reward .', 'Finally , a context-conditioning test ( CX test ) was performed , with no cues or rewards ( i . e . , "free run" ) , to probe the animals’ acquired contextual place preferences .', 'The key behavioral result of the animal study , which we aim to reproduce in the MB-RL agent model , is that the contextual preference for the south room goals is preserved in the last ( CX test ) phase .', 'We also aim to test whether the internal representations acquired by the two key components ( state-transition and state-value model ) of the MB-RL agent during learning , have coding properties that resemble HC and vStr activations , respectively , in the animal study .', 'The computational model ( MB-RL ) is fully explained in the Methods section; however , we shortly summarize it here for the sake of the reader .', 'The model essentially includes three interconnected components: a latent state categorization mechanism , which permits to learn the state representations that are useful to solve a task ( e . g . , place cells ) ; a state-transition model , which learns the contingencies between actions and future states ( e . g . , what location do I reach by going left ? ) and a state-value model , which learns the utility of the states ( e . g . , is reaching location x good or bad ? ) .', 'There are three important aspects of the model to note ( see the Methods section for further details ) .', 'First , we assume that state-transition and state-value models correspond to HC and vStr , respectively , and they jointly permit to steer look-ahead predictions ( analogous to hippocampal forward sweeps [20] ) .', 'As we will discuss below , our simulations will compare different versions of the same MB-RL agent , which use , or not use , look-ahead predictions for learning , control ( action selection ) or both .', "Second , the MB-RL agent includes a mechanism that adaptively selects whether or not to do a sweep , and the depth of the sweep , depending on the agent's uncertainty and confidence about the choice .", 'Essentially , the depth of sweeps decreases when the agent becomes sufficiently confident about its choice .', 'However , a change in reward contingencies ( e . g . , the location of reward ) would produce a reversal of this effect—and the generation of longer sweeps .', 'This is because after failing to reach a reward at the expected location , the agent would become again uncertain about its choice , hence triggering sweeps to minimize it before a choice [23] .', 'Here we are interested in validating the computational efficiency of this mechanism , which may potentially explain why VTE behavior increases rapidly when animals make errors following a switch in reward contingency [20] .', 'Third , most model-based planning algorithms start from a state space that is predefined by the programmers .', 'However , the brain has to simultaneously learn internal models to generate predictions , and the states to be used by the model .', 'In keeping , in the MB-RL agent , the learning processes required to acquire latent states and the ( state transition and state value ) models that use the latent states are interdependent ( see Eq 1 below ) .', 'Here we are interested in validating this learning scheme , which permits learning latent states that not only entail high perceptual accuracy but also afford good prediction and control when used by the state-transition and state-value models .', 'Successful context conditioning was evaluated in [19] with a CX test: a “free-run” target preference behavioral test in which the rats freely explored the maze for several minutes without any cue or reward .', "During this period , to probe the animals' learned preferences , statistics were collected about their visitations of ( and “nose pokes” in ) rooms that in the previous Contextual Conditioning phase were associated with high- and low-reward probabilities .", 'Analogously , to test the MB-RL agent’s learned preferences , we ran a free-run simulated test for 2 , 000 time steps , in which the agent started from a small central area ( 3x3 units ) and with random orientation .', 'In this simulation , action selection maximized the likelihood for ( expected ) reward pooled from all the 9 goal locations; and the agent did not learn .', 'To better clarify the behavioral effects of conditioning in the MB-RL agent , we performed the same free-run test ( and without learning ) also at the end of cue conditioning .', 'Finally , and importantly , we envisaged to compare different versions of the same MB-RL agent that uses , or not uses , look-ahead predictive mechanisms ( analogous to hippocampal forward sweeps [20] ) for both control / action selection and ( reward ) learning—in order to assess the computational importance of these mechanisms .', 'Specifically , we compared four different versions of the MB-RL agent using a 2x2 design , which considers 2 controllers ( i . e . , using or not using forward sweeps ) and 2 learning procedures ( using or not using forward sweeps ) ; see Section 4 for details .', 'We expected that look-ahead prediction ( or forward sweeps ) to provide the agent significant advantages for both control / action selection and ( reward ) learning; this result would be particularly intriguing for action selection , given that the baseline controller uses the full factorized distribution of reward and is thus computationally demanding .', 'The results shown in Fig 2 confirm our hypotheses , by showing an additive advantage of forward sweeps in both action selection and learning .', 'As shown in Fig 2A , the MB-RL agent that uses forward sweeps in both action selection and learning ( swControl+Reward ) , with average success of μ = 69% ( σ = 2 ) during the entire Cue Conditioning phase outperforms in amount of collected reward the agent that uses forward sweeps only for control ( swControl , μ = 62% ( σ = 3 ) , t ( 1 , 9 ) = 6 . 9 , n = 10 , p<0 . 001 ) , the agent that uses forward sweeps only for learning ( swReward , μ = 39% ( σ = 2 ) , t ( 1 , 9 ) = 33 . 4 , n = 10 , p<0 . 001 ) , and the agent that lacks them during both control and learning ( Baseline , μ = 36% ( σ = 1 ) , t ( 1 , 9 ) = 39 . 6 , n = 10 , p<0 . 001 ) and this advantage is evident from the very beginning of learning .', 'For all the four agents , performance decreases after the vertical dotted bar , in correspondence of the Contextual Conditioning phase—but this is expected , given that less reward is available in this phase .', 'Fig 2B illustrates the same advantage in overall path length needed to reach the reward site of the four agents .', 'Along with the increased performance , the length of sweeps used for control by the swControl+Reward and swControl agents decreases with time ( Fig 2C ) .', 'This is because the agent’s decision certainty progressively increases ( Fig 2D ) as their state-value model becomes more effective; in turn , the mechanism for sweep length control explained in the Methods Section decreases length ( we obtained equivalent accuracy and path length results using a controller having a constant length of 9 and thus requiring more computational results; results not shown ) .', 'This result aligns well with evidence that hippocampal forward sweeps at decision points progressively diminish during learning [20]—until they disappear when the animal develops a habit ( which we do not model here , but see [13] ) .', 'As the above analysis confirmed the advantages of the MB-RL agent that uses forward sweeps in both action selection and learning ( swControl+Reward ) , we used this MB-RL agent for the rest of our behavioral and neural analyses .', 'Our target rodent study reported that during "free run" in the CX test phase , animals showed an increased preference for the room that yielded more rewards in the previous ( Context ) phase [19] .', 'The MB-RL agent correctly reproduces this pattern of behavioral results .', "As shown on Fig 3A , in the CX test phase ( in the absence of reward ) , the agent shows a significant behavioral preference ( number of reward-site visits ) for the room that was previously most rewarded ( room1 vs . room3 , t ( 1 , 9 ) = 4 . 3 , n = 10 , p = 0 . 002; room2 vs . room3 , t ( 1 , 9 ) = 7 . 6 , n = 10 , p<0 . 001 ) , which matches the animal's learned preference ( and differed from an analogue test executed right after cue conditioning: room1 vs . room3 , t ( 1 , 9 ) = 1 . 5 , n = 10 , p = 0 . 16; room2 vs .", '–room3 , t ( 1 , 9 ) = 0 . 1 , n = 10 , p = 0 . 92 ) .', 'Fig 3B provides three representative trajectories before ( blue trajectories ) and after ( red trajectories ) context conditioning–the latter exemplifying the increased preference for the room that included highly rewarded locations in the Context phase ( black dots ) .', 'To further establish a parallel between the MB-RL agent and the HC—vStr circuit , we analyzed the content of the latent states that emerged in the agent’s transition model PM ( s′|s , a ) and value function PV ( r|g , s ) at the end of the Cue and Context Conditioning phases .', 'We decoded the spatial location corresponding to each latent state and created:', '( a ) value maps for each target showing for each spatial location the greatest reward across all head-directions and', '( b ) transition maps showing how position could change after applying an ( y ) action at any orientation .', 'We also analyzed the effect of conditioning at the neural level: how the learned internal models change and how this change affects behavior .', 'The neuronal analysis of HC and vStr cells after the cue conditioning task reported different coding characteristics in the two areas , with the former showing high spatial selectivity and the latter showing selectivity for reward- and task-related information , and place/reward combinations [19] .', 'The two components of the MB-RL agent acquire analogous coding preferences after conditioning .', 'Fig 4 shows the transition probabilities learned by the state-state ( transition ) model after Cue Conditioning ( Fig 4A ) and how they changed after Context Conditioning ( Fig 4B ) .', 'In Fig 4 , each green square corresponds to a place in the maze ( i . e . , the central square corresponds to the central place of the maze ) and the colors of the smaller squares code the greatest probability of transitioning from the location of that state , regardless the direction , to the location of any other state of the maze ( shown are only the possible successors , i . e . , nearby locations ) , by executing any of the 3 actions available to the agent ( see Methods section ) .', 'The more red the color of the cell , the higher the probability .', 'In other words , this map shows the spatial projection of the learned "successor states" ( small squares ) of each latent state ( green squares ) implicitly coded in the agent\'s probabilistic transition model—analogous to "successor representations" [35 , 36] .', 'For example , the green squares near the center include transitions to all directions while the green squares on the borders only include transitions towards the center .', 'In other words , the learned transition probabilities encode the actual transitions available in the environment , reflecting the idea of HC encoding a navigational "cognitive map" [5] .', 'Furthermore , importantly for our analysis , these codes only change ( remap ) to a minor extent between the Cue and Context Conditioning phases ( Fig 4B ) —in keeping to what was reported empirically [19] .', 'Essentially , the changes reflect a consolidation of the same transition model , not a remapping , due to additional training during the Context conditioning phase .', 'Fig 5 shows the value of each place ( i . e . , the learned probability of obtaining reward in the place ) when a goal location is selected or cued , as learned by the state-value function component of MB-RL agent after Cue Conditioning ( Fig 5A ) and how these values changed after Context Conditioning ( Fig 5B ) .', 'The nine outer big Y-shapes represent the possible goal sites; inside each of them , the colors code the probabilities of obtaining reward while starting from any specific location in the maze—the darker red the color , the higher the probability .', 'The central Y-shape represents the combined value function across all goal sites , which is effective when the goal is not cued , as in the behavioral CX test .', 'These probabilities relate well with key coding properties of vStr neurons in [19] .', 'If one assumes that the current goal is unknown , as in the CX-test , one can see the same rotational invariance in the model ( Fig 5A , the central Y-shape ) as in sample vStr neurons ( Fig 5C; for more samples , see [19] ) .', 'This aligns well with the idea that vStr neurons might encode place-reward associations and these in turn can be conditioned on a specific goal , possibly provided by prefrontal areas [37] or vicariously from hippocampal forward sweeps [21 , 38] .', 'Note that the probabilities emerging in the state-value model are closely related to reward functions of RL , which can be used to learn a policy from the current state [1] and , in a model-based scheme , to retrieve covert expectations of reward using mental simulation or forward sweeps [13 , 21 , 23] .', 'As usual in RL , the value of reward places is temporally discounted such that places farther from goal sites have lower values .', 'This creates a sort of gradient or "tessellation" of the task , which might reflect not only proximal distance but also other information such as phases or subtasks that are necessary to secure a reward [19] .', 'Importantly , the coding of value in the state-value model changes drastically after Contextual Conditioning ( Fig 5B ) .', 'While after Cue Conditioning one can observe full rotational symmetry , after the Context Conditioning a marked preference is evident for the goal locations in the most rewarded ( lower; south ) room—which also explains the behavioral results of Fig 2 .', 'This change of preference can be better appreciated if one considers the distribution of the change of values associated with latent states in the state-value model after Contextual Conditioning ( Fig 6 ) .', 'Specifically , there is a decrease of all the state values ( given that reward is less frequently available ) , but the value of states corresponding to the less rewarded rooms ( blue , mean change of -0 . 29 across all learners ) decreases significantly more than those of the more rewarded rooms ( red , mean change of -0 . 17 across all learners; t ( 1 , 9 ) = 25 . 0 , p<0 . 001 ) .', 'This result is coherent with a body of literature implying vStr in the coding of reward expectancies [21] and would imply a sensitivity for changing reward contingencies ( see Discussion ) .', 'This is in sharp contrast with the states in the state-state transition model ( black ) , which essentially does not change ( mean change of 0 . 01 across all learners ) .', 'Finally , Fig 7 provides analysis of the sweeps for action selection generated during the two conditioning phases .', 'During Cue Conditioning , sweep length increases at decision points , e . g . , the center of the maze and nearby densely located targets ( Fig 7A ) .', 'This simulates the rodent data that consistently show longer internally generated sequences at branch points [15] , [23] .', 'In turn , Fig 7B shows how sweep length changes after Context Conditioning .', 'Notably , when the agent is in a location that is far away from targets with low reward probability , it needs longer sweeps to accumulate sufficient evidence about the most valuable action .', 'Instead , sweep length remains the same or decreases for highly rewarded targets .'], ['Computational methods of reinforcement learning have been widely used to analyze neuronal data [2 , 39–41] .', 'However , this has been done far more systematically for model-free RL–mostly associated to habitual behavior—than for model-based RL methods [22 , 23 , 42]–more associated to goal-directed behavior and planning .', 'A challenge consists in mapping the components of a MB-RL agent to a neural circuit that offers a complete , systems-level solution to goal-directed choice , rather than a single aspect of it ( e . g . , the dynamics of dopaminergic neurons and their relations to reward prediction error ) .', 'Rodent research suggests that the neuronal circuit formed by the hippocampus ( HC ) and ventral striatum ( vStr ) might implement model-based computations in spatial navigation .', 'In this scheme , the HC learns a state-transition model ( permitting to predict future states ) and the vStr learns a state-value model ( encoding goal-dependent state values ) .', 'The HC might then predict future trajectories ( e . g . , using forward sweeps and ripple-associated replay ) and vStr might covertly evaluate them , permitting to select the best ( that is , reward-maximizing ) course of action [12 , 13 , 21] .', 'To provide a computationally-guided analysis of this idea , here we designed a Bayesian model-based reinforcement learning ( MB-RL ) agent and tested it in a challenging contextual conditioning experiment that is dependent on hippocampal and ventral striatal function [19] .', 'The results of our simulation show that the MB-RL agent procedure can reproduce animal data at multiple—behavioral , neural , systems—levels .', 'At the behavioral level , the agent shows place preferences that are analogous to rodents [19] .', "Key to this result is the fact that the agent's value function is conditioned on goal information .", 'Here , in other words , goal information acts as a “context” and permits the agent to learn multiple goals , as required by the experimental set-up [19] .', 'This possibility is precluded to other RL controllers that are only optimized for a single value function ( or goal ) and thus lack context sensitivity [1] but is important to scale up RL methods to real-world situations that involve multiple potential goals .', 'At the neural and systems levels , the agent develops internal codes that reproduce key signature of hippocampal and ventral striatal neurons [19] .', 'This is important to establish deep relations between formal methods and neurophysiology that potentially span multiple levels of analysis—from computational to algorithmic and implementational [43] .', 'Finally , in our proposed model , hippocampal sweeps can contribute to both action selection and learning .', 'Our comparison of multiple MB-RL schemes shows that mechanisms of look-ahead prediction that resemble hippocampal forward sweeps improve both control and learning .', 'There has been some recent skepticism around the idea of using look-ahead prediction for control and learning , based on the fact that prediction errors tend to accumulate over time [33] , which would cast into doubt the idea that hippocampal sequences may serve predictive or planning roles .', 'Furthermore , alternative proposals have cast model-based control ( and hippocampal function ) in terms of backward [42] or a combination of forward and backward inference [22] .', 'Our results suggest that the limited form of look-ahead prediction that we adopt is computationally advantageous , thus lending computational-level support for theories that assign forward sweeps a predictive role [44 , 45] .', 'The MB-RL agent is related to other computational models of model-based spatial decisions [22 , 23 , 46–52] that use various forms of forward planning .', 'Different from these architectures , the MB-RL agent includes a nonparametric module that learns a “state space” in an unsupervised manner while the agent learns the task , and which works synergistically with the other components of the architecture .', 'Indeed , the state space representations emerging from the nonparametric learning procedure ensure that only states that afford good prediction and control are retained .', 'Using end-to-end learning ( i . e . , from perception to action ) helps keeping the various learning procedures ( state space , action-state and state-value learning ) coordinated and is more effective and biologically realistic than using a staged approach—still popular in RL—in which unsupervised state space learning is seen as a generic preprocessing phase .', 'From a computational perspective , this approach can be considered to be a novel , model-based extension of a family of Bayesian methods that have been successfully applied to decision-making problems [53–57] .', 'It is also important to note that our approach does not require learning a separate state space ( or sensory mapping ) for each goal; rather , multiple spatial goals share the same state space—which implies that our MB-RL agent can deal natively with multiple goals .', 'This is different from most current deep RL approaches , which require learning a new state space for each problem or task ( but see [1] ) .', 'In our set-up , sharing the state space is effective because all goals correspond to spatial locations—or in other words , they all lie in the same low-dimensional state space that supports spatial navigation .', 'This reliance on a relatively low-dimensional state space makes spatial navigation quite special and tractable , both in machine learning and the brain; it is thus possible that brain areas that support goal-directed processing outside navigation domains rely on more complex computational solutions that remain to be fully established [10 , 58 , 59] .', "At the neurophysiological level , the latent states that emerged in the agent's state-state and state-value models during conditioning share important similarities with HC and vStr coding , respectively—and in particular the selectivity for space vs . reward-predictive information and place/reward combinations , respectively .", 'Our computational model thus provides a detailed ( yet to some extent abstract ) mapping between specific components of model-based controllers and neuronal circuits .', 'Establishing this sort of mappings is important if one aims to the cross-fertilization of current research in reinforcement learning / artificial intelligence and neurophysiology; for example , to probe in detail neuronal solutions to challenging machine learning problems .', 'Indeed , one should expect that the neuronal networks in the HC and vStr ( as well as other brain structures ) have been optimized over time to solve the challenging problems that animals must face every day , and cracking this neural code might be helpful to design artificial agents with similar abilities .', 'In this respect , a limitation of the model proposed here concerns the simplifying assumptions on HC and vStr coding and their respective roles in model-based control .', 'For simplicity , we mapped one-to-one HC to state-transition and vStr to state-value models .', 'However , this sharp division is likely to be simplistic and both neural coding and computational roles of the HC-vStr circuit are certainly more complex .', 'HC cells show some sensitivity to goal and reward information [60] and remap from cue-on to cue-off and spontaneous goal site approach [19] .', 'Furthermore , vStr can code information that goes beyond scalar reward expectations; it can code for example item- or object-specific information , such as the expected reward associated to a specific food or other characteristics of a task , including ( for example ) intermediate steps to goal locations [19] .', 'This suggests a more complex architecture in which HC and vStr do not map one-to-one to components of a model-based controller but may realize model-based computations in a more distributed manner; and in which they form a latent task model that encodes task variables that are more abstract than spatial codes , permitting “tessellating” tasks into behaviorally meaningful events ( e . g . , task phases that lend to reward delivery ) and forming hierarchies .', 'Understanding how the brain tessellates and organizes the state space hierarchically may be particularly important for the success of future model-based reinforcement learning algorithms .', 'Current model-based RL methods suffer from the problem that , during long look-ahead searches , errors sum up; for this , in practice , most practical applications use model-free solutions .', 'However , in principle , endowing model-based methods with the ability to exploit relevant task structure ( e . g . , milestones or subgoals ) may mitigate this problem , by making forward search more abstract and hierarchical ( or “saltatory” ) [61–63] or by avoiding chaining too many consecutive predictions [33] .', 'To this aim , unsupervised state representation methods ( including deep learning methods ) that are able to identify latent task structure may be productively incorporated into model-based control systems—yet it remains to be studied in future research how to handle or bound ( hierarchical ) state representations that may solve realistic problems [36 , 64–68] .', 'An alternative possibility consists in le
['Introduction', 'Results', 'Discussion', 'Methods']
['While the neurobiology of simple and habitual choices is relatively well known , our current understanding of goal-directed choices and planning in the brain is still limited .', 'Theoretical work suggests that goal-directed computations can be productively associated to model-based ( reinforcement learning ) computations , yet a detailed mapping between computational processes and neuronal circuits remains to be fully established .', 'Here we report a computational analysis that aligns Bayesian nonparametrics and model-based reinforcement learning ( MB-RL ) to the functioning of the hippocampus ( HC ) and the ventral striatum ( vStr ) –a neuronal circuit that increasingly recognized to be an appropriate model system to understand goal-directed ( spatial ) decisions and planning mechanisms in the brain .', 'We test the MB-RL agent in a contextual conditioning task that depends on intact hippocampus and ventral striatal ( shell ) function and show that it solves the task while showing key behavioral and neuronal signatures of the HC—vStr circuit .', 'Our simulations also explore the benefits of biological forms of look-ahead prediction ( forward sweeps ) during both learning and control .', 'This article thus contributes to fill the gap between our current understanding of computational algorithms and biological realizations of ( model-based ) reinforcement learning .']
['Computational reinforcement learning theories have contributed to advance our understanding of how the brain implements decisions—and especially simple and habitual choices .', 'However , our current understanding of the neural and computational principles of complex and flexible ( goal-directed ) choices is comparatively less advanced .', 'Here we design and test a novel ( model-based ) reinforcement learning model , and align its learning and control mechanisms to the functioning of the neural circuit formed by the hippocampus and the ventral striatum in rodents—which is key to goal-directed spatial cognition .', 'In a series of simulations , we show that our model-based reinforcement learning agent replicates multi-level constraints ( behavioral , neural , systems ) emerged from rodent cue- and context- conditioning studies , thus contributing to establish a map between computational and neuronal mechanisms of goal-directed spatial cognition .']
['learning', 'medicine and health sciences', 'decision making', 'nervous system', 'brain', 'social sciences', 'vertebrates', 'neuroscience', 'learning and memory', 'animals', 'mammals', 'simulation and modeling', 'cognitive psychology', 'systems science', 'mathematics', 'cognition', 'research and analysis methods', 'computer and information sciences', 'behavior', 'neural pathways', 'agent-based modeling', 'neostriatum', 'behavioral conditioning', 'psychology', 'hippocampus', 'eukaryota', 'rodents', 'neuroanatomy', 'anatomy', 'biology and life sciences', 'physical sciences', 'cognitive science', 'amniotes', 'organisms']