Cortical and Hippocampal Correlates of Deliberation during Model-Based Decisions for Rewards in Humans
[['Model-based decisions stand in contrast to a simpler sort of learned decision making whose neural instantiation is better understood: simply learning to repeat rewarded behaviors [4]–[6] .', 'To explain the former , more knowledge-driven path to decisions , researchers have long argued that the brain maintains internal representations of the contingency structure of a task — a “world model” or , in spatial tasks , a “cognitive map” — that can be adaptively applied to drive behavior .', 'Like a map of space , these representations describe the relationships between situations and actions , separate from any ties to reward .', 'The reliance on these representations is a defining characteristic of goal-directed decisions [1] , [2] .', 'Therefore , to identify the neural mechanisms of these decisions , researchers must first identify the representations that guide them .', 'Here , to examine in detail the process by which contingency representations are learned and inform action choice , we combined a sequential learning task [7] with an interleaved decision task in which rewards depended on contingencies learned in the first task .', 'In the learning task , participants were presented with one of four photograph images at a time , and asked simply to press the key corresponding to that image .', 'Which of the four images appeared next depended , probabilistically , on the image currently being viewed .', 'The sequential learning task allowed us to measure the gradual , trial-by-trial , acquisition of these probabilistic contingencies linking the four image stimuli .', "Participants' responses provided two observable measurements of learning: reaction time to identify each image , and image-specific BOLD activity in the ventral stream visual cortex .", 'Reaction times to identify an image indicated the degree to which subjects expected it , given the previous one — a classic and relatively direct measure of the learned predictive association [8]–[12] — and category-specific BOLD also reflected engagement of the neural representation of each image in anticipation of its presentation [13] .', 'By fitting computational models to this progression of subject expectations , we extracted a computational signature of the learning process , the learning rate , and used it to generate timeseries of decision variables based on these learned contingencies .', 'This enabled us to quantitatively characterize the influence of these associations when participants were asked , in the interleaved decision probes , to draw on them to make decisions .', 'Specifically , participants were told that one of the four images was , for a short period of time , to be associated with a reward .', 'They were then asked which of two other images would lead to that rewarded image as quickly as possible .', 'This manipulation has a form similar to a latent learning paradigm [14] , [15] , in which contingencies are learned separately from their link to reward .', 'By requiring subjects to use knowledge of the contingencies to guide their decisions , this design allows us to probe how and whether the contingencies are used to seek trial-specific goals — contingencies that are exclusively the realm of model-based decision processes .', 'Comparing the learning rates fit to behavior and BOLD responses we observed a striking match between hippocampal correlates of sequential learning and the learning underlying the reaction times , choices , prediction errors , and ventral visual stream activity , during both simple identification responses and deliberative decisions for reward .', 'These results suggest that regions involved in sequential learning , including hippocampus and ventral cortical areas , indeed provide the necessary contingency representations to support model-based choice — and , critically , demonstrate the use of particular associations learned by these regions during model-based decision making .'], ['We next identified neural correlates of each learning process .'], ['We are able to compare learning across different task phases ( learning and choice ) and sorts of measurements ( reaction times , choices , and BOLD correlates of different quantities ) by treating them all as different windows on a computational learning process .', 'We fit each sort of data with a standard computational model of how predictions are learned from recent experience , and compare the learning rate parameters that best explain these measurements .', 'The pattern of data in Figures 3 and 5 and Table 1 shows a striking consistency in these estimated learning rates between the different measurements .', 'However , there are a number of caveats to keep in mind about these analyses .', 'First , it is in principle not possible to conclude that any two of these learning rate estimates are “the same” as one another — only that they are not statistically distingishable .', 'But this pattern of negative findings is supported by positive ones , for instance that the differences between the various manifestations of “slow” and “fast” learning rates are significant ( Table 1 ) .', 'Also , our findings that apart from exhibiting similar learning rates , neural activity during choice and decisions implicate common neural structures support the interpretation that all this activity relates to a common underlying learning process .', 'Ultimately , however , establishing a definitive link between activity during learning and choice will require additional work using methods that can probe causal relationships between brain function and behavior .', 'A related point is that the estimates of learning rates from BOLD in Figure 5 consistently tend to be less extreme than their behavioral counterparts , i . e . slightly slower relative to the fast learning rate and faster relative to slow .', 'In a couple of cases , this difference between BOLD and behavioral estimates is significant , seeming to contradict the interpretation that all these measurements reflect a common learning process .', 'We believe this relates to another important set of caveats with this study , which is that it is methodologically challenging to estimate learning rates from BOLD data due to the nonlinear relationship between the learning rate and the decision variables that have BOLD correlates ( entropy , etc . ) .', 'To permit estimation , we approximate this relationship as linear using a first-order Taylor expansion [7] , [25] , [39] .', 'This allows us to estimate the learning rate in the context of the same standard fMRI analysis ( using a general linear model ) as the rest of our results , and in turn means these analyses cope in the standard ways with the many methodological complications of fMRI ( including for instance intersubject random effects , temporal and spatial autocorrelation , hemodynamics , and regressor colinearity ) .', 'This method appears to perform robustly in this and our previous study [7] and other closely related analyses of parametric brain-behavior relationships [38] , [40] , [41] , but there has not yet been a formal simulation study quantifying the error introduced by this approximation .', 'One key sort of approximation error that we have examined [7] arises from our choice of the midpoint between fast and slow learning rates as the point around which to linearize .', 'We choose this point to minimize the distance between the linearization point and the hypothetically relevant learning rates , since the error from linear extrapolation is expected to accumulate with distance .', 'However , this choice interacts with the way we identify voxels of interest for fitting the learning rate , by identifying peaks in activity assuming this midpoint learning rate .', 'Intuitively , this selection biases the estimated learning rates toward this midpoint ( see our previous study using this approach for a more thorough technical explanation [7] ) .', 'Although this effect is innocuous with respect to the conclusions in this article , it may account for some of the observed difference between neural and behavioral estimates in Figure 5 .', 'Our choice task has one of the key features of a latent learning task [15]: sequential contingency learning precedes the introduction of a new and unpracticed rewarding goal .', "In particular , given the sparse occurrence of the choice probes , and the different combinations of rewarded and starting images , these decisions implicate a model-based response strategy requiring participants to evaluate options' chances of reaching the new goal based on the predictive associations being continually learned in the sequential image presentation trials .", 'Conversely , choices of this sort leave little room for model-free reinforcement learning based only on the success of particular choices at earning money in previous choice trials .', 'Consistent with this , a key neural player in both the learning and decision phases in our results is the hippocampus .', 'The hippocampal system is associated with flexible memory for stimulus-stimulus relations [42]–[44] and is a longstanding candidate for maintaining contingency structure in the service of goal-directed decisions [2] , [19] , [45]–[48] .', 'In part , these suggestions are based on the analogy with spatial tasks , in which it has long been argued that the hippocampus implements a cognitive map [49] , [50] .', 'A suggestive connection of these ideas to nonspatial tasks is ubiquitous findings that the the hippocampal system is implicated in acquired equivalence , transitive inference , and sensory preconditioning effects [41] , [51]–[53] , as well as the flexible use of conceptual [54] and structured [55] knowledge .', 'All of these effects demonstrate a bias in novel choice probes caused by previously learned stimulus-stimulus relations .', 'Model-based decision making relies on a similar ability to flexibly chain together or recombine associations in novel ways , as exercised in latent learning tasks like our choice probes here .', 'Accordingly , we hypothesized that participants would draw on hippocampally-linked contingencies to make decisions .', 'Indeed , the learning rates that best explained both choices and BOLD signals during the decision trials were not distinguishable from those seen in hippocampus and nearby ventral stream visual cortex during sequential responding , while differing significantly from those seen in BOLD activity in caudate and the fast process in reaction times .', 'This quantitative convergence between learning processes examined during different tasks and through the lens of different observables substantiates the idea that model-based decisions and incidental stimulus-stimulus learning , like other sorts of relational learning and transfer [41] , [53]–[55] are supported by the same hippocampal memory system .', 'Interestingly , the literature concerning these tasks suggests what appear to be two distinct ( but potentially complementary ) mechanisms supporting the flexible transfer of relational knowledge to novel probes .', 'Some studies have demonstrated that better performance on transfer probes is predicted by hippocampal BOLD activity at learning but not test time [53] , [56] suggesting that transfer is somehow supported by processes that occur already during encoding .', 'One hypothesis is that such activity reflects the immediate transfer of learning , when information is first obtained , to other related associates by a process of spreading activation .', 'In other studies [54] , [55] , neural activity at probe time also related to correct performance or with the relational information itself .', 'This suggests the importance of processes occurring at the time of retrieval , and is consistent with theories ( as in the standard account of model-based RL ) that transfer is supported by some sort of active inference , planning or search at the time of the novel choice .', 'Our result ( discussed further below ) that hippocampal activity tracked the difficulty of the decision probes speaks to the latter mechanism , providing relatively direct evidence that the hippocampal system engages in more computation for harder transfer problems ( see also Simon & Daw [57] ) .', 'Altogether , these two distinct but complementary mechanisms appear to be each well supported across the literature , and could plausibly both contribute in different circumstances .', 'The type of model-based decision making studied here contrasts with “model-free” habit learning , of the sort associated with dorsolateral striatum [58] , predominant temporal-difference learning accounts of reward prediction error signal seen in dopamine neurons [6] , and the striatal BOLD response [29]–[31] .', 'That said , parts of striatum are clearly necessary for model-based decision making in rodents as well [59] , [60] .', 'Perhaps related , in human neuroimaging , even reward prediction errors observed in ventral striatum — though often characterized as reflecting the teaching signal for model-free stimulus-response learning — have recently been shown to report information about the state-state or relational structure of a task that would be known only to a model-based system [38] , [41] .', 'This may suggest some crosstalk between model-based and model-free learning in the brain .', 'The reward prediction errors in the decision phase of the present task are consistent with these results , in that they reflect stimulus-stimulus predictions combined with trial-specific rewards to which a purely model-free reinforcement learner would be blind .', 'The present results also extend these findings by showing that the stimulus-stimulus learning rate driving these prediction error effects matches that from the hippocampal system during the sequential response task , suggesting all these are indeed driven by a common learning process .', 'During the sequential response task , activity was not observed in the ventral striatal region commonly associated with reward prediction errors .', 'This may reflect the lack of overt reinforcement in this more implicit association task .', 'Instead , activity in a more dorsal/posterior region of striatum reflected a transient ( high learning rate ) adaptation process , which also had separate correlates in reaction times .', 'We speculate that this activity ( and the associated component of the reaction times ) may reflect a second process of response learning , which did not carry over into the decision task .', 'Indeed , the stimulus sequence in serial reaction time tasks of the sort we use is accompanied by an equivalent motor sequence ( of button presses ) , leading previous authors to suggest [61]–[63] that participants might learn either or both of two distinct types of sequential associations: stimulus-stimulus and response-response .', 'That these processes then are uniquely tied to separate brain systems — hippocampus and striatum — suggests that they reflect learning of information specialized to each of those systems .', 'Given the broader functional roles of both structures , it is tempting to hypothesize that hippocampus is associated with stimulus-stimulus associations and striatum with response-response [64]–[66] .', 'While we did not explicitly dissociate response-response and stimulus-stimulus associations , the weight of the literature tying each of these types of information to each brain structure suggests this hypothesis and encourages us to carry it forward throughout the below discussion .', 'Importantly , by asking participants to seek a particular stimulus given another , our decision probes isolate only stimulus-stimulus associations and cannot be solved on the basis of response-response associations .', "Thus , the finding that the hippocampal activity ( and its learning rate ) contributed to these choices , but not the striatal one , is consisistent with these structures' hypothesized involvement in stimulus and response prediction .", 'Further , the exclusive use of the slow-process associations in forward-looking , model-based choice suggest that these associations are of a type that may be flexibly recombined , a property long associated with hippocampal representations and not those of striatum [48] , [52] , [67] .', 'That this learning was ‘slow’ in the hippocampus may at first seem to run counter to the notion that this structure supports flexible , rapidly bound learning , as in episodic memory .', 'Model-based decisions are also characterized similarly , for instance because they tend to dominate behavior during initial learning but not following overtraining .', 'However , it is important to emphasize that the theoretical ‘flexibility’ of the model-based system is in its ability to recombine the learned associations , applying them in novel contexts to novel goals: it is fundamentally about what is learned ( e . g . , a world model rather than a fixed policy ) rather than how quickly .', 'The question over what timescale any associations are learned is distinct from this issue – indeed , much previous work [57] , [68] implies that the learning rate should normatively be controlled by factors such as the volatility of the environment and the reliability of observations .', 'In this context , the learning rate measures the degree to which the model-based system can draw on experiences learned from the far past , in applying them to these novel contexts .', 'A low learning rate indicates a long memory; a higher learning rate indicates a shorter memory .', 'The mechanisms which might give rise to these learning dynamics are an interesting topic for further research .', 'Here , we have provided evidence that hippocampally-learned information is used in behavior via fetching memories of past transition events .', 'That these candidate transition events might be drawn from memories stretching over tens of trials ( spanning under a minute ) into the past is well within understood capacity limitations of the hippocampal memory system .', '( For a further treatment of these issues , see the discussion provided in our previous paper using this task [7] . )', 'In category-selective regions of the ventral visual cortex , we observed reinstatement of stimulus-stimulus associations in a manner that was modulated by task demands , across our two different tasks .', 'Over the sequential response trials , we observed that BOLD activity correlated with stimulus expectations in category-selective regions of the ventral visual stream .', 'Specifically , activity in face- ( or house- ) selective regions of extrastriate visual cortex were also preferentially modulated by the expectation that the face ( or house ) image would appear next .', 'The finding that activity parametrically fluctuates with stimulus predictions in both hippocampus and the ventral visual areas — and that the learning rates explaining these effects match one another — provides evidence that both areas are participating in a common associative learning process .', 'At a more mechanistic level , it may be possible to interpret both entropy-related activity in hippocampus and probability-related activity in the ventral visual areas in terms of associative spreading that activates the representations of likely successors to the currently observed image .', 'On its face , the finding that anticipatory activity in the ventral areas decreases with conditional probability might seem to run counter to such a mechanism .', 'That is , one might expect that , if probability is attributed largely to a single image , then the representation of that image should be more strongly activated .', 'The contrary observation could be explained by a similar mechanism to the one that has been offered to explain ‘repetition suppression’ of BOLD ( and spiking ) responses [69] , [70] .', 'Here , a more narrowly tuned population could be recruited for more strongly expected stimuli .', 'However , this explanation is insufficient to explain the parallel anticipatory activation we observe during choice trials , which are presumably the result of a common mechanism for anticipatory retrieval in the service of behavior .', 'A different interpretation of the effect is suggested by envisioning stimulus prediction as an active process of accessing memories .', 'In particular , previously observed successors might be stochastically retrieved in a likelihood-weighted fashion to build up a statistical profile of the subsequent image , with this mnemonic evidence accumulated in a manner analogous to diffusion-to-bound models of perceptual discrimination [21] , [71] .', 'This idea is consistent with suggestions that anticipatory activity in category regions is driven by evidence accumulation [72] .', 'If such a process terminates when evidence reaches some threshold , then spiking activity would be elevated only over a shorter interval of time and , thus , on trials with strong evidence observed signal would be lower when integrated over the length of the hemodynamic response [73] .', 'The activity of these same category-selective regions during the decision trials could be understood in a similar manner , in terms of retrieving memories to evaluate candidate actions .', 'Here , activity in the face ( and house ) areas of ventral visual cortex correlated with our measure of the difficulty of deciding whether the choice of that stimulus would lead to reward .', 'This observation supports a model where evaluation of decision options occurs by bounded accumulation of evidence — memories stochastically sampled to evaluate the likely consequences of a choice ( here , the successor image and its reward status ) .', 'Our aggregate ( as opposed to stimulus-specific ) choice difficulty measure was also positively correlated with activity in the anterior MPFC and posterior cingulate cortex .', 'Activations under our reporting threshold were also observed in dorsal MPFC and anterior and posterior hippocampus .', 'These regions together comprise the fronto-temporal memory component of the well-known “default network” [28] .', 'Although originally characterized by its increased , coherent , activity during periods of rest , a role in deliberative evaluation is consistent with functional hypotheses for this network , in which activity is modulated by prospective or constructive memory .', 'Tying together experimental data from multiple levels of observation and across task and rest modalities , Buckner & Carroll [26] suggest the default network “enables mental exploration of alternative perspectives based on our past experiences” , a proposal they expanded on in later discussions [27] .', 'Burgess [74] offers a complementary suggestion for one component of the network , proposing that BA10 in particular acts as a ‘gateway’ between a focus on internal ( e . g . , mnemonic ) and external ( e . g . , sensory ) representations .', 'These proposals — along with observations of hippocampus and default network activity during look-ahead planning [75]–[77] — concord with our interpretation of the choice difficulty correlate as reflecting reinstatement of prior experiences .', 'Finally , by offering a closer look at how the brain employs associations in the service of model-based decision making , our study suggests a route toward addressing one key puzzle in this area .', 'To wit , whereas simple reward learning has a straightforward neural implementation ( embodied in model-free temporal difference theories and relatives [6] , [78] , [79] ) , and the inference that these be accompanied by model-based choice is well established [3] , the mechanism by which the brain actually implements such computations remains opaque .', 'The idea we have advanced above , that successor states are retrieved stochastically ( see also [45] ) , and their values integrated , connects directly with known neural mechanisms .', 'In particular , although the idea of model-based planning as a mnemonic version of evidence accumulation differs at least superficially from more abstract conceptualizations based on tree search [3] , [80] , [81] or Bayesian inference [82] , [83] , sampling from successor states provides a more realizable process-level account of model-based evaluation in circumstances ( such as chess ) when the full set of future trajectories is too large to explore systematically .', 'Moreover , it connects closely with evidence accumulation mechanisms that are well studied in the context of perceptual decision making , and comports with other suggestions that sampling or diffusion models apply to value-based decisions as well [28] , [84]–[87] .', 'It also joins those ideas with a literature suggesting that episodic memories can influence decisions [46] , [56] , [88] .'], ['Twenty-four right-handed individuals ( twelve female; ages 18–40 years , mean 28 ) participated in the study .', 'All had normal or corrected-to-normal vision .', 'All participants received a fixed fee of $40 unrelated to performance , for their participation in the experiment , plus additional compensation of between $0 and $40 depending on their performance in one pseudorandomly-selected decision round .', 'Participants were recruited from the New York University community as well as the surrounding area and gave informed consent in accordance with procedures approved by the New York University Committee on Activities Involving Human Subjects .', 'Participants performed a serial reaction time ( SRT ) task in which they observed a sequence of image presentations and were instructed to respond using a pre-trained keypress assigned to that image .', 'The experiment was controlled by a script written in Matlab ( Mathworks , Natick , MA , USA ) , using the Psychophysics Toolbox [90] .', 'The stimulus set consisted of four grayscale images that were matched for size , contrast , and luminance .', 'The images were chosen because they represent categories known to preferentially engage different areas of the ventral visual stream — bodies [32] , faces [33] , houses [34] , and household objects [35] .', 'Each participant viewed the same four images .', 'During behavioral training , the keys corresponded to the innermost fingers on the home keys of a standard USA-layout keyboard ( D , F , J , K ) .', 'Participants were instructed to learn the responses as linking a finger and an image , rather than a key and an image ( e . g . left index finger , rather than ‘F’ ) .', 'For the MRI sessions , the same fingers were used to respond on two MR-compatible button boxes .', 'The mappings between the four images and four responses were one-to-one , pseudorandomly generated for each participant prior to their training session , trained to the criterion prior to the fMRI session , and fixed throughout the course of training and experiment sessions .', 'Participants were informed that the key-to-image mapping was fixed , and that they were not being evaluated on the correctness of responses .', 'At each trial , one of the pictures was presented in the center of the screen , where it remained for three seconds , plus or minus uniformly distributed pseudorandom jitter , up to 474 ms in increments of 59 ms ( the length of one slice in the MRI session ) .', 'Participants were instructed to continue pressing keys until they responded correctly or ran out of time .', 'Correct responses triggered a gray bounding box which appeared around the image for the lesser of 300 ms or the remaining trial time ( Figure 1 ) .', 'Thus , each image presentation occurred for the programmed amount of time , regardless of participant response .', 'The inter-trial interval consisted of 237 ms of blank screen .', 'The test phase of the scanning session proceeded with three blocks of 250 trials: 210 sequential response trials , 20 reward display screens ( see Choice trials , below ) and 20 choice trials .', 'The first two blocks were followed by a rest period of participant-controlled length .', 'During the rest period , participants were presented with a screen that was blank except for a fixation cross .', 'Scan blocks after the first were initiated manually by the operator only after the participant pressed any of the relevant keys twice , to alert the operator that they were prepared to continue the task .', 'Total experiment time — inclusive of training , practice and test periods — was approximately 1 . 5 hours , conducted continuously .', 'Our analysis proceeded in several steps meant to first characterize the associative learning process , and then use this characterization to test behavioral and neural predictions about choices .', "Each participant's trial-by-trial RTs for correct identifications were regressed on explanatory variables including the estimated conditional probability of the picture currently being viewed given its predecessor — defined , in separate models ( described below ) , in a number of different ways representing different accounts of learning — together with several effects of no interest .", 'Trials on which the first keypress was not correct were excluded from behavioral analysis .', 'Effects of no interest included stimulus-self transitions , image identity effects and a linear effect of trial number .', 'Stimulus-self transitions were included to account for variance due to motor response readiness for the same keypress appearing twice in a row , above and beyond the preparation implied by any effect of the variables of interest .', 'Image identity effects were included to account for any differential response time by each finger .', 'Trial number effects were included to account for any monotonic shift in response time over the course of the experiment .', 'These nuisance effects were identical across all models considered; the models differed in how they specified the explanatory variable of interest , the conditional probability of each image .', 'In our initial analysis , the conditional probabilities were specified as the ground-truth contingencies: the probabilities actually encoded in the transition matrix .', 'Having established that RT reflected such learning by demonstrating a significant correlation with these idealized probabilities ( Figure 2 ) , subsequent analyses used computational models to generate a timeseries of probability estimates such as would be produced by different learning rules with the same experience history as the participant ( see Learning models for details ) .', 'Similarly , the learning rules for conditional probability were fit ( separately ) to choices in the decision trials , estimated so as to maximize the likelihood that the model would have selected the same options as did the participant , given the same series of experience ( see Choice models for details ) .', 'The learning models involved additional free parameters controlling the learning and decision processes ( e . g . learning rates ) , which were jointly estimated together with the regression weights by maximum likelihood .', 'For behavioral analysis , models were fit and parameters were estimated separately for each participant .', 'At the group level , regression weights were tested for significance using a t-test on the individual estimates across participants [91] .', 'To generate regressors for fMRI analysis ( below ) we refitted the behavioral model to estimate a single set of the parameters that optimized the RT and choice likelihoods aggregated over all participants ( i . e . treating the behavioral parameters as fixed effects ) .', 'This approach allowed us to characterize baseline learning-related activity separate from individual variation in neurally implied learning rates relative to this common baseline .', 'For the former , in our experience [22] , [25] , [38] , [92]–[95] , enforcing common model parameters provides a simple regularization that improves the reliability of population-level neural results .', 'Our neural model characterizes between-subjects variation in the learning rate parameter over this baseline , because it includes ( as additional random effects across participants ) the partial derivatives of each of the regressors of interest with respect to the learning rate .']]
['Introduction', 'Results', 'Discussion', 'Materials and Methods']
["How do we use our memories of the past to guide decisions we've never had to make before ?", 'Although extensive work describes how the brain learns to repeat rewarded actions , decisions can also be influenced by associations between stimuli or events not directly involving reward — such as when planning routes using a cognitive map or chess moves using predicted countermoves — and these sorts of associations are critical when deciding among novel options .', 'This process is known as model-based decision making .', 'While the learning of environmental relations that might support model-based decisions is well studied , and separately this sort of information has been inferred to impact decisions , there is little evidence concerning the full cycle by which such associations are acquired and drive choices .', 'Of particular interest is whether decisions are directly supported by the same mnemonic systems characterized for relational learning more generally , or instead rely on other , specialized representations .', 'Here , building on our previous work , which isolated dual representations underlying sequential predictive learning , we directly demonstrate that one such representation , encoded by the hippocampal memory system and adjacent cortical structures , supports goal-directed decisions .', 'Using interleaved learning and decision tasks , we monitor predictive learning directly and also trace its influence on decisions for reward .', 'We quantitatively compare the learning processes underlying multiple behavioral and fMRI observables using computational model fits .', 'Across both tasks , a quantitatively consistent learning process explains reaction times , choices , and both expectation- and surprise-related neural activity .', 'The same hippocampal and ventral stream regions engaged in anticipating stimuli during learning are also engaged in proportion to the difficulty of decisions .', 'These results support a role for predictive associations learned by the hippocampal memory system to be recalled during choice formation .']
['We are always learning regularities in the world around us: where things are , and in what order we might find them .', 'Our knowledge of these contingencies can be relied upon if we later want to use them to make decisions .', 'However , there is little agreement about the neurobiological mechanism by which learned contingencies are deployed for decision making .', 'These are different kinds of decisions than simple habits , in which we take actions that have in the past given us reward .', 'Neural mechanisms of habitual decisions are well-described by computational reinforcement learning approaches , but have not often been applied to ‘model-based’ decisions that depend on learned contingencies .', 'In this article , we apply reinforcement learning to investigate model-based decisions .', 'We tested participants on a serial reaction time task with changing sequential contingencies , and choice probes that depend on these contingencies .', 'Fitting computational models to reaction times , we show that two sets of predictions drive simple response behavior , only one of which is used to make choices .', 'Using fMRI , we observed learning and decision-related activity in hippocampal and ventral cortical areas that is computationally linked to the learned contingencies used to make choices .', 'These results suggest a critical role for a hippocampal-cortical network in model-based decisions for reward .']
[]