An Imperfect Dopaminergic Error Signal Can Drive Temporal-Difference Learning
[['In this article we focus on a specific variant of TD learning: the TD algorithm as implemented by the actor-critic architecture [36] .', 'Here , we summarize the basic principles; a thorough introduction can be found in [1] .', 'The goal of a TD learning agent , as for every reinforcement learning agent , is to maximize the accumulated reward it receives over time .', 'The actor-critic architecture ( see Fig . 1 ) achieves this goal by making use of two modules , the actor and the critic .', 'The actor module learns a policy , which gives the probability of selecting an action in a state .', 'A common method of defining a policy is given by the Gibbs softmax distribution:where is known as the preference of action in state and the index runs over all possible actions in state .', "The critic evaluates the consequences of the actor module's chosen actions with respect to a value function .", 'Once learning has reached equilibrium , the value function is the expected summed discounted future reward when starting from state and following policy .', 'During the learning process only estimates of the actual value function are available .', 'The performance of the agent on a task is improved by making successive updates to the policy and the value function .', 'These updates are usually formulated assuming a discretization of time and space: an error term is calculated based on the difference in estimations of the value function when moving from one discrete state to the next , : ( 1 ) where is the reward the agent receives when moving into state and is a discount factor .', 'This error signal , known as the TD error , is positive if the reward is greater than the expected discounted difference between and , indicating that the estimate of needs to be increased .', 'Conversely , is negative if the reward is less than the expected discounted difference , indicating that the estimate of needs to be decreased .', 'In the simplest version of TD learning , known as the TD ( ) algorithm , the critic improves its estimate of as follows: ( 2 ) where is a small positive step-size parameter .', 'For a given policy and a sufficiently small , the TD learning algorithm converges with probability [37] , [38] .', 'Additionally , the preference of the chosen action in state is adjusted to make the selection of this action correspondingly more or less likely the next time the agent visits that state .', 'One possibility to update the preference in the actor-critic architecture is given by: ( 3 ) where is another small step-size parameter .', 'For the purposes of this manuscript , we shall refer to the calculation of the error signal and the update of value function and policy described above as the classical discrete-time TD ( ) algorithm .'], ['Fig . 2 illustrates the architecture of our actor-critic spiking network model implementing temporal-difference learning ( see Introduction ) .', 'All neurons in the network are represented by current-based integrate-and-fire neurons with alpha shaped post-synaptic currents .', 'A tabular description of our model and its neuronal , synaptic and external stimulation parameters are given in Methods .', 'The network interacts with an environment , which is implemented purely algorithmically for the purpose of this work .', 'The input layer of the neural network represents the cortex; it encodes information about states , each represented by a population of neurons .', 'The environment stimulates the population associated with the current state of the agent with a constant DC input , causing the neurons to fire with a mean rate of ; in the inactivated state the neurons fire on average with .', 'The low background rate in the inactivated state is chosen for the sake of simplicity in developing the synaptic plasticity dynamics , but is not a critical assumption of the model ( see section “Synaptic-plasticity” ) .', 'Each population in the cortex projects to the actor and critic modules .', 'As the focus of our study is the consequences of a realistic dopaminergic signal for temporal-difference learning rather than action selection , we keep the actor model as simple as possible .', 'As in previous models [20] , [34] , [39] , the actor module consists of actor neurons , each corresponding to one action .', 'The synaptic weights between the cortical and the actor neurons represent the policy in our model .', 'Whichever action neuron fires first in response to the activation of the state neurons is interpreted by the environment as the chosen action ( for a review of first-spike coding , see [40] ) .', 'Immediately after an action has been chosen , i . e . after an actor neuron has spiked , the environment deactivates the previous state neurons and activates the neurons representing the new state resulting from the chosen action .', 'At the same time the environment inhibits the actor neurons for a short time period , during which no further action can be chosen , allowing the cortical signal from a newly entered state to build up .', 'For more sophisticated approaches to the action selection problem , see [41] , [42] .', 'Two key experimentally observed features of the activity of the dopaminergic neurons are a constant low background rate with phasic activity with asymmetric amplitude depending on whether a reward is given or withheld [2] .', 'As the basal ganglia dynamics generating this signal is unknown , we select the simplest possible network that generates these features; in general , multiple network configurations can produce the same dynamics [43] .', 'We adapt the circuit model proposed in [18] to perform the role of the critic module , which is responsible for generating a temporal-difference error .', 'The major model assumption here is that the weights of the synapses between the neurons representing a given state and the critic module encode the value of that state .', 'The circuit connects a population of neurons representing the striatum , the input layer of the basal ganglia , to a population of dopaminergic neurons directly and also indirectly via a population of neurons representing the ventral pallidum .', 'The direct and indirect pathways are both inhibitory .', 'Consequently , the synaptic input from the striatum via the indirect pathway has a net excitatory effect , whereas the delayed striatal synaptic input via the direct pathway has an inhibitory effect on the dopamine neurons .', 'This results in a phasic increase if the agent moves from a state with low cortico-striatal synaptic weights to a state with high weights ( see Fig . 3 ) and a phasic decrease if the agent moves from a state with high cortico-striatal synaptic weights to a state with low weights .', 'The length of the phasic activation is determined by the difference in the delays of the direct pathway and the indirect one .', 'We have chosen and which results in a duration of the phasic activation similar to that observed experimentally ( see Fig . 1 in [2] ) .', 'If the agent enters a rewarded state , the dopamine neurons receive an additional DC stimulation from the environment starting after the agent moves and lasting for the duration of the phasic activity , .', 'Assuming the cortico-striatal synaptic weights represent the value function , after each state transition the dopamine neurons integrate information about the current value function with a positive sign , information about the previous value function with a negative sign , and a reward signal .', 'Thus all the information necessary to calculate a form of temporal-difference error is present ( see Eq .', '( 1 ) ) .', 'The dopaminergic neurons project back and release dopamine into the extracellular space ( Fig . 2 purple arrows ) which modulates as a third factor the plasticity of the synapses between the cortex and the striatum and between the cortex and the actor neurons .', 'Later in this section we develop synaptic plasticity models using a top-down approach to implement TD learning .', 'In this section we show that our network is able to generate dopaminergic activity with realistic firing rates and discuss its similarities to , and differences from , the classical discrete-time algorithmic definition of the TD error signal given in Eq .', '( 1 ) .', 'It has been found that dopamine neurons fire with a low constant baseline activity ( approx . in rats [44] , [45] and in monkeys [46] ) as long as nothing unpredicted happens .', 'This is known as the tonic activity of the dopaminergic neurons .', 'For our model , this implies that the baseline firing rate should be independent of the strength of the cortical-striatal synapses associated with each state .', 'This condition can be fulfilled in our architecture for an infinite number of configurations assuming linear relationships between the firing rates of the neurons in the striatum and the ventral pallidum; for a derivation of these relationships , see Supplementary Text S1 .', 'We select the simplest rate relationship with a linear coefficient of one .', 'This relationship generates a constant baseline activity when and the synaptic weights connecting the striatum to the dopamine neurons are equal in strength to the synaptic weights between the ventral pallidum and the dopamine neurons .', 'For the parameters given in Methods the mean dopaminergic baseline firing rate in our network is approx .', ', which is close to the experimentally observed stationary dopaminergic firing rate .', 'When the agent transits from one state to another , the dopamine neurons exhibit phasic activity lasting for around in accordance with durations found experimentally [47] , [48] , see Fig . 3 .', 'Fig . 4 shows the amplitude of phasic activity of the dopaminergic neurons after the agent transits from state to state in dependence of the difference in the corresponding cortico-striatal synaptic weights .', 'In accordance with experimental observation [46] the dopamine neurons show a continuum of firing rates lower than the baseline for outcomes that are worse than predicted ( ) and higher than the baseline for outcomes better than expected ( ) .', 'Likewise , entering a state with an unpredicted reward induces a phasic increase of activity .', 'The amplitude of the phasic activity of the dopaminergic neurons therefore has similar properties to the algorithmic TD error signal given in Eq .', '( 1 ) .', 'However , the properties of the dopaminergic signal deviate from the TD error in the following points: Point 2 arises due to the nonlinearities inherent in spiking neuronal networks , particularly at low rates ( for a recent account see [49] ) .', 'If a linear rate-based model was assumed , the amplitude of the phasic response would also vary linearly until an amplitude of was reached for some negative value of .', 'Similarly , the addition of the reward signal could only affect the offset of the curve in a linear rate-based model ( point 3 ) .', 'A nonlinear rate-based model may well be able to capture these features , but in order to make the correct non-linear assumptions , the behavior of the system to be abstracted needs to be known first .', 'A nonlinear dependence of the dopaminergic firing rate on the reward prediction error has recently also been observed experimentally [46] .', 'As we show in the next subsection , point 4 can be compensated by introducing a discount factor in the synaptic plasticity dynamics .', 'A -discounted difference can also be obtained if the dopaminergic rate is assumed to depend on the striatal firing rate .', 'As this is not in accordance with experimental findings we do not make this assumption , however , a derivation of the relationship between the firing rates and is derived in Supplementary Text S1 .', 'In order for the network model to realize TD learning , the right synapses have to undergo the right changes in strength at the right time; this is also known as the credit assignment problem [1] .', 'Here , we derive synaptic plasticity dynamics in a top-down fashion for the cortico-striatal synapses and the synapses between the cortical populations and the actor module representing the value function and the policy respectively .', 'In the classical TD algorithm , when the agent transits from state into state , only the value and preference of the most recently exited state are updated ( see Eq .', '( 2 ) and Eq .', '( 3 ) ) .', 'For a synapse to implement this feature it requires a mechanism that enables plasticity for a short time period after the agent has left the state associated with the pre-synaptic neuron .', 'This situation is characterized by the pre-synaptic rate being initially high and then dropping , as the population of cortical neurons associated with a state is strongly stimulated when the agent is in that state and weakly stimulated otherwise .', 'An appropriate dynamics can be constructed if the synapse maintains two dynamic variables driven by the spikes of the pre-synaptic neuron as originally proposed in [34]: a pre-synaptic activity trace and a pre-synaptic efficacy trace : ( 4 ) ( 5 ) where denotes the th spike of the pre-synaptic neuron .', 'The pre-synaptic activity trace is an approximation of the pre-synaptic firing rate; it is incremented at every pre-synaptic spike and decays to with a time constant ( see top panel of Fig . 5 ) .', 'To restrict the plasticity to the period immediately following a state transition , we assume a value of such that the activity trace decays to zero before the agent performs a further state transition .', 'Efficacy traces as defined in Eq .', '( 5 ) have previously been postulated as part of a spike-timing dependent plasticity model that accounts for data obtained from triplet and quadruplet spike protocols [50] .', 'The efficacy trace is set to at every pre-synaptic spike and relaxes exponentially to with a time constant ( Fig . 5 , middle panel ) .', 'This time constant is assumed to be large such that is small in the presence of pre-synaptic activity .', 'When the agent is in the state associated with neuron , is high and is close to zero .', 'When the agent leaves the state , relaxes to and relaxes to .', 'A product of the two traces is therefore close to at all times except for the period shortly after the agent leaves the state associated with neuron ( Fig . 5 , bottom panel ) .', 'Therefore , a synaptic plasticity dynamics proportional to ensures that the right synapses are sensitive to modifications at the right time to implement TD learning .', 'This simple relationship only holds for a very low rate in the inactive state .', 'If the firing rate of cortical neurons in the inactive state were higher , then the product would be non-negligible at all times , resulting in permanent sensitivity of the synapse to irrelevant fluctations in the dopamine signal .', 'Of course , this could be compensated for without altering the functionality by requiring to exceed a threshold , or by adopting a triphasic approach based on successive pre-synaptic activity thresholds as in our earlier work [34] .', 'The low rate therefore does not constitute a requirement for our model .', 'However , to avoid additional factors in the plasticity dynamics , we prefer to keep the rate relationships as simple as possible .', 'In TD learning the value function and the policy are both updated proportionally to the TD error ( see Eq .', '( 2 ) and Eq .', '( 3 ) ) which in our network model is signalled by the deviation of the dopaminergic firing rate from its baseline .', 'For the sake of simplicity we model the dopamine concentration as the superposition of the activity traces of all dopaminergic neurons: ( 6 ) where is the th dopamine spike and is a time constant .', 'This simplified model captures the experimentally observed feature that the concentration of dopamine is dependent on the firing times of the dopaminergic neurons [51] , [52] .', 'Moreover , we set in agreement with experimental findings on the dopamine uptake time in the striatum [51] .', 'A more sophisticated approach to modelling the extracellular dopamine concentration can be found in [52] .', 'A suitable synaptic plasticity dynamics between a cortical neuron and a striatal neuron to implement value function updates is therefore given by: ( 7 ) where is the baseline concentration of dopamine and is a learning rate parameter .', 'As discussed in the previous section , one difference between the dopaminergic signal as generated by our network model and the TD error is that the dopaminergic firing rate depends on the total value of the current state , rather than the -discounted value ( compare Eq .', '( 2 ) ) .', 'However , it is possible to compensate for this discrepancy in the following way .', 'The firing rate of the striatum population expresses the value of the current state , as the value function is encoded by the cortico-striatal synaptic weights .', 'For a given cortico-striatal synapse , the current state value can therefore be approximated by a post-synaptic activity trace as defined in Eq .', '( 4 ) with a time constant , which can be chosen quite arbitrarily .', 'We therefore include a term in Eq .', '( 7 ) proportional to the post-synaptic activity trace : ( 8 ) where .', 'In our numerical simulations we assume a plasticity dynamics at the cortico-striatal synapses as given by Eq .', '( 8 ) .', 'During the short period after a transition from to , the cortico-striatal synapses associated with state are sensitive to modification .', 'As discussed in the previous section , the dopaminergic signal depends nonlinearly on successive reward predictions encoded in the cortico-striatal synaptic weights , whereas the TD error is a linear function on the value function of successive states .', 'Furthermore the slope of the non-linear function depends on the magnitude of any external reward .', 'This means that it is not possible to define a single mapping from the units of synaptic weights to the units of the value function that holds for all values of and all rewards , as in our previous study [34] .', 'However , it is possible to generate a piecewise mapping by approximating the nonlinear function for a given reward signal in a given range of by a linear function .', 'The mapping ( Eq .', '( 11 ) ) is derived in detail in the Supplementary Text S2 and consists of two steps .', 'First , the synaptic plasticity dynamics is integrated to calculate the net change in the mean outgoing synaptic weight of the neurons associated with a state when the agent moves from to .', 'Second , the net weight change is converted from units of synaptic weight to units of the value function according to the linear relationships: ( 9 ) ( 10 ) where is a proportionality parameter mapping the mean striatal firing rate to the units of the value function and is a proportionality factor mapping the mean cortico-striatal weights of a state to the mean striatal firing rate .', 'For our choice of parameters ( see Methods ) Eq .', '( 10 ) is fulfilled in the allowed range for the cortico-striatal weights with and .', 'Within a given range of , the mean net weight change of the synapses immediately after transition out of corresponds to a slightly modified version of the classical discrete-time value function update with an additional offset : ( 11 ) The learning parameters and of the equivalent TD ( ) algorithm and the offset depend on the synaptic parameters and as defined above .', 'They additionally depend on the slope and intercept of the linear approximation of the dopaminergic signal: ( 12 ) The constants depend on the synaptic time constants; see Supplementary Text S2 for the definitions .', 'Because and are dependent on the range of and the direct current applied to the dopamine neurons , the weight update can be interpreted as a TD learning value function update with self-adapting learning parameters and a self-adapting offset that depend on the current weight change and reward .', 'The greater the difference between the mean synaptic weights of successive states , the higher the learning rate and discount factor .', 'For the parameters used in our simulations , a range of can be realized by a range of .', 'A choice of results in a discount factor .', 'For a specific choice of , the learning rate can be determined by the synaptic parameter .', 'For , the range can be realized by the range .', 'As and can be chosen independently , all possible combinations of and can be realized .', 'If the current state is rewarded , the offset is a -dependent analog to the reward in the TD error Eq .', '( 1 ) .', 'Otherwise , for an appropriate choice of parameters ( see Methods ) is always smaller than and has no analog in classical TD learning .', 'Self-adjusting parameters have also been implemented in other three-factor learning rules such as the one in [53] based on the meta-learning algorithm proposed in [54] .', 'In contrast to meta-learning , in our model the values of the parameters do not adjust themselves to optimal parameters for a given task but vary according to the difference between the estimated values of successive states , , and the current reward value .', 'The range of possible learning parameters for a given and reward value depends on the current choice of synaptic parameters and , which can be set arbitrarily .', 'However , meta-learning could be an additional mechanism that adjusts the parameters and to optimal values for a given task .', 'The variable parameters suggest a similarity with value learning , another learning algorithm similar to TD but with a variable discount rate [55] .', 'However , in value learning the discount rate changes over time: it is lowest immediately after an unconditioned stimulus and increases in between them , making the algorithm more sensitive to long term rewards .', 'In our model the learning parameters do not depend on time but on the current reward and the difference in successive reward predictions encoded by .', 'Similarly to the update of the value function , in the classical discrete-time TD algorithm only the policy for the recently vacated state is updated .', 'As described earlier in this section , in the neuronal architecture an action is chosen by the first spike of an action neuron .', 'Therefore an appropriate plasticity dynamics for the synapse between a cortex neuron and an actor neuron is given by ( 13 ) where determines the learning speed , and is a post-synaptic activity trace as defined in Eq .', '( 4 ) with time constant .', 'The choice of post-synaptic time constant is not critical , but the activity trace should decay within the typical time an agent spends in a state in order to be selective for the most recently chosen action .', 'Unlike the cortico-striatal synapses described above , the lack of -discounting in the dopamine signal cannot be compensated for by the addition of an additional local term in the synaptic plasticity dynamics .', 'This is due to the fact that the post-synaptic activity here represents whether the encoded action was selected rather than the value function of the next state as in the previous case .', 'Information about the value of the new state could only arrive at the synapse through an additional non-local mechanism .', 'In order to ensure the agent continues to occasionally explore alternative directions to its preferred direction in any given state , we restrict the synaptic weights of the synapses between the cortex and the actor neurons to the range .', 'This results in a maximal probability of and a minimal probability of for any movement direction in any state ( see Supplementary Text S2 for a mapping of synaptic weights to probabilities ) .', 'The parameters for synaptic plasticity models used in our study are summarized in Methods .', 'The proposed cortico-striatal synaptic plasticity dynamics Eq .', '( 8 ) depends on three factors: the pre-synaptic firing rate , the post-synaptic firing rate and the dopamine concentration .', 'For cortico-striatal synapses the effect on the plasticity of each of these factors has experimentally been studied in vivo and in vitro ( see [9] for a review ) .', 'The long-term effects found on average across studies are summarized in column six of Table 1 .', 'These results show that in order to induce any long lasting changes in synaptic plasticity , a conjunction of pre- and post-synaptic activity is required .', 'Early studies on the effect of conjoined pre-synaptic and post-synaptic activity on the cortico-striatal plasticity reported exclusively long term depression ( LTD ) .', 'More recent studies have shown that long term potentiation ( LTP ) can also be obtained under some circumstances .', 'The expression of LTP or LTD seems to depend on methodological factors such as the age of the animal , the location of the neuron and the stimulating electrode and the stimulus parameters [9] .', 'Although in these studies it is assumed that dopamine is not involved , it cannot be ruled out as cortico-striatal high frequency stimulation causes dopamine release [56] .', 'The main findings resulting from studies involving all three factors can be summarized in the following three-factor rule [57]: under normal and low dopamine concentrations , the conjunction of pre- and post-synaptic activity leads to LTD , whereas a large phasic increase in dopamine concentration during pre- and post-synaptic activity results in LTP .', 'The predictions of the cortico-striatal synaptic dynamics given by Eq .', '( 8 ) for the various permutations of pre- and post-synaptic activity and dopamine concentration are summarized in column ( for , corresponding to ) and column ( for , corresponding to ) of Table 1 .', 'We assume that a value of in the first three columns denotes recent activity; due to the time constants of the activity traces this activation is still perceptible from the point of view of the synapse and can thus be assumed to have an active influence on plasticity .', 'Assuming the baseline dopamine concentration only changes on a long time scale , experiments involving no particular manipulations of the dopamine concentration ( denoted by in Table 1 ) will be characterized by .', 'The plasticity dynamics Eq .', '( 8 ) predicts LTD for an active influence of pre- and post-synaptic activity , and in accordance with the majority of the experimental findings; for no change in synaptic strength is predicted .', 'Furthermore , Eq .', '( 8 ) predicts that for simultaneous influence of pre- and post-synaptic activity , the direction of the synaptic change depends on the concentration of dopamine .', 'For normal ( ) as well as low dopamine concentration results in LTD ( see Fig . 6 ) , while a large phasic increase in the dopamine concentration results in LTP .', 'For the change from LTD to LTP occurs at , resulting in no change in synaptic strength under normal dopamine concentration in contrast to the experimental findings .', 'The theoretical model makes additional predictions in this case that go beyond the presence or absence of activity and the direction of change .', 'Due to the timing sensitivity of the plasticity dynamics given in Eq .', '( 8 ) , a weak synaptic weight change is predicted if the activity of the pre-synaptic neuron overlaps with the activity of the post-synaptic neuron in the presence of dopamine and a strong change if the pre-synaptic activity precedes the post-synaptic activity .', 'Such a dependency on timing involving extended periods of activation have so far not been tested experimentally .', 'However , protocols involving individual spike pairs have revealed comparable effects; for a review , see [58] .', 'The greatest difference between our predictions and the experimental findings is that a simultaneously active influence of pre-synaptic activity and dopamine is sufficient to induce LTD or LTP in the absence of post-synaptic activity .', 'However , this is quite an artificial case as pre-synaptic activity always generates post-synaptic activity in our network model dynamics .', 'The behavior of the model could be brought into better alignment with the experimental data by adding additional complexity .', 'For example , a multiplicative Heaviside function that evaluates to one when the post-synaptic activity exceeds a certain threshold would eliminate the generation of LTP/LTD in the absence of post-synaptic activity without altering the functionality of our model .', 'As the plasticity dynamics was derived to fulfil a particular computational function rather than to provide a phenomenological fit to the experimental data , we prefer to avoid this additional complexity .', 'Apart from this case , our predictions on the direction of cortico-striatal plasticity under the active conjunction of pre- and post-synaptic activity for are in good agreement with experimental findings .', 'As in our previous study [34] , we tested the learning capability of our neuronal network model on a grid-world task , a standard task for TD learning algorithms .', 'In our variant of this task , the grid consists of states arranged in a five by five grid ( see inset of Fig . 7 ) .', 'The agent can choose between four different actions ( south , north , east , west ) represented by actor neurons .', 'If the agent chooses an action which would lead outside the grid world , the action does not lead to a change in its position .', 'Only a single state is rewarded; when the agent enters it a direct current with amplitude is applied to the dopaminergic neurons corresponding to the real-valued reward sent to the critic module in a classical discrete-time TD algorithm ( see Introduction ) .', 'After the agent has found the reward and selected a new action , it is moved to a new starting position that is chosen randomly and independently of the selected action .', 'This is therefore a continuing task rather than an episodic task , as there are no terminal states .', 'To maximize its reward , the agent must find the reward from random starting positions in as few steps as possible .', 'The difficulty of the task is that the agent has to learn a series of several actions starting from each state in which only the last one results in a reward .', 'The grid world task is useful to visualize the behavior of a learning algorithm but is not intended to represent physical navigation task , as spatial information is not taken into consideration ( e . g . exploiting the knowledge of which states are neighbors ) .', 'To evaluate the performance of our model on the grid-world task , we separate the ongoing sequence of states and actions into trials , where a trial is defined as the period between the agent being placed in a starting position and the agent reaching the reward state .', 'We measure the latency for each trial , i . e . the difference between the number of steps the agent takes to reach the reward state and the minimum number of steps required to reach the reward state for the given starting position .', 'To provide a comparison , we also measure the performance of a classical discrete-time TD learning algorithmic implementation with corresponding parameters .', 'The specification of the discrete-time implementation is obtained by mapping the synaptic parameters to the discrete-time parameters for and determining the corresponding reward via a search algorithm ( see Supplementary Text S2 ) .', 'Fig . 7 shows the evolution of latency on the grid-world task for the neuronal network model with optimized parameters and the discrete-time algorithmic implementation with corresponding parameters .', 'Within the first trials the latency of the neuronal network model drops from around steps to steps .', 'After trials the agent has learnt the task; the latency is always below steps .', 'The learning speed and the equilibrium performance of the neuronal network model are as good as those of the corresponding discrete-time algorithmic implementation .', 'The performance of the discrete-time algorithmic implementation does not deteriorate if a discount factor is assumed for the updates to the policy in correspondence with the synaptic plasticity dynamics given by Eq .', '( 13 ) ( data not shown ) .', 'As discussed in section “Synaptic-plasticity” , we impose hard bounds on the weights of the synapses between the cortex and the actor to ensure that for a given state , no action becomes either impossible or certain .', 'For this task , it turns out that the lower bound is not necessary; restricting the weights to the range results in a similar learning performance ( data not shown ) .', 'However , the upper bound is necessary for the stability of the system .', 'In the absence of an upper bound , synaptic weig
['Introduction', 'Results', 'Discussion', 'Methods']
['An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning .', 'A promising framework in this context is temporal-difference ( TD ) learning .', 'Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine .', 'However , as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error , it is unclear whether it is capable of driving behavior adaptation in complex tasks .', 'Here , we present a spiking temporal-difference learning model based on the actor-critic architecture .', 'The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor .', 'The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine , pre- and post-synaptic activity .', 'An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset .', 'We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm .', 'However , the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards .', 'Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards .']
['What are the physiological changes that take place in the brain when we solve a problem or learn a new skill ?', 'It is commonly assumed that behavior adaptations are realized on the microscopic level by changes in synaptic efficacies .', 'However , this is hard to verify experimentally due to the difficulties of identifying the relevant synapses and monitoring them over long periods during a behavioral task .', 'To address this question computationally , we develop a spiking neuronal network model of actor-critic temporal-difference learning , a variant of reinforcement learning for which neural correlates have already been partially established .', 'The network learns a complex task by means of an internally generated reward signal constrained by recent findings on the dopaminergic system .', 'Our model combines top-down and bottom-up modelling approaches to bridge the gap between synaptic plasticity and system-level learning .', "It paves the way for further investigations of the dopaminergic system in reward learning in the healthy brain and in pathological conditions such as Parkinson's disease , and can be used as a module in functional models based on brain-scale circuitry ."]
['neuroscience/behavioral neuroscience', 'neuroscience', 'neuroscience/theoretical neuroscience']