How Attention Can Create Synaptic Tags for the Learning of Working Memories in Sequential Tasks
[['Animals like monkeys can be trained to perform complex cognitive tasks , simply by giving rewards at the right times .', 'They can learn to map sensory stimuli onto responses , to store task-relevant information and to integrate and combine unreliable sensory evidence .', 'Training induces new stimulus and memory representations in ‘multiple-demand’ regions of the cortex [1] .', 'For example , if monkeys are trained to memorize the location of a visual stimulus , neurons in lateral intra-parietal cortex ( LIP ) represent this location as a persistent increase of their firing rate [2 , 3] .', 'However , if the animals learn a visual categorization task , persistent activity of LIP cells becomes tuned to the boundary between categories [4] whereas the neurons integrate probabilistic evidence if the task is sensory decision making [5] .', 'Similar effects of training on persistent activity have been observed in the somatosensory system .', 'If monkeys are trained to compare frequencies of successive vibrotactile stimuli , working memory representations of analog variables are formed in somatosensory , prefrontal and motor cortex [6] .', 'Which learning mechanism induces appropriate working memories in these tasks ?', 'We here outline AuGMEnT ( Attention-Gated MEmory Tagging ) , a new reinforcement learning [7] scheme that explains the formation of working memories during trial-and-error learning and that is inspired by the role of attention and neuromodulatory systems in the gating of neuronal plasticity .', 'AuGMEnT addresses two well-known problems in learning theory: temporal and structural credit-assignment [7 , 8] .', 'The temporal credit-assignment problem arises if an agent has to learn actions that are only rewarded after a sequence of intervening actions , so that it is difficult to assign credit to the appropriate ones .', 'AuGMEnT solves this problem like previous temporal-difference reinforcement learning ( RL ) theories [7] .', 'It learns action-values ( known as Q-values [7] ) , i . e . the amount of reward that is predicted for a particular action when executed in a particular state of the world .', 'If the outcome deviates from the reward-prediction , a neuromodulatory signal that codes the global reward-prediction error ( RPE ) gates synaptic plasticity in order to change the Q-value , in accordance with experimental findings [9–12] .', 'The key new property of AuGMEnT is that it can also learn tasks that require working memory , thus going beyond standard RL models [7 , 13] .', 'AuGMEnT also solves the structural credit-assignment problem of networks with multiple layers .', 'Which synapses should change to improve performance ?', 'AuGMEnT solves this problem with an ‘attentional’ feedback mechanism .', 'The output layer has feedback connections to units at earlier levels that provide feedback to those units that were responsible for the action that was selected [14] .', 'We propose that this feedback signal tags [15] relevant synapses and that the persistence of tags ( known as eligibility traces [7 , 16] ) permits learning if time passes between the action and the RPE [see 17] .', 'We will here demonstrate the neuroscientific plausibility of AuGMEnT .', 'A preliminary and more technical version of these results has been presented at a conference [18] .'], ['We used AuGMEnT to train networks composed of three layers of units connected by two layers of modifiable synapses ( Fig . 1 ) .', 'Time was modeled in discrete steps .', 'At the start of every time step , feedforward connections propagate information from the sensory layer to the association layer through modifiable connections vij .', 'The sensory layer represents stimuli with instantaneous and transient units ( Fig . 1 ) .', 'Instantaneous units represent the current sensory stimulus x ( t ) and are active as long as the stimulus is present .', 'Transient units represent changes in the stimulus and behave like ‘on ( + ) ’ and ‘off ( - ) ’ cells in sensory cortices [19] .', 'They encode positive and negative changes in sensory inputs w . r . t . the previous time-step t - 1:', 'x+ ( t ) =[x ( t ) −x ( t−1 ) ]+ ,', '( 1 )', 'x− ( t ) =[x ( t−1 ) −x ( t ) ]+ ,', '( 2 )', 'where [⋅]+ is a threshold operation that returns 0 for all negative values , but leaves positive values unchanged .', 'Every input is therefore represented by three sensory units .', 'We assume that all units have zero activity at the start of the trial ( t = 0 ) , and that t = 1 at the first time-step of the trial .', 'The second ( hidden ) layer of the network models the association cortex , and contains regular units ( circles in Fig . 1 ) and memory units ( diamonds ) .', 'We use the term ‘regular unit’ to reflect the fact that these are regular sigmoidal units that do not exhibit persistent activity in the absence of input .', 'Regular units j are fully connected to instantaneous units i in the sensory layer by connections vijR ( the superscript R indexes synapses onto regular units , and v0jR is a bias weight ) .', 'Their activity yjR ( t ) is determined by:', 'inpjR ( t ) =∑ivijRxi ( t ) ,', '( 3 )', 'yjR ( t ) =σ ( inpjR ( t ) ) ,', '( 4 )', 'here inpjR ( t ) denotes the synaptic input and σ a sigmoidal activation function;', 'σ ( inpjR ( t ) ) =1/ ( 1\u200b+exp ( θ−inpjR ( t ) ) ) ,', '( 5 )', 'although our results do not depend on this particular choice of σ .', 'The derivative of yjR ( t ) can be conveniently expressed as:', 'y′jR ( t ) =σ′ ( inpjR ( t ) ) =∂yjR ( t ) ∂inpjR ( t ) =yjR ( t ) ( 1−yjR ( t ) ) .', '( 6 )', 'Memory units m ( diamonds in Fig . 1 ) are fully connected to the transient ( +/- ) units in the sensory layer by connections vlmM ( superscript M indexes synapses onto memory units ) and they integrate their input over the duration of the trial:', 'inpmM ( t ) =inpmM ( t−1 ) +∑lvlmMxl′ ( t ) ,', '( 7 )', 'ymM ( t ) =σ ( inpmM ( t ) ) ,', '( 8 )', 'where we use the shorthand xl′ that stands for both + and - cells , so ∑lvlmMxl′ ( t ) should be read as ∑lvlmM+xl+ ( t ) +∑lvlmM−xl− ( t ) The selective connectivity between the transient input units and memory cells is advantageous .', 'We found that the learning scheme is less stable when memory units also receive input from the instantaneous input units because in that case even weak constant input becomes integrated across time as an activity ramp .', 'We note , however , that there are other neuronal mechanisms which can prevent the integration of constant inputs .', 'For example , the synapses between instantaneous input units and memory units could be rapidly adapting , so that the memory units only integrate variations in their input .', 'The simulated integration process causes persistent changes in the activity of memory units .', 'It is easy to see that the activity of a memory unit equals the activity of a hypothetical regular unit that would receive input from all previous time-steps of the trial at the same time .', 'To keep the model simple , we do not simulate the mechanisms responsible for persistent activity , which have been addressed in previous work [20–22] .', 'Although the perfect integration assumed in Eqn .', '( 7 ) does not exist in reality , we suggest that it is an acceptable approximation for trials with a relatively short duration as in the tasks that will be described below .', 'Indeed , there are reports of single neuron integrators in entorhinal cortex with stable firing rates that persist for ten minutes or more [23] , which is orders of magnitude longer than the trials modeled here .', 'In neurophysiological studies in behaving animals , the neurons that behave like regular and memory units in e . g . LIP [2 , 3] and frontal cortex [24] would be classified as visual cells and memory cells , respectively .', 'The third layer receives input from the association layer through plastic connections wjk ( Fig . 1 ) .', 'Its task is to compute action-values ( i . e . Q-values [7] ) for every possible action .', 'Specifically , a Q-value unit aims to represent the ( discounted ) expected reward for the remainder of a trial if the network selects an action a in the current state s [7]:', 'Qπ ( s , a ) =Eπ{Rt|st=s , at=a} , withRt=∑p=0∞γprt+p+1 ,', '( 9 )', 'where the Eπ{⋅} term is the expected discounted future reward Rt given a and s , under action-selection policy π and γ∈[0 , 1] determines the discounting of future rewards r .', 'It is informative to explicitly write out the above expectation to see that Q-values are recursively defined as:', 'Qπ ( s , a ) =∑s′∈SPsas′[Rsas′+γ∑a′∈Aπ ( a′|s′ ) Qπ ( s′ , a′ ) ] ,', '( 10 )', 'where Psas′ is a transition matrix , containing the probabilities that executing action a in state s will move the agent to state s′ , Rsas′ is the expected reward for this transition , and S and A are the sets of states and actions , respectively .', 'Note that the action selection policy π is assumed to be stochastic in general .', 'By executing the policy π , an agent samples trajectories according to the probability distributions π , Psas′ and Rsas′ where every observed transition can be used to update the original prediction Q ( st , at ) .', 'Importantly , temporal difference learning schemes such as AuGMEnT are model-free , which means that they do not need explicit access to these probability distributions while improving their Q-values .', 'Q-value units k are fully connected to the association layer by connections wjkR ( from regular units , with w0kR as bias weight ) and wmkM ( from memory units ) .', 'The action value qk ( t ) is estimated as:', 'qk ( t ) =∑mwmkMymM ( t ) +∑jwjkRyjR ( t ) ,', '( 11 )', 'where qk ( t ) aims to represent the value of action k at time step t , i . e . if at = k .', 'In AuGMEnT , the state s in Eq .', '( 9 ) is represented by the vector of activations in the association layer .', 'Association layer units must therefore learn to represent and memorize information about the environment to compute the value of all possible actions a .', 'They transform a so-called partially observable Markov decision process ( POMDP ) where the optimal decision depends on information presented in the past into a simpler Markov decision process ( MDP ) by storing relevant information as persistent activity , making it available for the next decision .', 'The action-selection policy π is implemented by a stochastic winner-takes-all ( WTA ) competition biased by the Q-values .', 'The network usually chooses the action a with the highest value , but occasionally explores other actions to improve its value estimates .', 'We used a Max-Boltzmann controller [25] to implement the action selection policy π .', 'It selects the greedy action ( highest qk ( t ) , ties are broken randomly ) with probability 1 - ε , and a random action k sampled from the Boltzmann distribution PB with small probability ε: This controller ensures that the model explores all actions , but usually selects the one with the highest expected value .', 'We assume that the controller is implemented downstream , e . g . in the motor cortex or basal ganglia , but do not simulate the details of action selection , which have been addressed previously [26–30] .', 'After selecting an action a , the activity in the third layer becomes zk = δka , where δka is the Kronecker delta function ( 1 if k = a and 0 otherwise ) .', 'In other words , the selected action is the only one active after the selection process , and it then provides an “attentional” feedback signal to the association cortex ( orange feedback connections in Fig . 1A ) .', 'Learning in the network is controlled by two factors that gate plasticity: a global neuromodulatory signal ( described below ) and the attentional feedback signal .', 'Once an action is selected , the unit that codes the winning action a feeds back to earlier processing levels to create synaptic tags [31 , 32] , also known as eligibility traces [7 , 16] on the responsible synapses ( orange pentagons in Fig . 1 ) .', 'Tagging of connections from the association layer to the motor layer follows a form of Hebbian plasticity: the tag strength depends on presynaptic activity ( yj ) and postsynaptic activity after action selection ( zk ) and tags thus only form at synapses wja onto the winning ( i . e . selected ) motor unit a:', 'ΔTagjk=−αTagjk+yjzk , which is equivalent to:ΔTagja=−αTagja+yj , for the winning actiona , becauseza=1andΔTagjk=−αTagjk , fork≠a , becausezk≠a=0 ,', '( 13 )', 'where α controls the decay of tags .', 'Here , Δ denotes the change in one time-step , i . e Tag ( t+1 ) = Tag ( t ) +ΔTag ( t ) .', 'The formation of tags on the feedback connections waj′ follows the same rule so that the strength of feedforward and feedback connections becomes similar during learning , in accordance with neurophysiological findings [33] .', 'Thus , the association units that provided strong input to the winning action a also receive strongest feedback ( Fig . 1 , middle panel ) : they will be held responsible for the outcome of a .', 'Importantly , the attentional feedback signal also guides the formation of tags on connections vij so that synapses from the input layer onto responsible association units j ( strong waj′ ) are most strongly tagged ( Fig . 1B ) .', 'For regular units we propose:', 'ΔTagij=−αTagij+xiσ′ ( inpj ) waj′ ,', '( 14 )', 'where σ′ is the derivative of the association unit’s activation function σ ( Eq .', '( 5 ) ) , which determines the influence that a change in the input inpj has on the activity of unit j .', 'The idea has been illustrated in Fig . 1B .', 'Feedback from the winning action ( lower synapse in Fig . 1B ) enables the formation of tags on the feedforward connections onto the regular unit .', 'These tags can interact with globally released neuromodulators that inform all synapses about the RPE ( green cloud ‘δ’ in Fig . 1 ) .', 'Note that feedback connections only influence the plasticity of representations in the association layer but do not influence activity in the present version of the model .', 'We will come back to this point in the discussion .', 'In addition to synaptic tags , AuGMEnT uses synaptic traces ( sTrace , blue circle in Fig . 1A , C ) for the learning of new working memories .', 'These traces are located on the synapses from the sensory units onto memory cells .', 'Any pre-synaptic activity in these synapses leaves a trace that persists for the duration of a trial .', 'If one of the selected actions provides a feedback signal ( panel iv in Fig . 1C ) to the post-synaptic memory unit , the trace gives rise to a tag making the synapse plastic as it can now interact with globally released neuromodulators:', 'ΔsTraceij=xi ,', '( 15 )', 'ΔTagij=−αTagij+sTraceijσ′ ( inpj ) waj′', '( 16 )', 'We assume that the time scale of trace updates is fast compared to the tag updates , so that tags are updated with the latest traces .', 'The traces persist for the duration of the trial , but all tags decay exponentially ( 0<α<1 ) .', 'After executing an action , the network may receive a reward r ( t ) .', 'Moreover , an action a at time step ( t-1 ) may have caused a change in the sensory stimulus .', 'For example , in most studies of monkey vision , a visual stimulus appears if the animal directs gaze to a fixation point .', 'In the model , the new stimulus causes feedforward processing on the next time step t , which results in another set of Q-values .', 'To evaluate whether a was better or worse than expected , the model compares the predicted outcome Qa ( t-1 ) , which has to be temporarily stored in the system , to the sum of the reward r ( t ) and the discounted action-value Qa′ ( t ) of unit a′ that wins the subsequent stochastic WTA-competition .', 'This temporal difference learning rule is known as SARSA [7 , 34]:', 'δ ( t ) =r ( t ) +γqa′ ( t ) −qa ( t−1 ) .', '( 17 )', 'The RPE δ ( t ) is positive if the outcome of a is better than expected and negative if it is worse .', 'Neurons representing action values have been found in the frontal cortex , basal ganglia and midbrain [12 , 35 , 36] and some orbitofrontal neurons specifically code the chosen value , qa [37] .', 'Moreover , dopamine neurons in the ventral tegmental area and substantia nigra represent δ [9 , 10 , 38] .', 'In the model , the release of neuromodulators makes δ available throughout the brain ( green cloud in Fig . 1 ) .', 'Plasticity of all synapses depends on the product of δ and tag strength:', 'Δvij=βδ ( t ) Tagij , Δwjk=βδ ( t ) Tagjk ,', '( 18 )', 'where β is the learning rate , and where the latter equation also holds for the feedback weights wkj′ .', 'These equations capture the key idea of AuGMEnT: tagged synapses are held accountable for the RPE and change their strength accordingly .', 'Note that AuGMEnT uses a four-factor learning rule for synapses vij .', 'The first two factors are the pre- and postsynaptic activity that determine the formation of tags ( Eqns .', '( 14 ) – ( 16 ) ) .', 'The third factor is the “attentional” feedback from the motor selection stage , which ensures that tags are only formed in the circuit that is responsible for the selected action .', 'The fourth factor is the RPE δ , which reflects whether the outcome of an action was better or worse than expected and determines if the tagged synapses increase or decrease in strength .', 'The computation of the RPE demands the comparison of Q-values in different time-steps .', 'The RPE at time t depends on the action that the network selected at t-1 ( see Eqn .', '( 17 ) and the next section ) , but the activity of the units that gave rise to this selection have typically changed at time t .', 'The synaptic tags solve this problem because they labeled those synapses that were responsible for the selection of the previous action .', 'AuGMEnT is biologically plausible because the equations that govern the formation of synaptic tags ( Eqns .', '( 13 ) , ( 14 ) , ( 16 ) ) and traces ( Eq .', '( 15 ) ) and the equations that govern plasticity ( Eq .', '( 18 ) ) rely only on information that is available locally , at the synapse .', 'Furthermore , the hypothesis that a neuromodulatory signal , like dopamine , broadcasts the RPE to all synapses in the network is supported by neurobiological findings [9 , 10 , 38] .'], ['The aim of AuGMEnT is to reduce the RPE δ ( t ) because low RPEs for all network states imply reliable Q-values so that the network can choose the action that maximizes reward at every time-step .', 'The RPE δ ( t ) implies a comparison between two quantities: the predicted Q-value before the transition , qa ( t-1 ) , and a target Q-value r ( t ) +γqa′ ( t ) , which consists of the actually observed reward and the next predicted Q-value [7] .', 'If the two terms cancel , the prediction was correct .', 'SARSA aims to minimize the prediction error by adjusting the network weights w to improve the prediction qa ( t-1 ) to bring it closer to the observed value r ( t ) +γqa′ ( t ) .', 'It is convenient to do this through on-line gradient descent on the squared prediction error E ( qa ( t−1 ) ) =12 ( [r ( t ) +γqa′ ( t ) ]−qa ( t−1 ) ) 2 with respect to the parameters w [7 , 34]:', 'Δw∝−∂E ( qa ( t−1 ) ) ∂w=−∂E ( qa ( t−1 ) ) ∂qa ( t−1 ) ∂qa ( t−1 ) ∂w=δ ( t ) ∂qa ( t−1 ) ∂w ,', '( 19 )', 'where ∂qa ( t−1 ) ∂w is the gradient of the predicted Q-value Qa ( t-1 ) with respect to parameters w .', 'In Equation ( 19 ) we have used δ ( t ) =−∂E ( qa ( t−1 ) ) ∂qa ( t−1 ) , which follows from the definition of E ( qa ( t-1 ) ) .', 'Note that E is defined with regard to the sampled transition only so that the definition typically differs between successive transitions experienced by the network .', 'For notational convenience we will abbreviate E ( qa ( t-1 ) ) to Eqa in the remainder of this paper .', 'We will refer to the negative of Equation ( 19 ) as “error gradient” in the remainder of this paper .', 'The RPE is high if the sum of the reward r ( t ) and discounted qa′ ( t ) deviates strongly from the prediction qa ( t-1 ) on the previous time step .', 'As in other SARSA methods , the updating of synaptic weights is only performed for the transitions that the network actually experiences .', 'In other words , AuGMEnT is a so-called “on policy” learning method [7] .', 'We will first establish the equivalence of on-line gradient descent defined in Equation ( 19 ) and the AuGMEnT learning rule for the synaptic weights wjkR ( t ) from the regular units onto the Q-value units ( Fig . 1 ) .', 'According to Equation ( 19 ) , weights wjaR for the chosen action k = a on time step t-1 should change as:', 'ΔwjaR∝δ ( t ) ∂qa ( t−1 ) ∂wjaR ( t−1 ) ,', '( 20 )', 'leaving the other weights k≠a unchanged .', 'We will now show that AuGMEnT causes equivalent changes in synaptic strength .', 'It follows from Eq .', '( 11 ) that the influence of wjaR on qa ( t-1 ) ( i . e . ∂qa ( t−1 ) ∂wjaR ( t−1 ) in Eq .', '( 20 ) ) equals yjR ( t−1 ) , the activity of association unit j on the previous time step .', 'This result allows us to rewrite ( 20 ) as:', 'ΔwjaR∝−∂Eqa∂wjaR ( t−1 ) =δ ( t ) ∂qa ( t−1 ) ∂wjaR ( t−1 ) =δ ( t ) yjR ( t−1 ) .', '( 21 )', 'Recall from Eq .', '( 13 ) that the tags on synapses onto the winning output unit a are updated according to ΔTagja = - αTagja+yj ( orange pentagons in Fig . 1 ) .', 'In the special case α = 1 , it follows that on time step t , Tagja ( t ) =yjR ( t−1 ) and that tags on synapses onto output units k≠a are 0 . As a result ,', 'ΔwjaR∝δ ( t ) yjR ( t−1 ) =δ ( t ) Tagja ( t ) ,', '( 22 )', '=δ ( t ) Tagjk ( t ) ,', '( 23 )', 'for the synapses onto the selected action a , and the second , generalized , equation follows from the fact that ∂qk ( t−1 ) ∂wjkR ( t−1 ) =0 for output units k≠a that were not selected and therefore do not contribute to the RPE .', 'Inspection of Eqns .', '( 18 ) and ( 23 ) reveals that AuGMEnT indeed takes a step of size β in the direction opposite to the error gradient of Equation ( 19 ) ( provided α = 1; we discuss the case α≠1 below ) .', 'The updates for synapses between memory units m and Q-value units k are equivalent to those between regular units and the Q-value units .', 'Thus ,', 'ΔwmkM∝−∂Eqa∂wmkM ( t−1 ) =δ ( t ) ∂qk ( t−1 ) ∂wmkM ( t−1 ) =δ ( t ) Tagmk ( t ) .', '( 24 )', 'The plasticity of the feedback connections w′kjR and w′kmM from the Q-value layer to the association layer follows the same rule as the updates of connections wjkR and wmkM and the feedforward and feedback connections between two units therefore become proportional during learning [14] .', 'We will now show that synapses vijR between the input layer and the regular association units ( Fig . 1 ) also change according to the negative gradient of the error function defined above .', 'Applying the chain rule to compute the influence of vijR on qa ( t-1 ) results in the following equation:', 'ΔvijR∝δ ( t ) ∂qa ( t−1 ) ∂yjR ( t−1 ) ∂yjR ( t−1 ) ∂inpjR ( t−1 ) ∂inpjR ( t−1 ) ∂vijR ( t−1 ) , =δ ( t ) wjaRσ′ ( inpjR ( t−1 ) ) xi ( t−1 ) .', '( 25 )', 'The amount of attentional feedback that was received by unit j from the selected Q-value unit a at time t-1 is equal to w′ajR because the activity of unit a equals 1 once it has been selected .', 'As indicated above , learning makes the strength of feedforward and feedback connections similar so that wjaR can be estimated as the amount of feedback w′ajR that unit j receives from the selected action a ,', 'ΔvijR∝−∂Eqa∂vijR ( t−1 ) =δ ( t ) w′ajRσ′ ( inpjR ( t−1 ) ) xi ( t−1 ) .', '( 26 )', 'Recall from Eq .', '( 14 ) that the tags on synapses vijR are updated according to ΔTagij=−αTagij+xiσ′ ( inpj ) w′ajR .', 'Fig . 1B illustrates how feedback from action a controls the tag formation process .', 'If α = 1 , then on time step t , Tagij ( t ) =xi ( t−1 ) σ′ ( inpjR ( t−1 ) ) w′ajR so that Eq .', '( 26 ) can be written as:', 'ΔvijR∝−∂Eqa∂vijR ( t−1 ) =δ ( t ) Tagij ( t ) .', '( 27 )', 'A comparison to Eq .', '( 18 ) demonstrates that AuGMEnT also takes a step of size β in the direction opposite to the error gradient for these synapses .', 'The final set of synapses that needs to be considered are between the transient sensory units and the memory units .', 'We approximate the total input inpmM ( t ) of memory unit m as ( see Eq .', '( 7 ) ) :', 'inpmM ( t ) =∑lvlmM ( t ) xl′ ( t ) +∑l , t′=0t−1vlmM ( t′ ) xl′ ( t′ ) , ≈∑lvlmM ( t ) ∑t′=0txl′ ( t′ ) ,', '( 28 )', 'The approximation is good if synapses vlmM change slowly during a trial .', 'According to Equation ( 19 ) , the update for these synapses is:', 'ΔvlmM∝−∂Eqa∂vlmM ( t−1 ) =δ ( t ) ∂qa ( t−1 ) ∂ymM ( t−1 ) ∂ymM ( t−1 ) ∂inpmM ( t−1 ) ∂inpmM ( t−1 ) ∂vlmM ( t−1 ) , =δ ( t ) w′amMσ′ ( inpmM ( t−1 ) ) [∑t′=0t−1xl′ ( t′ ) ] .', '( 29 )', 'Eq .', '( 15 ) specifies that ΔsTracelm = xl so that sTracelm ( t−1 ) =∑t′=0t−1xl′ ( t′ ) , the total presynaptic activity of the input unit up to time t-1 ( blue circle in Fig . 1C ) .', 'Thus , Eq .', '( 29 ) can also be written as:', 'ΔvlmM∝δ ( t ) w′amMσ′ ( inpmM ( t−1 ) ) sTracelm ( t−1 ) .', '( 30 )', 'Eq .', '( 16 ) states that ΔTaglm=−αTaglm+sTracelmσ′ ( inpmM ) w′amM , because the feedback from the winning action a converts the trace into a tag ( panel iv in Fig . 1C ) .', 'Thus , if α = 1 then TaglmM ( t ) =w′amMσ′ ( inpmM ( t−1 ) ) sTracelm ( t−1 ) so that:', 'ΔvlmM∝δ ( t ) TaglmM ( t ) .', '( 31 )', 'Again , a comparison of Eqns .', '( 31 ) and ( 18 ) shows that AuGMEnT takes a step of size β in the direction opposite to the error gradient , just as is the case for all other categories of synapses .', 'We conclude that AuGMEnT causes an on-line gradient descent on all synaptic weights to minimize the temporal difference error if α = 1 . AuGMEnT provides a biological implementation of the well known RL method called SARSA , although it also goes beyond traditional SARSA [7] by', '( i ) including memory units', '( ii ) representing the current state of the external world as a vector of activity at the input layer', '( iii ) providing an association layer that aids in computing Q-values that depend non-linearly on the input , thus providing a biologically plausible equivalent of the error-backpropagation learning rule [8] , and', '( iv ) using synaptic tags and traces ( Fig . 1B , C ) so that all the information necessary for plasticity is available locally at every synapse .', 'The tags and traces determine the plasticity of memory units and aid in decreasing the RPE by improving the Q-value estimates .', 'If a memory unit j receives input from input unit i then a trace of this input is maintained at synapse vij for the remainder of the trial ( blue circle in Fig . 1C ) .', 'Suppose that j , in turn , is connected to action a which is selected at a later time point .', 'Now unit j receives feedback from a so that the trace on synapse vij becomes a tag making it sensitive to the globally released neuromodulator that codes the RPE δ ( panel iv in Fig . 1C ) .', 'If the outcome of a was better than expected ( δ>0 ) ( green cloud in panel', 'v ) , vij strengthens ( thicker synapse in panel', 'vi ) .', 'When the stimulus that activated unit i reappears on a later trial , the larger vij increases unit j ’s persistent activity which , in turn , enhances the activity of the Q-value unit representing a , thereby decreasing the RPE .', 'The synaptic tags of AuGMEnT correspond to the eligibility traces used in RL schemes .', 'In SARSA learning speeds up if the eligibility traces do not fully decay on every time step , but exponentially with parameter λ∈[0 , 1] [7]; the resulting rule is called SARSA ( λ ) .', 'In AuGMEnT , the parameter α plays an equivalent role and precise equivalence can be obtained by setting α = 1-λγ as can be verified by making this substitution in Eqn .', '( 13 ) ( 14 ) and ( 16 ) ( noting that Tag ( t+1 ) = Tag ( t ) +ΔTag ( t ) ) .', 'It follows that tags decay exponentially as Tag ( t+1 ) = λγTag ( t ) , equivalent to the decay of eligibility traces in SARSA ( λ ) .', 'These results establish the correspondence between the biologically inspired AuGMEnT learning scheme and the RL method SARSA ( λ ) .', 'A special condition occurs at the end of a trial .', 'The activity of memory units , traces , tags , and Q-values are set to zero ( see [7] ) , after updating of the weights with a δ that reflects the transition to the terminal state .', 'In the remainder of the results section we will illustrate how AuGMEnT can train multi-layered networks with the form of Fig . 1 to perform a large variety of tasks that have been used to study neuronal representations in the association cortex of monkeys .', 'We tested AuGMEnT on four different tasks that have been used to investigate the learning of working memory representations in monkeys .', 'The first three tasks have been used to study the influence of learning on neuronal activity in area LIP and the fourth task to study vibrotactile working memory in multiple cortical regions .', 'All tasks have a similar overall structure: the monkey starts a trial by directing gaze to a fixation point or by touching a response key .', 'Then stimuli are presented to the monkey and it has to respond with the correct action after a memory delay .', 'At the end of a trial , the model could choose between two possible actions .', 'The full task reward ( rf , 1 . 5 units ) was given if this choice was correct , while we aborted trials and gave no reward if the model made the wrong choice or broke fixation ( released the key ) before a go signal .', 'Researchers usually train monkeys on these tasks with a shaping strategy .', 'The monkey starts with simple tasks and then the complexity is gradually increased .', 'It is also common to give small rewards for reaching intermediate goals in the task , such as attaining fixation .', 'We encouraged fixation ( or touching the key in the vibrotactile task below ) by giving a small shaping reward ( ri , 0 . 2 units ) if the model directed gaze to the fixation point ( touched the key ) .', 'In the next section we will demonstrate that the training of networks with AuGMEnT is facilitated by shaping .', 'Shaping was not necessary for learning in any of the tasks , however , but it enhanced learning speed and increased the proportion of networks that learned the task within the alloted number of training trials .', 'Across all the simulations , we used a single , fixed configuration of the association layer ( three regular units , four memory units ) and Q-layer ( three units ) and a single set of learning parameters ( Tables 1 , 2 ) .', 'The number of input units varied across tasks as the complexity of the sensory stimuli differed .', 'We note , however , that the results described below would have been identical had we simulated a fixed , large input layer with silent input units in some of the tasks , because silent input units have no influence on activity in the rest of the network .', 'The first task ( Fig . 2A ) is a memory saccade/anti-saccade task modeled after Gottlieb and Goldberg [3] .', 'Every trial started with an empty screen , shown for one time step .', 'Then a fixation mark was shown that was either black or white , indicating that a pro- or anti-saccade would be required .', 'The model had to fixate within 10 time-steps , otherwise the trial was terminated without reward .', 'If the model fixated for two time-steps , we presented a cue on the left or the right side of the screen for one time-step and gave the fixation reward ri .', 'This was followed by a memory delay of two time steps during which only the fixation point was visible .', 'At the end of the memory delay the fixation mark turned off .', 'To collect the final reward rf in the pro-saccade condition , the model had to make an eye-movement to the remembered location of the cue and to the opposite location on anti-saccade trials .', 'The trial was aborted if the model failed to respond within eight time steps .', 'The input units of the model ( Fig . 2B ) represented the color of the fixation point and the presence of the peripheral cues .', 'The three Q-value units had to represent the value of directing gaze to the centre , left and right side of the screen .', 'This task can only be solved by storing cue location in working memory and , in addition , requires a non-linear transformation and can therefore not be solved by a linear mapping from the sensory units to the Q-value units .', 'We trained the models for maximally 25 , 000 trials , or until they learned the task .', 'We kept track of accuracy for all four trial types as the proportion correct responses in the last 50 trials .', 'When all accuracies reached 0 . 9 or higher , learning and exploration were disabled
['Introduction', 'Model', 'Results', 'Discussion']
['Intelligence is our ability to learn appropriate responses to new stimuli and situations .', 'Neurons in association cortex are thought to be essential for this ability .', 'During learning these neurons become tuned to relevant features and start to represent them with persistent activity during memory delays .', 'This learning process is not well understood .', 'Here we develop a biologically plausible learning scheme that explains how trial-and-error learning induces neuronal selectivity and working memory representations for task-relevant information .', 'We propose that the response selection stage sends attentional feedback signals to earlier processing levels , forming synaptic tags at those connections responsible for the stimulus-response mapping .', 'Globally released neuromodulators then interact with tagged synapses to determine their plasticity .', 'The resulting learning rule endows neural networks with the capacity to create new working memory representations of task relevant information as persistent activity .', 'It is remarkably generic: it explains how association neurons learn to store task-relevant information for linear as well as non-linear stimulus-response mappings , how they become tuned to category boundaries or analog variables , depending on the task demands , and how they learn to integrate probabilistic evidence for perceptual decisions .']
['Working memory is a cornerstone of intelligence .', 'Most , if not all , tasks that one can imagine require some form of working memory .', 'The optimal solution of a working memory task depends on information that was presented in the past , for example choosing the right direction at an intersection based on a road-sign some hundreds of meters before .', 'Interestingly , animals like monkeys readily learn difficult working memory tasks , just by receiving rewards such as fruit juice when they perform the desired behavior .', 'Neurons in association areas in the brain play an important role in this process; these areas integrate perceptual and memory information to support decision-making .', 'Some of these association neurons become tuned to relevant features and memorize the information that is required later as a persistent elevation of their activity .', 'It is , however , not well understood how these neurons acquire their task-relevant tuning .', 'Here we formulate a simple biologically plausible learning mechanism that can explain how a network of neurons can learn a wide variety of working memory tasks by trial-and-error learning .', 'We also show that the solutions learned by the model are comparable to those found in animals when they are trained on similar tasks .']
[]