Reinforcement Learning of Linking and Tracing Contours in Recurrent Neural Networks
[['Introspectively , visual perception appears to be remarkable effortless and automatic .', 'Our perceptual world is filled with familiar objects and we do not experience many difficulties in judging where an object ends and the next one begins .', 'The quality of image segmentation by the human brain surpasses segmentation in computer vision , which is known to be a hard problem [1] , yet the precise mechanisms responsible for image segmentation in humans are only partially understood .', 'In the present study we aim to explore the mechanisms that allow a neural network to learn to segment task-relevant image elements from a background of irrelevant elements .', 'Image processing in humans and non-human primates can be subdivided in at least two phases .', 'Research suggests that for many tasks the first phase is dominated by feedforward processing [2–5] ( see [6 , 7] for evidence for recurrent interactions in early vision ) .', 'When a new image is presented to the visual system , information is rapidly propagated from early to higher visual areas .', 'In this phase , the visual system extracts many elementary features such as colors , local orientations , contrasts , motion directions in low level areas and more complex features such as shape properties ( curvature , corners ) in higher areas [8] .', 'This early processing phase thereby produces a pattern of activity across the various areas of the visual cortex that has been called “base representation” [8] .', 'This early representation even includes certain object-categories such as animals or vehicles , which are detected very soon after the image has been presented [9 , 10] .', 'However , there are also many aspects of visual processing that require a more elaborate analysis than can be achieved during the first feedforward processing phase [4 , 11] .', 'Some tasks , such as image segmentation and contour grouping , demand the evaluation of the relations between items ( spatial as well as temporal ) .', 'Perceptual grouping and segmentation processes depend on a later serial processing phase where lateral connections between neurons in the same area and feedback connections that propagate information from higher areas back to lower areas come into play [2–5 , 12 , 13] .', 'In this second processing phase , the activity of neurons in lower areas of the visual cortex is not only determined by the visual information in the neurons’ receptive field itself , but there is an additional influence of the context given by the activity of other neurons in the same area as well as representations in higher visual areas that provide feedback [11] .', 'An example task where lateral and feedback connections play a role is contour grouping .', 'For example , if monkeys are trained to detect a string of collinear contour elements [14] , the neuronal responses in the primary visual cortex elicited by these elements are stronger than the responses elicited by line elements that are not part of such a perceptual group ( Fig 1A; c . f . [14–16] ) .', 'Interestingly , this contextual effect even occurs if the information in the V1 receptive field is held constant , which implies that it depends on the lateral influences of V1 neurons with different receptive fields and/or on feedback from higher visual areas where receptive fields are larger [8] .', 'In accordance with this view , the effect of grouping on V1 activity does not occur during the initial visual response but at an additional delay ( Fig 1A ) .', 'Another contour grouping task that is thought to rely on feedback and lateral connectivity is curve-tracing ( Fig 1B ) [8 , 17–19] .', 'In one version of this task , monkeys had to determine the endpoint of a target curve that started at the fixation point ( “FP” in Fig 1B ) as target for an eye movement , while ignoring another , distractor curve that was not connected to the fixation point .', 'Also in this task , the contour elements that belong to the target curve were labeled in the visual cortex with enhanced spiking activity , and again , only after a delay relative to the initial visual response .', 'The labeling of image elements with enhanced neuronal activity for grouping them into a coherent representation has been called incremental grouping [8] .', 'The delayed response modulation reflects the monkeys’ belief about the stimulus [20] .', 'It is much weaker if the monkey fails to perceive the perceptual group [15] .', 'Furthermore , if the monkey groups the wrong set of image elements in his perception , this erroneous set of contour elements is labeled with enhanced activity [17 , 21] .', 'A number of previous modeling studies have investigated how feedforward , lateral and feedback connections determine the response of visual cortical neurons in these contour grouping tasks [22–25] .', 'A key question that remains to be addressed , however , is how these grouping operations are learned during perceptual experience , because visual experience improves the detection and integration of image features [26–31] .', 'That visual experience aids in image segmentation and perceptual grouping also follows from the fact that these processes are more efficient when objects are presented in their familiar orientations than when they are shown upside down [32 , 33] .', 'Perceptual learning in the contour integration task ( Fig 1A ) has been well documented in previous work .', 'In this task , two weeks of experience greatly improve the accuracy of monkeys in detecting collinear image elements .', 'Importantly , this training also increases the strength of the neuronal response modulation in V1 [14 , 16] .', 'During learning , the only feedback that the monkeys receive about their performance is a reward when they correctly detect the string of collinear contour elements and the omission of a reward when they fail to make the appropriate response .', 'However , the neuronal mechanisms that underlie these reward-based perceptual grouping improvements are not well understood .', 'How does the visual brain change connections between the appropriate neurons when feedback about performance is so limited ?', 'In the present study we build upon previous models called AGREL ( Attention-Gated Reinforcement Learning ) and AuGMEnT ( Attention-Gated MEmory Tagging ) that have been proposed for the learning of feedforward connections from lower to higher areas in various cognitive tasks [34–36] and extend it to recurrent neural networks .', 'These previous learning models proposed a three-stage mechanism for the adaptation of synaptic weights .', 'In a first step , feedforward processing determines a winning unit in the output layer of the network that encodes the chosen action .', 'In a second step , an attentional feedback ( AFB ) signal originating from the winning unit assigns credit to those connections that were responsible for the chosen action by creating synaptic tags .', 'In the third step , a global learning signal determines the changes of the weights of those synapses that carry the plasticity tag .', 'This global learning signal presumably corresponds to a neuromodulator such as dopamine , serotonin or acetylcholine that encodes the reward-prediction error , i . e . the difference between the amount of reward that was expected and the amount of reward that was actually received by the network [37–39] .', 'These previous models only used attentional feedback signals originating from the output layer ( step two ) to highlight task relevant synapses , but did not use feedback or lateral connections for the labeling of image elements that belong to the same perceptual group with an enhancement of neuronal firing rates ( as in Fig 1 ) .', 'Thus the mechanisms that permit reward-based learning in perceptual grouping tasks that rely on activity propagation through feedback and lateral connections have remained relatively unexplored .', 'To address learning in tasks with recurrent networks that utilize feedforward , lateral and feedback connections we will here propose a new learning scheme called RELEARNN ( REinforcement LEArning in Recurrent Neural Networks ) where the activity of the network depends in a complex manner on the visual input pattern , as recurrent connections allow the recirculation of activity .', 'We will illustrate the capabilities of the new learning algorithm with the two contour grouping tasks of Fig 1 , namely contour linking and curve tracing [14 , 40] .', 'We tested if RELEARNN can train the same neural network layout ( and with the same parameters ) to perform either task if the only feedback is a reward for correct performance and the omission of a reward in case of an error .', 'We also investigated whether these networks develop coding strategies that resemble those in the visual cortex , where image elements of the same perceptual group are labeled with enhanced neuronal activity , and if and how these groups can be read out by higher areas to support the selection of the appropriate behavioral response .', 'Our results demonstrate that ( 1 ) the behavior of the neural networks during learning and the pattern of errors is similar to that of monkeys that are trained in these tasks [14 , 40] , ( 2 ) the model reproduces the changes in neuronal firing rates and the emergence of incremental grouping during the learning process , ( 3 ) that the labeling with enhanced neuronal activity for grouping is an efficient code that can be used to guide behavior , and ( 4 ) that RELEARNN is a comprehensive and powerful learning scheme , which captures fundamental aspects of synaptic plasticity in a recurrent neural network .', 'The model may thereby help to understand how neuromodulatory systems that code reward-prediction errors enable the learning of complex perceptual tasks that require the interactions between many units through feedforward , lateral and feedback connections .', 'RELEARNN may represent the first recurrent neural network learning scheme that can explain the learning of both contour linking and curve tracing with a biologically plausible learning scheme .', 'In the following , we will first describe the proposed learning algorithm and the architecture of the neural network model that can be trained to perform the contour linking and curve tracing tasks .', 'We will then present the simulations and compare them to the behavior and neuronal activity in monkeys trained on the same tasks , and will close with a comparison between RELEARNN and previous learning models .'], ['The aim of the model is to compute the value of actions when it is presented with a visual stimulus .', 'The model contains a number of output units ( Fig 2 ) , and it aims to approximate the value of each of the possible actions .', 'These action values ( known as Q-values [41] ) are coded by the activity of the output units .', 'The model usually chooses the action with the highest Q , but it will occasionally also explore other actions to promote learning .', 'To find an appropriate balance between biological detail and mathematical tractability we chose model units with a scalar activation value , but to not include spiking neurons in our model .', 'The model units represent the average activity in a cortical column with mean membrane potential p and mean firing rate g ( p ) .', 'As inputs , the model units receive excitation , inhibition as well as modulatory influences and the units , in turn , can inhibit , excite or modulate other model units .', 'The role of the modulatory connections is to amplify the influence of excitatory connections , but these modulatory connections are unable to drive the units themselves ( c . f . [42–46] ) .', 'Note that modulatory connections have an effect that is synapse specific .', 'Their role differs from the more global neuromodulatory signals like dopamine , acetylcholine or serotonine that are globally available in the network and gate plasticity .', 'The model units used here have proven to be versatile building blocks of neural networks in previous studies ( e . g . [47–56] ) .', 'The membrane potential p depends on the excitatory , inhibitory and modulatory inputs Iex , Iinh and Imod as follows ( Fig 2 , right ) :', 'd d t p = - α p + ( β - p ) · I e x · ( 1 + γ I m o d ) - ( ζ + p ) · I i n h .', '( 1 )', 'The decay rate of the potential of model units is controlled by α > 0 , the maximal potential by β > 0 , the minimal potential by ζ > 0 , and the parameter γ > 0 determines the impact of modulatory input .', 'The mean spike rate r depends on the potential and is calculated as', 'r = g ( p ) = { a + p , p ≥ 0 , a · exp ( p / a ) , p < 0 , ( 2 )', 'with a = 0 . 001 .', 'Because the membrane potential is bounded by β ( and −ζ ) , and because the network’s aim is to compute the action values as the activity p of the output units , we chose the reward of correctly performed trials below β and gave no reward if the model made an erroneous response .', 'We will now derive the learning algorithm , considering a network of N dynamically interacting model units with activities pi receiving excitatory input Iinp ( see Fig 2 for the general structure of such a network , which may or may not be fully connected ) .', 'Once the input has been given , the activity circulates through the excitatory , inhibitory and modulatory connections until the network activity stabilizes ( convergence to a stable state is required for the current version of the learning algorithm ) .', 'The overall dynamics are described by the following system of coupled differential equations ( same as Eq ( 1 ) , but now presented in vector notation )', 'd d t p = - α p + ( β - p ) · I e x · ( 1 + γ I m o d ) - ( ζ + p ) · I i n h .', '( 3 )', 'The excitatory , inhibitory and modulatory inputs Iex , Iinh and Imod depend on the presynaptic firing rates and the input into the network Iinp I e x = ( W e x ) T · g ( p ) + ( W i n p ) T · I i n p , I i n h / m o d = ( W i n h / m o d ) T · g ( p ) .', '( 4 )', 'Here , α , β , γ , ζ ≥ 0 and g ( applied element-wise ) are defined as in Eq ( 1 ) and p , Iinp ∈ ℝN are column vectors of the activations and inputs of each unit .', 'The positive elements W k l ( ⋅ ) ≥ 0 of the weight matrices W ( ⋅ ) ∈ ℝN × N determine the connection strength from unit k to unit l and W k l i n p determines the excitatory connection strength from feature k to input unit l ( here products of two column vectors like p ⋅ I ∈ ℝN are defined element-wise ) .', 'When the activity in the network has converged to a stable state ( usually within a few hundred iterations ) , the network chooses one action based on the activation of the output units that encode the action ( Q ) values .', 'We used the softmax rule to determine the probability φa of an output unit a to win the competition between all possible actions based on their values:', 'φ a = exp ( p a / τ ) ∑ j ∈ O exp ( p j / τ ) , ( 5 )', 'where O is the set of all output units and τ is called a temperature parameter [41] , which was adjusted to fit the experimental data ( c . f . Suppl . A in S1 Text; adjusted by grid search ) .', 'We here did not model how the softmax action selection process is implemented in the neural network , although this has been adressed in previous work [57] .', 'Moreover , the choice of softmax as an action selection rule is not critical .', 'We expect that other action selection mechanisms used in the reinforcement literature ( e . g . ɛ-greedy [41] or max-Bolzman [58] ) will give qualitatively similar results .', 'Although research in past years has increased our knowledge about the mechanisms for the long-term modification of synaptic strength ( e . g . [59 , 60] ) , we will here focus on rather simple rules for synaptic modification .', 'It is convenient to subdivide the mechanisms that lead to synaptic plasticity into a number of phases ( c . f . [34 , 35 , 61] ) .', 'Here we will distinguish between three phases .', 'Phase one starts in response to the input and ends when the network converges to a stable state p∞ and stochastically selects action a according to Eq ( 5 ) .', 'In the second phase , the selected output unit a causes an attentional feedback signal ( AFB ) that propagates through the network through a separate set of units ( one per column; small circles in Fig 3 ) that change their response by Δp during this phase so that their total activity becomes p∞ + Δp .', 'We call the network of units sensitive to the AFB the “accessory network” ( see below for details ) , which is important for the guidance of synaptic plasticity .', 'Although many neurophysiological studies on the neuronal correlates of action selection targeted frontal and parietal cortex [62 , 63] , more recent studies reported that some neurons in the visual cortex are indeed also influenced by action selection , which is in accordance with the propagation of selection signals through an accessory network [64 , 65] .', 'More generally , there is evidence for a ubiquitous bi-directional “counter stream” interaction between higher and lower cortical regions [66–69] also justified by theoretical considerations ( e . g . the selective tuning model for visual attention [70–72] ) .', 'Furthermore , there is anatomical data to support a dichotomy of cortico-cortical connections .', 'Studies from the lab of Sherman on the connectivity between areas of mouse cortex distinguished between class I and class II connections [73 , 74] .', 'Class I connections were called “drivers” because they are thought to activate neurons .', 'They project from lower to higher areas but also in the opposite direction , from higher areas back to lower areas .', 'Class II connections also project both in the feedforward and feedback direction , but they differ from class I connections because they are weaker and utilize the metabotropic glutamate receptor , involved in synaptic plasticity .', 'It is therefore conceivable that the accessory network might employ class II connections , while the regular network employs class I connections , although this mapping is presently still speculative .', 'We will assume that the strength of the connections between units in the accessory network is similar ( or proportional ) to the strength of connections between the regular units ( larger circles in Fig 3 ) .', 'This reciprocitity of regular and accessory connections is not a strong assumption because it can also be learned [34] .', 'As we will demonstrate below , the boost in the membrane potential Δpl of the accessory unit l during the second phase is proportional to the influence of a change in pl on the activity of the current winning unit p a ∞ during the first phase .', 'Therefore , the AFB can be used to assign credit to those units that had an impact on the decision to take action a .', 'Specifically , we will show that if an increase in the activity of unit nl would increase the activity of the winning unit p a ∞ during the first phase , then Δpl > 0 during the second phase .', 'If , on the other hand , unit ni decreases p a ∞ , this implies that Δpl < 0 .', 'The sign and magnitude of Δpl can be used to guide plasticity of synapses onto unit l if the aim is to decrease or increase p a ∞ , i . e . to adjust the value of this action .', 'Finally , in the third phase ( c . f . Fig 3 ) , the network receives a reward if it selects the correct action , but reward is omitted otherwise .', 'The output units of the network aim to represent the expected value if their action is chosen in the current sensory state ( c . f . [35 , 75] ) .', 'Neurons in the frontal cortex , basal ganglia and midbrain are known to code for action values [76 , 77] , i . e . their activity appears to approximate the so-called Q-value , defined as the expected return ϱ when choosing action a in state s ( c . f . [41]; note that rewards are often delayed; see discussion for extensions that deal with delayed rewards )', 'Q a = E π { ϱ | s , a } .', '( 6 )', 'When the network performs action a , it receives a reward ϱ and the aim of the learning rule is to adjust the current estimate of Qa , represented by the activity of the winning output unit p a ∞ .', 'To this aim , the network computes a reward prediction error δ by comparing the outcome of the trial ϱ to the predicted Q-value , i . e . a SARSA style prediction error for immediately rewarded tasks [41] .', 'In accordance with previous studies of reinforcement learning [78 , 79] , we assume that this reward prediction error is coded by a neuromodulatory signal that is globally released into the network so that it can influence the plasticity of all synapses ( Fig 3 , right panel ) .', 'δ = ϱ - Q a = ϱ - p a ∞ .', '( 7 )', 'Many dopamine neurons in the ventral tegmental area and substantia nigra encode the reward prediction error δ [37 , 80 , 81] .', 'Once the network has received feedback about the chosen action a , the learning rule changes the connections of the network in order to decrease the reward prediction error for this action .', 'Specifically , plasticity of the connection wkl from unit k to unit l depends on four factors: ( 1 ) the presynaptic activity r k ∞ , ( 2 ) the postsynaptic membrane potential p l ∞ , ( 3 ) the activity of the accessory unit l Δ p l ∞ , which represents the influence of unit l on the activity of a and ( 4 ) the reward prediction error δ:', 'Δ W k l = η · δ · Δ p l ∞ · f l ( p l ∞ ) · r k ∞ , ( 8 )', 'where η is the learning rate .', 'Note that the signals that determine plasticity are all available locally in the cortical column l and that Eq ( 8 ) implements a form of Hebbian plasticity , because it depends on the product of presynaptic activity r k ∞ and a function f ( ⋅ ) of the postsynaptic activity p l ∞ .', 'We will derive below that the form of f ( ⋅ ) differs between excitatory , inhibitory and modulatory connections projecting to column l:', 'f l e x ( p l ∞ ) = ( β - p l ∞ ) · ( 1 + γ · ( I ∞ m o d ) l ) , ( 9 ) f l m o d ( p l ∞ ) = γ · ( β - p l ∞ ) · ( I ∞ e x ) l , ( 10 ) f l i n h ( p l ∞ ) = - ( ζ + p l ∞ ) .', '( 11 )', 'These equations follow from Eqs ( 1 ) and ( 3 ) and can be obtained by differentiating with respect to the corresponding input type ( they represent the effective impact of the connection on the post-synaptic unit l ) .', 'The aim of the accessory network is to determine Δ p l ∞ for every unit l , which estimates the influence of l on the value of the selected action a .', 'In the sequel we will capitalize on results of Almeida and Pineda [82 , 83] who proposed a method to compute the error gradient in weight space for fully recurrent networks .', 'It is based on the error-backpropagation rule , which computes the required changes in synaptic weights but it is thought to be biologically implausible [84 , 85] .', 'However , previous work on the AGREL [34] and AuGMEnT [35] learning rules demonstrated that in case of a reinforcement learning problem it is possible to replace the backpropagation of errors by two factors that are biologically plausible ( reviewed by [86] ) : ( 1 ) an attentional feedback signal , which propagates activity rather than error signals from the output units to earlier processing levels , and ( 2 ) a globally released neuromodulatory signal that codes for the reward prediction error .', 'We will here provide an equivalent result for the learning rule proposed by [82 , 83] for fully recurrent networks .', 'Almeida [83 , 87 , 88] showed that once a fully recurrent network has settled in a stable state p∞ , the error gradient of all synapses can be computed by an accessory network if it is the transpose of the linearized original network .', 'Thus , in the accessory network of [83] connections Wkl are replaced by connections W l k ′ of equal strength but running in the opposite direction .', 'His method injects the error signal in every output unit , i . e . a positive signal for output units that are not active enough and a negative signal for output units that are too active .', 'The accessory network of Almeida then propagates the activity to units connected to the output units , and from there , successively to all other units in the network .', 'In RELEARNN only the winning output unit a injects activity into the accessory network and this activity circulates until the accessory units reach a stable state .'], ['Li , Piëch and Gilbert trained monkeys to group colinearly aligned contour elements using the task illustrated in Fig 1A [14] .', 'In every trial , the monkey saw two sets of short line elements within circular apertures .', 'One of the apertures contained a contour pattern composed of a number of colinear line elements and the other pattern contained line elements with a random orientation .', 'The overall orientation of the colinear contour was fixed during a training session , but it could vary across sessions .', 'At the start of a trial , the monkey had to direct gaze to the center of the display .', 'The animals’ task was to make an eye movement to the aperture with the contour pattern ( Fig 1 ) and the animals performed approximately 1500 trials per day .', 'The authors started with a pretraining phase of several days where the target contour was shown at a high luminance but the distractor elements were less visible because they had a lower luminance .', 'This was followed by the main training sessions where the distractor elements had the same luminance as the target elements so that only colinearity remained as a cue to solve the task .', 'We investigated whether RELEARNN could train the network of Fig 4 by rewarding saccades to the contour pattern .', 'Will the network learn to detect the colinearly aligned contour elements and will it reproduce the monkeys’ behavior during learning ?', 'Moreover , how does the activity of network units compare to the neuronal activity in the visual cortex of the monkeys [14] ?', 'Li et al . trained their monkeys to maintain fixation for several hundreds of milliseconds upon presentation of the patterns and to make the saccade after this delay [14] .', 'We trained the neural network on a version of the task that was simpler because we did not model the different phases of every trial .', 'The model could immediately select the eye-movement upon presentation of the pattern and convergence to a stable activity state ( learning of multiple task epochs by a feedforward network has been addressed by [35 , 36] ) .', 'The input to the network consisted of two patterns of bars with four possible orientations presented on a grid with 9 × 9 spatial positions on either side of the fixation point .', 'There were a total of 648 input units ( 81 locations per pattern × 4 orientations × 2 patterns ) that provided direct excitation and disynaptic inhibition via inhibitory units to the excitatory units of the linking layer , with 2 × 4 × 9 × 9 units , one for every orientation at each retinotopic location .', 'Excitatory units in the linking layer projected to units with adjacent receptive fields tuned to the same orientation ( four nearest neighbors ) and also to the three units with the same receptive field tuned to other orientations with modulatory connections .', 'Units of the linking layer propagated activity to a total of four units in the association layer , which , in turn , projected to the two units of the motor layer that had to learn the expected reward associated with a saccade to the left or right pattern .', 'To facilitate learning , we started with a short ( 500 stimulus presentations ) preliminary procedural training phase ( just like [14]; this was not necessary , however , to successfully learn the task ) .', 'During the pre-training phase , we made the background surrounding the contour elements less salient than the target pattern by reducing their contrast ( activity of input units representing background elements was set to 50% ) .', 'We then increased the contrast of the background contrast in increments of 10% ( 100 trials per contrast step ) until all elements had 100% contrast .', 'The pre-training phase was followed by training in the full task ( Fig 5 ) .', 'Training caused a gradual increase in the model’s accuracy and the effect of training was particularly pronounced for patterns with a larger number of colinear line elements .', 'The effect of training on the accuracy of the model resembled the improvement in the accuracy of the monkeys ( c . f . Fig 5 left vs . right ) .', 'The model was able to learn the contour linking task within ∼ 15 , 000 trials ( ∼ 3 , 000 trials per contour length ) , which is comparable to the time-course of the monkeys’ behavioral improvement ( 10 days with ∼ 1 , 500 trials per day ) .', 'The results of these simulations were not critically dependent on the exact parameter values , i . e . slight parameter changes did not qualitatively alter the outcome , although less optimal values of the parameters decreased learning speed .', 'We next investigated how learning of the contour integration task changed the activity of network units ( Fig 6 ) .', 'After the pre-training , but before the training in the full task , the activity of the excitatory units in the linking layer did not depend strongly on the number of colinearly aligned contours in the target pattern ( Fig 6A ) .', 'After training , however , the activity of these units clearly depended on the number of colinearly aligned contour elements ( Fig 6B ) .', 'Patterns with more colinearly aligned contour elements elicited stronger responses , thereby reproducing the effect of training in area V1 of monkeys ( Fig 6D and 6E ) .', 'It is also of interest to compare the time course of activity in the model’s linking layer to the time-course of activity in monkey V1 .', 'In the trained model , tuning to the number of colinear line elements was relatively weak during the initial response , but it became stronger when the network units started to converge to their stable activity level .', 'The feedforward activation from the input layer only conveys information about the orientation of the contour element in the receptive field and not about the context provided by image elements in the surround .', 'Thus , tuning to colinearity must be due to alterations in horizontal and feedback influences on the units , and this explains the delay in the emergence of this tuning .', 'The temporal profile of the response of model units is quite comparable to that of V1 neurons , for which tuning to the colinearity of image elements also emerges during the later phase of the response .', 'In their experiment , Li et al . [14] also anesthetized the monkeys after the training procedure with the aim to block the top-down and horizontal influences .', 'During anesthesia , the tuning to colinearity was indeed abolished in area V1 , whereas visually driven activity was maintained , albeit weaker ( Fig 6F ) .', 'We could replicate the effect of anesthesia in our model by removing the feedback and lateral connections ( Fig 6C ) , which confirmed that these contextual influences in the model are mediated by recurrent , i . e . lateral and feedback connections .', 'In addition , we tested the respective contribution of lateral and feedback interactions by deleting one set of connections or the other ( Fig 6G and 6H ) .', 'The results indicate that lateral interactions have a more pronounced effect than feedback interactions in this task , a prediction that can be tested in future experiments .', 'We next investigated how training in the contour linking task influenced the pattern of connectivity .', 'We first examined the horizontal connections in the linking layer of an example network that had been trained to criterion ( Fig 7 ) .', 'It can be seen that the lateral connections were strengthened along the direction corresponding to the axis of the unit’s orientation tuning .', 'We next examined the pattern of excitatory feedforward connections from the linking layer to units in the association layer .', 'In this analysis we focused on association units with a strong connection to motor units , which have an impact on the computation of the action values .', 'We found that the association units integrated the activity of units along the target contours , with a specific set of connections from each of the four orientation maps ( Fig 8A ) .', 'When we examined the pattern of modulatory feedback connections from the association layer back to the linking layer we found that connections of the regular network had become largely reciprocal ( in addition to the accessory network connections , which are enforced to be reciprocal ) .', 'Association units tended to have strong feedback connections to units that provide
['Introduction', 'Models', 'Results', 'Discussion']
['The processing of a visual stimulus can be subdivided into a number of stages .', 'Upon stimulus presentation there is an early phase of feedforward processing where the visual information is propagated from lower to higher visual areas for the extraction of basic and complex stimulus features .', 'This is followed by a later phase where horizontal connections within areas and feedback connections from higher areas back to lower areas come into play .', 'In this later phase , image elements that are behaviorally relevant are grouped by Gestalt grouping rules and are labeled in the cortex with enhanced neuronal activity ( object-based attention in psychology ) .', 'Recent neurophysiological studies revealed that reward-based learning influences these recurrent grouping processes , but it is not well understood how rewards train recurrent circuits for perceptual organization .', 'This paper examines the mechanisms for reward-based learning of new grouping rules .', 'We derive a learning rule that can explain how rewards influence the information flow through feedforward , horizontal and feedback connections .', 'We illustrate the efficiency with two tasks that have been used to study the neuronal correlates of perceptual organization in early visual cortex .', 'The first task is called contour-integration and demands the integration of collinear contour elements into an elongated curve .', 'We show how reward-based learning causes an enhancement of the representation of the to-be-grouped elements at early levels of a recurrent neural network , just as is observed in the visual cortex of monkeys .', 'The second task is curve-tracing where the aim is to determine the endpoint of an elongated curve composed of connected image elements .', 'If trained with the new learning rule , neural networks learn to propagate enhanced activity over the curve , in accordance with neurophysiological data .', 'We close the paper with a number of model predictions that can be tested in future neurophysiological and computational studies .']
['Our experience with the visual world allows us to group image elements that belong to the same perceptual object and to segregate them from other objects and the background .', 'If subjects learn to group contour elements , this experience influences neuronal activity in early visual cortical areas , including the primary visual cortex ( V1 ) .', 'Learning presumably depends on alterations in the pattern of connections within and between areas of the visual cortex .', 'However , the processes that control changes in connectivity are not well understood .', 'Here we present the first computational model that can train a neural network to integrate collinear contour elements into elongated curves and to trace a curve through the visual field .', 'The new learning algorithm trains fully recurrent neural networks , provided the connectivity causes the networks to reach a stable state .', 'The model reproduces the behavioral performance of monkeys trained in these tasks and explains the patterns of neuronal activity in the visual cortex that emerge during learning , which is remarkable because the only feedback for the model is a reward for successful trials .', 'We discuss a number of the model predictions that can be tested in future neuroscientific work .']
[]