Distributed Fading Memory for Stimulus Properties in the Primary Visual Cortex
[['A number of analysis methods have been designed to investigate how neuronal spiking activity correlates to sensory stimulation or behavior and , as expected , many relations have been found .', 'All of the following variables—firing rates [1]–[4] and the synchronization [4]–[7] and the differential timing of spiking activity [8]–[10]—have been reported to correlate , one way or another , with stimuli and/or with behavior .', 'Most of the efforts to distinguish between relevant and epiphenomenal variables face the problem that they have to rely on correlative rather than causal evidence .', 'In the absence of direct access to the information of causal nature , one way to begin addressing this issue is to complement the highly specialized statistical methods that have been developed to detect specific stimulus-related changes in responses [9] , [11]–[18] with analyses that make minimal assumptions about putative codes [19]–[21] , and examine how much information can be extracted by a single cortical neuron from the joint responses of its presynaptic neurons and then find out which variables were carrying the relevant information [22] , [23] .', 'To adjust to the conditions under which these cortical “readout” neurons are likely to operate , the data must be presented to the readout units in parallel , i . e . , the responses of the putative feeding neurons must be recorded simultaneously .', 'Also , analysis and classification of data should correspond to that conducted in real time .', 'Thus , the results should be available with short delays .', 'Hence , the analysis method should be allowed no more time for accumulating evidence than is available for cortical neurons to accomplish their task .', 'To fulfill these requirements , readout neurons need to be simulated on a computer and fed with data obtained in parallel recordings from a large number of cortical neurons , the assumption being that a certain fraction of these neurons provides input to neuronal classification .', 'The present study was designed to fulfill these premises .', 'Applying multielectrode technology , we recorded simultaneously from a large number of neurons in area 17 of lightly anesthetized cats and evoked responses with brief sequences of different , stationary flashed stimuli ( letters of the alphabet ) .', 'These responses were then fed as inputs to artificial readout neurons simulated on a computer ( also referred to as classifiers ) .', 'We first trained the artificial neurons to classify the visual stimuli that have generated the responses , and once the classification had reached criterion , we investigated which of the information-carrying variables were used by these artificial—but neuron-inspired—readout systems .', 'A recent study provided evidence that an artificial readout system can extract stimulus-specific information from the spiking activity of neurons that were sequentially recorded from macaque inferotemporal ( IT ) cortex [23] .', 'However , in this study , responses of different neurons that were supplied in parallel to the readout system had been obtained by recording sequentially from the different cells .', 'This approach eliminates temporal relations between neuronal responses that are not time locked to the stimulus , but established by internal interactions ( referred to also as “noise correlation” [24] ) .', 'Thus , it was not possible to determine whether stimulus-specific information is encoded by internal adjustment of spike timing , as this information cannot be retrieved unless responses are recorded simultaneously .', 'In addition to the identification of response variables containing stimulus-specific information , we were particularly interested in determining the duration over which this information was retrievable after the stimuli have disappeared .', 'In case there was evidence for “fading memory” already in primary visual cortex , the data would allow for inferences on the mechanisms underlying perceptual phenomena such as visual persistence and iconic storage [25]–[29] .', 'Moreover , the lack of evidence for fading memory would provide an important constraint for hypotheses on processing modes implemented in primary visual cortex .', 'Computer simulations of sparsely connected recurrent circuits of neuron-like processing units [30]–[33] as well as theoretical analyses of related circuits with simpler ( linear ) processing units [34] , [35] have shown that systems with fading memory exhibit powerful computational capabilities because they permit integration of stimulus information over time .', 'Negative evidence of fading memory would render such models unlikely and support more classical views that emphasize sequential step-by-step processing of single frames resulting from a pipelined organization of the circuits [36] , [37] .', 'In order to distinguish between processing with or without fading memory , we presented sequences of different stimuli and investigated whether information about preceding stimuli was erased by the presentation of subsequent stimuli or whether it persisted .', 'The latter case would provide strong evidence for the existence of fading memory , whereas the former would be supportive of models favoring frame-by-frame analysis of successive stimuli .'], ['We used silicon-based multielectrodes ( Michigan probes , see Materials and Methods ) to obtain highly parallel recordings from up to 124 randomly selected neurons in the primary visual cortex ( area 17 ) of lightly anesthetized cats .', 'These neurons , most of which had overlapping receptive fields ( RFs ) , were activated by presenting stationary flashed ( 100 ms ) uppercase letters ( A , B , C , D , and E ) .', 'The letters were always shown with maximal contrast and were either presented singly , one per trial , or in a sequence of two or three ( e . g . , ABC; and a 100-ms blank interval between letters in most experiments ) ( see Table S1 for experiments made on each cat ) .', 'The times at which letters were presented are indicated in all graphs by gray rectangles .', 'Each stimulation condition was presented 50 to 300 times in a randomized order .', 'To emulate classification processes realized with leaky integrate-and-fire ( I&F ) readout neurons ( but without reset or a refractory period ) , the time stamps of discharge sequences were first convolved with an exponentially decaying kernel ( if not specified otherwise , time constant , τ\u200a=\u200a20 ms ) that mimicked the time course of excitatory postsynaptic potentials ( EPSPs ) [38] , [39] ( see Materials and Methods ) .', 'The continuous convolved signal was fed to the classifier system , as illustrated in Figure 1C .', 'If not specified otherwise , the classification was made by a single I&F readout whose activation , at a time point , t , was computed as the weighted sum of the input signals at that time .', 'The classifier “fired” ( i . e . , detected the stimulus ) if its activation crossed a threshold .', 'The optimal weights and the firing thresholds of the classifiers were found ( learned ) by applying support vector machines ( SVM ) with linear kernels .', 'All reported results were obtained from test datasets derived from trials that were not part of the training datasets , only the latter being used to adjust the input weights for the readout neurons ( see Materials and Methods for a 10-fold cross-validation scheme ) .', 'In a few additional analyses ( reported later in “Superposition of Information about Different Stimuli” ) , we used also SVMs with polynomial and radial basis function kernels to investigate whether classification performance improved with these more sophisticated , nonlinear transformations of the input variables .', 'If not specified otherwise , the results are reported for classifiers trained only on a single instance in time , t , which were the values of the continuous signal resulting from the convolution of the spike trains by the exponential kernel ( Figure 1C ) .', 'The total number of values used to train each classifier depended on the number of trials ( see Table S2 ) .', 'Thus , for most analyses , different classifiers were trained for each time point along the trial , each classifier having its own unique set of weights .', 'These classifiers are denoted as Rt and should be distinguished from Rint classifiers , which were trained to generalize by using only one set of weights to classify neuronal responses over longer periods of time ( e . g . , 100-ms or 300-ms long ) .', 'In general , Rt classifiers have better performance than Rint classifiers because a single Rint classifier uses only one set of weights to accomplish a number of different classification tasks , each taking place at a different time point .', 'A corresponding collection of Rt classifiers approaches this problem by sharing the workload—each classifier having its own set of weights optimized for the unique properties of the responses at that time point , t .', 'Consequently , the recent history affecting the activity at time t imposes fewer constraints on the Rt than on the Rint classifier , the latter having to accommodate variability of response properties over an entire time interval .', 'All classifications were made as binary choices among two stimuli ( such as A and D ) , the chance level for correct classification being 50% .', 'In all graphs , the classification performance of Rt ( Rint ) readouts is shown as percentage correct classifications of the presented stimulus ( see Materials and Methods for more details on experimental and analysis procedures ) .', 'Responses that carry the information about past stimuli should also carry information about the most recent stimuli , i . e . , about those that evoked the responses .', 'The results of the following analyses indicate that this is the case .', 'In Figure 4A , a readout is fed with inputs from the same set of neurons as in Figure 3A ( and in Figure 2A ) , but this time , the readout is trained to classify the second stimulus in the sequence of three letters ( letter B vs . D ) .', 'The classification performance was high ( ∼80% ) despite the high degree of similarity of the shapes .', 'Consistent with our previous findings , classification performance correlated positively with Mfr ( r\u200a=\u200a0 . 39 for the entire period of 800 ms ) .', 'In Figure 4B and 4C , we show results for cats 2 and 3 in which both the first and second letter in the sequence were varied at the same time , allowing us to estimate whether information was available to classify simultaneously both the first and the second stimulus in the sequence .', 'We found that such sequence-specific information was available in the on- and off-responses to the second stimulus .', 'During this period , the classification performance for the first stimulus was not lower than the performance for the recent stimulus that evoked the response ( e . g . , around 70% correct in both cases for cat 3 ) .', 'We could also show that readouts could be trained to perform nonlinear exclusive OR ( XOR ) classification , i . e . , fire if either a sequence AB_ or CD_ was presented , but not if sequences AD_ or CB_ were presented ( for details , see “Nonlinear superposition of information” in Text S1 ) .', 'In Figure 4D and 4E , we show that these findings generalized across all the cats and , as a rule , also across different recordings made from the same cat .', 'For the classification of the first letter , the gross average maximum performance across the total of all 16 recordings peaked at ∼80% correct during the presentation of the second letter ( Figure 4D ) , and similarly , the classification of the second letter peaked at ∼75% correct during the presentation of the third letter ( Figure 4E ) .', 'In Figure 4D and 4E , one can also see one more remarkable result: classification performance is typically better for off- than for on-responses ( e . g . , in Figure 4D , the performance is overall higher at 250 ms than at 150 ms ) .', 'This result can also be seen in Figures 2A , 3A , and 3B ( i . e . , for cats 1 and 2 ) .', 'The results indicate that successful stimulus classification is a ubiquitous phenomenon and that off-responses contain substantial information about the stimuli even when interacting with on-responses to stimuli presented next in the sequence .', 'The information extracted by the classifier could be encoded either by slow changes in firing rates ( slow-rate code; e . g . , ∼100 ms ) or by precise timing of neuronal spiking events ( i . e . , fine temporal code; e . g . , ≤∼20 ms ) or both .', 'Our next step was to investigate the contribution of information encoded at these two different time scales .'], ['Olshausen and Field [63] have argued recently that fundamental aspects of visual cortex functions are still unknown .', 'The present results support this notion in that they force us to extend classical theories on visual processing .', 'These are based on the notion that the main functions of the visual processing hierarchy consist of the extraction and recombination of elementary visual features in divergent–convergent and hierarchically organized feedforward architectures ( e . g . , [37] ) .', 'In this case , scene analysis is essentially based on frame-by-frame computations in which each new input is processed independently of the previous one .', 'Hence , there is no memory for the past , and such memory is even considered disturbing rather than useful .', 'In contrast , we find evidence for fading memory in combination with nonlinear temporal interactions , at least in early visual areas .', 'Processes supporting both perception and action may benefit from such memory mechanisms because these offer a much wider spectrum of ( time-dependent ) computations than feedforward architectures .', 'As our data suggest , precise timing of action potentials plays an important role in these processes , suggesting that , besides firing rates , cortical processing also exploits time and temporal relations as coding space .'], ["All the experiments were conducted according to the guidelines of the Society for Neuroscience and the German law for the protection of animals , approved by the local government's ethical committee , and overseen by a veterinarian .", 'In five cats , anesthesia was induced with ketamine and maintained with a mixture of 70% N2O and 30% O2 and with halothane ( 0 . 4%–0 . 6% ) .', 'The cats were paralyzed with pancuronium bromide applied intravenously ( Pancuronium , Organon , 0 . 15 mg kg−1 h−1 ) .', 'Multiunit activity ( MUA ) was recorded from area 17 with multiple silicon-based 16-channel probes ( organized in a 4×4 spatial matrix ) , which were supplied by the Center for Neural Communication Technology at the University of Michigan ( Michigan probes ) .', 'The intercontact distances were 200 µm ( 0 . 3–0 . 5 MΩ impedance at 1 , 000 Hz ) .', 'Signals were amplified 1 , 000× and , to extract unit activity , were filtered between 500 Hz and 3 . 5 kHz .', 'Digital sampling was made with 32 kHz frequency , and the waveforms of threshold-detected action potentials were stored for an off-line spike sorting procedure .', 'The probes were inserted approximately perpendicular to the surface of the cortex , allowing us to record simultaneously from neurons at different cortical layers and at different columns .', 'Up to three probes were inserted simultaneously , allowing for recordings from up to 48 electrode contacts ( channels ) in parallel , each providing a signal with multiunit activity .', 'This arrangement produced a cluster of overlapping receptive fields ( RFs ) , all RFs being covered by the stimuli ( Figure 1A in the main text ) .', 'Results for one cat were obtained from the same preparation as in [9] and for another cat from [4] .', 'These studies also describe further details on recording techniques .', 'Stimuli were presented binocularly on a CRT monitor ( 21″ , HITACHI CM813ET ) with 100-Hz refresh rate and by using the software for visual stimulation ActiveSTIM ( http://www . ActiveSTIM . com ) .', 'After mapping the borders of the respective RFs , binocular fusion of the two eyes was achieved by aligning the optical axes with an adjustable prism placed in front of one eye .', 'The stimuli consisted of single white letters that spanned approximately 5° of visual angle and that were composed of elementary features suitable for evoking strong responses in area 17 .', 'The stimuli were presented with maximal brightness on a black background ( thus , also with the maximal contrast ) and were placed such that they stimulated the cluster of RFs near optimally .', 'For presentation of single letters , we used letters A and D , each presented for 100 ms . Stimulus sequences of three letters consisted of letters A , B , C , D , and E and were either presented as sequences ABC , DBC , and ADC ( cat 1 ) or sequences ABE , CBE , ADE , and CDE ( cats 2 and 3 ) .', 'Each member of a sequence was presented for 100 ms , and the blank period separating the presentation of letters also lasted , in most cases , 100 ms . For each stimulation condition ( single letter or a sequence ) , 50 ( cat 1 ) , 100 ( cat 4 ) , or 150 ( cats 2 , 3 , and 5 ) repetitions were made .', 'In a control experiment , we varied the blank period between two stimuli presented in a sequence , the periods having the values 100 , 200 , 300 , and 500 ms . In this experiment , the first stimulus in the sequence was always a letter ( A or C ) , whereas as the second stimulus , we randomly exchanged a letter ( B ) with a white blank screen that had the same luminance as the letter stimuli .', 'This control experiment was made on a separate cat , which ensured that we fully avoided repetitive presentations of identical blank periods , which were made in other cats .', 'The block randomization of the presentation order ensured that no more than two blank periods of identical duration occurred in a row .', 'Sinusoidal gratings were presented with full contrast , spanned 18° of visual angle , had spatial frequency of 2 . 6 cycles/degree , and drifted with the speed of 2 . 0°/s , thereby being suitable for evoking strong responses in area 17 .', 'For an overview of experiments performed on each cat , see Table S1 .']]
['Introduction', 'Results', 'Discussion', 'Materials and Methods']
['It is currently not known how distributed neuronal responses in early visual areas carry stimulus-related information .', 'We made multielectrode recordings from cat primary visual cortex and applied methods from machine learning in order to analyze the temporal evolution of stimulus-related information in the spiking activity of large ensembles of around 100 neurons .', 'We used sequences of up to three different visual stimuli ( letters of the alphabet ) presented for 100 ms and with intervals of 100 ms or larger .', 'Most of the information about visual stimuli extractable by sophisticated methods of machine learning , i . e . , support vector machines with nonlinear kernel functions , was also extractable by simple linear classification such as can be achieved by individual neurons .', 'New stimuli did not erase information about previous stimuli .', 'The responses to the most recent stimulus contained about equal amounts of information about both this and the preceding stimulus .', 'This information was encoded both in the discharge rates ( response amplitudes ) of the ensemble of neurons and , when using short time constants for integration ( e . g . , 20 ms ) , in the precise timing of individual spikes ( ≤∼20 ms ) , and persisted for several 100 ms beyond the offset of stimuli .', 'The results indicate that the network from which we recorded is endowed with fading memory and is capable of performing online computations utilizing information about temporally sequential stimuli .', 'This result challenges models assuming frame-by-frame analyses of sequential inputs .']
['Researchers usually assume that neuronal responses carry primarily information about the stimulus that evoked these responses .', 'We show here that , when multiple images are shown in a fast sequence , the response to an image contains as much information about the preceding image as about the current one .', 'Importantly , this memory capacity extends only to the most recent stimulus in the sequence .', 'The effect can be explained only partly by adaptation of neuronal responses .', 'These discoveries were made with the help of novel methods for analyzing high-dimensional data obtained by recording the responses of many neurons ( e . g . , 100 ) in parallel .', 'The methods enabled us to study the information contents of neural activity as accessible to neurons in the cortex , i . e . , by collecting information only over short time intervals .', 'This one-back memory has properties similar to the iconic storage of visual information—which is a detailed image of the visual scene that stays for a short while ( <1 s ) when we close our eyes .', 'Thus , one-back memory may be the neural foundation of iconic memory .', 'Our results are consistent with recent detailed computer simulations of local cortical networks of neurons ( “generic cortical microcircuits” ) , which suggested that integration of information over time is a fundamental computational operation of these networks .']
[]
The brain has a one-back memory for visual stimuli. Neural responses to an image contain as much information about the current image as it does about another image presented immediately before.