(C) PLOS One
This story was originally published by PLOS One and is unaltered.
. . . . . . . . . .



Attractive serial dependence overcomes repulsive neuronal adaptation [1]

['Timothy C. Sheehan', 'Neurosciences Graduate Program', 'University Of California San Diego', 'La Jolla', 'California', 'United States Of America', 'John T. Serences', 'Department Of Psychology', 'Kavli Institute For Brain', 'Mind']

Date: 2022-09

Sensory responses and behavior are strongly shaped by stimulus history. For example, perceptual reports are sometimes biased toward previously viewed stimuli (serial dependence). While behavioral studies have pointed to both perceptual and postperceptual origins of this phenomenon, neural data that could elucidate where these biases emerge is limited. We recorded functional magnetic resonance imaging (fMRI) responses while human participants (male and female) performed a delayed orientation discrimination task. While behavioral reports were attracted to the previous stimulus, response patterns in visual cortex were repelled. We reconciled these opposing neural and behavioral biases using a model where both sensory encoding and readout are shaped by stimulus history. First, neural adaptation reduces redundancy at encoding and leads to the repulsive biases that we observed in visual cortex. Second, our modeling work suggest that serial dependence is induced by readout mechanisms that account for adaptation in visual cortex. According to this account, the visual system can simultaneously improve efficiency via adaptation while still optimizing behavior based on the temporal structure of natural stimuli.

(A) Task schematic. An orientated stimulus is followed by a probe bar that is rotated <15° from the stimulus. Participants judged whether the bar was CW or CCW relative to the stimulus in a binary discrimination task. (B) Response bias: % of responses that were CCW as a function of Δθ = θn − 1 − θn (± SEM across participants). (C) Behavioral bias, green: average model-estimated bias as a function of Δθ (± SEM across participants); gray: average DoG fit to raw participant responses sorted by Δθ (± 1SEM across participants). (D) Response accuracy as a function of Δθ. (E) Responses are significantly more accurate for |Δθ|<30°. (F) Behavioral σ as a function of Δθ. (G) Behavioral variance is significantly less for |Δθ|<30°. Note that in computing variance, we “flip” the sign of errors following CCW inducing trials to avoid conflating bias with variance (see Methods ). (H) Bias is positively correlated with variance across participants. ***, p < 0.001. Data and code supporting this figure found here: https://osf.io/e5xw8/?view_only=e7c1da85aa684cc8830aec8d74afdcb4 . CCW, counterclockwise; CW, clockwise.

To determine what role visual cortex plays in driving serial dependence, we applied multivariate fMRI decoding techniques to data collected while participants performed a delayed orientation discrimination task ( Fig 1A ). We replicated classic serial dependence findings where behavioral reports were attracted to the orientation of the previous stimulus. However, this attractive behavioral bias was not accompanied by attractive biases in visual cortex, as predicted by early sensory models of serial dependence. Rather, we observed repulsive biases in early visual cortex that were consistent with adaptation. We then examined several possible read-out mechanisms and found that only decoding schemes that account for adaptation can reconcile the neural and behavioral biases found in our data. More generally, these results explain a mechanism where the visual system can reduce energy usage without sacrificing precision by optimizing sensory coding and behavioral readout relative to the temporal structure of natural environments.

Counter to studies reporting a perceptual locus of serial dependence that utilized brief or low contrast stimuli, other behavioral studies utilizing high-contrast spatial stimuli have found that serial dependence does not emerge immediately but instead emerges only, and increases with, a working memory maintenance period [ 34 – 36 ]. This observation suggests that serial dependence could be implemented by a later readout or memory maintenance circuit [ 34 , 37 – 39 ]. There is evidence that such a readout mechanism is Bayesian, as the influence of the “prior” (the previous stimulus) is larger when sensory representations are less precise due to either external or internal noise [ 4 , 40 ]. Thus, the existing behavioral evidence suggests that serial dependence can operate both on perceptual and working memory representations [ 26 , 34 , 41 ]. It is open question how and where past trial information interacts with incoming sensory and memory representations.

In contrast to the repulsive perceptual biases typically associated with neural adaptation, perceptual reports are sometimes attracted to recently presented items—a phenomenon termed “serial dependence.” Studies utilizing low contrast oriented stimuli suggest that serial dependence can be perceptual in nature as it operates before a peripheral tilt illusion, impacts the perception of simultaneously presented items, biases perceptual reports even when no probe is presented, and does not require a working memory delay [ 25 – 29 ]. This perceptual account could arise from activity changes in early visual cortex, consistent with a functional magnetic resonance imaging (fMRI) study that measured early sensory biases that match “attractive” behavioral reports [ 30 ]. This neural finding, however, is challenging to interpret as consecutive trials were always the same or orthogonal orientations, which, by definition, cannot distinguish attractive from repulsive biases. Related studies decoding past stimuli from electroencephalography (EEG) activity do not measure how current stimulus representations are biased, precluding a connection to behavioral biases [ 31 – 33 ].

Adaptation increases coding efficiency by modulating sensory tuning properties as a function of the recent past. For example, reducing the gain of neurons tuned to a recently seen adapting stimulus reduces the temporal autocorrelation of activity when similar stimuli are presented sequentially, improving the overall efficiency of sensory codes [ 7 , 13 – 16 ]. Importantly, adapted representations early in the processing stream (e.g. the Lateral Geniculate Nucleus, LGN) are inherited by later visual areas, meaning the changes in coding properties could, in turn, shape decision-making [ 8 , 17 , 18 ]. Although adaptation increases coding efficiency, it comes at a cost to perceptual fidelity as adaptation can lead to repulsion away from the adapting stimulus for features such as orientation and motion direction [ 19 – 21 ]. For example, after continuously viewing and adapting to motion in one direction, stationary objects will appear to be moving in the opposite direction (i.e., current perceptual representations are repelled away from recent percepts). However, this potentially deleterious aftereffect is accompanied by better discriminability around the adapting stimulus, which may be more important than absolute fidelity from a fitness perspective [ 16 , 22 – 24 ].

Natural stimuli are known to have strong statistical dependencies across both space and time, such as a prevalence of vertical and horizontal (cardinal) orientations and a higher probability of small orientation changes in given spatial region over short time intervals [ 1 – 4 ]. These regularities can be leveraged to improve the efficiency and accuracy of visual information processing. For example, regularities can yield attenuated neural responses to frequently occurring stimuli in early visual cortex (adaptation), reducing metabolic cost and redundancy in neural codes [ 5 – 9 ]. At readout, regularities might support the formation of Bayesian priors that can be used to bias decision-making in favor of higher probability stimuli [ 10 – 12 ]. While the effects of stimulus history on sensory coding and behavior have been studied extensively, it is unclear how changes in sensory coding shape behavior.

Finally, we examined the variance of our decoders to see if this mapped onto our empirically observed variance. As model coefficients were fit independent of observed variance, correspondence between model performance and BOLD/behavioral data would provide convergent support for the best model. While the models were trained using noiseless activity at encoding, we simulated responses using Poisson rates to induce response variability. We simulated 1,000 trials from each cross-validated fit and pooled the model outputs. We first confirmed that the variance of the unaware decoder was highest following small changes of Δθ ( Fig 5A , gray; Fig 5G t(5) = 3.93, p = 0.005, paired 1-tailed t test <30° versus >30°) matching the output of our neural decoder ( Fig 2G ) and providing additional support for gain adaptation causing the observed repulsion in the fMRI data. Next, we compared the different decoders and found that, matching real behavioral responses, all 3 decoders were more precise following small values of Δθ ( Fig 5G , Bayes-unaware, t(5) = 2.25, p = 0.037; Bayes-aware t(5) = 1.90, p = 0.058; and overaware t(5) = 5.43, p = 0.001). While the pattern of the Bayes-unaware variance matched behavior, its overall variance was much higher than our behavioral data such that it diverged from the behavioral data significantly more than either of the aware models ( Fig 5E–5G ; ps < 0.005, paired t test comparing Jenson–Shannon divergence of error distributions). Together, the variance data provide additional evidence in favor of adaptation driving the repulsive biases that were observed in the BOLD data and awareness of the current state of adaptation being a requisite condition for the observed attractive serial dependence. More generally, this model has notable advantages that can lead to enhanced discrimination, reduced energy usage, and improved discrimination in naturalistic conditions over a static labeled line representation.

We next considered 3 readout schemes of this adapted population to maximize the likelihood of our behavioral responses ( Fig 5B ). The Bayes-aware decoder is consistent with previous Bayesian accounts of serial dependence [ 4 ], but additionally asserts that Bayesian inference occurs after encoding and that readout must account for adaptation. Alternatively, the Bayes-unaware decoder tests whether this awareness is necessary to achieve attractive serial dependence. Both aware models achieved biases that were significantly more likely than the unaware model (t(5) = 6.53, p = 0.001, Bayes-aware; t(5) = 6.6, p = 0.001, overaware, t test on log-likelihood, Fig 5C ) but were indistinguishable from each other (p = 0.36). Thus, both aware models were able to explain the response biases while the unaware model did a relatively poor job, suggesting that some awareness of the adapted state is necessary.

(A–C) Neural/behavioral bias. (D–G) Neural/behavioral variance. (A) Unaware decoder (yellow) provides a good fit to neural bias (black outline). Decoded variance decreases monotonically with distance from previous stimulus. (± SEM across participants). (B) Perceptual bias (black outline) was well fit by the Bayes-aware and overaware models but not the Bayes-Unaware model (± SEM across participants). (C) Participant responses were significantly more likely under aware models. (D) Behavioral variance had a similar shape and magnitude to Bayes-aware and overaware model fits. Bayes-unaware model output was much less precise and had a different form. (E) Distribution of empirically predicted response errors (black line) and simulated model fits for an example participant. (F) The unaware model’s error distribution had significantly higher Jenson–Shannon divergence from BOLD decoder than either aware model. (G) Visualization of all uncertainties split as a function of close and far stimuli. Note that the Bayes-unaware model had an average uncertainty that was on average 6x that of perception. *, p < 0.05; **, p < 0.01; ***, p < 0.001. Data and code supporting this figure found here: https://osf.io/e5xw8/?view_only=e7c1da85aa684cc8830aec8d74afdcb4 .

For each participant, we fit the encoder–decoder model in 2 steps ( Fig 4D ). All model fitting was performed using the same cross-validation groups as our BOLD decoder and each stage had 2 free parameters that were fit using grid-search and gradient descent techniques. We first report results from the encoding stage of the model. The gain applied at encoding was adjusted to minimize the residual sum of squared errors (RSS) between the output of the unaware decoder and the residual errors of our BOLD decoder. The unaware readout of the adapted encoding process ( Fig 5A , yellow) provided a good fit to the average decoding errors obtained with the BOLD decoder ( Fig 5A , black outline, ρ = 0.99) and across individual participants ( S8A Fig , ranges: ρ = [0.84, 0.98]). The unaware readout provided a better fit to the outputs of our neural decoder than the null alternative of the presented orientation (t(5) = 3.41, p = 0.01) because it captured a significant proportion of the variance in decoding errors as a function of Δθ (t(5) = 7.5, p = 0.0007). This analysis demonstrates that our adaptation model does a reasonable job of recovering our empirical decoding data (both of which use a decoder unaware of sensory history).

(A) Encoding. Units with von Mises tuning curves encodes incoming stimuli. The gain of individual units undergoes adaptation such that their activity is reduced as a function of their distance from the previous stimulus. (B) Decoding. This activity is then read out using a scheme that assumes 1 of 3 adaptation profiles. The unaware decoder assumes no adaptation has taken place, the aware decoder assumes the true amount of adaptation while the overaware decoder overestimates the amount of adaptation (note center tuning curves dip lower than the minimum gain line from encoding). (C) Example stimulus decoding. Top: The resulting likelihood function for the unaware readout (dotted yellow line) has its representation for the current trial (θ n = −30°) biased away from the previous stimulus (θ n-1 = 0°). The aware readout (dotted green line) is not biased, while the overaware readout is biased toward the previous stimulus. These likelihood functions can be multiplied by a prior of stimulus contiguity (solid black line) to get a Bayesian posterior (bottom) where Bayes-unaware and Bayes-aware representations are shifted toward the previous stimulus. Tick marks indicate maximum likelihood or decoded orientation. (D) Summary of models and free parameters being fit to both BOLD decoder errors and behavioral bias. Data and code supporting this figure found here: https://osf.io/e5xw8/?view_only=e7c1da85aa684cc8830aec8d74afdcb4 .

The encoding stage consists of cells with uniformly spaced von Mises tuning curves whose amplitude is adapted by the identity of the previous stimulus (θ n−1 , Fig 4A ). The decoding stage reads out this activity using 1 of 3 strategies ( Fig 4B ). The unaware decoder assumes no adaptation has taken place and results in stimulus likelihoods p(m|θ) that are repelled from the previous stimulus ( Fig 4C , yellow, where m is the population activity at the encoding stage). This adaptation-naive decoder is a previously hypothesized mechanism for behavioral adaptation [ 51 ] and likely captures the process that gives rise to the repulsive bias we observe in visual cortex using a fMRI decoder that is agnostic to stimulus history ( Fig 2G ). Alternatively, the aware decoder ( Fig 4C , green) has perfect knowledge of the current state of adaptation and can thus account for and “undo” biases introduced during encoding. Finally, the overaware decoder knows the identity of the previous stimulus but overestimates the amount of gain modulation that takes place, resulting in a net attraction to the previous stimulus ( Fig 4C , red). We additionally built off of previous work showing stimuli are generally stable across time by implementing a prior of temporal contiguity [ 4 ]. In our implementation, a Bayesian prior centered on the previous stimulus ( Fig 4C , black) is multiplied by the decoded likelihood to get a Bayesian posterior ( Fig 4C , bottom). We applied this prior of temporal contiguity to both the aware decoder as well as the unaware decoder to test the importance of awareness at decoding. We did not apply a prior to the overaware model to balance the number of free parameters between the various decoders and to see if the overaware model could achieve attractive serial dependence without a Bayesian prior ( Fig 4 and S1 Table ).

We observed an attractive bias and low variability around the current stimulus feature in behavior, and a repulsive bias and high variability around the current feature in the fMRI decoding data. Thus, the patterns of bias and variability observed in the behavioral data are opposite to the patterns of bias and variability observed in visual cortex. To better understand these opposing effects, we reasoned that representations in early visual cortex do not directly drive behavior but instead are read out by later cortical regions that determine the correct response given the task [ 47 – 50 ]. In this construction, the decoded orientations from visual cortex represent only the beginning of a complex information processing stream that, in our task, culminates with the participant making a speeded button press response. Thus, we devised a 2-stage encoder–decoder model to describe observations in both early visual cortex and in behavior (see Modeling).

We additionally examined the time-course of the bias. Significant repulsive biases were observable through the duration of the trial, in all early visual ROIs ( S4 Fig ). As the undershoot portion of the HRF extended to approximately 25 seconds, we examined the bias relative to the time of the presentation of the previous stimulus. We included only trials with an interstimulus interval (ISI) greater than the median of 17.5 seconds and plotted bias as a function of the minimum time from the previous stimulus ( Fig 3C ). Notably, bias was still significantly repulsive for 30 seconds following the previous stimulus presentation in all early visual ROIs, further shrinking the possibility that our biases are driven by the slow time course of the HRF ( Fig 3C , last time point). Finally, we examined how far back previous stimuli shape early visual representations. We examined the influence of not just the N-1 stimulus, but N-2 and N-3 stimuli as well, corresponding to median ISIs of 35.1 and 52.5 seconds, respectively ( Fig 3D and 3E ). As any influence of these more distant stimuli should be diminished relative to N-1, we maximized our sensitivity by taking the average decoded representation from 4 to 12 seconds. While the control N+1 stimulus showed no impact on decoded orientation as expected, we continued to see biases that are significantly repulsive through the N-3 stimulus in V1 and V2 ( Fig 3E ). These neural biases were surprisingly persistent and are in line with recent studies that have found adaptation signatures extending 22 seconds in mouse visual cortex spiking activity [ 9 ]. It is not clear why our effects persist even longer, but it is likely driven in part by the long ISIs, resulting in fewer intervening stimuli compared to the paradigm utilized in [ 9 ]. We separately extended our analysis of behavioral biases and found no significant effect of trials except for N-1, although biases were trending toward being repulsive for N-2 and N-3 reflecting the pattern reported in [ 46 ] ( Fig 3F ). Together, these analyses suggest that our observed biases are driven by adaptation in the underlying neural population and provide additional evidence that behavior is not directly linked to early visual representations.

To further understand whether the time course of our task could lead to artifacts, we also simulated responses to our task using tuned voxels that were modeled after the task sequence and estimated HRFs observed in our experiment (see supplementary modeling section, S7 Fig ). These simulations show that repulsive biases like the ones we observed with both our time course and deconvolution-based decoders are only possible when the underlying tuning of voxels is adapted by past stimuli/responses.

(A) Average V1 HRF through deconvolution for stimulus and probe. Average best fit double gamma function overlaid in dotted lines. (B) (Left) Bias curves from decoder trained on response patterns from deconvolved double-gamma functions (± SEM across participants). Here excluding hV4 and IPS0 for clarity. (Right) Bias quantified with a DoG function across ROIs. (C) Bias across time including only trials with an ISI of at least 17.5 seconds. x-Axis reflects minimum time from previous stimulus. Repulsion significant in all ROIs at 32 seconds. (D) Bias as a function of various relative orientations for V1 and V3 (± SEM across participants). (E) Bias across early visual ROIs for N-1, N-2, and N-3. Color scheme same as C. N+1 control analysis to ensure effects not driven by some unknown structure in stimulus sequence. (F) Behavioral bias for various relative orientations. N-1 data same as data presented in Fig 2 . *, p < 0.05, **, p < 0.01, ***, p < 0.001. Data and code supporting this figure found here: https://osf.io/e5xw8/?view_only=e7c1da85aa684cc8830aec8d74afdcb4 . DoG, Derivative of Gaussian; HRF, hemodynamic response function; ROI, region of interest.

We considered whether the repulsive adaptation we observed in visual cortex could be explained by residual undershoot of the hemodynamic response function (HRF) from the previous stimulus. To address this concern, we directly modeled the evoked response in each voxel to the stimulus and probe using a deconvolution approach and used a parameterization of the resulting filter (double gamma function) to model the stimulus evoked response on each trial (see Kernel-based decoding). Notably, the stimulus response has an undershoot that extends up to 25 seconds following stimulus presentation (see Fig 3A for an example voxel and parameterization). Estimating responses using this filter on individual trials and using the resulting weights to train a decoder removes the linear contribution of previous stimulus/probe presentations [ 44 , 45 ]. Any bias in the resulting decoder should thus be due to changes in BOLD activity driven by neuronal activity rather than a hemodynamic artifact. We repeated all analyses after correcting for the shape of the HRF, and while the resulting decoder was less precise than one trained on the time course data (eg. V3 r circ = 0.19 ± 0.07 versus 0.32 ± 0.08 with time course decoder), it was still significantly predictive across all visual ROIs (ps < 0.05) except IPS0. Despite the noisier decoding, we still observed a significant repulsive bias in all visual ROIs that matched the pattern found when decoding the raw BOLD time course ( Fig 3B ).

We also examined how the precision of neural representations changed as a function of stimulus history. In sharp contrast to behavior, σ circ exhibited a monotonic trend such that neural decoding was least precise when the previous stimulus was similar ( Fig 2G , gray curve; see Neural variance). We quantified this difference in sensory uncertainty in a similar manner to the behavioral data and found that variance in the sensory representations was significantly greater following a similar stimulus (<30°, t(5) = 72.4, p = 4.8*10 −9 , paired 1-tailed t test, V3, Fig 2H ). This pattern was significant (p < 0.05) in all ROIs except IPS0 ( S6A Fig ). The results did not change qualitatively when we utilized vector length as a proxy for decoding precision derived directly from our channel estimates ( S6C and S6D Fig ) or when we used other thresholds between 20° and 40°. The repulsion of sensory representations and the corresponding reduction in decoding precision around the previous orientation is consistent with neural adaptation where recently active units are attenuated, thus leading to lower SNR responses in visual cortex.

The high SNR (signal to noise ratio) of the BOLD decoder additionally allowed us to examine residual errors on individual trials. When measuring the bias (circular mean, μ circ ; see Neural bias) of these decoding errors as a function of stimulus history (Δθ), we observed a strong repulsive bias reflecting neural adaptation (V3, Fig 2G , yellow). This bias was significant when quantified with a DoG (amplitude = −14.5° ± 2.9°, t(5) = −3.56, p = 0.0029; FWHM = 52.2° ± 2.94°, Fig 2G , black dotted line), and all ROIs had a significantly negative amplitude (p < 0.01, Fig 2I ). Critically, this bias was present across all TRs for both the task and localizer decoders and was visible in the bias curve computed for each individual participant ( S4 Fig ). In addition to the model-based analysis of responses in visual cortex, we also performed a model-free assessment of the dimensionality of activation patterns conditioned on the prior stimulus. Consistent with our main analysis, responses following close stimuli have a higher dimensionality than responses following far stimuli. This suggests that changes due to neural adaptation should assist pattern separation regardless of stimulus identity (see Dimensionality analysis; S5 Fig ).

We are interested in the how the identity of the previous stimulus influences representations of the current stimulus, akin to previous EEG studies that have demonstrated the ability to decode the previous stimulus during the current trial [ 32 ]. We performed a similar analysis by training and testing our task decoder on the identity of the previous stimulus using the same time points as the current trial decoder. This decoder was able to achieve above chance decoding in all ROIs examined indicating trial history information is present in the activity patterns ( Fig 2F ). As a control analysis, we attempted but were unable to decode the identity of the next stimulus using the same procedure ( S3F Fig ). The performance of the memory decoder for the previous stimulus peaked around 6 seconds after stimulus presentation but remained above chance throughout the delay period ( S4A Fig ). Notably, we were generally unable to decode the identity of the previous stimulus using our decoder trained on a localizer task suggesting representations of past trial stimuli are not in a “sensory code” ( S4B Fig ).

To examine how visual representations are affected by stimulus history, we trained a decoder on the orientation of the sample stimulus on each trial based on BOLD activation patterns in each ROI. We used the vector mean of the output of an inverted encoding model (IEM) as a single trial measure of orientation using a leave-one-run-out cross-validation across sets of 68 consecutive trials (4 blocks of 17 trials) that had orientations pseudo randomly distributed across all 180° of orientation space (see Orientation decoding for details). We first quantified single-trial decoding performance using circular correlation (r circ ) between the decoder-estimated orientations and the actual presented orientations and found that all ROIs had significant orientation information ( Fig 2C ). Our ability to decode extended for the duration of the trial, peaking around 12 seconds after stimulus presentation ( Fig 2D ). This memory signal seems to be largely in a “sensory code” as a decoder trained on a separate localizer task where participants viewed stimuli without holding them in memory achieved similar performance over a similar timescale (see fMRI localizer task; Fig 2E ). Thus, visual ROIs showed robust orientation information that could be decoded across the duration of the trial. For all analyses not shown across time, we used the average of 4 TRs (repetition time, spanning 4.8 to 8.0 seconds) following stimulus presentation to minimize the influence of the probe stimulus (which came up ≥6 seconds into the trial and thus should have a negligible influence on activity in the 4.8 to 8.0 seconds window after accounting for hemodynamic delay; see Fig 5A ).

(A) Left axis, behavioral serial dependence. Shaded green: average model-estimated bias as a function of Δθ (± SEM across participants); dotted black line: average DoG fit to raw participant responses sorted by Δθ. Right axis, variance. Purple shaded line: model-estimated variance as a function of Δθ (± SEM across participants). (B) Behavioral σ is significantly less for |Δθ|<30°. (C) Decoded orientation was significantly greater than chance when indexed with circular correlation for all ROIs examined. Error bars indicate ±SEM across participants. Dots show data from individual participants. (D) Decoding performance across time for a subset of ROIs. Vertical red line indicates time point used in most analysis. (E) Decoding performance across time for a decoder trained on a separate sensory localization task. (F) Performance of task decoder trained and tested on identity of previous stimulus across all ROIs. (G) Left axis, decoding bias. Shaded yellow line: decoded bias (μ circ of decoding errors) sorted by Δθ (± SEM across participants); dotted black line: average DoG fit to raw decoding errors sorted by Δθ. Right axis, decoded σ circ . Shaded gray line: average decoding variance (σ circ ) as a function of Δθ (± SEM across participants). Note that σ circ can range from [0, inf] and has no units. (H) Decoded variance is significantly greater for |Δθ|<30°. (I) Decoded errors are significantly repulsive when parameterized with a DoG in all ROIs. *, p < 0.05; **, p < 0.01; ***, p < 0.001. Data and code supporting this figure found here: https://osf.io/e5xw8/?view_only=e7c1da85aa684cc8830aec8d74afdcb4 . DoG, Derivative of Gaussian; ROI, region of interest.

To examine the influence of stimulus history on orientation-selective response patterns in early visual cortex, 6 participants completed between 748 and 884 trials (mean 838.7) of the task in the fMRI scanner over the course of four 2-hour sessions (average accuracy of 67.7% ± 0.4% with an average probe offset, δθ, of 3.65°). As with the behavior-only cohort, behavioral reports in these participants showed strong attractive serial dependence ( Fig 2A , green) that was significantly greater than 0 when parameterized with a DoG function (amplitude = 3.50° ± 0.27°, t(5) = 11.93, p = 0.00004; FWHM = 35.9° ± 2.34°, Fig 2A , black dotted line). This bias was not significantly modulated by intertrial interval, delay period, or an interaction between the 2 factors (all p-values > 0.5, mixed linear model grouping by participant). Similar to the behavioral cohort, we found that variance was generally lower around small values of Δθ. We quantified variance in the same manner as the behavioral cohort (flipping responses to match biases and down-sampling the larger group) and found that responses were more precise following close (<30°) relative to far stimuli (>30°, t(5) = −9.96, p = 0.00009, 1-tailed paired t test, Fig 2B ). This pattern was significant (p < 0.05) for thresholds between 20° and 40°. A subset of these participants completed some sessions where consecutive stimuli were not strictly independent as they were more likely to be between ±22.5 and 67.5° from the previous stimulus (see Methods , Behavioral discrimination task, 4 out of 6 participants had between 357 and 408 trials that were nonindependent accounting for between 40% and 50% of their trials and 32% of all trials completed). However, we replicated all of our main analysis excluding these sessions and found that our conclusion remained unchanged with the exception that our finding of reduced variance trended in the same direction but no longer reached significance ( S3 Fig ).

A subset of participants completed a version of the experiment with inhomogeneities in their stimulus sequences (such that consecutive orientations were more likely to be between ±22.5 and 67.5° from the previous stimulus). We repeated all of the above analyses excluding these participants and found all of our findings were qualitatively unchanged ( S2 Fig ).

Previous work has shown that serial dependence is greater when stimulus contrast is lower [ 28 ] and when internal representations of orientation are weaker due to stimulus independent fluctuations in encoding fidelity [ 4 ]. We tested a Bayesian interpretation of these findings by asking whether less precise individuals are more reliant on prior expectations and therefore more biased. Consistent with this account, we found a positive correlation between DoG amplitude and σ ( Fig 1H , r(45) = 0.52, p = 0.0001, 1-tailed Pearson correlation). This relationship was not dependent on our response parameterization as we report found similar relationships between DoG amplitude and both accuracy (r = −0.41, Pearson correlation, p < 0.005) and average task difficulty δθ (r = 0.44, p < 0.005).

We next examined how response precision (σ) varied as a function of Δθ and found that responses were more precise around small trial-to-trial orientation changes ( Fig 1F ), again consistent with previous reports [ 43 ]. We quantified this difference in precision by splitting trials into “close” and “far” bins (greater than or less than 30° separation) and confirmed that responses following “close” stimuli were more precise (t(46) = −3.72, p = 0.0003, paired 1-tailed t test, Fig 1G ; see Response precision). Note that the choice of 30° was arbitrary, but all threshold values between 20° and 40° yielded significant (p < 0.05) results. As with bias, this variance result was not an artifact of our parameterization as raw accuracy showed a similar pattern such that responses were more accurate following close stimuli (t(46) = 3.66, p = 0.0003; Fig 1D and 1E ). We additionally confirmed that our finding of reduced bias around small changes in orientation is not driven by a higher proportion of “cardinal” orientations (here defined as being ±22.5° of 0 or 90°) as the proportion of cardinal orientations did not differ between close and far bins of Δθ (mean % cardinal close: 50.6 ± 0.5%, far: 50.2 ± 0.3%, t(46) = 0.9, p = 0.39, paired t test).

Responses were robustly biased toward the previous stimulus ( Fig 1C , green curve), which we quantified by fitting a Derivative of Gaussian (DoG) function to the raw response data for each participant (gray curve; amplitude: 4.53° ± 0.42°, t(46) = 7.8, p = 5.9*10 −10 , 1-sample t test; full width at half maximum (FWHM): 42.9° ± 1.8°; see Serial dependence). The magnitude and shape of serial dependence are consistent with previous reports [ 25 , 42 ]. This bias is not an artifact of our parameterization as the same pattern is observable in the raw proportion of CCW responses ( Fig 1B ). Note that as participants are reporting the orientation of the probe relative to the grating stimulus, a greater proportion of reports that the probe was CCW corresponds to a CW shift in the perception of the grating.

To quantify the pattern of behavioral responses, we modeled the data as the product of a noisy encoding process described by a Gaussian distribution centered on the presented orientation with standard deviation σ and bias μ. Optimal values for σ and μ were found by maximizing the likelihood of responses for probes of varying rotational offsets from the remembered stimulus, thus converting pooled binary responses into variance and bias measured in degrees (see Response bias; S1 Fig ). This allowed us to measure precision for individual participants and also allowed us to measure how responses were biased as a function of the orientation difference between the remembered gratings on consecutive trials Δθ = θ n-1 − θ n , an assay of serial dependence.

To probe the behavioral effects of serial dependence, we designed a delayed discrimination task where participants judged whether a bar was tilted clockwise (CW) or counterclockwise (CCW) relative to the orientation of a remembered grating ( Fig 1A ). We first report the results from a behavior-only study (n = 47) followed by an analysis of neural activity for a cohort completing the same task in the fMRI scanner (n = 6). Task difficulty was adjusted for each participant by changing the magnitude of the probe offset (δθ) from the remembered grating and was titrated to achieve a mean accuracy of approximately 70% (accuracy 69.8 ± 0.82%, δθ: 4.61 ± 0.27°; all reported values mean ± 1 SEM unless otherwise noted). Fixing participants at this intermediate accuracy level helped to avoid floor/ceiling effects and improved our sensitivity to detect perceptual biases while keeping participants motivated.

Discussion

In this study, we sought to understand the neural underpinning of attractive serial dependence and how changes in tuning properties at encoding shape behavior. Based on previous behavioral and neural studies, we expected to observe attractive biases in line with observed behavior and decoding from early visual areas [30]. Instead, we found that representations were significantly repelled from the previous stimulus starting in primary visual cortex and continuing through IPS0 (Fig 2I). This repulsion is consistent with bottom up adaptation beginning either at or before V1 and cascading up the visual hierarchy [8,9,18]. As repulsive biases are in the opposite direction as behavioral biases, we built a model to link these conflicting patterns. The critical new insight revealed by the model is that only readout schemes that account for adaptation can explain the attractive behavioral bias observed in our paradigm. More generally, our BOLD data argue against an early sensory origin of serial dependence for orientation and instead suggest that serial dependence is driven by postperceptual or mnemonic circuits [38,39]. However, because we used a paradigm that required working memory, our results may not generalize to other situations in which serial dependence is observed even in the absence of a memory delay [25,26,29,52]. Thus, future work is needed to better understand the role of sensory representations in paradigms with low contrast stimuli, that do not require a memory delay period, and that utilize other features besides orientation.

There have been many prior studies arguing for either a perceptual or postperceptual origin of serial dependence. Some behavioral studies have found that serial dependence emerges almost immediately after the offset of a stimulus, pointing to an early perceptual origin of the effect [25–27,40]. One study additionally demonstrated that attraction to the previous stimulus seems to occur before the “tilt illusion” driven by concurrently presented flanking stimuli [27]. If history biases indeed operate before spatial context, this could point to a distinct assimilative mechanism for serial dependence in early visual processing that may only emerge under low stimulus drive. As our experiment always utilizes a working memory delay, it is unclear if the bias toward past stimuli is driven by a change in their perception of the stimulus itself or instead somehow biases their comparison with the probe stimulus only after the working memory maintenance period.

Others have found that serial dependence is repulsive at very short delays and only becomes attractive when items are held for an extended time in working memory [34,35]. This apparent discrepancy was reconciled by [28], who showed that attractive biases disappear without a working memory delay, unless the stimuli are rendered at a very low contrast. This observation suggests that serial dependence may emerge immediately when high sensory uncertainty is induced by low contrast stimuli, and it may emerge later if high sensory uncertainty is induced by extended working memory delay periods. It is curious that unlike some spatial working memory studies [34–36], we did not find that behavioral biases increased with delay time. One possible explanation is that this phenomenon is actually unique to spatial working memory due to either a more consistent increase in sensory uncertainty of spatial location due to eye movements or a separate mechanism of memory maintenance that becomes more susceptible to proactive interference relative to orientation memories. Separately, as our stimuli were presented at the fovea (unlike spatial paradigms) they are encoded by a larger population and thus may be less susceptible to degradation across time.

Evidence for an early sensory origin of serial dependence comes from an fMRI study with low contrast stimuli and a short (500 ms) delay period which reported that both behavioral responses and V1 representations were more precise following a matching stimulus [30]. This departure from our own finding could be driven by the stimuli that were rendered to have a very high uncertainty. Past work studying adaptation in nonhuman primates found repulsive patterns following long (4 seconds and 40 seconds) but attractive patterns following short (0.4 seconds) stimulus presentations, suggesting that stimulus duration may have a large influence on how past stimuli shape future sensory processing [53]. That said, the stimuli used in the fMRI study of [30] were always 1 of 2 orthogonal orientations, which, given a circular feature space like orientation, precludes an assessment of attraction or repulsion. Furthermore, correct motor responses were directly yoked to the stimulus so any behavioral tendency to report seeing the same stimulus on successive trials could be due to motor priming rather than stimulus based serial dependence (e.g., a “stay” bias). Related work has shown the ability to decode the previous stimulus from EEG activity patterns [31–33], but it is important to note that our study also showed robust decoding of the previous stimulus that did not also correspond with an attractive bias in the neural representation of the current stimulus (Figs 2F and 2G and S4). This is because the representations of current and past stimuli are not necessarily stored using the same code. Thus, while previous neural studies have argued that serial dependence emerges in visual cortex, no study has demonstrated an attraction toward the previous stimulus dependent on feature similarity consistent with behavioral biases. Further work examining neural biases using low contrast stimuli will shed further light on a potential role of coding changes in sensory cortex driving serial dependence.

In contrast to studies favoring an early sensory account—and more in line with the paradigm and findings reported in this manuscript—a single unit recording study in nonhuman primates used high-contrast stimuli and a longer working memory delay (1.4 to 5.6 seconds) [54]. Under these conditions, neural responses in the frontal eye fields (FEFs) were repelled from the previously remembered location even though saccades were attracted to the previously remembered location. Given the tight link between the FEF and attentional control [55–57], the authors speculated that the observed neural repulsion was due to residual attentional shifts carrying over from the previous trial. However, our observation of repulsive biases starting in V1 and persisting across later visual areas suggests that bottom-up adaptation may be a viable alternative explanation (which the authors also acknowledged). Further support for this account comes from a recent magnetoencephalography (MEG) study showing that representations were repelled from past stimuli both within the current trial and from the previous trial [58]. As in our study, this neural repulsion contrasts with attractive behavioral biases to the previous stimulus, suggesting that sensory representations do not directly shape behavior even in simple sensory paradigms [50]. Behavioral studies using similar high-contrast orientation stimuli to our own have also shown that responses are attracted to past decisions and repelled from past stimuli, further suggesting that these attractive biases do not emerge in early sensory areas [38,59,60]. Several modeling studies additionally suggest that serial biases are mediated by later readout circuits due to synaptic changes arising from persistent bump attractor dynamics as opposed to early sensory processing [37,39]. Thus, in line with our findings: behavioral, neuronal, and modeling studies utilizing high-contrast stimuli in working memory paradigms consistently point to attractive effects emerging in either memory or decision-making circuits and not early sensory areas.

In line with classic accounts, adaptation in visual cortex should lead to a reduction in energy usage during encoding [14]. However, the main advantage of adaptation may be to decorrelate inputs, thus enhancing the discriminability of incoming stimuli [14,61] and even acting as a form of short-term memory [62]. An optimal processing stream may emphasize differences at encoding and only favor stability once a stimulus has been selected by attention for more extensive postperceptual processing [38]. This motif of pattern separation followed by pattern completion would not be unique to adaptive visual processing. Similar mechanisms have been proposed as a critical component of long-term memory processing in the hippocampus and associative memory formation in the fly mushroom body [63]. Thus, the biases introduced by adaptation may be beneficial in part because they expand the dimensionality of the representational space as we found in our recordings (S5 Fig).

We did not explicitly define how awareness of adaptation is implemented, but it is clear that both attention to and conscious awareness of the previous stimulus are necessary for serial dependence to occur [25,64]. This is consistent with our model, and it suggests that some representation of information about stimulus history should be a minimum requirement for an aware decoding scheme. The identity of the previous stimulus for spatial position and angle has been shown to be decodable from the spiking activity of single units in the FEF and posterior parietal cortex (PPC) as well as large-scale activity patterns in human EEG and MEG [31–33,54,58,65]. We additionally demonstrate that information about the previous trial is encoded in patterns of fMRI activity in human visual cortex (Fig 2F), but not in a sensory-like code (S4A and S4B Fig). These signals could potentially be represented concurrently with representations of the current stimulus in the same populations of sensory neurons but in orthogonal codes analogous to what has been found for sequentially encoded items in primate prefrontal cortex and human EEG [66,67]. An alternate account holds that representations of stimulus history are maintained outside of early visual areas, consistent with findings from mouse parietal and primate prefrontal cortex [39,65]. This anatomical segregation could disambiguate incoming sensory drive from representations of stimulus history. Critically, optogenetically suppressing nonsensory representations of stimulus history eliminated history effects, thus providing strong support for some form of an aware readout mechanism [65].

For the decoding stage of our model, we established that only readout schemes that are aware of adaptation could explain attractive serial dependence. The Bayes-aware model is an extension of previously proposed models that employ an explicit prior but that did not consider effects of adaptation at encoding [4]. In contrast, the overaware model is a novel account that can achieve similar performance without needing an explicit prior based on stimulus history. While model fit metrics did not readily distinguish one of these 2 models as superior, the overaware model may prove to be more flexible. For example, one of our fMRI participants showed significant repulsion from far stimuli, an observation also reported by others [35,42]. While the overaware model can fit this repulsive regime, the Bayes-aware model is incapable of generating repulsive patterns (compare models fits for participant #3; S8 Fig). This limitation of a purely Bayesian account of serial dependence is also observable in prior work (Fig 6B in [4]).

The overaware model proposed in our study may instead be a special condition of a decoder with “fixed awareness” that is based on temporal transition probabilities in natural scenes that are steeply peaked around 0 (no change) over short timescales [1,2,4]. Such a readout would correct for the most encountered levels of adaptation by accounting for the transition probabilities of stimuli while being “fixed,” or inflexible, when stimuli violate these expectations. This decoder could account for additional phenomena not directly assessed in the present study such as the tilt aftereffect (TAE). The TAE and other forms of (repulsive) behavioral adaptation are often ascribed to an unaware decoder [7,51] but might instead reflect levels of adaptation that exceed the fixed level of adaptation expected by a “fixed-aware” decoder due to long presentations or high-contrast stimuli. This is supported by an apparent disconnect in the magnitude of repulsive biases between behavior and neural representations [5,19]. In contrast, the fixed awareness decoder would lead to attractive biases (serial dependence) when stimuli create less bottom-up drive than expected (e.g., through brief presentations or low contrast items). This “fixed-aware” decoder is consistent with previous findings of attractive biases disappearing or switching to repulsion when stimulus contrast or duration is increased [25,28]. This scheme could extend to spatial adaptation such as the tilt illusion where the joint probability of center and surround orientations being perfectly distinct would be vanishingly rare in natural scenes [68–70].

In this study, we extended previous descriptions of serial dependence by quantifying how both bias and variance are shaped by stimulus history. We report a robust pattern of perception being most precise following small changes in successive stimulus features (Figs 1F, 1G, 2A and 2B). This relationship violates a proposed perceptual “law” that bias is inversely proportional to the derivative of discrimination thresholds [71]. This account would assert that our attractive bias should come with a less precise representation following small changes (or a repulsive bias to account for our enhanced precision). We argue that serial dependence is not violating this law, but rather believe this is further evidence for delay dependent serial dependence being a postsensory phenomenon. Neural representations exhibit repulsive biases, expanding the perceptual space and allowing greater discriminability (S5 Fig). When these representations are read out by an aware decoder, the bias is undone but the enhanced discriminability remains (Fig 5D and 5G).

[END]
---
[1] Url: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001711

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/

You are viewing proxied material from magical.fish. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.