(C) PLOS One
This story was originally published by PLOS One and is unaltered.
. . . . . . . . . .



Efficient coding theory of dynamic attentional modulation [1]

['Wiktor Młynarski', 'Institute Of Science', 'Technology Austria', 'Klosterneuburg', 'Gašper Tkačik']

Date: 2023-01

Activity of sensory neurons is driven not only by external stimuli but also by feedback signals from higher brain areas. Attention is one particularly important internal signal whose presumed role is to modulate sensory representations such that they only encode information currently relevant to the organism at minimal cost. This hypothesis has, however, not yet been expressed in a normative computational framework. Here, by building on normative principles of probabilistic inference and efficient coding, we developed a model of dynamic population coding in the visual cortex. By continuously adapting the sensory code to changing demands of the perceptual observer, an attention-like modulation emerges. This modulation can dramatically reduce the amount of neural activity without deteriorating the accuracy of task-specific inferences. Our results suggest that a range of seemingly disparate cortical phenomena such as intrinsic gain modulation, attention-related tuning modulation, and response variability could be manifestations of the same underlying principles, which combine efficient sensory coding with optimal probabilistic inference in dynamic environments.

Funding: GT & WM were supported by the Austrian Science Fund Standalone Grant P 34015 "Efficient Coding with Biophysical Realism" ( https://pf.fwf.ac.at/ ) WM was additionally supported by the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 754411 ( https://ec.europa.eu/research/mariecurieactions/ ). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Adaptive codes can be viewed as the next iteration of the efficient coding paradigm, where the neural code is optimized not only to the statistical structure of the incoming stimuli but also to the statistical structure of the perceptual task [ 47 ]. In this way, the bits encoded about the stimulus are the meaningful bits that are essential for a given perceptual task, while the task-irrelevant bits are discarded (making adaptive code a lossy compression scheme) to save resources. The adaptive coding model reproduces known properties of neural coding in the visual cortex and generates novel testable predictions about neural correlations and the impact of perceptual uncertainty on the population code. Our results provide a theoretical account of how top-down modulation could contribute to increased efficiency of sensory representations in the visual system.

Building on these general principles, and by committing to specific assumptions and simplifications, we develop a model of adaptive sensory representations in the visual cortex. The model is optimized to infer the state of a changing environment from dynamic sequences of natural images. To minimize the amount of neural activity used to encode individual stimuli, the model utilizes top-down feedback to dynamically modulate the gain of individual neurons in the sensory population. This modulation gives rise to an “adaptive code”—a sensory representation that is dynamically adapted in a top-down manner to support perceptual inference in a changing environment.

Here we address this issue by developing a model of dynamic, top-down modulation of sensory codes. A theoretical grounding of our model is provided by a synthesis of two established normative theories of neural computation: probabilistic inference and efficient coding. Probabilistic inference specifies how task-relevant environmental states can be optimally estimated from unreliable sensory signals. Efficient coding specifies how finite neural resources should be allocated to encode these signals. A fusion of these two theories provides a natural framework to study attentional modulation of sensory codes: a process whose presumed purpose is to allocate finite resources to extract features of the stimulus, which are necessary to accurately estimate relevant properties of the environment [ 46 ].

Computational theories of attention have interpreted attention-related modulation of sensory neurons as a consequence of probabilistic inference [ 41 – 44 ], slow fluctuations in the brain state [ 38 ], or modulation of gain in hierarchical feed-forward pathways [ 45 ]. Despite this progress, we currently do not understand how top-down modulation could enable a key putative feature of attentional computations—namely, the efficient use of limited resources by sensory populations to dynamically encode only the task-relevant sensory information.

Attention is a particularly relevant internal state known to modulate sensory codes [ 5 ]. Its presumed purpose is to allocate finite neural resources to accurately represent stimuli relevant for the task at hand [ 5 , 6 ]. To account for task specificity, attentional processes are traditionally categorized by the task-relevant properties of the stimulus or the environment into, e.g., object-based attention [ 22 – 24 ], spatial attention [ 25 – 27 ], or feature-based attention [ 28 – 30 ]. Attentional processes are known to modulate neural tuning curves [ 31 ], receptive fields [ 32 ], and individual neuron firing rates [ 33 , 34 ]. Attentional and other modulatory processes can also influence the collective structure of the population activity, reflected in correlation patterns between pairs of neurons [ 35 – 38 ]. Furthermore, fluctuations in the attentional state can contribute to dynamic variability of neural firing that unfolds over long timescales [ 1 , 38 – 40 ].

The question of how internal states of the brain could modulate sensory neurons and contribute to variability of neural activity has been addressed by a number of theoretical studies [ 9 , 11 ]. Neural variability in the primary visual cortex has been linked to probabilistic inference and uncertainty of low-level image features [ 12 – 14 ], as well as to hierarchical inference, where sensory representations interact across different levels of visual pathway to represent progressively more abstract features [ 15 – 19 ]. Structured variability in sensory populations could also result from mechanistic constraints on neural circuit dynamics [ 20 , 21 ].

Activity of sensory neurons is highly variable, even in response to the same stimulus [ 1 – 3 ]. Key factors contributing to this variability in the visual cortex are top-down feedback signals from high-level visual areas [ 4 – 6 ]. These signals modulate neural responses to external stimuli and are believed to reflect a broad range of internal states, such as goals of the organism and its beliefs about the state of the environment [ 7 – 10 ].

Results

We consider a scenario depicted in Fig 1A, where the aim of the sensory system is to keep track of a changing latent state of the environment. This latent state, denoted by and evolving in time t, might correspond to a behaviorally relevant quantity, such as the position of a moving target. The brain does not have direct access to this latent state and has to infer it from a stream of high-dimensional stimuli . Stimuli are encoded by a resource-constrained population of sensory neurons whose instantaneous responses are denoted by . A sensory representation of the current stimulus is conveyed via feed-forward connections to a brain region that performs a specific inference (a perceptual observer). To solve this inference optimally, the observer combines the stimulus representation with its internal model of the world into a posterior distribution over the current state of the environment . The posterior distribution is used to extract a point-estimate of the state of the environment , and the predicted future distribution of stimuli, which we denote as . Based on this prediction, optimal parameters for the sensory population are computed and conveyed back upstream, via feedback connections. These optimal parameters are selected by the perceptual observer to minimize a general cost function schematized in Fig 1B. The cost function navigates a trade-off between two competing objectives: minimization of the expected error in perceptual inference and minimization of the amount of neural activity, which the system requires to encode the incoming stimuli. Parameters of the sensory code are chosen to optimize these two terms, averaged over the stimulus distribution conditioned on the predicted value of the latent state.

PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 1. Adaptation of the sensory code for perceptual inference in a dynamic environment. (A) Continually evolving state of the environment gives rise to a sequence of stimuli , which are encoded by a population of sensory neurons into neural responses . The properties of sensory neurons (e.g., their gain, receptive fields, recurrent interactions) are not fixed but can be adapted moment by moment via feedback connections from higher brain areas (the model considered here specifically adapts gain of individual neurons). The normative approach we study here considers a scenario where sensory neurons optimally adapt their activation thresholds, leading to maximally accurate inference of the state of the environment by the perceptual observer, at minimal activity cost in the sensory population. Illustrative natural images were taken from [48]. (B) Cost function used by the system to adapt the parameters of the sensory code. At each time step, parameters are selected to minimize this cost function. (C) A single round of parameter updates consists of multiple steps performed by the sensory system to infer the latent state of the environment from adaptively encoded stimulus stream. Colors correspond to distinct terms of the equation displayed in (B). https://doi.org/10.1371/journal.pbio.3001889.g001

Computations described above can be represented as a sequence of steps performed by the model sensory system at each time instant (Fig 1C). By implementing this procedure, the sensory population can use its finite resources to retain only those features of the stimulus, which are relevant to the perceptual observer at any given moment [46], which reflects our intuitions about the role of attention in perception [5].

In the following sections, we develop a model of population coding in the primary visual cortex that implements the general design principles outlined above. We describe first a specific model of neural populations in V1 and endow it with dynamic adaptation whereby the continually evolving perceptual belief adjusts the code to minimize unnecessary neural activity. We then simulate three inference tasks representative of the different kinds of attention studied previously. In the main part of the results, we describe properties of adaptive coding for these tasks and compare them to experimental data.

Adaptive coding enables accurate inference with minimal neural activity How do adaptive codes navigate the trade-off between minimizing neural activity and maximizing task performance? We simulated perceptual inference in dynamic environments over multiple time steps for all three tasks (Fig 4A). Adaptive coding results in drastic decreases of neural activity in the sensory population compared to the standard sparse coding (Fig 4B). Adaptive coding furthermore reveals interesting task-specific dynamics of population activity, locked to the switches in the environmental state. For example, in the object detection and orientation estimation tasks (Fig 4B, top and bottom panels, respectively), the neural activity is significantly decreased in “absent” and “horizontal” environmental states, respectively. This is because the sensory system needs to extract different kind of information to support downstream inferences in different environmental states. In contrast, the standard sparse code maintains a roughly constant level of activity (Fig 4B, red lines). PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 4. Adaptive coding significantly reduces activity cost with minimal impact on inference accuracy. (A) Rows correspond to inference tasks: object detection (top), target localization (middle), and orientation estimation (bottom). (B) Sensory population activity 〈|z n,t |〉 n in the standard sparse code optimized for image reconstruction (red = full code) or for a particular task (blue = adaptive code). Activities in object detection (top) and orientation estimation (bottom) tasks were averaged over 500 switches between different states of the environment. For the target localization task (middle), we plot a short nonaveraged activity segment (200 time steps out of a 104 time step simulation; see Methods). (C) Same as B but for feedback activity required to adapt the nonlinearities in the sensory population (see Methods). (D) Time-averaged activity of the full code (red bars) and adaptive code (blue bars). Pie charts show the total activity decomposed into contributions from two different environmental states (green and orange; top and bottom row only) and feedback (brown; adaptive codes only). (E) Inference accuracy (red = full code; blue = adaptive code). Estimates of the environmental state (“object present” in object detection task, top; “orientation horizontal” in orientation estimation task, bottom) were averaged over 100 environmental switches. For the target localization task (middle), inference accuracy is measured as mean squared error between the true and inferred position of the target cross. Text insets display the average inference error in each task (see Methods). https://doi.org/10.1371/journal.pbio.3001889.g004 We also quantified the cost of top-down feedback signaling (Fig 4C). In our model, feedback activity is commensurate with the amplitude and frequency of posterior belief updates in the perceptual observer (see Methods), making feedback activity patterns strongly task specific. In the object detection task, feedback activity peaks briefly during switches between environmental states (Fig 4C, top panel). In the orientation estimation task, the belief of the perceptual observer fluctuates strongly when vertical orientation dominates, leading to elevated feedback activity (Fig 4C, bottom panel). Since the signal statistics are more homogeneous in the target localization task, feedback activity (when nonzero) stays within a tight interval (Fig 4C, middle panel). Despite the additional cost of feedback signaling, the total activity of adaptive codes is drastically lower compared to the full sparse code, sometimes by more than an order of magnitude (Fig 4D). This dramatic reduction does not significantly impact the accuracy of the inferences (Fig 4E). Average trajectories of the posterior probability for the object detection and orientation estimation tasks are very similar (Fig 4E, top and bottom panels). In the target localization task, the instantaneous error of the target location estimate using the adaptive code closely follows the error of the full code (Fig 4E, middle panel). For all tasks, the time-averaged error values are comparable between the adaptive and the full code. Taken together, this demonstrates that adaptive coding enables accurate inferences while dramatically minimizing the cost of neural activity in the sensory population.

Statistical signatures of adaptive coding Dynamic adaptation significantly changes the statistical structure of a sensory code. The most prominent change is a large increase in the sparsity of the adaptive code compared to the standard sparse code across all tasks (Fig 5A and 5B). This finding is consistent with the observed suppression of average neural activity (Fig 4D). These two phenomena are, however, not exactly equivalent. Sparsity of neural responses (as measured by kurtosis) can be increased in many ways [49], and each would result in suppression of the average activity. In our case, sparsity increase in the adaptive code is induced specifically by a complete suppression of a subpopulation of neurons, resulting in the high spike at zero in the neural response distribution (Fig 5A). PPT PowerPoint slide

PNG larger image

TIFF original image Download: Fig 5. Statistical differences between the adaptive code and the standard sparse code. (A) Rows correspond to inference tasks: object detection (top), target localization (middle), and orientation estimation (bottom). (B) Distributions of neural responses z t,n for the standard sparse code code optimized for image reconstruction (full, red) and the adaptive code (blue); kurtosis as a measure of sparsness is displayed in inset. (C) Pairwise correlations of 10 example neurons whose activity is modulated by the task (different for each task). Correlations were computed over the entire stimulus trajectory used to generate plots in Fig 4. Upper triangle (red) of correlation matrices corresponds to the full code, bottom triangle (blue) to the adaptive code. (D) Belief-induced response variability in the adaptive code. Neural activation (grayscale proportional to |z n,t |0.5) for 32 example neurons chosen separately for each task, exposed to 1,000 presentations of the same stimulus (orange frame). Response variability at fixed stimulus originates from the fluctuations in the internal belief of the perceptual observer (top part of each panel). Here, these fluctuations are simulated as sinusoidal variations in the probability of environmental state (object detection and orientation estimation tasks; top and bottom row, respectively), or a random walk trajectory of the target for the localization task (middle row). (E) Belief-induced noise correlations in the adaptive code. Left column: correlation matrices of the same 100 neurons computed from responses to stimulus presentations displayed in (D). Right column: scaled singular values of correlation matrices of the adaptive code (blue). We compared this spectrum to the standard sparse coding in which a small amount of independent Gaussian noise is added to each neural activation. The normalized singular spectrum of noise correlations of the sparse code (red) is denser compared to that of the adaptive code. https://doi.org/10.1371/journal.pbio.3001889.g005 Coordinated top-down modulation of individual neurons leaves its imprint also on the collective statistics of the population activity. For example, different perceptual tasks engage different neurons and, among them, induce different patterns of pairwise correlation. This effect becomes apparent when we focus on a subset of neurons active in a task and compare their correlated activity under standard sparse code or under the adaptive code. In the standard sparse code, neural correlations are inherited solely from the stimulus (Fig 5C, top submatrices, red frame). In an adaptive code, they are additionally modulated by the task, leading to a very different correlation pattern (Fig 5C, bottom submatrices, blue frame). Changes in the stimulus are not the only factor that drives response variability in the visual cortex. Cortical responses are notoriously unreliable and can fluctuate widely over multiple presentations of the same stimulus [3], giving rise to “noise correlations” among sensory neurons [55–57]. Patterns of noise correlations can be task specific and driven by feedback [37]. Our framework provides a new normative hypothesis about the origin and functional relevance of response variability and noise correlations. In our model, neurons generate different responses even at fixed stimulus when the neural nonlinearities change due to fluctuations in the internal state of the perceptual observer. For example, at the beginning of each target localization trial—even though the stimulus is the same—the perceptual observer may have a different prior belief about where the target is, possibly influenced by preceding history of the neural dynamics or sampling noise that leads to stochastic information accumulation about target position. Trial-to-trial differences in this internal belief will result in a variable allocation of resources in the sensory population as directed by the perceptual observer via top-down feedback, leading to strong noise correlations. We simulated such a scenario by exposing our model to multiple presentations of a single stimulus, identical across the three tasks, while enabling the perceptual belief to vary. A clear pattern of response variability to multiple presentations of the same stimulus is visible in each case (Fig 5D). This task-specific and feedback-driven response variability manifests in distinct noise correlation structures (Fig 5E, left column). For the adaptive code, the noise correlation matrix is dominated by a small number of modes, reflecting a low-dimensional fluctuating internal state of the perceptual observer. This observation is consistent with the experimentally observed low dimensionality of task-specific correlations in the visual cortex [37,58]. In contrast, noise correlations are expected to be exactly zero for the standard sparse code, within the setting considered here. If independent noise is purposefully introduced into the standard sparse coding units (see Methods), the singular value spectrum is much denser than for the adaptive code (Fig 5E, right column), indicating that the presence low-rank noise correlations differentiates between adaptive and full sparse codes, within the framework described here. In a general setting, noise correlations may be caused by a number of different factors beyond the normative computations described here. For example, they can arise as a consequence of recurrent circuit mechanisms used to compute sparse representations [15,50], or due to the biophysical structure of a neuronal network [21,59–61]. Taken together, adaptive code is predicted to feature: first, a sparser response distribution compared to the standard sparse code; second, task-dependent response correlations compared to task-independent correlations for the standard sparse code; third, prominent yet low-rank noise correlations compared to zero noise correlations for the standard sparse code.

[END]
---
[1] Url: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001889

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/

You are viewing proxied material from magical.fish. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.