A global framework for a systemic view of brain modeling

doi:10.21203/rs.3.rs-59619/v1

Download PDF

Research

A global framework for a systemic view of brain modeling

https://doi.org/10.21203/rs.3.rs-59619/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 15 Feb, 2021

Read the published version in Brain Informatics →

Version 1

posted

You are reading this latest preprint version

Background: The brain is a complex system, due to the heterogeneity of its structure, the diversity of the functions in which it participates and to its reciprocal relationships with the body and the environment. Researchers in cognitive computational neuroscience generally focused on one specific task, lack this global view and a clear understanding of how their model is inserted in a global cognitive architecture. A systemic description of the brain is presented here, as a general framework where specific models in computational neuroscience can be integrated and associated with global information flows and cognitive functions.

Results: In an enactive view, this framework integrates the fundamental organization of the brain in sensorimotor loops with the internal and the external worlds, answering four fundamental questions (what, why, where and how). Our survival-oriented definition of behavior gives a prominent role to pavlovian and instrumental conditioning, augmented during phylogeny by the specific contribution of other kinds of learning, related to semantic memory in the posterior cortex, episodic memory in the hippocampus and working memory in the frontal cortex.

Conclusions: This framework highlights that responses can be prepared in different ways, from pavlovian reflexes and habitual behavior to deliberations for goal-directed planning and reasoning, and explains that these different kinds of responses coexist, collaborate and compete for the control of behavior. It also lays emphasis on the fact that cognition can be described as a dynamical system of interacting memories, some acting to provide information to others, to replace them when they are not efficient enough, or to help for their improvement. Describing the brain as an architecture of learning systems has also strong implications in Machine Learning. Our biologically informed view of pavlovian and instrumental conditioning can be very precious to revisit classical Reinforcement Learning and provide a basis to ensure really autonomous learning.

Cognitive Neuroscience

Brain modeling

cognitive functions

memory system

As stated by the French biologist and physician Henri Laborit (Laborit, 1976), the motivation of living beings is ... being, i.e. maintaining their organic structure. Whereas this statement is obvious for lower animals, it is too often neglected when studying high level cognitive functions, particularly in humans. Most of the time in computational neuroscience, such cognitive functions are associated to specific regions of the cortex and not to the bodily dimension or even to subcortical structures. Their characteristics are described as resulting from purely cortical dynamics, with no references to motivational or emotional groundings. In neuroscience, some authors and paradigmatic approaches have stressed the limitations of such corticocentric views, compared with a myopia (Parvizi, 2009), minimizing the essential role of subcortical structures. On the modeling side, the domain of embodied Artificial Intelligence has shown through robotic experiments (Pfeifer et al., 2007) that complex behaviors may result from elementary loops between sensors and actuators, exploiting the properties of the body instead of a complex representation of information. It is for example proposed in (Taouali et al., 2015) that the non homogeneous distribution of sensors in the retina can explain some visual target selection principles in a more parsimonious way than purely cortical mechanisms.

More fundamentally and anchored in cognitive science, enactivism, the theory of enaction (Varela et al., 1991), stresses principles like autonomy and ecological meaning of the behavior. In this theory, autonomous behavior is a central characteristic and is considered at different time scales. Fundamentally, a living being must choose on its own and at each moment the most adapted behavior and can only rely on previous learning (ontogeny) and on pre-established abilities (phylogeny, seen as learning at a long time scale). Ecological meaning refers to the motivational and emotional bases of behavior that have to be taken into account. Cats chase mice because they have such motivations, needs and goals — because they are cats.

In spite of their important role to define needs of the body and goals to be reached, the motivational and emotional dimensions of behavior are little studied in computational neuroscience and in cognitive science. Building on interoceptive information like visceral signals but also on somatosensory information (pain, pleasure, temperature), the insular cortex is hardly considered in cognitive modeling studies though reported to play a central role in defining motivations of the body to act, like feeding, breeding, preserving the integrity of the body (Craig, 2009). Biologically significant events important for survival signaled by such interoceptive signals but also by sensory information (e.g. related to the perception of a predator or of social signals) can be associated by learning with other neutral events that will elicit emotions useful to anticipate the former ones and to detect goals to be pursued or avoided. (Gros, 2010) suggests a specific role of information of reduced complexity for emotions that can become conscious feelings, also described as mental experiences of body states (Damasio and Carvalho, 2013).

Such a body of principles should make humans more conscious of their animal condition. It underlines the strong links between the brain, the body and the environment and, within the brain, is a strong motivation to consider large brain loops instead of cortical regions in isolation and to consider the multiple learning mechanisms at work within these loops. At the functional level, this is also a plea for defining a global cognitive architecture in which any cognitive operation in consideration should be delineated. Decision making, planning, selective attention or perceptual identification shouldn’t be studied, and models of the corresponding cerebral circuitry shouldn’t be elaborated, without a reference to a global framework relating cognition and the brain, seen as a whole and in relation to the body and the environment. Else, the risk is to just study mechanisms apart from the rationale for their existence and consequently to forget some of their fundamental characteristics and resources.

In agreement with these considerations, we present here in a systemic view, a general framework of brain modeling that has been elaborated from the analysis of the literature in cognitive, experimental and computational neuroscience. It is intended, for future specific studies of brain-inspired cognitive mechanisms, to serve as an outline in which each of these studies should be placed, for a better understanding of its contribution in general cognition and for consistency in this systemic view of cognition that we affirm here to be essential.

Based on strong neuroscientific and cognitive bases, this framework might be useful to help scientists in these domains have a more general view of the “big picture”. It is also and more specifically destined for modelers in computational cognitive neuroscience, with not necessarily a strong background in neuroscience, that would like to have a global and functional view of the cognitive architecture and its corresponding cerebral circuitry, within which they might display the topic on which they are presently working on. Generally, such scientists begin with an global view of a task, where the problem is to control an intelligent agent in its environment. As sketched in Fig. 1, this problem can be specified with several data flows organized under different poles: the perceptual and motor characteristics of the agent are respectively described in input and output data flows called the exteroceptive and motor poles, whereas task instructions are given to the agent in a limbic pole gathering specified goals and constraints and can be monitored within an interoceptive pole, through internal sensations corresponding to rewards, punishments and levels of energy. The aim of this paper is to propose a structured and comprehensive framework to help implement such a task or a part of it with a clear understanding of how it is inserted in a global cognitive architecture.

In the following, we present the main ingredients of this framework (Sect. 2), before specifying increasingly complex organizational principles under a phylogenetic perspective (Sect. 3) and corresponding functional and learning mechanisms to have it operational.

2.1 Three worlds to conciliate

The brain is facing complex and dynamic worlds, on each of which it can sense information and it can act, possibly resulting on modifications in that and other worlds. Of course, one of these worlds is the brain itself, often represented as a set of numerous processing units, each collecting information from some units and generating activities to modify others. In addition, it is often underestimated that the brain communicates with three other worlds. We call these worlds the external environment, the extended body and the internal body. The external environment corresponds to the external world, including objects subject to the laws of physics and beings also subject to the laws of nature, possibly including intentionality. These objects and agents exist in space and time and can be sensed by external sensors (i.e. seen, heard, touched, tasted or smelt), defining perception. The extended body considers the body as an agent in the external environment, in which it may act. The extended body is composed of parts (e.g. limbs, head) carrying the external sensors. Their positions in space can be sensed by proprioception and can be modified by elementary actions and integrated motor programs (e.g. walking, grasping, speaking). We call exteroception these external flows of information (perception and proprioception). The internal body refers to all the machinery that makes the body work internally at the visceral, chemical, hormonal levels, i.e. eat, drink, breath, digest, etc. This defines the fundamental needs of a being, depending on internal states that can be sensed by interoception. Homeostatic mechanisms and other internal and external responses can modify these states.

As sketched in Fig. 2, the brain has consequently exteroceptive (perceptual and proprioceptive) and interoceptive sensors to get information about these worlds and their inner dynamics. It can act on them through a series of processes that we will call responses at large. These responses can be voluntary or involuntary and be applied to the extended or internal body, respectively corresponding to motions of parts of the body with skeletal muscles (e.g. speaking or running) and to the activation of endocrine or exocrine glands (e.g. releasing an hormone or crying), of smooth muscles and of the heart. Impacts of these responses in the three worlds can also be perceived by sensors.

For both motor and sensory aspects, the central nervous system (including the brain) is connected to these worlds by the peripheral nervous system, including a somatic part (for perception and proprioception and for external responses) and a visceral part (also known as autonomic nervous system, for interoception and internal responses). The autonomic nervous system is itself divided in two parts, the parasympathetic system responsible for the feed and breed activities and the sympathetic system for the fight or flight activities. In a simplistic view, this can be categorized in a dichotomic way with positive (pleasant) situations and incentives to exploit them for the nourishment and the reproduction of the body, opposed to negative (unpleasant or painful) situations and incentives to get rid of them for the integrity of the body. As we will discuss later, though partly overlapping, it would be too simple to directly associate this dichotomy to rewards and punishments, since for example a lower punishment than expected can be seen as pleasant. This can be more nicely integrated in a two opponent process system (Dickinson and Balleine, 2002) with mutual inhibition between two classes with opposite properties, acting against a baseline.

Before describing more technically information processes in the brain, it is fundamental to stress again that they have been selected in an evolutionary scheme, particularly to enable living beings to maintain their structure, to optimize survival and reproduction. This sets a special emphasis on the internal body world that has been designed and complexified by evolution to represent special body states indicating critical situations (that we will call emotions below) and giving specific incentives for that aim (that we will can motivations). We will define accordingly in Sect. 2.2 below, the two behavioral processes in charge of selecting responses, based on these body states, namely pavlovian and instrumental conditioning. In both processes, signals that are received can be used to directly trigger responses, based on their intrinsic value or on their capacity to activate internal representations. They can also be used to modify internal representations or to create new ones, following several learning processes that will be described in the next sections.

In summary the processes for the transduction of signals into responses and for the elaboration of internal representations of information are based on the signals received from the three worlds, on the current state of the memories and on the architecture of the cerebral structures.

2.2 Pavlovian and instrumental conditioning

Considering the brain as a system integrating different kinds of sensations to decide for different kinds of responses, as sketched in Fig. 2, several mechanisms of increasing complexity have been aggregated to this system along evolution. A first set of mechanisms directly associated to emotional learning is related to pavlovian (or respondent) conditioning (Balleine and Killcross, 2006). Some biologically significant stimuli also called Unconditional Stimuli US (e.g. a predator or some food) can be automatically identified (without learning) and trigger Unconditional Responses UR (e.g. freezing or salivation). Pavlovian conditioning corresponds to learn that some initially neutral Conditional Stimuli CS (e.g. a tone or a light) announce the arrival of US. Pavlovian conditioning has been modeled by learning rules modifying CS-US associations as a function of prediction errors between the actual US and the US predicted by the CS (Le Pelley, 2004). If several CS predict a US, the more reliable predictors are taken into account (automatic processing, Mackintosh rule). In case of a US not explained by the CS, learning rather applies on new predictors (controlled processing, Pearce-Hall rule). Subsequently, the occurrence of CS can generate two phases of behavior (Balleine and Killcross, 2006). In the consummatory phase, the CS is associated to the sensory properties of the US and triggers specific responses like chewing or blinking. In the preparatory phase, if the CS is distant, it is associated to motivational properties of the US and to its valence (aversive or appetitive) and triggers non specific responses like arousal, heart rate increase and approach.

At this stage, it can be interpreted that pavlovian learning is a way to anticipate, upon CS arrival, the negative or positive characteristics of the US and to prepare the body to this inevitable event. In the pavlovian scheme, responses are stereotyped (also called pavlovian reflexes) and are consequences of the learned associations. Several mechanisms have been described, inserting other responses in the pavlovian process (Balleine and Killcross, 2006). In autoshaping, an action can be triggered to more easily get a CS. In pavlovian instrumental transfer (PIT), animals exposed to a CS associated to a US trigger more frequently the (instrumental, cf. below) response that was learned to obtain that US. Similarly to the distinction between specific and general responses evoked above for consummatory and preparatory phases of behavior, PIT can be built on the specific sensory features or on the general affective properties of the US (Balleine and Killcross, 2006). We will explain in Sect. 4.3 that these two kinds of processing are in fact mediated by different circuits, serving different purposes.

Whereas, based on CS, pavlovian conditioning passively predicts the US to occur and prepares the body to this event, another more active learning scheme called instrumental (or operant) conditioning considers that, by acting, some reinforcement might be obtained. In this case, the learned association is between a response and the outcome observed to be the consequence of the response. This principle is also defined as Thorndike’s law of effect: A response will be triggered more (resp. less) frequently when observed to lead to a positive (resp. negative) consequence. Alternatively, a response can be triggered more frequently if it leads to the avoidance or removal of a negative stimulus. The two opponent process system mentioned above is a good basis to consider these negative cases where, at the end, no reinforcement is given : the absence of negative outcome can be considered as a positive outcome, on which to build learning with so-called conditioned inhibitors. It has also been associated to the representation of safety (Genud-Gabai et al., 2013).

Since here the response is voluntary, it is possible to consider the corresponding level of need, devaluate the outcome and refrain from acting if the motivation is low. Whereas pavlovian conditioning simply defines how much an outcome is liked, instrumental conditioning considers how much it is currently wanted and chooses to trigger responses taking motivations into account. This can be extrinsic motivations, to get a desired (external or extrinsic) outcome satisfying fundamental needs, including integrity of the body, seen as a positive motivation in the framework of the two opponent process system. With the elaboration of more complex internal representations that will be described below, expressing intrinsic motivations (Oudeyer et al., 2007) will be also observed. They are related to a more abstract need of (intrinsic) information, to obtain from the exploration of the complex world and from the monitoring of internal activity, as it is the case with curiosity and attention toward novelty.

Instrumental conditioning can be performed under the control of (or conditional to) stimuli also called occasion setters, that can become conditioned reinforcers (Cardinal et al., 2002), leading to chaining in complex behavioral goal-directed sequences toward primary reinforcers (respectively defined as sub-goals and goals in planning). Conversely, these associations can be transformed in habits through extensive learning, where the conditional stimuli directly elicit responses without references to the outcomes to be obtained (Boraud et al., 2018). More generally, this refers to a dichotomy between goal-driven behavior (where the behavior is driven by internal goals and can adopt complex schemes) and stimulus-driven behavior (where the agent mainly reacts to perceived stimuli).

2.3 Four questions to be addressed

We have seen above that it is important to identify stimuli (or ’objects’) in the world and relate them to the corresponding need they can satisfy. But in case of more complex scenarios, it is also important to learn how to reach them and how to act on them. We sketch these pieces of information as answers to four fundamental questions: what, why, where and how. For illustration in this section, we relate each of these questions to a cortical area associated to the underlying information representation.

The why and what questions relate the internal body and external environment. The what question is a way to encode a CS and its emotional impact. The CS can be seen as the current goal of the behavior (for example a bottle of water). It is consequently important to encode its physical characteristics (shape, color) to be associated to its emotional value (preference, like, dislike). Several regions of the posterior ventral (or temporal) cortex have been reported to be selectively responsive to such physical characteristics (Rousselet et al., 2004). The why question corresponds to the characteristics of the motivational impact on the body and is useful to encode the impact of the US or the bodily cost of a response to get it. Consider for example the level of water deprivation or the intensity of a pain. Such information is represented in the posterior insular cortex (Craig, 2009). It can motivate the behavior, explaining ’why’ we act (for which purpose) and why (up to which level) we accept to spend our energy.

The where and how questions relate the external body and the external environment. Answering the where question provides information about the position of an ’object’ and particularly with regard to (some parts of) the body. It will be important to locate and orient toward it. The how question refers to the need to learn how objects can be modified (e.g. approached, moved, manipulated) by the action of some body parts. The posterior dorsal (or parietal) cortex has been reported to be involved in both where and how functions (Milner and Goodale, 1995), respectively in its superior and inferior parts.

These cortical areas were just mentioned for illustration because, as it will be detailed below, many other brain regions are involved in answering these questions. Structuring under these four questions provides the main ingredients of a simple goal-directed behavior: We evaluate why it is important to satisfy a need; accordingly, we describe the goal of our response (what are its characteristics) and locate it (where) for consumption (how). But of course, in the real world, things are not so easy. Several motivations and goals can be in competition. Their characteristics can be difficult to extract. Variable delays can exist between the main ingredients of the behavior (the US, CS and responses) and some abstract reasoning can be needed (the bottle of water is in the fridge and I have to find the kitchen beforehand). All these elements correspond to increasingly complex behaviors, made possible along evolution, as we describe in the next section.

It is not intended here to reactivate the outdated triune brain theory (McLean, 1973), promoting a purely hierarchical description of brain structures and functions through three stages of evolution each bringing more evolved behaviors. Instead, it will be clearly explained here that both on the sensorimotor (Sommer and Wurtz, 2004) and emotional and motivational (Cardinal et al., 2002) sides, complex and bidirectional relations exist between recent and older cerebral structures, organizing a variety of control loops with different constants of time. For the sake of clarity, it is proposed to refer to a series of evolutionary steps to organize the remaining of the paper as described in Fig. 3, keeping clearly in mind that these steps are very schematic and that, of course, in addition to the initial function of some structures evoked here, evolution has continued in all parts of the brain including in older structures, resulting in the imbricated control loops mentioned just above and decomposed in more details below.

3.1 Elementary association to responses

At an elementary level, key ancient subcortical structures, already existing in lower animals without a cortex, can be linked to the four questions mentioned above, directly associating responses to rough representations of sensory information.

What - The amygdala

The amygdala is an heterogeneous set of structures with sensory and motor aspects (Swanson and Petrovich, 1998). Among its nuclei (LeDoux, 2007), the lateral nucleus receives a wide spectrum of sensory inputs from the thalamus and the cortex and is generally reported as a place for storing CS-US associations. The central nucleus of the amygdala (CeA) is the main output region for the expression of innate emotional responses and related physiological responses, particularly in relation to the periaqueductal gray (PAG) and the lateral hypothalamus for resp. aversive and appetitive behaviors. Another major nucleus is the basal nucleus, particularly in charge of information exchange with higher level structures like the prefrontal cortex and the hippocampus (Carrere and Alexandre, 2015). Neurons in this nucleus encode a variety of information for aversive and appetitive stimuli, related to the sensory nature of the US, to conditioned inhibitors and, for instrumental conditioning, related to conditioned reinforcers (Belova et al., 2007). It also encodes the level of arousal, ambiguity and unpredictibility of information (Pessoa and Adolphs, 2010). Altogether, the lateral and basal nuclei, also called the basolateral complex (BLA), is both a place for representing the valence and the value of emotional stimuli and a relay for more elaborated processing in the cortex, ventral striatum and hippocampus, in direct association with the central nucleus CeA responsible for simple pavlovian responses and also involved in more elaborated emotional responses (Cardinal et al., 2002).

Why - The lateral hypothalamus

The lateral hypothalamus contains nuclei evaluating needs of the organism and responsive to appetitive US, and nuclei promoting responses related to digestive functions, blood pressure and other visceral functions (Craig, 2003). In the framework of the two opponent process system, it is also strongly linked to the PAG for pain control. Similarly, on the aversive side, the PAG also encodes corresponding US and mediates defensive responses (Bandler and Shipley, 1994). Both structures are consequently reported as low-level homeostatic centres and are involved in preparation of the sensory inputs to the insula and in the expression of more elaborated motivational behaviors (Cardinal et al., 2002). Similarly to other subcortical structures depicted here, these structures, initially organized in sensory and motor domains to promote direct associations from visceral and somatic signals towards simple appetitive and aversive behaviors, have evolved in association with more recent structures to enrich their representations and participate within more complex circuits to more elaborated behaviors.

Where - The superior colliculus

The superior colliculus (also called the tectum in lower animals) is a structure mainly studied for its involvement in eyes movement and gaze orientation (Lee et al., 1988). It is composed of several layers, some receiving mainly visual information from many regions in the brain, including directly from the retina. The more superficial sensory layers are topographic maps of the surrounding environment and are in direct association with deeper motor layers for eye movements towards the place elected by competition in the sensory layer (Taouali et al., 2015). It has been remarked that this structure can also perform direct sensorimotor associations for orientation of the whole body for tracking novel stimuli, for defensive movements and flight in case of a danger (Dean et al., 1989). For more complex oculomotor behavior, the superior colliculus remains an essential stage between the retina and the posterior dorsal cortex and the Frontal Eye Field FEF (Sommer and Wurtz, 2004) in the frontal cortex.

How - The cerebellum

The cerebellum is a cerebral structure known for its role in sensorimotor control (Manto et al., 2012). Its granular cells are sensory inputs arranged in somatotopy and receiving most kinds of sensory information including from proprioception. They are directly associated with Purkinje cells projecting to all output structures responsible for motor control, from movement execution to planning. Particularly, these circuits have been shown to be involved in limb movements, manipulation, speech, both for direct automatic movements (postural adjustment, balistic movements, conditioned reflexes) and for the control of voluntary movement and even more abstract cognitive functions, through higher level centres (Middleton and Strick, 2000).

In summary and in accordance with an enactive view of cognition, it is important to consider why the behavior is triggered, toward what goal, where it is situated and how it can be accessed. Each of these questions can be tackled independently by a simple sensorimotor association and we have reported here evidences that, for each question, one cerebral structure is particularly involved in elaborating such simple association. We have also indicated that, in each case, other higher level structures can build more complex relations on the association, in a classical framework of imbricated sensorimotor loops (Guillery, 2005), convenient both for incremental learning and for responding at anytime, as sketched in Fig. 4. Before bringing more information about other levels of imbrication, we first evoke another problem, related to a need of consistency between the selection of answers to these questions.

3.2 Selection of response in CBG loops

In mammals, the control of action is mainly cortical. In summary and as represented in Fig. 4 (cf. sections 4.2 and 4.3 for details), the cortex can be separated in a sensory pole in the posterior cortex (posterior to the central sulcus) and a pole representing responses (elementary actions, integrated behaviors or decisions impacting (acting on) internal variables) in the anterior (or frontal) cortex (Fuster, 1989). Section 4.3 explains how these responses in each region of the frontal cortex are learned, executed and monitored as transitions between sensory states in associated regions of the sensory cortex. Some frontal regions are agranular (with no or minor layer IV). They correspond to the motor and premotor cortex and to a part of the orbitofrontal cortex. They perform a selection of response from perceptual cues and internal state values when external contingencies are stable. This corresponds to stimulus driven and habitual behavior (Boraud et al., 2018). Other frontal regions, called together the prefrontal cortex (PFC), are granular and are only present in primates (though this is disputed (Uylings et al., 2003), as some features of the prefrontal cortex might be present sooner in evolution). They correspond to the ventral and dorsal regions of the medial prefrontal cortex (mPFC) and of the lateral prefrontal cortex (lPFC) and to the frontopolar cortex FPC, unique in humans (Koechlin, 2014). The prefrontal cortex is engaged in executive (or cognitive) control with such mechanisms as Task Sets and Working Memory described in Sect. 4.3. In summary, when the world is uncertain or when the behavior is guided by internal goals, the idea is to replace the dominant default behavior guided by stimuli, by the selection or the design of new rules (addressing specific tasks), from the retrospective or prospective evaluation of the situation.

In both cases (response or rule selection), the goal is to make a decision from limited resources in the motor case (we cannot at the same time consume pieces of food located in two distinct places) and in the cognitive case (the distributed cortical circuitry prevents from easily mixing pieces of information that should be aggregated for a complex decision). P. Redgrave and colleagues have nicely addressed this problem in (Redgrave et al., 1999) and have proposed that the basal ganglia (BG) is the cerebral structure responsible for solving this kind of conflict in the brain, underlining that, even if brain processing is generally distributed, the process of response selection is fundamentally centralised, which is rather rare in brain functioning.

The inner processing of the BG is very complex, involving a variety of internal structures, pathways and mechanisms, as evoked in (Redgrave, 2007), which are still topics of intense research. Basically, the BG can also be described as a sensorimotor set of nuclei, with, on the sensory side, the striatum as an input structure receiving sensory and motor information and, on the motor side, the internal part of the globus pallidus and the substancia nigra pars reticulata (called together GPi-SNr in primates; can also correspond to other names in other species) acting as output motor structures. Most of the models of response selection in the BG (Hazy et al., 2006; Chakravarthy et al., 2010; Boraud et al., 2018) describe inhibitory and excitatory pathways associating internal BG nuclei (like the external part of the globus pallidus, GPe) and defining loops between the BG and the Cortex, called CBG loops and sketched in Fig. 4.

For each CBG loop, a region of the striatum receives afferent information from frontal and posterior cortical regions and one among the responses represented in the frontal region is going to be selected by a funneling effect from the striatum to GPI-SNr and back to the frontal cortex after a thalamic step (Alexander et al., 1986). Competition between the internal inhibitory and excitatory pathways is responsible for either maintaining the current selection (maintaining a working memory or the execution of a response) or changing and updating the selection. This process of response selection is generally permitted by a kind of reinforcement learning, with a prominent role for the dopamine, sent by the ventral tegmental area (VTA) and the substantia nigra pars compacta (SNc) to modulate cortico-striatal connections. This process helps define elementary associations (the subtle combinations of cues, responses and contexts permitted by this circuitry that we call here rules) that minimize reward prediction errors (Joel et al., 2002). In addition to this fundamental role of dopamine in learning, another synergistic role of dopamine related to the control of performance also participates in the gating mechanism, deciding for the maintenance or updating of information and will be described in Sect. 4.3.

Several parallel loops (five in (Alexander et al., 1986)) have been described between the cortex and the BG, adapting this generic function of response selection to different kinds of information. But it is notable that in all cases the circuitry is similar. Its form in a generic virtual loop is represented in Fig. 4 and the extended representation in five loops in Fig. 5. The loops are displayed on a posterior-anterior axis in the frontal cortex and also correspond to different and well-identified regions of the striatum, as confirmed by many data reporting the topographical organization of projections and of information representation in these circuits (Alexander et al., 1986; Parent and Hazrati, 1995). The loops have been named depending on the frontal areas mainly engaged and consequently on the kind of responses selected by the loops (Alexander et al., 1986). In the motor loop, the dorsolateral striatum (mainly the putamen) receives information from the motor cortex and proprioceptive information from the sensory cortex. The loop is somatotopically organized, to select different classes of motor actions, e.g. involving the face, the arm or the legs (Alexander et al., 1986). The oculomotor loop (Hikosaka et al., 2006) participates in gaze orientation and involves also regions of the dorsolateral sensorimotor striatum (mainly the caudate nucleus) receiving projections from the posterior dorsal cortex together with the Frontal Eye Field FEF frontal area, known to encode gaze movement (Sommer and Wurtz, 2004). These two latter loops, both including the dorsolateral striatum and involved in sensorimotor behaviors, are sometimes called together motor loops.

The lateral prefrontal cortex loop, also called the cognitive or the associative loop, involves the dorsomedial striatum, the hippocampus and associative regions of the posterior and prefrontal cortex and is mainly engaged in cognitive control (Koechlin et al., 2003), related to the ability of the prefrontal cortex to manipulate abstract rules, as described below in Sect. 4.3. In the medial prefrontal loop, the ventral striatum (mainly the core of the nucleus accumbens, NAcc) receives sensory information about the needs of the body and the rewarding value of the CS experienced, and is rather involved in the level of ’wanting’ associated to instrumental learning, with the anterior cingulate cortex in the dmPFC cortex, monitoring performance with regard to the association of responses and reinforcement, to decide for a cost of response (fatigue, risk) adapted to the level of wanting (energizing role of motivation (Niv, 2007)), and the vmPFC possibly adapting the value of the goal to the needs (devaluation). In the orbitofrontal loop, the ventral striatum (mainly the shell of the nucleus accumbens) receives information from the posterior ventral cortex and from the hippocampus, giving sensory details about objects, and from the lateral part of the orbitofrontal cortex, reported to encode the sensory value of objects, typically of the US, and to define the ’liking’ of objects or their hedonic value (preference) (Kringelbach, 2005). Altogether, the latter two loops referring to values have been termed the limbic loops, as a reference to the limbic system (also including the amygdala, hypothalamus, hippocampus, also associated to emotions and motivations). This five-loops system is sometimes summarized into three more general loops (Parent and Hazrati, 1995), concerning sensorimotor (motor and occulomotor) control, limbic (emotional and motivational) control and associative (or cognitive) control.

The anatomical organization of the BG has been described by the parallel processing made by these segregated loops but also by a convergence of information (Parent and Hazrati, 1995) due to several characteristics. Within loops, a funneling effect is obvious when the reduction in the size of data flow from the cortical input to the output of the loops in GPi-SNr is considered (Joel et al., 2002). Between loops, overlapping schemes can be deduced from several principles, like the spiral principle proposed in (Haber et al., 2000), where pathways between loops can be observed through dopaminergic projections and through overlapping frontal representations from one loop to the next. This is also the case, considering the participation in the loops of the subcortical structures mentioned in Sect. 3.1 (McHaffie et al., 2005). These structures can be functionally associated, one to one, to the loops, as proposed in Fig. 5, whereas anatomical data suggest a wider scheme, for example with the cerebellum linked to the motor and oculomotor loops (Hikosaka et al., 2006) or the amygdala linked to orbitofrontal and medial prefrontal loops (Cardinal et al., 2002).

As it is proposed in Fig. 5, taking apart the cognitive loop that will be described in Sect. 4.3, each of the other four CBG loops can be seen independently as selecting a response (an elementary action, an integrated behavior or a decision impacting (acting on) internal variables, depending on the nature of information) with regard to its afferent information, and as participating to the answer to one of the four fundamental questions. In the orbitofrontal loop, What is the goal of my behavior? In the medial prefrontal loop, Why should I spend energy satisfying the corresponding need (and up to which level)? In the oculomotor loop, Where is this goal? In the motor loop, How should I behave (which response should I trigger) to get it?

Depending on the complexity of the task and on the richness of the environment, these decisions can be constrained and articulated in different ways. On one extreme, we are in the domain of goal-directed behavior, when there are several answers to each questions, and when loops must interact to find the best global solution: contingencies between local decisions and their consequences must be evaluated and corresponding reinforcements must be compared. This can be associated to the domain of planning (Pezzulo and Castelfranchi, 2009), with the classical steps of deciding for goal, motivation, strategy and execution, and of backtracking in the hierarchy when one step is impossible. On the other extreme, we are in the domain of habits, where the current state is enough to directly trigger the response with no need to refer a priori to a model of sensorimotor transitions or to the value of the outcome. Many behavioral experiments (Packard and Knowlton, 2002) have shown that both types of learning are present and in competition in the brain and probably that the longlasting learning of the later is dependent on a supervision by the former. It is consequently important to wonder how the rich representations of states and of sensorimotor contingencies needed by goal-directed behavior are built by learning and have been complexified through evolution by developing different kinds of memories, as we evoke now.

4.1 Keeping a memory of past episodes

Basically, we have explained above that, to give an ecological meaning to our behavior, our direct sensorimotor capabilities (being able to orient toward an object of interest (the where question) and being able to exploit the object with the body (the how question)) are enslaved by the motivational and emotional analysis of the situation (the why and what questions). At a first level of complexity, this can be performed by subcortical structures (amygdala, PAG and hypothalamus) learning simple pavlovian associations and having strong relations with the ventral striatum.

In the simplest cases, when the goal of the behavior has been identified in the sensory region of the amygdala (BLA) and is directly available for consumption, BLA activates the amygdalar output CeA for pavlovian response and sends also projections to the shell of NAcc for the corresponding consummatory behavior. Anatomical and functional considerations underline how these responses are similar. There is in fact anatomical continuity between CeA and the shell of NAcc with a proposed similar functional organization (Cassell et al., 1999) including strong dopaminergic innervation and projections to the same regions of motor output (including PAG and lateral hypothalamus).

When the target of the behavior is not directly identified, the general class of motivation can give information to energize a preparatory behavior that will result in selecting the target. This is allowed by projections from CeA to the core of NAcc and can result in simple autoshaping or in more complex goal-directed behavior. This view gives to the ventral striatum (or NAcc) a central role at the interface between pavlovian and instrumental learning (Mannella et al., 2013). That is the reason why it is particularly interesting to remark that, considering more elaborated information that was incorporated to the system along evolution and particularly from birds (Striedter, 2016), information of episodic memory originating from the hippocampus is projected to the striatum mainly in its ventral division (Voorn et al., 2004).

In mammals, within the medial temporal lobe generally reported as dedicated to declarative memory, the hippocampus is more precisely associated to episodic memory (Tulving, 1972), allowing to remember specific events in their context. Through its input structure, the entorhinal cortex, the hippocampus receives cortical information from the posterior ventral cortex related to the what and why questions (via the perirhinal cortex) and from the posterior dorsal cortex related to the where and how questions (via the postrhinal cortex, also called parahippocampic, depending on species) and aggregate them, including their organization in time (Jensen and Lisman, 2005), in an episode or event (Diana et al., 2007). This association of arbitrary information is made possible by the unique recurrent architecture of the hippocampal region CA3 that makes it work as an associative memory, learning very rapidly an event (Kassab and Alexandre, 2015). This recurrent structure appears in birds (Striedter, 2016). In reptiles, the ancestor of hippocampus is just a spatial memory.

Decision to memorize an event can be made intrinsically on the basis of its novelty and from extrinsic afferents, particularly originating directly or indirectly from the amygdala (Paz and Paré, 2013), signaling errors of reward prediction and consequently a need for a more precise learning. Errors might be due to ambiguities in the conjunction of features (O’Reilly and Rudy, 2001) or in their temporal ordering, as the hippocampus is also particularly critical for sequence and delay learning (Jensen and Lisman, 2005). The dentate gyrus (DG) appears in mammals (Striedter, 2016) as an intermediate step between the entorhinal cortex and CA3, with different functions of pattern separation (Kassab and Alexandre, 2018) during learning, to avoid errors in recall. In the recall process, thanks to direct projections between the entorhinal cortex and CA3, the hippocampus can be activated from partial information, evoke the complete episode and facilitate reactivation of other brain regions (Gruber and McDonald, 2012) via its output structures, CA1 and the subiculum. This has been for example reported as contextual signals sent to the amygdala for the extinction of pavlovian conditioning (Moustafa et al., 2013) or as predictive signals of possible paths sent to the entorhinal and prefrontal cortex and also to the ventral striatum during navigation of rats in a maze (Gruber and McDonald, 2012).

From its ability to store and later detect and recall complex multimodal episodes, particularly including delays between their constituents, the hippocampus provides the ventral striatum and the amygdala with more complex features than simple sensory cues sent by the thalamus or the cortex. It is for example reported that hippocampal inputs are critical to the amygdala in pavlovian trace conditioning (Paz and Paré, 2013), when the CS and the US are separated by a delay. This also allows to create conditioned reinforcers in the amygdala, corresponding to sub-goals or intermediate steps in a sequence of behaviors, sent to the ventral striatum and evoking surrogates of rewards when the actual reward is distant, as it is often the case in instrumental conditioning (Cardinal et al., 2002).

The distinction evoked above between the posterior ventral cortex (the what and why questions) representing perception for recognition and the posterior dorsal cortex (the how and where questions) rather representing perception for action (Milner and Goodale, 1995), has also been clearly reported in the hippocampus (Fanselow and Dong, 2015), with a dorsal region rather involved in navigation, with neurons coding for location (place cells) or head direction and a ventral region rather involved in emotional aspects and massively projecting to the amygdala and to the ventral striatum (mainly the shell). It must be noted that the dorsal hippocampus also projects to the core of the ventral (and the dorsomedial) striatum and to the anterior cingulate cortex (Pennartz et al., 2011), which underlines the special position of the medial prefrontal CBG loop, intermediate between pavlovian and instrumental conditioning and associating basically responses and outcomes. This will be discussed in more details in Sect. 4.3 below.

Recent hypotheses about the interplay between the hippocampus and the entorhinal cortex, associated to the computing formalism of successor representation (Stachenfeld et al., 2014) also propose a powerful basis for reasoning processes. In this view, CA3 is seen as a cognitive map encoding not only locations (classically associated to place cells) but also prediction of possible transitions to other locations under a probability distribution learnt through episodes stored in CA3. It is then proposed that this information is exploited in the entorhinal cortex, where so-called grid cells encoding various metric representations have been reported to perform path integration by performing a hierarchical decomposition of space. Now considering that the hippocampus also learns episodes of non spatial concepts, the same process will correspond to propose in the corresponding regions of the entorhinal cortex, a hierarchical decomposition as can be observed in planning, extracting subtasks and subgoals at various levels of description (Stachenfeld et al., 2017). It has been observed for some time (Pezzulo et al., 2014) that internally generated sequences resembling such episodes are replayed at certain key moments and particularly during rest. This replay mechanism is proposed to reinstantiate retrospective memory in the posterior cortex to improve training. It could also be a basis for prospective memory in the PFC, with such mental processes as planning, reasoning and more generally thoughts, as discussed in Sect. 4.3.

In summary, the hippocampus can represent complex events, corresponding to specific episodes, introducing rich and complex sensory information in pavlovian and instrumental conditioning. This gradient of complexity in sensory inputs, from specific cues encoded in the sensory cortex to cognitive maps and emotional episodes in the dorsal and ventral hippocampus is very nicely illustrated in (Voorn et al., 2004), gathering anatomical information in rats about hippocampal projections to the striatum, the amygdala and the frontal cortex, ordered along that gradient. Such complex information allows birds and mammals to learn pavlovian associations with a complex pattern in time. It is also critical in goal directed behavior which requires the prospective evocation of perception-response contingencies and of outcome values, as it has been reported in the hippocampus and the ventral striatum (Bornstein and Daw, 2011).

Experiments in rats (Packard and Knowlton, 2002) have shown that rapid and flexible goal-directed behavior involving the hippocampus and the dorsomedial striatum can be replaced by repetition by an habitual behavior involving the dorsolateral striatum and corresponding to a simple stimulus-response association insensitive to reward devaluation. Since the dorsolateral striatum has no hippocampal but only cortical sensory inputs, it can be thought that the slow habitual learning is constrained by the time for consolidation from the hippocampus to the sensory cortex, of the critical events triggering the response, as described in Sect. 4.2 below. In fact, when habits have been learned, the same experiments (Packard and Knowlton, 2002) show that both goal-directed and habitual learning coexist and are in competition. In a very interesting view (Penner and Mizumori, 2012) using the actor/critic framework where reinforcement learning is decomposed in an actor applying the current policy (rules of behavior) and a critic learning from errors of prediction the value of the outcomes and modifying the policy correspondingly, the dorsolateral striatum is proposed to be the actor for habitual behavior and the dorsomedial striatum to be the actor for goal-directed behavior. The shell, corresponding to the consummatory behavior and learning explicitly the value of the outcome, is proposed to be the critic of the dorsomedial striatum, learning explicitly the model of the world for goal-directed behavior, whereas the core, associated to preparatory behavior not specific of the outcome and learning only to associate a response to a motivational value, should be the critic of the dorsolateral striatum, associating directly in a habitual mode states with responses.

All these pieces of information give a very important role to the ventral striatum at the interface between limbic and motor systems. (Mannella et al., 2013) describes the ventral striatum as the place where motivational values are assigned to goals from their pavlovian value given by the amygdala and their salience and novelty given by the hippocampus. This results in associations between the outcomes and their motivational value in the shell and between responses and outcomes in the core, and the corresponding energizing effect on instrumental behavior. The dorsomedial striatum is also a key player in instrumental behavior and its role will become more clear as more details are given about the prefrontal cortex in Sect. 4.3.

4.2 Building abstract categories

Beyond the memory of specific episodes in the hippocampus, an important innovation has been brought in mammals by the cortex to build structured high-level information over simple signals: the elaboration of abstract categories composing a semantic memory. In the posterior cortex such a representation is built on data flows corresponding to the sensory dimensions evoked by the four questions discussed above (cf. also Fig. 5). This results in hierarchical cortical areas with neuronal populations responding to more and more complex objects (Rousselet et al., 2004), building more and more abstract categories in the ventral information flow relating the exteroceptive and interoceptive poles, for the What and Why questions. Based on considerations on the timing of information propagation (Nowak and Bullier, 1997), the information flow is described as parallel rather than serial in the dorsal pathway to elaborate categories between the exteroceptive and motor poles, related to the questions Where and How, even if intermediate strategies are also observed in associative areas, between a purely constructivist hierarchical and a purely purposive specialized view of information processing (Norman, 2002). This intricate representation is particularly useful to account for selective attention, a function of the posterior cortex particularly critical in primates (Fix et al., 2011), associating selection of spatial regions and implicit or explicit (covert or overt) involvement of body parts in the dorsal regions (for example corresponding to eye movements in the visual case) together with an anticipation of the subregion of the sensory space that will be available and the focused processing of critical features in the ventral regions.

In these associative regions, one crucial (and still open) question is about the choice of the compound objects to be represented since the combinatorics is obviously too large for a systematic representation. This selection is made by learning and in an ecological view, a simple (but vague) criterion is: ”Those which are the most useful to the organism”. A more precise specification must rely on the mechanisms triggering sensory learning in the posterior cortex (Ahissar and Hochstein, 1993), including the role of cholinergic modulation triggered by the amygdala, in case of error of prediction, to favor attentional process in the cortex (Pauli and O’Reilly, 2008), and the role of reinstatement (or replay) of episodes in the cortex, driven by the hippocampus in the consolidation process (McClelland et al., 1995).

Another important actor in the processes described in this section is the sensory thalamus (Sherman, 2007), for the critical role of its sensory part in the activation of the posterior cortex, conciliating feed-forward sensory input and feed-back cortical expectations, and also in cortical learning of new categories, particularly involving multimodal features. Nevertheless, it will not be described in details in this paper. Nor will be described the motor thalamus, even if it also has a critical role in the functioning of the frontal cortex presented below.

4.3 Building flexible sequences

The organization of the frontal lobe can be described in reference to regions of the posterior cortex in which frontal regions can control transitions of states (Burnod, 1989). In the motor cortex, neurons arranged in stripes symmetrical to the somatosensory cortical area can trigger elementary motor actions modifying the position of the bodily scheme until the sensory goal of the action (e.g. position of a limb, characteristics of the sound produced by the phonatory apparatus) is reached. Motor control is also reported in the premotor cortex, with a more integrated topography (Graziano, 2006), corresponding to more ecological categories of the behavioral repertoire like climbing, reaching, etc. Similarly, in oculomotor regions like the frontal eye field FEF (Sommer and Wurtz, 2004), transitions are between initial and final targeted eye positions.

The same process of control of transition between present and targeted states can be used to describe the functions of the limbic frontal regions. In the orbitofrontal cortex, lateral and caudal regions have been described (Padoa-Schioppa, 2009) as learning sensory features of rewarding stimuli and ordering them in a transitive way to define preferences for emotional stimuli, seen as potential goals of the behavior. This is the basis for emotional control, where the selection of a desired goal is sustained until that goal is obtained. Complementary to this consummatory behavior built on specific sensory features, the preparatory behavior can be organized in the medial prefrontal cortex on more general properties, with the ventromedial prefrontal cortex evaluating the rewarding value of stimuli for decision making and, in the dorsal part, the anterior cingulate cortex performing motivational control (Kouneiher et al., 2009). Basically, this region, described as associating responses to outcomes, is in charge of deciding if the energy required by the responses selected in the preparatory phase is worth the corresponding need. Accordingly, it is reported to energize the behavior, i.e. to evaluate up to which level it can be engaged and, when a strategy is selected, to maintain this selection until it is achieved (or given up). Here also motivational control can be defined as a transition from selection to satisfaction of the need, with maintenance of activity if not achieved.

It can be remarked that the medial prefrontal cortex is structured in a ventral part, deciding for the selection of a goal from its rewarding value and current motivations, and a dorsal part, selecting the response from its cost. The dorsal part also monitors the progress of the actual behavioral sequence (Holroyd et al., 2018) and, by comparing actual and predicted costs and rewards, is able to detect errors and conflicts indicating that the current control is not adapted. This can lead to direct adaptation of the motivational control or when this adaptation is not trivial and requires elaborated contextual rules, the needed cognitive control recruits additional circuits in the lateral prefrontal cortex (Badre, 2008). This region, increasingly large in primates, is also distributed in ventral and dorsal regions and is reported to elaborate complex rules (useful to address a specific task), with their complexity defined as the level of sequential arrangement of responses in the dorsolateral prefrontal cortex and as the level of precision in the definition of cues in the ventrolateral prefrontal cortex (O’Reilly, 2010), both together able to build a complex strategy, decomposing a goal and a level of engagement into subgoals and responses to get them. The same principle of maintenance of activity until satisfaction (or giving up), described as a working memory process (Fuster, 1989), results in resistence to distraction, another strong characteristic of the prefrontal cortex. Altogether, it has been proposed that the anterior cingulate cortex plays here a central role (Shenhav et al., 2013) by integrating from the orbitofrontal cortex, the ventromedial prefrontal cortex and the lateral prefrontal cortex three different factors (respectively the value of the reward, the cost of effort and the cost of cognitive control), to determine whether, where and how much control to allocate.

Three different levels of reasoning have been proposed in mammals, along evolution (Koechlin, 2014), relying on different kinds of associations learned in different cortical regions. In a first stage, rodents are able to learn to select the most rewarding response from the perceived stimuli and to correct errors. In their motor loops, they learn S-R associations, called the selective model, useful in the habitual behavior and for anticipating sensory states resulting from a response. Rodents are also able, thanks to their limbic loops, to predict the forthcoming outcomes and to adapt their future behavior in case of errors. These S-O associations are called the predictive model (and are similar to what is learned in pavlovian conditioning). In a second stage, thanks to their lateral PFC as evoked just above, primates can build more complex criteria to select a behavior adapted to specific contexts and to adapt their behavior to the specific situation (context) before committing errors. They accordingly build a contextual model and define rules adapted to specific contexts. In the third stage, according to (Koechlin, 2014), the frontopolar cortex allows humans to monitor several strategies in parallel and to perform hypothesis testing independently from the actual behavior, thanks to prospective mechanisms as described in Sect. 4.1.

Some generic mechanisms of the frontal cortex can be re-interpreted now. Each region of the frontal cortex has been described as preferentially linked to a posterior cortical region and composed of responses monitoring a transition from one state represented in this posterior region to another (from an initial to a final position; from need (e.g. water deprivation) to satisfaction of the need (satiety), etc.). This can be interpreted with the scheme S1-R-S2, where S1 is the initial condition eliciting R as a possible response (cf. (O’Regan and Noë, 2001) and the principle of affordance) and S2 is the consequence that can be anticipated if R is preactivated. Conversely, if S2 is a desired state, R is the response that has to be activated to obtain S2, which is possible if S1 is compatible with the current state. Else, R can display a sustained activity, as in working memory, and remain actively waiting until S1 is satisfied. This interpretation is reminiscent of behavioral studies where antecedents and consequences of goal-directed behaviors are seen as beliefs and desires (Balleine et al., 2009) and of more theoretical works on planning (Burnod, 1989; Pezzulo and Castelfranchi, 2009) explaining how goals (desires) can be decomposed into subgoals (S1 becomes desired) and recursively executed in such S1-R-S2 schemes. In our view, these intermediate steps with subgoals can also be provided by the hippocampus and the entorhinal cortex at various levels of description, as suggested in Sect. 4.1 for prospective memory. They are executed by cognitive control in lPFC: the goal remains active as a working memory in mPFC and activates subgoals and means (which can be seen as intentions) to get them in lPFC until good conditions are met (e.g. finding the kitchen seen as a subgoal to open the fridge), without forgetting the initial goal (of drinking a bottle of water), as ensured by the sustained activity insensitive to distraction.

This view is very consistent with an interpretation of the role of the BG for the dynamic gating of frontal representations (O’Reilly et al., 2010), switching from the updating of the choice of the best response to be selected (from the prediction of the value of its consequence) to the maintenance of its sustained activity until this consequence is obtained. This also explains why goal-directed behavior is defined by its sensitivity to goal devaluation and contingencies of responses (Balleine et al., 2009): in the habitual mode, S1 directly triggers R with no consideration of S2 and of the value of the goal obtained at the end of the process whereas in a goal-directed process, when a action is executed, its consequences are compared with its expected results and the corresponding contingencies are updated in case of a mismatch. Beyond real actions, the premotor theory of attention (Rizzolatti et al., 1987) proposes that attentional control is a weaker activation of motor control, allowing to explore the same situations by an access to the same learnt representations with no (or covert) action. The situations evoked above can consequently be examined in such a mode, corresponding to virtual thoughts instead of real actions in the world.

Globally, this heavy and structured process of the frontal (= prefrontal + premotor and motor) cortex can be summarized as follows: Exteroceptive and interoceptive stimuli can elicit response preactivations in the motor and limbic prefrontal cortex which can also evoke the anticipated consequences in exteroceptive and interoceptive terms. In simple and stable worlds, the elaborated model of the world can become of good predictive quality and at the end, the initial stimuli can be sufficient to trigger directly responses without evoking their consequences. This corresponds to the habitual mode, progressively shifting the control from the limbic to the motor loops (Hélie et al., 2015) and in the long term, only mobilizing the motor cortex in a basic stimulus-response scheme.

Nevertheless, in the early phases of learning or when the world is changing or when the best behavior to be selected does not correspond to the most frequent (for example in a specific context), a more precise analysis of the recent history of performance must be carried out, involving the limbic parts of the prefrontal cortex and of the basal ganglia. This is the reason why the dorsomedial prefrontal cortex is often reported to be involved in error detection and conflict monitoring (Rushworth et al., 2004) and the ventromedial prefrontal cortex to be sensitive to devaluation of outcome (Kringelbach, 2005) for example in case of reversal and extinction. The interoceptive preactivation of the limbic loops can evaluate and supervise this goal-directed learning, depending if gains or losses are observed between anticipated and actually obtained punishments and rewards, and results in the selection of the current goal and motivation.

In this goal-directed process, the role of the basal ganglia is prominent as a critic in the limbic loops to learn from errors of prediction and as an actor to explicitely trigger step by step the full plan of responses, as it was explained above. Concerning the transition between loops, note that both ventro- and dorsomedial prefrontal cortex project to the dorsomedial striatum (Gruber and McDonald, 2012) and that the exteroceptive preactivation of the motor loop is critical to offer affordances that help select the most appropriate preparatory behavior (Pezzulo and Castelfranchi, 2009), also supposed to be performed in the striatal region. The double role of the dopamine (Braver and Cohen, 2000) must be also particularly emphasized here. On the one hand, dopaminergic signals carry reward prediction errors that can be used for learning as it has been shown for a long time in reinforcement learning, with dopaminergic projections from VTA to the ventral striatum mainly for pavlovian aspects and from SNc to the dorsomedial striatum for the instrumental aspects (Yin et al., 2008). These pathways are also at the basis of the spiral principle by S. Haber evoked above (Haber et al., 2000), concerned with the articulation between CBG loops. On the other hand, dopaminergic projections from VTA to the PFC participate to the modulation of performance, by acting on the gating mechanism between maintenance and updating of activity in PFC (O’Reilly, 2006) in case of sudden changes in goal representations. This dual role of dopamine can also be seen as a dual contribution to, respectively, model-free and model-based reinforcement learning.

In a classical view (cf. for example (Dayan and Niv, 2008)), goal-directed behavior is associated to model-based reinforcement learning and habitual behavior to model-free reinforcement learning, in reference to computational learning mechanisms where contingencies of the world useful for decision are respectively gathered in an explicit model of the world or cached in variables summarizing the current state. The analogy refers to the fact that cached variables propose a more compact and less expensive representation than an explicit model and are less sensitive to accumulated approximations, and that in contrast they are very long to evaluate and to modify when the world changes. But the analogy has also been recently questioned (Miller et al., 2018). Model-free is not a perfect term, since a model has been built and even if it has been compiled in cached variables, they ultimately depend on rewarding values; accordingly, value-free (total independence on reinforcement) should be the good term for habits. Concerning model-based learning and the manipulation of explicit knowledge, such information can come from the temporal cortex (for semantic memory) or from the hippocampus (for episodic memory). Experiments have shown that the ventromedial prefrontal cortex is a key relay to associate this information in the process of cognitive control (Vaidya and Badre, 2020).

We have evoked above selective, predictive and contextual models learned in the motor, limbic and lateral loops of the frontal cortex by accumulating history of occurrences of corresponding motor, sensory and rewarding cues. Altogether, this has led to propose the concept of Task Set to describe the organization of frontal regions (Sakai, 2008) and to propose computational mechanisms for the coordination (cognitive control) and selection (decision making) of thoughts and responses for adaptive behavior (Domenech and Koechlin, 2015). In the ideal case, a configuration of cognitive processes in these loops has been selected as the behavior adapted to the situation and it will be actively maintained for subsequent task performance. In case of a problem, some processes will be adapted a posteriori and a new more adapted configuration will be selected or possibly created. These mechanisms are reported to be compatible with the observed arrangements and activations of frontal regions (Donoso et al., 2014); they still have to be consolidated at the computational level beyond stereotyped tasks, particularly concerning the question of creativity (Domenech and Koechlin, 2015). In the current view, applying the best Task Set from a previously learned repertoire rather suggests a strategy globally similar to model-free learning, with critical points where the strategy must be explicitly reconsidered in a model-based manner. In this insightful view, it has to be noted that model-based and model-free approaches are cooperative and not concurrent, thus minimizing their reported weaknesses. Such integrated architectures have already been proposed in the past in reinforcement learning (see the Dyna architecture in (Sutton, 1991)) and are presently extended with stronger biological bases (Hassabis et al., 2017).

All put together, we are still under the double constraint of goal driven and stimulus driven behaviors, with the general pre-eminence of the limbic side (O’Reilly et al., 2014) generating needs corresponding to motivations to be fulfilled. These motivations are then translated into desired goals to obtain from the environment. Reciprocally, stimuli can pre-activate motor responses by affordance, which will be directly triggered in case of habits and which will otherwise simply generate predictions of what could result of the response if triggered. In a pavlovian scheme, stimuli can also preactivate anticipated rewards. From this common basis, agranular frontal areas (motor, premotor and lateral orbitofrontal cortex) can directly make a pertinent decision if the world is stable enough. Else, cognitive control is needed with the help of medial and lateral prefrontal cortex, inhibiting the default behavior and imposing new rules adapted to the context, by an attentional process on the posterior cortex. Within this view, the behavior is seen as the control of perception, with affordances to select responses, depending on the desirability of the predicted outcomes (Pezzulo and Cisek, 2016). Interestingly, concepts like intentionality, thoughts and imagination (seen as a control of thoughts) can also be evoked in that view, as mechanisms of the cognitive control; higher cognition is in fact partly elaborated on the same basic sensorimotor and motivational loops.

In this paper, we have proposed a systemic description of the brain, as a general framework where specific models in computational neuroscience should be positioned before their development. Beyond its intrinsic interest, this framework is necessary, else the risk is to build models of particular neuronal structures in isolation without reference to more global information flows and cognitive functions and consequently to neglect some characteristics of the structure or to overload it with functions carried out in other parts of the cerebral network. This description in width rather than in depth also evokes a variety of sensorimotor loops and levels of representation, from pavlovian to instrumental conditioning, from goal-directed to habitual behavior, from episodic to semantic memory, from simple to complex rules, that can coexist and act in competition or in synergy. Having a global view of the underlying information flows can be useful to set a specific model back in a more general and dynamic cognitive context.

This framework considers several fundamental aspects of the brain, seen as the device controlling the behavior of the body, as summarized in Fig. 6: (i) In an enactive view, the brain has to elaborate loops with the internal and the external environments (cf. Figure 2) and to ensure their stability for the general goal of survival. This circular causality has already been expressed in many systemic views including, in computational neurocience, a very interesting approach exploiting the powerful formalism of thermodynamics (Friston, 2012). It confirms also the fundamental organization of the brain in sensorimotor loops and structures (Guillery, 2005). (ii) To provide a more precise account of the various characteristics of behavior, we have structured brain functions as answering four fundamental questions (what, why, where and how). From the basic what/where system (Ungerleider and Mishkin, 1982), this terminology has a long history in neuroscience, including more recent and precise views (Milner and Goodale, 1995; O’Reilly, 2010). In particular, a very close formulation is proposed in (Verschure et al., 2014) but is less accurate concerning the mapping to brain structures and functions. This latter paper insists on the need for the organism to answer these four fundamental questions (and also the When question, which is implicitly tackled in the present paper) but they are not related as it is the case here to specific regions of the striatum, frontal and sensory cortex and to specific subcortical structures. (iii) Functional and anatomical considerations are respectively added in Figs. 4 and 5 to explain how these questions can be addressed in brain structures, particularly taking phylogeny into account. Lower animals have the same fundamental problems of behavior organisation for survival; the main difference is about the quantity and quality of information that is provided and built in internal representations to define goals and needs and to elaborate answers. In this perspective, the mnemonic mechanisms described in Sect. 4 are seen as new representations provided by phylogeny to extend the power of the same pavlovian and instrumental mechanisms by applying them on past episodes and prospective cases (episodic memory), new categories (semantic memory) and flexible rules (working memory). The transformation of memory systems along evolution is presented in details in (Murray et al., 2016), in particular with more information about their impact from a social point of view in humans but with no reference to computational aspects which are central here.

Elaborating such a framework also provides opportunities to put together many facts at various levels of description and to elaborate on them principles of brain functioning and of cognitive architecture. Basically, the behavior is elaborated from cross-talk between the motor and limbic poles (cf. Figure 6). The motor pole receives affordances from the exteroceptive pole that might result in directly triggering habits or at least preactivating some responses. The limbic pole receives emotional and motivational information from the interoceptive pole which can directly trigger pavlovian reflexes and consummatory behavior or lead to the definition of goals and motivations. Then, instrumental conditioning will allow to specify a more appropriate preparatory behavior, based on preactivated responses and learned contingencies. In the simplest case, as permitted by the agranular frontal areas, an immediate decision is sufficient to trigger the behavior in a stable world, from the selective and predictive models. Else, as permitted by granular frontal areas, the sensory representation must be modified by cognitive control to be adapted to the situation, after an internal deliberation possibly exploiting episodic and semantic memory. What makes the picture difficult to comprehend here is that all these memory systems coexist, collaborate and compete for the control of behavior. This has been for example studied for pavlovian and instrumental conditioning (Dorfman et al., 2019) and for model-based and model-free learning (Daw et al., 2005), each time mentioning the dorsomedial prefrontal cortex as the place where the associations are weighted, depending on the characteristics of the predictive models and of the present situation.

As we might expect, this sketch gives a prominent role to pavlovian and instrumental conditioning, in this survival-oriented definition of behavior. Nevertheless, the contribution of other kinds of learning is obvious, specifically related to semantic memory in the posterior cortex, episodic memory in the hippocampus and working memory in the frontal cortex. More fundamentally, this also lays emphasis on the fact that cognition can be described as a dynamical system of interacting memories, some acting to provide information to others, to replace others when they are not efficient enough, or to help for the improvement of others. Such principles have already been described with a very strong impact in improving our understanding of cognitive mechanisms (McClelland et al., 1995). The framework that we have proposed here is a very convenient tool to study such mnemonic synergies, very difficult to delimit because not related to a unique cognitive function. We have for example seen that goal-directed behavior can generate a prospective memory (also called ”memory of the future” (Fuster, 1989)), as it is observed for places in the hippocampus and for rewards in the ventral striatum, and can also generate a retrospective memory that can participate in training the habitual system (Bornstein and Daw, 2011). Precisely understanding how these processes work and interact is an important challenge for future research.

It can also be remarked that, throughout the described mechanisms and representations built in the various kinds of memory, a central role is given to sensations. Apart from motor actions participating to the selective model (learning the S1-R-S2 associations that might drift toward the S-R habitual schemes), all the other learned structures are essentially related to sensations. This corresponds to the S-O predictive model but also, within the contextual model, to the idea that internal responses are going to bias sensory representations by attentional means. This is consistent with the idea of motivational priming (Bindra, 1978) proposing that motivations are in fact oriented toward sensory features, and more generally with the principle that behavior is the control of perception (Pezzulo and Castelfranchi, 2009).

Describing the brain as an architecture of learning systems has also strong implications in Machine Learning. In addition to the definition of interacting mnemonic synergies as a basis to ensure really autonomous learning, which is very poorly addressed in classical Machine Learning, a more realistic view of pavlovian and instrumental conditioning can be very precious to revisit classical Reinforcement Learning. Such a contribution is for example proposed in (Balasubramani et al., 2014) where the selection of responses is controlled by a utility function, defined by a weighted combination of value and risk. This can extend the classical actor/critic architecture and needs to be deepened, since many questions remain about the exact location of the critic (if any) and about the validity of the hypothesis of a double model-based and model-free actor-critic system evoked above (Penner and Mizumori, 2012).

Our systemic framework is also useful because it helps revisit the role of certain cerebral structures. Particularly, it appears from our analysis that the BG is best defined as a modulatory system that provides adaptive gating signals to the frontal cortex, instead of the prevalent ideas that BG directly encodes S-R associations and can be defined as a procedural learning system. This is confirmed by the (relatively) low impact of BG lesions in behavioral performances but rather in learning (Packard and Knowlton, 2002). One step further, the following phylogenetic interpretation could be proposed: Beyond simple reactions due to ancient subcortical structures, the hippocampus and the BG could be proposed as two structures endowed with rapid learning to adapt the behavior to specific cases, respectively corresponding to emotional episodes and to behaviors where the motivation must be explicitely reminded. But in both cases, the goal of the organism is to identify and learn criteria to automatize the behavior. In the same way as there is a consolidation from the hippocampus to the posterior cortex to create new categories in semantic memory, appropriate to discriminate the world, the transfer from goal-directed behavior involving the BG to habits in the motor pole of the frontal cortex might be seen as a way to create motor routines, giving a behavioral repertoire adapted to our needs. At the end, this would result in a system parallel to the ancient subcortical structures, except that the sensory and motor characteristics would have been selected and learned in a slow process, from interactions with the world.

Our framework also indicates some specific structures, which can be seen as a kind of hub in the cognitive architecture, because they are fundamentally multimodal and coordinate transfers between memory systems. For different reasons, this is particularly the case with the ventral striatum and the hippocampus which woud have to be studied more deeply in that perspective. Another domain which remains not detailed enough is the precise definition of the mechanisms of cognitive control, with the elaboration of complex rules in the lateral prefrontal cortex and their specific dorsal and ventral aspects. Preliminary theories have already been proposed (Badre, 2008; Koechlin et al., 2003) and should be more deeply explored in specific behavioral applications for a better understanding. Similarly, the role of neural structures like the frontopolar cortex or the thalamus and of neural mechanisms like neuromodulation should be more precisely studied and associated to the proposed framework.

It might finally be said that this paper proposes a very ”mechanical” view of the brain, neglecting highest cognitive functions of the brain, for example related to social aspects, language, not to evoke internal thought, mind wandering and consciousness. We think in fact that most of these functions rely on the same bases as the ones evoked here, since they also correspond to organize behavior in time and associate constraints emerging from internal and external worlds because, at the end, the motivation of a human being is also being well adapted to its mental, linguistic and social world.

Ethics approval and consent to participate :

Not applicable

Consent for publication :

Not applicable

Availability of data and materials :

Not applicable

Competing interests :

The author declares that he has no competing interests

Funding :

The author is an employee of the french government

Authors' contributions :

FA wrote the manuscript

Acknowledgements :

I thank all my former and present students, whose work contributed to build that framework

Ahissar M, Hochstein S. Attentional control of early perceptual learning. Proc Natl Acad Sci USA. 1993;90(12):5718–22.
Alexander G, DeLong M, Strick P. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Ann Rev Neurosci. 1986;9:357–81.
Badre D. Cognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes. Trends in Cognitive Sciences. 2008;12(5):193–200.
Balasubramani PP, Chakravarthy VS, Ravindran B, Moustafa AA. (2014). An extended reinforcement learning model of basal ganglia to understand the contributions of serotonin and dopamine in risk-based decision making, reward prediction and punishment learning. Frontiers in Computational Neuroscience, 8.
Balleine BW, Killcross S. Parallel incentive processing: an integrated view of amygdala function. Trends Neurosci. 2006;29(5):272–9.
Balleine BW, Liljeholm M, Ostlund SB. The integrative function of the basal ganglia in instrumental conditioning. Behav Brain Res. 2009;199(1):43–52+.
Bandler R, Shipley MT. Columnar organization in the midbrain periaqueductal gray: modules for emotional expression? Trends Neurosci. 1994;17(9):379–89.
Belova MA, Paton JJ, Morrison SE, Salzman CD. Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron. 2007;55(6):970–84.
Bindra D. How adaptive behavior is produced: A perceptual-motivational alternative to response reinforcements. Behav Brain Sci. 1978;1(1):4152.
Boraud T, Leblois A, Rougier NP. A natural history of skills. Prog Neurobiol. 2018;171:114–24.
Bornstein AM, Daw ND. Multiplicity of control in the basal ganglia: computational roles of striatal subregions. Curr Opin Neurobiol. 2011;21(3):374–80.
Braver TS, Cohen JD. (2000). On the control of control: The role of dopamine in regulating prefrontal function and working. MIT Press. Making Working Memory Work, 551–581.
Burnod Y. (1989). An adaptive neural network: the cerebral cortex. Masson.
Cardinal RN, Parkinson JA, Hall J, Everitt BJ. Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neuroscience Biobehavioral Reviews. 2002;26(3):321–52.
Carrere M, Alexandre F. (2015). A pavlovian model of the amygdala and its influence within the medial temporal lobe. Frontiers in Systems Neuroscience, 9(41).
Cassell MD, Freedman LJ, Shi C. (1999). The Intrinsic Organization of the Central Extended Amygdala. Annals of the New York Academy of Sciences, 877(1):217–241.
Chakravarthy VS, Joseph D, Bapi R. What do the basal ganglia do? A modeling perspective. Biol Cybern. 2010;103(3):237–53.
Craig A. How do you feel – now? the anterior insula and human awareness. Nat Rev Neurosci. 2009;10:59–70.
Craig AD. Interoception: the sense of the physiological condition of the body. Curr Opin Neurobiol. 2003;13(4):500–5.
Damasio AR, Carvalho GB. The nature of feelings: Evolutionary and neurobiological origins. Nat Rev Neurosci. 2013;14(2):143–52.
Daw ND, Niv Y, Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci. 2005;8(12):1704–11.
Dayan P, Niv Y. Reinforcement learning: the good, the bad and the ugly. Curr Opin Neurobiol. 2008;18(2):185–96.
Dean P, Redgrave P, Westby GW. Event or emergency? Two response systems in the mammalian superior colliculus. Trends in neurosciences. 1989;12(4):137–47.
Diana RA, Yonelinas AP, Ranganath C. Imaging recollection and familiarity in the medial temporal lobe: A three-component model. Trends in Cognitive Sciences. 2007;11(9):379–86.
Dickinson A, Balleine BW. (2002). The Role of Learning in the Operation of Motivational Systems. In Pashler, H. and Gallistel, R., editors, Stevens’ Handbook of Experimental Psychology, volume 3: Learning, Motivation and Emotion, pages 497–533. John Wiley & Sons, New York, 3rd edition.
Domenech P, Koechlin E. Executive control and decision-making in the prefrontal cortex. Current Opinion in Behavioral Sciences. 2015;1:101–6.
Donoso M, Collins AG, Koechlin E. (2014). Foundations of human reasoning in the prefrontal cortex. Science (New York, N.Y.), 344(6191):1481– 1486.
Dorfman HM, Gershman SJ. Controllability governs the balance between Pavlovian and instrumental action selection. Nat Commun. 2019;10(1):5826.
Fanselow MS, Dong H-W. Are the Dorsal and Ventral Hippocampus Functionally Distinct Structures? Neuron. 2015;65(1):7–19.
Fix J, Rougier NP, Alexandre F. A Dynamic Neural Field Approach to the Covert and Overt Deployment of Spatial Attention. Cognitive Computation. 2011;3:279–93.
Friston K. A Free Energy Principle for Biological Systems. Entropy (Basel Switzerland). 2012;14(11):2100–21.
Fuster J. The prefontal cortex. Anatomy, physiology and neurophysiology of the frontal lobe. New-York: Raven Press; 1989.
Genud-Gabai R, Klavir O, Paz R. Safety Signals in the Primate Amygdala. The Journal of Neuroscience. 2013;33(46):17986–94.
Graziano M. The organization of behavioral repertoire in motor cortex. Annu Rev Neurosci. 2006;29:105–34.
Gros C. Cognition and Emotion: Perspectives of a Closing Gap. Cognitive Computation. 2010;2(2):78–85.
Gruber AJ, McDonald RJ. (2012). Context, emotion, and the strategic pursuit of goals: Interactions among multiple brain systems controlling motivated behaviour. Frontiers in Behavioral Neuroscience, 6(50).
Guillery RW. (2005). Anatomical pathways that link perception and action, volume 149, pages 235–256. Elsevier.
Haber S, Fudge J, McFarland N. Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. The Journal of Neuroscience. 2000;20(6):2369–82.
Hassabis D, Kumaran D, Summerfield C, Botvinick MM. Neuroscience-Inspired Artificial Intelligence. Neuron. 2017;95:245–58.
Hazy TE, Frank MJ, O’Reilly RC. Banishing the homunculus: making working memory work. Neuroscience. 2006;139(1):105–18.
Hélie S, Ell SW, Ashby FG. Learning robust cortico-cortical associations with the basal ganglia: An integrative review. Cortex. 2015;64:123–35.
Hikosaka O, Nakamura K, Nakahara H. Basal Ganglia orient eyes to reward. J Neurophysiol. 2006;95(2):567–84.
Holroyd CB, Ribas-Fernandes JJF, Shahnazian D, Silvetti M, Verguts T. (2018). Human midcingulate cortex encodes distributed representations of task progress. Proceedings of the National Academy of Sciences, 115(25), 6398–6403.
Jensen O, Lisman JE. Hippocampal sequence-encoding driven by a cortical multi-item working memory buffer. Trends Neurosci. 2005;28(2):67–72.
Joel D, Niv Y, Ruppin E. Actor–critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw. 2002;15(4–6):535–47.
Kassab R, Alexandre F. (2015). Integration of exteroceptive and interoceptive information within the hippocampus: a computational study. Frontiers in Systems Neuroscience, 9(87).
Kassab R, Alexandre F. Pattern separation in the hippocampus: distinct circuits under different conditions. Brain Structure Function. 2018;223(6):2785–808.
Koechlin E. An evolutionary computational theory of prefrontal executive function in decision-making. Philos Trans R Soc Lond B Biol Sci. 2014;369(1655):20130474+.
Koechlin E, Ody C, Kouneiher F. The Architecture of Cognitive Control in the Human Prefrontal Cortex. Science. 2003;302(5648):1181–5.
Kouneiher F, Charron S, Koechlin E. Motivation and cognitive control in the human prefrontal cortex. Nat Neurosci. 2009;12(7):939–45.
Kringelbach ML. The human orbitofrontal cortex: linking reward to hedonic experience. Nat Rev Neurosci. 2005;6(9):691–702.
Laborit H. (1976). Eloge de la fuite. Folio Essais.
Le Pelley ME. The role of associative history in models of associative learning: a selective review and a hybrid model. The Quarterly Journal of Experimental Psychology. 2004;57(3):193–243.
LeDoux J. The amygdala. Curr Biol. 2007;17(20):R868–74.
Lee C, Rohrer WH, Sparks DL. Population coding of saccadic eye movements by neurons in the superior colliculus. Nature. 1988;332(6162):357–60.
Mannella F, Gurney K, Baldassarre G. (2013). The nucleus accumbens as a nexus between values and goals in goal-directed behavior: a review and a new hypothesis. Frontiers in behavioral neuroscience, 7.
Manto M, Bower JM, Conforto ABB, Delgado-Garcia JM, da Guarda SNFN, Gerwig M, Habas C, Hagura N, Ivry RB, Mariën P, Molinari M, Naito E, Nowak DA, Taib OB, Pelisson N, Tesche D, Tilikete CD, C., and Timmann D. Consensus paper: roles of the cerebellum in motor control–the diversity of ideas on cerebellar involvement in movement. Cerebellum. 2012;11(2):457–87.
McClelland JL, McNaughton BL, O’Reilly RC. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review. 1995;102(3):419–57.
McLean PD. (1973). An Evolutionary Approach to the Investigation of PsychoneuroEndocrine Functions. In K. Lissák (Éd.), Hormones and Brain Function (p. 379389). Springer US.
McHaffie JG, Stanford TR, Stein BE, Coizet V, Redgrave P. Subcortical loops through the basal ganglia. Trends in neurosciences. 2005;28(8):401–7.
Middleton FA, Strick PL. Basal ganglia and cerebellar loops: motor and cognitive circuits. Brain Res Rev. 2000;31(2):236–50.
Miller K, Ludvig E, Pezzulo G, Shenhav A. (2018). Realigning Models of Habitual and Goal-Directed Decision-Making. In Bornstein, A., Morris, R. W., and Shenhav, A., editors, Goal-Directed Decision Making. Academic Press.
Milner A, Goodale M. The visual brain in action. USA: Oxford University Press; 1995.
Moustafa AA, Gilbertson MW, Orr SP, Herzallah MM, Servatius RJ, Myers CE. A model of amygdala–hippocampal–prefrontal interaction in fear conditioning and extinction in animals. Brain Cogn. 2013;81(1):29–43.
Murray EA, Wise SP, Graham KS. (2016). The Evolution of Memory Systems: Ancestors, Anatomy, and Adaptations. Oxford University Press.
Niv Y. (2007). Cost, Benefit, Tonic, Phasic: What Do Response Rates Tell Us about Dopamine and Motivation. Annals of the New York Academy of Sci-ences, 1104(1):357–376.
Norman J. Two visual systems and two theories of perception: An attempt to reconcile the constructivist and ecological approaches. Behav Brain Sci. 2002;25:73–144.
Nowak LG, Bullier J. The timing of information transfer in the visual system. In: Rockland KS, Kaas JH, Peters A, editors. Extrastriate Cortex in Primates, volume 12 of Cerebral Cortex. New York: Plenum; 1997. pp. 205–41.
O’Regan JK, Noë A. (2001). A sensorimotor account of vision and visual consciousness. The Behavioral and brain sciences, 24(5).
O’Reilly RC. Biologically based computational models of high-level cognition. Science. 2006;314(5796):91–4.
O’Reilly RC. The What and How of prefrontal cortical organization. Trends Neurosci. 2010;33(8):355–61.
O’Reilly RC, Hazy TE, Mollick J, Mackie P, Herd S. (2014). Goal-Driven Cognition in the Brain: A Computational Framework. arXiv:1404.7591.
O’Reilly RC, Herd SA, Pauli WM. (2010). Computational models of cognitive control. Current Opinion in Neurobiology, 20(2).
O’Reilly RC, Rudy JW. Conjunctive Representations in Learning and Memory: Principles of Cortical and Hippocampal Function. Psychol Rev. 2001;108(2):311–45.
Oudeyer P-Y, Kaplan F, Hafner V. Intrinsic motivation systems for autonomous mental development. IEEE Trans Evol Comput. 2007;11(2):265–86.
Packard MG, Knowlton BJ. Learning and memory functions of the basal ganglia. Annu Rev Neurosci. 2002;25(1):563–93.
Padoa-Schioppa C. Range-Adapting Representation of Economic Value in the Orbitofrontal Cortex. The Journal of Neuroscience. 2009;29(44):14004–14.
Parent A, Hazrati LN. Functional anatomy of the basal ganglia. I. The cortico-basal ganglia-thalamo-cortical loop. Brain Res Brain Res Rev. 1995;20(1):91–127.
Parvizi J. Corticocentric myopia: old bias in new cognitive sciences. Trends in Cognitive Sciences. 2009;13(8):354–9.
Pauli WM, O’Reilly RC. Attentional control of associative learning–a possible role of the central cholinergic system. Brain Res. 2008;1202:43–53.
Paz R, Paré D. Physiological basis for emotional modulation of memory circuits by the amygdala. Curr Opin Neurobiol. 2013;23(3):381–6.
Pennartz CM, Ito R, Verschure PFMJ, Battaglia FP, Robbins TW. The hippocampal-striatal axis in learning, prediction and goal-directed behavior. Trends in neurosciences. 2011;34(10):548–59.
Penner MR, Mizumori SJY. Neural systems analysis of decision making during goal-directed navigation. Progress in neurobiology. 2012;96(1):96–135.
Pessoa L, Adolphs R. Emotion processing and the amygdala: from a ’low road’ to ’many roads’ of evaluating biological significance. Nat Rev Neurosci. 2010;11(11):773–83.
Pezzulo G, Castelfranchi C. Thinking as the control of imagination: a conceptual framework for goal-directed systems. Psychol Res. 2009;73(4):559–77.
Pezzulo G, Van der Meer MAA, Lansink CS, Pennartz CMA. Internally generated sequences in learning and executing goal-directed behavior. Trends in Cognitive Sciences. 2014;18(12):647–57.
Pezzulo G, Cisek P. Navigating the Affordance Landscape: Feed-back Control as a Process Model of Behavior and Cognition. Trends in Cognitive Sciences. 2016;20(6):414–24.
Pfeifer R, Bongard J, Grand S. (2007). How the body shapes the way we think: a new view of intelligence. Bradford Books. MIT Press.
Redgrave P. Basal Ganglia Scholarpedia. 2007;2(6):1825.
Redgrave P, Prescott TJ, Gurney K. The basal ganglia: A vertebrate solution to the selection problem? Neuroscience. 1999;89(4):1009–23.
Rizzolatti G, Riggio L, Dascola I, Umiltá C. Reorienting attention across the horizontal and vertical meridians: Evidence in favor of a premotor theory of attention. Neuropsychologia. 1987;25(1A):3140.
Rousselet G, Thorpe S, Fabre-Thorpe M. How parallel is visual processing in the ventral path? Trends in Cognitive Sciences. 2004;8(8):363–70.
Rushworth MFS, Walton ME, Kennerley SW, Bannerman DM. Action sets and decisions in the medial frontal cortex. Trends in Cognitive Sciences. 2004;8(9):410–7.
Sakai K. Task Set and Prefrontal Cortex. Annu Rev Neurosci. 2008;31(1):219–45.
Shenhav A, Botvinick MM, Cohen JD. The Expected Value of Control: An Integrative Theory of Anterior Cingulate Cortex Function. Neuron. 2013;79(2):217–40.
Sherman S. The thalamus is more than just a relay. Curr Opin Neurobiol. 2007;17:417–22.
Sommer M, Wurtz R. What the brain stem tells the frontal cortex. I. Oculomotor signals sent from superior colliculus to frontal eye field via mediodorsal thalamus. J Neurophysiol. 2004;91(3):1381–402.
Sutton RS. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin. 1991;2(4):160–3.
Stachenfeld KL, Botvinick M, Gershman SJ. (2014). Design Principles of the Hippocampal Cognitive Map. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems 27, pages 2528–2536. Curran Associates, Inc.
Stachenfeld KL, Botvinick MM, Gershman SJ. The hippocampus as a predictive map. Nat Neurosci. 2017;20(11):1643–53.
Striedter GF. Evolution of the hippocampus in reptiles and birds. J Comp Neurol. 2016;524(3):496–517.
Swanson LW, Petrovich GD. What is the amygdala? Trends Neurosci. 1998;21(8):323–31.
Taouali W, Goffart L, Alexandre F, Rougier NP. A parsimonious computational model of visual target position encoding in the superior colliculus. Biol Cybern. 2015;109(4):(8):549–59.
Tulving E. (1972). Episodic and semantic memory. Organization of Memory. Academic Press.
Ungerleider L, Mishkin M. (1982). Two cortical visual systems., pages 549–586. MIT Press.
Uylings HBM, Groenewegen HJ, Kolb B. Do rats have a prefrontal cortex? Behav Brain Res. 2003;146(1–2):3–17.
Vaidya AR, Badre D. (2020). Neural Systems for Memory-based Value Judgment and Decision-making. Journal of Cognitive Neuroscience, 1–28.
Varela F, Thompson E, Rosch E. The embodied mind: Cognitive science and human experience. Cambridge: MIT Press; 1991.
Verschure PFMJ, Pennartz CM, Pezzulo G. The why, what, where, when and how of goal-directed choice: neuronal and computational principles. Philosophical Transactions of the Royal Society B: Biological Sciences. 2014;369(1655):20130483+.
Voorn P, Vanderschuren LJ, Groenewegen HJ, Robbins TW, Pennartz CM. Putting a spin on the dorsal-ventral divide of the striatum. Trends Neurosci. 2004;27(8):468–74.
Yin HH, Ostlund SB, Balleine BW. Reward-guided learning beyond dopamine in the nucleus accumbens: the integrative functions of cortico-basal ganglia networks. Eur J Neurosci. 2008;28(8):1437–48.

Download PDF

Journal Publication

published 15 Feb, 2021

Read the published version in Brain Informatics →

Version 1

posted

You are reading this latest preprint version

A global framework for a systemic view of brain modeling

Status:

Journal Publication

Version 1

Abstract

Figures

1 Introduction

2 Main Ingredients

2.1 Three worlds to conciliate

2.2 Pavlovian and instrumental conditioning

2.3 Four questions to be addressed

3 Increasingly Complex Behaviors

3.1 Elementary association to responses

What - The amygdala

Why - The lateral hypothalamus

Where - The superior colliculus

How - The cerebellum

3.2 Selection of response in CBG loops

4 Associated Mnemonic Processes

4.1 Keeping a memory of past episodes

4.2 Building abstract categories

4.3 Building flexible sequences

5 Discussion

Declarations

Ethics approval and consent to participate :

Consent for publication :

Availability of data and materials :

Competing interests :

Funding :

Authors' contributions :

Acknowledgements :

References

Status:

Journal Publication

Version 1