ADHD is characterized by a difficulty to act in a goal-directed manner. While most environments require a sequence of actions for goal attainment, ADHD was never studied in the context of value-based sequence learning. Here, we made use of current advancements in hierarchical reinforcement-learning algorithms to track the internal value and choice policy of individuals with ADHD performing a three-stage sequence learning task. Specifically, 54 participants (28 ADHD, 26 controls) completed a value-based reinforcement-learning task that allowed us to estimate internal action values for each trial and stage using computational modeling. We found attenuated sensitivity to action values in ADHD compared to controls, both in choice and reaction-time variability estimates. Remarkably, this was found only for first stage actions (i.e., initiatory actions), while for action performed just before outcome delivery the two groups were strikingly indistinguishable. These results suggest a difficulty in following value estimation for initiatory actions in ADHD.
Article
Exploring the steps of learning: Computational modeling of initiatory-actions among individuals with Attention-Deficit/Hyperactivity Disorder
https://doi.org/10.21203/rs.3.rs-3132891/v1
This work is licensed under a CC BY 4.0 License
Journal Publication
published 08 Jan, 2024
You are reading this latest preprint version
Attention-Deficit/Hyperactivity Disorder (ADHD) is a neurodevelopmental disorder that is characterized by inattention, impulsivity, and hyperactivity 1,2. Approximately 10% of school-aged children are affected by ADHD 3 with the majority experiencing significant symptoms throughout adolescence and into adulthood 4. Individuals with ADHD frequently struggle with the completion of tasks that require the execution of multiple actions (e.g., doing homework) 5–7. Despite the known difficulties experienced by those with ADHD in completing multi-action tasks in natural environments, there has been no prior investigation of ADHD in the context of value-based sequence learning. In this study, we employed reinforcement learning modeling to examine the extent to which ADHD is associated with reduced value-based learning for initial actions (i.e., actions at the start of the sequence) using a multi-stage reinforcement learning task.
Reinforcement learning studies have demonstrated that human individuals estimate the value of potential actions based on their outcome history 8–10. This estimation process enables individuals to act persistently, even when the overall goal or outcome is distant 11,12. Specifically, reinforcement learning algorithms suggest that when making a sequence of actions that lead to a certain outcome, the value of the observed outcome is mentally "backpropagated" to preceding actions 13. For example, when making a cup of coffee, multiple actions must be completed before achieving the end goal (i.e., consumption of the beverage). These actions should be associated with the value of the reached outcome, thereby increasing the likelihood of repeating/avoiding the sequence based on the value of the outcome. The backpropagation of values to actions is considered a "glue" that binds the actions into a sequence 14. However, during the backpropagation process value is also discounted across the sequence, meaning that initiatory actions (i.e., actions performed at the start of a sequence) receive smaller value updates than terminal actions (i.e., actions performed immediately before outcome delivery10,15. The challenge is that without making correct value-based choices early in the sequence, the end goal cannot be reached, resulting in overall less consistent and more arbitrary value-based choices. Nevertheless, since initiatory actions are distant in time from the outcome, they are also assigned smaller value updates 10. Many studies have already suggested that the processes described by reinforcement learning equations, do indeed predict human behavior 15–17, and can explain neural activity 9,18–22. However, no study that we know of to date has used this approach to investigate whether value-based process can explain the difficulty of participants with ADHD to engage and maintain in a multi-stage task in order to reach a desired goal.
Here, we hypothesize that difficulties acting according to obtained rewards in ADHD will be more pronounced in initiatory actions (action that are performed in the start of a sequence) compared with terminal actions (that are performed at the end of e sequence). Specifically, we suggest that the process of updating the value of initiatory actions is less efficient in ADHD, leading to noisier choices at the start of a sequence. Difficulty holding and following distinct action values at the beginning of a sequence can led to less consistent choices and difficulty achieving the desired goal, which is highly relevant to ADHD symptomatology. For example, if the initiatory action of a sequence for making a beverage is not clearly defined, other competing actions that are closer to the outcome and assigned with similar value might take precedence. Alternatively, if the value of the initiatory action is internally well defined, it will lead more frequently to the selection of the best action, generating a more consistent and uninterrupted sequence of actions.
Although no studies have examined action-outcome associations in ADHD for initiatory actions, several findings in the literature support this hypothesis. First, ADHD individuals exhibit marked differences in reward-related neural processing 23–30. Second, ADHD individuals are known to show steeper temporal discounting of future values 31–33. Furthermore, ADHD shows inconsistent behavior when performing sustained attention/executive function tasks without immediate reward or feedback 34–40. Finally, when sustained attention/executive function tasks are performed under conditions of immediate reward (where every action can be considered a terminal action, immediately leading to reward), ADHD shows marked improvements 41. Therefore, there is reason to believe that individuals with ADHD have a reduced ability to assign and follow internal action value, mostly for actions that are at the beginning of a sequence and distance in time from outcome delivery. However, no empirical investigation has directly addressed this question.
The current study therefore examined to what extent individuals diagnosed with ADHD were able to update and act upon action-outcome associations across different stages of a sequential decision-making task. Specifically, we hypothesized that the value-base behavior of ADHD participants will be more inconsistent (noisier choices, and higher RT variability) compared with their healthy controls (HC) peers mostly for initiatory, but not terminal actions. We further hypothesize that the reason for such group differences will be due to reduced action value updating for outcome-distant stages in ADHD. To examine this question empirically, ADHD/HC participants performed a sequential reinforcement learning task, where each trial they were asked to make three choices to gain rewards. We then used computational modeling to assess the internal value each individual might be assigning for each action in each stage. We replicated previous findings showing that for all participants and across both groups, choice accuracy and RT variability estimates improved when the latent action values were more differentiated (larger internal value difference between the two available actions). We then examined the ability of ADHD individuals to act according to the reward history across stages in the task.
Participants. 54 Undergraduate Tel-Aviv University students completed the study in return for course credits or monetary compensation. (For demographic including gender, age, and IQ see Table 1; all participants were white caucasian; 53 came from the Jewish sector and 1 from the Arab - Christian sector). All participants reported normal or corrected vision and signed informed consent before participating in the study. Participants using ADHD medications were asked to abstain for at least 24 hours before the lab session. The study was approved by Tel-Aviv University IRB ethics committee. Participants ADHD assessment included a semi-structured interview using the Diagnostic Interview for ADHD in adults (DIVA) 42. Diva assessment conducted by a well-trained student under the supervision of a licensed clinical psychologist expert (See ADHD diagnosis in SI). Exclusion criteria included any neurological or psychiatric history past or current, history of significant head injury, and current drug or alcohol abuse.
Reinforcement learning task. Participants performed a multiple-stages reinforcement learning task (Fig. 1) that included three sequential stages. The participants were told that their task is to find a puppy that is hiding behind different chests, and that they need to make three choices each trial to try and locate the puppy. They were explained that the task includes two houses, each house contains two rooms, and each room contains two chests. accordingly, at the first stage of each trial participants chose one of two “houses” (denoted by a cartoon houses stimuli see Fig. 1B), each leading them deterministically to a second-stage state where a pair of “doors” (cartoon stimuli, see Fig. 1B) was offered. Choosing one of the doors led participants deterministically to a third-stage state where a pair of “chests” (cartoon stimuli; see Fig. 1B) was offered. After making all three choices, the participants were presented with the outcomes which was a cartoon image of a happy puppy, or an empty pet pillow with no puppy (Fig. 1A). Specifically, each chest delivered a reward (finding the puppy) on a drifting probability (Fig. 1C), and participants were asked to learn which house, door and chest are most likely to lead to reward throughout the task. (For more information see reinforcement learning task in SI).
Choice accuracy and difficulty. For the purposes of analysis, we wanted to code choice accuracy and choice difficulty for each trial and stage. In each stage, participants were asked to make a choice between two actions. A choice was coded as ‘correct’ if the participant selected in a certain stage the action that took part in the sequence with the highest true expected value. Choice difficulty was defined in a similar way, according to the true expected values of the sequence associated with each offered option. (For more information see choice accuracy and difficulty in SI).
Using ex-Gaussian distribution to estimate RT variability. An ex-Gaussian distribution is a non-theoretical statistical model that was extensively used in cognitive 43 and ADHD research 44–46 to describe the distribution of reaction times. It combines two different distributions: the Gaussian (normal) distribution, which represents the more frequent and typically quicker reaction time, and an exponential distribution, which represents the tail of slower reaction times. Specifically, τ is the parameter that describes the exponential portion of the distribution. By using the ex-Gaussian distribution, researchers can more accurately model the full range of reaction times, including the long tail of slower responses. (For more information see using ex-Gaussian distribution to estimate RT variability in SI).
Data Treatment. The first trial in each block, trials with implausibly quick reaction times (< 200ms), or exceptionally slow reaction times (> 4000ms) were omitted (5.15% of all trials). Participants with more than 25% excluded trials (one participant, HC group) or higher than 5% no response rate (none), were further excluded from analysis. Due to technical issues, we found after data collection that some participants did not receive choice feedback on a minority of trials. Those trials were omitted (69 trials overall), and participants with more than 5% of omitted trials were excluded (4 participants, 3 participants from HC group and 1 participant from ADHD group).
Data and code availability. Data, regression analysis, and computational modeling analysis codes are publicly available on https://osf.io/ws47k/.
Computational Modeling. To estimate the internal values participants might assign to each available action, we fitted a well-established hierarchical reinforcement learning model to participants’ choice behavior 16,47,48. First, we defined a prediction error for each state and action according to its expected value and the observed rewards as follows:
$$\left[1\right] P{E}_{1}={V}_{choice2}-{V}_{choice1}$$
$$\left[2\right] P{E}_{2}={V}_{choice3}-{V}_{choice2}$$
$$\left[3\right] P{E}_{3}=R-{V}_{choice3}$$
Where Vchoice1, Vchoice2, and Vchoice3 are the latent state-action values for the chosen actions in each state (in the reinforcement learning literature also known as Q values). 𝑅 refers to the outcome observed following the terminal state so that R∈{0,1}. We followed the convention in model-free reinforcement learning modeling 10 where each PE is backpropagated and assigned to all past actions, under a discounting factor (i.e., free parameter). Therefore, the first action is updated using PE1, PE2 and PE3. The second action is updated using PE2 and PE3. The third action is updated using PE3. Thus, the value of every action was updated as follows:
$$\left[4\right] {V}_{choice1}={V}_{choice1}+{\alpha }_{1}\left[{V}_{choice2}-{V}_{choice1}\right]+{\alpha }_{1}\lambda \left[{V}_{choice3}-{V}_{choice2}\right]+{\alpha }_{1}{\lambda }^{2}[R-{V}_{choice3}]$$
$$\left[5\right] {V}_{choice2}={V}_{choice2}+{\alpha }_{2}\left[{V}_{choice3}-{V}_{choice2}\right]+{\alpha }_{2}\lambda [R-{V}_{choice3}]$$
$$\left[6\right] {V}_{choice3}={V}_{choice3}+{\alpha }_{3}[R-{V}_{choice3}]$$
Where alpha 𝛼1, 𝛼2 and 𝛼3 are the learning rates for Stage 1, 2, and 3 respectively 𝜆 is an eligibility trace factor referring to the extent of prediction error back propagation. Specifically, for 𝜆 of 1 the agent will learn and update whole sequences, while for 𝜆=0 the agent will learn individual state action values for each stage. Finally, choices were predicted using a softmax decision policy (see Eq. 7 to 9):
were 𝛽1, 𝛽2 and 𝛽3 are the decision temperature for Stage 1, 2 and 3 respectively. Moreover, to ensure that the model can be used adequately to estimate latent parameters, we performed a parameter recovery analysis which indicated excellent parameter recovery with no indication of bias (see Figure S1, and model posterior sampling and parameter recovery analysis in SI).
Theory-independent analysis.
Accuracy rates - group difference analysis. To examine the difference in performance between ADHD and HC participants on accuracy rates, we performed a hierarchical Bayesian regression analysis (for more details see Bayesian parameters estimation in SI) where we predicted choice accuracy (0/1 for choosing the offer with the lower/higher true expected value, respectively), using choice difficulty (absolute difference between the true expected values of the two offers, see Methods), stage (1, 2 or 3) and group (HC vs. ADHD). When examining the interaction between group and choice difficulty, we found that the effect of choice difficulty on accuracy rates was smaller for ADHD compared to HC only for first stage choices. Specifically, a change of 0.5 in the difference between the expected value of the two offered actions in a first stage was associated with an increase of 22.6% in accuracy rates for the HC group, and only 1.8% for the ADHD group [posterior median of group effect in the first stage = -2.00 CI89% between − 3.35 to -0.71, pd = 99.30%]. Strikingly, group differences vanished completely, and even changed direction when examining second and third stage choice. Specifically, we found a tendency toward a group x choice difficulty interaction in the opposite direction to that of the first stage when examining the second stage [group effect posterior median = 1.45 CI89% between − 0.38 to 3.22, pd = 89.45%] and the third stage [median = 1.67, CI89% between − 0.19 to 3.58, pd = 92.15%]. Therefore, we conclude that ADHD participants were not sensitive to differences in choice difficulty as their HC group when making first stage choices. However, in second and third choices, we found no evidence for group differences in accuracy rates (see Fig. 2A).
Reaction-time variability - group difference analysis. We next examined the effect of choice difficulty (absolute difference between the true expected values of the two offers, see Methods), group (HC vs. ADHD) and stage (1, 2 or 3) on RT variability using a hierarchical Bayesian regression analysis (for more details see Bayesian parameters estimation in SI). We found that the effect of choice difficulty on RT variability was smaller for ADHD compared to HC, in first stage choices (see Fig. 2B). Specifically, a change of 0.5 in the difference between the expected value of the two offered actions was associated with an increase of 58ms in the tail of the RT distribution for the HC group. However, the ADHD showed only 11ms change for the same amount of difference [posterior median for group effect on τ estimates on the first stage = + 0.64 CI89% between 0.08 to 1.19, pd = 96.50%]. Strikingly, for the second and third stage this effect reversed with a slightly larger effect of choice difficulty on RT variability estimates for ADHD compared to HC. Specifically, the group effect on τ estimates was negative for both the second stage [posterior median = -0.42 CI89% between − 1.07 to 0.24, pd = 84.45%] and third stage [posterior median = -0.65 CI89% between − 1.32 to 0.01, pd = 94.25%; For coefficients report see Table S1].
Computational reinforcement learning modeling.
The previous analysis showed that ADHD individuals were less sensitive to the difference in the true expected values of the different choices in the first stage using both choice-accuracy and RT variability estimates. However, true expected values are quantified according to the pre-defined experimental design. Since the current task is a reinforcement learning task, the reward history is not only a function of the experimental design, but also a function of the participants choices. To track and gain a better estimate of the individual internal action value for each trial and stage (also referred as Q-values in reinforcement learning), we used a well-established hierarchical reinforcement learning modeling (for details see methods and computational reinforcement learning modeling in SI). We fitted the model to each group separately using hierarchical Bayesian parameter estimation (see model fitting in SI). We further performed a nested model comparison and found that a saturated seven parameter model had the best fit to the empirical data (for details see nested model comparisons in SI).
Accuracy rates estimated using internal action values. We used the expected value of the Q-value posterior distribution obtained for each action in our model Bayesian sampling as an estimate for the internal action values for each action (see Methods). To estimate choice accuracy, we first defined an accurate choice, as one that reflects a selection in the higher internal value action in each stage (defined as 1 if the action with the higher internal value was taken, and 0 otherwise). We repeated the same regression analyses reported in the previous section, with absolute difference in internal value (i.e., |ΔQ|, this notion is used due to the common notion of “Q-values” for internal values in the reinforcement learning literature) Group (ADHD vs HC), stage (1, 2 and 3) and their interactions as predictors for choice accuracy.
First, we found that higher Q-value differences were coupled with higher choice accuracy across stages and groups [posterior median = 5.39 CI89% between 4.19 to 6.50, pd ~ 100%]. Specifically, a change of 0.5 in the difference of two offered actions values was associated with an increase of 51% in accuracy rates. This finding shows that the model was able to capture well some estimate for values that indeed predicts participants’ behavior well. Second, we further found a substantial group x |ΔQ| interaction for Stage 1, so that ADHD exhibited lower sensitivity to Q-values difference in terms of choice accuracy compared with HC [posterior median of group effect in the first stage = -2.19 CI89% between − 3.75 to -0.63, pd = 98.65%; Fig. 3A]. Strikingly, similarly to former results, group differences vanished completely and even changed direction when examining second and third stage choice. Specifically, we found a tendency toward a group x |ΔQ| interaction in the opposite direction to that of the first stage when examining the second stage [posterior median = 3.11 CI89% between 1.21 to 5.13, pd = 99.30%] and the third stage [posterior median = 2.40 CI89% between 0.40 to 4.34, pd = 97%]. Overall, these findings resonated with the finding that ADHD had lower decision noise parameter at Stage 1 compared to their peers.
Reaction-time variability as a function of internal action values. We sampled an hierarchical Bayesian regression were absolute difference in Q-values (i.e., |ΔQ|) RT variability RT variability RT variability Group (ADHD vs HC), stage (1, 2 or 3) and their paired interaction as predictors for the τ parameter in an ex-Gaussian distribution fitted to trial-by-trial reaction-time estimates (see Methods). First, we found that higher Q-value differences were coupled with lower τ values across all stages [posterior median = -0.54 CI89% between − 0.67 to -0.41, pd ~ 100%]. This finding supports the validity of our |ΔQ|, since longer RT tails are expected for decisions with similar value 49–51. Furthermore, our model estimates of internal value action values were based only on choices, without introducing RTs to the model during parameter estimation. Thus, finding this expected association |ΔQ| serves as an important sanity check for our modeling approach.
Second, we found a substantial Group x |ΔQ| interaction for Stage 1, so that ADHD exhibited lower sensitivity to Q-values difference in terms of 𝜏 estimates [posterior median = 0.23 CI89% between 0.07 to 0.38, pd = 99%; Fig. 3]. A similar group x |ΔQ| interaction was found in Stage 2 [posterior median = -0.41 CI89% between − 0.63 to -0.18, pd = 99.9%]. However, for the third stage this effect reversed with a slightly smaller effect of Q-value difference on RT variability estimates for ADHD compared to HC [posterior median = 0.20 CI89% between − 0.42 to 0.03, pd = 91.2%; For coefficients report see Table S3].
The current study examined the ability of ADHD individuals to assign and act upon internal action values across stages of a sequential decision-making task. Participants with/without ADHD made decisions across a three-step deterministic tree task. First, we used reinforcement learning modeling to gain estimates of the latent and internal action values for each individual, trial and step. We replicated previous findings showing that across both groups, estimates of choice accuracy and RT variability improved when the two available actions were well differentiated in terms of their latent values. Second, we found that ADHD individuals were less sensitive to initiatory actions value differences across both choice accuracy and RT variability estimates compared with their HC. Specifically, a larger difference between the values of the two available first stage actions had smaller benefit on ADHD choices and RTs compared to HC. Remarkably, these group differences diminished and disappeared when making terminal third stage actions, to a point where the two groups were remarkably indistinguishable. Our results therefore suggested a striking attenuated sensitivity to value actions in ADHD, only in initiatory but not terminal task action.
The current finding extends previous literature in several ways. First, previous studies have already demonstrated that individuals with ADHD are uniquely characterized by inconsistent decision-making, as apparent in both choice and reaction times estimates 52,53. These studies further suggested that these behavioral inconsistencies in ADHD tend to wash out under conditions of immediate reward. For example, ADHD have been found to exhibit marked behavioral inconsistency in reaction times (i.e., increased RT variability) when performing sustained attention/executive function tasks (for review see 50,53,54). In a meta-analysis report, RT variability was further found to show the highest effect-size for group differences, thus the mechanisms that contribute to this ADHD phenotype can be of much importance to how we understand ADHD. The sustained attention/executive functions tasks that reported increased RT variability in ADHD were mostly performed in the absence of any immediate reward or feedback. Remarkably, when immediate feedback or reward was delivered, ADHD demonstrated a significant improvement in RT variability estimates. Improved RT variability with delivery of immediate reward was found in a variety of paradigms including go/no-go 54, choice reaction time task 55,56, simple reaction time 57, and time discrimination task 58,59. However, these studies are characterized almost exclusively with choice-feedback coupling, such that outcome is given after a single choice and use between-block manipulations (blocks with/without feedback). Here, we show both increased RT variability in ADHD compared to HC, and the absence of group difference in RT variability estimates within a single trial. This is important since it refutes claims that a general motivation induced across a block is the sole reason for improved performance in ADHD under conditions of immediate reward/feedback. Instead, our finding points towards a more refined value-action mechanism that leads to increased inconsistency in ADHD.
Second, we find that only for first stage choices, ADHD showed attenuated sensitivity to action values in terms of RT variability. Therefore, one explanation for both increased inconsistency in RTs in ADHD 50,53,54, and previous findings regarding the improvement in this estimate under immediate reward 54–57,59 might be an attenuated ability to assign and follow internal action-values. Specifically, when performing a task in the absence of immediate reward/feedback, the individual is required to hold in mind the overall value of task completion. This might lead to noisier action values in ADHD compared with HC, which in turn is reflected in noisier reaction times. Therefore, individuals need to hold an overall value of task completion to keep interest and guide the selection of accurate actions. Since individuals with ADHD have known deficits in reward neural processing, and tend to discount future outcomes 59, they may encounter particular difficulty in assigning value to current actions based on a distant goal. Indeed, reinforcement learning studies have confirmed both in simulations and empirical studies that a reduced ability to assign value to actions lead to prolonged and more inconsistent RTs, a finding that we also replicated in the current study across groups. Therefore, we speculate that the process of internally assigning value to actions, can be a major contributing mechanism to increased RT variability in ADHD.
Why do ADHD individuals exhibit noisier choices during first stage choices? The current study cannot offer a clear explanation, yet we will discuss a few explanations. First, it might be that ADHD individuals are more explorative in their behavior 6,25,52,60,61. Specifically, a few reinforcement learning studies suggested that less value-directed behavior in ADHD might be the results of directed decision to explore, rather than reduced ability to follow action values. For example, Dubois et al., 62 examined explore/exploit decision making between children, early and late adolescence. They found that participants with high scores in self-report symptoms of ADHD made more exploratory decisions. Next, Dubois & Hauser 63 further linked between exploration and the impulsivity component within ADHD, in an explore/exploit decision making task. Addicott et al., 64 also examined explore/exploit decision making using the 6-armed bandit task where on each trial, six bandit options were depicted on a computer screen and participants selected one and received a reward in various probabilities. They compared ADHD and HC groups of adults on explore/exploit decisions that were modeled using reinforcement learning algorithms. They found that ADHD participants made more exploratory decisions (i.e., chose options without the highest expected reward value) and earned fewer points than HC in all three study days. Frank et al., 25 examined go/no-go decision making between ADHD and HC groups of young adults. They additionally used computational models in order to predict participants' behavior. They found that ADHD participants were more likely to display inconsistent choices from trial to trial, and that increased inconsistent behavior was correlated with increased variability in reaction times.
Another explanation of the reduced tendency to act according to value actions in first stage choice in ADHD might be the temporal distance from the outcome. That is, in the current task we cannot rule out the fact that it is not the actual sequence or the fact that first stage choices were initiatory actions in a sequence of three decisions. Rather it might be that the temporal delay per-se is the reason for the reduced ability to follow action values. We note that a previous study suggested that counter intuitively, ADHD individuals improve in their ability to learn from feedback when action outcome epoch is prolonged 65. However, dedicated studies should carefully disentangle temporal delay from task stages to account for this explanation. Finally, it might be the complexity of the task that was difficult for the ADHD participants during first stage choices. Studies have shown reduced working memory in ADHD 36,40,66–70, which in turn might reduce their ability to hold in mind and follow the task structure during first stage choices.
Finally, a few limitations should be mentioned. First, our design does not allow us to differentiate whether ADHD individuals adopted different speed-accuracy response criteria in first stage choices. Further studies can address this by using dedicated models (e.g., evidence accumulation). Second, our study cannot disentangle whether temporal distance from the outcome, unique properties of first stage actions, or both are behind the effect. Finally, our study was performed in a college student population, and further studies are needed to demonstrate the generalization to the children population.
In conclusion, individuals with ADHD exhibit reduced choice accuracy and higher RT variability compared to HC individuals for first-stage, initiatory action, but not for the third stage, terminal action. Our computational modelling uncovered that individuals with and without ADHD assign and act upon internal action values in sequential decision-making tasks, and as shown here there are differences between the groups in the ability to do so. This highlights the need to address and deeper understand the internal action values as an important mechanism in ADHD.
Acknowledgement
We would like to acknowledge the undergraduate students Eli Greisman and Tali Bogachek for their assistance in data collection. The study was supported by the Minducate Center for the Science of Learning (Sagol School of Neuroscience, Tel Aviv University), the Israel Ministry of Science and Technology, and the Israeli Science Foundation (Grant 2536/20 awarded to NS). Both authors contributed equally to all stages of the study.
Conflict of interest
The authors declare that there were no conflicts of interest with respect to the authorship or the publication of this article.
- American Psychiatric Association, ed. Diagnostic and Statistical Manual of Mental Disorders: DSM-5. 5th ed. American Psychiatric Association; 2013.
- Faraone SV, Banaschewski T, Coghill D, et al. The World Federation of ADHD International Consensus Statement: 208 Evidence-based conclusions about the disorder. Neurosci Biobehav Rev. 2021;128:789-818. doi:10.1016/j.neubiorev.2021.01.022
- Thomas R, Sanders S, Doust J, Beller E, Glasziou P. Prevalence of Attention-Deficit/Hyperactivity Disorder: A Systematic Review and Meta-analysis. Pediatrics. 2015;135(4):e994-e1001. doi:10.1542/peds.2014-3482
- Lara C, Fayyad J, de Graaf R, et al. Childhood predictors of adult attention-deficit/hyperactivity disorder: results from the World Health Organization World Mental Health Survey Initiative. Biol Psychiatry. 2009;65(1):46-54. doi:10.1016/j.biopsych.2008.10.005
- Langberg JM, Smith ZR, Dvorsky MR, et al. Factor structure and predictive validity of a homework motivation measure for use with middle school students with attention-deficit/hyperactivity disorder (ADHD). Sch Psychol Q. 2018;33(3):390-398. doi:10.1037/spq0000219
- Seesjärvi E, Puhakka J, Aronen ET, et al. Quantifying ADHD Symptoms in Open-Ended Everyday Life Contexts With a New Virtual Reality Task. J Atten Disord. 2022;26(11):1394-1411. doi:10.1177/10870547211044214
- Sibley MH, Graziano PA, Ortiz M, Rodriguez L, Coxe S. Academic impairment among high school students with ADHD: The role of motivation and goal-directed executive functions. J Sch Psychol. 2019;77:67-76. doi:10.1016/j.jsp.2019.10.005
- Dayan P, Niv Y. Reinforcement learning: The Good, The Bad and The Ugly. Curr Opin Neurobiol. 2008;18(2):185-196. doi:10.1016/j.conb.2008.08.003
- Schultz W, Dayan P, Montague PR. A Neural Substrate of Prediction and Reward. Science. 1997;275(5306):1593-1599. doi:10.1126/science.275.5306.1593
- Sutton RS, Barto A. Reinforcement Learning: An Introduction. Second edition. The MIT Press; 2018.
- Lehmann MP, Xu HA, Liakoni V, Herzog MH, Gerstner W, Preuschoff K. One-shot learning and behavioral eligibility traces in sequential decision making. Gold JI, Kahnt T, Gläscher J, Jepma M, Ponte Costa R, eds. eLife. 2019;8:e47463. doi:10.7554/eLife.47463
- Singh SP, Sutton RS. Reinforcement learning with replacing eligibility traces. Mach Learn. 1996;22(1):123-158. doi:10.1007/BF00114726
- Botvinick MM, Niv Y, Barto AC. Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition. 2009;113(3):262-280. doi:10.1016/j.cognition.2008.08.011
- Schultz W, Dickinson A. Neuronal Coding of Prediction Errors. Annu Rev Neurosci. 2000;23(1):473-500. doi:10.1146/annurev.neuro.23.1.473
- Dezfouli A, Balleine BW. Actions, action sequences and habits: evidence that goal-directed and habitual action control are hierarchically organized. PLoS Comput Biol. 2013;9(12):e1003364. doi:10.1371/journal.pcbi.1003364
- Botvinick MM. Hierarchical reinforcement learning and decision making. Curr Opin Neurobiol. 2012;22(6):956-962. doi:10.1016/j.conb.2012.05.008
- Dezfouli A, Balleine BW. Habits, action sequences and reinforcement learning. Eur J Neurosci. 2012;35(7):1036-1051. doi:10.1111/j.1460-9568.2012.08050.x
- Fiorillo CD, Tobler PN, Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science. 2003;299(5614):1898-1902. doi:10.1126/science.1077349
- Houk JC, Davis JL, Beiser DG, eds. A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement. In: Models of Information Processing in the Basal Ganglia. The MIT Press; 1994. doi:10.7551/mitpress/4708.003.0020
- Montague PR, Hyman SE, Cohen JD. Computational roles for dopamine in behavioural control. Nature. 2004;431(7010):760-767. doi:10.1038/nature03015
- Schultz W. Dopamine reward prediction-error signalling: a two-component response. Nat Rev Neurosci. 2016;17(3):183-195. doi:10.1038/nrn.2015.26
- Tobler PN, Fiorillo CD, Schultz W. Adaptive Coding of Reward Value by Dopamine Neurons. Science. 2005;307(5715):1642-1645. doi:10.1126/science.1105370
- Biederman J, Faraone SV. Current concepts on the neurobiology of Attention-Deficit/Hyperactivity Disorder. J Atten Disord. 2002;6(1_suppl):7-16. doi:10.1177/070674370200601S03
- Del Campo N, Fryer TD, Hong YT, et al. A positron emission tomography study of nigro-striatal dopaminergic mechanisms underlying attention: implications for ADHD and its treatment. Brain. 2013;136(11):3252-3270. doi:10.1093/brain/awt263
- Frank MJ, Santamaria A, O’Reilly RC, Willcutt E. Testing Computational Models of Dopamine and Noradrenaline Dysfunction in Attention Deficit/Hyperactivity Disorder. Neuropsychopharmacology. 2007;32(7):1583-1599. doi:10.1038/sj.npp.1301278
- Luman M, Tripp G, Scheres A. Identifying the neurobiology of altered reinforcement sensitivity in ADHD: A review and research agenda. Neurosci Biobehav Rev. 2010;34(5):744-754. doi:10.1016/j.neubiorev.2009.11.021
- Plichta MM, Scheres A. Ventral–striatal responsiveness during reward anticipation in ADHD and its relation to trait impulsivity in the healthy population: A meta-analytic review of the fMRI literature. Neurosci Biobehav Rev. 2014;38:125-134. doi:10.1016/j.neubiorev.2013.07.012
- Sagvolden T, Johansen EB, Aase H, Russell VA. a dynamic developmental theory of attention-deficit/hyperactivity disorder (adhd) predominantly hyperactive/impulsive and combined subtypes. Behav Brain Sci. 2005;28(3):397-419. doi:10.1017/S0140525X05000075
- Solanto MV, Arnsten AFT, Castellanos FX. Stimulant Drugs and ADHD: Basic and Clinical Neuroscience. Oxford University Press; 2001.
- Volkow ND, Wang GJ, Newcorn JH, et al. Motivation deficit in ADHD is associated with dysfunction of the dopamine reward pathway. Mol Psychiatry. 2011;16(11):1147-1154. doi:10.1038/mp.2010.97
- Demurie E, Roeyers H, Baeyens D, Sonuga-Barke E. Common alterations in sensitivity to type but not amount of reward in ADHD and autism spectrum disorders. J Child Psychol Psychiatry. 2011;52(11):1164-1173. doi:10.1111/j.1469-7610.2010.02374.x
- Luman M, Oosterlaan J, Sergeant JA. The impact of reinforcement contingencies on AD/HD: A review and theoretical appraisal. Clin Psychol Rev. 2005;25(2):183-213. doi:10.1016/j.cpr.2004.11.001
- Sonuga-Barke EJS, Taylor E, Sembi S, Smith J. Hyperactivity and delay aversion--I. The effect of delay on choice. J Child Psychol Psychiatry. 1992;33(2):387-398. doi:10.1111/j.1469-7610.1992.tb00874.x
- Hart H, Radua J, Nakao T, Mataix-Cols D, Rubia K. Meta-analysis of Functional Magnetic Resonance Imaging Studies of Inhibition and Attention in Attention-deficit/Hyperactivity Disorder: Exploring Task-Specific, Stimulant Medication, and Age Effects. JAMA Psychiatry. 2013;70(2):185-198. doi:10.1001/jamapsychiatry.2013.277
- Krieger V, Amador-Campos JA, Guàrdia-Olmos J. Executive functions, Personality traits and ADHD symptoms in adolescents: A mediation analysis. PLoS ONE. 2020;15(5):e0232470. doi:10.1371/journal.pone.0232470
- Martinussen R, Hayden J, Hogg-johnson S, Tannock R. A Meta-Analysis of Working Memory Impairments in Children With Attention-Deficit/Hyperactivity Disorder. J Am Acad Child Adolesc Psychiatry. 2005;44(4):377-384. doi:10.1097/01.chi.0000153228.72591.73
- Mullane JC, Corkum PV. The Relationship Between Working Memory, Inhibition, and Performance on the Wisconsin Card Sorting Test in Children With and Without ADHD. Published 2007. Accessed October 6, 2022. https://journals.sagepub.com/doi/10.1177/0734282906297627
- Pedersen A, Ohrmann P. Impaired Behavioral Inhibition in Implicit Sequence Learning in Adult ADHD. J Atten Disord. 2018;22(3):250-260. doi:10.1177/1087054712464392
- Pievsky MA, McGrath RE. The Neurocognitive Profile of Attention-Deficit/Hyperactivity Disorder: A Review of Meta-Analyses. Arch Clin Neuropsychol. 2018;33(2):143-157. doi:10.1093/arclin/acx055
- Willcutt EG, Doyle AE, Nigg JT, Faraone SV, Pennington BF. Validity of the Executive Function Theory of Attention-Deficit/Hyperactivity Disorder: A Meta-Analytic Review. Biol Psychiatry. 2005;57(11):1336-1346. doi:10.1016/j.biopsych.2005.02.006
- Hulsbosch AK, De Meyer H, Beckers T, et al. Systematic Review: Attention-Deficit/Hyperactivity Disorder and Instrumental Learning. J Am Acad Child Adolesc Psychiatry. 2021;60(11):1367-1381. doi:10.1016/j.jaac.2021.03.009
- Kooij JJS, Francken MH, Bron TI. Diagnostic interview for ADHD in Adults 2.0 (DIVA 2.0). Published online 2010.
- Matzke D, Wagenmakers EJ. Psychological interpretation of the ex-Gaussian and shifted Wald parameters: A diffusion model analysis. Psychon Bull Rev. 2009;16(5):798-817. doi:10.3758/PBR.16.5.798
- Alderson RM, Rapport MD, Kofler MJ. Attention-Deficit/Hyperactivity Disorder and Behavioral Inhibition: A Meta-Analytic Review of the Stop-signal Paradigm. J Abnorm Child Psychol. 2007;35(5):745-758. doi:10.1007/s10802-007-9131-6
- Castellanos FX, Sonuga-Barke EJS, Scheres A, Di Martino A, Hyde C, Walters JR. Varieties of Attention-Deficit/Hyperactivity Disorder-Related Intra-Individual Variability. Biol Psychiatry. 2005;57(11):1416-1423. doi:10.1016/j.biopsych.2004.12.005
- Epstein JN, Langberg JM, Rosen PJ, et al. Evidence for higher reaction time variability for children with ADHD on a range of cognitive tasks including reward and event rate manipulations. Neuropsychology. 2011;25(4):427. doi:10.1037/a0022155
- Dezfouli A, Lingawi NW, Balleine BW. Habits as action sequences: hierarchical action control and changes in outcome value. Philos Trans R Soc B Biol Sci. 2014;369(1655):20130482. doi:10.1098/rstb.2013.0482
- Solway A, Diuk C, Córdova N, et al. Optimal Behavioral Hierarchy. PLOS Comput Biol. 2014;10(8):e1003779. doi:10.1371/journal.pcbi.1003779
- Peters J, D’Esposito M. The drift diffusion model as the choice rule in inter-temporal and risky choice: A case study in medial orbitofrontal cortex lesion patients and controls. PLoS Comput Biol. 2020;16(4):e1007615. doi:10.1371/journal.pcbi.1007615
- Shahar N, Teodorescu AR, Karmon-Presser A, Anholt GE, Meiran N. Memory for Action Rules and Reaction Time Variability in Attention-Deficit/Hyperactivity Disorder. Biol Psychiatry Cogn Neurosci Neuroimaging. 2016;1(2):132-140. doi:10.1016/j.bpsc.2016.01.003
- Shahar N, Hauser TU, Moutoussis M, et al. Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling. Gershman SJ, ed. PLOS Comput Biol. 2019;15(2):e1006803. doi:10.1371/journal.pcbi.1006803
- Hauser TU, Iannaccone R, Ball J, et al. Role of the Medial Prefrontal Cortex in Impaired Decision Making in Juvenile Attention-Deficit/Hyperactivity Disorder. JAMA Psychiatry. 2014;71(10):1165-1173. doi:10.1001/jamapsychiatry.2014.1093
- Kofler MJ, Rapport MD, Sarver DE, et al. Reaction time variability in ADHD: A meta-analytic review of 319 studies. Clin Psychol Rev. 2013;33(6):795-811. doi:10.1016/j.cpr.2013.06.001
- Adamo N, Hodsoll J, Asherson P, Buitelaar JK, Kuntsi J. Ex-Gaussian, Frequency and Reward Analyses Reveal Specificity of Reaction Time Fluctuations to ADHD and Not Autism Traits. J Abnorm Child Psychol. 2019;47(3):557-567. doi:10.1007/s10802-018-0457-z
- Tye C, Johnson KA, Kelly SP, et al. Response time variability under slow and fast-incentive conditions in children with ASD, ADHD and ASD+ADHD. J Child Psychol Psychiatry. 2016;57(12):1414-1423. doi:10.1111/jcpp.12608
- Vainieri I, Adamo N, Michelini G, Kitsune V, Asherson P, Kuntsi J. Attention regulation in women with ADHD and women with bipolar disorder: An ex-Gaussian approach. Psychiatry Res. 2020;285:112729. doi:10.1016/j.psychres.2019.112729
- Gopin CB, Berwid O, Marks DJ, Mlodnicka A, Halperin JM. ADHD Preschoolers With and Without ODD: Do They Act Differently Depending On Degree of Task Engagement/Reward? J Atten Disord. 2013;17(7):608-619. doi:10.1177/1087054711432140
- Marx I, Höpcke C, Berger C, Wandschneider R, Herpertz SC. The Impact of Financial Reward Contingencies on Cognitive Function Profiles in Adult ADHD. PLOS ONE. 2013;8(6):e67002. doi:10.1371/journal.pone.0067002
- Marx I, Hacker T, Yu X, Cortese S, Sonuga-Barke E. ADHD and the Choice of Small Immediate Over Larger Delayed Rewards: A Comparative Meta-Analysis of Performance on Simple Choice-Delay and Temporal Discounting Paradigms. J Atten Disord. 2021;25(2):171-187. doi:10.1177/1087054718772138
- Dubois M, Bowler A, Moses-Payne M, Habicht J, Steinbeis N, Hauser T. Tabula-Rasa Exploration Decreases during Youth and Is Linked to ADHD Symptoms. Neuroscience; 2020. doi:10.1101/2020.06.11.146019
- Hauser TU, Fiore VG, Moutoussis M, Dolan RJ. Computational Psychiatry of ADHD: Neural Gain Impairments across Marrian Levels of Analysis. Trends Neurosci. 2016;39(2):63-73. doi:10.1016/j.tins.2015.12.009
- Dubois M, Bowler A, Moses-Payne ME, et al. Exploration heuristics decrease during youth. Cogn Affect Behav Neurosci. 2022;22(5):969-983. doi:10.3758/s13415-022-01009-9
- Dubois M, Hauser TU. Value-free random exploration is linked to impulsivity. Nat Commun. 2022;13(1). doi:10.1038/s41467-022-31918-9
- Addicott MA, Pearson JM, Schechter JC, Sapyta JJ, Weiss MD, Kollins SH. Attention-deficit/hyperactivity disorder and the explore/exploit trade-off. Neuropsychopharmacology. 2021;46(3):614-621. doi:10.1038/s41386-020-00881-8
- Gabay Y, Shahbari-Khateb E, Mendelsohn A. Feedback Timing Modulates Probabilistic Learning in Adults with ADHD. Sci Rep. 2018;8:15524. doi:10.1038/s41598-018-33551-3
- Dovis S, Van der Oord S, Wiers RW, Prins PJM. What Part of Working Memory is not Working in ADHD? Short-Term Memory, the Central Executive and Effects of Reinforcement. J Abnorm Child Psychol. 2013;41(6):901-917. doi:10.1007/s10802-013-9729-9
- Ehlis AC, Bähne CG, Jacob CP, Herrmann MJ, Fallgatter AJ. Reduced lateral prefrontal activation in adult patients with attention-deficit/hyperactivity disorder (ADHD) during a working memory task: a functional near-infrared spectroscopy (fNIRS) study. J Psychiatr Res. 2008;42(13):1060-1067. doi:10.1016/j.jpsychires.2007.11.011
- Kofler MJ, Rapport MD, Bolden J, Sarver DE, Raiker JS, Alderson RM. Working memory deficits and social problems in children with ADHD. J Abnorm Child Psychol. 2011;39(6):805-817. doi:10.1007/s10802-011-9492-8
- Massat I, Slama H, Kavec M, et al. Working Memory-Related Functional Brain Patterns in Never Medicated Children with ADHD. PLOS ONE. 2012;7(11):e49392. doi:10.1371/journal.pone.0049392
- Rapport MD, Alderson RM, Kofler MJ, Sarver DE, Bolden J, Sims V. Working memory deficits in boys with attention-deficit/hyperactivity disorder (ADHD): the contribution of central executive and subsystem processes. J Abnorm Child Psychol. 2008;36(6):825-837. doi:10.1007/s10802-008-9215-y
- Zohar AH, Konfortes H. Diagnosing ADHD in Israeli Adults: The Psychometric Properties of the Adult ADHD Self Report Scale (ASRS) in Hebrew. Isr J Psychiatry Relat Sci. 2010;47(4):308-315.
- Ward MF, Wender PH, Reimherr FW. The Wender Utah Rating Scale: an aid in the retrospective diagnosis of childhood attention deficit hyperactivity disorder. Am J Psychiatry. 1993;150(6):885-890. doi:10.1176/ajp.150.6.885
- Foa EB, Kozak MJ, Salkovskis PM, Coles ME, Amir N. The validation of a new obsessive–compulsive disorder scale: The Obsessive–Compulsive Inventory. Psychol Assess. 1998;10:206-214. doi:10.1037/1040-3590.10.3.206
- Spielberger CD. State-Trait Anxiety Inventory for Adults. Published online 1983. doi:10.1037/t06496-000
- Beck AT, Steer RA, Brown G. Beck Depression Inventory–II. Published online 1996. doi:10.1037/t00742-000
- Baron-Cohen S, Wheelwright S, Skinner R, Martin J, Clubley E. The autism-spectrum quotient (AQ): evidence from Asperger syndrome/high-functioning autism, males and females, scientists and mathematicians. J Autism Dev Disord. 2001;31(1):5-17. doi:10.1023/a:1005653411471
- Raven J, Raven JC, Court JH. Manual for Raven’s Progressive Matrices and Vocabulary Scales. Pearson; 1998.
- Akam T, Costa R, Dayan P. Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task. PLOS Comput Biol. 2015;11(12):e1004648. doi:10.1371/journal.pcbi.1004648
- Daw ND, O’Doherty JP, Dayan P, Seymour B, Dolan RJ. Cortical substrates for exploratory decisions in humans. Nature. 2006;441(7095):876-879. doi:10.1038/nature04766
- Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ. Model-based influences on humans’ choices and striatal prediction errors. Neuron. 2011;69(6):1204-1215. doi:10.1016/j.neuron.2011.02.027
- Moran R, Dayan P, Dolan RJ. Human subjects exploit a cognitive map for credit assignment. Proc Natl Acad Sci. 2021;118(4):e2016884118. doi:10.1073/pnas.2016884118
Table 1. Characteristics of ADHD and Healthy control Groups
|
|
Healthy control |
ADHD |
p-value |
|
N |
26 (female: 20) |
28 (female: 15) |
|
|
Age |
19-34 (SD=4.50) |
20-32 (SD=2.72) |
.38 |
|
ASRS score |
38.2 (SD=10.5) |
61.7 (SD=6.6) |
< .001 |
|
WURS score |
20.0 (SD=18.5) |
44.4 (SD=13.1) |
< .001 |
|
OCI score |
40.6 (SD=28.1) |
32.3 (SD=23.6) |
.25 |
|
STAI score |
41.6 (SD=9.6) |
45.4 (SD=9.9) |
.16 |
|
BDI score |
9.1 (SD=9.7) |
11.3 (SD=7.2) |
.35 |
|
AQ score |
15.9 (SD=5.6) |
16.9 (SD=5.8) |
.44 |
|
Raven score |
12.2 (SD=3.4) |
12.5 (SD=3.7) |
.76 |
|
Note. ASRS - Adult ADHD Self Report Scale 71, WURS - Wender Utah Rating Scale for the Attention Deficit Hyperactivity 72, OCI - Obsessive-Compulsive Inventory 73, STAI - State Trait Anxiety Inventory 74, BDI - Beck Depression Inventory 75, AQ - Autism Spectrum Quotient 76, Raven - Raven Progressive Matrices 77. |
|||
The authors have declared there is NO conflict of interest to disclose
- SI.docx
Supporting information