RESEARCH DOSSIER
Reinforcement Learning and Model-Based Control
Abstract. Reinforcement learning explains how experienced consequences shape choice; model-based control explains how learned structure of environmental transitions supports prospective evaluation and adaptation. This fundamental research monograph examines the history of instrumental contingencies and cognitive maps, the formal distinction among rewards, values, policies, and transition models, temporal-difference learning, hierarchical actions, and planning under cognitive cost. It critically synthesizes devaluation experiments, human two-step tasks, behavioral and neural prediction-error evidence, manipulations of working memory and stress, alternative interpretations arising from task misunderstanding, and evidence on successor representations and computational replay through 2025. A source-based critical narrative synthesis identifies convergence in flexible use of learned relational knowledge alongside unresolved problems of latent-state representation, model identification, ecological generalization, and causal neural implementation. It distinguishes psychological evidence from engineered reinforcement-learning analogies and makes no claim of original experimentation, systematic-review protocol, or clinical validation.
Keywords: reinforcement learning; model-based control; model-free learning; action values; policy learning; instrumental conditioning; goal-directed action; habit formation; reward prediction error; state prediction error; two-step task; cognitive maps; successor representation; planning; metacontrol; exploration; volatility; computational modeling; working memory; Artificial Era
Introduction: Learning to Choose and Learning to Plan
Reinforcement learning describes how repeated interaction with an environment changes action preferences, expectations, and policies as consequences are experienced. Model-based control concerns the additional capacity to use representations of environmental transitions and outcomes to evaluate actions prospectively. The scientific object is their relation in the organization of adaptive choice: when an agent should rely on cached estimates, when it should simulate consequences, how it learns the relevant state structure, and how researchers can distinguish competing mechanisms from behavior. The framework connects instrumental psychology, cognitive maps, neural prediction errors, normative decision theory, and contemporary computational models. Its observable domain includes acquisition, reversal, outcome devaluation, sequential choice, and transfer to novel contingencies Balleine & Dickinson (1998) Daw et al. (2011) Kahn & Daw (2025).
The central question is not simply whether behavior is habitual or deliberative. One must identify which quantities were learned, which transitions were represented, whether incentive values were updated, what computational resources were available at choice, and what information the experiment actually made accessible. A fast response can be model-based if a transition map has been compiled into a simple policy; a slow response can reflect uncertainty or indecision without explicit planning. Likewise, immediate reward sensitivity does not identify a value-updating algorithm. Psychological explanation therefore requires linking task manipulations to discriminating predictions and evaluating alternative models rather than assigning an intuitive label to a single action pattern Dolan & Dayan (2013) Akam et al. (2015) Miller et al. (2019).
This monograph is a source-based critical narrative synthesis of foundational experiments, computational theories, human choice studies, neuroimaging findings, cognitive-resource manipulations, and recent work on predictive representations. It contains no original participant data, formal systematic-review protocol, or pooled meta-analysis. The argument develops from instrumental behavior to Markov representations, contrasts action-value learning with transition-based planning, examines their interaction with memory and cognitive effort, and closes with measurement validity and the Artificial Era. The evidential standard treats a computational fit as a constrained explanation of observations, neural correlates as informative associations, and experimental manipulations as stronger tests of causal processes only where their assumptions can be defended Collins & Frank (2012) Feher da Silva & Hare (2020) Karagoz et al. (2024).
Behaviorism, Cognitive Maps, and the History of the Problem
The historical roots of reinforcement learning encompass the psychology of learned consequences and a parallel tradition arguing that animals acquire relational knowledge about environments. Reward-dependent instrumental action supplies the central behavioral problem: actions alter the probability of valued outcomes, while prior experience changes which action is selected. Tolman's discussion of cognitive maps emphasized that learned spatial organization could support routes and adjustments whose content exceeds a single strengthened stimulus–response association. His 1948 article is a historical foundation for the question whether knowledge of transitions mediates action; its original maze observations should be understood in the methodological context of mid-twentieth-century animal research, not as a direct test of a modern algorithm Tolman (1948) Balleine & Dickinson (1998).
The Rescorla–Wagner account later formulated associative learning as the adjustment of predictions according to a discrepancy between expected and actual outcomes. This formal advance addressed cue competition in Pavlovian conditioning and supplied an influential prediction-error language. It did not in itself specify the full machinery of instrumental action selection, sequential state transitions, or prospective planning. The relationship between Pavlovian prediction and instrumental reinforcement learning is therefore one of mathematical and conceptual inheritance, with distinct behavioral observables and interventions. In instrumental settings, devaluation and contingency degradation became crucial ways to ask whether the subject represents the response–outcome relation and the current desirability of its consequence Rescorla & Wagner (1972) Balleine & Dickinson (1998).
Computational developments gave these psychological contrasts an explicit set of algorithms. Sutton's Dyna program demonstrated how a learning agent might alternate actual experience, learning an environmental model, and simulated experience generated from that model. The significance of this integration is conceptual as well as technical: planning can update cached action estimates, and learned models can accelerate action adaptation without requiring exhaustive online search every time an action is performed. Subsequent behavioral neuroscience adopted a contrast between prospective and retrospective valuation, often termed model-based and model-free reinforcement learning, as a testable working decomposition. It remains a family of hypotheses about computations and implementations rather than a proof that every mind contains exactly two independent controllers Sutton (1990) Daw et al. (2005) Dolan & Dayan (2013).
Goal-Directed Action, Devaluation, and Contingency
Instrumental goal-directedness is operationally investigated by altering an outcome's value after training and testing whether the associated action changes when direct new reinforcement is limited. If an action falls after the formerly desired outcome is devalued, this can indicate use of an action–outcome representation updated by present incentive value. Conversely, persistent responding can indicate reliance on earlier learned control or procedural associations. The logic is especially compelling when outcome identity, satiety or aversion procedures, and the absence of relearning opportunities are controlled. A single failure of devaluation sensitivity remains ambiguous if the subject does not encode the manipulation or if the test imposes unequal retrieval or expression demands Balleine & Dickinson (1998) Gillan et al. (2015).
Contingency degradation changes the causal effectiveness of an action by adding outcomes delivered independently of that action or altering the action–outcome relation. It tests a dimension distinguishable from reward frequency: outcomes can remain common while the action ceases to improve their probability. Successful adaptation therefore requires some representation of conditional outcome likelihoods rather than simple strengthening by temporal contiguity. Together, devaluation and degradation provide operational tests of prospective flexibility, although they do not uniquely determine whether the underlying representation is an explicit transition tree, a compact cached model, or another goal-sensitive learned controller. Their interpretation depends on the fidelity of the manipulations and the structure of the test Balleine & Dickinson (1998) Miller et al. (2019).
Goal-directed action and model-based reinforcement learning overlap but are not definitionally identical. In classical instrumental tasks, outcome identity and current desire are central, whereas sequential decision models usually specify an explicit state transition function and a reward function. A model can be used for a one-step goal computation, or a multistep rollout, or a successor-style prediction of future occupancy. Each predicts different transfer to a changed transition, reward, or policy. A robust psychological account therefore retains the behavioral concept of goal-directedness while distinguishing alternative computational routes to it. The opposite relation also requires care: habitual action may be cached value selection, stimulus–response association, or an automated sequence rather than one homogeneous mechanism Dezfouli & Balleine (2012) Miller et al. (2019) Kahn & Daw (2025).
Formal States, Actions, Transitions, and Rewards
A common formalization uses a Markov decision process with state set S, action set A, transition probabilities P(s' | s,a), reward function R(s,a,s'), discount factor gamma, and policy pi(a | s). The Markov assumption means the designated state contains the information necessary to predict the next transition and immediate reward given an action. Psychological experiments rarely guarantee that the participant internally uses the experimenter's state labels. A person might represent recent trial history, goals, rules, context, or perceived environmental structure as part of the state. The validity of a model consequently rests on the correspondence between formal variables and the learner's effective representation Daw et al. (2011) Akam et al. (2015).
The state-value function V under a policy is the expected discounted future reward when the current state is entered and subsequent actions follow that policy. An action-value function Q analogously conditions on an initial action. Model-based evaluation can use a Bellman recursion: Q(s,a) equals the expected immediate reward plus gamma times the future value of successor states. In a finite task with a learned transition model, changes to an outcome's value can propagate backward through the transition graph. This is the computational basis of a characteristic transfer advantage, but the prediction is conditional on the accuracy and accessibility of the state map and the agent's willingness to compute with it Sutton (1990) Gläscher et al. (2010).
The formal specification also clarifies why everyday uses of the word reward can be misleading. Experimenters define reward as an observable delivered outcome, such as monetary points or food, whereas a computational reward function represents how those consequences enter a model of optimization. Physiological need, satiation, incentives, social meaning, and instructions may alter experienced utility without changing the experimenter's nominal point count. A transition model might be correct while its incentive mapping is outdated, or the opposite. Learning a model and deploying it to act are distinct operations; errors in either stage can yield behavior that looks inflexible. Discounting, time cost, risk, and uncertainty further shape the relation between normative optimization and observed choice Niv et al. (2007) Dolan & Dayan (2013).
Model-Free Learning: Prediction Errors, Values, and Policies
Model-free temporal-difference learning updates predictions using local experience without requiring online evaluation of a learned transition tree. A standard state-value error is delta = r + gamma V(s') - V(s), with step-size alpha governing the next estimate. Action-value algorithms similarly update cached Q values, subject to their chosen on-policy or off-policy target. In a simple sequential task, repeated reward may strengthen a first-stage choice even if the transition leading to that reward was rare, because its cached estimate was reinforced through experienced trajectories. The resulting choice signature offers a hypothesis about retrospective valuation under conditions where the state labels and learning rule are well matched to the task Daw et al. (2011) Gläscher et al. (2010).
An action value is not itself a policy. Values express expected consequences in a specified decision process, whereas a policy specifies probabilities of actions conditional on information available to the agent. A softmax policy may convert Q differences into choice probabilities through an inverse-temperature parameter, while lapse rates, perseveration biases, and motor tendencies contribute further structure to observable behavior. An estimate that assigns high exploitation can reproduce deterministic choices whether the system used planned or cached values. Equating value learning with policy learning hides these distinctions and can make model comparison insensitive to the actual representation of action selection Collins & Frank (2012) Miller et al. (2019).
The link between temporal-difference error and dopamine became prominent through studies of reward-related neuronal responses, especially the analysis by Schultz, Dayan, and Montague. These results support an influential computational interpretation of certain phasic signals, yet neither every dopamine signal nor every striatal response is a unique assay of model-free learning. Dopamine also participates in motivational vigor and other computations; a theoretical account of average reward identifies the opportunity cost of time as relevant to response speed. Model-free algorithms are useful explanatory models, but identifying one in humans requires behavioral dissociation and carefully interpreted neural measurements Schultz et al. (1997) Niv et al. (2007) Daw et al. (2011).
Model-Based Planning: Transitions, Search, and Revaluation
A model-based controller learns or otherwise possesses a representation of how actions change the environment and which outcomes occur after those changes. Planning uses that representation to compare actions, often by expected return, before or during choice. A major empirical signature is sensitivity to changes in downstream reward or transition structure without an intervening complete history of direct reinforcement for the first action. Such transfer depends on an internal pathway from newly learned information to the choice. In practice planning can be approximated through limited-depth search, sampling, hierarchical structure, or cached results, so a behavioral model should specify which predictions are computed and when Sutton (1990) Daw et al. (2011) Karagoz et al. (2024).
Transition learning creates a distinct error signal when an observed successor state contradicts what the agent expected. Using a probabilistic transition representation, an update can compare the observed next state with its predicted probability; the relevant mismatch differs from a reward prediction error because the surprising event may contain no immediate incentive value. Gläscher and colleagues used human fMRI during sequential decisions to examine state prediction signals and reward-related signals. Their correlates in lateral prefrontal and parietal regions provide evidence for differentiated computational variables, while interpretation of blood-oxygen-level-dependent activity remains limited by temporal resolution and the dependence of regressors on behavioral model assumptions Gläscher et al. (2010).
Planning involves at least three interacting costs: collecting enough data to learn transitions, maintaining or retrieving those relations, and computing forward possibilities when a decision is needed. A deep rollout may be valuable in a small stable decision tree yet expensive in a volatile or branching environment. Even perfect transition knowledge can fail to yield optimal action if evaluation truncates search or if reward predictions are inaccurate. Model-based control is therefore both a knowledge problem and a resource-allocation problem. Its complexity creates the rationale for arbitration and computational shortcuts examined in later chapters; any claim about deliberation must locate the bottleneck in representation, calculation, or choice Daw et al. (2005) Kool et al. (2016) Jensen et al. (2024).
The Two-Step Task and Its Diagnostic Predictions
Sequential two-stage choice tasks were designed to separate reinforcement based on the last rewarded action from evaluation based on probabilistic state transitions. In the canonical arrangement, a first-stage choice commonly leads to one second-stage state and rarely to another; rewards associated with second-stage choices vary over trials. A model-free learner tends to repeat a recently rewarded first-stage choice regardless of whether the preceding transition was common or rare. A learner using transition probabilities instead recognizes that a rewarding outcome after a rare transition may favor the alternative first-stage action next time. The reward-by-transition interaction in first-stage stay choices consequently became a standard behavioral marker of model-based influence Daw et al. (2011) Akam et al. (2015).
The interpretation requires more than inspection of a fitted interaction coefficient. Participants must know or learn the transition structure; they must understand whether second-stage actions can change the reward probabilities; their response strategy must be stable enough for the assumed model to be meaningful. The task's trial sequence, instruction quality, reward volatility, and treatment of choice omissions all matter. Logistic stay regressions and computational model fits answer related but different questions: the former estimates statistical signatures of recent outcomes, while the latter assigns likelihood to a specified latent decision procedure. Neither procedure automatically identifies a mechanistic neural module Daw et al. (2011) Feher da Silva & Hare (2020).
The two-step task is particularly valuable when used as one component of triangulation. It permits investigators to impose controlled changes in the probability of reaching states and to fit competing strategies to the same choices, while reward history can be recorded precisely. However, the standard task often distributes small differences in expected value across many trials, making effortful planning yield only modest incremental benefit. That feature can produce limited model-based signatures even in participants able to plan. Modified tasks that create greater accuracy and monetary benefit from planning reveal an important dissociation between competence and strategic allocation of computation Kool et al. (2016) Kool et al. (2017).
Task Misunderstanding, Latent States, and Model Mimicry
An important methodological criticism is that simple hybrid model-free/model-based fits can misattribute behavior when the subject uses a different state representation than the experimenter assumes. Akam, Costa, and Dayan demonstrated analytically and by simulation that mechanisms depending on history, state classification, and transition learning can generate misleading signatures in the two-step task. An apparent residual model-free contribution may be produced by a richer controller that handles the task differently, while a weak model-based coefficient may coexist with complex planning. Researchers must therefore test alternative representational hypotheses rather than interpreting fitted weights as direct fractions of brain systems Akam et al. (2015).
Feher da Silva and Hare investigated whether richer task instructions alter how participants decide in the two-stage task. Their study found that better understanding led to choices primarily consistent with model-based inference and showed that inaccurate transition beliefs in model-based agents can generate apparently hybrid response patterns. This result bears on the interpretation of earlier mixed-strategy findings, rather than establishing that model-free learning never contributes to human instrumental choice. Different designs, subject populations, and outcome schedules may recruit different mixtures; the inference should follow the task and observed representational knowledge Feher da Silva & Hare (2020) Daw et al. (2011).
An empirical remedy is to elicit transition predictions and explicit understanding, estimate subjects' belief states, use diagnostic probe trials, manipulate transition statistics separately from rewards, and test recovery on simulated agents using the entire proposed model set. In latent-state tasks a participant may treat the environment as changing modes rather than updating every action value independently. Such an explanation predicts abrupt strategy changes, structured generalization, and sensitivity to cues signaling context. A responsible model comparison therefore reports both the richness of models tested and the specific alternatives it cannot discriminate, with simulation-based parameter recovery and out-of-sample choice prediction Akam et al. (2015) Karagoz et al. (2024) Kahn & Daw (2025).
Working Memory, Executive Load, and Learning Efficiency
Reinforcement-learning experiments can inadvertently measure working memory, explicit rule retrieval, or prospective attention in addition to incremental value updating. Collins and Frank found that human instrumental learning is shaped by working-memory capacity and set size, motivating hybrid accounts in which rapidly acquired short-lived representations interact with slower reinforcement learning. The same behavioral curve may be reproduced by different mixtures of memory and prediction-error learning unless the experiment manipulates set size, time delay, interference, or cue repetition to separate their characteristic limits. This distinction is important for age, cognitive capacity, and clinical comparisons Collins & Frank (2012).
Otto, Gershman, Markman, and Daw used an executive load manipulation in the context of sequential choice and reported reduced reliance on model-based contributions under increased cognitive demands. The finding provides experimental support for a computational resource cost of planning, yet load can also affect attention to transitions, retention of task instructions, or response control. Interpreting it as a selective intervention on a model-based module requires that auxiliary task performance and model identification assumptions be evaluated. Combining load effects with transfer, probe, and devaluation tests can distinguish a failure to compute from a failure to learn the relevant information Otto et al. (2013a) Daw et al. (2011).
An important consequence is that a small estimated planning weight cannot automatically be translated into low intelligence, poor goal-directed capacity, or a stable trait. It may describe an allocation decision in a task with little benefit to expensive prospective evaluation. Conversely, good task scores can arise from memorized policies, strategy instruction, or simple heuristics that have become effective in a stable environment. The most revealing studies contrast planning demand, model availability, and memory burden within the same individuals. They examine how the agent shifts strategies when the world changes, rather than ranking persons on one fitted coefficient alone Collins & Frank (2012) Kool et al. (2016) Kahn & Daw (2025).
Stress, Affect, and Control Allocation
Acute stress has been investigated as a moderator of sequential planning. Otto, Raio, Chiang, Phelps, and Daw reported that working-memory capacity moderated the relationship between acute stress and model-based performance in their experimental setting. This kind of result fits the hypothesis that prospective computation depends partly on resources that can be challenged by stress. The inference is bounded by the stress manipulation, the participants studied, and the assumptions connecting two-step behavior to latent strategies. It does not justify a universal claim that stress simply converts goal-directed decisions into habits Otto et al. (2013b) Otto et al. (2013a).
Affective states can also alter reward evaluation and attention to prediction errors, complicating a clean interpretation of planning coefficients. Eldar and Niv tested feedback between emotional state and evaluations of outcomes, offering evidence that mood and valuation can interact dynamically. Such findings suggest that observed changes in choices during emotional episodes can arise through altered utility signals as well as reduced calculation. To separate pathways, one needs measures of reward perception, physiological or self-reported affect, transition knowledge, and the time course of strategic switching. These are different latent variables and should be tested with manipulations that target each one Eldar & Niv (2015) Niv et al. (2007).
Stress and affect show the value of mechanistic restraint when moving toward clinical generalization. Resource scarcity, fatigue, and emotion can all covary with impulsive responding, but behavior in a point-based laboratory task may reveal only a narrow choice pattern. Generalization to everyday planning requires tasks with consequential goals, ecologically meaningful costs, and repeated measurement. A clinically interesting association is a premise for further study, not a diagnosis or direct treatment recommendation. Observational associations must be separated from the results of controlled experimental load and stress manipulations, especially when a theory claims a specific neural shift Gillan et al. (2016) Dolan & Dayan (2013).
Arbitration, Planning Cost, and the Expected Value of Computation
Dual-strategy theories raise an additional decision problem: when should an agent employ the costly strategy? Daw, Niv, and Dayan proposed an uncertainty-sensitive competition between prefrontal and striatal action-control systems. Their model treats the reliability of value estimates as relevant to arbitration, providing a principled explanation for why different control policies might dominate in different circumstances. The proposal predicts a dynamic allocation of control, although its identification with particular neural territories is a model-level hypothesis that requires separate empirical testing. The brain could implement several interacting approximations rather than a single switch Daw et al. (2005).
Kool, Cushman, and Gershman showed that conventional two-step structures can underreward planning relative to its cognitive cost and developed task modifications to improve the measurable consequences of using a model. In subsequent experiments Kool, Gershman, and Cushman found that people increased model-based choice when the accuracy benefit of planning was greater and especially when accurate performance had greater reward value. Reward magnification without an accuracy gain did not have the same effect. These data support cost–benefit arbitration over a simple view that model-based control is a fixed dispositional parameter Kool et al. (2016) Kool et al. (2017).
The expected value of computation framework generalizes the point. A planning operation has value when the expected improvement in decisions exceeds its time and cognitive cost. The agent must estimate the probability that additional consideration will change the favored action, how consequential the choice is, and whether the world model is likely to be accurate. This explains why effort may rise with task stakes but decline when further planning cannot meaningfully improve choice. A measured reaction-time increase is consistent with such allocation only when paired with evidence that the added time actually improves forward-sensitive decisions, because time can also index confusion, conflict, or lapses Kool et al. (2017) Jensen et al. (2024).
Volatility, Reversal, Exploration, and Structural Change
Learning in volatile settings requires distinguishing noise in outcomes from lasting changes in the reward-generating process. A surprising loss in a stable stochastic environment may carry limited information, while a comparable loss following a genuine reversal may warrant rapid updating. Models using adaptive learning rates, latent-state inference, or structured hypotheses predict different ways to respond to change. In sequential tasks, transition volatility and reward volatility must be manipulated separately: the agent may retain a correct transition map while reward values drift, or maintain accurate reward estimates while action-to-state mappings reverse Akam et al. (2015) Kahn & Daw (2025).
Exploration introduces further ambiguity. An agent may choose a seemingly inferior action to learn whether its rewards changed, to resolve uncertainty about a transition, or to diversify experience. The resulting choice can resemble random response noise under a simple softmax model. Distinguishing directed exploration from random exploration requires conditions that alter information value independently of expected immediate payoff. A model-based controller may explicitly plan to gather information, but an efficient exploration heuristic may appear equally flexible over a restricted experimental domain. This is a fertile setting for combining model recovery with carefully selected diagnostic probes Daw et al. (2005) Kool et al. (2016).
Reversal learning provides a stringent transfer test when diagnostic events reveal which dimension of the environment changed. If reward desirability changes, a controller with a correct transition model should often update upstream choices without relearning the structure. If transitions change, a previously cached successor representation can become misleading even when downstream rewards remain constant. Kahn and Daw's 2025 experiments directly examined adjustment between detailed model-based planning and temporally abstract prediction when environmental predictability changed. Their results argue for a wider family of adaptive predictive strategies than the traditional binary model-based/model-free classification Kahn & Daw (2025) Karagoz et al. (2024).
Neural Prediction Errors and the Limits of Localization
Computational neuroscience seeks neural signals that change with prediction errors, expected values, transitions, and planning demands. Dopamine-neuron recordings and subsequent human imaging encouraged the study of reward-prediction errors as candidate teaching signals, but measurement modalities differ profoundly. Electrophysiology samples firing or related local activity, whereas fMRI measures hemodynamic responses convolved over time. A regression between a model-derived variable and a brain signal is meaningful evidence for the variable's neural correlates; it is not a direct observation of the entire learning algorithm or a proof of the neural site's necessity for particular behavior. The temporal, spatial, and inferential level must remain explicit Schultz et al. (1997) Gläscher et al. (2010).
Gläscher and collaborators contrasted state prediction errors and reward prediction errors in a multistep human task. They reported state-prediction-related activity in lateral prefrontal and intraparietal regions and reward-prediction-related correlates in ventral striatum. Daw and colleagues later found that ventral striatal prediction-error responses incorporated model-based influences in proportions related to participants' choices. These results jointly complicate a strict localization picture in which a ventral striatal signal uniquely belongs to a model-free controller and prospective computation operates in a sealed separate region. The neural implementation may share training signals or communicate across representations Gläscher et al. (2010) Daw et al. (2011).
An inference about neural mechanism should distinguish correlational decoding, perturbation, and model-specific prediction. Signals can be statistically related to value because value correlates with reward history, attention, confidence, or effort. Selective perturbation offers a stronger route to causal claims but frequently affects multiple computations at once. Disentangling these alternatives requires experimental designs with independently varying transition surprise, reward surprise, action selection, and planning demand. It also requires neural analyses that evaluate confounds and uncertainty rather than choosing whichever regressor best fits a single voxel cluster. The question is how coordinated circuits implement a control computation across time Dolan & Dayan (2013) Jensen et al. (2024).
Prefrontal–Striatal Coordination and Action Systems
Anatomical accounts often associate outcome-sensitive and flexible action selection with networks involving prefrontal and associative striatal regions, while relatively automatic sensorimotor action patterns involve partly different corticostriatal loops. The architecture is better represented as coordinated, partially overlapping circuits than as one area for thinking and another for habit. Balleine and Dickinson's instrumental framework contributed to the analysis of action–outcome sensitivity and its cortical substrates, while later computational neuroscience asked how the resulting representations support multistep decision-making. The shift from behavioral categories to neural mechanisms requires species-specific evidence, homologous circuit considerations, and task correspondence Balleine & Dickinson (1998) Daw et al. (2005) Dolan & Dayan (2013).
Human imaging studies with sequential decision tasks show signals related to both transition knowledge and expected outcome. Their apparent regional distinctions depend on assumptions about what the subject learned and which computational variables covary across trials. Neural data cannot compensate for an incorrectly specified behavioral model: if the agent represents a task differently from the investigator, the estimated regressors may misidentify cognitive functions. Studies that measure subjects' structural beliefs and test independent behavioral transfer can better constrain neural interpretation. Analyses should report both competing regressors and the sensitivity of results to alternative task representations Daw et al. (2011) Akam et al. (2015) Feher da Silva & Hare (2020).
Motivational activation further complicates the mapping. A change in response vigor may arise from the opportunity cost of time and average reinforcement rather than a stronger preference for a particular action. Niv and colleagues' normative formulation links reward rate, response latency, and dopaminergic motivation and thereby separates action selection from the energy or urgency of responding. When a participant moves quickly toward a reward, researchers must not assume that urgency measures planning or habitual valuation without accounting for vigor. A complete account relates the content of choice, its timing, and resource expenditure while preserving distinctions among the relevant computational variables Niv et al. (2007) Schultz et al. (1997).
Cognitive Maps, Relational Structure, and Prospective Inference
A cognitive map is a structured representation of relationships among states or experiences that can support flexible inference. In spatial navigation it may relate locations and paths; in abstract decisions it may encode common causes, transition contingencies, or higher-order relations among choice options. The term connects Tolman's historical argument to modern model-based control without requiring literal spatial imagery. Its psychological importance is the capacity to recompute behavior from changed information about a structured environment, including possible paths not directly rewarded during learning Tolman (1948) Karagoz et al. (2024).
Karagoz, Reagh, and Kool coupled a sequential decision paradigm with behavioral representational similarity analysis to investigate how relationships between options are learned. They reported that stronger higher-order relational encoding predicted greater planning and better task performance, while simple visual co-occurrence was not the corresponding predictor. They also found more strongly encoded higher-order relations among objects encountered in high-reward contexts. This study offers evidence that motivation can influence the organization of a map, and that model availability is a graded learned achievement. Its representational findings remain dependent on how similarity responses and planning behavior index internal structure Karagoz et al. (2024).
A map is useful only insofar as its relational content matches the domain in which choices are made. A learner may store concrete transition probabilities, infer hidden task states, or abstract recurring roles across superficially different contexts. A rich map can enable rapid revaluation and generalization but requires memory and may be brittle when erroneous assumptions are held with confidence. Research must therefore contrast learning of the map with use of the map. Structural probes conducted before choice can test representation, while transfer or reversal probes test whether knowledge can guide decisions under changed contingencies Akam et al. (2015) Kahn & Daw (2025) Karagoz et al. (2024).
Successor Representations and Temporally Abstract Prediction
The successor representation describes expected future state occupancy under a policy, discounted over time. Instead of storing only immediate transition probabilities or directly caching an action's reward value, it stores predictive structure that can be combined with a reward function to evaluate future consequences. This creates a computational middle ground: if reward weights change while successor predictions remain valid, the agent can revalue quickly; if the policy or transition graph changes, old predictions may need revision. Such asymmetries provide experimental tests that discriminate successor-based valuation from complete online transition search and simple model-free value caching Kahn & Daw (2025).
Kahn and Daw constructed a dynamic multistep task designed to separate temporally abstract successor-based inference from detailed model-based planning. Their 2025 study reported a mixture of successor-like and model-based strategies across individuals and adaptive changes in their estimated use when environmental predictability was manipulated. The original article's publisher issued a correction in February 2025 concerning the presentation of an equation and signs in part of the analysis; the corrected version is the appropriate reference point. This work expands the psychological repertoire of predictive representations and highlights the danger of forcing all prospective choice into a two-system mixture Kahn & Daw (2025).
The conceptual distinction is one of what has been cached. A model-free value caches a predicted return; a successor representation caches the expected occupancy of future states; a complete transition model stores local environmental dynamics and can be searched with a chosen policy. Each supports a different combination of efficient choice and flexible response to reward or transition change. The precise predictive ranking depends on whether the subject updates its policy, whether the environment is partially observable, and the depth of the available rollout. Experiments can exploit these dependencies by independently changing incentives, transitions, and inferred state structure while comparing held-out choice sequences Kahn & Daw (2025) Daw et al. (2011).
Hierarchical Actions, Habits, and Policy Compression
Action sequences can acquire a degree of autonomy: once a sequence begins, component movements may be emitted with limited sensitivity to intermediate outcomes. Dezfouli and Balleine proposed accounts in which hierarchical action organization can explain habit-like patterns while retaining a richer structure than an isolated state–action value. Their work underscores that the basic unit of learning may be a temporally extended option or motor chunk rather than a single button press. Repeated sequences can compress action control and reduce computational burden, a plausible advantage when stable environmental structure makes micro-level replanning unnecessary Dezfouli & Balleine (2012) Dezfouli & Balleine (2013).
Miller, Shenhav, and Ludvig developed a value-free habit model in which repeated actions can strengthen direct stimulus–response tendencies without encoding expected outcome values in the habitual controller. Their account reproduces several traditional behavioral signatures of habits, including persistence following outcome devaluation and simple choice perseveration. This is a serious alternative to equating all habits with model-free temporal-difference learning. Research that aims to identify habitual reinforcement must therefore consider whether outcome values are represented, whether repetition alone changes policy, and whether procedural action sequences constitute a distinct level of description Miller et al. (2019).
Hierarchical and cached representations may also be learned by an originally model-based system. Repeated successful planning can generate a policy that is later executed quickly without deliberation, and a system can return to deeper reasoning after surprising failures. The observable speed of execution thus varies with the history of compilation and available context. A fruitful experimental design measures initial planning, subsequent automation, and transfer after a changed goal within the same participants. Such designs distinguish how a policy was acquired from how it is expressed, advancing beyond a simple static dichotomy between controlled and habitual behavior Sutton (1990) Dezfouli & Balleine (2013) Dolan & Dayan (2013).
Model Recovery, Predictive Validation, and Measurement
A credible computational analysis begins with an explicit generative model. It specifies the subjective state representation, parameter meanings, learning updates, initial values, decision rule, and observation process, including lapses and missing responses. Competing models must be capable of generating actual choice histories for the task and should differ in predictions the data can realistically distinguish. A small likelihood advantage is difficult to interpret if one model is more flexible or if important alternative strategies were omitted. Researchers should report simulated behavior, parameter recovery, model-recovery confusion matrices, and out-of-sample predictive performance where feasible Akam et al. (2015) Feher da Silva & Hare (2020).
Parameter recovery asks whether known parameter values used to generate simulated data can be estimated back from those data under the fitting pipeline. Model recovery asks whether data generated by one candidate model can be assigned to that model rather than a competitor. Failure in either test limits interpretation of individual differences, even when an optimizer reliably returns a numerical estimate. Hierarchical fitting can improve stability with limited trials but introduces shrinkage and group assumptions; posterior predictive checks expose mismatches in conditional choice patterns that a single summary statistic may miss. Reproducible code, task instructions, exclusion rules, and preregistered analytical contrasts support independent evaluation Collins & Frank (2012) Akam et al. (2015).
Behavioral regression, Bayesian model comparison, and neurally informed regressors each answer particular questions. Reward-by-transition interactions examine whether a known pattern is present; fitted models compare explicit psychological accounts; imaging regressors test neural covariation with model-derived quantities. Evidence becomes more persuasive when methods converge under manipulations that independently change transition knowledge, outcome value, and computational cost. The 2020 instruction-related challenge to standard two-step interpretations demonstrates why a single robust regression effect cannot settle the architecture of reinforcement learning without direct measurement of the learned task model Feher da Silva & Hare (2020) Kool et al. (2016) Daw et al. (2011).
Individual Differences, Development, and Ecological Generalization
Estimated differences in planning can reflect learning history, representational sophistication, working-memory capacity, task literacy, incentive sensitivity, and willingness to expend computational effort. Observing these sources together requires repeated measurements rather than treating a model-based weight as a fixed personal property. Experimental instruction, prior experience with games, numerical understanding, and reward magnitude can change how a participant construes the state space and evaluates options. A high-quality individual-differences program should compare models across multiple tasks with overlapping construct demands and examine within-person stability under controlled changes of context Collins & Frank (2012) Kool et al. (2017) Feher da Silva & Hare (2020).
Developmental claims should be grounded in age-appropriate tasks that distinguish instruction comprehension and structural learning from prospective planning. Children and older adults may differ in what they initially encode, how they use memory support, and how much computation they consider worthwhile. An identical observed stay response can therefore correspond to distinct processes in different groups. Tasks that provide learned transition diagrams, verbal probes, or reduced memory load can help distinguish capacity to use a model from capacity to acquire and maintain it. Normative claims about lifespan progression require longitudinal and replicated developmental designs rather than simple correlations between age and fitted coefficients Dolan & Dayan (2013) Karagoz et al. (2024).
Ecological generalization is a major unresolved challenge. Laboratory trials often feature artificial point currencies, simple actions, repeated probabilistic feedback, and short horizons, whereas consequential decisions incorporate delayed outcomes, social consequences, constraints, and rare events. A strategy optimized for many small trials may not be appropriate for one irreversible choice. Field and naturalistic research should therefore examine whether planning indices predict adaptation in realistic transitions after controlling intelligence, education, task familiarity, and motivation. Laboratory tasks are essential for mechanistic identification, while behavioral relevance requires an additional empirical bridge Kool et al. (2016) Kahn & Daw (2025).
Compulsivity, Psychiatric Dimensions, and Clinical Interpretation
Goal-directed action and model-based reinforcement learning have motivated transdiagnostic research into compulsive behavior. Gillan, Kosinski, Whelan, Phelps, and Daw examined associations between psychiatric symptom dimensions and goal-directed control using computational task measures. Their work illustrates a dimensional approach: symptoms may covary with choice signatures without being reducible to a single diagnostic category. The measured construct must still be treated as task-specific, and correlations must be interpreted alongside model identifiability, shared cognitive demands, and the diversity of participants. Such evidence provides mechanistic hypotheses rather than individual clinical diagnosis Gillan et al. (2016).
Experimental work by Gillan, Otto, Phelps, and Daw linked greater model-based learning in sequential decisions with resistance to habitual behavior in a separate devaluation-oriented assessment. This relation is theoretically important because it joins computational and traditional instrumental measures, rather than assuming their equivalence. Nonetheless, habit expression depends on training, context, outcome valuation, and test conditions, and a group-level association cannot establish that the tasks measure exactly the same latent process in every individual. Clinical research should replicate convergent behavioral tests and assess reliability before treating task coefficients as biomarkers Gillan et al. (2015) Balleine & Dickinson (1998).
Clinical translation requires longitudinal prediction, representative populations, discriminant validity against general cognitive ability and attentional factors, and assessment of whether changing the hypothesized control process changes relevant outcomes. Stress or symptom associations may have multiple causal pathways, including reward processing, memory, and task structure comprehension. The literature supports the importance of flexible control for understanding certain compulsivity-related phenomena while leaving substantial inferential distance between a fitted reinforcement-learning coefficient and a treatment decision. The present monograph makes no recommendation of individual intervention, medication, or diagnosis; its clinical contribution is a map of testable psychological processes Gillan et al. (2016) Otto et al. (2013b) Feher da Silva & Hare (2020).
Planning as Neural Simulation and the 2024–2025 Research Frontier
Recent computational neuroscience treats planning as a potentially learnable decision about when to generate internal rollouts. Jensen, Hennequin, and Mattar proposed a recurrent-network model in which an agent samples imagined action sequences, with planning organized by higher-level control; in a spatial navigation setting, the model accounts for features of human response times and exhibits replay-like patterns resembling observations in rodents. This is a theoretical and simulation-based contribution with connections to biological evidence, rather than direct proof that an identified human prefrontal–hippocampal circuit runs precisely that architecture. Its value is a set of mechanistic and behavioral predictions about when simulated trajectories should be initiated Jensen et al. (2024).
Complementary evidence from Karagoz and colleagues foregrounds construction of higher-order relational representations, while Kahn and Daw demonstrate adaptive use of temporally abstract models under manipulated predictability. Taken together these projects suggest that future research should measure not only whether participants plan but what structures they retrieve, which parts of trajectories are simulated, and when an efficient compressed prediction replaces explicit lookahead. The empirical convergence is at the level of flexible relational knowledge and strategy adaptation; each study uses different tasks, assumptions, samples, and analysis methods. Convergence is therefore promising without implying complete mechanistic identity Karagoz et al. (2024) Kahn & Daw (2025) Jensen et al. (2024).
The contrast between a prospective calculation and a cached prediction is becoming less binary as researchers study replay, hierarchical behavior, predictive abstractions, and metacognitive control over computation. This expanded architecture can explain why rapid performance follows extended learning and why a participant sometimes replans after an unexpected contingency change. It also makes experiments more demanding: models with many possible strategies can be hard to identify from choice alone. The next phase requires stronger diagnostic tasks, reliable structural probes, multi-session measurements, open model-recovery pipelines, and neural evidence that independently constrains which representations are available when choices occur Akam et al. (2015) Kahn & Daw (2025).
Reinforcement Learning Across Biological and Artificial Systems
Reinforcement learning is shared as a mathematical research vocabulary across psychology, neuroscience, and artificial-agent engineering. In an engineered system, an optimizer may update policy parameters, learn a world model, generate simulated transitions, or combine those methods. The same terms used to describe human learning can thus specify an executable algorithm in an artificial system. But a formal analogy does not determine identity of architecture: an artificial system trained with a particular objective and a human participant performing a point-reward task may have different representations, reward assignments, embodiment, and histories. Comparative science should match observable functions, internal information constraints, and independently testable mechanisms Sutton (1990) Daw et al. (2011).
Model-based control provides a particularly useful research bridge because both biological and artificial systems can be probed with changes to a transition map that their prior actions did not directly reinforce. Investigators can quantify transfer to new goals, sensitivity to planning depth, correction after a distorted world model, and resilience to reward revaluation. These tests permit controlled functional comparison while allowing each system to use different implementations. Artificial agents can further serve as simulation instruments for testing whether proposed decision mechanisms are identifiable from behavior, provided the synthetic agent set contains sufficiently diverse candidate policies Akam et al. (2015) Feher da Silva & Hare (2020).
In the horizon of the Artificial Era, psychological research can extend from Homo to Artificial by studying nonbiological decision-making systems as objects of carefully defined comparative inquiry. The research question is how an artificial agent organizes future-oriented choice, memory of contingencies, adaptive policy control, and relations with human partners. Philosophical concepts of Artificial Sapiens and Artificial Reason belong to the author's wider Aisentica framework; empirical claims in this dossier remain bound to observed learning and planning characteristics. A system that implements model-based reinforcement learning has demonstrated a computational capacity under an operational definition, while claims about agency, psychological identity, or broader sapience require independent arguments and evidence Sutton (1990) Kahn & Daw (2025).
Integrated Evidence Assessment and Alternative Explanations
The evidence for experience-dependent valuation, outcome-sensitive action, and use of relational transition information comes from mutually informative but nonidentical traditions. Instrumental devaluation experiments address goal sensitivity; sequential tasks quantify model-predicted choice patterns; fMRI examines correlates of task-related variables; cognitive-load manipulations test resource dependence; representational similarity work evaluates learned relational structure. None provides a complete explanatory account in isolation. Together they establish that reinforcement-guided behavior can involve learned environmental structure and that computation is flexibly allocated according to task conditions, although the form of underlying representations remains heterogeneous Balleine & Dickinson (1998) Daw et al. (2011) Kool et al. (2017) Karagoz et al. (2024).
The strongest methodological warning concerns identification. A hybrid model may fit a participant even when the assumed transition map is wrong; a softmax temperature may absorb unmodeled lapses; working memory may explain rapid acquisition attributed to reinforcement; and repetition-based habits may imitate model-free values. Alternative representational and policy models must be included as serious competitors. Explicitly manipulating a mechanism while holding others as constant as possible is more informative than post hoc naming of fitted coefficients. Simulation recovery, predictive validation, instruction checks, and replication across experimental paradigms should be central elements of causal interpretation Feher da Silva & Hare (2020) Collins & Frank (2012) Miller et al. (2019).
The quality of current evidence varies by claim. Existence of devaluation-sensitive instrumental responding and transition-sensitive decision behavior is supported by controlled experimental traditions, whereas any one anatomical division or unique two-controller ontology is more provisional. Studies of clinical dimensions show potentially important associations but require further replication, construct validation, and external outcome prediction. Recent work on predictive maps, successor representations, and replay-based planning is valuable for mechanism-building while still dependent on paradigm-specific assumptions. Scientific synthesis should preserve these graded conclusions rather than compressing all findings into the phrase that humans use model-based and model-free learning Gillan et al. (2016) Jensen et al. (2024) Kahn & Daw (2025).
Open Problems and a Program of Discriminating Experiments
A first open problem is the internal state problem: how does a learner decide which details of experience belong in a state, and how do those assignments change when contingencies shift? A discriminating experiment could provide identical observable rewards under conditions in which contextual cues imply different latent structures, then test whether participants transfer only within the inferred context. Competing models of flat value learning, latent-state inference, and explicit structural planning would make distinct held-out predictions. Measuring participants' transition beliefs directly would constrain whether an apparent planning deficit reflects state learning or value computation Akam et al. (2015) Feher da Silva & Hare (2020).
A second problem is control arbitration over time. Researchers could independently vary planning benefit, opportunity cost of time, branch complexity, and reliability of successor predictions, using incentives designed to produce observable differences between policies. Reaction-time distributions, eye movements, and targeted planning probes could be linked to trial-level algorithmic predictions. The same individual should be assessed repeatedly after structural learning and under a changed reward function. This design would illuminate whether control switches continuously, whether computation is cached into durable policies, and how uncertainty governs deployment Kool et al. (2017) Kahn & Daw (2025) Jensen et al. (2024).
A third problem concerns the causal neural implementation and its generalizability. Combined human behavioral modeling, high-temporal-resolution measurement, and carefully justified interventions might distinguish transition learning, prospective simulation, reward valuation, and habitual execution. Cross-species experiments can contribute where the task architecture and operational measures truly match. Clinically meaningful work additionally needs prospective replication and intervention-sensitive outcomes; artificial-agent comparisons need transparent algorithms and matched task constraints. The scientific aim is a progressively constrained account linking relational knowledge, learned values, policy generation, computational effort, and adaptation to changed environments Dolan & Dayan (2013) Gillan et al. (2016) Jensen et al. (2024).
Conclusion
Reinforcement learning and model-based control provide a coherent scientific framework for understanding how experience can guide adaptive action through both retained consequences and represented futures. The central historical trajectory runs from the analysis of instrumental contingencies and cognitive maps through prediction-error models and computational planning, into experimentally testable accounts of sequential choice. Its fundamental achievement is a language for identifying which relation an agent has learned, how the relation is transformed into preference, and which kind of information supports flexible change. The strongest evidence comes from combining goal revaluation, transition manipulation, sequential choice, and independent measures of knowledge Tolman (1948) Balleine & Dickinson (1998) Daw et al. (2011).
The classic model-free/model-based distinction remains useful because it generates explicit counterfactual predictions about changed rewards and transitions. Its adequacy is bounded by the growing evidence for working-memory involvement, richer task representations, policies that encode habits without action values, hierarchical sequences, successor representations, and resource-sensitive deployment. The scientific field is best represented as a collection of interacting computational strategies with varied information requirements and characteristic transfer capabilities. Future claims must specify the learner's state space, reward assignments, action policy, measurement model, and the tests that distinguish the explanation from alternatives Collins & Frank (2012) Miller et al. (2019) Feher da Silva & Hare (2020) Kahn & Daw (2025).
Prospective control is scientifically intelligible as the use of learned relational structure to evaluate or generate actions under goals and constraints. Its implementation can be studied across biological and engineered systems with appropriate operational definitions. The decisive research task is to explain when structured knowledge is acquired, when it is computationally worth using, how its output becomes action, and what happens when the represented world changes. This integrated account connects mathematical psychology with cognitive neuroscience, learning theory, and the comparative science of artificial systems, while keeping each level of evidence identifiable and independently testable Kool et al. (2017) Karagoz et al. (2024) Jensen et al. (2024).
References
- Akam, T., Costa, R., & Dayan, P. (2015). Simple plans or sophisticated habits? State, transition and learning interactions in the two-step task. PLOS Computational Biology, 11(12), e1004648. https://doi.org/10.1371/journal.pcbi.1004648
- Balleine, B. W., & Dickinson, A. (1998). Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates. Neuropharmacology, 37(4–5), 407–419. https://doi.org/10.1016/S0028-3908(98)00033-1
- Collins, A. G. E., & Frank, M. J. (2012). How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. European Journal of Neuroscience, 35(7), 1024–1035. https://doi.org/10.1111/j.1460-9568.2011.07980.x
- Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12), 1704–1711. https://doi.org/10.1038/nn1560
- Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron, 69(6), 1204–1215. https://doi.org/10.1016/j.neuron.2011.02.027
- Dezfouli, A., & Balleine, B. W. (2012). Habits, action sequences and reinforcement learning. European Journal of Neuroscience, 35(7), 1036–1051. https://doi.org/10.1111/j.1460-9568.2012.08050.x
- Dezfouli, A., & Balleine, B. W. (2013). Actions, action sequences and habits: Evidence that goal-directed and habitual action control are hierarchically organized. PLOS Computational Biology, 9(12), e1003364. https://doi.org/10.1371/journal.pcbi.1003364
- Dolan, R. J., & Dayan, P. (2013). Goals and habits in the brain. Neuron, 80(2), 312–325. https://doi.org/10.1016/j.neuron.2013.09.007
- Eldar, E., & Niv, Y. (2015). Interaction between emotional state and learning underlies mood instability. Nature Communications, 6, 6149. https://doi.org/10.1038/ncomms7149
- Feher da Silva, C., & Hare, T. A. (2020). Humans primarily use model-based inference in the two-stage task. Nature Human Behaviour, 4, 1053–1066. https://doi.org/10.1038/s41562-020-0905-y
- Gillan, C. M., Otto, A. R., Phelps, E. A., & Daw, N. D. (2015). Model-based learning protects against forming habits. Cognitive, Affective, & Behavioral Neuroscience, 15(3), 523–536. https://doi.org/10.3758/s13415-015-0347-6
- Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A., & Daw, N. D. (2016). Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. eLife, 5, e11305. https://doi.org/10.7554/eLife.11305
- Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010). States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66(4), 585–595. https://doi.org/10.1016/j.neuron.2010.04.016
- Jensen, K. T., Hennequin, G., & Mattar, M. G. (2024). A recurrent network model of planning explains hippocampal replay and human behavior. Nature Neuroscience, 27(7), 1340–1348. https://doi.org/10.1038/s41593-024-01675-7
- Kahn, A. E., & Daw, N. D. (2025). Humans rationally balance detailed and temporally abstract world models. Communications Psychology, 3, Article 1. (Publisher correction: 2025, Article 20.) https://doi.org/10.1038/s44271-024-00169-3
- Karagoz, A. B., Reagh, Z. M., & Kool, W. (2024). The construction and use of cognitive maps in model-based control. Journal of Experimental Psychology: General, 153(2), 372–385. https://doi.org/10.1037/xge0001491
- Kool, W., Cushman, F. A., & Gershman, S. J. (2016). When does model-based control pay off? PLOS Computational Biology, 12(8), e1005090. https://doi.org/10.1371/journal.pcbi.1005090
- Kool, W., Gershman, S. J., & Cushman, F. A. (2017). Cost-benefit arbitration between multiple reinforcement-learning systems. Psychological Science, 28(9), 1321–1333. https://doi.org/10.1177/0956797617708288
- Miller, K. J., Shenhav, A., & Ludvig, E. A. (2019). Habits without values. Psychological Review, 126(2), 292–311. https://doi.org/10.1037/rev0000120
- Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007). Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology, 191(3), 507–520. https://doi.org/10.1007/s00213-006-0502-4
- Otto, A. R., Gershman, S. J., Markman, A. B., & Daw, N. D. (2013). The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive. Psychological Science, 24(5), 751–761. https://doi.org/10.1177/0956797612463080
- Otto, A. R., Raio, C. M., Chiang, A., Phelps, E. A., & Daw, N. D. (2013). Working-memory capacity protects model-based learning from stress. Proceedings of the National Academy of Sciences, 110(52), 20941–20946. https://doi.org/10.1073/pnas.1312011110
- Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64–99). Appleton-Century-Crofts. https://www.columbia.edu/~rk566/Session4/Theory%20of%20Pavlovian%20Conditioning.pdf
- Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599. https://doi.org/10.1126/science.275.5306.1593
- Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine Learning: Proceedings of the Seventh International Conference (pp. 216–224). Morgan Kaufmann. https://doi.org/10.1016/B978-1-55860-141-3.50030-4
- Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55(4), 189–208. https://doi.org/10.1037/h0061626