Psychology by BogdanovaThe Encyclopedia of Psychology for the Artificial Era

RESEARCH DOSSIER

Computational Models of Associative and Reinforcement Learning

Abstract. Computational models of associative and reinforcement learning explain how humans and animals transform event contingencies, feedback, and sequential outcomes into predictions and action policies. This monograph examines the historical foundations and mathematical commitments of Rescorla–Wagner error correction, cue competition, attentional associability, temporal-difference learning, model-free and model-based control, successor representations, Bayesian latent-cause inference, and volatility-sensitive belief updating. It critically analyzes animal conditioning experiments, human revaluation and multistep decision tasks, neural prediction-error studies, and the methods required to recover, discriminate, and validate computational mechanisms. The synthesis identifies strong evidence for prediction-dependent learning and flexible state representations while clarifying the limits of parameter attribution, behavioral model comparison, neural interpretation, and generalization across biological and artificial systems.

Keywords: associative learning; reinforcement learning; Rescorla–Wagner model; prediction error; temporal-difference learning; model-based control; model-free control; Bayesian latent causes; uncertainty; reward prediction error; successor representation; computational model recovery; parameter identifiability

The Explanatory Problem: Learning from Contingency and Consequence

Computational models of associative and reinforcement learning specify how experience changes predictions, inferred environmental structure, action values, and subsequent choices. Their scientific object is the transformation from temporally ordered observations to adaptive expectations or policies. In Pavlovian learning an organism can acquire predictions about events that follow a stimulus; in instrumental learning it can acquire the consequences of acting in a state. These histories overlap, yet they generate different experimental observables. A model becomes psychological only when its input variables, updating assumptions, and output measures correspond to operations available in an experiment. The Rescorla–Wagner framework formalized outcome prediction as error-correcting change in associative strength, while temporal-difference learning extended the temporal horizon over which predictions can be learned Rescorla & Wagner (1972) Sutton (1988).

Association is an explanatory relation inferred from behavior under specified contingencies, rather than a synonym for mere repeated pairing. Blocking, extinction, context-dependent responding, reward devaluation, and reversal learning demonstrate that identical exposure counts may support different expectations or actions. Rescorla emphasized that the content of conditioning is best investigated through controlled manipulations of predictive relations, including tests conducted after acquisition Rescorla (1988). Computational models consequently make strong claims only when they predict differences among histories that simpler rules treat as equivalent. The formal object of analysis includes observed stimuli, possible hidden states, experienced outcomes, available actions, task timing, and the learner's uncertainty about these quantities.

The research synthesis developed here compares error-correction rules, attention-sensitive learning, temporal-difference prediction, model-free and model-based control, successor representations, and Bayesian latent-cause inference. It evaluates experimental support alongside identifiability, fitting, simulation, and transfer to new contingencies. Neural correlates are considered as converging constraints on computational explanations rather than direct proof that a particular mathematical rule is implemented in one brain region. The evidence spans animal conditioning, human sequential choice, uncertainty manipulations, and functional neuroimaging. A central conclusion of the methodological literature is that good fits to observed choices must be supplemented by parameter recovery, generated behavior, and tests designed to distinguish competing mechanisms Wilson & Collins (2019) Palminteri et al. (2017).

Historical Foundations: Conditioning, Prediction, and Cognitive Structure

The historical route toward formal learning models began with experimental control over cue–outcome relationships, response contingencies, and reward schedules. A decisive development was the recognition that prediction, rather than coincidence, can regulate acquisition. Kamin's analysis of blocking brought attention to the consequences of prior training: once one cue already predicts an outcome, pairing a compound containing an additional cue with that outcome may produce little learning about the new cue Kamin (1969). The phenomenon made the representation of competing predictors a central scientific problem. Rescorla and Wagner subsequently expressed this competition as a trialwise discrepancy between actual reinforcement and the summed predictions of present cues Rescorla & Wagner (1972).

The evolution of computational learning research also intersected with cognitive approaches to representation. Tolman's discussion of cognitive maps emphasized that organisms may acquire exploitable environmental relations that are expressed when task conditions change Tolman (1948). This tradition is important because a cached record of rewarded actions and a representation of environmental transitions support different counterfactual choices after changes in available paths or outcomes. Formal reinforcement learning later supplied a common vocabulary for state, action, outcome, expected return, and policy, allowing competing interpretations to be converted into explicit algorithms. The historical theories retain their distinctive experimental origins: one tradition controlled stimulus contingencies; another investigated navigation, latent learning, and flexible action.

A third development connected temporally separated consequences with predictions that evolve over multiple steps. Sutton's temporal-difference analysis established how a learner could adjust a current value estimate toward a reward plus a subsequent estimate, thereby assigning predictive importance to intermediate states Sutton (1988). Neurophysiological work on midbrain dopaminergic responses then motivated comparisons between formal reward prediction errors and event-locked neuronal activity Schultz et al. (1997). Those connections gave computational models unusual cross-level influence. Their significance rests on explicit correspondence among experimental events, model variables, and measurements, while the inference from correlated activity to a unique neural algorithm remains open. Historical chronology and present explanatory adequacy require separate assessment.

Formal Vocabulary: States, Events, Values, Beliefs, and Policies

A formal model first defines its state space. An observed cue may be treated as the state itself, as evidence about a hidden state, or as one component of a richer representation that includes task history and contextual information. An action is an available intervention by the learner; a transition is the probabilistic change from one state to another conditional on that action. Reward is a model variable for experienced or experimentally specified utility-related outcomes. State value, action value, and a decision policy have different mathematical roles: value estimates expected future return, whereas a policy specifies action probabilities under a decision rule. Confusing these roles can produce plausible graphs with incompatible psychological interpretations Niv (2009) Daw et al. (2011).

The simplest delta rule adjusts a prediction V after an outcome r through V(new) = V(old) + alpha × (r − V(old)), where alpha regulates the impact of discrepancy. In compound conditioning the predicted outcome may instead be a sum of associative strengths, and the outcome may be represented as a reinforcement asymptote. The prediction error is the difference between obtained and expected events; it is defined relative to a state representation and outcome scale. A high error can result from unexpected reward, mistaken state classification, or an unmodeled change in contingencies. Thus two agents displaying similar updates may have different representations and different psychological processes. Distinguishing these alternatives requires probes that change the informational structure of the task Rescorla & Wagner (1972) Gershman et al. (2010).

Sequential reinforcement models frequently include a discount factor, an exploration or inverse-temperature parameter, learning-rate parameters, and possibly forgetting or choice-perseveration terms. These parameters are descriptive within a specified likelihood and task design. A large fitted inverse temperature is not a direct biological measurement of motivation, and a high learning rate does not independently establish sensitivity to uncertainty. The same choices can arise through compensating parameter changes or through omitted processes such as working memory, response bias, or lapses. A complete analysis specifies the observation model, prior assumptions when relevant, model-fitting objective, parameter bounds, and diagnostic simulations. The statistical and psychological meaning of estimates must be established together Collins & Frank (2012) Wilson & Collins (2019).

From Learning Curves to Mechanistic Discrimination

Behavioral acquisition curves summarize how performance changes across trials, yet they are often insufficient to identify the process generating change. Several models can produce gradual response increases when rewards are stable. Their divergence becomes clearer when a conditioned cue is compounded with another predictor, an expected outcome is omitted, a contingency reverses, or the environment is reorganized without new direct reward experience. Strong evidence therefore emerges from experimentally constructed contrasts rather than from maximizing correspondence to a single trajectory. The classic assessment of the Rescorla–Wagner model identifies both its explanatory achievements and phenomena for which additional mechanisms are necessary Miller et al. (1995).

A discriminating experiment varies a causal or informational feature while preserving other determinants. For example, a reward devaluation procedure tests whether action selection can incorporate the present desirability of a known outcome; a transition-revaluation procedure tests whether a policy can exploit a changed environmental map. A reversal task measures flexibility, but reversal alone does not identify a Bayesian learner, a dynamic learning rate, or a state-switching account. Each can accelerate adaptation after unexpected outcomes. Experimental analysis needs predictions for initial trials after a change, subsequent learning, choice variability, and transfer to novel probes. Structural distinctions among algorithms are meaningful when expressed as differences in behavior under such manipulations Daw et al. (2011) Momennejad et al. (2017).

Computational adequacy has multiple dimensions. Descriptive adequacy concerns how much observed behavior an equation captures; predictive adequacy concerns performance on unseen observations; generative adequacy concerns whether fitted agents reproduce phenomena found in participants; explanatory adequacy concerns whether distinct model components map coherently to independently manipulable processes. These dimensions can diverge. A sophisticated model can forecast better through flexible nuisance parameters while providing weaker evidence for its proposed mechanisms. Palminteri and colleagues argue that model comparison should include the model's ability to generate the characteristic effect, not merely an information criterion Palminteri et al. (2017). Wilson and Collins make model simulation and recovery central steps in a reproducible workflow Wilson & Collins (2019).

Levels of Explanation and the Boundaries of the Dossier

Associative and reinforcement learning can be described at computational, algorithmic, and implementation levels. The computational level specifies what problem is being solved and what information is available; the algorithmic level proposes update rules, representations, and decision procedures; the implementation level investigates interacting neural and bodily processes. A temporal-difference error is first an algorithmic quantity. A signal correlated with it during neuroimaging or unit recording can constrain implementation hypotheses, but correspondence is incomplete without alternative-regressor analyses, temporal specificity, and causal intervention. The distinctive value of cross-level work lies in convergence across controlled behavior and targeted physiology Schultz et al. (1997) Gläscher et al. (2010).

This Research Dossier focuses on formal explanations of learning by human and animal subjects, including the experimental circumstances that identify or challenge those explanations. Detailed anatomy and the cellular origin of reward-related signals belong to the neighboring Neural Models of Reward and Prediction Errors dossier; artificial agents' engineering adaptation belongs to Artificial Learning and Behavioral Adaptation. Neural findings and machine implementations are included only where they clarify the interpretation or limits of psychological algorithms. The distinction protects the scientific identity of a shared computational language without assuming that applying the same equation to a person and a machine demonstrates equivalent mechanisms.

The literature reviewed here includes primary experiments, foundational formal theory, and methodological syntheses. Source appraisal distinguishes experimentally measured outcomes from mathematical simulations and from broader conceptual proposals. Where an article's available evidence is a published abstract, analysis is restricted to the verifiable design description and reported conclusions, rather than importing unverified quantitative claims. This approach is especially important in a field where model parameters, cortical activations, and individual learning strategies are often converted too quickly into psychological labels. A defensible synthesis treats every formalization as a set of falsifiable commitments concerning the conditions under which experience changes predictive or choice behavior Doll et al. (2015) Wilson & Collins (2019).

Rescorla–Wagner Error Correction and Cue Competition

The Rescorla–Wagner model describes associative change when conditioned stimuli jointly predict an unconditioned event. Its central expression is ΔV_i = α_i β (λ − ΣV_present): the change assigned to cue i depends on its salience or associability, a reinforcement-dependent scaling term, and the discrepancy between the maximum supportable association and the summed predictions of the cues available on that trial. The equation is important because it gives each concurrent cue access to the same overall prediction error. Under a common outcome, predictors therefore compete for associative strength. The published 1972 chapter situated this mechanism among phenomena such as blocking, conditioned inhibition, overshadowing, and differences between reinforcement and omission Rescorla & Wagner (1972).

Consider a preliminary training phase in which A reliably precedes food. During subsequent compound trials, both A and a new cue B accompany the same food. If A already accounts for most of the expected outcome, the summed prediction error may be small and the model predicts limited acquisition by B. A comparison group receiving B without the established predictor may acquire stronger responding. This is the formal logic of blocking and of Kamin's larger argument about surprise and attention Kamin (1969). The illustrative result is conditional on learning history, cue salience, outcome intensity, context, and the response test; a real experimental demonstration requires controls for perceptual generalization and unequal exposure.

The model makes additional predictions concerning overexpectation and inhibition because a compound can imply more reinforcement than is actually delivered or a cue can signal the omission of an otherwise predicted event. These predictions helped establish the model as a benchmark against which later associative and connectionist theories could be compared. However, fitting compound learning does not guarantee an adequate account of within-trial timing, sensory-specific outcome identity, contextual renewal, or changes in attention. Miller, Barnet, and Grahame's assessment emphasizes that the success of a parsimonious rule must be evaluated against carefully defined scope conditions and counterexamples Miller et al. (1995). Its historical influence follows from predictions that expose its assumptions to experiment.

Acquisition, Omission, Inhibition, and Overexpectation

Acquisition arises when the obtained event exceeds the current expectation and increments associative strength. Omission can produce a negative error and reduce the same strength, while a stimulus trained in the presence of an excitatory predictor may become a conditioned inhibitor if the compound reliably predicts absence of reinforcement. These interpretations depend on the specified representation of reinforcer magnitude and the test used to infer inhibition. Inhibitory learning is commonly assessed with summation or retardation tests rather than treated as established by absence of a conditioned response alone. The formal model supplies a set of predicted signs and magnitudes for updating but cannot by itself determine the response system or motivational state Rescorla & Wagner (1972) Pearce & Hall (1980).

Overexpectation is theoretically informative because it probes learning without requiring an objectively absent reward. If two independently reinforced cues are presented together with an outcome of ordinary magnitude, the compound can evoke an expectation that exceeds the obtained reinforcement. The Rescorla–Wagner mechanism predicts a negative discrepancy and subsequent reduction in the associative contributions of the two cues. A simple rule in which each stimulus changes only when its own expected outcome is violated cannot reproduce the same compound logic without additional assumptions. The distinctive test lies in controlled training of the components and comparison of individual-cue responding before and after compound experience, with attention to generalization and contextual changes Miller et al. (1995).

Associative strength is an abstract predictive quantity, not a directly observed psychological substance. The mapping from that quantity to response frequency, response vigor, response latency, or physiological preparation may differ across procedures. Consequently a failure of responding can reflect changes in motivation, masking by competing actions, a new context, or weak expression of an otherwise intact association. Laboratory studies that compare conditioned responding across response systems and carefully matched tests can separate some of these factors. Rescorla's analysis of Pavlovian conditioning stressed that acquiring relations among events need not be exhausted by the acquisition of a single overt conditioned response Rescorla (1988). Computational models benefit from representing that observation explicitly.

Attention, Associability, and the Dynamics of Cue Selection

Fixed cue salience offers an economical description for many acquisition effects, but organisms may change which stimuli receive effective learning as experience accumulates. The Pearce–Hall model introduced a rule linking associability to the degree of uncertainty or surprise associated with the outcome predicted by a cue. The principle is that cues with reliably predicted consequences progressively lose some capacity to support further learning, while cues involved in unexpected outcomes retain or regain attention Pearce & Hall (1980). This mechanism concerns the rate of learning from new observations and can be distinguished conceptually from a change in the expected outcome itself. Its experimental interpretation requires task manipulations that isolate attentional processing from the current strength of association.

Attention models predict interactions between a cue's past predictive usefulness and its future learning rate. Theories differ in whether attention preferentially tracks previously reliable predictors, unresolved errors, or task-relevant uncertainty. A response acceleration after surprise is compatible with several processes: greater sensory processing, altered state inference, response exploration, or higher effective learning rate. Theoretical ambiguity is reduced by designing transfer phases in which earlier training affects learning about new outcomes under matched exposures. Pearce and Hall argued that a conditioned stimulus can lose effectiveness even while its signal remains informative, making attention an independent theoretical variable Pearce & Hall (1980).

Computational treatments of selective attention connect feature weighting with statistical inference and reinforcement learning, but the connection should preserve the different scientific commitments of each theory. Dayan, Kakade, and Montague explored how learning and attention can be related across computational and neural levels Dayan et al. (2000). In practical model fitting, adding a dynamic associability parameter without independently informative manipulations may improve training fit while reducing parameter identifiability. A stronger model comparison establishes that the attention term predicts behavior in a new condition in which fixed-rate error correction and latent-state switching make different forecasts. Attention is most useful as a tested mechanism, rather than as an unrestricted adjustment for every deviation from a simple delta rule.

Extinction, Renewal, and the Multiplicity of Learned Causes

Extinction traditionally involves repeated presentations of a learned cue without its anticipated outcome. A pure error-correction account updates the cue's expected outcome downward as negative prediction errors accumulate. Such a rule describes declining conditioned responding within extinction, yet context-dependent recovery and related observations indicate that the earlier training history can continue to influence behavior. Extinction may support additional learning whose expression depends on context or hidden-state classification. Gershman, Blei, and Niv formalized contextual learning and extinction as probabilistic inference over latent causes, addressing renewal and latent inhibition within a unified framework Gershman et al. (2010).

A latent-cause account proposes that observations are assigned to inferred generating situations rather than obligatorily updating one undifferentiated association. When extinction experience is attributed to a new cause, original acquisition can remain available and may reappear under a context that favors retrieval of the acquisition cause. When acquisition and extinction are judged to arise from the same cause, updating can instead modify an existing predictive memory. Gershman, Monfils, Norman, and Niv develop this interpretation of memory modification through structure learning and associative updating Gershman et al. (2017). The account is theoretically attractive because it makes an experimental variable of contextual similarity, temporal spacing, and inferred identity of the causal situation.

Empirical interpretation remains demanding. A context shift can change perceptual similarity, arousal, retrieval, and response opportunity simultaneously; spontaneous recovery can emerge from several mechanistic theories. Latent-cause models add a structured explanation, but strong support requires novel predictions about which experiences are clustered together and how this grouping affects later expression. The model also depends on assumptions about the number and temporal persistence of causes, likelihoods assigned to observations, and inference approximations. Comparing extinction trajectories alone may be insufficient. Controlled renewal, reinstatement, and memory-retrieval manipulations provide a richer basis for distinguishing cause inference from a changing global learning rate Gershman et al. (2010) Gershman et al. (2017).

Representational Limits of Elemental Association

Elemental error-correction models represent a stimulus compound through the combined contributions of identifiable components. This design can explain cue competition, but compounds may also be experienced as configurations with properties that are not the sum of isolated elements. If a task demands distinguishing A, B, and AB as qualitatively different events, an exclusively additive representation can make incorrect predictions without new elemental or configural features. These cases remind researchers that the representational space chosen before updating is itself a substantive psychological hypothesis. A learning rule can be correct relative to a representation yet behaviorally unsuccessful because the organism partitions experience differently Miller et al. (1995).

Outcome identity introduces another constraint. An animal may distinguish food from water, or one sensory consequence from another, even when two outcomes have comparable immediately reinforcing value. A scalar associative strength cannot always preserve the information needed for outcome-specific revaluation or selective response preparation. The question is whether the experimental data require outcome-specific predictive representations, richer state variables, or a different action-value system. Rescorla's broader account of Pavlovian conditioning argues for attention to what is learned about event relationships instead of assuming that measured responding exhausts acquired knowledge Rescorla (1988).

These limitations suggest a productive analytical division: the update rule defines how beliefs or values change, while representation defines what can be learned and generalized. Adding configural nodes, feature interactions, outcome-specific values, or latent contexts can extend explanatory scope, yet each addition should be constrained by manipulations that expose its independent consequences. Otherwise the model risks becoming unfalsifiable through unconstrained representational engineering. A rigorous research program compares a minimal baseline, specified alternatives, and held-out tests. Miller and colleagues' comprehensive assessment remains valuable precisely because it treats empirical anomalies as evidence for refining the explanatory domain rather than as mere residual error Miller et al. (1995) Wilson & Collins (2019).

Temporal-Difference Learning and Predictive Time

Temporal-difference learning extends prediction-error reasoning to environments where informative events occur across a sequence of states. Its canonical one-step error is δ_t = r_t + γV(s_{t+1}) − V(s_t), where r_t is an immediate outcome, γ a discount factor, and V a prediction of future return. Instead of waiting for a final reward to evaluate every earlier moment, a learner updates from the difference between temporally neighboring predictions. This gives formal meaning to the propagation of value backward through a predictive sequence. Sutton's foundational analysis established the temporal-difference family and examined convergence and prediction performance under specified assumptions Sutton (1988).

The temporal-difference formulation incorporates two design commitments that deserve explicit psychological scrutiny. First, the learner must identify the successive states to which predictions attach; errors in temporal segmentation alter estimated predictions. Second, the learner must represent the relevant timescale, whether through state succession, eligibility traces, or other temporal machinery. A behavioral experiment that shows gradual anticipation of reward does not by itself identify a particular discount factor or state code. Computational work on representation and timing in dopamine theories shows how changes in temporal representation modify the expected pattern of error signals Daw et al. (2006).

An eligibility trace can assign credit to recently encountered states or actions, permitting the update to influence earlier steps more directly. Its decay determines how far backward recent experience continues to matter. Such a mechanism is valuable when cue and outcome are separated by a delay, yet delays also alter memory, attention, perception, and temporal expectation. Experiments must distinguish those contributions rather than treating a fitted trace constant as a self-contained neural measurement. The temporal-difference framework is most informative when it predicts both the trajectory of learning and the timing of responses under manipulated intervals, with generalization to untrained sequences as a further criterion Sutton (1988) Daw et al. (2006).

Instrumental Reinforcement Learning and Action Values

Instrumental reinforcement learning models how actions affect future outcomes and how this information guides choice. A simple action-value learner estimates Q(s,a) and updates a chosen action after observing consequences. The value may be revised by a reward prediction error incorporating future expected return, and a policy converts learned action values into choice probabilities. A softmax choice rule uses the relative values of available actions and an inverse-temperature parameter to express choice consistency. This formal apparatus clarifies the separation between learning from outcomes and selecting among possibilities, two processes that can vary independently in human or animal experiments Niv (2009).

Learning rules for actions differ in their treatment of unchosen alternatives, counterfactual outcomes, reward magnitude, omission, and perseveration. If outcomes for both chosen and unchosen options are revealed, a participant may learn counterfactual values that a chosen-only model omits. If the same option is selected repeatedly after its estimated value declines, one explanation is a perseveration bias rather than an anomalously small prediction error. Stated likelihoods should therefore incorporate the information participants actually see, the timing of feedback, action-specific accessibility, and experimental instructions. A useful baseline set includes both a value-updating model and a simpler stay–switch or choice-kernel alternative, each tested on simulated and real choice sequences Palminteri et al. (2017) Wilson & Collins (2019).

The distinction between reward learning and behavioral control becomes especially important under changing motivation. A previously rewarded action may continue even when the outcome becomes less desirable, while another action may incorporate the new outcome value immediately. Such observations support inquiry into habits, goal-directed action, and cached versus computed values, but the mapping is not one-to-one. Habitual behavior can reflect practice, action chunking, environmental constraints, or limited deliberative resources. Theories of multiple memory substrates emphasize that memory mechanisms available to the decision maker shape which algorithm best predicts the observed action Doll et al. (2015). A psychologically interpretable model must state whether it predicts outcome expectancy, subjective valuation, action policy, or all three.

Model-Free and Model-Based Control

Model-free reinforcement learning directly stores or incrementally updates action values from experienced outcomes. Model-based learning represents transition contingencies and prospective outcomes, enabling choices through forward evaluation. These terms specify algorithmic families, not two immutable populations of neurons or mutually exclusive human personality types. A flexible learner may combine learned state transitions with previously cached values and may change strategy under cognitive demands. Model-based flexibility is most clearly tested after selective changes to reward value or transition structure that make distinct predictions for a cached policy and for replanning. Formal comparisons must represent the participant's knowledge of the task, including possible misunderstanding or uncertainty Daw et al. (2011) Momennejad et al. (2017).

Daw and colleagues' multistep task distinguished choices sensitive to previous reward from those additionally sensitive to the transition structure between stages. Its human behavioral data showed both classes of influence, and associated striatal prediction-error signals reflected a mixture of model-based and model-free predictors Daw et al. (2011). The result is influential because it undermined a simple identification of ventral striatal responses with a computationally pure model-free learner. The reward-by-transition interaction in this task is an operational contrast conditioned on the analysis model; later model variants can introduce additional strategies that also influence that interaction. Interpretation therefore rests on model comparison, experimental controls, and replications rather than one diagnostic coefficient.

Model-based control may depend on resources needed to maintain transitions and evaluate prospective sequences. Otto and colleagues taxed central executive resources during sequential decision making and reported changes consistent with reduced contribution of model-based valuation Otto et al. (2013). The interpretation connects formal planning to cognitive processing constraints, although divided attention also changes task encoding, choice variability, and retention. Gillan and colleagues investigated relations between model-based learning and formation of habits, extending the research beyond a single task summary toward learning history and sensitivity to outcome changes Gillan et al. (2015). Together these studies demonstrate the value of combining cognitive manipulations with formal choices rather than inferring a stable neural division from model labels alone.

Arbitration, Reliability, and Cognitive Resource Allocation

Where multiple valuation strategies are available, a further question arises: which strategy governs a particular choice? An arbitration model assigns control according to factors such as expected accuracy, uncertainty, processing cost, and reliability of learned information. Daw, Niv, and Dayan proposed a normative account of competition between control systems grounded in uncertainty and flexibility Daw et al. (2005). Its importance is conceptual as well as formal, because the efficiency of a cached strategy depends on environmental stability while planning can respond to structural changes at greater computational expense. An arbitration weight is nevertheless a latent mathematical variable whose psychological meaning requires independent manipulation.

Lee, Shimojo, and O'Doherty investigated reliability-dependent arbitration in human learning and identified neural signals associated with the comparison and weighting of model-based and model-free predictions Lee et al. (2014). Their analysis supported a mechanism that adjusts behavioral control as the reliability of each strategy changes. The study provides evidence for computationally structured neural correlates; it does not establish that every observed individual uses one fixed arbitration algorithm across tasks. Reliability can be learned differently under stable and changing contingencies, and task experience may alter the cost of planning. Strong tests require explicit competing arbitration rules and independent variation of uncertainty or time pressure.

Resource allocation also raises a measurement problem. If a model-based strategy appears weaker under working-memory load, reduced planning is one explanation, while noisier transition learning or higher lapse rates are others. Collins and Frank showed that working memory processes can account for substantial features of trialwise instrumental learning usually attributed to reinforcement learning, providing a warning about omitted cognitive components Collins & Frank (2012). An adequate model should evaluate whether a control-weight parameter remains identifiable when separate memory capacity and response-noise terms are fitted. The scientific claim is stronger when the same parameter predicts multiple task manipulations and remains interpretable under recovery tests and independently constructed behavioral contrasts Wilson & Collins (2019).

Successor Representations Between Cached Value and Planning

The successor representation offers an intermediate form of predictive knowledge. Rather than storing only immediate action values or repeatedly simulating the complete transition system, an agent learns the expected discounted occupancy of future states from each current state. Expected rewards can then be attached to those anticipated occupancies to calculate a value estimate. In schematic notation V(s) = Σ_{s'} M(s,s') r(s'), where M represents expected future state visits. The resulting predictions can respond rapidly to a changed reward function while retaining limitations if the learned transition structure changes. This creates a distinctive testable profile of flexibility and inertia Momennejad et al. (2017).

Momennejad and collaborators compared behavioral sensitivity to reward and transition revaluation in human experiments and reported evidence compatible with successor-like computations Momennejad et al. (2017). The significance lies in experimentally separating changes that can be handled by reweighting stored future-state predictions from changes that demand a revision of those predictions themselves. A model-free versus model-based binary distinction would obscure this intermediate possibility. Nevertheless, successor representations encompass variants with different learning and replay capacities, so one experiment cannot uniquely specify the underlying architecture. Distinctive generalization patterns and richer tests of state transition knowledge remain necessary.

The successor representation also clarifies why apparent planning can arise from representations acquired incrementally through experience. A system may produce partially flexible behavior without calculating a fresh full search tree at choice time; likewise, a system described as habitual may preserve state predictions that support some counterfactual transfer. Computational economy and behavioral flexibility are continuous design dimensions shaped by available representations. This perspective is compatible with research linking multiple memory systems and decision strategies Doll et al. (2015). Scientific synthesis therefore compares how models encode transitions, rewards, contexts, and policy constraints, and how they update those quantities when environmental contingencies change. The question is which architecture generates the observed selective generalization, not which label sounds most deliberative.

Bayesian Latent-Cause Inference and State Construction

Many learning environments contain hidden changes: the same visible cue can signal different outcomes in different contexts, and the learner must decide whether new observations update an old association or indicate a new situation. Bayesian latent-cause models formalize this as inference over unobserved generating states. The posterior probability of a cause depends on prior assumptions about cause frequency and persistence as well as the likelihood of observed stimuli and outcomes under that cause. Prediction is therefore a weighted combination across candidate causal structures rather than a single scalar association updated uniformly. Gershman, Blei, and Niv developed this approach to account for context-sensitive conditioning and extinction Gershman et al. (2010).

Latent-state inference and reward learning answer interacting questions. The first concerns which situation is currently active; the second concerns what outcomes or action consequences belong to that situation. A dramatic prediction error may therefore trigger either strong change within a representation or a shift in the inferred cause responsible for current observations. These solutions can generate different later recovery and transfer effects. Gershman and colleagues extended the latent-cause framework to memory modification, proposing that retrieval updates an existing trace when observations are assigned to the same cause and can support new traces when distinct causes are inferred Gershman et al. (2017).

The scientific challenge lies in identifying evidence for hidden causes rather than assuming a new cause whenever the model encounters a difficult trial. Because unobserved states can be proliferated to accommodate any change, regularization, prior sensitivity, and predictions about contextual generalization are essential. A recovery test can vary physical context, time since training, or competing cues independently to distinguish a persistent cause representation from a variable learning rate. Model uncertainty should include uncertainty about assignment to causes, not merely variance in an estimated reward. These distinctions affect both experimental design and the interpretation of fitted latent-state sequences Gershman et al. (2010) Wilson & Collins (2019).

Learning Rates under Volatility and Uncertainty

A fixed learning rate assigns the same proportional weight to prediction errors regardless of whether the environment is stable or changing. In probabilistic environments, however, the diagnostic value of a new observation varies with uncertainty about the current value and the probability that contingencies have shifted. Bayesian and approximate Bayesian models can allow effective updating to depend on these conditions. Behrens and colleagues studied how human participants used information in environments with changing volatility and reported behavior consistent with adaptation to estimated environmental change Behrens et al. (2007). The empirical contribution concerns sensitivity to environmental structure rather than a universal single optimal value of alpha.

Nassar and colleagues developed an approximately Bayesian delta-rule account of belief updating in a changing environment, showing that adjustment of effective learning rates could explain trialwise human behavior in their task Nassar et al. (2010). The model connects ordinary prediction-error updating with normative variables that signal a possible change point and uncertainty in the estimated mean. The learning rate can increase when an observation indicates a structural change rather than an ordinary sample from stable noise. This distinction is critical because a large error under high outcome variability may be less diagnostic of a changed environment than a comparable error under low variability. Research designs can independently manipulate noise and volatility to discriminate these interpretations.

Uncertainty-sensitive models introduce additional estimation choices: the assumed hazard of environmental change, outcome noise, prior distribution, and possibility of stable periods. A fitting procedure that fixes incorrect hazard rates may compensate by inflating learning-rate or response-noise terms, weakening mechanistic inference. Researchers should measure how estimated parameters respond to manipulated change frequencies, assess alternative environments, and check whether simulations recover the qualitative distinction between volatility and stochastic noise. A claim that participants approximate Bayesian inference can be well supported for a particular experiment without establishing exact normative calculation in each individual brain. Such precision is central to comparing computational models fairly Behrens et al. (2007) Nassar et al. (2010).

Prediction Errors in Neural Data: Convergence and Limits

Computational neuroscience linked reward prediction error to phasic activity of some dopaminergic neurons, providing a prominent example of an algorithmic variable used to organize physiological evidence. Schultz, Dayan, and Montague compared neural responses during reward learning with the logic of prediction errors: unexpected reward, reward-predicting cues, and omitted expected outcomes have different implications under the computational account Schultz et al. (1997). The analysis helped establish an experimental program connecting trialwise expectations with neuronal timing. It does not make dopamine identical to reward, subjective pleasure, or every formal prediction-error signal. Neural responses vary across cell types, behavioral contexts, and measurement methods.

Human neuroimaging broadened this cross-level analysis. Gläscher and colleagues distinguished a prediction error concerning environmental states or transitions from a reward prediction error, reporting separable correlates in a probabilistic sequential task Gläscher et al. (2010). Daw and collaborators later found that striatal prediction-error regressors in a multistep task reflected influences from both model-free and model-based predictions Daw et al. (2011). These findings are informative because they connect measured behavior, explicit latent variables, and neural responses in the same experimental design. Yet fMRI blood-oxygenation responses are indirect physiological measures, and regressor correlations do not uniquely identify the cellular computation or establish a causal implementation.

Alternative accounts must consider correlated regressors, different state representations, model misspecification, individual variability, and task-specific prediction signals. Representation and timing models show that an observed neural response may depend strongly on whether reward expectations are expressed at cues, intermediate states, or elapsed-time representations Daw et al. (2006). Region labels alone therefore provide limited explanatory detail. Strong neural evidence includes independently validated behavioral models, explicit tests of alternative computational variables, replication across tasks, and intervention where feasible. The neighboring neural reward dossier addresses neuroanatomical and physiological depth; the present research treats these results as constraints on psychological models of learning.

Credit Assignment, Multicue Choice, and Representational Structure

Credit assignment identifies which antecedent events or actions should receive responsibility for an outcome. In elemental Pavlovian learning, a shared prediction error is distributed among present cues according to their effective salience; in sequential action learning, delayed outcomes must be attributed across choices and states. These versions share a problem of assigning explanatory influence, but differ in their causal structures and observables. Akaishi and colleagues studied credit assignment in a multicue environment, relating human behavior and neural measurements to the discrimination of relevant from irrelevant information Akaishi et al. (2016). The research extends learning beyond isolated one-cue trials toward the selective use of multiple competing predictors.

Different credit-assignment models can predict similarly during stable training yet disagree when an irrelevant cue becomes predictive or a previously reliable choice becomes disadvantageous. Cue-specific attention, eligibility traces, causal inference, and state abstraction are candidate solutions. These alternatives should be compared with controlled manipulations of temporal delay, sensory salience, cue correlation, and responsibility for outcomes. A model that fits choices after stable repeated training may perform poorly when contingency structures are deliberately rearranged. The explanatory test is whether the proposed architecture transfers the learned credit to newly informative predictors with a distinct pattern of errors and recovery Pearce & Hall (1980) Akaishi et al. (2016).

Representational flexibility also concerns hidden task variables. Wilson, Takahashi, Schoenbaum, and Niv advanced a theoretical account of orbitofrontal cortex as representing task states relevant for learning and choice, including states that must be inferred rather than directly observed Wilson et al. (2014). Their account provides a bridge between latent-state computational descriptions and the behavioral significance of context and history. It is a proposed organizing theory with supporting evidence and discriminating predictions, not a direct observation of an abstract map in every task. Formally separating state construction from reward updating allows investigators to ask whether a failed choice reflects inaccurate values, mistaken current-state inference, or limited access to the right context.

Memory, Exploration, and the Control of Updating

Computational models of human reinforcement learning are sensitive to working memory because many tasks permit rapid retention of recent stimulus–action–outcome episodes. A participant may use an explicit memory of the last feedback rather than the incremental policy assumed in a canonical reinforcement learner. Collins and Frank combined behavioral and computational analysis to investigate how working memory contributes to learning signatures that otherwise resemble reinforcement updating Collins & Frank (2012). A model omitting this mechanism may interpret rapid acquisition as an unusually high learning rate or derive misleading conclusions about neural learning deficits. Experimental manipulation of set size, delays, and interference can help distinguish these accounts.

Exploration introduces a related ambiguity. A participant may choose a poorly valued option to gain information, exhibit random lapses, prefer novelty, or follow a stable response bias. A softmax temperature alone cannot independently identify these processes. Their consequences for subsequent learning diverge because informational sampling changes what feedback becomes available. An uncertainty-sensitive learner can use exploration to improve knowledge, but that idea must be operationalized through the structure of observable alternatives and outcomes. Research on learning under environmental volatility provides a basis for distinguishing uncertainty about values from uncertainty about change points Behrens et al. (2007) Nassar et al. (2010).

Memory capacity, planning depth, feedback delay, and uncertainty often vary together in natural behavior. Experiments that isolate them can determine which model components have psychological meaning. For example, increasing memory load while holding reward contingencies fixed can test whether performance changes are due to working-memory support or reward updating. Alternatively, changing transition probabilities without changing immediate reward can test whether choice follows a learned environmental model. The strongest synthesis links such manipulations to simulation-based model predictions and held-out performance, rather than attributing every deviation from a baseline value learner to an unspecified general cognitive limitation Otto et al. (2013) Doll et al. (2015) Wilson & Collins (2019).

Experimental Paradigms and the Logic of Model Discrimination

Experimental paradigms are scientific instruments for separating predictions of formal accounts. Blocking and overshadowing interrogate compound predictors, outcome devaluation examines sensitivity to changed consequences, reversal learning tests adjustment after altered contingencies, and sequential two-stage tasks investigate reliance on state transitions. Each paradigm measures an observable contrast under particular assumptions. A high-quality design prespecifies the training schedule, knowledge available to the participant, dependent variables, and the contrast between computational alternatives. The same reversal learning curve might be generated by a fixed high learning rate, a change-point detector, a latent cause switch, or working-memory retrieval. Discrimination requires diagnostic probes and task variation chosen because the candidate models disagree Miller et al. (1995) Daw et al. (2011) Nassar et al. (2010).

Animal experiments often provide precise control of conditioning histories and outcome contingencies, while human experiments add verbal knowledge, working memory, strategic instruction, and self-reported interpretations. Species and task translation consequently require operational matching rather than treating similarly named behaviors as the same variable. The sequential-choice studies of Daw and colleagues and the state- and reward-error analysis of Gläscher and colleagues illustrate different uses of human behavior combined with model-based neural regressors Daw et al. (2011) Gläscher et al. (2010). Multicue credit-assignment experiments add a further test of which observed events are assigned responsibility for reward Akaishi et al. (2016). Converging evidence depends on whether each manipulation identifies a shared computational constraint.

Where causal claims are intended, investigators should randomize or counterbalance contingencies and examine learning at the trial level rather than treating a fitted participant difference as an intervention. Parameter estimates can be conditioned on task exposure, attrition, prior learning histories, and misunderstanding of transition probabilities. Comprehension checks, explicit assessments of inferred transitions, and out-of-sample validation help identify these problems. Predefining a primary model family and a small set of diagnostic contrasts reduces researcher degrees of freedom. The goal is a joint chain from an experimental difference to a distinct generative prediction and then to an appropriately qualified psychological interpretation Palminteri et al. (2017) Wilson & Collins (2019).

Likelihoods, Parameter Recovery, and Predictive Validation

Computational fitting evaluates how likely observed choices or responses are under an explicitly specified generative model. For a softmax learner, the likelihood combines action values updated from the actually experienced history with the probability of the chosen action on each trial. Optimization or Bayesian estimation can recover candidate parameters, but neither technique establishes that parameters are identifiable from the design. Learning rate and inverse temperature may trade off; a perseveration term may absorb some value dependence; hidden states may be weakly distinguished from dynamic learning rates. Before interpreting recovered differences, simulations should test whether the fitting procedure can distinguish known ground-truth values across the experimentally plausible range Wilson & Collins (2019).

Parameter recovery asks whether simulated data generated at known settings yield estimates close to those settings. Model recovery asks whether simulated data from one candidate can be correctly classified when fitted with the entire comparison set. Posterior or parametric predictive checks ask whether a fitted model reproduces theoretically important features such as reversal speed, reward sensitivity, response autocorrelation, and transfer. These procedures address related but distinct failure modes. Strong in-sample likelihood alone can select a model that does not generate the effect of interest, a problem highlighted through explicit falsification examples in the computational modeling literature Palminteri et al. (2017).

Out-of-sample validation should preserve the data-generating structure relevant to the scientific question. Holding out isolated trials can be misleading if trial histories are sequentially dependent and the fitted agent is allowed information that would not be available at prediction time. Holding out whole participants tests transfer across people, while holding out sessions or task contingencies asks whether the learning rule generalizes to new histories. Hierarchical estimation can stabilize participant-level parameters but also introduces assumptions about population distributions and between-subject variation. Reporting uncertainty intervals, prior sensitivity, and fitting stability is essential to interpreting the psychological implications of a model comparison Wilson & Collins (2019).

Scientific Evidence, Replicability, and Alternative Accounts

The evidential record is heterogeneous. The original Rescorla–Wagner and Sutton contributions are formal theories whose value depends partly on mathematical properties and the experimental patterns they organize. Blocking and related conditioning paradigms provide empirical constraints for associative models, while human planning tasks and neuroimaging studies test more complex computations. Each line carries a different level of inferential specificity. A successful qualitative simulation can demonstrate that a theory is capable of producing an effect; a behavioral comparison can support relative predictive adequacy; a neural correlate constrains biological plausibility. None automatically yields unique identification of the internal mechanism Rescorla & Wagner (1972) Sutton (1988) Daw et al. (2011).

Important alternatives include attentional models, multiple-memory accounts, model-free policies with richer state representations, hierarchical strategies, context inference, and trialwise exploration. In the same dataset, several may fit similarly because their generative predictions overlap under the tested conditions. Collins and Frank demonstrate why working memory must be represented in some instrumental learning tasks Collins & Frank (2012). Palminteri and colleagues explain how a quantitatively preferred model can fail to reproduce a core behavioral phenomenon Palminteri et al. (2017). Distinguishing models therefore requires manipulations directed at their causal commitments, such as resource load, reward revaluation, transition revaluation, and unexpected context changes.

Replication is strongest when it examines the diagnostic contrast across independent samples, environments, and implementations. Public code and recoverable task designs facilitate such work, but reproducibility of a fit is separate from replication of a claimed effect and from validation of a mechanism. Neural analyses require further controls for correlated regressors and choices of hemodynamic model. Translational claims require evidence about measurement stability and generalization beyond healthy experimental samples. Research reports should distinguish exploratory model development from preregistered confirmation, describe rejected models, and make data-processing choices inspectable. The modeler's responsibility is to expose uncertainty and competing explanations as parts of the scientific result Wilson & Collins (2019) Palminteri et al. (2017).

Artificial Systems and Cross-Domain Generalization

Reinforcement-learning algorithms also appear in artificial systems, creating an important opportunity to compare formal principles across biological and engineered learners. The same temporal-difference update can be implemented in software, used as a behavioral model for human choice, or proposed as a computational description of neural signaling. These uses share a mathematical vocabulary but require separate empirical validation. For an artificial agent, one can inspect implementation, reward definition, architecture, and training history; for a human or animal, those mechanisms must be inferred from experimentally accessible behavior and other measurements. Conceptual equivalence of an equation cannot by itself establish equivalence of memory, motivation, awareness, or agency Sutton (1988) Niv (2009).

Cross-domain research becomes scientifically productive when it specifies transferable questions. Does a learner represent hidden causes, update future-state occupancies, respond to contingency reversals, or allocate control by uncertainty? Can one architecture predict performance under a manipulation that another architecture fails? Such questions allow human and artificial behavior to be compared through matched tasks while preserving the different constraints of their implementation. A system that adjusts parameters from feedback might reproduce a human learning curve even while relying on wholly different representations. The Artificial Era perspective broadens the class of systems to which these questions are applied without allowing philosophical classifications to substitute for experimental evidence.

The present dossier confines its main inference to computational models explaining associative and reinforcement learning in psychology. Research on artificial learning architectures has independent scientific standing and is maintained in the related Artificial Learning and Behavioral Adaptation dossier. Comparative investigations can improve theories of learning by identifying which predictions arise from general computational constraints and which depend on organism-specific perception, development, and action. They also make model transparency particularly valuable: an artificial benchmark can demonstrate algorithmic sufficiency under explicit conditions, while a psychological experiment establishes whether subjects actually show the predicted pattern. Reliable transfer requires both forms of evidence, separately documented Momennejad et al. (2017) Wilson & Collins (2019).

Conclusion

Computational models of associative and reinforcement learning provide a disciplined account of how organisms revise expectations and actions across experience. Their foundational contributions are the formal representation of shared prediction error, competition among conditioned cues, temporal propagation of expected value, and the distinction between cached action preferences and planning from an environmental model. Subsequent approaches demonstrated the importance of attention, latent-state inference, uncertainty-sensitive updating, working memory, and intermediate successor representations. These advances show that learning depends on both the rule by which information changes and the representation to which the rule is applied Rescorla & Wagner (1972) Sutton (1988) Gershman et al. (2010) Momennejad et al. (2017).

The clearest empirical contributions emerge when experimental manipulations create distinct predictions. Compound conditioning tests whether cues share predictive error; extinction and renewal probe how contexts organize learning; reward and transition revaluation examine behavioral flexibility; uncertainty manipulations probe learning-rate adaptation; and sequential tasks test combinations of planning and incremental value learning. Neural data provide converging observations about error signals and state representations but demand cautious interpretation because algorithmic variables may be correlated and because physiological measurements do not reveal the entire generative process. The scientific task is to combine levels of evidence while retaining their different explanatory force Schultz et al. (1997) Daw et al. (2011) Gläscher et al. (2010).

The outstanding problems concern model identification, representation learning, cross-task generalization, and the degree to which fitted mechanisms are stable within individuals and across contexts. Progress requires transparent generative models, effective experimental contrasts, recovery tests, predictive validation, and independent replication. Mathematical parsimony is valuable when it preserves explanatory discrimination; richer models are warranted when they predict genuinely new phenomena. The resulting scientific synthesis is a coherent research program in which formal theories are evaluated by the distinctive behavioral and neural consequences they imply. Its future lies in explaining the organization of learning across timescales, species, and artificial systems through increasingly testable relationships among prediction, uncertainty, memory, and action Palminteri et al. (2017) Wilson & Collins (2019).

References