RESEARCH DOSSIER
Computational Model Comparison and Predictive Validation
Abstract. Computational model comparison is the empirical evaluation of alternative formal accounts of psychological data under explicit predictive targets, complexity assumptions, and experimental designs. This monograph reconstructs the development of model selection from information criteria and cross-validation to Bayesian predictive evaluation, simulation-based model recovery, and psychologically informative experimental discrimination. It analyzes generative likelihoods, complexity and flexibility, predictive scoring, information leakage, posterior predictive checks, Bayes factors, hierarchical models, identifiability, model mimicry, individual differences, and human–artificial comparative research. Primary mathematical and psychological research is integrated with contemporary evidence on reinforcement learning, memory, choices, response times, and hierarchical cognition. A critical narrative synthesis distinguishes descriptive fit, out-of-sample predictive adequacy, and warranted inference about mechanisms. The resulting scientific framework treats competing models as explicit, falsifiable families whose usefulness depends on both reliable predictions and diagnostic experiments. No new participant data, registered systematic review, or medical recommendation is presented.
Keywords: computational model comparison; predictive validation; mathematical psychology; generative cognitive models; cross-validation; held-out prediction; AIC; BIC; Bayesian model comparison; LOO; WAIC; PSIS; posterior predictive checks; model recovery; model mimicry; parameter recovery; hierarchical Bayesian models; experimental design; proper scoring rules; Artificial Era
Introduction: What Model Comparison Establishes
Computational model comparison evaluates competing mathematical descriptions of psychological observations by asking what each model predicts, how it accommodates uncertainty, and whether its apparent success survives assessment outside the data used to tune it. The relevant objects are generative specifications connecting tasks, latent quantities, individual responses, and observation noise. A model can predict choices, response times, ratings, learning curves, or neural covariates, provided that the observable target and evaluation rule are declared. The intellectual task is to determine what those predictions warrant. A high likelihood at fitted parameters is one source of evidence, while predictive validation and experimental discrimination supply further tests of the scientific interpretation Pitt & Myung (2002) Wilson & Collins (2019).
This monograph examines the meaning of model comparison across explanatory and predictive aims. It addresses competing hypotheses about cognitive mechanisms, the effect of model flexibility, held-out prediction, proper probabilistic scoring, posterior checks, and simulation-based investigations of model mimicry. It also treats hierarchical data, time dependence, model misspecification, and the difference between predictions for new trials of a familiar participant and predictions for previously unseen individuals. Such distinctions determine what successful generalization means. A central lesson of computational psychology is that choices among models must be understood relative to specified data-generating conditions and scientific questions, with explicit uncertainty surrounding both parameters and comparisons Shiffrin et al. (2008) Yarkoni & Westfall (2017).
The research method here is a critical narrative synthesis of original statistical work, methodological analyses, and published psychological experiments. Historical sources establish the foundations of information criteria, cross-validation, and prediction; cognitive studies illustrate how competing psychological accounts enter a likelihood; simulation papers investigate recovery and discriminability; and recent methodological work extends validation to hierarchical and computationally demanding models. No new participant results or meta-analytic pooled estimate is asserted. The interpretation proceeds from research questions to generative assumptions, from evaluation metrics to falsifying simulations, and from apparent model rankings to principled statements about psychological processes and their generalizability Navarro et al. (2004) Schad et al. (2021).
The Research Object: Generative Models, Predictions, and Observations
A computational psychological model specifies a conditional distribution for observations given experimental conditions, parameters, and often an individual or group identity. In a learning study, the observations may be trial-by-trial actions and outcomes; in a recognition task they may include accuracy and latency; in a rating design they may be ordinal or continuous measurements. The generative account can include memory updates, beliefs, decision thresholds, reinforcement values, and response noise. Its predictive distribution integrates uncertainty from the proposed mechanism and the observation process. Explicitly naming each component allows a model to be compared with alternatives that make similar predictions for different psychological reasons Daw et al. (2011) Wilson & Collins (2019).
The observation model defines what constitutes evidence. A Bernoulli choice likelihood, a multinomial response likelihood, and a joint response-time-and-choice likelihood can rank the same theoretical hypotheses differently because they evaluate different features of behavior. Conditional independence assumptions, lapse processes, trial ordering, and participant effects influence the predictions. A prediction for a future response conditional on the participant's learning history asks a different question from a prediction for a future participant drawn from the population. The scientist should specify whether prediction targets a point outcome, an entire probability distribution, a trajectory, or a theoretically decisive contrast Shiffrin et al. (2008) Vehtari et al. (2017).
The comparison set also matters. If two candidates share the same incorrect assumption about perception or measurement, the higher-ranked candidate can remain an inadequate description of the phenomenon. Statistical selection operates among named candidates; scientific evaluation separately asks whether the family spans psychologically plausible processes and produces diagnostically informative predictions. Model families should therefore be accompanied by baseline accounts, plausible nuisance alternatives, and checks of observable discrepancies. This makes model comparison a reproducible scientific argument rather than a contest between isolated fit scores. The requirements are particularly important when fitted latent variables are given strong psychological labels Palminteri et al. (2017) Roberts & Pashler (2000).
From Statistical Identification to Information Criteria
The modern analysis of model selection inherited the tension between fit and complexity from mathematical statistics. Akaike's 1974 formulation addressed statistical identification by relating maximum likelihood to an information-theoretic correction for the number of adjustable parameters. His criterion evaluates a model using the maximized log likelihood plus a penalty for estimated flexibility. Schwarz's 1978 asymptotic argument developed a distinct large-sample approximation motivated by Bayesian evidence and model dimension. These contributions provided widely usable benchmarks for selecting statistical structures, including psychological choice and response models, but their assumptions and objectives differ in ways that affect interpretation Akaike (1974) Schwarz (1978).
Stone's 1977 analysis connected model choice under a logarithmic predictive assessment to the behavior of Akaike's criterion in a particular asymptotic setting. This link made it possible to treat information criteria as approximations to predictive goals rather than universal grades of scientific truth. Later cognitive modeling research emphasized that the geometric shape of probability distributions can matter as much as the nominal number of parameters. Models with identical parameter counts can differ in the variety of data patterns they accommodate. Effective complexity therefore depends on how a model distributes probability over possible outcomes under the design being studied Stone (1977) Myung et al. (2000).
Historical development also reveals a shift in scientific emphasis. Conventional parameter estimation centers on fitting a prespecified structure; modern model evaluation asks whether predictions hold beyond calibration and whether candidate explanations remain distinguishable. Psychological scientists adapted information criteria, Bayesian evidence, and simulation to problems with nonlinear likelihoods, latent stages, and repeated behavioral observations. Their central methodological contribution was the recognition that a theory with a highly flexible prediction space can fit observed patterns for reasons unrelated to the proposed cognitive mechanism. The point is visible in influential work on the dangers of fit and model mimicry Pitt & Myung (2002) Wagenmakers et al. (2004).
Goodness of Fit and the Geometry of Flexibility
Observed goodness of fit is a property of an estimated model relative to a particular data set and discrepancy function. It can be impressive even when an account assigns high probability to almost every plausible behavioral pattern. Roberts and Pashler argued that the evidential force of a quantitative fit depends on the outcomes the theory excludes and the uncertainty of the measurement. Their critique exposes a general issue: an adjustable theory gains empirical content by placing consequential restrictions on observable distributions. A fit statistic alone cannot state which counterfactual outcomes would have contradicted the psychological account Roberts & Pashler (2000) Pitt & Myung (2002).
Formal complexity concerns the range and arrangement of predictions accessible to a candidate. Parameter count gives a convenient approximation in regular settings, yet nonlinearities, parameter bounds, latent mixtures, and correlated adjustments can alter the effective flexibility. Myung, Balasubramanian, and Pitt analyzed model selection through the geometry of probability distributions, demonstrating its relevance to recovering models in psychophysics. Their work provides a conceptual bridge from raw parameter enumeration to the shape of a model's predictive repertoire. A parameter that produces negligible observable variation under one task may be decisive under a more informative task Myung et al. (2000) Navarro et al. (2004).
Complexity must be evaluated for the actual inferential goal. A richly parameterized hierarchy may improve predictions for a partially observed participant by pooling information, while a simpler fixed-effect model may perform better on a completely new population. Regularization and shrinkage can reduce practical flexibility without changing the written parameter count. The appropriate penalty therefore depends on estimation, prior information, sample size, and prediction target. Simulation and held-out scoring are valuable because they evaluate these interacting elements under explicit conditions. Any claim of a model's superior mechanism requires more than numerical parsimony Shiffrin et al. (2008) Piironen & Vehtari (2017).
Likelihood, Latent Processes, and the Unit of Evidence
The likelihood encodes the assumptions that connect an internal computational account to observations. Sequential learning models commonly factor trial-wise choice probabilities conditional on preceding observations, while perceptual decision models may generate joint distributions of response choice and response time. A likelihood is a conditional model of the collected data and its ordering, so omissions can redirect the comparison. Treating correlated trials as independent, ignoring intermittent lapses, or discarding response times can favor a model for reasons external to the mechanism of interest. A scientifically useful comparison inspects the observation model as carefully as the cognitive update rule Daw et al. (2011) Wilson & Collins (2019).
The level at which data are held out should correspond to the intended scientific claim. Removing scattered trials tests interpolation into a participant's familiar task history; holding out contiguous future trials evaluates prospective adaptation; leaving out an entire person examines population-level transfer. Training-set aggregation can mask heterogeneity, and a participant-level likelihood can be dominated by frequently observed individuals. Hierarchical estimation changes both the predictive target and the uncertainty. Researchers must therefore report the unit of independence, the conditioning variables known at prediction time, and which latent quantities are recalibrated for the held-out unit Vehtari et al. (2017) Dao et al. (2024).
Predictive success can also depend on preprocessing, data exclusion rules, task transformations, and hyperparameter selection. When any part of those decisions uses the intended test set, the reported held-out score reflects some degree of information reuse. The complete modeling pipeline should be considered the predictor: data cleaning, feature construction, candidate selection, estimation, posterior draws, and scoring. This view helps separate an informative methodological advantage from an artifact of analysis choices. The requirement becomes especially consequential in small behavioral studies with many trial-level measurements but relatively few independent participants Yarkoni & Westfall (2017) Schad et al. (2021).
Prediction, Explanation, and Scientific Discrimination
Prediction evaluates how successfully a model anticipates observations under a specified distribution of tasks and participants. Explanation additionally interprets the structures and parameters that support those predictions as psychological mechanisms. Yarkoni and Westfall urged psychology to take predictive assessment seriously because explanatory research programs can accumulate elaborate accounts whose generalization remains unknown. A successful predictive test accordingly strengthens a scientific claim of generality, but does not uniquely establish the internal process generating the outcomes. Several mechanisms can be observationally equivalent within a given experimental design Yarkoni & Westfall (2017) Navarro et al. (2004).
Mechanism discrimination requires observable consequences that differ across models. A researcher comparing habit-based reinforcement learning with planning may manipulate transition structure, reward revaluation, or transfer conditions to expose predictions on which the accounts diverge. Cross-validation can reveal whether one candidate reliably predicts new choices; experimental contrasts ask whether the favored account captures a signature process. The combination is important because a complex model may predict well by absorbing nuisance patterns while leaving the central theoretical distinction unresolved. Simulation can identify where two models overlap and where a stronger experiment can separate them Daw et al. (2011) Palminteri et al. (2017).
The most defensible scientific conclusion states its comparison set, predictive target, uncertainty, and the design conditions supporting model discrimination. A ranking obtained in one laboratory task is evidence about that task's data distribution and candidate set; broader claims require independent tasks, participants, and populations. The same principle guides interpretation of machine-learning-based psychological predictions and computational comparisons between human and artificial behavior. Predictive validity supports transport to a specified environment, while explanatory validity depends on converging tests of the proposed process. Both are legitimate research aims, and the distinction directs how they should be tested Wilson & Collins (2019) Shiffrin et al. (2008).
Held-Out Prediction and the Meaning of Generalization
Held-out predictive validation asks how accurately a model describes observations that did not influence its fitting or selection. The predictive target must be specified before scoring: it may be a new response from an observed participant, a sequence from a known task, an unseen participant, or a task in a different environment. A held-out likelihood evaluates an entire probability distribution, so it rewards a model that assigns appropriate probabilities to the events that occur. For models that make conditional temporal predictions, the training/test separation must respect what information would actually be available at the moment of forecasting Stone (1977) Yarkoni & Westfall (2017).
The separation between training and evaluation data guards against optimization-induced overconfidence. Every candidate can adjust free parameters to exploit random features of training data, and repeated examination of the same test set can effectively turn that set into additional training information. Cross-validation distributes this evaluation across folds, but it should be treated as an estimate of performance under a declared sampling process. Its standard error depends on the observations and dependence structure used in the comparison. A difference in mean predictive score needs an uncertainty assessment before it becomes a decisive ranking Vehtari et al. (2017) Piironen & Vehtari (2017).
Cross-validation is especially informative when scientific claims concern transport. An alternative model may predict the next trial better while failing on reversal learning, a different sensory modality, or new participants. These are different generalization targets, each requiring separate validation data. The ideal program nests prediction within a sequence of increasing challenges: same-task held-out trials, temporal extrapolation, novel participants, and independent experimental regimes. Successful transfer beyond the original design strengthens claims about the scope of a computational account, while failures reveal which invariances its assumptions cannot support Wilson & Collins (2019) Yarkoni & Westfall (2017).
Training and Test Splits in Repeated-Measures Psychology
Psychological observations are often nested within trials, sessions, stimuli, participants, and laboratories. A random split across individual rows can permit the same person, item, or temporal trajectory to influence both training and testing. Such a split can validly estimate conditional prediction for new rows from known groups, but it does not test generalization to unseen groups. The investigator should match the splitting unit to the intended inference: participant-blocked splits for new people, stimulus-blocked splits for new items, forward-chaining splits for future time points, and cross-site splits for new recruitment settings Yarkoni & Westfall (2017) Vehtari et al. (2017).
Dependent trials require particular care because earlier feedback determines later beliefs in many reinforcement-learning models. If a validation trial is removed from the middle of a sequence, then subsequent observed choices may be conditioned on histories unavailable to a genuinely prospective predictor. A valid predictive exercise should reconstruct the conditioning history at each prediction point or explicitly declare the imputation scheme. Models that estimate participant-specific latent states also require a clear rule for how those states are inferred when a whole participant is held out. Different rules answer different questions about psychological generalization Daw et al. (2011) Collins & Frank (2012).
Nested validation handles settings where hyperparameters, feature choices, or even model families are selected from data. The outer evaluation units remain untouched until inner fitting and selection are complete. This principle is straightforward, yet implementation can be costly for hierarchical cognitive likelihoods, expensive simulation, or complex sampling schemes. Researchers should document all data-dependent decisions and can distinguish a development set from a final independent confirmatory sample. Where only one small dataset exists, numerical validation estimates remain conditional on the sampled design and cannot substitute for replication Dao et al. (2024) Schad et al. (2021).
Akaike Information Criterion and Bayesian Information Criterion
Akaike information criterion combines the maximized log likelihood with a penalty proportional to the number of fitted parameters in a regular likelihood-based setting. In its familiar notation, AIC equals minus twice the maximized log likelihood plus twice the parameter count. It approximates a predictive tradeoff under conditions defining its derivation. Schwarz's criterion, commonly called BIC, uses a penalty that increases with the logarithm of sample size, motivated by asymptotic Bayesian model evidence under regularity assumptions. Because the penalties differ, two criteria may select different cognitive models on the same observations Akaike (1974) Schwarz (1978).
The effective sample size requires judgment in dependent and hierarchical data. Thousands of trials from a few participants do not automatically supply thousands of independent pieces of evidence about population-level differences. Equally, a nominal parameter count may fail to express complex functional geometry when model parameters are weakly identified or constrained by a prior. AIC and BIC are useful screening summaries, yet neither provides a universal psychological mechanism test. Their predictive targets, estimation assumptions, and relationship to the data's dependence structure must remain visible in interpretation Myung et al. (2000) Shiffrin et al. (2008).
An information criterion should not be presented as the probability that a psychological theory is true. Differences in AIC or BIC express relative performance under particular approximations, and the ranking is conditional on candidates and observations. Analysts should inspect the associated patterns of error, recovery under simulated data, and stability to plausible model modifications. Stone's work on asymptotic links between cross-validation and Akaike's criterion gives a historical reason to understand information criteria through prediction; contemporary nonregular models motivate direct predictive simulation when analytic penalties are difficult to defend Stone (1977) Watanabe (2010).
Bayesian Predictive Evaluation and Leave-One-Out Methods
Bayesian predictive validation averages predictions over uncertainty in model parameters rather than evaluating only one maximum-likelihood estimate. In leave-one-out cross-validation, a model is repeatedly evaluated on a unit omitted from fitting, so that the sum of log predictive densities estimates an out-of-sample target. Modern computational practice often approximates these omissions from draws obtained from one fitted posterior. Vehtari, Gelman, and Gabry analyzed practical leave-one-out evaluation and the widely applicable information criterion, including the role of Pareto-smoothed importance sampling for influential observations Vehtari et al. (2017) Schad et al. (2021).
Pointwise log likelihood must be defined at the appropriate unit. A trial-level factorization may be valid for a model's conditional predictions within an observed participant, while participant-level generalization requires likelihood contributions aggregated over that participant and a corresponding refitting or approximation scheme. The posterior draws should be checked for convergence and adequate exploration before their predictive scores are trusted. If a few observations exert large influence, importance-sampling diagnostics can reveal that the approximation requires exact refitting or an alternative evaluation design. Computation is part of inferential validity, rather than a technical afterthought Vehtari et al. (2017) Dao et al. (2024).
The choice between LOO, K-fold validation, and a separate test set depends on model structure and the scientific target. Repeated estimation may be expensive when cognitive models contain hierarchical parameters or latent sequences, but approximate shortcuts should be validated against more exact calculations on representative cases. Bayesian predictive evaluation makes posterior uncertainty relevant to selection, although ranking candidates remains conditional on their likelihoods, priors, and the assumed future observations. An effective report includes pointwise differences or aggregate differences with uncertainty, the validation unit, and notable diagnostics rather than only an overall winner Piironen & Vehtari (2017) Vehtari et al. (2017).
WAIC, Singular Models, and the Limits of Simple Penalties
The widely applicable information criterion was developed for predictive assessment in models where familiar regular asymptotic approximations can break down. Many psychological models contain mixtures, redundant pathways, or latent structures that make their parameterization nonregular. Watanabe's analysis established an asymptotic relationship between Bayesian cross-validation and WAIC within singular learning theory, clarifying why prediction-oriented criteria can be useful beyond traditional parameter counting. This mathematical result concerns an asymptotic setting; empirical performance still depends on finite data, computational quality, and the model being fitted Watanabe (2010) Myung et al. (2000).
WAIC and approximate leave-one-out validation calculate summaries from posterior draws and pointwise log likelihoods. Their numerical similarity in favorable settings cannot guarantee that either estimates the desired external generalization target. When influential units, weak priors, and limited observations are present, PSIS-LOO diagnostics and actual refits can be more revealing than a criterion displayed without its uncertainty. Vehtari and colleagues demonstrated practical advantages and limitations of these computations, emphasizing that the reliability of the approximation is an empirical question Vehtari et al. (2017) Piironen & Vehtari (2017).
Complexity penalties also depend on what data are modeled jointly. If choices and reaction times are treated as one likelihood, their measurement noise and dependence affect the relative flexibility of candidate mechanisms. If each person has a latent state estimated from only a few trials, pooling assumptions shape effective complexity. No automatic criterion eliminates the need to inspect a model's generative behavior. In computational psychology, the strongest model comparison combines prediction-oriented measures with simulated evidence about whether the proposed models are distinguishable at the observed sample size Wagenmakers et al. (2004) Wilson & Collins (2019).
Proper Scoring, Calibration, and Discrimination
Probabilistic predictions should be assessed using scoring rules aligned with the reported target. A proper scoring rule rewards truthful expression of a predictive distribution in expectation under its own distributional assumptions. Gneiting and Raftery developed the theory of proper scoring rules for prediction and estimation, including the logarithmic score and other choices suited to different data types. Log predictive density measures the probability assigned to realized events; a classifier's accuracy at one threshold discards much of this uncertainty information. The evaluation criterion should reflect whether the scientific interest concerns probabilities, continuous outcomes, ranking, or concrete decisions Gneiting & Raftery (2007) Vehtari et al. (2017).
Calibration concerns the alignment between predictive probabilities and observed frequencies over relevant cases. A model that assigns a high confidence level to its favored choice on every trial can achieve apparent classification success while poorly representing uncertainty. Discrimination concerns the capacity to distinguish outcomes or conditions. Calibration, sharpness, and discrimination can move differently across candidate models, so a single score may obscure scientifically important deficiencies. Reliability diagrams, predictive intervals, and targeted checks of error by experimental condition can complement proper scoring without replacing the formal evaluation of probabilistic predictions Gneiting & Raftery (2007) Gelman et al. (1996).
Model comparison should distinguish prediction from downstream use. A policy optimized to classify a clinical risk group, for example, raises separate questions about prevalence, measurement validity, decision thresholds, and transport across populations. In this methods-focused monograph, these applications are discussed as inferential problems rather than as clinical prescriptions. Calibration on a development cohort is evidence only for that data regime. A scientific report should identify the score, baseline, uncertainty, selection procedure, and population to which the predictive claim is intended to apply Yarkoni & Westfall (2017) Bach (2025).
Posterior Predictive Checks and Absolute Model Adequacy
A relative ranking does not establish that the winning model represents the observations adequately. Posterior predictive checking addresses this omission by generating replicated observations from fitted models and comparing them with diagnostically chosen features of the actual data. Gelman, Meng, and Stern formalized posterior predictive assessment using realized discrepancies, allowing checks of statistics that depend on both observed data and latent parameters. For psychological tasks, these discrepancies can include reaction-time quantiles, learning-curve slopes, errors following reversals, sequential choice dependencies, and individual variation. The scientific value comes from specifying which features should be reproduced by the proposed mechanism Gelman et al. (1996) Schad et al. (2021).
Simulation clarifies the relationship between a model's formal equations and its behavioral implications. Even a model with favorable cross-validated performance may fail to generate a pronounced experimental effect that motivated its construction. Analysts can inspect entire distributions, conditional predictions, and the co-occurrence of multiple outcomes, rather than comparing only means. If an apparent anomaly is discovered after repeated exploration, the analysis should distinguish exploratory diagnosis from a prospectively declared confirmatory test. The goal is to reveal discrepancies that can guide theory revision and targeted experimental manipulation Palminteri et al. (2017) Wilson & Collins (2019).
Posterior predictive checks require interpretation in context. Replicated observations generated after fitting may share uncertainty with the data on which parameters were estimated; posterior predictive tail areas therefore have properties different from ordinary independent frequentist test p-values. Passing a chosen discrepancy check demonstrates adequacy only for that feature and the tested design. Researchers should use several scientifically motivated checks, clearly label each diagnostic, and preserve the simulated distributions and code needed to reproduce the comparison. Absolute checks and relative predictive rankings answer complementary questions about a computational account Gelman et al. (1996) Schad et al. (2021).
Falsification by Generating Behavioral Signatures
Computational accounts often enter psychology because they explain a named behavioral signature such as blocking, reward sensitivity, a speed–accuracy tradeoff, or planning in a multistep task. A model that fits trial data may nevertheless fail to generate the signature under experimentally realistic parameter values. Palminteri, Wyart, and Koechlin argued that simulating candidate models and checking their qualitative implications is essential for falsification. The appropriate procedure specifies the proposed diagnostic effect, generates simulated experiments under plausible parameter uncertainty, and inspects whether the predictions preserve the effect's direction and structure Palminteri et al. (2017) Wilson & Collins (2019).
Falsification requires clear counterfactuals. An investigator should ask which outcomes would count against the mechanism, what alternative mechanisms can generate the same outcome, and which experimental contrast would separate them. The simulation must reproduce the actual trial structure, feedback contingencies, missing responses, and measurement noise. A behavioral effect with a weak sample estimate may be compatible with several models, while a robust effect may still be uninformative if every candidate predicts it. Diagnosticity concerns the separation among predictions relative to expected observational variability Navarro et al. (2004) Roberts & Pashler (2000).
Successful falsification design treats an experiment as a deliberate probe of the candidate space. Changes to reward transitions, stimulus similarities, temporal gaps, or response deadlines can transform an otherwise underdetermined comparison into a discriminating one. This approach makes model construction accountable to the phenomena it purports to describe. It also creates a basis for future replication, because the diagnostic contrast, parameter ranges, and exclusion criteria can be specified before collecting a new sample. Predictive scoring and falsifying simulations should be integrated in the same evidence record Wagenmakers et al. (2004) Palminteri et al. (2017).
Model Recovery and Confusion Matrices
Model recovery asks whether a given experimental design and fitting pipeline can correctly identify a model when that model is known to have generated synthetic data. The analyst simulates many datasets under each candidate, refits all models, and records which candidate wins according to the intended criterion. A recovery matrix summarizes the probabilities of selecting model B when model A generated the data, across prespecified parameters and sample sizes. Off-diagonal values reveal confusion, which can arise from noisy observations, insufficient experimental contrast, estimation errors, or overlapping model predictions Wilson & Collins (2019) Navarro et al. (2004).
Model recovery results are conditional on the simulated parameter distribution and design. If recovery is checked only at extreme parameter values where models are easily separated, it can exaggerate diagnostic power for realistic participants. Conversely, a broad unrealistic range can imply poor recovery even when scientific interest centers on a narrower, well-supported regime. Researchers should explore plausible parameter ranges, account for boundary cases, and report asymmetries: data from one model may often be classified as another, although the reverse confusion is rare. Such asymmetry reflects the geometry of model predictions and can influence conclusions from real data Wagenmakers et al. (2004) Myung et al. (2000).
Recovery simulations must reproduce the intended inference pipeline in full, including parameter fitting, prior assumptions, candidate selection, and computational diagnostics. The simulations should be repeated under alternative plausible mechanisms and modest model misspecification to test robustness. A successful model recovery study establishes that the procedure can distinguish candidate models under simulated conditions; it cannot prove that one candidate is the actual psychological mechanism. Strong empirical claims combine recovery evidence with independent observation, targeted interventions, and transparent statements about competing models excluded from the analysis Wilson & Collins (2019) Palminteri et al. (2017).
Model Mimicry, Landscaping, and Experimental Informativeness
Model mimicry occurs when an account closely fits data produced by a competing account. Wagenmakers and collaborators proposed parametric-bootstrap cross-fitting to quantify expected differences in fit under alternative generating models. Navarro, Pitt, and Myung extended this logic through an analysis of distinguishability and informativeness, examining how one candidate accommodates patterns generated by another. Their work underscores that model discrimination depends on both the expressive regions of competing models and the experimental sample. The relevant question is whether a possible observation would reliably favor its generating mechanism under the planned procedure Wagenmakers et al. (2004) Navarro et al. (2004).
Landscaping studies model behavior across a distribution of possible outcomes. For example, two retention models may both fit the observed forgetting curve while differing sharply for particular delay schedules or interpolated learning conditions. Simulating both candidates across these regimes can reveal which new design makes their predictions separate beyond ordinary noise. As a result, model comparison becomes connected to experimental design: candidate models are used prospectively to choose observations that will be informative rather than merely retrodict data already collected. This is a powerful way of preventing a fit-driven cycle of increasingly elaborate theories Navarro et al. (2004) Roberts & Pashler (2000).
Model mimicry should be reported even when it complicates a clean conclusion. A strong observed advantage is less decisive if the design often misclassifies data generated by plausible alternatives. Equally, a nearly tied fit may become informative if simulations show the expected uncertainty under the particular sample and observational conditions. Comparative claims should include a confusion or cross-fitting analysis, a clear statement of the generating parameter ranges, and evidence that the proposed experimental signatures are identifiable. These steps support careful interpretation without demanding that every scientific model be uniquely recoverable in every task Wagenmakers et al. (2004) Shiffrin et al. (2008).
Experimental Design as a Model-Comparison Instrument
An experiment can be designed to maximize information about the difference between psychological accounts. Rather than collecting additional repetitions of the same weakly diagnostic condition, researchers can vary feedback structure, stimulus uncertainty, reward volatility, deadlines, or sequence composition in ways that models predict differently. The analysis begins by simulating the full proposed experiment under each candidate and quantifying anticipated discrimination under realistic participant and measurement variation. This approach turns theoretical divergence into a practical sampling and manipulation strategy Navarro et al. (2004) Wilson & Collins (2019).
Informative designs often require tradeoffs. More trials can stabilize parameter estimates but may introduce fatigue, learning, and temporal nonstationarity. Strong manipulations can increase between-model differences while changing the psychological process under investigation. Multiple tasks can broaden generality, yet also create measurement inconsistencies and heterogeneity in strategy. The appropriate design depends on the mechanism and its alternative explanations, not on a universal maximum of predictive accuracy. A prespecified simulation plan can show how proposed changes affect expected model recovery under plausible assumptions Palminteri et al. (2017) Myung et al. (2000).
Independent validation also belongs in experiment design. Candidate-generating simulations support the selection of a diagnostic task, after which a new dataset can evaluate whether the predicted contrasts occur. The research record should separate design-stage assumptions from observed outcomes and should preserve failures of discrimination as substantive results. Reliable comparison requires models capable of producing different predictions, instruments sensitive to those differences, and enough independent information to quantify uncertainty. When any one element is absent, claims about latent processes must be correspondingly narrow Roberts & Pashler (2000) Wagenmakers et al. (2004).
Reinforcement Learning as an Empirical Model-Comparison Case
Reinforcement-learning experiments provide a concrete illustration of psychological model comparison. Some models assign value through incremental reward prediction errors; others use knowledge of transition structure to evaluate prospective states. Daw and colleagues designed a multistep choice task intended to separate these influences and linked behavioral and neural findings to model-based and model-free computations. The study demonstrates the role of a diagnostic task: alternative accounts must predict different patterns after common and uncommon state transitions for the comparison to inform mechanism claims Daw et al. (2011) Wilson & Collins (2019).
The interpretation becomes more complex when working memory, perseverance, exploration, and lapses influence the same observed choices. Collins and Frank investigated the role of working memory in performance commonly analyzed through reinforcement-learning models. Their findings motivate alternative candidates that combine capacities and update rules rather than treating a single learning rate as a complete explanation. A more elaborate candidate may improve fit because it captures a genuine additional process, but its extra structure should be tested through held-out performance and signatures targeted to that process Collins & Frank (2012) Palminteri et al. (2017).
Results from one task should be interpreted through the fitted likelihood and the simulated discrimination of candidates. A two-step design supplies valuable evidence about control and learning, yet its parameters can depend on task-specific instructions, individual experience, and the measurement of choices. Model recovery can establish whether the comparison would separate candidate strategies under realistic parameters; posterior predictive checks show whether each candidate produces observed sequential patterns; transfer or intervention tests can examine generalization. Together these procedures yield a stronger empirical interpretation than merely reporting which model obtained the lowest criterion value Daw et al. (2011) Navarro et al. (2004).
Hierarchical Models and Population-Level Prediction
Hierarchical cognitive models express variation among participants while sharing information across a population. They can estimate group-level tendencies, individual learning rates, response thresholds, and uncertainty in both levels. Partial pooling changes predictions and therefore model comparison: a candidate may improve forecasts because it represents population heterogeneity rather than because its proposed cognitive mechanism is more accurate. Shiffrin and colleagues discussed hierarchical Bayesian evaluation as a way to address parameter interference, prediction, and generalization; subsequent work has explored efficient cross-validation for such models Shiffrin et al. (2008) Dao et al. (2024).
The prediction target determines how hierarchical parameters should be handled. Conditional prediction of another trial from a studied participant may use posterior estimates of that person's parameters, whereas prediction for a new participant must integrate over the group distribution without training on that person's responses. An omitted participant cannot contribute personal-level information through hyperparameters estimated with its outcomes. Grouped validation therefore tests a stricter inferential claim than scattered trial validation. A model that generalizes at one level may fail at another, and a complete report preserves this distinction Vehtari et al. (2017) Piironen & Vehtari (2017).
Hierarchical inference creates additional computational responsibilities. Sampling or approximation must adequately represent correlations, boundary parameters, and uncertainty across levels. Variational methods can make comparisons computationally feasible, but their predictive approximations require independent numerical scrutiny. Dao and collaborators investigated variational-Bayes cross-validation for selecting hierarchical cognitive models and used a decision-making application involving speed–accuracy tradeoffs. Such work supports practical model screening while reinforcing that inference quality, holdout unit, and scientific interpretation must be evaluated together Dao et al. (2024) Schad et al. (2021).
Sequential Decisions, Response Times, and Multivariate Prediction
Many psychological decisions produce multiple correlated outcomes: the selected action, its latency, a confidence judgment, and subsequent learning. Comparing computational models on choice alone tests a narrower account than comparing joint choice–time distributions. Decision models may assign different roles to evidence accumulation, response caution, drift variability, or nondecision processes. A model can obtain a similar choice likelihood through quite different temporal predictions, so response-time quantiles and conditional accuracy functions add diagnostic constraints. The observation model should preserve these joint features rather than treating each measured variable as a separate independent result Shiffrin et al. (2008) Wilson & Collins (2019).
Timing also matters within learning sequences. When an adaptive agent encounters changed contingencies, its choices after reversal, prediction errors, and exploration rates may reveal differences between learning and planning mechanisms. Models can agree during stable reward periods while diverging immediately after unexpected outcomes. Such transient periods carry disproportionate diagnostic information for mechanism comparison. Predictive validation should therefore evaluate both average fit and targeted conditional behavior across important phases. A single aggregate test-set score may conceal a systematic failure in a theoretically central state Daw et al. (2011) Collins & Frank (2012).
Multivariate prediction raises additional calibration questions. A model may reproduce average reaction times while underestimating long tails, or may capture choice proportions while failing to predict confidence distributions. Posterior predictive simulation supports checks of these errors; held-out proper scoring evaluates their consequences for probability forecasts. The appropriate decision about adding an outcome is scientific: a measurement should constrain a mechanism, be recorded reliably, and have a defensible observation model. When additional outcomes rely on poorly supported assumptions, apparent precision can obscure misspecification rather than improve understanding Gelman et al. (1996) Gneiting & Raftery (2007).
Measurement, Construct Validity, and Calibration
The comparison of models inherits limitations from the psychological measures on which it is based. If a task score mixes several cognitive capacities, a model parameter named attention, learning, or inhibition may capture a composite pattern rather than one isolated mechanism. Measurement validity asks whether observable indicators vary as the proposed latent construct requires. Predictive validation asks whether a fitted distribution anticipates new observations. Both questions are necessary when a computational parameter is used to characterize individuals, compare groups, or explain an intervention effect Roberts & Pashler (2000) Bach (2025).
Calibration can be designed experimentally by manipulating an inferred psychological quantity and comparing model-implied changes with measured changes. Bach's 2025 analysis of experiment-based calibration formalizes the relationship between manipulated latent quantities, theory-defined standard scores, and observed measurements. Its relevance to model comparison is conceptual: an accurate predictive model may need tests of whether its latent variables respond appropriately to controlled changes. This is particularly important for models with several substitutable parameters, where a change in one process may be absorbed numerically by another Bach (2025) Myung et al. (2000).
Between-group comparisons add measurement and sampling constraints. Differences in fitted rates or thresholds can arise from changes in task understanding, response policies, observation noise, or parameter recoverability. Independent reliability assessments and simulated recovery under each population's design are needed before interpreting contrasts as differences in cognitive architecture. Predictive validity may be strong even where parameter-level construct interpretation is uncertain; that result should be reported explicitly. An empirically useful model need not answer every psychological question, and claims should be limited to the aspects actually tested Wilson & Collins (2019) Schad et al. (2021).
Bayes Factors, Prior Sensitivity, and Marginal Likelihood
Bayesian model comparison can use marginal likelihoods that integrate the likelihood over each model's prior distribution. A Bayes factor expresses the ratio between these integrated quantities for named candidates and observed data. Because integration involves the entire prior predictive repertoire, the result can penalize flexibility differently from cross-validated predictive scores. The distinction is substantial: evidence for a model under a specified prior is a different target from expected accuracy on a future data unit. An analyst should select the method according to the research question and communicate the dependence on prior specification Shiffrin et al. (2008) Piironen & Vehtari (2017).
Marginal likelihood calculations are challenging for high-dimensional psychological models. Bridge sampling offers a numerical approach to estimating normalizing constants from fitted distributions, as described by Gronau, Singmann, and Wagenmakers. Computational accuracy depends on parameterization, sampling quality, and whether the estimator operates stably in the model's posterior geometry. Prior sensitivity analysis is essential because broad or weakly justified priors can materially alter Bayes factors while producing relatively similar conditional predictions. A single numerical ratio without sensitivity and computational checks gives an incomplete account of the evidence Gronau et al. (2020) Schad et al. (2021).
The scientific meaning of a Bayes factor remains conditional on candidate models, their priors, and the experiment's observation rule. It does not convert an untested model family into a uniquely identified cognitive mechanism. If every candidate omits a relevant process, relative evidence may favor the least inadequate model. Posterior predictive discrepancies and held-out transfer tests can reveal precisely this problem. For theory-oriented questions, marginal likelihood can be informative when assumptions are stated and computation is stable; predictive scores and model recovery test distinct aspects of the scientific claim Palminteri et al. (2017) Gelman et al. (1996).
Model Averaging, Stacking, and Uncertainty About the Candidate Set
Model comparison often culminates in one selected winner, although uncertainty may remain substantial. Predictive model averaging combines distributions from several candidates, and stacking selects weights to improve predictive performance under a chosen scoring rule. Yao, Vehtari, Simpson, and Gelman developed a method of averaging Bayesian predictive distributions suited to settings in which the actual data-generating process is outside the named candidate family. The resulting mixture can be a good predictor without establishing that its component weights are psychological frequencies or causal proportions Yao et al. (2018) Gneiting & Raftery (2007).
Combining models can reduce the instability of selecting a single candidate from near-tied alternatives. Yet an ensemble may obscure mechanistic interpretation if its components make contradictory assumptions about learning, memory, or decision control. Scientific reporting should distinguish the aim of building a useful forecasting distribution from the aim of testing a proposition about a specific internal process. A mixture's strong score may justify use as a predictive baseline while targeted interventions continue to test the mechanism. Stacking weights are optimized for a predictive target and should not be interpreted as universal posterior probabilities Yao et al. (2018) Yarkoni & Westfall (2017).
Candidate uncertainty also motivates broader sensitivity checks. Analysts can compare rankings across plausible priors, alternative noise models, task subsets, and new participants. If a mechanism-oriented claim survives these changes and passes model-recovery tests, the evidential picture is stronger than a single preferred score. If rankings reverse easily, the finding can still be scientifically useful by identifying an unstable boundary in the model family. A transparent comparison reports the candidate set and validation conditions under which each conclusion holds Piironen & Vehtari (2017) Wagenmakers et al. (2004).
Human–Artificial Comparison and the Scope of Functional Prediction
Artificial systems create a new empirical setting for comparing learned behavior, adaptive decisions, memory use, and strategy generalization. A human and an artificial agent may produce similar response distributions while implementing different computational structures. Predictive validation can assess functional similarity under matched tasks and held-out conditions; it cannot by itself establish identical internal mechanisms, experiences, or ontological status. The appropriate analysis identifies the observable behavior, the source of training or adaptation, and the level of representation to which the hypothesis refers. This preserves the distinction between functional prediction and mechanistic explanation Wilson & Collins (2019) Yarkoni & Westfall (2017).
Cross-system studies should use task designs that expose the relevant alternatives. A human participant may learn incrementally from trial experience, whereas an artificial system may exploit prior task examples, textual patterns, explicit planning, or persistent external tools. Matching the number of correct answers offers weak evidence if the systems differ in access to information. Researchers can test transfer to new contingencies, counterfactual interventions, calibration of confidence, and consistency across sessions while recording the informational histories of both systems. Performance similarity becomes more informative when it survives diagnostic changes designed to reveal different computational routes Navarro et al. (2004) Palminteri et al. (2017).
The Artificial Era provides a historical context for these questions within Psychology by Bogdanova. Comparative research can examine artificial cognition, memory, agency, and human–artificial relations through operationalized measures and controlled predictions. An authorial philosophical framework may motivate research questions, but the empirical status of each hypothesis depends on observed data and reproducible comparison. Standard tools of uncertainty estimation, recovery simulation, external validation, and construct assessment apply across systems with suitable adaptations. Their use enables rigorous investigation of emerging nonbiological forms of cognitive organization without presupposing what any behavioral resemblance ultimately means Gneiting & Raftery (2007) Schad et al. (2021).
Robustness to Misspecification and Alternative Explanations
Every computational comparison is conducted inside a set of assumptions about the task, participants, observation process, and permissible mechanisms. A model may win because an omitted lapse process affects competitors more severely, because a prior concentrates on realistic parameters, or because the experiment samples an unusually narrow behavioral regime. Robustness analysis deliberately changes scientifically plausible assumptions and records how inference responds. It tests noise distributions, parameter constraints, alternative choice policies, individual heterogeneity, and relevant task features. Stable conclusions across these changes are more persuasive than rankings obtained from a single preferred specification Roberts & Pashler (2000) Schad et al. (2021).
Misspecification must also be examined in simulation. Model recovery under perfectly specified candidate models tests procedural discrimination within that family; recovery when synthetic data include unmodeled lapses, parameter drift, or mixed strategies tests sensitivity to plausible departures. A confusion matrix that appears diagonal in the ideal case can become less informative under realistic observation noise. Comparing such conditions identifies which conclusions depend on structural assumptions. This diagnostic complements predictive scoring because a model can generalize adequately while assigning a misleading interpretation to one or more latent parameters Wagenmakers et al. (2004) Wilson & Collins (2019).
Critical discussion should explicitly recognize contradictions. A candidate may outperform another in predictive log density while failing a specific behavioral signature, or may recover under simulation while transferring poorly across participants. These patterns locate the limits of an account rather than collapsing all evidence into one score. Resolving them requires scientific priorities: a theory about a particular process needs diagnostic task effects, while an operational predictor needs validation in its intended deployment population. Reports should preserve both results and state why one criterion bears more directly on the hypothesis Palminteri et al. (2017) Yarkoni & Westfall (2017).
Computational Reliability, Reproducibility, and Audit Trails
Predictive validation depends on numerical as well as theoretical fidelity. Optimizers can converge to different local solutions, sampling chains can inadequately explore posterior regions, and simulation code can implement different update rules from those described in the paper. A reproducible comparison therefore preserves data processing, generative simulation, estimation routines, parameter bounds, random seeds, held-out assignments, predictive scores, and diagnostic outputs. Reporting only a final table of winner scores hides the steps that determine their meaning. Wilson and Collins discuss the importance of practical checks throughout computational behavioral modeling Wilson & Collins (2019) Schad et al. (2021).
Uncertainty in model rankings should be evaluated with the same care as parameter uncertainty. Comparing point estimates of expected predictive accuracy overlooks variability across observations, tasks, and participants. Statistical procedures for aggregating predictive differences depend on the independence unit and the particular resampling scheme. Where fitting is computationally unstable, the same analysis may return different model rankings across runs. Repeated fits and simulation-based diagnostics can distinguish numerical instability from genuine evidence of similarly predictive models. Approximations, such as variational Bayes or importance reweighting, require documented validation of their own accuracy Vehtari et al. (2017) Dao et al. (2024).
A complete audit trail links each theoretical claim to the operation that tests it. It records which experimental signature motivated comparison, which datasets entered fitting, what information was visible during prediction, and what generated the reported uncertainty. Research teams should retain code, metadata, and provenance for models as well as for empirical measurements. The methodological norm is especially valuable when a model's predictive ranking becomes a basis for individual assessment or a comparison of biological and artificial agents. Reproducibility safeguards the interpretation of observed performance and the possibility of independent critical reanalysis Shiffrin et al. (2008) Schad et al. (2021).
A Worked Research Protocol for Competing Learning Models
Consider a study comparing a reward-learning model, a planning model, and a model that supplements learning with working memory. The research question should name the intended psychological contrast: whether prospective knowledge of transitions and short-term capacity explain choices beyond incremental reinforcement updates. Before data collection, researchers can specify the experimental sequence, candidate update equations, parameter constraints, lapse rules, and joint observation model for choices and any recorded latencies. These decisions determine what constitutes a fair comparison. Published studies of model-based choice and memory contributions illustrate how such candidates can be constructed Daw et al. (2011) Collins & Frank (2012).
A preregistered computational assessment can begin with simulated datasets sampled over realistic parameter regimes and plausible numbers of trials and participants. All candidates are fit to each generated dataset, producing a recovery matrix and estimates of how reliably transition manipulations distinguish the theories. The protocol next evaluates each model's capacity to generate the target qualitative signature. When empirical data arrive, fitting is performed using the declared training units, predictive scoring is computed on truly held-out units, and posterior predictive checks compare simulated trajectories with observed learning and response patterns Wilson & Collins (2019) Wagenmakers et al. (2004).
The resulting scientific report would include relative predictive scores with uncertainty, meaningful calibration and distribution checks, recovery rates, sensitivity to lapse assumptions, and transfer to independent participants or changed contingencies. It would separately discuss how latent parameter estimates relate to psychological constructs. A winning model might establish a reliable predictive advantage for the studied task, while the mechanism claim could remain tentative if a competitor predicts the same key manipulation. This protocol is illustrative methodological guidance, not an assertion that new data or numerical results were collected for this monograph Navarro et al. (2004) Palminteri et al. (2017).
Critical Scientific Discussion: What the Evidence Supports
The literature supports a consistent methodological conclusion: an observed advantage in fit gains scientific force when its source is understood and its predictions are tested under credible conditions. Akaike and Schwarz supplied influential complexity-adjusted selection rules; Stone linked model selection to prediction; cognitive scientists demonstrated that geometric flexibility, observational mimicry, and design informativeness complicate raw fit rankings. Bayesian predictive evaluation, posterior checks, and recovery simulations now provide complementary methods of examining these problems. Their combined value lies in exposing assumptions behind comparative statements, not in supplying one universally decisive score Akaike (1974) Schwarz (1978) Navarro et al. (2004).
Evidence is uneven across applications and inferential targets. The mathematical properties of a criterion can be established under formal conditions, whereas a cognitive claim must survive the actual sampling process, psychometric quality, and experimental design. A method performing well in simulation is evidence for the simulated cases; a published model comparison is evidence about the tasks and participants studied. Even successful held-out prediction may fail to identify a unique mechanism. The study of model mimicry and cognitive falsification demonstrates why experiments that create divergent predictions are essential for mechanism-oriented science Wagenmakers et al. (2004) Palminteri et al. (2017).
The central methodological tension concerns the relationship between predictive optimization and theoretical restriction. Highly flexible models can anticipate complex data but may offer weak constraints on psychologically important counterfactuals. Restricted process models may predict fewer phenomena well while expressing stronger causal or mechanistic commitments. Neither advantage can be inferred from the complexity label alone. The evaluation must establish what question is being asked, which alternatives are available, what prediction target is relevant, and what evidence would change the interpretation. This is a discipline of explicit scientific commitments Roberts & Pashler (2000) Yarkoni & Westfall (2017).
Unresolved Research Problems and Future Directions
Several open problems concern validation across changes of scale and context. Psychological studies routinely combine many trials with modest participant samples, and computational models integrate choice, time, physiology, and narrative report at different levels of dependence. A robust evaluation must decide which dependencies belong in the likelihood, how uncertainty should be propagated, and how predictive performance transfers to entirely new people or environments. Developing more reliable validation procedures for hierarchical and sequential models remains an important area of research, particularly when direct refitting is computationally demanding Dao et al. (2024) Vehtari et al. (2017).
Another problem is how to evaluate model misspecification and the incompleteness of candidate sets. A candidate family can yield stable model rankings while excluding a process that becomes visible under a new task. Simulation-based recovery, targeted falsification, posterior predictive checks, and stacked predictive distributions offer partial answers. Their conclusions depend on which alternatives were simulated or included. Future work can improve systematic perturbation of candidate mechanisms, transparent assessment of parameter-level reliability, and integrated design strategies that prioritize observations with high theoretical discrimination Yao et al. (2018) Palminteri et al. (2017).
Research on artificial cognitive systems expands the comparison problem further. Different histories of training, embodiment, memory access, and external information can produce apparently similar behavior while changing what a prediction means. Future studies should identify matched and unmatched informational conditions, specify reproducible benchmark tasks, test causal interventions where feasible, and examine whether computational descriptions transfer across systems. The scientific objective is to develop comparable evidence about functional organization while preserving the limits of psychological inference. These problems connect mathematical psychology to wider debates about explanation, generalization, and the changing subject matter of psychology in the Artificial Era Yarkoni & Westfall (2017) Wilson & Collins (2019).
Conclusion
Computational model comparison is a structured scientific procedure for adjudicating among mathematical accounts of psychological observations. Its basic ingredients are explicit generative hypotheses, a defensible observation model, a declared predictive or evidential target, and an assessment of the uncertainty arising from limited and dependent data. Complexity-adjusted criteria, Bayesian marginal likelihood, cross-validation, and proper predictive scoring each measure particular aspects of a comparison. Their interpretation follows from their assumptions and intended purpose. A criterion that ranks candidate models does not automatically establish an exclusive psychological mechanism Akaike (1974) Gneiting & Raftery (2007) Shiffrin et al. (2008).
The most informative programs combine relative selection with absolute adequacy checks, model-recovery simulations, and experimental conditions designed to expose contrasting predictions. Posterior predictive discrepancies identify observed features a model struggles to generate. Parametric-bootstrap cross-fitting and recovery matrices quantify how often candidate accounts can be mistaken for one another. Independent held-out and transfer assessments show whether a predictive advantage persists beyond calibration. Taken together, these methods distinguish descriptive fit, predictive generalization, and mechanistic discrimination in a form that can be scrutinized and reproduced Gelman et al. (1996) Navarro et al. (2004) Wagenmakers et al. (2004) Wilson & Collins (2019).
The resulting scientific synthesis establishes a durable standard of interpretation: each computational comparison must state its competing models, population, task, prediction unit, data exclusions, estimation procedure, uncertainty, diagnostic simulations, and the counterfactual evidence that would challenge the favored explanation. Research on biological psychology, cognitive development, and artificial cognitive systems can employ this common logic while adapting experimental measures to their subjects. Psychological understanding advances when formal explanations produce calibrated, reproducible, and experimentally discriminable consequences across the conditions for which their claims are made Palminteri et al. (2017) Schad et al. (2021) Yarkoni & Westfall (2017).
References
- Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705
- Bach, D. R. (2025). Experiment-based calibration in psychology: Foundational and data-generating model. Journal of Mathematical Psychology, 127, Article 102950. https://doi.org/10.1016/j.jmp.2025.102950
- Collins, A. G. E., & Frank, M. J. (2012). How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. European Journal of Neuroscience, 35(7), 1024–1035. https://doi.org/10.1111/j.1460-9568.2011.07980.x
- Dao, V. H., Gunawan, D., Tran, M.-N., Kohn, R., Hawkins, G. E., & Brown, S. D. (2024). Efficient selection between hierarchical cognitive models: Cross-validation with variational Bayes. Psychological Methods, 29(1), 219–241. https://doi.org/10.1037/met0000458
- Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron, 69(6), 1204–1215. https://doi.org/10.1016/j.neuron.2011.02.027
- Gelman, A., Meng, X.-L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6(4), 733–760. https://www.jstor.org/stable/24306036
- Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. https://doi.org/10.1198/016214506000001437
- Gronau, Q. F., Singmann, H., & Wagenmakers, E.-J. (2020). bridgesampling: An R package for estimating normalizing constants. Journal of Statistical Software, 92(10), 1–29. https://doi.org/10.18637/jss.v092.i10
- Myung, I. J., Balasubramanian, V., & Pitt, M. A. (2000). Counting probability distributions: Differential geometry and model selection. Proceedings of the National Academy of Sciences, 97(21), 11170–11175. https://doi.org/10.1073/pnas.170283897
- https://doi.org/10.1016/j.cogpsych.2003.11.001
- Palminteri, S., Wyart, V., & Koechlin, E. (2017). The importance of falsification in computational cognitive modeling. Trends in Cognitive Sciences, 21(6), 425–433. https://doi.org/10.1016/j.tics.2017.03.011
- Piironen, J., & Vehtari, A. (2017). Comparison of Bayesian predictive methods for model selection. Statistics and Computing, 27(3), 711–735. https://doi.org/10.1007/s11222-016-9649-y
- Pitt, M. A., & Myung, I. J. (2002). When a good fit can be bad. Trends in Cognitive Sciences, 6(10), 421–425. https://doi.org/10.1016/S1364-6613(02)01964-2
- Roberts, S., & Pashler, H. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358–367. https://doi.org/10.1037/0033-295X.107.2.358
- Schad, D. J., Betancourt, M., & Vasishth, S. (2021). Toward a principled Bayesian workflow in cognitive science. Psychological Methods, 26(1), 103–126. https://doi.org/10.1037/met0000275
- Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. https://doi.org/10.1214/aos/1176344136
- Shiffrin, R. M., Lee, M. D., Kim, W., & Wagenmakers, E.-J. (2008). A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods. Cognitive Science, 32(8), 1248–1284. https://doi.org/10.1080/03640210802414826
- Stone, M. (1977). An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 44–47. https://doi.org/10.1111/j.2517-6161.1977.tb01603.x
- Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. https://doi.org/10.1007/s11222-016-9696-4
- Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48(1), 28–50. https://doi.org/10.1016/j.jmp.2003.11.004
- Watanabe, S. (2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11, 3571–3594. https://www.jmlr.org/papers/v11/watanabe10a.html
- Wilson, R. C., & Collins, A. G. E. (2019). Ten simple rules for the computational modeling of behavioral data. eLife, 8, Article e49547. https://doi.org/10.7554/eLife.49547
- Yao, Y., Vehtari, A., Simpson, D., & Gelman, A. (2018). Using stacking to average Bayesian predictive distributions (with discussion). Bayesian Analysis, 13(3), 917–1007. https://doi.org/10.1214/17-BA1091
- Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12(6), 1100–1122. https://doi.org/10.1177/1745691617693393