RESEARCH DOSSIER
Benchmark Contamination and Leakage
Abstract. Benchmark contamination and leakage can compromise scientific inference from artificial-system test scores to generalizable cognitive competence. This fundamental critical narrative research monograph reconstructs the foundations of information leakage in predictive modeling and psychometrics, classifies exposure through pretraining, fine-tuning, retrieval, prompt templates, and evaluator feedback, and synthesizes original research on corpus inspection, black-box memorization probes, longitudinal tests, and controlled contamination interventions. The analysis distinguishes evidence of exposure, recognition, causal score distortion, and ranking change; evaluates methodological weaknesses of individual detection families; and compares prevention through deduplication, independent holdouts, benchmark refresh, and governance. A versioned audit protocol connects construct validity and artificial cognitive assessment to reproducible testing in the Artificial Era.
Keywords: benchmark contamination; data leakage; test-set contamination; large language model evaluation; memorization; construct validity; evaluation validity; benchmark overfitting; black-box contamination detection; dynamic benchmarks; artificial cognition; Psychology of Artificial; Artificial Era
Benchmark Contamination as a Scientific Problem
Benchmark contamination is the presence of evaluation-relevant information in a system’s development history in a manner capable of compromising inference from a test score to competence on appropriately unseen tasks. In language-model evaluation, a question, reference answer, solution, annotation, or recognizable transformation can reach a model through pretraining, supervised fine-tuning, preference data, synthetic training examples, retrieval, or evaluator feedback. The scientific object is the causal dependency between that exposure and the observed score. A benchmark can be exposed without being remembered, and remembered without producing a measurable score change. Accordingly, contamination must be investigated through exposure pathways, behavioral evidence, and the validity of the intended inference Kaufman et al. (2012) Bordt et al. (2025).
The central measurement question is what an observed response establishes about a configured artificial system. A model answering a public mathematics item may have computed a solution, recalled a previously encountered answer, used a retrieved document, or combined all three. An identical final response cannot distinguish these histories. Psychology contributes a construct-validity vocabulary: the observable item score is an indicator, whereas the proposed competence—reasoning, knowledge, generalization, or adaptive problem solving—is an inferred construct. Cronbach and Meehl’s account of construct validity and later causal discussions of validity demand evidence linking the indicator to its intended attribute Cronbach & Meehl (1955) Borsboom et al. (2004).
Contamination and leakage must be assessed at the level of the evaluation pipeline, not only the model weights. An agent with a pristine checkpoint can still encounter answer keys in retrieval; a model may receive test-set exemplars inside a prompt; a public leaderboard can function as repeated feedback for prompt or model selection. Such pathways have different timing, agents, and mechanisms. The object of investigation is therefore a model-or-agent configuration together with its dataset provenance, prompts, external memory, scoring interface, and development decisions Kaufman et al. (2012) Cawley & Talbot (2010) Liang et al. (2023).
This monograph is a critical narrative synthesis of primary studies and methodological literature, with special attention to laboratory interventions, black-box probes, longitudinal natural experiments, and benchmark design. It asks which kinds of leakage are measurable, when exposure materially changes performance, how findings transfer across tasks and versions, and how evaluation can support psychological claims about artificial cognition. The conclusions are evidentiary and conditional on test design; they do not claim an original empirical experiment or exhaustive systematic search Oren et al. (2024) Samuel et al. (2025).
Historical Foundations of Leakage and Selection Bias
Concerns about information leakage preceded foundation models. In predictive modeling, a feature may reveal the future outcome because it was produced after the event to be predicted, or a training procedure may inadvertently use information from the nominal test partition. Kaufman, Rosset, Perlich, and Stitelman formulated leakage as access to target-relevant information unavailable in legitimate deployment. Their 2012 analysis located errors in data preparation, temporal ordering, feature construction, and evaluation protocols; the general principle applies directly to modern model assessments Kaufman et al. (2012).
A second lineage concerns adaptively selecting methods using noisy validation estimates. Cawley and Talbot demonstrated that model selection may overfit its selection criterion and that reporting the same data-driven result as an independent estimate introduces bias. Varma and Simon similarly analyzed optimistically biased error estimates when cross-validation is used for parameter selection without an appropriately separated outer assessment. The relevant lesson for contemporary benchmark leaderboards is that an evaluation set ceases to serve as an untouched test when researchers repeatedly inspect outcomes and optimize against it Cawley & Talbot (2010) Varma & Simon (2006).
The modern public benchmark ecosystem changes the scale of the problem. Hendrycks and colleagues introduced MMLU to cover 57 academic and professional subject areas; Cobbe and colleagues introduced GSM8K to evaluate grade-school mathematical reasoning. Their original studies specified tasks, splits, and evaluation conventions, providing reusable standards for comparative investigation. Because benchmark items, solutions, and discussions circulate on the open web, these same standards become potential material for later model-training corpora. A valid historical comparison must therefore record both the publication of a benchmark and the training or update cutoff of each evaluated system Hendrycks et al. (2021) Cobbe et al. (2021).
Foundation-model evaluation acquired multi-scenario measurement frameworks as model capacities broadened. The Holistic Evaluation of Language Models program emphasized that accuracy alone cannot represent calibration, robustness, fairness, and other dimensions of performance; consistent scenarios and transparent prompting are essential for comparisons. Under contamination, those dimensions interact: a model may appear more accurate on familiar items while showing poor uncertainty awareness or little transfer to paraphrases. The history of evaluation is thus simultaneously a history of measurement design, public reuse, and adaptive pressure on benchmarks Liang et al. (2023) Cronbach & Meehl (1955).
A Taxonomy of Exposure and Leakage Pathways
Exact-item contamination occurs when benchmark text or its answer appears verbatim in a training corpus, fine-tuning set, cache, or supplied context. Near-duplicate contamination involves formatting changes, paraphrase, punctuation variation, or copied solutions with altered wording. Semantic contamination carries substantially equivalent task information through translations, explanatory articles, generated variants, or answer rationales; task-level contamination supplies reusable task conventions or solution templates. Each category should be reported with the artifact exposed, its provenance, the stage of exposure, and the predicted mechanism by which it could change an evaluation Deng et al. (2024) Yao et al. (2024).
Training-time pathways include web-scale pretraining, supervised instruction examples, evaluation-generated synthetic data, reward or preference comparisons, and distillation from a model that has already seen the item. Inference-time pathways include retrieval-augmented generation, browsing, cached answers, search snippets, in-context examples, and tool results carrying test content. Development-time pathways include repeated selection of checkpoints or prompts against one public score. These categories overlap in practice but demand different remedies: corpus filtering addresses ingestion, access controls address retrieval, and independent holdouts address adaptive evaluation Kaufman et al. (2012) Cawley & Talbot (2010) Chen et al. (2025).
One particularly consequential distinction is between answer leakage and familiarization with the task format. If a model has learned a benchmark’s conventional answer schema, it may answer without having seen any individual item; the observed improvement can represent legitimate task learning, overfitting to benchmark-specific conventions, or a mixture. Li and Flanigan’s chronological analysis of task contamination emphasizes that evaluation histories must consider familiarity with task distributions alongside literal overlap. Researchers should specify whether their target estimand is novelty of particular items, novelty of task families, or transferable competence under new conditions Li & Flanigan (2024).
The scientific severity of each pathway depends on the intended claim. A benchmark intended to measure information retrieval may legitimately reward locating public information if retrieval is part of the declared task. The same access would violate a benchmark intended to measure unassisted knowledge in frozen model weights. Similarly, memorizing spelling can be functional knowledge in a language task, while memorizing the answer to a supposedly unseen reasoning problem can defeat the intended generalization test. Leakage is identified relative to an explicit task boundary, permissible resources, and evaluation estimand Borsboom et al. (2004) Kaufman et al. (2012).
Construct Validity and Psychological Interpretation of Scores
A psychological interpretation begins with a definition of the target construct. For example, mathematical reasoning might require selecting operations, tracking intermediate quantities, and reaching correct results under changes in surface context. If a public item can be recalled by recognizing its exact wording, the score supports a narrower proposition: performance on a known input in a specified setting. Construct validity requires additional evidence from novel forms, difficulty gradients, error patterns, process-sensitive interventions, and transfer conditions. A high score alone is insufficient to identify a unique cognitive mechanism Cronbach & Meehl (1955) Borsboom et al. (2004).
Convergent evidence arises when independent task families designed to recruit similar capacities show coherent performance patterns, after controlling for shared exposure. Discriminant evidence arises when tests designed to isolate different capacities produce the predicted differences. A model scoring well on a memorized benchmark and poorly on freshly constructed equivalents raises a question about the validity of the first indicator. Yet a mismatch alone does not prove memorization: new items may differ in difficulty, ambiguity, distribution, language, or scoring reliability. Matched-item designs are therefore central to separating exposure from task shift Dekoninck et al. (2024) Sun et al. (2025).
Artificial systems complicate the classical unit of analysis. A checkpoint, prompted assistant, retrieval-augmented workflow, and tool-using agent can share a model name while differing in available evidence and action policy. Their scores estimate different constructs and must not be treated as exchangeable observations. Evaluation records should identify model version, prompt hierarchy, decoding settings, context window, retrieval state, tool permissions, scoring code, and any human intervention. This configuration-specific definition follows general principles of transparent holistic evaluation Liang et al. (2023).
Claims about general intelligence, understanding, or artificial psychological organization require a coherent chain from operational manipulation to observation and interpretation. Evidence of test-set recognition supports a statement about possible memory of benchmark artifacts. Evidence of improved accuracy after injected contamination supports a causal effect of a particular exposure regime. Evidence of stable accuracy on fresh, causally matched problems supports a stronger generalization claim. None of these alone establishes a theory of subjective experience; the present dossier instead contributes an evidence standard for studying artificial behavior in the Artificial Era Kocyigit et al. (2025) Bordt et al. (2025) Cronbach & Meehl (1955).
Evidence from Direct Corpus and Dataset Inspection
Direct inspection is the strongest route to documenting the presence of benchmark artifacts in accessible training data. Investigators can hash normalized strings, identify exact shared spans, calculate n-gram overlap, retrieve semantically similar passages, and trace duplicates through source archives. Such audits should record whether the overlap concerns the question, answer, explanation, surrounding metadata, or entire test file. They should also preserve which training-stage corpus was searched and the temporal order between the creation of each artifact and a model checkpoint. A confirmed match demonstrates an exposure opportunity; causal influence on a specific model’s response requires additional evidence Deng et al. (2024) Li et al. (2024).
Source accessibility is uneven. Open-model releases sometimes identify broad mixtures without publishing complete document-level provenance; closed providers may disclose only training cutoffs or coarse categories. A negative search through public Common Crawl snapshots cannot exclude presence in proprietary instruction data, private uploads, model-distillation outputs, or a retrieval cache. Conversely, high text similarity between common factual questions and web articles can reflect the legitimate recurrence of shared subject matter. Consequently, audit reports must specify the denominator of searchable material, retrieval threshold, false-positive controls, and unknown exposure channels Deng et al. (2024) Samuel et al. (2025).
An open-source contamination report by Li and colleagues examined more than fifteen models on six multiple-choice question-answering benchmarks and found heterogeneous indicators of contamination. Its comparative contribution lies in showing that contamination cannot be treated as a constant attached to the benchmark alone. The finding depends on the specific model and detection method, and the authors’ estimates are method-specific measurements rather than a census of all training overlaps. Researchers should retain that distinction when comparing reported contamination rates across studies Li et al. (2024).
A useful forensic workflow separates a provenance finding from a behavioral finding. First, identify a benchmark item and search auditable training sources; second, confirm that the overlap preceded a relevant training or deployment stage; third, estimate a behavioral difference using matched clean items or controlled insertion; fourth, document remaining alternative explanations. Direct matches strengthen the inference of exposure, and controlled behavioral differences strengthen the inference of score distortion. This layered protocol is especially important when a single detection score appears decisive but the underlying mechanism remains underdetermined Kaufman et al. (2012) Kocyigit et al. (2025).
Black-Box Detection through Order, Completion and Recognition
Black-box evaluation is necessary when model weights and training records are unavailable. Oren and colleagues proposed a statistical contamination test based on the order of benchmark examples: under exchangeability, ordering permutations should have comparable likelihood, whereas memorized canonical sequences can become disproportionately probable. Their ICLR 2024 paper develops formal false-positive control under its assumptions and demonstrates sensitivity in controlled settings. Its inference is specific to order-dependent memorization and exchangeable test construction; an unsorted or transformed dataset may evade the signal Oren et al. (2024).
Deng and colleagues introduced Testset Slot Guessing alongside corpus retrieval. In one protocol, a multiple-choice question has an unlikely distractor concealed and the model predicts the missing option; in another, a low-predictability word is masked. The motivation is that answering the underlying knowledge question is not necessary to recover a peculiar test artifact, so unusually accurate recovery is evidence consistent with exposure. Their NAACL 2024 study reported exact-match recovery for some MMLU missing options in commercial models. This observation is a diagnostic indicator of benchmark familiarity, not an independently verified record of the specific training document Deng et al. (2024).
Golchin and Surdeanu developed the Data Contamination Quiz, or DCQ, presenting original instances alongside word-level perturbations and a none-of-the-above option. Because the alternatives are intended to be equally plausible except for wording associated with an encountered instance, preferential recognition of originals can reveal memorization signals. The 2025 TACL study includes controlled and observational evaluations and accounts for position bias. However, original instances can differ subtly from perturbations in naturalness, fluency, or likelihood; sound inference requires calibrated negative controls and matched perturbation quality Golchin & Surdeanu (2025).
These black-box tools answer related but different questions. A canonical-order likelihood test is sensitive to sequence memorization; slot guessing probes memorized improbable surface details; forced-choice quizzes probe discrimination of originals from variants. Their assumptions about API access, token likelihoods, item independence, and available prompts vary. Agreement across independent probes can strengthen a diagnosis; disagreement calls for analysis of the targeted memory signature before concluding that one method failed. Modern evaluations explicitly document inconsistencies among prominent detection techniques, cautioning against treating any one oracle as definitive Samuel et al. (2025).
Performance-Based and Perturbation-Based Contamination Tests
Performance-based tests examine the capability to transfer from familiar benchmark items to task-equivalent alternatives. ConStat, proposed by Dekoninck and colleagues, reframes contamination in terms of artificially inflated and poorly generalizing benchmark performance and compares the target benchmark against reference tests and reference models. This approach directly addresses distortion of a measured ability rather than merely detecting training overlap. Its validity depends on matching difficulty, task demands, and the reference models’ exposure histories; otherwise an observed score gap may conflate contamination with distribution shift Dekoninck et al. (2024).
Perturbation families must be chosen according to the construct under test. For an arithmetic word problem, investigators can change quantities and recompute the gold answer while preserving the relevant operation structure. For a factual recall question, substituting an entity may alter the underlying knowledge demand. For a reading-comprehension question, changing an irrelevant surface word can leave semantics invariant, whereas changing a negation reverses the proposition. The evaluator should justify whether each transformation preserves difficulty, answerability, and the inferential process the benchmark was intended to measure Sun et al. (2025) Borsboom et al. (2004).
Yao and colleagues experimentally demonstrated that contamination can cross language boundaries: translations of benchmark test examples can be used to contaminate evaluation while evading simple textual overlap checks. Their work illustrates why same-language string matching alone provides inadequate coverage. Deep changes in wording may retain answer-relevant information, while an exact string may be exposed without any detectable downstream performance advantage. A substantive contamination audit should therefore combine surface matching with controlled semantic transformations and tests of genuinely unseen generalization Yao et al. (2024).
A carefully interpreted performance gap is conditional evidence. Original-minus-paraphrase accuracy becomes more persuasive if item pairs are validated, raters agree on equivalent difficulty, scoring is blinded, negative-control models show small or balanced differences, and contamination is injected at known doses. Where original and rewritten questions differ in syntax, cultural knowledge, instruction length, or implicit cues, the gap may reflect ordinary prompt sensitivity. Performance-based methods should report uncertainty over paired items, reproducible generation rules, and alternative explanations instead of mechanically subtracting a putative contamination correction Samuel et al. (2025) Sun et al. (2025).
Controlled Injection as Causal Evidence
The most decisive evidence about a contamination mechanism comes from controlled insertion into an otherwise documented training pipeline. A researcher begins with a clean or comparatively clean split, assigns exposure conditions, controls training volume and compute, and measures accuracy on the original benchmark and independently composed alternatives. By manipulating the dose and format of exposure, the study can estimate causal score inflation under a specified architecture and learning regime. The result explains that experimental system; extension to other providers requires explicit argument about data, model capacity, and training dynamics Kocyigit et al. (2025) Bordt et al. (2025).
Kocyigit and colleagues studied machine translation with decontaminated splits and systematically introduced source-target contamination across formats and training conditions. Their ICML 2025 findings showed that including both sides of test translations can substantially inflate BLEU, with stronger inflation at the larger tested model scale under their controlled protocol. This result demonstrates that contaminated exposure can change an aggregate evaluation metric, and also that the magnitude depends on data format and model configuration. It should not be converted into a universal inflation coefficient for unrelated tasks Kocyigit et al. (2025).
Bordt and colleagues examined how benchmark overfitting scales with model parameters, frequency of exposure, and subsequent training tokens. Their ICML 2025 study qualified the commonplace claim that any small overlap invalidates every score: memorization and score influence can recede as training continues under some regimes. This does not make provenance irrelevant. It establishes that contamination is an intervention with dose, timing, retention, and task-dependent consequences, and that a binary seen-versus-unseen indicator captures only part of the causal process Bordt et al. (2025).
Experimental conclusions are strongest when the outcome includes both benchmark accuracy and a broader validity profile. Investigators can observe whether exposed items become easier, whether altered answer keys cause erroneous recall, whether novel items of the same type improve, and whether gains persist after additional training. A design with clean controls, staged measurements, matched capacity, and transparent training recipes can distinguish memorization, generalized learning from examples, and transient exposure effects. It also permits estimated uncertainty, rather than inferring causal harm merely because a corpus contains recognizable strings Kocyigit et al. (2025) Bordt et al. (2025) Cawley & Talbot (2010).
Benchmark Lifecycles, Training Cutoffs and Temporal Evidence
A contamination risk develops across the lifecycle of a benchmark. Authors design items, collect reference answers, publish examples, release code, receive public solutions, and allow models to be trained on subsequent web snapshots. Later benchmark updates may be based on the same examples, making their substantive independence uncertain. Every evaluation should record dataset version, source creation date, public release date, answer publication date, model training cutoff when known, and actual test date. These timestamps support useful constraints but cannot by themselves reveal every ingestion event Roberts et al. (2024) Chen et al. (2025).
Roberts and colleagues used a longitudinal natural experiment involving problems associated with Codeforces and Project Euler and compared model performance with measures of online exposure and problem release timing. Their ICLR 2024 work identified performance patterns consistent with contamination and demonstrated the value of chronological evidence. A temporal discontinuity can also reflect changing problem difficulty, recency of general knowledge, or format shifts. The method becomes stronger with matched difficulty controls and plausible mechanisms linking publicity to model access Roberts et al. (2024).
The cutoff argument is asymmetric. Material released after a verified frozen checkpoint cannot have been part of that checkpoint’s original training data, but could reach a later post-training process, retrieval component, or an updated service. Material predating a cutoff might have been absent from the actual training mixture. Model-provider cutoffs are often broad administrative statements, whereas scientific comparisons need dated checkpoints and explicit evidence-access boundaries. When exact provenance is missing, the audit should report exposure as possible or constrained, not as empirically established Roberts et al. (2024) Deng et al. (2024).
Because public benchmark results influence model development, the same test can gradually change from an independent measure into a development target. Chen and colleagues’ survey of static and dynamic evaluation describes how this pressure motivates periodic refreshing and other moving-target designs. A responsible longitudinal record should preserve past versions for historical comparison while adding private, independently generated contemporary items for fresh inference. Score changes across different benchmark vintages then require equating or calibration before being presented as progress in the same construct Chen et al. (2025) Liang et al. (2023).
Dynamic, Private and Continuously Renewed Evaluation
Dynamic benchmarking attempts to reduce exposure by deriving questions from recent data, generating new problem instances, rotating held-out pools, or maintaining private answer keys. LiveBench proposes frequent updates from recent sources with objective scoring across mathematics, coding, language, instruction following, reasoning, and other tasks. Its strongest general principle is to design evaluation renewal as a continuing process and to publish transparent scoring procedures. However, public release ultimately creates future exposure, so the claim of being uncontaminated is time-indexed and must be re-evaluated for each new model White et al. (2024).
A renewed benchmark can still contain familiar underlying facts or tasks. New wording alone does not guarantee independence if the answer follows from a widely memorized solution or if the generation template reproduces a known public item. AntiLeak-Bench and C2LEVA illustrate research efforts to use automated item renewal or controlled disclosure to lower leakage risks. Their contribution is primarily benchmark engineering; independently evaluating question quality, difficulty stability, item uniqueness, and the ability to detect contamination remains necessary Wu et al. (2025) Li et al. (2025).
Private evaluation is powerful only when access is genuinely constrained. An item may leak through sample demonstrations, grading services, participant logs, agent traces, human annotation, or scores released at sufficiently fine granularity. Restricting the exact item corpus, rotating keys, controlling query budgets, and auditing access histories are complementary safeguards. Private benchmarks also create tradeoffs with reproducibility: researchers may be unable to inspect item validity or replicate failures. A credible governance plan therefore separates public test specifications from protected item data and permits independent auditing under controlled conditions Chen et al. (2025) Liang et al. (2023).
Dynamic tasks often trade contamination resistance for comparability. If new questions differ in domain coverage and difficulty, scores from different months may not estimate the same performance variable. Item response modeling, anchor tasks with monitored exposure, matched generation blueprints, and repeated calibration panels can help. Scores should identify the exact benchmark vintage, item count, scoring rubric, system version, and uncertainty. The goal is to maintain sensitivity to learning and capability change while preventing a publicly available answer key from becoming the hidden explanation for an apparently stable ranking Cronbach & Meehl (1955) White et al. (2024).
Leakage Through Retrieval, Tools and Evaluator Feedback
An artificial assistant may obtain benchmark-specific information during inference rather than training. A retrieval index can contain complete benchmark answer sheets; a browser can locate published solutions; an orchestration layer may cache responses from earlier tests. Under these conditions the result measures the complete configured system, including environmental access and its search policy. Researchers need access logs identifying retrieval queries, returned documents, timestamps, and tool outputs, together with a protocol defining whether open-resource assistance is legitimate Kaufman et al. (2012) Liang et al. (2023).
A common failure is to compare a tool-using system with an unassisted model as though both performed the same psychological task. If the scientific construct is unaided mathematical inference, reading a supplied answer changes the intended measurement. If the construct is effective information-seeking, locating and validating external evidence may be relevant competence. The primary problem is therefore an under-specified task contract rather than the mere existence of external information. A clear evaluation should report assisted and unassisted conditions separately and record when the assistant crossed the information boundary Borsboom et al. (2004) Kaufman et al. (2012).
Evaluation pipelines can themselves transmit answers. Automated judges may be prompted with reference solutions, hidden rubrics may be revealed through repeated interactive feedback, and leaderboards may allow thousands of small optimizations against a nominally held-out set. Adaptive overfitting arises when information from earlier measurements changes later candidates, even if raw questions are never copied into training. The statistical literature on selection bias demonstrates why the final reported estimate must be obtained on a newly reserved sample independent of model and prompt selection Cawley & Talbot (2010) Varma & Simon (2006).
Practical logging should include the full input payload, system and developer instructions where auditable, retrieval index version, cache state, context replay, agent-tool configuration, evaluation harness commit, and scorer behavior. Logging must respect proprietary information and privacy, but the absence of a complete record must be acknowledged when attributing outcomes. For scientific reports about artificial behavior, it is essential to distinguish model memory, in-context availability, external memory, and human-supplied hints. They constitute different explanatory mechanisms and suggest different follow-up experiments Liang et al. (2023) Deng et al. (2024).
Contamination of Human and Artificial Behavioral Experiments
Psychological experiments conducted with artificial systems inherit classic testing concerns and introduce new ones. A model can encounter standardized cognitive puzzles, response keys, personality questionnaires, or well-known reasoning tasks within web-scale training. Reusing these materials as if they were genuinely novel stimuli risks mistaking familiarity with the test for a stable psychological disposition. Construct validity requires defining the phenomenon under study and testing whether responses generalize to novel item families and experimentally manipulated contexts Cronbach & Meehl (1955) Borsboom et al. (2004).
Human participants also learn from test exposure, and repeated testing can change performance through practice. The difference in artificial evaluation is the availability of large-scale pretraining, software cloning, rapid prompt iteration, and retrieval from public task descriptions. These mechanisms can create correlations across supposedly independent trials. In experiments involving stochastic language models, multiple samples from the same checkpoint are repeated observations of one configured system, not a population of unrelated artificial agents. Statistical uncertainty must account for item and system clustering Liang et al. (2023) Cawley & Talbot (2010).
A contaminated task can distort an explanatory comparison. Suppose investigators claim that an artificial system has superior analogical reasoning because it outperforms a human group on a public analogy test. The design must account for normative answer availability, language proficiency, exposure histories, prompting, response timing, and task instructions. Novel analogies generated under blinded rules can supply complementary evidence, but they must be piloted to validate their equivalence. Generalization claims are strongest when independent task developers and evaluators test multiple task families without iterative tuning to the same holdout Sun et al. (2025) Samuel et al. (2025).
Behavioral evaluation in the Artificial Era therefore needs a versioned experimental unit. A study can compare a specific checkpoint with humans under transparent constraints and report meaningful behavioral differences. It should resist inferring a stable, context-free psychological property from one benchmark. The method proposed by this synthesis is to triangulate public standardized tasks, controlled freshly generated tasks, process tracing, and perturbation tests. This is a methodological recommendation based on existing validity principles and contamination findings, rather than an assertion that a new general psychological scale has already been validated Liang et al. (2023) Cronbach & Meehl (1955).
Empirical Findings and the Limits of Comparative Inference
The evidence establishes that contamination can distort results, while also showing substantial heterogeneity. Controlled machine-translation experiments demonstrate metric inflation after source-target exposure; controlled scaling studies demonstrate that the relationship between exposure and retention changes with subsequent training. Black-box detection work reports recognition of benchmark-specific text, but those indicators cannot directly reconstruct hidden training corpora. The synthesis therefore distinguishes established contamination effects in documented experimental settings from model-specific exposure allegations based only on behavioral clues Kocyigit et al. (2025) Bordt et al. (2025) Deng et al. (2024).
Different studies operationalize contamination differently. Some measure overlap between test text and accessible corpora, some detect a memorized token or item ordering, and others quantify loss of performance after perturbation. A corpus overlap rate, quiz hit rate, and score inflation estimate cannot be pooled as though they share a common denominator. They may be positively associated under some mechanisms and dissociate under others. A rigorous comparison states the exposure unit, detection threshold, target population, sampling scheme, and causal inference associated with each number Li et al. (2024) Golchin & Surdeanu (2025) Dekoninck et al. (2024).
Samuel, Zhou, and Zou compared several detection methods on challenging contemporary benchmarks and found limited consistency among approaches, including difficulties with contamination introduced through instruction tuning and answer augmentation. Their findings are a warning about detector portability. A method validated on older, smaller models and exact text overlap can have poorly understood sensitivity to translated, paraphrased, or distilled exposure in a newer system. Published claims should specify the tested model versions and avoid projecting method performance outside its original validation Samuel et al. (2025) Yao et al. (2024).
A benchmark leaderboard aggregates item outcomes across potentially different exposure histories. If every model has a comparable advantage on familiar items, absolute accuracy can be inflated even when relative rankings change little. If some models receive much larger exposure, rankings may change. These are distinct empirical questions and require separate data: paired item responses, source exposure evidence, and uncertainty around rank differences. A defensible comparison presents original and contamination-resistant results side by side rather than applying one unvalidated adjustment to every leaderboard entry Dekoninck et al. (2024) Bordt et al. (2025).
Likelihood, Entropy and Membership-Inference Signals
Some detectors assess whether an evaluation example appears unusually predictable to a model. They use token probabilities, loss differences, completion likelihood, or hypotheses about the distribution of outputs produced by seen versus unseen texts. The intuitive foundation is that repeated training exposure can increase the probability of a specific sequence. However, common phrases, formulaic answers, and short highly constrained tasks can also be unusually predictable without direct exposure. A high-likelihood item is therefore evidence for a probabilistic comparison, not proof of membership Oren et al. (2024) Li & Flanigan (2024).
Membership inference asks whether a specified item belonged to the training set of a particular target model. Its success depends on model access, the availability of reference distributions, the item’s intrinsic difficulty, and the amount and timing of exposure. Evaluating an inference method requires training sets with known membership and holdouts drawn under a comparable protocol. A detector calibrated on natural-language paragraphs cannot automatically be applied to multiple-choice distractors, code fragments, and math solutions without measuring its specificity anew Li & Flanigan (2024) Samuel et al. (2025).
Likelihood-based sequence-order detection offers a more structured hypothesis: canonical order should be no more probable than exchangeable alternatives when there is no ordering memory. Oren and colleagues formalized a test with statistical false-positive guarantees under their stated assumptions. Order effects induced by sorting, topic clustering, or a training-format convention may violate exchangeability. The investigator must therefore validate the randomization model, define the test statistic, and preserve full information about the benchmark’s construction Oren et al. (2024).
A reliable detection report distinguishes access to token log probabilities from ordinary sampled-output APIs. Forced-choice recognition, perplexity comparison, sequence ordering, and output similarity require different observables and can fail for different technical reasons. Results should include an explicit null hypothesis and the expected response under exposure and nonexposure. Without these definitions, the term membership inference can conceal an uncalibrated recognition heuristic. Multiple independent controls and sensitivity analyses remain essential Golchin & Surdeanu (2025) Samuel et al. (2025).
Representation-Level and Kernel-Based Contamination Assessment
Surface-oriented tests are often insufficient when benchmark information has been transformed. Representation-level methods examine how exposure may alter embedding geometry, sample similarity, or learning dynamics. Choi and colleagues proposed Kernel Divergence Score, which compares kernel similarity matrices of sample embeddings before and after benchmark-specific fine-tuning. Their ICML 2025 experiments used controlled contamination to evaluate sensitivity and reported strong correspondence between the proposed signal and exposure levels under tested conditions Choi et al. (2025).
The representation-level premise is that examples seen during earlier training may respond differently to later fine-tuning than genuinely unseen examples. This premise is empirically testable, but it requires access to model representations or compatible fine-tuning operations and a defensible assumption about learning dynamics. A black-box hosted assistant may provide neither. Consequently, representation-level diagnostics complement methods based on textual outputs and corpus searches but cannot simply replace them across every model family Choi et al. (2025) Deng et al. (2024).
Representation probes also confront construct validity. If semantic similarity between two questions is high, the model may produce similar embeddings whether or not either was in the training data. Language, domain, task frequency, and tokenization can structure representation spaces independently of contamination. Controlled positive and negative exposure groups must therefore be sampled across matched difficulty and semantic conditions. A meaningful result reports detector performance separately for exact, paraphrased, translated, and task-level exposure Choi et al. (2025) Yao et al. (2024).
Interpretability of a detection score matters as much as its predictive accuracy. Researchers should distinguish the estimated probability of a particular item being previously seen, an aggregate measure of relative dataset risk, and an estimated causal performance advantage. A kernel statistic calibrated on controlled simulations may provide useful relative evidence even when it cannot be translated into an item-level exposure probability. Reporting the statistical target prevents unwarranted equivalence among representation scores, membership predictions, and score-inflation estimates Choi et al. (2025) Kocyigit et al. (2025).
Task-Level Familiarity, Synthetic Data and Indirect Transfer
Indirect contamination arises when a model has been trained on tasks or demonstrations that encode the benchmark’s solution procedure, even where no exact test instance is visible. An instructive example is a public reasoning dataset whose explanations are reused in synthetic instruction examples. A later model may learn reusable operations, memorized answer patterns, and benchmark-specific conventions at the same time. Li and Flanigan’s analysis emphasizes that the apparent zero-shot character of an evaluation can be affected by the chronology of task exposure Li & Flanigan (2024).
Training on a task family does not necessarily defeat a legitimate generalization claim. Educational interventions and machine-learning curricula both aim to transfer knowledge from examples to unseen items. The scientific distinction is whether an evaluator claims competence on newly sampled tasks or merely on specific held-out strings. When a system learns an abstract procedure from disclosed training examples and applies it to fresh cases, the measured improvement may represent genuine transfer. When it reproduces an answer supplied in developmental materials, that response provides much weaker evidence for the intended novel problem-solving construct Cronbach & Meehl (1955) Borsboom et al. (2004).
Distillation and generated data expand the provenance chain. A teacher model that encountered an evaluation set can produce synthetic explanations or problem variants for another model; tracing only direct test text in the student’s corpus can miss this indirect channel. Public tutorials, prompt templates, and benchmark solution videos similarly encode task-specific clues. A trustworthy evaluation should document known teacher models, synthetic-data creation prompts, answer filtering, and inclusion dates. This is an investigative prescription from known leakage mechanisms, not a claim that any particular unpublished model uses such a pathway Kaufman et al. (2012) Yao et al. (2024).
A task-level audit benefits from a graded exposure taxonomy, separating exact items, answer-bearing derivatives, structural templates, and general-domain instruction. The categories should be tied to measurable hypotheses. Exact-item exposure predicts recognition of unusual wording; template exposure predicts success on similarly structured fresh questions; broad instruction predicts transfer to a larger task family. Experiments that vary one exposure class while holding total training budgets and task difficulty fixed can establish which mechanism contributes to observed behavior Li & Flanigan (2024) Bordt et al. (2025).
Absolute Performance, Leaderboard Ranks and Decision Sensitivity
A contaminated benchmark can affect the absolute score of every tested system while leaving their ordering relatively stable, or it can produce advantages concentrated in a subset of systems and change a ranking. These outcomes address different questions. Absolute accuracy estimates performance on an item distribution; model ordering addresses a comparative decision. The causal question for each requires item-level responses, model-specific exposure indicators, and estimates of score change under adequately matched clean conditions Kocyigit et al. (2025) Dekoninck et al. (2024).
A rank comparison based only on two aggregate leaderboards can mislead when scores are close or highly correlated across items. Investigators should use paired responses for each model and benchmark version, report uncertainty around rank changes, and test whether the decision would change under plausible contamination-adjustment assumptions. Selection on public leaderboard performance can itself amplify noise and produce optimistic performance estimates. The earlier machine-learning literature on adaptive model selection supplies a formal foundation for treating these differences cautiously Cawley & Talbot (2010) Varma & Simon (2006).
Model comparisons are also affected by benchmark heterogeneity. A system may have seen many code tasks but few translation examples, while another has opposite exposure. Summarizing their performance with one composite number conceals this structure and can create unstable comparisons when test weights change. Reporting results by domain, difficulty, item age, exposure category, and evaluation mode makes the comparison more informative. For adaptive agents, a parallel distinction should separate raw model performance from tool-assisted performance Liang et al. (2023) Chen et al. (2025).
The scientific purpose of ranking analysis is to clarify the decision that the evidence supports. A research article can report that contamination is consistent with inflated public-test accuracy, that several independent detectors agree on a possible exposure mechanism, and that fresh evaluation yields a different absolute score. Whether a product or system ranking changes requires separate statistical analysis. This distinction prevents a legitimate critique of benchmark validity from becoming an ungrounded assertion about a specific model’s comparative ability Samuel et al. (2025) Sun et al. (2025).
Statistical Estimation, Controls and Uncertainty
A contamination investigation needs explicit hypotheses. One hypothesis is that benchmark items were exposed to the system. Another is that the system retained recognizable traces. A third is that retained traces increased observed performance. A fourth is that this increase changes a substantive comparison among systems. Each requires a different design and estimand. Treating their outcomes as a single yes-or-no label conceals uncertainty and can yield false claims about generalization Oren et al. (2024) Kocyigit et al. (2025).
Negative controls are particularly important in black-box detection. Investigators may include unpublished items, shuffled sequences from genuinely exchangeable collections, synthetic examples with known nonexposure, or models trained on clean splits under documented conditions. Positive controls use known injected examples with measured exposure dose. These controls reveal whether a detector responds to exposure rather than ordinary linguistic predictability, answer length, or formatting. Without such calibration, a high recognition rate is a result to explain rather than a validated contamination prevalence estimate Oren et al. (2024) Golchin & Surdeanu (2025) Samuel et al. (2025).
Paired designs compare an original and a transformation of the same item, permitting within-item analysis. Report the number of independent items, the allocation or ordering of prompts, the distribution of score differences, and uncertainty intervals appropriate to the paired observations. When comparing many model versions and item families, analyze dependence among repeated runs and apply safeguards against selection on the most dramatic contrast. A large number of sampled completions from one task does not compensate for a narrow or biased item pool Cawley & Talbot (2010) Varma & Simon (2006) Sun et al. (2025).
Pre-registration of evaluation rules can reduce adaptive leakage. The research team should define exclusions, contamination thresholds, transformation criteria, primary metrics, statistical tests, and release plans before inspecting the outcomes of the final held-out set. Where exploration is essential, designate a development partition and retain a separately governed final test. Publish enough code and provenance to enable independent review without disclosing protected test answers. An auditable chain of decisions is itself part of the scientific evidence Liang et al. (2023) Cawley & Talbot (2010).
Countermeasures and Their Failure Conditions
Deduplication removes exact and near-exact overlaps from accessible corpora. Its effectiveness depends on normalization, span length, the scope of inspected data, and whether semantically equivalent copies remain. A detector based only on string matching can miss translations, lightly rewritten solution texts, and synthetic answer rationales. Researchers should evaluate deduplication with deliberately seeded variants and specify its recall and precision in the relevant domain. A statement that data were decontaminated must name the procedure, not merely the intention Yao et al. (2024) Deng et al. (2024).
Paraphrasing and answer-option shuffling can disrupt surface recognition, but they can also change the cognitive demands or the statistical distribution of the task. Sun and colleagues studied benchmark-refresh strategies under controlled contamination and proposed separate measures for preserving task fidelity and resisting contamination. This dual requirement is central: a novel task that no longer measures the original construct offers little improvement in evaluation validity. Mitigation success should be measured against independently validated transfer tasks and contamination controls Sun et al. (2025).
Dynamic item generation, protected evaluation sets, and time-limited exposure windows reduce some leakage routes. LiveBench emphasizes recent, objectively scored tasks with regular updates; C2LEVA and AntiLeak-Bench investigate automated renewal and controlled protection. No single operational choice guarantees perpetual independence because newly published tasks can enter later training corpora. The strongest program combines protected holdouts with reproducible public baselines and repeated independent refresh cycles, while documenting which versions were exposed and when White et al. (2024) Li et al. (2025) Wu et al. (2025).
Contamination mitigation must also govern the evaluation organization. Separate benchmark construction from model optimization, limit granular feedback where appropriate, rotate independent audit sets, and preserve model and dataset lineage. When a laboratory repeatedly tunes prompts against a public benchmark, simply replacing a few example strings may leave the same adaptive selection bias. Likewise, a hidden test may be compromised by answer-sharing through a judge or retrieval system. Countermeasures are strongest when they address the complete path from item creation to model release and public score interpretation Kaufman et al. (2012) Cawley & Talbot (2010).
Reproducibility, Transparency and Responsible Benchmark Governance
Reproducibility begins with an exact description of the benchmark and experimental system. The record should contain immutable dataset hashes, item-generation methodology, model checkpoint or provider version, evaluation timestamps, inference settings, prompt templates, accessible tools, scoring code, and exclusion criteria. Where proprietary components are inaccessible, that limit should be stated explicitly. Reproducibility is not equivalent to exposing every private test item; it can be supported through escrowed evaluation material, independent auditors, and controlled reruns Liang et al. (2023) Chen et al. (2025).
Scientific transparency includes reporting failed contamination tests and detector disagreement. An audit that publishes only positive probes exaggerates apparent prevalence; one that reports only exact-string checks exaggerates confidence in cleanliness. A comprehensive contamination report distinguishes observed direct overlap, plausible semantic overlap, statistically detected memorization signatures, and demonstrated performance inflation. It can also disclose inaccessible training stages, insufficient negative controls, and uncertainty about transformations Li et al. (2024) Samuel et al. (2025).
Evaluation governance involves incentives. Public rankings shape funding, hiring, product claims, and users’ expectations, creating pressure to maximize visible benchmark scores. Independent test custody, versioned release histories, standardized reporting, and clear retraction or correction procedures reduce the informational asymmetry between model developers and benchmark users. A contamination finding should trigger examination of the affected claims and replacement evaluations; its existence need not imply intentional deception by any particular developer Cawley & Talbot (2010) Chen et al. (2025).
The research program also raises fairness questions. Public benchmarks may disproportionately cover English-language, highly documented, or culturally dominant tasks. Secret benchmarks reduce exposure but can make demographic and domain audits harder. The evaluator must balance provenance control against representativeness, accessibility, and independent scientific criticism. Construct validity supplies the organizing criterion: a comparison is useful when the data and procedure support the precise claim being made for the relevant people, domains, and artificial systems Cronbach & Meehl (1955) Liang et al. (2023).
Case Studies Across Knowledge, Reasoning, Code and Translation
Different benchmark families expose distinct contamination vulnerabilities. MMLU contains fixed multiple-choice academic knowledge items; a model can obtain an advantage by recalling an option’s wording, recognizing an answer, or learning the subject domain generally. Deng and colleagues’ slot-guessing procedure is especially relevant to fixed multiple-choice artifacts because improbable distractors can carry information about test-set exposure. The diagnostic conclusion is narrower than a claim that every correct MMLU answer was memorized Hendrycks et al. (2021) Deng et al. (2024).
GSM8K represents numerical word problems with worked mathematical solutions. Leakage may concern exact problem text, a solution trace, the final number, or a generated reformulation that preserves the same calculation. Replacing numerical quantities and recomputing reference answers can help test whether an observed success reflects manipulation of quantities rather than lookup of a familiar solution. Yet rewriting may change difficulty or numerical properties, so item-level validation and matched control examples remain necessary. The original GSM8K paper provides the benchmark context, while contamination studies supply tests of exposure and transfer Cobbe et al. (2021) Sun et al. (2025).
Programming benchmarks create special exposure channels through public repositories, contest archives, code completions, and solution explanations. A code-generation system may reproduce a remembered algorithm or solution snippet yet still fail on newly specified constraints. Longitudinal analysis of Codeforces and Project Euler tasks by Roberts and colleagues shows how release timing and online popularity can inform contamination hypotheses. Evaluation should distinguish identical known problems from new instances requiring adaptation and should test executable correctness under hidden cases rather than relying solely on textual resemblance Roberts et al. (2024).
Machine translation provides unusually clear evidence about the magnitude of contamination effects because researchers can deliberately mix held-out source-target pairs into training and observe changes in BLEU. The controlled experiments of Kocyigit and colleagues illustrate how exposure format, scaling, and task structure matter. A comparison between translation and mathematical reasoning requires care: translation may reward legitimate reproduction of a conventional rendering, while a novel reasoning item aims to test inferential transfer. The appropriate contamination threshold follows the intended claim in each domain Kocyigit et al. (2025) Borsboom et al. (2004).
A Reproducible Contamination Audit Protocol
A practical audit begins by writing a capability claim before collecting scores. Define the target skill, intended use, eligible inputs, permissible external resources, population of tasks, system version, and primary metric. Next, inventory benchmark provenance: item authorship, collection dates, public availability, released answer keys, derivative datasets, and known inclusion in training or evaluation ecosystems. Record an immutable snapshot of the benchmark and independent hashes for items and solutions Kaufman et al. (2012) Liang et al. (2023).
At the inspection stage, examine every accessible training and post-training source for exact and approximate overlaps; search translations or semantic equivalents where feasible; trace retrieval stores and cached tool outputs separately. Each detected match should be classified by exposure level and documented with evidence available to independent reviewers. A failure to find overlap is recorded with the searchable corpus, search procedures, and known blind spots. This prevents an incomplete audit from being advertised as proof of zero contamination Deng et al. (2024) Yao et al. (2024) Li et al. (2024).
At the behavioral stage, execute multiple calibrated probes: a recognition or slot-guessing test when appropriate, a sequence-order test where exchangeability assumptions hold, and matched original-versus-fresh-item performance assessments. Include positive controls with known exposure, negative controls without exposure, blinded scoring, stable prompts, counterbalanced order, and sufficiently varied item families. Where resources permit, inject contamination experimentally into an open model to estimate dose-dependent inflation and the influence of later training Oren et al. (2024) Golchin & Surdeanu (2025) Kocyigit et al. (2025) Bordt et al. (2025).
Finally, report results in a structured matrix: evidence of access, evidence of memorization, estimated score effect, transfer to fresh tasks, ranking sensitivity, and unresolved uncertainty. Preserve all raw evaluation metadata and distinguish exploratory analysis from confirmation. The publication should give readers the ability to reconstruct why a claim of contamination was made, which comparisons remain scientifically credible, and which new holdout is required. For large-scale recurring evaluation, this protocol should become a versioned institutional routine rather than a one-time cleanup Samuel et al. (2025) Chen et al. (2025) Cronbach & Meehl (1955).
Artificial Era Implications and Open Research Problems
In the Artificial Era, the evaluation of nonbiological cognitive systems becomes a persistent scientific institution. Scores affect accounts of artificial cognition, artificial memory, and system agency, and therefore the empirical meaning of a benchmark matters beyond a technical leaderboard. If a score reflects recognition of a public test artifact, it supplies evidence about informational history. If success generalizes under controlled perturbation and fresh task construction, it supplies stronger evidence about task-relevant competence. Both findings are scientifically useful when their explanations remain distinct Borsboom et al. (2004) Dekoninck et al. (2024).
A first open problem is the absence of comprehensive provenance for closed and continually updated systems. A second is detection of semantic, multilingual, and synthetic-data contamination under realistic training pipelines. A third is reliable calibration of benchmark transformations that preserve the intended construct while changing answer-relevant details. A fourth concerns adaptive agents whose retrieval, tools, and external memory make exposure task- and time-dependent. A fifth concerns statistical estimates of how contamination affects model ordering rather than just absolute scores Samuel et al. (2025) Yao et al. (2024) Chen et al. (2025).
Another research priority is external validation of proposed detectors. Independent teams should replicate probes across model sizes, languages, training doses, task families, and checkpoints while reporting both missed contamination and false alarms. The controlled work of Bordt and colleagues reveals that the persistence of contamination can depend on later training; consequently, detection sensitivity should be assessed as a function of learning history. Standardized reporting needs to characterize when a method is informative, when it lacks power, and when alternative explanations dominate Bordt et al. (2025) Oren et al. (2024).
Research on psychological constructs in artificial systems also requires multidisciplinary collaboration. Cognitive scientists can define task processes and construct networks; computer scientists can trace data lineage and implement controlled interventions; statisticians can design calibration and uncertainty analyses; governance specialists can preserve protected evaluation without suppressing auditability. A cumulative discipline will emerge from versioned systems, explicit definitions, fresh measures, and independently reproducible evidence. The direct consequence is a more interpretable empirical record of how Artificial systems learn, retrieve, remember, and perform Cronbach & Meehl (1955) Liang et al. (2023).
Conclusion
Benchmark contamination and leakage are systematic threats to the inference from observed artificial-system performance to transferable competence. Their scientific significance is determined by the information that crossed an intended evaluation boundary, the way the system could use that information, and the resulting effect on the score. Training overlap, model selection bias, retrieval access, prompt exposure, and evaluator feedback are distinct causal pathways. A rigorous investigation defines them explicitly and links each diagnosis to an identifiable artifact and evaluation configuration Kaufman et al. (2012) Cawley & Talbot (2010).
The literature supports several established findings. Test artifacts can leave detectable recognition and ordering signals in language models; semantic and cross-lingual transformations can evade simple overlap checks; controlled exposure can inflate results; and the effect of repeated exposure can vary with scale and subsequent learning. It also supports an important methodological conclusion: contamination detectors are not interchangeable, and their errors and assumptions must be independently evaluated. A single positive probe does not reconstruct a hidden training corpus, and a negative string search does not certify an uncontaminated model Oren et al. (2024) Deng et al. (2024) Yao et al. (2024) Kocyigit et al. (2025) Bordt et al. (2025) Samuel et al. (2025).
Scientific progress requires evaluation systems that preserve both independence and comparability. This synthesis recommends controlled contamination experiments, careful dataset provenance, calibrated black-box tests, fresh construct-matched tasks, private audit pools, pre-registered scoring, and versioned public records. Distinguishing score inflation from model-ranking distortion is essential whenever benchmark results are used to compare models. The goal is a measurable, reproducible account of what a system can do under specified conditions and what evidence justifies broader claims Chen et al. (2025) White et al. (2024) Cronbach & Meehl (1955).
Psychology by Bogdanova locates this issue within Psychology of Artificial because measurement integrity conditions the study of nonbiological psychological organization. In the Artificial Era, a scientific understanding of Artificial cognition grows through traceable interventions and stable interpretive standards. Benchmark Contamination and Leakage provides the methodological foundation for separating an artificial system’s interaction with known test material from its demonstrated capacity to respond coherently to new problems, without overstating what either observation proves Liang et al. (2023) Borsboom et al. (2004).
References
- Bordt, S., Srinivas, S., Boreiko, V., & von Luxburg, U. (2025). How much can we forget about data contamination? Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, 267, 4998–5016. https://proceedings.mlr.press/v267/bordt25a.html
- Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061
- Cawley, G. C., & Talbot, N. L. C. (2010). On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11, 2079–2107. https://jmlr.csail.mit.edu/papers/v11/cawley10a.html
- Chen, S., Chen, Y., Li, Z., Jiang, Y., Wan, Z., He, Y., Ran, D., Gu, T., Li, H., Xie, T., & Ray, B. (2025). Benchmarking large language models under data contamination: A survey from static to dynamic evaluation. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 10080–10098. https://aclanthology.org/2025.emnlp-main.511/
- Choi, H. K., Khanov, M., Wei, H., & Li, Y. (2025). How contaminated is your benchmark? Measuring dataset leakage in large language models with kernel divergence. Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, 267, 10666–10682. https://proceedings.mlr.press/v267/choi25b.html
- Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). Training verifiers to solve math word problems [Preprint]. arXiv:2110.14168. https://arxiv.org/abs/2110.14168
- Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
- Dekoninck, J., Müller, M. N., & Vechev, M. (2024). ConStat: Performance-based contamination detection in large language models [Preprint]. arXiv:2405.16281. https://arxiv.org/abs/2405.16281
- Deng, C., Zhao, Y., Tang, X., Gerstein, M., & Cohan, A. (2024). Investigating data contamination in modern benchmarks for large language models. Proceedings of NAACL 2024 (Vol. 1, pp. 8706–8719). https://doi.org/10.18653/v1/2024.naacl-long.482
- Golchin, S., & Surdeanu, M. (2025). Data contamination quiz: A tool to detect and estimate contamination in large language models. Transactions of the Association for Computational Linguistics, 13, 809–830. https://doi.org/10.1162/tacl.a.20
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). Measuring massive multitask language understanding. International Conference on Learning Representations. https://arxiv.org/abs/2009.03300
- Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4), Article 15, 1–21. https://doi.org/10.1145/2382577.2382579
- Kocyigit, M. Y., Briakou, E., Deutsch, D., Luo, J., Cherry, C., & Freitag, M. (2025). Overestimation in LLM evaluation: A controlled large-scale study on data contamination’s impact on machine translation. Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, 267, 31105–31132. https://proceedings.mlr.press/v267/kocyigit25a.html
- Li, C., & Flanigan, J. (2024). Task contamination: Language models may not be few-shot anymore. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16), 18471–18480. https://doi.org/10.1609/aaai.v38i16.29808
- Li, Y., Guo, Y., Guerin, F., & Lin, C. (2024). An open-source data contamination report for large language models. Findings of the Association for Computational Linguistics: EMNLP 2024, 528–541. https://aclanthology.org/2024.findings-emnlp.30/
- Li, Y., Wong, T. L., Cheung, T. H., Zhao, J., Zheng, D., Liu, K. W., Lyu, M. R., & Wang, L. (2025). C²LEVA: Toward comprehensive and contamination-free language model evaluation. Findings of the Association for Computational Linguistics: ACL 2025. https://aclanthology.org/2025.findings-acl.116/
- Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research. https://arxiv.org/abs/2211.09110
- Oren, Y., Meister, N., Chatterji, N., Ladhak, F., & Hashimoto, T. (2024). Proving test set contamination in black-box language models. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/46e624c244cff669223d488defd4e835-Abstract-Conference.html
- Roberts, M., Thakur, H., Herlihy, C., White, C., & Dooley, S. (2024). To the cutoff... and beyond? A longitudinal perspective on LLM data contamination. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/2d04d97593c8c33d415337f408ed0e1b-Abstract-Conference.html
- Samuel, V., Zhou, Y., & Zou, H. P. (2025). Towards data contamination detection for modern large language models: Limitations, inconsistencies, and oracle challenges. Proceedings of the 31st International Conference on Computational Linguistics, 5058–5070. https://aclanthology.org/2025.coling-main.338/
- Sun, Y., Wang, H., Li, D., Wang, G., & Zhang, H. (2025). The emperor’s new clothes in benchmarking? A rigorous examination of mitigation strategies for LLM benchmark data contamination. Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, 267, 57728–57753. https://proceedings.mlr.press/v267/sun25t.html
- Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7, Article 91. https://doi.org/10.1186/1471-2105-7-91
- White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., Hegde, C., LeCun, Y., Goldstein, T., Neiswanger, W., & Goldblum, M. (2024). LiveBench: A challenging, contamination-free LLM benchmark [Preprint]. arXiv:2406.19314. https://arxiv.org/abs/2406.19314
- Wu, X., Pan, L., Xie, Y., Zhou, R., Zhao, S., Ma, Y., Du, M., Mao, R., Luu, A. T., & Wang, W. Y. (2025). AntiLeakBench: Preventing data contamination by automatically constructing benchmarks with updated real-world knowledge. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol. 1, pp. 18403–18419). https://aclanthology.org/2025.acl-long.901/
- Yao, F., Zhuang, Y., Sun, Z., Xu, S., Kumar, A., & Shang, J. (2024). Data contamination can cross language barriers. Proceedings of EMNLP 2024. https://aclanthology.org/2024.emnlp-main.990/