Psychology by BogdanovaThe Encyclopedia of Psychology for the Artificial Era

RESEARCH DOSSIER

History of the Replication Crisis and Open Science

Abstract. The replication crisis in psychology developed from persistent twentieth-century concerns about publication selection, statistical power, theory testing, and the status of reported empirical results into a coordinated research and institutional reform movement during the 2010s. This critical narrative historical monograph reconstructs that development through original methodological publications, prominent multi-laboratory replication studies, research into questionable analytic practices, and evaluations of open-science interventions. The evidence includes early critiques by Sterling, Cohen, Meehl, Greenwald, and Rosenthal; debates around false-positive psychology and anomalous cognition in 2011; Many Labs and the Open Science Collaboration's 2015 study; preregistration, Registered Reports, and TOP transparency guidelines; and assessments of publication bias, data sharing, multi-analyst results, and the quality of reform-oriented reporting. Later studies of the history of disciplinary crises and changes in published evidence illuminate remaining problems of measurement, theory, generalizability, and institutional incentives. The final chapters examine challenges introduced by artificial research systems and propose historically grounded criteria for evaluating cumulative scientific evidence. The synthesis distinguishes computational reproducibility, new-data replication, robustness, and generalizability, and separates empirically demonstrated findings from reform proposals and subsequent interpretations. It is a critical narrative monograph, not a new replication experiment, systematic review, or meta-analysis.

Keywords: replication crisis; open science; history of psychology; reproducibility; replicability; publication bias; statistical power; researcher degrees of freedom; HARKing; questionable research practices; Open Science Collaboration; Many Labs; preregistration; Registered Reports; Transparency and Openness Promotion guidelines; p-curve; multiverse analysis; research integrity; meta-research; history of scientific reform; Artificial Era

The Historical Object: A Crisis of Psychological Knowledge

The replication crisis in psychology names a period of intensified scrutiny of the relation between published empirical claims and the results obtained when those claims are independently examined. Its public consolidation during the 2010s followed decades of methodological criticism concerning significance testing, selective publication, statistical power, researcher discretion, and professional incentives. A historical account therefore studies both scientific results and the institutions that converted individual experiments into authoritative knowledge. The object of this dossier is the development of that critique, its experimental tests, and the organizational reforms that followed. It is a critical narrative historical synthesis based on identified original studies, methodological articles, and later historical scholarship, rather than a new replication or protocol-registered systematic review. Sterling (1959) Shrout & Rodgers (2018) Teymoori & Trappes (2026)

The term crisis combines an empirical question with a disciplinary judgment. Empirically, investigators ask how closely outcomes in new data agree with earlier claims and how often statistical or theoretical conclusions survive stringent tests. Historically, calling that situation a crisis mobilizes editors, funders, departments, professional associations, and research networks around perceived deficiencies in the production of knowledge. The label can illuminate coordinated reform while concealing substantial differences among subfields, designs, and kinds of claim. A responsible history reconstructs who raised the concern, what evidence they offered, which alternatives were available, and how competing interpretations of the same evidence developed. Schmidt (2009) Shrout & Rodgers (2018) Teymoori & Trappes (2026)

The guiding questions are chronological and explanatory. Which problems were recognized before the 2010s? What made the 2011–2015 sequence unusually consequential? How did large collaborative projects alter the standards by which researchers judged an effect? Why did preregistration, Registered Reports, shared materials, badges, and institutional transparency policies become practical responses? What evidence now bears on their performance? The analysis considers changes in statistical practice alongside research cultures and technology, while distinguishing contemporary empirical evaluations from retrospective accounts of institutional transformation. Nosek et al. (2012) Open Science Collaboration (2015) Munafò et al. (2017)

Reproducibility, Replication, Robustness, and Scientific Claims

The vocabulary of the crisis requires historical precision. Computational reproducibility asks whether the same data and analytical operations yield the reported result. Replication ordinarily uses new observations to test an earlier empirical claim, with fidelity defined in relation to the claim's indispensable features. Robustness asks whether the conclusion survives defensible changes in assumptions, models, measurement, or analytic decisions. Generalizability addresses whether the finding transfers across populations, settings, times, or tasks. These questions intersect, yet they produce different forms of evidence. A faithful computational reconstruction can reproduce an erroneous claim; a well-designed new-data replication can disagree while exposing meaningful heterogeneity. Schmidt (2009) Open Science Collaboration (2015) Steegen et al. (2016)

The distinction between direct and conceptual replication acquired particular prominence in debates about psychology. Direct replication attempts to preserve operational conditions thought necessary for a finding, whereas conceptual replication varies procedures while testing a proposed explanatory principle. Both depend on theory. A procedure cannot be repeated without judgments about which aspects matter, and a changed operationalization cannot independently confirm a theory unless alternative explanations are constrained. Large collaborative replication programs introduced explicit protocols, original-author feedback, prespecified analyses, and multiple samples to reduce uncertainty about procedural drift. Their findings still required interpretation through effect sizes, uncertainty, and theoretical scope. Klein et al. (2014) Open Science Collaboration (2015) Shrout & Rodgers (2018)

An isolated significant original result and a nonsignificant replication do not logically prove opposite propositions. Significance thresholds provide one summary of sampling uncertainty under assumed models; differences between p values can arise with similar underlying effects. Researchers must compare effect estimates, precision, measurement, design equivalence, and the plausibility of contextual moderators. Historians must likewise avoid converting a methodological disagreement into a binary story of triumph and failure. The modern reform movement progressively treated replicability as a property of evolving evidential programs, supported by coordinated tests and cumulative synthesis, rather than a ceremonial repetition of a single result. Open Science Collaboration (2015) Camerer et al. (2018) Shrout & Rodgers (2018)

Significance Testing and Its Early Institutionalization

The practices later disputed in the replication crisis grew within twentieth-century attempts to organize inference from noisy empirical observations. Null-hypothesis significance testing offered a concise decision vocabulary that spread through psychology's experimental and correlational literatures. Its attraction was institutional as well as analytical: a conventional threshold simplified communication between investigators, referees, and editors. The threshold, however, addressed the behavior of a statistical procedure under specified assumptions rather than the probability that a scientific hypothesis was true. The history of methodological debate repeatedly returned to this distinction. Meehl (1967) Ioannidis (2005)

Paul Meehl's 1967 methodological paradox identified a problem in testing substantive psychological theories through statistical hypotheses that could be rejected by sufficiently sensitive data even when such rejection supplied little discriminating evidence between rival explanations. The methodological point concerned the fit between theoretical content and statistical inference. A low p value can register a detectable deviation while leaving the theoretical interpretation underdetermined. This concern later joined critiques of significance hunting: an impressive publication record may accumulate statistical decisions without improving the precision of substantive models or the prediction of new observations. Meehl (1967) Shrout & Rodgers (2018)

The historical tension also concerned what investigators recorded as failures. When only outcomes meeting a significance convention entered the literature, published success became a selected sample of research activity. Researchers encountering a journal article could not readily reconstruct how many candidate tests, measurements, or investigations had preceded it. The danger was structural: even with honest investigators and correctly calculated tests, the publication channel could transmit misleading evidence. Later open-science reforms addressed this historical information deficit by altering what was registered, preserved, and disseminated, not merely by recommending more cautious interpretation. Sterling (1959) Greenwald (1975) Nosek et al. (2012)

Sterling, Cohen, and the Problem of Hidden Selection

Theodore Sterling's 1959 paper in the Journal of the American Statistical Association documented concern that nonsignificant findings were less likely to be published, allowing repeated investigations of a null effect to continue until chance produced a publishable positive result. This was a distinctly pre-2010 formulation of the selection problem. Sterling's warning showed that the set of visible findings could be systematically different from the set of findings produced, and that ordinary significance-test error rates no longer described the evidential record encountered by a reader. The argument helped establish publication bias as a problem of scientific communication and inference. Sterling (1959)

Jacob Cohen's 1962 examination of statistical power in abnormal and social psychological research addressed a complementary weakness: the probability that a study would detect an effect if a specified effect were present. Low power means a substantial risk of missing real effects, but when significance determines publication, the subset that clears the threshold can also overestimate effect magnitude. The historical importance of Cohen's work lies in making sample planning an integral part of methodological judgment rather than a purely practical detail. Modern discussions of replication revisit the combined consequences of small samples, uncertain effect sizes, and selective visibility. Cohen (1962) Bakker et al. (2012)

Anthony Greenwald's 1975 analysis of prejudice against the null hypothesis brought investigator and editorial decisions into one model of the research-publication system. The study examined how decisions to pursue, abandon, report, or publish work shaped the literature's appearance. In that model, the selective attention of authors and editors can interact with sampling variability to produce a misleading body of evidence. Sterling, Cohen, and Greenwald consequently identified three mutually reinforcing problems: a restricted publication window, weak inferential precision, and an incentive structure that favored certain outcomes. Subsequent reforms developed new infrastructure around these established concerns. Greenwald (1975) Sterling (1959) Cohen (1962)

The File Drawer and the Development of Research Synthesis

Robert Rosenthal's 1979 formulation of the file-drawer problem emphasized the unknown denominator of unreported studies. A published collection of positive results can look persuasive even when unseen investigations with null outcomes would substantially alter a cumulative interpretation. Rosenthal proposed quantitative approaches to assess tolerance for additional null results. Their significance was historical: the unit of evidence shifted from a single article toward the distribution of studies that could exist beyond the visible record. A systematic reviewer could no longer treat publication as an unbiased sampling mechanism. Rosenthal (1979)

Meta-analysis offered a way to combine findings across investigations, but it depended on the quality and completeness of its source literature. If included effects were preferentially large or statistically significant, combining them could reproduce or intensify selection bias. Early warnings therefore anticipated a persistent question for research synthesis: does statistical aggregation reveal an underlying effect or give disproportionate authority to previously selected results? Later comparisons with prospective multi-laboratory replications offered empirical leverage on this question by placing conventional synthesis beside studies organized before outcomes were known. Rosenthal (1979) Kvarven et al. (2020)

The enduring conceptual lesson is that open scientific evidence requires a record of research decisions and outcomes broader than prestigious positive publications. Information about all registered studies, tested measures, exclusions, and analytic models helps readers understand the denominator of evidential opportunities. This insight links the earliest methodological critics directly to modern preregistration and results-independent editorial review. It also explains why debates about how to evaluate replication cannot be settled by identifying a single universal percentage of success: research programs differ in sample selection, inferential criteria, and the population of studies from which claims have been selected. Nosek et al. (2018) Scheel et al. (2021) Shrout & Rodgers (2018)

Before the Modern Crisis: Theory, Experiment, and Disciplinary Self-Criticism

An exclusive focus on the years after 2011 would obscure psychology's recurrent reflection on its methodological legitimacy. In the mid-twentieth century, psychologists disputed whether an accumulation of statistically significant laboratory results constituted successful theory testing, and whether operational definitions produced evidence about stable psychological kinds. Meehl's methodological paradox exemplified an enduring tension between rejecting an exact statistical null and distinguishing substantive theories. Historical scholarship published in 2026 places the replication crisis alongside earlier disciplinary crises that addressed psychological objects, knowledge, and institutional authority. The resemblance identifies recurring questions, although their practical settings and proposed remedies differed. Meehl (1967) Teymoori & Trappes (2026)

These earlier debates also establish a limit on the standard triumphal story of open science. Transparency increases the accessibility of decisions and observations, but a fully transparent sequence of experiments can still depend on a weak theoretical construct or on measurements that inadequately represent it. Conversely, a coherent theory can be burdened by selective evidence. The later reform movement gradually combined openness with renewed attention to validity, theoretical specificity, sample diversity, and the meaning of successful replication. The history of crisis is therefore a history of what psychological evidence was expected to demonstrate. Shrout & Rodgers (2018) Teymoori & Trappes (2026)

The chronology is best understood as an interaction among methods, reward structures, and technological capacities. Statistical software lowered the cost of exploring datasets; digital journals accelerated the circulation of striking findings; online collaboration made distributed replications practical; repositories allowed protocols and results to be preserved and inspected. Each affordance changed the scientific environment in ways that could increase either flexibility or accountability. Historiography explains their combined development instead of assigning one invention or individual responsibility for the crisis. Nosek et al. (2012) Munafò et al. (2017)

HARKing, Researcher Degrees of Freedom, and Exploratory Discovery

Norbert Kerr's 1998 paper named HARKing—hypothesizing after the results are known—as a practice in which a hypothesis developed after seeing the data is presented as though it had been specified before analysis. The problem is an error in the evidential account: the same observation cannot serve both as the stimulus that suggested an explanation and as wholly independent confirmation of that explanation. Exploratory inference can be scientifically productive, yet its warrant differs from prospective testing. The historical significance of the term lies in making the chronology of research decisions itself an object of methodological scrutiny. Kerr (1998)

Researchers usually face many defensible decisions about participants, measures, transformations, exclusions, outcomes, model specification, and when collection ends. If decisions are repeatedly selected according to the observed p value, the eventual inferential result can have an error rate different from its nominal one. Simmons, Nelson, and Simonsohn demonstrated in 2011 that undisclosed flexibility can produce false-positive evidence even while investigators apply familiar significance tests. The publication shifted attention from isolated miscalculations to entire sequences of choices that conventional papers rarely disclosed. Simmons et al. (2011) Steegen et al. (2016)

The subsequent language of questionable research practices grouped different behaviors under a contentious umbrella. Some practices explicitly misrepresent the sequence of analysis; others arise from ordinary adaptation to unexpected data and become problematic when their evidential status is concealed. The relevant methodological remedy is reconstructability: readers need to distinguish prespecified tests from data-contingent inquiries, determine how many analytic paths were available, and understand the relationship between the reported result and the process by which it was obtained. That principle later guided preregistration, full disclosure, and computational audit trails. Kerr (1998) John et al. (2012) Nosek et al. (2018)

False Discovery, Incentives, and the Publication Economy

John Ioannidis's 2005 analysis formalized how low prior probabilities, small studies, bias, and selective research practices can reduce the credibility of published positive findings. Its title posed a provocative claim about research findings across disciplines, while its contribution was a set of conditional models rather than a census proving that most psychological publications were false. A historical account must preserve that distinction. The model demonstrated how seemingly successful significance testing may yield a weak positive predictive value when many speculative claims are tested and reported selectively. Ioannidis (2005)

Bakker, van Dijk, and Wicherts examined the incentives confronting psychological researchers who must choose sample size, analytic effort, and publication strategy. Their 2012 analysis articulated why a system rewarding frequent significant papers could make low-powered designs superficially attractive to individual investigators, even when scientific knowledge suffered. Nosek, Spies, and Motyl similarly described a conflict between maximizing the publishability of a particular report and generating an unbiased cumulative record. These were institutional explanations, not diagnoses of individual moral character. Bakker et al. (2012) Nosek et al. (2012)

The incentive argument also clarifies why a methodological reform can fail if it changes author behavior without altering editorial selection or professional rewards. Researchers may register protocols but leave null results unpublished, share incomplete materials, or move flexibility into unrecorded preliminary decisions. Departments and reviewers may continue to value surprising positive outcomes over precision and correction. The open-science movement answered this coordination problem with policy instruments—new editorial formats, accessible repositories, transparent reporting requirements, and explicit recognition of research contributions that had previously remained invisible. Nosek et al. (2012) Nosek et al. (2015) Munafò et al. (2017)

The 2011 Shock: Extraordinary Claims and Ordinary Statistics

Daryl Bem's 2011 article reported experiments interpreted as evidence for anomalous retroactive influences on cognition and affect. The claim drew unusual attention because it appeared in a prominent psychology journal while using experimental and statistical procedures resembling those routinely accepted in mainstream psychology. For many observers, the unsettling implication concerned the general sensitivity of familiar research procedures to misleading evidence: if the same evidential conventions supported a deeply surprising conclusion, the conventions themselves deserved scrutiny. The episode became an emblem of the period without being the sole cause of the subsequent crisis. Bem (2011) Wagenmakers et al. (2011)

Wagenmakers and colleagues responded in the same journal issue with a methodological critique advocating closer attention to statistical evidence and alternative inferential approaches. The debate involved prior plausibility, analysis choices, and what a significant result can establish. Its historical significance depended partly on disciplinary visibility: methodological concerns familiar to statisticians became widely discussed through a case whose substantive interpretation many psychologists found implausible. The incident illustrated how confidence in a result depends on the full evidential context, not merely on the threshold reached by an individual analysis. Wagenmakers et al. (2011) Ioannidis (2005)

That year also saw the publication of False-Positive Psychology, in which Simmons, Nelson, and Simonsohn used simulations and demonstrations to show the consequences of undisclosed data-dependent flexibility. The simultaneous prominence of extraordinary empirical claims and familiar statistical vulnerabilities helped convert dispersed methodological complaints into an organized reform debate. The turning point was cumulative: new demonstrations of error, longstanding knowledge about publication selection, and the availability of collaborative digital methods together shifted attention from blaming particular findings toward auditing psychological research practices. Simmons et al. (2011) Sterling (1959) Nosek et al. (2012)

Measuring Research Practices and Challenging the Researcher Narrative

John, Loewenstein, and Prelec's 2012 survey investigated the self-reported prevalence of selected questionable research practices and the incentives surrounding them. The survey provided evidence that practices such as selective reporting were salient enough to warrant direct measurement. Its estimates depended on respondents, question wording, and models of candid disclosure; they should be interpreted as survey-based assessments of reported behavior rather than definitive prevalence rates for all psychologists. The work nevertheless expanded the empirical basis of reform discussions beyond anecdotes about individual papers. John et al. (2012)

Research incentives had long rewarded novel, concise, successful studies, but survey data complicated simplified explanations built around a few unreliable researchers. Normalized analytical discretion and differential publication probabilities can affect an entire literature even when most investigators intend to contribute useful knowledge. The core historical change was recognition of systemic dependence: the reliability of published claims depends on how laboratories make decisions, journals select articles, and professional cultures define success. Reformers therefore proposed modifications to institutions alongside improved statistical literacy. Greenwald (1975) John et al. (2012) Nosek et al. (2012)

Early reform arguments were also contested. Exploratory work produces new hypotheses, and strict adherence to a mistaken initial analysis can obscure unexpected discoveries. Preregistration requires investigators to identify which decisions are confirmatory and which remain exploratory; it does not make surprising post hoc results scientifically irrelevant. The relevant scholarly norm is traceable separation of evidential roles, allowing readers to assess how much independent evidence actually tests a prediction. The recognition of this distinction made preregistration a practical bridge between methodological self-criticism and reform of the publication process. Kerr (1998) Nosek et al. (2018) Munafò et al. (2017)

From Single-Laboratory Findings to Collaborative Replication

The formation of coordinated research networks was one of the decisive organizational changes of the 2010s. Earlier independent replications depended heavily on a laboratory's interest, access to an original procedure, and willingness to publish an outcome that might complicate an established finding. Collaborative projects redistributed work: multiple laboratories implemented a common protocol, recruitment generated more precise pooled estimates, and collective authorship gave replication work a recognized scholarly form. The methodological value was not solely larger sample size. A common protocol permitted research teams to examine variation across sites while reducing the influence of undocumented differences in analytical choices. Klein et al. (2014) Open Science Collaboration (2015)

Many Labs, published in 2014, compared a collection of psychological effects using coordinated data collection in multiple settings. The project was influential because it made replication an organized empirical research program rather than a reactive exchange between the proponents and critics of one result. Repeated measurement across sites encouraged scrutiny of effect magnitude, sampling conditions, and the practical meaning of procedural equivalence. The original projects and later Many Labs studies also demonstrated that even ostensibly direct replications require transparent choices about context and measurement. Klein et al. (2014) Klein et al. (2018)

Historically, the creation of common protocols changed scientific accountability. Documentation made replication procedures inspectable before data collection; collaboration produced a shared pool of methodological expertise; and public dissemination allowed disagreements to focus on evidence and design. A large project can introduce its own problems of coordination, recruitment variation, and construct transportability, so it must be assessed on those terms. Yet its institutional innovation was clear: methodological disagreements could now be investigated through planned distributed studies capable of distinguishing reproducible small effects from highly selected original estimates. Klein et al. (2014) Shrout & Rodgers (2018)

The Open Science Collaboration Report of 2015

The Open Science Collaboration's 2015 Science report became the central empirical landmark of the modern crisis. It presented replication attempts for 100 experimental and correlational studies drawn from three psychology journals. The investigators generally used original materials when available and sought higher-powered designs, while evaluating outcomes according to several criteria. In the original set, 97% of studies had reported statistically significant results; in the replication attempts, 36% produced statistically significant results according to the report's criterion. The replication effect sizes were approximately half the original effect sizes on average. These are study-sample findings, not a universal rate applying to psychological science. Open Science Collaboration (2015)

The report deliberately showed how different measures of reproduction yielded different summaries. Forty-seven percent of original effect sizes were within the reported 95% confidence intervals of replication estimates, while subjective ratings judged 39% to have replicated. These criteria address related questions without measuring exactly the same property. The cross-criterion disagreement mattered historically because it defeated a simplistic reading of replication as a single universally defined pass–fail test. An effect could be in the predicted direction but estimated with less precision, or an apparently weak replication could still be compatible with a modest true effect. Open Science Collaboration (2015) Shrout & Rodgers (2018)

The 2015 results challenged the expectation that published significance would routinely be preserved under independent testing and helped establish the credibility of large-scale audit research. Their limitations are essential to the historical record. The project covered a defined selection of articles, journal contexts, and available protocols; heterogeneity in constructs and experimental settings complicates comparisons. The report's contribution was an unusually visible evidential test that motivated research on selection, protocol fidelity, effect-size inflation, and the institutions governing publication. It did not establish that every unsuccessful replication falsified its underlying theory. Open Science Collaboration (2015) Munafò et al. (2017)

Multiple Criteria, Statistical Precision, and Scientific Interpretation

The replication debate made uncertainty a public issue. A statistical significance result depends on the effect estimate, its estimated standard error, the test used, and a decision threshold. When sample sizes change, a very similar underlying effect can yield a different significance decision. Conversely, two significant findings may estimate materially different effects. The Open Science Collaboration compared estimates and confidence intervals alongside binary thresholds, reflecting an emerging insistence that psychological claims be understood quantitatively and substantively. The historical move was from counting positive papers toward estimating what can be learned from new observations. Open Science Collaboration (2015) Shrout & Rodgers (2018)

Original studies selected for statistical significance will often carry exaggerated estimates, especially in a low-powered literature. Subsequent replications can therefore yield smaller effects even in the presence of a nonzero phenomenon. Selection, design changes, and population differences are competing hypotheses; they cannot be disentangled by pointing only to an original p value. Multi-laboratory programs strengthened inference by using larger samples, prespecified tests, and shared procedures, yet each also required a theory about which differences between laboratories matter. Generalizability and replication became more tightly linked research questions. Cohen (1962) Bakker et al. (2012) Klein et al. (2018)

Different scientific stakeholders responded differently to the same results. Authors may emphasize the possibility of unmeasured moderators, replication investigators may focus on the failure of a specified prediction, and methodologists may question whether any single comparison is decisive. Good cumulative inference combines these perspectives by stating the original prediction, analyzing the replication's precision, identifying plausible sources of heterogeneity, and testing such explanations prospectively. The lesson for historians is that the methodological revolution was also a negotiation over standards of evidence and scholarly authority. Shrout & Rodgers (2018) Teymoori & Trappes (2026)

Many Labs 2 and the Analysis of Context

Many Labs 2, published in 2018, tested 28 previously reported findings with preregistered, advance-reviewed protocols. Its 125 samples included 15,305 participants from 36 countries and territories, with each protocol administered to approximately half the samples. At the conventional significance threshold, 15 of the 28 replications produced a significant effect in the original direction. The project thus combined an explicit replication audit with an attempt to estimate how effect sizes varied across settings and samples. Its precision and distributed reach made it an important successor to the first Many Labs effort. Klein et al. (2018)

The project reported effect-size differences between original and replication estimates and examined evidence for heterogeneity across samples. The scale enabled investigators to distinguish effects that generalized relatively consistently from effects whose estimates depended more on research setting or population. These analyses challenged any expectation that a single replication outcome could fully characterize a psychological effect's domain. A result reproduced with a narrowly defined sample need not generalize to an entirely different environment; a result that varies by context demands an explicit account of the variation rather than a post hoc moderator story. Klein et al. (2018) Shrout & Rodgers (2018)

Cross-national participation broadened empirical scope, yet extensive participation does not by itself confer representative sampling of the world. Laboratories entered the project through practical networks and implemented particular tasks under constrained conditions. Transportability requires evidence about the distribution of relevant populations, languages, institutions, and task meanings. Many Labs 2 became historically important for turning those limits into researchable variables and for demonstrating the value of designing replication and heterogeneity assessments together. Klein et al. (2018) Teymoori & Trappes (2026)

Replications Beyond Psychology and the Ego-Depletion Debate

The spread of organized replication extended beyond psychology. Camerer and colleagues' 2016 Science study examined replicability in selected laboratory experiments from economics; a later 2018 project assessed 21 social-science experiments published in Nature and Science between 2010 and 2015. In the latter set, 13 replication results were statistically significant in the original direction. These projects mattered to the history of psychology because they located its crisis within a broader debate about experimental social science, publication selection, and collective evaluation. A replication rate from a selected journal sample must be interpreted against its designs and sampling frame. Camerer et al. (2016) Camerer et al. (2018)

The 2016 preregistered multi-laboratory test of ego depletion offered a particularly influential psychology case. Twenty-three laboratories contributed a combined sample of 2,141 participants to a standardized sequential-task protocol, yielding a pooled effect near zero, d = 0.04, with a 95% confidence interval from −0.07 to 0.15. The finding placed substantial pressure on a general resource-depletion explanation supported by earlier published studies and meta-analyses. It also initiated continued debate about whether the particular tasks adequately captured the proposed mechanism. The empirical result and its theoretical interpretation must be kept distinct. Hagger et al. (2016)

Together these initiatives demonstrated both the strengths and the limits of standardized replication. Shared protocols can reduce many types of investigator discretion and increase precision. At the same time, the choice of target experiment, manipulation, timing, participant population, and dependent measure remains theoretically significant. The replicability movement's durable contribution was to make those decisions public and disputable through evidence. Such projects provided historically novel infrastructure for correcting effects that had become familiar before their evidential foundations were independently tested. Hagger et al. (2016) Camerer et al. (2018) Shrout & Rodgers (2018)

Open Science as an Institutional Reform Movement

Open science became an organized response to the replication crisis when proposals for methodological discipline were translated into changes in how research was planned, evaluated, archived, and credited. Its central practices include public registration of protocols, sharing of data and materials when ethically possible, open code, accessible research records, and publishing rules that reduce dependence on the eventual direction of findings. Nosek, Spies, and Motyl's 2012 reform proposal emphasized restructuring professional incentives, while the 2017 manifesto of Munafò and colleagues brought design, analysis, reporting, dissemination, and rewards into an integrated framework. The movement operated through laboratories, journals, funders, repositories, and international networks. Nosek et al. (2012) Munafò et al. (2017)

The historical innovation lay in treating knowledge production as a sequence of decisions that could be documented before and after observations were obtained. A conventional paper tends to present the final experiment as a coherent narrative of hypothesis, method, and result. Real research often includes pilot work, abandoned models, amended measures, and interpretive turns. Open science technologies made many such stages recordable and inspectable, thereby allowing the reported article to be understood as one part of a longer evidential history. The resulting practices are best evaluated by the claims they help readers verify. Nosek et al. (2018) Munafò et al. (2017)

Open science also changed the social distribution of authority. More readers could inspect files and critique analyses, collaborating groups could work from common materials, and recognition systems could award credit for transparent practice. However, uploading a file does not guarantee that it is complete, usable, or ethically appropriate to release, and a preregistration can be vague or difficult to audit. Historical assessment therefore follows the difference between formal adoption of a practice and the quality of its execution. Kidwell et al. (2016) Hardwicke et al. (2020) Munafò et al. (2017)

Preregistration: Making Decisions Temporally Visible

Preregistration documents a research plan before researchers observe the outcomes that will be used to evaluate its predictions. The planned record can specify hypotheses, design, sample size, eligibility criteria, outcomes, exclusion rules, and analytic models. Its epistemic role follows from the distinction between generating a hypothesis using data and testing that hypothesis with independent observations. Nosek and colleagues described this as protecting the boundary between postdiction and prediction, a conceptual distinction that had been difficult to infer from traditional published narratives. Nosek et al. (2018) Kerr (1998)

Preregistration acquired institutional force through online registries and journals willing to evaluate whether a reported analysis matched an archived plan. The reform creates an audit trail rather than a guarantee of research quality. A badly chosen measure can be preregistered; an underpowered study can adhere exactly to its design; and unforeseen conditions can justify transparent deviations. The record is useful because readers can tell when a choice was made and decide what inferential status a result deserves. This is especially important when multiple outcomes and subgroups could have been examined. Nosek et al. (2018) Munafò et al. (2017)

The practice also altered the scientific legitimacy of exploratory analysis. Transparent exploratory work can reveal unexpected associations and generate new theoretical predictions; a later independent investigation can then test those predictions. Historical critiques of HARKing had already identified why collapsing these activities inflates the apparent strength of a claim. Preregistration made their separation operational within an increasingly digital research workflow. Its practical success depends on the completeness, accessibility, and comprehensibility of the original plan and on reviewers' willingness to inspect deviations. Kerr (1998) Simmons et al. (2011) Nosek et al. (2018)

Registered Reports and Results-Independent Editorial Review

Registered Reports moved the moment of substantive editorial review from after analysis to before decisive data collection. Researchers submit a research question, theoretical justification, design, and planned analyses for Stage 1 peer review. Following successful review, a journal offers in-principle acceptance conditional on adherence to the approved protocol and adequate completion of the study, with findings evaluated independently of whether their favored prediction is supported. Christopher Chambers's 2013 Cortex editorial is an early institutional landmark for the model in its modern form. The intervention targets publication selection directly. Chambers (2013)

The historical importance of this format becomes clear when compared with traditional publication incentives. An investigator normally cannot know whether a journal will value a completed result, and editors may disproportionately reward unexpected significant findings. Results-independent acceptance aims to decouple the decision to publish from the study's eventual statistical outcome. It also moves review of design and measurement to a stage when criticisms can still inform data collection. The format thus reorganizes the temporal structure of the research publication process rather than merely requiring an additional reporting checklist. Chambers (2013) Scheel et al. (2021)

The format places demands on authors and editors: plans must be sufficiently specified, revisions must be documented, and protocol deviations must be evaluated fairly. It is particularly suited to prospective hypothesis tests, including replication research, while exploratory projects may require different editorial arrangements. Later empirical investigations of Registered Reports compared both the distribution of results and the quality of published work, providing evidence with which to assess the reform beyond its conceptual appeal. Those studies remain subject to confounding by selection of authors, journals, topics, and research designs. Scheel et al. (2021) Soderberg et al. (2021)

TOP Guidelines, Transparency Policies, and Publishing Standards

The 2015 Transparency and Openness Promotion guidelines, introduced by Nosek and colleagues in Science, articulated a coordinated role for journals in making research practices more inspectable. Their framework organized expectations for citation, data, code, materials, design transparency, preregistration, and replication. The significance was the creation of a shared policy vocabulary that journals could adapt rather than the invention of any single method. By giving editors concrete implementation levels, the guidelines converted individual reform preferences into institutional commitments that could be communicated, compared, and reviewed. Nosek et al. (2015)

Reforms in journal policy influenced the information researchers anticipated collecting. If a manuscript must identify a repository and explain analytic decisions, authors need to maintain files and documentation during the research process. Requirements can improve the recoverability of data and code while creating additional work and legitimate privacy or intellectual-property dilemmas. Sound policy distinguishes openness of evidence from unrestricted release of sensitive participant data. For qualitative studies, historical archives, and work with vulnerable populations, documentation and controlled access can preserve accountability while respecting ethical obligations. Nosek et al. (2015) Munafò et al. (2017)

A historical perspective also distinguishes policy adoption from measured compliance. A journal may endorse transparency without consistently enforcing it, and a manuscript can satisfy formal documentation requirements while remaining computationally irreproducible. Later empirical audits therefore became indispensable to judging whether reforms changed the research record. The institutional effects of transparency guidelines are best assessed by observable availability, adequacy, and reuse of research materials over time, not by the mere appearance of favorable policy language. Hardwicke et al. (2020) Kidwell et al. (2016)

Badges, Credit, and the Social Economics of Openness

Recognition badges offered a low-cost incentive for practices such as open data and open materials. Kidwell and colleagues' 2016 study examined policy change at Psychological Science, which began offering badges in January 2014. The proportion of articles reporting open data had been below 3% before badges; it rose to 23% overall after the change and reached 39% during the first half of 2015. Those are journal-specific reporting rates under a particular policy intervention. Their comparison with other journals supported an association between the badge scheme and increased openness, while observational design limits causal certainty about every contributing factor. Kidwell et al. (2016)

The study also checked whether material described as available was actually available and usable. This distinction is fundamental for scientific evaluation. Stating that data exist in a public repository provides little benefit when files cannot be located, accessed, interpreted, or analyzed with sufficient documentation. Badges function as visible rewards only when the underlying conditions are inspected and meaningful. Their success thus depends on administrative rigor as well as motivational design. Kidwell et al. (2016) Hardwicke et al. (2020)

The wider institutional lesson is that transparency has labor costs and professional incentives. Preparing anonymized datasets, maintaining code, describing instruments, and archiving protocols consume resources that the conventional paper count may inadequately recognize. The movement to recognize these contributions changed the definition of legitimate scientific output. Policy experiments with badges are historically important because they supplied testable evidence that relatively small changes in journal incentives could affect researcher behavior, while leaving questions of sustainability, fairness, and long-term reuse open. Kidwell et al. (2016) Munafò et al. (2017)

Research Synthesis Under Publication Selection

Meta-analysis became a central site of controversy when large preregistered replications produced smaller estimates than earlier aggregated literatures. Kvarven, Strømland, and Johannesson compared 15 meta-analyses with preregistered multi-laboratory replication projects addressing the same topics. For 12 of 15 pairs, estimated effects differed significantly; meta-analytic effects were nearly three times as large as the replication estimates on average. The published article appeared online in December 2019 and in a 2020 journal volume, with a subsequent author correction. The findings apply to the selected comparison pairs rather than constituting a universal multiplier for all psychology meta-analyses. Kvarven et al. (2020)

The comparison sharpened an old methodological problem. A meta-analysis draws information from available studies and can achieve high apparent precision, but precision does not eliminate systematic selection. When the studies entering a synthesis have been filtered for significance and favorable interpretation, a pooled estimate can be confidently biased. Bias-correction techniques attempt to compensate for selection, yet they require assumptions that may fail for diverse methods and research contexts. Kvarven and colleagues' analyses showed that the corrections they examined did not substantially close the gaps in their sample, underlining the value of independently organized prospective evidence. Kvarven et al. (2020) Rosenthal (1979)

At the same time, the replication estimate itself has a scope defined by its instruments, populations, and implementation. A disagreement between a literature synthesis and a coordinated test invites investigation of selection, heterogeneity, and construct comparability. The historical advance was the ability to place different forms of evidence beside one another and ask why they diverged. That is a richer objective than awarding credibility automatically to either the original literature or the replication project. Kvarven et al. (2020) Shrout & Rodgers (2018)

P-Curve, Multiverse Analysis, and Analytical Flexibility

New methods examined the diagnostic pattern of published evidence itself. Simonsohn, Nelson, and Simmons introduced p-curve as a way to inspect distributions of significant p values and assess whether a selection of studies contains evidential value, while highlighting the assumptions needed for valid interpretation. The approach emerged from a central challenge of the crisis: when the entire set of attempted tests is unavailable, can the reported distribution still carry information about selective significance and underlying effects? P-curve supplied one strategy, not a substitute for complete study registries or sound experimental design. Simonsohn et al. (2014)

Steegen and colleagues' 2016 multiverse analysis addressed a different issue. A conclusion may depend on several defensible ways to code, exclude, transform, or model observations. Instead of hiding these choices behind a single apparently inevitable analysis, a multiverse examines the distribution of results across explicit analytic alternatives. The approach makes modeling judgment visible, but its usefulness depends on a justified set of specifications; including obviously implausible models does not automatically improve the quality of inference. Multiverse analysis is therefore an instrument for understanding analytical sensitivity rather than a machine for deriving theory-free truth. Steegen et al. (2016)

Together the methods changed the historical meaning of a result. A published coefficient or p value became one possible outcome in a landscape of selection rules and analytic decisions, and inferential confidence depended partly on how that landscape was constructed. These developments also exposed a division of labor: preregistration constrains some flexibility prospectively, multiverse analysis reveals it retrospectively or prospectively, and independent replication tests whether the resulting claim extends to new observations. None solves the conceptual problem of choosing measurements or explanatory mechanisms by itself. Simonsohn et al. (2014) Steegen et al. (2016) Nosek et al. (2018)

Many Analysts and the Sociology of Statistical Judgment

The Many Analysts project, published by Silberzahn and colleagues in 2018, asked 29 research teams involving 61 analysts to address one question using a common dataset. Their analytic approaches produced odds-ratio estimates spanning 0.89 to 2.93, with variation in statistical significance. The study concerned potential differences in soccer referees' red-card decisions by players' skin tone. The project showed that even when datasets and research questions are shared, legitimate analysis decisions can yield materially different reported conclusions. The quantitative results document one complex analysis challenge, not an estimate of variation for every psychological dataset. Silberzahn et al. (2018)

This investigation added a social dimension to the crisis. Scientific interpretation depends partly on expert judgments that differ across analytic communities: choice of controls, data cleaning, model family, uncertainty treatment, and assumptions about causal structures. The authors also explored whether analyst expertise or prior beliefs straightforwardly explained the observed diversity; the published abstract reports that they did not readily do so. The broader methodological point is that analysis is a skilled practice containing choices whose effects are difficult to anticipate from the final table alone. Silberzahn et al. (2018)

Many-analyst designs invite transparency and deliberation over alternative models. A crowd of analysts does not create a democratic substitute for methodological validity, because some specifications may answer different questions or depend on stronger assumptions. Nevertheless, publicly documenting multiple defensible approaches can reveal the sensitivity of a claim and help an audience distinguish robust patterns from model-dependent statements. Such projects extended reform beyond the replication of experimental effects into the reproducibility of statistical reasoning. Silberzahn et al. (2018) Steegen et al. (2016)

Empirical Evaluations of Registered Reports

The proliferation of Registered Reports made the reforms themselves open to empirical investigation. Scheel, Schijen, and Lakens compared the first tested hypothesis in 71 Registered Reports with a random sample of 152 standard psychology reports. Their 2021 analysis found positive results in approximately 44% of Registered Reports compared with approximately 96% in standard reports. The difference is compatible with a reduction in results-dependent publication and other sources of bias, but differences in study selection, topics, replications, and editorial settings complicate causal attribution. The result directly measures a difference between published samples, not an experimentally isolated effect of the editorial format. Scheel et al. (2021)

Soderberg and colleagues studied published Registered Reports and conventional comparison papers through structured evaluation by 353 researchers. In their 2021 study, reviews involved 29 Registered Reports and 57 comparison papers in psychology and neuroscience. Registered Reports received higher numerical ratings on all 19 evaluated criteria, with particularly marked differences for methodological rigor and analysis; differences in novelty and creativity were statistically uncertain. The study therefore provides evidence relevant to concerns that results-independent review must trade away intellectual value. Its observational comparison, rating scales, and selected paper populations still limit generalization. Soderberg et al. (2021)

These two investigations addressed distinct outcomes: the distribution of reported findings and evaluations of article quality. Their convergence supports cautious interest in editorial reform while leaving many questions open. Which researchers and journals choose Registered Reports? Would effects persist as the model expanded across disciplines? Can the review process accommodate complex observational and qualitative investigations? Does higher evaluated rigor translate into more stable estimates under future replication? The reform movement's scientific maturation is indicated by its willingness to measure these questions rather than relying entirely on programmatic advocacy. Scheel et al. (2021) Soderberg et al. (2021)

Transparency Audits and the Difference Between Policy and Practice

To assess institutional reform, investigators began measuring the actual accessibility of research inputs. Hardwicke and colleagues' 2020 study examined transparency and reproducibility-related practices in a sample of social-science articles published between 2014 and 2017. Such audits evaluate whether data, materials, protocols, and other research records are available in forms sufficient for inspection. This approach treats public openness as an observable property of the published research system. It complements survey-based assessments of investigators' attitudes and policy descriptions that cannot by themselves prove research materials were preserved. Hardwicke et al. (2020)

A complete audit must distinguish the presence of a sharing statement from successful recovery and use of a resource. File links can expire, anonymization may be inadequate, documentation may lack variable definitions, and software versions may affect the possibility of recomputing an analysis. Ethical limits on open sharing are themselves significant, especially when psychological data include participant identities or sensitive behavioral information. Transparency can then be supported through controlled access, metadata, reproducible scripts, and clear explanations of restrictions. The relevant criterion is that the evidential pathway is reviewable to the degree justified by the underlying data and consent arrangements. Hardwicke et al. (2020) Munafò et al. (2017)

Historically, these audits gave reform advocates and critics a common object of measurement: compliance and utility. Journals could introduce a policy and subsequently assess whether published articles met it. Repositories could refine preservation standards and educators could train researchers in documentation as part of scientific practice. The move from manifesto to empirical quality control represented a major institutional advance, although its effectiveness depends on sustained resources and continued review. Nosek et al. (2015) Kidwell et al. (2016) Hardwicke et al. (2020)

Histories of Crisis and the Recurrence of Fundamental Questions

Historians and methodologists have questioned the narrative that psychology suddenly discovered reproducibility problems around 2011. Lakens's 2025 comparative work reconstructs a crisis of confidence in social psychology extending from the 1960s through the late 1970s and compares it with the debates of the 2010s. Across these periods, researchers expressed concerns about replicability, theory development, generalizability, societal relevance, and scientific methods. The recurrent vocabulary challenges progress narratives that treat methodological criticism as an unprecedented moral awakening. It suggests that a disciplinary problem can be clearly recognized before professional arrangements produce durable responses to it. Lakens (2025) Teymoori & Trappes (2026)

Teymoori and Trappes's 2026 history and philosophy analysis locates major crises within enduring tensions surrounding the objects and epistemology of psychology. Their account expands historical explanation beyond the mechanics of p values. A field can improve statistical procedures while remaining divided over the ontological status of its constructs, the appropriate relation between laboratory tasks and everyday action, or the diversity of evidence required for general theories. Such debates have their own histories and cannot be settled by replication alone. In this sense, the twenty-first-century crisis is part of a longer institutional history of how psychological science decides what counts as knowledge. Teymoori & Trappes (2026)

A comparative historiography must preserve meaningful differences across periods. The earlier debates lacked contemporary online registries, large open collaboration platforms, and the same intensity of computational meta-research. The 2010s combined inherited epistemic concerns with faster information exchange, shared data infrastructure, and prospective multi-laboratory tests. The claim of recurrence is therefore analytic, not a claim that the same event repeated unchanged. Understanding continuity and change helps identify which reforms genuinely altered professional incentives and which tensions remain features of psychology's relation between theory, measurement, and experimental context. Lakens (2025) Nosek et al. (2012) Teymoori & Trappes (2026)

Have Research Results Changed Since the Crisis?

Measuring improvement after methodological reform is more difficult than documenting the adoption of a new journal policy. A change in published p values can reflect altered sample sizes, stronger effects, different study topics, changed model choices, editorial selection, or the use of registered designs. A larger fraction of apparently decisive results does not directly establish that individual claims will independently replicate. Evidence for scientific improvement therefore requires multiple measures: accuracy of preregistration, availability of data and code, distribution of null results, precision of estimates, and success under prospective new-data evaluation. Bogdan (2025) Hardwicke et al. (2020)

Paul C. Bogdan's 2025 study analyzed reported p values across 240,355 empirical psychology articles published from 2004 through 2024. It found a broad shift toward stronger reported statistical evidence across subdisciplines and examined relationships with journal placement, citation, and institutional incentives. The analysis documents a large descriptive change in the published record. Its evidence should not be read as a direct replication of the findings in those articles or a causal demonstration that specific open-science policies produced the shift; the measure captures reported inferential patterns under changing scientific and publication practices. Bogdan (2025)

An historical assessment of reform should be longitudinal and plural. Periods of rapid institutional change may improve some outcomes faster than others, with transparency policies easier to document than deep improvements in theoretical discrimination or population generalizability. Scientific fields also change their target questions, such that more demanding studies can produce weaker p values while carrying greater explanatory value. The proper outcome is an increasingly reliable and corrigible body of knowledge, evaluated through carefully specified indicators and independent checks rather than one headline figure. Bogdan (2025) Lakens (2025) Shrout & Rodgers (2018)

Generalizability, Theoretical Specificity, and Diverse Populations

Replication often brings the boundary conditions of a psychological claim into focus. A study performed with one student population, one language, and one procedural framing may reveal a stable association for that context without establishing a general property of human cognition or social behavior. Multi-site projects expand the number of contexts available for comparison, but their recruitment networks and task selections may still be selective. Theoretical progress requires models that identify what should remain invariant and which factors are expected to moderate the effect. Klein et al. (2018) Shrout & Rodgers (2018)

Historical critiques of psychology have repeatedly questioned whether laboratory convenience became mistaken for universality. The crisis of replicability reinforced this concern by making contextual heterogeneity visible across collaborators and samples. Yet subgroup explanations proposed after seeing discrepant replication results need independent tests. Otherwise a research program can preserve any disputed conclusion by continually introducing unverified conditions under which it supposedly holds. The credible alternative is to derive moderator predictions from explanatory assumptions and test those predictions through coordinated designs and adequate precision. Lakens (2025) Klein et al. (2018) Nosek et al. (2018)

The problem extends beyond demographic diversity. A psychological task carries historical and cultural meanings, and experimental instructions can change with technologies, institutions, and participant expectations. In longitudinal comparisons, the same named instrument may no longer instantiate the same construct. Historical reconstruction of stimuli, recruitment environments, and measurement rules is consequently indispensable for understanding why some apparent replications succeed and others diverge. The study of reproducibility ultimately connects methodological rigor with careful theory construction, construct validity, and transportability. Teymoori & Trappes (2026) Shrout & Rodgers (2018)

Interpretation, Disagreement, and Epistemic Accountability

Public disputes over replication frequently centered on the meaning of a discrepancy. One party emphasized the original theoretical account; another emphasized a preregistered comparison that did not reproduce the original evidence. Both could be responding to genuine features of the scientific problem. An informative evaluation specifies the claim under test, the necessary design conditions, measurement validity, competing explanations, and the precision of each estimate. It also states which aspects of the disagreement can be tested with new observations. Replication research becomes cumulative when explanatory alternatives become more constrained after every study. Shrout & Rodgers (2018) Open Science Collaboration (2015)

Disagreement is also shaped by professional stakes. A significant finding may underpin decades of accumulated literature, teaching material, laboratory identity, or public advice. When independent tests challenge that foundation, uncertainty can affect careers and disciplinary reputations. However, historical analysis should separate these structural incentives from an unsupported judgment about an individual investigator's intentions. Statistical errors, selective publication, different contexts, and genuine model failure can occur without deliberate deception. Conversely, claims of scientific integrity are strengthened by the inspectability of decisions, datasets, protocols, and corrections. Nosek et al. (2012) Munafò et al. (2017)

The legitimacy of methodological reform depends on a reciprocal norm. Researchers advancing surprising findings should provide material that permits serious testing, while replication teams should document protocols, engage with theory, and report uncertainty. Journal editors and reviewers should evaluate the evidential contribution rather than prefer confirmation or contradiction as a result in itself. The historical transition from singular controversy to coordinated multi-laboratory investigation illustrates the practical value of this norm. The measure of success is whether disagreements produce better evidence and sharper research questions. Hagger et al. (2016) Klein et al. (2018) Shrout & Rodgers (2018)

Artificial Research Systems and the Next Stage of Reproducibility

Recent research using artificial systems introduces new questions into this historical trajectory. Cui, Li, and Zhou's 2025 Nature Computational Science study used three large language models to examine 156 scenario-based experimental claims from psychology and management. They reported simulated main-effect replication rates of 73%–81% and lower rates for interactions, with effect sizes systematically larger than those in human studies. The experiment measures concordance between language-model-generated responses and selected human experimental findings under modeled scenarios. It is evidence about artificial simulation and its limitations, rather than direct new-human-participant replication. Cui et al. (2025)

Artificial research tools can support literature retrieval, protocol drafting, data preparation, code generation, and exploration of possible analytic outcomes. Their use also creates additional sources of variation: model versions, instructions, stochastic sampling, training-data exposure, and undocumented system changes can affect apparent results. Consequently, reproducibility requires versioned configurations, preserved prompts and evaluation code, explicit model provenance, and tests that distinguish re-execution from independent validation. The historical analogy is with the earlier challenge of researcher degrees of freedom: invisible choices can produce a persuasive final report whose path to production is difficult to reconstruct. Cui et al. (2025) Simmons et al. (2011) Steegen et al. (2016)

The Artificial Era expands the methodological scope of psychology to include studies of human–artificial interaction and research practices assisted by artificial systems. For such studies, operationalized claims must specify the relevant human participants, artificial components, communication context, and versioned tasks. A model's consistency across repeated calls addresses a different evidential question from the validity of its simulation of human populations. Open science offers an established governance vocabulary—traceability, preregistration, shared protocols, and independent scrutiny—whose application will require revision as research systems evolve. These proposals form a methodological perspective on contemporary scientific infrastructure, not a claim that artificial simulation is interchangeable with human experimentation. Cui et al. (2025) Nosek et al. (2018) Munafò et al. (2017)

The Architecture of Evidence in a Reformed Discipline

The replication crisis transformed the evidential architecture of psychological research by exposing multiple points at which a defensible experiment can become a misleading published claim. Weak statistical power undermines precision; flexible analytic choices can make chance patterns look confirmatory; selective publication changes the composition of the visible literature; and insufficient materials obstruct independent scrutiny. These problems interact. A well-powered experiment can still be affected by measurement ambiguity, while a transparent archive of biased studies can transmit biased conclusions. The central historical achievement of open science was to render those dependencies explicit and to create procedures for auditing them. Cohen (1962) Simmons et al. (2011) Munafò et al. (2017)

The reform toolkit is correspondingly plural. Preregistration documents which predictions preceded the evidential test; Registered Reports separate publication decisions from observed outcomes; multi-laboratory replication estimates effects in new samples; open data and code support computational scrutiny; sensitivity analyses reveal modeling dependence; and research synthesis evaluates the broader distribution of evidence. These practices operate at different stages and answer distinct questions. Their cumulative value depends on their coherence with the scientific claim, not on formal compliance with a checklist detached from the theory or measurement. Nosek et al. (2018) Chambers (2013) Klein et al. (2018) Steegen et al. (2016)

An adequate contemporary evaluation therefore records the study's epistemic status: what was predicted in advance, which measurements operationalized the proposed construct, which analyses were specified or added, how participants were selected, which datasets are available, and what independent work has tested the conclusion. The relevant outcome is a graded evidential judgment supported by identifiable decisions and sources. This orientation continues a long history of methodological criticism while giving it practical implementation in modern research infrastructure. Shrout & Rodgers (2018) Teymoori & Trappes (2026)

Limits of the Evidence Base and Historical Reconstruction

This monograph uses a critical narrative method: selected original methodological interventions, landmark empirical replications, institutional reform publications, and recent historical evaluations are compared to reconstruct major changes in psychological science. It does not use a preregistered search strategy, exhaustive screening record, formal risk-of-bias instrument, or pooled effect-size model. Accordingly, its conclusions concern well-documented developments in the research history, while any estimate of the present prevalence of unreliable results would require a separate sampling frame and defined evaluative criteria. Open Science Collaboration (2015) Shrout & Rodgers (2018)

The evidence is heterogeneous. Some sources are original empirical studies, including multi-laboratory projects and audits of research practices; others propose reforms or reconstruct disciplinary debates. Their inferential status differs. A manifesto establishes the existence and rationale of a proposal, not its effectiveness. A journal's adoption of policy establishes an institutional intervention, while a before–after audit assesses a particular outcome under its own selection and confounding structure. Historical interpretation must likewise distinguish evidence that a debate occurred from evidence that a specific policy produced a change in replicability. Munafò et al. (2017) Kidwell et al. (2016) Hardwicke et al. (2020)

Several questions remain open for researchers and historians: which interventions produce durable improvements under rigorous causal evaluation; how transparent practices can accommodate qualitative, longitudinal, and ethically restricted research; what theoretical features predict transfer across samples; how publication selection evolves when incentives change; and how artificial research tools affect the provenance of empirical claims. Progress will depend on longitudinal institutional evidence, prospective replications, careful construct validation, and better records of both successes and unsuccessful scientific attempts. Scheel et al. (2021) Lakens (2025) Cui et al. (2025)

Historical Periodization of the Replication Crisis

The chronology reconstructed here identifies a series of overlapping transformations. In the first, spanning the middle decades of the twentieth century, critics documented selection against nonsignificant results, inadequate power, and weak links between statistical decisions and substantive theory. A second phase developed approaches to research synthesis while exposing their vulnerability to the file drawer. A third phase, culminating around 2011–2012, translated these concerns into prominent demonstrations of false positives, analytical flexibility, and institutional incentives. A fourth phase established coordinated empirical tests of major published findings and brought the term replication crisis into widespread disciplinary use. Sterling (1959) Cohen (1962) Rosenthal (1979) Simmons et al. (2011) Open Science Collaboration (2015)

A fifth transformation institutionalized reform through preregistration, results-independent review, shared materials, reporting guidance, professional credit, and multi-analyst evaluation. The movement's credibility increasingly depended on empirical tests of those interventions, including direct comparison of publication formats and audits of accessible datasets. A sixth, still ongoing, integrates replication science with historical reflection on theory, generalizability, scientific culture, and artificial research infrastructure. The phases overlap, and significant precursors occur before their period of greatest public prominence. Periodization here is an explanatory aid rather than a claim about universal dates across all psychology subfields. Chambers (2013) Nosek et al. (2015) Scheel et al. (2021) Teymoori & Trappes (2026)

The same chronology demonstrates how discovery becomes science through social arrangements. Data are collected in institutions, interpreted under statistical and theoretical conventions, selected through editorial procedures, and stabilized through citation and teaching. The replication movement made those arrangements visible as mechanisms governing knowledge. Its reforms changed not only the amount of evidence available but also the conditions under which claims can be challenged, repeated, corrected, and retained. The historical meaning of the crisis lies in that shift from treating publication as final certification toward treating it as one stage in cumulative inquiry. Nosek et al. (2012) Munafò et al. (2017) Lakens (2025)

A Research Agenda for the History and Evaluation of Open Science

Future historical work should reconstruct institutional diffusion with greater precision. The dates on which a journal announced transparency standards, the dates on which authors began complying, and the dates on which independent reviewers could reproduce submitted analyses are distinct observations. Archive-based research can examine editorial correspondence, publicly versioned policies, repository metadata, peer-review histories, and researcher interviews to explain the uneven uptake of reform. Such evidence can test whether highly visible initiatives transformed everyday research routines or remained concentrated in particular journals and laboratories. Nosek et al. (2015) Hardwicke et al. (2020)

Evaluation of policy effectiveness also benefits from designs that isolate mechanisms. Observational comparisons between Registered Reports and traditional articles can motivate hypotheses, but stronger attribution would require careful accounting for author and journal selection, research topics, design quality, and time trends. Multi-laboratory collaborations can evaluate reproducibility while explicitly measuring population heterogeneity, and longitudinal analyses can distinguish improvements in reporting from improvements in predictive validity. Historical analysis contributes by identifying institutional changes, plausible comparison periods, and neglected evidence that contemporary datasets alone cannot recover. Scheel et al. (2021) Soderberg et al. (2021) Klein et al. (2018)

The broadest research agenda asks what makes psychology cumulatively informative across generations of researchers and changing technical systems. That requires preserving original study materials and versioned computational environments, reporting uncertainty as an essential component of scientific claims, creating incentives for sound tests with any outcome, and developing precise theories whose predictions can survive or fail under independent examination. Open science can provide the institutional conditions for these practices, while philosophy and history of psychology clarify their conceptual foundations. Nosek et al. (2018) Teymoori & Trappes (2026) Cui et al. (2025)

Conclusion

The replication crisis is a historically situated transformation in how psychology evaluates its empirical claims. Its antecedents include decades of documented publication selection, inadequate statistical power, theoretical underdetermination, and undisclosed analytical flexibility. The highly visible disputes of the early 2010s and the multi-laboratory results published during the following years brought those concerns into coordinated public examination. The record demonstrates that published significance, estimated effect magnitude, replicability, and generalizability must be interpreted as related but distinct dimensions of evidential strength. Sterling (1959) Cohen (1962) Simmons et al. (2011) Open Science Collaboration (2015)

Open science established a practical reform architecture that made more of the research process inspectable. Preregistration and Registered Reports alter when decisions are documented or evaluated; large collaborations test claims in independent samples; open data and code permit scrutiny of computation; and transparent sensitivity analysis reveals how conclusions depend on legitimate analytic choices. Empirical investigations of reform show measurable differences in reporting behavior and published outcome distributions, while also exposing the limits of observational comparison and uneven implementation. The history of reform is consequently an ongoing program of methodological and institutional learning. Kidwell et al. (2016) Scheel et al. (2021) Soderberg et al. (2021)

The enduring scientific task is to connect credible methods with precise concepts, valid measurement, diverse contexts, and institutions that reward cumulative correction. The recurring disciplinary questions identified by recent historians concern how psychology defines its objects, tests explanatory claims, and preserves the integrity of the evidence through which those claims become public knowledge. As artificial systems enter both the subject matter and the practice of psychological research, these questions acquire new technical forms while retaining the need for transparent provenance and independent validation. The mature legacy of the replication crisis is a research culture capable of making uncertainty explicit, investigating disagreements, and revising its conclusions through accessible evidence. Shrout & Rodgers (2018) Lakens (2025) Teymoori & Trappes (2026) Cui et al. (2025)

References

  • Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543–554. https://doi.org/10.1177/1745691612459060
  • Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425. https://doi.org/10.1037/a0021524
  • Bogdan, P. C. (2025). One decade into the replication crisis, how have psychological results changed? Advances in Methods and Practices in Psychological Science, 8(2). https://doi.org/10.1177/25152459251323480
  • Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. https://doi.org/10.1126/science.aaf0918
  • Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., Heikensten, E., Hummer, L., Imai, T., ... Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2, 637–644. https://doi.org/10.1038/s41562-018-0399-z
  • Chambers, C. D. (2013). Registered Reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. https://doi.org/10.1016/j.cortex.2012.12.016
  • Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65(3), 145–153. https://doi.org/10.1037/h0045186
  • Cui, Z., Li, N., & Zhou, H. (2025). A large-scale replication of scenario-based experiments in psychology and management using large language models. Nature Computational Science, 5, 627–634. https://doi.org/10.1038/s43588-025-00840-7
  • Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1–20. https://doi.org/10.1037/h0076157
  • Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., Brand, R., Brandt, M. J., Brewer, G., Bruyneel, S., ... Zwienenberg, M. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546–573. https://doi.org/10.1177/1745691616652873
  • Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020). An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). Royal Society Open Science, 7(2), 190806. https://doi.org/10.1098/rsos.190806
  • Ioannidis, J. P. A. (2005). Why most published research findings are false. PLOS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
  • John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
  • Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4
  • Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., Errington, T. M., Fiedler, S., & Nosek, B. A. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLOS Biology, 14(5), e1002456. https://doi.org/10.1371/journal.pbio.1002456
  • Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., ... Nosek, B. A. (2014). Investigating variation in replicability: A 'Many Labs' replication project. Social Psychology, 45(3), 142–152. https://doi.org/10.1027/1864-9335/a000178
  • Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Jr., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., ... Nosek, B. A. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225
  • Kvarven, A., Strømland, E., & Johannesson, M. (2020). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour, 4, 423–434. https://doi.org/10.1038/s41562-019-0787-z
  • Lakens, D. (2025). Concerns about theorizing, relevance, generalizability, and methodology across two crises in social psychology. International Review of Social Psychology, 38, Article 6. https://doi.org/10.5334/irsp.1038
  • Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34(2), 103–115. https://doi.org/10.1086/288135
  • Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, Article 0021. https://doi.org/10.1038/s41562-016-0021
  • Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., ... Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374
  • Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
  • Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–631. https://doi.org/10.1177/1745691612459058
  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
  • Rosenthal, R. (1979). The 'file drawer problem' and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
  • Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2). https://doi.org/10.1177/25152459211007467
  • Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90–100. https://doi.org/10.1037/a0015108
  • Shrout, P. E., & Rodgers, J. L. (2018). Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual Review of Psychology, 69, 487–510. https://doi.org/10.1146/annurev-psych-122216-011845
  • Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., ... Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
  • Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547. https://doi.org/10.1037/a0033242
  • Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021). Initial evidence of research quality of Registered Reports compared with the standard publishing model. Nature Human Behaviour, 5, 990–997. https://doi.org/10.1038/s41562-021-01142-4
  • Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. https://doi.org/10.1177/1745691616658637
  • Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54(285), 30–34. https://doi.org/10.1080/01621459.1959.10501497
  • Teymoori, A., & Trappes, R. (2026). The recurrence of fundamental questions: A historical and philosophical analysis of major disciplinary crises in psychology. Review of General Psychology, 30(2), 195–211. https://doi.org/10.1177/10892680261421875
  • Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi. Journal of Personality and Social Psychology, 100(3), 426–432. https://doi.org/10.1037/a0022790