Psychology by BogdanovaThe Encyclopedia of Psychology for the Artificial Era

RESEARCH DOSSIER

Replication and Robustness

Abstract. Replication and robustness concern the stability of psychological claims under independent data collection, reproducible computation, altered analytical assumptions, and carefully specified changes in research conditions. This fundamental research monograph reconstructs the historical emergence of reproducibility problems, analyzes the Open Science Collaboration, Many Labs, preregistered social-science replication programs, ego depletion, facial feedback, and power posing, and distinguishes statistical compatibility, substantive effect magnitude, and theoretical explanation. It examines selective reporting, statistical power, preregistration, Registered Reports, multiverse and specification-curve analysis, many-analyst variation, construct validity, context dependence, computational workflows, neuroimaging, causal identification, and evidence synthesis. The concluding analysis identifies unresolved methodological problems and requirements for replicable human–artificial psychological research. The study is a critical narrative synthesis of published literature, not a new experiment or registered systematic review.

Keywords: replication; reproducibility; robustness; replication crisis; psychological science; statistical power; preregistration; registered reports; multiverse analysis; specification curve; many-analyst studies; research integrity; metascience; human–artificial research; Artificial Era

Replication and Robustness as a Scientific Object

Replication and robustness concern the conditions under which a psychological claim remains evidentially credible when it is examined again. Replication asks whether new observations obtained under an intelligibly related protocol support an effect, relation, or prediction. Robustness asks how an inference changes when defensible decisions about measurement, exclusion, modeling, estimation, and implementation are varied. The unit of evaluation is a specified claim about a defined population, manipulation, outcome, and inferential target. A published significance test is only one observation of that claim's empirical behavior. Replication becomes informative when the target, the sampling mechanism, the comparison criterion, and the uncertainty associated with a new estimate are articulated before the result is known Open Science Collaboration (2015) Lakens (2022).

Four evidential questions organize this monograph. Can independent investigators recreate an analysis from the same inputs? Does another sample under a closely matched design produce compatible evidence? Does a claim survive plausible analytic and measurement alternatives? How should apparently discrepant outcomes change a theoretical judgment? These questions require separate answers because a perfectly reproducible computation may estimate a biased target, an effect that replicates in one laboratory may depend on an unmeasured moderator, and a robust association may still permit rival causal explanations. The methodological task is to identify the source of stability rather than attach a single success label to an entire research program Munafò et al. (2017) Poldrack et al. (2017).

This investigation is a critical narrative synthesis of published experimental replication projects, methodological studies, reporting interventions, and metascientific analyses. It does not report a newly executed replication, preregistered systematic search, quantitative meta-analysis, or exhaustive census of psychology. Its principal evidential material includes the Open Science Collaboration, Many Labs, registered replication reports, multiverse and many-analyst exercises, and prospective methodological reforms. Specific numerical results are identified as properties of particular source projects; they are not pooled into a universal percentage of psychological truth Open Science Collaboration (2015) Klein et al. (2018) Silberzahn et al. (2018).

Terminology, Targets, and Levels of Repetition

Computational reproducibility denotes the ability to obtain the same analytic output from the same data, code, preprocessing decisions, and computing environment, subject to numerical tolerances. Direct replication refers to new data collected through procedures designed to approximate the theoretically relevant conditions of an original investigation. Conceptual replication tests a related prediction through altered operationalizations. Statistical robustness concerns the persistence of an inference across specified admissible analyses. These labels describe different operations and should be tied to observable criteria, because a failed software rerun and an independently collected null result diagnose different possible faults Poldrack et al. (2017) Open Science Collaboration (2015).

A replication is interpretable only after the estimand is clarified. The original may estimate a mean treatment contrast, a standardized difference, an association, a latent-construct parameter, or a conditional effect within a selected subgroup. A follow-up that changes outcome scaling, stimulus exposure, participant recruitment, or estimand may address a nearby scientific question rather than the same one. Variations can be scientifically productive when documented and theorized. They become analytically misleading when all observations are treated as interchangeable evidence for an unspecified effect. Replication planning therefore begins with a precise mapping from theoretical claim to operational protocol and inferential estimand Lakens (2022) Simonsohn et al. (2020).

Robustness is graded and multidimensional. An effect might retain its direction under all reasonable codings while its magnitude varies substantially; it might remain statistically distinguishable from zero but fail a prespecified threshold of practical relevance; or it might depend on the removal of a small number of influential observations. Claims about robustness must state which parameter, loss function, direction, interval, or substantive conclusion is expected to remain stable. The set of permissible analytical variations is itself an object of justification and should not include clearly invalid models merely to inflate the apparent range of alternative estimates Steegen et al. (2016) Simonsohn et al. (2020).

Historical Antecedents of Cumulative Psychological Evidence

The contemporary replication movement emerges from much older problems in experimental inference. A scientific demonstration has always depended on the possibility that other investigators can inspect conditions, reproduce procedures, and challenge an interpretation. In quantitative psychology, repeated measurements and independent experimental comparisons gradually became central to evaluating regularities rather than singular demonstrations. Publication selection remained a recurring concern: Rosenthal (1979) formulated the file-drawer problem by emphasizing that a literature assembled from visible statistically favorable studies could misrepresent the set of studies conducted.

The problem was developed mathematically and methodologically over subsequent decades. Kerr (1998) described hypothesizing after results are known, a reporting practice that can present an explanation developed from observed data as though it had been independently predicted. Ioannidis (2005) analyzed how prior plausibility, power, bias, and multiplicity can influence the credibility of claimed discoveries. These contributions serve distinct purposes: a warning about unavailable results, an analysis of claim-generation and reporting, and a formal account of positive predictive value. None provides, by itself, an empirical census showing that a fixed fraction of psychological findings are false.

The institutional phase accelerated in the early 2010s, when investigative attention shifted from isolated disputes to the incentives and discretionary choices built into normal research workflows. Simulations and experiments by Simmons et al. (2011) showed how unreported flexibility can inflate false-positive findings, while the survey of psychologists by John et al. (2012) documented self-reported questionable practices. Bakker et al. (2012) examined the incentive to obtain significance through a series of small studies and found indications of excess significant findings within the samples of meta-analyses they studied. These studies illuminate mechanisms capable of affecting published evidence, while their designs and sampling frames limit how broadly their estimates should be extrapolated.

The Structure of a Replication Claim

A precise replication statement identifies the original target and explains what counts as an informative comparison. The target includes participant eligibility, recruitment source, assignment procedure, manipulation fidelity, stimulus properties, timing, outcome definition, missing-data rules, and statistical model. It also includes the smallest effect regarded as theoretically or practically meaningful. Without that last component, a conventionally nonsignificant replication may be interpreted as proof of absence even when its interval accommodates effects that matter Lakens (2017) Lakens (2022).

Different indicators provide different answers. A directional significance result tests whether data are unusual under a specified null model; an estimated effect and interval describe compatible magnitudes; an equivalence test assesses whether effects of a prespecified meaningful size can be excluded; and a Bayesian comparison assesses evidence between articulated hypotheses conditional on prior and model choices. An original estimate lying outside a replication interval is a different criterion from a same-direction significant result, particularly when the original is selected for significance and may be inflated. Open Science Collaboration (2015) expressly evaluated multiple operational criteria because replication success has no uniquely sufficient binary definition.

Evaluation should compare total evidence with the claim's intended scope, including statistical uncertainty and relevant design differences. A replication may reduce confidence in a large effect while leaving evidence for a smaller effect. Conversely, a nominally successful repetition may reveal a magnitude too unstable to support clinical, educational, or policy application. Epistemic assessment is continuous: study design, intervention integrity, data quality, inferential calibration, and substantive importance all matter. This approach is consistent with effect-size justification and equivalence methodology Lakens (2017) Lakens (2022).

Statistical Power, Precision, and the Winner's Curse

Replication outcomes are affected by the uncertainty of both the original study and its follow-up. A small study with low power can produce a highly variable estimate; selecting such estimates for significance tends to favor unusually large values among those published. This selection mechanism is often called the winner's curse. Consequently, a well-powered replication can obtain a smaller effect even when the underlying association or causal contrast is real. Bakker et al. (2012) modeled how the pursuit of significance through multiple small samples amplifies this mechanism, and Ioannidis (2005) placed power and selection into a broader model of scientific credibility.

Power calculations are useful only when their assumed target magnitude and decision rule are defensible. An estimated magnitude from a small selected original study may be a poor planning value. A design can instead target an effect size regarded as substantively worthwhile, a desired interval width, or another clearly specified inferential goal. Lakens (2022) explains that resource constraints, power, precision, and the smallest effect of interest can all be part of an explicit justification. An exact numerical sample-size recipe cannot be inferred from a generic replication label because it depends on design, variability, and the intended conclusion.

Precision matters when interpreting a null result. An interval that includes both meaningful benefit and meaningful harm supports different conclusions from an interval confined within a narrow equivalence region. Lakens (2017) develops equivalence testing to make the inference about sufficiently small effects explicit. Sensitivity analyses should also quantify how plausible attrition, exclusions, heteroskedasticity, clustering, and measurement unreliability alter the interval. A replication whose estimate is nominally close to zero but weakly identified still leaves the theory underdetermined; this is a property of the evidence, not a reason to discount the replication team.

The Open Science Collaboration and Its Interpretation

The Open Science Collaboration's Open Science Collaboration (2015) project is a landmark empirical examination of research claims published in three psychology journals. It organized replications of 100 experimental and correlational studies using greater statistical power and original materials when available. In that selected corpus, 97% of original studies reported statistically significant results, compared with 36% of replication studies under a conventional significance criterion. The mean replication effect size was substantially smaller than the original average; the authors also reported different outcomes under subjective, interval-based, and combined-effect criteria. Those figures belong to a defined selection of journals, study periods, designs, and replication protocols.

Two implications follow from the multiplicity of criteria. First, a replication rate is a conditional summary: changing the outcome definition changes the rate without changing any participant-level observations. Second, the project reveals distortions in the distribution of published findings but does not assign a universal reliability score to psychology. Selection into a replication project, statistical power, effect heterogeneity, and protocol changes can all affect observed rates. The original paper treated this as an initial estimate of reproducibility rather than the population prevalence of true and false psychological mechanisms Open Science Collaboration (2015).

The project's methodological contribution extends beyond any headline percentage. The infrastructure made study materials, independent teams, coordination, and openly described criteria central to assessment. It also forced a productive distinction between disagreement with a specific result and diagnosis of the investigative process. A field can learn from failures when the uncertainty is attributable to an identifiable sampling or procedural difference; ambiguous replications that omit their design rationale add far less information. The lasting lesson concerns the organization of cumulative inquiry and the need to design replications for inference Munafò et al. (2017).

Many Labs and Cross-Site Replication

Many Labs 1 tested 13 classic and contemporary effects across 36 independent samples comprising 6,344 participants. Klein et al. (2014) reported consistent replication of 10 effects, weak evidence for one, and failures for two; among the effects tested, the study found relatively limited explanatory contributions from several investigated laboratory and sample-context factors. This was a constrained set of phenomena with particular protocols, rather than a direct demonstration that context never moderates psychological effects. The experiment made it possible to compare effect variability within one coordinated research structure.

Many Labs 2 expanded the methodological question by testing 28 findings across 125 samples involving 15,305 participants from 36 countries and territories. Klein et al. (2018) found 15 of the 28 replications significant in the original direction under a conventional threshold, with study-specific patterns of heterogeneity. The project standardized protocols and reviewed them in advance, making it easier to separate procedural differences from variability across participating samples and locations. Yet the conclusions still depended on the selected effects, design fidelity, populations reached, and particular inferential indicator used.

Multisite coordination is valuable because it estimates more than a single pooled average. Investigators can examine between-site variance, differences in recruitment, instrument translation, experimenter training, delivery mode, and systematic departures from a shared protocol. A pooled result can obscure a small effect that is uniform, a larger effect present only in some settings, or heterogeneity dominated by measurement problems. Site-specific estimates and design documentation should accompany pooled conclusions. Replication across countries also does not automatically guarantee representation of all human populations, because participating samples remain selected Klein et al. (2018) Lakens (2022).

Social Science and Economics Replication Programs

Replicability extends across behavioral sciences, and comparisons across projects require attention to sampling and disciplinary differences. Camerer et al. (2016) investigated replicability of laboratory experiments in economics, demonstrating that the challenge is not limited to psychology. The project combined independent testing with explicit replication protocols. Its value for a psychological methodology dossier lies in the possibility of comparing institutional incentives, operational constraints, and behavioral tasks across related experimental traditions.

In a further project, Camerer et al. (2018) selected 21 social-science experiments published in Nature and Science between 2010 and 2015. Replication plans were reviewed by original authors and preregistered, and replication samples were on average substantially larger. Thirteen of the 21 studies, 62%, produced a significant effect in the original direction; the average replication effect magnitude was approximately half that of the original findings. The study also compared replicability indicators and community predictions. These findings characterize those selected high-profile experiments, not all published social science.

The cross-project comparison suggests a recurring empirical pattern of smaller follow-up estimates after original publication. However, raw comparisons between the 2015 psychology corpus and the later high-profile social-science corpus would be confounded by selection rules, experimental domains, sample types, and success definitions. Explanations may include publication selection, genuinely unstable effects, different populations, or implementation differences. A rigorous synthesis therefore resists a single transdisciplinary failure rate and instead asks which features of research design and reporting are associated with more stable estimates Open Science Collaboration (2015) Camerer et al. (2018).

Ego Depletion as a Replication Case

Ego depletion offers a case in which a prominent explanatory construct was operationalized through sequential self-control tasks. The resource model predicted that initial exertion would reduce performance on a subsequent task. Earlier studies and syntheses helped make this account influential, while concerns about publication bias, operational heterogeneity, and effect-size inflation motivated coordinated testing. The distinction between the effect as an empirical pattern and the resource theory as one explanation is essential. Even a stable sequential-task decrement would not uniquely demonstrate that a common mental resource had been depleted Hagger et al. (2016).

The preregistered multilaboratory report by Hagger et al. (2016) used 23 laboratories and 2,141 participants to test a common protocol. Its pooled standardized effect was d = 0.04, with a 95% confidence interval from −0.07 to 0.15. This result gave little support for a conventional positive effect under the selected sequential-task implementation. The protocol was intentionally specific, so the experiment should be understood as a strong test of that realization of the claim, with implications for claims of a broad, readily observable resource-depletion phenomenon.

The case illuminates how cumulative evidence changes theory. A small pooled estimate places pressure on strong general predictions and on reliance upon earlier selectively reported significant findings. It does not logically eliminate every possible fatigue, motivation, attention, or control mechanism. New variants must supply advance theoretical constraints and discriminating tests rather than indefinitely redefining the phenomenon after conflicting observations. Well-specified boundaries make refinement scientifically productive; unlimited post hoc rescue deprives the theory of independently testable content Hagger et al. (2016) Kerr (1998).

Facial Feedback and the Meaning of a Failed Protocol

Facial-feedback research investigates whether facial muscular configurations influence affective judgments or experiences. One influential protocol used the position of a pen held in the mouth to alter the musculature associated with smiling while participants evaluated cartoons. A broad facial-feedback theory admits multiple manipulations and outcomes, whereas the historical pen-in-mouth demonstration is one concrete test. Separating the theory from a particular apparatus helps locate the inferential target of later replications Wagenmakers et al. (2016).

The registered replication report coordinated by Wagenmakers et al. (2016) contained 17 independent direct replications of the original pen-in-mouth study. The report described a pooled difference in funniness ratings of approximately 0.03 units on a ten-point scale, with a 95% confidence interval from −0.11 to 0.16, compared with the original reported difference of 0.82 units. The findings challenged the reliability of that specific original effect under the standardized conditions. They also provided a more precise estimate than a single small follow-up could supply.

Interpretation requires specifying what would count as a boundary condition. Variations in camera presence, task demand, participant awareness, stimuli, and facial action measurement may matter, but their roles should be tested through prospective designs. A conceptual modification can probe mechanisms, while a closely matched protocol tests the historical result. Both are useful when their distinct targets are transparent. Treating any positive related finding as a direct replication, or any failure as definitive refutation of every version of facial-feedback theory, would collapse these methodological distinctions Wagenmakers et al. (2016) Simonsohn et al. (2020).

Questionable Research Practices and Analytical Flexibility

Analytical flexibility influences replication whenever researchers make consequential choices after seeing results. Decisions about exclusions, stopping rules, dependent variables, covariates, transformations, and subgroup selection multiply the set of possible tests. If only the most attractive result is reported as though it were the unique planned analysis, the nominal error rate no longer describes the full investigative process. Simmons et al. (2011) demonstrated through simulations and experimental examples how undisclosed flexibility can generate misleading positive results. The causal mechanism is a combination of selection among outcomes and a mismatch between the reported test and the procedure that produced it.

Questionable research practices are not a single behavioral category. Selective reporting, undisclosed exploratory analysis, data-dependent stopping, and retrospective hypothesis presentation affect inference in different ways. John et al. (2012) surveyed more than 2,000 psychologists using a design intended to encourage truthful reports of such practices. Survey admissions provide evidence that discretionary practices occurred among respondents, while uncertainty about sampling, memory, interpretation, and incentives prevents treating the findings as a precise timeless prevalence for the profession. Kerr (1998) describes how post hoc hypotheses can be misrepresented as prior predictions.

The practical response is to preserve an audit trail connecting decisions to their timing and rationale. An exploratory analysis may yield a valuable hypothesis if its status is reported; confirmatory credibility grows when the hypothesis and its evaluation are separated by independent data or prospectively specified testing. The reporting problem is therefore not the existence of researcher judgment. It is the invisibility of choices and their inferential consequences. Reporting all analyzed outcomes, disclosing exclusions and deviations, and evaluating sensitivity can make the chain of evidence inspectable Nosek et al. (2018) Munafò et al. (2017).

Preregistration and Temporal Separation of Inference

Preregistration records hypotheses, measures, exclusions, planned sample size, and analyses before relevant outcomes are observed. Its central purpose is to preserve the temporal distinction between prediction and postdiction. Nosek et al. (2018) explains this distinction and describes preregistration as a reform of the evidential record. The method is most valuable when the plan is sufficiently detailed to distinguish planned from opportunistic choices, is time-stamped before data-driven decisions, and remains accessible for comparison with the eventual report.

A preregistration may be incomplete, vague, or subsequently amended. Good practice treats amendments as dated versions, provides reasons for them, and reports the original plan alongside deviations. Prespecification cannot guarantee measurement validity, successful randomization, an unbiased sample, or a true causal model. A prespecified analysis of invalid data remains invalid. Likewise, preregistered significance does not eliminate uncertainty about effect magnitude or practical relevance. Preregistration should therefore be evaluated as a component of a broader design rather than a certification mark that resolves all methodological questions Nosek et al. (2018) Lakens (2022).

Replication investigators can prospectively register the criterion by which their study will change belief about an original claim. They may define the primary contrast, meaningful effect region, quality-control thresholds, and handling of missing data. They should also permit documented exploratory tests when unexpected patterns appear, because replication can generate new theoretical questions. The strongest evidential practice preserves both the confirmatory test and the exploratory follow-up as visible, separately interpretable components of the research record Munafò et al. (2017) Simonsohn et al. (2020).

Registered Reports and Publication Incentives

Registered Reports change the timing of editorial evaluation. Researchers submit their scientific rationale and methods for peer evaluation before outcomes are known, and an accepted protocol may receive an in-principle commitment to publication contingent on adherence to approved procedures and quality requirements. Chambers (2013) described an early implementation at Cortex. The model directly addresses the incentive to submit only positive, novel, or surprising outcomes, while encouraging scrutiny of design and analysis before study execution.

The distinction from ordinary preregistration matters. A researcher can preregister a study without seeking advance journal review or publication commitment; a Registered Report adds an editorial process and formal publication architecture. Both depend on thorough reporting of deviations and on meaningful assessment of evidence. For replication studies, registered review can reduce the chance that negative results are considered uninteresting after data collection. It can also improve comparability when original investigators and independent teams review procedural fidelity in advance Chambers (2013) Hagger et al. (2016).

Registered Reports involve tradeoffs in timing, institutional capacity, and types of research questions. Rapid exploratory inquiry, emergencies, archival work, and complex observational analyses may require adapted procedures. In-principle acceptance does not substitute for a study's execution quality, and reviewers can themselves disagree about which variables are essential. Evaluation should therefore focus on whether the format successfully isolates methodological merit from outcome desirability. The broader reform agenda calls for aligning rewards, training, and transparent evaluation across the research lifecycle Munafò et al. (2017).

Multiverse Analysis and the Robustness of Data Construction

An observed dataset results from a sequence of decisions, including variable construction, outlier rules, handling missing data, transformations, and coding. These decisions can be individually defensible while leading to different estimates. Steegen et al. (2016) introduced multiverse analysis as a way to run and display combinations of reasonable processing choices rather than reporting only one constructed dataset. The method turns previously hidden analytic discretion into an explicit empirical object whose sensitivity can be investigated.

A useful multiverse begins by defining an admissible decision space on theoretical and statistical grounds. Investigators identify decision nodes, combinations that remain methodologically defensible, dependencies among choices, and a common estimand across specifications. They then estimate an effect or model output across the chosen universe and report the distribution of magnitudes, uncertainty, signs, and substantive conclusions. If the specifications answer materially different questions, their numerical results cannot simply be counted as repeated estimates of one target. The chosen universe should be justified rather than designed to make an effect appear stable Steegen et al. (2016).

Multiverse output is descriptive unless a valid joint inference procedure and its assumptions are specified. Correlated estimates from the same participants do not constitute independent replications, and a majority of nominally significant specifications is not a universal probability that the hypothesis is correct. Nevertheless, the method can reveal which preprocessing choice most strongly changes a conclusion and guide a targeted future experiment. Robustness therefore becomes a basis for model criticism, new measurement, and focused replication, with transparency about the limits of the sensitivity space Steegen et al. (2016) Simonsohn et al. (2020).

Specification Curves and Model Dependence

Specification-curve analysis extends sensitivity assessment to the organized space of valid, nonredundant analytic specifications. Simonsohn et al. (2020) proposed identifying justified specifications, graphing their estimates, and conducting joint inferential assessment. The approach recognizes that different reasonable covariate sets, transformations, exclusions, and model definitions can yield different conclusions. It provides an intelligible depiction of which assumptions are consequential instead of treating every analytic choice as irrelevant background.

The graph is interpretable only when the estimation targets remain comparable. Models adjusting for a post-treatment variable, excluding an affected subgroup, or operationalizing a different construct may alter the causal or substantive question. Analysts must distinguish legitimate alternative operational choices from inadmissible conditioning and incompatible estimands. An appealing curve cannot repair a confounded observational design, and joint inference cannot validate an unsupported set of specifications. Robustness analysis must be embedded within the causal and measurement rationale of the study Simonsohn et al. (2020) Ioannidis (2005).

The practical value is especially high when an apparent effect is driven by one unstated modeling assumption. Researchers can present estimates by decision family, test influence of influential cases, report changes in theoretical conclusions, and identify where additional data would be most informative. The distinction between statistical stability and substantive stability should remain explicit: small coefficient changes may matter near a prespecified practical threshold, whereas changes in significance can occur with nearly identical estimates when sample size or standard error changes Simonsohn et al. (2020) Lakens (2017).

Many Analysts, One Dataset

Robustness can be evaluated by distributing the same question and dataset to independent analysts. Silberzahn et al. (2018) invited 29 teams to study whether soccer referees were more likely to issue red cards to darker-skinned than lighter-skinned players. Teams made different reasonable analytic decisions, producing substantially varied estimates; 20 reported a statistically significant positive association while nine did not. The project provided unusually direct evidence that analysts working with shared data and a common broad question may reach different conclusions without necessarily committing obvious errors.

This design measures researcher-mediated variation as a characteristic of the analytical process. It leaves other sources of replication uncertainty unchanged, because the observations are shared. Analysts' models may differ in covariate definitions, the functional form of a relationship, assumptions about clustering, outcome construction, and treatment of sparse strata. Some choices may actually target different quantities. The findings therefore argue for stating the estimand and making alternative decisions visible, not for mechanically averaging all analyst coefficients as though they were exchangeable independent replications Silberzahn et al. (2018).

Independent-analyst exercises have particular value for computational and observational psychology, where unobserved confounding, highly dimensional data, and complex models invite numerous defensible specifications. Their usefulness depends on clear task instructions, access to comparable information, and a transparent record of why each team made its decisions. Consensus among analysts does not certify causal identification; disagreement can instead expose where scientific questions and assumptions have not been adequately formalized. The method strengthens robustness assessment by exposing epistemic variation hidden in a single report Simonsohn et al. (2020) Silberzahn et al. (2018).

Power Posing and Differentiating Outcomes

Power posing illustrates how a popular psychological claim can combine several endpoints under one memorable description. Postural manipulations have been proposed to influence subjective experiences, social behavior, endocrine measures, or risk-taking. Those endpoints are distinct dependent variables requiring separate inferential evaluation. A study that supports a change in self-reported feeling does not automatically validate a hormonal mechanism or downstream behavioral outcome; nor does a null result on one endpoint settle all others Ranehill et al. (2015).

Ranehill et al. (2015) investigated the robustness of power-posing findings in a larger sample and reported no effects on hormones or risk tolerance under their protocol. The report's role is to challenge specific physiological and behavioral claims while encouraging investigators to differentiate measurement sensitivity, manipulation fidelity, and inferential scope. The absence of an abstract in one bibliographic index is a reminder to avoid inventing details not confirmed by the publication record. Independent replication should specify each outcome's primary status and meaningful effect threshold.

The case also illustrates communication risk. Public narratives often fuse a treatment's reported psychological and physiological consequences into a single package. When later evidence challenges part of the package, popular summaries may depict wholesale vindication or wholesale disproof. Scientific assessment instead follows the chain of measured outcomes, control conditions, outcome-specific estimates, and rival mechanisms. Robustness across multiple outcomes is a set of distinct evidential claims, not a branding property of a psychological intervention Simmons et al. (2011) Ranehill et al. (2015).

Measurement Reliability, Construct Validity, and Replication

A repeated study may precisely reproduce an operational score while failing to establish that the score measures the intended psychological construct. Construct identification requires a theory of indicators, relationships among measures, and boundary conditions. For replication, investigators should distinguish invariance of stimulus response from stability of an underlying cognitive, affective, or social mechanism. Changes in instruction, item wording, translation, testing platform, and participant familiarity may alter the measured variable. Repetition across such changes provides useful evidence only if measurement comparability is examined Munafò et al. (2017).

Measurement error affects effect-size precision and often attenuates observed associations, while differential reliability can make effects appear heterogeneous across settings. If a replication substitutes a short instrument for a longer original measure, differences may arise from score noise as well as substantive changes. A robust research program reports reliability under the new sample, evaluates whether instruments function similarly across groups, and considers alternative measurement models where feasible. Data transparency permits independent scrutiny of construction and scoring, but privacy, licensing, and participant protection must shape what can be shared Poldrack et al. (2017).

The replication question is therefore partly psychometric: what observable evidence warrants treating different scores as indicators of the same construct? This is especially important in personality, motivation, clinical symptoms, social attitudes, and complex cognitive tasks. Replication across one narrow indicator should not be advertised as proof of an entire latent psychological system. Stronger claims emerge when competing measures show coherent results and alternative measurement explanations are actively challenged through independent designs Lakens (2022) Munafò et al. (2017).

Heterogeneity, Context, and Theoretical Moderation

An effect's magnitude may vary between populations and settings without every difference indicating a defective study. A successful replication program should specify which contextual dimensions are theorized to matter: task instructions, language, cultural practices, recruitment, incentives, measurement regime, and baseline characteristics. Variation unconnected to an advance explanation is less informative than moderation predicted independently. Klein et al. (2014) and Klein et al. (2018) provide designs for examining such variation within standardized multi-site programs.

Heterogeneity has statistical and substantive dimensions. A random-effects model may estimate a distribution of study-specific effects, but its interpretation depends on defensible assumptions about the studies' exchangeability and sampling. A statistically significant heterogeneity statistic does not explain its source; a nonsignificant test may reflect limited ability to detect variation. Site-level covariates can be confounded with laboratory procedures, nationality, language, or sample composition, so a cross-site association need not establish a causal moderator. The useful question is which differences can be experimentally manipulated or prospectively held constant Klein et al. (2018).

A contextual explanation gains force when it constrains predictions before results become known. If a theory allows every replication success to confirm it and every failure to be attributed to an unspecified missing context, the range of potentially refuting observations disappears. Instead investigators can define a moderator, test its measurement, preregister an interaction, and replicate that interaction across samples. The result may be a narrower but more informative theory whose predictions differ across identified conditions Kerr (1998) Nosek et al. (2018).

Computational Reproducibility and Open Research Materials

Independent investigation begins with the possibility of reconstructing how numerical claims were obtained. Computational reproducibility requires accessible processing instructions, software dependencies, analytic code, variable dictionaries, and sufficient data or privacy-preserving alternatives. Version control records the history of changes to analyses; executable environments reduce dependence on undocumented local configurations. In neuroimaging, Poldrack et al. (2017) emphasizes how complex pipelines, software variation, and high-dimensional testing create particular transparency needs.

Availability of code and data is a practical condition, not a guarantee of scientific validity. A fully reproducible statistical pipeline can faithfully recreate a mistaken variable definition or a biased causal estimate. Conversely, legally protected or sensitive participant data may require secure access mechanisms, simulated demonstration datasets, or independent verification under disclosure restrictions. A balanced approach records computational steps and provenance while respecting consent, confidentiality, licenses, and institutional obligations Munafò et al. (2017) Poldrack et al. (2017).

Computational verification should be systematic. An external team can confirm file integrity, run documented commands, compare expected intermediate and final outputs, inspect random seeds and nondeterministic routines, and check whether reported tables match generated results. Tests may identify rounding, numerical-library differences, or missing processing stages before any new participants are recruited. This level of replication clarifies whether a later difference arises from data collection or from differences in software and calculation. The resulting audit trail is especially valuable for large-scale behavioral datasets and algorithmic psychological assessments Poldrack et al. (2017).

Meta-Analysis, Publication Selection, and Replication

Evidence synthesis may give a distorted impression when primary studies enter the published record selectively. The file-drawer problem Rosenthal (1979) concerns the difference between studies performed and studies available for aggregation. When significance or narrative appeal influences publication, a pooled result may overestimate the underlying parameter. Bakker et al. (2012) described how multiple underpowered investigations can interact with selection processes, and Ioannidis (2005) formalized conditions under which selective discoveries have low predictive value.

Kvarven et al. (2020) compared results of 15 meta-analyses with large preregistered multi-laboratory replication projects on corresponding psychological topics. Twelve pairs showed statistically significant differences, and the earlier meta-analytic estimates were, on average, much larger than the replication estimates. The comparison provides an empirical demonstration that apparent consensus across published small studies can diverge from a coordinated replication designed to reduce selective reporting. Its scope is the identified topic pairs and methods, and the magnitude of disagreement should not be generalized mechanically to all meta-analyses.

Meta-analyses remain essential when they account for study quality, selection mechanisms, heterogeneity, correlated outcomes, and the precise target estimand. A single multi-lab replication may have superior internal standardization yet offer a narrower range of manipulations than the historical literature. Integrative synthesis therefore belongs to a distinct evidence-aggregation task that cooperates with, but does not replace, individual replication. This dossier considers the implications for evaluating claim robustness; full literature-search protocols and quantitative synthesis architecture belong to a separate research object Kvarven et al. (2020) Munafò et al. (2017).

Statistical Inference Beyond a Dichotomous Success Label

Replication interpretation improves when uncertainty is represented by effect-size intervals and explicitly articulated decision thresholds. Two studies can differ in the conventional significance label despite producing compatible estimates; conversely, both can be significant while implying materially different magnitudes. The p value describes data compatibility with a particular null model under stated assumptions. It does not itself report the probability that the studied theory is true. A replication decision based only on crossing .05 discards information about precision, direction, size, and substantive importance Open Science Collaboration (2015) Lakens (2017).

Equivalence testing offers a route to positive evidence that effects of at least a chosen magnitude are unlikely under the model. Lakens (2017) discusses two one-sided tests using prespecified lower and upper bounds. These bounds require a scientific justification, such as the minimum effect capable of changing an applied decision or a theoretically predicted range. If an equivalence result depends on a very wide region, it may be numerically valid but substantively uninformative. If the region is implausibly narrow, a feasible replication may be unable to reach an informative conclusion Lakens (2022).

A complete evaluation should report the original estimate, replication estimate, interval, heterogeneity where applicable, and the interpretation of plausible effects. Alternative Bayesian or likelihood-based approaches can also be used when their assumptions and prior sensitivity are made explicit. No single statistic answers every question of theory appraisal. Where replication results conflict, analysts should inspect design differences and consider whether sequential sampling, outcome multiplicity, or model selection invalidated nominal inferential guarantees. Such scrutiny is more informative than post hoc movement between criteria to obtain a preferred answer Simmons et al. (2011) Nosek et al. (2018).

The Researcher, Institution, and Incentive Environment

Replication is a behavioral and institutional process as well as a statistical one. Researchers face incentives associated with novelty, career advancement, speed, funding, and journal selectivity. These incentives influence how questions are chosen, which outcomes are emphasized, how much effort is invested in method documentation, and whether replications are perceived as valuable scholarly contributions. Munafò et al. (2017) places reproducibility reforms across methods, reporting, dissemination, evaluation, and incentives. Technical tools alone cannot correct institutional conditions that systematically reward selectively favorable findings.

Reputational conflict can hinder scientific correction. Original investigators may regard a null replication as a personal accusation; replication teams may receive attention for disagreement; journals may prefer controversy to careful qualification. A well-designed workflow separates methodological assessment from personal judgment. Pre-data agreement on protocol details, independent validation, open exchange of analysis specifications, and transparent publication of protocol deviations can improve the credibility of both positive and negative outcomes. These practices are valuable even when investigators disagree about the final theoretical interpretation Chambers (2013) Nosek et al. (2018).

An accountable scientific culture recognizes replication, documentation, data stewardship, and methodological criticism as productive research. Institutional metrics should not presume that citation count or p-value strength alone indicates credibility. Bogdan (2025) examined how reported p-value patterns shifted across a very large corpus of psychology articles over time, while explicitly leaving questions about incentives and substantive robustness. Bibliometric indicators can monitor aspects of reporting practice but cannot replace evaluation of sample design, measurement quality, and causal identification. A field becomes more cumulative through the coordinated organization of all these practices Munafò et al. (2017).

Contemporary Evidence and the Limits of Improvement Metrics

Contemporary metascience examines whether methodological reforms alter research quality at scale. Bogdan (2025) studied reported p values across 240,355 empirical psychology articles published between 2004 and 2024, finding a shift toward stronger reported evidence across subdisciplines. The outcome is an important descriptive signal about the distribution of reported p values. However, reported p values are shaped by question difficulty, sample size, model specification, selective reporting, and changing publication practices. Their distribution cannot establish independently that causal claims are more reliable, or that replication rates improved by a known proportion.

Recent critical discussions continue to reveal concrete errors in otherwise prestigious research. Gelman & Brown (2024) examined how multilevel modeling choices and misreadings of existing literature can lead to strong public claims inadequately supported by a particular analysis. The lesson is methodological rather than a judgment on an entire field: complex models can obscure simple dependence structures, and prestigious institutional settings do not validate inferential assumptions. Rechecking a result against simpler transparent models can reveal the effective informational content of a design.

Evaluating reform requires outcomes tied to credibility, not merely uptake of labels. Useful indicators include prospectively registered hypotheses with high adherence, independently executable analyses, timely availability of study documentation, precision of effect estimates, and independent replication across scientifically relevant conditions. Such indicators should be audited for strategic compliance and for their own measurement validity. Longitudinal comparisons require stable definitions and careful treatment of shifts in journal portfolios and scientific topics. Evidence for changes in reporting culture should remain distinct from evidence for changes in the frequency of correct theoretical claims Munafò et al. (2017) Bogdan (2025).

Replicability in Neuroimaging and High-Dimensional Psychology

Brain imaging, digital phenotyping, and other high-dimensional psychological methods create many opportunities for analytic sensitivity. Preprocessing, artifact removal, smoothing, region definition, statistical thresholds, covariate adjustment, and statistical-model assumptions can substantially change a reported association. Poldrack et al. (2017) reviewed the need for transparent and reproducible neuroimaging workflows. These techniques often involve many processing stages, so a final image or coefficient may not communicate how the evidence was produced.

Replication with new neuroimaging data must consider scanner properties, recruitment, task implementation, head motion, physiological noise, and multiple-comparison control. Computational reproduction with existing data should separately verify preprocessing and inference. A common pipeline improves comparability, while a strategically constructed multiverse can identify whether a finding depends on a fragile preprocessing choice. Neither analysis establishes that a neural signal is uniquely attributable to one psychological construct; theoretical interpretation also requires convergence with behavioral evidence and discriminating design Poldrack et al. (2017) Steegen et al. (2016).

Modern psychological models using high-dimensional signals should include independent validation and leakage checks as part of robustness assessment. Selection of features, tuning of models, and preprocessing must occur within training folds when held-out predictions are evaluated. A replicated association within one dataset split may disappear under different partitions because of overfitting, or may remain stable while failing under a shifted population. Such problems create a substantive link between replication methods, formal modeling, and the separate scientific question of generalizability. Each claim should identify which kind of evidence it has actually survived Simonsohn et al. (2020) Poldrack et al. (2017).

Replication of Human–Artificial Psychological Research

The Artificial Era enlarges the methodological importance of replication because psychological claims increasingly involve interactions between human participants and adaptive computational systems. A study may examine how people trust an automated recommendation, attribute intention to a dialogue agent, delegate memory work, or develop patterns of cognitive partnership. Such findings may depend on an exact software model, prompt, interface, update state, sampling policy, and interaction history. A replication that merely reuses a commercial product name may test a different system once the underlying model changes. Robust inference requires versioned descriptions of the artificial system together with its observable behavior and experimental setting Munafò et al. (2017) Poldrack et al. (2017).

Replication of an artificial-system evaluation also requires distinguishing stochastic output variation from participant variation. The same prompt may generate different responses across repeated calls; model updates may shift distributions; and human participants may respond differently after discovering a system's identity. Research protocols should define model version, decoding settings where available, prompt construction, logging, data handling, exclusion policy, and the unit of statistical independence. Comparing repeated outputs generated by one model instance is not equivalent to independently sampling human participants or independently reproducing a psychological mechanism.

The conceptual significance is a disciplined expansion of what counts as a testable psychological claim. Artificial cognition, artificial agency, and psychologically meaningful human–artificial relations require distinct operational definitions and replicable measurements. A theoretical interpretation about Artificial as an emerging nonbiological order remains a philosophical framework until linked to specified observables and discriminating hypotheses. The appropriate empirical claim concerns measured performance, interaction patterns, or behavioral consequences under documented conditions. Robustness analysis can establish the stability of those observations without presuming a metaphysical conclusion Simonsohn et al. (2020) Lakens (2022).

Causal Identification and the Boundaries of Robustness

Robustness across statistical specifications can increase confidence that an observed relationship is not an artifact of one processing choice. It does not, by itself, identify a causal mechanism. Confounding, selection bias, inappropriate conditioning, and reverse causation may persist across many models. A family of similarly biased analyses can yield an impressively stable curve while all targeting the wrong quantity. Replication therefore needs a causal question: what intervention is compared with what alternative, for which units, over what time interval, and under what assumptions? Simonsohn et al. (2020) makes analytically defensible specifications an explicit prerequisite, while Gelman & Brown (2024) highlights how problematic modeling can sustain unwarranted claims.

Randomization strengthens causal identification when assignment is implemented and preserved, yet measurement error, interference, missing data, and treatment noncompliance can still matter. Observational replication is valuable for examining consistency across datasets and contexts but typically requires additional assumptions to infer a causal contrast. A second cohort may reproduce the same confounding structure; a different measurement regime may introduce another bias. Investigators should thus organize robustness checks around plausible threats to identification and explain what each check can and cannot resolve.

A theoretically informative replication targets alternative explanations. If a task is thought to change judgment through attentional allocation, researchers can measure or experimentally manipulate attention and distinguish that pathway from demand characteristics or response strategies. Merely observing the same total effect again is valuable, but does not uniquely validate its purported mechanism. Stronger explanatory claims connect experimental manipulation, intermediary processes, timing, discriminating predictions, and convergent methods. These requirements belong to the logic of causal psychological science rather than a supplementary ritual after a replication result Munafò et al. (2017) Lakens (2022).

Scientific Disagreement, Correction, and Cumulative Knowledge

Contradictory replication results create an opportunity to revise both theory and investigative practice. A useful adjudication reconstructs the two protocols, compares participant populations and measurement quality, assesses the precision and plausibility of estimates, and asks whether an overlooked moderator predicts the discrepancy. Independent review of code and data can reveal computational errors before psychological explanations are introduced. If discrepancies remain, a jointly planned discriminating study can be more informative than a sequence of commentaries about which laboratory is credible Poldrack et al. (2017) Klein et al. (2018).

Scientific correction includes changes to effect magnitude, causal interpretation, boundaries, or confidence. A smaller replicated effect may require reducing the prominence of a proposed intervention without abandoning all observations on which the theory was built. A successful result under one condition may support a conditional model, and a series of well-powered equivalence results may rule out effects larger than a prespecified meaningful threshold. Research programs grow cumulatively when their propositions are made more precise in response to evidence Lakens (2017) Hagger et al. (2016).

Transparency is also an ethical requirement toward participants and downstream users. Claims about education, therapy, workplace behavior, or automated assessment may influence consequential decisions. Communication should track the actual reliability, generalizability, and practical magnitude of the underlying evidence. Replication uncertainty is relevant to recommendations whenever a decision depends on a specific psychological effect. The scientific response is to disclose the quality and limits of evidence and update interpretations as the cumulative record changes Munafò et al. (2017) Kvarven et al. (2020).

Unresolved Questions and Research Priorities

Several major problems remain open. The field needs better models of when a replication differs sufficiently from an original protocol to change its estimand, and of how to allocate resources between repeated direct tests and strategically varied conceptual tests. It also needs reliable ways to identify and measure theoretically relevant moderators without generating unlimited post hoc explanations. Multisite studies provide an empirical platform, but observational variation across locations often cannot uniquely attribute differences to culture, laboratory, language, or recruitment Klein et al. (2018).

Robustness methods require principled specification spaces. Analysts need guidance on which decisions are genuinely defensible, how their dependencies should be represented, and how to quantify uncertainty when many correlated models are assessed. Measurement invariance, missing-data mechanisms, causal structure, and high-dimensional processing each demand domain-specific constraints. The many-analyst and multiverse traditions make this problem visible, while leaving a substantial agenda for formal methods and external validation Silberzahn et al. (2018) Steegen et al. (2016) Simonsohn et al. (2020).

Research on scientific reform also requires prospective evaluation. The uptake of preregistration, Registered Reports, open code, and independent replication should be linked to observable improvements in error detection, inferential precision, correction rates, and downstream decision quality. Greater significance strength in published articles is informative about reporting patterns, but not sufficient as a causal measure of reform impact Bogdan (2025). Adaptive artificial systems further require techniques for preserving model identity and measuring changes over software revisions. These questions can be addressed through coordinated protocols, transparent longitudinal datasets, and prospectively designed methodological experiments Nosek et al. (2018) Poldrack et al. (2017).

Conclusion

Replication and robustness establish a structured pathway by which psychological claims are exposed to independent evidence and deliberate variation. Their central object is a specified inference whose empirical and theoretical commitments can be inspected. The strongest research programs define targets clearly, justify sampling and effect-size thresholds, preserve planned decisions, make analytic alternatives visible, and investigate disagreements across independent observations. A repeated significant result, a computationally identical output, and an estimate stable across analytic specifications contribute different forms of evidence; each has a particular place in the cumulative assessment of a claim Open Science Collaboration (2015) Nosek et al. (2018) Simonsohn et al. (2020).

The historical record demonstrates both the capacity of experimental psychology to organize large collective tests and the difficulty of making broad generalizations from selected replication projects. Coordinated efforts revealed effect-size shrinkage and variable replication outcomes; preregistration and Registered Reports strengthened the distinction between forecast and retrospective explanation; multiverse and many-analyst approaches exposed variation hidden by singular analyses. The resulting scientific lesson is methodological specificity. Credibility depends on what was measured, which alternative hypotheses were excluded, which conditions were repeated, and how the evidence changes a substantively defined claim Klein et al. (2014) Klein et al. (2018) Hagger et al. (2016) Silberzahn et al. (2018).

Future psychological inquiry must integrate independent data collection, reproducible computation, valid constructs, principled robustness assessment, and transparent correction. This applies to experiments involving Homo and to emerging psychological investigations of human–artificial interaction, where software versions and interaction histories become part of the experimental protocol. The evidential standard remains a traceable relation between theory, observation, analysis, and independently testable prediction. Replication and robustness thereby function as continuing practices of scientific calibration and cumulative knowledge rather than as a final binary verdict on individual studies Munafò et al. (2017) Lakens (2022).

References

  • Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543–554. https://doi.org/10.1177/1745691612459060
  • Bogdan, P. C. (2025). One decade into the replication crisis, how have psychological results changed? Advances in Methods and Practices in Psychological Science, 8(2), 1–14. https://doi.org/10.1177/25152459251323480
  • Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. https://doi.org/10.1126/science.aaf0918
  • Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., Heikensten, E., Hummer, L., Imai, T., ... Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2, 637–644. https://doi.org/10.1038/s41562-018-0399-z
  • Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. https://doi.org/10.1016/j.cortex.2012.12.016
  • Gelman, A., & Brown, N. J. L. (2024). How statistical challenges and misreadings of the literature combine to produce unreplicable science: An example from psychology. Advances in Methods and Practices in Psychological Science, 7(4). https://doi.org/10.1177/25152459241276398
  • Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., Brand, R., Brandt, M. J., Brewer, G., Bruyneel, S., Calvillo, D. P., Campbell, W. K., Cannon, P. R., Carlucci, M., Carruth, N. P., Cheung, T., Crowell, A., De Ridder, D. T. D., Dewitte, S., ... Zwienenberg, M. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546–573. https://doi.org/10.1177/1745691616652873
  • Ioannidis, J. P. A. (2005). Why most published research findings are false. PLOS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
  • John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
  • Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4
  • Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., ... Nosek, B. A. (2014). Investigating variation in replicability: A Many Labs replication project. Social Psychology, 45(3), 142–152. https://doi.org/10.1027/1864-9335/a000178
  • Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Jr., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Bocian, K., Brandt, M. J., Busching, R., Cai, H., ... Nosek, B. A. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225
  • Kvarven, A., Strømland, E., & Johannesson, M. (2020). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour, 4, 423–434. https://doi.org/10.1038/s41562-019-0787-z
  • Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362. https://doi.org/10.1177/1948550617697177
  • Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1), Article 33267. https://doi.org/10.1525/collabra.33267
  • Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, Article 0021. https://doi.org/10.1038/s41562-016-0021
  • Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
  • Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017). Scanning the horizon: Towards transparent and reproducible neuroimaging research. Nature Reviews Neuroscience, 18, 115–126. https://doi.org/10.1038/nrn.2016.167
  • Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, S., & Weber, R. A. (2015). Assessing the robustness of power posing: No effect on hormones and risk tolerance in a large sample of men and women. Psychological Science, 26(5), 653–656. https://doi.org/10.1177/0956797614553946
  • Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://pages.ucsd.edu/~cmckenzie/Rosenthal1979PsychBulletin.pdf
  • Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., ... Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
  • Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour, 4, 1208–1214. https://doi.org/10.1038/s41562-020-0912-z
  • Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. https://doi.org/10.1177/1745691616658637
  • Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Jr., Albohn, D. N., Allard, E. S., Benning, S. D., Blouin-Hudon, E.-M., Bulnes, L. C., Caldwell, T. L., Calin-Jageman, R. J., Capaldi, C. A., Carfagno, N. S., Chasten, K. T., Cleeremans, A., Connell, L., DeCicco, J. M., ... Zwaan, R. A. (2016). Registered replication report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11(6), 917–928. https://doi.org/10.1177/1745691616674458