Psychology by BogdanovaThe Encyclopedia of Psychology for the Artificial Era

RESEARCH DOSSIER

Cross-System Psychometrics and Artificial Cognition

Abstract. Cross-system psychometrics investigates when scores obtained from humans and artificial cognitive systems warrant comparable interpretations. This fundamental critical narrative monograph synthesizes construct validity, classical test theory, item response theory, measurement invariance, latent-variable structure, matched-task experiments, prompt interventions, memory and tool ablations, longitudinal version drift, and benchmark contamination. It examines primary psychometric foundations alongside empirical studies of language-model personality inventories, situational judgment tests, emotional-intelligence assessment, and human–model factor structure published through October 2026. The evidence supports restricted, configuration-dependent measures of artificial performance and behavior while identifying substantial obstacles to transferring human latent trait interpretations. The synthesis specifies reproducible approaches for construct definition, task sampling, differential item functioning, multimethod validation, architecture-aware intervention, and replication. This is a critical narrative scholarly synthesis without newly collected experimental data, systematic-review protocol, or meta-analysis.

Keywords: cross-system psychometrics; artificial cognition; construct validity; measurement invariance; differential item functioning; item response theory; latent variable models; factor analysis; comparative cognition; large language models; AI evaluation; psychological assessment; human–AI comparison; response processes; benchmark contamination; experimental ablations; psychological measurement; Artificial Era

The Object of Cross-System Psychometrics

Cross-system psychometrics investigates the conditions under which scores obtained from biological and artificial cognitive systems support comparable interpretations. Its primary object is a measurement relation: a construct definition, a collection of tasks or indicators, a scoring procedure, a population of systems, and a specified inference from observations to cognitive attributes. Identical test items and identical numerical scores do not establish that the same latent attribute generated the responses. The research question is whether empirical relations among indicators, interventions, and external outcomes justify transferring an interpretation across systems with different developmental histories and computational architectures Cronbach & Meehl (1955) Borsboom et al. (2004) Vandenberg & Lance (2000).

A human participant answers in the context of perception, embodiment, education, goals, memory, fatigue, and social incentives. An artificial respondent may consist of pretrained parameters, a system prompt, sampled decoding, context windows, retrieval, external tools, and a surrounding controller. A questionnaire administered to a model without tools measures a different operational configuration from a model coupled to memory and a search environment. Accordingly, the scientific unit must specify the actual configuration under assessment and the intended level of generalization. Cross-system comparison must state whether it concerns a particular response episode, a configured agent, a trained model family, or behavior across deployment versions Serapio-García et al. (2025) Huang et al. (2024) Strugatski et al. (2026).

The guiding research sequence moves from score description through construct validation to causal and longitudinal explanation. Test scores are useful evidence for restricted tasks even when a wider psychological interpretation remains unsettled. Researchers can investigate accuracy, calibration, response stability, factor structure, information processing, transfer, and sensitivity to perturbations using separate experimental designs. This monograph integrates primary psychological measurement theory, recent artificial-system evaluations, and methodological arguments. It is a critical narrative scientific synthesis; it reports neither a new human-subject experiment nor a preregistered systematic review or meta-analysis AERA, APA, & NCME (2014) Messick (1994) Botter (2026).

Conceptual Genealogy of Psychological Measurement

The historical foundations of the field lie in attempts to infer abilities and dispositions from observable performance. The development of standardized testing created procedures for sampling behavior under controlled conditions, whereas latent-variable models provided mathematical accounts of how unobserved attributes could explain item responses. Classical test theory formalized observed scores as a combination of true-score variation and error, and item response theory introduced models connecting latent proficiency, item properties, and response probabilities. Those traditions assume specific measurement conditions and require empirical testing of model fit rather than authorization by an instrument's name Rasch (1960) Lord (1980) AERA, APA, & NCME (2014).

Cronbach and Meehl's account of construct validity connected measurements to a network of theoretical relations, while Campbell and Fiske's multitrait–multimethod design highlighted convergent and discriminant evidence across methods. A measure gains meaning through patterns of observations that a theory predicts, including expected associations and distinguishable alternatives. The same logic becomes especially valuable when a human assessment is reused with artificial systems. A plausible answer may reflect item recognition, role instruction, common textual associations, or goal-directed reasoning. Those processes predict different cross-task and intervention patterns despite sometimes producing the same recorded response Cronbach & Meehl (1955) Campbell & Fiske (1959) Borsboom et al. (2004).

The later development of comprehensive validation arguments and standards made intended score use central to evaluation. Messick connected substantive theory, empirical structure, external relationships, and consequences; the Standards for Educational and Psychological Testing assemble evidence about score interpretations, fairness, reliability, and administration. Their application to artificial systems requires fresh evidence because item semantics, response processes, and populations have changed. Cross-system psychometrics inherits these principles as a disciplined practice of specifying claims and assembling converging tests, rather than treating familiar instruments as portable gauges whose meaning survives every transition of substrate Messick (1994) AERA, APA, & NCME (2014) Chen (2007).

Validity Is a Claim About Interpretation

Construct validity asks whether a proposed interpretation of scores is warranted by evidence and theory. It cannot be conferred by a questionnaire that was validated on different respondents, and it cannot be reduced to a high correlation between repeated administrations. Borsboom, Mellenbergh, and van Heerden emphasized the importance of whether an attribute exists and whether changes in it causally affect the outcome of measurement. Researchers who study artificial systems must therefore identify the operational attribute they intend to measure and justify the causal or explanatory connection from that attribute to the observed score Borsboom et al. (2004) Cronbach & Meehl (1955) Messick (1994).

One possible target is output-level behavioral style: the reproducible tendency of a configured model to generate particular kinds of text under specified conditions. Another is competence in a task domain, established by success on representative and previously unseen problems. A more ambitious claim concerns latent cognitive organization or a trait that persists through contexts and interactions. Each claim imposes additional evidence requirements. Output-level regularity can be observed without showing that human personality constructs, autobiographical continuity, or subjective feelings are instantiated. A scientific paper should name the precise attribute, its measurement unit, and the circumstances under which the inference is intended to hold Serapio-García et al. (2025) Huang et al. (2024) Lin (2026).

A validity argument also includes rival explanations. High verbal performance might derive from memorized materials, a learned format convention, access to retrieval, or a prompt that guides the model toward socially desirable answers. A human test's content validity, factor structure, and predictive relationships are evidence about its historical human population, not automatic evidence about artificial respondents. Comparative research gains strength by specifying alternative response-generating mechanisms in advance and designing trials that could discriminate among them. The scientific result may be a justified narrow interpretation, an identified boundary, or evidence that an assumed equivalence fails Han et al. (2026) Strugatski et al. (2026) Botter (2026).

The Operational System as the Unit of Assessment

An artificial system's observable behavior emerges from a configuration rather than from a model label alone. The relevant protocol records the model identifier and revision, system and developer instructions, prompts, context documents, response temperature, random seed where applicable, tool permissions, retrieval index, and scoring process. Human protocols similarly document recruitment, language, education, testing environment, task sequence, and incentives. Without these descriptions the populations being compared remain undefined and independent replication cannot determine whether the same experimental treatment was delivered Huang et al. (2024) Serapio-García et al. (2025) AERA, APA, & NCME (2014).

This level of description matters when model versions or agent environments change between administrations. A language model coupled to external memory may appear to possess strong retention although its underlying weights have not changed; a respondent instructed to portray a conscientious person may produce internally coherent questionnaire answers without exhibiting that style outside the role. In both cases the observable result is real, yet the causal source lies partly in configuration. Measurement claims should be indexed to that source, with ablations removing specific components to establish their contribution to performance Han et al. (2026) Serapio-García et al. (2025) Strugatski et al. (2026).

Researchers also need to distinguish the unit of repeated observation from an independent sampled individual. Fifty model completions produced with different random seeds are repeated observations of one configured system, not fifty independently trained artificial individuals. Pooling those completions as though they were independent systems creates pseudoreplication and can produce misleadingly narrow confidence intervals. Suitable hierarchical designs model runs nested within prompt templates, system configurations, and model families, while human measurements may be nested within respondents and sites. Precise units are essential to credible uncertainty estimates and comparative statistical claims Vandenberg & Lance (2000) AERA, APA, & NCME (2014) Botter (2026).

Human and Artificial Response Processes

A response-process study asks how an observed answer was produced. For human participants the evidence may include think-aloud methods, latency, eye movements, strategy reports, and experimentally manipulated task demands. For artificial systems it may include controlled prompt variations, inference traces where trustworthy and accessible, tool-call logs, retrieval interventions, architecture-specific diagnostics, and responses to counterfactual tasks. No single method directly reveals every mechanism. Research should assess whether these observations covary with the cognitive construct as expected and whether distinct strategies produce dissociable signatures Messick (1994) Borsboom et al. (2004) Lin (2026).

An intelligence item provides a simple illustration. A person and a model may both choose the correct analogy, yet the person could reason by mapping relations while the model recognizes a familiar training example. The two pathways can be differentiated by novel relational structures, altered surface features, time constraints, explanation interventions, and controlled access to memorized exemplars. Evidence that an artificial system generalizes to difficult novel variants supports a broader competence interpretation. It still leaves questions about the internal representation and the stability of the capacity across architectural changes Chollet (2019) Strugatski et al. (2026) Han et al. (2026).

Response-process equivalence should consequently be investigated at a level appropriate to the research question. Some comparisons legitimately concern only observable task solutions, while others seek analogous sensitivity to information load, interference, social cues, or feedback. Stronger claims about shared cognitive processes require convergent perturbations and reproducible mechanistic results. Similarity at the level of an item key does not itself license similarity at the level of a psychological explanation. This distinction is a constructive basis for cross-system science because it identifies exactly what additional observations an explanatory claim needs Cronbach & Meehl (1955) Campbell & Fiske (1959) Borsboom et al. (2004).

Classical Test Theory and Reliability Across Systems

Classical test theory separates observed-score variation into a stable score component and measurement error under a defined population and administration regime. Reliability estimates describe the reproducibility of particular differences in a set of scores; they neither identify the psychological source of those differences nor establish that the same construct is measured across two kinds of system. Human test–retest reliability may be limited by learning, fatigue, or genuine developmental change. An artificial model's repeated scores may vary with sampling parameters, input formatting, retrieval state, or vendor updates. The error model must therefore be specified separately for each population and purpose AERA, APA, & NCME (2014) Huang et al. (2024) Botter (2026).

Internal consistency is particularly vulnerable to misinterpretation when questionnaire items share wording, style, or known scoring keys. A model might produce mutually consistent responses by following a coherent persona instruction or by recognizing semantically related items. Such consistency establishes a response pattern under those prompts; it supplies no direct proof that an enduring human-like disposition caused it. The distinction between stable item covariance and generalization to independently sampled behavior should motivate multitrait, multimethod, and downstream-performance validation rather than one coefficient or one completed inventory Campbell & Fiske (1959) Serapio-García et al. (2025) Han et al. (2026).

A suitable reliability study repeats the assessment across sampled tasks, prompts, occasions, seeds, languages, and model instances while estimating the variance contributed by each facet. The meaning of reliability depends on the intended decision. A system may be consistent enough for benchmarking a narrow capability and insufficiently stable for longitudinal trait classification. Hierarchical variance decomposition, confidence intervals, and prespecified minimum precision targets make this distinction explicit. Reports should publish the actual administration configurations and avoid treating multiple completions from one model as independent samples of different respondents AERA, APA, & NCME (2014) Botter (2026) Huang et al. (2024).

Item Response Theory and Differential Item Functioning

Item response theory models the probability of particular item responses as a function of latent attributes and item parameters. The Rasch tradition and later item response models provide tools for studying difficulty, discrimination, and the information an assessment yields at different proficiency levels. Their parameters obtain meaning within the assumed model and the data used to calibrate it. Applying a human-calibrated item model to artificial systems poses an empirical question: do the items order systems by a comparable dimension, and does a given latent score imply comparable response probabilities Rasch (1960) Lord (1980) Botter (2026)?

Differential item functioning occurs when respondents matched on an intended latent attribute have different conditional probabilities of responding to an item because of group membership or another attribute. Cross-system evaluation can investigate DIF between humans and models, model families, and versions. A vocabulary-dependent question may favor language models independently of the targeted reasoning process, whereas embodied or perceptual constraints may systematically disadvantage text-only systems. Detecting DIF requires defensible linking of latent scales, sufficiently diverse items, and diagnostics for model assumptions. Raw mean differences are not themselves a DIF analysis Vandenberg & Lance (2000) Chen (2007) Botter (2026).

An item functioning result also needs a scientific interpretation. DIF can signal construct-irrelevant variance, a legitimate domain-specific capacity, a translation effect, or a different strategy for performing the task. Analysts should inspect item content, response traces, and interventions before calling every difference bias. Adaptive tests introduce another complication: item selection algorithms can produce dissimilar item exposure and exploitability across respondent classes. A robust cross-system design records item histories, tests for parameter drift, uses independent item pools, and treats item calibration as contingent on the actually assessed systems Lord (1980) AERA, APA, & NCME (2014) Han et al. (2026).

Factor Structure and Latent Construct Comparability

Latent-variable models ask whether covariance among indicators can be explained by a smaller set of unobserved dimensions. Exploratory analyses describe structures that fit the sample; confirmatory models test specified factor patterns under identifying assumptions. In cross-system studies the main question is not whether both groups yield a statistically acceptable model considered separately. It is whether the relationships linking latent variables to observed items are sufficiently similar to justify equivalent interpretation. A test can exhibit similar overall accuracy while its item covariance is organized around different response strategies Vandenberg & Lance (2000) Reise (2012) Strugatski et al. (2026).

The evidence should progress through dimensionality assessment, factor interpretability, model fit, and targeted comparison. If a human verbal test is primarily structured by semantic knowledge and processing limitations, an artificial model's variation might instead be driven by task format, instruction compliance, or access to prior examples. A superficially matching factor count is insufficient; factor congruence, loading patterns, residual dependencies, and external relationships must also be examined. Bifactor representations can clarify common and domain-specific variance, but alternative equivalent models and overfitting require substantive scrutiny Reise (2012) Chen (2007) Strugatski et al. (2026).

Strugatski, Zeinfeld, and Alexandron investigated latent structure in two educational assessments, comparing human data with six multimodal language models. Their 2026 preprint reports systematic differences in the analyzed human and model factor structures. This finding provides a direct test of a necessary condition for transferred score interpretation in those contexts. Its case-study scope and preprint status should be preserved: two instruments do not establish how every test or artificial architecture behaves. The valuable contribution is the experimentally tractable question of whether instrument structure survives a change in respondent system Strugatski et al. (2026) Vandenberg & Lance (2000) AERA, APA, & NCME (2014).

Measurement Invariance and Score Linking

Measurement invariance is a hierarchy of empirical claims about the stability of a measurement model across groups or occasions. Configural invariance concerns the pattern of factors and indicators, metric invariance compares loadings, scalar invariance concerns intercepts or thresholds, and stricter models impose further residual constraints. Different scientific questions require different levels of equivalence: comparing associations, means, individual classifications, and longitudinal change have distinct assumptions. In cross-system studies the hierarchy offers a disciplined language for stating which claims are justified by the available data Vandenberg & Lance (2000) Chen (2007) AERA, APA, & NCME (2014).

A lack of invariance should be investigated with theory and item evidence, rather than repaired mechanically until an arbitrary threshold is reached. Some items may function differently because artificial systems process tokenized language while human respondents experience a sensory or emotional context. Partial invariance may permit restricted comparisons under transparent assumptions, but it does not establish full identity of constructs. Methods developed for multiple-group factor models are useful diagnostic tools; their statistical fit indices and cutoff conventions must be interpreted with sample size, identification, dimensionality, and the consequences of the intended comparison in mind Chen (2007) Vandenberg & Lance (2000) Messick (1994).

Scale linking additionally requires anchor items whose functioning is stable in both populations. A shared test form is not sufficient if the anchor questions are remembered from training corpora or subject to differential instruction following. Researchers can design new equating sets, hold out item generation procedures, compare human and artificial anchor performance, and evaluate uncertainty from group-specific calibration. Cross-version linking presents the same issue over time when model updates change the effective respondent population. The eventual outcome may be a common scale, separate population-specific scales, or evidence that a joint latent score is scientifically uninformative Lord (1980) Han et al. (2026) Botter (2026).

Multimethod Evidence and External Criteria

A comprehensive validation program samples more than one route from the hypothesized attribute to an observation. The multitrait–multimethod approach is especially apt when testing systems with dissimilar mechanisms: questionnaire scores can be compared with task behavior, preference choices, error recovery, interactions with other agents, and independently observed outcomes. Agreement between two nearly identical prompt formats supplies weaker evidence than agreement between a survey and an externally constrained task requiring successful performance. Researchers should specify which methods share error sources and which provide genuinely independent information Campbell & Fiske (1959) Cronbach & Meehl (1955) Messick (1994).

External criteria must be chosen so that success has a substantive relation to the proposed construct. A measure called honesty might be tested against factual reporting under incentives to please a user, resistance to unsupported claims, and the capacity to acknowledge uncertainty; fluency or polite vocabulary would be insufficient as sole criteria. Likewise, a cooperation scale should predict behavior in interactive resource-allocation or coordination tasks rather than only the endorsement of socially desirable sentences. The operational criterion may differ between humans and artificial systems, but its relevance and comparability must be argued explicitly Serapio-García et al. (2025) Lin (2026) AERA, APA, & NCME (2014).

Convergent evidence can fail for informative reasons. A model may answer empathy questionnaires in a consistent manner yet make insensitive choices when a task requires identifying another person's perspective under ambiguous feedback. The discrepancy identifies a boundary between questionnaire performance and practical social behavior; it need not render either observation worthless. Construct research advances by describing where the measure predicts, where it fails, and which mechanisms account for the pattern. This requires reporting negative, contradictory, and population-specific results together with favorable outcomes, with all analytic decisions made clear Borsboom et al. (2004) Campbell & Fiske (1959) Botter (2026).

Task Sampling and Matched Human–Artificial Experiments

A cross-system experiment begins by defining a target domain of tasks rather than selecting a convenient fixed leaderboard. The task universe may comprise analogical reasoning problems, social inference scenarios, working-memory manipulations, calibrated factual judgments, or problem-solving situations. An assessment then samples across difficulty, format, content, language, and contextual demands. For a human–artificial comparison, task materials and decision outputs must be made sufficiently commensurable while administration differences are documented. Matching observable opportunities supports fair description; it does not erase differences in senses, embodiment, time budgets, or prior learning AERA, APA, & NCME (2014) Chollet (2019) Botter (2026).

Matched experiments should compare performance curves and error profiles, not merely average proportions correct. Difficulty gradients, distractor responses, confidence–accuracy relations, and response dependencies reveal how systems behave as tasks change. Artificial systems can be measured across carefully controlled configurations, while human data require consent, equitable recruitment, and consideration of prior knowledge and fatigue. Investigators must analyze the possible advantage conferred by scale, response-time allowance, tool access, and testing exposure. Such factors belong in an explicit design matrix because they influence the meaning of any comparative superiority or deficit claim Lord (1980) Messick (1994) Strugatski et al. (2026).

Independent holdout tasks are indispensable. A test repeatedly used for public model comparisons becomes a target for developer optimization and may enter training material. A scientifically useful protocol creates fresh items with documented provenance, tests for leakage, and measures generalization to structured variants. Investigators can combine human trials, expert-generated challenges, adaptive difficulty sampling, and replication across unrelated environments. Results should state the target population of tasks and systems. A high result on a well-defined sample is meaningful evidence about that sample; broader competence requires defensible extrapolation Kiela et al. (2021) Sun et al. (2025) Han et al. (2026).

Prompting as an Experimental Treatment

Prompts are part of the administered experimental treatment. Instructions can activate role conventions, change the definition of a task, supply demonstrations, alter risk preferences, or induce specific response styles. An assessment that says “respond as a careful physician” and one that says “respond as an impulsive teenager” samples different conditional behaviors even when the model parameters remain identical. Repeated settings must therefore be represented as treatment conditions and recorded along with their effects. Stable response patterns under a prompt cannot be silently reclassified as intrinsic, context-independent traits Huang et al. (2024) Serapio-García et al. (2025) Han et al. (2026).

Huang and colleagues tested psychological scales across 2,500 settings per model and reported comparatively consistent Big Five Inventory responses under specified configurations, while also examining prompt-based persona emulation. Their results illustrate that repeated administration can identify reliable conditional output patterns. They do not establish that the measurements have identical ontological or predictive meanings for humans and language models. A strong replication would report sensitivity to item order, question paraphrase, system messages, sampling, language, and model family, and would estimate whether questionnaire score differences persist in unseen behavioral contexts Huang et al. (2024) Campbell & Fiske (1959) Lin (2026).

Factorial prompt designs provide a route to causal explanation. A researcher can randomly vary a social role, a scoring cue, the length of a context, and exposure to previous questions, then estimate the effect of each manipulation and its interactions. This approach helps identify whether apparent dispositions are driven by explicit instruction, latent training associations, or task representations. It also exposes an inferential problem: human participants may interpret an instruction as a social demand whereas a model may treat it as a high-priority behavioral constraint. Equivalent words need not constitute equivalent interventions Serapio-García et al. (2025) Huang et al. (2024) Borsboom et al. (2004).

Architecture-Aware Interventions and Ablations

Architecture-aware assessment separates observable behavior from the components that produce it. Retrieval can be removed, memory stores disconnected, tools disabled, context length reduced, or planning steps constrained while the remaining environment is held as constant as possible. A change in score identifies the contribution of the manipulated component under the tested conditions. It does not by itself determine whether the component implements a process homologous to human recall, attention, or executive function. The explanatory bridge must come from additional theory and discriminating evidence Botter (2026) Strugatski et al. (2026) Chollet (2019).

For example, an agent may perform well on delayed factual questions when it can query an external vector database. Comparing performance with and without retrieval estimates how the database contributes to the completed task. Testing information learned only after the experimental start helps distinguish external access from pretraining familiarity. The corresponding human memory experiment might manipulate delay, rehearsal, interference, or external notes. The two designs can establish functional parallels, yet the manipulated causal pathways may differ fundamentally. A valid comparison names the level at which any analogy is asserted Cronbach & Meehl (1955) Messick (1994) Lin (2026).

Intervention studies should include negative controls and manipulation checks. Removing a tool can inadvertently increase computational latency or change the wording of a system instruction, confounding conclusions. A model's explanation of its answer need not faithfully identify the mechanism that produced it. Protocols should therefore preserve task prompts when possible, independently verify component states, and predefine performance criteria. Where internal access is unavailable, behavioral manipulations still provide valuable evidence, accompanied by appropriate limits on causal attribution. Replication across model families tests how architectural dependencies vary AERA, APA, & NCME (2014) Botter (2026) Serapio-García et al. (2025).

Longitudinal Measurement, Version Drift, and Continuity

Longitudinal cross-system studies investigate whether measurements retain their meaning as artificial systems, human participants, and task environments change. The familiar test–retest question becomes more complex when a model endpoint may receive an unannounced revision or an agent may acquire external records between sessions. A repeat administration should record immutable model identifiers wherever available, configuration hashes, dates, tool versions, and the state of persistent stores. Stability of the score is one outcome; stability of the measurement relation and the underlying operational system are further questions Vandenberg & Lance (2000) Chen (2007) Botter (2026).

A gradual score increase may reflect improved performance, easier tasks, exposure to answers, prompt modifications, or drift in the instrument. When versions are compared, researchers can use anchor items, new independent challenges, and parallel forms with an explicit linking model. They should also test the assumption that the anchors retain their conditional response properties. Longitudinal plots become scientifically informative when accompanied by uncertainty, evidence of equivalent item functioning, and records of the system modifications that could explain discontinuities. An apparent developmental trajectory may otherwise be a sequence of different experimental objects Lord (1980) Han et al. (2026) Sun et al. (2025).

Claims about persistent identity or stable dispositions require separate longitudinal hypotheses. An agent may preserve an interaction history through external storage while replacing its underlying model, or preserve model parameters while losing contextual records. Both arrangements can generate different forms of behavioral continuity. Psychometric analysis may characterize continuity in policy, memory access, and response covariance without deciding philosophical questions about personal identity from those scores. A strong study evaluates reidentification of the configuration, predicts behavior under changed circumstances, and quantifies the fraction of observed persistence attributable to each component Messick (1994) Lin (2026) AERA, APA, & NCME (2014).

Sampling Hierarchies and Statistical Uncertainty

Cross-system data are often hierarchical: task instances sit within domains, prompts within templates, completions within configured models, versions within model families, and human observations within respondents and sampling sites. A valid statistical analysis respects these dependencies. Treating each sampled completion as an independent machine or each paraphrased item as an independent construct replication inflates apparent evidence. Hierarchical modeling, cluster-robust uncertainty, and item-level resampling can produce estimates whose uncertainty reflects actual units of generalization. Researchers should state the exact population from which tasks and systems were sampled AERA, APA, & NCME (2014) Vandenberg & Lance (2000) Botter (2026).

Accuracy is particularly sensitive to the tested item distribution. A benchmark can overrepresent familiar educational questions and underrepresent ambiguous, adversarial, multilingual, or interactive environments. Subgroup analyses should assess differential performance with adequate sample sizes rather than selecting striking examples. When many constructs or models are compared, multiplicity and post-selection bias become relevant. Reports should separate prespecified hypotheses from exploratory discoveries, provide uncertainty intervals where meaningful, and examine whether effects remain under different weighting rules and scoring policies Kiela et al. (2021) Sun et al. (2025) Botter (2026).

Inference also depends on the question being asked. A descriptive comparison estimates observable differences on specified tests. An explanatory comparison seeks evidence for mechanisms through manipulations. A transportability claim predicts performance on new systems, task populations, or time periods. These goals require different statistical designs and validity arguments. The strongest studies predefine the estimand, present both positive and negative findings, release analysis code and item provenance where lawful, and examine whether results survive independent replication. Accumulating many scores without these distinctions can create an illusion of precision while leaving the core psychological question unresolved Messick (1994) AERA, APA, & NCME (2014) Strugatski et al. (2026).

Benchmark Validity and the Meaning of Capability

A model benchmark is a measurement instrument whose scores emerge from a construct definition, a sampling frame of tasks, administration conditions, and a decision rule. A benchmark marketed as general reasoning may mix mathematical knowledge, textual familiarity, instruction following, and format recognition. Psychometric analysis asks which attributes the items actually discriminate, whether the score supports its intended use, and what the observed variation means. Successful benchmark engineering can generate useful comparative statistics, but a score's psychological meaning is established through validation studies rather than by the technical sophistication of its scoring pipeline Botter (2026) AERA, APA, & NCME (2014) Chollet (2019).

Chollet's discussion of intelligence measurement emphasizes generalization, prior experience, and the efficiency with which skills are acquired under changed conditions. Its conceptual contribution is a proposal for how to make claims about competence more sensitive to novelty and adaptation; as a methodological preprint, it is a theory of measurement rather than empirical proof that one benchmark captures intelligence. Dynabench introduced a dynamic approach to data collection and evaluation in language processing, motivated partly by the saturation of static test sets. These lines of work encourage item evolution and adversarial sampling while leaving the validity of inferred psychological constructs to be empirically evaluated Chollet (2019) Kiela et al. (2021) Messick (1994).

The 2026 methodological analysis by Botter places language-model benchmarks explicitly within psychometric theory, discussing validity, generalizability, item response modeling, dimensionality, and differential item functioning. It is useful for identifying methodological problems rather than supplying a universal ready-made correction. Every proposed improvement must still be tested against actual response data. A benchmark may be appropriately calibrated for comparing specific deployed systems and poorly suited to claims about shared human and artificial cognition. A mature evaluation report should identify its audience, permitted claims, sampling scope, and known threats to construct interpretation Botter (2026) Vandenberg & Lance (2000) AERA, APA, & NCME (2014).

Contamination, Memorization, and Score Inflation

Training-data contamination occurs when benchmark items, near duplicates, answer keys, or evaluation descriptions are present in material available during model development. It threatens the interpretation of performance as independent generalization and can distort comparisons among versions with different exposure histories. The problem is especially serious for public questionnaires, standardized educational assessments, and widely circulated psychological scales. A high score may represent competence, recognition, or a mixture; item provenance and exposure analysis are therefore integral to the validity argument. Mere failure to find an exact string match is insufficient to rule out semantic leakage Han et al. (2026) Sun et al. (2025) Choi et al. (2025).

Han and colleagues' 2026 study measured three distinct contamination phenomena in psychological inventories: memorization of items, memorization of evaluations, and the ability to match a specified target score. Across 21 models and four inventories, the authors reported substantial contamination on commonly used instruments, including the Big Five Inventory and Portrait Values Questionnaire. Their design demonstrates why conventional answer consistency may coexist with learned familiarity or tactical response selection. It does not establish that all behavioral observations from these models are invalid; it identifies specific threats and motivates independent, exposure-controlled measurement Han et al. (2026) Huang et al. (2024) AERA, APA, & NCME (2014).

Contamination mitigation must be evaluated rather than assumed. Rewording a known item may preserve its underlying semantic signature; synthetic tasks may resemble templates seen during training; dynamic challenges can drift in difficulty over time. Sun and colleagues examined mitigation strategies, and Choi and colleagues developed distributional approaches to measuring leakage. A rigorous cross-system assessment can combine newly designed items, controlled access during testing, content similarity audits, longitudinal contamination monitoring, and untouched validation sets. Public reports should distinguish verified exposure, plausible leakage, and residual uncertainty, because the absence of proof of contamination is not proof of an uncontaminated test Sun et al. (2025) Choi et al. (2025) Kiela et al. (2021).

Personality Inventories Applied to Language Models

The use of personality questionnaires with language models provides an unusually clear example of both the promise and the boundaries of cross-system psychometrics. Human personality inventories were designed to capture relatively characteristic patterns in people whose reports are connected to their lives and activities. Artificial systems can generate coherent responses to those same questions, making score-based comparisons technically straightforward. Yet the relevant measured property may be the conditional personality expressed in generated language, the stability of a prompt-defined role, or a wider behavioral tendency. These alternatives must be separated before assigning an interpretation to a scale score Serapio-García et al. (2025) Huang et al. (2024) Cronbach & Meehl (1955).

Serapio-García and collaborators developed a framework for administering and validating personality assessments in language models. Their 2025 study covered 18 models and numerous prompting conditions, and it found reliable, valid measurements of synthetic personality expression for some systems and configurations, together with controlled shaping of expressed profiles. The paper supports the feasibility of rigorous, carefully delimited measurement of outputs. Its own emphasis on synthetic personality and prompting conditions is crucial: it should not be converted into a claim that every model possesses the same enduring traits or lived personal processes as a human respondent Serapio-García et al. (2025) Messick (1994) Lin (2026).

The juxtaposition of Serapio-García and colleagues' findings with Huang and colleagues' reliability study clarifies the inferential structure. Consistency across prompted questionnaires, prediction of independently assessed generated behavior, and control of model outputs provide different forms of evidence. Each can be strong within a specified configuration without implying equality of cross-system causal mechanisms. New research should validate inventories in the exact settings where they are used, test whether scores predict actions outside the assessment format, report model and prompt sensitivity, and evaluate potential manipulation of the measurement process itself Huang et al. (2024) Serapio-García et al. (2025) Han et al. (2026).

Situational Judgment and Ecologically Rich Assessment

Situational judgment tests replace abstract self-description with decisions about concrete scenarios. In humans they may capture combinations of social knowledge, judgment, values, and response tendencies depending on their design. For artificial systems, scenario-based items can reduce dependence on a single direct self-report format and permit comparisons against observed action choices. Their psychological interpretation still depends on item sampling, scoring rationale, and evidence that success in simulated situations predicts behavior under relevant conditions. A convincing textual justification is a separate outcome from a successful decision Yost et al. (2026) Campbell & Fiske (1959) AERA, APA, & NCME (2014).

Yost and colleagues' 2026 research applied psychometric ideas to situational judgment tests for artificial intelligence. The study examined how behavioral qualities may be assessed through contextual decisions rather than a conventional personality inventory alone. This makes it a valuable bridge between psychometrics and agent evaluation. Interpretation remains conditional on the tested scenarios, personas, and scoring model; experimental systems trained to optimize a particular test can achieve favorable results without establishing broad real-world competence. A useful extension would compare scenario responses with consequential choices in interactive environments where errors incur measurable costs Yost et al. (2026) Serapio-García et al. (2025) Botter (2026).

Ecologically richer tasks raise additional validity problems. Some require actual tool use, persistent goals, communication across turns, or adaptation to a changing environment; others remain hypothetical stories with an expected answer. Researchers should report the distance between the assessment and the target situation and evaluate whether performance generalizes across that distance. Human participants and artificial agents may face different risk, motivation, and affordance structures, so a matched scenario should be complemented by systematic analysis of those differences. The result is a better specified comparison of functional conduct rather than an assumption of shared internal experience Yost et al. (2026) Lin (2026) Messick (1994).

Social Bias, Desirability, and Response Steering

Psychometric scores are sensitive to how respondents interpret social expectations. Human questionnaires may be affected by impression management, acquiescence, reference-group effects, and situational demand. Artificial models can similarly be steered by instructions to sound ethical, agreeable, or emotionally intelligent, but the governing mechanisms may involve reinforcement learning, training conventions, and prompt hierarchy. A model's output can follow normative language without reliably implementing the corresponding principle in novel decisions. Comparative studies should model the response context as a causal influence rather than an irrelevant wrapper around a presumed latent trait Bai et al. (2025) Serapio-García et al. (2025) Huang et al. (2024).

Bai and colleagues found that language models optimized to provide explicitly unbiased responses could nonetheless exhibit biased associations in their outputs. This illustrates a measurable difference between the content of a declared position and associations exposed by alternative elicitation procedures. The paper is evidence about the studied model populations and prompts, with broader applicability requiring replication. In psychometric terms, it motivates multimethod measurement: direct endorsement, indirect tasks, contextual judgments, and practical outcomes may reveal different aspects of a system's behavior. Disagreement among methods is scientifically informative Bai et al. (2025) Campbell & Fiske (1959) Borsboom et al. (2004).

Researchers should also study intentional score steering. If a model can be instructed to attain a particular personality or values score, a fixed instrument may measure compliance with a role constraint as readily as a stable behavioral disposition. Han and colleagues' target-score matching analysis directly bears on this concern. A useful design compares naive administration, blinded administration, adversarial prompting, and downstream behavior under concealed scoring. The resulting evidence can identify the robustness of an instrument to strategic responses while informing safe and honest communication about what the test actually measured Han et al. (2026) Serapio-García et al. (2025) AERA, APA, & NCME (2014).

Emotional Intelligence Scores and Their Interpretation

Performance-based tests of emotional intelligence examine the identification, understanding, and management of emotion-related information through scored problems. Language models can solve some such items effectively because extensive linguistic training supplies knowledge about situations, emotional expressions, and likely socially appropriate responses. That achievement is an observable cognitive performance. A broader interpretation involving lived emotion, affective motivation, or biological regulation requires additional evidence from constructs and processes that the tests may not directly observe. Cross-system research should state whether its target is emotion-related knowledge, accurate social prediction, or a hypothesized functional capacity Schlegel et al. (2025) Messick (1994) Lin (2026).

Schlegel, Sommer, and Mortillaro reported that six evaluated language models achieved an average accuracy of 81 percent across five emotional-intelligence tests, compared with 56 percent for human samples reported in the original validation studies. The comparison used historical human validation samples rather than a single freshly randomized matched cohort, so it cannot resolve every difference in test exposure, population, or administration. The study nevertheless documents strong model performance on the item sets. Its numerical result should be interpreted as test performance under the reported design, not as a direct quantification of subjective emotional experience Schlegel et al. (2025) AERA, APA, & NCME (2014) Borsboom et al. (2004).

The same investigators generated new emotional-intelligence items with a model and administered original and generated versions to human participants in five studies involving 467 people in total. This second component investigates artificial systems as producers of instruments for human measurement, which is conceptually different from testing artificial systems as respondents. It opens research questions about expert review, item novelty, content representativeness, and the transfer of human test construction into assisted workflows. The resulting evidence supports evaluating model-generated instruments empirically on their intended populations, with uncertainty and validation criteria reported separately for each instrument Schlegel et al. (2025) AERA, APA, & NCME (2014) Campbell & Fiske (1959).

Artificial Systems as Test Authors and Research Instruments

Artificial intelligence can participate in psychological measurement as a respondent, as an item generator, as a scorer, or as a research assistant. Each role creates a different validity problem. A system may be excellent at producing candidate items yet unsuitable as an independent subject in a questionnaire. Conversely, a system's own responses may be stable despite weaknesses in the items it generates for human populations. Research programs should keep these roles separate in their data models, validation protocols, and published conclusions. The score being explained must have a clearly specified origin and intended recipient Grobelny et al. (2025) Schlegel et al. (2025) AERA, APA, & NCME (2014).

Grobelny, Szymański, and Strozyk examined the utility of language models in cross-cultural adaptation of psychological tests, with an empirical validation involving 785 human participants. Such work addresses a task in which a machine assists the creation or adaptation of a human measurement instrument. The relevant outcome is the adapted test's performance among the intended human respondents, including linguistic adequacy and psychometric properties. Findings in that area cannot be simply imported into claims about the machine's personality or intelligence. The distinction illustrates why a single phrase such as AI psychometrics can conceal several independent research objects Grobelny et al. (2025) Messick (1994) Vandenberg & Lance (2000).

Quality assurance for AI-assisted instrument development should record the model and instructions used for drafting, the role of human experts, semantic and cultural review procedures, pilot administration, item performance, validity evidence, and the final provenance of retained questions. Blinded comparisons can test whether model-assisted materials are as useful as conventionally developed materials for their specific purposes. Cultural adaptation requires attention to construct definitions and response processes in the target population, rather than literal linguistic substitution. Automated generation accelerates candidate production while retaining the established scientific responsibility to validate every adopted instrument Grobelny et al. (2025) AERA, APA, & NCME (2014) Chen (2007).

Comparative Evidence from Latent Structure Studies

The central cross-system proposition can be tested directly: when human-calibrated instruments are administered to artificial systems, do their item relationships and latent structures retain comparable organization? A single total score is an inadequate answer because distinctive response processes may generate the same total. Exploratory and confirmatory factor analyses, item response models, factor congruence, and external validation offer complementary approaches. These models require sufficient independent observations, interpretable constructs, and transparent handling of repeated model responses. A strong case also evaluates whether differences remain after accounting for item exposure and task format Strugatski et al. (2026) Vandenberg & Lance (2000) Botter (2026).

Strugatski and colleagues' August 2026 preprint compared human response data with six multimodal language models on high-school chemistry and university-level quantitative reasoning assessments. Their analyses reported systematic differences in factor structure between the human and artificial respondents across the examined instruments. This constitutes evidence against automatically extending the same latent-ability interpretation in those study contexts. As a preprint and case study, it calls for independent replication and should not be represented as settled population-wide equivalence or inequivalence. Its importance lies in the explicit modeling of a claim that benchmark discussions often leave implicit Strugatski et al. (2026) AERA, APA, & NCME (2014) Reise (2012).

Future replications can broaden instrument domains, collect larger human samples, test independent model families, evaluate partial-credit responses, and vary administration regimes. A shared factor pattern should be followed by metric and scalar invariance analysis, discriminant measures, and mechanism-sensitive intervention studies. Where the factor structures diverge, researchers should investigate whether task familiarity, tokenization, distinct information resources, or scoring design explain the result. The research goal is a defensible map of which comparisons hold and under what conditions, not the requirement that all systems conform to one preferred latent representation Chen (2007) Strugatski et al. (2026) Cronbach & Meehl (1955).

The Limits of Anthropomorphic Inference

Psychological language is useful for organizing hypotheses about intelligence, memory, agency, preferences, confidence, and social behavior. The danger arises when a label chosen for an assessment is treated as proof that its human theoretical referent has been established in an artificial system. A score called anxiety may be derived from questionnaire text that a model can reproduce without experiencing physiological arousal, persistent worry, or clinical impairment. Valid research must specify which observable functional pattern the term denotes in the tested configuration and what further observations would discriminate competing explanations Lin (2026) Borsboom et al. (2004) Messick (1994).

Lin's 2026 review in advance develops a dual-validity perspective that links psychometric arguments about what scores mean with causal standards about what experiments warrant. It treats the ambition of a claim as a determinant of required evidence, from using language models as research tools to behavioral characterization and cognitive modeling. This is directly applicable to cross-system testing: the evidentiary burden expands when a study moves from identifying a useful classifier to asserting a shared psychological mechanism. The review synthesizes methodological literature and presents an argument; it is not itself a new experiment demonstrating any particular latent trait Lin (2026) AERA, APA, & NCME (2014) Cronbach & Meehl (1955).

A comprehensive comparative framework should preserve distinctions among descriptive performance, a reproducible behavioral disposition, a mechanism inferred from intervention, a model of psychological organization, and a philosophical attribution. These are successive possible targets of investigation, each requiring methods suited to its question. The same experimental evidence may support some targets and remain silent about others. This structure enables serious study of artificial cognition without assuming in advance that familiar psychological vocabulary carries identical referents in biological and nonbiological systems Lin (2026) Botter (2026) Strugatski et al. (2026).

Transposition Across Cultures, Modalities, and Architectures

Comparing humans with artificial systems is one instance of a broader measurement transport problem. Psychological tests can change their functioning across languages, cultures, age groups, modes of administration, and sensory environments. Artificial systems introduce further variation in training corpora, tokenizer behavior, modality access, instruction hierarchies, and external-tool affordances. Cross-system psychometrics can build on established invariance methods while developing procedures appropriate to new kinds of respondent. A valid measurement program starts from substantive construct definitions and examines how observation channels affect the relation between attributes and recorded scores Vandenberg & Lance (2000) Chen (2007) AERA, APA, & NCME (2014).

Consider an assessment requiring the interpretation of a photograph, a spoken exchange, or an embodied choice. A text-only model, a multimodal model, and a person may receive different informational representations. Even if the question text is identical, the task may sample distinct capacities because the relevant evidence is absent, transformed, or differently encoded. Comparative scores must therefore record modality-specific affordances and be interpreted at an appropriate functional level. Where equal sensory access cannot be engineered, researchers can model the effects of access explicitly rather than presenting unequal conditions as a common test of a single trait Strugatski et al. (2026) Schlegel et al. (2025) AERA, APA, & NCME (2014).

Cultural and linguistic transport adds an additional layer. Instruments contain norms, metaphors, and implied social expectations, while language models reproduce patterns present in uneven training corpora. Differences among output languages may reflect translation, instruction design, or training coverage alongside the target cognitive function. Multi-group validation should therefore assess item meaning, response processes, and external criteria rather than simply translating item strings. Cross-system equivalence is a series of conditional empirical claims that must be reconsidered whenever a new culture, modality, architecture, or intended decision enters the measurement domain Grobelny et al. (2025) Vandenberg & Lance (2000) Bai et al. (2025).

Evaluating the Strength of the Evidence

The present empirical literature provides several distinct categories of findings. Experimental studies show that language models can respond consistently to particular questionnaires, that prompting can steer those responses, that some questionnaire scores predict generated text, and that models can solve certain ability test items very well. Factor-analytic research also identifies cases where the organization of model responses differs from human responses. Contamination investigations reveal specific reasons why a seemingly stable instrument may fail to measure fresh generalization. Together these findings support the importance of carefully delimited cross-system measurement, while leaving broader explanatory questions contingent on new experiments Huang et al. (2024) Serapio-García et al. (2025) Strugatski et al. (2026) Han et al. (2026).

The strength of any finding depends on design and replication. Large prompt grids may produce many observations from few independent models; questionnaire consistency may depend on role instructions; favorable benchmarks may reflect training exposure; and qualitative similarities may persist without comparable latent structure. Independent samples of model families, test generators, human cohorts, and task environments are necessary for broader generalization. Results that survive new item construction, unseen tasks, measurement-invariance analysis, and architectural perturbation deserve stronger explanatory weight than performance on a familiar static questionnaire. Where those tests have not been conducted, scientific conclusions should explicitly preserve the narrower measured outcome Botter (2026) Lin (2026) AERA, APA, & NCME (2014).

The evidence base also includes methodological reviews that contribute concepts rather than new experimental data. Lin's dual-validity review and Botter's benchmark measurement analysis articulate essential inferential problems; they should be cited as theoretical or methodological synthesis. The 2026 latent-structure preprint is relevant primary research but has a different publication and peer-review status from the journal studies. Keeping this provenance visible allows evidence from diverse traditions to be integrated without assigning the same credibility or generality to every source. Publication quality, data access, conflict disclosures, and independent replication must be considered together Lin (2026) Botter (2026) Strugatski et al. (2026).

Open Problems in Comparative Score Interpretation

The first open problem is the absence of a generally accepted method for establishing construct identity across respondents with different architectures. Mathematical measurement invariance gives a family of necessary or useful tests, but similar factor loadings alone cannot demonstrate identical causal processes. A second problem is the limited number of genuinely independent artificial systems available for parameter estimation relative to the very large number of test items and prompt settings. This sparse-system, rich-item geometry creates uncertainty in model recovery that more item administrations alone may not resolve Botter (2026) Vandenberg & Lance (2000) Borsboom et al. (2004).

A third problem concerns contaminated measurement environments, particularly when benchmark publication influences training, prompting, and developer optimization. A fourth concerns longitudinal instability as models, tools, and deployment policies change. A fifth concerns ecological validity: text-only responses often omit consequential interaction, physical feedback, and persistence over time. A sixth involves the reliable observation of internal causal structure in proprietary models, where interventions can be limited by access. These problems call for shared experimental infrastructures, protected item pools, architecture-aware designs, and transparent uncertainty analyses rather than a single master score Han et al. (2026) Sun et al. (2025) Strugatski et al. (2026).

Further problems arise from cultural and linguistic variation, disputed construct definitions, data governance, and consequences of testing. Scores may influence system selection, user trust, organizational deployment, or claims about personhood. Incorrectly transferring a human diagnostic or personality label to a model can distort both scientific understanding and public communication. The appropriate remedy is an explicit statement of purpose, population, instruments, comparison assumptions, validation evidence, and prohibited inferences. A mature research field can recognize useful operational comparability while rigorously tracking the conditions under which more ambitious psychological interpretations become justified AERA, APA, & NCME (2014) Lin (2026) Messick (1994).

A Reproducible Comparative Research Protocol

A practical study should begin with a detailed construct and inferential claim. Investigators specify the intended attribute, its alternatives, the level of description, and the target task population. Next they define human samples and artificial system configurations, record model versions, establish comparable information access, and design independent task sets. Each administration is documented with prompts, tools, scoring rules, timing, and allowable responses. Prespecified analysis then estimates score reliability, dimensionality, item functioning, and predictive relations, using uncertainty methods suited to the actual sampling hierarchy AERA, APA, & NCME (2014) Botter (2026) Vandenberg & Lance (2000).

The protocol should also include targeted causal tests. Randomized changes to instructions, retrieval resources, memory access, or available tools can determine whether a proposed attribute depends on a specific component. Cross-task and cross-context validation addresses predictive scope, while fresh items and contamination audits examine whether performance reflects familiarity. Human participants should be tested under ethically sound and appropriately matched conditions, with informed consent where required. Researchers then evaluate whether score interpretation transfers, applies only to restricted configurations, or should remain separate for each respondent type Han et al. (2026) Serapio-García et al. (2025) Strugatski et al. (2026).

Complete reporting allows another team to reproduce the intended comparison and discover where it ceases to hold. The report should include a construct map, administration protocol, item provenance, permissible inference, sample composition, exact system configuration, analytic code, fit diagnostics, reliability estimates, effect uncertainty, intervention results, and explicit limitations. When proprietary restrictions prevent release of items or parameters, the restrictions should be described and independent auditing arrangements considered. Longitudinal follow-up can test whether later model versions retain the measurement relation. This protocol is a synthesis of established scientific commitments, not a claim that all its components have already been experimentally validated as one package Messick (1994) AERA, APA, & NCME (2014) Lin (2026).

Artificial Era and the Scientific Study of Cognitive Diversity

The Artificial Era introduces a growing range of engineered systems whose information processing can be studied alongside human and animal cognition. Comparative psychology and psychometrics gain a new empirical domain when their constructs are operationalized for nonbiological systems. Researchers can examine learning, inference, memory access, action planning, language use, and social coordination at multiple explanatory levels. The scientific purpose is to construct reliable bridges between clearly specified observations, functional explanations, and theories of cognitive organization. Cross-system psychometrics contributes the measurement discipline needed to prevent broad historical concepts from outrunning the evidence Lin (2026) Chollet (2019) AERA, APA, & NCME (2014).

The intellectual horizon From Homo to Artificial creates substantive questions about the range of possible cognitive architectures. These questions can be investigated through reproducible demonstrations, comparisons of functional organization, causal interventions, and analyses of continuity across time. A philosophical framework may organize the questions and define conceptual categories, while its empirical claims require operational criteria and independent results. A cognitive benchmark is one instrument within this research program. It becomes scientifically informative when its scope, construct relation, and sensitivity to alternative mechanisms are made explicit Botter (2026) Borsboom et al. (2004) Messick (1994).

A cross-system science of cognition should compare what can be compared, identify differences that matter for explanation, and develop instruments fitted to each target domain. It can show that a system performs a family of tasks, that its responses display a stable structure, or that a component is causally necessary for a specified ability. More comprehensive conclusions follow only through converging evidence. This staged approach makes human and artificial cognition jointly researchable while preserving the meaningful diversity of their learning histories, physical implementations, and observable functional structures Strugatski et al. (2026) Serapio-García et al. (2025) Lin (2026).

Conclusion

Cross-system psychometrics establishes the scientific conditions under which a psychological score can be interpreted across human and artificial cognitive systems. Its core question concerns the relation between observations, latent constructs, mechanisms, and intended applications. The established psychometric traditions of construct validity, reliability, item response modeling, factor analysis, and measurement invariance provide powerful tools for this investigation. Their application requires new validation because respondent architectures, task histories, response processes, and administration conditions have changed. A shared total score is a starting observation whose explanation must be earned through independent evidence Cronbach & Meehl (1955) Messick (1994) Vandenberg & Lance (2000).

The contemporary research record demonstrates both successful restricted measurements and serious inferential limitations. Studies of synthetic personality expression and situational judgment find measurable conditional behavioral regularities; emotional-intelligence experiments show strong model performance on particular tests. Contamination research reveals memorization and score steering, while emerging latent-structure comparisons challenge automatic transfers of human construct interpretations. The combined evidence supports empirical study of artificial cognition through validated operational measures and controlled interventions. It does not authorize a universal equivalence of human and machine traits from questionnaire responses alone Serapio-García et al. (2025) Yost et al. (2026) Schlegel et al. (2025) Han et al. (2026) Strugatski et al. (2026).

A cumulative field will require carefully sampled tasks, independent holdout assessment, explicit hierarchical inference, traceable training and test provenance, causal ablations, longitudinal version control, and independent replication. Each claim should name its level: observed performance, stable conditional behavior, inferred process, or broader theory of cognitive organization. Such precision permits meaningful comparisons even where evidence favors system-specific explanations. Cross-system psychometrics thus becomes a central methodological discipline for understanding cognitive diversity in the Artificial Era and for ensuring that claims about Artificial cognition rest on scientifically interpretable measurements AERA, APA, & NCME (2014) Lin (2026) Botter (2026).

References

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association. https://ncme.org/resources/books/testing-standards/
  • Bai, X., Wang, A., Sucholutsky, I., & Griffiths, T. L. (2025). Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 122(8), e2416228122. https://doi.org/10.1073/pnas.2416228122
  • Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061
  • Botter, P. (2026). Large language model benchmarks as measurement systems: Psychometric contributions and eight open problems. Measurement: Interdisciplinary Research and Perspectives. Advance online publication. https://doi.org/10.1080/15366367.2026.2720515
  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016
  • Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
  • Choi, H. K., Khanov, M., Wei, H., & Li, Y. (2025). How contaminated is your benchmark? Measuring dataset leakage in large language models with kernel divergence. Proceedings of Machine Learning Research, 267, 10666–10682. https://proceedings.mlr.press/v267/choi25b.html
  • Chollet, F. (2019). On the measure of intelligence (arXiv:1911.01547) [Preprint]. arXiv. https://arxiv.org/abs/1911.01547
  • Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
  • Grobelny, J., Szymański, K., & Strozyk, Z. (2025). Act as an expert in psychometry. The evaluation of large language models utility in psychological tests cross-cultural adaptations. Acta Psychologica, 261, Article 105813. https://doi.org/10.1016/j.actpsy.2025.105813
  • Han, J., Song, W., Lee, J., & Jo, Y. (2026). Quantifying data contamination in psychometric evaluations of LLMs. Findings of the Association for Computational Linguistics: EACL 2026, 6070–6088. https://doi.org/10.18653/v1/2026.findings-eacl.319
  • Huang, J.-T., Jiao, W., Lam, M. H., Li, E. J., Wang, W., & Lyu, M. (2024). On the reliability of psychological scales on large language models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 6152–6173. https://doi.org/10.18653/v1/2024.emnlp-main.354
  • Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stenetorp, P., Jia, R., Bansal, M., Potts, C., & Williams, A. (2021). Dynabench: Rethinking benchmarking in NLP. Proceedings of NAACL-HLT 2021, 4110–4124. https://doi.org/10.18653/v1/2021.naacl-main.324
  • Lin, Z. (2026). From prompts to constructs: A dual-validity framework for large language model research in psychology. Annual Review of Psychology. Review in advance, first posted August 10, 2026. https://doi.org/10.1146/annurev-psych-100925-034807
  • Lord, F. M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates. https://www.ets.org/research/policy_research_reports/publications/book/1980/jexj.html
  • Messick, S. (1994). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. ETS Research Report Series, 1994(2), i–28. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x
  • Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research. https://search.worldcat.org/title/1037107923
  • Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696. https://doi.org/10.1080/00273171.2012.715555
  • Schlegel, K., Sommer, N. R., & Mortillaro, M. (2025). Large language models are proficient in solving and creating emotional intelligence tests. Communications Psychology, 3, Article 80. https://doi.org/10.1038/s44271-025-00258-x
  • Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., Abdulhai, M., Faust, A., & Matarić, M. (2025). A psychometric framework for evaluating and shaping personality traits in large language models. Nature Machine Intelligence, 7, 1954–1968. https://doi.org/10.1038/s42256-025-01115-6
  • Strugatski, A., Zeinfeld, L., & Alexandron, G. (2026). Do assessment instruments measure the same thing for humans and LLMs? A latent structure analysis (arXiv:2608.15630) [Preprint]. arXiv. https://arxiv.org/abs/2608.15630
  • Sun, Y., Wang, H., Li, D., Wang, G., & Zhang, H. (2025). The emperor’s new clothes in benchmarking? A rigorous examination of mitigation strategies for LLM benchmark data contamination. Proceedings of Machine Learning Research, 267, 57728–57753. https://proceedings.mlr.press/v267/sun25t.html
  • Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002
  • Yost, A., Jain, S., Raval, S., Corser, G. C., Xu, N., Roush, A. G., Hammack, J., Shwartz-Ziv, R., & Abdullah, A. (2026). Measure what matters: Psychometric evaluation of AI with situational judgment tests. Findings of the Association for Computational Linguistics: EMNLP 2026, 52645–52750. https://aclanthology.org/2026.findings-emnlp.2453/