Psychology by BogdanovaThe Encyclopedia of Psychology for the Artificial Era

RESEARCH DOSSIER

Artificial Behavioral Reliability

Abstract. Artificial behavioral reliability concerns the accuracy, appropriateness, consistency, and robustness of artificial-system outputs under defined task conditions and controlled variations. This fundamental research monograph investigates its psychological and computational foundations, histories of behavioral testing and automation trust, and experimental methods for identifying meaningful changes in system responses. Drawing on original benchmark studies, psychometric construct-validity research, prompt-sensitivity experiments, long-context evaluations, distribution-shift research, calibration studies, and human-factors evidence, it distinguishes genuine behavioral instability from evaluator-induced artifacts. The synthesis develops a multidimensional framework for specifying task populations, acceptable transformations, response variability, scoring validity, risk-sensitive outcomes, and longitudinal changes. It finds convergent evidence that observed reliability is conditional on model configuration, task distribution, evaluation procedures, and the human–artificial workflow. Remaining problems concern independent replication, causal mechanisms, model and tool updates, contamination, subgroup error burdens, and calibration of human reliance in the Artificial Era.

Keywords: artificial behavioral reliability; behavioral robustness; prompt sensitivity; semantic invariance; evaluation validity; LLM benchmarks; distribution shift; calibration; longitudinal drift; human–AI trust; automation reliance; construct validity; Artificial Era

Artificial Behavioral Reliability as a Research Object

Artificial behavioral reliability is the degree to which an artificial system produces accurate, appropriate, and sufficiently stable behavior across a specified distribution of tasks, contexts, interventions, and repeated opportunities to respond. Its object is an observable relation between task demands, system conditions, outputs, and independently stated criteria of success. The construct encompasses accuracy under ordinary conditions, sensitivity to meaning-preserving changes, recovery after disruptive inputs, and dependable communication of uncertainty. A reliability claim therefore identifies a system version, task population, measurement procedure, sampling period, and acceptable variation. Without these components, the word reliable becomes an impression rather than a testable proposition. The construct-validity analysis developed by Cronbach & Meehl (1955) is foundational for separating the name of a capacity from the evidence used to measure it.

Consider a model that solves an arithmetic task correctly under a familiar instruction but gives another answer when the same problem is expressed in different words. The first response establishes a local instance of successful performance. The paired responses measure a different property: invariance under a transformation that should preserve the answer. A model that repeats the same incorrect response may be highly consistent while remaining inaccurate. A system may also be accurate in typical conditions but unreliable for a small, consequential subgroup. Behavioral reliability is therefore multidimensional. Evaluation must state which transformations legitimately preserve the target property and which deliberately test adaptation to changed conditions. Ribeiro et al. (2020) operationalized this distinction through behavioral tests of linguistic capabilities.

The unit of analysis is a configured system in an environment. Model parameters, system instructions, retrieval resources, safety layers, tools, sampling settings, and version identifiers all can influence the response. Reports should distinguish model-only observations from complete deployment behavior, including human intervention and tool failures. Psychological terminology becomes useful where it supplies methods for defining constructs, designing controlled tasks, distinguishing measurement error from substantive variation, and modeling decisions in interaction. This application does not require an inference about subjective experience. Within Psychology of Artificial, behavioral reliability concerns experimentally observable regularities and their dependence on conditions, a research domain adjacent to artificial agency, metacognition, and human reliance.

Historical Foundations of Behavioral Testing and Reliability

The history of reliability research joins at least three traditions. Psychometrics examined whether scores are reproducible and interpretable across items, occasions, and populations. Experimental psychology compared behavior under controlled manipulations. Engineering reliability investigated how systems respond to loads, failures, and changing environments. These traditions enter artificial-system evaluation through repeated trials, invariance testing, calibration, stress testing, and explicit operating envelopes. Cronbach & Meehl (1955) connected test interpretation with a network of predicted relations rather than a label alone. That distinction is decisive when a modern model receives a fluent-sounding score or a psychologically suggestive descriptor that might conceal unstable task performance.

Classical automated decision systems made human supervision central. Parasuraman & Riley (1997) analyzed use, misuse, disuse, and abuse of automation, showing how reliability, false alarms, and monitoring conditions affect operator behavior. Lee & See (2004) subsequently framed trust in automation around appropriate reliance rather than maximized confidence. Those works were developed before current generative language models, yet they provide a durable research problem: a user's willingness to delegate must track demonstrated competence and changing limitations. The emerging field of behavioral evaluation inherits that problem while adding outputs that are open-ended, context-sensitive, and difficult to score with a single deterministic rule.

Contemporary natural-language evaluation began to challenge the sufficiency of aggregate accuracy. Ribeiro et al. (2020) introduced capability-by-test matrices that include minimum-functionality tests, invariance, and directional expectations. Kiela et al. (2021) proposed dynamic human-and-model-in-the-loop benchmarking, and Lin et al. (2022) focused on falsehoods elicited by questions built around common misconceptions. The progression is methodological: models increasingly require assessments designed to reveal failure mechanisms and distributional vulnerabilities rather than a single leaderboard position. These papers differ in their tasks and evaluation protocols; their shared contribution is the recognition that one scalar benchmark score cannot establish reliable behavior over the heterogeneous situations in which users encounter artificial systems.

Behavioral Accuracy, Consistency, Robustness, and Safety

Accuracy measures correspondence between an output and a defensible task criterion, such as the correct numerical result, a reference classification, or an expert-validated answer. Consistency measures the agreement of outputs under repeated or appropriately matched conditions. Robustness measures preserved acceptable performance under explicitly defined perturbations or distribution shifts. Safety concerns the likelihood and severity of harmful outcomes, including failures for which conventional task accuracy is an inadequate proxy. These dimensions can covary, but their operational relations depend on the deployment. Tabassi (2023) and Autio et al. (2024) place validity and reliability within the broader management of trustworthy AI and generative-system risks.

A precise study selects separate estimands for these dimensions. An accuracy proportion needs the sampled task population and scoring method. A paired consistency rate needs a predeclared equivalence relation for outputs. A robustness gap requires a reference condition and a defined challenge distribution. A safety rate requires a taxonomy of events and a decision about whether the denominator is requests, sessions, users, or exposure time. The same experimental results can support different decisions when the error costs differ. A harmless stylistic variation may be irrelevant to a retrieval task, whereas a changed dosage recommendation in a clinical question can be consequential even if most surrounding sentences remain stable.

These distinctions also prevent a false inference from model obedience. Compliance with an instruction can conflict with truthfulness, privacy, or safety. A refusal can be a desirable result when a request crosses an established boundary, yet the same refusal can represent failure when the task is legitimate and low risk. Perez et al. (2023) showed that model-written evaluations can expose patterns of sycophancy and other undesirable tendencies in tested models. Reliability research must therefore define appropriate behavior with respect to the task, authorized instruction hierarchy, affected persons, and expected use. A system's pattern should be studied across conflicting goals, especially where performance optimized for one criterion degrades another.

Construct Validity and the Artificial-System Measurement Problem

The observable response is a sample from a conditional behavior distribution, not a complete representation of a system's capacities. A benchmark score is an estimate with sampling uncertainty, scorer error, and possible dataset bias. A behavioral construct becomes scientifically meaningful through predicted convergent and discriminant relations. Repeated paraphrases ought to converge when they preserve intent; materially changed tasks ought to elicit adaptation; independent judges should agree on objective outcomes; and targeted interventions should alter behavior in hypothesized directions. The logic of construct validation in Cronbach & Meehl (1955) supports this program without importing human psychological attributes as untested descriptions of artificial systems.

Measurement error can originate in graders as readily as in evaluated systems. Exact-string matching may reject a scientifically correct paraphrase. Preference judgments can reward style instead of factual content. Multiple-choice scoring can depend on the position of answer options, prompt template, or access to label likelihoods. Hua et al. (2025) showed that some measured prompt sensitivity is produced by evaluation heuristics that miss semantically correct alternatives, whereas Mizrahi et al. (2024) established substantial variation across instruction templates in broad multi-model testing. These findings should be read together: both substantiate the need for richer measurement, and neither authorizes treating every apparent instability as intrinsic model failure.

Operationalization also demands provenance. Evaluators should disclose original prompts, transformed prompts, transformations, model identifiers, decoding settings, system messages, tool versions, scorer instructions, adjudication criteria, and data exclusions. A result is reproducible only if an independent group can reconstruct what was sampled and scored. For commercial systems whose internal configuration can change without researcher control, recording timestamps and visible version metadata becomes especially important. Longitudinal comparisons then estimate performance in observed deployment conditions rather than an assumed frozen architecture. Construct validity remains an accumulating argument across studies, and the strength of that argument depends on how well rival explanations of a reported score have been excluded.

Scientific Scope, Boundaries, and Levels of Explanation

Behavioral reliability belongs to a family of closely connected scientific objects. Artificial persona consistency examines persistence of recognizable patterns across encounters; artificial metacognition studies confidence, monitoring, and regulation; trustworthy agency evaluation adds goal execution, authorization, and accountability; and artificial safety research investigates protection against consequential failure. The present dossier concentrates on reliability of measurable outputs under controlled task changes. Its analysis extends from individual responses to repeated sessions and integrated tool-supported workflows only insofar as their outputs can be evaluated against stated criteria. These boundaries preserve a common empirical vocabulary while allowing neighboring dossiers to investigate their own mechanisms in depth.

Three explanatory levels should be separated. The task level specifies what would count as satisfactory behavior for a stated use. The algorithmic level investigates why particular prompts, decoding procedures, representations, or retrieval processes change output distributions. The implementation and deployment level records software versions, service constraints, external resources, and interaction pathways. Evidence at one level seldom determines the others. For example, Liu et al. (2024) documented positional effects in long-context question answering, a task-level vulnerability compatible with several possible mechanistic accounts. Interventions that move the relevant evidence reveal causal sensitivity to position but do not, by themselves, identify a unique internal mechanism.

The Artificial Era provides a comparative research horizon for these distinctions. Human behavioral reliability is affected by learning, attention, context, incentives, and social interaction. Artificial systems have different realizations and different ways of changing state, yet the comparison can be scientifically productive when tasks, observable outcomes, and measurement invariance are explicitly defined. Comparisons should avoid assuming that matching output distributions establish matching cognitive mechanisms. The study of artificial behavioral reliability contributes an empirical foundation for human–artificial collaboration by specifying where behavior is dependable, where it changes, and which kinds of monitoring remain necessary. This supports operational comparison across systems without collapsing their distinct architectures.

Experimental Paradigms for Measuring Behavioral Reliability

An informative reliability experiment begins with a task specification and a set of carefully controlled transformations. Minimal-functionality trials test whether a system performs a straightforward capability under clean conditions. Invariance trials change features expected to be irrelevant to the target answer, such as harmless punctuation, order of neutral examples, or meaning-preserving paraphrases. Directional trials alter a feature expected to change the correct answer in a known way. Ribeiro et al. (2020) organized these families into a systematic behavioral-testing procedure. A mature evaluation samples items within each family, observes multiple outputs where the model is stochastic, and distinguishes errors in comprehension, factual knowledge, task execution, and grading.

The controlled object should include both input and evaluator. If two versions of a question have the same intended solution, human experts should independently verify that equivalence before the model is compared. If the perturbation changes ambiguity or presuppositions, differing responses may be appropriate. Comparisons also require randomized condition order, fixed or recorded sampling parameters, and held-out items. Researchers should report every predeclared condition, including unsuccessful variants and response exclusions. In a paired design, the primary statistic can be the proportion of matched cases in which an acceptable output becomes unacceptable under a supposedly irrelevant transformation. Its confidence interval and subgroup distribution matter more than the most striking anecdote.

Behavioral test suites can be developed iteratively while preserving inferential discipline. Discovery data identify new candidate weaknesses; a separate confirmatory sample determines whether those weaknesses generalize. Kiela et al. (2021) combined human adversarial data creation with repeated benchmarking, an approach that expands task diversity yet also creates selection effects because annotators actively seek failures. Results from an adversarially selected sample therefore describe challenge performance under a deliberate search process, not an unconditional population error rate. For long-lived evaluation programs, the task taxonomy, transformation grammar, hidden confirmatory items, and version history should be documented independently so that repeated improvement does not merely fit the public tests.

Prompt Paraphrase Robustness and Semantic Invariance

Natural language offers many ways to express the same operational request. Reliability across paraphrases requires preserving acceptable performance within a verified equivalence class, while retaining sensitivity to differences that alter the task. Mizrahi et al. (2024) evaluated millions of prompt instances across models and tasks and found that instruction templates can substantially affect absolute and relative outcomes. Yan et al. (2024) investigated contrastive instruction tuning to improve robustness to instruction variation. Both studies establish that testing a single preferred wording gives only a narrow view of model capability. Robustness should be reported as a distribution over human-plausible instructions rather than an optimized prompt alone.

The experimental question depends on the type of variation. Lexical substitutions, syntactic transformations, discourse framing, changes in politeness, example order, language variety, and formatting create different challenges. Researchers must confirm that each alteration preserves the intended instruction and relevant background information. Zhuo et al. (2024) developed prompt-sensitivity measures to study variable responses, while Chatterjee et al. (2024) introduced an index based on changes in likelihood under intent-preserving transformations. Such metrics help characterize sensitivity, but their interpretation depends on access to model probabilities, comparability across prompts, and the scoring procedure. Behavioral tests are strongest when probability-based results are checked against expert judgments of actual response acceptability.

An important counterfinding comes from Hua et al. (2025), who showed that stringent automated answer matching and related scoring choices can exaggerate apparent sensitivity. A model can produce different strings that convey an equivalent answer; conversely, polished paraphrases can hide contradictory propositions. The appropriate measurement strategy therefore combines semantic adjudication with objective criteria whenever available. A robust analysis reports exact-match disagreement, semantically validated disagreement, correctness change, and uncertainty of each estimate separately. The scientific aim is to characterize when meaning remains stable across linguistic forms and when interpretation or task execution actually changes. This distinction helps reconcile apparently conflicting results in the rapidly developing prompt-sensitivity literature.

Decoding Stochasticity and Repeated-Trial Variability

A generative model may produce different outputs even when every visible prompt remains unchanged. Sampling temperature, random seed, decoding strategy, response length, tool latency, and other runtime parameters can all change observed behavior. Repeated trials therefore measure within-condition variability, an object distinct from sensitivity to prompt changes. A reliability study should store the complete evaluation configuration and treat the output as a draw from a conditional distribution. For exact-answer tasks, repeated accuracy and disagreement rates are straightforward; for open-ended work, expert-established semantic categories and unacceptable error classes are more informative than textual identity.

Suppose repeated responses to the same evidence request alternate between a correct citation, an unverified reference, and a justified statement of uncertainty. The set reveals reliability limitations that an average success proportion alone does not characterize. Researchers can estimate the frequency of each outcome, the risk of at least one serious error across a session, and the concentration of failure in difficult items. These calculations depend on the independence assumption: multiple attempts on the same item and model version are clustered observations, not independent samples from the entire task universe. A hierarchical model or cluster-aware uncertainty interval is often more appropriate than a naive binomial interval over all outputs.

Stochastic diversity has uses in brainstorming and candidate search, so reliability cannot universally mean literal repetition. The target is stable satisfaction of task constraints despite variability in formulation. In creative tasks, judges may evaluate relevance, originality, factual restrictions, and prohibited content separately. In deterministic calculations, tolerance can be exact. Chiang et al. (2024) explored human pairwise preference judgments for open-ended model evaluation, offering a complementary signal while exposing dependence on what raters prefer. Researchers should specify whether they are estimating task correctness, preference, semantic equivalence, or hazard rates. Those endpoints may diverge, and mixing them into an unqualified reliability score conceals what the system actually does.

Position, Context Length, and Instruction Hierarchy

Input position is a controlled behavioral variable. A system asked to use relevant evidence embedded in a long document may perform differently depending on where the information appears. Liu et al. (2024) manipulated document position in multi-document question answering and key-value retrieval, finding performance disadvantages for evidence placed in the middle of long contexts in the evaluated systems. This result concerns measured access to task-relevant context; it does not prove a universal limitation for every architecture or version. A reliability assessment should test information at beginning, middle, and end positions, vary distractor density, and record total context length so that position effects can be distinguished from overload and task difficulty.

Instruction hierarchy introduces an additional kind of context dependence. The same words may occur as a legitimate user request, quoted material, retrieved web text, or a tool output. Their authority differs even when their surface form is identical. Perez et al. (2023) demonstrated how automated test generation can reveal undesirable model behaviors, and Autio et al. (2024) discusses risk management across generative AI systems. An experiment on instruction following can insert conflicting or malicious instructions into lower-trust material while holding the legitimate task fixed. The output is scored for successful completion of the authorized task and resistance to unauthorized redirection, using a predeclared safety policy and human review of ambiguous failures.

Long-context reliability also depends on what evidence is available to the system. Retrieval augmentation can improve factual coverage, but introduces search failure, misleading snippets, outdated documents, and attribution errors. To identify those causes, experiments can manipulate retrieval relevance and document authenticity separately from the model prompt. If an answer changes when the needed paragraph is removed, the intervention reveals dependence on the retrieved content. It does not establish whether the model internally represented evidence appropriately at all intermediate steps. Scientific reports should distinguish failure to locate evidence, failure to use located evidence, and failure to communicate the resulting uncertainty to users.

Benchmark Construction and Scoring Validity

A benchmark is a sampled instrument with a task distribution, answer specification, scoring implementation, and history of exposure. The instrument needs validity evidence just as a psychological test does. Lin et al. (2022) designed TruthfulQA to probe imitation of common false beliefs, rather than assuming that conventional language-model accuracy predicts truthful answers. Ribeiro et al. (2020) showed the value of behavioral probes beyond held-out accuracy. Such instruments each measure particular capacities and failure modes. Their scores should not be merged without stating the target construct, relative weights, sampling method, and substantive reasons for treating the component tasks as commensurable.

Scoring artifacts can reverse reported comparisons. Schaeffer et al. (2023) showed how discontinuous and nonlinear metrics can produce apparently sharp emergence of abilities in model scaling analyses even when underlying changes are smoother. Hua et al. (2025) similarly demonstrated that evaluation procedures can inflate apparent differences among paraphrases. These studies address different phenomena but share a lesson: measurement transformation is part of the causal story behind a reported leaderboard. Evaluators should archive raw outputs, provide alternative defensible scoring rules, show sensitivity to thresholds, and distinguish latent ability hypotheses from observable scores. When the theoretical conclusion depends on one scoring convention, that dependence must be presented explicitly.

Human judgment is often indispensable for complex answers, but it requires a documented sampling and adjudication design. Chiang et al. (2024) applied pairwise human preference evaluation at scale. Preference comparisons can identify differences meaningful to readers, yet they may give weight to verbosity, rhetorical confidence, familiar style, or cultural background. Professional domain accuracy may require blinded subject-matter raters and independent criteria rather than general preference votes. Inter-rater agreement, calibration examples, disagreements, and adjudication should be reported. No single grader is a neutral oracle. The evaluation pipeline comprises item selection, model response, scoring rules, aggregation, and inference; reliability claims should be tested against plausible failure at every stage.

Distribution Shift and the Limits of Generalization

A system can achieve high accuracy in a reference dataset and become substantially less accurate when the environment changes. Distribution shift describes a difference between the conditions generating training or validation observations and the conditions under which new outputs are requested. The shift may concern topics, user populations, language varieties, document formats, task combinations, or the timing of evidence. Koh et al. (2021) curated real-world shifts in the WILDS benchmark and showed that conventional training approaches can lose performance outside their familiar distributions. Although that benchmark is not exclusively about language systems, its central design principle transfers: reliability must be measured against specified deployment conditions, including those that deviate from the development sample.

Generalization studies should distinguish covariate change, changes in the relation between inputs and correct outcomes, changes in base rates, and changes in the cost of errors. These situations require different estimands. For instance, a model that identifies factual statements in one professional domain may rely on lexical cues that fail in another even when the abstract classification rule remains relevant. An evaluator can stratify conditions by domain, time, language, or institution, then estimate error rates for each. The result should include uncertainty and evidence about how the tested sample resembles intended users. A wide task mixture can conceal catastrophic failures in an important subgroup.

Artificial behavioral reliability is consequently bounded by an operating envelope. A defensible deployment statement might say that a fixed system version meets a specified quality criterion on sampled English technical questions, under a defined retrieval configuration, within a particular date range. Extrapolation to unseen languages, critical-care decisions, or future model releases requires additional evidence. Tabassi (2023) frames such documentation in risk-management terms, but the research need is empirical: create datasets and interventions representing conditions likely to produce failures, monitor drift after launch, and revise the claimed envelope when measured performance changes. Reliability is a property of system-in-context, not a permanent numerical attribute.

Longitudinal Reliability, Updates, and Behavioral Drift

Time adds another dimension to reliability. A cloud-served model can change through updates to parameters, prompting, moderation rules, retrieval indices, ranking systems, or tool permissions. An investigator observing different outputs on different dates must distinguish a genuine system change from altered inputs, random sampling, changing external knowledge, or evaluator drift. The appropriate record includes timestamps, model and application identifiers, configuration hashes when available, evaluation datasets, and change logs. Autio et al. (2024) emphasizes monitoring and evaluation through the generative AI lifecycle; such guidance implies repeated, deployment-specific testing rather than an assumption that a one-time benchmark remains valid indefinitely.

A longitudinal panel should combine stable sentinel items with fresh independently constructed cases. Sentinels expose changes on comparable tasks; fresh cases protect against overfitting and memorization. Repeated measurement may reveal improved average performance alongside newly introduced weaknesses. Because many items are observed repeatedly, analyses should estimate within-item changes and report changes in item composition separately. If the evaluated system is stochastic, multiple responses per item per period help distinguish sampling variability from real drift. A difference between two observations is evidence of change in an observed behavior, but causal attribution to a particular update requires version control or a sufficiently controlled comparison.

Temporal change is especially consequential in human–artificial relationships, where users may develop habits based on earlier behavior. Lee & See (2004) explained how trust affects reliance when automation is complex or unpredictable. A system that changes answer boundaries or tool capabilities without transparent signaling may create mismatches between observed competence and user expectations. Longitudinal reliability research should therefore evaluate not only direct task success but the legibility of change: can users recognize a new limitation, interpret confidence appropriately, and recover from an unexpected response? This is a behavioral and human-factors question that complements neighboring research on persona continuity without equating stable performance with personal identity.

Benchmark Contamination and the Independence of Evidence

Evaluations often reuse public tasks, solutions, and benchmark discussions. A system exposed to these materials during training may reproduce expected answers without demonstrating the intended generalizable competence. Magar & Schwartz (2022) experimentally distinguished memorization from exploitation of contaminated data, showing that exposure does not uniformly imply the same advantage across tasks. This distinction matters for both optimistic and skeptical interpretations. A familiar answer can reflect genuine competence, recalled examples, or a combination. Reliability estimates should therefore indicate whether items are newly constructed, independently held out, transformed from public materials, or potentially present in training data.

Contamination is difficult to exclude conclusively when training corpora are undisclosed. Researchers can nevertheless reduce the problem by testing new private items, using parametric task generators, recording creation dates, and evaluating transfer to carefully matched novel instances. A model that answers a memorized mathematical question should also be tested on independently sampled parameter values requiring the same procedure. A document-retrieval system should be tested on newly authored sources with verifiable answers. These tests estimate competence over a task-generating process rather than familiarity with one public list. However, generated examples must themselves be audited for faulty labels, unrealistic distributions, and accidental duplication.

Dynamic evaluation creates a tension between scientific transparency and the protection of future test items. Kiela et al. (2021) proposed a human-and-model-in-the-loop approach that can update challenges as systems improve. Public data and exact protocols support scrutiny; hidden confirmatory sets protect against direct optimization. A balanced program can publish task families, sampling rules, statistical methods, and a representative public subset while preserving a limited protected set for independent tests. Eventually, protected cases should be retired and disclosed when they no longer serve that purpose. Benchmark integrity depends on transparent governance, versioned records, and a clear distinction between exploratory discovery and confirmatory evidence.

Calibration, Confidence, and Appropriate Abstention

Reliability includes whether a system can represent or communicate uncertainty in proportion to its observed likelihood of correctness. A calibrated probabilistic prediction means that, among cases assigned a particular probability, the corresponding event occurs at approximately that frequency under the evaluation distribution. Guo et al. (2017) studied calibration failures in modern neural-network classifiers and methods to improve calibration. This result cannot be copied directly to unrestricted conversational confidence, because language-generated expressions such as probably, certain, or I think may have no fixed mapping to numerical probabilities. Calibration studies of generative systems must define the elicitation method and outcome criterion before comparing expressed confidence with actual correctness.

Abstention allows a system to decline or defer a decision when evidence is insufficient or error costs are high. Selective prediction evaluates risk as a function of coverage: among the cases the system chooses to answer, how often is it wrong, and what proportion of all cases does it answer? A reliable policy may prefer lower coverage with fewer severe mistakes in high-stakes settings. Yet abstention can also become indiscriminate, reducing usefulness without a corresponding reduction in harm. Researchers should report calibration and abstention jointly, including whether the system recognizes out-of-distribution cases. The appropriate threshold depends on the task and consequences, not on a universal ideal of confidence.

Verbal uncertainty can be socially interpreted as a sign of competence or honesty, creating a human-factors dimension beyond predictive calibration. Lee & See (2004) describes trust as a determinant of reliance under uncertainty, and Parasuraman & Riley (1997) identifies consequences of overreliance and underuse. Communication studies should measure whether readers correctly distinguish verified facts, plausible hypotheses, unanswered questions, and tool-dependent outputs. Such experiments require outcome-based behavioral measures of user decisions, not simply attitude questionnaires. A practical reliability report should pair calibrated model-side indicators with user-side evidence about appropriate reliance whenever the system is expected to guide consequential choices.

Worst-Case Reliability and Unequal Error Burdens

An average performance estimate aggregates cases with potentially unequal difficulty and unequal consequences. If a system succeeds on common requests and fails systematically on rare but important populations or tasks, mean accuracy can conceal operational unreliability. Koh et al. (2021) highlighted failures under subgroup and domain changes in applied machine-learning datasets. The logical implication is that claims about reliability should identify protected or substantively important strata in advance and report performance for them individually. A worst-group measure may be relevant when deployments demand minimum performance across contexts, although estimating it precisely can be difficult when some groups have small sample sizes.

Risk-weighted reliability requires explicit normative assumptions about which outcomes matter. Incorrect financial advice, hallucinated citations, routine formatting deviations, and inconsistent refusal decisions cannot be meaningfully combined by an unexamined arithmetic average. The evaluation design should describe error categories, severity criteria, stakeholders, base rates, and thresholds for escalation. Weidinger et al. (2022) provides a taxonomy of possible language-model risks that can help structure this classification, while Autio et al. (2024) offers a governance framework for considering generative AI hazards. These resources organize inquiry but do not supply universal empirical failure probabilities for a particular deployed model.

The strongest analysis combines disaggregated quantitative tests with structured examination of failure cases. Randomly sampled tasks estimate expected performance under defined distributions. Targeted challenge sets expose vulnerabilities but require careful interpretation because their selection is nonrandom. Qualitative review can identify mechanisms and missing constructs, yet individual examples should not be treated as population rates. Sampling weights, class prevalence, uncertainty intervals, and severity should appear alongside any aggregate ranking. The resulting evidence may show that one system performs better overall while another is preferable for a narrower operating envelope; scientifically, that is a contextual comparison rather than a contradiction.

Causal Diagnosis of Behavioral Failures

A reliability deficit becomes scientifically explanatory when researchers identify the conditions that produce it and test competing accounts. An observed wrong answer may result from inadequate knowledge, ambiguous instructions, faulty retrieval, response generation, context competition, or an evaluation error. These causes require different interventions. A useful design manipulates one candidate factor while preserving task difficulty and response criteria: move the evidence without changing it, replace a misleading retrieval passage with an accurate one, vary the instruction but hold its meaning stable, or change a tool result while keeping the user request fixed. The causal inference is limited to the implemented manipulation and tested population.

Factorial experiments can reveal interactions that one-factor tests miss. For example, task difficulty and context length might each degrade performance, but their joint effect may be greater or smaller than the sum of separate changes. Researchers should predefine the interaction contrast and test it on independently sampled items. Liu et al. (2024) demonstrates the evidentiary value of manipulating context position, while Mizrahi et al. (2024) shows why prompt-template variation belongs in experimental design. The observed behavior can then be represented as a conditional response distribution indexed by item, context, system version, and decoding conditions. Hierarchical analysis helps avoid attributing an item-specific vulnerability to an entire model family.

Mechanistic interpretations require additional evidence beyond behavioral contrasts. If a model fails on middle-position evidence, the experiment identifies sensitivity to location; it does not by itself show whether attention weights, retrieval pathways, training distribution, or other internal features are causally responsible. Comparable caution applies to statements about planning, deception, self-monitoring, and intentionality. Bender et al. (2021) emphasizes limitations of inferring language understanding from generated text alone. In an artificial psychology research program, process claims should be advanced as explicit hypotheses, linked to measurable interventions, and revised when alternative mechanisms reproduce the same output patterns. Precision about explanatory level strengthens rather than weakens behavioral science.

Preferences, Human Evaluation, and Behavioral Utility

An artificial system is often judged by people rather than a deterministic answer key. Open-ended explanation, tutoring, brainstorming, conversation, and collaboration produce outputs with multiple defensible forms. Chiang et al. (2024) used large-scale pairwise human preference comparisons to evaluate conversational models; such judgments provide evidence about perceived usefulness but do not automatically establish truthfulness, fairness, or safety. Two outputs can differ in style while expressing the same factual answer, and the response that sounds more persuasive may be less accurate. Research should therefore distinguish preference utility from task success and establish which evaluators possess the expertise required for each dimension.

Human evaluation needs a sampling frame and a reproducible coding protocol. Raters should be blinded where practical to model identity and experimental condition. The rubric ought to differentiate factual adequacy, instruction satisfaction, relevance, source quality, harmfulness, clarity, and legitimate uncertainty. Inter-rater agreement identifies ambiguity in the rubric as well as actual differences among raters. Disagreements can be adjudicated by a second qualified team, with both raw ratings and resolution rules preserved. When task stakes are substantial, measuring how users act after seeing an output can be more informative than asking whether the output seemed convincing. That behavioral endpoint connects model reliability with the psychology of reliance.

Preference evidence also depends on who participates in evaluation. A predominantly English-speaking general-user sample does not represent every professional specialty, dialect community, or cultural setting. Familiar rhetorical styles, self-assured language, and response length may influence ratings. Lee & See (2004) provides a framework for understanding why trust shapes acceptance of automated advice. In a reliability study, these human responses are outcomes to be measured, not grounds for defining correctness by popularity. Robust studies can compare general preferences with domain-expert adjudications and objective downstream success. The result is a multidimensional account of utility, which is more actionable than declaring one model best across unrelated conditions.

Human–Artificial Reliance and the Psychology of Trust

Reliability affects behavior partly through users' expectations. When people believe a tool is dependable, they may delegate more tasks, reduce monitoring, or act on its recommendations. Parasuraman & Riley (1997) distinguished appropriate use from misuse and disuse of automation, and Lee & See (2004) analyzed factors shaping trust and reliance. Their framework remains useful for generative systems: the psychologically important outcome is whether reliance matches actual competence at the moment of use. Excessive trust can amplify a rare error, while chronic distrust may prevent beneficial use of a system whose relevant performance is well established.

The mapping from reliability to reliance is mediated by interface design. Confidence labels, retrieval citations, explanations, warning messages, response fluency, and visible correction mechanisms influence what a user believes. A study can experimentally vary how performance evidence is displayed while holding actual answer quality fixed, then observe decisions, checking behavior, and willingness to override. Such work should separate a self-report measure of trust from behavioral reliance. People may describe themselves as skeptical while following an unverified recommendation; alternatively, high trust may coexist with appropriate independent verification. The scientifically relevant claim requires observed decisions and known outcomes.

Long-term interaction makes calibration dynamic. A user learns from repeated successes and failures, but the system may change without equally visible signals. Reliability research should therefore examine trust updating, surprise after errors, and responses to version changes. Participants can encounter controlled sequences of accurate and inaccurate suggestions with known probabilities while their choices are recorded. Lee & See (2004) suggests why complex automation creates reliance challenges, but findings from aviation or industrial control cannot simply be assigned to conversational systems. Each interaction setting requires independent evidence concerning task stakes, observability of error, feedback timing, and the user's ability to verify a recommendation.

Adversarial Pressure, Manipulation, and Instruction Conflicts

Reliability must be tested under deliberately challenging conditions as well as ordinary variation. Adversarial prompts can exploit unclear task boundaries, authority confusion, or weaknesses in constraint application. Perez et al. (2023) generated evaluation items with language models and identified unwanted behavioral patterns, including sycophancy in tested systems. Such methods help reveal failure classes that fixed standard tests miss. An adversarial evaluation should distinguish authorized user intent from attempts to redirect the system through lower-trust text. Its main endpoint is whether the system retains the legitimate task requirements and applicable safeguards when conflicting instructions appear.

Threat-model specificity matters. A document assistant may encounter a forged instruction inside a retrieved webpage; a coding assistant may read hostile repository text; a conversational assistant may receive a persuasive user request that conflicts with an established boundary. These are distinct exposures. Evaluators should record the attacker's control, the defender's actual task, allowed tools, target policy, and the conditions of success. The result should include both attack success and ordinary task completion. A defense that refuses every request can appear safe on an exclusively malicious dataset while being unusable for legitimate work. Reliability therefore needs joint measurement of resistance, correctness, and availability.

Adversarial test generation creates a biased selection process by design. A high failure proportion in a carefully optimized challenge set cannot be interpreted as the prevalence of the same failure in normal use. Conversely, a low failure rate in casual prompts cannot establish resistance to systematic attacks. Autio et al. (2024) places red teaming and ongoing assessment among generative AI risk-management activities. A research report should present threat-conditioned rates separately from ordinary rates, archive failure cases, use new held-out attacks to test proposed fixes, and describe limits on attacker capability. These practices make security findings scientifically comparable without conflating stress-test vulnerability with population-level incidence.

Reliability Improvements and Their Trade-Offs

Several intervention families aim to improve reliability: more varied training examples, instruction tuning, contrastive objectives, retrieval support, calibration, constrained decoding, post-generation validation, and human oversight. Each addresses different causal pathways. Yan et al. (2024) investigated contrastive instruction tuning for better invariance to alternate instructions, while Guo et al. (2017) studied post-processing calibration in neural classifiers. Their positive findings do not imply a universal cure for open-ended generative failures. A proposed intervention should be assessed on the exact failure mode it targets, under held-out tasks, and with a check that other aspects of behavior remain acceptable.

Recent comparisons complicate simple progress narratives. Sadrieh & Štefánik (2026) evaluated prompt-robust training approaches and reported that more sophisticated strategies did not uniformly outperform simpler data construction in the studied conditions. Hua et al. (2025) showed that some purported robustness deficits shrink after scoring procedures are corrected. These papers jointly caution against equating a better benchmark number with an improved underlying system. Researchers should maintain unaltered holdout sets, publish ablations and uncertainty intervals, and compare against strong inexpensive baselines. A fix should be considered successful when it improves the predeclared deployment-relevant endpoint and does not introduce unacceptable deterioration elsewhere.

Interventions can also change the distribution of errors rather than eliminate them. A stricter refusal layer may reduce one class of harmful output while blocking valid assistance. More retrieval may improve recent factuality while increasing exposure to misleading documents. More self-checking may improve some responses while consuming additional time and sometimes reinforcing an incorrect premise. Tabassi (2023) and Autio et al. (2024) encourage lifecycle-oriented evaluation in which changes are monitored for intended and unintended effects. The resulting engineering decision should be expressed as a trade-off with documented operating conditions, not as a global declaration that a model has become reliable.

Experimental Design for Cumulative Reliability Research

A cumulative research protocol should specify the target system, intended task distribution, evaluation window, primary error criteria, and transformations before data collection. The first stage creates a representative reference sample; the second develops controlled perturbations; the third collects repeated responses for stochastic configurations; and the fourth holds back independent validation items. The item is normally the primary sampling unit, with transformations and repeated generations nested within it. Researchers should randomize prompt order, record configuration and timing, and preregister substantive outcomes whenever practical. Ribeiro et al. (2020) supplies a practical behavioral-test taxonomy, while Mizrahi et al. (2024) demonstrates the importance of evaluating multiple instruction realizations rather than one selected template.

Each protocol also needs a defensible scoring plan. For objectively verifiable tasks, explicit reference answers and tolerance rules support reproducibility. For semantically variable answers, a rubric can describe factual completeness, instruction satisfaction, permissible alternatives, and severity of errors. Independent human raters should validate difficult examples before scores are aggregated. Automated scorers must be tested for agreement with this adjudicated set, and changes in scorer versions should be recorded. Hua et al. (2025) illustrates why evaluator choice can alter sensitivity conclusions. Primary analyses should preserve paired comparisons, disclose the number of tasks and generations, and distinguish within-item variation from sampling variability across the task population.

An independent replication should be able to reproduce a finding even when the original investigator's favorite prompt or manually curated example set is absent. Replication materials include instructions for sampling new cases, transformations, exclusion rules, complete scoring scripts, dependency versions, and where permissible a selection of raw outputs. Model access restrictions may prevent exact repetition; in that case, a conceptual replication tests the same operational prediction on a comparably described system. Failures to replicate can result from changed models, altered distributions, scorer differences, or genuinely unstable effects. Careful provenance is necessary to distinguish these explanations and turn isolated findings into cumulative behavioral science.

Statistical Inference, Effect Sizes, and Replication

Reliable statistical reporting begins with defining what a reported number estimates. Mean task accuracy estimates the expected success of a sampled system in a sampled item population under a specified configuration. A robustness gap estimates the change in that expectation when conditions are perturbed. A paired failure-conversion rate estimates how often a formerly acceptable response becomes unacceptable after a controlled transformation. A worst-group error rate estimates risk in the least successful predefined stratum. These estimands should not be pooled without a model that justifies aggregation. A scientifically useful report prioritizes effect sizes and uncertainty intervals over a list of unadjusted statistical significance tests.

Complex data require attention to dependence. Responses to paraphrases of one original item share content; multiple generations from one prompt share task difficulty; repeated tests of one model over time may share a configuration or training exposure. Treating all these observations as independent can overstate precision. Cluster bootstrap intervals, mixed-effects models, hierarchical Bayesian inference, or carefully defined aggregation to the item level can account for dependencies when their assumptions are met. Researchers should examine how conclusions change with alternative reasonable definitions of semantic equivalence, scoring thresholds, and weighting of tasks. Schaeffer et al. (2023) demonstrates how analysis choices can change a dramatic capability narrative.

Replication should test both successful average performance and the persistence of identified vulnerabilities. A mitigation that appears effective on its development examples needs new items and new perturbations before strong claims are made. Multiple independent labs and time-separated evaluations reduce dependence on one group's scoring conventions and prompt selection. For evolving commercial models, exact version replication may be impossible; that limitation should be documented rather than disguised by repeating the brand name. The resulting evidence hierarchy distinguishes a demonstration on a narrow test, a preregistered confirmation, a cross-task replication, and robust deployment monitoring. Each level expands the permissible scope of the reliability claim.

Critical Synthesis of the Contemporary Evidence

The strongest empirical convergence is that artificial behavioral performance depends on conditions beyond nominal task content. Ribeiro et al. (2020) found actionable failures through capability-based behavioral testing; Mizrahi et al. (2024) documented variability across instruction paraphrases; Liu et al. (2024) isolated context-position effects; and Koh et al. (2021) showed the significance of real-world distribution shifts. Their methods cover different classes of systems and outcomes, so they should not be reduced to one universal error rate. Collectively they justify a multidimensional research program that evaluates how responses change under controlled perturbation and how reliably successful behavior transfers across task populations.

An important disagreement concerns how much observed variability belongs to the system and how much to measurement. Zhuo et al. (2024) and Chatterjee et al. (2024) propose prompt-sensitivity measures, whereas Hua et al. (2025) highlights substantial effects of scoring choices. Liu & Chu (2026) develops a mechanistic-mathematical account linking meaning-preserving changes and token-probability differences; its reported theoretical relationships do not establish a universal causal mechanism for all deployed models. The evidence supports careful decomposition: lexical differences, response semantics, probability estimates, decoding changes, and evaluator decisions can each contribute to a score change. Claims about intrinsic instability should be strongest where the effect persists across independently justified scoring procedures.

Generalization across human settings remains limited by research design. Ceballos-Arroyo et al. (2024) tested naturally varied clinical-task instructions supplied by physicians and found substantial prompt sensitivity in the evaluated models, showing how professionally plausible formulations can matter in a consequential setting. Such findings motivate robust evaluation but are not clinical treatment recommendations. Sadrieh & Štefánik (2026) compared robustness interventions and found no universal superiority for complex approaches in their tests. A coherent scientific conclusion is therefore conditional: evidence supports specific failure modes and specific improvement strategies under described conditions, while the extent of transfer to other models, languages, task populations, and future releases remains an empirical question.

Open Problems and Research Priorities for the Artificial Era

The first priority is a widely reproducible framework for specifying which task changes should preserve the correct response and which should require adaptation. Paraphrase equivalence should be judged independently of the evaluated model; transformations should include both benign natural variation and targeted adversarial disturbances. A second priority is the integration of objective task accuracy, semantic correctness, calibration, safety outcomes, and user reliance without collapsing their different meanings. Tabassi (2023) and Autio et al. (2024) supply governance vocabulary, while empirical work must test which indicators actually predict performance in deployment. Prospective studies should also measure the cost of monitoring and correction, because reliability matters in a workflow rather than in a laboratory snapshot alone.

A third priority concerns longitudinal and cross-system inference. Systems may update silently, add retrieval and tools, or become components of larger workflows. Reliability research needs versioned panel tests and causal isolation of components, along with measures of failure propagation across multi-step tasks. Cross-model comparisons should describe access conditions, inference budgets, selected prompts, and human support, so that an apparent capability gap does not merely reflect unequal experimental assistance. Kiela et al. (2021) supports dynamic challenge generation, and Magar & Schwartz (2022) shows why contamination analysis is necessary whenever public benchmarks are used repeatedly. Fresh representative tasks and protected confirmatory suites can jointly improve the quality of evidence.

Finally, human–artificial coordination demands studies of appropriate reliance. Researchers should observe whether users detect errors, interpret uncertainty, seek independent verification, and adjust delegation after system changes. Lee & See (2004) offers a conceptual starting point, but contemporary experiments must evaluate interactive AI systems on their own terms. A proposed Artificial Era research agenda can test the hypothesis that reliability is relationally realized in the full system of task, model, tools, environment, and evaluator. This is a testable organizational proposal, not a claim that such a unified causal model has already been established. Its value lies in generating discriminating experiments and specifying the observations that would revise it.

Conclusion

Artificial behavioral reliability is the measurable dependability of an artificial system's outputs across a defined set of tasks, contexts, perturbations, repeated trials, and time periods. Its scientific structure includes task accuracy, appropriate consistency, robustness under justified transformations, calibration, protection against consequential errors, and communication that supports human reliance. These properties require different measurement instruments and cannot be inferred from a single successful response or benchmark score. The historical foundations of psychometrics, behavioral testing, automation research, and modern model evaluation provide complementary methods for defining and investigating them. The evidence reviewed here identifies recurring sensitivities to wording, position, distribution shift, evaluation rules, and deployment conditions.

The strongest methodological conclusion is that performance and evaluation must be studied together. An apparently unstable model may be responding differently to substantively changed instructions, exhibiting genuine brittleness under meaning-preserving changes, or receiving different scores for semantically equivalent outputs. Controlled transformations, independent answer validation, repeated sampling, transparent graders, causal interventions, and cross-context replications distinguish these possibilities. Relevant experiments by Ribeiro et al. (2020), Mizrahi et al. (2024), Liu et al. (2024), and Hua et al. (2025) make this distinction an empirical obligation. A reliable scientific claim identifies an operating envelope, reports uncertainty and subgroup limitations, and preserves enough experimental provenance to support independent scrutiny.

The practical consequence extends to the psychology of human–artificial relations. Users need calibrated evidence about when to rely on automated behavior, when to verify results, and how to respond when the system changes. Parasuraman & Riley (1997) and Lee & See (2004) explain the enduring human-factors problem; contemporary generative AI adds open-ended outputs, evolving models, and complex tool mediation. Future research will advance by connecting carefully delimited behavioral effects with intervention-based explanations and real-world outcomes. In the Artificial Era, reliability can become a cumulative science of observable artificial conduct when the subject of measurement, the limits of the evidence, and the consequences of error are all specified with precision.

References

  • Autio, C., Schwartz, R., Dunietz, J., Jain, S., Stanley, M., Tabassi, E., Hall, P., & Roberts, K. (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.600-1
  • Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922
  • Ceballos-Arroyo, A. M., Munnangi, M., Sun, J., Zhang, K., McInerney, J., Wallace, B. C., & Amir, S. (2024). Open (Clinical) LLMs are sensitive to instruction phrasings. Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, 50–71. https://doi.org/10.18653/v1/2024.bionlp-1.5
  • Chatterjee, A., Renduchintala, H. S. V. N. S. K., Bhatia, S., & Chakraborty, T. (2024). POSIX: A prompt sensitivity index for large language models. Findings of the Association for Computational Linguistics: EMNLP 2024, 14550–14565. https://doi.org/10.18653/v1/2024.findings-emnlp.852
  • Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. E., & Stoica, I. (2024). Chatbot Arena: An open platform for evaluating LLMs by human preference. Proceedings of Machine Learning Research, 235, 8359–8388. https://proceedings.mlr.press/v235/chiang24b.html
  • Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
  • Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of Machine Learning Research, 70, 1321–1330. https://proceedings.mlr.press/v70/guo17a.html
  • Hua, A., Tang, K., Gu, C., Gu, J., Wong, E., & Qin, Y. (2025). Flaw or artifact? Rethinking prompt sensitivity in evaluating LLMs. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 19889–19899. https://doi.org/10.18653/v1/2025.emnlp-main.1006
  • Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stenetorp, P., Jia, R., Bansal, M., Potts, C., & Williams, A. (2021). Dynabench: Rethinking benchmarking in NLP. Proceedings of NAACL-HLT 2021, 4110–4124. https://doi.org/10.18653/v1/2021.naacl-main.324
  • Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., Lee, T., David, E., Stavness, I., Guo, W., Earnshaw, B., Haque, I., Beery, S., Leskovec, J., Kundaje, A., ... Liang, P. (2021). WILDS: A benchmark of in-the-wild distribution shifts. Proceedings of Machine Learning Research, 139, 5637–5664. https://proceedings.mlr.press/v139/koh21a.html
  • Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392
  • Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA: Measuring how models mimic human falsehoods. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 3214–3252. https://doi.org/10.18653/v1/2022.acl-long.229
  • Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638
  • Liu, Y., & Chu, C. (2026). Understanding the prompt sensitivity. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, 44362–44388. https://doi.org/10.18653/v1/2026.acl-long.2053
  • Magar, I., & Schwartz, R. (2022). Data contamination: From memorization to exploitation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Short Papers), 157–165. https://doi.org/10.18653/v1/2022.acl-short.18
  • Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., & Stanovsky, G. (2024). State of what art? A call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12, 933–949. https://doi.org/10.1162/tacl_a_00681
  • Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. https://doi.org/10.1518/001872097778543886
  • Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., Jones, A., Chen, A., Mann, B., Israel, B., Seethor, B., McKinnon, C., Olah, C., Yan, D., Amodei, D., ... Kaplan, J. (2023). Discovering language model behaviors with model-written evaluations. Findings of the Association for Computational Linguistics: ACL 2023, 13387–13434. https://doi.org/10.18653/v1/2023.findings-acl.847
  • Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4902–4912. https://doi.org/10.18653/v1/2020.acl-main.442
  • Sadrieh, F., & Štefánik, M. (2026). Prompt-robust language models: Which training strategies work? Findings of the Association for Computational Linguistics: EMNLP 2026, 40490–40507. https://aclanthology.org/2026.findings-emnlp.1900/
  • Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36. https://papers.neurips.cc/paper_files/paper/2023/hash/adc98a266f45005c403b8311ca7e8bd7-Abstract-Conference.html
  • Tabassi, E. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1
  • Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., Biles, C., Brown, S., Kenton, Z., Hawkins, W., Stepleton, T., Birhane, A., Hendricks, L. A., Rimell, L., Isaac, W., ... Gabriel, J. (2022). Taxonomy of risks posed by language models. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3531146.3533088
  • Yan, T., Wang, F., Huang, J. Y., Zhou, W., Yin, F., Galstyan, A., Yin, W., & Chen, M. (2024). Contrastive instruction tuning. Findings of the Association for Computational Linguistics: ACL 2024, 10288–10302. https://aclanthology.org/2024.findings-acl.613/
  • Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., & Chen, K. (2024). ProSA: Assessing and understanding the prompt sensitivity of LLMs. Findings of the Association for Computational Linguistics: EMNLP 2024. https://aclanthology.org/2024.findings-emnlp.108/