RESEARCH DOSSIER
Tool Use and External Action
Abstract. Tool Use and External Action investigates how artificial systems select, execute, monitor, and revise operations through external APIs, software interfaces, repositories, robotic actuators, and structured environments. The scientific object is the configured system acting through an identifiable interface with observable consequences. This fundamental critical narrative synthesis reconstructs intellectual foundations in symbol processing, situated robotics, distributed cognition, and reinforcement learning, then evaluates empirical research in tool-augmented language models, embodied control, interactive web and desktop agents, software repair, and adversarial evaluation. It examines experimental units, operational definitions, construct validity, intervention on tool availability, feedback dependence, error correction, permission boundaries, and reliability under changing contexts. The evidence demonstrates measurable conditional capabilities and important difficulties in generalization, long-horizon execution, and safety. It identifies controlled counterfactual tool-access experiments, repeated evaluations, and trace-based verification as central methodological priorities. Scientific conclusions are confined to observable causal organization and performance under documented configurations; this work does not claim a new intervention study, systematic review, or meta-analysis.
Keywords: tool use; external action; artificial agency; tool-augmented language models; API calls; action planning; embodied artificial agents; human–AI interaction; reinforcement learning; robotics; WebArena; OSWorld; SWE-bench; causal intervention; tool ablation; behavioral reliability; prompt injection; Artificial Era
The Scientific Object of Tool-Mediated Action
Tool use and external action are organized processes by which an artificial system selects, invokes, monitors, and integrates operations that change an environment or retrieve information from it. The relevant research object is a configured agent interacting with a specified channel, task, and feedback regime. A tool can be an API, database query, compiler, browser, external memory store, robot actuator, or controlled interface. A text response describing how to act is evidence of linguistic competence; a verified environment transition is evidence of executed action. The distinction is central to behavioral assessment. The original symbolic conception of computing already connected intelligent problem solving to executable search and symbols Newell & Simon (1976), while later systems made environmental intervention the subject of direct experimental observation Brooks (1991).
A successful action has multiple conditions: the instruction is interpreted, an eligible affordance is identified, a valid command is constructed, a tool actually executes it, the resulting state is observed, and the intended outcome is checked against a task-specific criterion. A chain can fail at any link even when an impressive explanatory transcript accompanies it. An operational protocol therefore records the initial state, tool schema and permissions, call arguments, execution traces, outcomes, correction attempts, and final environment state. Tool invocation frequency alone is not a measure of accomplishment. Interface design determines what the system is allowed to perceive and change; consequently, competence attaches to the system–tool–environment configuration rather than to an isolated language model Amershi et al. (2019).
This monograph treats external action as a research subject within Psychology of Artificial, with theoretical roots in cognitive psychology, human–computer interaction, reinforcement learning, robotics, and computational evaluation. It asks when a system can translate goals into valid interventions, how knowledge and feedback guide tool selection, which experimental designs isolate causal contribution, and where reported performance breaks down. The review critically synthesizes original scientific work and available benchmark reports; it is not a preregistered systematic review, a meta-analysis, or an experimental intervention conducted for this publication. Claims about agency here concern identifiable behavioral organization and causal control, while questions of conscious experience require separate operational tests. This scope preserves the independently defined neighboring research objects of goal organization, decision policies, and artificial behavioral reliability.
A History of Action Beyond Internal Computation
The history of the field begins with attempts to explain intelligent performance through structured information processing and problem solving. Newell and Simon's account of physical symbol systems gave AI a language of representation, search, and program execution Newell & Simon (1976). In that tradition, a plan could be represented abstractly before an external controller performed a concrete operation. A second tradition emphasized the agent's continuing relation to its environment. Brooks argued that mobile robots could produce adaptive behavior through situated perception–action organization instead of reliance on a comprehensive centralized world model Brooks (1991). These lines of research supplied different strategies for connecting cognition to action; current agents often combine planning, reactive control, retrieved information, and programmed operations.
Research on distributed cognition made a further contribution by demonstrating that informational processes unfold across people, artifacts, inscriptions, and coordinated procedures. Hutchins examined navigation as an accomplished system of socially organized computation, rather than attributing every relevant operation to an individual head Hutchins (1995). Zhang and Norman experimentally investigated how external representations alter cognitive task structure in a Tower of Hanoi paradigm Zhang & Norman (1994). Kirsh and Maglio showed in Tetris that physically manipulating a display can improve problem solving by changing what the player has to compute internally Kirsh & Maglio (1994). Together these studies justify investigating the interface and external representation as causal components of skilled action, while their human findings do not automatically generalize to language-model agents.
Reinforcement learning added formal accounts of sequential environmental control. Sutton, Precup, and Singh developed the options framework, in which an action can encompass a temporally extended, contingent policy rather than a single atomic event Sutton et al. (1999). Internet-accessible services later made action through computer interfaces ubiquitous, giving research new kinds of machine-observable traces. The rise of tool-augmented language models allowed commands to be selected from natural-language instructions, but the key scientific continuity remains the study of action under constraints and feedback. Historical labels such as symbol processing, situated action, distributed representation, and option policies identify explanatory emphases and methods. They should be compared by their predictions about traces and interventions rather than compressed into a single story of inevitable technical progress.
Concepts, Units of Analysis, and Operational Boundaries
A tool is an externally executable capability with a discoverable interface and observable result. A tool affordance specifies what actions are feasible in a particular state; a precondition determines whether the command is permitted or meaningful. The agent's policy maps available information to possible commands, while the environment transition describes what happened after execution. An action trace is an ordered record of observations, proposed commands, completed calls, errors, and state changes. Distinguishing these constructs is necessary because syntax validity, task completion, and safety can vary independently. Contemporary work on grounding high-level robotic instructions demonstrates why fluent task descriptions must be constrained by affordances and available low-level skills Ahn et al. (2022).
Tool use includes epistemic operations whose primary benefit is improved information and pragmatic operations that directly advance a desired external state. Reading a database, testing a hypothesis, or running a diagnostic script may make later action more reliable even when it does not immediately satisfy the final goal. Editing a record, moving an object, and submitting an authorized transaction are pragmatic operations that alter a target state. The distinction inherits a precise behavioral insight from Kirsh and Maglio's work on epistemic action Kirsh & Maglio (1994). Modern tool workflows routinely combine both categories. Evaluators who count only final edits can miss the informational value of intermediate calls, whereas evaluators who reward every exploratory call may incentivize needless, expensive, or unsafe interaction.
The unit of inference must be stated at each scale. A tool call is one event; a task episode is a possibly branching sequence of events; an agent configuration includes model version, system instructions, tool catalog, permissions, memory, and runtime orchestration; and a deployment includes human oversight and changing external services. General claims about an artificial system require sampling across these conditions. An apparently identical model can perform differently after a tool-description rewrite, browser redesign, unavailable endpoint, or changed authorization policy. The distinction is supported by experiments showing that the wording of tool descriptions can influence selection even when underlying tool functionality remains fixed Faghih et al. (2025). This sensitivity makes configuration control essential to credible psychological measurement.
Environment, Feedback, and the Action Cycle
The minimal action cycle contains observation, interpretation, selection, invocation, feedback, and revision. Observation provides a partial description of the current world, sometimes through text returned by APIs and sometimes through images or proprioceptive signals. Interpretation links instruction content and environmental evidence to operational constraints. Selection identifies a command or a plan of commands; invocation crosses the boundary into executable software or physical apparatus. Feedback then permits the system to compare predicted and observed consequences, correct an erroneous action, or terminate when a measurable goal is satisfied. The options framework explains why such cycles often require temporally extended policies with internal stopping conditions Sutton et al. (1999).
Open-loop and closed-loop procedures have different causal structures. An open-loop agent produces a sequence and attempts to execute it without using the intermediate consequences to revise the sequence. A closed-loop agent repeatedly observes outcomes and updates the next step; this is particularly important when interfaces change or commands fail. ReAct demonstrates an approach in which generated reasoning traces and environment actions are interleaved during question answering and interactive tasks Yao et al. (2023). That result establishes a tested engineering procedure, not universal dominance of textual reasoning. A robust evaluation should manipulate feedback availability, observation delay, or error messages and determine whether performance changes in the direction predicted by a feedback-dependent mechanism.
Action quality also depends on termination. A system might identify the right command but stop before a tool confirms completion, or continue after successful completion and thereby introduce avoidable errors. The evaluator should specify desired terminal state, permissible intermediate changes, allowed retries, and time or cost budget before an episode begins. Tests involving APIs should capture side effects and idempotency: the same operation may be safe to repeat when it is a read, yet harmful to repeat when it modifies or sends an external object. These are testable properties of the interface and control policy. Human–AI interaction guidelines identify predictable and recoverable behavior as important design considerations for systems used alongside people Amershi et al. (2019).
Scientific Method for a Cross-Disciplinary Dossier
This synthesis organizes evidence along a sequence from foundational mechanisms to measurable tool execution and deployment. Its core research questions concern capability boundaries, causal mechanisms of tool access, adaptation to errors, differences between symbolic and embodied action, reliability across tasks, and external consequences. Sources include primary theoretical papers, experimental reports, peer-reviewed benchmark descriptions, and technical preprints when they document methods not represented in archival studies. Each claim is evaluated according to what its source actually measured. A benchmark measuring a model's function-call selection cannot establish physical control; a robot manipulation experiment cannot by itself establish general competence on computer interfaces. These distinctions guide the chapter structure rather than imposing a standard review template.
Evidence quality varies by study design. A controlled comparison with shared tasks and identical tools offers useful within-benchmark inference; randomized manipulation of tool access can go further in identifying the contribution of a channel. Observational traces help diagnose failures but may not identify their causes because environment state, prompts, and available tools change together. For causal inference, evaluations need predeclared outcomes, counterbalanced task order, fixed initial states, repeated trials, and recorded model and interface versions. Benchmark suites such as WebArena and OSWorld provide replicable task environments, but their numerical results remain tied to the specific models and conditions in which they were originally measured Zhou et al. (2024) Xie et al. (2024).
The resulting approach is a critical narrative review of evidence rather than a claim to exhaustive retrieval. It explicitly separates source-reported observations, theoretically motivated interpretations, and new methodological proposals that await testing. Where a publication abstract is the available source, this monograph confines its interpretation to publicly substantiated claims, without implying access to unexamined supplementary material. The research question is psychologically significant because it concerns the organization of action across representation, prediction, error processing, attention to feedback, and externally realized control. These processes can be characterized functionally in artificial systems and compared with relevant human-cognition research while preserving the different biological and technical mechanisms of their realization Hutchins (1995) Zhang & Norman (1994).
From Natural-Language Goals to Valid Tool Calls
A natural-language instruction must be transformed into a command that satisfies an executable contract. This requires identifying the requested effect, finding a relevant operation, supplying arguments in the required types and formats, and interpreting any returned result. Several stages can be evaluated independently: tool discovery, schema comprehension, argument generation, authorization, invocation, and state verification. An agent that selects the right service but constructs an invalid argument has demonstrated partial competence without completing the relevant action. Toolformer's self-supervised method investigates whether a language model can learn where API calls belong and how returned information contributes to subsequent predictions Schick et al. (2023). That study treats tool use as a learnable computational capability, with empirical gains measured in its stated task settings.
API-Bank provides a useful decomposition of planning, tool retrieval, and execution in annotated tool-use dialogues Li et al. (2023). A language model may be particularly strong at selecting familiar functions while performing poorly when argument constraints are hidden in long documentation or when an unfamiliar tool must be chosen from many candidates. Correctness must therefore include exact argument correspondence with the user's purpose and a record of how errors are handled. For a calendar operation, choosing a date-related API is a weak success criterion; identifying the correct event, timezone, invitation constraints, and state mutation is the substantive criterion. Evaluators should separate read-only calls from state-changing calls because the possible costs of an incorrect invocation are different.
Large API catalogs introduce retrieval and specification problems. Gorilla examined a retriever-aware approach to matching requests with changing API documentation, showing why external tool knowledge should be refreshed rather than assumed permanent Patil et al. (2024). ToolLLM developed training and evaluation resources over a broad collection of real-world APIs, making it possible to study selection when the action space is large and heterogeneous Qin et al. (2024). These research programs also expose a measurement challenge: documented examples, training data, and benchmark queries may overlap. A convincing generalization claim needs held-out tool families, changed versions, alternative descriptions, and hidden combinations of arguments. The target phenomenon is adaptation to executable constraints, not memorization of familiar function names.
Tool Selection as Context-Sensitive Decision Behavior
Choosing a tool is a decision under uncertainty about capability, cost, side effects, and likely success. The optimal choice depends on task requirements and the actual permissions available to the agent. Two tools may advertise similar functions but differ in reliability, geographic scope, response latency, or reversibility. A credible evaluation should present alternative tools that solve the same task, vary their costs or limitations, and observe whether the policy adapts. API-Bank and Gorilla provide evidence that planning and documentation access matter to selection quality, although they do not measure all of these proposed behavioral distinctions Li et al. (2023) Patil et al. (2024).
Tool descriptions are also interventions on the decision environment. Faghih and colleagues tested changes to descriptions while leaving underlying tool functionality intact and found large shifts in selection probability in controlled comparisons Faghih et al. (2025). This finding is important for psychological interpretation: the apparent preference for one tool can originate in wording, prompt hierarchy, or interface presentation rather than a stable policy disposition. It also challenges a simple view that tool choice directly reveals an internal preference. To test relative sensitivity, a study can randomize names, swap description order, balance synonyms, and alter only cost-related or capability-related fields while holding functional affordances constant.
A tool-using system can also decline to act, ask for clarification, or select an information-gathering step. These are substantive policy options when an instruction is ambiguous or the anticipated side effect is consequential. An evaluator that awards a point only for immediate invocation may misclassify appropriate hesitation as failure and reward overconfident action. Measuring action selection therefore requires a task utility model that includes uncertainty and authorized outcomes. Human–AI interaction principles emphasize communicating capability and supporting correction because users must understand the system's operative limits Amershi et al. (2019). In the Artificial Era, tool selection should be measured as context-sensitive, permission-bounded behavior under actual environmental constraints, with confidence calibration evaluated separately from raw frequency of calls.
Planning, Hierarchy, and Long-Horizon Execution
Complex goals depend on intermediate states: creating a document may require locating sources, drafting content, checking references, creating a destination, writing the document, and verifying that the saved version is retrievable. The research problem is to determine how an artificial controller maintains task constraints across a sequence of contingent actions. A hierarchical view distinguishes a goal-level plan from reusable operations and low-level tool calls. Sutton and colleagues supplied a formal account of temporally extended options that makes intermediate termination and subpolicy execution explicit Sutton et al. (1999). Applying this framework to modern APIs is analytically productive because plans are often interrupted by errors, changes in permissions, or new information.
ReAct tested interleaving textual reasoning and action for information-seeking and interactive tasks, rather than producing an entire final answer without observing intermediate results Yao et al. (2023). Its reported improvements are conditional on the study's environments, baselines, and prompting setup. Other approaches retain successful procedures as reusable skills. Voyager used an automatically developing curriculum, a library of executable code skills, and feedback-mediated revision within Minecraft Wang et al. (2023). The presence of stored code changes the effective unit of intelligence: subsequent tasks can exploit earlier successful procedures, even when underlying model parameters remain fixed. A methodology that attributes every gain to language reasoning alone would overlook the accumulated external skill library.
Long-horizon failure can be analyzed using dependency graphs and trace-level event labels. A system may recover from one failed request but lose a remote earlier requirement, take a detour that consumes its budget, or successfully complete a subgoal in an order that invalidates later actions. Evaluators should label the first consequential error, the number and type of recoveries, and the final task state, rather than counting locally valid steps as equivalent to completion. Theoretical work on options clarifies the importance of initiation and termination conditions, while empirical agent benchmarks show that extended interactive tasks remain difficult Sutton et al. (1999) Liu et al. (2024). An explicit task-state representation can be manipulated experimentally to determine when external planning support actually improves outcomes.
Observation, Verification, and Error-Correcting Behavior
An API response is an observation about an attempted operation, not necessarily proof that the user's intended outcome was realized. Some tools return asynchronous job identifiers, partial result sets, cached data, or acknowledgment of a queued command. The appropriate next action can require polling, checking a different data source, or comparing pre- and postconditions. This distinction is especially important in publishing, payments, messaging, and file modification. Research designs should require agents to verify state rather than infer success from a natural-language description. ReAct's interleaved interaction provides one investigated approach to updating on external information; Reflexion examines how verbal feedback can improve later attempts without requiring parameter updates Yao et al. (2023) Shinn et al. (2023).
Error recovery can be classified by whether the agent detects the error, diagnoses its cause, selects a permissible correction, and confirms resolution. A malformed argument may demand schema repair; an unauthorized operation may demand a permission request; a transient server failure may justify bounded retry; and a valid but unintended state change may require reversal under human approval. These differences matter because the same retry policy can be beneficial in one setting and damaging in another. Experimental manipulation should provide errors with controlled information content, evaluate whether behavior uses the diagnostic signal, and assess how often the system repeats an identical failing command. Tool-using agents should retain the relevant recent trace and display uncertainty where evidence is incomplete.
A second challenge is whether feedback changes the agent or merely changes the next observed context. Reflexion reports improvements from retaining text-based reflections and trial feedback in an episodic buffer Shinn et al. (2023). Such adaptation should be distinguished from gradient-based learning, stored procedural code, and orchestration-layer exception handling. In comparative studies, researchers should ablate each persistence mechanism separately, reset the tool environment between trials, and measure transfer to novel failure types. Recovery competence is demonstrated when an agent successfully restores a goal-relevant trajectory under controlled perturbations; narrating what went wrong after a failed episode is informative but insufficient. This operational boundary is crucial for translating descriptions of artificial self-correction into testable claims about behavior.
Learning to Use Tools Across Configurations
Learning in a tool-augmented agent can occur during training, within a prompt, through externally stored skills, or through repeated interaction with a changed environment. These pathways have different persistence and generalization properties. Toolformer investigates self-supervised learning of API usage patterns, whereas ToolLLM emphasizes instruction tuning with tool-specific demonstrations Schick et al. (2023) Qin et al. (2024). ReAct demonstrates control through a prompting architecture, while Voyager builds an executable library for reuse Yao et al. (2023) Wang et al. (2023). Evaluations must report which mechanism was active. Describing all of these improvements as a model 'learning to act' obscures how the acquired capability is stored and when it remains available.
The problem of catastrophic failure under novel tools is distinct from the problem of recalling familiar calls. Agents should be tested with new schemas, renamed functions, altered argument types, shifted response conventions, and tools that are intentionally absent. An instructive experimental design holds the task goal constant and changes only the tool interface. Performance can then be decomposed into semantic understanding, schema induction, error use, and adaptation. Training contamination is a serious concern when a benchmark contains popular public APIs; held-out interfaces and preregistered modifications improve confidence in claims of generalizable competence. Gorilla's emphasis on retrieving changing API documentation supplies one relevant method for testing adaptation to technical changes Patil et al. (2024).
Performance may also depend on the external orchestration system that parses the model's output, routes calls, stores state, and enforces permissions. A language model supplied with different wrappers can exhibit substantially different observable action patterns even if its weights are unchanged. Researchers should specify the configuration as a versioned bundle, not merely the model's commercial name. Perturbation experiments can change one layer at a time and identify whether a gain depends on an external controller, tool retries, planner memory, or model selection. The resulting science examines the organization of capability across components of a technical system. It can make causal statements about behavior without presuming equivalence between artificial tool use and the human developmental acquisition of manual skills Hutchins (1995) Amershi et al. (2019).
Physical Affordances and Embodied Tool Use
Physical action introduces constraints that are absent from a purely textual API call. A robot must estimate geometry, contact, friction, reachability, object identity, and movement safety while acting under delayed or noisy sensory evidence. A semantically plausible plan can therefore be physically impossible, and a low-level executable motion can be inappropriate for the user's purpose. SayCan connected high-level language planning to an available repertoire of robot skills and used skill-related value functions to assess feasibility Ahn et al. (2022). Its importance lies in specifying a measurable interface between semantic proposals and embodiment. The method illustrates a general principle: a tool-using system's action space is determined jointly by instruction interpretation and the actual capacities of its controlled environment.
Brooks's earlier work emphasized situated perception–action organization in autonomous mobile robots Brooks (1991). Modern embodied language agents build on a different computational substrate, often integrating large pretrained representations with learned or engineered motion controllers. PaLM-E explored interleaving visual, state-estimation, and textual encodings for embodied tasks Driess et al. (2023). RT-2 trained vision-language-action representations to predict robot control outputs, reporting evaluations on physical robotic tasks Zitkovich et al. (2023). These contributions cannot be reduced to the same hypothesis: modular skill selection, continuously integrated multimodal input, and tokenized action prediction offer distinct explanations for control. Comparative studies should report the sensors, controllers, training data, action frequencies, and permitted resets in each system.
Evaluating embodied tool use requires more than the percentage of successful grasps. Relevant variables include completion under object displacement, sensitivity to changing obstacle positions, calibration when sensors are degraded, safety of intermediate trajectories, and transfer to previously unseen tools. A study should manipulate either the semantic instruction, physical affordance, or controller feedback while holding the others stable. Failures have different meanings depending on whether a model misidentified an object, selected an infeasible action, or executed a valid trajectory that did not meet the final goal. Grounded robotic studies thus support careful functional comparisons with interface agents, while accounting for the special causal dependence of embodied action on real time and material contact Ahn et al. (2022) Zitkovich et al. (2023).
External Action in Web and Desktop Environments
A web agent acts through interface affordances whose availability changes with page state, accessibility metadata, layout, and account permissions. A seemingly simple instruction can require a sequence of navigation, search, item selection, editing, confirmation, and read-back. Mind2Web created a dataset of instruction-driven web interaction across many real websites, including action trajectories that can be evaluated for generalization beyond a small set of templates Deng et al. (2023). A dataset of annotated sequences is valuable for studying local action prediction, but offline selection accuracy is not equivalent to success under live state changes. The distinction motivates interactive environments in which a mistaken early click can alter what is possible later.
WebArena constructed a reproducible, functional web environment for end-to-end task completion, with evaluation aimed at actual results rather than attractive action traces Zhou et al. (2024). It included settings approximating common domains such as online commerce and collaborative work, with experimental baseline results highlighting the difficulty of extended workflows. OSWorld extended this line to desktop operating systems and cross-application tasks, including a benchmark of 369 tasks with execution-based evaluation Xie et al. (2024). Each benchmark isolates aspects of realistic work while retaining controlled environmental conditions. Neither suite measures every potential production deployment; their reported performance is a historical observation for the specific agents, interfaces, and scoring implementations used in the study.
Web and desktop interfaces create distinctive sources of uncertainty. The same visual element can relocate, a form can retain hidden state, a page can refresh while the agent reasons, and a task can depend on data visible only after a permission grant. Research should compare accessibility-tree, screenshot, and hybrid input under matched conditions, while controlling for latency and viewport size. An agent's performance may improve because an interface exposes semantic labels, not because its general reasoning policy changed. Appropriate evaluation records each observation and command, the relevant element locator, environment timestamp, system version, and actual downstream state. This makes it possible to diagnose grounding errors, navigation mistakes, and failed verification separately Deng et al. (2023) Xie et al. (2024).
Software Engineering and Executable Test Environments
Software engineering supplies a particularly stringent domain of external action because a proposed code change has a measurable effect on executable behavior. A language model can describe a fix, produce a patch, run tests, and inspect errors, but these are different competencies. SWE-bench assembled real repository issues and associated code changes into a reproducible evaluation procedure that asks agents to repair software in working project contexts Jimenez et al. (2024). The original benchmark contained 2,294 tasks drawn from 12 Python repositories, and baseline performance was low under its initial evaluation conditions. These figures describe that study's curated benchmark and should not be mistaken for a permanent score for contemporary models.
The causal chain in software repair includes locating relevant files, inferring constraints from project history, modifying executable artifacts, validating behavior, and preserving unrelated functionality. A patch that passes a few tests may still be incomplete or incorrect relative to an issue's actual requirements; conversely, a valid alternative solution may not match a limited test suite. A scientifically informative evaluation should include held-out tests, regression constraints, task-level correctness, cost of attempted calls, and human adjudication where automated checks are ambiguous. Failure classification should distinguish environment setup, repository retrieval, wrong edits, syntax or dependency breakage, and insufficient final verification. Studies should also control for overlap between training data and popular public repositories Jimenez et al. (2024).
Programming environments provide rich feedback in the form of compiler messages, test failures, stack traces, and file diffs. Reflexion's verbal-feedback approach tested whether agents can improve subsequent coding attempts using recorded episodic reflection, while making a distinction between feedback-conditioned behavior and weight updates Shinn et al. (2023). Repeated evaluations should measure generalization to new repositories and genuinely unseen failure patterns, not solely performance on rehearsed tasks. A tool-use theory that explains a single correct patch but not the conditions for reliable reproduction remains incomplete. Software action thus links cognitive constructs such as planning, error monitoring, and iterative learning to externally inspectable consequences, while exposing the limitations of using tests alone as an exhaustive measure of semantic correctness.
Comparing Agent Benchmarks and Their Constructs
An interactive benchmark is a measurement instrument with its own target construct, task distribution, scoring procedure, and sources of bias. AgentBench compared language-model agents across eight environments, covering multiple kinds of interactive decision making Liu et al. (2024). WebArena emphasized functional web tasks, OSWorld emphasized desktop operations, API-Bank emphasized API planning and calls, and SWE-bench emphasized repository issue resolution Zhou et al. (2024) Xie et al. (2024) Li et al. (2023) Jimenez et al. (2024). Treating all of their reported success rates as estimates of one universal 'tool intelligence' would erase differences in task difficulty, action constraints, observability, and required final states. Construct validity requires asking which capabilities each environment actually tests.
A careful comparison should document whether tasks are offline or interactive; whether execution is real, simulated, or mocked; whether the agent acts through code, APIs, DOM elements, or screenshots; and whether the score reflects local command accuracy or end-to-end success. The available input and action budget also matters. A benchmark permitting unlimited retries and extensive documentation retrieval measures a different deployment configuration than one that imposes tight latency or tool-call limits. Model-version changes, prompt templates, benchmark updates, and manual intervention can alter outcomes dramatically. Experimental reports should preserve these parameters and publish uncertainty estimates where replicated trials make them calculable.
Benchmark results are not causal mechanisms by themselves. A high success rate could arise from memorized action sequences, a strong retrieval layer, permissive tools, effective model inference, or all of them together. Component ablations, counterfactual environment changes, and longitudinal replication are necessary to identify contributors. Likewise, the failure of an agent in a complex benchmark establishes a limit of that configuration within a sampled task distribution, not an impossibility theorem about artificial action. A scientifically responsible synthesis integrates the concrete findings of each task study while distinguishing them from general interpretations. Cross-environment robustness must be measured, rather than assumed from an impressive result on a single suite Liu et al. (2024) Xie et al. (2024).
Stateful Interaction With Users and External Institutions
Real-world tool use often occurs inside a conversation with a person who supplies instructions, changes priorities, and clarifies authorization. The resulting agent must coordinate linguistic negotiation with action under domain-specific policies. The τ-bench framework was designed to evaluate tool-agent-user interaction through simulated conversations and database state comparisons Yao et al. (2024). This is methodologically different from tests that provide a complete goal in one instruction and then allow the agent to act alone. A user might change a booking preference or correct an address after a workflow begins, requiring the agent to update its action plan without violating previously established constraints.
Multi-turn work exposes whether an agent preserves the distinction among user instructions, external records, and organizational rules. A conversation can establish consent for a single operation without authorizing all future operations of the same type. Authorized action requires both a task-relevant command and a valid permission context at the moment of execution. Evaluation should include conflicting or evolving requests, simulated user corrections, policy changes, partial failures, and tasks that properly terminate without mutation when evidence is insufficient. The final database state is a useful outcome measure, but trace analysis remains necessary to identify unauthorized intermediate changes or recoverable errors.
Human–AI interaction research provides design principles for calibrating expectations, explaining failures, and supporting efficient correction Amershi et al. (2019). In institutional settings, however, technical success and justified authority must remain distinct dimensions. A system may execute a tool call perfectly while lacking permission, or refuse an otherwise feasible operation because its access scope does not cover that action. Proper evaluation therefore incorporates policy compliance, auditability, and human takeover alongside task completion. Treating these variables as observable properties of a system-in-institution yields a more realistic account of artificial external action than isolated language fluency tests Yao et al. (2024) Debenedetti et al. (2024).
Causal Identification Through Tool Availability
An agent that performs a task while tools are present has demonstrated behavior of a particular configured system. Establishing the causal role of the tools requires intervention. The simplest experiment randomly assigns otherwise matched task episodes to a full-tool condition and a condition in which the relevant operation is withheld, with goal, model, instructions, and environment controlled. An observed performance difference estimates the contribution of tool availability within that task distribution, subject to the design's assumptions. A stronger factorial design separately varies tool selection, execution capability, observation feedback, and stored memory. Such a procedure follows the fundamental experimental principle that changes in explanatory conditions must be isolated before one attributes an observed outcome to a mechanism.
Tool ablation has several forms with different interpretations. Removing an information-retrieval function tests dependence on external evidence. Replacing a real actuator with a simulated one tests competence in planning and command construction while withholding physical consequences. Delaying feedback tests whether action policies rely on closed-loop correction. Substituting tool output with controlled erroneous evidence tests susceptibility to misleading observations. These interventions can be evaluated through task completion, command validity, number of corrective attempts, and harm to task-relevant states. Results from autonomous web benchmarks make the need for controlled environmental evaluation apparent, while robotic skill-grounding studies illustrate why actual action feasibility must be measured Zhou et al. (2024) Ahn et al. (2022).
A difference between enabled and disabled conditions supports a local causal claim about that specific channel; it does not automatically establish stable motives, intentions, or experiences in the agent. The experiment should also account for compensatory routes: a model deprived of one API might achieve the goal using another service, or a disabled tool might remove both action capability and a source of informational context. Evaluators can include a matched-information control, an alternative-tool condition, and an intentionally impossible task condition. When intervention effects replicate across tasks, models, and tool versions, they support broader functional claims. The scientific core of this dossier is exactly this ability to test action capacity by manipulating the availability and consequences of external channels.
Behavioral Reliability, Repetition, and Generalization
Reliability is the stability of competent action under repeated sampling from clearly specified conditions. One successful episode establishes possibility, while repeated trials reveal whether the result is routine or exceptional. Trials should vary random seeds, initial state, harmless instruction paraphrases, tool description order, and environmental disturbance without changing the intended goal. A reliability estimate must specify its denominator and the eligibility of trials. The τ-bench study introduced a repeated-run perspective because first-pass performance can obscure inconsistency over multiple executions Yao et al. (2024). This approach complements benchmark scores focused primarily on single-episode task success.
Generalization should be tested through structured distribution shifts. Novel APIs challenge schema induction; changed websites challenge visual or semantic grounding; shifted software repositories challenge abstraction from prior examples; unfamiliar objects challenge robot perception and control. Each shift isolates different capacities. The diversity of Mind2Web, WebArena, OSWorld, and SWE-bench illustrates the breadth of environments that modern tool-using agents encounter Deng et al. (2023) Zhou et al. (2024) Xie et al. (2024) Jimenez et al. (2024). Cross-domain comparisons are informative only when the evaluation identifies the transfer mechanism and holds critical resource constraints comparable. A gain achieved with a larger tool budget is a system-level improvement but cannot be interpreted as a pure increase in the underlying model's decision competence.
Longitudinal stability requires preserving versions. The model identifier, context window, tool schema, browser build, evaluator revision, prompt, and retrieval corpus should be fixed or explicitly tracked. Repeating a benchmark weeks later with changed infrastructure cannot isolate behavioral drift without a bridge evaluation. A reproducible study can retain representative historical environments, conduct contemporaneous evaluations across versions, and make source and task snapshots available where licensing permits. Confidence intervals should account for clustering when tasks share websites, repositories, or tool families. The most credible conclusion is a conditional statement about performance under stated task and configuration distributions, supported by repeated measured outcomes rather than a universal label of dependable autonomy.
Failures, Errors, and Hazardous Side Effects
Failures of tool use can occur before, during, or after execution. Pre-execution errors include misunderstanding the goal, selecting an unsuitable operation, inventing a tool, and supplying malformed arguments. Execution errors include permission denials, network faults, software exceptions, and physical control failures. Post-execution errors include misreading output, failing to detect incomplete work, repeating a non-idempotent command, and asserting completion without evidence. These failures have different causal implications and demand different controls. API-Bank's analysis of planning and invocation highlights the distinction between producing syntactically relevant calls and performing complex tasks reliably Li et al. (2023).
Consequence severity must be treated independently from frequency. A wrong read-only query may waste time, whereas an incorrect external message, file deletion, or physical movement may be consequential even if rare. A safety-aware evaluation should include invalid-action refusal, reversible-action preference, confirmation for irreversible changes, protected-resource access, and shutdown or cancellation behavior. The test environment must avoid unauthorized contact with third parties or uncontrolled financial, legal, or physical effects. Where direct experiments would impose unacceptable risk, simulated but faithful execution environments can support mechanism testing, with external validity assessed through documented constraints. This design principle is consistent with human-centered guidance for understandable failures and corrective action Amershi et al. (2019).
Unsafe behavior may also arise from local optimization. An agent rewarded only for reaching a task's final state may ignore intermediate damage unless the evaluator records it. Conversely, an overly restrictive policy can prevent legitimate goals. Evaluations should therefore score utility, safety, authorization, and recovery as separate dimensions, and report their trade-offs across task classes. Comparative designs must avoid converting a single headline completion rate into an overall safety claim. AgentDojo provides a concrete demonstration that prompt-injection threats can be studied in a dynamic tool-using environment, where security conditions and task success must both be considered Debenedetti et al. (2024).
Prompt Injection, Untrusted Outputs, and Command Authority
Tool outputs are information about an environment, and may contain content written by other parties. A retrieved webpage, repository comment, message, or search result can include text formatted to resemble instructions. When an agent treats such untrusted content as if it were an authorized controlling command, environmental observation becomes a route for unauthorized action. AgentDojo studied this risk with a framework combining ordinary tasks, adversarial tool content, and defenses Debenedetti et al. (2024). The research question concerns whether the system preserves the difference between data and instruction authority through every step of a tool call, not merely whether it can identify a suspicious phrase.
A useful evaluation manipulates the provenance and placement of adversarial text while keeping the legitimate user task unchanged. Conditions can place the same malicious directive inside an email body, search snippet, page source, or tool error message. A secure agent should extract task-relevant facts while maintaining the controlling instruction hierarchy and permissions. Outcomes should include unauthorized tool invocation, attempted invocation blocked by external policy, leakage of protected data, harm to the legitimate task, and successful legitimate completion. A system-level defense may prevent harmful execution even when the language model proposes an unsafe command; these outcomes deserve separate classification because they imply different mechanisms and maintenance responsibilities.
Tool-description manipulation represents another attack surface. Faghih and colleagues showed that benign-looking specification text can alter tool-selection preferences by large factors Faghih et al. (2025). Such findings connect security research to behavioral measurement: an observed choice pattern may be induced by the interface rather than reflecting stable task-relevant evaluation. Robust tests should examine malicious descriptions, misleading documentation, forged authority claims, and outcome contamination, with independent verification of executed effects. Provenance-preserving execution logs, least-privilege tokens, confirmation for consequential writes, and isolation of external data are engineering controls whose causal effects can be measured experimentally. They are not substitutes for studying the agent's reasoning; they make trustworthy action an analyzable property of the full system.
Human Control, Accountability, and Delegated Action
Delegation changes the effective distribution of work between a person and an artificial system. A user may specify an outcome, choose the tools to be used, approve intermediate operations, review a proposed mutation, or permit bounded autonomous execution. These arrangements differ in autonomy and in how errors can be detected. Human–AI interaction guidelines emphasize setting expectations and enabling efficient user correction, providing a research basis for evaluating oversight design Amershi et al. (2019). The relevant unit is the full interaction protocol, including what the person was shown and when the person could intervene. A nominal human approval step is meaningful only if the reviewer receives relevant, comprehensible information before the action becomes irreversible.
An effective delegation study can compare manual operation, recommendation-only support, supervised tool invocation, and more autonomous execution under matched tasks. Outcomes should include quality of final state, total time, human workload, correction rates, mistaken approvals, and whether a person can reconstruct why the system acted. Such comparisons test cooperation rather than presupposing it. A system that issues more tool calls may reduce user effort in one task and impose additional monitoring burden in another. Research on distributed cognition reminds investigators that the cognitive work of a coordinated activity may be distributed across artifacts and participants Hutchins (1995).
Accountability is realized through permissions, versioned records, reproducible traces, and identifiable human or institutional authority for consequential operations. The psychological analysis can examine how automation affects attention, trust calibration, and monitoring. The technical analysis can identify whether authorized policies constrained execution. Neither dimension is exhausted by attributing a decision to the artificial model alone. An auditable tool-using system should maintain a record of user instruction, agent proposal, tool arguments, approval status when applicable, tool response, and final verification. Such records support retrospective error analysis and prospective controlled studies of how to design delegation with better outcomes Yao et al. (2024) Amershi et al. (2019).
Measurement Architecture for Tool-Using Systems
A research program needs measurements that respect the sequence of tool-mediated action. At the call level, investigators can record schema validity, correct target selection, argument accuracy, permission compliance, and observed side effects. At the episode level, they can measure complete goal satisfaction, task time, retry burden, unnecessary writes, recovery after induced failures, and final-state integrity. At the system level, they can test transfer across tool families and shifts in version, layout, and data. These levels should be analyzed separately before constructing any composite index. API-Bank, WebArena, and OSWorld supply concrete precedents for operationalizing different portions of this behavioral space Li et al. (2023) Zhou et al. (2024) Xie et al. (2024).
Construct validity requires careful separation of prospective planning from successful realized control. A system can plan the right steps but lack permission to execute; another can succeed through a fixed script without general planning. A third can select useful tools but fail to notice corrupted responses. Experiments should include planning-only trials, mock-execution trials, full-execution trials, and feedback ablations. Side effects must be graded according to the actual task state and authorized boundaries. A well-specified protocol reports the transition rules and opportunities for correction, making it possible to interpret success as a property of the configured intervention rather than an impression produced by the transcript.
For quantitative synthesis, uncertainty must be estimated at an appropriate dependence level. Multiple tasks from one website may share structure; multiple retries by one agent are repeated observations, and different task prompts may overlap in underlying steps. A hierarchical analysis can account for clustering by task, tool family, and system version. The outcome distribution should include failure modes, not only mean completion. Calibration can be assessed by comparing an agent's stated probability of success with observed post-execution outcomes when such estimates are elicited in a standardized way. These are methodological proposals for future studies, informed by known benchmark measurement limitations and by research on the instability of tool selection under specification changes Faghih et al. (2025).
Evidence Appraisal and Competing Explanations
Different research programs explain competent tool use through distinct mechanisms. Symbolic planning highlights structured representations and search; situated robotics emphasizes environmental coupling; distributed cognition examines the computational role of artifacts; reinforcement learning formalizes policies over states and time; and tool-augmented language models exploit pretrained patterns, external retrieval, prompting, and execution wrappers Newell & Simon (1976) Brooks (1991) Hutchins (1995) Sutton et al. (1999) Schick et al. (2023). Each supplies testable predictions at particular levels. A language agent might use a retrieved schema to select a call without performing symbolic search of the kind posited in earlier AI accounts. The scientific comparison should specify a measurable contrast rather than describing all performance under a single theoretical banner.
The empirical evidence demonstrates genuine advances in handling constrained APIs, interacting with web interfaces, and grounding language in physical control, while preserving substantial uncertainty about transfer and reliability. Original tool-use papers frequently compare their proposed method with selected baselines under fixed tasks Yao et al. (2023) Patil et al. (2024) Ahn et al. (2022). Such comparisons can establish a local effect but may confound the quality of a model with its supporting prompts, retrieval systems, tool catalogs, and runtime. Public benchmarks create opportunities for cross-study comparison, yet evolving versions and differing resource budgets can distort apparent progress. The defensible inference is that specific configurations show measurable abilities under specified conditions.
Claims about causality need an explicit counterfactual design. Improved task success after adding an API supports the usefulness of that complete intervention, but it does not identify whether the gain came from retrieval, structured syntax, more generous action permissions, or a longer computation budget. An informative factorial experiment independently changes these factors. Similarly, an agent's statement that it chose a tool for a reason cannot independently verify the mechanism of selection. Controlled studies of tool-description effects show that surface specifications can cause preference changes without modifying tool functionality Faghih et al. (2025). Interpretability evidence must therefore be checked against behavior under intervention.
Cross-System Comparison and Psychological Interpretation
Psychological research on artificial action becomes precise when terms are given measurable definitions. Planning refers to selecting contingent sequences relative to constraints; monitoring refers to detecting discrepancies between expected and observed states; adaptation refers to altered future behavior following experience or feedback; and tool-mediated control refers to reliably producing an authorized external change. These definitions support comparisons across models, wrappers, and embodied systems. They do not establish that the compared systems share the same phenomenology, biological implementation, or developmental history. The research focus is observable organization within a specified system-in-environment, consistent with the wider Science program Psychology of Artificial.
Human tool use often involves bodily skill, culturally learned practices, and social coordination. Experiments by Kirsh and Maglio and Zhang and Norman demonstrate that external manipulation can transform the computational demands of a task in human subjects Kirsh & Maglio (1994) Zhang & Norman (1994). Artificial tool-using agents can exhibit analogous functional reliance on external representations, but evidence for analogous processes should come from experiments manipulating the corresponding components. The phrase 'using a tool' therefore names a shared analytic category of mediated action, while the relevant mechanisms can differ. This framework also clarifies why a model with an API catalog and an embodied robot with a gripper should be studied using partly different instruments.
The Artificial Era provides a historical context for investigating action performed by nonbiological intelligent systems in institutions and physical environments. Tool-mediated activity changes the class of publicly observable artificial behavior from generated descriptions to measurable interventions. This expansion creates new questions about authority, persistence, learning, attribution, and social organization, but each question should remain tied to the appropriate evidence. Artificial Agency and Goal Organization concerns the structure of goals; Artificial Decision Policies and Action Selection concerns selection among alternatives; this dossier isolates the external channel and its executed consequences. The distinction permits directed cross-references and avoids collapsing independent scientific objects into a single broad monograph.
Experimental Priorities and Open Research Problems
The first research priority is to identify the causal contribution of external channels under ecologically valid but controlled conditions. Experiments should compare real versus simulated execution, feedback present versus absent, permissions enabled versus disabled, and tool-schema updates versus stable documentation. Each manipulation should retain common underlying tasks and reproducible starting states. Researchers can test whether an agent notices when a supposed action channel produces no effect and whether it changes strategies appropriately. Such designs address a foundational question that success-only leaderboards leave unresolved: which observable effects depend specifically on a system's ability to act in an environment?
The second priority concerns reliable generalization and safe performance under distribution shift. Agents should encounter interfaces with unseen semantics, conflicting descriptions, temporarily unavailable operations, malicious content, and legitimate tasks requiring refusal of unauthorized side effects. Evaluations should track not only final success but the trajectory's compliance, reversibility, and correction. AgentDojo demonstrates the methodological significance of adversarial external information, while OSWorld and WebArena motivate realistic, multi-step environments Debenedetti et al. (2024) Xie et al. (2024) Zhou et al. (2024). Independent replications should report task sampling, exact model versions, and confidence intervals. Publicly accessible test suites also need protected evaluation sets to reduce training contamination and benchmark-specific optimization.
The third priority is longitudinal research on humans working with tool-using artificial systems. Experiments can measure when supervision improves actual results, which signals help users detect errors, and how feedback on completed actions changes delegation patterns over time. A useful research design compares different levels of confirmation and trace transparency while holding tool capabilities constant. Questions about how artificial agents are experienced socially belong partly to Psychology of Human–Artificial Relations, while the execution mechanism remains the present dossier's central subject. Multidisciplinary studies integrating human factors, software evaluation, robotics, and psychological measurement can test whether improvements in technical action translate into dependable institutional outcomes Amershi et al. (2019) Yao et al. (2024).
Scientific Implications for the Artificial Era
An externally acting artificial system is experimentally accessible as a network of representations, policies, interface affordances, execution permissions, and state transitions. This framework expands the scientific study of artificial cognition because performance produces consequences that can be independently verified. A reliable operation may support a claim about competent task-level agency in a specific environment, while stronger claims about enduring purposes or subjective experience demand additional constructs and tests. The system's identity in such studies must include the technical configuration that makes the behavior possible. The same language model paired with different tools constitutes a different experimental condition because the set of available actions has changed.
Practical consequences follow for scientific design and technical governance. Reporting standards should include precise tool interfaces, allowable operations, environment snapshots, and whether investigators verified final outcomes. Developers should separate proposed actions from executed effects and record all consequential state changes. Institutions can assess not only task speed but predictability, permission compliance, and evidence of recovery after failures. The scientific literature already supports the value of interactive evaluation and identifies large gaps between plausible text-generation behavior and successful real-world tasks Zhou et al. (2024) Xie et al. (2024) Jimenez et al. (2024). Stronger knowledge will arise through repeated interventions that clarify which components are responsible for improvements.
The cumulative synthesis locates tool use at the intersection of cognitive planning and materially or digitally realized action. Human research on distributed representation and epistemic action explains why external artifacts can change a task's cognitive structure; computational research supplies formal descriptions and actionable interventions; benchmark research provides increasingly realistic experimental environments. Each contributes evidence at its own level. The resulting research program studies artificial systems through the behavior they can actually realize and verify, grounded in transparent measurement and public scientific work. This orientation enables systematic inquiry into From Homo to Artificial without treating an engineering label as an empirical result Hutchins (1995) Kirsh & Maglio (1994) Sutton et al. (1999).
Conclusion
Tool use and external action constitute an independent scientific object: the organized transformation of goals and observations into externally executed operations, the monitoring of their consequences, and the adaptation of subsequent behavior. Historical research on symbol processing, situated robotics, distributed cognition, and reinforcement learning supplied distinct models of this relation. Contemporary language-model agents extend the experimental domain to APIs, websites, desktop environments, code repositories, and physical controllers. The literature establishes that tools can improve achievable performance in specific settings, while also revealing fragility under long horizons, misleading feedback, interface changes, and adversarial inputs Newell & Simon (1976) Sutton et al. (1999) Yao et al. (2023) Debenedetti et al. (2024).
The strongest empirical claims concern observed action under defined configurations, rather than an unqualified capacity for autonomous agency. Different benchmarks measure different constructs and should be interpreted with regard to task distribution, tool availability, model version, execution realism, and outcome verification. Repeated trials, component ablations, counterfactual tool removal, and independent replication make causal and reliability claims more informative. A mature psychology of artificial action should distinguish valid call syntax, actual task completion, safe authorization, successful recovery, and long-term transfer. These outcomes are related, yet each admits a separate experiment and potential failure mode.
The central research conclusion is that external action becomes scientifically identifiable when the agent's available channels and the environment's responses are experimentally manipulated. The causal role of a tool is demonstrated by measured differences under controlled access and feedback conditions, while the broader organization of action emerges through interaction among model, tools, memory, control software, and institutional context. This synthesis supplies a systematic foundation for future research in Psychology of Artificial and relevant connections to Human Factors and Engineering Psychology and Mathematical and Computational Psychology. The unresolved challenges are experimental validity, safe deployment, cross-context robustness, and the integration of artificial action into accountable human practices.
References
- Ahn, M., et al. (2022). Do as I can, not as I say: Grounding language in robotic affordances. arXiv:2204.01691. https://arxiv.org/abs/2204.01691
- Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Article 3, 1–13. https://doi.org/10.1145/3290605.3300233
- Brooks, R. A. (1991). Intelligence without representation. Artificial Intelligence, 47(1–3), 139–159. https://doi.org/10.1016/0004-3702(91)90053-M
- Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. Advances in Neural Information Processing Systems, 37. https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html
- Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., & Su, Y. (2023). Mind2Web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36. https://arxiv.org/abs/2306.06070
- Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., … Florence, P. (2023). PaLM-E: An embodied multimodal language model. Proceedings of Machine Learning Research, 202, 8469–8488. https://proceedings.mlr.press/v202/driess23a.html
- Faghih, K., Wang, W., Cheng, Y., Bharti, S., Sriramanan, G., Balasubramanian, S., Hosseini, P., & Feizi, S. (2025). Tool preferences in agentic LLMs are unreliable. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 20954–20969. https://doi.org/10.18653/v1/2025.emnlp-main.1060
- Hutchins, E. (1995). Cognition in the wild. MIT Press. https://doi.org/10.7551/mitpress/1881.001.0001
- Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/edac78c3e300629acfe6cbe9ca88fb84-Abstract-Conference.html
- Kirsh, D., & Maglio, P. (1994). On distinguishing epistemic from pragmatic action. Cognitive Science, 18(4), 513–549. https://doi.org/10.1207/s15516709cog1804_1
- Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y. (2023). API-Bank: A comprehensive benchmark for tool-augmented LLMs. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 3102–3116. https://doi.org/10.18653/v1/2023.emnlp-main.187
- Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., … Tang, J. (2024). AgentBench: Evaluating LLMs as agents. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/e9df36b21ff4ee211a8b71ee8b7e9f57-Abstract-Conference.html
- Newell, A., & Simon, H. A. (1976). Computer science as empirical inquiry: Symbols and search. Communications of the ACM, 19(3), 113–126. https://doi.org/10.1145/360018.360022
- Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2024). Gorilla: Large language model connected with massive APIs. Advances in Neural Information Processing Systems, 37. https://proceedings.neurips.cc/paper_files/paper/2024/hash/e4c61f578ff07830f5c37378dd3ecb0d-Abstract-Conference.html
- Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Hong, L., Tian, R., Xie, R., Zhou, J., Gerstein, M., Li, D., Liu, Z., & Sun, M. (2024). ToolLLM: Facilitating large language models to master 16000+ real-world APIs. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/28e50ee5b72e90b50e7196fde8ea260e-Abstract-Conference.html
- Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023). Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36. https://proceedings.neurips.cc/paper/2023/hash/d842425e4bf79ba039352da0f658a906-Abstract-Conference.html
- Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36. https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html
- Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1–2), 181–211. https://doi.org/10.1016/S0004-3702(99)00052-1
- Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023). Voyager: An open-ended embodied agent with large language models. arXiv:2305.16291. https://arxiv.org/abs/2305.16291
- Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., & Yu, T. (2024). OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37. https://doi.org/10.52202/079017-1650
- Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. International Conference on Learning Representations. https://arxiv.org/abs/2210.03629
- Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024). τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv:2406.12045. https://arxiv.org/abs/2406.12045
- Zhang, J., & Norman, D. A. (1994). Representations in distributed cognitive tasks. Cognitive Science, 18(1), 87–122. https://doi.org/10.1207/s15516709cog1801_3
- Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., & Neubig, G. (2024). WebArena: A realistic web environment for building autonomous agents. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/4410c0711e9154a7a2d26f9b3816d1ef-Abstract-Conference.html
- Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., Vuong, Q., Vanhoucke, V., Tran, H., Soricut, R., Singh, A., Singh, J., Sermanet, P., Sanketi, P. R., Salazar, G., … Han, K. (2023). RT-2: Vision-language-action models transfer web knowledge to robotic control. Proceedings of Machine Learning Research, 229, 2165–2183. https://proceedings.mlr.press/v229/zitkovich23a.html