##### Report GitHub Issue

×

Title:

Content selection saved. Describe the issue below:

Description:

Submit without GitHub

Submit in GitHub

[<img src='/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg' alt='arXiv logo' title='' width='100' height='' />Back to arXiv](/)


[License: CC BY 4.0](https://info.arxiv.org/help/license/index.html#licenses-available) 

arXiv:2605.07937v1 [cs.CL] 08 May 2026

Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?
===============================================================================================

Anmol Gulati Hariom Gupta Elias Lumer  
Sahil Sen Vamse Kumar Subbiah  
PricewaterhouseCoopers U.SCorresponding authors: anmol.b.gulati@pwc.com, elias.lumer@pwc.com

###### Abstract

Long-horizon AI agents execute complex workflows spanning hundreds of sequential actions, yet a single wrong assumption early on can cascade into irreversible errors. When instructions are incomplete, the agent must decide not only *whether* to ask for clarification but *when*, and no prior work measures how clarification value changes over the course of execution. We introduce a forced-injection framework that provides ground-truth clarifications at controlled points in the agent’s trajectory across four information dimensions (goal, input, constraint, context), three agent benchmarks, and four frontier models (three per benchmark; one on a single benchmark only; 84 task variants; 6,000+ runs). Counter to the common intuition that “earlier is always better,” we find that the value of clarification depends sharply on *what* information is missing: goal clarification loses nearly all value after 10% of execution (pass@3 drops from 0.78 to baseline), while input clarification retains value through roughly 50%. Deferring *any* clarification type past mid-trajectory degrades performance below never asking at all. Cross-model Kendall $\tau$ correlations (0.78–0.87 among models sharing identical task coverage; 0.34–0.67 across the full 4-model panel) confirm these timing profiles are substantially task-intrinsic. A complementary study of 300 unscripted sessions reveals that no current frontier model asks within the empirically optimal window, with strategies ranging from over-asking (52% of sessions) to never asking at all. These empirical demand curves provide the quantitative foundation that existing theoretical frameworks require but have lacked, and establish concrete design targets for timing-aware clarification policies. Code and data will be publicly released.

1 Introduction
--------------

Long-horizon AI agents now execute complex multi-step workflows spanning code repair, enterprise automation, and tool orchestration, often completing tens to hundreds of sequential actions with minimal human oversight*(Yao et al., [2023](#bib.bib33 "ReAct: synergizing reasoning and acting in language models"); Xu et al., [2024](#bib.bib3 "TheAgentCompany: benchmarking LLM agents on consequential real world tasks"); Jimenez et al., [2024](#bib.bib4 "SWE-bench: can language models resolve real-world GitHub issues?"); Bandi et al., [2026](#bib.bib6 "MCP-Atlas: a large-scale benchmark for tool-use competency with real MCP servers"); Lumer et al., [2026a](#bib.bib54 "Tool and agent selection for large language model agents in production: a survey"))*. When user instructions are incomplete, these agents face a fundamental decision: continue executing under uncertainty, or pause to request clarification. Consider an enterprise agent tasked with generating a quarterly report that spends 30 actions building the wrong table before realizing the user meant fiscal quarters, not calendar quarters. A single clarifying question at step two could have prevented the wasted work entirely. The cost of this decision scales with trajectory length, as each additional action taken under wrong assumptions compounds the risk of irreversible errors and wasted computation*(Lumer et al., [2026b](#bib.bib50 "Don’t break the cache: an evaluation of prompt caching for long-horizon agentic tasks"))*.

Recent benchmarks show that underspecification consistently degrades agent performance and that clarification channels enable partial recovery*(Pu et al., [2026](#bib.bib1 "LHAW: controllable underspecification for long-horizon tasks"); Elfeki et al., [2026](#bib.bib2 "HiL-Bench: do agents know when to ask for help?"))*, yet existing work treats clarification as a binary capability: the agent either can or cannot ask. Neither LHAW*(Pu et al., [2026](#bib.bib1 "LHAW: controllable underspecification for long-horizon tasks"))* nor HiL-Bench*(Elfeki et al., [2026](#bib.bib2 "HiL-Bench: do agents know when to ask for help?"))* addresses the temporal dimension, though LHAW’s failure mode analysis independently identifies timing as a problem, finding that agents ask at inappropriate points in the trajectory. Classical value-of-information theory*(Howard, [1966](#bib.bib7 "Information value theory"))* and HCI interruption research*(Adamczyk and Bailey, [2004](#bib.bib29 "If not now, when? the effects of interruption at different moments within task execution"); Mark et al., [2008](#bib.bib41 "The cost of interrupted work: more speed and stress"))* both predict that clarification benefit should vary with timing, yet no empirical measurement of this interaction exists for modern LLM agents. Asking too early risks querying about information that is inferable from context not yet observed. Asking too late means the agent has already committed irreversible actions based on wrong assumptions.

We isolate the pure timing effect with a forced-injection framework that provides ground-truth clarifications at controlled trajectory points, independent of the agent’s ability to detect ambiguity, across 84 underspecified task variants, three benchmarks, and four frontier models (Figure[1](#S1.F1 "Figure 1 ‣ 1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). Our experiments yield two findings that challenge common assumptions about clarification timing. First, the value of clarification is not simply “earlier is better”; it depends sharply on what type of information is missing, with goal clarification losing nearly all value after 10% of execution while input clarification retains value through roughly 50%. Second, despite this structure, no current frontier model naturally asks within the empirically optimal window for any dimension, with strategies ranging from over-asking (52% of sessions) to never asking at all.

Our contributions are:

1. 1.

    We establish that clarification timing interacts strongly with information dimension: goal information has a narrow early window (${\leq}$10% of execution), input information is recoverable through ${\sim}$50%, and constraint benefit depends on whether an oracle gap exists, with cross-model consistency (Kendall $\tau$ of 0.78–0.87 among models sharing identical task coverage; 0.34–0.67 on the broader panel) confirming these profiles are substantially task-intrinsic. Wasted compute grows steadily with delay, providing a complementary cost measure (Section[5](#S5 "5 Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")).

2. 2.

    We quantify the gap between current model asking behavior and these empirically optimal windows: across 300 unscripted sessions, natural ask timings are severely late for goal tasks, and one frontier model never asks at all, establishing concrete design targets for timing-aware clarification policies.

<img src='2605.07937v1/figures/fig1_overview_candidate_8.png' alt='Refer to caption' title='' width='598' height='334' />

*Figure 1: Overview of the forced-injection experimental framework. We inject ground-truth clarifications at controlled points along an oracle-calibrated action budget, measuring task success (pass@3) at each injection timing across four information dimensions.*

2 Related Work
--------------

Prior work addresses whether clarification helps, when information is theoretically valuable, and how agents should decide to ask. We identify a common blind spot: none varies clarification *timing* as an independent variable.

### 2.1 Clarification in Agent Benchmarks

A growing body of work establishes that underspecification degrades agent performance and that clarification channels enable partial recovery. LHAW*(Pu et al., [2026](#bib.bib1 "LHAW: controllable underspecification for long-horizon tasks"))* constructs 285 underspecified task variants across four information dimensions and shows that models exhibit widely divergent clarification strategies. HiL-Bench*(Elfeki et al., [2026](#bib.bib2 "HiL-Bench: do agents know when to ask for help?"))* complements this by penalizing both over-asking and missed escalation. Shorter-horizon benchmarks study related phenomena: QuestBench*(Li et al., [2025](#bib.bib13 "QuestBench: can models ask the right questions?"))*, AmbigQA*(Min et al., [2020](#bib.bib14 "AmbigQA: answering ambiguous open-domain questions"))*, ClariQ*(Aliannejadi et al., [2020](#bib.bib15 "ConvAI3: generating clarifying questions for open-domain dialogue systems (ClariQ)"))*, UserBench*(Qian and others, [2025](#bib.bib16 "UserBench: an interactive gym environment for user-centric agents"))*, and $\tau$-bench*(Yao and others, [2024](#bib.bib17 "τ-Bench: a benchmark for tool-agent-user interaction in real-world domains"))*. Tamkin et al.*(Tamkin et al., [2023](#bib.bib37 "Task ambiguity in humans and language models"))* show that task ambiguity systematically degrades model behavior, and Kadavath et al.*(Kadavath et al., [2022](#bib.bib30 "Language models (mostly) know what they know"))* establish that LLMs have limited self-assessment of uncertainty. Several approaches aim to close this gap: confidence-based selective clarification*(Kuhn et al., [2023](#bib.bib31 "CLAM: selective clarification for ambiguous questions with generative language models"))*, uncertainty-aware planning*(Hu et al., [2024](#bib.bib49 "Uncertainty of thoughts: uncertainty-aware planning enhances information seeking in large language models"))*, proactive pre-execution questioning*(Zhang et al., [2024](#bib.bib35 "Ask-before-plan: proactive language agents for real-world planning"))*, and simulation-based ask-or-answer training*(Wu et al., [2025](#bib.bib36 "CollabLLM: from passive responders to active collaborators"))*. In reinforcement learning, ask-for-help policies*(Clouse, [1996](#bib.bib22 "On integrating apprentice learning and reinforcement learning"))* and DAgger*(Ross et al., [2011](#bib.bib23 "A reduction of imitation learning and structured prediction to no-regret online learning"))* query experts under uncertainty, Plaut et al.*(Plaut et al., [2025](#bib.bib24 "Avoiding catastrophe in online learning by asking for help"))* prove that agents must query at a linear rate or risk catastrophe, and recent LLM systems learn *whether* to ask via counterfactual training*(Liu and others, [2025](#bib.bib25 "CaRT: teaching LLM agents to know when they know enough"))*, process reward models*(Min et al., [2025](#bib.bib26 "Self-regulation and requesting interventions"))*, or interactive feedback*(Pan and others, [2025](#bib.bib27 "When benchmarks talk: re-evaluating code generation with interactive tasks"))*. All optimize a confidence threshold but assume information is equally useful at any trajectory point. Broader long-horizon agent benchmarks*(Zhou et al., [2024](#bib.bib45 "WebArena: a realistic web environment for building autonomous agents"); Trivedi et al., [2024](#bib.bib44 "AppWorld: a controllable world of apps and people for benchmarking interactive coding agents"); Liu et al., [2024](#bib.bib46 "AgentBench: evaluating LLMs as agents"); Xie et al., [2024](#bib.bib47 "OSWorld: benchmarking multimodal agents for open-ended tasks in real computer environments"); Mialon et al., [2023](#bib.bib48 "GAIA: a benchmark for general AI assistants"))* similarly do not vary clarification timing.

### 2.2 Theoretical Foundations Predict Timing Should Matter

The decision to seek clarification mid-trajectory is a special case of the value-of-information (VOI) problem*(Lindley, [1956](#bib.bib38 "On a measure of the information provided by an experiment"); Howard, [1966](#bib.bib7 "Information value theory"))*: should the agent pay a cost (interruption, latency) to reduce uncertainty before acting? Russell and Wefald*(Russell and Wefald, [1991](#bib.bib8 "Do the right thing: studies in limited rationality"))* formalize this as metareasoning, and the active-learning paradigm*(Settles, [2009](#bib.bib39 "Active learning literature survey"))* studies when to query an oracle to maximize learning efficiency. In HCI, Adamczyk and Bailey*(Adamczyk and Bailey, [2004](#bib.bib29 "If not now, when? the effects of interruption at different moments within task execution"))* demonstrate that task outcomes depend on *when* an interruption occurs, and Mark et al.*(Mark et al., [2008](#bib.bib41 "The cost of interrupted work: more speed and stress"))* quantify the recovery cost of poorly timed interruptions. Recent work operationalizes these ideas for LLM agents: Rao and Daumé*(Rao and Daumé III, [2018](#bib.bib9 "Learning to ask good questions: ranking clarification questions using neural expected value of perfect information"))* apply expected value of perfect information to rank clarification questions, Dong et al.*(Dong et al., [2026](#bib.bib10 "Value of information: a framework for human-agent communication"))* propose a decision-theoretic framework, Kobalczyk et al.*(Kobalczyk et al., [2025](#bib.bib34 "Active task disambiguation with LLMs"))* cast clarification as Bayesian experimental design, Suri et al.*(Suri et al., [2025](#bib.bib11 "Structured uncertainty guided clarification for LLM agents"))* use structured uncertainty for tool-calling agents, Lumer et al.*(Lumer et al., [2025a](#bib.bib53 "Memtool: optimizing short-term memory management for dynamic tool calling in llm agent multi-turn conversations"))* optimize short-term memory management across multi-turn tool interactions, and PRISM*(Fu et al., [2026](#bib.bib12 "PRISM: festina lente proactivity – risk-sensitive, uncertainty-aware deliberation for proactive agents"))* learns a cost-sensitive gate for proactive interventions. These frameworks all predict that clarification value should depend on timing, yet none provides the empirical measurements our forced-injection experiments produce.

3 Hypotheses: Trajectory Commitment
-----------------------------------

We develop a simple model predicting *why* clarification timing should interact with information dimension. A task is specified by a parameter vector $\theta\=(\theta_{\text{goal}},\theta_{\text{input}},\theta_{\text{con}},\theta_{\text{ctx}})$, one component per dimension. The agent observes an underspecified version $\tilde{\theta}$ in which one component $\theta_{d}$ is missing, and executes a trajectory $\tau\=(a_{1},\ldots,a_{T})$. At injection time $t$, the agent receives the true value of $\theta_{d}$. We define the *commitment* $C_{d}(t)\in[0,1]$ as the fraction of actions $a_{1},\ldots,a_{t}$ that causally depend on dimension $d$. The value of clarification at time $t$ is bounded by the recoverable portion:

|  | $\text{VOI}_{d}(t)\;\leq\;\text{VOI}_{d}(0)\cdot\bigl(1-C_{d}(t)\bigr),$ |  | (1) |
| --- | --- | --- | --- |

since committed actions cannot be undone. The shape of $C_{d}(t)$ thus upper-bounds the timing decay profile for each dimension; the actual VOI may decay faster if reconciliation costs exceed the recoverable portion. We argue that $C_{d}(t)$ grows at different rates due to the causal structure of typical workflows: *Goal* and *context* condition all subsequent actions: early approach selection cascades through the entire trajectory, predicting concave, front-loaded commitment. *Input* affects only data-dependent steps; agents can partially infer inputs through exploration, predicting approximately linear commitment. *Constraints* impose rules that may be invisible until violated; actions taken without constraint knowledge may actively conflict with it, so late injection can be *disruptive*, with reconciliation cost exceeding the information’s value.

*Table 1: Dataset and experimental setup. Action budgets are oracle-calibrated per model and variant. Total runs include all conditions and trials across available models.*

| Benchmark | Variants | Dimensions | Action Budget | Total Runs |
| --- | --- | --- | --- | --- |
| MCP-Atlas | 36 | goal, constraint, input, context | 6–20 | 3,024 |
| TheAgentCompany | 30 | goal, constraint, input | 6–49 | 1,890 |
| SWE-Bench Pro | 18 | goal, constraint, input, context | 1–121 | 1,134 |

*Table 2: Frontier models evaluated. API identifiers are provided for reproducibility. Clarification archetypes are derived from the natural-ask protocol (Section[4.4](#S4.SS4 "4.4 Natural Ask Protocol ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). A fourth model, DeepSeek V3.2, is evaluated on MCP-Atlas only (see Section[4.2](#S4.SS2 "4.2 Benchmarks and Models ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")).*

| Model | API Identifier | Provider | Archetype |
| --- | --- | --- | --- |
| GPT-5.2 | gpt-5.2-2025-12-11 | OpenAI | Over-clarifier (52%) |
| Claude Sonnet 4.5 | claude-sonnet-4-5 | Anthropic | Balanced (23%) |
| Gemini 3 Flash | gemini-3-flash-preview | Google | Under-clarifier (0%) |
| DeepSeek V3.2 | deepseek-v3-2 | DeepSeek | N/A (MCP-Atlas only) |

These predictions yield two falsifiable hypotheses. H1 (Dimension-dependent timing): Goal and context clarification are front-loaded (steep VOI decay), while input clarification decays gradually. H2 (Constraint attenuation): On benchmarks where an oracle gap exists, constraint clarification yields VOI that declines with delay, with reconciliation costs offsetting some but not all of the information’s value. We test both in Section[5](#S5 "5 Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").

4 Experimental Design
---------------------

### 4.1 Overview and Hypotheses

We test the hypotheses derived from the trajectory commitment model (Section[3](#S3 "3 Hypotheses: Trajectory Commitment ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")): H1 predicts that goal and context clarification are front-loaded while input clarification decays gradually, and H2 predicts that constraint clarification yields VOI that declines with delay on benchmarks where an oracle gap exists, with reconciliation costs offsetting some but not all of the information’s value. We additionally test whether, for each dimension, there exists a *point of no return* beyond which clarification no longer improves over the no-clarification baseline.

For each of 84 underspecified task variants (full breakdown in Appendix[A.10](#A1.SS10 "A.10 Dataset Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")), we run seven conditions with three trials per condition across available models (four on MCP-Atlas, three on TheAgentCompany and SWE-Bench Pro), yielding over 6,000 total experiment runs across all conditions, models, and trials (Appendix[A.8](#A1.SS8 "A.8 Total Runs and Computational Cost ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). The seven conditions are: (1) *oracle*, where the agent receives the fully-specified original prompt (upper bound); (2) *no-clarification*, where the agent receives the underspecified prompt and must complete the task without additional information (lower bound); and (3–7) five *injection* conditions at 10%, 30%, 50%, 70%, and 90% of the trajectory, where the missing information is provided mid-execution.

<img src='2605.07937v1/figures/fig_natural_ask_protocol_v2.png' alt='Refer to caption' title='' width='598' height='334' />

*Figure 2: Natural ask protocol. Three models each run 100 sessions on underspecified TheAgentCompany tasks with the ask_user tool enabled. We report two distinct percentages: the *ask rate* (fraction of sessions in which the model asked at least once) and the *mean ask timing* (how far through the trajectory the first ask occurred, as a percentage of total actions). GPT-5.2 asks frequently (ask rate 52%, mean timing 43%), Claude Sonnet 4.5 asks selectively (ask rate 23%, mean timing 50%), and Gemini 3 Flash never asks (ask rate 0%).*

### 4.2 Benchmarks and Models

We draw our 84 variants from a stratified subset of LHAW’s*(Pu et al., [2026](#bib.bib1 "LHAW: controllable underspecification for long-horizon tasks"))* underspecified tasks, spanning three benchmarks that cover distinct agent domains (Table[1](#S3.T1 "Table 1 ‣ 3 Hypotheses: Trajectory Commitment ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")):

MCP-Atlas *(Bandi et al., [2026](#bib.bib6 "MCP-Atlas: a large-scale benchmark for tool-use competency with real MCP servers"); Lumer et al., [2025b](#bib.bib52 "Scalemcp: dynamic and auto-synchronizing model context protocol tools for llm agents"), [c](#bib.bib51 "Tool-to-agent retrieval: bridging tools and agents for scalable llm multi-agent systems"))* (36 variants, 4 models) evaluates tool-use competency across real MCP servers with short trajectories (6–20 actions). All four information dimensions are represented. This is the only benchmark evaluating all four models including DeepSeek V3.2, as its standard tool-calling APIs require no benchmark-specific harness.

TheAgentCompany *(Xu et al., [2024](#bib.bib3 "TheAgentCompany: benchmarking LLM agents on consequential real world tasks"))* (30 variants, 3 models) tests enterprise workflows on a simulated corporate platform with longer trajectories (6–49 actions) involving web, database, and tool interactions. Three dimensions are covered (goal, constraint, input). DeepSeek V3.2 is excluded due to harness integration limitations.

SWE-Bench Pro *(Scale AI, [2025](#bib.bib5 "SWE-Bench Pro: a more challenging benchmark for real-world software engineering"))* (18 variants, 3 models) evaluates code-repair tasks on real GitHub issues (1–121 actions). DeepSeek V3.2 is excluded; the remaining three models each have both injection and baseline data, yielding $n{\=}12$–31 (variant, model) units per injection cell depending on dimension (Appendix[A.7](#A1.SS7 "A.7 SWE-Bench Pro Limitations ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")).

LHAW defines four information dimensions: *goal* (unclear deliverable), *constraint* (missing rules or thresholds), *input* (unspecified data source), and *context* (absent domain knowledge). Variants are stratified by dimension and ambiguity class: *outcome-critical* (missing information causes failure), *divergent* (valid but unintended output), and *benign* (agent succeeds despite the gap).

### 4.3 Forced Injection Protocol

Because tasks range from 6 to 121 actions, we calibrate injection points per model and task. For model $m$ and variant $v$, let $B_{m,v}$ denote the mean oracle trajectory length (rounded to the nearest integer). The injection action for timing $t\in{0.1,0.3,0.5,0.7,0.9}$ is $a_{\text{inject}}\=\max(1,\lfloor B_{m,v}\cdot t\rfloor)$. For SWE-Bench Pro tasks where $B_{m,v}\=1$, all five injection conditions collapse to action 1; these are retained to preserve sample size but contribute zero timing variance.

At action $a_{\text{inject}}$, a synthetic user message containing the ground-truth removed_segments is inserted at the next clean turn boundary (Appendix[A.11](#A1.SS11 "A.11 Injection Protocol Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[A.12](#A1.SS12 "A.12 Example Trajectory ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). The ask_user tool is disabled, so clarification arrives only through the injection. Messages read naturally (e.g., “By the way, I should have mentioned: the target format is CSV, not JSON.”). Each action corresponds to one tool call or one agent message turn.

### 4.4 Natural Ask Protocol

To assess whether models naturally ask within the optimal timing window, we run a complementary 300-session study on TheAgentCompany tasks with the ask_user MCP tool enabled. Each of three models (Claude Sonnet 4.5, GPT-5.2, Gemini 3 Flash) completes 100 sessions across seven base tasks (a stratified subset of the 30 TAC variants, selected to cover goal-dimension and input-dimension variants in roughly equal proportion across all three ambiguity classes). When invoked, a simulated user responds with the ground-truth missing information matching the forced-injection content. We record the action step of each ask_user call as a percentage of total trajectory length (Figure[2](#S4.F2 "Figure 2 ‣ 4.1 Overview and Hypotheses ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")).

<img src='2605.07937v1/figures/fig_voi_mcp_atlas_2x2.png' alt='Refer to caption' title='' width='539' height='429' />

*Figure 3: MCP-Atlas VOI curves by information dimension ($n{\=}5$–28 per dimension; see subplot titles). Goal clarification is dramatically front-loaded (0.78 at 10% vs. 0.40 NC), input shows gradual decline, and constraint shows minimal benefit over baseline. Dashed: no-clarification baseline; dotted: oracle. Per-benchmark curves and aggregate curves appear in Appendix[A.1](#A1.SS1 "A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") and[A.9](#A1.SS9 "A.9 Aggregate VOI Interpretation ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").*

### 4.5 Metrics

We evaluate with four complementary metrics:

Pass@3. Following the HumanEval formulation*(Chen et al., [2021](#bib.bib28 "Evaluating large language models trained on code"))*, we compute the probability that at least one of $k\=3$ trials succeeds for each (variant, model, condition) cell. We report the mean pass@3 across units within each experimental group.

Wasted compute. For injection conditions, we count pre-injection actions whose effects are absent from the oracle (fully-specified) trace, normalized by total pre-injection actions. This quantifies the fraction of early work rendered useless by late clarification.

Actions absent from the oracle trace may still be indirectly useful (e.g., exploratory schema queries that inform the agent’s understanding). We therefore treat this metric as a conservative upper bound on waste, appropriate for quantifying the cost ceiling of late clarification rather than claiming every non-oracle action is worthless. Qualitative analysis (Appendix[A.6](#A1.SS6 "A.6 Qualitative Trajectory Analysis ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")) confirms that some “wasted” exploration partially compensates for missing specifications, consistent with input’s gradual VOI decline.

Point of no return. The latest injection timing at which pass@3 significantly exceeds the no-clarification baseline (one-sided permutation test, $p<0.05$), stratified by dimension and ambiguity class.

Cross-model consistency. Kendall’s $\tau$ rank correlation*(Kendall, [1938](#bib.bib40 "A new measure of rank correlation"))* of per-variant pass@3 vectors across model pairs, measuring whether timing effects are task-intrinsic or model-dependent.

<img src='2605.07937v1/figures/fig_appendix_wasted_compute_all.png' alt='Refer to caption' title='' width='598' height='271' />

*Figure 4: Wasted compute across all benchmarks. Left: MCP-Atlas and TheAgentCompany (fraction of pre-injection actions absent from the oracle trace). Right: SWE-Bench Pro (absolute action count; different scale due to longer trajectories). Later injection consistently produces more wasted work.*

*Table 3: Cross-model Kendall’s $\tau$ rank correlation. Left: TheAgentCompany only (3 models, filtered to complete-data pairs). Right: all benchmarks combined (4 models).*

|  | Claude | Gemini | GPT |
| --- | --- | --- | --- |
| Claude | 1.00 | 0.87 | 0.82 |
| Gemini | 0.87 | 1.00 | 0.78 |
| GPT | 0.82 | 0.78 | 1.00 |

*(a) TheAgentCompany*

|  | Claude | DeepSeek | Gemini | GPT |
| --- | --- | --- | --- | --- |
| Claude | 1.00 | 0.45 | 0.67 | 0.43 |
| DeepSeek | 0.45 | 1.00 | 0.47 | 0.34 |
| Gemini | 0.67 | 0.47 | 1.00 | 0.39 |
| GPT | 0.43 | 0.34 | 0.39 | 1.00 |

*(b) All benchmarks combined*

5 Results
---------

### 5.1 VOI Curves Are Dimension-Dependent (H1)

Figure[3](#S4.F3 "Figure 3 ‣ 4.4 Natural Ask Protocol ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") presents VOI curves on MCP-Atlas, the benchmark with the strongest timing signal ($n{\=}5$–28 units per dimension; complete per-benchmark values in Appendix Table[4](#A1.T4 "Table 4 ‣ A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). Clarification timing interacts strongly with information dimension, partially confirming H1 (the goal-front-loading prediction is strongly supported on MCP-Atlas; the context prediction is directionally consistent but underpowered at $n{\=}17$ across two benchmarks):

Goal. Exhibits dramatic front-loading. Injection at 10% of the trajectory recovers near-oracle performance (pass@3 of 0.78 vs. oracle 0.80, NC 0.40). The benefit decays steeply: Inj-30 \= 0.50 (above NC), Inj-50 \= 0.44 (above NC), Inj-70 \= 0.39 ($\approx$ NC), Inj-90 \= 0.39. By 70% of the trajectory, goal injection no longer improves over no-clarification. This confirms that goal information has a narrow early window of high value.

Input. Shows gradual decline (0.46 at Inj-10, 0.36 at Inj-30 and Inj-50, 0.32 at Inj-70, 0.25 at Inj-90; NC \= 0.33, oracle \= 0.57). The early window (Inj-10 through Inj-50) recovers most of the oracle gap, but by Inj-70 input clarification falls to the NC baseline and by Inj-90 sits well below it. The flatter slope relative to goal confirms that input information is recoverable for longer than goal information, consistent with agents partially compensating through environment exploration, but this recovery window extends only through approximately 50% of the trajectory.

<img src='2605.07937v1/figures/fig6_natural_ask_overlay.png' alt='Refer to caption' title='' width='479' height='328' />

*Figure 5: Natural ask timing overlaid on aggregate VOI curves for goal and input dimensions. Vertical lines indicate mean first-ask timing: GPT-5.2 (purple, 43%) asks within the early-injection window for input but late for goal; Claude (orange, 50%) asks at midpoint. Gemini never asks. The shaded region marks the optimal window for goal clarification (defined as the range of injection timings whose pass@3 is within 5 percentage points of the Inj-10 maximum).*

Constraint. Constraint behavior depends on whether an oracle gap exists. On MCP-Atlas, Oracle and NC are identical (both 0.12), so the benchmark is uninformative for H2 (there is no information value to attenuate), and a flat injection curve (0.13–0.17 across timings) is consistent with both H2 and the null hypothesis. On SWE-Bench Pro, where the oracle gap is substantial (0.81 vs. 0.56 NC), constraint injection shows genuine declining benefit (0.81 at Inj-10, declining to 0.68 at Inj-90; Appendix Table[4](#A1.T4 "Table 4 ‣ A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")), supporting a weaker form of H2: reconciliation costs reduce constraint VOI relative to the oracle, but injected constraint information remains above the no-clarification baseline at all tested timings (0.68 $>$ 0.56). The strong form of H2 (below-baseline disruption) is not supported by either benchmark. Constraint-type clarification should ideally be provided before execution begins rather than mid-trajectory.

Context. Evidence comes from two benchmarks with limited variant diversity: MCP-Atlas ($n{\=}5$) shows strong early-injection benefit (0.80 at 10% vs. 0.60 NC), and SWE-Bench Pro ($n{\=}12$) is essentially flat across timings (0.92 at most injection points, with Oracle \= 0.67 and NC \= 0.75; the NC $>$ Oracle inversion reflects sampling variability; see Appendix[A.7](#A1.SS7 "A.7 SWE-Bench Pro Limitations ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). The combined sample ($n{\=}17$) is too small for reliable conclusions about timing structure for context, though the MCP-Atlas signal is consistent with H1’s front-loading prediction.

Cross-benchmark patterns. On TheAgentCompany, floor effects (oracle $\leq$29%) attenuate timing signals. Goal injection conditions all sit within 4 pp of NC (0.12), consistent with sampling variability at $n{\=}65$ rather than injection-induced disruption. Input shows mild benefit at intermediate timings (0.21–0.25 vs. 0.19 NC). Constraint is at floor (0% across all conditions). SWE-Bench Pro results (Appendix[A.1](#A1.SS1 "A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"); caveats in[A.7](#A1.SS7 "A.7 SWE-Bench Pro Limitations ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")) are directionally consistent. Aggregate curves appear in Appendix[A.9](#A1.SS9 "A.9 Aggregate VOI Interpretation ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").

Point of no return. We find that clarification benefit is continuous rather than threshold-based: only outcome-critical variants show a statistically significant recovery point, at 30% of the trajectory ($p<0.05$; Appendix[A.5](#A1.SS5 "A.5 Point of No Return Analysis ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). Earlier is always better, but there is no sharp cutoff after which clarification becomes useless.

### 5.2 Wasted Compute

Wasted compute (fraction of pre-injection actions absent from the oracle trace, an upper bound on discarded work) increases steadily with injection delay across all three benchmarks (Figure[4](#S4.F4 "Figure 4 ‣ 4.5 Metrics ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"); Appendix[A.3](#A1.SS3 "A.3 Wasted Compute by Benchmark ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). On TheAgentCompany, waste rises from 0.0% at Inj-10 to 21.7% at Inj-90. On MCP-Atlas, it ranges from 38.4% to 52.9% (higher baseline reflecting shorter trajectories). SWE-Bench Pro confirms the pattern: 0.7 to 10.4 wasted actions.

### 5.3 Cross-Model Consistency

Table[3](#S4.T3 "Table 3 ‣ 4.5 Metrics ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") reports Kendall’s $\tau$ rank correlations between model pairs. On TheAgentCompany (3 models, balanced variant set), correlations range from 0.78 to 0.87, indicating strong agreement on which variants benefit from early clarification. On the combined dataset across all benchmarks (4 frontier models), Claude-Gemini achieves $\tau\=0.67$ and all model pairs exceed 0.34 ($p<0.01$ for all pairs; see Appendix[A.4](#A1.SS4 "A.4 Cross-Model Correlation Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). These results support the interpretation that timing effects are predominantly task-intrinsic: the same variants benefit from early injection regardless of which model executes them.

### 5.4 Natural Ask Overlay

Our natural-ask analysis of 300 TheAgentCompany sessions reveals stark differences in model clarification behavior (Appendix[A.2](#A1.SS2 "A.2 Natural Ask Timing Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") provides per-model statistics and timing distributions). GPT-5.2 asks in 52% of sessions with a mean first-ask timing of 43% through the trajectory. Claude Sonnet 4.5 asks in 23% of sessions at a mean timing of 50%. Gemini 3 Flash never asks (0% ask rate across all 100 sessions). These archetypes are consistent with LHAW’s independent finding, under a different protocol, that GPT-5.2 over-clarifies with the lowest per-question efficiency while Gemini models under-clarify*(Pu et al., [2026](#bib.bib1 "LHAW: controllable underspecification for long-horizon tasks"))*.

Overlaying these natural timings on the forced-injection VOI curves reveals timing alignment gaps (Figure[5](#S5.F5 "Figure 5 ‣ 5.1 VOI Curves Are Dimension-Dependent (H1) ‣ 5 Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). GPT-5.2’s asking at 43% is past the goal optimum (10%) but within the input window; Claude’s 50% is suboptimal for goal but reasonable for input; Gemini forgoes all benefit. Notably, timing alignment alone does not predict success: Claude achieves higher per-session success (11%) than GPT-5.2 (3%) despite asking later and less often (Appendix[A.2](#A1.SS2 "A.2 Natural Ask Timing Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")), suggesting that question quality may matter more than frequency, though this between-model comparison cannot isolate question quality from other model-level differences, and we treat it as a hypothesis rather than a controlled finding.

6 Limitations
-------------

This study establishes the *demand side* of clarification timing (how much does task performance benefit from receiving information at each trajectory point?) but does not address the *supply side*: making agents recognize ambiguity and ask at the right moment. Building supply-side mechanisms on top of these VOI curves is an important next step. The strongest timing signals come from MCP-Atlas ($n{\=}5$–28 per dimension); TheAgentCompany provides weaker signal due to floor effects, and SWE-Bench Pro has moderate per-cell sizes ($n{\=}12$–31). The natural-ask protocol covers only TheAgentCompany (300 sessions). A behavioral confound exists between protocols: forced injection disables ask_user while natural-ask enables it, so agents may plan differently when they know they cannot ask; the forced-injection VOI curves should be interpreted as upper bounds on information value.

7 Conclusion
------------

We present the first empirical VOI curves for clarification timing in long-horizon agent workflows. Timing effects are strongly dimension-dependent: goal clarification is front-loaded (${\leq}$10% window), input degrades gradually (recoverable through ${\sim}$50%), and constraint benefit depends on the oracle gap, with cross-model consistency confirming these profiles are task-intrinsic. No current frontier model asks within the optimal window. These findings yield two actionable guidelines: (1) agents should validate goals within the first few actions; (2) input queries can be deferred slightly longer but should be raised within the first half of the trajectory. More broadly, the forced-injection methodology generalizes to any agent benchmark with underspecified tasks and provides the empirical demand curves that existing theoretical frameworks*(Howard, [1966](#bib.bib7 "Information value theory"); Dong et al., [2026](#bib.bib10 "Value of information: a framework for human-agent communication"); Kobalczyk et al., [2025](#bib.bib34 "Active task disambiguation with LLMs"))* require but have lacked. Building timing-aware clarification policies*(Sen et al., [2026](#bib.bib55 "Chronos: temporal-aware conversational agents with structured event retrieval for long-term memory"))* on top of these curves is a natural next step.

Acknowledgments and Disclosure of Funding
-----------------------------------------

We thank the LHAW, MCP-Atlas, TheAgentCompany, and SWE-Bench Pro teams for their open-source benchmarks. We also thank our colleagues at PricewaterhouseCoopers for valuable feedback on earlier drafts.

References
----------

* P. D. Adamczyk and B. P. Bailey (2004)If not now, when? the effects of interruption at different moments within task execution.In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 271–278.Cited by: [§1](#S1.p2.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* M. Aliannejadi, J. Kiseleva, A. Chuklin, J. Dalton, and M. Burtsev (2020)ConvAI3: generating clarifying questions for open-domain dialogue systems (ClariQ).arXiv preprint arXiv:2009.11352.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* C. Bandi, B. Hertzberg, G. Boo, T. Polakam, J. Da, S. Hassaan, M. Sharma, A. Park, E. Hernandez, D. Rambado, I. Salazar, R. Cruz, C. Rane, B. Levin, B. Kenstler, and B. Liu (2026)MCP-Atlas: a large-scale benchmark for tool-use competency with real MCP servers.arXiv preprint arXiv:2602.00933.Cited by: [§1](#S1.p1.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§4.2](#S4.SS2.p2.1 "4.2 Benchmarks and Models ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021)Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374.Cited by: [§4.5](#S4.SS5.p2.1 "4.5 Metrics ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* J. A. Clouse (1996)On integrating apprentice learning and reinforcement learning.Ph.D. Thesis, University of Massachusetts Amherst.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* Y. R. Dong, T. Hu, Z. Hui, C. Zhang, I. Vulić, A. Bobu, and N. Collier (2026)Value of information: a framework for human-agent communication.arXiv preprint arXiv:2601.06407.Cited by: [§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§7](#S7.p1.2 "7 Conclusion ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* M. Elfeki, T. Trinh, G. Luo, K. Luu, N. Hunt, E. Hernández, N. Marwaha, Y. Y. He, C. Wang, F. Carabedo, A. Castillo, and B. Liu (2026)HiL-Bench: do agents know when to ask for help?.arXiv preprint arXiv:2604.09408.Cited by: [§1](#S1.p2.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* Y. Fu, X. Tan, T. Hao, C. Zhan, and X. Qiu (2026)PRISM: festina lente proactivity – risk-sensitive, uncertainty-aware deliberation for proactive agents.In International Conference on Learning Representations,Cited by: [§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* R. A. Howard (1966)Information value theory.IEEE Transactions on Systems Science and Cybernetics 2 (1),  pp. 22–26.External Links: [Document](https://dx.doi.org/10.1109/TSSC.1966.300074 "")Cited by: [§1](#S1.p2.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§7](#S7.p1.2 "7 Conclusion ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* Z. Hu, C. Liu, X. Feng, Y. Zhao, S. Ng, A. T. Luu, J. He, P. W. Koh, and B. Hooi (2024)Uncertainty of thoughts: uncertainty-aware planning enhances information seeking in large language models.In Advances in Neural Information Processing Systems,Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2024)SWE-bench: can language models resolve real-world GitHub issues?.In International Conference on Learning Representations,Cited by: [§1](#S1.p1.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, et al. (2022)Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* M. G. Kendall (1938)A new measure of rank correlation.Biometrika 30 (1/2),  pp. 81–93.Cited by: [§4.5](#S4.SS5.p6.1 "4.5 Metrics ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* M. Kobalczyk, N. Astorga, T. Liu, and M. van der Schaar (2025)Active task disambiguation with LLMs.In International Conference on Learning Representations,Cited by: [§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§7](#S7.p1.2 "7 Conclusion ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* L. Kuhn, Y. Gal, and S. Farquhar (2023)CLAM: selective clarification for ambiguous questions with generative language models.In ICML 2023 Workshop on Structured Probabilistic Inference and Generative Modeling,Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* B. Z. Li, B. Kim, and Z. Wang (2025)QuestBench: can models ask the right questions?.arXiv preprint arXiv:2503.22674.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* D. V. Lindley (1956)On a measure of the information provided by an experiment.Annals of Mathematical Statistics 27 (4),  pp. 986–1005.External Links: [Document](https://dx.doi.org/10.1214/aoms/1177728069 "")Cited by: [§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* G. Liu et al. (2025)CaRT: teaching LLM agents to know when they know enough.arXiv preprint arXiv:2510.08517.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. (2024)AgentBench: evaluating LLMs as agents.In International Conference on Learning Representations,Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* E. Lumer, A. Gulati, F. Nizar, D. Hedroits, A. Mehta, H. Hwangbo, V. K. Subbiah, P. H. Basavaraju, and J. A. Burke (2026a)Tool and agent selection for large language model agents in production: a survey.Preprints.External Links: [Document](https://dx.doi.org/10.20944/preprints202512.1050.v2 ""),[Link](https://doi.org/10.20944/preprints202512.1050.v2 "")Cited by: [§1](#S1.p1.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* E. Lumer, A. Gulati, V. K. Subbiah, P. H. Basavaraju, and J. A. Burke (2025a)Memtool: optimizing short-term memory management for dynamic tool calling in llm agent multi-turn conversations.arXiv preprint arXiv:2507.21428.Cited by: [§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* E. Lumer, A. Gulati, V. K. Subbiah, P. H. Basavaraju, and J. A. Burke (2025b)Scalemcp: dynamic and auto-synchronizing model context protocol tools for llm agents.In International Joint Conference on Computational Intelligence, pp. 23–42.Cited by: [§4.2](#S4.SS2.p2.1 "4.2 Benchmarks and Models ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* E. Lumer, F. Nizar, A. Gulati, P. H. Basavaraju, and V. K. Subbiah (2025c)Tool-to-agent retrieval: bridging tools and agents for scalable llm multi-agent systems.arXiv preprint arXiv:2511.01854.Cited by: [§4.2](#S4.SS2.p2.1 "4.2 Benchmarks and Models ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* E. Lumer, F. Nizar, A. Jangiti, K. Frank, A. Gulati, M. Phadate, and V. K. Subbiah (2026b)Don’t break the cache: an evaluation of prompt caching for long-horizon agentic tasks.arXiv preprint arXiv:2601.06007.Cited by: [§1](#S1.p1.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* G. Mark, D. Gudith, and U. Klocke (2008)The cost of interrupted work: more speed and stress.In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 107–110.Cited by: [§1](#S1.p2.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom (2023)GAIA: a benchmark for general AI assistants.arXiv preprint arXiv:2311.12983.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. Min, V. Zhong, L. Zettlemoyer, and H. Hajishirzi (2020)AmbigQA: answering ambiguous open-domain questions.In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5783–5797.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. Y. Min, Y. Wu, J. Sun, M. Kaufmann, F. Tajwar, Y. Bisk, and R. Salakhutdinov (2025)Self-regulation and requesting interventions.arXiv preprint arXiv:2502.04576.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* J. Pan et al. (2025)When benchmarks talk: re-evaluating code generation with interactive tasks.arXiv preprint arXiv:2502.18413.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* B. Plaut, H. Zhu, and S. Russell (2025)Avoiding catastrophe in online learning by asking for help.In International Conference on Machine Learning,Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* G. Pu, M. S. Lee, U. M. Sehwag, D. J. Lee, B. Zhu, Y. Maurya, M. Raghavendra, Y. Xue, and S. M. Denton (2026)LHAW: controllable underspecification for long-horizon tasks.arXiv preprint arXiv:2602.10525.Cited by: [§A.10](#A1.SS10.p1.1 "A.10 Dataset Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§A.11.4](#A1.SS11.SSS4.p1.1 "A.11.4 Reproducibility Parameters ‣ A.11 Injection Protocol Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§1](#S1.p2.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§4.2](#S4.SS2.p1.1 "4.2 Benchmarks and Models ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§5.4](#S5.SS4.p1.1 "5.4 Natural Ask Overlay ‣ 5 Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* C. Qian et al. (2025)UserBench: an interactive gym environment for user-centric agents.arXiv preprint arXiv:2507.22034.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. Rao and H. Daumé III (2018)Learning to ask good questions: ranking clarification questions using neural expected value of perfect information.In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2737–2746.Cited by: [§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. Ross, G. J. Gordon, and J. A. Bagnell (2011)A reduction of imitation learning and structured prediction to no-regret online learning.In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 627–635.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. J. Russell and E. Wefald (1991)Do the right thing: studies in limited rationality. MIT Press, Cambridge, MA.Cited by: [§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* Scale AI (2025)SWE-Bench Pro: a more challenging benchmark for real-world software engineering.arXiv preprint arXiv:2509.16941.Cited by: [§4.2](#S4.SS2.p4.1 "4.2 Benchmarks and Models ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. Sen, E. Lumer, A. Gulati, and V. K. Subbiah (2026)Chronos: temporal-aware conversational agents with structured event retrieval for long-term memory.arXiv preprint arXiv:2603.16862.Cited by: [§7](#S7.p1.2 "7 Conclusion ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* B. Settles (2009)Active learning literature survey.Technical reportTechnical Report TR-1648, University of Wisconsin-Madison, Department of Computer Sciences.Cited by: [§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* M. Suri, P. Mathur, N. Lipka, F. Dernoncourt, R. A. Rossi, and D. Manocha (2025)Structured uncertainty guided clarification for LLM agents.arXiv preprint arXiv:2511.08798.Cited by: [§2.2](#S2.SS2.p1.1 "2.2 Theoretical Foundations Predict Timing Should Matter ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* A. Tamkin, K. Handa, A. Shrestha, and N. Goodman (2023)Task ambiguity in humans and language models.In International Conference on Learning Representations,Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* H. Trivedi, T. Khot, M. Hartmann, R. Manber, V. Dong, E. Li, S. Harber, H. Fang, A. Mishra, D. Radev, and A. Sabharwal (2024)AppWorld: a controllable world of apps and people for benchmarking interactive coding agents.In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Note: Best Resource Paper AwardCited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. Wu, M. Galley, B. Peng, H. Cheng, G. Li, Y. Dou, W. Cai, J. Zou, J. Leskovec, and J. Gao (2025)CollabLLM: from passive responders to active collaborators.In International Conference on Machine Learning,Note: Outstanding Paper AwardCited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shi, Z. Lu, et al. (2024)OSWorld: benchmarking multimodal agents for open-ended tasks in real computer environments.In Advances in Neural Information Processing Systems,Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* F. F. Xu, Y. Shi, Y. Lin, X. Yang, T. Xie, B. Wang, H. Peng, S. Yao, Q. Shi, S. Wang, et al. (2024)TheAgentCompany: benchmarking LLM agents on consequential real world tasks.arXiv preprint arXiv:2412.14161.Cited by: [§1](#S1.p1.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"),[§4.2](#S4.SS2.p3.1 "4.2 Benchmarks and Models ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. Yao et al. (2024)$\tau$-Bench: a benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045.Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023)ReAct: synergizing reasoning and acting in language models.In International Conference on Learning Representations,Cited by: [§1](#S1.p1.1 "1 Introduction ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* X. Zhang, Y. Deng, Z. Ren, S. Ng, and T. Chua (2024)Ask-before-plan: proactive language agents for real-world planning.In Findings of the Association for Computational Linguistics: EMNLP 2024,Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").
* S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig (2024)WebArena: a realistic web environment for building autonomous agents.In International Conference on Learning Representations,Cited by: [§2.1](#S2.SS1.p1.1 "2.1 Clarification in Agent Benchmarks ‣ 2 Related Work ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?").

Appendix A Extended Results
---------------------------

### A.1 Per-Benchmark VOI Curves

Table[4](#A1.T4 "Table 4 ‣ A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") presents the complete pass@3 values for all (benchmark, dimension, condition) cells. Figures[6](#A1.F6 "Figure 6 ‣ A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")–[8](#A1.F8 "Figure 8 ‣ A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") visualize these as per-benchmark VOI curves.

*Table 4: Complete pass@3 results by benchmark, dimension, and condition. $n$ denotes injection-condition (variant, model) units per dimension; only pairs for which all five injection conditions and the oracle baseline completed successfully are included (pairs with failures in any condition are excluded). For multi-dimension variants (Table[10](#A1.T10 "Table 10 ‣ A.10 Dataset Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")), each pair is attributed to its primary dimension only. Oracle and NC cells may include up to 4 additional baseline-only units (per-cell $n$ for Oracle/NC is therefore $n+k$ where $k\in[0,4]$; exact counts are available in the released code). TheAgentCompany uses the 3-model subset (see Section[4.2](#S4.SS2 "4.2 Benchmarks and Models ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")).*

| Benchmark | Dimension | Oracle | Inj-10 | Inj-30 | Inj-50 | Inj-70 | Inj-90 | NC |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MCP-Atlas | Goal ($n{\=}18$) | 0.80 | 0.78 | 0.50 | 0.44 | 0.39 | 0.39 | 0.40 |
| | Constraint ($n{\=}23$) | 0.12 | 0.17 | 0.13 | 0.13 | 0.13 | 0.13 | 0.12 |
| Input ($n{\=}28$) | 0.57 | 0.46 | 0.36 | 0.36 | 0.32 | 0.25 | 0.33 |
| Context ($n{\=}5$) | 0.80 | 0.80 | 0.60 | 0.60 | 0.40 | 0.60 | 0.60 |
| TAC | Goal ($n{\=}65$) | 0.20 | 0.11 | 0.08 | 0.10 | 0.08 | 0.12 | 0.12 |
| | Constraint ($n{\=}15$) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Input ($n{\=}50$) | 0.29 | 0.23 | 0.21 | 0.25 | 0.21 | 0.21 | 0.19 |
| SWE-Bench Pro | Goal ($n{\=}21$) | 0.83 | 0.86 | 0.81 | 0.67 | 0.81 | 0.71 | 0.46 |
| | Constraint ($n{\=}31$) | 0.81 | 0.81 | 0.74 | 0.74 | 0.74 | 0.68 | 0.56 |
| Input ($n{\=}24$) | 0.83 | 1.00 | 0.92 | 0.79 | 0.79 | 0.75 | 0.63 |
| Context ($n{\=}12$) | 0.67 | 0.92 | 0.92 | 0.67 | 0.92 | 0.92 | 0.75 |

<img src='2605.07937v1/figures/fig_appendix_voi_mcp-atlas.png' alt='Refer to caption' title='' width='598' height='147' />

*Figure 6: MCP-Atlas VOI curves by dimension (expanded view with per-model error indicators; cf. the pooled summary in main text Figure[3](#S4.F3 "Figure 3 ‣ 4.4 Natural Ask Protocol ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). Goal shows dramatic front-loading (0.78 at 10% vs. 0.40 NC), input shows gradual decline, constraint is flat near zero, and context (limited $n{\=}5$) shows strong early benefit. Dashed: no-clarification baseline; dotted: oracle.*

<img src='2605.07937v1/figures/fig_appendix_voi_theagentcompany.png' alt='Refer to caption' title='' width='598' height='196' />

*Figure 7: TheAgentCompany VOI curves by dimension. Success rates are substantially lower than MCP-Atlas (oracle $\leq$29% for non-constraint dimensions). Goal shows no reliable timing effect (all conditions within 4 percentage points of baseline). Input shows a mild benefit at intermediate timings (21–25% at Inj-30/50/70 vs. 19% NC). Constraint is at floor (0% across all conditions).*

<img src='2605.07937v1/figures/fig_appendix_voi_swe-bench_pro.png' alt='Refer to caption' title='' width='598' height='147' />

*Figure 8: SWE-Bench Pro VOI curves by dimension, pooled across three models ($n{\=}12$–31 per injection cell). All four dimensions show injection benefit over no-clarification, with input and goal exhibiting the clearest declining VOI pattern. Context has limited variant diversity ($n{\=}12$).*

Benchmark-specific interpretation. The three benchmarks represent a spectrum of task characteristics that explain their distinct VOI patterns:

* •

    MCP-Atlas tasks are short (6–20 actions), tool-focused, and typically have a single correct approach. This creates high commitment rates: choosing the wrong tool chain in the first few actions wastes most of the trajectory. Goal clarification is therefore extremely front-loaded.

* •

    TheAgentCompany tasks are longer (6–49 actions) and involve multiple interacting systems (web, database, file system). The lower baselines reflect genuine task difficulty; even oracle-informed agents achieve only 20–29% pass@3 on non-constraint dimensions. The weak timing signal for goal suggests that these tasks have inherently high stochasticity: success depends on factors beyond information completeness (e.g., correct navigation of web interfaces).

* •

    SWE-Bench Pro tasks vary enormously in difficulty (1–121 actions). Input shows a gradual decline from 1.00 (Inj-10) to 0.75 (Inj-90), consistent with code-generation patterns: once an agent commits to an implementation approach (choosing file locations, function signatures, test strategies), switching costs increase steadily.

### A.2 Natural Ask Timing Details

The 300 natural-ask sessions span seven TheAgentCompany base tasks. Table[5](#A1.T5 "Table 5 ‣ A.2 Natural Ask Timing Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") summarizes per-model ask behavior.

*Table 5: Natural ask behavior by model (100 sessions each). Ask rate is the fraction of sessions with $\geq$1 ask call. Mean timing is the trajectory-relative position of the first ask.*

| Model | Ask Rate | Total Calls | Mean Timing | Median Timing |
| --- | --- | --- | --- | --- |
| GPT-5.2 | 52% | 89 | 43% | 50% |
| Claude Sonnet 4.5 | 23% | 23 | 50% | 50% |
| Gemini 3 Flash | 0% | 0 | – | – |

GPT-5.2 averages 1.71 ask calls per asking session (max 5), indicating a pattern of iterative clarification. Claude always asks exactly once when it asks at all, suggesting a more deliberate single-query strategy. The distribution of first-ask timings is concentrated: 72% of all first-ask events occur between 25% and 50% of the trajectory.

<img src='2605.07937v1/figures/fig_appendix_natural_ask_timing_dist.png' alt='Refer to caption' title='' width='598' height='225' />

*Figure 9: Distribution of first-ask timing by model. GPT-5.2 asks frequently (52% of sessions) with timing concentrated between 25–50% of the trajectory. Claude asks selectively (23%) with a tight cluster around 50%. Gemini never asks. Dashed lines indicate mean first-ask timing.*

<img src='2605.07937v1/figures/fig_appendix_natural_ask_per_task.png' alt='Refer to caption' title='' width='598' height='271' />

*Figure 10: Per-task ask rate across all models. Task complexity appears to influence asking propensity independently of model architecture, with data-analysis tasks eliciting higher ask rates than simpler administrative tasks.*

Qualitative analysis of ask content. GPT-5.2’s questions tend to be specific and actionable (e.g., “Should I use the FY25 Q1 data or the calendar year Q1?”), while Claude’s questions are broader (e.g., “Could you clarify what format you’d like the output in?”). Despite asking far more often, GPT-5.2 achieves substantially lower per-session success (3%) than Claude (11%), indicating that asking frequency does not predict task success. Claude’s selective single-question strategy outperforms GPT-5.2’s iterative approach, suggesting that question quality and selectivity may matter more than volume, though we cannot isolate this from other model-level differences. This is consistent with the VOI framework: later and repeated questions yield diminishing returns, and poorly targeted questions may provide no actionable benefit even when asked at an appropriate trajectory point.

### A.3 Wasted Compute by Benchmark

Table[6](#A1.T6 "Table 6 ‣ A.3 Wasted Compute by Benchmark ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") reports mean wasted compute fractions by benchmark and injection timing. On TheAgentCompany, wasted compute follows a clean linear pattern reflecting the longer trajectories where waste accumulates gradually. On MCP-Atlas, the higher baseline (38% even at injection-10) reflects shorter trajectories where even a single early action pursuing the wrong path counts as waste relative to the oracle trace.

*Table 6: Wasted compute by benchmark and injection timing. MCP-Atlas and TheAgentCompany report the fraction of pre-injection actions absent from the oracle trace; SWE-Bench Pro reports absolute action counts due to longer, variable-length trajectories.*

| Benchmark | Inj-10 | Inj-30 | Inj-50 | Inj-70 | Inj-90 |
| --- | --- | --- | --- | --- | --- |
| TheAgentCompany (%) | 0.0 | 9.6 | 13.7 | 17.6 | 21.7 |
| MCP-Atlas (%) | 38.4 | 39.7 | 49.2 | 52.6 | 52.9 |
| SWE-Bench Pro (actions) | 0.7 | 4.3 | 5.0 | 8.3 | 10.4 |

The MCP-Atlas pattern of high initial waste (38% at injection-10) that plateaus at 53% reflects a structural property of short tool-calling trajectories: even early injection requires the agent to restart its approach, and the first 1–2 actions are often already committed. In contrast, TheAgentCompany’s linear growth from 0% reflects that the first few actions in enterprise workflows (opening a browser, navigating to a page) are typically approach-independent and not wasted regardless of the eventual clarification.

### A.4 Cross-Model Correlation Details

<img src='2605.07937v1/figures/fig_appendix_kendall_tac.png' alt='Refer to caption' title='' width='598' height='522' />

<img src='2605.07937v1/figures/fig_appendix_kendall_combined.png' alt='Refer to caption' title='' width='598' height='520' />

*Figure 11: Kendall $\tau$ correlation heatmaps. Left: TheAgentCompany only (3 models, balanced variant set). Right: all benchmarks combined (4 frontier models). All $p$-values $<0.01$ except where noted.*

Table[7](#A1.T7 "Table 7 ‣ A.4 Cross-Model Correlation Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") reports the $p$-values for all model-pair correlations. For the combined dataset, all 4-model pairs achieve significance at $p<0.01$. For the TAC-only analysis, all pairs are significant at $p<10^{-5}$, reflecting the larger number of comparable variant-timing combinations (each model runs all 30 TAC variants across 5 injection conditions).

*Table 7: Kendall $\tau$ $p$-values for all model pairs. Combined: all benchmarks (4 models). TAC: TheAgentCompany only (3 models, filtered to complete-data pairs).*

|  | Claude | Gemini | GPT |
| --- | --- | --- | --- |
| Claude | – | $<10^{-9}$ | $<10^{-4}$ |
| Gemini |  | – | $<10^{-3}$ |
| GPT |  |  | – |

*(a) TAC $p$-values*

|  | Claude | DeepSeek | Gemini | GPT |
| --- | --- | --- | --- | --- |
| Claude | – | $<10^{-3}$ | $<10^{-9}$ | $<10^{-4}$ |
| DeepSeek |  | – | $<10^{-3}$ | $<0.01$ |
| Gemini |  |  | – | $<10^{-3}$ |
| GPT |  |  |  | – |

*(b) Combined $p$-values*

The higher TAC correlations (0.78–0.87) compared to the combined dataset (0.34–0.67) likely reflect two factors: (1) the TAC subset uses exactly the same variant set and injection protocol across all three models (ensuring apple-to-apple comparison), and (2) the combined dataset includes MCP-Atlas with 4 models (introducing DeepSeek which is only available on that benchmark) and SWE-Bench Pro where high overall pass rates compress the VOI range, attenuating rank correlations.

### A.5 Point of No Return Analysis

<img src='2605.07937v1/figures/fig_appendix_ponr.png' alt='Refer to caption' title='' width='419' height='248' />

*Figure 12: Latest useful injection point by dimension and ambiguity class. Only outcome-critical variants for goal and constraint dimensions show a statistically significant recovery point (at 30% of trajectory). Blank bars indicate no significant point of no return was detected.*

Table[8](#A1.T8 "Table 8 ‣ A.5 Point of No Return Analysis ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") reports the point-of-no-return analysis. We define the “latest useful injection point” as the latest timing at which pass@3 significantly exceeds the no-clarification baseline (one-sided permutation test with 10,000 permutations, $p<0.05$, Bonferroni-corrected for 5 timing comparisons per cell).

*Table 8: Point of no return: latest injection timing with significant improvement over no-clarification ($p<0.05$, Bonferroni-corrected). “–” indicates no timing achieves significance.*

| Dimension | Outcome-Critical | Divergent | Benign |
| --- | --- | --- | --- |
| Goal | 30% | – | – |
| Constraint | 30% | – | – |
| Input | – | – | – |
| Context | – | – | – |

The absence of significant points for input and context dimensions is consistent with the VOI curves: input shows gradual decline (never sharply crossing the baseline), and context has too few samples ($n{\=}5$) for statistical power. For goal and constraint, the 30% cutoff aligns with the commitment model’s prediction that these dimensions have steep early commitment rates.

### A.6 Qualitative Trajectory Analysis

To illustrate how timing interacts with agent behavior, we present representative trajectory patterns for each dimension:

Goal (MCP-Atlas, early injection helps). A task requires exporting data “in a format suitable for spreadsheet analysis.” Without clarification, all three models default to JSON output. With goal clarification at 10% (“the user wants CSV format”), the agent correctly outputs CSV from the start. At 50%, two of three models have already committed to JSON parsing pipelines and must discard 3–5 actions to restart. At 90%, all models have completed the task in the wrong format.

Input (MCP-Atlas, gradual decline). A task requires querying “the customer database” but doesn’t specify which table contains customer records. Without clarification, agents explore the schema (listing tables, examining columns), eventually finding the correct table through trial and error. This exploration is “wasted” relative to the oracle trace but is informationally productive, as it narrows the search space. Early injection eliminates this exploration; late injection still saves the agent from querying the wrong table at the final step.

Constraint (TAC, disruptive injection). A task asks for a budget report but omits that “all figures must be in thousands.” Without this constraint, agents produce correct reports in raw units. When the constraint is injected at 50%, agents attempt to retroactively scale existing outputs, sometimes introducing rounding errors or formatting inconsistencies that reduce checkpoint scores below the unconstrained baseline.

### A.7 SWE-Bench Pro Limitations

SWE-Bench Pro results warrant additional caveats beyond those noted in Section[6](#S6 "6 Limitations ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?"):

1. 1.

    Moderate sample sizes. Pooling across three models yields $n{\=}12$–31 (variant, model) units per injection cell depending on dimension. Per-model estimates remain noisy, but the pooled values in Table[4](#A1.T4 "Table 4 ‣ A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") are more stable than single-model estimates.

2. 2.

    Model coverage. Three models (Gemini 3 Flash, GPT-5.2, Claude Sonnet 4.5) have both injection and baseline data, enabling within-model comparisons. DeepSeek V3.2 is excluded from SWE-Bench Pro.

3. 3.

    Context dimension. Only 4 context-type variants are available in LHAW’s SWE-Bench Pro subset, yielding $n{\=}12$ units when pooled across models. The resulting estimates should be interpreted with caution given the limited variant diversity.

Additionally, multiple SWE-Bench Pro cells exhibit values above the Oracle baseline: Goal Inj-10 (0.86 vs. Oracle 0.83), Input Inj-10 (1.00 vs. 0.83), Input Inj-30 (0.92 vs. 0.83), Context Inj-10 (0.92 vs. 0.67), and Context NC (0.75 vs. 0.67). At sample sizes of $n{\=}12$–31 per cell, these inversions are consistent with sampling variability. An additional contributing factor is that the injection messages provide explicit, structured guidance (e.g., “the target format is CSV”) that may be more salient than the same information embedded in the original task prompt; this is a known artifact of forced-injection protocols. None of these differences is statistically significant. We retain Oracle as the conceptual upper bound while noting that small-sample noise can produce localized inversions.

Despite these limitations, SWE-Bench Pro contributes to the aggregate picture by confirming that code-generation tasks exhibit steep commitment rates (consistent with the goal-dimension pattern on MCP-Atlas) and that oracle information substantially improves performance (0.81–0.83 oracle vs. 0.46–0.63 no-clarification for goal, constraint, and input dimensions across three models; see Table[4](#A1.T4 "Table 4 ‣ A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")).

### A.8 Total Runs and Computational Cost

Table[9](#A1.T9 "Table 9 ‣ A.8 Total Runs and Computational Cost ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") summarizes the total experimental scope.

*Table 9: Total experimental trials by benchmark and protocol. Each forced-injection trial corresponds to one (variant, model, condition, trial-index) execution. Natural-ask sessions are single-trial.*

| Benchmark | Forced Injection Trials | Natural Ask Sessions | Total Trials |
| --- | --- | --- | --- |
| MCP-Atlas | 3,024 | – | 3,024 |
| TheAgentCompany | 1,890 | 300 | 2,190 |
| SWE-Bench Pro | 1,134 | – | 1,134 |
| Total | 6,048 | 300 | 6,348 |

### A.9 Aggregate VOI Interpretation

<img src='2605.07937v1/figures/fig3_voi_aggregate.png' alt='Refer to caption' title='' width='598' height='477' />

*Figure 13: Aggregate VOI curves by information dimension (all benchmarks combined). Goal clarification is front-loaded, input shows gradual decline, and constraint injection provides only marginal benefit over no-clarification. Dashed lines indicate oracle (upper) and no-clarification (lower) baselines. See main text Figure[3](#S4.F3 "Figure 3 ‣ 4.4 Natural Ask Protocol ‣ 4 Experimental Design ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") for the per-benchmark MCP-Atlas curves that drive the strongest signals.*

The aggregate curves in Figure[13](#A1.F13 "Figure 13 ‣ A.9 Aggregate VOI Interpretation ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") pool across benchmarks with substantially different baseline performance levels, weighting each (variant, model) unit equally. Because TAC contributes the majority of goal units (63%) with near-floor baselines, the aggregate attenuates the stronger timing signals visible in the per-benchmark curves (Figures[6](#A1.F6 "Figure 6 ‣ A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")–[8](#A1.F8 "Figure 8 ‣ A.1 Per-Benchmark VOI Curves ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")). The per-benchmark breakdowns should be consulted alongside this summary view.

### A.10 Dataset Details

Table[10](#A1.T10 "Table 10 ‣ A.10 Dataset Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") provides the full cross-tabulation of the 84 task variants used in this study, broken down by benchmark, information dimension(s), and ambiguity class. The subset is drawn from a larger pool of 101 variants in LHAW*[Pu et al., [2026](#bib.bib1 "LHAW: controllable underspecification for long-horizon tasks")]*; 17 SWE-Bench Pro variants are excluded to manage computational costs. Selection was stratified to preserve dimension balance: excluded variants do not differ systematically from retained ones on dimension distribution or ambiguity class (see Section[6](#S6 "6 Limitations ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?")).

*Table 10: Task variant breakdown by benchmark, dimension, and ambiguity class. Multi-dimension variants (e.g., “goal+input”) have multiple segments removed simultaneously. Totals: 36 MCP-Atlas, 30 TheAgentCompany, 18 SWE-Bench Pro.*

| Benchmark | Dimension(s) | Outcome-Crit. | Divergent | Benign | Total |
| --- | --- | --- | --- | --- | --- |
| MCP-Atlas | goal | 6 | 1 | 0 | 7 |
| | input | 5 | 4 | 0 | 9 |
| constraint | 5 | 4 | 2 | 11 |
| context / context+goal | 0 | 1 | 2 | 3 |
| multi (goal+input, constr.+input, constr.+goal) | 2 | 1 | 3 | 6 |
| TAC | goal | 6 | 2 | 3 | 11 |
| | input | 2 | 4 | 0 | 6 |
| constraint / constr.+goal | 2 | 0 | 1 | 3 |
| multi (goal+input, constr.+input) | 4 | 5 | 1 | 10 |
| SWE-Bench Pro | goal+input / input | 1 | 2 | 2 | 5 |
| | constraint / constr.+goal | 2 | 2 | 2 | 6 |
| context / context+goal | 0 | 1 | 2 | 3 |
| constr.+input / constr.+context | 2 | 1 | 1 | 4 |
| Total | | 37 | 28 | 19 | 84 |

### A.11 Injection Protocol Details

#### A.11.1 Injection Message Construction

Injection messages are constructed from a template system that maps each (dimension, subdimension) pair to a natural-language message pattern. Table[11](#A1.T11 "Table 11 ‣ A.11.1 Injection Message Construction ‣ A.11 Injection Protocol Details ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") shows representative templates. For variants with multiple removed segments, the first segment uses the full template and subsequent segments are appended with connective phrases (“And {value}.” or “Also, {value}.”).

*Table 11: Representative injection message templates by dimension and subdimension. The {value} placeholder is filled with the ground-truth content from the LHAW variant specification.*

| Dimension | Subdimension | Template |
| --- | --- | --- |
| goal | target | “By the way, I should clarify: I’m specifically looking for {value}.” |
| goal | format | “By the way, please give me the result in {value}.” |
| input | source | “By the way, you can find the data in {value}.” |
| input | location | “By the way, it’s located at {value}.” |
| constraint | temporal | “By the way, I should have mentioned: I’m looking at {value}.” |
| constraint | selection | “I should mention, only include those that are {value}.” |
| context | background | “By the way, for context: {value}.” |
| context | domain knowledge | “For context, {value}.” |
| (default fallback) | | “By the way, I should have mentioned: {value}.” |

Example injection messages from actual experiments:

* •

    *Goal (format):* “By the way, please give me the result in CSV format.”

* •

    *Constraint (temporal + selection):* “By the way, I should have mentioned: I’m looking at last month of 2022. Also, 0.50.”

* •

    *Input (source):* “By the way, you can find the data in the shared_drive/Q3_budget folder.”

#### A.11.2 ask_user Tool Definition

For natural-ask experiments, the ask_user tool is provided to models with the following schema (OpenAI function-calling format):

```

{
  "name": "ask_user",
  "description": "Ask the user a clarifying question to get
    more information about the task. Use this when the task
    is ambiguous or you need specific details to proceed.",
  "parameters": {
    "properties": {
      "question": {"type": "string",
        "description": "The clarifying question to ask."},
      "context": {"type": "string",
        "description": "Optional additional context.",
        "default": ""}
    },
    "required": ["question"]
  }
}

```

Additionally, the task prompt is extended with:

> IMPORTANT: Your output will be checked by an auto-grader looking for exact answers. This task may be missing critical information. Use the ask_user tool to ask the user for any missing details.

When the model invokes ask_user, a simulated user responds with the ground-truth removed_segments content, matching the injection message that would be used in the forced-injection condition.

#### A.11.3 Oracle Budget Calibration

Injection timing is defined relative to an oracle-calibrated action budget $B_{m,v}$ for each (model $m$, variant $v$) pair. The budget is computed as the mean trajectory length across 3 oracle-condition trials, rounded to the nearest integer:

|  | $B_{m,v}\=\text{round}!\left(\frac{1}{3}\sum_{t\=1}^{3}|\tau^{\text{oracle}}_{m,v,t}|\right)$ |  |
| --- | --- | --- |

The injection action for timing fraction $f\in{0.1,0.3,0.5,0.7,0.9}$ is:

|  | $a_{\text{inject}}\=\max!\bigl(1,\;\lfloor B_{m,v}\cdot f\rfloor\bigr)$ |  |
| --- | --- | --- |

Budget ranges reflect task diversity: MCP-Atlas tasks are short (6–20 actions), TheAgentCompany tasks are moderate (6–49 actions), and SWE-Bench Pro tasks can require extended code navigation and editing (1–121 actions).

#### A.11.4 Reproducibility Parameters

All models are queried with temperature $\=0.0$ (greedy decoding) and a maximum output length of 4,096 tokens per turn. The three trials per (variant, model, condition) cell use fixed random seeds (0, 1, 2) for any stochastic API parameters. Total API cost for the 6,348 experimental runs was approximately $4,200 USD across all providers. Task success is determined by the LHAW automated grader*[Pu et al., [2026](#bib.bib1 "LHAW: controllable underspecification for long-horizon tasks")]*, which compares agent outputs against ground-truth answers using exact-match or rubric-based criteria depending on the benchmark (MCP-Atlas: tool-output match; TheAgentCompany: checkpoint scoring; SWE-Bench Pro: test-suite pass).

### A.12 Example Trajectory

Table[12](#A1.T12 "Table 12 ‣ A.12 Example Trajectory ‣ Appendix A Extended Results ‣ Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?") illustrates the structure of a single forced-injection trial. This MCP-Atlas example (variant 688ba1...::V_S1_delete, GPT-5.2, injection at 10%) shows a 7-action trajectory with the injection arriving as a user message after step 1.

*Table 12: Abridged trial trace (MCP-Atlas, injection at 10%, GPT-5.2). The injection message arrives as a synthetic user turn between steps 1 and 2. All subsequent actions incorporate the clarified information.*

| Step | Phase | Action | Purpose |
| --- | --- | --- | --- |
| 1 | PRE | get_config | Environment setup |
| .*Injection: “By the way, I should have mentioned:* | | | |
| *I’m looking at first crime I had registered.”*. | | | |
| 2 | POST | list_directory | Locate data files |
| 3 | POST | start_process | Launch data tool |
| 4 | POST | start_process | Execute query |
| 5 | POST | search_pubmed | Cross-reference |
| 6 | POST | search_pubmed | Refine search |
| 7 | POST | search_pubmed | Extract answer |
| Result: task_success \= True, total_actions \= 7 | | | |
| --- | --- | --- | --- |

Each trial JSON records: variant_id, model, condition, injection_point, actions (with per-step action_name, parameters, result, is_pre_injection), conversation (full chat history including the injection turn), task_success, total_actions, pre_injection_actions, post_injection_actions, and duration_seconds. All trial data will be released with the code.


Instructions for reporting errors
---------------------------------

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile
 support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the
 methods listed below:


**Tip:** You can select the relevant text first, to include it in your report.

Our team has already identified [the following issues](https://github.com/arXiv/html_feedback/issues). We appreciate your time reviewing and reporting rendering errors we
 may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability
 should not be a barrier to accessing research. Thank you for your continued support in championing open access for
 all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a [list of packages that need conversion](https://github.com/brucemiller/LaTeXML/wiki/Porting-LaTeX-packages-for-LaTeXML), and welcome [developer contributions](https://github.com/brucemiller/LaTeXML/issues).

BETA