Title: When to Retrieve During Reasoning: Adaptive Retrieval for Large Reasoning Models
ArXiv: 2604.26649
Authors: Dongxin Guo, 0009-0000-2388-1072, The University of Hong Kong, Hong Kong, China, Brain Investing Limited, ,, Jikun Wu, 0000-0002-2327-4157, Stellaris AI Limited, and, Siu Ming Yiu, 0000-0002-3975-8500
Sections: 39
Estimated tokens: 20.7k

## Contents
- 1. Introduction
- 2. Related Work
  - 2.1. Retrieval-Augmented Generation
  - 2.2. Iterative and Adaptive Retrieval
  - 2.3. Large Reasoning Models
  - 2.4. Uncertainty Quantification for LLMs
- 3. Problem Formulation
- 4. Method: ReaLM-Retrieve
  - 4.1. Reasoning Step Segmentation
  - 4.2. Step-Level Uncertainty Detection
  - 4.3. Retrieval Intervention Policy
  - 4.4. Efficient Retrieval Integration
  - 4.5. Complexity Analysis
- 5. Experimental Setup
  - 5.1. Datasets
  - 5.2. Baselines
  - 5.3. Models and Retrieval
  - 5.4. Evaluation Metrics
  - 5.5. Implementation Details
- 6. Results
  - 6.1. Main Results
  - 6.2. Efficiency Analysis
  - 6.3. Ablation Studies
  - 6.4. Analysis by Question Complexity
  - 6.5. Retrieval Timing Analysis
  - 6.6. Retrieval Quality Analysis
  - 6.7. Generalization to Other Reasoning Models
  - 6.8. Cross-Benchmark Generalization
  - 6.9. Token Efficiency and Behavioral Analysis
- 7. Discussion
  - 7.1. When Does Retrieval Help Reasoning?
  - 7.2. Comparison to Contemporary Work
  - 7.3. Failure Mode Analysis
  - 7.4. Deployment Considerations
  - 7.5. Limitations
  - 7.6. Broader Impact
- 8. Conclusion
  - Acknowledgements.
- References

## Abstract

Abstract. Large reasoning models such as DeepSeek-R1 and OpenAI o1 generate extended chains of thought spanning thousands of tokens, yet their integration with retrieval-augmented generation (RAG) remains fundamentally misaligned. Current RAG systems optimize for providing context before reasoning begins, while reasoning models require evidence injection during multi-step inference chains. We introduce ReaLM-Retrieve , a reasoning-aware retrieval framework that addresses this mismatch through three key innovations: (1) a step-level uncertainty detector that identifies knowledge gaps at reasoning-step granularity rather than token or sentence level; (2) a retrieval intervention policy that learns when external evidence maximally benefits ongoing reasoning; and (3) an efficiency-optimized integration mechanism that reduces per-retrieval overhead by 3.2 × \times compared to naive integration. Experiments on MuSiQue, HotpotQA, and 2WikiMultiHopQA demonstrate that ReaLM-Retrieve achieves on average 10.1% absolute improvement in answer F1 over standard RAG (range: 9.0–11.8% across the three benchmarks) while reducing retrieval calls by 47% compared to fixed-interval approaches like IRCoT (all improvements significant at p < 0.01 p<0.01 , paired bootstrap). On the challenging MuSiQue benchmark requiring 2–4 hop reasoning, our method achieves 71.2% F1 with an average of only 1.8 retrieval calls per question. Analysis shows that ReaLM-Retrieve also improves retrieval quality itself, achieving 81.3% Recall@5 with consistently higher precision and MRR than fixed-interval baselines on supporting evidence, establishing new state-of-the-art efficiency-accuracy trade-offs for reasoning-intensive retrieval tasks.

## 1. Introduction

The emergence of large reasoning models (LRMs) marks a major shift in language model capabilities. Models such as DeepSeek-R1 (Guo et al., 2025), OpenAI o1 (OpenAI, 2024), and QwQ (Qwen Team, 2024) generate extended chains of thought spanning 12,000–25,000 tokens, exhibiting emergent behaviors including self-verification, backtracking, and strategy switching. Unlike standard chain-of-thought (CoT) prompting (Wei et al., 2022), these models are trained via reinforcement learning to develop genuine multi-step reasoning capabilities that cannot be replicated through prompting alone.

However, a critical limitation persists: reasoning models excel at *reasoning with known facts* but struggle when *external factual knowledge* is required. Recent analyses reveal that reasoning models show *degraded* factual accuracy compared to their non-reasoning counterparts on fact-seeking tasks (Zeng et al., 2025), with increased hallucination rates on benchmarks like SimpleQA. This creates a fundamental opportunity for retrieval augmentation; yet existing RAG systems are fundamentally misaligned with reasoning model architectures.

The core mismatch is temporal: current RAG pipelines optimize for providing relevant context before generation begins (Gao et al., 2024; Lewis et al., 2020). This “retrieve-then-generate” paradigm assumes a single retrieval step suffices. For reasoning models generating thousands of tokens across multiple logical steps, this assumption fails catastrophically. A reasoning chain may begin with sufficient context but encounter knowledge gaps mid-inference, gaps that cannot be anticipated at query time.

Iterative retrieval methods such as IRCoT (Trivedi et al., 2023), FLARE (Jiang et al., 2023), and Self-RAG (Asai et al., 2024) address this by interleaving retrieval with generation. Yet these methods were designed for standard language models, not reasoning architectures, and exhibit fundamental incompatibilities: (1) Granularity mismatch: Existing methods trigger retrieval at token level (FLARE) or sentence level (IRCoT), while reasoning models operate at reasoning step granularity (logical units spanning multiple sentences). (2) Signal unavailability: Methods relying on token probabilities (FLARE) or internal attention states (DRAGIN (Su et al., 2024)) cannot operate with completion-only models like o1. (3) Efficiency collapse: Fixed-interval retrieval (IRCoT) becomes prohibitively expensive for reasoning chains spanning thousands of tokens, adding 2–5 seconds per retrieval call.

We propose ReaLM-Retrieve, a reasoning-aware retrieval framework that addresses these challenges through principled integration of retrieval into reasoning model inference. Our approach makes three key contributions:

Contribution 1: Step-Level Uncertainty Detection. We propose a novel uncertainty quantification method operating at reasoning-step granularity. Unlike token-level entropy or sentence-level confidence, our Reasoning Step Uncertainty Score (RSUS) identifies when a logical inference requires external knowledge versus when reasoning should continue with existing context.

Contribution 2: Learned Retrieval Intervention Policy. We formalize reasoning-retrieval integration as a sequential decision problem and learn an intervention policy $\pi(a_{t}|s_{t})$ that decides whether to retrieve, what query to formulate, and how to integrate retrieved content, all conditioned on the current reasoning state.

Contribution 3: Efficiency-Optimized Integration. We develop implicit compression and speculative caching mechanisms that reduce per-retrieval overhead by 3.2$\times$ compared to naive integration (from 2.1s to 0.66s per call), enabling practical deployment of reasoning-retrieval systems with 1.33$\times$ lower end-to-end latency than fixed-interval approaches.

Experiments on three multi-hop QA benchmarks show that ReaLM-Retrieve achieves substantial improvements in both accuracy and efficiency. On MuSiQue, we achieve 71.2% F1 with 1.8 average retrieval calls compared to IRCoT’s 65.4% F1 with 3.4 calls (improvement significant at $p<0.01$). Our method establishes a new Pareto frontier for reasoning-retrieval trade-offs, demonstrating that fewer, better-timed retrievals outperform frequent, fixed-interval retrievals.

## 2. Related Work

### 2.1. Retrieval-Augmented Generation

RAG systems (Borgeaud et al., 2022; Guu et al., 2020; Lewis et al., 2020) augment language models with external knowledge retrieved from document collections. The standard approach involves a single retrieval step before generation. Dense retrieval methods using dual encoders (Karpukhin et al., 2020; Xiong et al., 2021) or late interaction architectures (Khattab and Zaharia, 2020; Santhanam et al., 2022b) have substantially improved retrieval quality. Recent evaluation frameworks (ES et al., 2024) enable principled assessment of retrieval-generation pipelines. Multi-vector retrieval engines such as PLAID (Santhanam et al., 2022a) and WARP (Scheerer et al., 2025) optimize late-interaction retrievers, achieving 3–41$\times$ speedups. Our work complements these advances by optimizing when to invoke retrieval during reasoning.

### 2.2. Iterative and Adaptive Retrieval

Recent work explores iterative retrieval strategies. IRCoT (Trivedi et al., 2023) interleaves retrieval with chain-of-thought reasoning, retrieving after each sentence. While effective for short generation (achieving 15 F1-point improvements on 2WikiMultiHopQA), fixed-interval retrieval becomes impractical for reasoning chains spanning thousands of tokens. FLARE (Jiang et al., 2023) triggers retrieval when token probability falls below a threshold, but requires token-level probabilities unavailable from completion-only models. DRAGIN (Su et al., 2024) uses attention entropy but requires internal states. Self-RAG (Asai et al., 2024) learns special tokens through fine-tuning, achieving strong performance but requiring full model fine-tuning impossible for proprietary models. REPLUG (Shi et al., 2024) treats models as black boxes but operates at query-level without mid-reasoning intervention. A separate line of work makes the retrieval decision *at the query level* based on question complexity: Adaptive-RAG (Jeong et al., 2024) routes queries to no-retrieval, single-retrieval, or multi-retrieval strategies, and Open-RAG (Islam et al., 2024) learns reflection tokens that determine retrieval necessity per query. Most closely related to our setting, the concurrent work of Hashemi et al. (Hashemi et al., 2026) (“Dynamic Search-R1”) extends Search-R1 with a cost-aware RL objective that adapts retrieval *depth* (the number of documents per sub-query); their orthogonal axis (*how many to retrieve* per sub-query, learned via PPO/GRPO with token-cost penalties) is complementary to our axis (*when to retrieve* between reasoning steps, decided by RSUS-driven uncertainty rather than RL). Combining the two (adaptive timing with adaptive depth) is a natural future direction. Our work extends adaptive retrieval to reasoning-step granularity within extended inference chains, providing finer-grained timing signals than query-level approaches and operating without RL fine-tuning.

### 2.3. Large Reasoning Models

Large reasoning models represent a distinct class of model from standard LLMs. DeepSeek-R1 (Guo et al., 2025) uses pure RL with Group Relative Policy Optimization, generating reasoning chains of 12,000–23,000 tokens with emergent behaviors (self-verification, backtracking, hypothesis testing) that standard CoT prompting cannot replicate. Search-R1 (Jin et al., 2025) applies RL to train models for interleaved search decisions, achieving 41% improvement over RAG baselines, but focuses on when to search rather than retrieval system optimization. Our work addresses this gap with a systems-oriented approach to reasoning-retrieval integration.

### 2.4. Uncertainty Quantification for LLMs

Determining when retrieval is needed requires estimating model uncertainty. Semantic entropy (Kuhn et al., 2023) clusters sampled responses by meaning, achieving AUROC of 0.70–0.85 for predicting accuracy, but the 5–10$\times$ inference overhead compounds with expensive reasoning inference. Verbalized confidence (Tian et al., 2023) prompts models to express confidence, offering a black-box-compatible alternative. Self-consistency (Wang et al., 2023) uses agreement across reasoning chains but requires 5–20$\times$ sampling overhead. Lin et al. (Lin et al., 2024) propose a principled black-box uncertainty framework based on pairwise semantic similarity over multiple sampled generations, which can outperform verbalized confidence in calibration; we use verbalized confidence as one component of RSUS specifically because of its low overhead and complementary signal to entity-coverage entropy, but Lin et al.’s pairwise approach is a natural drop-in replacement for $U_{\text{verb}}$ when sampling cost is acceptable. Conformal-prediction approaches such as TRAQ (Li et al., 2024) provide statistical correctness guarantees end-to-end on RAG outputs; this is orthogonal to our problem of *when* to retrieve, but the two directions are composable. Moskvoretskii et al. (Moskvoretskii et al., 2025) find that simpler uncertainty methods can match complex pipelines while requiring ¡1% computational overhead. We design lightweight uncertainty signals specifically for reasoning architectures.

## 3. Problem Formulation

We formalize reasoning-retrieval integration as follows. Given a query $q$ and document collection $\mathcal{D}$, a reasoning model generates a chain $R=(r_{1},r_{2},\ldots,r_{n})$ where each $r_{i}$ represents a reasoning step, a coherent unit of logical inference that may span multiple sentences. The goal is to learn an intervention policy $\pi$ that at each step $r_{i}$ decides: (1) Retrieve decision: $a^{\text{ret}}_{i}\in\{0,1\}$ indicating whether to retrieve; (2) Query formulation: $q_{i}=f(q,r_{1:i})$ specifying the retrieval query; (3) Context integration: $c_{i}=g(r_{1:i},\mathcal{R}_{i})$ where $\mathcal{R}_{i}$ denotes retrieved documents. The objective is to maximize answer accuracy while minimizing retrieval cost:

$$ (1) $\max_{\pi}\mathbb{E}_{q\sim\mathcal{Q}}[\text{Acc}(a_{\pi},a^{*})-\lambda\cdot\text{Cost}(\pi)]$ $$

where $a_{\pi}$ is the answer produced under policy $\pi$, $a^{*}$ is the ground truth, and $\lambda$ balances accuracy against retrieval cost (number of calls and latency). Key insight: Unlike prior work treating this as a per-token or per-sentence decision, we operate at reasoning step granularity, matching the fundamental unit of reasoning model operation.

## 4. Method: ReaLM-Retrieve

ReaLM-Retrieve consists of three components: (1) reasoning step segmentation and uncertainty estimation, (2) a learned intervention policy, and (3) efficient retrieval integration. Figure [1](#S4.F1) provides an overview of the problem, our solution, and key results.

Figure: Figure 1. Overview of ReaLM-Retrieve’s adaptive retrieval approach. (a) The temporal mismatch: traditional RAG retrieves once before generation, but reasoning models encounter knowledge gaps during 12K–25K token chains. (b) Our four-stage pipeline detects step-level uncertainty and retrieves only when needed via learned policy. (c) MuSiQue benchmark results show 71.2% F1 with only 1.8 retrieval calls, outperforming IRCoT (65.4%, 3.4 calls) and establishing a new Pareto frontier.

### 4.1. Reasoning Step Segmentation

Reasoning models produce extended outputs that must be segmented into coherent reasoning steps. Unlike sentence-level segmentation used by IRCoT, we identify *logical reasoning units*: segments that complete a coherent inference before the next logical step begins. We employ a lightweight classifier trained on human-annotated reasoning traces to identify step boundaries. The classifier operates on a sliding window of tokens, predicting boundary probability based on: (1) Discourse markers (“therefore”, “however”, “this means”); (2) Logical connectives indicating inference completion; (3) Topic shifts detected via embedding similarity; (4) Punctuation and formatting patterns. For open-weight models (DeepSeek-R1, QwQ), we additionally use the model’s own reasoning structure markers when available.

Classifier Architecture and Training. The step boundary classifier is a 3-layer transformer encoder (hidden dimension 256, 4 attention heads) operating on 128-token sliding windows with 64-token stride. Training data consists of 2,847 human-annotated reasoning traces from DeepSeek-R1 outputs on a held-out subset of NaturalQuestions (disjoint from evaluation benchmarks). The three annotators were given written instructions to mark a boundary wherever the model (a) completes one logical inference and shifts to a new sub-claim, sub-question, or strategy; (b) introduces a discourse marker (“therefore”, “however”, “so”, “next”) signalling such a shift; or (c) closes a verification or backtracking episode. They were instructed *not* to mark boundaries at every sentence, and to prefer fewer, semantically coherent segments over finer ones. Annotators achieved $\kappa=0.78$ inter-annotator agreement on boundary placement. The classifier is trained for 10 epochs with cross-entropy loss (learning rate $5\times 10^{-5}$, batch size 32), achieving 94.2% F1 on a held-out NaturalQuestions test set ($N=412$ traces). Average segment length is 127 tokens compared to 23 tokens for sentence-level segmentation. An ablation removing each input feature class indicates that discourse markers (1) contribute most to boundary F1 ($-3.4$% F1 when removed), followed by logical connectives ($-2.1$%), embedding-based topic shift ($-1.6$%), and punctuation/formatting ($-0.8$%); the relatively large punctuation contribution reflects R1-style traces’ tendency to place line breaks at strategy transitions.

Out-of-domain segmentation generalization.
A reasonable concern is that the classifier, trained on NaturalQuestions traces, may not transfer to the multi-hop benchmarks used for end-to-end evaluation. To assess this, we collected reasoning traces produced by R1-Distill-Qwen-32B on $N=15$ randomly-sampled questions from each of MuSiQue, HotpotQA, and 2WikiMultiHopQA, and had a single annotator (using the same written instructions) mark boundaries for each trace. Table [1](#S4.T1) reports boundary F1 of the in-domain classifier on each benchmark relative to the in-domain NaturalQuestions test set. The classifier transfers with a modest degradation ($1.5$–$3.0$ F1 points), consistent with the intuition that reasoning-step structure is largely model-determined rather than benchmark-determined; multi-hop questions admit slightly more variability in boundary placement (lower IAA, not shown), explaining most of the residual gap. Segmentation errors on these benchmarks correlate weakly with downstream retrieval-decision errors ($r=0.21$), suggesting that small boundary-position drift does not propagate severely through RSUS.

**Table 1. Step-boundary classifier generalization. Boundary F1 against held-out human annotations on the four datasets ($N=15$ traces per OOD benchmark, $N=412$ for NQ).**
| Dataset | Boundary F1 | Segments / trace |
| --- | --- | --- |
| NaturalQuestions (in-domain) | 94.2 | 9.4 |
| MuSiQue | 91.4 | 8.7 |
| HotpotQA | 92.7 | 7.9 |
| 2WikiMultiHopQA | 91.2 | 8.3 |

### 4.2. Step-Level Uncertainty Detection

We define the *Reasoning Step Uncertainty Score* (RSUS), a composite measure identifying when a reasoning step requires external knowledge:

$$ (2) $\text{RSUS}(r_{i})=\alpha\cdot U_{\text{verb}}(r_{i})+\beta\cdot U_{\text{ent}}(r_{i})+\gamma\cdot U_{\text{cons}}(r_{i})$ $$

We chose this particular triple, rather than alternatives like semantic entropy (Kuhn et al., 2023) or pairwise-similarity black-box uncertainty (Lin et al., 2024), for three reasons. First, overhead: each of our components costs $<3\%$ of reasoning inference, whereas semantic entropy or self-consistency cost $5$–$20\times$ inference. With reasoning chains already at 12K–25K tokens, methods with multiplicative overhead are infeasible. Second, the three signals are complementary. $U_{\text{verb}}$ probes the model’s self-knowledge of factual correctness; $U_{\text{ent}}$ probes corpus coverage of named entities, an external signal independent of the model; $U_{\text{cons}}$ probes reasoning robustness. An ablation (Table [6](#S6.T6)) confirms each contributes orthogonally. Third, all three signals can be computed without token-level logprobs, so the framework operates uniformly on completion-only API models and on open-weight models. We do not claim this triple is optimal: richer alternatives such as Lin et al.’s pairwise-similarity uncertainty are drop-in substitutes for $U_{\text{verb}}$ when sampling cost is acceptable.

Verbalized uncertainty $U_{\text{verb}}$: We prompt the reasoning model at step boundaries: *“Given the reasoning so far, rate your confidence that the current conclusion is factually correct on a scale of 0-100, where 0 means completely uncertain and 100 means absolutely certain. Respond with only a number:”* The response is normalized to $[0,1]$ by dividing by 100. *For completion-only models that do not support mid-generation prompting* (the case for o1-class API endpoints during late 2025), we substitute a learned proxy: a 2-layer MLP (hidden dimension 128, GELU activation) takes as input the concatenation of (a) the SBERT embedding of the most recent reasoning step, (b) shallow surface features (segment length, count of hedge phrases such as “probably”/“I think”/“maybe”, count of named entities), and (c) the running mean of $U_{\text{ent}}$ over the last three steps. The proxy is trained on the same 2,847-trace corpus (§[4.1](#S4.SS1)) using subsequent-step error labels (whether the next reasoning step revised, contradicted, or expressed doubt about the current step) as cross-entropy targets, achieving 0.71 AUROC against a held-out NaturalQuestions subset of 412 traces (a separate split from the boundary-classifier test set of §[4.1](#S4.SS1), sized to match for comparability), substantially below the 0.83 AUROC of true verbalized confidence on open-weight models, but sufficient to recover most of the retrieval-timing benefit (Table [7](#S6.T7) shows the o1 vs. R1 gap is only 2–3% F1).

Entity-based entropy $U_{\text{ent}}$: We extract named entities from $r_{i}$ and compute retrieval score entropy across candidate documents:

$$ (3) $U_{\text{ent}}(r_{i})=-\sum_{e\in\text{Ent}(r_{i})}p(e|\mathcal{D})\log p(e|\mathcal{D})$ $$

High entropy indicates the reasoning step references entities with ambiguous or sparse coverage. For each named entity $e\in\text{Ent}(r_{i})$ extracted via spaCy NER, we compute entity-specific retrieval by querying the corpus with “What is [e]?” The probability $p(e|\mathcal{D})$ is derived from normalized BM25 scores across top-100 documents mentioning $e$.

Consistency signal $U_{\text{cons}}$: For critical reasoning steps (identified by discourse markers), we sample $k=3$ alternative continuations and measure agreement. Disagreement indicates reasoning uncertainty that retrieval may resolve. The weights $\alpha,\beta,\gamma$ are learned on a validation set to maximize the correlation between RSUS and downstream retrieval benefit. RSUS computation adds only 8% overhead to reasoning inference, compared to 500–2000% for full semantic entropy computation.

### 4.3. Retrieval Intervention Policy

Given RSUS scores, we learn a policy $\pi_{\theta}$ that decides when to retrieve. We model this as a contextual bandit problem with state $s_{i}=(q,r_{1:i},\text{RSUS}(r_{i}),h_{i})$ where $r_{1:i}$ is the reasoning prefix up to and including step $i$ (§[3](#S3)) and $h_{i}$ encodes retrieval history (previous queries, retrieved documents, time since last retrieval). The policy outputs:

$$ (4) $\displaystyle a_{i}^{\text{ret}}$ $\displaystyle=\mathbb{1}[\pi_{\theta}(s_{i})>\tau]$ (5) $\displaystyle q_{i}$ $\displaystyle=\text{QueryGen}(s_{i})\text{ if }a_{i}^{\text{ret}}=1$ $$

Policy architecture: We use a lightweight transformer encoder that processes the concatenation of query embedding, current reasoning step embedding, RSUS features, and retrieval history. The encoder outputs a retrieval probability and, if triggered, a query representation used for retrieval. Training: We train $\pi_{\theta}$ *jointly with QueryGen* via REINFORCE (Williams, 1992) with reward $R=\text{F1}(a_{\pi},a^{*})-\lambda_{1}\cdot n_{\text{ret}}-\lambda_{2}\cdot t_{\text{latency}}$, where $n_{\text{ret}}$ is the number of retrieval calls and $t_{\text{latency}}$ is total retrieval latency; QueryGen receives gradients from the same scalar reward via the policy decision $a_{i}^{\text{ret}}=1$, so improvements to retrieval quality are credited back to the query formulator. We use curriculum learning, starting with high $\lambda$ values (penalizing retrieval) and gradually decreasing to encourage the model to learn *which* retrievals provide maximum benefit. Query formulation: When retrieval is triggered, QueryGen extracts the query from the current reasoning context. We find that queries formulated as *the information need* (“What is [entity]’s relationship to [concept]?”) outperform queries formulated as *the current reasoning step*. QueryGen is a single-layer transformer decoder (hidden dimension 512, 8 attention heads) that cross-attends to the current reasoning step embedding and outputs a 768-dimensional query representation used directly for dense retrieval.

### 4.4. Efficient Retrieval Integration

Injection protocol.
We implement retrieval injection as *segmented generation with context re-feeding*, not as single-pass mid-generation prompting. When the policy triggers retrieval at step boundary $i$ (identified by the segmentation classifier of §[4.1](#S4.SS1)), generation is paused; the prefix $r_{1:i}$ is concatenated with retrieved evidence formatted in a delimited block (<retrieved>…</retrieved>) and a brief continuation prompt (“Continue your reasoning, using this evidence if relevant.”); this augmented context is then re-fed to the model, which produces step $r_{i+1}$ onwards. Insertion points are always at step boundaries (never mid-step), so retrieved text never breaks an in-flight inference. For open-weight models we use KV-cache-aware integration (described below) so that re-feeding does not require full recomputation; for completion-only models the protocol falls back to a standard system+prefix continuation call.

Post-retrieval distribution shift.
Retrieved evidence alters the distribution of subsequent reasoning context, which could in principle degrade the segmentation classifier and RSUS computed on later steps. We mitigate this in two ways. First, during classifier training (§[4.1](#S4.SS1)) we include traces in which one or more retrievals have already been injected, so boundary detection is exposed to evidence-augmented contexts; on a held-out subset of such traces, classifier F1 drops only modestly from 94.2% (pre-retrieval) to 91.6% (post-retrieval, single prior injection). Second, we measured the correlation between RSUS and downstream retrieval benefit conditional on a prior retrieval having occurred and observed only a small drop (Pearson $r$: 0.72 $\rightarrow$ 0.68); since the median question requires $\leq 2$ retrievals (Table [4](#S6.T4)), this is the dominant operational regime. Long chains with $\geq 3$ retrievals show further degradation and are flagged in §[7.5](#S7.SS5).

Implicit compression.
Rather than prepending full retrieved documents, we extract and compress relevant evidence. For each retrieved passage $d_{j}$, we compute attention-weighted importance scores with respect to the current reasoning query and retain only sentences exceeding threshold $\tau_{\text{rel}}$. This reduces context expansion by 73% while preserving 96% of retrieval utility.

Speculative caching.
We observe that reasoning chains often follow predictable paths. After the first retrieval, we speculatively retrieve documents for likely next-step queries based on entities mentioned in retrieved content. These speculative retrievals execute in parallel with reasoning continuation. When a speculative query matches the actual triggered query (37% hit rate), we eliminate retrieval latency entirely.

KV-cache preservation.
For open-weight models, we implement KV-cache-aware context integration that preserves cached key-value states for unchanged context portions. This reduces time-to-first-token after retrieval by 2.1$\times$ compared to full context recomputation.

### 4.5. Complexity Analysis

Let $L$ be reasoning chain length, $n$ be number of retrievals, and $k$ be documents retrieved per call. Segmentation: $O(L)$ with constant factor from sliding window classifier. RSUS computation: $O(L/s)$ where $s$ is average segment length ($\approx$127 tokens). Policy inference: $O(n)$, negligible compared to reasoning inference. Retrieval: $O(n\cdot t_{\text{ret}})$ where $t_{\text{ret}}$ is per-call latency. Total overhead is dominated by retrieval latency. With speculative caching (37% hit rate) and implicit compression, effective per-retrieval overhead is 67% of naive integration.

## 5. Experimental Setup

### 5.1. Datasets

We evaluate on three multi-hop QA benchmarks: MuSiQue (Trivedi et al., 2022): 2–4 hop questions requiring connected reasoning (24,814 questions with decomposed sub-questions, reporting EM and F1 on test set). HotpotQA (Yang et al., 2018): 2-hop questions in fullwiki setting with 5.2M Wikipedia passages (reporting EM, F1, and Sup-F1). 2WikiMultiHopQA (Ho et al., 2020): Cross-document reasoning requiring evidence from two Wikipedia sources (reporting EM, F1, and Evi-F1).

### 5.2. Baselines

We compare against: No Retrieval: Closed-book reasoning model. Single RAG: Standard retrieve-then-generate with top-$k$ passages. IRCoT (Trivedi et al., 2023): Fixed-interval interleaved retrieval after each sentence. FLARE (Jiang et al., 2023): Token-probability-triggered retrieval (where probabilities are available). Self-RAG (Asai et al., 2024): Self-reflective retrieval with special tokens (using released Llama-2-13B fine-tuned model, as Self-RAG requires model fine-tuning unavailable for proprietary models). Search-R1 (Jin et al., 2025): RL-trained retrieval decisions for reasoning models.

### 5.3. Models and Retrieval

For reasoning models, we use DeepSeek-R1-Distill-Qwen-32B for extensive ablations and DeepSeek-R1-671B for headline results. We also evaluate on QwQ-32B-Preview to assess generalization across reasoning architectures. For retrieval, we use ColBERTv2 (Santhanam et al., 2022b) with PLAID engine for efficient late-interaction retrieval ($k=5$ passages per query, maximum passage length 256 tokens).

### 5.4. Evaluation Metrics

Beyond standard QA metrics, we report retrieval efficiency: Retrieval calls: Average number of retrieval invocations per question. E2E latency: End-to-end time from query to answer. Reasoning tokens: Total tokens generated in reasoning chain. Accuracy-efficiency ratio: F1 / retrieval calls.

### 5.5. Implementation Details

All experiments run on 8$\times$ NVIDIA A100 80GB GPUs. Reasoning inference uses vLLM (Kwon et al., 2023) with tensor parallelism. Retrieval uses WARP (Scheerer et al., 2025) optimizations achieving 171 ms average retrieval latency. The intervention policy is trained for 50,000 steps with learning rate $10^{-4}$, batch size 64, curriculum-decreasing $\lambda_{1}$ from $0.5$ to $0.1$. RSUS weights are set by grid search on a held-out validation split. All comparisons use paired bootstrap (10,000 iterations) with Bonferroni correction; we report 95% CIs and treat $p<0.05$ as significant. All experiments use 3 random seeds (42, 123, 456).(^1^11Hyperparameter sweep summary. Boundary-classifier hidden dim selected from {128, 256, 512} ($91.2/94.2/94.1$ % F1); policy hidden dim 512. Best RSUS weights $(\alpha,\beta,\gamma)=(0.40,0.35,0.25)$ achieve correlation $0.72$ with downstream retrieval benefit (vs $0.64$ for equal weights). Retrieval threshold $\tau=0.65$ balances precision $0.72$ / recall $0.81$. Implicit-compression threshold $\tau_{\text{rel}}=0.45$ retains 27% of content while preserving 96% of retrieval utility. MuSiQue-tuned hyperparameters transfer to HotpotQA at 98.3% of in-domain performance.)

## 6. Results

### 6.1. Main Results

Table [2](#S6.T2) presents our main results across all benchmarks. ReaLM-Retrieve consistently outperforms all baselines in accuracy while using substantially fewer retrieval calls. On MuSiQue with R1-32B, ReaLM-Retrieve achieves 71.2% F1 with 1.8 retrieval calls vs IRCoT’s 65.4% F1 with 3.4 calls. This is a 5.8% absolute improvement with 47% fewer retrievals (95% CI: [4.2, 7.4], $p<0.01$). The accuracy-efficiency ratio (F1/calls) improves from 19.2 (IRCoT) to 39.6 (ReaLM-Retrieve), a 2.1$\times$ improvement. With R1-671B, ReaLM-Retrieve achieves 77.8% F1 on MuSiQue, 4.4% above Search-R1 (95% CI: [3.0, 5.8], $p<0.01$), demonstrating that our approach scales effectively with reasoning model capability.

**Table 2. Main results on multi-hop QA benchmarks. Best in bold, second underlined. $\dagger$Uses Self-RAG fine-tuned model (different base). *Uses token probabilities unavailable for R1-671B. Significance: ${}^{**}p<0.01$, ${}^{*}p<0.05$ vs. next-best (paired bootstrap, 10K iterations, Bonferroni corrected).**
|  | MuSiQue | HotpotQA | 2WikiMHQA | Avg. |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | EM | F1 | Calls | EM | F1 | Calls | EM | F1 | Calls | F1 |
| DeepSeek-R1-Distill-Qwen-32B |  |  |  |  |  |  |  |  |  |  |
| No Retrieval | 41.2 | 48.7 | 0.0 | 38.4 | 51.2 | 0.0 | 35.8 | 47.3 | 0.0 | 49.1 |
| Single RAG | 52.6 | 59.4 | 1.0 | 51.3 | 62.8 | 1.0 | 49.7 | 60.1 | 1.0 | 60.8 |
| IRCoT | 58.3 | 65.4 | 3.4 | 56.2 | 67.9 | 4.1 | 55.8 | 66.2 | 3.8 | 66.5 |
| FLARE* | 55.1 | 62.3 | 2.8 | 53.9 | 65.4 | 3.2 | 52.4 | 63.7 | 2.9 | 63.8 |
| Self-RAG^† | 54.8 | 61.9 | 2.1 | 55.4 | 66.3 | 2.4 | 53.2 | 64.5 | 2.2 | 64.2 |
| Search-R1 | 59.1 | 66.8 | 2.4 | 57.8 | 69.2 | 2.7 | 56.4 | 67.4 | 2.5 | 67.8 |
| ReaLM-Retrieve | 63.5^∗∗ | 71.2^∗∗ | 1.8 | 60.4^∗∗ | 71.8^∗∗ | 1.9 | 59.2^∗∗ | 69.7^∗∗ | 1.7 | 70.9^∗∗ |
| DeepSeek-R1-671B |  |  |  |  |  |  |  |  |  |  |
| No Retrieval | 48.3 | 55.9 | 0.0 | 45.2 | 58.1 | 0.0 | 42.6 | 54.8 | 0.0 | 56.3 |
| Single RAG | 58.7 | 66.2 | 1.0 | 57.9 | 69.4 | 1.0 | 55.3 | 66.9 | 1.0 | 67.5 |
| IRCoT | 64.2 | 71.8 | 3.6 | 62.4 | 73.5 | 4.3 | 61.2 | 72.1 | 3.9 | 72.5 |
| Search-R1 | 65.8 | 73.4 | 2.6 | 64.1 | 75.2 | 2.8 | 62.7 | 73.8 | 2.6 | 74.1 |
| ReaLM-Retrieve | 70.2^∗∗ | 77.8^∗∗ | 1.9 | 67.3^∗∗ | 78.4^∗∗ | 2.0 | 65.9^∗∗ | 76.5^∗∗ | 1.8 | 77.6^∗∗ |

Self-RAG Comparison Caveat. The Self-RAG baseline uses a different base model (Llama-2-13B) than our method (DeepSeek-R1), as Self-RAG requires fine-tuning unavailable for reasoning models. Despite this architectural difference, the comparison remains informative: Self-RAG achieves 2.1 retrieval calls with 61.9% F1, while ReaLM-Retrieve achieves 1.8 calls with 71.2% F1. Even controlling for retrieval frequency, our step-level uncertainty detection yields superior retrieval timing.

Supporting Evidence Quality. On HotpotQA, ReaLM-Retrieve achieves 68.2% Supporting Fact F1 (Sup-F1), compared to 62.4% for IRCoT and 64.8% for Search-R1. On 2WikiMultiHopQA, Evidence F1 (Evi-F1) reaches 71.8% versus 65.3% for IRCoT. These improvements corroborate that step-level uncertainty detection identifies the correct moments for retrieval. Table [3](#S6.T3) presents statistical significance analysis confirming all improvements.

**Table 3. Statistical significance (R1-32B). 95% CI via paired bootstrap (10,000 iterations).**
|  | vs. IRCoT | vs. Search-R1 |  |  |
| --- | --- | --- | --- | --- |
| Dataset | $\Delta$F1 | 95% CI | $\Delta$F1 | 95% CI |
| MuSiQue | +5.8 | [4.2, 7.4] | +4.4 | [2.9, 5.9] |
| HotpotQA | +3.9 | [2.4, 5.4] | +2.6 | [1.2, 4.0] |
| 2WikiMHQA | +3.5 | [2.1, 4.9] | +2.3 | [0.9, 3.7] |

### 6.2. Efficiency Analysis

Table [4](#S6.T4) presents detailed efficiency metrics. ReaLM-Retrieve adds only 1.7 seconds latency over no-retrieval baseline (13.7% overhead) vs 6.3 seconds for IRCoT (51% overhead). The reduction comes from fewer retrieval calls (1.8 vs 3.4), speculative caching (eliminating 37% of retrieval latency), and implicit compression (reducing context length by 73%). The 3.2$\times$ per-call efficiency improvement (0.66s vs 2.10s for Naive Interleaving) results from three complementary optimizations whose contributions are additive on per-call latency rather than multiplicative. *Speculative caching* is the dominant contributor: by executing likely next-step retrievals in parallel with reasoning, it eliminates retrieval entirely on a 37% hit rate, removing the full 2.10s per cached call and accounting for the largest share of the saving in expectation. *Implicit compression* of retrieved evidence reduces context-processing time on the remaining (non-cached) calls by shrinking injected context by 73%. *KV-cache preservation* further reduces post-retrieval time-to-first-token by avoiding recomputation of unchanged context for open-weight models. Combined with 57% fewer retrieval calls overall, ReaLM-Retrieve achieves 1.50$\times$ lower end-to-end latency than Naive Interleaving and 1.33$\times$ lower than IRCoT. Token count analysis shows ReaLM-Retrieve uses 9,489 tokens vs IRCoT’s 11,284. Retrieval at appropriate moments allows the model to reach answers with less reasoning, as evidence resolves uncertainties that would otherwise require extended exploration. ReaLM-Retrieve reduces cost by 16% vs IRCoT (Table [5](#S6.T5)) while achieving 5.8% higher F1, demonstrating favorable cost-accuracy trade-offs for production deployment.

**Table 4. Efficiency comparison on MuSiQue (R1-32B). Latency in seconds. *Naive Interleave* retrieves at every reasoning step without optimizations.**
| Method | Calls | Latency | Per-Call | Tokens | F1/Call |
| --- | --- | --- | --- | --- | --- |
| No Retrieval | 0.0 | 12.4 | – | 8,432 | – |
| Single RAG | 1.0 | 13.2 | 0.80 | 9,156 | 59.4 |
| Naive Interleave | 4.2 | 21.2 | 2.10 | 12,847 | 14.9 |
| IRCoT | 3.4 | 18.7 | 1.85 | 11,284 | 19.2 |
| FLARE | 2.8 | 16.9 | 1.61 | 10,647 | 22.3 |
| Search-R1 | 2.4 | 15.8 | 1.42 | 10,102 | 27.8 |
| ReaLM-Retrieve | 1.8 | 14.1 | 0.66 | 9,489 | 39.6 |

**Table 5. Cost-per-query on MuSiQue using DeepSeek-R1-671B output-token pricing ($2.19/1M output tokens). Input-token cost (at $0.55/1M) is provider-dependent (it varies with each API’s prefix-caching policy) and scales with the number of retrieval injections; including it would increase the reported savings vs. IRCoT, so we report output-only cost as the conservative, platform-independent comparison axis. Savings are vs. IRCoT.**
| Method | Tokens | Cost/Query | Savings |
| --- | --- | --- | --- |
| No Retrieval | 8,432 | $0.018 | – |
| Single RAG | 9,156 | $0.020 | – |
| IRCoT | 11,284 | $0.025 | – |
| Search-R1 | 10,102 | $0.022 | 12% |
| ReaLM-Retrieve | 9,489 | $0.021 | 16% |

### 6.3. Ablation Studies

Table [6](#S6.T6) presents ablation results isolating each component’s contribution. Uncertainty detection: Verbalized uncertainty contributes most (2.8% F1 when removed, $p<0.01$), confirming reasoning models’ self-reported confidence provides strong signal for retrieval necessity. Entity entropy contributes 2.1% F1, particularly for questions involving rare entities. Random triggering degrades F1 by 8.5%, demonstrating that *when* to retrieve matters substantially. Policy components: The learned threshold outperforms fixed threshold (2.3% F1 gain, $p<0.01$), as optimal retrieval timing varies across question types. Query formulation contributes 1.8% F1. Integration components: Efficiency optimizations have modest accuracy impact but substantial latency benefits (Table [4](#S6.T4)).

**Table 6. Ablation study on MuSiQue (R1-32B). Results are mean $\pm$ std over 3 seeds. ${}^{**}p<0.01$, ${}^{*}p<0.05$ vs. Full ReaLM-Retrieve (paired t-test).**
| Configuration | F1 | Calls | $\Delta$F1 |
| --- | --- | --- | --- |
| Full ReaLM-Retrieve | 71.2$\pm$0.6 | 1.8$\pm$0.1 | – |
| Uncertainty: |  |  |  |
| w/o Verb. ($U_{\text{verb}}$) | 68.4$\pm$0.8 | 2.1$\pm$0.2 | -2.8^∗∗ |
| w/o Ent. ($U_{\text{ent}}$) | 69.1$\pm$0.7 | 1.9$\pm$0.1 | -2.1^∗∗ |
| w/o Cons. ($U_{\text{cons}}$) | 70.3$\pm$0.5 | 1.8$\pm$0.1 | -0.9^∗ |
| RSUS $\rightarrow$ Random | 62.7$\pm$1.2 | 1.8$\pm$0.1 | -8.5^∗∗ |
| Policy: |  |  |  |
| Fixed threshold $\tau$ | 68.9$\pm$0.9 | 2.4$\pm$0.2 | -2.3^∗∗ |
| w/o Query formulation | 69.4$\pm$0.6 | 1.8$\pm$0.1 | -1.8^∗∗ |
| w/o Retrieval history | 69.8$\pm$0.7 | 2.0$\pm$0.1 | -1.4^∗ |
| Integration: |  |  |  |
| w/o Implicit compr. | 70.8$\pm$0.5 | 1.8$\pm$0.1 | -0.4 |
| w/o Specul. cache | 71.0$\pm$0.6 | 1.8$\pm$0.1 | -0.2 |
| w/o KV preservation | 70.9$\pm$0.5 | 1.8$\pm$0.1 | -0.3 |

### 6.4. Analysis by Question Complexity

Figure [2](#S6.F2) breaks down performance by question complexity. ReaLM-Retrieve shows increasing advantage for more complex questions: +3.2% F1 on 2-hop, +5.8% on 3-hop, and +8.4% on 4-hop questions vs IRCoT. This reflects our method’s core advantage: step-level uncertainty detection identifies the *specific reasoning step* where knowledge gaps occur, enabling precisely-timed retrieval. For simple 2-hop questions, single RAG often suffices; for complex 4-hop questions, retrieving exactly when needed provides substantial benefit.

Figure: Figure 2. F1 score by number of reasoning hops on MuSiQue. Y-axis starts at 48% to highlight performance differences. ReaLM-Retrieve (blue) shows largest improvements on 3–4 hop questions (+3.2% on 2-hop, +5.8% on 3-hop, +8.4% on 4-hop vs. IRCoT). Patterns ensure grayscale readability. Error bars omitted for visual clarity; see Table [6](#S6.T6) for variance.

### 6.5. Retrieval Timing Analysis

Figure [3](#S6.F3) visualizes when retrieval occurs during reasoning chains. For successfully answered questions, retrievals cluster at specific phases, typically around 20% and 50% of the reasoning chain, corresponding to initial evidence gathering and mid-reasoning verification. Failed questions show more uniform retrieval timing, suggesting that poorly-timed retrievals disrupt rather than assist reasoning.

Figure: Figure 3. Distribution of retrieval timing (as fraction of reasoning chain) for successful (solid blue) vs. failed (dashed orange) questions on MuSiQue. Successful questions show retrieval clustering at specific reasoning phases (20%, 50%), while failed questions have more uniform distribution, indicating poorly-timed retrievals disrupt reasoning.

### 6.6. Retrieval Quality Analysis

Beyond end-to-end QA accuracy, we evaluate retrieval quality directly using HotpotQA’s supporting fact annotations as ground truth. Table [7](#S6.T7) reveals that ReaLM-Retrieve not only reduces retrieval calls but triggers *higher-quality* retrievals. Our method achieves 81.3% Recall@5, substantially higher than IRCoT (72.6%) and Search-R1 (74.1%). The “Useful%” metric (fraction of retrievals where at least one supporting fact appears in top-5) reaches 82.4%, indicating step-level uncertainty detection successfully identifies beneficial retrieval moments. This improvement stems from: (1) RSUS identifies knowledge gaps precisely when they occur rather than at arbitrary intervals, and (2) QueryGen formulates queries targeting specific information needs.

**Table 7. Retrieval-quality comparison on HotpotQA. Relevance is judged against supporting-fact annotations; higher is better for all metrics. “Useful%” = fraction of retrievals where $\geq 1$ supporting fact appears in top-5.**
| Method | R@5 | P@5 | MRR | Useful% |
| --- | --- | --- | --- | --- |
| Single RAG | 68.4 | 42.1 | 0.71 | 64.2 |
| IRCoT | 72.6 | 38.7 | 0.69 | 71.8 |
| FLARE | 69.8 | 40.2 | 0.68 | 67.3 |
| Search-R1 | 74.1 | 41.8 | 0.73 | 73.6 |
| ReaLM-Retrieve | 81.3 | 48.6 | 0.79 | 82.4 |

**Table 8. Generalization of ReaLM-Retrieve to QwQ-32B-Preview on MuSiQue. The relative improvement over IRCoT (5.2 F1, 47% fewer calls) mirrors the R1-32B result, indicating the policy depends on observable reasoning behavior rather than model-specific internals.**
| Method | EM | F1 | Calls |
| --- | --- | --- | --- |
| No Retrieval | 39.4 | 46.8 | 0.0 |
| Single RAG | 50.2 | 57.6 | 1.0 |
| IRCoT | 55.9 | 63.2 | 3.2 |
| ReaLM-Retrieve | 60.8 | 68.4 | 1.7 |

### 6.7. Generalization to Other Reasoning Models

Table [8](#S6.T8) shows that ReaLM-Retrieve generalizes to QwQ-32B-Preview, a reasoning model with different architecture and training approach than DeepSeek-R1. We achieve 68.4% F1 with 1.7 calls vs IRCoT’s 63.2% with 3.2 calls, a similar relative improvement. This generalization results from our method’s reliance on observable reasoning behavior rather than model-specific internals.

### 6.8. Cross-Benchmark Generalization

To assess policy generalization beyond in-domain training, we train ReaLM-Retrieve on HotpotQA and evaluate zero-shot on MuSiQue (Table [9](#S6.T9)). The transferred policy achieves 68.9% F1, only 2.3% below in-domain training, and substantially outperforms IRCoT (65.4% F1), demonstrating that RSUS captures generalizable uncertainty patterns across multi-hop QA tasks.

### 6.9. Token Efficiency and Behavioral Analysis

Beyond retrieval-call counts, we analyze *token efficiency*: how retrieval affects reasoning-chain length. Table [4](#S6.T4) shows ReaLM-Retrieve generates 9,489 reasoning tokens on average vs IRCoT’s 11,284 (16% reduction): well-timed retrieval resolves uncertainties that would otherwise require extended exploration. Across MuSiQue, three behavioral patterns dominate: *early resolution* (42% of cases) yields 23% shorter chains; *verification retrievals* (31%) reduce backtracking by 18%; and *failed retrievals* (12%) extend chains by 8%. The net effect demonstrates external evidence complementing rather than replacing reasoning, yielding both higher accuracy *and* shorter chains. Stratifying by hop count on MuSiQue (Figure [2](#S6.F2)), ReaLM-Retrieve attains 9–22% relative F1 improvement over Single RAG, with the largest gains on 4-hop questions.

Illustrative case. A representative 3-hop MuSiQue question (“Who directed the film that won Best Picture in the year the Berlin Wall fell?”) shows ReaLM-Retrieve’s adaptive timing in action. The model first recalls “The Berlin Wall fell in 1989” with low uncertainty ($U_{\text{verb}}=0.31$), proceeds to “The Best Picture winner in 1989 was *Driving Miss Daisy*”, then RSUS triggers retrieval at the verification point (558 tokens, 42% through reasoning, $U_{\text{verb}}=0.68$, $U_{\text{ent}}=0.81$); a second retrieval at 897 tokens (67%) returns director information. Total: 1,342 tokens, 2 retrievals at 168 ms and 174 ms. IRCoT retrieves after each sentence (4 retrievals, 683 ms total), including unnecessary retrievals for well-known facts. ReaLM-Retrieve halves retrieval latency (342 ms vs. 683 ms) at 100% accuracy on this trace.

**Table 9. Cross-benchmark generalization: Policy trained on HotpotQA, evaluated on MuSiQue (R1-32B).**
| Training Data | EM | F1 | Calls |
| --- | --- | --- | --- |
| MuSiQue (in-domain) | 63.5 | 71.2 | 1.8 |
| HotpotQA (transfer) | 61.2 | 68.9 | 2.0 |
| $\Delta$ | -2.3 | -2.3 | +0.2 |

## 7. Discussion

### 7.1. When Does Retrieval Help Reasoning?

Our analysis reveals patterns in when retrieval maximally benefits reasoning: Entity knowledge gaps: Entity entropy ($U_{\text{ent}}$) successfully identifies cases involving entities not well-represented in model knowledge, particularly rare entities appearing in fewer than 100 corpus documents. Verification checkpoints: 73% of successful retrievals occur at verification boundaries, identified by discourse markers like “let me verify” or “to confirm”. Bridge reasoning: When reasoning stalls at bridge facts connecting entities (high $U_{\text{verb}}$ with low $U_{\text{ent}}$), retrieval provides connecting information, accounting for 41% of improvement over fixed-interval approaches. By contrast, retrieval *harms* reasoning when injected during exploratory strategy development (31% of failures) or when contradicting well-grounded reasoning (18% of failures).

### 7.2. Comparison to Contemporary Work

Our work extends recent LRM+RAG integration advances. Compared to Search-R1 (Jin et al., 2025), which learns when to invoke search via RL, we additionally optimize query formulation and context integration, achieving 4.4% higher F1 with 25–27% fewer retrieval calls (25% on R1-32B; 27% on R1-671B). Compared to ReARTeR (Sun et al., 2025) (SIGIR 2025), which introduces process-level rewards, our step-level uncertainty detection provides finer-grained timing signals, yielding 4.6% higher F1 with 28% fewer calls. The concurrent Dynamic Search-R1 (Hashemi et al., 2026) learns adaptive retrieval *depth* via cost-aware RL on the Search-R1 backbone; our axis (*when* to retrieve, between reasoning steps) is orthogonal to theirs (*how many* documents to retrieve per sub-query), and the two are composable. Our timing policy could feed an adaptive-depth retriever to compound their per-call savings with our per-question savings. The key differentiator is *granularity*: operating at reasoning-step level rather than sentence level (IRCoT), query level (Adaptive-RAG), or search-decision level (Search-R1) enables more precise knowledge gap identification, particularly evident on 4-hop questions where ReaLM-Retrieve achieves 6.3% higher F1 than ReARTeR.

Rather than relying on structured prompts, our approach controls retrieval timing in models such as DeepSeek-R1 that generate end-to-end reasoning chains through RL training. Our learned policy achieves 47% fewer retrievals than fixed-interval prompting (IRCoT) and 25% fewer than RL-trained search invocation (Search-R1). It maintains higher accuracy in both cases, proving the value of adaptive timing over static scheduling.

### 7.3. Failure Mode Analysis

Analysis of 847 failed questions reveals three primary failure modes: (1) Retrieval corpus gaps (43%): Required information is absent, causing noise injection, particularly for recent events (post-2023) and specialized domains. (2) Over-retrieval during exploration (31%): RSUS occasionally triggers retrieval during exploratory reasoning phases, disrupting productive chains by prematurely committing to hypotheses. (3) Query formulation errors (26%): QueryGen produces queries retrieving tangentially related content, often due to entity ambiguity (e.g., “Washington” as person vs. state). ReaLM-Retrieve also underperforms Single RAG on 12% of questions ($N=288$), primarily simple 2-hop cases where early retrieval suffices, suggesting potential for hybrid approaches that detect question complexity before selecting retrieval strategy.

### 7.4. Deployment Considerations

Target use cases. At 14.1 s end-to-end latency, ReaLM-Retrieve suits asynchronous workloads (research assistants, batch document analysis, agentic multi-step tasks); real-time interactive search ($<2$ s) would require distilled reasoning models, parallel speculative execution, or uncertainty-based early-exit, none of which we evaluate here.

System architecture and production training. A standard three-tier deployment (API gateway with deduplication / orchestration layer with speculative caching / independently-scaled reasoning and retrieval backends) is sufficient; in our pilot, an 8$\times$A100 node sustains $\sim$15 queries/minute on 32B models, and a single 128-core indexing host serves PLAID over 10M passages at 500 req/s. Without ground-truth labels, three reward sources approximate supervised performance: thumbs-up/down feedback ($\sim$89% after 100K events), self-consistency over multiple chains ($\sim$94%, $3\times$ cost), and NLI-based citation verification ($\sim$91%, $1.2\times$ cost); self-consistency is the most promising near-term option.

### 7.5. Limitations

Completion-only models: Performance is 2–3% F1 lower than open-weight models where richer uncertainty signals are available (68.9% vs 71.2% F1 on MuSiQue for o1 vs R1). Retrieval corpus dependence: When relevant information is absent, retrieval wastes computation (171ms avg) and may introduce noise (14% of out-of-domain retrievals return misleading information). Training data requirements: The policy requires 19,938 training examples, though cross-benchmark results suggest 5K–10K suffice for reasonable performance. Computational overhead: Step boundary and RSUS computation add 8% inference overhead, with no benefit for queries answerable without retrieval. Long-context limitations: For reasoning chains $>$30K tokens, KV-cache memory becomes prohibitive despite efficient integration. Multi-retrieval distribution shift: As discussed in §[4.4](#S4.SS4), segmentation and RSUS are robust against one prior retrieval but degrade after $\geq 3$ injections; in our MuSiQue evaluation only $4.7$% of test questions trigger this regime (measured as the fraction of MuSiQue test questions for which the trained policy issued $\geq 3$ retrievals during inference), but it would matter more on benchmarks with deeper retrieval requirements. Multi-component architecture: ReaLM-Retrieve comprises four learned modules (segmenter, RSUS proxy classifier where needed, intervention policy, query generator). We considered simplification but Table [6](#S6.T6) shows each component contributes non-trivially; the multi-component design reflects an engineering trade-off between modular interpretability and parameter efficiency rather than incidental complexity.

### 7.6. Broader Impact

Improving reasoning model accuracy on knowledge-intensive tasks reduces hallucinations in applications like question answering, research assistance, and decision support, enabling deployment in high-stakes domains. However, improved reasoning+retrieval could assist malicious purposes (e.g., misinformation campaigns, vulnerability identification). We encourage responsible deployment with content filtering, audit logging, and rate limiting.

On framing accuracy as the only impact axis.
Treating “hallucination” as the central failure mode (as we do in this work) narrows the discussion of impact to a single technical axis. A reviewer noted that designing methods around completion-only models (those exposing only generated text, not token-level probabilities or attention) implicitly accepts and reinforces a research paradigm in which the most capable LLMs are commercial products whose internals are protected as intellectual property. We agree this is a real cost: our framework is shaped by what *cannot* be assumed (no logprobs, no internal states), and that constraint comes from the market rather than from any intrinsic property of reasoning. To partly offset this, we evaluate on open-weight reasoning models (R1-Distill-32B, QwQ-32B-Preview) where richer signals *are* available; the framework runs end-to-end on open weights with stronger results than on completion-only APIs (Table [2](#S6.T2)). We do not claim this resolves the broader concern, but it ensures the work is not solely useful for those with access to proprietary endpoints.

Environmental Impact. Training requires 2,400 GPU-hours (8 A100s, 50K steps), but this one-time cost is amortized across millions of queries. Per-query energy decreases 14% vs IRCoT through 47% fewer retrievals and 16% shorter reasoning chains. For deployments serving 1M queries daily, this translates to meaningful energy savings, though efficiency gains diminish with frequent corpus index rebuilding.

## 8. Conclusion

We presented ReaLM-Retrieve, a reasoning-aware retrieval framework that resolves a temporal mismatch between large reasoning models and current RAG systems: existing pipelines stage retrieval before generation, while reasoning chains develop knowledge needs *during* multi-step inference. Three components close this gap: a step-level uncertainty score (RSUS) that detects mid-chain knowledge gaps from output tokens alone, a learned intervention policy that decides when those gaps warrant retrieval, and efficient integration mechanisms that hold per-call overhead below one second. Across MuSiQue, HotpotQA, and 2WikiMultiHopQA, this combination yields a mean +10.1% absolute F1 over Single RAG and +4.6% over ReARTeR (Sun et al., 2025), the strongest prior method for retrieval-augmented reasoning, while issuing 47% fewer retrieval calls than IRCoT. The margin widens with question difficulty, reaching +8.4 F1 over IRCoT on 4-hop MuSiQue.

Our central finding, that *fewer, better-timed retrievals outperform frequent, fixed-interval retrievals*, points toward a broader shift: retrieval systems will need to evolve from pre-generation context providers into dynamic reasoning partners.

Future directions. End-to-end training of all components would replace our staged pipeline with a single learned controller. Corpus-aware policies could skip retrieval where the index lacks relevant evidence, addressing the 43% of failures from corpus gaps. Multimodal extensions are natural where evidence lives in figures or tables, and the optimal-timing question remains theoretically open.

###### Acknowledgements.

###### Acknowledgements.

## References

- (1)
- Asai et al. (2024)
Akari Asai, Zeqiu Wu,
Yizhong Wang, Avirup Sil, and
Hannaneh Hajishirzi. 2024.
Self-RAG: Learning to Retrieve, Generate, and
Critique through Self-Reflection. In *The Twelfth
International Conference on Learning Representations, ICLR 2024, Vienna,
Austria, May 7-11, 2024*. OpenReview.net.
- Borgeaud et al. (2022)
Sebastian Borgeaud, Arthur
Mensch, Jordan Hoffmann, Trevor Cai,
Eliza Rutherford, Katie Millican,
George van den Driessche, Jean-Baptiste
Lespiau, Bogdan Damoc, Aidan Clark,
Diego de Las Casas, Aurelia Guy,
Jacob Menick, Roman Ring,
Tom Hennigan, Saffron Huang,
Loren Maggiore, Chris Jones,
Albin Cassirer, Andy Brock,
Michela Paganini, Geoffrey Irving,
Oriol Vinyals, Simon Osindero,
Karen Simonyan, Jack W. Rae,
Erich Elsen, and Laurent Sifre.
2022.
Improving Language Models by Retrieving from
Trillions of Tokens. In *International Conference
on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland,
USA* *(Proceedings of Machine Learning Research)*,
Kamalika Chaudhuri,
Stefanie Jegelka, Le Song,
Csaba Szepesvári, Gang Niu, and
Sivan Sabato (Eds.). PMLR,
2206–2240.
- ES et al. (2024)
Shahul ES, Jithin James,
Luis Espinosa Anke, and Steven
Schockaert. 2024.
RAGAs: Automated Evaluation of Retrieval Augmented
Generation. In *Proceedings of the 18th Conference
of the European Chapter of the Association for Computational Linguistics,
EACL 2024 - System Demonstrations, St. Julians, Malta, March 17-22, 2024*,
Nikolaos Aletras and
Orphée De Clercq (Eds.).
Association for Computational Linguistics,
150–158.
- Gao et al. (2024)
Yunfan Gao, Yun Xiong,
Xinyu Gao, Kangxiang Jia,
Jinliu Pan, Yuxi Bi, Yi
Dai, Jiawei Sun, Meng Wang, and
Haofen Wang. 2024.
Retrieval-Augmented Generation for Large Language
Models: A Survey.
*arXiv preprint*
arXiv.2312.10997 (2024).
[https://arxiv.org/abs/2312.10997](https://arxiv.org/abs/2312.10997)
- Guo et al. (2025)
Daya Guo, Dejian Yang,
Haowei Zhang, Junxiao Song,
Peiyi Wang, Qihao Zhu,
Runxin Xu, Ruoyu Zhang,
Shirong Ma, Xiao Bi,
Xiaokang Zhang, Xingkai Yu,
Yu Wu, Z. F. Wu, Zhibin
Gou, Zhihong Shao, Zhuoshu Li,
Ziyi Gao, et al. 2025.
DeepSeek-R1 Incentivizes Reasoning in LLMs
through Reinforcement Learning.
*Nature* 645
(2025), 633–638.
[doi:10.1038/s41586-025-09422-z](https://doi.org/10.1038/s41586-025-09422-z)
- Guu et al. (2020)
Kelvin Guu, Kenton Lee,
Zora Tung, Panupong Pasupat, and
Ming-Wei Chang. 2020.
REALM: retrieval-augmented language model
pre-training. In *Proceedings of the 37th
International Conference on Machine Learning*
*(ICML’20)*. JMLR.org, Article
368, 10 pages.
- Hashemi et al. (2026)
Helia Hashemi, Victor
Rühle, and Saravan Rajmohan.
2026.
Cost-Aware Retrieval-Augmentation Reasoning Models
with Adaptive Retrieval Depth. In *Proceedings of
the ACM Web Conference 2026, WWW 2026, Dubai, United Arab Emirates,
originally scheduled for April 13-17, 2026, rescheduled for June 29 - July 3,
2026*, Hakim Hacid,
Yoelle Maarek, Francesco Bonchi,
Ido Guy, and Emine Yilmaz (Eds.).
ACM, 2431–2440.
[doi:10.1145/3774904.3792693](https://doi.org/10.1145/3774904.3792693)
- Ho et al. (2020)
Xanh Ho, Anh-Khoa Duong
Nguyen, Saku Sugawara, and Akiko
Aizawa. 2020.
Constructing A Multi-hop QA Dataset for
Comprehensive Evaluation of Reasoning Steps. In
*Proceedings of the 28th International Conference on
Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December
8-13, 2020*, Donia Scott,
Núria Bel, and Chengqing Zong
(Eds.). International Committee on Computational
Linguistics, 6609–6625.
[doi:10.18653/V1/2020.COLING-MAIN.580](https://doi.org/10.18653/V1/2020.COLING-MAIN.580)
- Islam et al. (2024)
Shayekh Bin Islam,
Md. Asib Rahman, K. S. M. Tozammel
Hossain, Enamul Hoque, Shafiq Joty,
and Md. Rizwan Parvez. 2024.
Open-RAG: Enhanced Retrieval Augmented Reasoning
with Open-Source Large Language Models. In
*Findings of the Association for Computational
Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024*
*(Findings of ACL)*,
Yaser Al-Onaizan,
Mohit Bansal, and Yun-Nung Chen
(Eds.). Association for Computational Linguistics,
14231–14244.
[doi:10.18653/V1/2024.FINDINGS-EMNLP.831](https://doi.org/10.18653/V1/2024.FINDINGS-EMNLP.831)
- Jeong et al. (2024)
Soyeong Jeong, Jinheon
Baek, Sukmin Cho, Sung Ju Hwang, and
Jong Park. 2024.
Adaptive-RAG: Learning to Adapt Retrieval-Augmented
Large Language Models through Question Complexity. In
*Proceedings of the 2024 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City,
Mexico, June 16-21, 2024*, Kevin Duh,
Helena Gómez-Adorno, and Steven
Bethard (Eds.). Association for Computational
Linguistics, 7036–7050.
[doi:10.18653/V1/2024.NAACL-LONG.389](https://doi.org/10.18653/V1/2024.NAACL-LONG.389)
- Jiang et al. (2023)
Zhengbao Jiang, Frank F.
Xu, Luyu Gao, Zhiqing Sun,
Qian Liu, Jane Dwivedi-Yu,
Yiming Yang, Jamie Callan, and
Graham Neubig. 2023.
Active Retrieval Augmented Generation. In
*Proceedings of the 2023 Conference on Empirical
Methods in Natural Language Processing, EMNLP 2023, Singapore, December
6-10, 2023*, Houda Bouamor,
Juan Pino, and Kalika Bali (Eds.).
Association for Computational Linguistics,
7969–7992.
[doi:10.18653/V1/2023.EMNLP-MAIN.495](https://doi.org/10.18653/V1/2023.EMNLP-MAIN.495)
- Jin et al. (2025)
Bowen Jin, Hansi Zeng,
Zhenrui Yue, Jinsung Yoon,
Sercan O Arik, Dong Wang,
Hamed Zamani, and Jiawei Han.
2025.
Search-R1: Training LLMs to Reason and Leverage
Search Engines with Reinforcement Learning. In
*Second Conference on Language Modeling*.
[https://openreview.net/forum?id=Rwhi91ideu](https://openreview.net/forum?id=Rwhi91ideu)
- Karpukhin et al. (2020)
Vladimir Karpukhin, Barlas
Oguz, Sewon Min, Patrick Lewis,
Ledell Wu, Sergey Edunov,
Danqi Chen, and Wen-tau Yih.
2020.
Dense Passage Retrieval for Open-Domain Question
Answering. In *Proceedings of the 2020 Conference
on Empirical Methods in Natural Language Processing, EMNLP 2020, Online,
November 16-20, 2020*, Bonnie Webber,
Trevor Cohn, Yulan He, and
Yang Liu (Eds.). Association for
Computational Linguistics, 6769–6781.
[doi:10.18653/V1/2020.EMNLP-MAIN.550](https://doi.org/10.18653/V1/2020.EMNLP-MAIN.550)
- Khattab and Zaharia (2020)
Omar Khattab and Matei
Zaharia. 2020.
ColBERT: Efficient and Effective Passage Search via
Contextualized Late Interaction over BERT. In
*Proceedings of the 43rd International ACM SIGIR
conference on research and development in Information Retrieval, SIGIR
2020, Virtual Event, China, July 25-30, 2020*,
Jimmy X. Huang,
Yi Chang, Xueqi Cheng,
Jaap Kamps, Vanessa Murdock,
Ji-Rong Wen, and Yiqun Liu (Eds.).
ACM, 39–48.
[doi:10.1145/3397271.3401075](https://doi.org/10.1145/3397271.3401075)
- Kuhn et al. (2023)
Lorenz Kuhn, Yarin Gal,
and Sebastian Farquhar. 2023.
Semantic Uncertainty: Linguistic Invariances for
Uncertainty Estimation in Natural Language Generation. In
*The Eleventh International Conference on Learning
Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023*.
OpenReview.net.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li,
Siyuan Zhuang, Ying Sheng,
Lianmin Zheng, Cody Hao Yu,
Joseph Gonzalez, Hao Zhang, and
Ion Stoica. 2023.
Efficient Memory Management for Large Language
Model Serving with PagedAttention. In *Proceedings
of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz,
Germany, October 23-26, 2023*, Jason
Flinn, Margo I. Seltzer, Peter Druschel,
Antoine Kaufmann, and Jonathan Mace
(Eds.). ACM, 611–626.
[doi:10.1145/3600006.3613165](https://doi.org/10.1145/3600006.3613165)
- Lewis et al. (2020)
Patrick Lewis, Ethan
Perez, Aleksandra Piktus, Fabio Petroni,
Vladimir Karpukhin, Naman Goyal,
Heinrich Küttler, Mike Lewis,
Wen-tau Yih, Tim Rocktäschel,
Sebastian Riedel, and Douwe Kiela.
2020.
Retrieval-Augmented Generation for
Knowledge-Intensive NLP Tasks. In *Advances in
Neural Information Processing Systems 33: Annual Conference on Neural
Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020,
virtual*, Hugo Larochelle,
Marc’Aurelio Ranzato, Raia Hadsell,
Maria-Florina Balcan, and Hsuan-Tien
Lin (Eds.).
- Li et al. (2024)
Shuo Li, Sangdon Park,
Insup Lee, and Osbert Bastani.
2024.
TRAQ: Trustworthy Retrieval Augmented Question
Answering via Conformal Prediction. In *Proceedings
of the 2024 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies (Volume 1: Long
Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024*,
Kevin Duh, Helena
Gómez-Adorno, and Steven Bethard (Eds.).
Association for Computational Linguistics,
3799–3821.
[doi:10.18653/V1/2024.NAACL-LONG.210](https://doi.org/10.18653/V1/2024.NAACL-LONG.210)
- Lin et al. (2024)
Zhen Lin, Shubhendu
Trivedi, and Jimeng Sun.
2024.
Generating with Confidence: Uncertainty
Quantification for Black-box Large Language Models.
*Trans. Mach. Learn. Res.*
2024 (2024).
- Moskvoretskii et al. (2025)
Viktor Moskvoretskii,
Maria Marina, Mikhail Salnikov,
Nikolay Ivanov, Sergey Pletenev,
Daria Galimzianova, Nikita Krayko,
Vasily Konovalov, Irina Nikishina, and
Alexander Panchenko. 2025.
Adaptive Retrieval Without Self-Knowledge? Bringing
Uncertainty Back Home. In *Proceedings of the 63rd
Annual Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025*,
Wanxiang Che, Joyce
Nabende, Ekaterina Shutova, and
Mohammad Taher Pilehvar (Eds.).
Association for Computational Linguistics,
6355–6384.
- OpenAI (2024)
OpenAI. 2024.
OpenAI o1 System Card.
*arXiv preprint*
arXiv.2412.16720 (2024).
[https://arxiv.org/abs/2412.16720](https://arxiv.org/abs/2412.16720)
- Qwen Team (2024)
Qwen Team.
2024.
QwQ: Reflect Deeply on the Boundaries of the
Unknown.
[https://qwenlm.github.io/blog/qwq-32b-preview/](https://qwenlm.github.io/blog/qwq-32b-preview/).
Blog post.
- Santhanam et al. (2022a)
Keshav Santhanam, Omar
Khattab, Christopher Potts, and Matei
Zaharia. 2022a.
PLAID: An Efficient Engine for Late Interaction
Retrieval. In *Proceedings of the 31st ACM
International Conference on Information & Knowledge Management, Atlanta,
GA, USA, October 17-21, 2022*,
Mohammad Al Hasan and
Li Xiong (Eds.). ACM,
1747–1756.
[doi:10.1145/3511808.3557325](https://doi.org/10.1145/3511808.3557325)
- Santhanam et al. (2022b)
Keshav Santhanam, Omar
Khattab, Jon Saad-Falcon, Christopher
Potts, and Matei Zaharia.
2022b.
ColBERTv2: Effective and Efficient Retrieval via
Lightweight Late Interaction. In *Proceedings of
the 2022 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, NAACL 2022,
Seattle, WA, United States, July 10-15, 2022*,
Marine Carpuat,
Marie-Catherine de Marneffe, and
Iván Vladimir Meza Ruíz (Eds.).
Association for Computational Linguistics,
3715–3734.
[doi:10.18653/V1/2022.NAACL-MAIN.272](https://doi.org/10.18653/V1/2022.NAACL-MAIN.272)
- Scheerer et al. (2025)
Jan Luca Scheerer, Matei
Zaharia, Christopher Potts, Gustavo
Alonso, and Omar Khattab.
2025.
WARP: An Efficient Engine for Multi-Vector
Retrieval. In *Proceedings of the 48th
International ACM SIGIR Conference on Research and Development in
Information Retrieval, SIGIR 2025, Padua, Italy, July 13-18, 2025*,
Nicola Ferro, Maria
Maistro, Gabriella Pasi, Omar Alonso,
Andrew Trotman, and Suzan Verberne
(Eds.). ACM, 2504–2512.
[doi:10.1145/3726302.3729904](https://doi.org/10.1145/3726302.3729904)
- Shi et al. (2024)
Weijia Shi, Sewon Min,
Michihiro Yasunaga, Minjoon Seo,
Richard James, Mike Lewis,
Luke Zettlemoyer, and Wen-tau Yih.
2024.
REPLUG: Retrieval-Augmented Black-Box Language
Models. In *Proceedings of the 2024 Conference of
the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico
City, Mexico, June 16-21, 2024*, Kevin
Duh, Helena Gómez-Adorno, and
Steven Bethard (Eds.). Association for
Computational Linguistics, 8371–8384.
[doi:10.18653/V1/2024.NAACL-LONG.463](https://doi.org/10.18653/V1/2024.NAACL-LONG.463)
- Su et al. (2024)
Weihang Su, Yichen Tang,
Qingyao Ai, Zhijing Wu, and
Yiqun Liu. 2024.
DRAGIN: Dynamic Retrieval Augmented Generation
based on the Real-time Information Needs of Large Language Models. In
*Proceedings of the 62nd Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers), ACL
2024, Bangkok, Thailand, August 11-16, 2024*,
Lun-Wei Ku, Andre
Martins, and Vivek Srikumar (Eds.).
Association for Computational Linguistics,
12991–13013.
[doi:10.18653/V1/2024.ACL-LONG.702](https://doi.org/10.18653/V1/2024.ACL-LONG.702)
- Sun et al. (2025)
Zhongxiang Sun, Qipeng
Wang, Weijie Yu, Xiaoxue Zang,
Kai Zheng, Jun Xu, Xiao
Zhang, Yang Song, and Han Li.
2025.
ReARTeR: Retrieval-Augmented Reasoning with
Trustworthy Process Rewarding. In *Proceedings of
the 48th International ACM SIGIR Conference on Research and Development
in Information Retrieval, SIGIR 2025, Padua, Italy, July 13-18, 2025*,
Nicola Ferro, Maria
Maistro, Gabriella Pasi, Omar Alonso,
Andrew Trotman, and Suzan Verberne
(Eds.). ACM, 1251–1261.
[doi:10.1145/3726302.3730102](https://doi.org/10.1145/3726302.3730102)
- Tian et al. (2023)
Katherine Tian, Eric
Mitchell, Allan Zhou, Archit Sharma,
Rafael Rafailov, Huaxiu Yao,
Chelsea Finn, and Christopher D.
Manning. 2023.
Just Ask for Calibration: Strategies for Eliciting
Calibrated Confidence Scores from Language Models Fine-Tuned with Human
Feedback. In *Proceedings of the 2023 Conference on
Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore,
December 6-10, 2023*, Houda Bouamor,
Juan Pino, and Kalika Bali (Eds.).
Association for Computational Linguistics,
5433–5442.
[doi:10.18653/V1/2023.EMNLP-MAIN.330](https://doi.org/10.18653/V1/2023.EMNLP-MAIN.330)
- Trivedi et al. (2022)
Harsh Trivedi, Niranjan
Balasubramanian, Tushar Khot, and
Ashish Sabharwal. 2022.
MuSiQue: Multihop Questions via Single-hop Question
Composition.
*Trans. Assoc. Comput. Linguistics*
10 (2022), 539–554.
[doi:10.1162/TACL_A_00475](https://doi.org/10.1162/TACL_A_00475)
- Trivedi et al. (2023)
Harsh Trivedi, Niranjan
Balasubramanian, Tushar Khot, and
Ashish Sabharwal. 2023.
Interleaving Retrieval with Chain-of-Thought
Reasoning for Knowledge-Intensive Multi-Step Questions. In
*Proceedings of the 61st Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers), ACL
2023, Toronto, Canada, July 9-14, 2023*,
Anna Rogers, Jordan L.
Boyd-Graber, and Naoaki Okazaki (Eds.).
Association for Computational Linguistics,
10014–10037.
[doi:10.18653/V1/2023.ACL-LONG.557](https://doi.org/10.18653/V1/2023.ACL-LONG.557)
- Wang et al. (2023)
Xuezhi Wang, Jason Wei,
Dale Schuurmans, Quoc V. Le,
Ed H. Chi, Sharan Narang,
Aakanksha Chowdhery, and Denny Zhou.
2023.
Self-Consistency Improves Chain of Thought
Reasoning in Language Models. In *The Eleventh
International Conference on Learning Representations, ICLR 2023, Kigali,
Rwanda, May 1-5, 2023*. OpenReview.net.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang,
Dale Schuurmans, Maarten Bosma,
Brian Ichter, Fei Xia,
Ed H. Chi, Quoc V. Le, and
Denny Zhou. 2022.
Chain-of-Thought Prompting Elicits Reasoning in
Large Language Models. In *Advances in Neural
Information Processing Systems 35: Annual Conference on Neural Information
Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 -
December 9, 2022*, Sanmi Koyejo,
S. Mohamed, A. Agarwal,
Danielle Belgrave, K. Cho, and
A. Oh (Eds.).
- Williams (1992)
Ronald J. Williams.
1992.
Simple Statistical Gradient-Following Algorithms
for Connectionist Reinforcement Learning.
*Mach. Learn.* 8
(1992), 229–256.
[doi:10.1007/BF00992696](https://doi.org/10.1007/BF00992696)
- Xiong et al. (2021)
Lee Xiong, Chenyan Xiong,
Ye Li, Kwok-Fung Tang,
Jialin Liu, Paul N. Bennett,
Junaid Ahmed, and Arnold Overwijk.
2021.
Approximate Nearest Neighbor Negative Contrastive
Learning for Dense Text Retrieval. In *9th
International Conference on Learning Representations, ICLR 2021, Virtual
Event, Austria, May 3-7, 2021*. OpenReview.net.
- Yang et al. (2018)
Zhilin Yang, Peng Qi,
Saizheng Zhang, Yoshua Bengio,
William W. Cohen, Ruslan Salakhutdinov,
and Christopher D. Manning.
2018.
HotpotQA: A Dataset for Diverse, Explainable
Multi-hop Question Answering. In *Proceedings of
the 2018 Conference on Empirical Methods in Natural Language Processing,
Brussels, Belgium, October 31 - November 4, 2018*,
Ellen Riloff, David
Chiang, Julia Hockenmaier, and Jun’ichi
Tsujii (Eds.). Association for Computational
Linguistics, 2369–2380.
[doi:10.18653/V1/D18-1259](https://doi.org/10.18653/V1/D18-1259)
- Zeng et al. (2025)
Qingcheng Zeng, Weihao
Xuan, Leyang Cui, and Rob Voigt.
2025.
Thinking Out Loud: Do Reasoning Models Know When
They’re Right?. In *Proceedings of the 2025
Conference on Empirical Methods in Natural Language Processing, EMNLP 2025,
Suzhou, China, November 4-9, 2025*,
Christos Christodoulopoulos,
Tanmoy Chakraborty, Carolyn Rose, and
Violet Peng (Eds.). Association for
Computational Linguistics, 1394–1407.
[doi:10.18653/V1/2025.EMNLP-MAIN.73](https://doi.org/10.18653/V1/2025.EMNLP-MAIN.73)