##### Report GitHub Issue
×
Title:
Content selection saved. Describe the issue below:
Description:
Submit without GitHub
Submit in GitHub
[
Back to arXiv](/)
[License: CC BY 4.0](https://info.arxiv.org/help/license/index.html#licenses-available)
arXiv:2605.05806v1 [cs.LG] 07 May 2026
Retrieval from Within: An Intrinsic Capability of Attention-Based Models
==========================================================================
Elad Hoffer1,
Yochai Blau1,
Edan Kinderman1,
Ron Banner1,
Daniel Soudry1,2,
Boris Ginsburg1
1NVIDIA
2Department of Electrical Engineering, Technion, Haifa, Israel
{elad.hoffer, daniel.soudry}@gmail.com
{yblau,ekinderman,rbanner,bginsburg}@nvidia.com
###### Abstract
Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems.
We ask whether an attention-based encoder-decoder can instead retrieve directly from its own internal representations.
We introduce *INTRA* (*INTrinsic Retrieval via Attention*),
a framework where decoder attention queries score pre-encoded evidence chunks that are then directly reused as context for generation.
By construction, INTRA unifies retrieval and generation, eliminating the retriever-generator mismatch typical of RAG pipelines.
This design also amortizes context encoding by reusing precomputed encoder states across queries.
On question-answering benchmarks, INTRA outperforms strong engineered retrieval pipelines on both evidence recall and end-to-end answer quality.
Our results demonstrate that attention-based models already
possess a retrieval mechanism that can be elicited, rather than added as an
external module.
1 Introduction
--------------
### 1.1 Motivation
Large language models are increasingly used in settings where the information
needed to answer a query is sparse relative to the full available corpus. This is the regime in which retrieval-augmented generation (RAG) has become the default design:
a retriever selects candidate evidence, which is then used by a generator to produce an answer *[lewis2021retrievalaugmentedgenerationknowledgeintensivenlp]*.
This decomposition is practical because naively concatenating all available context
into a single long prompt is computationally expensive, and even large-context
models remain brittle when the relevant evidence is sparse and distributed *[yen2024helmetevaluatelongcontextlanguage, modarressi2025nolimalongcontextevaluationliteral]*.
At the same time, this standard framing encourages a strong architectural
separation between retrieval and generation. The retriever operates over indexed
text or embeddings, while the language model consumes the selected evidence only
after that selection is complete. In practice this
modularity is often helpful, but it can obscure an important fact: attention is
already a query-conditioned mechanism for selecting and weighting relevant
information. This motivates the central question of the paper: can a single
pretrained encoder-decoder retrieve the needed evidence and use it to answer a
query? More broadly, how much of RAG can be handled inside the model itself,
without a separate retriever?
### 1.2 Retrieval as an intrinsic capability
We study question answering using a fixed knowledge base and ask
whether a pretrained encoder-decoder can retrieve, prioritize, and use evidence
drawn from its own representation space. Our central hypothesis is that
pretrained attention-based models already possess an intrinsic retrieval
capability in this setting. We call this regime *INTRA* (*INTrinsic Retrieval via Attention*): rather than relying on an external retriever, the model selects evidence and generates
answers over the representations produced by its own encoder.
The connection between attention and retrieval is made concrete in
Section[2.2](#S2.SS2 "2.2 Attention-based retrieval ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models"): both are query-conditioned
matching operations over candidate states. Within this framing, INTRA transforms the decoder’s cross-attention queries into scores for chunk-level retrieval. This perspective does not suggest that attention mechanisms constitute a complete solution for large-corpus retrieval. Rather, it suggests that a pretrained encoder-decoder contains the
right interface for retrieval: query states that express what the decoder needs,
and encoded evidence states that can be selected and consumed without translation into another representation space.
This design has several practical advantages. The same encoded chunk states
are used for both evidence selection and answer generation, reducing the
mismatch between a separately trained retriever and the generator it serves.
Because those states are encoder memories, static evidence can be encoded once
and reused across queries instead of being repeatedly packed into a long decoder
context. Finally, the retrieval mechanism can be adapted with lightweight
decoder-side retrieval queries rather than a separately trained retriever, reducing the need for a dedicated retrieval system.
*Figure 1: Left: Standard RAG pipeline. An external retriever selects documents which are then re-encoded by the decoder
to produce an answer. Retrieval and generation operate in separate representation spaces. Right: Our method, INTRA, uses a pretrained frozen encoder-decoder for
both retrieval and generation. The decoder retrieves relevant chunks through its
cross-attention queries, augmented with learnable retrieval tokens. The retriever and
generator share a representation space, allowing pre-encoded evidence to be reused
across queries. No external retriever is required.
Numbers indicate the sequence of operations.*
### 1.3 Contributions
* •
We formulate *INTRA*, in which a
single pretrained encoder-decoder model uses one shared representation space
to couple evidence selection with answer generation.
* •
We identify a minimal architectural recipe for exposing this
capability: the pretrained encoder’s native chunk representations are
reused directly, encoder-side late interaction performs coarse retrieval,
and decoder-side retrieval queries refine evidence without introducing a
separate retriever or compression model, as shown in
Figure[1](#S1.F1 "Figure 1 ‣ 1.2 Retrieval as an intrinsic capability ‣ 1 Introduction ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models").
* •
We show empirically that this unified retrieval-generation design is especially strong on multi-hop QA. It rivals strong engineered retrieval pipelines despite their use of large-scale training data, while utilizing the same latent evidence for both selection and generation.
* •
We characterize the computational profile of this design, including
the reusable-context regime that emerges when static evidence is encoded
once and reused across queries.
2 Method
--------
### 2.1 Framework formulation
We consider a retrieval-and-generation setting in which the model generates an
output from a small set of relevant evidence chunks. Let
$\mathcal{T}\={t_{i}}_{i\=1}^{M}$ denote the corpus of text chunks, and let
$\mathcal{S}\subseteq{1,\ldots,M}$ denote the selected chunk indices. For a
selected set, the retrieved text context is
| | $T(\mathcal{S})\=\Big[\,t_{i}:i\in\mathcal{S}\,\Big],$ | |
| --- | --- | --- |
where $[\cdot]$ denotes concatenation. Given an input $x$ (e.g., a question), a
decoder $\mathrm{Dec}$ produces the output $y$ conditioned on this context:
| | $y\=\mathrm{Dec}\bigl(x,T(\mathcal{S})\bigr).$ | |
| --- | --- | --- |
In a standard RAG pipeline, the selected set $\mathcal{S}$ is obtained from an *external* retrieval function,
$\mathcal{S}\=\mathrm{retrieve}(x,\mathcal{T})$, and $\mathrm{Dec}$ is usually a
separate LLM that generates from the retrieved text.
We focus on *encoder-decoder* models, where the same pretrained model can
encode evidence and decode the answer. The encoder $\mathrm{Enc}$ maps a text
sequence $t$ to token representations
| | $k\=\mathrm{Enc}(t)\in\mathbb{R}^{L_{c}\times d},$ | |
| --- | --- | --- |
where $d$ is the representation dimension. For each corpus chunk, we write
$k_{i}\=\mathrm{Enc}(t_{i})$ and denote the pre-encoded chunk set by
$\mathcal{K}\={k_{i}}_{i\=1}^{M}$. For a selected set $\mathcal{S}$, the encoded
context is
| | $K(\mathcal{S})\=\Big[\,k_{i}:i\in\mathcal{S}\,\Big],$ | |
| --- | --- | --- |
where the same concatenation notation is now applied to token representations.
Generation in our setting conditions on the encoded context rather than on raw
retrieved text, so
| | $y\=\mathrm{Dec}(x,K(\mathcal{S})).$ | |
| --- | --- | --- |
To make the decoder queries explicit, we use a view that isolates
the cross-attention computation. Let
| | $\mathrm{Attention}(q,k,v)\triangleq\mathrm{softmax}!\left(\frac{qk^{\top}}{\sqrt{d}}\right)v.$ | |
| --- | --- | --- |
During the forward pass that computes $\mathrm{Dec}(x,K(\mathcal{S}))$, let $h_{0}\=x$ and
let $q_{\ell}$ denote the query-side state consumed by cross-attention in decoder
layer $\ell$. The simplified
internal recurrence is
| | $\begin{array}[]{rcl}q_{\ell}\&\=\&\Psi_{\ell}(h_{\ell-1}),\\ z_{\ell}\&\=\&\mathrm{Attention}(q_{\ell},K(\mathcal{S}),K(\mathcal{S})),\\ h_{\ell}\&\=\&\Phi_{\ell}(h_{\ell-1},z_{\ell}),\qquad\ell\=1,\ldots,L.\end{array}$ | | (1) |
| --- | --- | --- | --- |
Here $\Psi_{\ell},\Phi_{\ell}$ denotes the layer-specific transformation, including residual updates, self-attention, feed-forward transformations, and normalization. The decoder output is then produced from the final internal state, where $\mathrm{Out}$ denotes the model’s final text-generation head.
| | $y\=\mathrm{Dec}(x,K(\mathcal{S}))\=\mathrm{Out}(h_{L}).$ | |
| --- | --- | --- |
We also mark $\widetilde{\mathrm{Dec}}$ for the same decoder forward pass, with the
intermediate query states exposed:
| | $\forall\ell\=1,\ldots,L:q_{\ell}\=\widetilde{\mathrm{Dec}}_{\ell}(x,K(\mathcal{S})).$ | |
| --- | --- | --- |
### 2.2 Attention-based retrieval
Cross-attention already scores decoder-side query states against encoder-side
token representations. We use the same matching signal to rank chunks before
generation. Our goal is to convert the token-level comparison in
Eq.[1](#S2.E1 "In 2.1 Framework formulation ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") into a single retrieval score for each encoded chunk
$k_{i}$.
To obtain these scores with a pre-trained *frozen* decoder, we augment the
input $x$ with $R$ trainable retrieval tokens
$\rho\in\mathbb{R}^{R\times d}$. Given token embeddings
${x_{j}}_{j\=1}^{L_{q}}$, the retrieval input is
| | $x_{\mathrm{retrieval}}\=\bigl[x_{1},\dots,x_{L_{q}},\rho_{1},\dots,\rho_{R}\bigr].$ | | (2) |
| --- | --- | --- | --- |
To move from token-level attention scores to chunk-level retrieval scores, we
use a scaled ColBERT-style late-interaction score $\mathrm{MaxSim}$ *[colbert]*111The factor $1/\sqrt{d}$ has no effect on MaxSim ranking; it is included only to make the similarity to the attention score explicit..
For sequences $u\in\mathbb{R}^{L_{u}\times d}$ and
$v\in\mathbb{R}^{L_{v}\times d}$,
| | $\mathrm{MaxSim}(u,v)\triangleq\sum_{a\=1}^{L_{u}}\max_{1\leq b\leq L_{v}}\left(\frac{uv^{\top}}{\sqrt{d}}\right)_{a,b}\,,$ | |
| --- | --- | --- |
MaxSim uses the same scaled token-level dot product as attention, but aggregates
by taking the best-matching chunk token for each query token rather than
applying a softmax over all tokens.
With learned per-layer aggregation weights $\alpha_{\ell}$, the score for chunk $i$ is
| | $s_{i}\triangleq\sum_{\ell}{\alpha_{\ell}\mathrm{MaxSim}(q_{\ell},k_{i})}\qquad\mathrm{where}\qquad q_{\ell}\=\widetilde{\mathrm{Dec}}_{\ell}(x_{\mathrm{retrieval}},K(\mathcal{S}_{0}))\,,$ | | (3) |
| --- | --- | --- | --- |
where $\mathcal{S}_{0}$ is the initial chunk selection (See section[2.3](#S2.SS3 "2.3 Initial context selection for retrieval ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") for how $\mathcal{S}_{0}$ is constructed).
We then select the chunks with the largest scores:
| | $\mathcal{S}_{\mathrm{INTRA}}\=\left{i\in{1,\ldots,M}:s_{i}\text{ is among the top-}n\text{ scores}\right}.$ | | (4) |
| --- | --- | --- | --- |
Thus, $\mathcal{S}_{\mathrm{INTRA}}$ is the set of selected chunk indices. Then the ordinary decoder generates from that
selected context:
| | $y\=\mathrm{Dec}\left(x,K(\mathcal{S}_{\mathrm{INTRA}})\right).$ | | (5) |
| --- | --- | --- | --- |
Inference thus consists of two decoder forward passes over the pre-encoded chunk set $\mathcal{K}\={k_{i}}_{i\=1}^{M}$: a retrieval pass $\widetilde{\mathrm{Dec}}(x_{\mathrm{retrieval}},K(\mathcal{S}_{0}))$ that exposes the query states ${q_{\ell}}$ used in Eq.[3](#S2.E3 "In 2.2 Attention-based retrieval ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") to score all chunks, followed by a generation pass over the selected context $K(\mathcal{S}_{\mathrm{INTRA}})$.
### 2.3 Initial context selection for retrieval
A natural initial chunk set $\mathcal{S}_{0}$ in our setting is to select chunks whose encoded representations are most similar to the encoded input. Let $k_{x}\=\mathrm{Enc}(x)$. We define
| | $s_{i}^{(0)}\=\mathrm{MaxSim}(k_{x},k_{i}),\hskip 20.00003pt\mathcal{S}_{0}\=\left{i\in{1,\ldots,M}:s_{i}^{(0)}\text{ is among the top-}n_{0}\text{ scores}\right}.$ | | (6) |
| --- | --- | --- | --- |
This initialization provides the decoder with a useful starting context, but it
does not restrict the final retrieval set. The set $\mathcal{S}_{\mathrm{INTRA}}$
is still selected by scoring the full corpus as in
Eq.[4](#S2.E4 "In 2.2 Attention-based retrieval ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models"), and can therefore include chunks outside
$\mathcal{S}_{0}$. This differs from reranking methods, which only reorder an
initially retrieved candidate set.
Another possible initial selection is the empty set $\mathcal{S}_{0}\=\emptyset$, in which case cross-attention is the identity function and $z_{\ell}\=q_{\ell}$ in Eq.[1](#S2.E1 "In 2.1 Framework formulation ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models").
3 Practical Implementation
--------------------------
We now describe the practical changes needed to adapt pretrained encoder-decoder attention models to the INTRA framework. Our implementation starts from T5Gemma2 *[zhang2025t5gemma]* and modifies the decoder cross-attention so that pre-encoded chunk states can be reused directly for retrieval and generation.
### 3.1 Shared context representations across layers
In T5Gemma2, as in other Transformer-based encoder-decoder models, the cross-attention computation $z_{\ell}\=\mathrm{Attention}(q_{\ell},K(\mathcal{S}),K(\mathcal{S}))$ defined in Eq.[1](#S2.E1 "In 2.1 Framework formulation ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") does not take dot products directly against the raw stored encoder states $K(\mathcal{S})$. Instead, the key inputs to the attention function are subject to layer-specific transformations. The corresponding keys are typically computed by applying an RMSNorm with learned scale $\gamma_{K,\ell}$ and a linear projection matrix $W_{K,\ell}$. Thus, in Eq.[1](#S2.E1 "In 2.1 Framework formulation ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") and related equations, we would need to replace $K(\mathcal{S})$ with $K_{\ell}(\mathcal{S})\=(\mathrm{RMSNorm}(K(\mathcal{S}))\odot\gamma_{K,\ell})W_{K,\ell}$. This would require computing layer-specific representations $K_{\ell}(S)$ to evaluate MaxSim for retrieval, rather than evaluating against a single reusable context across all layers.
To avoid this overhead, we propose reversed query-key projection *Reverse-QWK* (or *RQWK*), a novel technique that stores one normalized encoder representation $\bar{K}(\mathcal{S})\=\mathrm{RMSNorm}(K(\mathcal{S}))$ and moves the learned key scale $\gamma_{K,\ell}$ and projection matrix $W_{K,\ell}$ to the query side, defining a modified query transformation:
| | $\widetilde{q}_{\ell}\=(q_{\ell}W_{K,\ell}^{\top})\odot\gamma_{K,\ell}.$ | |
| --- | --- | --- |
Cross-attention can then be computed against the same normalized encoder states for all layers, maintaining equivalence with Eq.[1](#S2.E1 "In 2.1 Framework formulation ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models"):
| | $z_{\ell}\=\mathrm{Attention}_{\mathrm{RQWK}}(\widetilde{q}_{\ell},\bar{K}(\mathcal{S}),\bar{K}(\mathcal{S}))\=\mathrm{softmax}\left(\frac{\widetilde{q}_{\ell}\bar{K}(\mathcal{S})^{\top}}{\sqrt{d}}\right)\bar{K}(\mathcal{S})$ | | (7) |
| --- | --- | --- | --- |
The MaxSim score in Eq.[3](#S2.E3 "In 2.2 Attention-based retrieval ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") is computed using these same quantities, $\mathrm{MaxSim}(\widetilde{q}_{\ell},\bar{k}_{i})$ (where $\bar{k}_{i}\=\mathrm{RMSNorm}(k_{i})$), so retrieval and attention operate in the same representation space. This preserves the attention scores while allowing retrieval queries from different decoder layers to operate on the same stored chunk pool $K(\mathcal{S})$.
We use Reverse-QWK only as an implementation device; the full derivation including per-head treatment under Group-Query Attention, handling of positional embeddings, and the resulting memory savings are deferred to AppendixLABEL:appendix:reverse_qwk.
### 3.2 Retrieval training objective
Let $\mathcal{O}(x)\subseteq{1,\dots,M}$ denote the oracle evidence chunks
for input $x$ (e.g., a question). When explicit retrieval supervision is available, we train the
retrieval scores $s_{i}$ from Eq.[3](#S2.E3 "In 2.2 Attention-based retrieval ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") with a soft
cross-entropy objective that assigns equal target mass to all oracle chunks:
| | $\mathcal{L}_{\mathrm{retrieval}}\=-\frac{1}{|\mathcal{O}(x)|}\sum_{j\in\mathcal{O}(x)}\log\left(\mathrm{softmax}(s)_{j}\right),$ | |
| --- | --- | --- |
where $s\=[s_{1},\ldots,s_{M}]$. With the decoder frozen, this objective updates
the retrieval tokens $\rho$ in Eq.[2](#S2.E2 "In 2.2 Attention-based retrieval ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") and the layer aggregation weights $\alpha$ in Eq.[3](#S2.E3 "In 2.2 Attention-based retrieval ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models"),
teaching the induced decoder queries to place probability mass on the oracle
evidence set.
### 3.3 Approximate similarities with pooled chunk embeddings
Computing $\mathrm{MaxSim}$ against every token is expensive when chunk length $L_{c}$ is large. For efficient scoring, we replace each encoded chunk $k_{i}\in\mathbb{R}^{L_{c}\times d}$ with a fixed-length mean-pooled sequence $\widehat{k}_{i}\in\mathbb{R}^{L_{p}\times d}$ where $L_{p}\ll L_{c}$. Retrieval scores are computed using $\widehat{k}_{i}$ in place of $k_{i}$.
This approximation is natural for INTRA because the pooled vectors are fixed averages of the model’s own encoder states, requiring no additional compressor or compression-specific training (distinct from latent-compression approaches *[he2026clarabridgingretrievalgeneration]*). We find that small values such as $L_{p}\in{3,5,7}$ substantially reduce MaxSim cost while preserving the shared-representation design.
4 Benchmarks and Experimental Setup
-----------------------------------
We evaluate INTRA on four Wikipedia-based QA benchmarks: HotPotQA *[yang2018hotpotqa]*, 2WikiMultihopQA *[ho2020constructing]*,
MuSiQue *[trivedi2022musique]*, and Natural Questions *[kwiatkowski2019natural]*. Together they span bridge and comparison
reasoning, cleaner two-hop evidence chains, compositionally harder multi-hop
questions, and single-hop open-domain QA. We build one shared retrieval candidate pool for all four benchmarks under a
fixed budget of approximately 100M tokens, containing 759K chunks in total. Full details on the context pool construction and split statistics are provided in AppendixLABEL:appendix:clara_dataset_details.
We compare INTRA against nine retrieval baselines, including sparse lexical methods (TF-IDF *[salton1988termweighting]*, BM25 *[robertson2009bm25]*), dense single-vector models (BGE-large *[xiao2023cpack]*, Qwen3-Embedding-0.6B/4B *[zhang2025qwen3embedding]*), reranking (Jina reranker *[jinaai2024jinarerankerv2]*), hybrid RAG (RRF *[cormack2009reciprocal]*), and a ColBERT-style MaxSim late-interaction baseline *[colbert]* (details in AppendixLABEL:appendix:baseline_details). For retrieval, we report complete-evidence recall at $k\in{5,10,20}$, defined as the fraction of examples where *all* oracle chunks are retrieved. For end-to-end QA, we take the top-5 retrieved chunks, pack their pre-encoded T5Gemma2 representations as cross-attention context, and generate answers with the T5Gemma2 model, reporting exact match (EM) and token-level F1. All experiments use the open retrieval setting.
Implementation Details. We initialize from a T5Gemma2 4B-4B checkpoint, warm-started on the CLaRa QA pretraining dataset *[he2026clarabridgingretrievalgeneration]* and adapted on our training splits. During retrieval training, the encoder and decoder backbones are frozen, optimizing only the retrieval token embeddings $\rho_{i}$ ($\sim 164$K parameters) and layer aggregation weights $\alpha_{l}$ (272 parameters). Initial context $\mathcal{S}_{0}$ uses $n_{0}\=20$ and pooled chunks of length $L_{p}\=7$. At evaluation, QA builds a five-chunk context: the top four retrieved chunks from $\mathcal{S}_{\mathrm{INTRA}}$ plus the top initial context chunk from $\mathcal{S}_{0}$. We then generate with deterministic greedy decoding. Further details and ablations are reported in
AppendicesLABEL:appendix:implementation_details andLABEL:appendix:ablation_study.
5 Results
---------
We organize the results around the three empirical questions that motivate the
paper. First, does INTRA improve retrieval of complete evidence
sets (Section[5.1](#S5.SS1 "5.1 Retrieval Results ‣ 5 Results ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models"))? Second, do those gains translate into better end-to-end answer quality (Section[5.2](#S5.SS2 "5.2 End-to-End Question-Answering Results ‣ 5 Results ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models"))?
Third, what efficiency advantage appears once chunk representations are
reused rather than re-encoded from raw text (Section[5.3](#S5.SS3 "5.3 Efficiency Results ‣ 5 Results ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models"))?
### 5.1 Retrieval Results
Figure[2](#S5.F2 "Figure 2 ‣ 5.1 Retrieval Results ‣ 5 Results ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") reports complete-evidence recall for $k\in{5,10,20}$ across the four evaluation benchmarks.
Complete-evidence recall@k is the fraction of examples for which *all* annotated supporting chunks are retrieved within the top-$k$ results.
We view this metric as the clearest proxy for retrieval quality, because it rewards recovering the full supporting set rather than only incomplete supporting evidence.
*Figure 2: Complete-evidence recall: the percentage of examples for which *all* supporting facts are retrieved.
INTRA performs best on multi-hop benchmarks (HotPotQA, 2Wiki, MuSiQue) that require evidence assembly.
NQ’s single-hop nature minimizes this benefit.*
The main pattern is that INTRA is strongest on multi-hop retrieval settings that
require assembling multiple pieces of evidence (HotPotQA, 2Wiki, MuSiQue). INTRA’s ranking leverages decoder attention weights, which serve as a proxy for the informational requirements of the answer generation process. This advantage is less pronounced on NQ, where
retrieval often reduces to finding one directly supporting passage, leaving less room for
decoder-guided evidence assembly.
AppendixLABEL:appendix:additional_results reports the full
retrieval results.
In Fig.[3](#S5.F3 "Figure 3 ‣ 5.1 Retrieval Results ‣ 5 Results ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") we also compare three top-5 evidence sets: the initial retrieval set $\mathcal{S}_{0}$, the same initial set
reranked by the decoder scores $s_{i}$ from
Eq.[3](#S2.E3 "In 2.2 Attention-based retrieval ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models"), and the final INTRA set
$\mathcal{S}_{\mathrm{INTRA}}$ from Eq.[4](#S2.E4 "In 2.2 Attention-based retrieval ‣ 2 Method ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models").
The results show that reranking $\mathcal{S}_{0}$ is beneficial, but full-corpus INTRA scoring yields the largest gains by recovering evidence absent from the initial pool.
*Figure 3: Complete-evidence recall@5 for the initial set $\mathcal{S}_{0}$, $\mathcal{S}_{0}$ reranked, and the final set $\mathcal{S}_{\mathrm{INTRA}}$. INTRA performs corpus-wide scoring and can recover evidence outside the initial candidate pool.*
### 5.2 End-to-End Question-Answering Results
Table[1](#S5.T1 "Table 1 ‣ 5.2 End-to-End Question-Answering Results ‣ 5 Results ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") evaluates the end-to-end retrieval-and-generation behavior of INTRA.
We vary the retrieval method while keeping the same T5Gemma2 decoder for generation, reporting both exact match (EM) and token-level F1 (full results are in AppendixLABEL:appendix:additional_results).
INTRA surpasses all baselines on multi-hop benchmarks (HotPotQA, 2Wiki, MuSiQue), consistent with the results of Section[5.1](#S5.SS1 "5.1 Retrieval Results ‣ 5 Results ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models").
This is notable because INTRA’s retrieval signal comes from a frozen decoder pretrained only for generation, whereas baselines such as BGE and Qwen-Embedding are pretrained for retrieval on large-scale retrieval corpora (that include HotPotQA and NQ as supervision *[thakur_bge_full_data, zhang2025qwen3embedding]*).
*Table 1: End-to-end question-answering performance across retrieval methods with a fixed T5Gemma2 generator.
INTRA performs best on all multi-hop benchmarks.*
| | HotPotQA | | 2Wiki | | MuSiQue | | NQ | | Average | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Retrieval method | EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM | F1 |
| TF-IDF | 34.2 | 44.5 | 39.0 | 42.7 | 5.3 | 14.6 | 34.9 | 42.9 | 28.4 | 36.2 |
| BM25 | 40.5 | 52.0 | 41.7 | 45.2 | 7.7 | 16.8 | 43.4 | 51.8 | 33.3 | 41.5 |
| MaxSim | 40.7 | 52.2 | 41.6 | 45.8 | 10.1 | 19.8 | 48.4 | 57.8 | 35.2 | 43.9 |
| Hybrid RAG | 43.4 | 54.3 | 46.0 | 49.9 | 10.6 | 20.1 | 50.5 | 59.1 | 37.6 | 45.8 |
| BGE | 41.9 | 53.0 | 46.1 | 49.6 | 10.8 | 19.7 | 52.2 | 61.2 | 37.8 | 45.9 |
| Qwen3-Emb-0.6B | 37.0 | 47.7 | 45.7 | 49.8 | 11.1 | 20.0 | 36.4 | 44.6 | 32.6 | 40.5 |
| Qwen3-Emb-4B | 40.3 | 51.2 | 46.0 | 50.1 | 12.7 | 21.8 | 54.5 | 63.7 | 38.4 | 46.7 |
| Qwen3-Emb-4B + Reranker | 41.6 | 53.6 | 46.8 | 50.8 | 13.3 | 22.5 | 55.1 | 64.2 | 39.2 | 47.5 |
| INTRA | 46.4 | 58.0 | 49.2 | 53.2 | 14.0 | 23.0 | 51.2 | 60.3 | 40.2 | 48.6 |
Table[2](#S5.T2 "Table 2 ‣ 5.2 End-to-End Question-Answering Results ‣ 5 Results ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") compares using a shared decoder for retrieval and generation against coupling an INTRA retriever with a stronger generator. While superior reasoning and parametric knowledge allow more capable generators to boost performance, INTRA retrieval enhances performance by aligning evidence with the decoder’s specific attention patterns. To isolate the impact of generator strength, we measure how much of the EM gap between random and complete-evidence contexts is closed by INTRA:
| | $\mathrm{GapClosure}\=100\cdot\frac{\mathrm{EM}(\mathrm{INTRA})-\mathrm{EM}(\mathrm{random})}{\mathrm{EM}(\mathrm{complete})-\mathrm{EM}(\mathrm{random})},$ | |
| --- | --- | --- |
where the parenthetical term
indicates whether generation uses chunks from $\mathcal{S}_{\mathrm{INTRA}}$, random chunks, or
the complete-evidence (oracle) chunks.
Utilizing the same T5Gemma2 decoder for both retrieval and generation closes the largest average gap, demonstrating the benefit of coupling the retriever and generator. This highlights the need for stronger INTRA backbones, given that open-source encoder-decoder models are currently scarcer and weaker than decoder-only options.
*Table 2: Generator compatibility with a T5Gemma2-INTRA retriever.
Gap closure measures the percentage of the EM gap from random chunks to complete-evidence chunks recovered by INTRA retrieval.
Sharing a decoder across retrieval and generation aligns the retrieved evidence with the generator’s attention, closing the largest gap.*
| Generator | HotpotQA | 2Wiki | MuSiQue | NQ | Average |
| --- | --- | --- | --- | --- | --- |
| Mistral0.3-7B | 50.0% | 21.6% | 24.0% | 56.8% | 38.1% |
| Phi4-3.8B | 55.9% | 18.4% | 18.9% | 68.7% | 40.5% |
| Llama3.1-8B | 61.9% | 38.8% | 23.1% | 71.1% | 48.7% |
| Gemma4-E2B | 63.0% | 47.5% | 25.7% | 74.2% | 52.6% |
| Qwen3.5-9B | 64.0% | 48.0% | 26.1% | 78.5% | 54.1% |
| Qwen3.5-27B | 63.2% | 51.8% | 23.9% | 76.5% | 53.8% |
| T5Gemma2-4B (same as retriever) | 66.4% | 56.8% | 38.5% | 75.9% | 59.4% |
### 5.3 Efficiency Results
INTRA’s encoder-decoder design also yields a direct efficiency benefit. Standard
RAG typically retrieves text, so after retrieval the generator re-encodes the
selected chunks before decoding. INTRA retrieves pre-encoded chunks from
$\mathcal{K}$ instead, and those states feed into generation as decoder
cross-attention memory. Retrieval and generation incur their usual costs111Retrieval complexity scales as $\mathcal{O}(\sqrt{M}L_{q}L_{c})$ in practice using inverted file (IVF) approximate nearest-neighbor (ANN) search e.g. with FAISS or cuVS, *[colbert, Johnson2019, cuVS2026]*.,
but the selected evidence is no longer re-encoded at query time. Table[5.3](#S5.SS3 "5.3 Efficiency Results ‣ 5 Results ‣ Retrieval from Within: An Intrinsic Capability of Attention-Based Models") summarizes this computational trade-off (detailed analysis in AppendixLABEL:appendix:compute_analysis). Furthermore, storing these representations is practical, as storing a 1-billion-token corpus quantized to 8-bit precision requires around 2.5 TB of storage (see AppendixLABEL:appendix:memory_efficiency for details).
*Table 3: Computational trade-offs across full-context prompting,
standard RAG, and INTRA. Variables denote number of corpus chunks $M$, chunk length $L_{c}$, query length $L_{q}$, retrieved chunks $k$, and generation length $L_{g}$. INTRA has the same retrieval and generation terms as
RAG, but avoids re-encoding retrieved evidence during prefilling when chunk
representations are reusable across queries. In our setting where $M\gg k$, full-context prefilling is computationally infeasible.*
\__nicematrix_patch_booktabs:\__nicematrix_revert_colortbl:
Instructions for reporting errors
---------------------------------
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile
support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the
methods listed below:
**Tip:** You can select the relevant text first, to include it in your report.
Our team has already identified [the following issues](https://github.com/arXiv/html_feedback/issues). We appreciate your time reviewing and reporting rendering errors we
may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability
should not be a barrier to accessing research. Thank you for your continued support in championing open access for
all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a [list of packages that need conversion](https://github.com/brucemiller/LaTeXML/wiki/Porting-LaTeX-packages-for-LaTeXML), and welcome [developer contributions](https://github.com/brucemiller/LaTeXML/issues).
BETA