--- title: "Multimodal Latent Reasoning via Predictive Embeddings" authors: ["Ashutosh Adhikari & Mirella Lapata", "School of Informatics", "University of Edimburgh"] url: "https://arxiv.org/abs/2604.08065" sections: 29 estimated_tokens: "16.8k" --- ## Contents - 1 Introduction - 2 Related Work - 3 PEARL: Predictive Latent Reasoning - 3.1 Problem Formulation and Overview - 3.2 Trajectory Encoding - 3.3 Latent Trajectory Predictor - 3.4 Predictive Embedding Objective - 3.5 Next-Latent Prediction - 3.6 Autoregressive Generation Objective - 3.7 Training Objective - 3.8 Inference - 4 Experimental Setting - 5 Results - 5.1 How Does Pearl Compare to Reconstruction-based Methods? - 5.2 What is the Effect of Training Regime on Pearl ? - 5.3 Does Pearl Generalise Across Model Sizes and Architectures? - 5.4 Do Reconstruction-based Methods Actually “Imagine” Images? - 6 Conclusion - Ethics Statement - References - Appendix A Additional Results and Hyperparameter Settings - A.1 Hyperparameter Settings - A.2 Embedding Visualisation - A.3 Ablation for Next-Latent Prediction - Appendix B Examples and Dataset Statistics - B.1 Reconstruction-based Multimodal Latent Reasoning - B.2 Training Data - B.3 Evaluation Data - Appendix C Limitations and Future Work ## Abstract Abstract Tool-augmented multimodal reasoning enables visual language models (VLMs) to improve perception by interacting with external tools (e.g., cropping, depth estimation). However, such approaches incur substantial inference overhead, require specialized supervision, and are prone to erroneous tool calls. We propose Pearl ( P redictive E mbedding A lignment for R easoning in L atent space), a JEPA-inspired framework that learns from expert tool-use trajectories entirely in the latent space, eliminating the need for explicit tool invocation at inference time. Unlike reconstruction-based latent reasoning methods, which autoregressively generate latent tokens and suffer from training–inference mismatch and limited support for multi-step tool use, Pearl directly learns predictive embeddings from multimodal trajectories while preserving the standard vision-language generation pipeline: it is model-agnostic, simple to train, and naturally supports trajectories with multiple tool calls. Experiments across multiple perception benchmarks show that Pearl matches or outperforms standard supervised fine-tuning and reconstruction-based latent reasoning approaches. Furthermore, we provide empirical evidence that reconstruction-based methods primarily learn embeddings rather than image edits in latent space, motivating predictive embedding learning as a more principled alternative. ## 1 Introduction Recent work in multimodal reasoning has explored augmenting vision-language models (VLMs) with external tools (e.g., cropping, object detection, depth estimation) to improve grounded reasoning (Su et al., 2024; Wu et al., 2025b; Su et al., 2025). By enabling models to iteratively manipulate visual inputs, these approaches allow VLMs to “think with images” rather than relying solely on textual reasoning. While interacting with expert tools to edit images is an effective strategy for grounding LLMs in visual context (Yue et al., 2024; Hao et al., 2025), tool-based approaches introduce practical and conceptual challenges. First, invoking external tools incurs substantial inference-time latency and compute overhead (Nichols et al., 2025; Wu et al., 2025a). Second, learning to correctly select and parameterize tools requires specialized supervision (Wu et al., 2025b; Su et al., 2024), and even after such training, erroneous calls waste inference compute and pollute the context with irrelevant information. Third, most approaches assume a homogeneous tool set, whereas handling the diverse tools exposed by complex MCP servers requires advanced planning and instruction-following (Wu et al., 2025a). A promising direction replaces explicit tool use with latent reasoning, where models operate in a continuous embedding space instead of generating discrete intermediate outputs (Hao et al., 2024; Tan et al., 2025; Gozeten et al., 2026). Prior work has explored *reconstruction-based* latent reasoning, in which models autoregressively generate latent tokens intended to represent intermediate visual transformations (Li et al., 2025; Yang et al., 2025; Gu et al., 2025). Borrowing from Coconut (Hao et al., 2024), these methods supervise continuous latent tokens with a reconstruction objective against the outputs of visual tools, while preserving the standard transformer architecture (see Figure [5](#A2.F5) for an overview). Despite their appeal, these approaches suffer from two fundamental limitations. First, they exhibit a training–inference mismatch: during training, models are supervised with as many latent tokens as there are image patch tokens in the tool output, yet at inference only a small, fixed number of latent tokens is decoded, often without improving, and sometimes degrading, performance (Yang et al., 2025; Li et al., 2025). Second, they are typically confined to single-step transformations, failing to support multi-step reasoning over sequences of tool operations. These observations suggest that reconstruction-based methods primarily learn useful *embeddings* rather than genuinely simulating visual transformations in latent space. In this work, we propose Pearl (Predictive Embedding Alignment for Reasoning in Latent space), a JEPA-inspired framework that learns predictive representations from expert tool-use trajectories. Rather than autoregressively generating latent tokens, Pearl predicts trajectory embeddings from an image–question pair (see Figure [1](#S0.F1), right), allowing the model to internalize the effects of tool use without explicit tool invocation. The framework operates entirely in latent space during training, avoids training–inference mismatch, and supports multi-step reasoning over trajectories with multiple tool calls. We instantiate Pearl by jointly optimizing a standard vision–language generation objective with a predictive embedding objective over interleaved multimodal trajectories (see Figure [1](#S0.F1), left), enabling the model to retain its text generation capabilities while learning both the *effects* and *sequencing* of task-relevant transformations. We evaluate Pearl across a range of multimodal reasoning benchmarks, including settings with single and multiple tool calls. Pearl consistently matches or outperforms supervised fine-tuning and reconstruction-based latent reasoning approaches. Moreover, our analysis shows that reconstruction-based methods primarily learn embeddings rather than performing genuine latent “imagination”, supporting predictive embedding learning as a more principled alternative. ## 2 Related Work Tool-augmented Multi-modal Reasoning. A prominent line of work augments vision-language models with external visual tools to improve grounded reasoning (Su et al., 2024; Huang et al., 2025b; Zheng et al., 2025). These approaches enable models to iteratively manipulate images through operations such as cropping, object detection, or spatial transformations, effectively allowing them to “think with images” rather than relying solely on textual reasoning. More advanced systems further integrate specialized tools such as depth estimation or multi-step visual editing pipelines, often interleaving tool execution with chain-of-thought reasoning. To determine when and how to invoke tools, early methods rely on supervised fine-tuning with expert trajectories (Su et al., 2025; Chung et al., 2026) while more recent approaches leverage reinforcement learning to acquire tool-use policies and support multi-step reasoning (Zheng et al., 2025; Su et al., 2024; Geng et al., 2026). Despite their effectiveness, these methods introduce significant practical challenges: tool invocation incurs substantial inference-time overhead, requires specialized supervision for correct tool selection and parameterization, and remains brittle to errors that can propagate through the reasoning process. Furthermore, many approaches assume a fixed or homogeneous tool set, limiting scalability to diverse or dynamically evolving tool environments. We instead explore latent visual reasoning, where the model internalizes the effects of tool use directly in representation space, eliminating the need for explicit tool invocation at inference time. Reconstruction-based Latent Reasoning. Existing work on latent reasoning in multimodal models autoregressively generates latent tokens under a reconstruction objective, aiming to “imagine” intermediate image edits. Concretely, these latent tokens are trained to reconstruct tool outputs, borrowing from approaches such as Coconut (Hao et al., 2024) in the text domain. This requires models to switch between latent and discrete tokens at inference, a complication that Pearl avoids. However, this line of work is largely confined to single tool calls within a reasoning trajectory (Li et al., 2025; Yang et al., 2025; Gu et al., 2025). COVT (Qin et al., 2025) extends this setting to multiple tool calls, but applies a fixed sequence of operations regardless of the input query. This design imposes two key limitations. First, COVT avoids dynamic planning by restricting tool calls to parameter-free operations, precluding actions such as cropping that require query-specific arguments. Second, all tool calls are applied directly to the original image $\mathcal{I}_{0}$ rather than to the outputs of preceding operations, resulting in a shallow tree of independent branches rather than a chain of dependent steps. In contrast, Pearl avoids autoregressive latent generation by predicting a single embedding of the full expert trajectory, naturally supporting multi-step tool use as the prediction target encodes the entire trajectory $\mathcal{R}$ rather than a single transformation. Joint Embedding Predictive Architectures for Language Models. The Joint Embedding Predictive Architecture (JEPA) is a self-supervised learning framework that trains a model to predict the embedding of one view of the data from another, rather than reconstructing raw inputs. It has shown promise as a pre-training objective for multimodal models with V-JEPA2 (Assran et al., 2025) and VL-JEPA (Chen et al., 2026) achieving competitive performance; however, both require substantial pre-training compute to match the performance of current state-of-the-art models. In the textual domain, LLM-JEPA (Huang et al., 2025a) adapts JEPA to fine-tune off-the-shelf language models, avoiding the need to train from scratch. Concretely, LLM-JEPA minimises a distance between the encoded representations of paired text and code, encouraging the model to develop modality-agnostic representations conducive to text-to-code generation. Similarly, V-JEPA2 aligns embeddings from video frames and robotic actions with predicted future frames for video model pre-training. Pearl adapts the JEPA objective to fine-tune an off-the-shelf VLM from expert multimodal tool-use trajectories, treating the image-question pair and the full reasoning trajectory as two views of the same problem. This allows Pearl to internalize the effect of sequential visual tool use in the latent space, without incurring the prohibitive compute of pre-training or departing from the standard image-text-to-text inference pipeline. ## 3 PEARL: Predictive Latent Reasoning ### 3.1 Problem Formulation and Overview We consider a training setting where each example consists of an image-question pair $\mathbf{x}=\langle\mathcal{I}_{0},\mathcal{Q}\rangle$ and an expert multimodal reasoning trajectory $$ $\mathcal{R}=(\mathcal{I}_{1},\mathcal{T}_{1},\mathcal{I}_{2},\mathcal{T}_{2},\dots,\mathcal{I}_{N},\mathcal{T}_{N}),$ $$ where each $\mathcal{I}_{i}$ is an intermediate image produced by an expert visual tool (e.g., crop, highlight, spatial transformation) and each $\mathcal{T}_{i}$ is the associated reasoning text, with the final step containing the answer to $\mathcal{Q}$ (see Figure [1](#S0.F1)). Our goal is to train a VLM that benefits from such tool-use trajectories *without* invoking tools at inference time. In contrast to prior reconstruction-based approaches, which autoregressively generate latent tokens intended to reconstruct intermediate visual edits (Li et al., 2025; Yang et al., 2025; Gu et al., 2025), Pearl (Predictive Embedding Alignment for Reasoning in Latent space) directly predicts a latent representation of the full trajectory from the original image-question pair, preserving the standard VLM inference pipeline while internalizing information from tool-based reasoning during training. Concretely, Pearl encodes $\mathbf{x}$ and $\mathcal{R}$ independently and trains a lightweight predictor to anticipate the trajectory embedding from the input alone. Intuitively, the predictor asks: given only the image and question, can the model anticipate what the expert tool-use trajectory would look like in latent space? This design has three advantages. First, it avoids explicit tool invocation at inference. Second, it avoids the training–inference mismatch of reconstruction-based methods, where models are trained with many latent tokens but decode only a small fixed number at test time. Third, because the prediction target encodes the *entire* multimodal trajectory rather than a single image edit, Pearl naturally supports multiple tool calls. ### 3.2 Trajectory Encoding We instantiate both $\mathrm{Enc}(\mathbf{x})$ and $\mathrm{Enc}(\mathcal{R})$ using the hidden states of an off-the-shelf autoregressive VLM, serializing the two views as follows (see also Figure [1](#S0.F1)): - • the input view consists of the original image-question pair $\langle\mathcal{I}_{0},\mathcal{Q}\rangle$; - • the trajectory view consists of the interleaved sequence $(\mathcal{I}_{1},\mathcal{T}_{1},\ldots,\mathcal{I}_{N},\mathcal{T}_{N})$. For each view, we run a forward pass through the VLM and take the final hidden state of the last token as its sequence representation, following prior work on JEPA-style fine-tuning of decoder-only language models (Huang et al., 2025a): $$ $h_{\mathbf{x}}=\mathrm{Enc}(\mathbf{x}),\qquad h_{\mathcal{R}}=\mathrm{Enc}(\mathcal{R}).$ $$ Using separate forward passes avoids cross-view information leakage and keeps the method architecture-agnostic, at the cost of additional training-time compute. This overhead applies only during training; inference remains identical to standard VLM decoding. A lightweight predictor network then takes the input encoding $h_{\mathbf{x}}$ and produces a predicted version of the trajectory embedding. ### 3.3 Latent Trajectory Predictor To map the input representation $h_{\mathbf{x}}$ to the trajectory latent space, we use a predictor built from the VLM itself. Following prior tied-weights JEPA formulations (Huang et al., 2025a), we append $K$ learnable special tokens [PRED] to the serialized input $\mathbf{x}$, and define the predicted trajectory representation as the hidden state of the final predictor token: $$ $\hat{h}_{\mathcal{R}}=\mathrm{Pred}(h_{\mathbf{x}}).$ $$ Intuitively, the predictor allows the model to perform additional nonlinear computation over the image-question representation before producing the target latent (see visualization in Figure [1](#S0.F1)). When $K=0$, the predictor reduces to the identity map. In practice, using predictor tokens lets us reuse the VLM’s existing self-attention stack rather than introducing a separate MLP or auxiliary transformer, thereby keeping the method simple and parameter-efficient. ### 3.4 Predictive Embedding Objective Our central training signal is a JEPA-style predictive embedding loss that aligns the predicted latent $\hat{h}_{\mathcal{R}}$ with the encoded expert trajectory $h_{\mathcal{R}}$. We define $$ $\mathcal{L}_{\rm JEPA}=D\!\left(\hat{h}_{\mathcal{R}},\mathrm{sg}[h_{\mathcal{R}}]\right),$ (1) $$ where $D(\cdot,\cdot)$ is a distance function and $\mathrm{sg}[\cdot]$ denotes stop-gradient. In our experiments, we use SmoothL1 loss for $D$. This objective encourages the model to learn a compact representation of the *effect* of expert tool use and multimodal reasoning, rather than explicitly reconstructing intermediate image edits. In this sense, PEARL learns predictive trajectory embeddings rather than latent image generation (see Figure [1](#S0.F1)). ### 3.5 Next-Latent Prediction A potential limitation of using the final hidden state of a decoder as a sequence representation is that it may not reliably summarize all relevant preceding context. To encourage hidden states to behave as predictive summary states, we add a next-latent prediction objective inspired by recent work on latent dynamics in transformers (Teoh et al., 2025). Let $h_{t}$ denote the hidden state at time step $t$ within the serialized trajectory. A lightweight latent predictor is trained to forecast future hidden states $\hat{h}_{t+i}$ from the current state $h_{t}$, for a prediction horizon $d$. We optimize $$ $\mathcal{L}_{\text{NextLat}}=\mathbb{E}_{t}\left[\frac{1}{d}\sum_{i=1}^{d}\mathrm{SmoothL1Loss}\left(\mathrm{sg}[h_{t+i}],\hat{h}_{t+i}\right)\right].$ (2) $$ This objective encourages hidden states to be informative about future trajectory evolution, making them better suited for sequence-level latent alignment. Specifically, Teoh et al. (2025) show that optimizing the hidden state transitions as in Equation ([2](#S3.E2)) causes them to converge to belief states, which Kaelbling et al. (1998) define as sufficient statistics of the past history. We view this term as a regularizer that improves the quality of the learned latent representations, rather than as a separate reasoning mechanism. ### 3.6 Autoregressive Generation Objective In addition to latent alignment, we retain the standard VLM training objective over the textual portions of the expert trajectory. Given the interleaved multimodal context, the model is trained to autoregressively predict each token in the textual segments $\mathcal{T}_{1},\dots,\mathcal{T}_{N}$: $$ $\mathcal{L}_{\rm VLM}=-\sum_{n=1}^{N}\sum_{t=1}^{|\mathcal{T}_{n}|}\log p_{\theta}\!\left(\mathcal{T}_{n}^{(t)}\mid\mathcal{I}_{0},\mathcal{Q},\mathcal{I}_{1},\mathcal{T}_{1},\dots,\mathcal{I}_{n},\mathcal{T}_{n}^{(