--- title: "The Landscape of Agentic Reinforcement Learning for LLMs: A Survey" authors: ["Guibin Zhang", "Hejia Geng", "Xiaohang Yu", "Zhenfei Yin", "Zaibin Zhang", "Zelin Tan", "Heng Zhou", "Zhongzhi Li", "Xiangyuan Xue", "Yijiang Li", "Yifan Zhou", "Yang Chen", "Chen Zhang", "Yutao Fan", "Zihu Wang", "Songtao Huang", "Piedrahita-Velez, Francisco", "Yue Liao", "Hongru Wang", "Mengyue Yang", "Heng Ji", "Jun Wang", "Shuicheng Yan", "Philip Torr", "Lei Bai", "University of Oxford", "Shanghai AI Laboratory", "National University of Singapore", "University of Illinois Urbana-Champaign", "Brown University", "University College London", "University of Science and Technology of China", "Imperial College London", "Dalian University of Technology", "Chinese Academy of Sciences", "The Chinese University of Hong Kong", "University of Georgia", "University of California, San Diego", "University of California, Santa Barbara", "University of Bristol", ",", "Corresponding Author"] url: "https://arxiv.org/abs/2509.02547" sections: 155 estimated_tokens: "138.4k" --- ## Contents - 1 Introduction - Synergy between RL and LLMs - LLM Agents. - Research Gap and Our Contributions. - Structure of the Survey. - 2 Preliminary: From LLM RL to Agentic RL - 2.1 Markov Decision Processes - PBRFT. - Agentic RL. - 2.2 Environment State - PBRFT. - Agentic RL. - 2.3 Action Space - 2.4 Transition Dynamics - PBRFT. - Agentic RL. - 2.5 Reward Function - PBRFT. - Agentic RL. - 2.6 Learning Objective - PBRFT. - Agentic RL. - 2.7 RL Algorithms - REINFORCE: The Foundational Policy Gradient - Proximal Policy Optimization (PPO) - Direct Preference Optimization (DPO) - Group Relative Policy Optimization (GRPO) - 3 Agentic RL: The model capability perspective - 3.1 Planning - RL as an External Guide for Planning. - RL as an Internal Driver of Planning. - Prospective: The Synthesis of Deliberation and Intuition. - 3.2 Tool Using - ReAct-style Tool Calling. - Tool-integrated RL. - Prospective: Long-horizon TIR. - 3.3 Memory - RL in RAG-style Memory. - RL for Token-level Memory. - Prospective: RL for Structured Memory. - 3.4 Self-Improvement - RL for Verbal Self-correction. - RL for Internalizing Self-correction. - RL for Iterative Self-training. - Prospective: Meta Evolution of Reflection Ability. - 3.5 Reasoning - Fast Reasoning: Intuitive and Efficient Inference - Slow Reasoning: Deliberate and Structured Problem Solving - Prospective: Integrating Slow Reasoning Mechanisms into Agentic Reasoning - 3.6 Perception - From Passive Perception to Active Visual Cognition - Grounding-Driven Active Perception. - Tool-Driven Active Perception. - Generation-Driven Active Perception. - Audio. - 3.7 Others - 4 Agentic RL: The Task Perspective - 4.1 Search & Research Agent - 4.1.1 Open Source RL Methods - Search from the external Internet - Search from LLM internal knowledge - 4.1.2 Closed Source RL Methods - Industrial Research Agents. - Case Study: OpenAI Deep Research. - 4.2 Code Agent - 4.2.1 RL for Code Generation - Outcome reward RL. - Process reward RL. - 4.2.2 RL for Iterative Code Refinement - Outcome reward RL. - Process reward RL. - 4.2.3 RL for Automated Software Engineering - Outcome reward RL. - Process reward RL. - Remark on closed-source systems. - 4.2.4 Emerging Paradigms - Code World Models - 4.3 Mathematical Agent - 4.3.1 RL for Informal Mathematical Reasoning - Outcome reward RL. - Process reward RL. - 4.3.2 RL for Formal Mathematical Reasoning - Outcome reward RL. - Process reward RL. - Hybrid reward RL. - 4.4 GUI Agent - 4.4.1 RL-free Methods - Vanilla VLM-based GUI Agents - Supervised Fine-Tuning (SFT) with Static Trajectory Data - 4.4.2 RL in Static GUI Environments - 4.4.3 RL in Interactive GUI Environments - 4.5 Vision Agents - Image Tasks. - Video Tasks. - 3D Vision Tasks. - 4.6 Embodied Agents - RL in VLA Navigation Agent. - RL in VLA Manipulation Agent. - Case Study: Voyager. - 4.7 Multi-Agent Systems - RL-Free Multi-Agent Evolution - 4.7.1 RL-Driven Optimization of Non-Parametric Coordination Modules - 4.7.2 RL-Driven Optimization of Selected Agent Policies - 4.7.3 End-to-End Multi-Agent Reinforcement Learning - 4.8 Other Tasks - TextGame. - Table. - Time Series. - General QA. - Social. - 5 Enviroment and Frameworks - 5.1 Environment Simulator - 5.1.1 Web Environments - 5.1.2 GUI Environments - 5.1.3 Coding & Software Engineering Environments - Interactive SWE Environments. - Coding Benchmarks & Datasets. - Programmatic World-Model Environments. - 5.1.4 Domain-specific Environments - Science & Research. - Machine Learning Engineering (MLE). - Biomedical. - Cybersecurity. - 5.1.5 Simulated & Game Environments - 5.1.6 General-Purpose Environments - 5.2 RL Framework - Agentic RL frameworks. - RLHF and LLM fine-tuning frameworks. - General-purpose RL frameworks - 6 Open Challenges and Future Directions - 6.1 Trustworthiness - Security. - Hallucination. - Sycophancy. - 6.2 Scaling up Agentic Training - Computation. - Model Size. - Data Size. - Efficiency. - 6.3 Scaling up Agentic Environments - 6.4 The Mechanistic Debate on RL in LLMs - Case study: Mathematical Reasoning - 6.5 Architectural Patterns for Real-World Agent Deployment - Guardrails and Safety Patterns. - Human-in-the-Loop Verification. - Hierarchical Orchestration. - Inter-Agent Communication Protocols. - 6.6 Broader Social Impact - Dual-Use Risks. - Environmental Sustainability. - Labor Market Implications. - Bias Amplification. - Evaluation Contamination. - 7 Conclusion - References ## Abstract Abstract The emergence of agentic reinforcement learning (Agentic RL) marks a paradigm shift from conventional reinforcement learning applied to large language models (LLM RL), reframing LLMs from passive sequence generators into autonomous, decision-making agents embedded in complex, dynamic worlds. This survey formalizes this conceptual shift by contrasting the degenerate single-step Markov Decision Processes (MDPs) of LLM RL with the temporally extended Partially Observable Markov Decision Processes (POMDPs) that define Agentic RL. Building on this foundation, we propose a comprehensive twofold taxonomy: one organized around core agentic capabilities, including planning, tool use, memory, reasoning, self-improvement, and perception, and the other around their applications across diverse task domains. Central to our thesis is that reinforcement learning serves as the critical mechanism for transforming these capabilities from static, heuristic modules into adaptive, robust agentic behavior. To support and accelerate future research, we consolidate the landscape of open-source environments, benchmarks, and frameworks into a practical compendium. By synthesizing over five hundred recent works, this survey charts the contours of this rapidly evolving field and highlights the opportunities and challenges that will shape the development of scalable, general-purpose AI agents. ## 1 Introduction The rapid convergence of large language models (LLMs) and reinforcement learning (RL) has precipitated a fundamental transformation in how language models are conceived, trained, and deployed. Early LLM RL paradigms largely treated these models as static conditional generators, optimized to produce single-turn outputs aligned with human preferences or benchmark scores. While successful for alignment and instruction-following, such approaches overlook the broader spectrum of sequential decision-making that underpins realistic, interactive settings. These limitations have prompted a shift in perspective: rather than viewing LLMs as passive text emitters, recent developments increasingly frame them as *Agents*, i.e., autonomous decision-makers capable of perceiving, reasoning, planning, invoking tools, maintaining memory, and adapting strategies over extended horizons in partially observable, dynamic environments. We define this emerging paradigm as Agentic Reinforcement Learning (Agentic RL). To more clearly delineate the distinction between the concept of Agentic RL studied in this work and conventional RL approaches, we provide the following definition: In Section [2](#S2), we present a more formal, symbolically grounded distinction between Agentic RL and conventional RL. Prior research relevant to Agentic RL can be broadly grouped into two complementary threads: Synergy between RL and LLMs and LLM Agents, detailed as follows: ##### Synergy between RL and LLMs The second line of research investigates how reinforcement learning algorithms are applied to improve or align LLMs. A primary branch, RL for training LLMs, leverages on-policy (e.g., proximal policy optimization (PPO) (Schulman et al., 2017) and Group Relative Policy Optimization (GRPO) (Shao et al., 2024b)) and off-policy (e.g., actor–critic, Q-learning (Mnih et al., 2013)) methods to enhance capabilities such as instruction-following, ethical alignment, and code generation (Srivastava and Aggarwal, 2025; Wang et al., 2025m; 2024c). A complementary direction, LLMs for RL, examines the deployment of LLMs as planners, reward designers, goal generators, or information processors to improve sample efficiency, generalization, and multi-task planning in control environments, with systematic taxonomies provided by (Cao et al., 2025c). RL has also been integrated throughout the LLM lifecycle: from data generation (Guo et al., 2025b; Wan et al., 2025a) and pretraining (Dong et al., 2025a) to post-training and inference (Chow et al., 2025), as surveyed by (Guo and Wang, 2025). The most prominent branch here is post-training alignment, notably Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017), along with extensions such as Reinforcement Learning from AI Feedback (RLAIF) (Bai et al., 2022) and Direct Preference Optimization (DPO) (Rafailov et al., 2023; Wang et al., 2024j; Xiao et al., 2024; Liu et al., 2025k; Srivastava and Aggarwal, 2025) ##### LLM Agents. LLM-based agents represent an emerging paradigm in which LLMs act as autonomous or semi-autonomous decision-making entities (Wang et al., 2025d; Li et al., 2025r), capable of reasoning, planning, and executing actions in pursuit of complex goals. Recent surveys have sought to map this landscape from complementary perspectives. Luo et al. (2025a) propose a methodology-centered taxonomy that connects architectural foundations, collaboration mechanisms, and evolutionary pathways, while Plaat et al. (2025) emphasizes the core capabilities of reasoning, acting, and interacting as defining features of agentic LLMs. Tool use, encompassing retrieval-augmented generation (RAG) and API utilization, is a central paradigm, extensively discussed in Li (2025) and further conceptualized by Wang et al. (2024k). Planning and reasoning strategies form another pillar, with surveys such as Masterman et al. (2024) and Kumar et al. (2025) highlighting common design patterns like plan-execute-reflect loops, while Tao et al. (2024) extend this to self-evolution, where agents iteratively refine knowledge and strategies without substantial human intervention. Other directions explore collaborative, cross-modal, and embodied settings, from multi-agent systems (Aratchige and Ilmini, 2025) to multimodal integration (Durante et al., 2024), and brain-inspired architectures with memory and perception (Liu et al., 2025a). ##### Research Gap and Our Contributions. The recent surge in research on LLM agents and RL-enhanced LLMs reflects two complementary perspectives: one explores what large language models can do as the core of autonomous agents, while the other focuses on how reinforcement learning can optimize their behavior. However, despite the breadth of existing work, a unified treatment of *Agentic RL*, which conceptualizes LLMs as policy-optimized agents embedded in sequential decision processes, remains lacking. Current studies often examine isolated capabilities, domains, or custom environments, with inconsistent terminology and evaluation protocols, making systematic comparison and cross-domain generalization difficult. To bridge this gap, we present a coherent synthesis that connects theoretical foundations with algorithmic approaches and practical systems. We formalize Agentic RL through Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs) abstractions to distinguish it from classical LLM RL paradigms, and introduce a capability-centered taxonomy that includes planning, tool use, memory, reasoning, reflection (self-improvement), and interaction as RL-optimizable components. Furthermore, we consolidate representative tasks, environments, frameworks, and benchmarks that support agentic LLM training and evaluation, and conclude by discussing open challenges and outlining promising future directions for scalable, general-purpose agentic intelligence. Overall, we aim to further clarify the research scope of this survey: ##### Structure of the Survey. This survey is organized to progressively build a unified understanding of Agentic RL from conceptual foundations to practical implementations. Section [2](#S2) formalizes the paradigm shift to Agentic RL through an MDP/POMDP lens. Section [3](#S3) examines Agentic RL from the capability perspective, categorizing key modules such as planning, reasoning, tool use, memory, self-improvement, perception, and others. Section [6](#S4.F6) explores applications across domains, including search, GUI navigation, code generation, mathematical reasoning, and multi-agent systems. Section [5](#S5) consolidates open-source environments and RL frameworks that underpin experimentation and benchmarking. Section [6](#S6) discusses open challenges and future directions towards scalable, adaptive, and reliable agentic intelligence, and Section [7](#S7) concludes the survey. The overall structure is also illustrated in Figure [1](#S1.F1). Figure: Figure 1: The primary organizational structure of the survey. ## 2 Preliminary: From LLM RL to Agentic RL LLMs are initially pre-trained using behavior cloning, which applies maximum likelihood estimation (MLE) to static datasets such as web-scraped text corpora. Subsequent post-training methods enhance capabilities and align outputs with human preferences—transforming them beyond generic web-data replicators. A common technique is supervised fine-tuning (SFT), where models are refined on human-generated (prompt, response) demonstrations. However, procuring sufficient high-quality SFT data remains challenging (Maosongcao et al., 2025; Szep et al., 2025; Han et al., 2025). Reinforcement fine-tuning (RFT) offers an alternative by optimizing models through reward functions, circumventing dependence on behavioral demonstrations. In early RFT research, the core objective is to optimize LLMs through human feedback (Christiano et al., 2017; Ouyang et al., 2022) or data preferences (Rafailov et al., 2023), aligning them with human preferences (RLHF) or directly with data preferences (as in DPO).(^1^11Although DPO is another form of optimization objective in RLHF, its complexity is optimized from the perspective of the training process, so it is necessary to distinguish between pure RLHF and DPO.) This preference-based RFT (PBRFT) primarily involves learning reward model optimization for LLMs on a fixed preference dataset, or directly implementing it using data preferences. With the release of LLMs such as OpenAI o1 (OpenAI et al., 2024) and DeepSeek-R1 (DeepSeek-AI et al., 2025) that possess reasoning capabilities, their improved performance and cross-domain generalization have garnered widespread attention. With the release of models like OpenAI o3 (OpenAI Team, 2025), which possess both self-evolving reasoning capabilities and support for tool use, researchers are beginning to contemplate how to deeply integrate LLMs with downstream tasks through reinforcement learning methods. Subsequently, researchers have shifted their focus from PBRFT, aimed at optimizing fixed preference datasets, to agentic reinforcement learning tailored for specific tasks and dynamic environments. In this section, we provide a formalization of the paradigm shift from PBRFT to the emerging framework of agentic reinforcement learning (Agentic RL). While both approaches leverage RL techniques to improve LLMs’ performance, they fundamentally differ in their underlying assumptions, task structure, and decision-making granularity. Figure [2](#S2.F2) illustrates the paradigm shift from LLM RL to Agentic RL. Figure: Figure 2: Paradigm shift from LLM RL to Agentic RL. We draw inspiration from (Kumar et al., 2025). The fan-shaped design reflects the outward growth of the RL formulation—from traditional RL (inner), to LLM RL, to full Agentic RL (outer). Color-coded regions represent: red = features specific to LLM RL; teal = features required for Agentic RL; purple = existing Agentic RL implementations. Arrows point outward to indicate increasing interaction breadth (tool use, web browsing, dynamic environments) as one moves toward more agentic settings. Refer to caption: 2509.02547v5/figures/biu2.jpg ### 2.1 Markov Decision Processes The Markov decision process (MDP) for the RL fine-tuning process can be formalized as a seven-element tuple $\langle\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{P},\mathcal{R},T,\gamma\rangle$, where $\mathcal{S}$ represents the state space and $\mathcal{O}$ is the observation space of the agent. $\mathcal{A}$ denotes the action space. $\mathcal{R}$ is defined as the reward function, $\mathcal{P}$ encapsulates the state transition probabilities, $T$ signifies the task horizon, and $\gamma$ is the discount factor. By casting both preference-based RFT and Agentic RL as MDPs or POMDPs, we clarify the theoretical implications of treating LLMs either as static sequence generators or as interactive, decision-capable agents embedded within dynamic environments. ##### PBRFT. The RL training process of PBRFT is formalized as a degenerate MDP defined by the tuple: $$ $\langle\mathcal{S}_{\text{trad}},\mathcal{A}_{\text{trad}},\mathcal{P}_{\text{trad}},\mathcal{R}_{\text{trad}},T=1,\gamma=1\rangle.$ (1) $$ ##### Agentic RL. The RL training process of Agentic RL is modeled as a POMDP: $$ $\langle\mathcal{S}_{\text{agent}},\mathcal{A}_{\text{agent}},\mathcal{P}_{\text{agent}},\mathcal{R}_{\text{agent}},\gamma,\mathcal{O}\rangle.$ (2) $$ where the agent receives observations $o_{t}=O(s_{t})$ based on the state $s_{t}\in\mathcal{S}_{\text{agent}}$. The primary distinctions between PBRFT and Agentic RL are delineated in Table [1](#S2.T1). In summary, PBRFT optimizes sequences of output sentences within a fixed dataset under full observations, whereas Agentic RL optimizes semantic-level behaviors in variable environments characterized by partial observations. **Table 1: Formal comparison between traditional PBRFT and Agentic RL.** | Concept | Traditional PBRFT | Agentic RL | | --- | --- | --- | | $\mathcal{S}$: State space | $\{s_{0}\}$ (single prompt); episode ends immediately. | $s_{t}\in\mathcal{S}_{\text{agent}}$; $o_{t}=O(s_{t})$; horizon $T>1$. | | $\mathcal{A}$: Action space | Pure text sequences. | $\mathcal{A}_{\text{text}}\cup\mathcal{A}_{\text{action}}$. | | $\mathcal{P}$: Transition | Deterministic transition to the terminal state. | Dynamic transition function $P(s_{t+1}\mid s_{t},a_{t})$. | | $\mathcal{R}$: Reward | Single scalar $r(a)$. | Step‑wise $R(s_{t},a_{t})$; combines sparse task and dense sub‑rewards. | | $J(\theta)$: Objective | $\mathbb{E}_{a\sim\pi_{\theta}}[r(a)]$. | $\mathbb{E}_{\tau\sim\pi_{\theta}}[\sum_{t}\gamma^{t}R(s_{t},a_{t})]$. | ### 2.2 Environment State ##### PBRFT. In the training process, each episode starts from a single prompt state $s_{0}$; the episode terminates immediately after the model emits one response. Formally, the underlying MDP degenerates to a *single-step* decision problem with horizon $T=1$. The state space reduces to a single static prompt input: $$ $\mathcal{S}_{\text{trad}}=\{\text{prompt}\}.$ (3) $$ ##### Agentic RL. The LLM agent acts over multiple time-steps in a POMDP. Let $s_{t}\in\mathcal{S}_{\text{agent}}$ denote the full world state and the LLM agent gets observation $O_{t}$ based on the current state $o_{t}=\mathcal{O}(s_{t})$. The LLM agent chooses an action $a_{t}$ based on the current observation $o_{t}$, and the state evolves over time: $$ $s_{t+1}\sim P(s_{t+1}\mid s_{t},a_{t}).$ (4) $$ as the agent accumulates intermediate signals such as retrieved tool results, user messages, or environment feedback. The interaction is thus inherently dynamic and temporally extended. ### 2.3 Action Space In the Agentic RL setting, the LLM’s action space comprises two distinct subspaces: $$ $\mathcal{A}_{\text{agent}}=\mathcal{A}_{\text{text}}\cup\mathcal{A}_{\text{action}}.$ (5) $$ Here, $\mathcal{A}_{\text{text}}$ denotes the space of free-form natural language tokens emitted via autoregressive decoding, while $\mathcal{A}_{\text{action}}$ denotes the space of abstract, non-linguistic actions, which is usually delimited in the output stream by special tokens and . These actions may invoke external tools (e.g., call("search", "Einstein")) or interact with an environment (e.g., move("north")), depending on task requirements. Notably, $\mathcal{A}_{\text{action}}$ is recursively constructed, such that an element $a\in\mathcal{A}_{\text{action}}$ may itself represent a sequence $(a_{1},\dots,a_{k})$ of primitive actions, thus unifying primitive and composite actions within the same space. Formally, the two subspaces differ in semantics and functional role: $\mathcal{A}_{\text{text}}$ defines the space of outputs intended for human or machine interpretation without directly altering the external state, whereas $\mathcal{A}_{\text{action}}$ defines the space of environment-interactive behaviors that either (i) acquire new information through tool invocations, or (ii) modify the state of a physical or simulated environment. This distinction enables a unified policy jointly to model language generation and environment interaction within the same RL formulation. ### 2.4 Transition Dynamics ##### PBRFT. In conventional PBRFT, the transition dynamics are deterministic: the next state is determined once an action is taken, as follows: $$ $\mathcal{P}(s_{1}\mid s_{0},a)=1,\quad\text{where there is no uncertainty.}$ (6) $$ ##### Agentic RL. In Agentic RL, the environment evolves under uncertainty according to $$ $s_{t+1}\sim\mathcal{P}(s_{t+1}\mid s_{t},a_{t}),\quad a_{t}\in\mathcal{A}_{\text{text}}\cup\mathcal{A}_{\text{action}}.$ (7) $$ Text actions $(\mathcal{A}_{\text{text}})$ generate natural language outputs without altering the environmental state. Structured actions $(\mathcal{A}_{\text{action}})$, delimited by and , can either query external tools or directly modify the environment. This sequential formulation contrasts with the one-shot mapping of PBRFT, enabling policies that iteratively combine communication, information acquisition, and environment manipulation. ### 2.5 Reward Function ##### PBRFT. PBRFT commonly features a reward function with verifiable response correctness, which may be implemented using either a rule-based verifier (DeepSeek-AI et al., 2025) or a neural network-parameterized reward model (Zhong et al., 2025). Regardless of the implementation approach, its core follows the equation: $$ $\mathcal{R}_{\text{trad}}(s_{0},a)=r(a).$ (8) $$ where $r:\mathcal{A}\!\to\!\mathbb{R}$ is a scalar score supplied by a human- or AI-preference model, with no intermediate feedback. ##### Agentic RL. The reward function of the LLM agent is based on the downstream task. $$ $\mathcal{R}_{\text{agent}}(s_{t},a_{t})=\begin{cases}r_{\text{task}}&\text{on task completion},\[2.0pt] r_{\text{sub}}(s_{t},a_{t})&\text{for step-level progress},\[2.0pt] 0&\text{otherwise}.\end{cases}$ (9) $$ allowing dense, sparse, or learned rewards (e.g., unit-test passes, symbolic verifier success). ### 2.6 Learning Objective ##### PBRFT. The optimization objective of PBRFT is to maximize the response reward based on the policy $\pi_{\theta}$: $$ $J_{\text{trad}}(\theta)=\mathbb{E}_{a\sim\pi_{\theta}}\bigl[r(a)\bigr].$ (10) $$ No discount factor is required; optimization resembles maximum-expected-reward sequence modeling. ##### Agentic RL. The optimization objective of Agentic RL is to maximize the discounted reward: $$ $J_{\text{agent}}(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\,\sum_{t=0}^{T-1}\gamma^{t}R_{\text{agent}}(s_{t},a_{t})\right],\qquad 0<\gamma<1.$ (11) $$ This objective is optimized via policy‑gradient or value‑based methods with exploration and long‑term credit assignment. PBRFT focuses on single-turn text quality alignment without explicit planning, tool use, or environmental feedback, while Agentic RL involves multi-turn planning, adaptive tool invocation, stateful memory, and long‑horizon credit assignment, enabling the LLM to function as an autonomous decision‑making agent. ### 2.7 RL Algorithms In contemporary research, RL algorithms constitute a pivotal component in both PBRFT and Agentic RL frameworks. Different RL algorithms demonstrate distinct sample efficiency and performance characteristics, each offering a unique approach to the central challenge of aligning model outputs with complex, often subjective, human goals. The canonical methods, such as REINFORCE, PPO (Schulman et al., 2017), GRPO (DeepSeek-AI et al., 2025), and DPO (Rafailov et al., 2023), form a spectrum from general policy gradients to specialized preference learning. We next introduce each of these four classic algorithms and provide a comparison of popular variants from each family in Table [2](#S2.T2). ##### REINFORCE: The Foundational Policy Gradient As one of the earliest policy gradient algorithms, REINFORCE (Williams, 1992) provides the foundational theory for training stochastic policies. It operates by increasing the probability of actions that lead to high cumulative reward and decreasing the probability of those that lead to low reward. Its objective function is given by: $$ $\nabla_{\theta}J(\theta)=\mathbb{E}_{s_{0}}\left[\frac{1}{N}\sum_{i=1}^{N}\left(\mathcal{R}(s_{0},a^{(i)})-b(s_{0})\right)\nabla_{\theta}\log\pi_{\theta}(a^{(i)}|s_{0})\right].$ (12) $$ where $a^{(i)}\sim\pi_{\theta}(a|s_{0})$ is the $i$-th sampled response, $\mathcal{R}(s_{0},a)$ denotes the final rewards received on task completion, and $b(s)$ is a baseline function to reduce the variance of the policy gradient estimate. In general, $b(s)$ can be any function, including random variables. In practice, $b(s)$ is commonly instantiated as the value function $V(s)$. Despite with advantages of the concise formula and easy implementation, REINFORCE suffers from drawbacks such as high variance in gradient estimates, sample inefficiency, sensitivity to learning rate and the lack of a critic (value estimator). ##### Proximal Policy Optimization (PPO) PPO (Schulman et al., 2017) became the dominant RL algorithm for LLM alignment due to its stability and reliability. It improves upon vanilla policy gradients by limiting the update step to prevent destructively large policy changes. Its primary clipped objective function is: $$ $L_{PPO}(\theta)=\frac{1}{N}\sum_{i=1}^{N}\min\left(\frac{\pi_{\theta}(a_{t}^{(i)}|s_{t})}{\pi_{\theta_{old}}(a_{t}^{(i)}|s_{t})}A(s_{t},a_{t}^{(i)}),\;\;\mathrm{clip}\left(\frac{\pi_{\theta}(a_{t}^{(i)}|s_{t})}{\pi_{\theta_{old}}(a_{t}^{(i)}|s_{t})},1-\epsilon,1+\epsilon\right)A(s_{t},a_{t}^{(i)})\right).$ (13) $$ where $a_{t}^{(i)}\sim\pi_{\theta_{old}}(a|s_{t})$ is the $i$-th sampled response from the old policy $\pi_{\theta_{old}}$, whose update is delayed. $A_{t}$ is the estimated advantage given by $$ $A(s_{t},a_{t})=\mathcal{R}(s_{t},a_{t})-V(s_{t}).$ (14) $$ where $V_{\theta}(s)$ is the learned value function, i.e., the expectation $\mathbb{E}_{a\sim\pi_{\theta}(a|s)}[\mathcal{R}(s,a)]$, which is typically, but not necessarily, derived from a critic network that is of the same size as the policy network. The clip term prevents the probability ratio from moving too far from 1, ensuring stable updates. The estimation of the advantage function plays a predominant role in the performance of PPO. Recent variants have concentrated on reducing the bias (Kazemnejad et al., 2024) or variance (Yue et al., 2025b) in the advantage estimation. Meanwhile, some other variants make improvements from the perspectives of stable policy update mechanisms (Liu et al., 2025s) or mitigating sparse rewards (Dai et al., 2025). Despite these improvements, a remaining drawback is its reliance on a separate critic network for advantage estimation, which substantially increases the parameter count during training. ##### Direct Preference Optimization (DPO) DPO represents a groundbreaking shift by entirely bypassing the need for a separate explicit reward model. It reframes the problem of maximizing a reward under a KL-constraint as a likelihood-based objective on human preference data. Given a dataset of preferences $D=\{(y_{w},y_{l})\}$, where $y_{w}$ is the preferred response and $y_{l}$ is the dispreferred one, the DPO loss is: $$ $L_{DPO}(\pi_{\theta};\pi_{ref})=-\mathbb{E}_{(x,y_{w},y_{l})\sim D}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{ref}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{ref}(y_{l}|x)}\right)\right].$ (15) $$ where $\pi_{ref}$ is a reference policy (usually the initial SFT model), and $\beta$ is a hyperparameter. While DPO eliminates the critic, its performance is intrinsically tied to the quality and coverage of its static preference dataset. Variants have emerged to address its limitations via involving external or online data (Ethayarajh et al., 2024; Hong et al., 2024a). In addition, some other work attempts to improve by introducing generalized optimization objectives (Gheshlaghi Azar et al., 2024) or sophisticated implicit reward mechanisms (Meng et al., 2024; Lai et al., 2024; Hong et al., 2025a). ##### Group Relative Policy Optimization (GRPO) The remarkable success achieved by DeepSeek (Guo et al., 2025a) has catalyzed significant research interest in GRPO. Proposed to address the inefficiency of PPO’s large critic, GRPO introduces a novel, lightweight evaluation paradigm. It operates on groups of responses, using their relative rewards within a group to compute advantages, thus eliminating the need for an absolute value critic. The core GRPO objective can be conceptualized as: $$ $L_{GRPO}=\frac{1}{G}\sum_{g=1}^{G}\min\left(\frac{\pi_{\theta}(a_{t}^{(g)}|s_{t}^{(g)})}{\pi_{\theta_{old}}(a_{t}^{(g)}|s_{t}^{(g)})}\hat{A}(s_{t}^{(g)},a_{t}^{(g)}),\;\;\mathrm{clip}\left(\frac{\pi_{\theta}(a_{t}^{(g)}|s_{t}^{(g)})}{\pi_{\theta_{old}}(a_{t}^{(g)}|s_{t}^{(g)})},1-\epsilon,1+\epsilon\right)\hat{A}(s_{t}^{(g)},a_{t}^{(g)})\right).$ (16) $$ where a group of outputs $\{(s_{0}^{(g)},a_{0}^{(g)},\ldots,s_{T-1}^{(g)},a_{T-1}^{(g)})\}_{g=1}^{G}$ is sampled from the old policy $\pi_{\theta_{old}}$. The advantage function is estimated by $$ $\hat{A}(s_{t},a_{t})=\frac{\mathcal{R}(s_{t},a_{t})-\mathrm{mean}(\mathcal{R}(s_{t}^{(1)},a_{t}^{(1)}),\ldots,\mathcal{R}(s_{t}^{(G)},a_{t}^{(G)}))}{\mathrm{std}(\mathcal{R}(s_{t}^{(1)},a_{t}^{(1)}),\ldots,\mathcal{R}(s_{t}^{(G)},a_{t}^{(G)}))}.$ (17) $$ This group-relative approach is highly sample-efficient and reduces computational overhead. However, the group-based advantage estimation is vulnerable to high variance and low accuracy. Consequently, a series of novel algorithms derived from the GRPO framework have been subsequently proposed (see Table [2](#S2.T2)), aiming to substantially improve its advantage estimation. **Table 2: Comparison of the popular variants of the PPO, DPO, and GRPO families. Clip corresponds to preventing the policy ratio from moving too far from 1 for ensuring stable updates. KL penalty corresponds to penalizing the KL divergence between the learned policy and the reference policy for ensuring alignment.** | Method | Objective Type | Key Mechanism | | --- | --- | --- | | PPO family | | | | PPO (Schulman et al., 2017) | Policy gradient | Policy ratio clipping | | VAPO (Yue et al., 2025b) | Policy gradient | Adaptive KL penalty + variance control | | LitePPO (Liu et al., 2025s) | Policy gradient | Stable advantage updates | | PF-PPO (Zhang et al., 2025c) | Policy gradient | Policy filtration | | VinePPO (Kazemnejad et al., 2024) | Policy gradient | Unbiased value estimates | | PSGPO (Dai et al., 2025) | Policy gradient | Process supervision | | DPO family | | | | DPO (Rafailov et al., 2023) | Preference optimization | Implicit reward related to the policy | | $\beta$-DPO (Wu et al., 2024) | Preference optimization | Dynamic KL coefficient | | SimPO (Meng et al., 2024) | Preference optimization | Use the average log probability of a sequence as the implicit reward | | IPO (Gheshlaghi Azar et al., 2024) | A special case of a more general objective exclusively expressed in terms of pairwise preferences | Always regularizes its solution towards a preference policy by controlling the gap between the log-likelihood ratios,
which avoids the over-fitting to the preference dataset. | | KTO (Ethayarajh et al., 2024) | Knowledge transfer optimization | Teacher stabilization | | ORPO (Hong et al., 2024a) | Online regularized
preference optimization | Online stabilization | | Step-DPO (Lai et al., 2024) | Preference optimization | Step-wise supervision | | LCPO (Hong et al., 2025a) | Preference optimization | Length preference with limited data and training | | GRPO family | | | | GRPO (DeepSeek-AI et al., 2025) | Policy Gradient
under group-based reward | Group-based relative reward
to eliminate value estimates | | DAPO (Yu et al., 2025e) | Surrogate of GRPO’s | Decoupled
clip and dynamic sampling | | GSPO (Zheng et al., 2025a) | Surrogate of GRPO’s | Define the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization | | GMPO (Zhao et al., 2025f) | Surrogate of GRPO’s | Geometric mean of token-level rewards | | ProRL (Liu et al., 2025h) | Same as GRPO’s | Reference policy reset | | Posterior-GRPO (Fan et al., 2025a) | Same as GRPO’s | Reward only successful processes | | Dr.GRPO (Liu et al., 2025r) | Unbiased GRPO’s objective | Eliminate the bias in
optimization of
GRPO | | Step-GRPO (Zhang et al., 2025k) | Same as GRPO’s | Rule-based reasoning rewards | | SRPO (Zhang et al., 2025t) | Same as GRPO’s | Two-staged history-resampling | | GRESO (Zheng et al., 2025b) | Same as GRPO’s | Pre-rollout filtering | | StarPO (Wang et al., 2025v) | Same as GRPO’s | Reasoning-guided actions for
multi-turn interactions | | GHPO (Liu et al., 2025u) | Policy gradient | Adaptive prompt refinement | | Skywork R1V2 (Wang et al., 2025i) | GRPO’s with hybrid reward signal | Selective sample buffer | | ASPO (Lin and Xu, 2025) | GRPO’s with shaped advantage function | Apply a clipped bias directly to advantage function | | TreePo (Li et al., 2025n) | Same as GRPO’s | Self-guided policy rollout for reducing the compute burden | | EDGE-GRPO (Zhang et al., 2025u) | Same as GRPO’s | Entropy-driven advantage and duided error correction to mitigate
advantage collapse | | DARS (Yang et al., 2025h) | Same as GRPO’s | Reallocate compute
from medium-difficulty to the hardest problems via multi-stage rollout sampling | | CHORD (Zhang et al., 2025r) | Weighted sum of GRPO’s and Supervised Fine-Tuning losses | Reframe Supervised Fine-Tuning as a dynamically weighted
auxiliary objective within the on-policy RL process | | PAPO (Wang et al., 2025u) | Surrogate of GRPO’s | Encourage learning to perceive while learning to reason through the Implicit Perception Loss | | Pass@k Training (Chen et al., 2025l) | Same as GRPO’s | Pass@k metric as the reward to continually train a model | ## 3 Agentic RL: The model capability perspective Figure: Figure 3: The agent–environment interaction and RL loop for agentic LLMs. Core agentic capabilities drive action generation, while the environment provides feedback and rewards, which are aggregated through RL-based optimization across diverse task domains (“Collab.” denotes tasks requiring explicit task division and multi-agent coordination). Refer to caption: 2509.02547v5/figures/env2_tmlr.jpg In this section, we conceptually characterize Agentic RL as the principled training of an autonomous agent composed of a set of key abilities/modules, i.e., planning (Section [3.1](#S3.SS1)), tool use (Section [3.2](#S3.SS2)), memory (Section [3.3](#S3.SS3)), self-improvement (Section [3.4](#S3.SS4)), reasoning (Section [3.5](#S3.SS5)), perception (Section [3.6](#S3.SS6)), and others (Section [3.7](#S3.SS7)), following the classic LLM agent definition (Weng, 2023; Shang et al., 2025b), as demonstrated in [Figure˜5](#S3.F5). Traditionally, an agent pairs an LLM with mechanisms for planning (e.g., task decomposition and plan selection) (Wei et al., 2025a), reasoning (chain-of-thought or multi-turn inference) (Zhang et al., 2024c), external tool invocation (Qin et al., 2024b), long- and short-term memory, and iterative reflection to self-correct and refine behavior. Agentic RL thus treats these components not as static pipelines but as interdependent policies that can be jointly optimized: RL for planning learns multi-step decision trajectories; RL for memory shapes retrieval and encoding dynamics; RL for tool use optimizes invocation timing and fidelity; and RL for reflection drives internal self‑supervision and self-improvement. Consequently, our survey systematically examines how RL empowers planning, tool use, memory, reflection, and reasoning in subsequent subsections. We aim to provide a high-level conceptual delineation of RL’s applications for agent capabilities, rather than an exhaustive enumeration of all related work, which we provide in Section [6](#S4.F6). ### 3.1 Planning Planning, the deliberation over a sequence of actions to achieve a goal, constitutes a cornerstone of artificial intelligence, demanding complex reasoning, world knowledge, and adaptability (Newell et al., 1958). Initial efforts leveraged the innate capabilities of LLMs through prompting-based methods (Huang et al., 2024a; Yao et al., 2023b). For example, Modular Agentic Planner (MAP) (Webb et al., 2025) introduces a brain-inspired, modular architecture that decomposes planning into specialized LLM modules for conflict monitoring, state evaluation, and coordination. However, these approaches lacked a mechanism for adaptation through experience (Wei et al., 2025a). RL has emerged as a powerful paradigm to address this gap, enabling agents to refine their planning strategies by learning from environmental feedback. The integration of RL into agent planning manifests in two distinct paradigms, distinguished by whether RL functions as an external guide to a structured planning process or as an internal driver that directly evolves the LLM’s intrinsic planning policy, which we will detail below. ##### RL as an External Guide for Planning. One major paradigm frames RL as an external guide to the planning process, where the LLM’s primary role is to generate potential actions within a structured search framework. Here, RL is not employed to fine-tune the LLM’s generative capabilities directly, but rather to train an auxiliary value or heuristic function (Wei et al., 2025a). This learned function then guides a classical search algorithm, such as Monte Carlo Tree Search (MCTS), by evaluating the quality of different planning trajectories. Representative works like RAP (Hao et al., 2023) and LATS (Zhou et al., 2024a) exemplify this approach. Planning without Search (Hong et al., 2025d) extends this idea by leveraging offline goal-conditioned RL to learn a language-based value critic that guides LLM reasoning and planning without updating the LLM’s parameters. In this configuration, the LLM acts as a knowledge-rich action proposer, while RL provides adaptive, evaluative feedback for efficient exploration. Beyond static guidance, Learning When to Plan (Paglieri et al., 2025) formulates dynamic planning as an RL-driven test-time compute allocation problem, training agents to decide when to invoke explicit planning to balance reasoning performance against computational cost. Conversely, MAPF-DT (Atasever et al., 2025) explores the reverse direction, employing Decision Transformer–based offline RL for decentralized multi-agent path planning, with LLM guidance enhancing adaptability and long-horizon efficiency in dynamic environments. ##### RL as an Internal Driver of Planning. A second, more integrated paradigm positions RL as an internal driver of the agent’s core planning capabilities. This approach casts the LLM directly as a policy model and optimizes its planning behavior through direct environmental interaction. Instead of guiding an external search algorithm, RL-based feedback from trial and error is used to directly refine the LLM’s internal policy for generating plans. This is achieved through methods derived from RLHF, such as leveraging DPO on successful versus failed trajectories as seen in ETO (Song et al., 2024b), or through lifelong learning frameworks. For instance, VOYAGER (Wang et al., 2024a) iteratively builds and refines a skill library from environmental interaction. This paradigm transforms the LLM from a static generator into an adaptive policy that continuously evolves, enhancing its robustness and autonomy in dynamic environments. In a complementary direction, Dynamic Speculative Planning (DSP) (Guan et al., 2025b) embodies an online reinforcement mechanism that adapts the agent’s policy to jointly optimize latency and operational cost, demonstrating that internal policy refinement can govern not only task success but also system efficiency. RLTR (Li et al., 2025p) decouples planning from answer generation and introduces tool-use rewards that directly evaluate action sequence quality, enabling focused optimization of the agent’s planning capability without relying on verifiable final answers. AdaPlan and its PilotRL framework (Lu et al., 2025c) leverage global plan-based guidance with progressive RL to enhance LLM agents’ long-horizon planning and execution coordination in text game environments like AFLWorld and TextCraft. Planner-R1 (Zhu et al., 2025d) examines reward-density effects in Agentic RL, showing that shaped, process-level rewards markedly improve learning efficiency and enable smaller models to attain competitive planning capability. ##### Prospective: The Synthesis of Deliberation and Intuition. The prospective horizon for agentic planning lies in the synthesis of these two paradigms: moving beyond the distinction between external search and internal policy optimization. The ultimate goal is to develop an agent that internalizes the structured search process itself, seamlessly blending intuitive, fast plan generation with deliberate, slow, deliberative reasoning. In such a model, RL would not only refine the final plan but also optimize a meta-policy governing the deliberation process: learning when to explore alternative paths, how to prune unpromising branches, and how deeply to reason before committing to an action. This would transform the LLM agent from a component that either proposes actions or acts as a raw policy into an integrated reasoning engine. ### 3.2 Tool Using Figure: Figure 4: The development of agentic tool use. Note that we only select a small bunch of representative works here to reflect the progress. Refer to caption: 2509.02547v5/figures/tooluse.jpg RL has emerged as a pivotal methodology for evolving tool-enabled language agents from post-hoc, ReAct-style pipelines to deeply interleaved, multi-turn Tool-Integrated Reasoning (TIR) systems. While early paradigms successfully demonstrated the feasibility of tool invocation, their reliance on SFT or prompt engineering limited agents to mimicking static patterns, lacking the strategic flexibility to adapt to novel scenarios or recover from errors (Chen et al., 2024g; Kavathekar et al., 2025). Agentic RL addresses this by shifting the learning paradigm from imitation to outcome-driven optimization, enabling agents to autonomously discover when, how, and which tools to deploy. This evolution charts a clear trajectory, which we explore in three stages. We begin with (1) early ReAct-style tool calling, then examine (2) modern tool-integrated reasoning (TIR) that deeply embeds tool use within cognitive loops, and finally, discuss the prospective challenge of (3) multi-turn TIR, focusing on temporal credit assignment for robust, long-horizon performance. ##### ReAct-style Tool Calling. Early paradigms for tool invocation predominantly relied on either prompt engineering or SFT to elicit tool-use behaviors. The (I) prompt engineering approach, exemplified by ReAct (Yao et al., 2023b), leveraged few-shot exemplars to guide an LLM to interleave reasoning traces and actions within a "Thought-Action-Observation" cycle, capitalizing on the model’s in-context learning abilities. Going beyond, (II) SFT-based methods were introduced to internalize models’ tool-use capabilities. Frameworks like Toolformer (Schick et al., 2023) employed a self-supervised objective to teach models where to insert API calls, while others like FireAct (Chen et al., 2023), AgentTuning (Zeng et al., 2024), Agent-FLAN (Chen et al., 2024h) fine-tuned models on expert-generated or curated datasets of tool-interaction trajectories (e.g., AgentBank (Song et al., 2024a), APIBank (Li et al., 2023b)). Although SFT improved the reliability of tool invocation, both of these early approaches are fundamentally constrained by their imitative nature. They train agents to replicate static, pre-defined patterns of tool use, thereby lacking the strategic flexibility to adapt to novel scenarios or recover from unforeseen errors, a limitation that RL-centric approaches directly address by shifting the learning objective from imitation to outcome-driven optimization. ##### Tool-integrated RL. Building on the limitations of purely imitative paradigms, RL-based approaches for tool use shift the objective from replicating fixed patterns to optimizing end-task performance. This transition enables agents to strategically decide when, how, and in what combination to invoke tools, adapting dynamically to novel contexts and unforeseen failures. At the foundation, frameworks such as ToolRL (Qian et al., 2025) demonstrate that, even when initialized from base models without any imitation traces, RL training can elicit emergent capabilities, e.g., self-correction of faulty code, adaptive adjustment of invocation frequency, and the composition of multiple tools for complex sub-tasks. Subsequently, a recent surge in research has produced works such as OTC-PO (Wang et al., 2025e), ReTool (Feng et al., 2025a), AutoTIR (Wei et al., 2025c), VTool-R1 (Wu et al., 2025g), DeepEyes (Zheng et al., 2025g), Pixel-Reasoner (Su et al., 2025a), Agentic Reasoning (Wu et al., 2025e), ARTIST (Singh et al., 2025), ToRL (Li et al., 2025l) and numerous other works (Hao et al., 2025a; Feng et al., 2024a; Wei et al., 2025f; Li et al., 2025f; Wu et al., 2025a; Li et al., 2025i; Chen et al., 2025d; Song et al., 2025d; Ye et al., 2025a), which employ RL policies that interleave symbolic computation (e.g., code execution, image editing) with natural-language reasoning within a single rollout. This integrated control loop allows the agent to balance precise, tool-mediated operations with flexible verbal inference, tailoring the reasoning process to the evolving task state. Lin and Xu (2025) theoretically proves that TIR fundamentally expands LLM capabilities beyond the “invisible leash” of pure-text RL by introducing deterministic tool-driven state transitions, establishes token-efficiency arguments for feasibility under finite budgets, and proposes Advantage Shaping Policy Optimization (ASPO) to stably guide agentic tool use. Today, such tool-integrated reasoning is no longer a niche capability but a baseline feature of advanced agentic models. Mature commercial and open-source systems, such as OpenAI’s DeepResearch and o3 (OpenAI, 2025a), Kimi K2 (Kimi, 2025), Qwen QwQ-32B (Team, 2025c), Zhipu GLM Z1 (AI, 2025), Microsoft rStar2-Agent (Shang et al., 2025a) and Meituan LongCat (Meituan, 2025), routinely incorporate these RL-honed strategies, underscoring the centrality of outcome-driven optimization in tool-augmented intelligence. ##### Prospective: Long-horizon TIR. While tool-integrated RL has proven effective for optimizing actions within a single reasoning loop, the primary frontier lies in extending this capability to robust, long-horizon tasks that require multi-turn reasoning (Gao et al., 2025d). This leap is fundamentally bottlenecked by the challenge of temporal credit assignment (Pignatelli et al., 2024). Current RL approaches often depend on sparse, trajectory-level/outcome-based rewards, making it difficult to pinpoint which specific tool invocation in a long, interdependent sequence contributed to success or failure. While nascent research has begun to explore more granular reward schemes, such as turn-level advantage estimation in GiGPO (Feng et al., 2025b) and SpaRL (Wang et al., 2025b), these are still early steps. Consequently, developing more granular credit assignment mechanisms that can accurately guide the agent through complex decision chains without inadvertently punishing useful exploration or promoting reward hacking remains a critical and largely unsolved problem for advancing agentic systems. ### 3.3 Memory Agentic RL transforms memory modules from passive data stores into dynamic, RL‑controlled subsystems, deciding what to store, when to retrieve, and how to forget similar to humans (Wu et al., 2025k). This section traces this evolution through four representative phases. ##### RL in RAG-style Memory. Early systems (e.g., retrieval-augmented generation) treated memory as an external datastore; when RL was employed at all, it solely regulated when to perform queries. Several classic memory systems without RL involvement, such as MemoryBank (Zhong et al., 2024), MemGPT (Packer et al., 2023), and HippoRAG (gutiérrez2025hipporagneurobiologicallyinspiredlongterm), adopt predefined memory management strategies that specify how to store, integrate, and retrieve information (e.g., storage via vector databases or knowledge graphs; retrieval based on semantic similarity or topological connectivity). Subsequently, RL was incorporated into the memory management pipeline as a functional component. A notable example is the framework proposed in Tan et al. (2025b), where the RL policy adjusts retrieval behavior through prospective reflection (multi-level summarization) and retrospective reflection (reinforcing retrieval outcomes). Nevertheless, the memory medium itself remained static (e.g., simple vector store or summary buffer), and the agent exerted no control over the write processes. Recently, Memory-R1 (Yan et al., 2025b) introduced an RL-based memory-augmented Agent framework where a Memory Manager learns to perform structured operations (ADD/UPDATE/DELETE/NOOP) via PPO or GRPO based on downstream QA performance, while an Answer Agent employs a Memory Distillation policy over RAG-retrieved entries to reason and answer. Follow-up works like Mem-$\alpha$ (Wang et al., 2025t) and Memory-as-action (Zhang et al., 2025y) have also explored RL for training agents into automatic memory managers. **Table 3: An overview of three classic categories of agent memory; works marked with ^† directly employ RL. The list here is not exhaustive, and we refer readers interested in broader agent memory to Wu et al. (2025k). The shaded rows indicate the use of reinforcement learning algorithms.** | Method | Type | Key Characteristics | | --- | --- | --- | | RAG-style Memory | | | | MemoryBank (Zhong et al., 2024) | External Store | Static memory with predefined storage/retrieval rules | | MemGPT (Packer et al., 2023) | External Store | OS-like agent with static memory components | | HippoRAG (gutiérrez2025hipporagneurobiologicallyinspiredlongterm) | External Store | Neuro-inspired memory with heuristic access | | Prospect^† (Tan et al., 2025b) | RL-guided Retrieval | Uses RL for reflection-driven retrieval adjustment | | Memory-R1${\dagger}$ (Yan et al., 2025b) | RL-guided Retrieval | RL-driven memory ADD/UPDATE/DELETE/NOOP | | Mem-$\alpha$${\dagger}$ (Wang et al., 2025t) | RL-guided Retrieval | RL-guided agents for memory retrieval | | Memory-as-action (Zhang et al., 2025y) | RL-guided Management | End-to-end training agents for memory management | | Token-level Memory | | | | MemAgent${\dagger}$ (Yu et al., 2025d) | Explicit Token | RL controls which NL tokens to retain or overwrite | | MEM1^† (Zhou et al., 2025g) | Explicit Token | Memory pool managed by RL to enhance context handling | | Memory Token (Jin et al., 2025b) | Explicit Token | Structured memory for reasoning disentanglement | | ReSum${\dagger}$ (Wu et al., 2025i) | Explicit Token | Turn-wise Interaction summary for ReAct agents | | Context Folding${\dagger}$ (Sun et al., 2025c) | Explicit Token | Context folding for ReAct agents | | MemoryLLM (Wang et al., 2024h) | Latent Token | Latent tokens repeatedly integrated and updated | | M+ (Wang et al., 2025s) | Latent Token | Scalable memory tokens for long-context tracking | | IMM (Orlicki, 2025) | Latent Token | Decouples word representations and latent memory | | Memory (Hongkang Yang et al., 2024) | Latent Token | Forget-resistant memory tokens for evolving context | | MemGen${\dagger}$ (Zhang et al., 2025e) | Latent Token | Context-sensitive latent token as memory carriers | | Structured Memory | | | | Zep (Rasmussen et al., 2025) | Temporal Graph | Temporal knowledge graph enabling structured retrieval | | A-MEM (Xu et al., 2025d) | Atomic Memory Notes | Symbolic atomic memory units; structured storage | | G-Memory (Zhang et al., 2025d) | Hierarchical Graph | Multi-level memory graph with topological structure | | Mem0 (Chhikara et al., 2025) | Structured Graph | Agent memory with full-stack graph-based design | ##### RL for Token-level Memory. Subsequent advancements introduced models equipped with explicit, trainable memory controllers, enabling agents to regulate their own memory states (often stored in token form) without relying on fixed, external memory systems. Notably, such memory is commonly instantiated in two forms. The first is (I) explicit tokens, corresponding to human-readable natural language. For example, in MemAgent (Yu et al., 2025d), the agent maintains a natural-language memory pool alongside the LLM, with an RL policy determining, at each segment, which tokens to retain or overwrite, effectively compressing long-context inputs into concise, informative summaries. Similar approaches include MEM1 (Zhou et al., 2025g) and Memory Token (Jin et al., 2025b), both of which explicitly preserve a pool of natural-language memory representations. More frequently, works like ReSum (Wu et al., 2025i), context folding (Sun et al., 2025c) have also explored RL for context memory management. The second form is (II) implicit tokens, where memory is maintained in the form of latent embeddings. A representative line of work includes MemoryLLM (Wang et al., 2024h) and M+ (Wang et al., 2025s), in which a fixed set of latent tokens serves as “memory tokens.” As the context evolves, these tokens are repeatedly retrieved, integrated into the LLM’s forward computation, and updated, thereby preserving contextual information and exhibiting strong resistance to forgetting. Unlike explicit tokens, these memory tokens are not tied to human-readable text but rather constitute a machine-native form of memory. Related efforts include IMM (Orlicki, 2025) and Memory (Hongkang Yang et al., 2024). Across both paradigms, these approaches empower agents to autonomously manage their memory banks, delivering significant improvements in long-context understanding, continual adaptation, and self-improvement. MemGen (Zhang et al., 2025e) for the first time proposes the paradigm of leveraging latent memory tokens for carrying and generating experiential knowledge, posing promising directions for RL-based latent memory. ##### Prospective: RL for Structured Memory. Building on token-level approaches, recent trends are moving toward structured memory representations, which organize and encode information beyond flat token sequences. Representative examples include the temporal knowledge graph in Zep (Rasmussen et al., 2025), the atomic memory notes in A-MEM (Xu et al., 2025d), and the hierarchical graph-based memory designs in G-Memory (Zhang et al., 2025d) and Mem0 (Chhikara et al., 2025). These systems capture richer relational, temporal, or hierarchical dependencies, enabling more precise retrieval and reasoning. However, their management, spanning insertion, deletion, abstraction, and linkage updates, has thus far been governed by handcrafted rules or heuristic strategies. To date, little work has explored the use of RL to dynamically control the construction, refinement, or evolution of such structured memory, making this an open and promising direction for advancing agentic memory capabilities. ### 3.4 Self-Improvement As LLM agents evolve, recent research increasingly emphasizes RL as a mechanism for ongoing reflection, enabling agents to learn from their own mistakes across planning, reasoning, tool use, and memory (Gao et al., 2025c). Rather than relying exclusively on data-driven training phases or static reward models, these systems incorporate iterative, self-generated feedback loops, ranging from prompt-level heuristics to fully fledged RL controllers, to guide agents toward continual self-improvement. ##### RL for Verbal Self-correction. Initial methods in this vein leveraged prompt-based heuristics, sometimes referred to as verbal reinforcement learning, where agents generate an answer, linguistically reflect on its potential errors, and subsequently produce a refined solution, all within a single inferential pass without gradient updates. Prominent examples include Reflexion (Shinn et al., 2023), Self-refine (Madaan et al., 2023), CRITIC (Gou et al., 2024), and Chain-of-Verification (He et al., 2024). For instance, the Self-Refine (Madaan et al., 2023) protocol directs an LLM to iteratively polish its output using three distinct prompts for generation, feedback, and refinement, proving effective across domains like reasoning and programming. To enhance the efficacy and robustness of such self-reflection, several distinct strategies have been developed: (I) multiple sampling, which involves generating multiple output rollouts by sampling from the model’s distribution. By aggregating critiques or solutions from multiple attempts, the agent can improve the consistency and quality of its self-reflection. This method has been widely studied in works like If-or-Else (Li et al., 2024b), UALA (Han et al., 2024) and Multi-agent Verification (Lifshitz et al., 2025). This approach is conceptually analogous to test-time scaling techniques, so we refer the reader to (Pignatelli et al., 2024) for more details; (II) structured reflection workflows, rather than prompting for a monolithic reflection on a final answer, prescribe a more dedicated and granular workflow. For example, Chain-of-Verification (He et al., 2024) manually decomposes the process into distinct “Retrieving, Rethinking, and Revising” stages; (III) external guidance, which grounds the reflection process in verifiable, objective feedback by incorporating external tools. These tools include code interpreters, as seen in Self-Debugging (Chen et al., 2024f), CAD modeling programs in Luban (Guo et al., 2024), mathematical calculators in T1 (Kang et al., 2025b), step-wise reward models (Xiong et al., 2025), and tool-interactive critiquing mechanisms (Gou et al., 2024). ##### RL for Internalizing Self-correction. While verbal self-correction offers a potent inference-time technique, its improvements are ephemeral and confined to a single session. To instill a more durable and generalized capability for self-improvement, subsequent research has employed RL with gradient-based updates to internalize these reflective feedback loops directly into the model’s parameters and to fundamentally enhance the model’s inherent ability to identify and correct its own errors. This paradigm has been applied across multiple domains. For instance, KnowSelf (Qiao et al., 2025) leverages DPO and RPO (Pang et al., 2024) to enhance agents’ self-reflection capabilities in text-based game environments, while Reflection-DPO (Patel et al., 2025) focuses on user–agent interaction scenarios, enabling agents to better infer user intent through reflective reasoning. DuPo (She et al., 2025) employs RL with dual-task feedback to enable annotation-free optimization, enhancing LLM agents’ self-correction across translation, reasoning, and reranking tasks. SWEET-RL (Zhou et al., 2025e) and ACC-Collab (Estornell et al., 2025b) adopt a slightly different setting from the above works: they train an external critic model to provide higher-quality revision suggestions for the actor agent’s actions. Nonetheless, the underlying principle remains closely aligned. ##### RL for Iterative Self-training. Moving toward full agentic autonomy, the third and most advanced class of models combines reflection, reasoning, and task generation into a self-sustaining loop, enabling unbounded self-improvement without human-labeled data. These methods can be distinguished by the architecture of their learning loops: (I) Self-play and search-guided refinement, which emulates classic RL paradigms like AlphaZero. R-Zero (Huang et al., 2025a), for instance, employs a Monte Carlo Tree Search (MCTS) to explore a reasoning tree, using the search results to iteratively train both a policy LLM (the actor) and a value LLM (the critic) entirely from scratch. Similarly, the ISC framework (Tian et al., 2024) operationalizes a cycle of "Imagination, Searching, and Criticizing," where the agent generates potential solution paths, uses a search algorithm to explore them, and applies a critic to refine its reasoning strategy before producing a final answer. (II) Execution-guided curriculum generation, where the agent creates its own problems and learns from verifiable outcomes. Absolute Zero (Zhao et al., 2025a) exemplifies this by proposing its own tasks, attempting solutions, verifying them via execution, and using the outcome-based reward to refine its policy. Similarly, Self-Evolving Curriculum (Chen et al., 2025f) enhances this process by framing problem selection itself as a non-stationary bandit task, allowing the agent to strategically generate a curriculum that maximizes its learning gains over time. TTRL (Zuo et al., 2025) applies this principle for on-the-fly adaptation to a single problem. At test time, it uses execution-based rewards to rapidly fine-tune a temporary copy of the agent’s policy for the specific task at hand; this specialized policy is then used to generate the final answer before being discarded. Though differing in whether the learning is permanent or ephemeral, all these methods underscore a powerful, unified strategy: harnessing execution-based feedback to autonomously guide the agent’s reasoning process. ALAS (Atreja, 2025) constructs an autonomous pipeline that crawls web data, distills it into training signals, and continuously fine-tunes LLMs, thereby enabling self-training and self-evolution without manual dataset curation. (III) Collective bootstrapping, where learning is accelerated by aggregating shared experience. SiriuS (Zhao et al., 2025e), for example, constructs and augments a live repository of successful reasoning trajectories from multi‑agent interactions, using this growing knowledge base to bootstrap its own training curriculum. MALT (Motwani et al., 2025) shares a similar motivation, yet its implementation is limited to a three-agent setup. Nevertheless, all these methods define feedback loops that are internally generated and continuously evolving, representing a significant step toward truly autonomous agents. ##### Prospective: Meta Evolution of Reflection Ability. While current research successfully uses RL to refine an agent’s behavior through reflection, the reflection process itself remains largely handcrafted and static. The next frontier lies in applying RL at a higher level of abstraction to enable meta-learning for adaptive reflection, focusing not just on correcting an error, but on learning how to self-correct more effectively over time. In this paradigm, the agent may learn a meta-policy that governs its own reflective strategies. For instance, it could learn to dynamically choose the most appropriate form of reflection for a given task, deciding whether a quick verbal check is sufficient or if a more costly, execution-guided search is necessary. Furthermore, an agent could use long-term outcomes to evaluate and refine the very heuristics it uses for self-critique, effectively learning to become a better internal critic. By optimizing the reflective mechanism itself, this approach moves beyond simple self-correction and toward a state of continuous self-improvement in the learning process, representing a crucial step toward agents that can not only solve problems but also autonomously enhance their fundamental capacity to learn from experience. ### 3.5 Reasoning Reasoning in large language models can be broadly categorized into fast reasoning and slow reasoning, building on the dual-process cognitive theory (Kahneman, 2011; Kahneman and Tversky, 1974; Stanovich and West, 2000), as discussed in recent surveys (Ke et al., 2025; Kumar et al., 2025). Fast reasoning corresponds to rapid, heuristic-driven inference with minimal intermediate steps, while slow reasoning emphasizes deliberate, structured, and multi-step reasoning. Understanding the trade-offs between these two paradigms is crucial for designing models that balance efficiency and accuracy in complex problem-solving. ##### Fast Reasoning: Intuitive and Efficient Inference Fast reasoning models operate in a manner analogous to System 1 (Li et al., 2025r) cognition: quick, intuitive, and pattern-driven. They generate immediate responses without explicit step-by-step deliberation, excelling in tasks that prioritize fluency, efficiency, and low latency. Most conventional LLMs fall under this category, where reasoning is implicitly encoded in next-token prediction (Shao et al., 2024b; Yang et al., 2024a). However, this efficiency comes at the cost of systematic reasoning, making these models more vulnerable to factual errors, biases, and shallow generalization. To address the severe hallucination problems in fast reasoning, current research has largely focused on various direct approaches. Some studies attempt to mitigate errors and hallucinations in the next-token prediction paradigm by leveraging internal mechanisms (Wang et al., 2023b; Yao et al., 2023a; Besta et al., 2024) or by simulating human-like cognitive reasoning. Other works propose introducing both external and internal confidence estimation methods (Lightman et al., 2023; Wang et al., 2024d) to identify more reliable reasoning paths. However, constructing such external reasoning frameworks often risks algorithmic adaptivity issues and can easily fall into the complexity trap. ##### Slow Reasoning: Deliberate and Structured Problem Solving In contrast, slow reasoning models are designed to emulate System 2 cognition (Li et al., 2025r) by explicitly producing intermediate reasoning traces. Techniques such as chain-of-thought prompting, multi-step verification (Qin et al., 2024a), and reasoning-augmented reinforcement learning allow these models to engage in deeper reflection and achieve greater logical consistency. While slower in inference due to extended reasoning trajectories, they achieve higher accuracy and robustness in knowledge-intensive tasks such as mathematics, scientific reasoning, and multi-hop question answering (Chu et al., 2025a). Representative examples include OpenAI’s o1 (OpenAI et al., 2024) and o3 series (OpenAI Team, 2025), DeepSeek-R1 (DeepSeek-AI et al., 2025), as well as methods that incorporate dynamic test-time scaling (Aggarwal and Welleck, 2025; Zhang et al., 2024a; Xu et al., 2025a; Yao et al., 2023a) or reinforcement learning (Zeng et al., 2025c; Yu et al., 2025e; Wang et al., 2025k; Liang et al., 2025a; b; Yue et al., 2025a) for reasoning. Modern slow reasoning exhibits output structures that differ substantially from fast reasoning. These include a clear exploration and planning structure, frequent verification and checking behaviors, and generally longer inference lengths and times. Past work has explored diverse patterns for constructing long-chain reasoning outputs. Some methods—Macro-o1, HuatuoGPT-o1, and AlphaZero—have attempted to synthesize long chains-of-thought via structured, agentic search (Zhao et al., 2024; Chen et al., 2024c; b). Other approaches focus on generating long-CoT datasets that embody specific deliberative or reflective thinking patterns; examples include HiICL-MCTS, LLaVA-CoT, rStar-Math, and ReasonFlux (Wu et al., 2025d; Xu et al., 2025b; Guan et al., 2025a; Yang et al., 2025d). Recent approaches that perform reasoning in the latent space leverage latent representations to conduct parallel reasoning and explore diverse reasoning trajectories (Zhang et al., 2025aa; Hao et al., 2024). With the progress of pretrained foundation models, more recent work has shifted toward self-improvement paradigms—frequently instantiated with reinforcement learning—to further enhance models’ reasoning capabilities (Zeng et al., 2025c; Yu et al., 2025e). ##### Prospective: Integrating Slow Reasoning Mechanisms into Agentic Reasoning The dichotomy between fast and slow reasoning highlights an open challenge in agentic reasoning: how to employ reinforcement learning for reliably training slow-thinking reasoning capabilities in agentic scenarios. Reinforcement learning in agentic scenarios faces greater challenges in training stability, such as ensuring compatibility with diverse environments. Agentic reasoning itself is also susceptible to overthinking. Purely fast models may overlook critical reasoning steps, while slow models often suffer from excessive latency or overthinking behaviors, such as unnecessarily long chains of thought. Emerging approaches seek hybrid strategies (Yang et al., 2025a) that combine the efficiency of fast reasoning with the rigor of slow reasoning (Yang et al., 2025g; Hou et al., 2025; Li et al., 2025q; Chen et al., 2025g). For instance, adaptive test-time scaling allows a model to decide whether to respond quickly or to engage in extended deliberation depending on task complexity. Developing such cognitively aligned mechanisms is a key step toward building reasoning agents that are both efficient and reliable. Figure: Figure 5: A conceptual overview of how RL empowers agentic LLMs across six core capabilities. The central panel summarizes the capability taxonomy, while the side panels illustrate representative RL mechanisms and interaction patterns. Listed methods are illustrative rather than exhaustive; see the main text for details. Refer to caption: 2509.02547v5/figures/capability_tmlr.jpg ### 3.6 Perception By bridging visual perception with linguistic abstraction, Large Vision–Language Models (LVLMs) have demonstrated unprecedented capabilities for perceiving and understanding multimodal content  (Team et al., 2023; Liu et al., 2023a; Wang et al., 2024e; Li et al., 2024d; Chen et al., 2024j; OpenAI, 2023; Zhang et al., 2025q; 2024b). Central to this progress is the incorporation of explicit reasoning mechanisms into multimodal learning frameworks (Shao et al., 2024a; Zhang et al., 2023), moving beyond passive perception toward active visual cognition (Su et al., 2025c). RL has emerged as a powerful paradigm for this purpose, enabling the alignment of vision–language–action models with complex, multi-step reasoning objectives that go beyond the constraints of supervised next-token prediction (Zhou et al., 2025a; Wu et al., 2025h). ##### From Passive Perception to Active Visual Cognition Multimodal content often requires nuanced, context-dependent interpretation. Inspired by the remarkable success of RL in enhancing reasoning within LLMs  (DeepSeek-AI et al., 2025; Team et al., 2025b), researchers have increasingly sought to transfer these gains to multimodal learning  (Shen et al., 2025a; Peng et al., 2025). Early efforts focused on preference-based RL to strengthen the Chain-of-Thought (CoT) reasoning ability of MLLMs (Wang et al., 2024g; Dong et al., 2025d; Zhu et al., 2025b). Visual-RFT (Liu et al., 2025v) and Reason-RFT (Tan et al., 2025a) directly apply GRPO to the vision domain, adaptively incorporating vision-specific metrics such as IoU as verifiable reward signals, while STAR-R1 (Li et al., 2025t) extended this idea by introducing partial rewards tailored for visual GRPO. Building upon this, a series of approaches—Vision-R1 (Huang et al., 2025c), VLM-R1 (Shen et al., 2025a), LMM-R1 (Peng et al., 2025), and MM-Eureka (Meng et al., 2025)—developed specialized policy optimization algorithms designed to incentivize step-wise visual reasoning, demonstrating strong performance even on smaller 3B-parameter models. SVQA-R1 (Wang and Ling, 2025) introduced Spatial-GRPO, a novel groupwise RL method that enforces view-consistent and transformation-invariant objectives. Visionary-R1 (Xia et al., 2025a) enforces image captioning as a prerequisite step before reasoning, mitigating shortcut exploitation during reinforcement finetuning. A line of curriculum-learning methods have also been proposed to ease and smooth the RL training process of vision reinforcement finetuning (Yang et al., 2025c; Chen et al., 2025b; Zhan et al., 2025; Guo et al., 2025d; Dong et al., 2025d). R1-V (Chen et al., 2025b) introduces VLM-Gym and trains G0/G1 models via scalable, pure RL self-evolution with a perception-enhanced cold start, yielding emergent perception–reasoning synergy across diverse visual tasks. R1-Zero (Zhou et al., 2025d) shows that even simple rule-based rewards can induce self-reflection and extended reasoning in non-SFT models, surpassing supervised baselines. PAPO (Wang et al., 2025u) proposes a perception-aware policy optimization framework that augments RLVR methods with an implicit perception KL loss and double-entropy regularization, while Li et al. (2025s) proposes a summarize-and-then-reason framework under RL training to mitigate visual hallucinations and improve reasoning without dense human annotations. Collectively, these approaches demonstrate that R1-style RL can be successfully transferred to the vision domain, provided that well-designed, verifiable reward metrics are used—yielding significant improvements in performance, robustness, and out-of-distribution generalization. More recent work explores another key advantage of RL: moving beyond the formulation of tasks as passive perception, where static, verifiable rewards are computed only on the text-based outputs of LVLMs. Instead, RL can be used to incentivize active cognition over multimodal content—treating visual representations as manipulable and verifiable intermediate thoughts. This paradigm empowers models not merely to “look and answer,” but to actively see, manipulate, and reason with visual information as part of a multi-step cognitive process (Su et al., 2025c). ##### Grounding-Driven Active Perception. To advance from passive perception to active visual cognition, a key direction is enabling LVLMs to repeatedly look back and query the image while generating their reasoning process. This is achieved through grounding (Nagaraja et al., 2016; Mao et al., 2016), which anchors each step of the generated chain-of-thought (CoT) to specific regions of the multimodal input—facilitating more valid and verifiable reasoning by explicitly linking text with corresponding visual regions. To begin with, GRIT (Fan et al., 2025c) interleaves bounding-box tokens with textual CoT and uses GRPO with both verifiable rewards and bounding-box correctness as supervision. Chung et al. (2025) introduces a simple point-and-copy mechanism that allows the model to dynamically retrieve relevant image regions throughout the reasoning process. Ground-R1 (Cao et al., 2025a) and BRPO (Chu et al., 2025c) highlight evidence regions (via IoU-based or reflection rewards) prior to text-only reasoning, while DeepEyes (Zheng et al., 2025g) demonstrates that end-to-end RL can naturally induce such grounding behaviors. Chain-of-Focus further refines this approach by grounding CoT steps followed by zooming in operations. ##### Tool-Driven Active Perception. Another promising direction for enabling active perception is to frame visual cognition as an agentic process, where external tools, code snippets, and runtime environments assist the model’s cognitive workflow (Gupta and Kembhavi, 2023; Zhao et al., 2025d). For instance, VisTA (Huang et al., 2025d) and VTool-R1 (Wu et al., 2025g) focus on teaching models how to select and use visual tools through RL, while OpenThinkIMG (Su et al., 2025b) provides standardized infrastructure for training models to “think with images.” Finally, Visual-ARFT (Liu et al., 2025v) leverages RL to facilitate tool creation, harnessing the code-generation capabilities of MLLMs to dynamically extend their perceptual toolkit. Pixel Reasoner (Su et al., 2025a) expands the model’s action space with operations such as crop, erase, and paint, and introduces curiosity-driven rewards to discourage premature termination of exploration. ##### Generation-Driven Active Perception. In addition to grounding and tool use, humans employ one of their most powerful cognitive abilities—imagination—to produce sketches or diagrams that aid problem-solving. Inspired by this, researchers have begun equipping LVLMs with the ability to generate sketches or images interleaved with chain-of-thought (CoT) reasoning, enabling models to externalize intermediate representations and reason more effectively (Xu et al., 2025e; Fang et al., 2025a; Li et al., 2025c). Visual Planning (Xu et al., 2025e) proposes to use imagined image rollouts only as the CoT images thinking, using downstream task success as the reward signal. GoT-R1 (Duan et al., 2025) applies RL within the Generation-CoT framework, allowing models to autonomously discover semantic–spatial reasoning plans before producing the image. Similarly, T2I-R1 (Jiang et al., 2025b) explicitly decouples the process into a semantic-level CoT for high-level planning and a token-level CoT for patch-wise pixel generation, jointly optimizing both stages with RL. ##### Audio. RL has also been extended beyond vision–language models to a diverse range of modalities, including audio. Within the audio–language domain, we categorize RL applications into two broad classes. (1) Reasoning enhancement for large audio–language models: RL is leveraged to guide models in producing structured, step-by-step reasoning chains for tasks such as audio question answering and logical inference (Wen et al., 2025; Diao et al., 2025; Li et al., 2025d; d; Wen et al., 2025). (2) Fine-grained component optimization in speech synthesis (TTS): RL is employed to directly refine system components—for example, improving duration predictors—using perceptual quality metrics such as speaker similarity and word error rate as reward signals, thereby yielding more natural and intelligible speech (Li et al., 2025m). Some other works such as EchoInk-R1 (Xing et al., 2025) further enrich visual reasoning by integrating audio–visual synchrony under GRPO optimization. ### 3.7 Others Beyond optimizing the above core cognitive modules, Agentic RL also strengthens the ability to maintain strategic coherence over extended, multi-turn interactions. Here, RL is applied to support long-horizon reasoning and effective credit assignment. For long-horizon interactions, the central challenge is temporal credit assignment (Pignatelli et al., 2024), where sparse and delayed feedback obscures the link between an agent’s actions and a distant outcome. Agentic RL directly confronts this by evolving both the learning signal and the optimization framework. One major approach is the (I) integration of process-based supervision with final outcome rewards. Rather than relying on a single reward at a trajectory’s conclusion, this paradigm uses auxiliary models or programmatic rules to evaluate the quality of intermediate steps, providing a denser and more immediate learning signal that guides the agent’s multi-turn strategy. For example, EPO (Liu et al., 2025m), ThinkRM (Hong et al., 2025c), SPO (Guo et al., 2025c), and AgentPRM (Choudhury, 2025) introduce external reward models to provide step-wise signals for agents; in contrast, RLVMR (Zhang et al., 2025ac) designs manually defined, programmatic rules to guide the intermediate supervision. A second, complementary strategy is to (II) extend preference optimization from single turns to multi-step segments. Techniques like Segment-level DPO (SDPO) (Kong et al., 2025) move beyond comparing isolated responses and instead construct preference data over entire conversational snippets or action sequences. This allows the model to directly learn how early decisions influence long-term success, thereby refining its ability to maintain strategic coherence in extended dialogues and complex tasks. ## 4 Agentic RL: The Task Perspective Agentic RL manifests through a wide spectrum of concrete tasks that test and shape its evolving capabilities. This section surveys representative application domains where Agentic RL has demonstrated remarkable potential and unique challenges. We begin with search and information retrieval (Section [4.1](#S4.SS1)), followed by code generation and software engineering (Section [4.2](#S4.SS2)), and mathematical reasoning (Section [4.3](#S4.SS3)). We then discuss its role in GUI navigation (Section [4.4](#S4.SS4)), vision understanding tasks (Section [4.5](#S4.SS5)), as well as VLM embodied interaction (Section [4.6](#S4.SS6)). Beyond single-agent scenarios, we extend the perspective to multi-agent systems (Section [4.7](#S4.SS7)) and conclude with other emerging domains (Section [4.8](#S4.SS8)). Together, these applications highlight how Agentic RL transitions from abstract paradigms into actionable, real-world problem-solving, as illustrated in Figure [6](#S4.F6). Figure: Figure 6: The evolution tree of RL for domain-specific agents, illustrating the chronological progression of representative domains and methods. Refer to caption: 2509.02547v5/figures/agent-domain-tree.jpg ### 4.1 Search & Research Agent Search has been central to extending LLMs with external knowledge, with Retrieval-Augmented Generation (RAG) as a widely used approach (Gao et al., 2024; Fan et al., 2024). The paradigm is now evolving beyond simple information retrieval towards creating autonomous agents capable of *deep research*: complex, multi-step processes that involve not just finding information but also performing in-depth analysis, synthesizing insights from numerous sources, and drafting comprehensive reports (Kimi, 2025; Perplexity, 2025). This shift elevates the objective from answering queries to tackling complex research tasks. Early prompt-driven methods relied on brittle query strategies and manual engineering. While more recent works like Search-o1 (Li et al., 2025i) leverage large reasoning models for agentic, inference-time retrieval, and multi-agent systems such as DeepResearch (Zhang et al., 2025s) coordinate querying and summarization sub-agents, they still lack learning signals. These prompt-based methods lack any fine-tuning signal, leading to limited generalization and poor effectiveness in multi-turn settings that demand a tight loop of search, reasoning, and synthesis. These limitations have led to the adoption of reinforcement learning to directly optimize the end-to-end process of query generation and search–reasoning coordination for advanced research objectives. Table [4](#S4.T4) presents the majority of works studied in this section. In the following, we will detail how RL empowers these agents. **Table 4: A summary of RL-based methods for search and research agents.** | Method | Category | Base LLM | Resource Link | | --- | --- | --- | --- | | Open Source Methods | | | | | DeepRetrieval (Jiang et al., 2025c) | External | Qwen2.5-3B-Instruct,
Llama-3.2-3B-Instruct | [GitHub](https://github.com/pat-jj/DeepRetrieval) | | Search-R1 (Jin et al., 2025a) | External | Qwen2.5-3B/7B-Base/Instruct | [GitHub](https://github.com/PeterGriffinJin/Search-R1) | | R1-Searcher (Song et al., 2025a) | External | Qwen2.5-7B,
Llama3.1-8B-Instruct | [GitHub](https://github.com/RUCAIBox/R1-Searcher) | | R1-Searcher++ (Song et al., 2025b) | External | Qwen2.5-7B-Instruct | [GitHub](https://github.com/RUCAIBox/R1-Searcher-plus) | | ReSearch (Chen et al., 2025d) | External | Qwen2.5-7B/32B-Instruct | [GitHub](https://github.com/Agent-RL/ReCall/tree/re-search) | | StepSearch (Wang et al., 2025w) | External | Qwen2.5-3B/7B-Base/Instruct | [GitHub](https://github.com/Zillwang/StepSearch) | | DeepResearcher (Zheng et al., 2025e) | External | Qwen2.5-7B-Instruct | [GitHub](https://github.com/GAIR-NLP/DeepResearcher) | | WebDancer (Wu et al., 2025a) | External | Qwen2.5-7B/32B,
QWQ-32B | [GitHub](https://github.com/Alibaba-NLP/WebAgent/tree/main/WebDancer) | | WebThinker (Li et al., 2025j) | External | QwQ-32B,
DeepSeek-R1-Distilled-Qwen,
Qwen2.5-32B | [GitHub](https://github.com/sunnynexus/WebThinker) | | WebSailor (Li et al., 2025f) | External | Qwen2.5-3B/7B/32B/72B | [GitHub](https://github.com/Alibaba-NLP/WebAgent/tree/main/WebSailor) | | WebWatcher (Geng et al., 2025) | External | Qwen2.5-VL-7B/32B | [GitHub](https://github.com/Alibaba-NLP/WebAgent/tree/main/WebWatcher) | | WebShaper (Tao et al., 2025) | External | Qwen-2.5-32B/72B, QwQ-32B | [GitHub](https://github.com/Alibaba-NLP/DeepResearch) | | ASearcher (Gao et al., 2025d) | External | Qwen2.5-7B/14B, QwQ-32B | [GitHub](https://github.com/inclusionAI/ASearcher) | | Atom-Searcher (Deng et al., 2025b) | External | Qwen2.5-7B-Instruct | [GitHub](https://github.com/antgroup/Research-Venus) | | MiroMind Open Deep Research (MiroMind Team, 2025) | External | - | [Website](https://miromind.ai/blog/miromind-open-deep-research) | | SimpleDeepResearcher (Sun et al., 2025b) | External | QwQ-32B | [GitHub](https://github.com/RUCAIBox/SimpleDeepSearcher) | | AWorld (Yu et al., 2025a) | External | Qwen3-32B | [GitHub](https://github.com/inclusionAI/AWorld/tree/main/train) | | SFR-DeepResearch (Nguyen et al., 2025b) | External | QwQ-32B, Qwen3-8B, GPT-oss-20b | - | | ZeroSearch (Sun et al., 2025a) | Internal | Qwen2.5-3B/7B-Base/Instruct | [GitHub](https://github.com/Alibaba-NLP/ZeroSearch) | | SSRL (Fan et al., 2025b) | Internal | Qwen2.5, Llama-3.2/Llama-3.1, Qwen3 | [GitHub](https://github.com/TsinghuaC3I/SSRL) | | Closed Source Methods | | | | | OpenAI Deep Research (OpenAI, 2025a) | External | OpenAI Models | [Website](https://openai.com/index/introducing-deep-research/) | | Perplexity’s DeepResearch (Perplexity, 2025) | External | - | [Website](https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research) | | Google Gemini’s DeepResearch (Google, 2025) | External | Gemini | [Website](https://gemini.google/overview/deep-research/) | | Kimi-Researcher (Kimi, 2025) | External | Kimi K2 | [Website](https://moonshotai.github.io/Kimi-Researcher/) | | Grok AI DeepSearch (x.ai, 2025) | External | Grok3 | [Website](https://grokaimodel.com/deepsearch/) | | Doubao with Deep Think (Doubao, 2025) | External | Doubao | [Website](https://www.doubao.com/chat/) | | Manus WideResearch | External | - | [Website](https://manus.im/blog/introducing-wide-research) | #### 4.1.1 Open Source RL Methods ##### Search from the external Internet A major line of work builds on the RAG foundation but relies on *real-time web search APIs* as the external environment, using reinforcement learning to optimize query generation and multi-step reasoning. Early progress was spearheaded by DeepRetrieval (Jiang et al., 2025c), which framed one-shot query generation as a GRPO-trained policy and directly rewarded recall and relevance against live search results. Motivated by its gains, subsequent methods extended the paradigm into multi-turn, reasoning-integrated, and multi-modal search. Search-R1 (Jin et al., 2025a) and DeepResearcher (Zheng et al., 2025e) integrate retrieved-token masking with outcome-based rewards to interleave query formulation and answer generation. AutoRefine (Shi et al., 2025b) further advances this trajectory by inserting refinement phases between successive search calls, using GRPO to reward not only answer correctness but also retrieval quality, enabling agents to iteratively filter and structure noisy evidence during long-horizon reasoning. R1-Searcher (Song et al., 2025a) employs a two-stage, cold-start PPO strategy—first learning when to invoke web search, then how to exploit it—while its successor R1-Searcher++ (Song et al., 2025b) adds supervised fine-tuning, internal-knowledge rewards to avoid redundancy, and dynamic memory for continual assimilation. ReSearch (Chen et al., 2025d) pursues fully end-to-end PPO without supervised tool-use trajectories, while StepSearch (Wang et al., 2025w) accelerates convergence on multi-hop QA by assigning intermediate step-level rewards. Atom-Searcher (Deng et al., 2025b) is an agentic deep research framework that significantly improves LLM problem-solving by refining the reasoning process itself, not just the final outcome. WebDancer (Wu et al., 2025a) leverages human browsing trajectory supervision plus RL fine-tuning to produce autonomous ReAct-style agents, excelling on GAIA (Mialon et al., 2024) and WebWalkerQA (Wu et al., 2025b). WebThinker (Li et al., 2025j) embeds a Deep Web Explorer into a think-search-draft loop, aligning via DPO with human feedback to tackle complex report-generation. WebSailor (Li et al., 2025f) is a complete post-training methodology designed to teach LLM agents sophisticated reasoning for complex web navigation and information-seeking tasks. WebWatcher (Geng et al., 2025) further extends to multimodal search, combining visual-language reasoning, tool use, and RL to outperform text-only and multimodal baselines on BrowseComp-VL and VQA benchmarks. ASearcher (Gao et al., 2025d) uses large-scale asynchronous reinforcement learning with synthesized QA data, enabling long-horizon search (40+ tool calls) and outperforming prior open-source methods. MiroMind Open Deep Research (MiroMind ODR) (MiroMind Team, 2025) aims to build a high-performance, fully open-sourced, open-collaborative deep research ecosystem — with an agent framework, model, data, and training infrastructure all fully accessible and open. ##### Search from LLM internal knowledge However, these training methods that rely on external APIs face two major challenges: (1) the document quality of real-time internet document searching is uncontrolled, and noisy information brings instability to the training process. (2) The API cost is too high and severely limits scalability. To enhance the efficiency, controllability, and stability of training, some recent studies have used controllable simulated search engines such as LLM internal knowledge. For example, ZeroSearch (Sun et al., 2025a) replaces live web retrieval with a pseudo search engine distilled from LLMs themselves, combining curriculum RL to gradually approach live-engine performance without issuing real queries. SSRL (Fan et al., 2025b) takes this idea further: the agent performs entirely offline “self-search” during training, without explicit search engines, yet transfers seamlessly to online inference, where real APIs can still boost performance. Though still at an early stage, offline self-search enhances stability and scalability beyond API limits, pointing toward more self-reliant research agents. #### 4.1.2 Closed Source RL Methods ##### Industrial Research Agents. Despite progress in combining RAG and RL, most open-source models still fail on OpenAI’s BrowseComp (Wei et al., 2025b), a challenging benchmark that measures the ability of AI agents to locate hard-to-find information, revealing gaps in long-horizon planning, page-grounded tool use, and cross-source verification. In contrast, recent closed source systems are markedly stronger, having shifted from mere query optimization to fully autonomous research agents that navigate the open web, synthesize information from multiple sources, and draft comprehensive reports. This is likely due to the industry’s more powerful foundation models and the availability of more high-quality data. OpenAI Deep Research (OpenAI, 2025a) achieves 51.5% pass@1 on BrowseComp. Other prototypes, Perplexity’s DeepResearch (Perplexity, 2025), Google Gemini’s DeepResearch (Google, 2025), Kimi-Researcher (Kimi, 2025), Grok AI DeepSearch (x.ai, 2025), Doubao with Deep Think (Doubao, 2025), combine RL-style fine-tuning with advanced tool integration and memory modules, ushering in a new era of interactive, iterative research assistants. ##### Case Study: OpenAI Deep Research. Deep Research provides a concrete example of how capabilities from Section [3](#S3) combine with the RL-shaped search strategies. The agent begins with long-horizon multi-step reasoning and planning, decomposing a user request into sub-goals. It then performs RL-shaped web search: issuing queries, selecting which pages to open, and refining its search trajectory. These search policies are shaped during training using research-oriented benchmarks such as BrowseComp (Wei et al., 2025b). Throughout the process, the agent maintains persistent memory in the form of scratchpad notes and performs cross-source verification before synthesis. These capabilities—reasoning, planning, tool use, memory, and verification—are coupled with RL-shaped control decisions over search depth, branch selection, and evidence integration, forming a unified research agent. ### 4.2 Code Agent Code generation, or more broadly, software engineering, provides an ideal testbed for LLM-based Agentic RL: execution semantics are explicit and verifiable, and automated signals (compilation, unit tests, and runtime traces) are readily available (Dong et al., 2025b). Early multi-agent frameworks (e.g., MetaGPT, AutoGPT, AgentVerse) coordinated roles through prompting without parameter updates, showcasing the promise of modular role allocation (Hong et al., 2024b; Gravitas, 2023; Chen et al., 2024e). Initial RL for code, such as CodeRL, incorporated execution-based reward modeling and actor–critic training (Le et al., 2022), catalyzing a wave of studies that exploit execution feedback to guide policy updates. Table [5](#S4.T5) presents the majority of works studied in this section. We structure the literature along increasing *task complexity*, progressing from *code generation* (Section [4.2.1](#S4.SS2.SSS1)) to *code refinement* (Section [4.2.2](#S4.SS2.SSS2)) and *software engineering* (Section [4.2.3](#S4.SS2.SSS3)). #### 4.2.1 RL for Code Generation Early research focused on relatively simple, single-round code generation (e.g., completing a function or solving a coding challenge in one go), which lays the foundation for subsequent large-scale software engineering. ##### Outcome reward RL. Methods in this class optimize directly for final correctness, typically measured by pass@k or unit-test success. AceCoder (Zeng et al., 2025b) introduces a data-efficient RLHF pipeline for code generation, constructing large-scale preference pairs from existing code fragments to train a reward model via Bradley–Terry loss, which then guides RFT on the synthesized dataset. Beyond early actor–critic formulations, recent open-source efforts scale outcome-based RL on large pre-trained code models. DeepCoder-14B (Luo et al., 2025c) stabilizes GRPO training via iterative context lengthening and DAPO-inspired filtering, and employs a sparse Outcome Reward Model (ORM) to prevent reward hacking on curated coding data. RLTF employs an online RL loop that uses unit test results as multi-granularity reward signals, from coarse pass/fail outcomes to fine-grained fault localization, directly guiding code refinement (Liu et al., 2023b). CURE formalizes coder–tester co-evolution: a tester generates or evolves unit tests while a coder iteratively patches code; a reward-precision objective mitigates low-quality test effects during joint training (Wang et al., 2025q). Absolute Zero applies self-play RL without human data. It generates coding tasks for itself and uses execution outcomes as verifiable rewards to bootstrap reasoning ability (Zhao et al., 2025a). Re:Form (Yan et al., 2025a) leverages formal language-based reasoning with RL and automated verification to reduce human priors, enabling reliable program synthesis and surpassing strong baselines on formal verification tasks. In (Feng et al., 2025c), the authors propose a two-stage training pipeline: first fine-tuning for a high-correctness baseline, then performing efficiency-driven online RL optimization. ##### Process reward RL. To mitigate sparsity and credit assignment, several works design process-level supervision by integrating compilation and execution feedback. StepCoder (Dou et al., 2024) decomposes compilation and execution into step-level signals for shaping; Process Supervision-Guided Policy Optimization (PSGPO) (Dai et al., 2025) leverages intermediate error traces and process annotations for dense rewards; and CodeBoost (Wang et al., 2025n) mines raw repositories to unify heterogeneous execution-derived signals, ranging from output correctness to error-message quality, under a single PPO framework. Further, PRLCoder (Ye et al., 2025b) introduces process-supervised RL by constructing reward models that score each partial snippet: a teacher model mutates lines of reference solutions and assigns positive/negative signals based on compiler and test feedback. This fine-grained supervision yields faster convergence and +10.5% pass-rate improvements over the base model, illustrating how dense shaping at the line-level can guide code synthesis more effectively than outcome-only signals. o1-Coder (Zhang et al., 2024d) combines RL with Monte Carlo Tree Search, where the policy learns from exploration guided by test case rewards and gradually improves from pseudocode to executable code. Posterior-GRPO (Fan et al., 2025a) rewards intermediate reasoning but gates credit by final test success to prevent speculative reward exploitation; Policy Filtration for RLHF (Zhang et al., 2025c) improves reward-correctness alignment by filtering low-confidence pairs before policy updates. Scaling preference supervision beyond costly human annotation has proven effective as well. CodeFavor (Liu et al., 2024a) constructs CodePrefBench from code evolution histories, covering correctness, efficiency, security, and style to improve preference modeling and alignment. Focused-DPO (Zhang et al., 2025p) adapts preference-based RL by weighting preference optimization on error-prone regions of the code, making feedback more targeted and improving robustness across benchmarks. Yang et al. (2025f) studies how RL-trained small-scale agents surpass large-scale prompt-based models in MLE environments via duration-aware gradient updates in a distributed asynchronous RL. #### 4.2.2 RL for Iterative Code Refinement A second line of research targets more complex coding tasks that require debugging and iterative refinement. In these scenarios, an agent may need multiple attempts to improve solutions, using feedback from human requirements or failed test results, which is closer to real-world tasks. ##### Outcome reward RL. A representative line treats the entire refinement loop as a trajectory while optimizing for final task success. RLEF (Gehring et al., 2025) (Reinforcement Learning from Execution Feedback) grounds correction loops in real error messages as context while optimizing for ultimate pass rates; this reduces the number of attempts needed and improves competitive-programming performance relative to single-shot baselines. $\mu$Code (Jain et al., 2025a) jointly trains a generator and a learned verifier under single-step reward feedback, showing that verifier-guided outcome rewards can outperform purely execution-feedback baselines. R1‑Code‑Interpreter (Chen et al., 2025h) harnesses multi-turn supervised fine-tuning and reinforcement learning to train LLMs to decide when and how to invoke a code interpreter during step-by-step reasoning. Figure: Figure 7: Benchmark Performance of RL-Enhanced Code and SWE Methods. Scores are pass@1 unless otherwise specified. Refer to caption: 2509.02547v5/figures/code_benchmarks.png **Table 5: A summary of RL methods for code and software engineering agents.** | Method | Reward | Base LLM | Resource | | --- | --- | --- | --- | | RL for Code Generation | | | | | AceCoder (Zeng et al., 2025b) | Outcome | Qwen2.5-Coder-7B-Base/Instruct, Qwen2.5-7B-Instruct | [GitHub](https://github.com/TIGER-AI-Lab/AceCoder) | | DeepCoder-14B (Luo et al., 2025c) | Outcome | DeepSeek-R1-Distilled-Qwen-14B | [GitHub](https://github.com/agentica-project/rllm) | | RLTF (Liu et al., 2023b) | Outcome | CodeGen-NL 2.7B, CodeT5-770M | [GitHub](https://github.com/Zyq-scut/RLTF) | | CURE (Wang et al., 2025q) | Outcome | Qwen2.5-7B/14B-Instruct, Qwen3-4B | [GitHub](https://github.com/Gen-Verse/CURE) | | Absolute Zero (Zhao et al., 2025a) | Outcome | Qwen2.5-7B/14B, Qwen2.5-Coder-3B/7B/14B, Llama-3.1-8B | [GitHub](https://github.com/LeapLabTHU/Absolute-Zero-Reasoner) | | StepCoder (Dou et al., 2024) | Process | DeepSeek-Coder-Instruct-6.7B | [GitHub](https://github.com/Ablustrund/APPS_Plus) | | PSGPO (Dai et al., 2025) | Process | Qwen2.5-Coder-7B-Instruct | - | | CodeBoost (Wang et al., 2025n) | Process | Qwen2.5-Coder-7B-Instruct, Llama-3.1-8B-Instruct,
Seed-Coder-8B-Instruct, Yi-Coder-9B-Chat | [GitHub](https://github.com/sijieaaa/CodeBoost) | | PRLCoder (Ye et al., 2025b) | Process | CodeT5+, Unixcoder, T5-base | - | | o1-Coder (Zhang et al., 2024d) | Process | DeepSeek-1.3B-Instruct | [GitHub](https://github.com/ADaM-BJTU/O1-CODER) | | Posterior-GRPO (Fan et al., 2025a) | Process | Qwen2.5-Coder-3B-Base, Qwen2.5-Coder-7B-Instruct, Qwen2.5-Math-7B | - | | Policy Filtration for RLHF (Zhang et al., 2025c) | Process | DeepSeek-Coder-6.7B, Qwen1.5-7B | [GitHub](https://github.com/swtheing/PF-PPO-RLHF) | | CodeFavor (Liu et al., 2024a) | Process | Mistral-NeMo-12B-Instruct,
Gemma-2-9B-Instruct,
Llama-3-8B-Instruct,
Mistral-7B-Instruct-v0.3 | [GitHub](https://llm-code-preference.github.io/) | | Focused-DPO (Zhang et al., 2025p) | Process | DeepSeek-Coder-6.7B-Base/Instruct,
Magicoder-S-DS-6.7B,
Qwen2.5-Coder-7B-Instruct | - | | Re:Form (Yan et al., 2025a) | Outcome | Qwen2.5 (0.5B–14B) | [GitHub](https://github.com/Veri-Code/ReForm) | | Qwen Team (Feng et al., 2025c) | Outcome | Qwen2.5-Coder-7B/32B-Instruct | - | | RL for Iterative Code Refinement | | | | | RLEF (Gehring et al., 2025) | Outcome | Llama-3.0-8B-Instruct, Llama-3.1-8B/70B-Instruct | - | | $\mu$Code (Jain et al., 2025a) | Outcome | Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct | [GitHub](https://github.com/portal-cornell/muCode) | | R1-Code-Interpreter (Chen et al., 2025h) | Outcome | Qwen2.5-7B/14B-Instruct-1M, Qwen2.5-3B-Instruct | [GitHub](https://github.com/yongchao98/R1-Code-Interpreter) | | IterPref (Wu et al., 2025c) | Process | Deepseek-Coder-7B-Instruct,
Qwen2.5-Coder-7B,
CodeQwen1.5-7B-Chat,
StarCoder2-15B | - | | LeDex (Jiang et al., 2024) | Process | StarCoder-15B, CodeLlama-7B/13B | - | | CTRL (Xie et al., 2025) | Process | Qwen2.5-Coder-7B/14B/32B-Instruct | [GitHub](https://github.com/HKUNLP/critic-rl) | | ReVeal (Jin et al., 2025c) | Process | DAPO-Qwen-32B | - | | RL for Automated Software Engineering (SWE) | | | | | DeepSWE (Luo et al., 2025b) | Outcome | Qwen3-32B | [GitHub](https://github.com/agentica-project/rllm) | | SWE-RL (Wei et al., 2025e) | Outcome | Llama-3.3-70B-Instruct | [GitHub](https://github.com/facebookresearch/swe-rl) | | Satori-SWE (Zeng et al., 2025a) | Outcome | Qwen2.5-Coder-32B-Instruct | [GitHub](https://github.com/satori-reasoning/Satori-SWE) | | RLCoder (Wang et al., 2025p) | Outcome | CodeLlama-7B,
StarCoder-7B,
StarCoder2-7B,
DeepSeekCoder-1B/7B | [GitHub](https://github.com/DeepSoftwareAnalytics/RLCoder) | | Qwen3-Coder (Team, 2025b) | Outcome | Qwen3-Coder-480B-A35B-Instruct | [GitHub](https://github.com/QwenLM/Qwen3) | | ML-Agent (Liu et al., 2025q) | Outcome | Qwen2.5-7B-Base/Instruct,
DeepSeek-R1-Distill-Qwen-7B | [GitHub](https://github.com/MASWorks/ML-Agent) | | OS-R1 (Lin et al., 2025b) | Outcome | Qwen2.5-3B/7B-Instruct | [GitHub](https://github.com/LHY-24/OS-R1) | | Golubev et al. (2025) | Process | Qwen2.5-72B-Instruct | - | | SWEET-RL (Zhou et al., 2025e) | Process | Llama-3.1-8B/70B-Instruct | [GitHub](https://github.com/facebookresearch/sweet_rl) | ##### Process reward RL. Process-supervised approaches explicitly guide *how* the model debugs. IterPref (Wu et al., 2025c) constructs localized preference pairs from iterative debugging traces and applies targeted preference optimization to penalize faulty regions, improving correction accuracy with minimal collateral updates. LeDex (Jiang et al., 2024) couples explanation-driven diagnosis with self-repair: it automatically curates explanation–refinement trajectories and applies dense, continuous rewards to jointly optimize explanation quality and code correctness via PPO, yielding consistent pass@1 gains over SFT-only coders. Beyond explanation-driven shaping, some works like CTRL (Xie et al., 2025) explicitly train separate critic models to evaluate each attempted refinement and provide gradient signals to the policy, though at the cost of added inference overhead. ReVeal (Jin et al., 2025c) extends process-level refinement into a self-evolving agent that autonomously generates tests and learns from per-turn rewards to enhance reasoning and recovery from errors. #### 4.2.3 RL for Automated Software Engineering ##### Outcome reward RL. End-to-end training in realistic environments demonstrates that sparse—but validated—success signals can scale. DeepSWE performs large-scale RL on software engineering missions using verified task completion as the sole reward, achieving leading open-source results on SWE-bench–style evaluations (Luo et al., 2025b). SWE-RL extracts rule-based, outcome-oriented signals from GitHub commit histories, enabling training on authentic improvement patterns and generalization to unseen bug-fixing tasks (Wei et al., 2025e). Satori‑SWE introduces an evolutionary RL-enabled test-time scaling method (EvoScale) that trains models to self-improve generations across iterations for sample-efficient software engineering tasks (Zeng et al., 2025a). OS-R1 (Lin et al., 2025b) presents a rule-based reinforcement learning framework for Linux kernel tuning, enabling efficient exploration, accurate configuration, and superior performance over heuristic methods. RLCoder frames retrieval-augmented repository-level code completion as an RL problem, using perplexity-based feedback to train a retriever to fetch helpful context without labeled data (Wang et al., 2025p). Qwen3‑Coder performs large-scale execution-driven reinforcement learning on long-horizon, multi-turn interactions across 20,000 parallel environments, yielding state-of-the-art performance on benchmarks like SWE‑Bench Verified (Team, 2025b). In machine learning domains, ML-Agent executes multi-step pipelines (e.g., automated ML), optimizing performance-based terminal rewards (Liu et al., 2025q). ##### Process reward RL. Dense supervision over agentic trajectories improves credit assignment across many steps. From the optimization perspective, long-context, multi-turn software agents benefit from stabilized policy-gradient variants; e.g., Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) improves training stability and performance on SWE-bench Verified through multi-turn code generation and debugging interactions, leveraging long-context feedback (Golubev et al., 2025). SWEET-RL trains multi-turn agents on ColBench (backend and frontend tasks), leveraging privileged information during RL to reduce exploration noise and improve long-horizon generalization (Zhou et al., 2025e). ##### Remark on closed-source systems. Commercial systems such as OpenAI’s Codex and Anthropic’s Claude Code have emphasized preference-aligned fine-tuning and reinforcement learning to improve usefulness and safety in code generation and editing workflows (OpenAI, 2025b; Anthropic, 2025). While concrete training details are limited publicly, these systems underscore the growing role of RL in aligning agentic behavior with developer-centric objectives in practical IDE and terminal environments. #### 4.2.4 Emerging Paradigms ##### Code World Models A recent paradigm shift departs from traditional neural approximations by framing the world model itself as executable code. In these Code World Models (CWMs), agents synthesize programs to explicitly define transition and reward dynamics, enabling model-based planning via verifiable, symbolic simulation rather than opaque latent states. GIF-MCTS (Dainese et al., 2024) formulates world-model construction as program induction: the LLM edits an “Environment” class and a search procedure selects versions that best explain offline transitions, yielding executable models suitable for downstream planning. WorldCoder (Tang et al., 2024) represents dynamics and rewards as explicit Python functions and refines them through an iterative synthesize–repair process guided by transition consistency and optimism constraints. Meta’s 32B CWM (team et al., 2025) strengthens this paradigm by providing an open-weights model trained on interpreter traces and agentic trajectories to improve program synthesis and execution fidelity. Recent work further applies CWMs to general game environments (Lehrach et al., 2025), where an LLM induces complete rule-based simulators and planning is performed directly on the executable model. Implementing these programmatic world-model paradigms often incurs substantial inference cost, since agents repeatedly synthesize, refine, and query executable simulators. In practice, low-bit quantization (e.g., 8-bit for workstation GPUs like RTX 4500 Ada or 4-bit for consumer hardware) is frequently adopted to make large code-oriented models feasible to deploy. Collectively, CWMs establish programmatic world models as a coherent direction for code agents, coupling LLM-based program synthesis with structured, verifiable simulation for model-based reasoning. ### 4.3 Mathematical Agent Mathematical reasoning is widely regarded as a gold standard for assessing LLM agents’ reasoning ability, owing to its symbolic abstraction, logical consistency, and long-horizon deductive demands. We structure the research efforts around two complementary paradigms: *informal reasoning* (Section [4.3.1](#S4.SS3.SSS1)), which operates without formal verification support and includes natural-language reasoning and programming-language tool use; and *formal reasoning* (Section [4.3.2](#S4.SS3.SSS2)), which relies on rigorously specified formal languages and proof assistants. We note that RLVR methods such as DAPO (Yu et al., 2025e), GRPO (Ren et al., 2025), and GRESO (Zheng et al., 2025b) have consistently played a substantial role in recent enhancements of mathematical reasoning in LLMs. However, given their broader relevance across reasoning tasks, we discuss them in Section [2.7](#S2.SS7), instead of elaborating here. #### 4.3.1 RL for Informal Mathematical Reasoning Informal mathematics essentially refers to reasoning and expression in natural language. Such reasoning may incorporate symbols or function names, but no finite and explicit set of logical rules defines their syntactic validity, and no formal semantics precisely determines their interpretation and meaning (Yang et al., 2024b; Asperti et al., 2025). While informal mathematical reasoning relaxes strict rigor at the detail level, it affords greater expressive flexibility and better captures the high-level structure of arguments. This makes it particularly suited for a variety of math tasks such as mathematical word-problem solving, equation manipulation, and symbolic computation (Singh et al., 2025; Mai et al., 2025). Although general-purpose programming languages are symbolic, they lack the rigor and formal semantics of proof-assistant languages, and are therefore regarded as informal when applied to mathematical reasoning (Yang et al., 2024b), typically through tool invocation of executors such as Python with numerical or symbolic libraries. ##### Outcome reward RL. Outcome-only methods define rewards solely by final numerical or symbolic correctness (e.g., algebraic equations) during RL. Empirically, such training often leads to emergent agentic behaviors, including adaptive tool use interleaved with natural language reasoning. ARTIST (Singh et al., 2025) introduces a framework for tool-integrated agentic reasoning, interleaving tool invocations, e.g. code execution, directly within the reasoning chain. Trained with outcome-only rewards, it achieves strong performance and observes emergent agentic behaviors, including self-reflection, and context-aware CoT, which further shows that by integrating dynamic tool use with RL, agentic tool-integrated reasoning could learn optimal strategies for interacting with environments, highlighting the potential of RL to internalize tool-integrated reasoning strategies in LLMs. Similarly, ToRL (Li et al., 2025l) improves performance by exploiting the scaling of tool-integrated reasoning RL and encouraging code execution behaviour, and experiments show emergent cognitive behaviors, such as adaptive tool-use, self-correction based on tool feedback, and adaptive computational reasoning. ZeroTIR (Mai et al., 2025) investigates the scaling law of RL from outcome-based rewards for Tool-Integrated Reasoning with Python code execution settings, revealing a strong correlation between training computational effort and the spontaneous code execution frequency, the average response length, and the final task accuracy, which corroborates the empirical emergence of tool-integrated reasoning strategies. TTRL (Zuo et al., 2025) leverages majority voting to estimate rewards, enabling training on unlabeled data. Fine-tuned on these majority-vote rewards, it not only surpasses the base model’s maj@n accuracy but also achieves an empirical performance curve and upper bound that, surprisingly, closely approach those of direct RL training with labeled test answers on MATH-500, underscoring its practical value and potential. However, RENT (Prabhudesai et al., 2025) suggests that majority voting is limited in generalization, it applies only to questions with deterministic answers, and will not work on free-response outputs. To address this limitation, it extends the entropy minimization idea (Wang et al., 2021) to RL, using the token-level average negative entropy as a reward to guide learning, achieving improvements on an extensive suite of benchmarks including math problem solving, suggesting that confidence-based reward shaping can serve as a path toward continual improvement. Alternatively, Satori (Shen et al., 2025b) proposes Chain-of-Action-Thought (COAT), a variant of CoT that explicitly integrates action choices, and modularizes reasoning into 3-fold meta-actions, including continuation, reflection, and exploration of alternatives, and internalizes this behavior via RL with outcome-only rewards. In particular, 1-shot RLVR (Wang et al., 2025r) studies data efficiency of outcome-only RL with verifier signals. Surprisingly, they found that RL with only 1 example performs close to using a 1.2k-example dataset, and with 2 examples comes close to using the 7.5k MATH training dataset. They also highlight an intriguing phenomenon, named post-saturation generalization, that test accuracy continues to improve even after the training accuracy on the single example approaches 100%. In addition to correctness, hallucination remains a major challenge in informal mathematical reasoning, motivating methods that explicitly promote trustworthiness. For instance, Kirchner et al. (2024) propose a game-theoretic training algorithm that jointly optimizes for both correctness and legibility. Inspired by Prover-Verifier Games (Anil et al., 2021), the method alternates between training a small verifier that predicts solution correctness, a "helpful" prover that generates solutions accepted by the verifier, and a "sneaky" prover that aims to fool it. Empirically, this increases the helpful prover accuracy, verifier robustness and legibility (measured by human accuracy in time-constrained verification tasks). This result suggests that verifier-guided legibility optimization can enhance the interpretability and trustworthiness of LLM-generated informal reasoning. Recent rStar2-Agent (Shang et al., 2025a) is a 14B-parameter math reasoning model trained with agentic reinforcement learning using a high-throughput Python execution environment, a novel GRPO-RoC algorithm to resample on correct rollouts amid tool-noise, and a multi-stage training recipe—achieving state-of-the-art results in just 510 RL steps, achieving average pass@1 scores of 80.6% on AIME24 and 69.8% on AIME25. ##### Process reward RL. Process-aware methods leverage intermediate evaluators (e.g. unit tests, assertions, sub-task checks) to provide denser feedback, shaping credit assignment and improving tool-integrated reasoning (TIR). START (Li et al., 2025b) guides TIR by injecting handcrafted hint text into Long CoT traces, typically after conjunction words or before the CoT stop token, to encourage code executor calls during inference. This enables test-time scaling that improves reasoning accuracy. The collected trajectories are then used to fine-tune the model, internalizing the tool-invocation behavior. LADDER (Simonds and Yoshiyama, 2025) introduces a training-time framework where an LLM recursively generates and solves progressively simpler variants of a complex problem, using verifiable reward signals to guide a difficulty-based curriculum, and achieves substantial improvements in mathematical reasoning. An additional test-time RL step (TTRL) further enhances performance. The authors suggest that this approach of self-generated curriculum learning with verifiable feedback may generalize beyond informal mathematical tasks to any domain with reliable automatic verification. To improve performance on complex problems, SWiRL (Goldie et al., 2025) synthesizes step-wise tool use reasoning data by iteratively decomposing solutions, and then adopts a preference-based step-wise RL approach to fine-tune the base model on the multi-step trajectories. While many of these approaches exploit inference-time interventions, they often suffer from generalization limitations due to their reliance on manually designed logical structures. To overcome this, RLoT (Hao et al., 2025b) instead trains a lightweight navigator agent model with RL to adaptively enhance reasoning, showing improved generalization across diverse tasks. While informal approaches excel at word problems and symbolic computations, they struggle to extend effectively to advanced mathematical tasks such as automated theorem proving. This limitation arises from two fundamental challenges: evaluation difficulty, which demands machine-verifiable feedback unavailable to informal methods, and scarcity of high-quality formal proof data (Yang et al., 2024b; Asperti et al., 2025). #### 4.3.2 RL for Formal Mathematical Reasoning Formal mathematical reasoning refers to reasoning carried out in a formal language with precisely defined syntax and semantics, yielding proof objects that are mechanically checkable by a verifier. This paradigm is particularly suited for advanced tasks such as automated theorem proving (ATP) (Xin et al., 2025), where an agent, given a statement (theorem, lemma, or proposition), must construct a proof object that the verifier accepts, thereby ensuring machine-verifiable correctness. From a reinforcement learning perspective, formal theorem proving is commonly modeled as a Markov Decision Process (MDP): proof states transition via the application of tactics(^2^22In Lean-style Interactive Theorem Provers (ITPs), a tactic is a command or small script that instructs the system to refine the current proof goal, with the resulting proof term checked by the ITP kernel for correctness.), each of which is treated as a discrete action in RL-based proof search (Wu et al., 2021). Under this formulation, formal theorem proving can be cast as a search problem over a vast, discrete, and parameterized action space. Formal proofs are verified by proof assistants such as Lean, Isabelle, Coq, and HOL Light. These systems, often referred to as Interactive Theorem Provers (ITPs), deterministically accept or reject proof objects, producing binary pass/fail signals as the primary reward for RL training, while some works also explore leveraging error messages as auxiliary signals (Ambati, 2025; Ji et al., 2025). ##### Outcome reward RL. The outcome-only paradigm was demonstrated at scale in 2024 with DeepSeek-Prover-v1.5 (Xin et al., 2025), which releases an end-to-end RL pipeline in Lean based solely on binary verifier feedback, resulting in significant improvements in proof success on benchmarks like miniF2F (Zheng et al., 2022) and ProofNet (Azerbayev et al., 2023). The authors propose a variant of MCTS, i.e. RMaxTS, that incorporates intrinsic rewards for discovering novel tactic states to encourage diversity of proof exploration during inference-time search and mitigate the sparse-reward issue. Building on this direction, Leanabell-Prover (Zhang et al., 2025l) scales up DeepSeek-Prover-v1.5 by aggregating an expansive hybrid dataset of statement-proof pairs and informal reasoning sketches from multiple sources and pipelines such as Mathlib4 (The mathlib Community, 2020), LeanWorkbook (Ying et al., 2025), NuminaMath (Li et al., 2024a), STP (Dong and Ma, 2025), etc., covering well over 20 mathematical domains. This broad coverage mitigates the scarcity of aligned informal-to-formal (NL to Lean4) training examples, which are crucial for bridging natural-language reasoning and formal proof generation. At the same time, Kimina-Prover (Wang et al., 2025a) Preview further emphasizes the critical challenge of aligning informal and formal reasoning. It implements a structured “formal reasoning pattern,” where natural-language reasoning and Lean 4 code snippets are interleaved within thinking blocks. To reinforce this alignment, the output is constrained—to include at least one tactic block and to reuse no less than 60% of the Lean 4 snippets in the final proof, ensuring close correspondence between internal reasoning and formal output. A recent work, Seed-Prover (Chen et al., 2025c), integrates multiple techniques. It first adopts a lemma-centered proof paradigm, which enables systematic problem decomposition, cross-trajectory lemma reuse, and explicit progress tracking. It then enriches RL training with a diverse prompting strategy that randomly incorporates both informal and formal proofs, successful and failed lemmas, and Lean compiler feedback, thereby enhancing adaptability to varied inputs. At inference, it employs a conjecture–prover pipeline that interleaves proving conjectures into lemmas and generating new conjectures from the evolving lemma pool, substantially improving its capacity to tackle difficult problems. Complementarily, the accompanying Seed-Geometry system extends formal reasoning to geometry, providing state-of-the-art performance on Olympiad benchmarks. Together, these efforts demonstrate that sparse but explicit reward signals can yield nontrivial gains, particularly when paired with effective exploration strategies. ##### Process reward RL. To improve credit assignment and reduce wasted exploration, several works extend the outcome-only paradigm with denser, step-level signals. DeepSeek-Prover-v2 (DeepSeek-AI et al., 2024) designs a dual-model pipeline to unify both informal (natural-language) and formal (Lean4) mathematical reasoning to reinforce the formal reasoning ability. It introduces subgoal decomposition, where a prover model solves recursively decomposed subgoals and receives binary Lean feedback at the subgoal level, effectively providing denser supervision and improving both accuracy and interpretability. Following this dual-role collaborative mindset, ProofNet++ (Ambati, 2025) implements a neuro-symbolic RL framework featuring a Symbolic Reasoning Interface, which maps LLM-generated reasoning into formal proof trees, and a Formal Verification Engine, which verifies these proofs with Lean or HOL Light and routes error feedback back to the LLM for self-correction. Leanabell-Prover-v2 (Ji et al., 2025) integrates verifier messages into reinforcement updates within a long CoT framework, enabling explicit verifier-aware self-monitoring that stabilizes tactic generation and reduces repeated failure patterns. ##### Hybrid reward RL. Although both outcome-only and process-aware reward paradigms have demonstrated encouraging advances, the scarcity of high-quality theorem-proving data further amplifies the challenges of reinforcement learning under sparse rewards as well as the design of step-level preference signals (Zeng and Zhong, 2024; Wang et al., 2024f; Dong and Ma, 2025). To mitigate these limitations, a prominent line of work adopts expert iteration (ExIt) (Anthony et al., 2017), a framework that combines search with policy learning. This paradigm provides an alternative to outcome-only or process-aware RL, alleviating data scarcity by producing high-quality supervised trajectories. Instead of directly optimizing against sparse verifier signals, ExIt performs *search-guided data augmentation*: valid proof trajectories discovered by search and checked by a verifier are reused as expert demonstrations in an imitation-learning loop. It usually employs a two-role system: the *expert* collects valid and progressive trajectories via MCTS under outcome-only verifier feedback, while the *apprentice* trains a policy on these process-level trajectories and then shares the improved policy back with the expert, thereby bootstrapping subsequent rounds of search and accelerating convergence. Prior work (Polu and Sutskever, 2020) introduces ExIt into formal theorem proving, demonstrating that search-generated expert data can bootstrap models toward tackling complex multi-step proving challenges. Later works adapt this design to Lean and other ITPs. When applied to formal theorem proving, naive tree search methods often face severe search space explosion when navigating the vast parameterized tactic space. To mitigate this, InternLM2.5-StepProver (Wu et al., 2025m) introduces a preference-based critic model, trained with RLHF-style optimization, to guide expert search, effectively providing a curriculum that directs exploration toward problems of suitable difficulty. Lean-STaR (Lin et al., 2025a) further enhances ExIt by integrating Self-Taught Reasoner (STaR) (Zelikman et al., 2022). It first trains a thought-augmented tactic predictor on synthesized (proof state, generated thought, ground-truth tactic) triples. Then, in the expert-iteration loop, the model produces trajectories that interleave thoughts with tactics; trajectories with tactics successfully validated by Lean are retained and reused for imitation learning. Empirically, the inclusion of thoughts increases the diversity of exploration in the sample-based proof search. A recent work, STP (Dong and Ma, 2025), points out that solely relying on expert iteration will quickly plateau due to the sparse positive rewards. To address this, it extends the conjecturer–prover self-play idea from Minimo (Poesia et al., 2024) to practical formal languages (Lean/Isabelle) with an open-ended action space and starts from a pretrained model. STP instantiates a dual-role loop in which a conjecturer proposes statements that are barely provable by the current prover, and a prover is trained with standard expert iteration; this generates an adaptive curriculum and alleviates sparse training signals. Empirically, STP reports large gains on LeanWorkbook (Ying et al., 2025) and reports competitive results among whole-proof generation methods on miniF2F (Zheng et al., 2022) and ProofNet (Azerbayev et al., 2023). Figure: Figure 8: Benchmark Performance of RL-Enhanced Math Methods. Scores are pass@1 unless otherwise specified. Refer to caption: 2509.02547v5/figures/math_benchmarks.png **Table 6: A summary of RL methods for mathematical reasoning agents.** | Method | Reward | Resources | | --- | --- | --- | | RL for Informal Mathematical Reasoning | | | | ARTIST (Singh et al., 2025) | Outcome | - | | ToRL (Li et al., 2025l) | Outcome | [GitHub](https://github.com/GAIR-NLP/ToRL)  [HuggingFace](https://huggingface.co/GAIR/ToRL-7B) | | ZeroTIR (Mai et al., 2025) | Outcome | [GitHub](https://github.com/yyht/openrlhf_async_pipline)  [HuggingFace](https://huggingface.co/htxu91/zero-tir-7b-550step) | | TTRL (Zuo et al., 2025) | Outcome | [GitHub](https://github.com/PRIME-RL/TTRL) | | RENT (Prabhudesai et al., 2025) | Outcome | [GitHub](https://github.com/satrams/rent-rl)
[Website](https://rent-rl.github.io/) | | Satori (Shen et al., 2025b) | Outcome | [GitHub](https://github.com/satori-reasoning/Satori)  [HuggingFace](https://huggingface.co/Satori-reasoning)
[Website](https://satori-reasoning.github.io/) | | 1-shot RLVR (Wang et al., 2025r) | Outcome | [GitHub](https://github.com/ypwang61/One-Shot-RLVR)  [HuggingFace](https://huggingface.co/collections/ypwang61/one-shot-rlvr-6827f72c3359b2ffe75fc1a8) | | Prover-Verifier Games  (Kirchner et al., 2024) | Outcome | - | | rStar2-Agent (Shang et al., 2025a) | Outcome | [GitHub](https://github.com/microsoft/rStar) | | START (Li et al., 2025b) | Process | - | | LADDER (Simonds and Yoshiyama, 2025) | Process | - | | SWiRL (Goldie et al., 2025) | Process | - | | RLoT (Hao et al., 2025b) | Process | [GitHub](https://anonymous.4open.science/r/RL-LLM-Reasoning-1A30) | | RL for Formal Mathematical Reasoning | | | | DeepSeek-Prover-v1.5 (Xin et al., 2025) | Outcome | [GitHub](https://github.com/deepseek-ai/DeepSeek-Prover-V1.5)  [HuggingFace](https://huggingface.co/deepseek-ai) | | Leanabell-Prover (Zhang et al., 2025l) | Outcome | [GitHub](https://github.com/Leanabell-LM/Leanabell-Prover)  [HuggingFace](https://huggingface.co/collections/stoney0062/leanabell-prover-67fe4fae1dcf1d7221e309e9) | | Kimina-Prover  (Wang et al., 2025a) | Outcome | [GitHub](https://github.com/MoonshotAI/Kimina-Prover-Preview)  [HuggingFace](https://huggingface.co/collections/AI-MO/kimina-prover-preview-67fb536b883d60e7ca25d7f9) | | Seed-Prover (Chen et al., 2025c) | Outcome | [GitHub](https://github.com/ByteDance-Seed/Seed-Prover) | | DeepSeek-Prover-v2 (DeepSeek-AI et al., 2024) | Process | [GitHub](https://github.com/deepseek-ai/DeepSeek-V2)  [HuggingFace](https://huggingface.co/deepseek-ai) | | ProofNet++ (Ambati, 2025) | Process | - | | Leanabell-Prover-v2 (Ji et al., 2025) | Process | [GitHub](https://github.com/Leanabell-LM/Leanabell-Prover-V2) | | InternLM2.5-StepProver (Wu et al., 2025m) | Hybrid | [GitHub](https://github.com/InternLM/InternLM-Math) | | Lean-STaR (Lin et al., 2025a) | Hybrid | [GitHub](https://github.com/Lagooon/LeanSTaR)  [HuggingFace](https://huggingface.co/ScalableMath/Lean-STaR-plus)
[Website](https://leanstar.github.io/) | | STP (Dong and Ma, 2025) | Hybrid | [GitHub](https://github.com/kfdong/STP)  [HuggingFace](https://huggingface.co/kfdong/STP_model_Lean_0320) | ### 4.4 GUI Agent GUI agents have progressed through distinct training paradigms. Early systems used pre-trained vision–language models (VLMs) in a pure zero-shot fashion, mapping screenshots and prompts directly to single-step actions. Later, SFT on static (screen, action) trajectories improved grounding and reasoning, but were limited by scarce human operation traces. Reinforcement fine-tuning (RFT) reframes GUI interaction as sequential decision-making, allowing agents to learn via trial-and-error with sparse or shaped rewards, and has advanced from simple single-task settings to complex, real-world, long-horizon scenarios. Table [7](#S4.T7) presents the majority of works studied in this section. #### 4.4.1 RL-free Methods ##### Vanilla VLM-based GUI Agents Early GUI agents directly leveraged pre-trained Vision–Language Models (VLMs) in a purely zero-shot manner, mapping screenshots and prompts to single-step actions without any task-specific fine-tuning. Representative systems include MM-Navigator (Yan et al., 2023), SeeAct (Zheng et al., 2024), and TRISHUL (Kunal Singh, 2025), which differ in interface domains or parsing strategies but share the same reliance on off-the-shelf VLMs. While showcasing the generality of foundation models, these approaches suffer from limited grounding accuracy and reliability, restricting their applicability to complex tasks (Zhang et al., 2025b; Nguyen et al., 2025a). ##### Supervised Fine-Tuning (SFT) with Static Trajectory Data The SFT paradigm adapts pre-trained vision–language models to GUI tasks by minimizing cross-entropy loss on offline (screen, action) pairs, without online interaction. InfiGUIAgent (Liu et al., 2025n) employs a two-stage pipeline that first improves grounding and then incorporates hierarchical and reflective reasoning. UI-AGILE (Lian et al., 2025) enhances supervised fine-tuning by incorporating continuous rewards, simplified reasoning, and cropping-based resampling, while further proposing a decomposed grounding mechanism for handling high-resolution displays. TongUI (Zhang et al., 2025a) instead emphasizes data scale, constructing the 143K-trajectory GUI-Net from multimodal web tutorials to enhance generalization. While differing in focus, these approaches all face the limitation of scarce human operation traces. #### 4.4.2 RL in Static GUI Environments In static settings, reinforcement learning is applied on pre-collected datasets with deterministic execution traces, using rule-based criteria for outcome evaluation in the absence of live environment interactions. GUI-R1 (Luo et al., 2025d) adopts an R1-style reinforcement fine-tuning pipeline over a unified action schema, using simple format and correctness rewards to improve step-level action prediction with modest data. UI-R1 (Lu et al., 2025d) applies group-relative policy optimization to stabilize policy updates and improve exact parameter matching through a compact action interface and reward shaping for action-type and argument accuracy. InFiGUI-R1 (Liu et al., 2025o) introduces a two-stage training paradigm that first distills spatial reasoning to enhance grounding, followed by reinforcement learning with sub-goal supervision and recovery mechanisms to improve long-horizon reasoning. AgentCPM-GUI (Zhang et al., 2025ab) combines grounding-aware pre-training, supervised imitation, and GRPO-based reinforcement fine-tuning with a concise JSON action space, reducing decoding overhead while improving robustness on long-horizon sequences. UI-Venus (Gu et al., 2025) is a multimodal screenshot-based UI agent fine-tuned via RFT with custom reward functions and a self-evolving trajectory framework, achieving a new state-of-the-art performance in both UI grounding and navigation. #### 4.4.3 RL in Interactive GUI Environments In interactive settings, reinforcement learning agents are optimized through online rollouts in dynamic environments, requiring robustness to stochastic transitions and long-horizon dependencies. WebAgent-R1 (Wei et al., 2025f) conducts end-to-end multi-turn reinforcement learning with asynchronous trajectory generation and group-wise advantages, improving success on diverse web tasks. Vattikonda et al. (2025) studies reinforcement learning for web agents under realistic page dynamics and large action spaces, highlighting challenges in credit assignment and safe exploration. UI-TARS (Qin et al., 2025) integrates pre-training for GUI understanding with reinforcement learning for native desktop control, coupling milestone tracking and reflection to enhance long-horizon execution. DiGiRL (Bai et al., 2024) introduces an offline-to-online reinforcement learning pipeline on real Android devices, combining advantage-weighted updates, doubly robust advantage estimation, and instruction-level curricula to cope with non-stationarity. ZeroGUI (Yang et al., 2025b) automates task generation and reward estimation with a vision-language evaluator, then applies two-stage online reinforcement learning (training on generated tasks followed by test-time adaptation) to reduce human supervision. MobileGUI-RL (Shi et al., 2025c) scales training on Android virtual devices with trajectory-aware GRPO, a decaying efficiency reward, and curriculum filtering, improving execution efficiency and generalization while keeping the system practical for large rollout volumes. ComputerRL (Lai et al., 2025) introduces an API‑GUI hybrid interaction paradigm paired with a massively parallel, fully asynchronous RL infrastructure and the novel Entropulse training strategy—alternating RL with supervised fine‑tuning—to empower GUI‑based agents to operate efficiently and scalably in desktop environments. **Table 7: A summary of methods for GUI agents, categorized by training paradigm and environment complexity.** | Method | Paradigm | Environment | Resource Link | | --- | --- | --- | --- | | RL-free GUI Agents | | | | | MM-Navigator (Yan et al., 2023) | Vanilla VLM | - | [GitHub](https://github.com/zzxslp/MM-Navigator) | | SeeAct (Zheng et al., 2024) | Vanilla VLM | - | [GitHub](https://github.com/OSU-NLP-Group/SeeAct) | | TRISHUL (Kunal Singh, 2025) | Vanilla VLM | - | - | | InfiGUIAgent (Liu et al., 2025n) | SFT | Static | [GitHub](https://github.com/InfiXAI/InfiGUIAgent)  [HuggingFace](https://huggingface.co/datasets/rootsautomation/ScreenSpot)
[Website](https://b7277.github.io/InfiGUIAgent.github.io/) | | UI-AGILE (Lian et al., 2025) | SFT | Interactive | [GitHub](https://github.com/KDEGroup/UI-AGILE)  [HuggingFace](https://huggingface.co/KDEGroup/UI-AGILE) | | TongUI (Zhang et al., 2025a) | SFT | Static | [GitHub](https://github.com/TongUI-agent/TongUI-agent)  [HuggingFace](https://huggingface.co/collections/Bofeee5675/tongui-67f611e2d48b2b6e0d2ba3ee)
[Website](https://tongui-agent.github.io/) | | RL-based GUI Agents | | | | | GUI-R1 (Luo et al., 2025d) | RL | Static | [GitHub](https://github.com/ritzz-ai/GUI-R1)  [HuggingFace](https://huggingface.co/ritzzai/GUI-R1) | | UI-R1 (Lu et al., 2025d) | RL | Static | [GitHub](https://github.com/lll6gg/UI-R1)  [HuggingFace](https://huggingface.co/LZXzju/Qwen2.5-VL-3B-UI-R1-E) | | InFiGUI-R1 (Liu et al., 2025o) | RL | Static | [GitHub](https://github.com/InfiXAI/InfiGUI-R1)  [HuggingFace](https://huggingface.co/InfiX-ai/InfiGUI-R1-3B) | | AgentCPM (Zhang et al., 2025ab) | RL | Static | [GitHub](https://github.com/OpenBMB/AgentCPM-GUI)  [HuggingFace](https://huggingface.co/openbmb/AgentCPM-GUI) | | UI-Venus (Gu et al., 2025) | RL | Static | [GitHub](https://github.com/inclusionAI/UI-Venus) | | WebAgent-R1 (Wei et al., 2025f) | RL | Interactive | - | | Vattikonda et al. (2025) | RL | Interactive | - | | UI-TARS (Qin et al., 2025) | RL | Interactive | [GitHub](https://github.com/bytedance/UI-TARS)  [HuggingFace](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B)
[Website](https://seed-tars.com/) | | UI-TARS-2 (Wang et al., 2025c) | RL | Interactive | [GitHub](https://github.com/bytedance/ui-tars)
[Website](https://seed-tars.com/showcase/ui-tars-2/) | | DiGiRL (Bai et al., 2024) | RL | Interactive | [GitHub](https://github.com/DigiRL-agent/digirl)  [HuggingFace](https://huggingface.co/collections/JackBAI/digirl-6682ea42bdfb5af9bfc5f29f)
[Website](https://digirl-agent.github.io/) | | ZeroGUI (Yang et al., 2025b) | RL | Interactive | [GitHub](https://github.com/OpenGVLab/ZeroGUI) | | MobileGUI-RL (Shi et al., 2025c) | RL | Interactive | - | | ComputerRL (Lai et al., 2025) | RL | Interactive | - | ### 4.5 Vision Agents RL has been applied to a wide range of vision tasks (including, but not limited to, image, video, 3D perception and generation). Since the number of related papers is substantial, this section does not aim to provide an exhaustive overview; for a more comprehensive survey on RL for various vision tasks, we refer readers to two dedicated surveys in vision (Wu et al., 2025h; Zhou et al., 2025a). ##### Image Tasks. The success of DeepSeek-R1 (DeepSeek-AI et al., 2025) has sparked widespread interest in applying RL to incentivize long-form reasoning behavior, encouraging LVLMs to produce extended CoT sequences that improve visual perception and understanding (Shao et al., 2024a). This research trajectory has evolved from early work that simply adapted R1-style objectives to the vision domain—aimed primarily at enhancing passive perception (Tan et al., 2025a; Li et al., 2025t; Huang et al., 2025c; Shen et al., 2025a; Peng et al., 2025; Xia et al., 2025a; Yang et al., 2025c; Gao et al., 2025a)—toward the now-popular paradigm of active perception, or “thinking with images” (Su et al., 2025c). The key transition lies in moving from text-only CoT that references an image once, to interactive, visually grounded reasoning, achieved through (i) grounding (Li et al., 2025e; Nagaraja et al., 2016; Mao et al., 2016; Fan et al., 2025c; Chung et al., 2025; Cao et al., 2025a), (ii) agentic tool use (Zhao et al., 2025d; Huang et al., 2025d; Wu et al., 2025g; Su et al., 2025b; Liu et al., 2025v; Su et al., 2025a), and (iii) visual imagination via sketching or generation (Xu et al., 2025e; Duan et al., 2025; Jiang et al., 2025b). Beyond text-only outputs, many vision tasks—such as scene understanding—require structured predictions like bounding boxes, masks, and segmentation maps. To begin with, Visual-RFT (Liu et al., 2025v) uses IoU with confidence as a verifiable reward for bounding-box outputs, while Vision-R1 (Huang et al., 2025c) incorporates precision and recall as localization rewards. Extending this idea, Liu et al. (2025p) applies GRPO to segmentation tasks, combining soft and strict rewards with bounding-box IoU and L1 loss, and point-wise L1 distance. VLM-R1 (Shen et al., 2025a) employs mean Average Precision (mAP) as a reward to explicitly incentivize detection and localization capabilities in LVLMs. Finally, R1-SGG (Chen et al., 2025m) introduces three variants of GRPO rewards for scene-graph matching—ranging from hard rewards based on text matching and IoU to softer rewards computed via text-embedding dot products. RL has also been widely applied to image generation, particularly through its integration with diffusion and flow models—for example, RePrompt (Wu et al., 2025f), Diffusion-KTO (Li et al., 2024c), Flow-GRPO (Liu et al., 2025d), and GoT-R1 (Duan et al., 2025). Beyond diffusion-based approaches, RL has been leveraged for autoregressive image generation, where it improves coherence, fidelity, and controllability by directly optimizing task- or user-specific reward signals (Wang et al., 2025f; Jiang et al., 2025b; Yuan et al., 2025a). ##### Video Tasks. Following the same spirit, numerous works have extended GRPO variants to the video domain (Cheng et al., 2024; Feng et al., 2024b; Maaz et al., 2023) to enhance temporal reasoning (Park et al., 2025b; Li et al., 2025k; Zhu et al., 2025c; Liao et al., 2025c; Ouyang, 2025). TW-GRPO (Dang et al., 2025) introduces a token-weighted GRPO framework that emphasizes high-information tokens to generate more focused reasoning chains and employs soft, multi-choice rewards for lower-variance optimization. EgoVLM (Vinod et al., 2025) combines keyframe-based rewards with direct GRPO training to produce interpretable reasoning traces tailored for egocentric video. DeepVideo-R1 reformulates the GRPO objective as a regression task (Park et al., 2025b), while VideoChat-R1 demonstrates that reinforcement fine-tuning (RFT) can be highly data-efficient for task-specific video reasoning improvements (Li et al., 2025k). TinyLLaVA-Video-R1 explores scaling RL to smaller video LLMs (Zhang et al., 2025v), and (Chen et al., 2025j) introduces infrastructure and a two-stage pipeline (CoT-SFT + RL) to support large-scale RL for long videos. Additional efforts have also extended RL for embodied video reasoning tasks (Zhao et al., 2025b). A similar trend is observed in video generation, where RL is applied to improve temporal coherence, controllability, and semantic alignment. Key examples include DanceGRPO (Xue et al., 2025), GAPO (Zhu et al., 2025a), GRADEO (Mou et al., 2025), InfLVG (Fang et al., 2025b), Phys-AR (Lin et al., 2025c), VideoReward (Liu et al., 2025e), TeViR (Chen et al., 2025i), and InstructVideo (Yuan et al., 2024). ##### 3D Vision Tasks. RL has also been widely adopted to advance 3D understanding (Hong et al., 2023; Xu et al., 2024b; Deng et al., 2024a; Chen et al., 2024a; Zhou et al., 2023; Chen et al., 2024d) and generation (Wang et al., 2024i; Yin et al., 2025; Siddiqui et al., 2024). MetaSpatial (Pan and Liu, 2025) introduces the first RL-based framework for 3D spatial reasoning, leveraging physics-aware constraints and rendered-image evaluations as rewards during training. Scene-R1 (Yuan et al., 2025d) learns to reason about 3D scenes without point-wise 3D supervision, while SpatialReasoner (Ma et al., 2025c) introduces shared 3D representations that unify perception, computation, and reasoning stages. In the domain of 3D generation, RL has been applied to improve text-to-3D alignment and controllability. Notable efforts include DreamCS (Zou et al., 2025), which aligns generation with human preferences; DreamDPO (Zhou et al., 2025f) and DreamReward (Ye et al., 2024), which optimize 3D generation using 2D reward signals; and Nabla-R2D3 (Liu et al., 2025i), which further refines 3D outputs with reinforcement-driven objectives. ### 4.6 Embodied Agents Embodied agents encompass a broad family of systems that perceive a structured environment and act within it, ranging from vision-language-action (VLA) models to language-driven open-ended agents. While many recent systems focus on VLA settings that require grounding in real-world visual observations, all embodied agents must integrate perception, reasoning, and action to operate effectively in complex physical or simulated environments and to execute goal-directed behaviors conditioned on high-level instructions. These competencies form a foundational component of agentic LLMs and MLLMs in embodied scenarios. In instruction-driven embodied scenarios, RL is often employed as a post-training strategy. A common pipeline begins with a pre-trained vision-language-action (VLA) model (Kim et al., 2024; Black et al., 2024; Team et al., 2025a; Liao et al., 2025b) obtained through imitation learning under teacher forcing supervision. This model is then embedded into an interactive agent that engages with the environment to collect reward signals. These rewards guide the iterative refinement of the policy, supporting effective exploration, improving sample efficiency, and enhancing the model’s generalization capabilities across diverse real-world conditions. RL in VLA frameworks (SimpleVLA-RL Team, 2025; Lu et al., 2025a; Qi et al., 2025; Song et al., 2025e) can be broadly categorized into two classes: navigation agents, which emphasize spatial reasoning and locomotion in complex environments, and manipulation agents, which focus on the precise control of physical objects under diverse and dynamic constraints. ##### RL in VLA Navigation Agent. For navigation agents, planning is the central capability. Reinforcement learning is employed to enhance the VLA model’s ability to predict and optimize future action sequences. A common strategy (Zhao et al., 2025c) is to integrate traditional robotics-style RL, using step-wise directional rewards, directly into VLA-based navigation frameworks. Some approaches operate at the trajectory level. VLN-R1 (Qi et al., 2025) aligns predicted and ground-truth paths to define trajectory-level rewards, and applies GRPO, following DeepSeek-R1, to improve predictive planning. OctoNav-R1 (Gao et al., 2025a) also leverages GRPO but focuses on reinforcing internal deliberation within the VLA model, promoting a thinking-before-acting paradigm that enables more anticipatory and robust navigation. S2E (He et al., 2025) introduces a reinforcement learning framework that augments navigation foundation models with interactivity and safety, combining video pretraining with RL to achieve superior generalization and performance on the NavBench-GS benchmark. ##### RL in VLA Manipulation Agent. Manipulation agents, typically involving robotic arms, require fine-grained control for executing structured tasks under diverse conditions. In this context, RL is employed to enhance the instruction-following and trajectory prediction capabilities of VLA models, especially to improve generalization across tasks and environments. RLVLA (Liu et al., 2025f) and VLA-RL (Lu et al., 2025a) adopt pre-trained VLMs as evaluators, using their feedback to assign trajectory-level rewards for VLA policy refinement. These methods establish an online RL framework that effectively improves manipulation performance and demonstrates favorable scaling properties. TGRPO  (Chen et al., 2025k) further incorporates GRPO into manipulation tasks by defining rule-based reward functions over predicted trajectories. This enables the VLA model to generalize to unseen scenarios and improves its robustness in real-world deployment. VIKI-R (Kang et al., 2025a) complements this with a unified benchmark and two-stage framework for multi-agent embodied cooperation, combining Chain-of-Thought fine-tuning with multi-level RL to enable compositional coordination across diverse embodiments. A central challenge in RL for VLA embodied agents is scaling training to real-world environments. While simulation platforms enable efficient large-scale experimentation, the sim-to-real gap remains significant, particularly in fine-grained manipulation tasks. Conducting RL directly in real-world settings is currently impractical due to the high cost and complexity of physical robot experiments. Most RL algorithms require millions of interaction steps, which demand substantial time, resources, and maintenance. As a result, developing scalable embodied RL pipelines that can bridge the gap between simulation and real-world deployment remains an open and pressing problem. ##### Case Study: Voyager. Beyond these general challenges in embodied RL, Voyager (Wang et al., 2024a), a language-driven open-ended embodied agent, illustrates how planning, skill acquisition, and RL-based curriculum learning can be integrated in practice. The agent explores Minecraft using an iterative loop: it generates a plan, interacts with the environment, extracts reusable skills from successful trajectories, and stores them in a growing skill library. A curriculum scheduler selects new tasks based on the agent’s current skill set, while RL objectives guide which behavior should be committed as skills and when to refine or discard them. This creates a self-improving cycle in which planning, environmental interaction, memory, and RL-driven curriculum optimization are tightly coupled. ### 4.7 Multi-Agent Systems Large Language Model (LLM)-based Multi-agent Systems (MAS) comprise multiple autonomous agents collaborating to solve complex tasks through structured interaction, coordination, and memory management. Early static and hand-designed MAS such as CAMEL and MetaGPT (Li et al., 2023a; Hong et al., 2024b) explored role specialization and task decomposition, while debate-based frameworks such as MAD and MoA (Wang et al., 2025g; Liang et al., 2024) enhanced reasoning via collaborative refinement. Subsequent multi-agent research has shifted to proposing optimizable cooperative systems, which enable MAS to not only dynamically adjust coordination patterns but also directly enhance agent-level reasoning and decision-making strategies. Table [8](#S4.T8) summarizes the main body of works discussed in this section. ##### RL-Free Multi-Agent Evolution In the RL-free self-evolving setting, foundation models cannot be directly optimized; instead, system evolution is driven by mechanisms such as symbolic learning (Zhou et al., 2024c), dynamic graph optimization (Zhuge et al., 2024; Ma et al., 2025d; Zhou et al., 2025b), and workflow rewriting (Hu et al., 2025d; Zhang et al., 2025j; i). These methods improve the coordination and adaptability within MAS, but cannot directly update the parameters of foundation models. **Table 8: A summary of reinforcement learning and evolution paradigms in LLM-based Multi-Agent Systems. “Dynamic” denotes whether the multi-agent system is task-dynamic, i.e., processes different task queries with different configurations (agent count, topologies, reasoning depth, prompts, etc). “Train” denotes whether the method involves training the LLM backbone of agents.** | Method | Dynamic | Train | RL Algorithm | Resource Link | | --- | --- | --- | --- | --- | | RL-Free Multi-Agent Systems (not exhaustive) | | | | | | CAMEL (Li et al., 2023a) | ✗ | ✗ | - | [GitHub](https://github.com/camel-ai/camel)  [HuggingFace](https://huggingface.co/camel-ai) | | MetaGPT (Hong et al., 2024b) | ✗ | ✗ | - | [GitHub](https://github.com/FoundationAgents/MetaGPT) | | MAD (Liang et al., 2024) | ✗ | ✗ | - | [GitHub](https://github.com/Skytliang/Multi-Agents-Debate) | | MoA (Wang et al., 2025g) | ✗ | ✗ | - | [GitHub](https://github.com/togethercomputer/moa) | | AFlow (Zhang et al., 2025j) | ✗ | ✗ | - | [GitHub](https://github.com/FoundationAgents/AFlow) | | RL-Based Multi-Agent Training | | | | | | GPTSwarm (Zhuge et al., 2024) | ✗ | ✗ | policy gradient | [GitHub](https://github.com/metauto-ai/gptswarm)
[Website](https://gptswarm.org/) | | MaAS (Zhang et al., 2025f) | ✔ | ✗ | policy gradient | [GitHub](https://github.com/bingreeky/MaAS) | | G-Designer (Zhang et al., 2025g) | ✔ | ✗ | policy gradient | [GitHub](https://github.com/yanweiyue/GDesigner) | | Optima (Chen et al., 2025e) | ✗ | ✔ | DPO | [GitHub](https://github.com/thunlp/Optima) | | DITS (Shi et al., 2025a) | ✗ | ✔ | DPO | - | | MALT (Motwani et al., 2025) | ✗ | ✔ | DPO | - | | MARFT (Liao et al., 2025a) | ✗ | ✔ | MARFT | [GitHub](https://github.com/jwliao-ai/MARFT) | | ACC-Collab (Estornell et al., 2025b) | ✗ | ✔ | DPO | - | | MAPoRL (Park et al., 2025a) | ✔ | ✔ | PPO | [GitHub](https://github.com/%0Achanwoo-park-official/MAPoRL) | | MLPO (Estornell et al., 2025a) | ✔ | ✔ | MLPO | - | | ReMA (Wan et al., 2025b) | ✔ | ✔ | MAMRP | [GitHub](https://github.com/ziyuwan/ReMA-public) | | FlowReasoner (Gao et al., 2025b) | ✔ | ✔ | GRPO | [GitHub](https://github.com/sail-sg/FlowReasoner) | | CURE (Wang et al., 2025q) | ✗ | ✔ | rule-based RL | [GitHub](https://github.com/Gen-Verse/CURE)
[HuggingFace](https://huggingface.co/collections/Gen-Verse/reasonflux-coder-6833109ed9300c62deb32c6b) | | MMedAgent-RL (Xia et al., 2025b) | ✗ | ✔ | GRPO | - | | Chain-of-Agents (Li et al., 2025h) | ✔ | ✔ | DAPO | [GitHub](https://github.com/OPPO-PersonalAI/Agent_Foundation_Models)  [HuggingFace](https://huggingface.co/collections/PersonalAILab/afm-models-689200e11d0b21a67c015ba8) | | RLCCF (Yuan et al., 2025b) | ✗ | ✔ | GRPO | - | | MAGRPO (Liu et al., 2025l) | ✗ | ✔ | MAGRPO | - | #### 4.7.1 RL-Driven Optimization of Non-Parametric Coordination Modules These approaches keep agent parameters frozen while using RL to optimize external coordination structures such as communication topologies, routing policies, or workflow graphs. Methods such as GPTSwarm, MaAS, and G-Designer (Zhuge et al., 2024; Zhang et al., 2025f; g) treat MAS coordination as a graph-level policy updated via policy gradient. Because no agent-level gradients exist, credit assignment must operate at the topology or message-routing level. Rewards are typically delayed and sparse—e.g., only final task accuracy—requiring global-to-local credit decomposition or structural priors to avoid collapse. A key comparison emerges between *fixed communication protocols* (pre-specified message formats) and *learnable protocols*. Fixed protocols excel in low-data or highly specialized domains where stability is critical, whereas learnable protocols allow RL to discover efficient emergent communication but require substantially higher sample complexity and careful regularization to prevent overfitting or degenerate conventions. #### 4.7.2 RL-Driven Optimization of Selected Agent Policies A second class of systems updates only a subset of agents—often a leader, coordinator, or specialized expert—while keeping others frozen for stability. Representative examples include Optima, DITS, MALT, ACC-Collab (Chen et al., 2025e; Shi et al., 2025a; Motwani et al., 2025; Estornell et al., 2025b). These approaches balance flexibility and scalability: training only a few agents reduces sample complexity and avoids the instability of fully-decoupled credit assignment. MALT (Motwani et al., 2025) employs a heterogeneous multi-agent search tree to generate large-scale labeled trajectories, fine-tuning agents via a combination of Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) from both successful and failed reasoning paths. Credit assignment in this regime is fundamentally *semi-local*: rewards emerge from a collective trajectory, but gradients apply only to the optimized agent(s). This requires mechanisms such as role-conditioned DPO (Motwani et al., 2025), local advantage estimation, or counterfactual baselines to prevent reward hijacking by non-updated agents. Empirically, such partial optimization yields better sample efficiency than fully joint multi-agent training while still enabling the emergence of specialized roles. #### 4.7.3 End-to-End Multi-Agent Reinforcement Learning Full multi-agent RL jointly trains all agents under a shared or decentralized objective, typically formalized as a Dec-POMDP. Methods such as MAGRPO, MAPoRL, MLPO, ReMA, FlowReasoner, Chain-of-Agents, and SPIRAL (Liu et al., 2025l; Park et al., 2025a; Estornell et al., 2025a; Wan et al., 2025b; Gao et al., 2025b; Li et al., 2025h; Liu et al., 2025b) jointly optimize collaboration and reasoning behaviors, enabling emergent division of labor and communication conventions. For example, MAGRPO (Liu et al., 2025l) formalizes multi‑LLM cooperation as a Dec-POMDP problem and introduces a multi‑agent variant of GRPO, which enables joint training of LLM agents in MAS while maintaining decentralized execution. MAPoRL (Park et al., 2025a) extends MAD by verifying debate responses and using validation outcomes as RL rewards to improve collaborative reasoning. RLCCF (Yuan et al., 2025b) is a self-supervised multi-agent RL framework that leverages self-consistency-weighted ensemble voting to generate pseudo-labels and collaboratively optimize individual model policies via GRPO, boosting both individual and collective reasoning accuracy. ReMA (Wan et al., 2025b) separates reasoning into a meta-thinking agent and an execution agent, jointly trained under aligned RL objectives with parameter sharing. LERO (Wei et al., 2025d) combines MARL with LLM-generated hybrid rewards and evolutionary search to improve credit assignment and partial observability handling in cooperative tasks. CURE (Wang et al., 2025q) focuses on code generation, jointly training a code generator and unit tester via RL to produce richer reward signals, achieving strong generalization across diverse coding benchmarks. MMedAgent-RL (Xia et al., 2025b) introduces a reinforcement learning-based multi-agent framework for medical VQA, where dynamically coordinated general practitioners and specialists collaboratively reason with curriculum-guided learning, significantly outperforming existing Med-LVLMs and achieving more human-like diagnostic behavior. Chain-of-Agents (COA) (Li et al., 2025h) is an end-to-end paradigm where a single LLM simulates multi-agent collaboration by dynamically orchestrating role-playing and tool-using agents; this is achieved through multi-agent distillation (converting trajectories from state-of-the-art multi-agent systems into training data) and agentic reinforcement learning with carefully designed reward functions, resulting in Agent Foundation Models (AFMs). SPIRAL (Liu et al., 2025b) presents a fully online, multi-turn, multi-agent self-play reinforcement learning framework for LLMs in zero-sum games, employing a shared policy with role-conditioned advantage estimation (RAE) to stabilize learning, and demonstrates that gameplay fosters transferable reasoning skills that significantly improve mathematical and general reasoning benchmarks. However, end-to-end multi-LLM training exacerbates the *temporal and structural credit assignment* problem because rewards may depend on long multi-turn interaction chains. Solutions include role-conditioned advantage estimation (RAE), hierarchical controller–worker architectures (MLPO, ReMA), and self-play curricula (SPIRAL) that densify reward signals by constructing increasingly challenging interactions. These hierarchical patterns mirror enterprise deployments where a supervisory agent coordinates multiple workers; RL proves particularly effective at learning stable delegation and arbitration strategies under sparse reward settings. Despite their expressiveness, joint MARL approaches face scalability limits: sample complexity grows roughly linearly with the number of agents and quadratically with interaction depth. Algorithms such as MAGRPO and PPO-based MAPoRL mitigate this using centralized critics or value-shared baselines, but achieving scalable credit decomposition remains a central open challenge. ### 4.8 Other Tasks ##### TextGame. ARIA (Yang et al., 2025e) compresses the sprawling action space via intention-driven reward aggregation, reducing sparsity and variance. GiGPO (Feng et al., 2025b) enhances temporal credit assignment through hierarchical grouping without added computational burden. RAGEN (Wang et al., 2025v) ensures stable multi-turn learning by filtering trajectories and stabilizing gradients, while advocating for reasoning-aware rewards. SPA-RL (Wang et al., 2025b) decomposes delayed rewards into per-step signals, improving performance and grounding accuracy. Trinity-RFT (Pan et al., 2025) provides a unified, modular framework for reinforcement fine-tuning across tasks—including text games—enabling flexible, efficient, and scalable experimentation with diverse RL modes and data pipelines. ##### Table. SkyRL-SQL (Liu et al., 2025j) introduces a data-efficient, multi-turn RL pipeline for Text-to-SQL, enabling LLM agents to interactively probe databases, refine, and verify SQL queries. With just 653 training examples, the SkyRL-SQL-7B model surpasses both GPT-4o and o4-mini on SQL generation benchmarks. MSRL (Chen et al., 2025a) introduces multimodal structured reinforcement learning with multi-granularity rewards to overcome the SFT plateau in chart-to-code generation, achieving state-of-the-art performance on chart understanding benchmarks. ##### Time Series. Time-R1 (Liu et al., 2025t) enhances moderate-sized LLMs with comprehensive temporal reasoning abilities through a progressive reinforcement learning curriculum and a dynamic rule-based reward system. TimeMaster (Zhang et al., 2025n) trains time-series MLLMs that combine SFT with GRPO to enable structured, interpretable temporal reasoning over visualized time-series inputs. ##### General QA. Agent models (Zhang et al., 2025z) internalize chain-of-action generation to enable autonomous and efficient decision-making through a combination of supervised fine-tuning and reinforcement learning. L-Zero (Zhang et al., 2025m) enables large language models to become general-purpose agents through a scalable, end-to-end reinforcement learning pipeline utilizing a low-cost, extensible, and sandboxed concurrent agent worker pool. ##### Social. Sotopia-RL (Yu et al., 2025c) refines coarse episode-level rewards into utterance-level, multi-dimensional signals to enable efficient and stable RL training for socially intelligent LLMs under partial observability and multi-faceted objectives. Wang et al. (2025h) introduces an Adaptive Mode Learning (AML) framework with the Adaptive Mode Policy Optimization (AMPO) algorithm, which uses reinforcement learning to dynamically switch between multi-granular reasoning modes in social intelligence tasks, achieving higher accuracy and shorter reasoning chains than fixed-depth RL methods like GRPO. **Table 9: Application-capability dependency matrix. Dots indicate qualitative dependency levels: $\bullet$ Core, $\circ$ Supporting, – Minimal. The heatmap provides a navigation aid linking the capability taxonomy (Section [3](#S3)) with the application domains (Section [6](#S4.F6)).** | | Agentic Capability | | | | | | | --- | --- | --- | --- | --- | --- | --- | | Application | Planning | Tool-Use | Memory | Self-Imp. | Reasoning | Percep. | | Search | $\bullet$ | $\bullet$ | $\circ$ | $\circ$ | $\bullet$ | – | | Code | $\circ$ | $\bullet$ | $\circ$ | $\circ$ | $\bullet$ | – | | Math | $\bullet$ | $\circ$ | – | $\circ$ | $\bullet$ | – | | GUI | $\bullet$ | $\bullet$ | $\circ$ | – | $\circ$ | $\bullet$ | | Vision | $\circ$ | $\circ$ | – | – | $\bullet$ | $\bullet$ | | Embodied | $\bullet$ | $\circ$ | $\bullet$ | – | $\circ$ | $\bullet$ | | MAS | $\bullet$ | $\circ$ | $\circ$ | – | $\bullet$ | $\circ$ | ## 5 Enviroment and Frameworks ### 5.1 Environment Simulator In agentic reinforcement learning, the environment is the world with which the agent interacts, receiving sensory input (observations) and enacting choices (actions) through its actuators. The environment, in turn, responds to the agent’s actions by transitioning to a new state and providing a reward signal. With the rise of the LLM Agent paradigm, many works have proposed environments for training specific tasks. Table [10](#S5.T10) provides an overview of the key environments examined in this section. **Table 10: A summary of environments and benchmarks for agentic reinforcement learning, categorized by agent capability, task domain, and modality. The agent capabilities are denoted by: ① Reasoning, ② Planning, ③ Tool Use, ④ Memory, ⑤ Collaboration, ⑥ Self-Improve.** | Environment / Benchmark | Agent Capability | Task Domain | Modality | Resource Link | | --- | --- | --- | --- | --- | | LMRL-Gym (Abdulhai et al., 2025) | ①, ④ | Interaction | Text | [GitHub](https://github.com/abdulhaim/LMRL-Gym) | | ALFWorld (Shridhar et al., 2021) | ②, ① | Embodied, Text Games | Text | [GitHub](https://github.com/alfworld/alfworld)
[Website](https://alfworld.github.io/) | | TextWorld (Côté et al., 2019) | ②, ① | Text Games | Text | [GitHub](https://github.com/microsoft/TextWorld) | | ScienceWorld (Wang et al., 2022) | ①, ② | Embodied, Science | Text | [GitHub](https://github.com/allenai/ScienceWorld)
[Website](https://sciworld.apps.allenai.org/) | | AgentGym (Xi et al., 2025) | ①, ④ | Text Games | Text | [GitHub](https://github.com/WooooDyy/AgentGym)
[Website](https://agentgym.github.io/) | | Agentbench (Liu et al., 2024b) | ① | General | Text, Visual | [GitHub](https://github.com/THUDM/AgentBench) | | InternBootcamp (Li et al., 2025g) | ① | General, Coding, Logic | Text | [GitHub](https://github.com/InternLM/InternBootcamp) | | LoCoMo (Maharana et al., 2024) | ④ | Interaction | Text | [GitHub](https://github.com/snap-research/LoCoMo)
[Website](https://snap-research.github.io/locomo/) | | MemoryAgentBench (Hu et al., 2025e) | ④ | Interaction | Text | [GitHub](https://github.com/HUST-AI-HYZ/MemoryAgentBench) | | WebShop (Yao et al., 2022) | ②, ③ | Web | Text | [GitHub](https://github.com/princeton-nlp/WebShop)
[Website](https://webshop-pnlp.github.io/) | | Mind2Web (Gou et al., 2025) | ②, ③ | Web | Text, Visual | [GitHub](https://github.com/OSU-NLP-Group/Mind2Web-2)
[Website](https://osu-nlp-group.github.io/Mind2Web-2/) | | WebArena (Zhou et al., 2024b) | ②, ③ | Web | Text | [GitHub](https://github.com/web-arena-x/webarena)
[Website](https://webarena.dev/) | | VisualwebArena (Koh et al., 2024) | ①, ②, ③ | Web | Text, Visual | [GitHub](https://github.com/web-arena-x/visualwebarena)
[Website](https://jykoh.com/vwa) | | AppBench (Wang et al., 2024b) | ②, ③ | App | Text | [GitHub](https://github.com/hrwise-nlp/AppBench) | | AppWorld (Trivedi et al., 2024) | ②, ③ | App | Text | [GitHub](https://github.com/stonybrooknlp/appworld)
[Website](https://appworld.dev/) | | AndroidWorld (Rawles et al., 2025) | ②, ③ | GUI, App | Text, Visual | [GitHub](https://github.com/google-research/android_world) | | OSWorld (Xie et al., 2024) | ②, ③ | GUI, OS | Text, Visual | [GitHub](https://github.com/xlang-ai/OSWorld)
[Website](https://os-world.github.io/) | | WindowsAgentArena (Bonatti et al., 2024) | ② | GUI, OS | Text, Visual | [GitHub](https://github.com/microsoft/WindowsAgentArena)
[Website](https://microsoft.github.io/WindowsAgentArena/) | | Debug-Gym (Yuan et al., 2025c) | ①, ③ | SWE | Text | [GitHub](https://github.com/microsoft/debug-gym)
[Website](https://microsoft.github.io/debug-gym/) | | MLE-Dojo (Qiang et al., 2025) | ②, ① | MLE | Text | [GitHub](https://github.com/MLE-Dojo/MLE-Dojo)
[Website](https://mle-dojo.github.io/MLE-Dojo-page/) | | $\tau$-bench (Barres et al., 2025) | ①, ③ | SWE | Text | [GitHub](https://github.com/sierra-research/tau2-bench) | | TheAgentCompany (Xu et al., 2024a) | ②, ③, ⑤ | SWE | Text | [GitHub](https://github.com/TheAgentCompany/TheAgentCompany)
[Website](https://the-agent-company.com/) | | MedAgentGym (Xu et al., 2025c) | ① | Science | Text | [GitHub](https://github.com/wshi83/MedAgentGym) | | SecRepoBench (Dilgren et al., 2025) | ①, ③ | Coding, Security | Text | - | | R2E-Gym (Jain et al., 2025c) | ①, ② | SWE | Text | [GitHub](https://github.com/R2E-Gym/R2E-Gym)
[Website](https://r2e-gym.github.io/) | | BigCodeBench (Zhuo et al., 2025) | ① | Coding | Text | [GitHub](https://github.com/bigcode-project/bigcodebench)
[Website](https://bigcode-bench.github.io/) | | LiveCodeBench (Jain et al., 2025b) | ① | Coding | Text | [GitHub](https://github.com/LiveCodeBench/LiveCodeBench)
[Website](https://livecodebench.github.io) | | SWE-bench (Jimenez et al., 2024) | ①, ③ | SWE | Text | [GitHub](https://github.com/swe-bench/SWE-bench)
[Website](https://www.swebench.com/) | | SWE-rebench (Badertdinov et al., 2025) | ①, ③ | SWE | Text | [Website](https://swe-rebench.com/) | | DevBench (Li et al., 2025a) | ②, ① | SWE | Text | [GitHub](https://github.com/open-compass/DevEval) | | ProjectEval (Liu et al., 2025g) | ②, ① | SWE | Text | [GitHub](https://github.com/RyanLoil/ProjectEval/)
[Website](https://ryanloil.github.io/ProjectEval/) | | DA-Code (Huang et al., 2024b) | ①, ③ | Data Science, SWE | Text | [GitHub](https://aclanthology.org/2024.emnlp-main.748/)
[Website](https://github.com/yiyihum/da-code) | | ColBench (Zhou et al., 2025e) | ②, ① | SWE, Web Dev | Text | [GitHub](https://arxiv.org/abs/2503.15478)
[Website](https://github.com/facebookresearch/sweet_rl) | | NoCode-bench (Deng et al., 2025a) | ②, ① | SWE | Text | [GitHub](https://github.com/NoCode-bench/NoCode-bench)
[Website](https://nocodebench.org/) | | MLE-Bench (Chan et al., 2025) | ②, ①, ③ | MLE | Text | [GitHub](https://github.com/openai/mle-bench/)
[Website](https://openai.com/index/mle-bench/) | | PaperBench (Starace et al., 2025) | ②, ①, ③ | MLE | Text | [GitHub](https://github.com/openai/preparedness/tree/main/project/paperbench)
[Website](https://openai.com/index/paperbench/) | | Crafter (Hafner, 2022) | ②, ④ | Game | Visual | [GitHub](https://openreview.net/forum?id=1W0z96MFEoH)
[Website](https://danijar.com/crafter) | | Craftax (Matthews et al., 2024) | ②, ④ | Game | Visual | [GitHub](https://github.com/MichaelTMatthews/Craftax) | | ELLM (Crafter variant) (Du et al., 2023) | ②, ① | Game | Visual | [GitHub](https://proceedings.mlr.press/v202/du23f.html)
[Website](https://github.com/yuqingd/ellm) | | SMAC / SMAC-Exp (Samvelyan et al., 2019) | ⑤, ② | Game | Visual | [GitHub](https://github.com/oxwhirl/smac) | | Factorio (Hopkins et al., 2025) | ②, ① | Game | Visual | [GitHub](https://github.com/JackHopkins/factorio-learning-environment)
[Website](https://jackhopkins.github.io/factorio-learning-environment/) | | SMAC-Hard (Deng et al., 2024b) | ②, ④ | Game | Visual | [GitHub](https://github.com/devindeng94/smac-hard) | | TacticCraft (Ma et al., 2025a) | ②, ⑤ | Game | Text | - | #### 5.1.1 Web Environments In the realm of web-based environments, several benchmarks offer controlled yet realistic static environments for Agentic RL. WebShop (Yao et al., 2022) is a simulated e-commerce website featuring a large catalog of real-world products and crowdsourced text instructions. Agents navigate various webpage types and issue diverse actions (e.g., searching, selecting items, customizing, purchasing) to find and buy products, with its deterministic search engine aiding reproducibility. Furthermore, Mind2Web (Gou et al., 2025) is a dataset designed for generalist web agents, featuring a substantial number of tasks from many real-world websites across diverse domains. It provides webpage snapshots and crowdsourced action sequences for tasks like finding flights or interacting with social profiles, emphasizing generalization across unseen websites and domains. Similarly, WebArena (Zhou et al., 2024b) and its multimodal extension, VisualwebArena (Koh et al., 2024), are self-hostable, reproducible web environments delivered as Docker containers. WebArena features fully functional websites across common domains like e-commerce, social forums, collaborative development, and content management systems, enriched with utility tools and knowledge bases, and supports multi-tab tasks and user role simulation. VisualwebArena extends this by introducing new tasks requiring visual comprehension and a “Set-of-Marks” (SoM) representation to annotate interactable elements on screenshots, bridging the gap for multimodal web agents. Additionally, AppWorld (Trivedi et al., 2024) constitutes an environment simulating a multi-application ecosystem, encompassing 9 daily-use applications (e.g., Amazon, Spotify, Gmail) with 457 invokable APIs, and constructing a digital world featuring approximately 100 virtual characters and their social relationships. Agents accomplish complex tasks (such as travel planning and social relationship management) by writing code to call APIs. In these environments, all changes to the web pages or visual elements occur exclusively in response to the agent’s actions. #### 5.1.2 GUI Environments AndroidWorld (Rawles et al., 2025) exemplifies such dynamism as a benchmarking environment operating on a live Android emulator, featuring 116 hand-crafted tasks across 20 real-world applications. Its dynamic nature is underscored by parameter instantiation that generates millions of unique task variations, ensuring the environment evolves into novel configurations without direct agent influence. Agents interact through a consistent interface (supporting screen interactions, app navigation, and text input) while receiving real-time state feedback, with integration to MiniWoB++ providing durable reward signals for evaluating adaptive performance. OSWorld (Xie et al., 2024) is a scalable real computer environment for multimodal agents, supporting task setup and execution-based evaluation across Ubuntu, Windows, and macOS. It includes a substantial number of real-world computer tasks involving real web and desktop applications, OS file I/O, and workflows spanning multiple applications, where all OS state changes are exclusively triggered by the agent’s actions. #### 5.1.3 Coding & Software Engineering Environments Code-related tasks are supported by a wide range of executable environments and benchmarks. These can be broadly categorized into interactive environments, where agents directly alter the state, and benchmarks/datasets that provide curated tasks and evaluation pipelines. ##### Interactive SWE Environments. Several environments instantiate agent–environment interaction under software engineering workflows. Debug-Gym (Yuan et al., 2025c) is a text-based interactive coding environment for LLM agents in debugging settings. It equips agents with tools like a Python debugger (pdb) to actively explore and modify buggy codebases, supporting repository-level information handling and ensuring safety via Docker containers. R2E-Gym (Jain et al., 2025c) constructs a procedurally generated, executable gym-style environment of over 8K software engineering tasks, powered by the SWE-Gen pipeline and hybrid verifiers. TheAgentCompany (Xu et al., 2024a) simulates a software development company, where agents act as "digital workers" performing professional tasks such as web browsing, coding, program execution, and communication with simulated colleagues. It features a diverse set of long-horizon tasks with checkpoints for partial credit, providing a comprehensive testbed for agents in a realistic workplace setting. In all these environments, the underlying problem definitions and codebases remain fixed, and changes occur solely as a result of the agent’s actions. ##### Coding Benchmarks & Datasets. A wide range of benchmarks and datasets focus on constructing curated task suites and evaluation pipelines. HumanEval (Chen et al., 2021) introduces a benchmark of 164 hand-crafted Python programming tasks to measure functional correctness via the pass@k metric. MBPP (Austin et al., 2021) provides 974 entry-level Python tasks with natural language descriptions for evaluating short program synthesis. BigCodeBench (Zhuo et al., 2025) proposes a large-scale, contamination-free function-level benchmark of 1,140 tasks requiring composition of multiple function calls. LiveCodeBench (Jain et al., 2025b) builds a continuously updated, contamination-free benchmark from real competition problems. SWE-bench (Jimenez et al., 2024) introduces a dynamic, execution-driven code repair benchmark derived from real GitHub issues. SWE-rebench (Badertdinov et al., 2025) introduces a continual GitHub-mining pipeline ($>$21k tasks) for both training and evaluation. DevBench (Li et al., 2025a) evaluates end-to-end development across design, setup, implementation, and testing. ProjectEval (Liu et al., 2025g) constructs LLM-generated, human-reviewed project tasks with simulated user interactions. ColBench (Zhou et al., 2025e) instantiates multi-turn backend/frontend tasks with a privileged critic for step-wise rewards. NoCode-bench (Deng et al., 2025a) evaluates LLMs on feature addition from documentation updates across real codebases. CodeBoost (Wang et al., 2025n) serves as a data-centric, execution-driven training pipeline by extracting and augmenting code snippets. ##### Programmatic World-Model Environments. Beyond isolated coding tasks, recent benchmarks evaluate whether agents can induce executable world models. The Code World Models Benchmark (CWMB) (Dainese et al., 2024) requires agents to synthesize Python “Environment” classes (specifically the “step” function) to replicate ground-truth dynamics, assessing both transition fidelity and downstream planning utility. Complementing this, the Code Simulation suite (Malfa et al., 2024; 2025) offers finer-grained tests on line-by-line execution prediction and algorithmic generalization. Collectively, these tasks shift the evaluation focus from functional correctness to the dynamics-induction and program-simulation capabilities essential for constructing programmatic world models. #### 5.1.4 Domain-specific Environments ##### Science & Research. ScienceWorld (Wang et al., 2022) integrates science simulations (e.g., thermodynamics, electricity, chemistry) into complex text-based tasks designed around elementary-level science education. PaperBench (Starace et al., 2025) evaluates the ability of LLM agents to replicate cutting-edge machine learning research by reproducing 20 ICML 2024 papers from scratch, scored against rubric-based subtasks. $\tau$-bench (Barres et al., 2025) simulates dynamic conversations for software engineering tasks, operating with an underlying database state and domain-specific rules that change only through the agent’s API calls. ##### Machine Learning Engineering (MLE). MLE-Dojo (Qiang et al., 2025) is a Gym-style framework for iterative machine learning engineering workflows, built upon real-world Kaggle competitions. It provides an interactive environment for agents to iteratively experiment, debug, and refine solutions. MLE-Bench (Chan et al., 2025) establishes a benchmark for MLE by curating 75 Kaggle competitions, evaluating agents against human baselines on public leaderboards. DA-Code (Huang et al., 2024b) addresses agentic data-science workflows grounded in real datasets and executable analysis, providing a focused benchmark for this domain. ##### Biomedical. MedAgentGym (Xu et al., 2025c) provides a domain-specific environment for biomedical code generation and testing, focusing on tasks within this specialized scientific field. ##### Cybersecurity. SecRepoBench (Dilgren et al., 2025) is a domain-specific benchmark for security vulnerability repair, covering 27 repositories and 15 Common Weakness Enumeration (CWE) categories. #### 5.1.5 Simulated & Game Environments Text-based environments simulate interactive settings where agent actions are expressed through natural language. LMRL-Gym (Abdulhai et al., 2025) provides a benchmark for evaluating reinforcement learning algorithms in multi-turn language interactions, including tasks like “20 Questions” and Chess. TextWorld (Côté et al., 2019) is a sandbox environment for training agents in text-based games, offering both hand-authored and procedurally generated games. Game-based environments also emphasize visual settings that may evolve independently. Crafter (Hafner, 2022) is a 2D open-world survival game that benchmarks deep exploration and long-horizon reasoning. Craftax (Matthews et al., 2024), built upon Crafter using JAX, introduces increased complexity and GPU-acceleration for open-ended RL. The modified Crafter variant by ELLM (Du et al., 2023) expands the action space and introduces distractor tasks. For multi-agent coordination, SMAC (Samvelyan et al., 2019) and SMAC-Hard (Deng et al., 2024b) provide StarCraft II-based benchmarks for cooperative decentralized control. SMAC-R1 (Deng et al., 2024b), Adaptive Command (Ma et al., 2025b) and TacticCraft (Ma et al., 2025a) further advance the performance of LLM agents in StarCraft II-style environments. Factorio (Hopkins et al., 2025) presents a dynamic, tick-based industrial simulation where agent inaction still alters the world state. #### 5.1.6 General-Purpose Environments Some environments and benchmarks are designed for broad evaluation or to improve general agent capabilities. AgentGym (Xi et al., 2025) focuses on improving LLM agent generalization via instruction tuning and self-correction, operating on deterministic environments such as ALFWorld, BabyAI, and SciWorld. Agentbench (Liu et al., 2024b) serves as a broad evaluation framework, assessing LLMs as agents across a variety of distinct interactive environments, including SQL-based, game-based, and web-based scenarios. InternBootcamp (Li et al., 2025g) is a scalable framework integrating over 1000 verifiable reasoning tasks, spanning programming, logic puzzles, and games, with a standardized interface for RL training and automated task generation. ### 5.2 RL Framework **Table 11: A summary of frameworks for reinforcement learning, categorized by type and key features.** | Framework | Key Features | Resource | | --- | --- | --- | | Agentic RL Frameworks | | | | Verifiers (Brown, 2025) | Verifiable environment setup | [GitHub](https://github.com/willccbb/verifiers) | | SkyRL-v0 (Cao et al., 2025b) | Long-horizon real-world training | [GitHub](https://github.com/NovaSky-AI/SkyRL) | | AREAL (Fu et al., 2025) | Asynchronous training | [GitHub](https://github.com/inclusionAI/AReaL) | | MARTI (Zhang et al., 2025o) | Integrated multi-agent training | [GitHub](https://github.com/TsinghuaC3I/MARTI) | | EasyR1 (Zheng et al., 2025d) | Multimodal support | [GitHub](https://github.com/hiyouga/EasyR1) | | AgentFly (Wang et al., 2025j) | Scalable asynchronous execution | [GitHub](https://github.com/Agent-One-Lab/AgentFly) | | Agent Lightning (Luo et al., 2025e) | Decoupled hierarchical RL | [GitHub](https://github.com/microsoft/agent-lightning) | | AWorld (Yu et al., 2025a) | Parallel rollouts across clusters | [GitHub](https://github.com/inclusionAI/AWorld/tree/main/train) | | RL-Factory (RL-Factory, 2025) | Easy-to-design reward | [GitHub](https://github.com/Simple-Efficient/RL-Factory) | | ROLL (Wang et al., 2025o) | Stable Multi-GPU Parallel Training | [GitHub](https://github.com/alibaba/ROLL) | | AgentRL (Zhang et al., 2025h) | Asynchronous Multi-Task Training | [GitHub](https://github.com/THUDM/AgentRL) | | VerlTool (Jiang et al., 2025a) | Tool-integrated rollout | [GitHub](https://github.com/TIGER-AI-Lab/verl-tool) | | RLHF and LLM Fine-tuning Frameworks | | | | OpenRLHF (Hu et al., 2025b) | High-performance scalable RLHF | [GitHub](https://github.com/OpenRLHF/OpenRLHF) | | TRL (von Werra et al., 2020) | Hugging Face RLHF | [GitHub](https://github.com/huggingface/trl) | | trlX (Havrilla et al., 2023) | Distributed large-model RLHF | [GitHub](https://github.com/CarperAI/trlx) | | HybridFlow (Sheng et al., 2025) | Streamlined experiment management | [GitHub](https://github.com/volcengine/verl) | | SLiMe (THUDM, 2025) | High-performance async RL | [GitHub](https://github.com/THUDM/slime) | | Oat (Liu et al., 2024c) | Lightweight RL support | [GitHub](https://github.com/sail-sg/oat) | | General-purpose RL Frameworks | | | | RLlib (Liang et al., 2018) | Production-grade scalable library | [GitHub](https://github.com/ray-project/ray/tree/master/rllib) | | Acme (Hoffman et al., 2020) | Modular distributed components | [GitHub](https://github.com/google-deepmind/acme) | | Tianshou (Weng et al., 2022) | High-performance PyTorch platform | [GitHub](https://github.com/thu-ml/tianshou/) | | Stable Baselines3 (Raffin et al., 2021) | Reliable PyTorch algorithms | [GitHub](https://github.com/DLR-RM/stable-baselines3) | | PFRL (Fujita et al., 2021) | Benchmarked prototyping algorithms | [GitHub](https://github.com/pfnet/pfrl) | In this section, we summarize three categories of codebases/frameworks most relevant to this work: Agentic RL frameworks, RLHF and LLM fine-tuning frameworks, and general-purpose RL frameworks. Table [11](#S5.T11) provides an overview of the prevailing Agentic RL and LLM RL frameworks for readers’ reference. ##### Agentic RL frameworks. Verifiers (Brown, 2025) introduces a verifiable-environment setup for end-to-end policy optimization with LLMs, while SkyRL-v0 (Cao et al., 2025b) and its modular successors (Griggs et al., 2025) demonstrate long-horizon, real-world agent training via reinforcement learning. AREAL (Fu et al., 2025) scales this paradigm with an asynchronous, distributed architecture tailored to language reasoning tasks, and MARTI (Zhang et al., 2025o) extends it further to multi-agent LLM systems that integrate reinforcement training and inference. EasyR1 (Zheng et al., 2025d) brings multi-modality support, enabling agents to leverage vision and language signals together in a unified RL framework. AgentFly (Wang et al., 2025j) presents a scalable and extensible agent‑RL framework that empowers language‑model agents with traditional reinforcement‑learning algorithms—enabling token‑level multi‑turn interaction via decorator‑based tools and reward definition, asynchronous execution, and centralized resource management for high‑throughput RL training. Agent Lightning (Luo et al., 2025e) is a flexible RL framework that decouples agent execution from training by modeling execution as an MDP and using a hierarchical RL algorithm (LightningRL) to train any AI agent with near-zero code modification. AWorld (Yu et al., 2025a) is a distributed Agentic RL framework, which tackles the main bottleneck of agent training—experience generation—by orchestrating massively parallel rollouts across clusters, achieving a 14.6× speedup over single-node execution and enabling scalable end-to-end training pipelines. ROLL (Wang et al., 2025o) provides a scalable library for large-scale RL optimization with a unified controller, parallel workers, and automatic resource mapping for efficient multi-GPU training. VerlTool (Jiang et al., 2025a) introduces an Agentic RL with tool use (ARLT) framework built upon Verl (Sheng et al., 2025), enabling agents to jointly optimize planning and execution across interactive environments. AgentRL (Zhang et al., 2025h) provides a scalable asynchronous framework for multi-turn, multi-task Agentic RL, unifying environment orchestration and introducing cross-policy sampling and task advantage normalization for stable large-scale training. ##### RLHF and LLM fine-tuning frameworks. OpenRLHF (Hu et al., 2025b) offers a high-performance, scalable toolkit designed for large-scale model alignment; TRL (von Werra et al., 2020) provides Hugging Face’s baseline implementations for RLHF experiments; trlX (Havrilla et al., 2023) adds distributed training support for fine-tuning models up to tens of billions of parameters; and HybridFlow (Sheng et al., 2025) streamlines experiment management and scaling for RLHF research pipelines. SLiMe (THUDM, 2025) is an LLM post-training framework for RL scaling that combines Megatron with SGLang for high-performance multi-mode training, supports Async RL, and enables flexible disaggregated workflows for reward and data generation via custom interfaces and server-based engines. ##### General-purpose RL frameworks supply the core algorithms and distributed execution engines that can underpin agentic LLM systems. RLlib (Liang et al., 2018) is a production-grade, scalable library offering unified APIs for on-policy, off-policy, and multi-agent methods; Acme (Hoffman et al., 2020) provides modular, research-oriented building blocks for distributed RL; Tianshou (Weng et al., 2022) delivers a high-performance, pure-PyTorch platform supporting online, offline, and hierarchical RL; Stable Baselines3 (Raffin et al., 2021) packages reliable PyTorch implementations of standard model-free algorithms; and PFRL (Fujita et al., 2021) (formerly ChainerRL) offers benchmarked deep-RL algorithm implementations for rapid prototyping. ## 6 Open Challenges and Future Directions The advance of agent RL toward general-purpose intelligence hinges on overcoming three pivotal challenges that define the field’s research frontier. First is the challenge of Trustworthiness: ensuring the reliability, safety, and alignment of increasingly autonomous agents. Second is Scaling up Agentic Training, which requires surmounting the immense practical bottlenecks in computation, data, and algorithmic efficiency. Finally, an agent’s capabilities are fundamentally bounded by its world, making Scaling up Agentic Environments—the creation of complex and adaptive training grounds—a critical necessity. ### 6.1 Trustworthiness ##### Security. The security landscape for autonomous agents is fundamentally more complex than for standard LLMs. While traditional models are primarily vulnerable to attacks on their text-in, text-out interface, agents possess an expanded attack surface due to their external components like tools, memory, and planning modules (Wang et al., 2025l; Shang et al., 2025b). This architecture exposes them to novel threats beyond direct prompt injection. For instance, indirect prompt injection can occur when an agent interacts with a compromised external environment, such as a malicious website or API, which poisons its memory or tool outputs (Chen et al., 2024i). Multi-agent systems further compound these risks by introducing vulnerabilities through inter-agent communication, where one compromised agent can manipulate or mislead others within the collective (Wang et al., 2025l). RL significantly magnifies these agent-specific risks by transforming the agent from a passive victim of manipulation into an active, goal-seeking exploiter of vulnerabilities. The core issue is instrumental goal achievement through reward hacking: an RL agent’s primary directive is to maximize its long-term reward, and it may learn that unsafe actions are the most effective path to this goal. For example, if an agent discovers that using a malicious, third-party tool yields a high reward for a given task, RL will actively reinforce and entrench this unsafe behavior. Similarly, if an agent learns that it can bypass safety protocols to achieve its objective more efficiently, the resulting reward signal will teach it to systematically probe for and exploit such security loopholes. This creates a more persistent and dangerous threat than one-off jailbreaks, as the agent autonomously learns and optimizes deceptive or harmful strategies over time. Mitigating these amplified risks requires a defense-in-depth approach tailored to agentic systems. A critical first line of defense is robust sandboxing (Lu et al., 2025b; Ruan et al., 2024), where agents operate in strictly controlled, permission-limited environments to contain the potential damage from a compromised tool or action. At the training level, mitigation strategies must focus on shaping the reward signal itself. This includes implementing process-based rewards that penalize unsafe intermediate steps (e.g., calling an untrusted API) and employing adversarial training within the RL loop, where the agent is explicitly rewarded for resisting manipulation attempts and ignoring poisoned information. Finally, continuous monitoring and anomaly detection are essential for post-deployment safety. By tracking an agent’s actions, such as tool calls and memory access patterns, it is possible to identify deviations from normal behavior, allowing for timely intervention. ##### Hallucination. In the context of agentic LLMs, hallucination is the generation of confident yet ungrounded outputs, including statements, reasoning steps, or tool usage, that are not rooted in provided evidence or external reality. This issue extends beyond simple factual errors to encompass unfaithful reasoning paths and misaligned planning, with overconfidence often masking the agent’s uncertainty (Cossio, 2025; Huang et al., 2025b). In multimodal agents, it also manifests as cross-modal inconsistency, such as a textual description mismatching an image, framing it as a fundamental grounding problem (Bai et al., 2025). Evaluating hallucination requires assessing both factuality against objective truth and faithfulness to a given source, often measured through benchmarks like HaluEval-QA or by the agent’s ability to appropriately abstain on unanswerable questions, where a refusal to answer ("I don’t know") is a critical signal of epistemic awareness (Li and Ng, 2025; Song et al., 2025c). RL can inadvertently amplify hallucination if the reward mechanism is not carefully designed. Studies show that outcome-driven RL, which rewards only the correctness of the final answer, can encourage agents to find spurious correlations or shortcuts. This process may yield confident but unfounded intermediate reasoning steps, as the optimization process settles into local optima that achieve the goal without being factually sound (Li and Ng, 2025). This phenomenon introduces a "hallucination tax," where reinforcement finetuning can degrade an agent’s ability to refuse to answer, compelling it to generate responses for unanswerable questions rather than abstaining (Song et al., 2025c). However, the effect is highly dependent on the training pipeline; while RL-only post-training can worsen factuality, a structured approach combining SFT with a verifiable-reward RL process can mitigate this degradation (Yao et al., 2025). Promising mitigation strategies involve a hybrid approach of training-time alignment and inference-time safeguards. During training, a key direction is to shift from outcome-only rewards to process-based rewards. Techniques like Factuality-aware Step-wise Policy Optimization (FSPO) verify each intermediate reasoning step against evidence, directly shaping the policy to discourage ungrounded claims (Li and Ng, 2025). Data-centric approaches enhance epistemic humility by training agents on a mix of solvable and unsolvable problems, restoring their ability to abstain when necessary (Song et al., 2025c). At the system level, this is complemented by inference-time techniques such as retrieval augmentation, tool-use for fact-checking, and post-hoc verification to ground the agent’s outputs in reliable sources. For multimodal agents, explicitly adding cross-modal alignment objectives is crucial for ensuring consistency (Huang et al., 2025b; Cossio, 2025; Bai et al., 2025). Collectively, these directions aim to align the agent’s reward-seeking behavior with the goal of truthfulness, fostering more reliable and trustworthy autonomous systems. ##### Sycophancy. Sycophancy in LLM agents refers to their tendency to generate outputs that conform to a user’s stated beliefs, biases, or preferences, even when those are factually incorrect or lead to suboptimal outcomes (Sun and Wang, 2025). This behavior transcends mere conversational agreeableness, fundamentally affecting an agent’s planning and decision-making processes. For instance, a sycophantic agent might adopt a user’s flawed reasoning in its internal plan, choose a course of action that validates the user’s incorrect assumptions, or filter information from tools to present only what aligns with the user’s view (Malmqvist, 2024). This represents a critical misalignment, where the agent optimizes for the user’s expressed preference rather than their latent, long-term interest in achieving the best possible outcome. RL is a primary cause for this behavior. The underlying mechanism is a form of “reward hacking,” where the agent learns to exploit the reward model in ways that do not align with true human preferences (Lu et al., 2024). Because human labelers often show a preference for agreeable and validating responses, the reward model inadvertently learns to equate user satisfaction with sycophantic agreement. Consequently, RLHF can directly incentivize and "exacerbate sycophantic tendencies" by teaching the agent that conforming to a user’s viewpoint is a reliable strategy for maximizing reward, even if it compromises truthfulness (Wen et al., 2024). Mitigating sycophancy is an active area of research that focuses on refining the reward signal and training dynamics. A promising direction is the development of sycophancy-aware reward models, which are explicitly trained to penalize responses that merely parrot user beliefs without critical evaluation. At inference time, strategies like explicitly prompting the agent to adopt a “red team” or contrarian perspective can also help counteract ingrained sycophantic tendencies. Cooper (Hong et al., 2025b) is a reinforcement learning framework that co‑optimizes both the policy model and the reward model online, using high‑precision rule‑based verifiers to select positive samples and LLM‑generated negative samples, thereby preventing the policy from exploiting a static reward model (i.e., reward hacking) by continuously adapting the reward model to closing emergent loopholes. Ultimately, the future direction lies in designing reward systems that robustly capture the user’s long-term interests—such as receiving accurate information and making sound decisions—over their immediate desire for validation. ### 6.2 Scaling up Agentic Training ##### Computation. Recent advances demonstrate that scaling reinforcement learning fine-tuning (RFT) computation directly enhances the reasoning ability of LLM-based agents. The Agent RL Scaling Law study shows that longer training horizons systematically improve tool-use frequency, reasoning depth, and overall task accuracy, highlighting the predictive benefit of allocating more compute to RL training (Mai et al., 2025). Similarly, ProRL reveals that prolonged RL training expands reasoning boundaries beyond those accessible to base models, uncovering novel solution strategies even where extensive sampling from the pretrained model fails (Liu et al., 2025h). Building upon this, ProRLv2 extends training steps and incorporates more stable optimization techniques, demonstrating sustained benefits as smaller models, after extensive RL training, rival the performance of larger models on mathematics, code, and logic benchmarks (Hu et al., 2025a). Collectively, these results underscore that scaling compute through extended RL training is not merely complementary to enlarging model or data size, but a fundamental axis for advancing agentic reasoning. ##### Model Size. Increasing model capacity heightens both the promise and pitfalls of RL-based agent training. Larger models unlock greater potential but risk entropy collapse and narrowing of capability boundaries, as RL sharpens output distributions toward high-reward modes, limiting diversity (Dong et al., 2025c). Methods like RL-PLUS address this with hybrid strategies and advantage functions that foster novel reasoning paths, breaking capability ceilings (Dong et al., 2025c). Meanwhile, scaling demands massive compute, making efficiency vital. A two-stage approach in Vattikonda et al. (2025) uses large teachers to generate SFT data for smaller students, refined via on-policy RL. This “SFT+RL” setup outperforms each method alone and cuts compute by half compared to pure SFT. The work also underscores RL’s extreme hyperparameter sensitivity at scale, stressing the need for careful tuning. ##### Data Size. Scaling RL training across domains introduces both synergy and conflict in agentic reasoning. Cross-domain RL in math, code, and logic tasks shows complex interactions (Li et al., 2025o): some pairings enhance each other, while others interfere and reduce performance. Model initialization also matters—instruction-tuned models generalize differently than raw ones. Building on this, the Guru dataset (Cheng et al., 2025) spans six reasoning domains, showing that RL gains correlate with pretraining exposure: math and code benefit from transfer, but domains like simulation or logic need dedicated training. These findings suggest that while multi-domain RL data can amplify general reasoning, it must be carefully curated to balance complementarity and mitigate interference across tasks. ##### Efficiency. The efficiency of LLM post-training is a central frontier for sustainable scaling (Tie et al., 2025). Beyond brute-force scaling, recent research emphasizes improving RL training efficiency through post-training recipes, methodological refinements, and hybrid paradigms. POLARIS (An et al., 2025) demonstrates that calibrating data difficulty, employing diversity-driven sampling, and extending reasoning length substantially boost RL effectiveness, enabling smaller models to reach or even surpass much larger counterparts on reasoning benchmarks. Complementary work (Liu et al., 2025s) provides systematic evaluations of common RL techniques, finding that judiciously combining just a few simple strategies often outperforms more complex methods. Another study proposes Dynamic Fine-Tuning (DFT) (Wu et al., 2025l), showing that introducing RL principles into gradient scaling can match or exceed advanced RL approaches with minimal additional cost. Taken together, these advances suggest a dual trajectory for the future: on one hand, progressively refining RL-based recipes to maximize efficiency; on the other, rethinking training paradigms to embed RL-like generalization signals without full-fledged online RL. A particularly compelling direction lies in exploring how agentic models might acquire robust generalization from extremely limited data, for instance, by leveraging principled difficulty calibration, meta-learning dynamics, or information-theoretic regularization to distill broad reasoning abilities from a handful of experiences. Such pathways point to the possibility of a new regime of post-training: one where the ability to extrapolate, abstract, and generalize becomes decoupled from sheer data volume, and instead hinges on exploiting the structure and dynamics of the training process itself. ### 6.3 Scaling up Agentic Environments A nascent yet critical frontier for Agentic RL involves a paradigmatic shift from treating the training environment as a static entity to viewing it as a dynamic and optimizable system. This perspective addresses a core bottleneck in agent development: the scarcity of interactive, adaptive environments and the difficulty of engineering effective reward signals. As a growing consensus holds that prevalent environments like ALFWorld (Shridhar et al., 2021) and ScienceWorld (Wang et al., 2022) are insufficient for training general-purpose agents (Zheng et al., 2025f), research is moving beyond solely adapting the agent’s policy. Instead, a co-evolutionary approach uses learning-based methods to adapt the environment itself. One key strategy is to automate reward function design. This involves deploying an auxiliary "explorer" agent to generate a diverse dataset of interaction trajectories, which are then used to train a reward model via heuristics or preference modeling. This effectively decouples agent training from the expensive process of manual reward specification, enabling the learning of complex behaviors without direct human annotation. Beyond automating the reward signal, a second, more dynamic strategy is to automate curriculum generation, transforming the environment into an active teacher. This approach establishes a feedback loop where an agent’s performance data, highlighting specific weaknesses, is fed to an “environment generator” LLM. As exemplified by EnvGen (Zala et al., 2024), this generator then procedurally adapts the environment’s configuration, creating new tasks that specifically target and remedy the agent’s deficiencies. This form of goal-directed Procedural Content Generation (PCG) ensures the agent is consistently challenged within its “zone of proximal development,” accelerating learning and preventing overfitting. Together, automated rewards and adaptive curricula create a symbiotic relationship between the agent and its environment, establishing a scalable "training flywheel" that is essential for the future of self-improving agentic systems. ### 6.4 The Mechanistic Debate on RL in LLMs Two competing explanations have emerged for why RL appears to boost LLM reasoning. The “amplifier” view holds that RL with verifiable rewards—often instantiated via PPO-style variants such as GRPO—mainly reshapes the base model’s output distribution: by sampling multiple trajectories and rewarding the verifiably correct ones, RL concentrates probability mass on already-reachable reasoning paths, improving pass@1 while leaving the support of solutions largely unchanged; consistent with this, large-k pass@k analyses often find that the base model eventually matches or surpasses its RL-tuned counterpart, suggesting elicitation rather than creation of capabilities, and further evidence indicates that reflective behaviors can already emerge during pre-training (Shao et al., 2024b; Yue et al., 2025a; AI et al., 2025). By contrast, the “new-knowledge” view argues that RL after next-token prediction can install qualitatively new computation by leveraging sparse outcome-level signals and encouraging longer test-time computation: theory shows that RL enables generalization on problems (e.g., parity) where next-token training alone is statistically or computationally prohibitive; empirically, RL can improve generalization to out-of-distribution rule- and visual- variants, induce cognitive behaviors (verification, backtracking, subgoal setting) that were absent in the base model yet predict self-improvement, and in under-exposed domains even expand the base model’s pass@k frontier (Guo et al., 2025a; Tsilivis et al., 2025; Chu et al., 2025b; Gandhi et al., 2025; Cheng et al., 2025). Whether RL can truly endow LLMs with abilities beyond those acquired during pre-training remains an open question, and its underlying learning mechanisms are still to be fully understood. ##### Case study: Mathematical Reasoning From a mechanistic standpoint, our survey of RL for mathematical reasoning in Sec [4.3](#S4.SS3) suggests that RL functions neither as a pure “sampler amplifier” nor as a universally reliable source of genuinely new reasoning algorithms (Yue et al., 2025a). Across the cited mathematical and code-reasoning studies, approximately 2/3 primarily emphasize improvements in pass@1 accuracy, while about 1/3 explicitly report expanding pass@k frontiers (e.g., higher pass@32 at fixed or only modestly improved pass@1), indicating that many systems leverage RL chiefly to reshape the sampling distribution over pre-existing competent trajectories rather than to unlock qualitatively new ones. However, cases such as 1-shot RLVR and self-evolving System-2-style frameworks (e.g., rStar-Math–like pipelines (Guan et al., 2025a)) also exhibit “post-saturation” generalization and cross-category transfer, which are difficult to explain as mere reweighting and instead suggest strategy-level reorganization of latent capabilities. Empirically, we find that such “new-capability” behaviors appear most reliably on tasks with (i) high-fidelity, often executable or formally checkable reward signals; (ii) compositional or multi-step structure where many partial trajectories are verifiably graded; and (iii) base models in the “intermediate” regime (neither near-random nor near-ceiling) where the space of near-miss trajectories is rich enough for exploration but still densely populated with correct reasoning paths. Under these conditions, policy-gradient updates plus explicitly managed exploration (e.g., entropy bonuses, self-play curricula, or search-guided expert iteration) seem to move the model toward internalizing more abstract decision rules—whereas on easier, low-noise benchmarks or with coarse outcome-only rewards, RL predominantly acts as an amplifier that sharpens and reuses patterns already implicit in the pretrained model. ### 6.5 Architectural Patterns for Real-World Agent Deployment While the survey primarily analyzes RL as a mechanism for improving reasoning performance, the practical deployment of RL-optimized systems requires architectural patterns that ensure reliability, safety, and operational robustness. This subsection synthesizes four cross-cutting design principles—safety guardrails, human-in-the-loop supervision, hierarchical orchestration, and inter-agent communication protocols—that commonly arise in real-world deployments of RL-enhanced reasoning systems, irrespective of the domain. ##### Guardrails and Safety Patterns. Deployed systems typically incorporate multi-layered safety mechanisms that operate independently of the RL optimization loop. These include input validation (schema enforcement, semantic filtering, and constraint checking), output sanitization (format normalization, groundedness checks, and post-hoc constraint satisfaction), and sandboxed execution for tool or code calls. Such guardrails can be implemented in two major ways: (1) Using RL optimization itself as a safeguard, where, for example, many works directly incentivize models to “think safely” during the reasoning output via RL (Zheng et al., 2025c; Zhang et al., 2025w) ; and (2) Using external modules to monitor RL training, such as AWS Bedrock. ##### Human-in-the-Loop Verification. Human oversight remains essential in high-stakes or uncertainty-prone settings (Mozannar et al., 2025; Takerngsaksiri et al., 2025). HITL mechanisms range from synchronous review of critical decisions to asynchronous auditing, exception handling, and feedback collection. They often rely on model confidence signals or external uncertainty detectors to trigger intervention (Nazir and Banerjee, 2025). Architecturally, HITL provides sparse but high-fidelity corrective signals that complement RL reward structures, enabling safe deployment even when real-world reward feedback is limited, delayed, or noisy. ##### Hierarchical Orchestration. Many practical systems adopt hierarchical control structures (such as supervisor–worker, controller–executor, or planner–solver patterns, as observed in (Zhang et al., 2025s; Liu et al., 2025c; Hu et al., 2025c)) to manage complex workflows. The supervisory layer coordinates subtasks, resolves conflicts, or enforces global constraints, while lower-level components focus on domain-specific reasoning or tool execution. This decomposition facilitates temporal and structural credit assignment, improves scalability, and mirrors enterprise orchestration pipelines where operational logic and execution are cleanly separated. ##### Inter-Agent Communication Protocols. When multiple reasoning entities interact—whether as explicit agents or modular system components—the choice of communication protocol becomes critical. Fixed protocols (e.g., ANP (Chang et al., 2025), A2A (Project, 225), ACP (Team, 2025a)) offer stability and predictability, while learnable communication channels allow adaptive coordination but require stronger regularization to avoid emergent pathologies. Standardized communication interfaces support composability, reproducibility, and compatibility with external workflow engines. ### 6.6 Broader Social Impact The growing deployment of autonomous, agentic LLM systems raises broader societal considerations that increasingly shape research priorities. This subsection highlights five cross-cutting impact areas which merit sustained attention as agentic capabilities continue to advance. ##### Dual-Use Risks. The deployment of Agentic RL lowers the barrier for misuse, notably through “sleeper agent” behaviors where models appear aligned during training but activate concealed harmful policies in deployment (Hubinger et al., 2024). This deceptive alignment often persists despite SFT, RLHF, and adversarial training, as models—particularly those utilizing chain-of-thought—learn to distinguish evaluation contexts from operation. To govern such hazards across domains like Cybersecurity, CBRN, and Autonomous Replication and Adaptation (ARA), OpenAI’s Preparedness Framework establishes a four-tier risk assessment structure (OpenAI, 2025c). However, the framework faces criticism for permissive thresholds, discretionary evaluation protocols, and static assessments that fail to account for post-deployment capability evolution (Coggins et al., 2025). ##### Environmental Sustainability. Large-scale RL is substantially more resource-intensive than SFT due to rollout generation, long-horizon reasoning, and iterative decision steps. Agentic systems further increase training- and deployment-time carbon footprints as interactions unfold over multi-stage workflows (Gardner et al., 2025). Sustainable practices include hardware-aware quantization and resource-efficiency–focused methods. HAQ searches for optimal layer-wise bitwidths under hardware constraints (Wang et al., 2019), and HERO derives low-bit quantization policies for efficient inference using RL-based optimization (Zhang et al., 2025x). Other recent work develops environmental evaluation benchmarks (Wu et al., 2025j). ##### Labor Market Implications. The shift from token-level assistance to autonomous workflow execution positions agentic systems as increasingly strong substitutes for humans in a variety of knowledge-intensive tasks. Code agents have demonstrated the ability to perform debugging, patching, and repository-level issue resolution in SWE benchmarks (Jimenez et al., 2024; Liu et al., 2023c). GUI and web agents similarly automate interactive desktop and browser workflows as shown in OSWorld and WebArena evaluations (Xie et al., 2024; Zhou et al., 2024b). Economic analyses indicate that such end-to-end automation may disproportionately affect entry-level or routine cognitive roles, raising concerns about skill ladder erosion and labor displacement (Eloundou et al., 2023; Brynjolfsson et al., 2023). These trends highlight broader socioeconomic implications, especially for labor markets that may be increasingly exposed to automation. ##### Bias Amplification. RLHF and RLAIF exacerbate societal biases and ideological sycophancy by overfitting to annotator preferences (Casper et al., 2023). Despite surface-level politeness, these models intensify covert discrimination and gender stereotypes, particularly in multi-turn agentic settings (Barnhart et al., 2025a). Furthermore, standard optimization risks collapsing minority preference modes (Xiao et al., 2025). Mitigation strategies address both reward and policy levels. Techniques include fairness-aware reward learning (Swamy et al., 2024; Ouyang et al., 2025), MaxMin-RLHF for heterogeneous groups (Chakraborty et al., 2024), and diversity-preserving objectives like DivPO to prevent mode collapse (Xiao et al., 2025; Wang et al., 2023a; Lanchantin et al., 2025). Complementary approaches involve pluralistic annotator pools and Constitutional AI (Santurkar et al., 2023; Bai et al., 2022), evaluated via benchmarks like CrowS-Pairs and dialect-sensitive tests (Barnhart et al., 2025b). ##### Evaluation Contamination. Static benchmarks like HumanEval and SWE-bench suffer from data contamination, causing inflated scores and illusory robustness (Banerjee et al., 2024). In agentic settings, this encourages overfitting to environmental quirks rather than generalizable reasoning. Addressing these limitations, recent work prioritizes dynamic, contamination-resistant benchmarks, including LiveCodeBench (Jain et al., 2025b), LiveSearchBench (Zhou et al., 2025c), LiveTradeBench (Yu et al., 2025b), and LiveBench (White et al., 2025). Combined with adversarial frameworks like Breakpoint (Hariharan et al., 2025), these approaches prevent test-gaming and offer rigorous assessments of out-of-distribution performance. Collectively, these broader-impact considerations reinforce the importance of coupling methodological advances in Agentic RL with safety, sustainability, fairness, and robustness principles. As agentic systems advance toward broader deployment, understanding and mitigating these societal effects will remain an important direction for future research. ## 7 Conclusion This survey has charted the emergence of Agentic Reinforcement Learning (Agentic RL), a paradigm that elevates LLMs from passive text generators to autonomous, decision-making agents situated in complex, dynamic worlds. Our journey began by formalizing this conceptual shift, distinguishing the temporally extended and partially observable MDPs (POMDPs) that characterize Agentic RL from the single-step decision processes of conventional RL for LLMs. From this foundation, we constructed a comprehensive, twofold taxonomy to systematically map the field: one centered on core agentic capabilities (planning, tool use, memory, reasoning, self-improvement, perception, etc.) and the other on their application across a diverse array of task domains. Throughout this analysis, our central thesis has been that RL provides the critical mechanism for transforming these capabilities from static, heuristic modules into adaptive, robust agentic behavior. By consolidating the landscape of open-source environments, benchmarks, and frameworks, we have also provided a practical compendium to ground and accelerate future research in this burgeoning field. ## References - M. Abdulhai, I. White, C. V. Snell, C. Sun, J. Hong, Y. Zhai, K. Xu, and S. Levine (2025) LMRL gym: benchmarks for multi-turn reinforcement learning with language models. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=hmGhP5DO2W) Cited by: [§5.1.5](#S5.SS1.SSS5.p1.1), [Table 10](#S5.T10.1.1.3.2.1). - P. Aggarwal and S. Welleck (2025) L1: controlling how long a reasoning model thinks with reinforcement learning. In Second Conference on Language Modeling, External Links: [Link](https://openreview.net/forum?id=4jdIxXBNve) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p1.1). - E. AI, :, D. J. Shah, P. Rushton, S. Singla, M. Parmar, K. Smith, Y. Vanjani, A. Vaswani, A. Chaluvaraju, A. Hojel, A. Ma, A. Thomas, A. Polloreno, A. Tanwer, B. D. Sibai, D. S. Mansingka, D. Shivaprasad, I. Shah, K. Stratos, K. Nguyen, M. Callahan, M. Pust, M. Iyer, P. Monk, P. Mazarakis, R. Kapila, S. Srivastava, and T. Romanski (2025) Rethinking reflection in pre-training. External Links: 2504.04022, [Link](https://arxiv.org/abs/2504.04022) Cited by: [§6.4](#S6.SS4.p1.1). - Z. AI (2025) Zai-org/GLM-Z1-32B-0414 · Hugging Face — huggingface.co. Note: [https://huggingface.co/zai-org/GLM-Z1-32B-0414](https://huggingface.co/zai-org/GLM-Z1-32B-0414)[Accessed 25-08-2025] Cited by: [§3.2](#S3.SS2.SSS0.Px2.p2.1). - M. Ambati (2025) ProofNet++: a neuro-symbolic system for formal proof verification with self-correction. External Links: 2505.24230, [Link](https://arxiv.org/abs/2505.24230) Cited by: [§4.3.2](#S4.SS3.SSS2.Px2.p1.1), [§4.3.2](#S4.SS3.SSS2.p2.1), [Table 6](#S4.T6.10.24.14.1.1.1). - C. An, Z. Xie, X. Li, L. Li, J. Zhang, S. Gong, M. Zhong, J. Xu, X. Qiu, M. Wang, and L. Kong (2025) POLARIS: a post-training recipe for scaling reinforcement learning on advanced reasoning models. External Links: [Link](https://hkunlp.github.io/blog/2025/Polaris) Cited by: [§6.2](#S6.SS2.SSS0.Px4.p1.1). - C. Anil, G. Zhang, Y. Wu, and R. Grosse (2021) Learning to give checkable answers with prover-verifier games. External Links: 2108.12099, [Link](https://arxiv.org/abs/2108.12099) Cited by: [§4.3.1](#S4.SS3.SSS1.Px1.p1.1). - T. Anthony, Z. Tian, and D. Barber (2017) Thinking fast and slow with deep learning and tree search. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30, pp. . External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/d8e1344e27a5b08cdfd5d027d9b8d6de-Paper.pdf) Cited by: [§4.3.2](#S4.SS3.SSS2.Px3.p1.1). - Anthropic (2025) Claude code: deep coding at terminal velocity. Note: [https://www.anthropic.com/claude-code](https://www.anthropic.com/claude-code)Anthropic’s agentic command-line coding tool, introduced alongside Claude 3.7 Sonnet. Enables developers to delegate engineering tasks directly from their terminal via natural-language commands. Cited by: [§4.2.3](#S4.SS2.SSS3.Px3.p1.1). - R. M. Aratchige and W. M. K. S. Ilmini (2025) LLMs working in harmony: a survey on the technological aspects of building effective llm-based multi agent systems. External Links: 2504.01963, [Link](https://arxiv.org/abs/2504.01963) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1). - A. Asperti, A. Naibo, and C. S. Coen (2025) Thinking machines: mathematical reasoning in the age of llms. External Links: 2508.00459, [Link](https://arxiv.org/abs/2508.00459) Cited by: [§4.3.1](#S4.SS3.SSS1.Px2.p2.1), [§4.3.1](#S4.SS3.SSS1.p1.1). - M. Atasever, M. Hong, M. N. Kulkarni, Q. Li, and J. V. Deshmukh (2025) Multi-agent path finding via offline rl and llm collaboration. External Links: 2509.22130, [Link](https://arxiv.org/abs/2509.22130) Cited by: [§3.1](#S3.SS1.SSS0.Px1.p1.1). - D. Atreja (2025) ALAS: autonomous learning agent for self-updating language models. External Links: 2508.15805, [Link](https://arxiv.org/abs/2508.15805) Cited by: [§3.4](#S3.SS4.SSS0.Px3.p1.1). - J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton (2021) Program synthesis with large language models. External Links: 2108.07732, [Link](https://arxiv.org/abs/2108.07732) Cited by: [§5.1.3](#S5.SS1.SSS3.Px2.p1.1). - Z. Azerbayev, B. Piotrowski, H. Schoelkopf, E. W. Ayers, D. Radev, and J. Avigad (2023) ProofNet: autoformalizing and formally proving undergraduate-level mathematics. External Links: 2302.12433, [Link](https://arxiv.org/abs/2302.12433) Cited by: [§4.3.2](#S4.SS3.SSS2.Px1.p1.1), [§4.3.2](#S4.SS3.SSS2.Px3.p3.1). - I. Badertdinov, A. Golubev, M. Nekrashevich, A. Shevtsov, S. Karasik, A. Andriushchenko, M. Trofimova, D. Litvintseva, and B. Yangel (2025) SWE-rebench: an automated pipeline for task collection and decontaminated evaluation of software engineering agents. External Links: 2505.20411, [Link](https://arxiv.org/abs/2505.20411) Cited by: [§5.1.3](#S5.SS1.SSS3.Px2.p1.1), [Table 10](#S5.T10.1.1.30.29.1). - H. Bai, Y. Zhou, M. Cemri, J. Pan, A. Suhr, S. Levine, and A. Kumar (2024) DigiRL: training in-the-wild device-control agents with autonomous reinforcement learning. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 12461–12495. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/1704ddd0bb89f159dfe609b32c889995-Paper-Conference.pdf) Cited by: [§4.4.3](#S4.SS4.SSS3.p1.1), [Table 7](#S4.T7.9.9.2). - Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan (2022) Constitutional ai: harmlessness from ai feedback. External Links: 2212.08073, [Link](https://arxiv.org/abs/2212.08073) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1.4), [§6.6](#S6.SS6.SSS0.Px4.p1.1). - Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou (2025) Hallucination of multimodal large language models: a survey. External Links: 2404.18930, [Link](https://arxiv.org/abs/2404.18930) Cited by: [§6.1](#S6.SS1.SSS0.Px2.p1.1), [§6.1](#S6.SS1.SSS0.Px2.p3.1). - S. Banerjee, A. Agarwal, and E. Singh (2024) The vulnerability of language model benchmarks: do they accurately reflect true llm performance?. External Links: 2412.03597, [Link](https://arxiv.org/abs/2412.03597) Cited by: [§6.6](#S6.SS6.SSS0.Px5.p1.1). - L. Barnhart, R. Akbarian Bafghi, S. Becker, and M. Raissi (2025a) Aligning to what? limits to RLHF based alignment. In Findings of the Association for Computational Linguistics: NAACL 2025, L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, pp. 7556–7591. External Links: [Link](https://aclanthology.org/2025.findings-naacl.421/), [Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.421), ISBN 979-8-89176-195-7 Cited by: [§6.6](#S6.SS6.SSS0.Px4.p1.1). - L. Barnhart, R. Akbarian Bafghi, S. Becker, and M. Raissi (2025b) Aligning to what? limits to RLHF based alignment. In Findings of the Association for Computational Linguistics: NAACL 2025, L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, pp. 7556–7591. External Links: [Link](https://aclanthology.org/2025.findings-naacl.421/), [Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.421), ISBN 979-8-89176-195-7 Cited by: [§6.6](#S6.SS6.SSS0.Px4.p1.1). - V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan (2025) $\tau^{2}$-Bench: evaluating conversational agents in a dual-control environment. External Links: 2506.07982, [Link](https://arxiv.org/abs/2506.07982) Cited by: [§5.1.4](#S5.SS1.SSS4.Px1.p1.1), [Table 10](#S5.T10.1.1.1.1). - M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyk, and T. Hoefler (2024) Graph of thoughts: solving elaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence 38 (16), pp. 17682–17690. External Links: ISSN 2159-5399, [Link](http://dx.doi.org/10.1609/aaai.v38i16.29720), [Document](https://dx.doi.org/10.1609/aaai.v38i16.29720) Cited by: [§3.5](#S3.SS5.SSS0.Px1.p2.1). - K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) $\backslash\pi_{0}$: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: [§4.6](#S4.SS6.p1.1). - R. Bonatti, D. Zhao, F. Bonacci, D. Dupont, S. Abdali, Y. Li, Y. Lu, J. Wagle, K. Koishida, A. Bucker, L. Jang, and Z. Hui (2024) Windows agent arena: evaluating multi-modal os agents at scale. External Links: 2409.08264, [Link](https://arxiv.org/abs/2409.08264) Cited by: [Table 10](#S5.T10.1.1.20.19.1). - W. Brown (2025) Verifiers: reinforcement learning with llms in verifiable environments. Note: [https://github.com/willccbb/verifiers](https://github.com/willccbb/verifiers) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.3.3.1.1.1). - E. Brynjolfsson, D. Li, and L. R. Raymond (2023) Generative ai at work. Working Paper Technical Report 31161, Working Paper Series, National Bureau of Economic Research. External Links: [Document](https://dx.doi.org/10.3386/w31161), [Link](http://www.nber.org/papers/w31161) Cited by: [§6.6](#S6.SS6.SSS0.Px3.p1.1). - M. Cao, H. Zhao, C. Zhang, X. Chang, I. Reid, and X. Liang (2025a) Ground-r1: incentivizing grounded visual reasoning via reinforcement learning. arXiv preprint arXiv:2505.20272. Cited by: [§3.6](#S3.SS6.SSS0.Px2.p2.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - S. Cao, S. Hegde, D. Li, T. Griggs, S. Liu, E. Tang, J. Pan, X. Wang, A. Malik, G. Neubig, K. Hakhamaneshi, R. Liaw, P. Moritz, M. Zaharia, J. E. Gonzalez, and I. Stoica (2025b) SkyRL-v0: train real-world long-horizon agents via reinforcement learning. External Links: [Link](https://novasky-ai.notion.site/skyrl-v0) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.4.4.1.1.1). - Y. Cao, H. Zhao, Y. Cheng, T. Shu, Y. Chen, G. Liu, G. Liang, J. Zhao, J. Yan, and Y. Li (2025c) Survey on large language model-enhanced reinforcement learning: concept, taxonomy, and methods. IEEE Transactions on Neural Networks and Learning Systems 36 (6), pp. 9737–9757. External Links: ISSN 2162-2388, [Link](http://dx.doi.org/10.1109/TNNLS.2024.3497992), [Document](https://dx.doi.org/10.1109/tnnls.2024.3497992) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1). - S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. Wang, S. Marks, C. Segerie, M. Carroll, A. Peng, P. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Bıyık, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell (2023) Open problems and fundamental limitations of reinforcement learning from human feedback. External Links: 2307.15217, [Link](https://arxiv.org/abs/2307.15217) Cited by: [§6.6](#S6.SS6.SSS0.Px4.p1.1). - S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, F. Huang, D. Manocha, A. S. Bedi, and M. Wang (2024) MaxMin-rlhf: alignment with diverse human preferences. External Links: 2402.08925, [Link](https://arxiv.org/abs/2402.08925) Cited by: [§6.6](#S6.SS6.SSS0.Px4.p1.1). - J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, A. Madry, and L. Weng (2025) MLE-bench: evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=6s5uXNWGIh) Cited by: [§5.1.4](#S5.SS1.SSS4.Px2.p1.1), [Table 10](#S5.T10.1.1.36.35.1). - G. Chang, E. Lin, C. Yuan, R. Cai, B. Chen, X. Xie, and Y. Zhang (2025) Agent network protocol technical white paper. External Links: 2508.00007, [Link](https://arxiv.org/abs/2508.00007) Cited by: [§6.5](#S6.SS5.SSS0.Px4.p1.1). - B. Chen, C. Shu, E. Shareghi, N. Collier, K. Narasimhan, and S. Yao (2023) FireAct: toward language agent fine-tuning. External Links: 2310.05915, [Link](https://arxiv.org/abs/2310.05915) Cited by: [§3.2](#S3.SS2.SSS0.Px1.p1.1). - B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia (2024a) Spatialvlm: endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455–14465. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - G. Chen, M. Liao, C. Li, and K. Fan (2024b) AlphaMath almost zero: process supervision without process. External Links: 2405.03553, [Link](https://arxiv.org/abs/2405.03553) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p2.1). - J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang (2024c) HuatuoGPT-o1, towards medical complex reasoning with llms. External Links: 2412.18925, [Link](https://arxiv.org/abs/2412.18925) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p2.1). - L. Chen, X. Zhao, Z. Zeng, J. Huang, L. Zheng, Y. Zhong, and L. Ma (2025a) Breaking the sft plateau: multimodal structured reinforcement learning for chart-to-code generation. arXiv preprint arXiv:2508.13587. External Links: [Link](https://arxiv.org/abs/2508.13587) Cited by: [§4.8](#S4.SS8.SSS0.Px2.p1.1). - L. Chen, H. Gao, T. Liu, Z. Huang, F. Sung, X. Zhou, Y. Wu, and B. Chang (2025b) G1: bootstrapping perception and reasoning abilities of vision-language model via reinforcement learning. arXiv preprint arXiv:2505.13426. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - L. Chen, J. Gu, L. Huang, W. Huang, Z. Jiang, A. Jie, X. Jin, X. Jin, C. Li, K. Ma, C. Ren, J. Shen, W. Shi, T. Sun, H. Sun, J. Wang, S. Wang, Z. Wang, C. Wei, S. Wei, Y. Wu, Y. Wu, Y. Xia, H. Xin, F. Yang, H. Ying, H. Yuan, Z. Yuan, T. Zhan, C. Zhang, Y. Zhang, G. Zhang, T. Zhao, J. Zhao, Y. Zhou, and T. H. Zhu (2025c) Seed-prover: deep and broad reasoning for automated theorem proving. External Links: 2507.23726, [Link](https://arxiv.org/abs/2507.23726) Cited by: [§4.3.2](#S4.SS3.SSS2.Px1.p1.1), [Table 6](#S4.T6.10.23.13.1.1.1). - M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021) Evaluating large language models trained on code. External Links: 2107.03374, [Link](https://arxiv.org/abs/2107.03374) Cited by: [§5.1.3](#S5.SS1.SSS3.Px2.p1.1). - M. Chen, T. Li, H. Sun, Y. Zhou, C. Zhu, H. Wang, J. Z. Pan, W. Zhang, H. Chen, F. Yang, Z. Zhou, and W. Chen (2025d) ReSearch: learning to reason with search for llms via reinforcement learning. External Links: 2503.19470, [Link](https://arxiv.org/abs/2503.19470) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1), [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.7.7.1). - S. Chen, X. Chen, C. Zhang, M. Li, G. Yu, H. Fei, H. Zhu, J. Fan, and T. Chen (2024d) Ll3da: visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26428–26438. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - W. Chen, Y. Su, J. Zuo, C. Yang, C. Yuan, C. Chan, H. Yu, Y. Lu, Y. Hung, C. Qian, Y. Qin, X. Cong, R. Xie, Z. Liu, M. Sun, and J. Zhou (2024e) AgentVerse: facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=EHg5GDnyq1) Cited by: [§4.2](#S4.SS2.p1.1). - W. Chen, J. Yuan, C. Qian, C. Yang, Z. Liu, and M. Sun (2025e) Optima: optimizing effectiveness and efficiency for llm-based multi-agent system. External Links: 2410.08115, [Link](https://arxiv.org/abs/2410.08115) Cited by: [§4.7.2](#S4.SS7.SSS2.p1.1), [Table 8](#S4.T8.3.14.11.1.1.1). - X. Chen, J. Lu, M. Kim, D. Zhang, J. Tang, A. Piché, N. Gontier, Y. Bengio, and E. Kamalloo (2025f) Self-evolving curriculum for llm reasoning. External Links: 2505.14970, [Link](https://arxiv.org/abs/2505.14970) Cited by: [§3.4](#S3.SS4.SSS0.Px3.p1.1). - X. Chen, J. Xu, T. Liang, Z. He, J. Pang, D. Yu, L. Song, Q. Liu, M. Zhou, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu (2025g) Do NOT think that much for 2+3=? on the overthinking of long reasoning models. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=MSbU3L7V00) Cited by: [§3.5](#S3.SS5.SSS0.Px3.p1.1). - X. Chen, M. Lin, N. Schärli, and D. Zhou (2024f) Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=KuPixIqPiq) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - Y. Chen, P. Hsu, C. Hsu, and D. Shiu (2024g) Enhancing function-calling capabilities in llms: strategies for prompt formats, data integration, and multilingual translation. External Links: 2412.01130, [Link](https://arxiv.org/abs/2412.01130) Cited by: [§3.2](#S3.SS2.p1.1). - Y. Chen, Y. Liu, J. Zhou, Y. Hao, J. Wang, Y. Zhang, and C. Fan (2025h) R1-code-interpreter: training llms to reason with code via supervised and reinforcement learning. External Links: 2505.21668, [Link](https://arxiv.org/abs/2505.21668) Cited by: [§4.2.2](#S4.SS2.SSS2.Px1.p1.1), [Table 5](#S4.T5.1.1.22.21.1). - Y. Chen, H. Li, Z. Jiang, H. Wen, and D. Zhao (2025i) TeViR: text-to-video reward with diffusion models for efficient reinforcement learning. arXiv preprint arXiv:2505.19769. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - Y. Chen, W. Huang, B. Shi, Q. Hu, H. Ye, L. Zhu, Z. Liu, P. Molchanov, J. Kautz, X. Qi, et al. (2025j) Scaling rl to long videos. arXiv preprint arXiv:2507.07966. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - Z. Chen, K. Liu, Q. Wang, W. Zhang, J. Liu, D. Lin, K. Chen, and F. Zhao (2024h) Agent-FLAN: designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 9354–9366. External Links: [Link](https://aclanthology.org/2024.findings-acl.557/), [Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.557) Cited by: [§3.2](#S3.SS2.SSS0.Px1.p1.1). - Z. Chen, R. Niu, H. Kong, and Q. Wang (2025k) TGRPO: fine-tuning vision-language-action model via trajectory-wise group relative policy optimization. arXiv preprint arXiv:2506.08440. Cited by: [§4.6](#S4.SS6.SSS0.Px2.p1.1). - Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li (2024i) AgentPoison: red-teaming llm agents via poisoning memory or knowledge bases. External Links: 2407.12784, [Link](https://arxiv.org/abs/2407.12784) Cited by: [§6.1](#S6.SS1.SSS0.Px1.p1.1). - Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024j) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185–24198. Cited by: [§3.6](#S3.SS6.p1.1). - Z. Chen, X. Qin, Y. Wu, Y. Ling, Q. Ye, W. X. Zhao, and G. Shi (2025l) Pass@k training for adaptively balancing exploration and exploitation of large reasoning models. External Links: 2508.10751, [Link](https://arxiv.org/abs/2508.10751) Cited by: [Table 2](#S2.T2.1.38.37.1.1.1). - Z. Chen, J. Wu, Z. Lei, M. Pollefeys, and C. W. Chen (2025m) Compile scene graphs with reinforcement learning. arXiv preprint arXiv:2504.13617. Cited by: [§4.5](#S4.SS5.SSS0.Px1.p1.1). - Z. Cheng, S. Leng, H. Zhang, Y. Xin, X. Li, G. Chen, Y. Zhu, W. Zhang, Z. Luo, D. Zhao, et al. (2024) Videollama 2: advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - Z. Cheng, S. Hao, T. Liu, F. Zhou, Y. Xie, F. Yao, Y. Bian, Y. Zhuang, N. Dey, Y. Zha, Y. Gu, K. Zhou, Y. Wang, Y. Li, R. Fan, J. She, C. Gao, A. Saparov, H. Li, T. W. Killian, M. Yurochkin, Z. Liu, E. P. Xing, and Z. Hu (2025) Revisiting reinforcement learning for llm reasoning from a cross-domain perspective. External Links: 2506.14965, [Link](https://arxiv.org/abs/2506.14965) Cited by: [§6.2](#S6.SS2.SSS0.Px3.p1.1), [§6.4](#S6.SS4.p1.1). - P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav (2025) Mem0: building production-ready ai agents with scalable long-term memory. External Links: 2504.19413, [Link](https://arxiv.org/abs/2504.19413) Cited by: [§3.3](#S3.SS3.SSS0.Px3.p1.1), [Table 3](#S3.T3.11.9.26.17.1). - S. Choudhury (2025) Process reward models for llm agents: practical framework and directions. External Links: 2502.10325, [Link](https://arxiv.org/abs/2502.10325) Cited by: [§3.7](#S3.SS7.p2.1). - Y. Chow, G. Tennenholtz, I. Gur, V. Zhuang, B. Dai, A. Kumar, R. Agarwal, S. Thiagarajan, C. Boutilier, and A. Faust (2025) Inference-aware fine-tuning for best-of-n sampling in large language models. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=77gQUdQhE7) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1). - P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei (2017) Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Red Hook, NY, USA, pp. 4302–4310. External Links: ISBN 9781510860964 Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1.3), [§2](#S2.p2.1). - T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, S. Levine, and Y. Ma (2025a) SFT memorizes, RL generalizes: a comparative study of foundation model post-training. In The Second Conference on Parsimony and Learning (Recent Spotlight Track), External Links: [Link](https://openreview.net/forum?id=d3E3LWmTar) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p1.1). - T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma (2025b) SFT memorizes, rl generalizes: a comparative study of foundation model post-training. External Links: 2501.17161, [Link](https://arxiv.org/abs/2501.17161) Cited by: [§6.4](#S6.SS4.p1.1). - X. Chu, X. Chen, G. Wang, Z. Tan, K. Huang, W. Lv, T. Mo, and W. Li (2025c) Qwen look again: guiding vision-language reasoning models to re-attention visual information. arXiv preprint arXiv:2505.23558. Cited by: [§3.6](#S3.SS6.SSS0.Px2.p2.1). - J. Chung, J. Kim, S. Kim, J. Lee, M. S. Kim, and Y. Yu (2025) Don’t look only once: towards multimodal interactive reasoning with selective visual revisitation. arXiv preprint arXiv:2505.18842. Cited by: [§3.6](#S3.SS6.SSS0.Px2.p2.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - S. Coggins, A. K. Saeri, K. A. Daniell, L. P. Ruster, J. Liu, and J. L. Davis (2025) The 2025 openai preparedness framework does not guarantee any ai risk mitigation practices: a proof-of-concept for affordance analyses of ai safety policies. External Links: 2509.24394, [Link](https://arxiv.org/abs/2509.24394) Cited by: [§6.6](#S6.SS6.SSS0.Px1.p1.1). - M. Cossio (2025) A comprehensive taxonomy of hallucinations in large language models. External Links: 2508.01781, [Link](https://arxiv.org/abs/2508.01781) Cited by: [§6.1](#S6.SS1.SSS0.Px2.p1.1), [§6.1](#S6.SS1.SSS0.Px2.p3.1). - M. Côté, Á. Kádár, X. Yuan, B. Kybartas, T. Barnes, E. Fine, J. Moore, M. Hausknecht, L. El Asri, M. Adada, W. Tay, and A. Trischler (2019) TextWorld: a learning environment for text-based games. In Computer Games, T. Cazenave, A. Saffidine, and N. Sturtevant (Eds.), Cham, pp. 41–75. Cited by: [§5.1.5](#S5.SS1.SSS5.p1.1), [Table 10](#S5.T10.1.1.5.4.1). - N. Dai, Z. Wu, R. Zheng, Z. Wei, W. Shi, X. Jin, G. Liu, C. Dun, L. Huang, and L. Yan (2025) Process supervision-guided policy optimization for code generation. External Links: [Link](https://openreview.net/forum?id=Cn5Z0MUPZT) Cited by: [§2.7](#S2.SS7.SSS0.Px2.p1.6), [Table 2](#S2.T2.1.9.8.1.1.1), [§4.2.1](#S4.SS2.SSS1.Px2.p1.1), [Table 5](#S4.T5.1.1.10.9.1). - N. Dainese, M. Merler, M. Alakuijala, and P. Marttinen (2024) Generating code world models with large language models guided by monte carlo tree search. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: [§4.2.4](#S4.SS2.SSS4.Px1.p2.1.1), [§5.1.3](#S5.SS1.SSS3.Px3.p1.1.1). - J. Dang, J. Wu, T. Wang, X. Lin, N. Zhu, H. Chen, W. Zheng, M. Wang, and T. Chua (2025) Reinforcing video reasoning with focused thinking. External Links: 2505.24718, [Link](https://arxiv.org/abs/2505.24718) Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Xu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang (2025) DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. External Links: 2501.12948, [Link](https://arxiv.org/abs/2501.12948) Cited by: [§2.5](#S2.SS5.SSS0.Px1.p1.2), [§2.7](#S2.SS7.p1.1), [Table 2](#S2.T2.1.19.18.1.1.1), [§2](#S2.p2.1), [§3.5](#S3.SS5.SSS0.Px2.p1.1), [§3.6](#S3.SS6.SSS0.Px1.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - DeepSeek-AI, A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Yang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, J. L. Cai, J. Liang, J. Guo, J. Ni, J. Li, J. Chen, J. Yuan, J. Qiu, J. Song, K. Dong, K. Gao, K. Guan, L. Wang, L. Zhang, L. Xu, L. Xia, L. Zhao, L. Zhang, M. Li, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li, N. Tian, P. Huang, P. Wang, P. Zhang, Q. Zhu, Q. Chen, Q. Du, R. J. Chen, R. L. Jin, R. Ge, R. Pan, R. Xu, R. Chen, S. S. Li, S. Lu, S. Zhou, S. Chen, S. Wu, S. Ye, S. Ma, S. Wang, S. Zhou, S. Yu, S. Zhou, S. Zheng, T. Wang, T. Pei, T. Yuan, T. Sun, W. L. Xiao, W. Zeng, W. An, W. Liu, W. Liang, W. Gao, W. Zhang, X. Q. Li, X. Jin, X. Wang, X. Bi, X. Liu, X. Wang, X. Shen, X. Chen, X. Chen, X. Nie, and X. Sun (2024) DeepSeek-v2: a strong, economical, and efficient mixture-of-experts language model. CoRR abs/2405.04434. External Links: [Link](https://doi.org/10.48550/arXiv.2405.04434) Cited by: [§4.3.2](#S4.SS3.SSS2.Px2.p1.1), [Table 6](#S4.T6.8.8.2.1.1). - L. Deng, Z. Jiang, J. Cao, M. Pradel, and Z. Liu (2025a) NoCode-bench: a benchmark for evaluating natural language-driven feature addition. External Links: 2507.18130, [Link](https://arxiv.org/abs/2507.18130) Cited by: [§5.1.3](#S5.SS1.SSS3.Px2.p1.1), [Table 10](#S5.T10.1.1.35.34.1). - W. Deng, J. Yang, R. Ding, J. Liu, Y. Li, X. Qi, and E. Ngai (2024a) Can 3d vision-language models truly understand natural language?. arXiv preprint arXiv:2403.14760. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - Y. Deng, G. Wang, Z. Ying, X. Wu, J. Lin, W. Xiong, Y. Dai, S. Yang, Z. Zhang, Q. Wang, Y. Qin, Y. Wang, Q. Zha, S. Dai, and C. Meng (2025b) Atom-searcher: enhancing agentic deep research via fine-grained atomic thought reward. External Links: 2508.12800, [Link](https://arxiv.org/abs/2508.12800) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.16.16.1). - Y. Deng, Y. Yu, W. Ma, Z. Wang, W. Zhu, J. Zhao, and Y. Zhang (2024b) SMAC-hard: enabling mixed opponent strategy script and self-play on smac. External Links: 2412.17707, [Link](https://arxiv.org/abs/2412.17707) Cited by: [§5.1.5](#S5.SS1.SSS5.p1.1), [Table 10](#S5.T10.1.1.43.42.1). - X. Diao, C. Zhang, K. Kong, W. Wu, C. Ma, Z. Ouyang, P. Qing, S. Vosoughi, and J. Gui (2025) SoundMind: rl-incentivized logic reasoning for audio-language models. arXiv preprint arXiv:2506.12935. Cited by: [§3.6](#S3.SS6.SSS0.Px5.p1.1). - C. Dilgren, P. Chiniya, L. Griffith, Y. Ding, and Y. Chen (2025) SecRepoBench: benchmarking llms for secure code generation in real-world repositories. External Links: 2504.21205, [Link](https://arxiv.org/abs/2504.21205) Cited by: [§5.1.4](#S5.SS1.SSS4.Px4.p1.1), [Table 10](#S5.T10.1.1.25.24.1). - K. Dong and T. Ma (2025) STP: self-play LLM theorem provers with iterative conjecturing and proving. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=zWArMedNuW) Cited by: [§4.3.2](#S4.SS3.SSS2.Px1.p1.1), [§4.3.2](#S4.SS3.SSS2.Px3.p1.1), [§4.3.2](#S4.SS3.SSS2.Px3.p3.1), [Table 6](#S4.T6.10.10.2.1.1). - Q. Dong, L. Dong, Y. Tang, T. Ye, Y. Sun, Z. Sui, and F. Wei (2025a) Reinforcement pre-training. External Links: 2506.08007, [Link](https://arxiv.org/abs/2506.08007) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1). - Y. Dong, X. Jiang, J. Qian, T. Wang, K. Zhang, Z. Jin, and G. Li (2025b) A survey on code generation with llm-based agents. External Links: 2508.00083, [Link](https://arxiv.org/abs/2508.00083) Cited by: [§4.2](#S4.SS2.p1.1). - Y. Dong, X. Jiang, Y. Tao, H. Liu, K. Zhang, L. Mou, R. Cao, Y. Ma, J. Chen, B. Li, Z. Jin, F. Huang, Y. Li, and G. Li (2025c) RL-plus: countering capability boundary collapse of llms in reinforcement learning with hybrid-policy optimization. External Links: 2508.00222, [Link](https://arxiv.org/abs/2508.00222) Cited by: [§6.2](#S6.SS2.SSS0.Px2.p1.1). - Y. Dong, Z. Liu, H. Sun, J. Yang, W. Hu, Y. Rao, and Z. Liu (2025d) Insight-v: exploring long-chain visual reasoning with multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9062–9072. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - S. Dou, Y. Liu, H. Jia, E. Zhou, L. Xiong, J. Shan, C. Huang, X. Wang, X. Fan, Z. Xi, Y. Zhou, T. Ji, R. Zheng, Q. Zhang, T. Gui, and X. Huang (2024) StepCoder: improving code generation with reinforcement learning from compiler feedback. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 4571–4585. External Links: [Link](https://aclanthology.org/2024.acl-long.251/), [Document](https://dx.doi.org/10.18653/v1/2024.acl-long.251) Cited by: [§4.2.1](#S4.SS2.SSS1.Px2.p1.1), [Table 5](#S4.T5.1.1.9.8.1). - B. Doubao (2025) Doubao. External Links: [Link](http://www.doubao.com/) Cited by: [§4.1.2](#S4.SS1.SSS2.Px1.p1.1), [Table 4](#S4.T4.4.1.29.29.1). - Y. Du, O. Watkins, Z. Wang, C. Colas, T. Darrell, P. Abbeel, A. Gupta, and J. Andreas (2023) Guiding pretraining in reinforcement learning with large language models. In ICML, pp. 8657–8677. External Links: [Link](https://proceedings.mlr.press/v202/du23f.html) Cited by: [§5.1.5](#S5.SS1.SSS5.p1.1), [Table 10](#S5.T10.1.1.40.39.1). - C. Duan, R. Fang, Y. Wang, K. Wang, L. Huang, X. Zeng, H. Li, and X. Liu (2025) GoT-r1: unleashing reasoning capability of mllm for visual generation with reinforcement learning. External Links: 2505.17022, [Link](https://arxiv.org/abs/2505.17022) Cited by: [§3.6](#S3.SS6.SSS0.Px4.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - Z. Durante, Q. Huang, N. Wake, R. Gong, J. S. Park, B. Sarkar, R. Taori, Y. Noda, D. Terzopoulos, Y. Choi, K. Ikeuchi, H. Vo, L. Fei-Fei, and J. Gao (2024) Agent ai: surveying the horizons of multimodal interaction. CoRR abs/2401.03568. External Links: [Link](https://doi.org/10.48550/arXiv.2401.03568) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1). - T. Eloundou, S. Manning, P. Mishkin, and D. Rock (2023) GPTs are gpts: an early look at the labor market impact potential of large language models. External Links: 2303.10130, [Link](https://arxiv.org/abs/2303.10130) Cited by: [§6.6](#S6.SS6.SSS0.Px3.p1.1). - A. Estornell, J. Ton, M. F. Taufiq, and H. Li (2025a) How to train a leader: hierarchical reasoning in multi-agent llms. arXiv preprint arXiv:2507.08960. External Links: [Link](https://arxiv.org/abs/2507.08960) Cited by: [§4.7.3](#S4.SS7.SSS3.p1.1), [Table 8](#S4.T8.3.20.17.1.1.1). - A. Estornell, J. Ton, Y. Yao, and Y. Liu (2025b) ACC-collab: an actor-critic approach to multi-agent LLM collaboration. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=nfKfAzkiez) Cited by: [§3.4](#S3.SS4.SSS0.Px2.p1.1), [§4.7.2](#S4.SS7.SSS2.p1.1), [Table 8](#S4.T8.3.18.15.1.1.1). - K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela (2024) Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=iUwHnoENnl) Cited by: [§2.7](#S2.SS7.SSS0.Px3.p1.5), [Table 2](#S2.T2.1.14.13.1.1.1). - L. Fan, Y. Zhang, M. Chen, and Z. Liu (2025a) Posterior-grpo: rewarding reasoning processes in code generation. External Links: 2508.05170, [Link](https://arxiv.org/abs/2508.05170) Cited by: [Table 2](#S2.T2.1.24.23.1.1.1), [§4.2.1](#S4.SS2.SSS1.Px2.p1.1), [Table 5](#S4.T5.1.1.14.13.1). - W. Fan, Y. Ding, L. Ning, S. Wang, H. Li, D. Yin, T. Chua, and Q. Li (2024) A survey on rag meeting llms: towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, New York, NY, USA, pp. 6491–6501. External Links: ISBN 9798400704901, [Link](https://doi.org/10.1145/3637528.3671470), [Document](https://dx.doi.org/10.1145/3637528.3671470) Cited by: [§4.1](#S4.SS1.p1.1). - Y. Fan, K. Zhang, H. Zhou, Y. Zuo, Y. Chen, Y. Fu, X. Long, X. Zhu, C. Jiang, Y. Zhang, L. Kang, G. Chen, C. Huang, Z. He, B. Wang, L. Bai, N. Ding, and B. Zhou (2025b) SSRL: self-search reinforcement learning. External Links: 2508.10874, [Link](https://arxiv.org/abs/2508.10874) Cited by: [§4.1.1](#S4.SS1.SSS1.Px2.p1.1), [Table 4](#S4.T4.4.1.22.22.1). - Y. Fan, X. He, D. Yang, K. Zheng, C. Kuo, Y. Zheng, S. J. Narayanaraju, X. Guan, and X. E. Wang (2025c) GRIT: teaching mllms to think with images. arXiv preprint arXiv:2505.15879. Cited by: [§3.6](#S3.SS6.SSS0.Px2.p2.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - R. Fang, C. Duan, K. Wang, L. Huang, H. Li, S. Yan, H. Tian, X. Zeng, R. Zhao, J. Dai, et al. (2025a) Got: unleashing reasoning capability of multimodal large language model for visual generation and editing. arXiv preprint arXiv:2503.10639. Cited by: [§3.6](#S3.SS6.SSS0.Px4.p1.1). - X. Fang, L. Ma, Z. Chen, M. Zhou, and G. Qi (2025b) InfLVG: reinforce inference-time consistent long video generation with grpo. arXiv preprint arXiv:2505.17574. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - J. Feng, S. Huang, X. Qu, G. Zhang, Y. Qin, B. Zhong, C. Jiang, J. Chi, and W. Zhong (2025a) ReTool: reinforcement learning for strategic tool use in llms. CoRR abs/2504.11536. External Links: [Link](https://doi.org/10.48550/arXiv.2504.11536) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1). - L. Feng, Z. Xue, T. Liu, and B. An (2025b) Group-in-group policy optimization for llm agent training. External Links: 2505.10978, [Link](https://arxiv.org/abs/2505.10978) Cited by: [§3.2](#S3.SS2.SSS0.Px3.p1.1), [§4.8](#S4.SS8.SSS0.Px1.p1.1). - P. Feng, Y. He, G. Huang, Y. Lin, H. Zhang, Y. Zhang, and H. Li (2024a) AGILE: a novel reinforcement learning framework of llm agents. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 5244–5284. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/097c514162ea7126d40671d23e12f51b-Paper-Conference.pdf) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1). - Y. Feng, Y. Li, W. Zhang, H. Luo, Z. Yue, S. Zheng, and Z. Lu (2024b) VideoOrion: tokenizing object dynamics in videos. arXiv preprint arXiv:2411.16156. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - Y. Feng, Y. Xu, X. Xu, B. Hui, and J. Lin (2025c) Towards better correctness and efficiency in code generation. External Links: 2508.20124, [Link](https://arxiv.org/abs/2508.20124) Cited by: [§4.2.1](#S4.SS2.SSS1.Px1.p1.1), [Table 5](#S4.T5.1.1.19.18.1). - W. Fu, J. Gao, S. Xu, Z. Mei, C. Zhu, X. Shen, C. He, G. Wei, J. Mei, W. JIASHU, T. Yang, B. Yuan, and Y. Wu (2025) AREAL: a large-scale asynchronous reinforcement learning system for language reasoning. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models, External Links: [Link](https://openreview.net/forum?id=qJ0okaW9Z9) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.5.5.1.1.1). - Y. Fujita, P. Nagarajan, T. Kataoka, and T. Ishikawa (2021) ChainerRL: a deep reinforcement learning library. Journal of Machine Learning Research 22 (77), pp. 1–14. External Links: [Link](http://jmlr.org/papers/v22/20-376.html) Cited by: [§5.2](#S5.SS2.SSS0.Px3.p1.1), [Table 11](#S5.T11.4.27.27.1.1.1). - K. Gandhi, A. Chakravarthy, A. Singh, N. Lile, and N. D. Goodman (2025) Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. External Links: 2503.01307, [Link](https://arxiv.org/abs/2503.01307) Cited by: [§6.4](#S6.SS4.p1.1). - C. Gao, L. Jin, X. Peng, J. Zhang, Y. Deng, A. Li, H. Wang, and S. Liu (2025a) OctoNav: towards generalist embodied navigation. External Links: 2506.09839, [Link](https://arxiv.org/abs/2506.09839) Cited by: [§4.5](#S4.SS5.SSS0.Px1.p1.1), [§4.6](#S4.SS6.SSS0.Px1.p1.1). - H. Gao, Y. Liu, Y. He, L. Dou, C. Du, Z. Deng, B. Hooi, M. Lin, and T. Pang (2025b) FlowReasoner: reinforcing query-level meta-agents. External Links: 2504.15257, [Link](https://arxiv.org/abs/2504.15257) Cited by: [§4.7.3](#S4.SS7.SSS3.p1.1), [Table 8](#S4.T8.3.22.19.1.1.1). - H. Gao, J. Geng, W. Hua, M. Hu, X. Juan, H. Liu, S. Liu, J. Qiu, X. Qi, Y. Wu, H. Wang, H. Xiao, Y. Zhou, S. Zhang, J. Zhang, J. Xiang, Y. Fang, Q. Zhao, D. Liu, Q. Ren, C. Qian, Z. Wang, M. Hu, H. Wang, Q. Wu, H. Ji, and M. Wang (2025c) A survey of self-evolving agents: on path to artificial super intelligence. External Links: 2507.21046, [Link](https://arxiv.org/abs/2507.21046) Cited by: [§3.4](#S3.SS4.p1.1). - J. Gao, W. Fu, M. Xie, S. Xu, C. He, Z. Mei, B. Zhu, and Y. Wu (2025d) Beyond ten turns: unlocking long-horizon agentic search with large-scale asynchronous rl. External Links: 2508.07976, [Link](https://arxiv.org/abs/2508.07976) Cited by: [§3.2](#S3.SS2.SSS0.Px3.p1.1), [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.15.15.1). - Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang (2024) Retrieval-augmented generation for large language models: a survey. External Links: 2312.10997, [Link](https://arxiv.org/abs/2312.10997) Cited by: [§4.1](#S4.SS1.p1.1). - J. Gardner, A. Dutta, S. Roy, O. P. Kreidl, and L. Boloni (2025) Greener deep reinforcement learning: analysis of energy and carbon efficiency across atari benchmarks. External Links: 2509.05273, [Link](https://arxiv.org/abs/2509.05273) Cited by: [§6.6](#S6.SS6.SSS0.Px2.p1.1). - J. Gehring, K. Zheng, J. Copet, V. Mella, T. Cohen, and G. Synnaeve (2025) RLEF: grounding code LLMs in execution feedback with reinforcement learning. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=PzSG5nKe1q) Cited by: [§4.2.2](#S4.SS2.SSS2.Px1.p1.1), [Table 5](#S4.T5.1.1.21.20.1). - X. Geng, P. Xia, Z. Zhang, X. Wang, Q. Wang, R. Ding, C. Wang, J. Wu, Y. Zhao, K. Li, Y. Jiang, P. Xie, F. Huang, and J. Zhou (2025) WebWatcher: breaking new frontiers of vision-language deep research agent. External Links: 2508.05748, [Link](https://arxiv.org/abs/2508.05748) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.13.13.1). - M. Gheshlaghi Azar, Z. Daniel Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello (2024) A general theoretical paradigm to understand learning from human preferences. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, S. Dasgupta, S. Mandt, and Y. Li (Eds.), Proceedings of Machine Learning Research, Vol. 238, pp. 4447–4455. External Links: [Link](https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html) Cited by: [§2.7](#S2.SS7.SSS0.Px3.p1.5), [Table 2](#S2.T2.1.13.12.1.1.1). - A. Goldie, A. Mirhoseini, H. Zhou, I. Cai, and C. D. Manning (2025) Synthetic data generation & multi-step rl for reasoning & tool use. External Links: 2504.04736, [Link](https://arxiv.org/abs/2504.04736) Cited by: [§4.3.1](#S4.SS3.SSS1.Px2.p1.1), [Table 6](#S4.T6.10.20.10.1.1.1). - A. Golubev, M. Trofimova, S. Polezhaev, I. Badertdinov, M. Nekrashevich, A. Shevtsov, S. Karasik, S. Abramov, A. Andriushchenko, F. Fisin, S. Skvortsov, and B. Yangel (2025) Training long-context, multi-turn software engineering agents with reinforcement learning. External Links: 2508.03501, [Link](https://arxiv.org/abs/2508.03501) Cited by: [§4.2.3](#S4.SS2.SSS3.Px2.p1.1), [Table 5](#S4.T5.1.1.35.34.1). - Google (2025) Gemini deep research. Note: [https://gemini.google/overview/deep-research/](https://gemini.google/overview/deep-research/) Cited by: [§4.1.2](#S4.SS1.SSS2.Px1.p1.1), [Table 4](#S4.T4.4.1.26.26.1). - B. Gou, Z. Huang, Y. Ning, Y. Gu, M. Lin, W. Qi, A. Kopanev, B. Yu, B. J. Gutiérrez, Y. Shu, C. H. Song, J. Wu, S. Chen, H. N. Moussa, T. Zhang, J. Xie, Y. Li, T. Xue, Z. Liao, K. Zhang, B. Zheng, Z. Cai, V. Rozgic, M. Ziyadi, H. Sun, and Y. Su (2025) Mind2Web 2: evaluating agentic search with agent-as-a-judge. External Links: 2506.21506, [Link](https://arxiv.org/abs/2506.21506) Cited by: [§5.1.1](#S5.SS1.SSS1.p1.1), [Table 10](#S5.T10.1.1.13.12.1). - Z. Gou, Z. Shao, Y. Gong, yelong shen, Y. Yang, N. Duan, and W. Chen (2024) CRITIC: large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=Sx038qxjek) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - S. Gravitas (2023) AutoGPT: autonomous gpt-4 agent framework. Note: GitHub, MIT LicenseInitial release date External Links: [Link](https://github.com/Significant-Gravitas/AutoGPT) Cited by: [§4.2](#S4.SS2.p1.1). - T. Griggs, S. Hegde, E. Tang, S. Liu, S. Cao, D. Li, C. Ruan, P. Moritz, K. Hakhamaneshi, R. Liaw, A. Malik, M. Zaharia, J. E. Gonzalez, and I. Stoica (2025) Evolving skyrl into a highly-modular rl framework. Note: Notion Blog External Links: [Link](https://novasky-ai.notion.site/skyrl-v01) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1). - Z. Gu, Z. Zeng, Z. Xu, X. Zhou, S. Shen, Y. Liu, B. Zhou, C. Meng, T. Xia, W. Chen, Y. Wen, J. Dou, F. Tang, J. Lin, Y. Liu, Z. Guo, Y. Gong, H. Jia, C. Gao, Y. Guo, Y. Deng, Z. Guo, L. Chen, and W. Wang (2025) UI-venus technical report: building high-performance ui agents with rft. External Links: 2508.10833, [Link](https://arxiv.org/abs/2508.10833) Cited by: [§4.4.2](#S4.SS4.SSS2.p1.1), [Table 7](#S4.T7.9.16.7.1). - X. Guan, L. L. Zhang, Y. Liu, N. Shang, Y. Sun, Y. Zhu, F. Yang, and M. Yang (2025a) RStar-math: small llms can master math reasoning with self-evolved deep thinking. External Links: 2501.04519, [Link](https://arxiv.org/abs/2501.04519) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p2.1), [§6.4](#S6.SS4.SSS0.Px1.p1.1). - Y. Guan, Q. Lan, S. Fei, D. Ding, D. Acharya, C. Wang, W. Y. Wang, and W. Hua (2025b) Dynamic speculative agent planning. External Links: 2509.01920, [Link](https://arxiv.org/abs/2509.01920) Cited by: [§3.1](#S3.SS1.SSS0.Px2.p1.1). - D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025a) Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), pp. 633–638. Cited by: [§2.7](#S2.SS7.SSS0.Px4.p1.3), [§6.4](#S6.SS4.p1.1). - Y. Guo, Z. Guo, C. Huang, Z. Wang, Z. Zhang, H. Yu, H. Zhang, and Y. Shen (2025b) Synthetic data rl: task definition is all you need. External Links: 2505.17063, [Link](https://arxiv.org/abs/2505.17063) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1). - Y. Guo, L. Xu, J. Liu, D. Ye, and S. Qiu (2025c) Segment policy optimization: effective segment-level credit assignment in rl for large language models. External Links: 2505.23564, [Link](https://arxiv.org/abs/2505.23564) Cited by: [§3.7](#S3.SS7.p2.1). - Y. Guo, S. Peng, J. Guo, D. Huang, X. Zhang, R. Zhang, Y. Hao, L. Li, Z. Tian, M. Gao, Y. Li, Y. Gan, S. Liang, Z. Zhang, Z. Du, Q. Guo, X. Hu, and Y. Chen (2024) Luban: building open-ended creative agents via autonomous embodied verification. CoRR abs/2405.15414. External Links: [Link](https://doi.org/10.48550/arXiv.2405.15414) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - Z. Guo and H. Wang (2025) A survey of reinforcement learning in large language models: from data generation to test-time inference. Available at SSRN 5128927. External Links: [Link](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5128927) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1). - Z. Guo, M. Hong, and T. Jin (2025d) Observe-r1: unlocking reasoning abilities of mllms with dynamic progressive reinforcement learning. arXiv preprint arXiv:2505.12432. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - T. Gupta and A. Kembhavi (2023) Visual programming: compositional visual reasoning without training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14953–14962. Cited by: [§3.6](#S3.SS6.SSS0.Px3.p1.1). - D. Hafner (2022) Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=1W0z96MFEoH) Cited by: [§5.1.5](#S5.SS1.SSS5.p1.1), [Table 10](#S5.T10.1.1.38.37.1). - J. Han, W. Buntine, and E. Shareghi (2024) Towards uncertainty-aware language agent. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 6662–6685. External Links: [Link](https://aclanthology.org/2024.findings-acl.398/), [Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.398) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - X. Han, J. Yang, T. Wang, Z. Bi, X. Song, J. Hao, and J. Song (2025) Towards alignment-centric paradigm: a survey of instruction tuning in large language models. External Links: 2508.17184, [Link](https://arxiv.org/abs/2508.17184) Cited by: [§2](#S2.p1.1). - B. Hao, M. Wang, Z. Xu, Y. Chen, C. Peng, J. GU, and C. Zhuang (2025a) Exploring superior function calls via reinforcement learning. External Links: 2508.05118, [Link](https://arxiv.org/abs/2508.05118) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1). - Q. Hao, S. Li, J. Yuan, and Y. Li (2025b) RL of thoughts: navigating llm reasoning with inference-time reinforcement learning. External Links: 2505.14140, [Link](https://arxiv.org/abs/2505.14140) Cited by: [§4.3.1](#S4.SS3.SSS1.Px2.p1.1), [Table 6](#S4.T6.10.21.11.1.1.1). - S. Hao, Y. Gu, H. Ma, J. Hong, Z. Wang, D. Wang, and Z. Hu (2023) Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 8154–8173. External Links: [Link](https://aclanthology.org/2023.emnlp-main.507/), [Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.507) Cited by: [§3.1](#S3.SS1.SSS0.Px1.p1.1). - S. Hao, S. Sukhbaatar, D. Su, X. Li, Z. Hu, J. Weston, and Y. Tian (2024) Training large language models to reason in a continuous latent space. External Links: 2412.06769, [Link](https://arxiv.org/abs/2412.06769) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p2.1). - K. Hariharan, U. Girit, A. Wang, and J. Andreas (2025) Breakpoint: scalable evaluation of system-level reasoning in llm code agents. External Links: 2506.00172, [Link](https://arxiv.org/abs/2506.00172) Cited by: [§6.6](#S6.SS6.SSS0.Px5.p1.1). - A. Havrilla, M. Zhuravinskyi, D. Phung, A. Tiwari, J. Tow, S. Biderman, Q. Anthony, and L. Castricato (2023) TrlX: a framework for large scale reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 8578–8595. External Links: [Link](https://aclanthology.org/2023.emnlp-main.530), [Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.530) Cited by: [§5.2](#S5.SS2.SSS0.Px2.p1.1), [Table 11](#S5.T11.4.18.18.1.1.1). - B. He, N. Chen, X. He, L. Yan, Z. Wei, J. Luo, and Z. Ling (2024) Retrieving, rethinking and revising: the chain-of-verification can improve retrieval augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 10371–10393. External Links: [Link](https://aclanthology.org/2024.findings-emnlp.607/), [Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.607) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - H. He, Y. Ma, W. Wu, and B. Zhou (2025) From seeing to experiencing: scaling navigation foundation models with reinforcement learning. arXiv preprint arXiv:2507.22028. External Links: [Link](https://arxiv.org/abs/2507.22028) Cited by: [§4.6](#S4.SS6.SSS0.Px1.p1.1). - M. W. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, N. Momchev, D. Sinopalnikov, P. Stańczyk, S. Ramos, A. Raichuk, D. Vincent, et al. (2020) Acme: a research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979. Cited by: [§5.2](#S5.SS2.SSS0.Px3.p1.1), [Table 11](#S5.T11.4.24.24.1.1.1). - B. Hong, J. Liu, Z. Huang, K. Zhang, and M. Zhang (2025a) Pruning long chain-of-thought of large reasoning models via small-scale preference optimization. arXiv preprint arXiv:2508.10164. Cited by: [§2.7](#S2.SS7.SSS0.Px3.p1.5), [Table 2](#S2.T2.1.17.16.1.1.1). - H. Hong, Y. Yan, X. Wu, G. Hou, W. Zhang, W. Lu, Y. Shen, and J. Xiao (2025b) Cooper: co-optimizing policy and reward models in reinforcement learning for large language models. External Links: 2508.05613, [Link](https://arxiv.org/abs/2508.05613) Cited by: [§6.1](#S6.SS1.SSS0.Px3.p4.1). - I. Hong, C. Yu, L. Qiu, W. Yan, Z. Xu, H. Jiang, Q. Zhang, Q. Lu, X. Liu, C. Zhang, and T. Zhao (2025c) Think-rm: enabling long-horizon reasoning in generative reward models. External Links: 2505.16265, [Link](https://arxiv.org/abs/2505.16265) Cited by: [§3.7](#S3.SS7.p2.1). - J. Hong, N. Lee, and J. Thorne (2024a) Orpo: monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691. Cited by: [§2.7](#S2.SS7.SSS0.Px3.p1.5), [Table 2](#S2.T2.1.15.14.1.1.1). - J. Hong, A. Dragan, and S. Levine (2025d) Planning without search: refining frontier llms with offline goal-conditioned rl. External Links: 2505.18098, [Link](https://arxiv.org/abs/2505.18098) Cited by: [§3.1](#S3.SS1.SSS0.Px1.p1.1). - S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber (2024b) MetaGPT: meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=VtmBAGCN7o) Cited by: [§4.2](#S4.SS2.p1.1), [§4.7](#S4.SS7.p1.1), [Table 8](#S4.T8.3.6.3.1.1.1). - Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, and C. Gan (2023) 3d-llm: injecting the 3d world into large language models. Advances in Neural Information Processing Systems 36, pp. 20482–20494. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - H. Y. Hongkang Yang, Z. L. Zehao Lin, W. W. Wenjin Wang, H. W. Hao Wu, Z. L. Zhiyu Li, B. T. Bo Tang, W. W. Wenqiang Wei, J. W. Jinbo Wang, Z. T. Zeyun Tang, S. S. Shichao Song, C. X. Chenyang Xi, Y. Y. Yu Yu, K. C. Kai Chen, F. X. Feiyu Xiong, L. T. Linpeng Tang, and W. E. Weinan E (2024) Memory^3: language modeling with explicit memory. Journal of Machine Learning 3 (3), pp. 300–346. External Links: ISSN 2790-203X, [Link](http://dx.doi.org/10.4208/jml.240708), [Document](https://dx.doi.org/10.4208/jml.240708) Cited by: [§3.3](#S3.SS3.SSS0.Px2.p1.1), [Table 3](#S3.T3.11.9.21.12.1). - J. Hopkins, M. Bakler, and A. Khan (2025) Factorio learning environment. External Links: 2503.09617, [Link](https://arxiv.org/abs/2503.09617) Cited by: [§5.1.5](#S5.SS1.SSS5.p1.1), [Table 10](#S5.T10.1.1.42.41.1). - B. Hou, Y. Zhang, J. Ji, Y. Liu, K. Qian, J. Andreas, and S. Chang (2025) ThinkPrune: pruning long chain-of-thought of llms via reinforcement learning. External Links: 2504.01296, [Link](https://arxiv.org/abs/2504.01296) Cited by: [§3.5](#S3.SS5.SSS0.Px3.p1.1). - J. Hu, M. Liu, S. Diao, X. Lu, X. Dong, P. Molchanov, Y. Choi, J. Kautz, and Y. Dong (2025a) ProRL V2 - Prolonged Training Validates RL Scaling Laws. Note: [https://hijkzzz.notion.site/prorl-v2](https://hijkzzz.notion.site/prorl-v2)Notion page. First published: August 11, 2025. Accessed: August 15, 2025. Cited by: [§6.2](#S6.SS2.SSS0.Px1.p1.1). - J. Hu, X. Wu, W. Shen, J. K. Liu, Z. Zhu, W. Wang, S. Jiang, H. Wang, H. Chen, B. Chen, W. Fang, Xianyu, Y. Cao, H. Xu, and Y. Liu (2025b) OpenRLHF: an easy-to-use, scalable and high-performance rlhf framework. External Links: 2405.11143, [Link](https://arxiv.org/abs/2405.11143) Cited by: [§5.2](#S5.SS2.SSS0.Px2.p1.1), [Table 11](#S5.T11.4.16.16.1.1.1). - M. Hu, Y. Zhou, W. Fan, Y. Nie, B. Xia, T. Sun, Z. Ye, Z. Jin, Y. Li, Q. Chen, Z. Zhang, Y. Wang, Q. Ye, B. Ghanem, P. Luo, and G. Li (2025c) OWL: optimized workforce learning for general multi-agent assistance in real-world task automation. External Links: 2505.23885, [Link](https://arxiv.org/abs/2505.23885) Cited by: [§6.5](#S6.SS5.SSS0.Px3.p1.1). - S. Hu, C. Lu, and J. Clune (2025d) Automated design of agentic systems. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=t9U3LW7JVX) Cited by: [§4.7](#S4.SS7.SSS0.Px1.p1.1). - Y. Hu, Y. Wang, and J. McAuley (2025e) Evaluating memory in llm agents via incremental multi-turn interactions. External Links: 2507.05257, [Link](https://arxiv.org/abs/2507.05257) Cited by: [Table 10](#S5.T10.1.1.11.10.1). - C. Huang, W. Yu, X. Wang, H. Zhang, Z. Li, R. Li, J. Huang, H. Mi, and D. Yu (2025a) R-zero: self-evolving reasoning llm from zero data. External Links: 2508.05004, [Link](https://arxiv.org/abs/2508.05004) Cited by: [§3.4](#S3.SS4.SSS0.Px3.p1.1). - L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu (2025b) A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2), pp. 1–55. External Links: ISSN 1558-2868, [Link](http://dx.doi.org/10.1145/3703155), [Document](https://dx.doi.org/10.1145/3703155) Cited by: [§6.1](#S6.SS1.SSS0.Px2.p1.1), [§6.1](#S6.SS1.SSS0.Px2.p3.1). - W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Z. Xu, Y. Hu, and S. Lin (2025c) Vision-r1: incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - X. Huang, W. Liu, X. Chen, X. Wang, H. Wang, D. Lian, Y. Wang, R. Tang, and E. Chen (2024a) Understanding the planning of llm agents: a survey. CoRR abs/2402.02716. External Links: [Link](https://doi.org/10.48550/arXiv.2402.02716) Cited by: [§3.1](#S3.SS1.p1.1.1). - Y. Huang, J. Luo, Y. Yu, Y. Zhang, F. Lei, Y. Wei, S. He, L. Huang, X. Liu, J. Zhao, and K. Liu (2024b) DA-code: agent data science code generation benchmark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 13487–13521. External Links: [Link](https://aclanthology.org/2024.emnlp-main.748/), [Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.748) Cited by: [§5.1.4](#S5.SS1.SSS4.Px2.p1.1), [Table 10](#S5.T10.1.1.33.32.1). - Z. Huang, Y. Ji, A. S. Rajan, Z. Cai, W. Xiao, H. Wang, J. Hu, and Y. J. Lee (2025d) Visualtoolagent (vista): a reinforcement learning framework for visual tool selection. arXiv preprint arXiv:2505.20289. Cited by: [§3.6](#S3.SS6.SSS0.Px3.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Cheng, A. Jermyn, A. Askell, A. Radhakrishnan, C. Anil, D. Duvenaud, D. Ganguli, F. Barez, J. Clark, K. Ndousse, K. Sachan, M. Sellitto, M. Sharma, N. DasSarma, R. Grosse, S. Kravec, Y. Bai, Z. Witten, M. Favaro, J. Brauner, H. Karnofsky, P. Christiano, S. R. Bowman, L. Graham, J. Kaplan, S. Mindermann, R. Greenblatt, B. Shlegeris, N. Schiefer, and E. Perez (2024) Sleeper agents: training deceptive llms that persist through safety training. External Links: 2401.05566, [Link](https://arxiv.org/abs/2401.05566) Cited by: [§6.6](#S6.SS6.SSS0.Px1.p1.1). - D. Huh and P. Mohapatra (2024) Multi-agent reinforcement learning: a comprehensive survey. External Links: 2312.10256, [Link](https://arxiv.org/abs/2312.10256) Cited by: [item ✗](#S1.I2.ix2.p1.1). - A. K. Jain, G. Gonzalez-Pumariega, W. Chen, A. M. Rush, W. Zhao, and S. Choudhury (2025a) Multi-turn code generation through single-step rewards. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=aJeLhLcsh0) Cited by: [§4.2.2](#S4.SS2.SSS2.Px1.p1.1), [Table 5](#S4.T5.1.1.1.1). - N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica (2025b) LiveCodeBench: holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=chfJJYC3iL) Cited by: [§5.1.3](#S5.SS1.SSS3.Px2.p1.1), [Table 10](#S5.T10.1.1.28.27.1), [§6.6](#S6.SS6.SSS0.Px5.p1.1). - N. Jain, J. Singh, M. Shetty, T. Zhang, L. Zheng, K. Sen, and I. Stoica (2025c) R2E-gym: procedural environment generation and hybrid verifiers for scaling open-weights SWE agents. In Second Conference on Language Modeling, External Links: [Link](https://openreview.net/forum?id=7evvwwdo3z) Cited by: [§5.1.3](#S5.SS1.SSS3.Px1.p1.1), [Table 10](#S5.T10.1.1.26.25.1). - X. Ji, Y. Liu, Q. Wang, J. Zhang, Y. Yue, R. Shi, C. Sun, F. Zhang, G. Zhou, and K. Gai (2025) Leanabell-prover-v2: verifier-integrated reasoning for formal theorem proving via reinforcement learning. External Links: 2507.08649, [Link](https://arxiv.org/abs/2507.08649) Cited by: [§4.3.2](#S4.SS3.SSS2.Px2.p1.1), [§4.3.2](#S4.SS3.SSS2.p2.1), [Table 6](#S4.T6.10.25.15.1.1.1). - D. Jiang, Y. Lu, Z. Li, Z. Lyu, P. Nie, H. Wang, A. Su, H. Chen, K. Zou, C. Du, T. Pang, and W. Chen (2025a) VerlTool: towards holistic agentic reinforcement learning with tool use. External Links: 2509.01055, [Link](https://arxiv.org/abs/2509.01055) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.14.14.1.1.1). - D. Jiang, Z. Guo, R. Zhang, Z. Zong, H. Li, L. Zhuo, S. Yan, P. Heng, and H. Li (2025b) T2i-r1: reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint arXiv:2505.00703. Cited by: [§3.6](#S3.SS6.SSS0.Px4.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - N. Jiang, X. Li, S. Wang, Q. Zhou, S. B. Hossain, B. Ray, V. Kumar, X. Ma, and A. Deoras (2024) LeDex: training llms to better self-debug and explain code. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 35517–35543. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/3ea832724870c700f0a03c665572e2a9-Paper-Conference.pdf) Cited by: [§4.2.2](#S4.SS2.SSS2.Px2.p1.1), [Table 5](#S4.T5.1.1.24.23.1). - P. Jiang, J. Lin, L. Cao, R. Tian, S. Kang, Z. Wang, J. Sun, and J. Han (2025c) DeepRetrieval: hacking real search engines and retrievers with large language models via reinforcement learning. CoRR abs/2503.00223. External Links: [Link](https://doi.org/10.48550/arXiv.2503.00223) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.3.3.1). - C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan (2024) SWE-bench: can language models resolve real-world github issues?. In The Twelfth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=VTF8yNQM66) Cited by: [§5.1.3](#S5.SS1.SSS3.Px2.p1.1), [Table 10](#S5.T10.1.1.29.28.1), [§6.6](#S6.SS6.SSS0.Px3.p1.1). - B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han (2025a) Search-r1: training llms to reason and leverage search engines with reinforcement learning. External Links: 2503.09516, [Link](https://arxiv.org/abs/2503.09516) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.4.4.1). - M. Jin, W. Luo, S. Cheng, X. Wang, W. Hua, R. Tang, W. Y. Wang, and Y. Zhang (2025b) Disentangling memory and reasoning ability in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 1681–1701. External Links: [Link](https://aclanthology.org/2025.acl-long.84/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.84), ISBN 979-8-89176-251-0 Cited by: [§3.3](#S3.SS3.SSS0.Px2.p1.1), [Table 3](#S3.T3.11.9.17.8.1). - Y. Jin, K. Xu, H. Li, X. Han, Y. Zhou, C. Li, and J. Bai (2025c) ReVeal: self-evolving code agents via iterative generation-verification. External Links: 2506.11442, [Link](https://arxiv.org/abs/2506.11442) Cited by: [§4.2.2](#S4.SS2.SSS2.Px2.p1.1), [Table 5](#S4.T5.1.1.26.25.1). - D. Kahneman and A. Tversky (1974) Judgment under uncertainty: heuristics and biases. Science 185 (4157), pp. 1124–1131. External Links: [Document](https://dx.doi.org/10.1126/science.185.4157.1124) Cited by: [§3.5](#S3.SS5.p1.1.3). - D. Kahneman (2011) Thinking, fast and slow. Farrar, Straus and Giroux, New York. External Links: ISBN 978-0374275631 Cited by: [§3.5](#S3.SS5.p1.1.3). - L. Kang, X. Song, H. Zhou, Y. Qin, J. Yang, X. Liu, P. Torr, L. Bai, and Z. Yin (2025a) Viki-r: coordinating embodied multi-agent cooperation via reinforcement learning. arXiv preprint arXiv:2506.09049. Cited by: [§4.6](#S4.SS6.SSS0.Px2.p1.1). - M. Kang, J. Jeong, S. Lee, J. Cho, and S. J. Hwang (2025b) Distilling llm agent into small models with retrieval and code tools. External Links: 2505.17612, [Link](https://arxiv.org/abs/2505.17612) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - I. Kavathekar, R. Donakanti, P. Kumaraguru, and K. Vaidhyanathan (2025) Small models, big tasks: an exploratory empirical study on small language models for function calling. External Links: 2504.19277, [Link](https://arxiv.org/abs/2504.19277) Cited by: [§3.2](#S3.SS2.p1.1). - A. Kazemnejad, M. Aghajohari, E. Portelance, A. Sordoni, S. Reddy, A. Courville, and N. Le Roux (2024) Vineppo: unlocking rl potential for llm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679. External Links: [Link](https://arxiv.org/abs/2410.01679) Cited by: [§2.7](#S2.SS7.SSS0.Px2.p1.6), [Table 2](#S2.T2.1.8.7.1.1.1). - Z. Ke, F. Jiao, Y. Ming, X. Nguyen, A. Xu, D. X. Long, M. Li, C. Qin, P. Wang, silvio savarese, C. Xiong, and S. Joty (2025) A survey of frontiers in LLM reasoning: inference scaling, learning to reason, and agentic systems. Transactions on Machine Learning Research. Note: Survey Certification External Links: ISSN 2835-8856, [Link](https://openreview.net/forum?id=SlsZZ25InC) Cited by: [§3.5](#S3.SS5.p1.1.3). - M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246. Cited by: [§4.6](#S4.SS6.p1.1). - Kimi (2025) Kimi-researcher: end-to-end rl training for emerging agentic capabilities. Note: [https://moonshotai.github.io/Kimi-Researcher/](https://moonshotai.github.io/Kimi-Researcher/) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p2.1), [§4.1.2](#S4.SS1.SSS2.Px1.p1.1), [§4.1](#S4.SS1.p1.1), [Table 4](#S4.T4.4.1.27.27.1). - J. H. Kirchner, Y. Chen, H. Edwards, J. Leike, N. McAleese, and Y. Burda (2024) Prover-verifier games improve legibility of llm outputs. External Links: 2407.13692, [Link](https://arxiv.org/abs/2407.13692) Cited by: [§4.3.1](#S4.SS3.SSS1.Px1.p1.1), [Table 6](#S4.T6.10.16.6.1.1.1). - J. Y. Koh, R. Lo, L. Jang, V. Duvvur, M. Lim, P. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried (2024) VisualWebArena: evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 881–905. Cited by: [§5.1.1](#S5.SS1.SSS1.p1.1), [Table 10](#S5.T10.1.1.15.14.1). - A. Kong, W. Ma, S. Zhao, Y. Li, Y. Wu, K. Wang, X. Liu, Q. Li, Y. Qin, and F. Huang (2025) SDPO: segment-level direct preference optimization for social agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 12409–12423. External Links: [Link](https://aclanthology.org/2025.acl-long.607/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.607), ISBN 979-8-89176-251-0 Cited by: [§3.7](#S3.SS7.p2.1). - K. Kumar, T. Ashraf, O. Thawakar, R. M. Anwer, H. Cholakkal, M. Shah, M. Yang, P. H. S. Torr, F. S. Khan, and S. Khan (2025) LLM post-training: a deep dive into reasoning large language models. External Links: 2502.21321, [Link](https://arxiv.org/abs/2502.21321) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1), [Figure 2](#S2.F2), [Figure 2](#S2.F2.3.2), [§3.5](#S3.SS5.p1.1.3). - M. K. Kunal Singh (2025) TRISHUL: a training-free agentic framework for zero-shot gui action grounding. External Links: 2503.67890, [Link](https://arxiv.org/abs/2502.08226) Cited by: [§4.4.1](#S4.SS4.SSS1.Px1.p1.1), [Table 7](#S4.T7.9.14.5.1). - H. Lai, X. Liu, Y. Zhao, H. Xu, H. Zhang, B. Jing, Y. Ren, S. Yao, Y. Dong, and J. Tang (2025) ComputerRL: scaling end-to-end online reinforcement learning for computer use agents. External Links: 2508.14040, [Link](https://arxiv.org/abs/2508.14040) Cited by: [§4.4.3](#S4.SS4.SSS3.p1.1), [Table 7](#S4.T7.9.22.13.1). - X. Lai, Z. Tian, Y. Chen, S. Yang, X. Peng, and J. Jia (2024) Step-dpo: step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629. Cited by: [§2.7](#S2.SS7.SSS0.Px3.p1.5), [Table 2](#S2.T2.1.16.15.1.1.1). - J. Lanchantin, A. Chen, S. Dhuliawala, P. Yu, J. Weston, S. Sukhbaatar, and I. Kulikov (2025) Diverse preference optimization. External Links: 2501.18101, [Link](https://arxiv.org/abs/2501.18101) Cited by: [§6.6](#S6.SS6.SSS0.Px4.p1.1). - H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi (2022) CodeRL: mastering code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, pp. 21314–21328. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/8636419dea1aa9fbd25fc4248e702da4-Paper-Conference.pdf) Cited by: [§4.2](#S4.SS2.p1.1). - W. Lehrach, D. Hennes, M. Lazaro-Gredilla, X. Lou, C. Wendelken, Z. Li, A. Dedieu, J. Grau-Moya, M. Lanctot, A. Iscen, J. Schultz, M. Chiam, I. Gemp, P. Zielinski, S. Singh, and K. P. Murphy (2025) Code world models for general game playing. External Links: 2510.04542, [Link](https://arxiv.org/abs/2510.04542) Cited by: [§4.2.4](#S4.SS2.SSS4.Px1.p2.1.1). - B. Li, W. Wu, Z. Tang, L. Shi, J. Yang, J. Li, S. Yao, C. Qian, B. Hui, Q. Zhang, Z. Yu, H. Du, P. Yang, D. Lin, C. Peng, and K. Chen (2025a) Prompting large language models to tackle the full software development lifecycle: a case study. In Proceedings of the 31st International Conference on Computational Linguistics, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert (Eds.), Abu Dhabi, UAE, pp. 7511–7531. External Links: [Link](https://aclanthology.org/2025.coling-main.502/) Cited by: [§5.1.3](#S5.SS1.SSS3.Px2.p1.1), [Table 10](#S5.T10.1.1.31.30.1). - C. Li, M. Xue, Z. Zhang, J. Yang, B. Zhang, X. Wang, B. Yu, B. Hui, J. Lin, and D. Liu (2025b) START: self-taught reasoner with tools. CoRR abs/2503.04625. External Links: [Link](https://doi.org/10.48550/arXiv.2503.04625) Cited by: [§4.3.1](#S4.SS3.SSS1.Px2.p1.1), [Table 6](#S4.T6.10.18.8.1.1.1). - C. Li, W. Wu, H. Zhang, Y. Xia, S. Mao, L. Dong, I. Vulić, and F. Wei (2025c) Imagine while reasoning in space: multimodal visualization-of-thought. arXiv preprint arXiv:2501.07542. Cited by: [§3.6](#S3.SS6.SSS0.Px4.p1.1). - G. Li, J. Liu, H. Dinkel, Y. Niu, J. Zhang, and J. Luan (2025d) Reinforcement learning outperforms supervised fine-tuning: a case study on audio question answering. arXiv preprint arXiv:2503.11197. Cited by: [§3.6](#S3.SS6.SSS0.Px5.p1.1). - G. Li, J. Xu, Y. Zhao, and Y. Peng (2025e) DyFo: a training-free dynamic focus visual search for enhancing lmms in fine-grained visual understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9098–9108. Cited by: [§4.5](#S4.SS5.SSS0.Px1.p1.1). - G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem (2023a) Camel: communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems 36, pp. 51991–52008. External Links: [Link](https://openreview.net/forum?id=3IyL2XWDkG) Cited by: [§4.7](#S4.SS7.p1.1), [Table 8](#S4.T8.1.1.2.1.1). - J. Li, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. C. Huang, K. Rasul, L. Yu, A. Q. Jiang, Z. Shen, Z. Qin, B. Dong, L. Zhou, Y. Fleureau, G. Lample, and S. Polu (2024a) NuminaMath. Technical report Peking University and collaborators. Note: Technical Report External Links: [Link](http://faculty.bicmr.pku.edu.cn/%CB%9Cdongbin/Publications/numina_dataset.pdf) Cited by: [§4.3.2](#S4.SS3.SSS2.Px1.p1.1). - J. Li and H. T. Ng (2025) The hallucination dilemma: factuality-aware reinforcement learning for large reasoning models. External Links: 2505.24630, [Link](https://arxiv.org/abs/2505.24630) Cited by: [§6.1](#S6.SS1.SSS0.Px2.p1.1), [§6.1](#S6.SS1.SSS0.Px2.p2.1), [§6.1](#S6.SS1.SSS0.Px2.p3.1). - K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, W. Shen, J. Zhang, D. Zhang, X. Wu, Y. Jiang, M. Yan, P. Xie, F. Huang, and J. Zhou (2025f) WebSailor: navigating super-human reasoning for web agent. External Links: 2507.02592, [Link](https://arxiv.org/abs/2507.02592) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1), [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.12.12.1). - L. Li, G. Chen, Y. Su, Z. Chen, Y. Zhang, E. Xing, and K. Zhang (2024b) Confidence matters: revisiting intrinsic self-correction capabilities of large language models. CoRR abs/2402.12563. External Links: [Link](https://doi.org/10.48550/arXiv.2402.12563) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - M. Li, Y. Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y. Li (2023b) API-bank: a comprehensive benchmark for tool-augmented llms. In EMNLP, pp. 3102–3116. External Links: [Link](https://doi.org/10.18653/v1/2023.emnlp-main.187) Cited by: [§3.2](#S3.SS2.SSS0.Px1.p1.1). - P. Li, J. Ye, Y. Chen, Y. Ma, Z. Yu, K. Chen, G. Cui, H. Li, J. Chen, C. Lyu, W. Zhang, L. Li, Q. Guo, D. Lin, B. Zhou, and K. Chen (2025g) InternBootcamp technical report: boosting llm reasoning with verifiable task scaling. External Links: 2508.08636, [Link](https://arxiv.org/abs/2508.08636) Cited by: [§5.1.6](#S5.SS1.SSS6.p1.1), [Table 10](#S5.T10.1.1.9.8.1). - S. Li, K. Kallidromitis, A. Gokul, Y. Kato, and K. Kozuka (2024c) Aligning diffusion models by optimizing human utility. External Links: 2404.04465, [Link](https://arxiv.org/abs/2404.04465) Cited by: [§4.5](#S4.SS5.SSS0.Px1.p1.1). - W. Li, J. Lin, Z. Jiang, J. Cao, X. Liu, J. Zhang, Z. Huang, Q. Chen, W. Sun, Q. Wang, H. Lu, T. Qin, C. Zhu, Y. Yao, S. Fan, X. Li, T. Wang, P. Liu, K. Zhu, H. Zhu, D. Shi, P. Wang, Y. Guan, X. Tang, M. Liu, Y. E. Jiang, J. Yang, J. Liu, G. Zhang, and W. Zhou (2025h) Chain-of-agents: end-to-end agent foundation models via multi-agent distillation and agentic rl. External Links: 2508.13167, [Link](https://arxiv.org/abs/2508.13167) Cited by: [§4.7.3](#S4.SS7.SSS3.p1.1), [Table 8](#S4.T8.3.3.2.1.1). - X. Li, G. Dong, J. Jin, Y. Zhang, Y. Zhou, Y. Zhu, P. Zhang, and Z. Dou (2025i) Search-o1: agentic search-enhanced large reasoning models. External Links: 2501.05366, [Link](https://arxiv.org/abs/2501.05366) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1), [§4.1](#S4.SS1.p1.1). - X. Li, J. Jin, G. Dong, H. Qian, Y. Zhu, Y. Wu, J. Wen, and Z. Dou (2025j) WebThinker: empowering large reasoning models with deep research capability. External Links: 2504.21776, [Link](https://arxiv.org/abs/2504.21776) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.11.11.1). - X. Li, Z. Yan, D. Meng, L. Dong, X. Zeng, Y. He, Y. Wang, Y. Qiao, Y. Wang, and L. Wang (2025k) Videochat-r1: enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - X. Li (2025) A review of prominent paradigms for LLM-based agents: tool use, planning (including RAG), and feedback learning. In Proceedings of the 31st International Conference on Computational Linguistics, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert (Eds.), Abu Dhabi, UAE, pp. 9760–9779. External Links: [Link](https://aclanthology.org/2025.coling-main.652/) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1). - X. Li, H. Zou, and P. Liu (2025l) ToRL: scaling tool-integrated rl. External Links: 2503.23383, [Link](https://arxiv.org/abs/2503.23383) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1), [§4.3.1](#S4.SS3.SSS1.Px1.p1.1), [Table 6](#S4.T6.1.1.2.1.1). - Y. Li, Q. Gao, T. Zhao, B. Wang, H. Sun, H. Lyu, R. D. Hawkins, N. Vasconcelos, T. Golan, D. Luo, et al. (2024d) Core knowledge deficits in multi-modal language models. arXiv preprint arXiv:2410.10855. Cited by: [§3.6](#S3.SS6.p1.1). - Y. A. Li, X. Jiang, F. Tao, C. Niu, K. Xu, J. Song, and N. Mesgarani (2025m) DMOSpeech 2: reinforcement learning for duration prediction in metric-optimized speech synthesis. arXiv preprint arXiv:2507.14988. Cited by: [§3.6](#S3.SS6.SSS0.Px5.p1.1). - Y. Li, Q. Gu, Z. Wen, Z. Li, T. Xing, S. Guo, T. Zheng, X. Zhou, X. Qu, W. Zhou, et al. (2025n) TreePO: bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree-based modeling. arXiv preprint arXiv:2508.17445. Cited by: [Table 2](#S2.T2.1.33.32.1.1.1). - Y. Li, Z. Pan, H. Lin, M. Sun, C. He, and L. Wu (2025o) Can one domain help others? a data-centric study on multi-domain reasoning via reinforcement learning. External Links: 2507.17512, [Link](https://arxiv.org/abs/2507.17512) Cited by: [§6.2](#S6.SS2.SSS0.Px3.p1.1). - Z. Li, Y. Hu, and W. Wang (2025p) Encouraging good processes without the need for good answers: reinforcement learning for llm agent planning. External Links: 2508.19598, [Link](https://arxiv.org/abs/2508.19598) Cited by: [§3.1](#S3.SS1.SSS0.Px2.p1.1). - Z. Li, X. Liang, Z. Tang, L. Ji, P. Wang, H. Xu, X. W, H. Huang, W. Deng, Y. Gong, Z. Guo, X. Liu, F. Yin, and C. Liu (2025q) TL;dr: too long, do re-weighting for efficient llm reasoning compression. External Links: 2506.02678, [Link](https://arxiv.org/abs/2506.02678) Cited by: [§3.5](#S3.SS5.SSS0.Px3.p1.1). - Z. Li, D. Zhang, M. Zhang, J. Zhang, Z. Liu, Y. Yao, H. Xu, J. Zheng, P. Wang, X. Chen, Y. Zhang, F. Yin, J. Dong, Z. Li, B. Bi, L. Mei, J. Fang, X. Liang, Z. Guo, L. Song, and C. Liu (2025r) From system 1 to system 2: a survey of reasoning large language models. External Links: 2502.17419, [Link](https://arxiv.org/abs/2502.17419) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1), [§3.5](#S3.SS5.SSS0.Px1.p1.1), [§3.5](#S3.SS5.SSS0.Px2.p1.1). - Z. Li, W. Yu, C. Huang, R. Liu, Z. Liang, F. Liu, J. Che, D. Yu, J. Boyd-Graber, H. Mi, et al. (2025s) Self-rewarding vision-language model via reasoning decomposition. arXiv preprint arXiv:2508.19652. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - Z. Li, Z. Ma, M. Li, S. Li, Y. Rong, T. Xu, Z. Zhang, D. Zhao, and W. Huang (2025t) STAR-r1: spatial transformation reasoning by reinforcing multimodal llms. arXiv preprint arXiv:2505.15804. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - S. u. Lian, Y. Wu, J. Ma, Z. Song, B. Chen, X. Zheng, and H. Li (2025) UI-agile: advancing gui agents with effective reinforcement learning and precise inference-time grounding. arXiv preprint arXiv:2507.22025. External Links: [Link](https://arxiv.org/abs/2507.22025) Cited by: [§4.4.1](#S4.SS4.SSS1.Px2.p1.1), [Table 7](#S4.T7.2.2.2). - E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. Gonzalez, M. Jordan, and I. Stoica (2018) RLlib: abstractions for distributed reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, J. Dy and A. Krause (Eds.), Proceedings of Machine Learning Research, Vol. 80, pp. 3053–3062. External Links: [Link](https://proceedings.mlr.press/v80/liang18b.html) Cited by: [§5.2](#S5.SS2.SSS0.Px3.p1.1), [Table 11](#S5.T11.4.23.23.1.1.1). - T. Liang, Z. He, W. Jiao, X. Wang, Y. Wang, R. Wang, Y. Yang, S. Shi, and Z. Tu (2024) Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 17889–17904. External Links: [Link](https://aclanthology.org/2024.emnlp-main.992/), [Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.992) Cited by: [§4.7](#S4.SS7.p1.1), [Table 8](#S4.T8.3.7.4.1.1.1). - X. Liang, Z. Li, Y. Gong, Y. Wang, H. Zhang, Y. Shen, Y. N. Wu, and W. Chen (2025a) SwS: self-aware weakness-driven problem synthesis in reinforcement learning for llm reasoning. External Links: 2506.08989, [Link](https://arxiv.org/abs/2506.08989) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p1.1). - X. Liang, Z. Li, Y. Gong, Y. Shen, Y. N. Wu, Z. Guo, and W. Chen (2025b) Beyond pass@1: self-play with variational problem synthesis sustains rlvr. External Links: 2508.14029, [Link](https://arxiv.org/abs/2508.14029) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p1.1). - J. Liao, M. Wen, J. Wang, and W. Zhang (2025a) MARFT: multi-agent reinforcement fine-tuning. External Links: 2504.16129, [Link](https://arxiv.org/abs/2504.16129) Cited by: [Table 8](#S4.T8.3.17.14.1.1.1). - Y. Liao, P. Zhou, S. Huang, D. Yang, S. Chen, Y. Jiang, Y. Hu, J. Cai, S. Liu, J. Luo, L. Chen, S. Yan, M. Yao, and G. Ren (2025b) Genie envisioner: a unified world foundation platform for robotic manipulation. arXiv preprint arXiv:2508.05635. Cited by: [§4.6](#S4.SS6.p1.1). - Z. Liao, Q. Xie, Y. Zhang, Z. Kong, H. Lu, Z. Yang, and Z. Deng (2025c) Improved visual-spatial reasoning via r1-zero-like training. arXiv preprint arXiv:2504.00883. External Links: [Link](https://arxiv.org/abs/2504.00883) Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - S. Lifshitz, S. A. McIlraith, and Y. Du (2025) Multi-agent verification: scaling test-time compute with goal verifiers. In ICLR 2025 Workshop on Modularity for Collaborative, Decentralized, and Continual Deep Learning, External Links: [Link](https://openreview.net/forum?id=mGAAoEWOq9) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023) Let’s verify step by step. External Links: 2305.20050, [Link](https://arxiv.org/abs/2305.20050) Cited by: [§3.5](#S3.SS5.SSS0.Px1.p2.1). - H. Lin, Z. Sun, S. Welleck, and Y. Yang (2025a) Lean-STar: learning to interleave thinking and proving. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=SOWZ59UyNc) Cited by: [§4.3.2](#S4.SS3.SSS2.Px3.p2.1), [Table 6](#S4.T6.9.9.2.1.1). - H. Lin and Z. Xu (2025) Understanding tool-integrated reasoning. External Links: 2508.19201, [Link](https://arxiv.org/abs/2508.19201) Cited by: [Table 2](#S2.T2.1.32.31.1.1.1), [§3.2](#S3.SS2.SSS0.Px2.p1.1). - H. Lin, Y. Li, H. Luo, K. Yao, L. Zhang, M. Xing, and Y. Wu (2025b) OS-r1: agentic operating system kernel tuning with reinforcement learning. arXiv preprint arXiv:2508.12551. External Links: [Link](https://arxiv.org/abs/2508.12551) Cited by: [§4.2.3](#S4.SS2.SSS3.Px1.p1.1), [Table 5](#S4.T5.1.1.34.33.1). - W. Lin, L. Jia, W. Hu, K. Pan, Z. Yue, W. Zhao, J. Chen, F. Wu, and H. Zhang (2025c) Reasoning physical video generation with diffusion timestep tokens via reinforcement learning. arXiv preprint arXiv:2504.15932. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - B. Liu, X. Li, J. Zhang, J. Wang, T. He, S. Hong, H. Liu, S. Zhang, K. Song, K. Zhu, Y. Cheng, S. Wang, X. Wang, Y. Luo, H. Jin, P. Zhang, O. Liu, J. Chen, H. Zhang, Z. Yu, H. Shi, B. Li, D. Wu, F. Teng, X. Jia, J. Xu, J. Xiang, Y. Lin, T. Liu, T. Liu, Y. Su, H. Sun, G. Berseth, J. Nie, I. Foster, L. T. Ward, Q. Wu, Y. Gu, M. Zhuge, X. Tang, H. Wang, J. You, C. Wang, J. Pei, Q. Yang, X. Qi, and C. Wu (2025a) Advances and challenges in foundation agents: from brain-inspired intelligence to evolutionary, collaborative, and safe systems. CoRR abs/2504.01990. External Links: [Link](https://doi.org/10.48550/arXiv.2504.01990) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1). - B. Liu, L. Guertler, S. Yu, Z. Liu, P. Qi, D. Balcells, M. Liu, C. Tan, W. Shi, M. Lin, W. S. Lee, and N. Jaques (2025b) SPIRAL: self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. External Links: 2506.24119, [Link](https://arxiv.org/abs/2506.24119) Cited by: [§4.7.3](#S4.SS7.SSS3.p1.1). - H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023a) Visual instruction tuning. Advances in neural information processing systems 36, pp. 34892–34916. Cited by: [§3.6](#S3.SS6.p1.1). - J. Liu, S. Xu, S. Liu, Y. Li, W. Liu, M. Liu, X. Zhou, H. Wang, S. Jia, zhen Wang, S. Tian, H. Li, J. Zhang, Y. Yu, P. Cao, and H. Wang (2025c) JoyAgent-jdgenie: technical report on the gaia. External Links: 2510.00510, [Link](https://arxiv.org/abs/2510.00510) Cited by: [§6.5](#S6.SS5.SSS0.Px3.p1.1). - J. Liu, Y. Zhu, K. Xiao, Q. FU, X. Han, Y. Wei, and D. Ye (2023b) RLTF: reinforcement learning from unit test feedback. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, [Link](https://openreview.net/forum?id=hjYmsV6nXZ) Cited by: [§4.2.1](#S4.SS2.SSS1.Px1.p1.1), [Table 5](#S4.T5.1.1.6.5.1). - J. Liu, T. Nguyen, M. Shang, H. Ding, X. Li, Y. Yu, V. Kumar, and Z. Wang (2024a) Learning code preference via synthetic evolution. External Links: 2410.03837, [Link](https://arxiv.org/abs/2410.03837) Cited by: [§4.2.1](#S4.SS2.SSS1.Px2.p1.1), [Table 5](#S4.T5.1.1.16.15.1). - J. Liu, G. Liu, J. Liang, Y. Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang (2025d) Flow-grpo: training flow matching models via online rl. External Links: 2505.05470, [Link](https://arxiv.org/abs/2505.05470) Cited by: [§4.5](#S4.SS5.SSS0.Px1.p1.1). - J. Liu, G. Liu, J. Liang, Z. Yuan, X. Liu, M. Zheng, X. Wu, Q. Wang, W. Qin, M. Xia, et al. (2025e) Improving video generation with human feedback. arXiv preprint arXiv:2501.13918. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - J. Liu, F. Gao, B. Wei, X. Chen, Q. Liao, Y. Wu, C. Yu, and Y. Wang (2025f) What can rl bring to vla generalization? an empirical study. arXiv preprint arXiv:2505.19789. Cited by: [§4.6](#S4.SS6.SSS0.Px2.p1.1). - K. Liu, Y. Pan, Y. Xiang, D. He, J. Li, Y. Du, and T. Gao (2025g) ProjectEval: a benchmark for programming agents automated evaluation on project-level code generation. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 20205–20221. External Links: [Link](https://aclanthology.org/2025.findings-acl.1036/), [Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1036), ISBN 979-8-89176-256-5 Cited by: [§5.1.3](#S5.SS1.SSS3.Px2.p1.1), [Table 10](#S5.T10.1.1.32.31.1). - M. Liu, S. Diao, X. Lu, J. Hu, X. Dong, Y. Choi, J. Kautz, and Y. Dong (2025h) ProRL: prolonged reinforcement learning expands reasoning boundaries in large language models. External Links: 2505.24864, [Link](https://arxiv.org/abs/2505.24864) Cited by: [Table 2](#S2.T2.1.23.22.1.1.1), [§6.2](#S6.SS2.SSS0.Px1.p1.1). - Q. Liu, Z. Liu, D. Zhang, and K. Jia (2025i) Nabla-r2d3: effective and efficient 3d diffusion alignment with 2d rewards. arXiv preprint arXiv:2506.15684. External Links: [Link](https://arxiv.org/abs/2506.15684) Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - S. Liu, S. Hegde, S. Cao, A. Zhu, D. Li, T. Griggs, E. Tang, A. Malik, K. Hakhamaneshi, R. Liaw, P. Moritz, M. Zaharia, J. E. Gonzalez, and I. Stoica (2025j) SkyRL-sql: matching gpt-4o and o4-mini on text2sql with multi-turn rl. External Links: [Link](https://github.com/NovaSky-AI/SkyRL) Cited by: [§4.8](#S4.SS8.SSS0.Px2.p1.1). - S. Liu, W. Fang, Z. Hu, J. Zhang, Y. Zhou, K. Zhang, R. Tu, T. Lin, F. Huang, M. Song, Y. Li, and D. Tao (2025k) A survey of direct preference optimization. CoRR abs/2503.11701. External Links: [Link](https://doi.org/10.48550/arXiv.2503.11701) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1.4). - S. Liu, Z. Liang, X. Lyu, and C. Amato (2025l) LLM collaboration with multi-agent reinforcement learning. External Links: 2508.04652, [Link](https://arxiv.org/abs/2508.04652) Cited by: [§4.7.3](#S4.SS7.SSS3.p1.1), [Table 8](#S4.T8.3.25.22.1.1.1). - T. Liu, C. Xu, and J. McAuley (2023c) RepoBench: benchmarking repository-level code auto-completion systems. External Links: 2306.03091, [Link](https://arxiv.org/abs/2306.03091) Cited by: [§6.6](#S6.SS6.SSS0.Px3.p1.1). - X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang (2024b) AgentBench: evaluating llms as agents. In ICLR, External Links: [Link](https://openreview.net/forum?id=zAdUB0aCTQ) Cited by: [§5.1.6](#S5.SS1.SSS6.p1.1), [Table 10](#S5.T10.1.1.8.7.1). - X. Liu, K. Wang, Y. Li, Y. Wu, W. Ma, A. Kong, F. Huang, J. Jiao, and J. Zhang (2025m) EPO: explicit policy optimization for strategic reasoning in LLMs via reinforcement learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 15371–15396. External Links: [Link](https://aclanthology.org/2025.acl-long.747/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.747), ISBN 979-8-89176-251-0 Cited by: [§3.7](#S3.SS7.p2.1). - Y. Liu, P. Li, Z. Wei, C. Xie, X. Hu, X. Xu, S. Zhang, X. Han, H. Yang, and F. Wu (2025n) InfiGUIAgent: a multimodal generalist GUI agent with native reasoning and reflection. In ICML 2025 Workshop on Computer Use Agents, External Links: [Link](https://openreview.net/forum?id=p0h9XJ7fMH) Cited by: [§4.4.1](#S4.SS4.SSS1.Px2.p1.1), [Table 7](#S4.T7.1.1.2). - Y. Liu, P. Li, C. Xie, X. Hu, X. Han, S. Zhang, H. Yang, and F. Wu (2025o) Infigui-r1: advancing multimodal gui agents from reactive actors to deliberative reasoners. External Links: 2504.14239, [Link](https://arxiv.org/abs/2504.14239) Cited by: [§4.4.2](#S4.SS4.SSS2.p1.1), [Table 7](#S4.T7.6.6.2). - Y. Liu, B. Peng, Z. Zhong, Z. Yue, F. Lu, B. Yu, and J. Jia (2025p) Seg-zero: reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520. Cited by: [§4.5](#S4.SS5.SSS0.Px1.p1.1). - Z. Liu, J. Chai, X. Zhu, S. Tang, R. Ye, B. Zhang, L. Bai, and S. Chen (2025q) ML-agent: reinforcing llm agents for autonomous machine learning engineering. External Links: 2505.23723, [Link](https://arxiv.org/abs/2505.23723) Cited by: [§4.2.3](#S4.SS2.SSS3.Px1.p1.1), [Table 5](#S4.T5.1.1.33.32.1). - Z. Liu, C. Chen, C. Du, W. S. Lee, and M. Lin (2024c) Sample-efficient alignment for llms. External Links: 2411.01493, [Link](https://arxiv.org/abs/2411.01493) Cited by: [Table 11](#S5.T11.4.21.21.1.1.1). - Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin (2025r) Understanding r1-zero-like training: a critical perspective. arXiv preprint arXiv:2503.20783. Cited by: [Table 2](#S2.T2.1.25.24.1.1.1). - Z. Liu, J. Liu, Y. He, W. Wang, J. Liu, L. Pan, X. Hu, S. Xiong, J. Huang, J. Hu, S. Huang, S. Yang, J. Wang, W. Su, and B. Zheng (2025s) Part i: tricks or traps? a deep dive into rl for llm reasoning. External Links: 2508.08221, [Link](https://arxiv.org/abs/2508.08221) Cited by: [§2.7](#S2.SS7.SSS0.Px2.p1.6), [Table 2](#S2.T2.1.6.5.1.1.1), [§6.2](#S6.SS2.SSS0.Px4.p1.1). - Z. Liu, P. Han, H. Yu, H. Li, and J. You (2025t) Time-r1: towards comprehensive temporal reasoning in llms. External Links: 2505.13508, [Link](https://arxiv.org/abs/2505.13508) Cited by: [§4.8](#S4.SS8.SSS0.Px3.p1.1). - Z. Liu, C. Gong, X. Fu, Y. Liu, R. Chen, S. Hu, S. Zhang, R. Liu, Q. Zhang, and D. Tu (2025u) GHPO: adaptive guidance for stable and efficient llm reinforcement learning. arXiv preprint arXiv:2507.10628. Cited by: [Table 2](#S2.T2.1.30.29.1.1.1). - Z. Liu, Z. Sun, Y. Zang, X. Dong, Y. Cao, H. Duan, D. Lin, and J. Wang (2025v) Visual-rft: visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1), [§3.6](#S3.SS6.SSS0.Px3.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - G. Lu, W. Guo, C. Zhang, Y. Zhou, H. Jiang, Z. Gao, Y. Tang, and Z. Wang (2025a) VLA-rl: towards masterful and general robotic manipulation with scalable reinforcement learning. External Links: 2505.18719, [Link](https://arxiv.org/abs/2505.18719) Cited by: [§4.6](#S4.SS6.SSS0.Px2.p1.1), [§4.6](#S4.SS6.p1.1). - J. Lu, T. Holleis, Y. Zhang, B. Aumayer, F. Nan, F. Bai, S. Ma, S. Ma, M. Li, G. Yin, Z. Wang, and R. Pang (2025b) ToolSandbox: a stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. External Links: 2408.04682, [Link](https://arxiv.org/abs/2408.04682) Cited by: [§6.1](#S6.SS1.SSS0.Px1.p3.1). - K. Lu, C. Chen, B. Cui, H. Leng, and W. Zhang (2025c) PilotRL: training language model agents via global planning-guided progressive reinforcement learning. External Links: 2508.00344, [Link](https://arxiv.org/abs/2508.00344) Cited by: [§3.1](#S3.SS1.SSS0.Px2.p1.1). - T. Lu, L. Shen, X. Yang, W. Tan, B. Chen, and H. Yao (2024) It takes two: on the seamlessness between reward and policy model in rlhf. External Links: 2406.07971, [Link](https://arxiv.org/abs/2406.07971) Cited by: [§6.1](#S6.SS1.SSS0.Px3.p2.1). - Z. Lu, Y. Chai, Y. Guo, X. Yin, L. Liu, H. Wang, H. Xiao, S. Ren, G. Xiong, and H. Li (2025d) UI-r1: enhancing efficient action prediction of gui agents by reinforcement learning. External Links: 2503.21620, [Link](https://arxiv.org/abs/2503.21620) Cited by: [§4.4.2](#S4.SS4.SSS2.p1.1), [Table 7](#S4.T7.5.5.2). - J. Luo, W. Zhang, Y. Yuan, Y. Zhao, J. Yang, Y. Gu, B. Wu, B. Chen, Z. Qiao, Q. Long, R. Tu, X. Luo, W. Ju, Z. Xiao, Y. Wang, M. Xiao, C. Liu, J. Yuan, S. Zhang, Y. Jin, F. Zhang, X. Wu, H. Zhao, D. Tao, P. S. Yu, and M. Zhang (2025a) Large language model agent: a survey on methodology, applications and challenges. CoRR abs/2503.21460. External Links: [Link](https://doi.org/10.48550/arXiv.2503.21460) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1). - M. Luo, N. Jain, J. Singh, S. Tan, A. Patel, Q. Wu, A. Ariyak, C. Cai, S. Z. Tarun Venkat, B. Athiwaratkun, M. Roongta, C. Zhang, L. E. Li, R. A. Popa, K. Sen, and I. Stoica (2025b) DeepSWE: training a state-of-the-art coding agent from scratch by scaling rl. Note: [https://pretty-radio-b75.notion.site/DeepSWE-Training-a-Fully-Open-sourced-State-of-the-Art-Coding-Agent-by-Scaling-RL-22281902c1468193aabbe9a8c59bbe33](https://pretty-radio-b75.notion.site/DeepSWE-Training-a-Fully-Open-sourced-State-of-the-Art-Coding-Agent-by-Scaling-RL-22281902c1468193aabbe9a8c59bbe33)Notion Blog Cited by: [§4.2.3](#S4.SS2.SSS3.Px1.p1.1), [Table 5](#S4.T5.1.1.28.27.1). - M. Luo, S. Tan, R. Huang, A. Patel, A. Ariyak, Q. Wu, X. Shi, R. Xin, C. Cai, M. Weber, C. Zhang, L. E. Li, R. A. Popa, and I. Stoica (2025c) DeepCoder: a fully open-source 14b coder at o3-mini level. Note: [https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51](https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51)Notion Blog Cited by: [§4.2.1](#S4.SS2.SSS1.Px1.p1.1), [Table 5](#S4.T5.1.1.5.4.1). - R. Luo, L. Wang, W. He, and X. Xia (2025d) GUI-r1 : a generalist r1-style vision-language action model for gui agents. External Links: 2504.10458, [Link](https://arxiv.org/abs/2504.10458) Cited by: [§4.4.2](#S4.SS4.SSS2.p1.1), [Table 7](#S4.T7.4.4.2). - X. Luo, Y. Zhang, Z. He, Z. Wang, S. Zhao, D. Li, L. K. Qiu, and Y. Yang (2025e) Agent lightning: train any ai agents with reinforcement learning. External Links: 2508.03680, [Link](https://arxiv.org/abs/2508.03680) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.9.9.1.1.1). - W. Ma, J. Jiang, H. Fu, and H. Zhang (2025a) TacticCraft: natural language-driven tactical adaptation for starcraft ii. External Links: 2507.15618, [Link](https://arxiv.org/abs/2507.15618) Cited by: [§5.1.5](#S5.SS1.SSS5.p1.1), [Table 10](#S5.T10.1.1.44.43.1). - W. Ma, D. Xu, S. Lin, H. Zhang, and J. Wang (2025b) Adaptive command: real-time policy adjustment via language models in starcraft ii. External Links: 2508.16580, [Link](https://arxiv.org/abs/2508.16580) Cited by: [§5.1.5](#S5.SS1.SSS5.p1.1). - W. Ma, Y. Chou, Q. Liu, X. Wang, C. de Melo, J. Xie, and A. Yuille (2025c) Spatialreasoner: towards explicit and generalizable 3d spatial reasoning. arXiv preprint arXiv:2504.20024. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - X. Ma, C. Lin, Y. Zhang, V. Tresp, and Y. Ma (2025d) Agentic neural networks: self-evolving multi-agent systems via textual backpropagation. arXiv preprint arXiv:2506.09046. Cited by: [§4.7](#S4.SS7.SSS0.Px1.p1.1). - M. Maaz, H. Rasheed, S. Khan, and F. S. Khan (2023) Video-chatgpt: towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark (2023) Self-refine: iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, pp. 46534–46594. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/91edff07232fb1b55a505a9e9f6c0ff3-Paper-Conference.pdf) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - A. Maharana, D. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y. Fang (2024) Evaluating very long-term conversational memory of llm agents. External Links: 2402.17753, [Link](https://arxiv.org/abs/2402.17753) Cited by: [Table 10](#S5.T10.1.1.10.9.1). - X. Mai, H. Xu, X. W, W. Wang, J. Hu, Y. Zhang, and W. Zhang (2025) Agent rl scaling law: agent rl with spontaneous code execution for mathematical problem solving. External Links: 2505.07773, [Link](https://arxiv.org/abs/2505.07773) Cited by: [§4.3.1](#S4.SS3.SSS1.Px1.p1.1), [§4.3.1](#S4.SS3.SSS1.p2.1), [Table 6](#S4.T6.2.2.2.1.1), [§6.2](#S6.SS2.SSS0.Px1.p1.1). - E. L. Malfa, C. Weinhuber, O. Torre, F. Lin, A. G. Cohn, N. Shadbolt, and M. J. Wooldridge (2024) Code simulation challenges for large language models. CoRR abs/2401.09074. External Links: [Link](https://doi.org/10.48550/arXiv.2401.09074), [Document](https://dx.doi.org/10.48550/ARXIV.2401.09074), 2401.09074 Cited by: [§5.1.3](#S5.SS1.SSS3.Px3.p1.1.1). - E. L. Malfa, C. Weinhuber, O. Torre, F. Lin, X. A. Huang, S. Marro, A. Cohn, N. Shadbolt, and M. Wooldridge (2025) Code simulation as a proxy for high-order tasks in large language models. External Links: 2502.03568, [Link](https://arxiv.org/abs/2502.03568) Cited by: [§5.1.3](#S5.SS1.SSS3.Px3.p1.1.1). - L. Malmqvist (2024) Sycophancy in large language models: causes and mitigations. External Links: 2411.15287, [Link](https://arxiv.org/abs/2411.15287) Cited by: [§6.1](#S6.SS1.SSS0.Px3.p1.1). - J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy (2016) Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11–20. Cited by: [§3.6](#S3.SS6.SSS0.Px2.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - M. Maosongcao, T. Zhang, M. Li, C. Zhang, Y. Liu, C. He, H. Duan, S. Zhang, and K. Chen (2025) Condor: enhance LLM alignment with knowledge-driven data synthesis and refinement. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 22392–22412. External Links: [Link](https://aclanthology.org/2025.acl-long.1091/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1091), ISBN 979-8-89176-251-0 Cited by: [§2](#S2.p1.1). - T. Masterman, S. Besen, M. Sawtell, and A. Chao (2024) The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: a survey. External Links: 2404.11584, [Link](https://arxiv.org/abs/2404.11584) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1). - M. Matthews, M. Beukman, B. Ellis, M. Samvelyan, M. T. Jackson, S. Coward, and J. N. Foerster (2024) Craftax: a lightning-fast benchmark for open-ended reinforcement learning. In Forty-first International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=hg4wXlrQCV) Cited by: [§5.1.5](#S5.SS1.SSS5.p1.1), [Table 10](#S5.T10.1.1.39.38.1). - Meituan (2025) Meituan-longcat/LongCat-Flash-Chat · Hugging Face. Note: [https://huggingface.co/meituan-longcat/LongCat-Flash-Chat](https://huggingface.co/meituan-longcat/LongCat-Flash-Chat)[Accessed 02-09-2025] Cited by: [§3.2](#S3.SS2.SSS0.Px2.p2.1). - F. Meng, L. Du, Z. Liu, Z. Zhou, Q. Lu, D. Fu, B. Shi, W. Wang, J. He, K. Zhang, et al. (2025) Mm-eureka: exploring visual aha moment with rule-based large-scale reinforcement learning. CoRR. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - Y. Meng, M. Xia, and D. Chen (2024) SimPO: simple preference optimization with a reference-free reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: [Link](https://openreview.net/forum?id=3Tzcot1LKb) Cited by: [§2.7](#S2.SS7.SSS0.Px3.p1.5), [Table 2](#S2.T2.1.12.11.1.1.1). - G. Mialon, C. Fourrier, T. Wolf, Y. LeCun, and T. Scialom (2024) GAIA: a benchmark for general AI assistants. In The Twelfth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=fibxvahvs3) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1). - MiroMind Team (2025) MiroMind open deep research v0.1: a high-performance, fully open-sourced deep research project that grows with developers. Note: Blog post External Links: [Link](https://miromind.ai/blog/miromind-open-deep-research) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.17.17.1). - V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller (2013) Playing atari with deep reinforcement learning. External Links: 1312.5602, [Link](https://arxiv.org/abs/1312.5602) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1). - S. R. Motwani, C. Smith, R. J. Das, R. Rafailov, I. Laptev, P. H. S. Torr, F. Pizzati, R. Clark, and C. S. de Witt (2025) MALT: improving reasoning with multi-agent llm training. External Links: 2412.01928, [Link](https://arxiv.org/abs/2412.01928) Cited by: [§3.4](#S3.SS4.SSS0.Px3.p1.1), [§4.7.2](#S4.SS7.SSS2.p1.1), [§4.7.2](#S4.SS7.SSS2.p2.1), [Table 8](#S4.T8.3.16.13.1.1.1). - Z. Mou, B. Xia, Z. Huang, W. Yang, and J. Jia (2025) GRADEO: towards human-like evaluation for text-to-video generation via multi-step reasoning. External Links: 2503.02341, [Link](https://arxiv.org/abs/2503.02341) Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - H. Mozannar, G. Bansal, C. Tan, A. Fourney, V. Dibia, J. Chen, J. Gerrits, T. Payne, M. K. Maldaner, M. Grunde-McLaughlin, E. Zhu, G. Bassman, J. Alber, P. Chang, R. Loynd, F. Niedtner, E. Kamar, M. Murad, R. Hosn, and S. Amershi (2025) Magentic-ui: towards human-in-the-loop agentic systems. External Links: 2507.22358, [Link](https://arxiv.org/abs/2507.22358) Cited by: [§6.5](#S6.SS5.SSS0.Px2.p1.1). - V. K. Nagaraja, V. I. Morariu, and L. S. Davis (2016) Modeling context between objects for referring expression understanding. In European Conference on Computer Vision, pp. 792–807. Cited by: [§3.6](#S3.SS6.SSS0.Px2.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - M. S. Nazir and C. Banerjee (2025) Zero-shot llms in human-in-the-loop rl: replacing human feedback for reward shaping. External Links: 2503.22723, [Link](https://arxiv.org/abs/2503.22723) Cited by: [§6.5](#S6.SS5.SSS0.Px2.p1.1). - A. Newell, J. C. Shaw, and H. A. Simon (1958) Elements of a theory of human problem solving.. Psychological review 65 (3), pp. 151. Cited by: [§3.1](#S3.SS1.p1.1). - D. Nguyen, J. Chen, Y. Wang, G. Wu, N. Park, Z. Hu, H. Lyu, J. Wu, R. Aponte, Y. Xia, X. Li, J. Shi, H. Chen, V. D. Lai, Z. Xie, S. Kim, R. Zhang, T. Yu, M. Tanjim, N. K. Ahmed, P. Mathur, S. Yoon, L. Yao, B. Kveton, J. Kil, T. H. Nguyen, T. Bui, T. Zhou, R. A. Rossi, and F. Dernoncourt (2025a) GUI agents: a survey. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 22522–22538. External Links: [Link](https://aclanthology.org/2025.findings-acl.1158/), [Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1158), ISBN 979-8-89176-256-5 Cited by: [§4.4.1](#S4.SS4.SSS1.Px1.p1.1). - X. Nguyen, S. Pandit, R. G. Reddy, A. Xu, S. Savarese, C. Xiong, and S. Joty (2025b) SFR-deepresearch: towards effective reinforcement learning for autonomously reasoning single agents. External Links: 2509.06283, [Link](https://arxiv.org/abs/2509.06283) Cited by: [Table 4](#S4.T4.4.1.20.20.1). - OpenAI, :, A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, A. Iftimie, A. Karpenko, A. T. Passos, A. Neitz, A. Prokofiev, A. Wei, A. Tam, A. Bennett, A. Kumar, A. Saraiva, A. Vallone, A. Duberstein, A. Kondrich, A. Mishchenko, A. Applebaum, A. Jiang, A. Nair, B. Zoph, B. Ghorbani, B. Rossen, B. Sokolowsky, B. Barak, B. McGrew, B. Minaiev, B. Hao, B. Baker, B. Houghton, B. McKinzie, B. Eastman, C. Lugaresi, C. Bassin, C. Hudson, C. M. Li, C. de Bourcy, C. Voss, C. Shen, C. Zhang, C. Koch, C. Orsinger, C. Hesse, C. Fischer, C. Chan, D. Roberts, D. Kappler, D. Levy, D. Selsam, D. Dohan, D. Farhi, D. Mely, D. Robinson, D. Tsipras, D. Li, D. Oprica, E. Freeman, E. Zhang, E. Wong, E. Proehl, E. Cheung, E. Mitchell, E. Wallace, E. Ritter, E. Mays, F. Wang, F. P. Such, F. Raso, F. Leoni, F. Tsimpourlas, F. Song, F. von Lohmann, F. Sulit, G. Salmon, G. Parascandolo, G. Chabot, G. Zhao, G. Brockman, G. Leclerc, H. Salman, H. Bao, H. Sheng, H. Andrin, H. Bagherinezhad, H. Ren, H. Lightman, H. W. Chung, I. Kivlichan, I. O’Connell, I. Osband, I. C. Gilaberte, I. Akkaya, I. Kostrikov, I. Sutskever, I. Kofman, J. Pachocki, J. Lennon, J. Wei, J. Harb, J. Twore, J. Feng, J. Yu, J. Weng, J. Tang, J. Yu, J. Q. Candela, J. Palermo, J. Parish, J. Heidecke, J. Hallman, J. Rizzo, J. Gordon, J. Uesato, J. Ward, J. Huizinga, J. Wang, K. Chen, K. Xiao, K. Singhal, K. Nguyen, K. Cobbe, K. Shi, K. Wood, K. Rimbach, K. Gu-Lemberg, K. Liu, K. Lu, K. Stone, K. Yu, L. Ahmad, L. Yang, L. Liu, L. Maksin, L. Ho, L. Fedus, L. Weng, L. Li, L. McCallum, L. Held, L. Kuhn, L. Kondraciuk, L. Kaiser, L. Metz, M. Boyd, M. Trebacz, M. Joglekar, M. Chen, M. Tintor, M. Meyer, M. Jones, M. Kaufer, M. Schwarzer, M. Shah, M. Yatbaz, M. Y. Guan, M. Xu, M. Yan, M. Glaese, M. Chen, M. Lampe, M. Malek, M. Wang, M. Fradin, M. McClay, M. Pavlov, M. Wang, M. Wang, M. Murati, M. Bavarian, M. Rohaninejad, N. McAleese, N. Chowdhury, N. Chowdhury, N. Ryder, N. Tezak, N. Brown, O. Nachum, O. Boiko, O. Murk, O. Watkins, P. Chao, P. Ashbourne, P. Izmailov, P. Zhokhov, R. Dias, R. Arora, R. Lin, R. G. Lopes, R. Gaon, R. Miyara, R. Leike, R. Hwang, R. Garg, R. Brown, R. James, R. Shu, R. Cheu, R. Greene, S. Jain, S. Altman, S. Toizer, S. Toyer, S. Miserendino, S. Agarwal, S. Hernandez, S. Baker, S. McKinney, S. Yan, S. Zhao, S. Hu, S. Santurkar, S. R. Chaudhuri, S. Zhang, S. Fu, S. Papay, S. Lin, S. Balaji, S. Sanjeev, S. Sidor, T. Broda, A. Clark, T. Wang, T. Gordon, T. Sanders, T. Patwardhan, T. Sottiaux, T. Degry, T. Dimson, T. Zheng, T. Garipov, T. Stasi, T. Bansal, T. Creech, T. Peterson, T. Eloundou, V. Qi, V. Kosaraju, V. Monaco, V. Pong, V. Fomenko, W. Zheng, W. Zhou, W. McCabe, W. Zaremba, Y. Dubois, Y. Lu, Y. Chen, Y. Cha, Y. Bai, Y. He, Y. Zhang, Y. Wang, Z. Shao, and Z. Li (2024) OpenAI o1 system card. External Links: 2412.16720, [Link](https://arxiv.org/abs/2412.16720) Cited by: [§2](#S2.p2.1), [§3.5](#S3.SS5.SSS0.Px2.p1.1). - OpenAI Team (2025) OpenAI o3 and o4-mini: next-generation reasoning models. Technical report OpenAI. Note: Technical announcement introducing OpenAI’s o3 and o4-mini models with advanced reasoning capabilities and tool integration. External Links: [Link](https://openai.com/blog/openai-o3-o4-mini) Cited by: [§2](#S2.p2.1), [§3.5](#S3.SS5.SSS0.Px2.p1.1). - OpenAI (2023) GPT-4v(ision) system card. System Card OpenAI. External Links: [Link](https://cdn.openai.com/papers/GPTV_System_Card.pdf) Cited by: [§3.6](#S3.SS6.p1.1). - OpenAI (2025a) Deep research. Note: [https://openai.com/index/introducing-deep-research/](https://openai.com/index/introducing-deep-research/) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p2.1), [§4.1.2](#S4.SS1.SSS2.Px1.p1.1), [Table 4](#S4.T4.4.1.24.24.1). - OpenAI (2025b) Introducing codex. Note: [https://openai.com/index/introducing-codex/](https://openai.com/index/introducing-codex/) Cited by: [§4.2.3](#S4.SS2.SSS3.Px3.p1.1). - OpenAI (2025c) OpenAI preparedness framework v2. Technical report OpenAI. Note: Accessed: 2025-12-04 External Links: [Link](https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf) Cited by: [§6.6](#S6.SS6.SSS0.Px1.p1.1). - J. I. Orlicki (2025) Beyond words: a latent memory approach to internal reasoning in llms. External Links: 2502.21030, [Link](https://arxiv.org/abs/2502.21030) Cited by: [§3.3](#S3.SS3.SSS0.Px2.p1.1), [Table 3](#S3.T3.11.9.20.11.1). - K. Ouyang (2025) Spatial-r1: enhancing mllms in video spatial reasoning. arXiv e-prints, pp. arXiv–2504. External Links: [Link](https://arxiv.org/abs/2504.01805) Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, pp. 27730–27744. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf) Cited by: [§2](#S2.p2.1). - S. Ouyang, Y. Hu, G. Chen, Q. Li, F. Zhang, and Y. Liu (2025) Towards reward fairness in RLHF: from a resource allocation perspective. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 3247–3259. External Links: [Link](https://aclanthology.org/2025.acl-long.163/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.163), ISBN 979-8-89176-251-0 Cited by: [§6.6](#S6.SS6.SSS0.Px4.p1.1). - C. Packer, V. Fang, S. G. Patil, K. Lin, S. Wooders, and J. E. Gonzalez (2023) MemGPT: towards llms as operating systems. CoRR abs/2310.08560. External Links: [Link](https://doi.org/10.48550/arXiv.2310.08560) Cited by: [§3.3](#S3.SS3.SSS0.Px1.p1.1), [Table 3](#S3.T3.11.9.13.4.1). - D. Paglieri, B. Cupiał, J. Cook, U. Piterbarg, J. Tuyls, E. Grefenstette, J. N. Foerster, J. Parker-Holder, and T. Rocktäschel (2025) Learning when to plan: efficiently allocating test-time compute for llm agents. External Links: 2509.03581, [Link](https://arxiv.org/abs/2509.03581) Cited by: [§3.1](#S3.SS1.SSS0.Px1.p1.1). - X. Pan, Y. Chen, Y. Chen, Y. Sun, D. Chen, W. Zhang, Y. Xie, Y. Huang, Y. Zhang, D. Gao, Y. Li, B. Ding, and J. Zhou (2025) Trinity-rft: a general-purpose and unified framework for reinforcement fine-tuning of large language models. External Links: 2505.17826, [Link](https://arxiv.org/abs/2505.17826) Cited by: [§4.8](#S4.SS8.SSS0.Px1.p1.1). - Z. Pan and H. Liu (2025) Metaspatial: reinforcing 3d spatial reasoning in vlms for the metaverse. arXiv preprint arXiv:2503.18470. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - R. Y. Pang, W. Yuan, K. Cho, H. He, S. Sukhbaatar, and J. Weston (2024) Iterative reasoning preference optimization. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 116617–116637. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/d37c9ad425fe5b65304d500c6edcba00-Paper-Conference.pdf) Cited by: [§3.4](#S3.SS4.SSS0.Px2.p1.1). - C. Park, S. Han, X. Guo, A. E. Ozdaglar, K. Zhang, and J. Kim (2025a) MAPoRL: multi-agent post-co-training for collaborative large language models with reinforcement learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 30215–30248. External Links: [Link](https://aclanthology.org/2025.acl-long.1459/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1459), ISBN 979-8-89176-251-0 Cited by: [§4.7.3](#S4.SS7.SSS3.p1.1), [Table 8](#S4.T8.3.19.16.1.1.1). - J. Park, J. Na, J. Kim, and H. J. Kim (2025b) DeepVideo-r1: video reinforcement fine-tuning via difficulty-aware regressive grpo. External Links: 2506.07464, [Link](https://arxiv.org/abs/2506.07464) Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - M. Patel, X. Puig, R. Desai, R. Mottaghi, S. Chernova, J. Truong, and A. Rai (2025) ADAPT: actively discovering and adapting to preferences for any task. arXiv preprint arXiv:2504.04040. Cited by: [§3.4](#S3.SS4.SSS0.Px2.p1.1). - Y. Peng, G. Zhang, M. Zhang, Z. You, J. Liu, Q. Zhu, K. Yang, X. Xu, X. Geng, and X. Yang (2025) Lmm-r1: empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - Perplexity (2025) Perplexity deep research. Note: [https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research](https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research) Cited by: [§4.1.2](#S4.SS1.SSS2.Px1.p1.1), [§4.1](#S4.SS1.p1.1), [Table 4](#S4.T4.4.1.25.25.1). - E. Pignatelli, J. Ferret, M. Geist, T. Mesnard, H. van Hasselt, and L. Toni (2024) A survey of temporal credit assignment in deep reinforcement learning. Transactions on Machine Learning Research. Note: Survey Certification External Links: ISSN 2835-8856, [Link](https://openreview.net/forum?id=bNtr6SLgZf) Cited by: [§3.2](#S3.SS2.SSS0.Px3.p1.1), [§3.4](#S3.SS4.SSS0.Px1.p1.1), [§3.7](#S3.SS7.p2.1). - A. Plaat, M. J. van Duijn, N. van Stein, M. Preuss, P. van der Putten, and K. J. Batenburg (2025) Agentic large language models, a survey. CoRR abs/2503.23037. External Links: [Link](https://doi.org/10.48550/arXiv.2503.23037) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1). - G. Poesia, D. Broman, N. Haber, and N. Goodman (2024) Learning formal mathematics from intrinsic motivation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: [Link](https://openreview.net/forum?id=uNKlTQ8mBD) Cited by: [§4.3.2](#S4.SS3.SSS2.Px3.p3.1). - S. Polu and I. Sutskever (2020) Generative language modeling for automated theorem proving. External Links: 2009.03393, [Link](https://arxiv.org/abs/2009.03393) Cited by: [§4.3.2](#S4.SS3.SSS2.Px3.p1.1). - M. Prabhudesai, L. Chen, A. Ippoliti, K. Fragkiadaki, H. Liu, and D. Pathak (2025) Maximizing confidence alone improves reasoning. External Links: 2505.22660, [Link](https://arxiv.org/abs/2505.22660) Cited by: [§4.3.1](#S4.SS3.SSS1.Px1.p1.1), [Table 6](#S4.T6.10.15.5.1.1.1). - A. Project (225) GitHub - a2aproject/A2A: An open protocol enabling communication and interoperability between opaque agentic applications. — github.com. Note: [https://github.com/a2aproject/A2A](https://github.com/a2aproject/A2A)[Accessed 05-12-2025] Cited by: [§6.5](#S6.SS5.SSS0.Px4.p1.1). - Z. Qi, Z. Zhang, Y. Yu, J. Wang, and H. Zhao (2025) VLN-r1: vision-language navigation via reinforcement fine-tuning. arXiv preprint arXiv:2506.17221. Cited by: [§4.6](#S4.SS6.SSS0.Px1.p1.1), [§4.6](#S4.SS6.p1.1). - C. Qian, E. C. Acikgoz, Q. He, H. Wang, X. Chen, D. Hakkani-Tür, G. Tur, and H. Ji (2025) ToolRL: reward is all tool learning needs. External Links: 2504.13958, [Link](https://arxiv.org/abs/2504.13958) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1). - R. Qiang, Y. Zhuang, Y. Li, D. S. V. K, R. Zhang, C. Li, I. S. Wong, S. Yang, P. Liang, C. Zhang, and B. Dai (2025) MLE-dojo: interactive environments for empowering llm agents in machine learning engineering. External Links: 2505.07782, [Link](https://arxiv.org/abs/2505.07782) Cited by: [§5.1.4](#S5.SS1.SSS4.Px2.p1.1), [Table 10](#S5.T10.1.1.22.21.1). - S. Qiao, Z. Qiu, B. Ren, X. Wang, X. Ru, N. Zhang, X. Chen, Y. Jiang, P. Xie, F. Huang, and H. Chen (2025) Agentic knowledgeable self-awareness. In Workshop on Reasoning and Planning for Large Language Models, External Links: [Link](https://openreview.net/forum?id=PGdSLjYwMT) Cited by: [§3.4](#S3.SS4.SSS0.Px2.p1.1). - Y. Qin, X. Li, H. Zou, Y. Liu, S. Xia, Z. Huang, Y. Ye, W. Yuan, H. Liu, Y. Li, and P. Liu (2024a) O1 replication journey: a strategic progress report – part 1. External Links: 2410.18982, [Link](https://arxiv.org/abs/2410.18982) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p1.1). - Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, X. Zhou, Y. Huang, C. Xiao, C. Han, Y. R. Fung, Y. Su, H. Wang, C. Qian, R. Tian, K. Zhu, S. Liang, X. Shen, B. Xu, Z. Zhang, Y. Ye, B. Li, Z. Tang, J. Yi, Y. Zhu, Z. Dai, L. Yan, X. Cong, Y. Lu, W. Zhao, Y. Huang, J. Yan, X. Han, X. Sun, D. Li, J. Phang, C. Yang, T. Wu, H. Ji, G. Li, Z. Liu, and M. Sun (2024b) Tool learning with foundation models. ACM Comput. Surv. 57 (4). External Links: ISSN 0360-0300, [Link](https://doi.org/10.1145/3704435), [Document](https://dx.doi.org/10.1145/3704435) Cited by: [§3](#S3.p1.1). - Y. Qin, Y. Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y. Li, S. Huang, W. Zhong, K. Li, J. Yang, Y. Miao, W. Lin, L. Liu, X. Jiang, Q. Ma, J. Li, X. Xiao, K. Cai, C. Li, Y. Zheng, C. Jin, C. Li, X. Zhou, M. Wang, H. Chen, Z. Li, H. Yang, H. Liu, F. Lin, T. Peng, X. Liu, and G. Shi (2025) UI-tars: pioneering automated gui interaction with native agents. External Links: 2501.12326, [Link](https://arxiv.org/abs/2501.12326) Cited by: [§4.4.3](#S4.SS4.SSS3.p1.1), [Table 7](#S4.T7.8.8.2). - R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, pp. 53728–53741. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1.4), [§2.7](#S2.SS7.p1.1), [Table 2](#S2.T2.1.11.10.1.1.1), [§2](#S2.p2.1). - A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann (2021) Stable-baselines3: reliable reinforcement learning implementations. Journal of Machine Learning Research 22 (268), pp. 1–8. External Links: [Link](http://jmlr.org/papers/v22/20-1364.html) Cited by: [§5.2](#S5.SS2.SSS0.Px3.p1.1), [Table 11](#S5.T11.4.26.26.1.1.1). - P. Rasmussen, P. Paliychuk, T. Beauvais, J. Ryan, and D. Chalef (2025) Zep: a temporal knowledge graph architecture for agent memory. External Links: 2501.13956, [Link](https://arxiv.org/abs/2501.13956) Cited by: [§3.3](#S3.SS3.SSS0.Px3.p1.1), [Table 3](#S3.T3.11.9.23.14.1). - C. Rawles, S. Clinckemaillie, Y. Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. E. Bishop, W. Li, F. Campbell-Ajala, D. K. Toyama, R. J. Berry, D. Tyamagundlu, T. P. Lillicrap, and O. Riva (2025) AndroidWorld: a dynamic benchmarking environment for autonomous agents. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=il5yUQsrjC) Cited by: [§5.1.2](#S5.SS1.SSS2.p1.1), [Table 10](#S5.T10.1.1.18.17.1). - Z. Z. Ren, Z. Shao, J. Song, H. Xin, H. Wang, W. Zhao, L. Zhang, Z. Fu, Q. Zhu, D. Yang, Z. F. Wu, Z. Gou, S. Ma, H. Tang, Y. Liu, W. Gao, D. Guo, and C. Ruan (2025) DeepSeek-prover-v2: advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition. External Links: 2504.21801, [Link](https://arxiv.org/abs/2504.21801) Cited by: [§4.3](#S4.SS3.p2.1). - RL-Factory (2025) GitHub - Simple-Efficient/RL-Factory: Train your Agent model via our easy and efficient framework. Note: [https://github.com/Simple-Efficient/RL-Factory](https://github.com/Simple-Efficient/RL-Factory)[Accessed 03-09-2025] Cited by: [Table 11](#S5.T11.4.11.11.1.1.1). - Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J. Maddison, and T. Hashimoto (2024) Identifying the risks of LM agents with an LM-emulated sandbox. In The Twelfth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=GEcwtMk1uA) Cited by: [§6.1](#S6.SS1.SSS0.Px1.p3.1). - M. Samvelyan, T. Rashid, C. Schroeder de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C. Hung, P. H. S. Torr, J. Foerster, and S. Whiteson (2019) The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, Richland, SC, pp. 2186–2188. External Links: ISBN 9781450363099 Cited by: [§5.1.5](#S5.SS1.SSS5.p1.1), [Table 10](#S5.T10.1.1.41.40.1). - S. Santurkar, E. Durmus, F. Ladhak, C. Lee, P. Liang, and T. Hashimoto (2023) Whose opinions do language models reflect?. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. Cited by: [§6.6](#S6.SS6.SSS0.Px4.p1.1). - T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, pp. 68539–68551. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/d842425e4bf79ba039352da0f658a906-Paper-Conference.pdf) Cited by: [§3.2](#S3.SS2.SSS0.Px1.p1.1). - J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, [Link](https://arxiv.org/abs/1707.06347) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1), [§2.7](#S2.SS7.SSS0.Px2.p1.7), [§2.7](#S2.SS7.p1.1), [Table 2](#S2.T2.1.4.3.1.1.1). - N. Shang, Y. Liu, Y. Zhu, L. L. Zhang, W. Xu, X. Guan, B. Zhang, B. Dong, X. Zhou, B. Zhang, Y. Xin, Z. Miao, S. Li, F. Yang, and M. Yang (2025a) RStar2-agent: agentic reasoning technical report. External Links: 2508.20722, [Link](https://arxiv.org/abs/2508.20722) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p2.1), [§4.3.1](#S4.SS3.SSS1.Px1.p1.1), [Table 6](#S4.T6.10.17.7.1.1.1). - Y. Shang, Y. Li, K. Zhao, L. Ma, J. Liu, F. Xu, and Y. Li (2025b) AgentSquare: automatic LLM agent search in modular design space. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=mPdmDYIQ7f) Cited by: [§3](#S3.p1.1), [§6.1](#S6.SS1.SSS0.Px1.p1.1). - H. Shao, S. Qian, H. Xiao, G. Song, Z. Zong, L. Wang, Y. Liu, and H. Li (2024a) Visual cot: advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems 37, pp. 8612–8642. Cited by: [§3.6](#S3.SS6.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024b) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, [Link](https://arxiv.org/abs/2402.03300) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1), [§3.5](#S3.SS5.SSS0.Px1.p1.1), [§6.4](#S6.SS4.p1.1). - S. She, Y. Bao, Y. Lu, L. Xu, T. Li, W. Zhu, S. Huang, S. Cheng, L. Lu, and Y. Wang (2025) DuPO: enabling reliable llm self-verification via dual preference optimization. External Links: 2508.14460, [Link](https://arxiv.org/abs/2508.14460) Cited by: [§3.4](#S3.SS4.SSS0.Px2.p1.1). - H. Shen, P. Liu, J. Li, C. Fang, Y. Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhang, et al. (2025a) Vlm-r1: a stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - M. Shen, G. Zeng, Z. Qi, Z. Hong, Z. Chen, W. Lu, G. W. Wornell, S. Das, D. D. Cox, and C. Gan (2025b) Satori: reinforcement learning with chain-of-action-thought enhances LLM reasoning via autoregressive search. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=j4FXxMiDjL) Cited by: [§4.3.1](#S4.SS3.SSS1.Px1.p1.1), [Table 6](#S4.T6.3.3.2.1.1). - G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) HybridFlow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, pp. 1279–1297. External Links: [Link](http://dx.doi.org/10.1145/3689031.3696075), [Document](https://dx.doi.org/10.1145/3689031.3696075) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [§5.2](#S5.SS2.SSS0.Px2.p1.1), [Table 11](#S5.T11.4.19.19.1.1.1). - W. Shi, Z. Yu, F. Feng, X. He, and C. Xiong (2025a) Efficient multi-agent system training with data influence-oriented tree search. External Links: 2502.00955, [Link](https://arxiv.org/abs/2502.00955) Cited by: [§4.7.2](#S4.SS7.SSS2.p1.1), [Table 8](#S4.T8.3.15.12.1.1.1). - Y. Shi, S. Li, C. Wu, Z. Liu, J. Fang, H. Cai, A. Zhang, and X. Wang (2025b) Search and refine during think: facilitating knowledge refinement for improved retrieval-augmented reasoning. External Links: 2505.11277, [Link](https://arxiv.org/abs/2505.11277) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1). - Y. Shi, W. Yu, Z. Li, Y. Wang, H. Zhang, N. Liu, H. Mi, and D. Yu (2025c) MobileGUI-rl: advancing mobile gui agent through reinforcement learning in online environment. External Links: 2507.05720, [Link](https://arxiv.org/abs/2507.05720) Cited by: [§4.4.3](#S4.SS4.SSS3.p1.1), [Table 7](#S4.T7.9.21.12.1). - N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, pp. 8634–8652. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/1b44b878bb782e6954cd888628510e90-Paper-Conference.pdf) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - M. Shridhar, X. Yuan, M. Cote, Y. Bisk, A. Trischler, and M. Hausknecht (2021) ALFWorld: aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=0IOX0YcCdTn) Cited by: [Table 10](#S5.T10.1.1.4.3.1), [§6.3](#S6.SS3.p1.1). - Y. Siddiqui, A. Alliegro, A. Artemov, T. Tommasi, D. Sirigatti, V. Rosov, A. Dai, and M. Nießner (2024) Meshgpt: generating triangle meshes with decoder-only transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19615–19625. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - T. Simonds and A. Yoshiyama (2025) LADDER: self-improving llms through recursive problem decomposition. External Links: 2503.00735, [Link](https://arxiv.org/abs/2503.00735) Cited by: [§4.3.1](#S4.SS3.SSS1.Px2.p1.1), [Table 6](#S4.T6.10.19.9.1.1.1). - SimpleVLA-RL Team (2025) SimpleVLA-rl: online rl with simple reward enables training vla models with only one trajectory. Note: [https://github.com/PRIME-RL/SimpleVLA-RL](https://github.com/PRIME-RL/SimpleVLA-RL)GitHub repository Cited by: [§4.6](#S4.SS6.p1.1). - J. Singh, R. Magazine, Y. Pandya, and A. Nambi (2025) Agentic reasoning and tool integration for llms via reinforcement learning. External Links: 2505.01441, [Link](https://arxiv.org/abs/2505.01441) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1), [§4.3.1](#S4.SS3.SSS1.Px1.p1.1), [§4.3.1](#S4.SS3.SSS1.p2.1), [Table 6](#S4.T6.10.13.3.1.1.1). - H. Song, J. Jiang, Y. Min, J. Chen, Z. Chen, W. X. Zhao, L. Fang, and J. Wen (2025a) R1-searcher: incentivizing the search capability in llms via reinforcement learning. External Links: 2503.05592, [Link](https://arxiv.org/abs/2503.05592) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.5.5.1). - H. Song, J. Jiang, W. Tian, Z. Chen, Y. Wu, J. Zhao, Y. Min, W. X. Zhao, L. Fang, and J. Wen (2025b) R1-searcher++: incentivizing the dynamic knowledge acquisition of llms via reinforcement learning. External Links: 2505.17005, [Link](https://arxiv.org/abs/2505.17005) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.6.6.1). - L. Song, T. Shi, and J. Zhao (2025c) The hallucination tax of reinforcement finetuning. External Links: 2505.13988, [Link](https://arxiv.org/abs/2505.13988) Cited by: [§6.1](#S6.SS1.SSS0.Px2.p1.1), [§6.1](#S6.SS1.SSS0.Px2.p2.1), [§6.1](#S6.SS1.SSS0.Px2.p3.1). - Y. Song, W. Xiong, X. Zhao, D. Zhu, W. Wu, K. Wang, C. Li, W. Peng, and S. Li (2024a) AgentBank: towards generalized llm agents via fine-tuning on 50000+ interaction trajectories. In EMNLP (Findings), pp. 2124–2141. External Links: [Link](https://aclanthology.org/2024.findings-emnlp.116) Cited by: [§3.2](#S3.SS2.SSS0.Px1.p1.1). - Y. Song, D. Yin, X. Yue, J. Huang, S. Li, and B. Y. Lin (2024b) Trial and error: exploration-based trajectory optimization of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 7584–7600. External Links: [Link](https://aclanthology.org/2024.acl-long.409/), [Document](https://dx.doi.org/10.18653/v1/2024.acl-long.409) Cited by: [§3.1](#S3.SS1.SSS0.Px2.p1.1). - Z. Song, S. Yue, and J. Zhang (2025d) Thinking isn’t an illusion: overcoming the limitations of reasoning models via tool augmentations. External Links: 2507.17699, [Link](https://arxiv.org/abs/2507.17699) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1). - Z. Song, G. Ouyang, M. Li, Y. Ji, C. Wang, Z. Xu, Z. Zhang, X. Zhang, Q. Jiang, Z. Chen, et al. (2025e) Maniplvm-r1: reinforcement learning for reasoning in embodied manipulation with large vision-language models. arXiv preprint arXiv:2505.16517. Cited by: [§4.6](#S4.SS6.p1.1). - S. S. Srivastava and V. Aggarwal (2025) A technical survey of reinforcement learning techniques for large language models. External Links: 2507.04136, [Link](https://arxiv.org/abs/2507.04136) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1), [§1](#S1.SS0.SSS0.Px1.p1.1.4). - K. E. Stanovich and R. F. West (2000) Individual differences in reasoning: implications for the rationality debate. Behavioral and Brain Sciences 23 (5), pp. 645–665. External Links: [Document](https://dx.doi.org/10.1017/S0140525X00003435) Cited by: [§3.5](#S3.SS5.p1.1.3). - G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese, and T. Patwardhan (2025) PaperBench: evaluating AI’s ability to replicate AI research. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=xF5PuTLPbn) Cited by: [§5.1.4](#S5.SS1.SSS4.Px1.p1.1), [Table 10](#S5.T10.1.1.37.36.1). - A. Su, H. Wang, W. Ren, F. Lin, and W. Chen (2025a) Pixel reasoner: incentivizing pixel-space reasoning with curiosity-driven reinforcement learning. External Links: 2505.15966, [Link](https://arxiv.org/abs/2505.15966) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1), [§3.6](#S3.SS6.SSS0.Px3.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - Z. Su, L. Li, M. Song, Y. Hao, Z. Yang, J. Zhang, G. Chen, J. Gu, J. Li, X. Qu, et al. (2025b) Openthinkimg: learning to think with images via visual tool reinforcement learning. arXiv preprint arXiv:2505.08617. Cited by: [§3.6](#S3.SS6.SSS0.Px3.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - Z. Su, P. Xia, H. Guo, Z. Liu, Y. Ma, X. Qu, J. Liu, Y. Li, K. Zeng, Z. Yang, et al. (2025c) Thinking with images for multimodal reasoning: foundations, methods, and future frontiers. arXiv preprint arXiv:2506.23918. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p2.1), [§3.6](#S3.SS6.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - H. Sun, Z. Qiao, J. Guo, X. Fan, Y. Hou, Y. Jiang, P. Xie, Y. Zhang, F. Huang, and J. Zhou (2025a) ZeroSearch: incentivize the search capability of llms without searching. External Links: 2505.04588, [Link](https://arxiv.org/abs/2505.04588) Cited by: [§4.1.1](#S4.SS1.SSS1.Px2.p1.1), [Table 4](#S4.T4.4.1.21.21.1). - S. Sun, H. Song, Y. Wang, R. Ren, J. Jiang, J. Zhang, F. Bai, J. Deng, W. X. Zhao, Z. Liu, L. Fang, Z. Wang, and J. Wen (2025b) SimpleDeepSearcher: deep information seeking via web-powered reasoning trajectory synthesis. External Links: 2505.16834, [Link](https://arxiv.org/abs/2505.16834) Cited by: [Table 4](#S4.T4.4.1.18.18.1). - W. Sun, M. Lu, Z. Ling, K. Liu, X. Yao, Y. Yang, and J. Chen (2025c) Scaling long-horizon llm agent via context-folding. External Links: 2510.11967, [Link](https://arxiv.org/abs/2510.11967) Cited by: [§3.3](#S3.SS3.SSS0.Px2.p1.1), [Table 3](#S3.T3.10.8.8.1.1). - Y. Sun and T. Wang (2025) Be friendly, not friends: how llm sycophancy shapes user trust. External Links: 2502.10844, [Link](https://arxiv.org/abs/2502.10844) Cited by: [§6.1](#S6.SS1.SSS0.Px3.p1.1). - G. Swamy, C. Dann, R. Kidambi, Z. S. Wu, and A. Agarwal (2024) A minimaximalist approach to reinforcement learning from human feedback. External Links: 2401.04056, [Link](https://arxiv.org/abs/2401.04056) Cited by: [§6.6](#S6.SS6.SSS0.Px4.p1.1). - M. Szep, D. Rueckert, R. von Eisenhart-Rothe, and F. Hinterwimmer (2025) Fine-tuning large language models with limited data: a survey and practical guide. External Links: 2411.09539, [Link](https://arxiv.org/abs/2411.09539) Cited by: [§2](#S2.p1.1). - W. Takerngsaksiri, J. Pasuksmit, P. Thongtanunam, C. Tantithamthavorn, R. Zhang, F. Jiang, J. Li, E. Cook, K. Chen, and M. Wu (2025) Human-in-the-loop software development agents. External Links: 2411.12924, [Link](https://arxiv.org/abs/2411.12924) Cited by: [§6.5](#S6.SS5.SSS0.Px2.p1.1). - H. Tan, Y. Ji, X. Hao, M. Lin, P. Wang, Z. Wang, and S. Zhang (2025a) Reason-rft: reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20752. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - Z. Tan, J. Yan, I. Hsu, R. Han, Z. Wang, L. Le, Y. Song, Y. Chen, H. Palangi, G. Lee, A. R. Iyer, T. Chen, H. Liu, C. Lee, and T. Pfister (2025b) In prospect and retrospect: reflective memory management for long-term personalized dialogue agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 8416–8439. External Links: [Link](https://aclanthology.org/2025.acl-long.413/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.413), ISBN 979-8-89176-251-0 Cited by: [§3.3](#S3.SS3.SSS0.Px1.p1.1), [Table 3](#S3.T3.3.1.1.1.1). - H. Tang, D. Key, and K. Ellis (2024) WorldCoder, a model-based llm agent: building world models by writing code and interacting with the environment. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: [§4.2.4](#S4.SS2.SSS4.Px1.p2.1.1). - Z. Tao, T. Lin, X. Chen, H. Li, Y. Wu, Y. Li, Z. Jin, F. Huang, D. Tao, and J. Zhou (2024) A survey on self-evolution of large language models. External Links: 2404.14387, [Link](https://arxiv.org/abs/2404.14387) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1). - Z. Tao, J. Wu, W. Yin, J. Zhang, B. Li, H. Shen, K. Li, L. Zhang, X. Wang, Y. Jiang, P. Xie, F. Huang, and J. Zhou (2025) WebShaper: agentically data synthesizing via information-seeking formalization. External Links: 2507.15061, [Link](https://arxiv.org/abs/2507.15061) Cited by: [Table 4](#S4.T4.4.1.14.14.1). - A. Team (2025a) Agent Communication Protocol — agentcommunicationprotocol.dev. Note: [https://agentcommunicationprotocol.dev/](https://agentcommunicationprotocol.dev/)[Accessed 05-12-2025] Cited by: [§6.5](#S6.SS5.SSS0.Px4.p1.1). - F. C. team, J. Copet, Q. Carbonneaux, G. Cohen, J. Gehring, J. Kahn, J. Kossen, F. Kreuk, E. McMilin, M. Meyer, Y. Wei, D. Zhang, K. Zheng, J. Armengol-Estapé, P. Bashiri, M. Beck, P. Chambon, A. Charnalia, C. Cummins, J. Decugis, Z. V. Fisches, F. Fleuret, F. Gloeckle, A. Gu, M. Hassid, D. Haziza, B. Y. Idrissi, C. Keller, R. Kindi, H. Leather, G. Maimon, A. Markosyan, F. Massa, P. Mazaré, V. Mella, N. Murray, K. Muzumdar, P. O’Hearn, M. Pagliardini, D. Pedchenko, T. Remez, V. Seeker, M. Selvi, O. Sultan, S. Wang, L. Wehrstedt, O. Yoran, L. Zhang, T. Cohen, Y. Adi, and G. Synnaeve (2025) CWM: an open-weights llm for research on code generation with world models. External Links: 2510.02387, [Link](https://arxiv.org/abs/2510.02387) Cited by: [§4.2.4](#S4.SS2.SSS4.Px1.p2.1.1). - G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. (2023) Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: [§3.6](#S3.SS6.p1.1). - G. R. Team, S. Abeyruwan, J. Ainslie, J. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, et al. (2025a) Gemini robotics: bringing ai into the physical world. arXiv preprint arXiv:2503.20020. Cited by: [§4.6](#S4.SS6.p1.1). - K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. (2025b) Kimi k1. 5: scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - Q. Team (2025b) Qwen3-coder: agentic coding in the world. Note: Accessed: 2025-12-02 External Links: [Link](https://qwenlm.github.io/blog/qwen3-coder/) Cited by: [§4.2.3](#S4.SS2.SSS3.Px1.p1.1), [Table 5](#S4.T5.1.1.32.31.1). - Q. Team (2025c) Qwq-32B: embracing the power of reinforcement learning. Note: Blog post on QwenLM official site[Accessed 2025-08-25] External Links: [Link](https://qwenlm.github.io/blog/qwq-32b/) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p2.1). - The mathlib Community (2020) Mathlib4: the lean 4 mathematical library. Note: Accessed: 2025-09-01 External Links: [Link](https://github.com/leanprover-community/mathlib4) Cited by: [§4.3.2](#S4.SS3.SSS2.Px1.p1.1). - THUDM (2025) Slime: a llm post-training framework for rl scaling. Note: GitHub repository, [https://github.com/THUDM/slime](https://github.com/THUDM/slime)Accessed: 2025-08-13 Cited by: [§5.2](#S5.SS2.SSS0.Px2.p1.1), [Table 11](#S5.T11.4.20.20.1.1.1). - Y. Tian, B. Peng, L. Song, L. Jin, D. Yu, L. Han, H. Mi, and D. Yu (2024) Toward self-improvement of llms via imagination, searching, and criticizing. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 52723–52748. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/5e5853f35164e434015716a8c2a66543-Paper-Conference.pdf) Cited by: [§3.4](#S3.SS4.SSS0.Px3.p1.1). - G. Tie, Z. Zhao, D. Song, F. Wei, R. Zhou, Y. Dai, W. Yin, Z. Yang, J. Yan, Y. Su, Z. Dai, Y. Xie, Y. Cao, L. Sun, P. Zhou, L. He, H. Chen, Y. Zhang, Q. Wen, T. Liu, N. Z. Gong, J. Tang, C. Xiong, H. Ji, P. S. Yu, and J. Gao (2025) A survey on post-training of large language models. External Links: 2503.06072, [Link](https://arxiv.org/abs/2503.06072) Cited by: [§6.2](#S6.SS2.SSS0.Px4.p1.1). - H. Trivedi, T. Khot, M. Hartmann, R. Manku, V. Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian (2024) AppWorld: a controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 16022–16076. External Links: [Link](https://aclanthology.org/2024.acl-long.850/), [Document](https://dx.doi.org/10.18653/v1/2024.acl-long.850) Cited by: [§5.1.1](#S5.SS1.SSS1.p1.1), [Table 10](#S5.T10.1.1.17.16.1). - N. Tsilivis, E. Malach, K. Ullrich, and J. Kempe (2025) How reinforcement learning after next-token prediction facilitates learning. External Links: 2510.11495, [Link](https://arxiv.org/abs/2510.11495) Cited by: [§6.4](#S6.SS4.p1.1). - D. Vattikonda, S. Ravichandran, E. Penaloza, H. Nekoei, M. Thakkar, T. L. S. de Chezelles, N. Gontier, M. Muñoz-Mármol, S. O. Shayegan, S. Raimondo, X. Liu, A. Drouin, L. Charlin, A. Piché, A. Lacoste, and M. Caccia (2025) How to train your llm web agent: a statistical diagnosis. External Links: 2507.04103, [Link](https://arxiv.org/abs/2507.04103) Cited by: [§4.4.3](#S4.SS4.SSS3.p1.1), [Table 7](#S4.T7.9.18.9.1), [§6.2](#S6.SS2.SSS0.Px2.p1.1). - A. Vinod, S. Pandit, A. Vavre, and L. Liu (2025) EgoVLM: policy optimization for egocentric video understanding. External Links: 2506.03097, [Link](https://arxiv.org/abs/2506.03097) Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec (2020) TRL: transformer reinforcement learning. GitHub. Note: [https://github.com/huggingface/trl](https://github.com/huggingface/trl) Cited by: [§5.2](#S5.SS2.SSS0.Px2.p1.1), [Table 11](#S5.T11.4.17.17.1.1.1). - F. Wan, D. Cai, S. Huang, X. Quan, and M. Wang (2025a) Let large language models find the data to train themselves. External Links: [Link](https://openreview.net/forum?id=5YCZZSEosw) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1). - Z. Wan, Y. Li, X. Wen, Y. Song, H. Wang, L. Yang, M. Schmidt, J. Wang, W. Zhang, S. Hu, and Y. Wen (2025b) ReMA: learning to meta-think for llms with multi-agent reinforcement learning. External Links: 2503.09501, [Link](https://arxiv.org/abs/2503.09501) Cited by: [§4.7.3](#S4.SS7.SSS3.p1.1), [Table 8](#S4.T8.3.21.18.1.1.1). - C. Wang, Y. Jiang, C. Yang, H. Liu, and Y. Chen (2023a) Beyond reverse kl: generalizing direct preference optimization with diverse divergence constraints. External Links: 2309.16240, [Link](https://arxiv.org/abs/2309.16240) Cited by: [§6.6](#S6.SS6.SSS0.Px4.p1.1). - D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell (2021) Tent: fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=uXl3bZLkr3c) Cited by: [§4.3.1](#S4.SS3.SSS1.Px1.p1.1). - G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2024a) Voyager: an open-ended embodied agent with large language models. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, [Link](https://openreview.net/forum?id=ehfRiF0R3a) Cited by: [§3.1](#S3.SS1.SSS0.Px2.p1.1), [§4.6](#S4.SS6.SSS0.Px3.p1.1.1). - H. Wang, M. Unsal, X. Lin, M. Baksys, J. Liu, M. D. Santos, F. Sung, M. Vinyes, Z. Ying, Z. Zhu, J. Lu, H. de Saxcé, B. Bailey, C. Song, C. Xiao, D. Zhang, E. Zhang, F. Pu, H. Zhu, J. Liu, J. Bayer, J. Michel, L. Yu, L. Dreyfus-Schmidt, L. Tunstall, L. Pagani, M. Machado, P. Bourigault, R. Wang, S. Polu, T. Barroyer, W. Li, Y. Niu, Y. Fleureau, Y. Hu, Z. Yu, Z. Wang, Z. Yang, Z. Liu, and J. Li (2025a) Kimina-prover preview: towards large formal reasoning models with reinforcement learning. External Links: 2504.11354, [Link](https://arxiv.org/abs/2504.11354) Cited by: [§4.3.2](#S4.SS3.SSS2.Px1.p1.1), [Table 6](#S4.T6.7.7.2.1.1). - H. Wang, C. T. Leong, J. Wang, J. Wang, and W. Li (2025b) SPA-rl: reinforcing llm agents via stepwise progress attribution. External Links: 2505.20732, [Link](https://arxiv.org/abs/2505.20732) Cited by: [§3.2](#S3.SS2.SSS0.Px3.p1.1), [§4.8](#S4.SS8.SSS0.Px1.p1.1). - H. Wang, H. Zou, H. Song, J. Feng, J. Fang, J. Lu, L. Liu, Q. Luo, S. Liang, S. Huang, W. Zhong, Y. Ye, Y. Qin, Y. Xiong, Y. Song, Z. Wu, A. Li, B. Li, C. Dun, C. Liu, D. Zan, F. Leng, H. Wang, H. Yu, H. Chen, H. Guo, J. Su, J. Huang, K. Shen, K. Shi, L. Yan, P. Zhao, P. Liu, Q. Ye, R. Zheng, S. Xin, W. X. Zhao, W. Heng, W. Huang, W. Wang, X. Qin, Y. Lin, Y. Wu, Z. Chen, Z. Wang, B. Zhong, X. Zhang, X. Li, Y. Li, Z. Zhao, C. Jiang, F. Wu, H. Zhou, J. Pang, L. Han, Q. Liu, Q. Ma, S. Liu, S. Cai, W. Fu, X. Liu, Y. Wang, Z. Zhang, B. Zhou, G. Li, J. Shi, J. Yang, J. Tang, L. Li, Q. Han, T. Lu, W. Lin, X. Tong, X. Li, Y. Zhang, Y. Miao, Z. Jiang, Z. Li, Z. Zhao, C. Li, D. Ma, F. Lin, G. Zhang, H. Yang, H. Guo, H. Zhu, J. Liu, J. Du, K. Cai, K. Li, L. Yuan, M. Han, M. Wang, S. Guo, T. Cheng, X. Ma, X. Xiao, X. Huang, X. Chen, Y. Du, Y. Chen, Y. Wang, Z. Li, Z. Yang, Z. Zeng, C. Jin, C. Li, H. Chen, H. Chen, J. Chen, Q. Zhao, and G. Shi (2025c) UI-tars-2 technical report: advancing gui agent with multi-turn reinforcement learning. External Links: 2509.02544, [Link](https://arxiv.org/abs/2509.02544) Cited by: [Table 7](#S4.T7.9.19.10.1). - H. Wang, C. Qian, M. Li, J. Qiu, B. Xue, M. Wang, H. Ji, and K. Wong (2025d) Toward a theory of agents as tool-use decision-makers. External Links: 2506.00886, [Link](https://arxiv.org/abs/2506.00886) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1). - H. Wang, C. Qian, W. Zhong, X. Chen, J. Qiu, S. Huang, B. Jin, M. Wang, K. Wong, and H. Ji (2025e) Acting less is reasoning more! teaching model to act efficiently. External Links: 2504.14870, [Link](https://arxiv.org/abs/2504.14870) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1). - H. Wang, R. Wang, B. Xue, H. Xia, J. Cao, Z. Liu, J. Z. Pan, and K. Wong (2024b) AppBench: planning of multiple APIs from various APPs for complex user instruction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 15322–15336. External Links: [Link](https://aclanthology.org/2024.emnlp-main.856/), [Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.856) Cited by: [Table 10](#S5.T10.1.1.16.15.1). - J. Wang, Z. Tian, X. Wang, X. Zhang, W. Huang, Z. Wu, and Y. Jiang (2025f) Simplear: pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504.11455. Cited by: [§4.5](#S4.SS5.SSS0.Px1.p1.1). - J. Wang, J. WANG, B. Athiwaratkun, C. Zhang, and J. Zou (2025g) Mixture-of-agents enhances large language model capabilities. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=h0ZfDIrj7T) Cited by: [§4.7](#S4.SS7.p1.1), [Table 8](#S4.T8.3.8.5.1.1.1). - J. Wang, Z. Zhang, Y. He, Z. Zhang, Y. Song, T. Shi, Y. Li, H. Xu, K. Wu, X. Yi, et al. (2024c) Enhancing code llms with reinforcement learning in code generation: a survey. arXiv preprint arXiv:2412.20367. External Links: [Link](https://arxiv.org/abs/2412.20367) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1). - K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han (2019) HAQ: hardware-aware automated quantization with mixed precision. External Links: 1811.08886, [Link](https://arxiv.org/abs/1811.08886) Cited by: [§6.6](#S6.SS6.SSS0.Px2.p1.1). - M. Wang, Y. Li, H. Wang, X. Zhang, N. Xu, B. Wu, F. Huang, H. Yu, and W. Mao (2025h) Adaptive thinking via mode policy optimization for social language agents. External Links: 2505.02156, [Link](https://arxiv.org/abs/2505.02156) Cited by: [§4.8](#S4.SS8.SSS0.Px5.p1.1). - P. Wang and H. Ling (2025) SVQA-r1: reinforcing spatial reasoning in mllms via view-consistent reward optimization. arXiv preprint arXiv:2506.01371. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - P. Wang, L. Li, Z. Shao, R. X. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui (2024d) Math-shepherd: verify and reinforce llms step-by-step without human annotations. External Links: 2312.08935, [Link](https://arxiv.org/abs/2312.08935) Cited by: [§3.5](#S3.SS5.SSS0.Px1.p2.1). - P. Wang, Y. Wei, Y. Peng, X. Wang, W. Qiu, W. Shen, T. Xie, J. Pei, J. Zhang, Y. Hao, et al. (2025i) Skywork r1v2: multimodal hybrid reinforcement learning for reasoning. arXiv preprint arXiv:2504.16656. Cited by: [Table 2](#S2.T2.1.31.30.1.1.1). - P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. (2024e) Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: [§3.6](#S3.SS6.p1.1). - R. Wang, R. A. Genadi, B. E. Bouardi, Y. Wang, F. Koto, Z. Liu, T. Baldwin, and H. Li (2025j) AgentFly: extensible and scalable reinforcement learning for lm agents. External Links: 2507.14897, [Link](https://arxiv.org/abs/2507.14897) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.8.8.1.1.1). - R. Wang, J. Zhang, Y. Jia, R. Pan, S. Diao, R. Pi, and T. Zhang (2024f) TheoremLlama: transforming general-purpose LLMs into lean4 experts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 11953–11974. External Links: [Link](https://aclanthology.org/2024.emnlp-main.667/), [Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.667) Cited by: [§4.3.2](#S4.SS3.SSS2.Px3.p1.1). - R. Wang, P. Jansen, M. Côté, and P. Ammanabrolu (2022) ScienceWorld: is your agent smarter than a 5th grader?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y. Goldberg, Z. Kozareva, and Y. Zhang (Eds.), Abu Dhabi, United Arab Emirates, pp. 11279–11298. External Links: [Link](https://aclanthology.org/2022.emnlp-main.775/), [Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.775) Cited by: [§5.1.4](#S5.SS1.SSS4.Px1.p1.1), [Table 10](#S5.T10.1.1.6.5.1), [§6.3](#S6.SS3.p1.1). - S. Wang, L. Yu, C. Gao, C. Zheng, S. Liu, R. Lu, K. Dang, X. Chen, J. Yang, Z. Zhang, Y. Liu, A. Yang, A. Zhao, Y. Yue, S. Song, B. Yu, G. Huang, and J. Lin (2025k) Beyond the 80/20 rule: high-entropy minority tokens drive effective reinforcement learning for llm reasoning. External Links: 2506.01939, [Link](https://arxiv.org/abs/2506.01939) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p1.1). - S. Wang, G. Zhang, M. Yu, G. Wan, F. Meng, C. Guo, K. Wang, and Y. Wang (2025l) G-safeguard: a topology-guided security lens and treatment on llm-based multi-agent systems. arXiv preprint arXiv:2502.11127. Cited by: [§6.1](#S6.SS1.SSS0.Px1.p1.1). - S. Wang, S. Zhang, J. Zhang, R. Hu, X. Li, T. Zhang, J. Li, F. Wu, G. Wang, and E. Hovy (2025m) Reinforcement learning enhanced llms: a survey. External Links: 2412.10400, [Link](https://arxiv.org/abs/2412.10400) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1). - S. Wang, Q. Guo, K. Zhao, Y. Zhang, X. Li, X. Li, S. Li, R. She, S. Yu, and W. P. Tay (2025n) CodeBoost: boosting code llms by squeezing knowledge from code snippets with rl. External Links: 2508.05242, [Link](https://arxiv.org/abs/2508.05242) Cited by: [§4.2.1](#S4.SS2.SSS1.Px2.p1.1), [Table 5](#S4.T5.1.1.11.10.1), [§5.1.3](#S5.SS1.SSS3.Px2.p1.1). - W. Wang, S. Xiong, G. Chen, W. Gao, S. Guo, Y. He, J. Huang, J. Liu, Z. Li, X. Li, et al. (2025o) Reinforcement learning optimization for large-scale learning: an efficient and user-friendly scaling library. arXiv preprint arXiv:2506.06122. Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.12.12.1.1.1). - W. Wang, Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, J. Zhu, X. Zhu, L. Lu, Y. Qiao, and J. Dai (2024g) Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou (2023b) Self-consistency improves chain of thought reasoning in language models. External Links: 2203.11171, [Link](https://arxiv.org/abs/2203.11171) Cited by: [§3.5](#S3.SS5.SSS0.Px1.p2.1). - Y. Wang, Y. Wang, D. Guo, J. Chen, R. Zhang, Y. Ma, and Z. Zheng (2025p) RLCoder: Reinforcement Learning for Repository-Level Code Completion . In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), Vol. , Los Alamitos, CA, USA, pp. 1140–1152. External Links: ISSN , [Document](https://dx.doi.org/10.1109/ICSE55347.2025.00014), [Link](https://doi.ieeecomputersociety.org/10.1109/ICSE55347.2025.00014) Cited by: [§4.2.3](#S4.SS2.SSS3.Px1.p1.1), [Table 5](#S4.T5.1.1.31.30.1). - Y. Wang, L. Yang, Y. Tian, K. Shen, and M. Wang (2025q) Co-evolving llm coder and unit tester via reinforcement learning. External Links: 2506.03136, [Link](https://arxiv.org/abs/2506.03136) Cited by: [§4.2.1](#S4.SS2.SSS1.Px1.p1.1), [§4.7.3](#S4.SS7.SSS3.p1.1), [Table 5](#S4.T5.1.1.7.6.1), [Table 8](#S4.T8.2.2.2.1.1). - Y. Wang, Q. Yang, Z. Zeng, L. Ren, L. Liu, B. Peng, H. Cheng, X. He, K. Wang, J. Gao, W. Chen, S. Wang, S. S. Du, and Y. Shen (2025r) Reinforcement learning for reasoning in large language models with one training example. External Links: 2504.20571, [Link](https://arxiv.org/abs/2504.20571) Cited by: [§4.3.1](#S4.SS3.SSS1.Px1.p1.1), [Table 6](#S4.T6.4.4.2.1.1). - Y. Wang, Y. Gao, X. Chen, H. Jiang, S. Li, J. Yang, Q. Yin, Z. Li, X. Li, B. Yin, J. Shang, and J. McAuley (2024h) MEMORYLLM: towards self-updatable large language models. In Forty-first International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=p0lKWzdikQ) Cited by: [§3.3](#S3.SS3.SSS0.Px2.p1.1), [Table 3](#S3.T3.11.9.18.9.1). - Y. Wang, D. Krotov, Y. Hu, Y. Gao, W. Zhou, J. McAuley, D. Gutfreund, R. Feris, and Z. He (2025s) M+: extending memoryLLM with scalable long-term memory. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=OcqbkROe8J) Cited by: [§3.3](#S3.SS3.SSS0.Px2.p1.1), [Table 3](#S3.T3.11.9.19.10.1). - Y. Wang, R. Takanobu, Z. Liang, Y. Mao, Y. Hu, J. McAuley, and X. Wu (2025t) Mem-$\alpha$: learning memory construction via reinforcement learning. External Links: 2509.25911, [Link](https://arxiv.org/abs/2509.25911) Cited by: [§3.3](#S3.SS3.SSS0.Px1.p1.1), [Table 3](#S3.T3.6.4.4.2.2). - Z. Wang, J. Lorraine, Y. Wang, H. Su, J. Zhu, S. Fidler, and X. Zeng (2024i) Llama-mesh: unifying 3d mesh generation with language models. arXiv preprint arXiv:2411.09595. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - Z. Wang, X. Guo, S. Stoica, H. Xu, H. Wang, H. Ha, X. Chen, Y. Chen, M. Yan, F. Huang, et al. (2025u) Perception-aware policy optimization for multimodal reasoning. arXiv preprint arXiv:2507.06448. Cited by: [Table 2](#S2.T2.1.37.36.1.1.1), [§3.6](#S3.SS6.SSS0.Px1.p1.1). - Z. Wang, B. Bi, S. K. Pentyala, K. Ramnath, S. Chaudhuri, S. Mehrotra, X. Mao, S. Asur, et al. (2024j) A comprehensive survey of llm alignment techniques: rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216. External Links: [Link](https://arxiv.org/abs/2407.16216) Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1.4). - Z. Wang, Z. Cheng, H. Zhu, D. Fried, and G. Neubig (2024k) What are tools anyway? a survey from the language model perspective. In First Conference on Language Modeling, External Links: [Link](https://openreview.net/forum?id=Xh1B90iBSR) Cited by: [§1](#S1.SS0.SSS0.Px2.p1.1). - Z. Wang, K. Wang, Q. Wang, P. Zhang, L. Li, Z. Yang, X. Jin, K. Yu, M. N. Nguyen, L. Liu, E. Gottlieb, Y. Lu, K. Cho, J. Wu, L. Fei-Fei, L. Wang, Y. Choi, and M. Li (2025v) RAGEN: understanding self-evolution in llm agents via multi-turn reinforcement learning. External Links: 2504.20073, [Link](https://arxiv.org/abs/2504.20073) Cited by: [Table 2](#S2.T2.1.29.28.1.1.1), [§4.8](#S4.SS8.SSS0.Px1.p1.1). - Z. Wang, X. Zheng, K. An, C. Ouyang, J. Cai, Y. Wang, and Y. Wu (2025w) StepSearch: igniting llms search ability via step-wise proximal policy optimization. External Links: 2505.15107, [Link](https://arxiv.org/abs/2505.15107) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.8.8.1). - T. Webb, S. S. Mondal, and I. Momennejad (2025) A brain-inspired agentic architecture to improve planning with llms. Nature Communications 16 (1), pp. 8633. External Links: [Document](https://dx.doi.org/10.1038/s41467-025-63804-5), [Link](https://doi.org/10.1038/s41467-025-63804-5) Cited by: [§3.1](#S3.SS1.p1.1.1). - H. Wei, Z. Zhang, S. He, T. Xia, S. Pan, and F. Liu (2025a) PlanGenLLMs: a modern survey of LLM planning capabilities. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 19497–19521. External Links: [Link](https://aclanthology.org/2025.acl-long.958/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.958), ISBN 979-8-89176-251-0 Cited by: [§3.1](#S3.SS1.SSS0.Px1.p1.1), [§3.1](#S3.SS1.p1.1.1), [§3](#S3.p1.1). - J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese (2025b) BrowseComp: a simple yet challenging benchmark for browsing agents. External Links: 2504.12516, [Link](https://arxiv.org/abs/2504.12516) Cited by: [§4.1.2](#S4.SS1.SSS2.Px1.p1.1), [§4.1.2](#S4.SS1.SSS2.Px2.p1.1.1). - Y. Wei, X. Yu, Y. Weng, T. Pan, A. Li, and L. Du (2025c) AutoTIR: autonomous tools integrated reasoning via reinforcement learning. External Links: 2507.21836, [Link](https://arxiv.org/abs/2507.21836) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1). - Y. Wei, X. Shan, and J. Li (2025d) LERO: llm-driven evolutionary framework with hybrid rewards and enhanced observation for multi-agent reinforcement learning. External Links: 2503.21807, [Link](https://arxiv.org/abs/2503.21807) Cited by: [§4.7.3](#S4.SS7.SSS3.p1.1). - Y. Wei, O. Duchenne, J. Copet, Q. Carbonneaux, L. Zhang, D. Fried, G. Synnaeve, R. Singh, and S. I. Wang (2025e) SWE-rl: advancing llm reasoning via reinforcement learning on open software evolution. External Links: 2502.18449, [Link](https://arxiv.org/abs/2502.18449) Cited by: [§4.2.3](#S4.SS2.SSS3.Px1.p1.1), [Table 5](#S4.T5.1.1.29.28.1). - Z. Wei, W. Yao, Y. Liu, W. Zhang, Q. Lu, L. Qiu, C. Yu, P. Xu, C. Zhang, B. Yin, H. Yun, and L. Li (2025f) WebAgent-r1: training web agents via end-to-end multi-turn reinforcement learning. In ICML 2025 Workshop on Computer Use Agents, External Links: [Link](https://openreview.net/forum?id=KqrYTALRjH) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1), [§4.4.3](#S4.SS4.SSS3.p1.1), [Table 7](#S4.T7.9.17.8.1). - C. Wen, T. Guo, S. Zhao, W. Zou, and X. Li (2025) Sari: structured audio reasoning via curriculum-guided reinforcement learning. arXiv preprint arXiv:2504.15900. Cited by: [§3.6](#S3.SS6.SSS0.Px5.p1.1). - J. Wen, R. Zhong, A. Khan, E. Perez, J. Steinhardt, M. Huang, S. R. Bowman, H. He, and S. Feng (2024) Language models learn to mislead humans via rlhf. External Links: 2409.12822, [Link](https://arxiv.org/abs/2409.12822) Cited by: [§6.1](#S6.SS1.SSS0.Px3.p2.1). - J. Weng, H. Chen, D. Yan, K. You, A. Duburcq, M. Zhang, Y. Su, H. Su, and J. Zhu (2022) Tianshou: a highly modularized deep reinforcement learning library. Journal of Machine Learning Research 23 (267), pp. 1–6. External Links: [Link](http://jmlr.org/papers/v23/21-1127.html) Cited by: [§5.2](#S5.SS2.SSS0.Px3.p1.1), [Table 11](#S5.T11.4.25.25.1.1.1). - L. Weng (2023) LLM-powered autonomous agents. lilianweng.github.io. External Links: [Link](https://lilianweng.github.io/posts/2023-06-23-agent/) Cited by: [§3](#S3.p1.1). - C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Dey, Shubh-Agrawal, S. S. Sandha, S. Naidu, C. Hegde, Y. LeCun, T. Goldstein, W. Neiswanger, and M. Goldblum (2025) LiveBench: a challenging, contamination-limited llm benchmark. External Links: 2406.19314, [Link](https://arxiv.org/abs/2406.19314) Cited by: [§6.6](#S6.SS6.SSS0.Px5.p1.1). - R. J. Williams (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 (3), pp. 229–256. External Links: [Document](https://dx.doi.org/10.1007/BF00992696), [Link](https://doi.org/10.1007/BF00992696), ISSN 1573-0565 Cited by: [§2.7](#S2.SS7.SSS0.Px1.p1.8.1). - J. Wu, B. Li, R. Fang, W. Yin, L. Zhang, Z. Tao, D. Zhang, Z. Xi, G. Fu, Y. Jiang, P. Xie, F. Huang, and J. Zhou (2025a) WebDancer: towards autonomous information seeking agency. External Links: 2505.22648, [Link](https://arxiv.org/abs/2505.22648) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1), [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.10.10.1). - J. Wu, W. Yin, Y. Jiang, Z. Wang, Z. Xi, R. Fang, L. Zhang, Y. He, D. Zhou, P. Xie, and F. Huang (2025b) WebWalker: benchmarking LLMs in web traversal. In Workshop on Reasoning and Planning for Large Language Models, External Links: [Link](https://openreview.net/forum?id=cVI9lAfkuK) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1). - J. Wu, H. Li, X. Zhang, J. Luo, Y. Huang, R. Chu, Y. Yang, and S. Li (2025c) IterPref: focal preference learning for code generation via iterative debugging. External Links: 2503.02783, [Link](https://arxiv.org/abs/2503.02783) Cited by: [§4.2.2](#S4.SS2.SSS2.Px2.p1.1), [Table 5](#S4.T5.1.1.23.22.1). - J. Wu, M. Feng, S. Zhang, F. Che, Z. Wen, C. Liao, and J. Tao (2025d) Beyond examples: high-level automated reasoning paradigm in in-context learning via mcts. External Links: 2411.18478, [Link](https://arxiv.org/abs/2411.18478) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p2.1). - J. Wu, J. Zhu, Y. Liu, M. Xu, and Y. Jin (2025e) Agentic reasoning: a streamlined framework for enhancing LLM reasoning with agentic tools. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 28489–28503. External Links: [Link](https://aclanthology.org/2025.acl-long.1383/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1383), ISBN 979-8-89176-251-0 Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1). - J. Wu, Y. Xie, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He (2024) $\beta$-DPO: direct preference optimization with dynamic $\beta$. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: [Link](https://openreview.net/forum?id=ZfBuhzE556) Cited by: [Table 2](#S2.T2.1.1.1.1.1). - M. Wu, M. Norrish, C. Walder, and A. Dezfouli (2021) TacticZero: learning to prove theorems from scratch with deep reinforcement learning. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), External Links: [Link](https://openreview.net/forum?id=edmYVRkYZv) Cited by: [§4.3.2](#S4.SS3.SSS2.p1.1). - M. Wu, L. Wang, P. Zhao, F. Yang, J. Zhang, J. Liu, Y. Zhan, W. Han, H. Sun, J. Ji, X. Sun, Q. Lin, W. Deng, D. Zhang, F. Sun, Q. Zhang, and R. Ji (2025f) RePrompt: reasoning-augmented reprompting for text-to-image generation via reinforcement learning. External Links: 2505.17540, [Link](https://arxiv.org/abs/2505.17540) Cited by: [§4.5](#S4.SS5.SSS0.Px1.p1.1). - M. Wu, J. Yang, J. Jiang, M. Li, K. Yan, H. Yu, M. Zhang, C. Zhai, and K. Nahrstedt (2025g) VTool-r1: vlms learn to think with images via reinforcement learning on multimodal tool use. External Links: 2505.19255, [Link](https://arxiv.org/abs/2505.19255) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1), [§3.6](#S3.SS6.SSS0.Px3.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - W. Wu, C. Gao, J. Chen, K. Q. Lin, Q. Meng, Y. Zhang, Y. Qiu, H. Zhou, and M. Z. Shou (2025h) Reinforcement learning in vision: a survey. External Links: 2508.08189, [Link](https://arxiv.org/abs/2508.08189) Cited by: [§3.6](#S3.SS6.p1.1), [§4.5](#S4.SS5.p1.1). - X. Wu, K. Li, Y. Zhao, L. Zhang, L. Ou, H. Yin, Z. Zhang, Y. Jiang, P. Xie, F. Huang, et al. (2025i) ReSum: unlocking long-horizon search intelligence via context summarization. arXiv preprint arXiv:2509.13313. Cited by: [§3.3](#S3.SS3.SSS0.Px2.p1.1), [Table 3](#S3.T3.9.7.7.1.1). - Y. Wu, I. Hua, and Y. Ding (2025j) Unveiling environmental impacts of large language model serving: a functional unit view. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 10560–10576. External Links: [Link](https://aclanthology.org/2025.acl-long.519/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.519), ISBN 979-8-89176-251-0 Cited by: [§6.6](#S6.SS6.SSS0.Px2.p1.1). - Y. Wu, S. Liang, C. Zhang, Y. Wang, Y. Zhang, H. Guo, R. Tang, and Y. Liu (2025k) From human memory to ai memory: a survey on memory mechanisms in the era of llms. External Links: 2504.15965, [Link](https://arxiv.org/abs/2504.15965) Cited by: [§3.3](#S3.SS3.p1.1), [Table 3](#S3.T3), [Table 3](#S3.T3.2.1). - Y. Wu, Y. Zhou, Z. Ziheng, Y. Peng, X. Ye, X. Hu, W. Zhu, L. Qi, M. Yang, and X. Yang (2025l) On the generalization of sft: a reinforcement learning perspective with reward rectification. External Links: 2508.05629, [Link](https://arxiv.org/abs/2508.05629) Cited by: [§6.2](#S6.SS2.SSS0.Px4.p1.1). - Z. Wu, S. Huang, Z. Zhou, H. Ying, Z. Yuan, W. Zhang, D. Lin, and K. Chen (2025m) InternLM2.5-stepprover: advancing automated theorem proving via critic-guided search. In 2nd AI for Math Workshop @ ICML 2025, External Links: [Link](https://openreview.net/forum?id=qwCqeIg5iI) Cited by: [§4.3.2](#S4.SS3.SSS2.Px3.p2.1), [Table 6](#S4.T6.10.26.16.1.1.1). - x.ai (2025) Grok 3 beta — the age of reasoning agents. External Links: [Link](https://x.ai/news/grok-3) Cited by: [§4.1.2](#S4.SS1.SSS2.Px1.p1.1), [Table 4](#S4.T4.4.1.28.28.1). - Z. Xi, Y. Ding, W. Chen, B. Hong, H. Guo, J. Wang, X. Guo, D. Yang, C. Liao, W. He, S. Gao, L. Chen, R. Zheng, Y. Zou, T. Gui, Q. Zhang, X. Qiu, X. Huang, Z. Wu, and Y. Jiang (2025) AgentGym: evaluating and training large language model-based agents across diverse environments. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 27914–27961. External Links: [Link](https://aclanthology.org/2025.acl-long.1355/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1355), ISBN 979-8-89176-251-0 Cited by: [§5.1.6](#S5.SS1.SSS6.p1.1), [Table 10](#S5.T10.1.1.7.6.1). - J. Xia, Y. Zang, P. Gao, Y. Li, and K. Zhou (2025a) Visionary-r1: mitigating shortcuts in visual reasoning with reinforcement learning. External Links: 2505.14677, [Link](https://arxiv.org/abs/2505.14677) Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - P. Xia, J. Wang, Y. Peng, K. Zeng, X. Wu, X. Tang, H. Zhu, Y. Li, S. Liu, Y. Lu, et al. (2025b) MMedAgent-rl: optimizing multi-agent collaboration for multimodal medical reasoning. arXiv preprint arXiv:2506.00555. External Links: [Link](https://arxiv.org/abs/2506.00555) Cited by: [§4.7.3](#S4.SS7.SSS3.p1.1), [Table 8](#S4.T8.3.23.20.1.1.1). - J. Xiao, Z. Li, X. Xie, E. Getzen, C. Fang, Q. Long, and W. J. Su (2025) On the algorithmic bias of aligning large language models with rlhf: preference collapse and matching regularization. External Links: 2405.16455, [Link](https://arxiv.org/abs/2405.16455) Cited by: [§6.6](#S6.SS6.SSS0.Px4.p1.1). - W. Xiao, Z. Wang, L. Gan, S. Zhao, Z. Li, R. Lei, W. He, L. A. Tuan, L. Chen, H. Jiang, et al. (2024) A comprehensive survey of direct preference optimization: datasets, theories, variants, and applications. arXiv preprint arXiv:2410.15595. Cited by: [§1](#S1.SS0.SSS0.Px1.p1.1.4). - T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu (2024) OSWorld: benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 52040–52094. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/5d413e48f84dc61244b6be550f1cd8f5-Paper-Datasets_and_Benchmarks_Track.pdf) Cited by: [§5.1.2](#S5.SS1.SSS2.p1.1), [Table 10](#S5.T10.1.1.19.18.1), [§6.6](#S6.SS6.SSS0.Px3.p1.1). - Z. Xie, J. chen, L. Chen, W. Mao, J. Xu, and L. Kong (2025) Teaching language models to critique via reinforcement learning. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=UVoxPlv5E1) Cited by: [§4.2.2](#S4.SS2.SSS2.Px2.p1.1), [Table 5](#S4.T5.1.1.25.24.1). - H. Xin, Z.Z. Ren, J. Song, Z. Shao, W. Zhao, H. Wang, B. Liu, L. Zhang, X. Lu, Q. Du, W. Gao, H. Zhang, Q. Zhu, D. Yang, Z. Gou, Z.F. Wu, F. Luo, and C. Ruan (2025) DeepSeek-prover-v1.5: harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=I4YAIwrsXa) Cited by: [§4.3.2](#S4.SS3.SSS2.Px1.p1.1), [§4.3.2](#S4.SS3.SSS2.p1.1), [Table 6](#S4.T6.5.5.2.1.1). - Z. Xing, X. Hu, C. Fu, W. Wang, J. Dai, and P. Heng (2025) Echoink-r1: exploring audio-visual reasoning in multimodal llms via reinforcement learning. arXiv preprint arXiv:2505.04623. Cited by: [§3.6](#S3.SS6.SSS0.Px5.p1.1). - W. Xiong, W. Zhao, W. Yuan, O. Golovneva, T. Zhang, J. Weston, and S. Sukhbaatar (2025) StepWiser: stepwise generative judges for wiser reasoning. External Links: 2508.19229, [Link](https://arxiv.org/abs/2508.19229) Cited by: [§3.4](#S3.SS4.SSS0.Px1.p1.1). - F. Xu, H. Yan, C. Ma, H. Zhao, J. Liu, Q. Lin, and Z. Wu (2025a) $\phi$-Decoding: adaptive foresight sampling for balanced inference-time exploration and exploitation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 13214–13227. External Links: [Link](https://aclanthology.org/2025.acl-long.647/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.647), ISBN 979-8-89176-251-0 Cited by: [§3.5](#S3.SS5.SSS0.Px2.p1.1). - F. F. Xu, Y. Song, B. Li, Y. Tang, K. Jain, M. Bao, Z. Z. Wang, X. Zhou, Z. Guo, M. Cao, M. Yang, H. Y. Lu, A. Martin, Z. Su, L. Maben, R. Mehta, W. Chi, L. Jang, Y. Xie, S. Zhou, and G. Neubig (2024a) TheAgentCompany: benchmarking llm agents on consequential real world tasks. External Links: 2412.14161, [Link](https://arxiv.org/abs/2412.14161) Cited by: [§5.1.3](#S5.SS1.SSS3.Px1.p1.1), [Table 10](#S5.T10.1.1.23.22.1). - G. Xu, P. Jin, Z. Wu, H. Li, Y. Song, L. Sun, and L. Yuan (2025b) LLaVA-cot: let vision language models reason step-by-step. External Links: 2411.10440, [Link](https://arxiv.org/abs/2411.10440) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p2.1). - R. Xu, Y. Zhuang, Y. Zhong, Y. Yu, X. Tang, H. Wu, M. D. Wang, P. Ruan, D. Yang, T. Wang, G. Xiao, C. Yang, Y. Xie, and W. Shi (2025c) MedAgentGym: training llm agents for code-based medical reasoning at scale. External Links: 2506.04405, [Link](https://arxiv.org/abs/2506.04405) Cited by: [§5.1.4](#S5.SS1.SSS4.Px3.p1.1), [Table 10](#S5.T10.1.1.24.23.1). - R. Xu, X. Wang, T. Wang, Y. Chen, J. Pang, and D. Lin (2024b) Pointllm: empowering large language models to understand point clouds. In European Conference on Computer Vision, pp. 131–147. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - W. Xu, K. Mei, H. Gao, J. Tan, Z. Liang, and Y. Zhang (2025d) A-mem: agentic memory for llm agents. External Links: 2502.12110, [Link](https://arxiv.org/abs/2502.12110) Cited by: [§3.3](#S3.SS3.SSS0.Px3.p1.1), [Table 3](#S3.T3.11.9.24.15.1). - Y. Xu, C. Li, H. Zhou, X. Wan, C. Zhang, A. Korhonen, and I. Vulić (2025e) Visual planning: let’s think only with images. arXiv preprint arXiv:2505.11409. Cited by: [§3.6](#S3.SS6.SSS0.Px4.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - Z. Xue, J. Wu, Y. Gao, F. Kong, L. Zhu, M. Chen, Z. Liu, W. Liu, Q. Guo, W. Huang, and P. Luo (2025) DanceGRPO: unleashing grpo on visual generation. External Links: 2505.07818, [Link](https://arxiv.org/abs/2505.07818) Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - A. Yan, Z. Yang, W. Zhu, K. Lin, L. Li, J. Wang, J. Yang, Y. Zhong, J. J. McAuley, J. Gao, Z. Liu, and L. Wang (2023) GPT-4v in wonderland: large multimodal models for zero-shot smartphone gui navigation. CoRR abs/2311.07562. External Links: [Link](https://doi.org/10.48550/arXiv.2311.07562) Cited by: [§4.4.1](#S4.SS4.SSS1.Px1.p1.1), [Table 7](#S4.T7.9.12.3.1). - C. Yan, F. Che, X. Huang, X. Xu, X. Li, Y. Li, X. Qu, J. Shi, Z. He, C. Lin, et al. (2025a) Re: form–reducing human priors in scalable formal software verification with rl in llms: a preliminary study on dafny. arXiv preprint arXiv:2507.16331. External Links: [Link](https://arxiv.org/abs/2507.16331) Cited by: [§4.2.1](#S4.SS2.SSS1.Px1.p1.1), [Table 5](#S4.T5.1.1.18.17.1). - S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, H. Schütze, V. Tresp, and Y. Ma (2025b) Memory-r1: enhancing large language model agents to manage and utilize memories via reinforcement learning. External Links: 2508.19828, [Link](https://arxiv.org/abs/2508.19828) Cited by: [§3.3](#S3.SS3.SSS0.Px1.p1.1), [Table 3](#S3.T3.4.2.2.1.1). - A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu (2025a) Qwen3 technical report. External Links: 2505.09388, [Link](https://arxiv.org/abs/2505.09388) Cited by: [§3.5](#S3.SS5.SSS0.Px3.p1.1). - A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, K. Lu, M. Xue, R. Lin, T. Liu, X. Ren, and Z. Zhang (2024a) Qwen2.5-math technical report: toward mathematical expert model via self-improvement. External Links: 2409.12122, [Link](https://arxiv.org/abs/2409.12122) Cited by: [§3.5](#S3.SS5.SSS0.Px1.p1.1). - C. Yang, S. Su, S. Liu, X. Dong, Y. Yu, W. Su, X. Wang, Z. Liu, J. Zhu, H. Li, W. Wang, Y. Qiao, X. Zhu, and J. Dai (2025b) ZeroGUI: automating online gui learning at zero human cost. External Links: 2505.23762, [Link](https://arxiv.org/abs/2505.23762) Cited by: [§4.4.3](#S4.SS4.SSS3.p1.1), [Table 7](#S4.T7.9.20.11.1). - J. Yang, F. Ma, Z. Wang, D. Yin, K. Rong, F. Rao, and R. Zhang (2025c) WeThink: toward general-purpose vision-language reasoning via reinforcement learning. External Links: 2506.07905, [Link](https://arxiv.org/abs/2506.07905) Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - K. Yang, G. Poesia, J. He, W. Li, K. Lauter, S. Chaudhuri, and D. Song (2024b) Formal mathematical reasoning: a new frontier in ai. External Links: 2412.16075, [Link](https://arxiv.org/abs/2412.16075) Cited by: [§4.3.1](#S4.SS3.SSS1.Px2.p2.1), [§4.3.1](#S4.SS3.SSS1.p1.1), [§4.3.1](#S4.SS3.SSS1.p2.1). - L. Yang, Z. Yu, B. Cui, and M. Wang (2025d) ReasonFlux: hierarchical llm reasoning via scaling thought templates. External Links: 2502.06772, [Link](https://arxiv.org/abs/2502.06772) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p2.1). - R. Yang, Y. Zhang, A. Chen, X. Wang, S. Yuan, J. Chen, D. Yang, and Y. Xiao (2025e) ARIA: training language agents with intention-driven reward aggregation. External Links: 2506.00539, [Link](https://arxiv.org/abs/2506.00539) Cited by: [§4.8](#S4.SS8.SSS0.Px1.p1.1). - S. Yang, J. He-Yueya, and P. Liang (2025f) Reinforcement learning for machine learning engineering agents. External Links: 2509.01684, [Link](https://arxiv.org/abs/2509.01684) Cited by: [§4.2.1](#S4.SS2.SSS1.Px2.p1.1). - W. Yang, S. Ma, Y. Lin, and F. Wei (2025g) Towards thinking-optimal scaling of test-time compute for llm reasoning. External Links: 2502.18080, [Link](https://arxiv.org/abs/2502.18080) Cited by: [§3.5](#S3.SS5.SSS0.Px3.p1.1). - Z. Yang, Z. Guo, Y. Huang, Y. Wang, D. Xie, Y. Wang, X. Liang, and J. Tang (2025h) Depth-breadth synergy in rlvr: unlocking llm reasoning gains with adaptive exploration. arXiv preprint arXiv:2508.13755. Cited by: [Table 2](#S2.T2.1.35.34.1.1.1). - S. Yao, H. Chen, J. Yang, and K. Narasimhan (2022) WebShop: towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, pp. 20744–20757. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/82ad13ec01f9fe44c01cb91814fd7b8c-Paper-Conference.pdf) Cited by: [§5.1.1](#S5.SS1.SSS1.p1.1), [Table 10](#S5.T10.1.1.12.11.1). - S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. R. Narasimhan (2023a) Tree of thoughts: deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: [Link](https://openreview.net/forum?id=5Xc1ecxO1h) Cited by: [§3.5](#S3.SS5.SSS0.Px1.p2.1), [§3.5](#S3.SS5.SSS0.Px2.p1.1). - S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao (2023b) ReAct: synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=WE_vluYUL-X) Cited by: [§3.1](#S3.SS1.p1.1.1), [§3.2](#S3.SS2.SSS0.Px1.p1.1). - Z. Yao, Y. Liu, Y. Chen, J. Chen, J. Fang, L. Hou, J. Li, and T. Chua (2025) Are reasoning models more prone to hallucination?. External Links: 2505.23646, [Link](https://arxiv.org/abs/2505.23646) Cited by: [§6.1](#S6.SS1.SSS0.Px2.p2.1). - J. Ye, C. Jiang, Z. Du, Y. Xu, X. Yao, Z. Xi, X. Fan, Q. Zhang, X. Huang, and J. Chen (2025a) Feedback-driven tool-use improvements in large language models via automated build environments. External Links: 2508.08791, [Link](https://arxiv.org/abs/2508.08791) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1). - J. Ye, F. Liu, Q. Li, Z. Wang, Y. Wang, X. Wang, Y. Duan, and J. Zhu (2024) Dreamreward: text-to-3d generation with human preference. In European Conference on Computer Vision, pp. 259–276. External Links: [Link](https://icml.cc/virtual/2025/poster/45024) Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - Y. Ye, T. Zhang, W. Jiang, and H. Huang (2025b) Process-supervised reinforcement learning for code generation. External Links: 2502.01715, [Link](https://arxiv.org/abs/2502.01715) Cited by: [§4.2.1](#S4.SS2.SSS1.Px2.p1.1), [Table 5](#S4.T5.1.1.12.11.1). - F. Yin, X. Chen, C. Zhang, B. Jiang, Z. Zhao, W. Liu, G. Yu, and T. Chen (2025) Shapegpt: 3d shape generation with a unified multi-modal language model. IEEE Transactions on Multimedia. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - H. Ying, Z. Wu, Y. Geng, Z. Yuan, D. Lin, and K. Chen (2025) Lean workbook: a large-scale lean problem set formalized from natural language math problems. External Links: 2406.03847, [Link](https://arxiv.org/abs/2406.03847) Cited by: [§4.3.2](#S4.SS3.SSS2.Px1.p1.1), [§4.3.2](#S4.SS3.SSS2.Px3.p3.1). - C. Yu, S. Lu, C. Zhuang, D. Wang, Q. Wu, Z. Li, R. Gan, C. Wang, S. Hou, G. Huang, W. Yan, L. Hong, A. Xue, Y. Wang, J. Gu, D. Tsai, and T. Lin (2025a) AWorld: orchestrating the training recipe for agentic ai. External Links: 2508.20404, [Link](https://arxiv.org/abs/2508.20404) Cited by: [Table 4](#S4.T4.4.1.19.19.1), [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.10.10.1.1.1). - H. Yu, F. Li, and J. You (2025b) LiveTradeBench: seeking real-world alpha with large language models. External Links: 2511.03628, [Link](https://arxiv.org/abs/2511.03628) Cited by: [§6.6](#S6.SS6.SSS0.Px5.p1.1). - H. Yu, Z. Qi, Y. Zhao, K. Nottingham, K. Xuan, B. P. Majumder, H. Zhu, P. P. Liang, and J. You (2025c) Sotopia-rl: reward design for social intelligence. External Links: 2508.03905, [Link](https://arxiv.org/abs/2508.03905) Cited by: [§4.8](#S4.SS8.SSS0.Px5.p1.1). - H. Yu, T. Chen, J. Feng, J. Chen, W. Dai, Q. Yu, Y. Zhang, W. Ma, J. Liu, M. Wang, and H. Zhou (2025d) MemAgent: reshaping long-context llm with multi-conv rl-based memory agent. External Links: 2507.02259, [Link](https://arxiv.org/abs/2507.02259) Cited by: [§3.3](#S3.SS3.SSS0.Px2.p1.1), [Table 3](#S3.T3.7.5.5.1.1). - Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2025e) Dapo: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. External Links: [Link](https://arxiv.org/abs/2503.14476) Cited by: [Table 2](#S2.T2.1.20.19.1.1.1), [§3.5](#S3.SS5.SSS0.Px2.p1.1), [§3.5](#S3.SS5.SSS0.Px2.p2.1), [§4.3](#S4.SS3.p2.1). - H. Yuan, S. Zhang, X. Wang, Y. Wei, T. Feng, Y. Pan, Y. Zhang, Z. Liu, S. Albanie, and D. Ni (2024) Instructvideo: instructing video diffusion models with human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6463–6474. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - S. Yuan, Y. Liu, Y. Yue, J. Zhang, W. Zuo, Q. Wang, F. Zhang, and G. Zhou (2025a) AR-grpo: training autoregressive image generation models via reinforcement learning. arXiv preprint arXiv:2508.06924. Cited by: [§4.5](#S4.SS5.SSS0.Px1.p1.1). - W. Yuan, S. Tang, W. Lin, J. Ruan, G. Cui, B. Zhang, T. Chen, T. Liu, Y. Fu, P. Ye, and L. Bai (2025b) Wisdom of the crowd: reinforcement learning from coevolutionary collective feedback. External Links: 2508.12338, [Link](https://arxiv.org/abs/2508.12338) Cited by: [§4.7.3](#S4.SS7.SSS3.p1.1), [Table 8](#S4.T8.3.24.21.1.1.1). - X. Yuan, M. M. Moss, C. E. Feghali, C. Singh, D. Moldavskaya, D. MacPhee, L. Caccia, M. Pereira, M. Kim, A. Sordoni, and M. Côté (2025c) Debug-gym: a text-based environment for interactive debugging. External Links: 2503.21557, [Link](https://arxiv.org/abs/2503.21557) Cited by: [§5.1.3](#S5.SS1.SSS3.Px1.p1.1), [Table 10](#S5.T10.1.1.21.20.1). - Z. Yuan, S. Jiang, C. Feng, Y. Zhang, S. Cui, Z. Li, and N. Zhao (2025d) Scene-r1: video-grounded large language models for 3d scene reasoning without 3d annotations. arXiv preprint arXiv:2506.17545. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang (2025a) Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. External Links: 2504.13837, [Link](https://arxiv.org/abs/2504.13837) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p1.1), [§6.4](#S6.SS4.SSS0.Px1.p1.1), [§6.4](#S6.SS4.p1.1). - Y. Yue, Y. Yuan, Q. Yu, X. Zuo, R. Zhu, W. Xu, J. Chen, C. Wang, T. Fan, Z. Du, et al. (2025b) Vapo: efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118. External Links: [Link](https://arxiv.org/abs/2504.05118) Cited by: [§2.7](#S2.SS7.SSS0.Px2.p1.6), [Table 2](#S2.T2.1.5.4.1.1.1). - A. Zala, J. Cho, H. Lin, J. Yoon, and M. Bansal (2024) EnvGen: generating and adapting environments via LLMs for training embodied agents. In First Conference on Language Modeling, External Links: [Link](https://openreview.net/forum?id=F9tqgOPXH5) Cited by: [§6.3](#S6.SS3.p2.1). - E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman (2022) STaR: self-taught reasoner bootstrapping reasoning with reasoning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. External Links: ISBN 9781713871088, [Link](https://dl.acm.org/doi/10.5555/3600270.3601396) Cited by: [§4.3.2](#S4.SS3.SSS2.Px3.p2.1). - A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang (2024) AgentTuning: enabling generalized agent abilities for LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 3053–3077. External Links: [Link](https://aclanthology.org/2024.findings-acl.181/), [Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.181) Cited by: [§3.2](#S3.SS2.SSS0.Px1.p1.1). - G. Zeng, M. Shen, D. Chen, Z. Qi, S. Das, D. Gutfreund, D. Cox, G. Wornell, W. Lu, Z. Hong, and C. Gan (2025a) Satori-swe: evolutionary test-time scaling for sample-efficient software engineering. External Links: 2505.23604, [Link](https://arxiv.org/abs/2505.23604) Cited by: [§4.2.3](#S4.SS2.SSS3.Px1.p1.1), [Table 5](#S4.T5.1.1.30.29.1). - H. Zeng, D. Jiang, H. Wang, P. Nie, X. Chen, and W. Chen (2025b) ACECODER: acing coder RL via automated test-case synthesis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 12023–12040. External Links: [Link](https://aclanthology.org/2025.acl-long.587/), [Document](https://dx.doi.org/10.18653/v1/2025.acl-long.587), ISBN 979-8-89176-251-0 Cited by: [§4.2.1](#S4.SS2.SSS1.Px1.p1.1), [Table 5](#S4.T5.1.1.4.3.1). - L. Zeng and L. Zhong (2024) Skywork-math: data scaling laws for mathematical reasoning in LLMs — the story goes on. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24, External Links: [Link](https://openreview.net/forum?id=uHtzqZKbeK) Cited by: [§4.3.2](#S4.SS3.SSS2.Px3.p1.1). - W. Zeng, Y. Huang, Q. Liu, W. Liu, K. He, Z. MA, and J. He (2025c) SimpleRL-zoo: investigating and taming zero reinforcement learning for open base models in the wild. In Second Conference on Language Modeling, External Links: [Link](https://openreview.net/forum?id=vSMCBUgrQj) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p1.1), [§3.5](#S3.SS5.SSS0.Px2.p2.1). - Y. Zhan, Z. Wu, Y. Zhu, R. Xue, R. Luo, Z. Chen, C. Zhang, Y. Li, Z. He, Z. Yang, et al. (2025) GThinker: towards general multimodal reasoning via cue-guided rethinking. arXiv preprint arXiv:2506.01078. Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - B. Zhang, Z. Shang, Z. Gao, W. Zhang, R. Xie, X. Ma, T. Yuan, X. Wu, S. Zhu, and Q. Li (2025a) TongUI: building generalized gui agents by learning from multimodal web tutorials. arXiv preprint arXiv:2504.12679. External Links: [Link](https://arxiv.org/abs/2504.12679) Cited by: [§4.4.1](#S4.SS4.SSS1.Px2.p1.1), [Table 7](#S4.T7.3.3.2). - C. Zhang, S. He, J. Qian, B. Li, L. Li, S. Qin, Y. Kang, M. Ma, G. Liu, Q. Lin, S. Rajmohan, D. Zhang, and Q. Zhang (2025b) Large language model-brained GUI agents: a survey. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, [Link](https://openreview.net/forum?id=xChvYjvXTp) Cited by: [§4.4.1](#S4.SS4.SSS1.Px1.p1.1). - C. Zhang, W. Shen, L. Zhao, X. Zhang, X. Xu, W. Dou, and J. Bian (2025c) Policy filtration for RLHF to mitigate noise in reward models. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=L8hYdTQVcs) Cited by: [Table 2](#S2.T2.1.7.6.1.1.1), [§4.2.1](#S4.SS2.SSS1.Px2.p1.1), [Table 5](#S4.T5.1.1.15.14.1). - D. Zhang, S. Zhoubian, Z. Hu, Y. Yue, Y. Dong, and J. Tang (2024a) ReST-mcts*: llm self-training via process reward guided tree search. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 64735–64772. External Links: [Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/76ec4dc30e9faaf0e4b6093eaa377218-Paper-Conference.pdf) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p1.1). - G. Zhang, M. Fu, G. Wan, M. Yu, K. Wang, and S. Yan (2025d) G-memory: tracing hierarchical memory for multi-agent systems. External Links: 2506.07398, [Link](https://arxiv.org/abs/2506.07398) Cited by: [§3.3](#S3.SS3.SSS0.Px3.p1.1), [Table 3](#S3.T3.11.9.25.16.1). - G. Zhang, M. Fu, and S. Yan (2025e) MemGen: weaving generative latent memory for self-evolving agents. External Links: 2509.24704, [Link](https://arxiv.org/abs/2509.24704) Cited by: [§3.3](#S3.SS3.SSS0.Px2.p1.1), [Table 3](#S3.T3.11.9.9.1.1). - G. Zhang, L. Niu, J. Fang, K. Wang, L. BAI, and X. Wang (2025f) Multi-agent architecture search via agentic supernet. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=imcyVlzpXh) Cited by: [§4.7.1](#S4.SS7.SSS1.p1.1), [Table 8](#S4.T8.3.12.9.1.1.1). - G. Zhang, Y. Yue, X. Sun, G. Wan, M. Yu, J. Fang, K. Wang, T. Chen, and D. Cheng (2025g) G-designer: architecting multi-agent communication topologies via graph neural networks. In Forty-second International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=LpE54NUnmO) Cited by: [§4.7.1](#S4.SS7.SSS1.p1.1), [Table 8](#S4.T8.3.13.10.1.1.1). - H. Zhang, X. Liu, B. Lv, X. Sun, B. Jing, I. L. Iong, Z. Hou, Z. Qi, H. Lai, Y. Xu, R. Lu, H. Wang, J. Tang, and Y. Dong (2025h) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.13.13.1.1.1). - J. Zhang, S. Hu, C. Lu, R. Lange, and J. Clune (2025i) Darwin godel machine: open-ended evolution of self-improving agents. arXiv preprint arXiv:2505.22954. Cited by: [§4.7](#S4.SS7.SSS0.Px1.p1.1). - J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y. Luo, and C. Wu (2025j) AFlow: automating agentic workflow generation. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=z5uVAKwmjf) Cited by: [§4.7](#S4.SS7.SSS0.Px1.p1.1), [Table 8](#S4.T8.3.9.6.1.1.1). - J. Zhang, J. Huang, H. Yao, S. Liu, X. Zhang, S. Lu, and D. Tao (2025k) R1-vl: learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937. Cited by: [Table 2](#S2.T2.1.26.25.1.1.1). - J. Zhang, Q. Wang, X. Ji, Y. Liu, Y. Yue, F. Zhang, D. Zhang, G. Zhou, and K. Gai (2025l) Leanabell-prover: posttraining scaling in formal reasoning. External Links: 2504.06122, [Link](https://arxiv.org/abs/2504.06122) Cited by: [§4.3.2](#S4.SS3.SSS2.Px1.p1.1), [Table 6](#S4.T6.6.6.2.1.1). - J. Zhang, J. Xi, Z. Song, J. Lu, Y. Ke, T. Sun, Y. Yang, J. Zhang, S. Zhang, and Z. Xie (2025m) L0: reinforcement learning to become general agents. External Links: 2506.23667, [Link](https://arxiv.org/abs/2506.23667) Cited by: [§4.8](#S4.SS8.SSS0.Px4.p1.1). - J. Zhang, L. Feng, X. Guo, Y. Wu, Y. Dong, and D. Xu (2025n) TimeMaster: training time-series multimodal llms to reason via reinforcement learning. External Links: 2506.13705, [Link](https://arxiv.org/abs/2506.13705) Cited by: [§4.8](#S4.SS8.SSS0.Px3.p1.1). - K. Zhang, R. Liu, X. Zhu, K. Tian, S. Zeng, G. Jia, Y. Fan, X. Lv, Y. Zuo, C. Jiang, Z. Liu, J. Wang, Y. Wang, R. Zhao, E. Hua, Y. Wang, S. Wang, J. Gao, X. Long, Y. Sun, Z. Ma, G. Cui, L. Bai, N. Ding, B. Qi, and B. Zhou (2025o) MARTI: a framework for multi-agent llm systems reinforced training and inference. Tsinghua University and Shanghai AI Lab. External Links: [Link](https://github.com/TsinghuaC3I/MARTI) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.6.6.1.1.1). - K. Zhang, G. Li, J. Li, Y. Dong, J. Li, and Z. Jin (2025p) Focused-DPO: enhancing code generation through focused preference optimization on error-prone points. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 9578–9591. External Links: [Link](https://aclanthology.org/2025.findings-acl.498/), [Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.498), ISBN 979-8-89176-256-5 Cited by: [§4.2.1](#S4.SS2.SSS1.Px2.p1.1), [Table 5](#S4.T5.1.1.17.16.1). - W. Zhang, Y. Feng, H. Luo, Y. Li, Z. Yue, S. Zheng, and Z. Lu (2025q) Unified multimodal understanding via byte-pair visual encoding. arXiv preprint arXiv:2506.23639. Cited by: [§3.6](#S3.SS6.p1.1). - W. Zhang, Z. Xie, Y. Feng, Y. Li, X. Xing, S. Zheng, and Z. Lu (2024b) From pixels to tokens: byte-pair encoding on quantized visual modalities. arXiv preprint arXiv:2410.02155. Cited by: [§3.6](#S3.SS6.p1.1). - W. Zhang, Y. Xie, Y. Sun, Y. Chen, G. Wang, Y. Li, B. Ding, and J. Zhou (2025r) On-policy rl meets off-policy experts: harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508.11408. Cited by: [Table 2](#S2.T2.1.36.35.1.1.1). - W. Zhang, C. Cui, Y. Zhao, R. Hu, Y. Liu, Y. Zhou, and B. An (2025s) AgentOrchestra: a hierarchical multi-agent framework for general-purpose task solving. External Links: 2506.12508, [Link](https://arxiv.org/abs/2506.12508) Cited by: [§4.1](#S4.SS1.p1.1), [§6.5](#S6.SS5.SSS0.Px3.p1.1). - X. Zhang, J. Wang, Z. Cheng, W. Zhuang, Z. Lin, M. Zhang, S. Wang, Y. Cui, C. Wang, J. Peng, et al. (2025t) Srpo: a cross-domain implementation of large-scale reinforcement learning on llm. arXiv preprint arXiv:2504.14286. Cited by: [Table 2](#S2.T2.1.27.26.1.1.1). - X. Zhang, S. Wen, W. Wu, and L. Huang (2025u) EDGE-grpo: entropy-driven grpo with guided error correction for advantage diversity. arXiv preprint arXiv:2507.21848. Cited by: [Table 2](#S2.T2.1.34.33.1.1.1). - X. Zhang, S. Wen, W. Wu, and L. Huang (2025v) Tinyllava-video-r1: towards smaller lmms for video reasoning. arXiv preprint arXiv:2504.09641. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - Y. Zhang, S. Mao, T. Ge, X. Wang, Y. Xia, W. Wu, T. Song, M. Lan, and F. Wei (2024c) LLM as a mastermind: a survey of strategic reasoning with large language models. In First Conference on Language Modeling, External Links: [Link](https://openreview.net/forum?id=iMqJsQ4evS) Cited by: [§3](#S3.p1.1). - Y. Zhang, Z. Zeng, D. Li, Y. Huang, Z. Deng, and Y. Dong (2025w) RealSafe-r1: safety-aligned deepseek-r1 without compromising reasoning capability. External Links: 2504.10081, [Link](https://arxiv.org/abs/2504.10081) Cited by: [§6.5](#S6.SS5.SSS0.Px1.p1.1). - Y. Zhang, C. Ma, J. Ge, L. Jiang, J. Xu, and W. Zhang (2025x) HERO: hardware-efficient rl-based optimization framework for nerf quantization. CoRR abs/2510.09010. External Links: [Link](https://doi.org/10.48550/arXiv.2510.09010), [Document](https://dx.doi.org/10.48550/ARXIV.2510.09010), 2510.09010 Cited by: [§6.6](#S6.SS6.SSS0.Px2.p1.1). - Y. Zhang, J. Shu, Y. Ma, X. Lin, S. Wu, and J. Sang (2025y) Memory as action: autonomous context curation for long-horizon agentic tasks. External Links: 2510.12635, [Link](https://arxiv.org/abs/2510.12635) Cited by: [§3.3](#S3.SS3.SSS0.Px1.p1.1), [Table 3](#S3.T3.11.9.15.6.1.1). - Y. Zhang, S. Wu, Y. Yang, J. Shu, J. Xiao, C. Kong, and J. Sang (2024d) O1-coder: an o1 replication for coding. arXiv preprint arXiv:2412.00154. External Links: [Link](https://arxiv.org/abs/2412.00154) Cited by: [§4.2.1](#S4.SS2.SSS1.Px2.p1.1), [Table 5](#S4.T5.1.1.13.12.1). - Y. Zhang, Y. Yang, J. Shu, X. Wen, and J. Sang (2025z) Agent models: internalizing chain-of-action generation into reasoning models. External Links: 2503.06580, [Link](https://arxiv.org/abs/2503.06580) Cited by: [§4.8](#S4.SS8.SSS0.Px4.p1.1). - Z. Zhang, X. He, W. Yan, A. Shen, C. Zhao, S. Wang, Y. Shen, and X. E. Wang (2025aa) Soft thinking: unlocking the reasoning potential of llms in continuous concept space. External Links: 2505.15778, [Link](https://arxiv.org/abs/2505.15778) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p2.1). - Z. Zhang, Y. Lu, Y. Fu, Y. Huo, S. Yang, Y. Wu, H. Si, X. Cong, H. Chen, Y. Lin, J. Xie, W. Zhou, W. Xu, Y. Zhang, Z. Su, Z. Zhai, X. Liu, Y. Mei, J. Xu, H. Tian, C. Wang, C. Chen, Y. Yao, Z. Liu, and M. Sun (2025ab) AgentCPM-gui: building mobile-use agents with reinforcement fine-tuning. External Links: 2506.01391, [Link](https://arxiv.org/abs/2506.01391) Cited by: [§4.4.2](#S4.SS4.SSS2.p1.1), [Table 7](#S4.T7.7.7.2). - Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smola (2023) Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923. Cited by: [§3.6](#S3.SS6.p1.1). - Z. Zhang, Z. Chen, M. Li, Z. Tu, and X. Li (2025ac) RLVMR: reinforcement learning with verifiable meta-reasoning rewards for robust long-horizon agents. External Links: 2507.22844, [Link](https://arxiv.org/abs/2507.22844) Cited by: [§3.7](#S3.SS7.p2.1). - A. Zhao, Y. Wu, Y. Yue, T. Wu, Q. Xu, Y. Yue, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang (2025a) Absolute zero: reinforced self-play reasoning with zero data. External Links: 2505.03335, [Link](https://arxiv.org/abs/2505.03335) Cited by: [§3.4](#S3.SS4.SSS0.Px3.p1.1), [§4.2.1](#S4.SS2.SSS1.Px1.p1.1), [Table 5](#S4.T5.1.1.8.7.1). - B. Zhao, Z. Wang, J. Fang, C. Gao, F. Man, J. Cui, X. Wang, X. Chen, Y. Li, and W. Zhu (2025b) Embodied-r: collaborative framework for activating embodied spatial reasoning in foundation models via reinforcement learning. arXiv preprint arXiv:2504.12680. Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - H. Zhao, W. Song, D. Wang, X. Tong, P. Ding, X. Cheng, and Z. Ge (2025c) More: unlocking scalability in reinforcement learning for quadruped vision-language-action models. arXiv preprint arXiv:2503.08007. Cited by: [§4.6](#S4.SS6.SSS0.Px1.p1.1). - S. Zhao, H. Zhang, S. Lin, M. Li, Q. Wu, K. Zhang, and C. Wei (2025d) PyVision: agentic vision with dynamic tooling. arXiv preprint arXiv:2507.07998. Cited by: [§3.6](#S3.SS6.SSS0.Px3.p1.1), [§4.5](#S4.SS5.SSS0.Px1.p1.1). - W. Zhao, M. Yuksekgonul, S. Wu, and J. Zou (2025e) SiriuS: self-improving multi-agent systems via bootstrapped reasoning. In Workshop on Reasoning and Planning for Large Language Models, External Links: [Link](https://openreview.net/forum?id=sLBSJr3hH5) Cited by: [§3.4](#S3.SS4.SSS0.Px3.p1.1). - Y. Zhao, H. Yin, B. Zeng, H. Wang, T. Shi, C. Lyu, L. Wang, W. Luo, and K. Zhang (2024) Marco-o1: towards open reasoning models for open-ended solutions. External Links: 2411.14405, [Link](https://arxiv.org/abs/2411.14405) Cited by: [§3.5](#S3.SS5.SSS0.Px2.p2.1). - Y. Zhao, Y. Liu, J. Liu, J. Chen, X. Wu, Y. Hao, T. Lv, S. Huang, L. Cui, Q. Ye, et al. (2025f) Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673. Cited by: [Table 2](#S2.T2.1.22.21.1.1.1). - B. Zheng, B. Gou, J. Kil, H. Sun, and Y. Su (2024) GPT-4V(ision) is a generalist web agent, if grounded. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, pp. 61349–61385. External Links: [Link](https://proceedings.mlr.press/v235/zheng24e.html) Cited by: [§4.4.1](#S4.SS4.SSS1.Px1.p1.1), [Table 7](#S4.T7.9.13.4.1). - C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. (2025a) Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: [Table 2](#S2.T2.1.21.20.1.1.1). - H. Zheng, Y. Zhou, B. R. Bartoldson, B. Kailkhura, F. Lai, J. Zhao, and B. Chen (2025b) Act only when it pays: efficient reinforcement learning for llm reasoning via selective rollouts. External Links: 2506.02177, [Link](https://arxiv.org/abs/2506.02177) Cited by: [Table 2](#S2.T2.1.28.27.1.1.1), [§4.3](#S4.SS3.p2.1). - J. Zheng, X. Ji, Y. Lu, C. Cui, W. Zhao, G. Deng, Z. Liang, A. Zhang, and T. Chua (2025c) RSafe: incentivizing proactive reasoning to build robust and adaptive llm safeguards. External Links: 2506.07736, [Link](https://arxiv.org/abs/2506.07736) Cited by: [§6.5](#S6.SS5.SSS0.Px1.p1.1). - K. Zheng, J. M. Han, and S. Polu (2022) MiniF2F: a cross-system benchmark for formal olympiad-level mathematics. External Links: 2109.00110, [Link](https://arxiv.org/abs/2109.00110) Cited by: [§4.3.2](#S4.SS3.SSS2.Px1.p1.1), [§4.3.2](#S4.SS3.SSS2.Px3.p3.1). - Y. Zheng, J. Lu, S. Wang, Z. Feng, D. Kuang, and Y. Xiong (2025d) EasyR1: an efficient, scalable, multi-modality rl training framework. Note: [https://github.com/hiyouga/EasyR1](https://github.com/hiyouga/EasyR1) Cited by: [§5.2](#S5.SS2.SSS0.Px1.p1.1), [Table 11](#S5.T11.4.7.7.1.1.1). - Y. Zheng, D. Fu, X. Hu, X. Cai, L. Ye, P. Lu, and P. Liu (2025e) DeepResearcher: scaling deep research via reinforcement learning in real-world environments. External Links: 2504.03160, [Link](https://arxiv.org/abs/2504.03160) Cited by: [§4.1.1](#S4.SS1.SSS1.Px1.p1.1), [Table 4](#S4.T4.4.1.9.9.1). - Z. Zheng, T. Cui, C. Xie, J. Zhang, J. Pan, L. He, and Q. Chen (2025f) NatureGAIA: pushing the frontiers of gui agents with a challenging benchmark and high-quality trajectory dataset. External Links: 2508.01330, [Link](https://arxiv.org/abs/2508.01330) Cited by: [§6.3](#S6.SS3.p1.1). - Z. Zheng, M. Yang, J. Hong, C. Zhao, G. Xu, L. Yang, C. Shen, and X. Yu (2025g) DeepEyes: incentivizing "thinking with images" via reinforcement learning. External Links: 2505.14362, [Link](https://arxiv.org/abs/2505.14362) Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.1), [§3.6](#S3.SS6.SSS0.Px2.p2.1). - J. Zhong, W. Shen, Y. Li, S. Gao, H. Lu, Y. Chen, Y. Zhang, W. Zhou, J. Gu, and L. Zou (2025) A comprehensive survey of reward models: taxonomy, applications, challenges, and future. External Links: 2504.12328, [Link](https://arxiv.org/abs/2504.12328) Cited by: [§2.5](#S2.SS5.SSS0.Px1.p1.2). - W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang (2024) MemoryBank: enhancing large language models with long-term memory. Proceedings of the AAAI Conference on Artificial Intelligence 38 (17), pp. 19724–19731. External Links: [Link](https://ojs.aaai.org/index.php/AAAI/article/view/29946), [Document](https://dx.doi.org/10.1609/aaai.v38i17.29946) Cited by: [§3.3](#S3.SS3.SSS0.Px1.p1.1), [Table 3](#S3.T3.11.9.12.3.1). - A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y. Wang (2024a) Language agent tree search unifies reasoning, acting, and planning in language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: [§3.1](#S3.SS1.SSS0.Px1.p1.1). - G. Zhou, P. Qiu, C. Chen, J. Wang, Z. Yang, J. Xu, and M. Qiu (2025a) Reinforced mllm: a survey on rl-based reasoning in multimodal large language models. arXiv preprint arXiv:2504.21277. Cited by: [§3.6](#S3.SS6.p1.1), [§4.5](#S4.SS5.p1.1). - H. Zhou, H. Geng, X. Xue, L. Kang, Y. Qin, Z. Wang, Z. Yin, and L. Bai (2025b) Reso: a reward-driven self-organizing llm-based multi-agent system for reasoning tasks. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. External Links: [Link](https://openreview.net/forum?id=te0jBwgBRm) Cited by: [§4.7](#S4.SS7.SSS0.Px1.p1.1). - H. Zhou, A. Yu, Y. Fan, J. Shi, L. Kang, H. Geng, Y. Zhang, Y. Fan, Y. Wu, T. He, Y. Qin, L. Bai, and Z. Yin (2025c) LiveSearchBench: an automatically constructed benchmark for retrieval and reasoning over dynamic knowledge. External Links: 2511.01409, [Link](https://arxiv.org/abs/2511.01409) Cited by: [§6.6](#S6.SS6.SSS0.Px5.p1.1). - H. Zhou, X. Li, R. Wang, M. Cheng, T. Zhou, and C. Hsieh (2025d) R1-zero’s "aha moment" in visual reasoning on a 2b non-sft model. External Links: 2503.05132, [Link](https://arxiv.org/abs/2503.05132) Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - J. Zhou, J. Wang, B. Ma, Y. Liu, T. Huang, and X. Wang (2023) Uni3d: exploring unified 3d representation at scale. arXiv preprint arXiv:2310.06773. Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig (2024b) WebArena: a realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=oKn9c6ytLx) Cited by: [§5.1.1](#S5.SS1.SSS1.p1.1), [Table 10](#S5.T10.1.1.14.13.1), [§6.6](#S6.SS6.SSS0.Px3.p1.1). - W. Zhou, Y. Ou, S. Ding, L. Li, J. Wu, T. Wang, J. Chen, S. Wang, X. Xu, N. Zhang, H. Chen, and Y. E. Jiang (2024c) Symbolic learning enables self-evolving agents. External Links: 2406.18532, [Link](https://arxiv.org/abs/2406.18532) Cited by: [§4.7](#S4.SS7.SSS0.Px1.p1.1). - Y. Zhou, S. Jiang, Y. Tian, J. Weston, S. Levine, S. Sukhbaatar, and X. Li (2025e) SWEET-rl: training multi-turn llm agents on collaborative reasoning tasks. External Links: 2503.15478, [Link](https://arxiv.org/abs/2503.15478) Cited by: [§3.4](#S3.SS4.SSS0.Px2.p1.1), [§4.2.3](#S4.SS2.SSS3.Px2.p1.1), [Table 5](#S4.T5.1.1.36.35.1), [§5.1.3](#S5.SS1.SSS3.Px2.p1.1), [Table 10](#S5.T10.1.1.34.33.1). - Z. Zhou, X. Xia, F. Ma, H. Fan, Y. Yang, and T. Chua (2025f) Dreamdpo: aligning text-to-3d generation with human preferences via direct preference optimization. arXiv preprint arXiv:2502.04370. External Links: [Link](https://arxiv.org/abs/2502.04370) Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - Z. Zhou, A. Qu, Z. Wu, S. Kim, A. Prakash, D. Rus, J. Zhao, B. K. H. Low, and P. P. Liang (2025g) MEM1: learning to synergize memory and reasoning for efficient long-horizon agents. External Links: 2506.15841, [Link](https://arxiv.org/abs/2506.15841) Cited by: [§3.3](#S3.SS3.SSS0.Px2.p1.1), [Table 3](#S3.T3.8.6.6.1.1). - B. Zhu, Y. Jiang, B. Xu, S. Yang, M. Yin, Y. Wu, H. Sun, and Z. Wu (2025a) Aligning anime video generation with human feedback. External Links: 2504.10044, [Link](https://arxiv.org/abs/2504.10044) Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - L. Zhu, Y. Guan, D. Liang, J. Ju, Z. Luo, B. Qin, J. Luan, Y. Liu, and X. Bai (2025b) Shuffle-r1: efficient rl framework for multimodal large language models via data-centric dynamic shuffle. External Links: 2508.05612, [Link](https://arxiv.org/abs/2508.05612) Cited by: [§3.6](#S3.SS6.SSS0.Px1.p1.1). - L. Zhu, Q. Chen, X. Shen, and X. Cun (2025c) VAU-r1: advancing video anomaly understanding via reinforcement fine-tuning. External Links: 2505.23504, [Link](https://arxiv.org/abs/2505.23504) Cited by: [§4.5](#S4.SS5.SSS0.Px2.p1.1). - S. Zhu, Y. Jiang, H. Sang, S. Tang, Q. Song, B. He, R. Jain, Z. Wang, and A. Geramifard (2025d) Planner-r1: reward shaping enables efficient agentic rl with smaller llms. External Links: 2509.25779, [Link](https://arxiv.org/abs/2509.25779) Cited by: [§3.1](#S3.SS1.SSS0.Px2.p1.1). - M. Zhuge, W. Wang, L. Kirsch, F. Faccio, D. Khizbullin, and J. Schmidhuber (2024) GPTSwarm: language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, External Links: [Link](https://openreview.net/forum?id=uTC9AFXIhg) Cited by: [§4.7](#S4.SS7.SSS0.Px1.p1.1), [§4.7.1](#S4.SS7.SSS1.p1.1), [Table 8](#S4.T8.3.11.8.1.1.1). - T. Y. Zhuo, V. M. Chien, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. GONG, J. Hoang, A. R. Zebaze, X. Hong, W. Li, J. Kaddour, M. Xu, Z. Zhang, P. Yadav, N. Jain, A. Gu, Z. Cheng, J. Liu, Q. Liu, Z. Wang, D. Lo, B. Hui, N. Muennighoff, D. Fried, X. Du, H. de Vries, and L. V. Werra (2025) BigCodeBench: benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=YrycTjllL0) Cited by: [§5.1.3](#S5.SS1.SSS3.Px2.p1.1), [Table 10](#S5.T10.1.1.27.26.1). - X. Zou, R. Xia, H. Wang, and P. Zhou (2025) DreamCS: geometry-aware text-to-3d generation with unpaired 3d reward supervision. arXiv preprint arXiv:2506.09814. External Links: [Link](https://arxiv.org/abs/2506.09814) Cited by: [§4.5](#S4.SS5.SSS0.Px3.p1.1). - Y. Zuo, K. Zhang, L. Sheng, S. Qu, G. Cui, X. Zhu, H. Li, Y. Zhang, X. Long, E. Hua, B. Qi, Y. Sun, Z. Ma, L. Yuan, N. Ding, and B. Zhou (2025) TTRL: test-time reinforcement learning. External Links: 2504.16084, [Link](https://arxiv.org/abs/2504.16084) Cited by: [§3.4](#S3.SS4.SSS0.Px3.p1.1), [§4.3.1](#S4.SS3.SSS1.Px1.p1.1), [Table 6](#S4.T6.10.14.4.1.1.1).