Title: 2511.16108 ArXiv: 2511.16108 SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent Shiyi Cao §∗ , Dacheng Li § , Fangzhou Zhao § † {}^{\S^{\dagger}} , Shuo Yuan § † {}^{\S^{\dagger}} , Sumanth R Hegde ¶ † {}^{\P^{\dagger}} , Connor Chen § † {}^{\S^{\dagger}} , Charlie Ruan § † {}^{\S^{\dagger}} , Tyler Griggs § , Shu Liu § , Eric Tang ¶ , Richard Liaw ¶ , Philipp Moritz ¶ , Matei Zaharia § , Joseph E. Gonzalez § , Ion Stoica § NovaSky AI, § UC Berkeley, ¶ Anyscale GitHub: https://github.com/NovaSky-AI/SkyRL HuggingFace: https://huggingface.co/NovaSky-AI/SA-SWE-32B Co-lead the project. † Main Contributors. Abstract We introduce SkyRL-Agent , a framework for efficient, multi-turn, long-horizon agent training and evaluation. It provides efficient asynchronous dispatching, lightweight tool integration, and flexible backend interoperability, enabling seamless use with existing RL frameworks such as SkyRL-train, VeRL, and Tinker. Using SkyRL-Agent , we train SA-SWE-32B, a SWE agent trained from Qwen3-32B (24.4% Pass@1) purely with RL. We introduce two key components: an optimized asynchronous pipeline dispatcher that achieves a 1.55 × \times speedup over naive asynchronous batching, and a tool-enhanced training recipe leveraging an AST-based search tool to facilitate code navigation, boost rollout Pass@K, and improve training efficiency. Together, these optimizations enable SA-SWE-32B to reach 39.4% Pass@1 on SWE-Bench Verified 1 1 1 Evaluated using a simplified ReAct loop with only the file-editor and bash tools, under 40k context length and 100 max steps. with more than 2 × \times cost reduction than prior models reaching similar performance. Despite being trained solely on SWE tasks, SA-SWE-32B generalizes effectively to other agentic tasks, including Terminal-Bench, BrowseComp-Plus, and WebArena. We further demonstrate SkyRL-Agent ’s extensibility through case studies on deep research, computer use, and memory agents, each trained using a different training backend. (a) Training metrics for SA-SWE-32B and baselines. We use the checkpoint at step 125 for final evaluation. Curves for DeepSWE are taken from their wandb logs. (b) GPU utilization during generation for Async Pipeline and Async Batch (Bounded) . Evaluated under batch size 64 and 8 rollouts (512 total) with 2 × \times 8 H100 GPUs for SA-SWE-32B training. Figure 1: Training dynamics and system performance for SA-SWE-32B. (a) compares training metrics across ablations; (b) shows GPU utilization comparison between asynchronous dispatching strategies. 1 Introduction Recent progress in post-training techniques, especially reinforcement learning from verifiable rewards (RLVR) (Guo et al., 2025 ; Hu et al., 2025 ; Wen et al., 2025 ) , has spurred a growing effort to move beyond single-turn language model optimization toward the training of multi-turn, long-horizon, tool-augmented agents (Jin et al., 2025 ; Chen et al., 2017 ; Gao et al., 2025 ; Li et al., 2025b ; c ; Luo et al., 2025b ; Cao et al., 2025 ) . These agents are capable of performing complex, multi-step reasoning and acting across diverse environments such as code repositories, browsers, or operating systems. As tasks become increasingly interactive and open-ended, training such agents efficiently and reliably has emerged as a major challenge for both the reinforcement learning (RL) and systems communities. Recent RL training frameworks such as SkyRL-train (Griggs et al., 2025 ) , AReaL (Fu et al., 2025 ) , VeRL (Sheng et al., 2025 ) , and SLIME (Zhu et al., 2025 ) focus on improving the systems efficiency of large-scale reinforcement learning for language models. These frameworks manage device placement and data flow across heterogeneous clusters, coordinate efficient weight synchronization between inference and training engines, and employ diverse parallelization strategies (e.g., tensor, data, and pipeline parallelism) together with asynchronous or hybrid execution plans to maximize hardware utilization. Tinker (Lab, 2025 ) builds on this ecosystem by offering modular APIs for fine-tuning and sampling, allowing users to experiment with customized training algorithms and data pipelines while abstracting away the complexity of distributed execution. Unfortunately, what is still missing is a modular and performant agentic rollout orchestration layer that can generate, schedule, and evaluate multi-turn agent trajectories at scale. In practice, researchers and practitioners are forced to build ad-hoc solutions for coordinating asynchronous rollouts, managing environment and agent states, and integrating to existing training frameworks. Such fragmented setups result in inefficiencies, brittle execution, and debugging challenges, especially given the inherent instability and fragility of RL training. Specifically, the framework should satisfy the following key properties to enable efficient, scalable, and extensible agent training: 1. Flexible tool and task integration. LLM-based agents rely on a wide range of tools to complete tasks, including stateless utilities (e.g, python interpreters), environment-modifying tools (e.g., file editors), and agent-state-modifying operations (e.g., summarization and history truncation), each with distinct runtime requirements. The framework should allow new tasks and tools to be integrated easily with minimum modifications to the main agent loop. 2. Efficient rollout scheduling. To achieve high hardware utilization and minimize latency, the framework should support fine-grained scheduling rather than treating each rollout as a monolithic job. Intra-rollout scheduling breaks a rollout into stages (e.g., initialization, LLM generation, reward computation) and schedules these stages independently, enabling their heterogeneous CPU- and GPU-bound operations to overlap with stages from other rollouts. Inter-rollout scheduling determines the global dispatch order and prioritization of rollouts so that workloads across CPU and GPU remain balanced over time, preventing hardware idling and reducing overall makespan. Together, these mechanisms improve heterogeneous resource utilization and overall generation throughput. 3. Training-backend agnostic. Developers should be able to specify the agent design, tool execution logic, and training backend independently. Decoupling these components allows a single agent implementation to run unmodified across different frameworks, to flexibly leverage their unique capabilities and seamlessly transition between local deployments and API-based services. SkyRL-Agent builds on these observations by introducing an efficient execution framework for heterogeneous agent tasks. It consists of three key components: (1) a tool-centric task interface that supports dynamic registration of user-defined tools, instruction builders, and verifiers for different tasks, enabling seamless integration of new tasks and tools with minimal code changes; (2) a fine-grained asynchronous dispatcher abstraction that provides a unified interface for designing scheduling policies for a batch of multi-stage rollout jobs; and (3) a backend bridge that connects to RL training systems such as Tinker, VeRL, and SkyRL-train for advantage estimation and policy optimization. To demonstrate the effectiveness of SkyRL-Agent , we use it to train SA-SWE-32B, a software-engineering agent derived from Qwen3-32B (24.4% Pass@1), using pure RL on 4.5K instances from R2E-Gym (Jain et al., 2025 ) . SA-SWE-32B achieves 39.4% Pass@1 on SWE-Bench Verified (Jimenez et al., 2024 ) , matching the performance of state-of-the-art models of similar scale while reducing the total training cost by more than 2 × \times . This efficiency gain stems from two key contributions: an asynchronous pipeline dispatcher that delivers a 1.55 × \times speedup over naive asynchronous batching by overlapping CPU- and GPU-bound operations ( Section 3.2 ), and a tool-enhanced training recipe leveraging an AST-based search tool that facilitates code navigation, leading to higher Pass@K for rollouts and improved sample efficiency ( Section 4.2 ). Although trained exclusively on software-engineering tasks, SA-SWE-32B generalizes effectively to other agentic benchmarks ( Section 4.3 ) such as Terminal-Bench (Team, 2025b ) , BrowseComp-Plus (Chen et al., 2025b ) , and WebArena (Zhou et al., 2023 ) . Beyond the SWE agent, we further demonstrate the versatility of SkyRL-Agent in Section 5 by training several additional agents in smaller scale, including the Deep Research Agent , the Computer Use Agent , and the Memory Agent , each connected to distinct RL backends, showcasing the framework’s flexibility and interoperability. 2 Background Table 1: Comparison of SkyRL-Agent with existing agent training frameworks. SkyRL-Agent offers unified tool interface, efficient rollout scheduling, backend portability, runtime scaling utilities, and dynamic trajectory construction for scalable agent training. Framework Backend Agent Execution Interface Runtime Scaling Trajectory VeRL-Tool (Jiang et al., 2025 ) VeRL Data-parallel Tool Ray Mask rLLM (Tan et al., 2025 ) VeRL Data-parallel Gym K8s Mask / Transition GEM (Liu et al., 2025c ) Multi Data-parallel Gym – Transition Agent-Lightning (Luo et al., 2025b ) Multi Data-parallel – – Transition SkyRL-Agent (Ours) Multi Data + Pipeline (extensible) Tool Ray/K8s Mask / Transition Multi-Turn LLM Agent Training. Reinforcement learning (RL) has shown strong potential for improving single-turn reasoning tasks such as mathematics, logic, and coding (Shao et al., 2024 ; Guo et al., 2025 ; Hu et al., 2025 ; An et al., 2025 ; Yu et al., 2025b ) . In these settings, the model generates a complete response to a single prompt and receives a scalar reward reflecting its correctness or quality. Building on these successes, researchers have further explored tool-integrated reasoning , where the model interacts with external tools such as search engines, calculators, or code executors (Li et al., 2025c ; Jin et al., 2025 ; Chen et al., 2017 ) to enhance its reasoning abilities, showing significant improvement on related reasoning benchmarks. Since tool responses (e.g., retrieved documents or computed results) are interleaved with the model’s own reasoning, most approaches employ a mask-based loss construction , where non-model-generated tokens are masked during optimization to ensure that only generated segments contribute to the policy gradient. Recent works have also applied RL to long-horizon, multi-turn agent training (Cao et al., 2025 ; Luo et al., 2025a ; Gao et al., 2025 ; Lu et al., 2025 ) , enabling optimization of agent scaffolds such as ReAct (Yao et al., 2022 ) , CodeAct (Wang et al., 2024a ) , and MemGPT (Packer et al., 2023 ) beyond supervised learning (Pan et al., 2024 ; Yang et al., 2025a ) . A multi-turn agent can be formulated as a partially observable Markov decision process (POMDP) (Kaelbling et al., 1998 ) , where at each step t t , the agent observes a context o t o_{t} , generates an action a t a_{t} from a policy π θ ​ ( a t | o t ) \pi_{\theta}(a_{t}|o_{t}) , and receives a scalar reward r t r_{t} from the environment. The objective is to maximize the expected cumulative return J ​ ( θ ) = 𝔼 π θ ​ [ ∑ t = 1 T r t ] , J(\theta)=\mathbb{E}_{\pi_{\theta}}\Big[\sum_{t=1}^{T}r_{t}\Big], where T T is the number of agent turns per task. In multi-turn settings, agents often modify their context between steps through summarization, truncation, or selective incorporation of retrieved information, which makes the overall interaction history non-stationary. Such dynamic context management breaks the assumption of a single continuous text sequence, making the conventional mask-based method difficult to generalize beyond short, tool-integrated reasoning tasks. Building on the POMDP formulation, recent work instead adopts a more general transition-based data representation (Tan et al., 2025 ; Luo et al., 2025b ) , where each model invocation and its resulting feedback are represented as a tuple ( o t , a t , r t ) (o_{t},a_{t},r_{t}) . This representation supports flexible reward assignment and decouples learning from the details of agent execution and context management. It also naturally applies to multi-agent or sub-agent systems, where different policies interact within a shared environment. Training Frameworks for LLM Agents. Existing agent training frameworks such as VeRL-Tool (Jiang et al., 2025 ) , rLLM (Tan et al., 2025 ) , GEM (Liu et al., 2025c ) , and Agent-Lightning (Luo et al., 2025b ) have advanced large-scale RL training for LLM agents, yet remain limited by naive asynchronous batching (i.e., data-parallel) execution. SkyRL-Agent supports flexible rollout scheduling in multiple dimensions (e.g., data-parallel + pipeline-parallel). Beyond execution efficiency, SkyRL-Agent unifies stateless tools (e.g., python interpreter), environment-modifying actions (e.g., file editor), and agent-state-modifying operations (e.g., summarization) within a single tool, unlike Gym-centric loops (Tan et al., 2025 ; Liu et al., 2025c ) , which manage agent state outside env.step with ad hoc implementations. As summarized in Tab. 1 , Agent-Lightning serves as a middle layer between training backends and workflow frameworks such as LangChain (LangChain, 2025 ) and AutoGen (AutoGen, 2025 ) , but lacks native support for user-defined tools and new task integration. rLLM and VeRL-Tool provide stronger extensibility, but they are tied to the VeRL (Sheng et al., 2025 ) backend, and VeRL-Tool’s mask-based trajectory construction limits its applicability to settings such as multi-agent systems or memory agents. 3 Framework Architecture Figure 2: Overview Architecture of SkyRL-Agent . The framework decomposes each rollout into three stages: (1) runtime initialization for tool execution runtime setup, (2) agent run where the agent performs actions through the tool interface, and (3) reward calculation for outcome evaluation. During execution, the inputs and outputs of LLM calls are recorded as transitions and stored in a buffer, while post_process aggregates these transitions together with their rewards into formatted data compatible with multiple RL training backends such as SkyRL-train, VeRL, and Tinker. The dispatcher schedules jobs across the three stages according to predefined policies. SkyRL-Agent is designed as a modular framework for training and evaluating tool-use agents at scale. The system architecture is illustrated in Fig. 2 , consisting of three main components: a tool-centric agent loop ( Section 3.1 ) for flexible tool and task integration, a fine-grained dispatcher ( Section 3.2 ) for heterogeneous scheduling, and a backend bridge ( Section 3.3 ) for seamless connection with RL training systems. 3.1 Tool-centric agent loop. In SkyRL-Agent , agents act purely via OpenAI-style function calls. Each tool implements its own execution logic and specifies its runtime. Existing Gym-style environments (Liu et al., 2025c ; Jain et al., 2025 ; Xie et al., 2024 ) , can be integrated by wrapping their step() as a tool, for example, an implementation of tools for the computer use agent is shown in Fig. 3 . This design yields several practical benefits: (i) unified management of agent and environment states : unlike Gym-centric agent loops (Tan et al., 2025 ; Liu et al., 2025c ) , where agent state is managed outside env.step through ad hoc code, SkyRL-Agent brings it under the same tool abstraction as other actions, making agent-state-modifying operations such as context management modular and learnable. (ii) convenient multi‑task training : different datasets bind to different tool sets and verifiers without modifying the agent loop, allowing the runner to multiplex tasks within a single training job; and (iii) minimal‑change task integration : adding a new task reduces to providing a small set of tool implementations together with task‑specific instruction builder and verifiers, leaving agent code unchanged and without touching other parts of the system. ⬇ @register_tool ( "osworld_action" ) class OSWorldActionTool ( BaseTool ): name = "osworld_action" description = ( "Execute desktop automation actions using pyautogui. " "Provide Python code such as ’pyautogui.click(500, 300)’ " "or ’pyautogui.typewrite(\"Hello World\")’." ) parameters = { "type" : "object" , "properties" : { "code" : { "type" : "string" }}, "required" : [ "code" ] } \ pardef call ( self , params : Union [ str , dict ], runtime = None , ** kwargs ) -> str : ... code = params . get ( "code" , "" ). strip () obs , reward , done , info = runtime . step ( code , TIMEOUT ) ... Figure 3: Example definition for desktop tools in the Computer Use agent. 3.2 Fine‑grained heterogeneous scheduling via the dispatcher. Multi‑turn RL rollouts comprise operations with disparate costs and device affinities. As illustrated in Fig. 2 , SkyRL-Agent decomposes each trajectory into stage jobs: 1 runtime initialization , 2 agent run , and 3 reward calculation . The dispatcher maintains bounded queues per stage and routes jobs according to predefined policies, balancing workloads on heterogeneous devices to improve overall resource utilization. Figure 4: Examples of Supported Dispatching Methods. Async Batch is normally used for reasoning tasks where runtime initialization and reward computation are lightweight. Async Batch (Bounded) schedules trajectories sequentially with capped concurrency, leading to unbalanced GPU utilization across stages, but remains effective when runtime reset is inexpensive, such as in computer-use tasks. Async Pipeline overlaps the three stages to maintain high GPU utilization, suitable for tasks with expensive runtime or reward stages. We provide several dispatching strategies under a unified interface, allowing practitioners to conveniently select the policy that best matches the characteristics of their tasks, as illustrated in Fig. 4 . Async Batch launches all trajectories concurrently and is effective when both runtime initialization and reward calculation are lightweight, such as in search-integrated reasoning tasks (Jin et al., 2025 ) . Async Batch (Bounded) limits concurrency to a configurable pool size, which is suitable for environments where concurrency must be restricted to prevent overloading the runtime or where persistent resources (e.g., long-lived virtual machines in computer-use tasks) can be efficiently reused across rollouts. Async Pipeline employs three bounded queues of different, configurable sizes to overlap CPU-bound stages with GPU-bound inference, preventing CPU overhead from throttling the agent loop and overload of runtime services. Existing agent training frameworks typically adopt the asynchronous batching approaches. In our SWE agent training ( Section 4.2 ), the Async Pipeline dispatching strategy achieves approximately a 1.55 × \times speedup over the naive Async Batch (Bounded) approach, as shown in Fig. 1(b) . With the async pipeline method, GPU utilization remains stable at around 90% throughout the generation stage. In contrast, the async batch strategy exhibits large fluctuations and frequent drops in utilization, revealing bubbles on the GPU side caused by CPU-bound operations such as runtime initialization and reward computation. During these intervals, the GPU stays idle while waiting for CPU-side preparation to complete. The asynchronous pipeline approach eliminates these idle periods by overlapping CPU- and GPU-bound stages, keeping the GPU consistently active and achieving significantly higher overall hardware efficiency. ⬇ @register_dispatcher ( "async_batch_bounded" ) async def async_batch_bounded ( cfg , trajs , init_fn , run_fn , eval_fn ): sem = asyncio . Semaphore ( cfg . get ( "max_parallel_agents" , 8)) async def one ( i , j ): async with sem : tr = trajs [ i ][ j ] await getattr ( tr , init_fn )() await getattr ( tr , run_fn )() await getattr ( tr , eval_fn )() await asyncio . gather (*[ asyncio . create_task ( one ( i , j )) for i in trajs for j in trajs [ i ]]) \ par Figure 5: Simplified Implementation for Async Batch (Bounded). An example implementation of Async Batch (Bounded) is provided in Fig. 5 , demonstrating that new dispatching strategies can be added with minimal code under the unified interface. This design makes it easy to implement customized parallelization strategies for different workload characteristics, for example, priority-based scheduling that prioritizes trajectories with higher evaluation cost (e.g., more test cases), preventing them from becoming long-tail stragglers and reducing the overall makespan. 3.3 Connecting to training backends. We adopt a transition-based design that records each LLM invocation as an individual transition containing the input tokens, output tokens, and their corresponding log probabilities (if provided by the inference endpoint). These transitions form the foundation for constructing the final training data and enable several key capabilities for robust RL training. First, it can help mitigate consistency between inference and training. By explicitly recording log probabilities, we can apply recent techniques such as Flash-RL (Liu et al., 2025a ) to correct for inference–training engine mismatch (Yao et al., 2025 ) . Second, it guarantees token-level fidelity. The transition format naturally supports token-in/token-out processing, eliminating off-policy drift caused by re-tokenization or text postprocessing (Team, 2025a ) . Third, it enables algorithmic flexibility. Transition-level data enables a wider range of RL algorithms beyond mask-based concatenation approaches, which are often inefficient and rigid. For example, we can dynamically summarize prior context (see examples in Section 5.2 ), insert structured prompts, or even modify the agent’s roles across turns, thereby enabling more diverse and flexible agent training methods. The recording mechanism is implemented via a lightweight decorator that transparently captures model inputs and outputs during generation, creating transition objects without requiring any modification to agent logic: ⬇ @record_transition async def _generate ( self , input_ids , sampling_params ): return await self . backend . generate ( input_ids , sampling_params ) Additionally, for computational efficiency, we dynamically pack transitions that share the same prefix into a single training sample using a standard masking-based method. As a result, for trajectories without any context modification during execution, the system naturally falls back to the traditional concatenation-and-mask approach. After execution, the runner aggregates the training samples into a unified intermediate format, as shown in Fig. 6 . SkyRL-Agent transparently converts this format into the required input structure for different backends for advantage estimation and policy updates. As a result, switching from local evaluation to distributed training only requires adjusting configuration rather than modifying agent or task code, preserving a clean separation of concerns while enabling backend specialization. ⬇ def post_process (...): return { "prompt_token_ids" : ..., "response_ids" : ..., "logprobs" : ..., "loss_masks" : ..., "traj_rewards" : ..., "traj_idx" : ..., "rollout_metrics" : ..., } Figure 6: Backend-agnostic training data returned by post_process . By abstracting away the training backend, researchers can compare alternative approaches on identical tasks without confounds from differing implementations, accelerating algorithmic progress. Equally, the abstraction preserves access to backend-specific strengths such as advanced model support, novel training algorithms, specialized hardware support (e.g., GPU/TPU/AMD accelerators), and API-based execution for very large models, so practitioners can exploit these capabilities through configuration without changing the code. 3.4 Error Handling Agent execution in realistic environments inevitably encounters various failures: LLM generation issues, malformed tool calls, execution timeouts, and resource exhaustion. While our tool‑centric design already localizes many environment‑specific errors to the tool level, SkyRL-Agent additionally handles two classes of failures in the agent loop : terminal conditions (e.g., context‑window exceeded, max‑iterations reached), which immediately stop the episode, and recoverable conditions (e.g., parse failures, incorrect parameters), for which the framework injects corrective feedback into the conversation history to enable self‑correction on the next step, so that agents can recover from these failures, improving overall success rates. 4 SWE Agent We will now present the full training recipe for SA-SWE-32B, implemented using SkyRL-Agent . 4.1 Background Recent work has rapidly advanced the use of large language models (LLMs) for automating software engineering (SWE) tasks (Jimenez et al., 2024 ; Yang et al., 2024 ; 2025b ; Pan et al., 2024 ) . SWE‑Bench (Jimenez et al., 2024 ) has emerged as a central testbed: given a real GitHub repository and a bug report, a system must produce a patch and is judged by whether the project’s unit tests pass after applying that patch. Figure 7: Illustration of an SWE Agent. Current solutions on SWE tasks cluster into two families. Agent‑based systems (Wang et al., 2024b ; Yang et al., 2024 ) place the model in an executable environment with a set of tools (e.g., shell, search, and file viewer/editor) and let it iteratively inspect, edit, build, and test until completion, as illustrated in Fig. 7 . This mirrors developer workflows and supports incremental diagnosis and repair. Workflow‑based systems such as Agentless (Xia et al., 2024 ) , by contrast, prescribe a fixed pipeline (e.g., localize → \rightarrow repair → \rightarrow test), reducing the problem to a chain of verifiable single‑turn sub-tasks. The former favors flexibility and fidelity to practice; the latter simplifies orchestration and evaluation at the cost of interactivity and generalizability. SkyRL-Agent is designed to facilitate end‑to‑end RL training of long‑horizon, multi‑turn, multi‑tool agents across diverse agentic tasks. We posit that the tool‑use competence acquired in SWE, where agents must iteratively inspect, edit, and test real codebases, transfers to other domains that similarly require stateful tool interaction. Consequently, in this work, we focus more on training the full interactive loop in an end-to-end manner. 4.2 Training Recipe Bootstrapping training with better tools. Recent studies have shown that agents often struggle with error localization, a key bottleneck in software-enginerring tasks (Chen et al., 2025a ; Zan et al., 2025 ) . We observe that this failure largely arises from the agent’s inability to use commands to effectively navigate the codebase. Specifically, agents tend to over-rely on viewing files directly instead of leveraging search utilities (e.g., grep , find ), and often fail to refine their queries with informative patterns or precise keywords, leading to repeated failures to locate relevant context. As a result, the agent frequently falls back to using the view tool to look through files chunk by chunk, or retrieves large amounts of irrelevant content, both of which consume excessive context and reduce overall efficiency. Based on these observations, we implement an AST-based search tool that provides powerful retrieval capabilities supporting both fuzzy matching and structural pattern search, inspired by LocAgent (Chen et al., 2025a ) . To further encourage the search tool usage, we append contextual hints at the end of each search result (e.g., suggesting which specific terms or patterns to search next) to guide the agent toward more precise queries. We find that such tool enhancements are critical for efficient RL training. Without them, end-to-end RL becomes extremely challenging due to the sparse and delayed rewards inherent in long-horizon tasks. In particular, minimal-tool configurations, such as the bash-only setup used in Mini-SWE-Agent (min, 2025 ) , exhibit very high non-resolved rates (e.g., 50/64) on the R2E-Gym data, making the learning process challenging and inefficient. In contrast, our optimized, tool-guided setup significantly accelerates convergence, achieving comparable or even higher performance than prior models of similar size within only 125 steps of RL training ( Fig. 1(a) ). For evaluation, to assess generalization, we ablate the search tool. Remarkably, the model exhibits emergent, internally realized search behavior and attains comparable performance without explicit access to a dedicated search tool. We also observe in the training that as training progresses, there is a steady increase in the average number of search invocations per trajectory, indicating that the agent gradually learns to rely more on search for efficient code localization, as shown in Fig. 1(a) . RL algorithms and hyper-parameters. For stable training, we adopt a fully on-policy setup where the training batch size equals the mini-batch size (both set to 64), and the number of rollouts per task is 8. Trajectories that terminate due to external constraints, such as the maximum context length (32K tokens), step limit (50 turns), are masked out during gradient updates to prevent the model from being biased against trajectories with more actions or reasoning steps. Importantly, this masking does not modify the reward or advantage estimation; it only excludes those samples from the gradient computation. Following prior works (Luo et al., 2025a ; Liu et al., 2025b ) , we apply leave-one-out advantage estimation and remove both standard deviation and length normalization in advantage computation. We disable KL and entropy loss and train with a learning rate of 1 ​ e − 6 1e-6 . The Qwen3 chat template is modified to preserve the “thinking” from previous turns, maintaining reasoning continuity across steps. Hints for agent to recover and proceed. Multi-turn agent reinforcement learning often suffers from agents getting stuck in repetitive or unproductive behaviors. Prior work, such as SWE-Gym (Pan et al., 2024 ) and Kimi-Dev (Yang et al., 2025b ) observes that starting from a generic pre-trained model can lead to brittle behaviors: failing to invoke tools, generating no or incorrect function calls, or looping over similar actions without progress. Agents may also lose track of the task context or remaining steps, and tend to stop solving the problem prematurely without reaching the maximum turn or context limit. During training, hints are introduced as structured cues that help agents recover from failed actions and re-enter valid trajectories. Such hints include suggestions about potential actions to take upon tool execution failure, notifications that the remaining step budget or context window is about to be exceeded, corrections for invalid or incomplete function calls, or prompts to re-examine failed edits. These hints substantially improve trajectory quality, stabilize rollout collection, and increase the proportion of successful trajectories used for policy optimization. 4.3 Evaluations To demonstrate the effectiveness of our RL training, we evaluate SA-SWE-32B across 4 representative agentic benchmarks. These include SWE-Bench Verified (Jimenez et al., 2024 ) as the in-domain evaluation, and three out-of-domain benchmarks: Terminal-Bench (Team, 2025b ) , WebArena (Zhou et al., 2023 ) , and BrowseComp-Plus (Chen et al., 2025b ) , which assess the model’s generalization to diverse interactive environments involving command-line reasoning, web navigation, and search-integrated reasoning. SWE-Bench Verified We evaluate SA-SWE-32B and several baseline models on SWE-Bench verified using a simple ReAct (Yao et al., 2022 ) scaffold, and the models are provided with only a bash tool and a file editor tool 2 2 2 https://github.com/All-Hands-AI/openhands-aci/blob/main/openhands_aci/editor/editor.py . The maximum context length is set to 40K (the default for Qwen3-32B), and the maximum number of steps to 100. The main results are reported in Section 4.3 . We highlight that, SA-SWE-32B achieves state-of-the-art performance across open-recipe models at its scale, on SWE-Bench Verified, under an end-to-end interactive evaluation setting, demonstrating strong agentic tool-use ability, while incurring substantially lower training cost and without relying on distillation from a stronger teacher model. Among the baseline models, SWE-agent-LM-32B (Yang et al., 2025a ) is fine-tuned from Qwen2.5-Coder-32B-Instruct on 5,016 SWE-smith (Yang et al., 2025a ) trajectories generated by Claude Sonnet 3.7. DeepSWE (Luo et al., 2025a ) , on the other hand, is trained purely with reinforcement learning on the same 4.5K R2E-Gym dataset using the default R2E-Gym scaffold. Compared to DeepSWE, we enhance the training recipe with tool-guided RL, which significantly improves training efficiency. As shown in Fig. 1(a) , the non-resolved rate of SA-SWE-32B remains consistently lower and decreases faster than that of DeepSWE. Combined with the system-level optimizations introduced in Section 3.2 , our approach achieves 50% lower training cost while delivering superior performance. Models such as Kimi-dev (Yang et al., 2025b ) and SWE-Swiss (He et al., 2025 ) , which are trained under the Agentless (Xia et al., 2024 ) scaffold, struggle to follow tool-call instructions in the ReAct setup, leading to substantially lower scores than those reported in their original works, and we therefore only include their reported results in the table for reference. Table 2: Pass@1 performance on SWE-Bench Verified for reference and open-source SWE models. “–” denotes unavailable values. “ × \times ” denotes omitted results. Reported scores are taken from the corresponding papers/blogs; because different works use distinct scaffolds and evaluation setups, the numbers may not be strictly comparable. Simple ReAct denotes a minimal ReAct agent loop that exposes only a bash tool and a file‑editing tool and the model generates only one patch per instance. Model Model Size Recipe SWE-Bench Verified Cost (H100 Hours) Simple ReAct Reported \rowcolor gray!12 Reference Qwen3-32B 32B – 24.4 – – Qwen3-Coder-30B 30B – 45.0 – – \rowcolor gray!12 Open-source models < < 100B SWE-agent-LM-32B 32B 3.7 Sonnet Distill 38 40.2 – SWE-Swiss 32B R1 Distill + RL × \times 45.0 – Kimi-dev 72B R1 Distill + RL × \times 48.6 – DeepSWE 32B RL 36.4 42.2 9180 \rowcolor blue!8 SA-SWE-32B 32B RL 39.4 – 4601 Terminal-Bench Terminal-Bench (version 0.1.1) is a command-line interaction benchmark designed to evaluate agents on system-level tasks that require sequential command execution, environment manipulation, and output verification. It contains 80 tasks spanning categories such as software engineering, system administration, and security, measuring an agent’s ability to navigate file systems, execute shell commands, and complete multi-step technical workflows. In our evaluation, we use the OpenHands agent 3 3 3 https://github.com/laude-institute/terminal-bench/tree/main/terminal_bench/agents/installed_agents/openhands to compare the baseline Qwen3-32B model against SA-SWE-32B. WebArena WebArena is a web-based interaction benchmark designed to evaluate agents on realistic website navigation and task completion across multiple domains. It contains 812 tasks spanning e-commerce, social forums, collaborative software, and content management systems, measuring an agent’s ability to interpret web interfaces, execute multi-step workflows, and achieve specified goals through browser interactions. The benchmark uses fully functional websites with dynamic content to simulate real-world web automation scenarios. In our evaluation, we compare the baseline Qwen3-32B model against SA-SWE-32B. Table 3: SA-SWE-32B’s performance on other tasks. Model Terminal Bench BrowseComp-Plus WebArena Acc. Avg. Turn \rowcolor gray!12 Reference Qwen3-32B 13.75 18.1 3.68 15.8 \rowcolor blue!8 SA-SWE-32B 16.25 19.4 4.6 17.0 BrowseComp-Plus BrowseComp-Plus is a search-oriented benchmark designed to evaluate agents on iterative search, reasoning, and planning tasks. It uses a fixed, human-verified corpus containing both supporting and hard-negative documents to ensure fair and reproducible evaluation. The benchmark consists of 830 questions and measures an agent’s ability to plan multi-step searches, reason over retrieved evidence, and generate accurate final answers. In our evaluation, we use Qwen3-Embedding-8B (Zhang et al., 2025 ) as the retriever and configure the search tool to return top-5 search results. As demonstrated in Section 4.3 , SA-SWE-32B demonstrates improved performance over the base model across all of the benchmarks. During SWE agent training (refer to Fig. 1(a) ), we observe that the average number of search calls increases from 3 to 4, and the average number of turns grows from 18 to 25, indicating that the agent learns to rely more on external search and iterative reasoning as training progresses. Remarkably, on BrowseComp-Plus, which requires extensive search and reasoning across multiple documents, our model, although trained only on the SWE task, exhibits a similar trend by making more search calls than the base model. 5 Other Agents In this section, we provide additional examples implemented in SkyRL-Agent , including deep research agents, a special agent that manages its own memory, and computer use agents. 5.1 Deep Research Deep Research agents (OpenAI, 2025 ; Gao et al., 2025 ; Li et al., 2025b ; a ) are designed to tackle comprehensive question-answering tasks where the required knowledge is not contained within the model parameters. These agents typically rely on web-scale search (e.g., Google Search) and browser-based interaction to gather, reason over, and synthesize external information. In SkyRL-Agent , we support both searching and web-browsing tools under the unified tool-centric interface. Specifically, we adopt the ReAct framework, using SerperAPI for search and Jina Reader API (Jina.ai, 2025 ) for web-page retrieval. To improve efficiency, we introduce caching to avoid repeated fetches and a two-stage summarization pipeline that splits long pages into chunks, summarizes each, and merges the results while preserving references. These optimizations significantly accelerate rollouts and reduce repeated failures during multi-turn research tasks. Experiment setup. Recent work (An et al., 2025 ; Cheng et al., 2025 ) highlights the importance of balancing both difficulty and diversity in reinforcement learning data. To this end, we sample 50K problems from (Fan et al., 2025 ) and perform offline difficulty estimation using Qwen3-8B in thinking mode, generating four rollouts per example. Each problem is categorized by the number of successful rollouts: 0/4 (Impossible), 1/4 (Hard), 2/4 (Medium), 3/4 (Easy), and 4/4 (Perfect). Following the mirrored difficulty distribution proposed in Polaris, we construct a dataset composed of 25% Impossible, 30% Hard, 30% Medium, and 15% Easy examples, further balanced across STEM disciplines (computer science, biology, physics, and economics) to ensure broad coverage and representational diversity. All experiments are trained using the SkyRL-train (Griggs et al., 2025 ) backend. Figure 8: Training Curve for Deep Research Agent on SkyRL-train backend. We train Qwen3-8B with GRPO Shao et al. ( 2024 ) using the general verifier from (Ma et al., 2025 ) , a global batch size of 64 with 8 rollouts, a mini-batch size of 64, and a learning rate of 5 × 10 − 6 5\times 10^{-6} . The training uses Qwen3-235B in non-reasoning mode as the summarizer. Reward and validation curves during training are shown in Fig. 8 . For evaluation, we use LLM-as-a-Judge on the HLE-500 benchmark (Li et al., 2025b ) . Since the general verifier (Ma et al., 2025 ) tends to be more lenient, we additionally use gpt-oss-20b as the primary judge. Under the general verifier, scores improve from 12.6% to 18.8%, and under gpt-oss-20b, from 9.2% to 10.2% / 11.0%. Challenges. We encountered two challenges in deep-research training. First , heavy tools introduce significant compute overhead and, if under-provisioned, harm both throughput and stability. Web-scraping pipelines ( ∼ \sim 10k tokens/request) paired with large summarizers (e.g., Qwen-3-32B) slowed training drastically: serving Qwen-3-32B on a single H200 took 2,101.6 s per iteration, whereas switching to a Qwen API reduced this to 592.1 s ( ∼ \sim 3.5 × \times faster). Our final stage used four GH200s with a data-parallel router. Insufficient serving caused timeouts that contaminated rollouts (e.g., a dip around iteration 30) and required multiple steps (30–33) to recover, motivating both adequate tool capacity and robust handling (e.g., masking abnormal trajectories) to protect the reward signal. Second , online search can leak benchmark answers. During evaluation on GPQA-Diamond, models sometimes retrieved solutions directly from public pages (e.g., Hugging Face) instead of reasoning, inflating scores via shortcut retrieval. We therefore enforce domain blocks (e.g., Hugging Face, GitHub, GitLab, Chegg) to limit leakage and encourage authentic, tool-augmented reasoning. 5.2 Memory Agent The fixed context length remains a fundamental limitation for contemporary LLMs. Recent work such as MemAgent (Yu et al., 2025a ) proposes a recursive long-text processing mechanism, where the model incrementally summarizes content into a fixed-size memory buffer. We implemented the MemAgent scaffold by introducing a Next tool, as illustrated in Fig. 8(a) . Experimental Setup. We evaluate on the RULER-HotpotQA dataset (Hsieh et al., 2024 ) , which provides a golden supporting paragraph accompanied by several distractor paragraphs. Following the setup of MemAgent, the model is presented with chunks of text up to 4k tokens each. (a) MemAgent Scaffold in SkyRL-Agent . The agent processes document chunks using the Next tool, which summarizes the current chunk and appends the summary to the next chunk as input for the following turn. (b) Training Reward and Validation Score. Results on RULER-HotpotQA for the memory agent trained from the Tinker backend. Figure 9: MemAgent Results. During training, it processes sequences of up to 28k tokens, and during evaluation, up to 112k tokens. We use GPT-5-nano as the verifier for answer correctness. We train the model using the Tinker backend from Qwen3-8B (non-thinking mode), using a LoRA rank of 128, batch size 32, 8 rollouts, and a maximum of 8K tokens per turn. Training curves are shown in Fig. 8(b) . For reference, MemAgent reports a 79.69% accuracy when training Qwen2.5-7B-Instruct under the same dataset using a batch size of 128 and group size 16, evaluated with an exact-match verifier. 5.3 Computer Use Agent Computer Use agents aim to complete tasks via interacting with an actual computer interface. We implement Computer Use agent training using the OSWorld environment (Xie et al., 2024 ) where the model can interact with a virtual computer through the PyAutoGUI APIs and get environment feedback through accessibility trees generated from the virtual desktop. We integrate the OSWorld environment as a tool in SkyRL-Agent , as shown in Fig. 3 , enabling agents to perform GUI interactions such as mouse clicks, keystrokes, scrolling, and window manipulation directly through generated Python codes with PyAutoGUI APIs. Figure 10: Training Curve for Computer Use Agent on VeRL backend. Experiement Setup. Due to the lack of data, we mirror the setup in ARPO (Lu et al., 2025 ) , which chooses a subset from the benchmark to be the training data. Similar to the data filtering process for the Deep Research task ( Section 5.1 ), we filter the OSWorld benchmark and collect a subset of 32 tasks with Hard, Medium, and Easy difficulties. We train Qwen3-8B with GRPO and implement the rollout process using the Async Batch (Bounded) dispatcher with fixed 32 virtual desktop environments. We scale the OSWorld environments by launching its virtual-machine as Ray (Moritz et al., 2018 ) remote tasks, disaggregating CPU-bound feedback loops from GPU-bound model generation. The training batch size is set to 8 with 8 rollouts for each task. The evaluation is directly performed by the environment’s native evaluation function, with training curves shown in Fig. 10 . Observations. We observe that for OSWorld tasks, while the training reward steadily improves, the validation accuracy shows little to no gain. This suggests that the tasks are inherently difficult for Qwen3-8B, and the learned policy struggles to generalize beyond the training environments. Similar trends have also been reported in (Lu et al., 2025 ) . Acknowledgment This work is supported by the kind compute support from Anyscale, Amazon, Lambda, Thinking Machine Lab, and gifts from AMD, Google, Mayfield, Mithril, Laude Institute, Accenture, Broadcom, Cisco, IBM, Intel, Intesa Sanpaolo, Lightspeed, Mibura, Microsoft, NVIDIA, Samsung SDS, and SAP. We also thank John Yang, Naman Jain, Xingyao Wang, Jiayi Pan, Chuan Li and Eric Tu for helpful discussion and feedback. References min (2025) The 100 line ai agent that solves github issues & more. https://github.com/SWE-agent/mini-swe-agent , 2025. An et al. (2025) Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. URL https://hkunlp.github.io/blog/2025/Polaris . AutoGen (2025) AutoGen. AutoGen. https://github.com/microsoft/autogen, 2025. Cao et al. (2025) Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, Akshay Malik, Graham Neubig, Kourosh Hakhamaneshi, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Skyrl-v0: Train real-world long-horizon agents via reinforcement learning, 2025. Chen et al. (2017) Chen Chen, Xiaojun Meng, Shengdong Zhao, and Morten Fjeld. Retool: Interactive microtask and workflow design through demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems , pp. 3551–3556, 2017. Chen et al. (2025a) Zhaoling Chen, Xiangru Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Arman Cohan, and Xingyao Wang. Locagent: Graph-guided llm agents for code localization. arXiv preprint arXiv:2503.09089 , 2025a. Chen et al. (2025b) Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent. arXiv preprint arXiv:2508.06600 , 2025b. Cheng et al. (2025) Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, et al. Revisiting reinforcement learning for llm reasoning from a cross-domain perspective. arXiv preprint arXiv:2506.14965 , 2025. Fan et al. (2025) Run-Ze Fan, Zengzhi Wang, and Pengfei Liu. Megascience: Pushing the frontiers of post-training datasets for science reasoning. arXiv preprint arXiv:2507.16812 , 2025. Fu et al. (2025) Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 2025. URL https://arxiv.org/abs/2505.24298 . Gao et al. (2025) Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv preprint arXiv:2508.07976 , 2025. Griggs et al. (2025) Tyler Griggs, Sumanth Hegde, Eric Tang, Shu Liu, Shiyi Cao, Dacheng Li, Charlie Ruan, Philipp Moritz, Kourosh Hakhamaneshi, Richard Liaw, et al. Evolving skyrl into a highly-modular rl framework. Notion Blog , 2025. Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025. He et al. (2025) Zhenyu He, Qingping Yang, Wei Sheng, Xiaojian Zhong, Kechi Zhang, Chenxin An, Wenlei Shi, Tianle Cai, Di He, Jiaze Chen, and Jingjing Xu. Swe-swiss: A multi-task fine-tuning and rl recipe for high-performance issue resolution. https://www.notion.so/SWE-Swiss-A-Multi-Task-Fine-Tuning-and-RL-Recipe-for-High-Performance-Issue-Resolution-21e174dedd4880ea829ed4c861c44f88 , 2025. Notion Blog. Hsieh et al. (2024) Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 , 2024. Hu et al. (2025) Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290 , 2025. Jain et al. (2025) Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica. R2e-gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents. arXiv preprint arXiv:2504.07164 , 2025. Jiang et al. (2025) Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, et al. Verltool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01055 , 2025. Jimenez et al. (2024) Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=VTF8yNQM66 . Jin et al. (2025) Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 , 2025. Jina.ai (2025) Jina.ai. Jina, 2025. URL https://jina.ai/ . Kaelbling et al. (1998) Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence , 101(1-2):99–134, 1998. Lab (2025) Thinking Machines Lab. Tinker api. https://tinker-docs.thinkingmachines.ai/ , 2025. LangChain (2025) LangChain. LangChain. https://www.langchain.com/, 2025. Li et al. (2025a) Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, et al. Websailor: Navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02592 , 2025a. Li et al. (2025b) Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yongkang Wu, Ji-Rong Wen, Yutao Zhu, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776 , 2025b. Li et al. (2025c) Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383 , 2025c. Liu et al. (2025a) Liyuan Liu, Feng Yao, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Flashrl: 8bit rollouts, full power rl, August 2025a. URL https://fengyao.notion.site/flash-rl . Liu et al. (2025b) Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783 , 2025b. Liu et al. (2025c) Zichen Liu, Anya Sims, Keyu Duan, Changyu Chen, Simon Yu, Xiangxin Zhou, Haotian Xu, Shaopan Xiong, Bo Liu, Chenmien Tan, et al. Gem: A gym for agentic llms. arXiv preprint arXiv:2510.01051 , 2025c. Lu et al. (2025) Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Arpo: End-to-end policy optimization for gui agents with experience replay. arXiv preprint arXiv:2505.16282 , 2025. Luo et al. (2025a) Michael Luo, Naman Jain, Jaskirat Singh, Sijun Tan, Ameen Patel, Qingyang Wu, Alpay Ariyak, Colin Cai, Tarun Venkat, Shang Zhu, Ben Athiwaratkun, Manan Roongta, Ce Zhang, Li Erran Li, Raluca Ada Popa, Koushik Sen, and Ion Stoica. Deepswe: Training a state-of-the-art coding agent from scratch by scaling rl. https://pretty-radio-b75.notion.site/DeepSWE-Training-a-Fully-Open-sourced-State-of-the-Art-Coding-Agent-by-Scaling-RL-22281902c1468193aabbe9a8c59bbe33 , 2025a. Notion Blog. Luo et al. (2025b) Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K Qiu, and Yuqing Yang. Agent lightning: Train any ai agents with reinforcement learning. arXiv preprint arXiv:2508.03680 , 2025b. Ma et al. (2025) Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing llm reasoning across all domains. arXiv preprint arXiv:2505.14652 , 2025. Moritz et al. (2018) Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging { \{ AI } \} applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18) , pp. 561–577, 2018. OpenAI (2025) OpenAI. Deep research system card, 2025. URL https://cdn.openai.com/deep-research-system-card.pdf . Packer et al. (2023) Charles Packer, Vivian Fang, Shishir_G Patil, Kevin Lin, Sarah Wooders, and Joseph_E Gonzalez. Memgpt: Towards llms as operating systems. 2023. Pan et al. (2024) Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139 , 2024. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024. Sheng et al. (2025) Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems , pp. 1279–1297, 2025. Tan et al. (2025) Sijun Tan, Michael Luo, Colin Cai, Tarun Venkat, Kyle Montgomery, Aaron Hao, Tianhao Wu, Arnav Balyan, Manan Roongta, Chenguang Wang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. rllm: A framework for post-training language agents. https://pretty-radio-b75.notion.site/rLLM-A-Framework-for-Post-Training-Language-Agents-21b81902c146819db63cd98a54ba5f31 , 2025. Notion Blog. Team (2025a) The Agent Lightning (AGL) Team. No more retokenization drift: Returning token ids via the openai compatible api matters in agent rl, 2025a. URL https://blog.vllm.ai/2025/10/22/agent-lightning.html . Team (2025b) The Terminal-Bench Team. Terminal-bench: A benchmark for ai agents in terminal environments, Apr 2025b. URL https://github.com/laude-institute/terminal-bench . Wang et al. (2024a) Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning , 2024a. Wang et al. (2024b) Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations , 2024b. Wen et al. (2025) Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245 , 2025. Xia et al. (2024) Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489 , 2024. Xie et al. (2024) Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. Yang et al. (2024) John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems , 37:50528–50652, 2024. Yang et al. (2025a) John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. arXiv preprint arXiv:2504.21798 , 2025a. Yang et al. (2025b) Zonghan Yang, Shengjie Wang, Kelin Fu, Wenyang He, Weimin Xiong, Yibo Liu, Yibo Miao, Bofei Gao, Yejie Wang, Yingwei Ma, et al. Kimi-dev: Agentless training as skill prior for swe-agents. arXiv preprint arXiv:2509.23045 , 2025b. Yao et al. (2025) Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, August 2025. URL https://fengyao.notion.site/off-policy-rl . Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations , 2022. Yu et al. (2025a) Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, et al. Memagent: Reshaping long-context llm with multi-conv rl-based memory agent. arXiv preprint arXiv:2507.02259 , 2025a. Yu et al. (2025b) Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 , 2025b. Zan et al. (2025) Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving. arXiv preprint arXiv:2504.02605 , 2025. Zhang et al. (2025) Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 , 2025. Zhou et al. (2023) Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854 , 2023. Zhu et al. (2025) Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime , 2025. GitHub repository. Corresponding author: Xin Lv.