Title: 2510.00023 ArXiv: 2510.00023 ToolBrain: A Flexible Reinforcement Learning Framework for Agentic Tools ToolBrain: A Flexible Reinforcement Learning Framework for Agentic Tools Quy Minh Le 1 Minh Sao Khue Luu 1 1 1 footnotemark: 1 Khanh-Tung Tran 2 1 1 footnotemark: 1 Duc-Hai Nguyen 2 Hoang-Quoc-Viet Pham 1 Quan Le 3 Hoang Thanh Lam 4 Hoang D. Nguyen 2 2 2 footnotemark: 2 1 ToolBrain Research, Ireland 2 University College Cork, Ireland 3 CeADAR University College Dublin, Ireland 4 IBM Research Lab, Dublin, Ireland Equal contribution.Corresponding authors: t.l.hoang@ie.ibm.com , hn@cs.ucc.ie . Abstract Effective tool use is essential for agentic AI, yet training agents to utilize tools remains challenging due to manually designed rewards, limited training data, and poor multi-tool selection, resulting in slow adaptation, wasted computational resources, and suboptimal performance. We introduce ToolBrain , a lightweight and user-friendly framework for training tool use in agentic models with flexible reinforcement learning, thereby easing the barriers for researchers and practitioners to adapt LLM-based agents to specific domains. It supports a wide range of training strategies, including reinforcement learning algorithms such as GRPO and DPO, as well as supervised learning. ToolBrain enables custom reward callables directly on an agent’s execution traces or simply utilizes an automated LLM-as-a-judge system for reward generation. It is packed with useful capabilities, including knowledge distillation from large to small models, automatic task generation from tool descriptions, seamless tool retrieval, efficient fine-tuning pipelines with QLoRA through Unsloth, and quantized inference via bitsandbytes. We demonstrate ToolBrain through an Email Search Agent case study, showing measurable improvements in tool-use skills under a realistic workflow, while keeping the codebase simple and extensible. Our framework is publicly available at https://toolbrain.org/ ToolBrain: A Flexible Reinforcement Learning Framework for Agentic Tools Quy Minh Le 1 † † thanks: Equal contribution. Minh Sao Khue Luu 1 1 1 footnotemark: 1 Khanh-Tung Tran 2 1 1 footnotemark: 1 Duc-Hai Nguyen 2 Hoang-Quoc-Viet Pham 1 Quan Le 3 Hoang Thanh Lam 4 † † thanks: Corresponding authors: t.l.hoang@ie.ibm.com , hn@cs.ucc.ie . Hoang D. Nguyen 2 2 2 footnotemark: 2 1 ToolBrain Research, Ireland 2 University College Cork, Ireland 3 CeADAR University College Dublin, Ireland 4 IBM Research Lab, Dublin, Ireland 1 Introduction LLM-based agents have become a viable technological wave, capable of executing complex tasks, ranging from planning, code generation, interaction with APIs, to scientific discovery, through the use of tools Shinn et al. ( 2023 ); Yao et al. ( 2023 ); Schick et al. ( 2023 ); LangChain ( 2025 ) . However, many agentic systems rely on supervised fine-tuning or prompt engineering to adapt behavior OpenAI ( 2023 ); Wei et al. ( 2022 ) , preventing continuous improvements through experience in perplexing environments. Reinforcement Learning (RL) enables adaptive policies based on system traces, enhancing LLM capabilities for preference optimization Ouyang et al. ( 2022 ); Rafailov et al. ( 2024 ) and reasoning tasks Nakano et al. ( 2021 ) , but its integration into agentic tool-use workflows remains underdeveloped, especially for complex and self-evolving tools. Several key challenges hinder the broader adoption of RL in agent and tool development. First, existing frameworks such as ART Hilton et al. ( 2025 ) and Agent Lightning Luo et al. ( 2025 ) lack a lightweight and user-friendly interface that allows users to define and apply RL reward signals directly on an agent execution trace. Second, although tool calling with large language models is highly effective, these models remain computationally expensive, while smaller models perform substantially worse. As a result, knowledge distillation from large models becomes critical for industrial deployment and cost efficiency. Third, the tool ecosystem is often extremely large, making it inefficient to learn effective behavior in the presence of many irrelevant tools. Finally, collecting high-quality training data is typically very costly. Addressing the challenge of teaching models to use tools effectively, therefore, requires treating all of these issues in a unified manner. ToolBrain is designed to address these pain points by providing a simple API that connects (i) a tool-using agent, (ii) a flexible reward system (§ 3.2.1 ), and (iii) RL training algorithms such as GRPO and DPO (§ 3.2.2 ). The framework is further extended with a suite of powerful features, including intelligent tool retrieval (§ 3.2.3 ), zero-shot task generation (§ 3.2.4 ), knowledge distillation (§ 3.2.5 ), and a highly efficient training backend (§ 3.2.6 ). We demonstrate ToolBrain’s effectiveness and flexibility through a comprehensive evaluation across multiple tasks. Our main experiment focuses on a complex, multi-step information retrieval task (Email Search), where we show significant performance improvements on both 3B and 7B parameter models. To showcase the framework’s rapid adaptation capabilities, we conduct two supplementary experiments on quantitative reasoning (Finance) and real-world grounding (API) tasks, demonstrating how Knowledge Distillation can quickly uplift the performance of a compact 0.5B model. The remainder of this paper is structured as follows. We first situate our work within the existing landscape of agent frameworks (§ 2 ). We then describe the technical details of ToolBrain, presenting our core Coach-Athlete architectural paradigm and its key features (§ 3 ). Subsequently, we provide a comprehensive empirical evaluation to validate our framework’s performance on the aforementioned tasks (§ 4 ). Finally, we conclude with a summary of our contributions (§ 5 ). 2 Related Work Aspect ToolBrain LangChain / LangGraph ART Agent Lightning Training Approach ✓ Native RL (GRPO, DPO) with iterative fine-tuning. Supervised learning and prompt chaining. GRPO-based RL with RULER evaluator. Hierarchical RL with credit assignment. Reward System ✓ Hybrid: Python callable plus ranking-based LLM. Manual heuristic scoring. RULER: automated LLM-as-judge with relative scoring. Credit-aware reward assignment. Tool Management ✓ Integrated Tool Retriever automatically selects relevant tools. Manual tool definition and passing. Manual tool definition and passing. Manual tool definition and passing. Advanced Strategies ✓ Supports knowledge distillation and Zero-Learn task generation. — — — Efficiency & Usability ✓ Simple Brain API ; integrated Unsloth/QLoRA optimizations. Code-centric; complex context management. Minimal code changes; requires separate server setup. Requires MDP design; steep RL expertise. Table 1: Comparison of ToolBrain with other agent training frameworks. ToolBrain builds upon a rich landscape of agent and RL frameworks. Although numerous systems facilitate agent development, they often present trade-offs in usability, flexibility, and the steep learning curve associated with reinforcement learning. To position our contributions, we compare ToolBrain with three representative approaches in Table 1 : LangChain/LangGraph LangChain ( 2025 ) , representing popular code-centric systems; ART Hilton et al. ( 2025 ) , a contemporary RL-focused framework; and Agent Lightning Luo et al. ( 2025 ) , which focuses on hierarchical RL. ToolBrain aims to complement these systems by offering a lightweight, RL-centric interface that exposes reward signals at the level of agent traces and integrates practical features such as distillation and tool retrieval. 3 System Description 3.1 Architecture Overview ToolBrain’s design is structured around the Coach–Athlete paradigm , a conceptual abstraction that cleanly separates training orchestration from task execution responsibilities . As illustrated in Figure 1 , the paradigm consists of three main components: a high-level Brain (the Coach) that manages the training loop; an Agent (the Athlete) responsible for executing tasks using external tools; and a lightweight internal Adapter (the Interpreter) that provides a standardized communication interface. The training workflow progresses through a complete, well-defined cycle. First, the Brain issues a command to the Agent. The Agent executes the task, and its actions are monitored by the Adapter, which translates them into a standardized, high-fidelity Execution Trace . This trace is the foundation of the learning loop: the Brain uses it to compute rewards and determine the optimal policy updates. Crucially, the Brain then applies these updates directly to the parameters of the Agent’s underlying model, thereby improving its capabilities for future tasks and completing the cycle. Although this paradigm shares conceptual similarities with actor-learner architectures such as IMPALA (Espeholt et al. , 2018 ) , it establishes a different boundary tailored for iterative agent development. The linchpin of this architecture is the internal Adapter, which implements the classic Adapter design pattern (Gamma et al. , 1995 ) . Its primary responsibility is to act as a structural bridge, converting the proprietary, framework-specific memory of a user-provided agent into ToolBrain’s standardized Execution Trace format. This decoupling is critical: it allows the Brain (the client) to interact with any Agent (the adaptee) through a single, consistent interface, regardless of the agent’s internal implementation (e.g., smolagents , langchain ). By handling this translation automatically, the Adapter makes the framework agnostic to the user’s choice of agent implementation. This ensures that the trace data fed to downstream RL algorithms is uniform and high-fidelity, regardless of the underlying agent’s architecture. Comprehensive schema definitions for the Execution Trace are provided in Appendix A . Figure 1: The Coach-Athlete-Interpreter paradigm in ToolBrain. The Brain orchestrates the process, the user-provided Agent executes the task, and the Adapter acts as a standardized communication layer, translating interactions into a high-fidelity trace for learning. Figure 2 illustrates the streamlined API workflow of ToolBrain, which encapsulates the entire training process. The design emphasizes modularity and user control. The user first configures the core components: a tool-using agent , a reward_func , and optionally a custom ToolRetriever to define the tool selection strategy. These components are then passed to the central Brain orchestrator, which manages the entire pipeline — from data generation and knowledge distillation to the final one-line brain.train() command. Figure 2: The ToolBrain API workflow, demonstrating how a user composes a training setup and executes the entire RL pipeline with a single command. It highlights key features such as the configurable tool retrieval, flexible rewards, one-parameter algorithm selection, and advanced strategies like knowledge distillation. 3.2 Key Features and Innovations ToolBrain introduces several features intended to make RL for tool-using agents more accessible in practice. 3.2.1 Flexible Reward Interface A central goal of ToolBrain is to simplify reward design, which remains a significant challenge in agent training. To this end, our framework provides a flexible, hybrid reward system that supports two complementary approaches. The first is user-defined heuristic rewards, where any Python callable can operate directly on an agent’s full execution trace. This allows for precise, objective, and domain-specific feedback based on clear success criteria. For more nuanced tasks without clear ground-truth, ToolBrain integrates an optional LLM-as-a-Judge mechanism. This aligns with the growing body of work on using powerful LLMs for evaluation, a practice increasingly validated in recent research (Zheng et al. , 2023 ; Liu et al. , 2023 ; Gu et al. , 2025 ) . Instead of asking for an unreliable absolute score, ToolBrain’s judge ranks a group of traces for the same query from best to worst. This relative feedback is then converted into normalized scalar rewards, a robust approach inspired by the principles of learning from human preferences (Ouyang et al. , 2022 ) . Internally, a small wrapper harmonizes both heuristic and judge-based functions into a unified interface, abstracting away the complexity from the training loop. Implementation details and example reward functions are provided in Appendix B . 3.2.2 Learning Algorithms: GRPO and DPO ToolBrain supports two state-of-the-art algorithms for agent alignment. For scenarios with explicit scalar rewards, it implements Group Relative Policy Optimization (GRPO) (Shao et al. , 2024 ) . GRPO is a modern variant of the widely-used Proximal Policy Optimization (PPO) family of algorithms (Schulman et al. , 2017 ) , which optimizes a policy by generating groups of responses and normalizing rewards to obtain stable relative advantages. For preference-based learning, ToolBrain implements Direct Preference Optimization (DPO) (Rafailov et al. , 2024 ) . DPO offers a more direct and often more stable approach to the problem of learning from human feedback (Christiano et al. , 2023 ) , as it learns directly from chosen vs. rejected pairs without needing to train a separate reward model. We provide full pseudocode for both methods in Appendix C . 3.2.3 Intelligent Tool Retrieval As the number of available tools grows, an agent’s ability to select the most relevant ones for a given task becomes critical. Providing an LLM with a vast and mostly irrelevant tool library increases context length, computational cost, and the likelihood of hallucinated or incorrect tool calls. This challenge is a central theme in recent tool-use research (Schick et al. , 2023 ; Qin et al. , 2023 ) . To address this, ToolBrain implements an Intelligent Tool Retrieval mechanism. Our approach externalizes the selection logic from the agent itself, a design inspired by recent systems that employ a dedicated LLM as a tool filter or retriever (Qin et al. , 2023 ; Huang et al. , 2025 ) . Instead of a simple boolean flag, ToolBrain uses a dependency injection pattern: the user instantiates and configures a separate ToolRetriever object, which leverages a powerful LLM (e.g., GPT-4) to act as a tool selection module. This retriever is then passed to the Brain during initialization. Before each task, the Brain uses this retriever to dynamically select a small, relevant subset of tools from the agent’s full library. This design provides significant flexibility, reduces context length, and improves agent accuracy by focusing its attention on the most pertinent tools for the task at hand. An example is provided in Appendix F . 3.2.4 Zero-Learn Task Generation A primary obstacle in fine-tuning agents is the scarcity of high-quality, domain-specific training data. To address this, ToolBrain incorporates a Zero-Learn task generation pipeline, enabling it to bootstrap its own training data. This approach is inspired by a growing body of work demonstrating that large language models can effectively self-improve by learning from their own generated outputs (Wang et al. , 2023 ; Huang et al. , 2022 ) . Given a set of tool definitions and an optional high-level task description, ToolBrain’s generate_training_examples method prompts its own underlying LLM to synthesize a multiple set of tool-using queries. This self-supervised methodology aligns with the principles of models that teach themselves to use tools (Schick et al. , 2023 ) . To further enhance data quality, the framework includes a self-ranking mechanism, allowing the model to evaluate its own generations and prioritize tasks that are concrete and well-aligned with the provided tools. A detailed breakdown of the generation process and query categories is provided in Appendix D . 3.2.5 Knowledge Distillation for Policy Initialization Training smaller, more efficient models to perform complex tasks is a primary goal in applied NLP. To facilitate this, ToolBrain implements a knowledge distillation pipeline, a powerful technique for transferring capabilities from a large teacher model to a smaller student model (Hinton et al. , 2015 ) . This approach has proven highly effective for creating compact and fast language models, such as DistilBERT, without a significant loss in performance (Sanh et al. , 2020 ) . In ToolBrain’s workflow, a large teacher model (e.g., 7B parameters) first generates high-quality execution traces for a given set of tasks. These expert demonstrations are then filtered for correctness and used as a dataset to train a smaller student model (e.g., 0.5B parameters) via a standard supervised learning objective. This process provides the student with a well-initialized policy, yielding meaningful action distributions and stable behavior prior to subsequent reinforcement learning fine-tuning. As demonstrated in our supplementary experiments (Section 4 ), this technique significantly uplifts the performance of small models on specialized tasks. The full algorithm and code examples are provided in Appendix E . 3.2.6 Efficient Training Optimization Making RL fine-tuning practical on consumer hardware is a core design principle of ToolBrain. To achieve this, the framework integrates a suite of state-of-the-art optimization techniques, abstracting them behind simple, high-level parameters. A key strategy is the support for Low-Rank Adaptation (LoRA) (Hu et al. , 2021 ) , which dramatically reduces the number of trainable parameters by freezing the base model and only training small, injectable matrices. Building upon this, ToolBrain natively supports QLoRA (Dettmers et al. , 2023 ) , an even more memory-efficient approach that uses 4-bit quantization for the frozen base model, making it feasible to fine-tune large models on a single GPU. Furthermore, the framework supports standard mixed-precision training (fp16) to accelerate computation (Micikevicius et al. , 2018 ) , and seamlessly integrates with accelerated backends such as Unsloth (Han and Han, 2023 ) to improve training efficiency. By combining these powerful optimizations, ToolBrain significantly lowers the barrier to entry for training capable, tool-using agents. Implementation examples are provided in Appendix F.2 . 4 Experiments and Results To demonstrate the effectiveness and flexibility of ToolBrain, we conduct a comprehensive evaluation centered on three distinct tasks. Our main experiment provides a deep dive into a complex, multi-step reasoning task, showcasing the framework’s core training capabilities. This is followed by two additional experiments that assess the framework’s versatility and efficiency in adapting agents to specialized domains. 4.1 Main Experiment: Email Search Agent 4.1.1 Task This experiment centers on a challenging information retrieval and multi-step reasoning task, inspired by the ART ⋅ \cdot E project (Corbitt, 2025a ) . The agent’s objective is to answer natural language questions by navigating a large-scale email database. We use the well-established Enron Email Corpus (Klimt and Yang, 2004 ) , a dataset containing approximately 0.5 million real-world emails from senior management at Enron. For training and evaluation, we utilize a publicly available set of question-answer pairs derived from this corpus (Corbitt, 2025b ) . This setup requires the agent to perform a sophisticated tool-using workflow, including searching, reading, and synthesizing information to derive the correct answer. 4.1.2 Evaluation and Results We trained two sizes of the Qwen2.5 model (3B and 7B parameters) for 60 steps using the GRPO algorithm Shao et al. ( 2024 ) with an LLM-as-a-Judge Zheng et al. ( 2023 ); Ouyang et al. ( 2022 ) . To provide a comprehensive view of performance, we tracked multiple metrics: task success rate, hallucination rate (the percentage of incorrect answers among all attempted answers), and the average number of turns per query. The results summarized in Table 2 indicate a substantial improvement in the agent’s overall performance. The learning dynamics are further illustrated in Figure 3 . The untrained 3B model fails to solve the task, whereas the 7B model demonstrates a small degree of zero-shot competence. After training, both models achieve notable gains in task success. Importantly, training enhances both accuracy and reliability, reducing the hallucination rate of the 7B model from 60.0% to 35.0%. Moreover, the 7B model shows improved efficiency, completing the task in fewer turns, while the 3B model typically engages in more turns to reach a solution. These contrasting behaviors appear to reflect the effects of scale on the two models. Finally, the continued upward trajectory of correctness for both models at the 60-step mark suggests that further training could yield additional improvements. Table 2: Comprehensive Evaluation of the Email Search Agent (from a single representative run). The table compares key performance metrics at the start (Step 0) and end (Step 60) of training. Arrows (↓) indicate that lower values are better. Before Training (Step 0) After Training (Step 60) Model Success Rate (%) Hallucination Rate (%) ↓ Avg. Turns ↓ Success Rate (%) Hallucination Rate (%) ↓ Avg. Turns ↓ Qwen2.5-3B 0.0 100.0 4.63 16.7 66.7 5.57 Qwen2.5-7B 13.3 60.0 7.03 43.3 35.0 4.77 Figure 3: Learning curve showing the Correctness Rate from a representative training run on the email search task. 4.2 Supplementary Experiments: Demonstrating Flexibility and Efficiency 4.2.1 Methodology and Tasks To showcase the framework’s versatility, we evaluated a 0.5B parameter agent on two supplementary tasks, each representing a different challenge for language agents. We used a simple Before vs. After methodology to measure the impact of our training process. The tasks are as follows: • Finance Agent (Quantitative Reasoning): The agent must correctly map natural language questions to structured financial calculation tools, a task that requires high precision and strict adherence to tool specifications. • API Agent (Real-World Grounding): The agent must call an external weather API to answer real-time queries, demonstrating its ability to connect to and utilize live, external data sources. 4.2.2 Dataset and Training To prepare for training and evaluation, we used ToolBrain’s built-in generate_training_examples feature to automatically synthesize datasets. For each of the two additional tasks, we generated a training set of 40 queries and a held-out test set of 10 queries by providing a task-specific description. The agents were trained briefly and efficiently using knowledge distillation from a 7B teacher model. This approach demonstrates the Zero-Learn capability of our framework while ensuring a consistent and reproducible methodology. Details on the data-generation API and the distillation pipeline are provided in Appendix D and Appendix E , respectively. 4.2.3 Results and Analysis Evaluation was conducted on a held-out test set of 10 unseen queries for each agent. As shown in Table 3 , the untrained baseline models exhibited limited capability. After the distillation phase, the agents demonstrated clear improvements in calling the correct tool. The Finance Agent improved from 20% to 40%, while the API Agent improved from 30% to 60%. Although the results are not perfect, they indicate that ToolBrain — and especially its distillation component — offers an efficient mechanism for rapidly adapting small language models to specialized domains. Table 3: Demonstrating Framework Flexibility on Secondary Tasks. Success Rate (%) is measured by the agent’s ability to call the correct tool on a held-out test set of 10 queries. Case Study Untrained Trained (w/ Distill) Finance (Quant. Reasoning) 20.0% 40.0% API (Real-World Grounding) 30.0% 60.0% 5 Conclusion The effective use of tools is fundamental to advancing the capabilities of agentic AI systems. However, developing robust tool-augmented agents presents several challenges, including the inherent complexity of reinforcement learning frameworks, the difficulties associated with designing appropriate reward functions, and the substantial computational cost required for training. In this work, we introduce ToolBrain, a framework that aims to bridge the gap between agent design and iterative, experience-driven improvement through reinforcement learning. We introduce the Coach–Athlete paradigm as a central architectural principle, offering a streamlined, high-level API that abstracts away underlying implementation complexity. We show how ToolBrain’s flexible hybrid reward system enables users to provide effective feedback through both user-defined code and a ranking-based LLM-as-a-Judge mechanism. In addition, we describe a set of advanced capabilities — such as intelligent tool retrieval, knowledge distillation, and zero-learn task generation — that operate alongside modern training optimizations, including Unsloth and QLoRA, to make agent training more practical and accessible. Through our central case study of training an Email Search Agent, we provided both quantitative and qualitative evidence of our framework’s efficacy. The experimental results show that agents trained with ToolBrain demonstrate significant and consistent performance improvements over their initial baselines. The learning curves validate our training pipeline’s effectiveness, and the final agent’s ability to handle complex, multi-step workflows showcases the sophisticated skills acquired. Although ToolBrain provides a flexible foundation, several promising avenues remain for future work. These include expanding the tool retrieval mechanism to handle even more complex, dynamic tool libraries, extending the Coach-Athlete paradigm to multi-agent scenarios, and exploring online RL algorithms for continuous, real-time agent adaptation. Ultimately, ToolBrain offers a practical and effective basis for the broader community. By reducing the barrier to agent-centric RL, we aim to enable more developers and researchers to design, refine, and deploy the next generation of capable, reliable, and domain-adapted autonomous systems. Acknowledgements This publication has emanated from research supported in part by grants from Research Ireland under Grant [12-RC-2289-P2] and [18/CRT/6223] which is co-funded under the European Regional Development Fund. This work is also supported by ISY Labs. For the purpose of Open Access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. References P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei (2023) Deep reinforcement learning from human preferences . External Links: 1706.03741 , Link Cited by: §3.2.2 . K. Corbitt (2025a) ART·e: how we built an email research agent that beats o3 . Note: OpenPipe Blog External Links: Link Cited by: §4.1.1 . K. Corbitt (2025b) Enron emails sample questions . Note: Hugging Face Datasets External Links: Link Cited by: §4.1.1 . T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) QLoRA: efficient finetuning of quantized llms . External Links: 2305.14314 , Link Cited by: §3.2.6 . L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu (2018) IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures . External Links: 1802.01561 , Link Cited by: §3.1 . E. Gamma, R. Helm, R. Johnson, and J. Vlissides (1995) Design patterns: elements of reusable object-oriented software . Addison-Wesley Longman Publishing Co., Inc. , Boston, MA, United States . External Links: ISBN 978-0-201-63361-0 Cited by: §3.1 . J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Y. Wang, W. Gao, L. Ni, and J. Guo (2025) A survey on llm-as-a-judge . External Links: 2411.15594 , Link Cited by: §3.2.1 . D. Han and M. Han (2023) Unsloth . Note: https://github.com/unslothai/unsloth GitHub repository Cited by: §3.2.6 . B. Hilton, K. Corbitt, D. Corbitt, S. Gandhi, A. William, B. Kovalenskyi, and A. Jones (2025) ART: agent reinforcement trainer . GitHub . Note: https://github.com/openpipe/art Cited by: §1 , §2 . G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network . External Links: 1503.02531 , Link Cited by: §3.2.5 . E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021) LoRA: low-rank adaptation of large language models . External Links: 2106.09685 , Link Cited by: §3.2.6 . J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han (2022) Large language models can self-improve . External Links: 2210.11610 , Link Cited by: §3.2.4 . K. Huang, S. Zhang, H. Wang, Y. Qu, Y. Lu, Y. Roohani, R. Li, L. Qiu, G. Li, J. Zhang, D. Yin, S. Marwaha, J. N. Carter, X. Zhou, M. Wheeler, J. A. Bernstein, M. Wang, P. He, J. Zhou, M. Snyder, L. Cong, A. Regev, and J. Leskovec (2025) Biomni: a general-purpose biomedical ai agent . Note: bioRxiv preprint External Links: Document , Link Cited by: §3.2.3 . B. Klimt and Y. Yang (2004) The enron corpus: a new dataset for email classification research . In Machine Learning: ECML 2004 , J. Boulicaut, F. Esposito, F. Giannotti, and D. Pedreschi (Eds.) , Lecture Notes in Computer Science , Vol. 3201 , Berlin, Heidelberg , pp. 217–226 . External Links: ISBN 978-3-540-30115-8 , Document Cited by: §4.1.1 . Inc. LangChain (2025) LangGraph . Note: https://www.langchain.com/langgraph Cited by: §1 , §2 . Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-eval: nlg evaluation using gpt-4 with better human alignment . External Links: 2303.16634 , Link Cited by: §3.2.1 . X. Luo, Y. Zhang, Z. He, Z. Wang, S. Zhao, D. Li, L. K. Qiu, and Y. Yang (2025) Agent lightning: train any ai agents with reinforcement learning . External Links: 2508.03680 , Link Cited by: §1 , §2 . P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu (2018) Mixed precision training . External Links: 1710.03740 , Link Cited by: §3.2.6 . R. Nakano, J. Hilton, O. Balaji, A. Chowdhery, C. Hashme, L. Jiang, V. Kosaraju, G. Krueger, G. Navarro, A. Power, et al. (2021) WebGPT: browser-assisted question-answering with human feedback . arXiv preprint arXiv:2112.09332 . Cited by: §1 . OpenAI (2023) GPT-4 technical report . Note: arXiv preprint arXiv:2303.08774 Cited by: §1 . L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback . arXiv preprint arXiv:2203.02155 . Cited by: §1 , §3.2.1 , §4.1.2 . Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun (2023) ToolLLM: facilitating large language models to master 16000+ real-world apis . External Links: 2307.16789 , Link Cited by: §3.2.3 , §3.2.3 . R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2024) Direct Preference Optimization: Your Language Model is Secretly a Reward Model . arXiv . Note: arXiv:2305.18290 [cs] External Links: Link , Document Cited by: §1 , §3.2.2 . V. Sanh, L. Debut, J. Chaumond, and T. Wolf (2020) DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter . External Links: 1910.01108 , Link Cited by: §3.2.5 . T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, T. Scialom, A. Sridhar, I. Beltagy, J. Launay, J. Schmidhuber, et al. (2023) Toolformer: language models can teach themselves to use tools . arXiv preprint arXiv:2302.04761 . Cited by: §1 , §3.2.3 , §3.2.4 . J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms . External Links: 1707.06347 , Link Cited by: §3.2.2 . Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models . arXiv . Note: arXiv:2402.03300 [cs] External Links: Link , Document Cited by: §3.2.2 , §4.1.2 . N. Shinn, B. Labash, and D. Gopinath (2023) Reflexion: language agents with verbal reinforcement learning . arXiv preprint arXiv:2303.11366 . Cited by: §1 . Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi (2023) Self-instruct: aligning language models with self-generated instructions . External Links: 2212.10560 , Link Cited by: §3.2.4 . J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou (2022) Chain-of-thought prompting elicits reasoning in large language models . In Advances in Neural Information Processing Systems , Cited by: §1 . S. Yao, J. Zhao, D. Yu, S. Peng, K. Narasimhan, E. Cambria, and Y. Yang (2023) ReAct: synergizing reasoning and acting in language models . In Proceedings of the International Conference on Learning Representations (ICLR) , Cited by: §1 . L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging llm-as-a-judge with mt-bench and chatbot arena . External Links: 2306.05685 , Link Cited by: §3.2.1 , §4.1.2 . Appendix Appendix A Implementation Details A.1 Core API for the Email Search Agent Listing 1 demonstrates the core API used to set up the Email Search Agent experiment described in Section 4 . It shows the agent definition and the Brain initialization. Crucially, it utilizes a custom reward function, reward_art_style_judge , which was specifically designed to replicate the direct-assessment logic of the ART ⋅ \cdot E project for a fair comparison of agent performance. This highlights ToolBrain’s flexibility in accommodating highly specialized, user-defined reward signals. ⬇ 1 from smolagents import tool , CodeAgent 2 from toolbrain . models import UnslothModel 3 from toolbrain import Brain 4 from my_project import custom_rewards # Custom reward functions 5 6 # 1. Define the agent’s tools 7 @tool 8 def search_emails ( keywords : list [ str ]) -> list [ dict ]: 9 "" "Searches the email DB for given keywords." "" 10 # ... logic to query the email database ... 11 return email_db . search ( keywords ) 12 13 # 2. Define the agent using the UnslothModel wrapper 14 email_search_agent = CodeAgent ( 15 model = UnslothModel ( model_id = "Qwen/Qwen2.5-7B-Instruct" ), 16 tools =[ search_emails , read_email ] 17 ) 18 19 # 3. Initialize the Brain with the custom, ART-style judge 20 brain = Brain ( 21 agent = email_search_agent , 22 algorithm = "GRPO" , 23 reward_func = custom_rewards . reward_art_style_judge , 24 reward_kwargs ={ "judge_model" : "gpt-4o-mini" }, 25 learning_rate =3 e -5 26 ) 27 28 # 4. Start the training process 29 brain . train ( dataset = email_questions_dataset ) Listing 1: Core API setup for the Email Search Agent experiment. A.2 The Execution Trace: Core Data Structures The Execution Trace is the standardized, high-fidelity "source of truth" for all learning in ToolBrain. It is a list of Turn objects, where each Turn captures a complete interaction cycle. This data structure is generated by the internal Agent Adapter , which acts as an interpreter, translating an agent’s framework-specific memory into this universal format. Listing 2 shows a simplified definition of these core structures, which are crucial for enabling accurate reward computation and RL training. ⬇ 1 class Turn ( TypedDict ): 2 prompt_for_model : str 3 model_completion : str 4 parsed_completion : ParsedCompletion 5 tool_output : Optional [ str ] 6 7 class ParsedCompletion ( TypedDict ): 8 thought : Optional [ str ] 9 tool_code : Optional [ str ] 10 final_answer : Optional [ str ] Listing 2: A simplified definition of the core data structures. Appendix B Reward Design ToolBrain supports two primary paradigms for reward design: simple, user-defined Python functions for straightforward tasks, and powerful LLM-based judges for more complex, nuanced evaluations. B.1 User-Defined Heuristic Rewards For tasks with clear, objective success criteria, users can provide any Python callable as a reward function. This function receives the full execution trace and returns a scalar score. Listing 3 shows an example that rewards efficiency. ⬇ 1 from toolbrain . core_types import Trace 2 3 def reward_step_efficiency ( trace : Trace , ** kwargs ) -> float : 4 "" "Rewards higher for shorter traces." "" 5 max_turns = int ( kwargs . get ( "max_turns" , 5)) 6 num_turns = len ( trace ) 7 8 if num_turns <= max_turns : 9 return 1.0 10 penalty = ( num_turns - max_turns ) * 0.1 11 return max (0.0, 1.0 - penalty ) Listing 3: A simple user-defined reward function. B.2 LLM-as-a-Judge Reward Function For complex tasks without a clear ground-truth, ToolBrain provides a built-in, ranking-based LLM-as-a-judge. This function operates on a batch of traces and does not require a gold_answer , making it ideal for unsupervised learning scenarios. As shown in Listing 4 , the user simply passes this function to the Brain . The framework then automatically handles the process of collecting multiple traces, prompting a judge model to rank them, and converting these ranks into scalar rewards. While our main experiment (Section 4 ) used a custom direct-assessment judge for methodological consistency with prior work, this built-in approach is the recommended, general-purpose solution. ⬇ 1 from toolbrain . rewards import reward_llm_judge_via_ranking 2 3 brain = Brain ( 4 agent = my_agent , 5 algorithm = "GRPO" , 6 reward_func = reward_llm_judge_via_ranking , 7 reward_kwargs ={ "judge_model" : "gemini/gemini-1.5-flash" } 8 ) Listing 4: Using ToolBrain’s built-in, ranking-based LLM judge. Appendix C Learning Algorithms We provide here the GRPO and DPO training procedures referenced in Section 3.2 . Algorithm 1 GRPO training for a single query q q 0: Policy model π θ \pi_{\theta} , reward function R R , group size G G , hyperparameters ϵ , β \epsilon,\beta 1: For i = 1 , … , G i=1,\dots,G , run the agent to obtain a Trace τ i \tau_{i} 2: Compute a scalar reward r i = R ​ ( τ i ) r_{i}=R(\tau_{i}) for each trace 3: Compute group-normalized advantage A ^ i = r i − mean ⁡ ( { r j } j = 1 G ) std ⁡ ( { r j } j = 1 G ) \hat{A}_{i}=\frac{r_{i}-\operatorname{mean}(\{r_{j}\}_{j=1}^{G})}{\operatorname{std}(\{r_{j}\}_{j=1}^{G})} 4: Assemble the GRPO loss ℒ GRPO ​ ( θ ) = − 1 G ​ ∑ i = 1 G 1 | o i | ​ ∑ t = 1 | o i | [ min ⁡ ( ρ i , t ​ A ^ i , t , clip ⁡ ( ρ i , t , 1 − ϵ , 1 + ϵ ) ​ A ^ i , t ) − β ​ D KL ​ ( π θ ∥ π ref ) ] \mathcal{L}_{\text{GRPO}}(\theta)=-\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\Big[\min(\rho_{i,t}\hat{A}_{i,t},\operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t})-\beta D_{\mathrm{KL}}(\pi_{\theta}\|\pi_{\mathrm{ref}})\Big] 5: Update θ ← θ − η ​ ∇ θ ℒ GRPO ​ ( θ ) \theta\leftarrow\theta-\eta\,\nabla_{\theta}\mathcal{L}_{\text{GRPO}}(\theta) Algorithm 2 DPO training for a single query q q 0: Policy π θ \pi_{\theta} , reference policy π ref \pi_{\text{ref}} , group size G G , hyperparameter β \beta 1: For i = 1 , … , G i=1,\dots,G , run the agent to obtain a Trace τ i \tau_{i} 2: Compute rewards and sample a preferred y w y_{w} and dispreferred y ℓ y_{\ell} for each query 3: Define r θ ​ ( y ∣ x ) = log ⁡ π θ ​ ( y ∣ x ) π ref ​ ( y ∣ x ) r_{\theta}(y\mid x)=\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid x)} 4: Assemble the DPO loss ℒ DPO ​ ( θ ) = − log ⁡ σ ​ ( β ​ ( r θ ​ ( y w ∣ x ) − r θ ​ ( y ℓ ∣ x ) ) ) \mathcal{L}_{\text{DPO}}(\theta)=-\log\sigma\left(\beta\big(r_{\theta}(y_{w}\mid x)-r_{\theta}(y_{\ell}\mid x)\big)\right) 5: Update θ ← θ − η ​ ∇ θ ℒ DPO ​ ( θ ) \theta\leftarrow\theta-\eta\,\nabla_{\theta}\mathcal{L}_{\text{DPO}}(\theta) Appendix D Zero-Learn Task Generation Details This section expands on the Zero-Learn mechanism briefly described in Section 3.2 . D.1 Generation API ⬇ 1 from smolagents import CodeAgent 2 from toolbrain import Brain , get_transformer_model 3 4 agent = CodeAgent ( 5 model = TransformersModel ( model_id = "Qwen/Qwen2.5-0.5B-Instruct" ), 6 tools =[ 7 calculate_compound_interest , 8 calculate_loan_payment , 9 calculate_cagr , 10 calculate_npv 11 ], 12 ) 13 14 brain = Brain ( 15 agent = agent , 16 algorithm = "GRPO" 17 # reward_func is not needed for generation 18 ) 19 20 generated_examples = brain . generate_training_examples ( 21 task_description = "Generate tasks to learn to use simple finance tools." , 22 num_examples =100, 23 min_tool_calls =2, 24 max_words =80, 25 self_rank = True 26 ) Listing 5: Generating training examples with Brain and a Qwen model. D.2 Example Query Categories We observed three main categories of generated queries: (i) Executable tool calls ⬇ " Calculate Loan Payment with annual rate of 5%, 7 years , principal of $10 ,000." " Calculate Compound Interest : Principal = 1000, Rate = 0.05, Times Compounded = 12, Years = 10" " What is the compound interest on $10 ,000 at an annual interest rate of 5% for 3 years ?" (ii) Formula or explanatory requests ⬇ " What is the formula for calculating compound interest ?" " What is the formula to calculate the future value of an investment ?" (iii) Out-of-scope or noisy queries ⬇ " Calculate the total cost of a car purchase including insurance and maintenance over 5 years ..." " Calculate Compound Interest on $10 ,000 for 3 years at an annual rate of 5%, then convert this amount to USD using the current exchange rate and compute the NPV ." Out of 100 generated queries, approximately 63% were directly executable, 27% were formula or explanatory, and 10% were noisy or out-of-scope. These statistics informed our filtering and rewriting strategy. Appendix E Knowledge Distillation Pipeline E.1 Algorithm Algorithm 3 ToolBrain Distillation Pipeline 1: Input: Teacher model π T \pi_{T} , student brain B S B_{S} , tool function 𝒯 \mathcal{T} , query set q q 2: Parameters: N = 100 N=100 traces, quality threshold ρ \rho 3: if cached traces exist then 4: Load ( { τ i } , { x i } , { r i } ) (\{\tau_{i}\},\{x_{i}\},\{r_{i}\}) from disk 5: else 6: Initialize teacher agent with π T \pi_{T} and tool 𝒯 \mathcal{T} 7: for i = 1 i=1 to N N do 8: Execute teacher agent on query q q 9: Collect trace τ i \tau_{i} , RL input x i x_{i} , reward r i r_{i} 10: end for 11: Cache ( { τ i } , { x i } , { r i } ) (\{\tau_{i}\},\{x_{i}\},\{r_{i}\}) to disk 12: end if 13: Filter high-quality traces: ℱ = { x i ∣ r i > ρ } \mathcal{F}=\{x_{i}\mid r_{i}>\rho\} 14: if | ℱ | > 0 |\mathcal{F}|>0 then 15: Train student π S \pi_{S} with cross-entropy on ℱ \mathcal{F} : ℒ distill ​ ( θ ) = − 1 | ℱ | ​ ∑ x ∈ ℱ ∑ t = 1 | y | log ⁡ π S ​ ( y t | x , y < t ) \mathcal{L}_{\text{distill}}(\theta)=-\frac{1}{|\mathcal{F}|}\sum_{x\in\mathcal{F}}\sum_{t=1}^{|y|}\log\pi_{S}(y_{t}|x,y_{