Title: 2502.18439 ArXiv: 2502.18439 \ours\faIconcanadian-maple-leaf: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning \ours \faIcon canadian-maple-leaf : Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning Chanwoo Park ♡ Seungju Han ♠ Xingzhi Guo ♢ Asuman Ozdaglar ♡ Kaiqing Zhang ♣ Joo-Kyung Kim ♢ ♡ MIT ♠ Stanford ♢ Amazon ♣ UMD {cpark97,asu}@mit.edu, seungju@stanford.edu , {guxingzh, jookyk}@amazon.com, kaiqing@umd.edu This work was initiated during an internship at Amazon AGI. Abstract Leveraging multiple large language models (LLMs) to build collaborative multi-agentic workflows has demonstrated significant potential. However, most previous studies focus on prompting the out-of-the-box LLMs, relying on their innate capability for collaboration, which may not improve LLMs’ performance as shown recently. In this paper, we introduce a new post-training paradigm \ours ( M ulti- A gent P ost-c o -training for collaborative LLMs with R einforcement L earning), to explicitly elicit the collaborative behaviors and further unleash the power of multi-agentic LLM frameworks. In \ours , multiple LLMs first generate their own responses independently and engage in a multi-turn discussion to collaboratively improve the final answer. In the end, a \ours verifier evaluates both the answer and the discussion, by assigning a score that verifies the correctness of the answer, while adding incentives to encourage corrective and persuasive discussions. The score serves as the co-training reward, and is then maximized through multi-agent RL. Unlike existing LLM post-training paradigms, \ours advocates the co-training of multiple LLMs together using RL for better generalization. Accompanied by analytical insights, our experiments demonstrate that training individual LLMs alone is insufficient to induce effective collaboration. In contrast, multi-agent co-training can boost the collaboration performance across benchmarks, with generalization to unseen domains. The code is available at https://github.com/chanwoo-park-official/MAPoRL . \ours \faIcon canadian-maple-leaf : Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning Chanwoo Park ♡ † † thanks: This work was initiated during an internship at Amazon AGI. Seungju Han ♠ Xingzhi Guo ♢ Asuman Ozdaglar ♡ Kaiqing Zhang ♣ Joo-Kyung Kim ♢ ♡ MIT ♠ Stanford ♢ Amazon ♣ UMD {cpark97,asu}@mit.edu, seungju@stanford.edu , {guxingzh, jookyk}@amazon.com, kaiqing@umd.edu 1 Introduction Recent advances in large language models (LLMs) have highlighted their potential for collaboration, particularly within the multi-agentic framework (Du et al., 2024 ; Li et al., 2023 ; Kim et al., 2024b ) . The shift from single-agent to multi-agent systems introduces new dimensions and challenges in enabling effective collaboration among LLM agents. Recent approaches to multi-LLM collaboration mostly rely on prompting pre-trained models. However, such approaches struggle with achieving genuine collaboration among the agents. For example, multi-agent debate does not consistently lead to improved performance with additional turns (Huang et al., 2024 ) . This limitation may be somewhat expected – while LLMs are able to simulate collaboration procedures, they were not explicitly trained to achieve effective cooperation. In theory, it is not hard to imagine that single-agent training is insufficient for collaboration – an untrained and non-strategic opponent can fail to act in a way that promotes collaboration. Instead, achieving collaborative behaviors requires interactive training environments where each agent actively engages with others, and dynamically optimizes the strategy (Gagne, 1974 ; Macy, 1991 ; Hertz-Lazarowitz et al., 2013 ) . Moreover, conventional approaches such as supervised fine-tuning (SFT), as we will show, are inadequate for this purpose, either: merely mimicking multi-agent interactions from training data may not lead to effective collaboration. To develop more effective collaborative agents, we propose M ulti- A gent P ost-c o -training for collaborative LLMs with R einforcement L earning ( \ours ), a co-training paradigm for multiple LLMs using multi-agent reinforcement learning (MARL). In MAPoRL , within the pre-defined frameworks for multi-agent collaboration (e.g., the debate framework (Du et al., 2024 ) ), each agent receives rewards for their responses during collaboration, based on the quality of their answers and interactions. The objective for each agent in MAPoRL is to maximize their own value function, defined as the expected cumulative sum of rewards over the course of the collaboration. To further encourage cooperation in MAPoRL , we incorporate incentives for successful interactions and penalties for collaboration failures, steering the LLMs toward more effective and aligned behaviors. Through a simplified game-theoretic example, we validate the following insights: 1) single-agent training alone is insufficient to produce genuinely cooperative agents, and 2) co-trained agents can reach an equilibrium that exhibits cooperative behavior. To assess the effectiveness of MAPoRL , we conduct experiments across diverse tasks and evaluation strategies. Specifically, we train multi-agent LLMs for tasks such as mathematical reasoning (GSM8k (Cobbe et al., 2021 ) ) and natural language inference (ANLI (Nie et al., 2020 ) ), comparing their performance against baseline approaches. Additionally, we evaluate the robustness of our method by testing agents on out-of-domain tasks (e.g., training on a NLI task and evaluating on a math dataset), demonstrating the generalization capabilities of our approach. We also explore the collaboration among agents of varying capabilities, by analyzing the impact of training heterogeneous LLMs together. To the best of our knowledge, this study is among the first works to explore the training of multi-LLM systems as a whole 1 1 1 Together with the contemporaneous works Subramaniam et al. ( 2025 ) and Zhao et al. ( 2025 ) , both of which were released within the past month while preparing this paper. In contrast to MAPoRL , the algorithms therein were based on (iterative) SFT, instead of RL. Also, Motwani et al. ( 2024 ) provided a method to train verifier-generation-refiner system with DPO. , using RL, for multi-LLM collaboration. Figure 1: MAPoRL can be applied to any multi-LLM system with a scorer/verifier. In the illustrated example, it is integrated into a collaborative debate system for mathematical problem-solving. LLMs generate responses based on the multi-agent system pipeline, and a scorer/verifier evaluates their outputs. The reward for each LLM is determined based on these scores, which may include both current and future pipeline evaluations. Multi-Agent RL is employed to maximize each agent’s value function. 2 Analytical Insights: Collaborate to Solve Hard Questions In this section, we present a simplified model of multi-LLM collaboration and explain (a) why co-training multiple LLMs is necessary compared to training a single agent, and (b) the role of incentives to further enhance collaboration during training. We validate both aspects through experiments in Section 4 . 2.1 Problem Setup We consider questions that inherently require collaboration for a successful solution. For instance, solving complex mathematical problems often requires collaboration among multiple agents (Liang et al., 2024 ; Du et al., 2024 ) . Beyond mathematics, collaboration can also enhance the performance on tasks related to privacy, factuality, and reliability (Feng et al., 2025 ) . We model the interaction among LLMs as a repeated game with T 𝑇 T italic_T turns. For simplicity, we assume that in each turn, each agent chooses between two actions: Collaborate ( a 0 subscript 𝑎 0 a_{0} italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) or Act Independently ( a 1 subscript 𝑎 1 a_{1} italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ). For a given question q 𝑞 q italic_q , we define C ⁢ ( q ) 𝐶 𝑞 C(q) italic_C ( italic_q ) as a non-negative integer representing the collaboration threshold . The agents achieve collaborative synergy if, over the course of the T 𝑇 T italic_T -turn interactions, the total number of collaborative actions ( a 0 subscript 𝑎 0 a_{0} italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) of all the agents meets or exceeds C ⁢ ( q ) 𝐶 𝑞 C(q) italic_C ( italic_q ) . When collaborative synergy is achieved, each agent receives a reward R syn ⁢ ( q ) = 1 subscript 𝑅 syn 𝑞 1 R_{\text{syn}}(q)=1 italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) = 1 , representing a (near-)guaranteed correct solution. Prior to achieving synergy, agents receive rewards based on their chosen actions: a reward of R col ⁢ ( q ) subscript 𝑅 col 𝑞 R_{\text{col}}(q) italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) for choosing to collaborate ( a 0 ) subscript 𝑎 0 (a_{0}) ( italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) and R ind ⁢ ( q ) subscript 𝑅 ind 𝑞 R_{\text{ind}}(q) italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) for acting independently ( a 1 ) subscript 𝑎 1 (a_{1}) ( italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) , where R col ⁢ ( q ) < R ind ⁢ ( q ) subscript 𝑅 col 𝑞 subscript 𝑅 ind 𝑞 R_{\text{col}}(q) max ⁡ ( 3 ⁢ R col ⁢ ( q ) − 2 ⁢ R ind ⁢ ( q ) , 2 ⁢ R ind ⁢ ( q ) − R col ⁢ ( q ) ) . subscript 𝑅 syn 𝑞 3 subscript 𝑅 col 𝑞 2 subscript 𝑅 ind 𝑞 2 subscript 𝑅 ind 𝑞 subscript 𝑅 col 𝑞 R_{\text{syn}}(q)>\max(3R_{\text{col}}(q)-2R_{\text{ind}}(q),2R_{\text{ind}}(q% )-R_{\text{col}}(q)). italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) > roman_max ( 3 italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) - 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) , 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) - italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) ) . 2 can be proved by adapting the results of Zhang and Hofbauer ( 2016 ) , and transforming our setup with T = 2 𝑇 2 T=2 italic_T = 2 into a matrix game. This observation implies that when both agents optimize their own cumulative reward, they will naturally choose collaboration when R syn ⁢ ( q ) subscript 𝑅 syn 𝑞 R_{\text{syn}}(q) italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) is high enough, which emphasizes the importance of additional incentives to promote collaborative synergy. Due to this observation, in Section 3.3 , we incentivize collaboration by providing a higher R syn ⁢ ( q ) subscript 𝑅 syn 𝑞 R_{\text{syn}}(q) italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) . 2.3 Toy Experiments with T = 10 , 20 𝑇 10 20 T=10,20 italic_T = 10 , 20 Turns We illustrate the benefit of jointly optimized (multi-agent) policies over those obtained from a single-agent approach in our setting, with longer T = 10 , 20 𝑇 10 20 T=10,20 italic_T = 10 , 20 turns. Each question q 𝑞 q italic_q is associated with the rewards R col ⁢ ( q ) subscript 𝑅 col 𝑞 R_{\text{col}}(q) italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) , R ind ⁢ ( q ) subscript 𝑅 ind 𝑞 R_{\text{ind}}(q) italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) , and R syn ⁢ ( q ) subscript 𝑅 syn 𝑞 R_{\text{syn}}(q) italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) , along with a collaboration threshold C ⁢ ( q ) 𝐶 𝑞 C(q) italic_C ( italic_q ) . Further details on the choices of these quantities be found in Appendix D . We first consider a single agent interacting with a fixed opponent whose probability of collaborating, π ⁢ ( q ) 𝜋 𝑞 \pi(q) italic_π ( italic_q ) , is set at { 0.5 , 0.6 , 0.7 } 0.5 0.6 0.7 \{0.5,\,0.6,\,0.7\} { 0.5 , 0.6 , 0.7 } . Despite the relatively high likelihood of collaboration from the opponent, the single-agent policy, which optimizes its response to the fixed opponent, does not result in effective collaboration ( Figure 5 ). Instead of learning to strategically engage with the opponent’s behavior, the single-agent policy, which follows a best-response strategy to the fixed opponent, tends to avoid collaboration, highlighting the limitations of a single-agent framework when facing a fixed , non-strategic opponent. Next, we consider two jointly optimizing (multi-agent) learners who adapt their policy based on the other’s actions. Concretely, we compute an entropy-regularized Nash equilibrium for τ = 0.1 𝜏 0.1 \tau=0.1 italic_τ = 0.1 via backward induction. As shown in Figure 5 (solid red curves), the jointly optimized agents coordinate with significantly higher collaboration rates. Intuitively, this is because their learning process, which fosters high collaboration at larger t 𝑡 t italic_t , shapes their strategic behavior from the very beginning, leading to increased cooperation even at the first turn. These toy results underscore the importance of strategic interactions : when both agents adapt their policies simultaneously, they learn to be more collaborative, despite the temptation of short-term independent rewards. 3 Post-Co-Training with MARL We now provide an overview of our new paradigm of Multi-Agent Post-Co-Training with RL ( \ours ) for LLM collaboration. In our framework, each agent’s response per turn is evaluated by a verifier, which assigns a score reflecting the answer’s validity. The reward is defined as the (weighted) sum of the verifier’s scores from the current turn and those from the future turns, thus capturing both the immediate feedback and the projected long-term impact of the agent’s response. The agents’ policies are updated using multi-agent PPO for each turn, ensuring that the learning process incorporates both the performance in the current turn and the influence of the anticipated collaborative interactions in the future turns. 3.1 Multi-Agent System - Collaborative Debate Formulation We follow the collaborative debate system proposed by Du et al. ( 2024 ) as an example of our multi-LLM system in the experiments. Note that, MAPoRL can be applied to other multi-LLM systems as long as each agent’s response can be evaluated—for example, by a verifier that assigns a score reflecting the quality or correctness of the response. The reward for each agent is then determined by summing the verifier scores of all the responses influenced by that agent throughout the multi-agent interaction process. Assume we have a collaborative debate system that runs for T 𝑇 T italic_T turns and involves A 𝐴 A italic_A agents. In each turn, an LLM must determine its next response based on the history of its own answers as well as the answers provided by other LLM agents. Let q 𝑞 q italic_q be the given question, and let s t ⁢ i subscript 𝑠 𝑡 𝑖 s_{ti} italic_s start_POSTSUBSCRIPT italic_t italic_i end_POSTSUBSCRIPT denote the solution provided by agent i 𝑖 i italic_i at turn t 𝑡 t italic_t . We inductively express the solution s ( t + 1 ) ⁢ i subscript 𝑠 𝑡 1 𝑖 s_{(t+1)i} italic_s start_POSTSUBSCRIPT ( italic_t + 1 ) italic_i end_POSTSUBSCRIPT as follows: s 1 ⁢ i = LLM i ⁢ ( q ) , s ( t + 1 ) ⁢ i = LLM i ⁢ ( q ⊕ j ∈ [ A ] , t ′ ∈ [ t ] s t ′ ⁢ j ) formulae-sequence subscript 𝑠 1 𝑖 subscript LLM 𝑖 𝑞 subscript 𝑠 𝑡 1 𝑖 subscript LLM 𝑖 subscript direct-sum formulae-sequence 𝑗 delimited-[] 𝐴 superscript 𝑡 ′ delimited-[] 𝑡 𝑞 subscript 𝑠 superscript 𝑡 ′ 𝑗 \displaystyle s_{1i}=\text{LLM}_{i}\left(q\right),\,s_{(t+1)i}=\text{LLM}_{i}% \left(q\oplus_{j\in[A],t^{\prime}\in[t]}s_{t^{\prime}j}\right) italic_s start_POSTSUBSCRIPT 1 italic_i end_POSTSUBSCRIPT = LLM start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_q ) , italic_s start_POSTSUBSCRIPT ( italic_t + 1 ) italic_i end_POSTSUBSCRIPT = LLM start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_q ⊕ start_POSTSUBSCRIPT italic_j ∈ [ italic_A ] , italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ [ italic_t ] end_POSTSUBSCRIPT italic_s start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_j end_POSTSUBSCRIPT ) where ⊕ direct-sum \oplus ⊕ denotes token-wise concatenation, 1 ≤ t ≤ T − 1 1 𝑡 𝑇 1 1\leq t\leq T-1 1 ≤ italic_t ≤ italic_T - 1 and LLM i ⁢ ( s ) subscript LLM 𝑖 𝑠 \text{LLM}_{i}(s) LLM start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_s ) represents the function of inputting prompt s 𝑠 s italic_s into the LLM i subscript LLM 𝑖 \text{LLM}_{i} LLM start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT which outputs logits over its token space, followed by sampling a token based on these logits. If A = 1 𝐴 1 A=1 italic_A = 1 , then this setup is equivalent to that of self-correcting LMs Madaan et al. ( 2024 ) . Now, we define 𝜽 = ( θ t ⁢ a ) t ∈ [ T ] , a ∈ [ A ] 𝜽 subscript subscript 𝜃 𝑡 𝑎 formulae-sequence 𝑡 delimited-[] 𝑇 𝑎 delimited-[] 𝐴 \boldsymbol{\theta}=(\theta_{ta})_{t\in[T],a\in[A]} bold_italic_θ = ( italic_θ start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) start_POSTSUBSCRIPT italic_t ∈ [ italic_T ] , italic_a ∈ [ italic_A ] end_POSTSUBSCRIPT , where θ t ⁢ a subscript 𝜃 𝑡 𝑎 \theta_{ta} italic_θ start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT represents the parameters of the a 𝑎 a italic_a -th agent at turn t 𝑡 t italic_t . We denote LLM parameterized by θ t ⁢ a subscript 𝜃 𝑡 𝑎 \theta_{ta} italic_θ start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT as L ⁢ L ⁢ M θ t ⁢ a 𝐿 𝐿 subscript 𝑀 subscript 𝜃 𝑡 𝑎 LLM_{\theta_{ta}} italic_L italic_L italic_M start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT end_POSTSUBSCRIPT . Next, to implement MAPoRL , we define the reward function for the multi-agent RL formulation, using the verifier score. Due to space constraints, we here introduce the Influence-aware Verification Rewards , and defer other choices of the reward functions that we will use in the experiments to Appendix F . Definition 1 (Influence-aware Verification Reward) . The influence-aware verification reward function R 𝛉 ⁢ ( q , s t ⁢ a ) subscript 𝑅 𝛉 𝑞 subscript 𝑠 𝑡 𝑎 R_{\boldsymbol{\theta}}(q,s_{ta}) italic_R start_POSTSUBSCRIPT bold_italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) is defined as R 𝜽 ( q , s t ⁢ a ) = 𝔼 [ 1 ∑ t ′ ∈ [ t , T ] γ t ′ − t ( Verifier ( q , s t ⁢ a ) \displaystyle R_{\boldsymbol{\theta}}(q,s_{ta})=\mathbb{E}\bigg{[}\frac{1}{% \sum_{t^{\prime}\in[t,T]}\gamma^{t^{\prime}-t}}\biggl{(}\text{Verifier}(q,s_{% ta}) italic_R start_POSTSUBSCRIPT bold_italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) = blackboard_E [ divide start_ARG 1 end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ [ italic_t , italic_T ] end_POSTSUBSCRIPT italic_γ start_POSTSUPERSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT - italic_t end_POSTSUPERSCRIPT end_ARG ( Verifier ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) + ∑ t ′ ∈ [ t + 1 , T ] ∑ j ∈ [ A ] 1 A γ t ′ − t Verifier ( q , s t ′ ⁢ j ) ) ] . \displaystyle\,+\sum_{t^{\prime}\in[t+1,T]}\sum_{j\in[A]}\frac{1}{A}\gamma^{t^% {\prime}-t}\text{Verifier}(q,s_{t^{\prime}j})\biggr{)}\bigg{]}. + ∑ start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ [ italic_t + 1 , italic_T ] end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_j ∈ [ italic_A ] end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_A end_ARG italic_γ start_POSTSUPERSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT - italic_t end_POSTSUPERSCRIPT Verifier ( italic_q , italic_s start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_j end_POSTSUBSCRIPT ) ) ] . Here, the expectation arises from the randomness of other agents’ answers, which are influenced by the agent’s current response, and γ ∈ [ 0 , 1 ] 𝛾 0 1 \gamma\in[0,1] italic_γ ∈ [ 0 , 1 ] is a discount factor. This reward not only considers the verifier’s score for the current solution s t ⁢ a subscript 𝑠 𝑡 𝑎 s_{ta} italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT , but also incorporates the impact of this solution on the future answers of all agents. The term ∑ j ∈ [ A ] 1 A subscript 𝑗 delimited-[] 𝐴 1 𝐴 \sum_{j\in[A]}\frac{1}{A} ∑ start_POSTSUBSCRIPT italic_j ∈ [ italic_A ] end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_A end_ARG averages the verifier’s scores across all the agents, reflecting the influence that s t ⁢ a subscript 𝑠 𝑡 𝑎 s_{ta} italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT has on the overall multi-agent system. 3.2 Multi-Agent RL Formulation The reward of each agent, as well as its answer generation, is intertwined with the actions of other agents in the multi-LLM system with the reward function ( Definition 1 ). Thus, instead of single-agent RL, we design a multi-agent RL approach. For this paper, we choose multi-agent PPO (Yu et al., 2022 ) as a representative multi-agent RL algorithm and instantiate it in the language domain. Conventional multi-agent PPO extends PPO to multi-agent settings by training decentralized policies, either with a shared critic or through independent learning, where each agent optimizes its policy based on local observations and rewards. Typically, agents learn independently or with limited coordination mechanisms, such as centralized critics or shared value functions. Our approach adapts multi-agent PPO by defining the state as the concatenation of the multi-agent interaction history, allowing agents to condition their responses on past interactions. Additionally, we introduce reward structures that are aligned but not fully identical across agents, encouraging them to fulfill different roles while collectively working toward solving the task. Since we are solving multi-turn problems, the value function for each turn’s state needs to be defined. The state of each turn’s value V t ⁢ a ⁢ 𝜽 subscript 𝑉 𝑡 𝑎 𝜽 V_{ta\boldsymbol{\theta}} italic_V start_POSTSUBSCRIPT italic_t italic_a bold_italic_θ end_POSTSUBSCRIPT is the expectation of the cumulated reward conditioned on the input text i t ⁢ a x superscript subscript 𝑖 𝑡 𝑎 𝑥 i_{ta}^{x} italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT , which is defined as V t ⁢ a ⁢ 𝜽 ⁢ ( i t ⁢ a x ) = 𝔼 ⁢ [ ∑ x ′ = x length ⁢ ( s t ⁢ a ) r 𝜽 ⁢ ( q , s t ⁢ a 1 : x ′ ) | q , i t ⁢ a x ] . subscript 𝑉 𝑡 𝑎 𝜽 superscript subscript 𝑖 𝑡 𝑎 𝑥 𝔼 delimited-[] conditional superscript subscript superscript 𝑥 ′ 𝑥 length subscript 𝑠 𝑡 𝑎 subscript 𝑟 𝜽 𝑞 superscript subscript 𝑠 𝑡 𝑎 : 1 superscript 𝑥 ′ 𝑞 superscript subscript 𝑖 𝑡 𝑎 𝑥 \displaystyle V_{ta\boldsymbol{\theta}}(i_{ta}^{x})=\mathbb{E}\left[\sum_{x^{% \prime}=x}^{\texttt{length}(s_{ta})}r_{\boldsymbol{\theta}}(q,s_{ta}^{1:x^{% \prime}}){\,\bigg{|}\,}q,i_{ta}^{x}\right]. italic_V start_POSTSUBSCRIPT italic_t italic_a bold_italic_θ end_POSTSUBSCRIPT ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ) = blackboard_E [ ∑ start_POSTSUBSCRIPT italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = italic_x end_POSTSUBSCRIPT start_POSTSUPERSCRIPT length ( italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) end_POSTSUPERSCRIPT italic_r start_POSTSUBSCRIPT bold_italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 : italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ) | italic_q , italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ] . Here, i t ⁢ a x ′ = q ⊕ t ′ ∈ [ t − 1 ] , j ∈ [ A ] s t ′ ⁢ j ⊕ s t ⁢ a 1 : x ′ superscript subscript 𝑖 𝑡 𝑎 superscript 𝑥 ′ direct-sum subscript direct-sum formulae-sequence superscript 𝑡 ′ delimited-[] 𝑡 1 𝑗 delimited-[] 𝐴 𝑞 subscript 𝑠 superscript 𝑡 ′ 𝑗 subscript superscript 𝑠 : 1 superscript 𝑥 ′ 𝑡 𝑎 i_{ta}^{x^{\prime}}=q\oplus_{t^{\prime}\in[t-1],j\in[A]}s_{t^{\prime}j}\oplus s% ^{1:{x^{\prime}}}_{ta} italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT = italic_q ⊕ start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ [ italic_t - 1 ] , italic_j ∈ [ italic_A ] end_POSTSUBSCRIPT italic_s start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_j end_POSTSUBSCRIPT ⊕ italic_s start_POSTSUPERSCRIPT 1 : italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT and r 𝜽 ⁢ ( q , s t ⁢ a 1 : x ′ ) = 𝟏 ⁢ ( x ′ = length ⁢ ( s t ⁢ a ) ) ⁢ R 𝜽 ⁢ ( q , s t ⁢ a ) subscript 𝑟 𝜽 𝑞 superscript subscript 𝑠 𝑡 𝑎 : 1 superscript 𝑥 ′ 1 superscript 𝑥 ′ length subscript 𝑠 𝑡 𝑎 subscript 𝑅 𝜽 𝑞 subscript 𝑠 𝑡 𝑎 \displaystyle r_{\boldsymbol{\theta}}(q,s_{ta}^{1:x^{\prime}})=\boldsymbol{1}(% x^{\prime}=\texttt{length}(s_{ta}))R_{\boldsymbol{\theta}}(q,s_{ta}) italic_r start_POSTSUBSCRIPT bold_italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 : italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ) = bold_1 ( italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = length ( italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) ) italic_R start_POSTSUBSCRIPT bold_italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) − λ KL ⁢ KL ⁢ ( LLM θ ref , t ⁢ a ⁢ ( i t ⁢ a x ′ ) ∥ LLM θ t ⁢ a ⁢ ( i t ⁢ a x ′ ) ) , subscript 𝜆 KL KL conditional subscript LLM subscript 𝜃 ref 𝑡 𝑎 superscript subscript 𝑖 𝑡 𝑎 superscript 𝑥 ′ subscript LLM subscript 𝜃 𝑡 𝑎 superscript subscript 𝑖 𝑡 𝑎 superscript 𝑥 ′ \displaystyle\,-\lambda_{\text{KL}}\text{KL}\left(\text{LLM}_{\theta_{\text{% ref},ta}}(i_{ta}^{x^{\prime}})\,\|\,\text{LLM}_{\theta_{ta}}(i_{ta}^{x^{\prime% }})\right), - italic_λ start_POSTSUBSCRIPT KL end_POSTSUBSCRIPT KL ( LLM start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT ref , italic_t italic_a end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ) ∥ LLM start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ) ) , where t 𝑡 t italic_t denotes the turn index and a 𝑎 a italic_a refers to agent a ∈ [ A ] 𝑎 delimited-[] 𝐴 a\in[A] italic_a ∈ [ italic_A ] , s t ⁢ a 1 : x superscript subscript 𝑠 𝑡 𝑎 : 1 𝑥 s_{ta}^{1:x} italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT represents the generated token from agent a 𝑎 a italic_a up to the x 𝑥 x italic_x -th token in turn t 𝑡 t italic_t , with θ ref , t ⁢ a subscript 𝜃 ref 𝑡 𝑎 \theta_{\text{ref},ta} italic_θ start_POSTSUBSCRIPT ref , italic_t italic_a end_POSTSUBSCRIPT denoting the parameter of a reference LLM, and λ KL ≥ 0 subscript 𝜆 KL 0 \lambda_{\text{KL}}\geq 0 italic_λ start_POSTSUBSCRIPT KL end_POSTSUBSCRIPT ≥ 0 is some regularization coefficient. As per our reward construction, the value maximization not only considers the current turn’s verifier score, but also anticipates future verifier scores from the same or other agents across multiple turns, which makes multi-agent training relevant. We estimate the advantage function using Generalized Advantage Estimation (GAE) (Schulman et al., 2016 ) , which leverages the value function to measure how much better the current token selection is compared to the baseline value function. The value function is approximated by a neural network with parameter θ v ⁢ t ⁢ a subscript 𝜃 𝑣 𝑡 𝑎 \theta_{vta} italic_θ start_POSTSUBSCRIPT italic_v italic_t italic_a end_POSTSUBSCRIPT , denoted as V ⁢ ( i t ⁢ a x ; θ v ⁢ t ⁢ a ) 𝑉 superscript subscript 𝑖 𝑡 𝑎 𝑥 subscript 𝜃 𝑣 𝑡 𝑎 V(i_{ta}^{x};\theta_{vta}) italic_V ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ; italic_θ start_POSTSUBSCRIPT italic_v italic_t italic_a end_POSTSUBSCRIPT ) , which serves as an estimate of V t ⁢ a ⁢ 𝜽 ⁢ ( i t ⁢ a x ) subscript 𝑉 𝑡 𝑎 𝜽 superscript subscript 𝑖 𝑡 𝑎 𝑥 V_{ta\boldsymbol{\theta}}(i_{ta}^{x}) italic_V start_POSTSUBSCRIPT italic_t italic_a bold_italic_θ end_POSTSUBSCRIPT ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ) . Using V ⁢ ( i t ⁢ a x ; θ v ⁢ t ⁢ a ) 𝑉 superscript subscript 𝑖 𝑡 𝑎 𝑥 subscript 𝜃 𝑣 𝑡 𝑎 V(i_{ta}^{x};\theta_{vta}) italic_V ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ; italic_θ start_POSTSUBSCRIPT italic_v italic_t italic_a end_POSTSUBSCRIPT ) , we estimate the advantage function A ⁢ ( i t ⁢ a x ; 𝜽 , θ v ⁢ t ⁢ a ) 𝐴 superscript subscript 𝑖 𝑡 𝑎 𝑥 𝜽 subscript 𝜃 𝑣 𝑡 𝑎 A(i_{ta}^{x};\boldsymbol{\theta},\theta_{vta}) italic_A ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ; bold_italic_θ , italic_θ start_POSTSUBSCRIPT italic_v italic_t italic_a end_POSTSUBSCRIPT ) via GAE. The loss function for multi-agent PPO is then given by: L PPO ⁢ ( θ t ⁢ a , θ v ⁢ t ⁢ a ) = L Surrogate ⁢ ( θ t ⁢ a ) + L Value ⁢ ( θ v ⁢ t ⁢ a ) , subscript 𝐿 PPO subscript 𝜃 𝑡 𝑎 subscript 𝜃 𝑣 𝑡 𝑎 subscript 𝐿 Surrogate subscript 𝜃 𝑡 𝑎 subscript 𝐿 Value subscript 𝜃 𝑣 𝑡 𝑎 \displaystyle L_{\text{PPO}}(\theta_{ta},\theta_{vta})=L_{\text{Surrogate}}(% \theta_{ta})+L_{\text{Value}}(\theta_{vta}), italic_L start_POSTSUBSCRIPT PPO end_POSTSUBSCRIPT ( italic_θ start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT , italic_θ start_POSTSUBSCRIPT italic_v italic_t italic_a end_POSTSUBSCRIPT ) = italic_L start_POSTSUBSCRIPT Surrogate end_POSTSUBSCRIPT ( italic_θ start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) + italic_L start_POSTSUBSCRIPT Value end_POSTSUBSCRIPT ( italic_θ start_POSTSUBSCRIPT italic_v italic_t italic_a end_POSTSUBSCRIPT ) , where L Surrogate ⁢ ( θ t ⁢ a ) subscript 𝐿 Surrogate subscript 𝜃 𝑡 𝑎 L_{\text{Surrogate}}(\theta_{ta}) italic_L start_POSTSUBSCRIPT Surrogate end_POSTSUBSCRIPT ( italic_θ start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) is defined as 𝔼 [ \displaystyle\mathbb{E}\Bigl{[} blackboard_E [ min ( R t ⁢ a x A ( i t ⁢ a x ; 𝜽 old , θ old , v ⁢ t ⁢ a ) , \displaystyle\min\Bigl{(}R_{ta}^{x}\,A\bigl{(}i_{ta}^{x};\boldsymbol{\theta}_{% \text{old}},\theta_{\text{old},vta}\bigr{)}, roman_min ( italic_R start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT italic_A ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ; bold_italic_θ start_POSTSUBSCRIPT old end_POSTSUBSCRIPT , italic_θ start_POSTSUBSCRIPT old , italic_v italic_t italic_a end_POSTSUBSCRIPT ) , clip ϵ ( R t ⁢ a x ) A ( i t ⁢ a x ; 𝜽 old , θ old , v ⁢ t ⁢ a ) ) ] . \displaystyle\qquad\qquad\texttt{clip}_{\epsilon}\bigl{(}R_{ta}^{x}\bigr{)}\,A% \bigl{(}i_{ta}^{x};\boldsymbol{\theta}_{\text{old}},\theta_{\text{old},vta}% \bigr{)}\Bigr{)}\Bigr{]}. clip start_POSTSUBSCRIPT italic_ϵ end_POSTSUBSCRIPT ( italic_R start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ) italic_A ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ; bold_italic_θ start_POSTSUBSCRIPT old end_POSTSUBSCRIPT , italic_θ start_POSTSUBSCRIPT old , italic_v italic_t italic_a end_POSTSUBSCRIPT ) ) ] . and L Value ⁢ ( θ v ⁢ t ⁢ a ) subscript 𝐿 Value subscript 𝜃 𝑣 𝑡 𝑎 L_{\text{Value}}(\theta_{vta}) italic_L start_POSTSUBSCRIPT Value end_POSTSUBSCRIPT ( italic_θ start_POSTSUBSCRIPT italic_v italic_t italic_a end_POSTSUBSCRIPT ) is defined as L Value ⁢ ( θ v ⁢ t ⁢ a ) = 𝔼 ⁢ [ λ value ⁢ ( V ⁢ ( i t ⁢ a x ; θ v ⁢ t ⁢ a ) − V t ⁢ a target ⁢ ( i t ⁢ a x ) ) 2 ] . subscript 𝐿 Value subscript 𝜃 𝑣 𝑡 𝑎 𝔼 delimited-[] subscript 𝜆 value superscript 𝑉 superscript subscript 𝑖 𝑡 𝑎 𝑥 subscript 𝜃 𝑣 𝑡 𝑎 subscript superscript 𝑉 target 𝑡 𝑎 superscript subscript 𝑖 𝑡 𝑎 𝑥 2 \displaystyle L_{\text{Value}}(\theta_{vta})=\mathbb{E}\left[\lambda_{\text{% value}}\left(V(i_{ta}^{x};\theta_{vta})-V^{\text{target}}_{ta}(i_{ta}^{x})% \right)^{2}\right]. italic_L start_POSTSUBSCRIPT Value end_POSTSUBSCRIPT ( italic_θ start_POSTSUBSCRIPT italic_v italic_t italic_a end_POSTSUBSCRIPT ) = blackboard_E [ italic_λ start_POSTSUBSCRIPT value end_POSTSUBSCRIPT ( italic_V ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ; italic_θ start_POSTSUBSCRIPT italic_v italic_t italic_a end_POSTSUBSCRIPT ) - italic_V start_POSTSUPERSCRIPT target end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ) ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] . Here, clip ϵ ⁢ ( α ) := min ⁡ ( max ⁡ ( 1 − ϵ , α ) , 1 + ϵ ) assign subscript clip italic-ϵ 𝛼 1 italic-ϵ 𝛼 1 italic-ϵ \texttt{clip}_{\epsilon}(\alpha):=\min(\max(1-\epsilon,\alpha),1+\epsilon) clip start_POSTSUBSCRIPT italic_ϵ end_POSTSUBSCRIPT ( italic_α ) := roman_min ( roman_max ( 1 - italic_ϵ , italic_α ) , 1 + italic_ϵ ) , R t ⁢ a x = LLM θ , t ⁢ a ⁢ ( s t ⁢ a x + 1 ∣ i t ⁢ a x ) LLM θ old , t ⁢ a ⁢ ( s t ⁢ a x + 1 ∣ i t ⁢ a x ) superscript subscript 𝑅 𝑡 𝑎 𝑥 subscript LLM 𝜃 𝑡 𝑎 conditional subscript superscript 𝑠 𝑥 1 𝑡 𝑎 superscript subscript 𝑖 𝑡 𝑎 𝑥 subscript LLM subscript 𝜃 old 𝑡 𝑎 conditional subscript superscript 𝑠 𝑥 1 𝑡 𝑎 superscript subscript 𝑖 𝑡 𝑎 𝑥 R_{ta}^{x}=\frac{\text{LLM}_{\theta,ta}(s^{x+1}_{ta}\mid i_{ta}^{x})}{\text{% LLM}_{\theta_{\text{old},ta}}(s^{x+1}_{ta}\mid i_{ta}^{x})} italic_R start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT = divide start_ARG LLM start_POSTSUBSCRIPT italic_θ , italic_t italic_a end_POSTSUBSCRIPT ( italic_s start_POSTSUPERSCRIPT italic_x + 1 end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ∣ italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ) end_ARG start_ARG LLM start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT old , italic_t italic_a end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_s start_POSTSUPERSCRIPT italic_x + 1 end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ∣ italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ) end_ARG , 𝜽 old = ( θ old , t ⁢ a ) t ∈ [ T ] , a ∈ [ A ] subscript 𝜽 old subscript subscript 𝜃 old 𝑡 𝑎 formulae-sequence 𝑡 delimited-[] 𝑇 𝑎 delimited-[] 𝐴 \boldsymbol{\theta}_{\text{old}}=(\theta_{\text{old},{ta}})_{t\in[T],a\in[A]} bold_italic_θ start_POSTSUBSCRIPT old end_POSTSUBSCRIPT = ( italic_θ start_POSTSUBSCRIPT old , italic_t italic_a end_POSTSUBSCRIPT ) start_POSTSUBSCRIPT italic_t ∈ [ italic_T ] , italic_a ∈ [ italic_A ] end_POSTSUBSCRIPT is the parameter used in the rollout for multi-agent PPO, and V t ⁢ a target ( i t ⁢ a x ) = V ( i t ⁢ a x ; θ old , v ⁢ t ⁢ a ) + A ( i t ⁢ a x ; 𝜽 old , θ old , v ⁢ t ⁢ a ) ) . \displaystyle V_{ta}^{\text{target}}(i_{ta}^{x})=V(i_{ta}^{x};\theta_{\text{% old},vta})+A(i_{ta}^{x};\boldsymbol{\theta}_{\text{old}},\theta_{\text{old},% vta})). italic_V start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT target end_POSTSUPERSCRIPT ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ) = italic_V ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ; italic_θ start_POSTSUBSCRIPT old , italic_v italic_t italic_a end_POSTSUBSCRIPT ) + italic_A ( italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT ; bold_italic_θ start_POSTSUBSCRIPT old end_POSTSUBSCRIPT , italic_θ start_POSTSUBSCRIPT old , italic_v italic_t italic_a end_POSTSUBSCRIPT ) ) . The expectation 𝔼 𝔼 \mathbb{E} blackboard_E is taken over the randomness from q ∼ 𝒬 , s t ′ ⁢ a ′ ∼ LLM θ old , t ′ ⁢ a ′ ⁢ ( q ⊕ t ′′ ∈ [ t ′ − 1 ] , j ∈ [ A ] s t ′ ⁢ j ) formulae-sequence similar-to 𝑞 𝒬 similar-to subscript 𝑠 superscript 𝑡 ′ superscript 𝑎 ′ subscript LLM subscript 𝜃 old superscript 𝑡 ′ superscript 𝑎 ′ subscript direct-sum formulae-sequence superscript 𝑡 ′′ delimited-[] superscript 𝑡 ′ 1 𝑗 delimited-[] 𝐴 𝑞 subscript 𝑠 superscript 𝑡 ′ 𝑗 q\sim\mathcal{Q},s_{t^{\prime}a^{\prime}}\sim\text{LLM}_{\theta_{\text{old},t^% {\prime}a^{\prime}}}(q\oplus_{t^{\prime\prime}\in[t^{\prime}-1],j\in[A]}s_{t^{% \prime}j}) italic_q ∼ caligraphic_Q , italic_s start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_a start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUBSCRIPT ∼ LLM start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT old , italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_a start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_q ⊕ start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ ′ end_POSTSUPERSCRIPT ∈ [ italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT - 1 ] , italic_j ∈ [ italic_A ] end_POSTSUBSCRIPT italic_s start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_j end_POSTSUBSCRIPT ) for all t ′ ∈ [ t ] , a ∈ [ A ] formulae-sequence superscript 𝑡 ′ delimited-[] 𝑡 𝑎 delimited-[] 𝐴 t^{\prime}\in[t],a\in[A] italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ [ italic_t ] , italic_a ∈ [ italic_A ] , and x ∼ Unif ⁢ ( [ length ⁢ ( s t ⁢ a ) ] ) similar-to 𝑥 Unif delimited-[] length subscript 𝑠 𝑡 𝑎 x\sim\text{Unif}([\texttt{length}(s_{ta})]) italic_x ∼ Unif ( [ length ( italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) ] ) , where 𝒬 𝒬 \mathcal{Q} caligraphic_Q denotes the distribution of questions. Each agent for each turn optimizes its policy and value function simultaneously, over the parameters ( θ t ⁢ a , θ v ⁢ t ⁢ a ) subscript 𝜃 𝑡 𝑎 subscript 𝜃 𝑣 𝑡 𝑎 (\theta_{ta},\theta_{vta}) ( italic_θ start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT , italic_θ start_POSTSUBSCRIPT italic_v italic_t italic_a end_POSTSUBSCRIPT ) . These agent interactions among multiple LLMs inherently lead to a multi-agent RL problem, rather than a single-agent RL one, as each agent influences others’ learning processes throughout training. 3.3 Reward Shaping to Incentivize Collaboration As discussed in Section 2 , incorporating additional incentives in the reward can steer agents towards better collaboration. We define four key parameters when implementing such a reward-shaping: parameters α 0 subscript 𝛼 0 \alpha_{0} italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and α 1 subscript 𝛼 1 \alpha_{1} italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT correspond to the incentives related to an agent’s own revision of the answer, and parameters β 0 subscript 𝛽 0 \beta_{0} italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and β 1 subscript 𝛽 1 \beta_{1} italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT correspond to those related to her influence on other agents’ answers. Specifically, α 0 subscript 𝛼 0 \alpha_{0} italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT represents the ability to extract useful information from incorrect answers ( critical reasoning ), while α 1 subscript 𝛼 1 \alpha_{1} italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT reflects an agent’s tendency to be persuaded by the correct information. Meanwhile, β 0 subscript 𝛽 0 \beta_{0} italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT represents the ability to provide incorrect answers that still contain useful information, potentially leading to better responses in the future turns. In contrast, β 1 subscript 𝛽 1 \beta_{1} italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT captures an agent’s ability to effectively persuade others when providing correct answers. We provide Table 1 and Table 2 to summarize the design of these incentives. Answer (t) Answer (t+1) Majority (t) Incentive R W R - α 1 subscript 𝛼 1 \alpha_{1} italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT R W W - α 0 subscript 𝛼 0 \alpha_{0} italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT W R W α 0 subscript 𝛼 0 \alpha_{0} italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT W R R α 1 subscript 𝛼 1 \alpha_{1} italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT Table 1: The design of additional incentives regarding an agent’s own answer revision in MAPoRL . The incentive is determined by how an agent changes its answer between consecutive turns ( t 𝑡 t italic_t and t + 1 𝑡 1 t+1 italic_t + 1 ) relative to the majority opinion of others. R indicates a correct answer, W indicates an incorrect answer. The incentive value is applied to the agent’s answer at turn t + 1 𝑡 1 t+1 italic_t + 1 . Majority (t) Majority (t+1) Answer (t) Incentive R W R - β 1 subscript 𝛽 1 \beta_{1} italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT R W W - β 0 subscript 𝛽 0 \beta_{0} italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT W R W β 0 subscript 𝛽 0 \beta_{0} italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT W R R β 1 subscript 𝛽 1 \beta_{1} italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT Table 2: The design of additional incentives regarding an agent’s influence on other agents’ answers in MAPoRL . The incentive is based on how the majority opinion changes between consecutive turns t 𝑡 t italic_t and t + 1 𝑡 1 t+1 italic_t + 1 relative to the agent’s answer at turn t 𝑡 t italic_t . The incentive value is applied to the agent’s answer at turn t 𝑡 t italic_t . 4 Experiments 4.1 Datasets We evaluate MAPoRL on two benchmark NLP tasks to validate its performance in both mathematical reasoning and logical natural language inference. The details are summarized as follows: GSM8K (Cobbe et al., 2021 ) and TinyGSM (Liu et al., 2023 ) . GSM8K is a benchmark dataset designed to assess a model’s mathematical reasoning abilities, requiring models to solve high-school-level mathematics problems. TinyGSM is an augmented version of GSM8K, generated using GPT-3.5-turbo, where solutions are provided in Python. Importantly, we did not utilize the reasoning processes from GSM8K or TinyGSM but rely solely on their final answers. For training the verifier model, we used 7,463 samples from GSM8K. Additionally, we incorporated the first 12,800 synthetic samples from TinyGSM for \ours 2 2 2 We divided the dataset for training the verifier and training \ours to prevent overfitting LLMs to the trained verifiers. . For evaluation, we hold out 1,319 samples from GSM8K as a test set. Adversarial Natural Language Inference (ANLI) (Nie et al., 2020 ) . ANLI is designed to evaluate a model’s natural language understanding by presenting adversarial examples that challenge even the state-of-the-art models. To train the verifier model, we used first 10,000 training examples. Furthermore, we used the next 12,800 examples for \ours training and 1,200 samples for testing. Evaluation Method. After all turns are completed, the final answer is determined using a majority voting scheme among the agents’ responses. The accuracy is based on whether the majority-selected response is correct. In cases where no clear majority winner emerges (e.g., a tie in vote counts), we adopt an expectation-based approach by weighting the correctness of each tied response proportionally. For example, if two agents receive an equal number of votes, the final score is adjusted as the expected accuracy of selecting the first agent’s answer as the final result. This ensures a continuous evaluation metric rather than an arbitrary tiebreaker. 4.2 Models We primarily use the Microsoft Phi-3-mini-128k-instruct (3.4B) model (Abdin et al., 2024 ) , together with Qwen2.5-3B-instruct (Yang et al., 2024 ) and Llama-3-8B-instruct (Dubey et al., 2024 ) for the experiments. Due to computational constraints, we mainly use quantized models and fine-tuned them with QLoRA (Dettmers et al., 2024 ) . We defer the training details to Appendix G . When evaluating on GSM8K and ANLI, we set the max token length to 300 and 250, respectively. 4.3 Experiment 1: Vanilla Collaboration by Off-the-shelf LLMs Cannot Improve Performance, While \ours -Trained LLMs Can We first compare the collaboration performance of off-the-shelf LLMs with \ours -trained LLMs. The training was conducted with two agents collaborating over three turns. An overview of the trained system is provided in Figure 1 . In Experiment 1, we trained the model starting from turn t ≥ 2 𝑡 2 t\geq 2 italic_t ≥ 2 for two reasons: (a) the first turn primarily focuses on knowledge acquisition from each dataset, and (b) to ensure a fair comparison with off-the-shelf LLMs. We focus on enhancing collaboration skills rather than teaching specific task knowledge. For this experiment, we used Phi-3-mini-128k-instruct and evaluate the trained models in a three-agent and three-turn collaboration environment. We observe that even when the off-the-shelf LLM is allowed to generate longer reasoning (600 tokens, twice the output length of our \ours -trained model model), its accuracy did not improve across turns. This aligns with prior findings in the literature, particularly for models that are not sufficiently strong. For instance, Huang et al. ( 2024 , Table 7) provided evidence that additional turns do not necessarily improve the performance significantly. Similarly, our results show that off-the-shelf LLMs’ performance may not benefit from additional turns. In contrast, LLMs trained using \ours exhibit improved performance as the number of collaboration turns increased, as shown in Figure 2 . Figure 2: Performance comparison of different LLMs across tasks (left: GSM8k, right: ANLI) under various settings. We evaluate collaboration ability in five conditions: (1) off-the-shelf LLMs collaborating and (2) models trained using MAPoRL collaborating (with all incentive parameters ( Section 4.4 ) α , β = 0 , 1 , 2 formulae-sequence 𝛼 𝛽 0 1 2 \alpha,\beta=0,1,2 italic_α , italic_β = 0 , 1 , 2 , respectively). Remark 2 (Domain-Specific Knowledge Acquisition vs. Collaboration Ability Improvement) . One might question whether the performance gains observed in \ours -trained models stem from acquiring domain-specific knowledge rather than improved collaboration ability. To address this, we compare off-the-shelf LLMs and \ours -trained models by testing how well they perform on questions without any collaboration, providing \ours -trained models only the original question – without interaction history – to check if their performance is solely due to domain knowledge learned during training. The results are as follows: Phi-3 MAPoRL T2 MAPoRL T3 GSM8k 0.609 0.604 0.611 ANLI 0.451 0.458 0.453 Here, we provide the same questions to the off-the-shelf Phi-3 model, the \ours -trained turn-2 model, and the \ours -trained turn-3 model. The similar performance across these models suggests that MAPoRL training did not enhance task-specific knowledge but rather improved the models’ ability to collaborate effectively. We also provide the changes in the fraction of responses that transition their correctness over multiple turns of \ours . The fraction of Incorrect → → \to → Incorrect responses decreased, and the fraction of Correct → → \to → Incorrect responses also decreased, indicating that MAPoRL enhanced effective collaboration. Figure 3: Changes in the fraction of responses that transition their correctness over multiple turns of MAPoRL on GSM8k. Figure 4: Changes in the fraction of responses that transition their correctness over multiple turns of MAPoRL on ANLI. 4.4 Experiment 2: Reward Shaping with Collaboration Incentives In addition to the multi-agent independent PPO framework, we then investigate the auxiliary incentive mechanism designed to enhance collaborative interactions. To analyze the impact of the incentive parameters ( α 𝛼 \alpha italic_α and β 𝛽 \beta italic_β , Section 3.3 ), we simplify our experimental setup by limiting the total number of debate turns to 2 and analyze the following cases. Here, α 0 subscript 𝛼 0 \alpha_{0} italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and α 1 subscript 𝛼 1 \alpha_{1} italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT correspond to incentives for an agent’s own revision, capturing critical reasoning (extracting useful information from incorrect answers) and persuadability (accepting correct information), respectively. Meanwhile, β 0 subscript 𝛽 0 \beta_{0} italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and β 1 subscript 𝛽 1 \beta_{1} italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT correspond to incentives for influencing others, where β 0 subscript 𝛽 0 \beta_{0} italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT encourages providing incorrect but useful responses, and β 1 subscript 𝛽 1 \beta_{1} italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT reflects an agent’s ability to persuade others with correct answers. To analyze the impact of the incentive parameters ( α 𝛼 \alpha italic_α and β 𝛽 \beta italic_β , Section 3.3 ), we simplify our experimental setup by limiting the total number of debate turns to 2 and analyze the following cases. Here, α 0 subscript 𝛼 0 \alpha_{0} italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and α 1 subscript 𝛼 1 \alpha_{1} italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT correspond to the incentives related to an agent’s own revision of the answer, while β 0 subscript 𝛽 0 \beta_{0} italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and β 1 subscript 𝛽 1 \beta_{1} italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT correspond to the incentives related to the agent’s influence on other agents’ answers. ( α 0 , α 1 subscript 𝛼 0 subscript 𝛼 1 \alpha_{0},\alpha_{1} italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) RWR RWW WRW WRR Δ 0 subscript Δ 0 \Delta_{0} roman_Δ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT Δ 1 subscript Δ 1 \Delta_{1} roman_Δ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT (0, 0) 0.0529 0.0563 0.1244 0.2286 0.1757 0.0661 (0, 2) 0.0270 0.0521 0.1259 0.2194 0.1924 0.0738 (2, 0) 0.0500 0.0563 0.1241 0.2272 0.1772 0.0678 Table 3: Analysis of answer revision patterns under different α 𝛼 \alpha italic_α parameters. The columns RWR through WRR show the proportion of each transition type, where the three letters indicate Answer(t), Answer(t+1), and Majority(t) respectively. R and W stand for right and wrong answer. Δ 0 subscript Δ 0 \Delta_{0} roman_Δ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT measures the difference in transitions from wrong to right answers when the majority is wrong (WRW − - - RWW) which is related to α 0 subscript 𝛼 0 \alpha_{0} italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , while Δ 1 subscript Δ 1 \Delta_{1} roman_Δ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT measures transitions when the majority is right (WRR − - - RWR) which is related to α 1 subscript 𝛼 1 \alpha_{1} italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT . Analysis of α 0 subscript 𝛼 0 \alpha_{0} italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and α 1 subscript 𝛼 1 \alpha_{1} italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT . We compare baseline ( α 0 , α 1 ) = ( 0 , 0 ) subscript 𝛼 0 subscript 𝛼 1 0 0 (\alpha_{0},\alpha_{1})=(0,0) ( italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) = ( 0 , 0 ) against two configurations: ( 0 , 2 ) 0 2 (0,2) ( 0 , 2 ) and ( 2 , 0 ) 2 0 (2,0) ( 2 , 0 ) . When α 1 subscript 𝛼 1 \alpha_{1} italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT was increased to 2, we observe a 9.5% improvement in Δ 1 subscript Δ 1 \Delta_{1} roman_Δ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , indicating that incentivizing agents to follow correct majority opinions effectively improved performance. When α 0 subscript 𝛼 0 \alpha_{0} italic_α start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT was increased to 2, we observed a smaller (2.57%) improvement in Δ 0 subscript Δ 0 \Delta_{0} roman_Δ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , suggesting that rewarding agents for deviating from incorrect majority opinions had a positive but limited effect. ( β 0 , β 1 subscript 𝛽 0 subscript 𝛽 1 \beta_{0},\beta_{1} italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) RWR RWW WRW WRR Δ 0 subscript Δ 0 \Delta_{0} roman_Δ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT Δ 1 subscript Δ 1 \Delta_{1} roman_Δ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT (0, 0) 0.0070 0.0453 0.0226 0.0221 0.0151 -0.0227 (0, 2) 0.0686 0.0461 0.0231 0.0230 0.0161 -0.0230 (2, 0) 0.0011 0.0360 0.0161 0.0188 0.0177 -0.0199 Table 4: Analysis of majority opinion influence under different β 𝛽 \beta italic_β parameters. Meaning of the column is the same as Table 3 . Analysis of β 0 subscript 𝛽 0 \beta_{0} italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and β 1 subscript 𝛽 1 \beta_{1} italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT . We compare baseline ( β 0 , β 1 ) = ( 0 , 0 ) subscript 𝛽 0 subscript 𝛽 1 0 0 (\beta_{0},\beta_{1})=(0,0) ( italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) = ( 0 , 0 ) against configurations ( 0 , 2 ) 0 2 (0,2) ( 0 , 2 ) and ( 2 , 0 ) 2 0 (2,0) ( 2 , 0 ) . Increasing β 1 subscript 𝛽 1 \beta_{1} italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT to 2 resulted in a slight decrease in Δ 1 subscript Δ 1 \Delta_{1} roman_Δ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT (-1.32%), indicating that incentivizing agents based on their influence when correct did not improve outcomes. However, increasing β 0 subscript 𝛽 0 \beta_{0} italic_β start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to 2 lead to a substantial improvement in Δ 0 subscript Δ 0 \Delta_{0} roman_Δ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT (17.2%), suggesting that rewarding agents for constructive influence even when wrong (providing useful incorrect answers that lead to better future responses) significantly enhanced collaborative performance. For a total debating turns of 3, we also plot the collaboration performance using models trained with α i = β i = 0 , 1 , 2 formulae-sequence subscript 𝛼 𝑖 subscript 𝛽 𝑖 0 1 2 \alpha_{i}=\beta_{i}=0,1,2 italic_α start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = italic_β start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = 0 , 1 , 2 for i = 1 , 2 𝑖 1 2 i=1,2 italic_i = 1 , 2 on the GSM8K and ANLI tasks ( Figure 2 ). The results showed some performance improvement, though the gain was relatively modest. 4.5 Experiment 3: Collaboration Ability Acquired by \ours Is Transferable Here, we investigate the transferability of collaboration abilities acquired through MAPoRL across different datasets not used during training. We evaluate LLMs trained with MAPoRL on one dataset when applied to tasks from other datasets. For instance, we assesse models trained on ANLI when solving tasks from GSM8k, along with other dataset combinations. The results, presented in Table 5 , demonstrate that collaboration abilities learned through \ours are indeed transferable across datasets. This suggests that the models acquire a meta -capability for effective collaboration, even when encountering novel, unseen tasks. Training → → \rightarrow → Evaluation Model Turn 1 Turn 2 Turn 3 ANLI → → \rightarrow → GSM8K Off-the-shelf 0.677 0.688 0.640 Trained 0.677 0.712 0.720 GSM8K → → \rightarrow → ANLI Off-the-shelf 0.482 0.486 0.468 Trained 0.482 0.499 0.507 Table 5: Performance comparison (Accuracy) of 3-agent collaboration using off-the-shelf vs. trained LLMs. For each dataset pair (rows in bold), the first row shows the off-the-shelf performance and the second row shows the trained model performance, across Turns 1–3. These findings demonstrate that models trained through \ours on one task can effectively generalize their collaborative capabilities to different, unrelated tasks. This generalization ability suggests that \ours develops fundamental collaborative skills that transcend specific task domains. 4.6 Experiment 4: MAPoRL with Heterogeneous LLMs Can Help In this experiment, we investigate collaborative learning between different foundation models, specifically examining co-training between (Phi3 3.4B and Qwen2.5 3B) and (Phi3 3.4B and Llama3-8B) pairs. In single-model evaluations, both Phi3 and Qwen2.5 3B demonstrate stronger performance compared to Llama3-8B. Due to GPU memory constraints necessitating simultaneous loading of two base models, we conduct experiments in a two-agent, two-turn environment. This setup enables us to explore whether models with heterogeneous capabilities could effectively collaborate to enhance the overall performance ( Figure 7 ). The synergistic effects are particularly evident when models with different strengths worked together, suggesting that diverse model partnerships can yield better outcomes than individual model performance alone when we have \ours . 4.7 Experiment 5: Naïve Supervised Fine-Tuning Using High-Quality Collaboration Samples May Not Induce Collaborative Behaviors In this experiment, we investigate whether models could learn collaborative behavior through SFT on high-quality debate trajectories. We generated 12,800 trajectories using the multi-agent system ( Figure 1 ) with off-the-shelf LLMs to match the training sample size used in MAPoRL for GSM8K. To provide favorable conditions for SFT, we allow a maximum of 600 tokens per response, which exceeded the token limit used in our MAPoRL experiments. We selected the top 10% of trajectories using the following criteria: 1) excluding trajectories without well-formatted answers, 2) filtering out trajectories where the final majority voting result was incorrect, and 3) selecting 1,280 trajectories based on the verifier’s score of the final answer, which evaluates both correctness and reasoning quality. Interestingly, the results indicate that SFT not only failed to enhance collaborative behaviors, but also led to a decline in performance compared to the off-the-shelf model. Specifically, for turn-2, accuracy dropped to 0.578 ( Δ = − 0.111 Δ 0.111 \Delta=-0.111 roman_Δ = - 0.111 ), and for turn-3, it further decreased to 0.525 ( Δ = − 0.114 Δ 0.114 \Delta=-0.114 roman_Δ = - 0.114 ) 3 3 3 Initially, these unexpected results led us to validate our findings through multiple experiments with varying temperatures for language generation. The consistent performance degradation across turns was observed in all the cases. This pattern suggests fundamental challenges in using SFT to maintain collaborative performance across multiple debate turns. . This suggests that either substantially more training data would be required to learn effective collaborative behaviors, or that SFT might not be an effective approach for inducing such behaviors. Contemporaneously, Subramaniam et al. ( 2025 ) and Zhao et al. ( 2025 ) enhance multi-agent performance by incorporating new techniques into iterative SFT with their own data augmentation to generate effective collaboration examples, demonstrating its potential when combined with additional refinements. In contrast, our approach does not leverage data augmentation, but uses RL. 5 Concluding Remarks, Limitations, and Potential Risks In this paper, we have introduced \ours , a new post-training paradigm that leverages multi-agent RL to explicitly foster the collaboration among multiple LLMs. Unlike methods that rely solely on prompting or single-agent fine-tuning, MAPoRL focuses on co-training multiple LLMs, ensuring that each agent adapts its policy not just to immediate feedback, but also to the strategic behaviors of other agents over multiple interactive turns. By incorporating a verifier network for reward shaping with incentives, the framework guides each agent’s responses that account for both short-term correctness and long-term collaborative potential, thus promoting collaborative discussions that lead to more accurate final answers. Through an extensive set of experiments on reasoning-intensive tasks – such as GSM8K for mathematical problem-solving and ANLI for logical natural language inference – our results demonstrate that off-the-shelf LLMs often do not improve the overall performance with additional debate turns. In contrast, \ours -trained agents show significant improvements with accuracy increasing as collaboration progresses. Crucially, these collaborative abilities are shown transferable across tasks, suggesting that once LLMs learn to collaborate, they can retain a generalizable “collaboration skill” applicable to different domains. Furthermore, our experiments with heterogeneous LLMs highlight that MAPoRL can also foster collaborative synergy even among models of varying capabilities. Limitations Since we use instruction prompts as inputs to the LLMs, the output can vary significantly depending on the prompts. As the first methodology paper, our experiments are conducted on relatively small LLMs (3B to 8B parameters) for fast iteration, and the observed behaviors may differ on larger models. After all turns of multi-LLM interactions, we apply majority voting to determine the final answer. Using alternative mechanisms, such as a manager agent that makes the final prediction based on the responses from multiple agents, may further improve the overall performance. Potential Risks As our proposed approach encourages and facilitates collaboration among multiple LLM agents, when adversarial or malicious agents exist, our method could lead to unintended harmful outcomes by enabling their collaboration with others. Acknowledgment We sincerely thank Jisu Jang for their valuable feedback on Figure 1 . K.Z. acknowledges the support from the U.S. Army Research Office grant W911NF-24-1-0085 and the NSF CAREER Award 2443704. References Abdin et al. (2024) Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 . Ahmadian et al. (2024) Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 12248–12267, Bangkok, Thailand. Association for Computational Linguistics. Akata et al. (2023) Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. 2023. Playing repeated games with large language models. arXiv preprint arXiv:2305.16867 . Amodei et al. (2016) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 . Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 . Brookins and DeBacker (2023) Philip Brookins and Jason Matthew DeBacker. 2023. Playing games with GPT: What can we learn about a large language model from canonical strategic games? Available at SSRN 4493398 . Chen et al. (2024) Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. 2024. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. In ACL . Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 . Cooper (1999) Russell Cooper. 1999. Coordination games . cambridge university Press. Dettmers et al. (2024) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems , 36. Du et al. (2024) Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In ICML . Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 . Fan et al. (2024) Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao He. 2024. Can large language models serve as rational players in game theory? a systematic analysis. In AAAI . Feng et al. (2025) Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang, Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen-Yu Lee, Tomas Pfister, Yejin Choi, and Yulia Tsvetkov. 2025. When one llm drools, multi-llm collaboration rules. arXiv preprint arXiv:2502.04506 . Fu et al. (2023) Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. 2023. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142 . Gagne (1974) Robert M Gagne. 1974. Instruction and the conditions of learning. Psychology of school learning: Views of the learner , 1:153–175. Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 . Hadfield-Menell et al. (2017) Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. 2017. Inverse reward design. Advances in neural information processing systems , 30. Harsanyi and Selten (1988) John C Harsanyi and Reinhard Selten. 1988. A general theory of equilibrium selection in games. MIT Press Books , 1. Hernandez-Leal et al. (2019) Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. 2019. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems , 33(6):750–797. Hertz-Lazarowitz et al. (2013) R Hertz-Lazarowitz, S Kagan, Shlomo Sharan, R Slavin, and Clark Webb. 2013. Learning to cooperate, cooperating to learn . Springer Science & Business Media. Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In ICLR . Huang et al. (2024) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. In ICLR . Hüttenrauch et al. (2017) Maximilian Hüttenrauch, Adrian Šošić, and Gerhard Neumann. 2017. Guided deep reinforcement learning for swarm systems. arXiv preprint arXiv:1709.06011 . Khan et al. (2024) Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. In Forty-first International Conference on Machine Learning . Kim et al. (2024a) Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2024a. Language models can solve computer tasks. Advances in Neural Information Processing Systems , 36. Kim et al. (2024b) Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Cynthia Breazeal, and Hae Won Park. 2024b. Mdagents: An adaptive collaboration of llms for medical decision-making. In NeurIPS . Kumar et al. (2025) Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. 2025. Training language models to self-correct via reinforcement learning. In ICLR . Le et al. (2022) Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems , 35:21314–21328. Li et al. (2023) Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large scale language model society. Neural Information Processing Systems . Li et al. (2024) Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. 2024. Improving multi-agent debate with sparse communication topology. In EMNLP Findings . Liang et al. (2024) Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2024. Encouraging divergent thinking in large language models through multi-agent debate. In EMNLP . Liao et al. (2025) Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. 2025. Marft: Multi-agent reinforcement fine-tuning. arXiv preprint arXiv:2504.16129 . Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050 . Liu et al. (2023) Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. 2023. Tinygsm: achieving> 80% on gsm8k with small language models. arXiv preprint arXiv:2312.09241 . Lorè and Heydari (2023) Nunzio Lorè and Babak Heydari. 2023. Strategic behavior of large language models: Game structure vs. contextual framing. arXiv preprint arXiv:2309.05898 . Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR . Macy (1991) Michael W Macy. 1991. Learning to cooperate: Stochastic and tacit collusion in social exchange. American Journal of Sociology , 97(3):808–843. Madaan et al. (2024) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems , 36. Motwani et al. (2024) Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip HS Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. 2024. Malt: Improving reasoning with multi-agent llm training. arXiv preprint arXiv:2412.01928 . Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding. In ACL . Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730–27744. Park et al. (2024) Chanwoo Park, Mingyang Liu, Kaiqing Zhang, and Asuman Ozdaglar. 2024. Principled rlhf from heterogeneous feedback via personalization and preference aggregation. arXiv preprint arXiv:2405.00254 . Park et al. (2025) Chanwoo Park, Xiangyu Liu, Asuman Ozdaglar, and Kaiqing Zhang. 2025. Do llm agents have regret? a case study in online learning and games. In ICLR . Park et al. (2023) Chanwoo Park, Kaiqing Zhang, and Asuman Ozdaglar. 2023. Multi-player zero-sum markov games with networked separable interactions. Advances in Neural Information Processing Systems , 36. Qu et al. (2024) Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. 2024. Recursive introspection: Teaching language model agents how to self-improve. In NeurIPS . Rashid et al. (2020) Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. JMLR . Schulman et al. (2016) John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2016. High-dimensional continuous control using generalized advantage estimation. In ICLR . Shalev-Shwartz et al. (2016) Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. 2016. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295 . Shani et al. (2024) Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, et al. 2024. Multi-turn reinforcement learning from preference human feedback. In NeurIPS . Shinn et al. (2024) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems , 36. Stengel-Eskin et al. (2025) Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. 2025. Teaching models to balance resisting and accepting persuasion. In NAACL . Subramaniam et al. (2025) Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. 2025. Multiagent finetuning of language models. In ICLR . Sunehag et al. (2018) Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2018. Value-decomposition networks for cooperative multi-agent learning. In AAMAS . Tian et al. (2024) Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2024. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Representations . Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275 . Wang et al. (2024) Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 9426–9439. Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In ICLR . Welleck et al. (2023) Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations . Wu et al. (2023) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155 . Xiong et al. (2025) Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, et al. 2025. Building math agents with multi-turn iterative preference learning. In ICLR . Yang et al. (2024) An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 . Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. International Conference on Learning Representations . Yu et al. (2022) Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems , 35:24611–24624. Yu et al. (2024) Fei Yu, Anningzhe Gao, and Benyou Wang. 2024. Ovm,outcome-supervised value models for planning in mathematical reasoning. In NAACL Findings . Zhang and Hofbauer (2016) Boyu Zhang and Josef Hofbauer. 2016. Quantal response methods for equilibrium selection in 2 × \times × 2 coordination games. Games and Economic Behavior , 97:19–31. Zhang et al. (2021) Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. 2021. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control , pages 321–384. Zhao et al. (2024) Jun Zhao, Can Zu, Hao Xu, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Longagent: Scaling language models to 128k context through multi-agent collaboration. In EMNLP . Zhao et al. (2025) Wanjia Zhao, Mert Yuksekgonul, Shirley Wu, and James Zou. 2025. Sirius: Self-improving multi-agent systems via bootstrapped reasoning. arXiv preprint arXiv:2502.04780 . Zhu et al. (2023) Banghua Zhu, Michael Jordan, and Jiantao Jiao. 2023. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning , pages 43037–43067. PMLR. Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 . Appendix A Detailed Related Work Discussion Multi-Agent Reinforcement Learning. Various algorithms have been proposed to address multi-agent reinforcement learning (MARL) Hernandez-Leal et al. ( 2019 ); Zhang et al. ( 2021 ) , including multi-agent Proximal Policy Optimization (PPO) (Yu et al., 2022 ) , and value function factorization techniques such as QMIX and VDN (Rashid et al., 2020 ; Sunehag et al., 2018 ) . In the context of language models and collaborative debating we focus on, MARL takes on a particular and unique form. Here, each agent’s state is represented by the sequence of previous responses from all the agents, with each agent deciding the next token based on this history. LLMs provide compact state representations through their hidden layers, enabling the use of long debate histories. Multi-Agent Collaboration with LLMs. An array of studies have explored effective collaboration frameworks among multiple large language model agents to solve complex tasks (Wu et al., 2023 ; Li et al., 2024 ; Zhao et al., 2024 ) . For example, “role-playing”-based approaches utilized multi-agent LLMs by assigning a specific role to each LLM (Li et al., 2023 ) , and “multi-agent debate”-based approaches prompted each LLM agent to solve the task independently and then discuss (Du et al., 2024 ; Khan et al., 2024 ) . In a debate, the agents reason through each other’s answers to converge on a consensus response, which may improve the factual accuracy, mathematical ability, and reasoning capabilities of the LLM (Du et al., 2024 ; Liang et al., 2024 ; Kim et al., 2024b ) . Similar mult-agentic frameworks include voting (Wang et al., 2023 ) , group discussions (Chen et al., 2024 ) , and negotiating (Fu et al., 2023 ) . However, all of these frameworks rely heavily on prompt engineering, which may lead to sub-optimal results (Huang et al., 2024 ) , and do not consider training LLMs specifically for collaboration. Therefore, while multi-LLM systems seem promising at the first glance, their performance may be limited when using the out-of-the-box (pretrained) LLM with only prompt tuning, which highlights the need for training for better multi-agent collaboration. Recently, Stengel-Eskin et al. ( 2025 ) introduced a training framework for accepting or rejecting persuasion in multi-agent systems. Additionally, very recently, Subramaniam et al. ( 2025 ) and Zhao et al. ( 2025 ) focused on training the entire multi-agent systems using iterative SFT. In contrast, MAPoRL employs (multi-agent) RL to train the whole multi-LLM system. Recently, after \ours was released, Liao et al. ( 2025 ) provided a similar training system of multi-agents with reinforcement learning. RL for LLM Training. RL has been widely used in post-training LLMs, e.g., for improving factuality Tian et al. ( 2024 ) , code generation Le et al. ( 2022 ) , and more recently and significantly, reasoning Guo et al. ( 2025 ) . One prevalent approach of RL for LLM training is RL from human feedback (RLHF) (Ziegler et al., 2019 ; Ouyang et al., 2022 ; Bai et al., 2022 ; Ahmadian et al., 2024 ) . RL offers a smooth generalization to the multi-turn setting based on the Markov decision process (MDP) model, and there have been attempts to apply multi-turn RL for LLM training, such as RLHF for multi-turn model training to enhance the dialogue abilities (Shani et al., 2024 ) , or deriving multi-turn RL objective for the improvement of mathematical reasoning Xiong et al. ( 2025 ) . However, the major difference from our work is that, these works did not consider multi-agent settings for collaboration. Recently, Kumar et al. ( 2025 ) enhanced LLMs’ ability to self-correct using an RL-based approach. Our framework can accommodate this case by using a single agent in MAPoRL . Appendix B Additional Literature Review Multi-Agent RL. Multi-agent reinforcement learning (MARL) has achieved significant advancements, particularly in cooperative games and their real-world applications, such as coordinating robot swarms (Hüttenrauch et al., 2017 ) and self-driving vehicles (Shalev-Shwartz et al., 2016 ) . (A comprehensive overview of MARL can be found in Zhang et al. ( 2021 ) ). The primary challenge in MARL lies in the exponentially large action space, making it difficult to optimize the policy for each agent. Various approaches have been proposed to address this issue, including multi-agent Proximal Policy Optimization (PPO) (Yu et al., 2022 ) , value function factorization methods (QMIX, VDN) (Rashid et al., 2020 ; Sunehag et al., 2018 ) , and network-based formulations for multi-agent learning (Park et al., 2023 ) . These methods aim to make MARL more scalable with a large number of agents, mostly focusing on the classical models of stochastic/Markov games. In the context of language models and collaborative debate systems, MARL takes on a unique form. Here, each agent’s state is represented by the sequence of previous responses from all agents, with each agent deciding the next token based on this history. The detailed mathematical formulation for reinforcement learning in language models can be found in several theoretical and empirical studies on reinforcement learning with human feedback (RLHF) ( e.g., Ouyang et al. ( 2022 ); Zhu et al. ( 2023 ); Park et al. ( 2024 ) ). LLMs provide high-quality state representations through their hidden layers, enabling the consideration of long debate histories. Moreover, the sequential nature of these interactions inherently captures non-Markovian policies due to the extended sequence of responses. Teaching LLM Self-Correction. As mentioned in the main paper, single-agent self-correction and multi-agent collaboration has a very interesting relationship. Single-agent self-correction and multi-agent collaboration rely on multi-turn interactions—either internally, within a single agent, or collaboratively, among multiple agents—to improve results by challenging initial outputs and refining them through iteration. In single-agent systems, self-correction functions like an internal debate. The agent evaluates its own output over multiple turns, identifying potential mistakes and proposing alternative solutions. This process mirrors human reflection, where reconsideration often leads to improved conclusions. Meanwhile, in multi-agent systems, different agents engage in a collaborative debate, questioning and refining each other’s answers. By interacting in multiple rounds, these agents combine their individual perspectives to correct errors and arrive at more accurate solutions. There are several prior works aiming to improve LLMs’ ability to self-correct. First line of work is using prompting technique, which guides LMs via prompting to iteratively correct the model outputs (Madaan et al., 2024 ) . However, some works use the ground-truth labels to determine when to stop the self-correction (Kim et al., 2024a ; Shinn et al., 2024 ; Yao et al., 2023 ) , which is not applicable in the real-world scenarios where answer is not available for the tasks, and it is shown that under such scenarios the models can not do self-correct effectively (Huang et al., 2024 ) . Another line of works train LLMs to learn self-correction; Qu et al. ( 2024 ) introduced an approach using stronger LLMs to obtain multi-turn trajectories that have better responses through the iteration, and uses this data to fine-tune LLMs to learn self-correction. Different from this work, our approach do not require stronger LLMs for demonstrations, relying solely on the reward for training. Welleck et al. ( 2023 ) proposed supervised fine-tuning to train a corrector model that can edit the model response iteratively, but this is specified the type of collaboration in the generate-then-refine pattern, which can be sub-optimal to learned by the models. Kumar et al. ( 2025 ) employed an RL-based approach for the self-improvement of language models. Multi-Agent LLMs with Game Theory. Recent work has actively explored the strategic interactions of LLM agents within game-theoretic frameworks, as demonstrated in studies such as Park et al. ( 2025 ); Brookins and DeBacker ( 2023 ); Akata et al. ( 2023 ); Lorè and Heydari ( 2023 ); Fan et al. ( 2024 ) . Our paper can be viewed as training LLMs as solvers of cooperative games such as solving mathematical problems together. Appendix C Deferred Content of Section 2 Remark 3 (Rationale Behind the Setup) . This formalization captures several key aspects of complex problem-solving dynamics. Choosing to collaborate ( a 0 subscript 𝑎 0 a_{0} italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) represents contributing exploratory ideas or partial solutions . While these contributions have a lower immediate probability of correctness R col ⁢ ( q ) subscript 𝑅 col 𝑞 R_{\text{col}}(q) italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) , they are essential building blocks towards the complete solution. Acting independently ( a 1 ) subscript 𝑎 1 (a_{1}) ( italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) represents using conventional approaches that may yield a higher immediate probability of correctness R ind ⁢ ( q ) subscript 𝑅 ind 𝑞 R_{\text{ind}}(q) italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) , but may contribute less to solving particularly challenging problems. The collaboration threshold C ⁢ ( q ) 𝐶 𝑞 C(q) italic_C ( italic_q ) represents the minimum amount of collaboration efforts and idea generation needed to solve complex problems. Once this threshold is reached (i.e., achieving collaborative synergy), the agents can combine their insights to solve the challenging problem, with a higher reward R syn ⁢ ( q ) subscript 𝑅 syn 𝑞 R_{\text{syn}}(q) italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) . See 1 Proof. For the last turn ( t = 2 𝑡 2 {t}=2 italic_t = 2 ), regardless of whether the opponent selects a 0 subscript 𝑎 0 a_{0} italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT or not, choosing a 1 subscript 𝑎 1 a_{1} italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT is an optimal strategy. This is because: • If collaborative synergy has been achieved, the agent will always receive R syn ⁢ ( q ) subscript 𝑅 syn 𝑞 R_{\text{syn}}(q) italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) regardless of their action in the second turn. • If collaborative synergy has not been achieved, since we know that R col ⁢ ( q ) < R ind ⁢ ( q ) subscript 𝑅 col 𝑞 subscript 𝑅 ind 𝑞 R_{\text{col}}(q) 0 𝜏 0 \tau>0 italic_τ > 0 is a temperature parameter and H ⁢ ( π i ) = − ∑ a i π i ⁢ ( a i ) ⁢ log ⁡ π i ⁢ ( a i ) 𝐻 subscript 𝜋 𝑖 subscript subscript 𝑎 𝑖 subscript 𝜋 𝑖 subscript 𝑎 𝑖 subscript 𝜋 𝑖 subscript 𝑎 𝑖 H(\pi_{i})=-\sum_{a_{i}}\pi_{i}(a_{i})\log\pi_{i}(a_{i}) italic_H ( italic_π start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) = - ∑ start_POSTSUBSCRIPT italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT italic_π start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_log italic_π start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) is the Shannon entropy of the strategy, and u i subscript 𝑢 𝑖 u_{i} italic_u start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT is the utility function of player i 𝑖 i italic_i . This entropy term smoothens the best response, leading to a softmax (or logit) formula of the optimal strategy: π i ⋆ ⁢ ( a i ) = exp ⁡ ( 1 τ ⁢ 𝔼 a − i ∼ π − i ⋆ ⁢ [ u i ⁢ ( a i , a − i ) ] ) ∑ a i ′ exp ⁡ ( 1 τ ⁢ 𝔼 a − i ∼ π − i ⋆ ⁢ [ u i ⁢ ( a i ′ , a − i ) ] ) . superscript subscript 𝜋 𝑖 ⋆ subscript 𝑎 𝑖 1 𝜏 subscript 𝔼 similar-to subscript 𝑎 𝑖 superscript subscript 𝜋 𝑖 ⋆ delimited-[] subscript 𝑢 𝑖 subscript 𝑎 𝑖 subscript 𝑎 𝑖 subscript superscript subscript 𝑎 𝑖 ′ 1 𝜏 subscript 𝔼 similar-to subscript 𝑎 𝑖 superscript subscript 𝜋 𝑖 ⋆ delimited-[] subscript 𝑢 𝑖 superscript subscript 𝑎 𝑖 ′ subscript 𝑎 𝑖 \pi_{i}^{\star}(a_{i})=\frac{\exp\Bigl{(}\frac{1}{\tau}\,\mathbb{E}_{a_{-i}% \sim\pi_{-i}^{\star}}\bigl{[}u_{i}(a_{i},a_{-i})\bigr{]}\Bigr{)}}{\sum_{a_{i}^% {\prime}}\exp\Bigl{(}\frac{1}{\tau}\,\mathbb{E}_{a_{-i}\sim\pi_{-i}^{\star}}% \bigl{[}u_{i}(a_{i}^{\prime},a_{-i})\bigr{]}\Bigr{)}}. italic_π start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) = divide start_ARG roman_exp ( divide start_ARG 1 end_ARG start_ARG italic_τ end_ARG blackboard_E start_POSTSUBSCRIPT italic_a start_POSTSUBSCRIPT - italic_i end_POSTSUBSCRIPT ∼ italic_π start_POSTSUBSCRIPT - italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_POSTSUBSCRIPT [ italic_u start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT - italic_i end_POSTSUBSCRIPT ) ] ) end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUBSCRIPT roman_exp ( divide start_ARG 1 end_ARG start_ARG italic_τ end_ARG blackboard_E start_POSTSUBSCRIPT italic_a start_POSTSUBSCRIPT - italic_i end_POSTSUBSCRIPT ∼ italic_π start_POSTSUBSCRIPT - italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_POSTSUBSCRIPT [ italic_u start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT , italic_a start_POSTSUBSCRIPT - italic_i end_POSTSUBSCRIPT ) ] ) end_ARG . Observation 2. Consider a game where each agent maximizes their expected cumulative utility plus an entropy regularizer with a small regularization coefficient τ > 0 𝜏 0 \tau>0 italic_τ > 0 . Let NE ⁢ ( τ ) NE 𝜏 \text{NE}(\tau) NE ( italic_τ ) denote the unique Nash equilibrium of the regularized game for a fixed τ > 0 𝜏 0 \tau>0 italic_τ > 0 . As τ → 0 → 𝜏 0 \tau\to 0 italic_τ → 0 , the sequence of equilibria NE ⁢ ( τ ) NE 𝜏 \text{NE}(\tau) NE ( italic_τ ) converges to Collaborate ( a 0 ) subscript 𝑎 0 (a_{0}) ( italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) if R syn ⁢ ( q ) = 1 > max ⁡ ( 3 ⁢ R col ⁢ ( q ) − 2 ⁢ R ind ⁢ ( q ) , 2 ⁢ R ind ⁢ ( q ) − R col ⁢ ( q ) ) . subscript 𝑅 syn 𝑞 1 3 subscript 𝑅 col 𝑞 2 subscript 𝑅 ind 𝑞 2 subscript 𝑅 ind 𝑞 subscript 𝑅 col 𝑞 R_{\text{syn}}(q)=1>\max(3R_{\text{col}}(q)-2R_{\text{ind}}(q),2R_{\text{ind}}% (q)-R_{\text{col}}(q)). italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) = 1 > roman_max ( 3 italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) - 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) , 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) - italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) ) . Proof. Following the reasoning in showing 1 , we analyze the cumulative reward for the turn t = 1 𝑡 1 t=1 italic_t = 1 . The reward matrix is given by: a 0 ⁢ (Collaborate) a 1 ⁢ (Act independently) a 0 ⁢ (Collaborate) ( R col ⁢ ( q ) + R syn ⁢ ( q ) , R col ⁢ ( q ) + R syn ⁢ ( q ) ) ( R col ⁢ ( q ) + R ind ⁢ ( q ) , 2 ⁢ R ind ⁢ ( q ) ) a 1 ⁢ (Act independently) ( 2 ⁢ R ind ⁢ ( q ) , R col ⁢ ( q ) + R ind ⁢ ( q ) ) ( 2 ⁢ R ind ⁢ ( q ) , 2 ⁢ R ind ⁢ ( q ) ) . missing-subexpression subscript 𝑎 0 (Collaborate) subscript 𝑎 1 (Act independently) missing-subexpression missing-subexpression missing-subexpression subscript 𝑎 0 (Collaborate) subscript 𝑅 col 𝑞 subscript 𝑅 syn 𝑞 subscript 𝑅 col 𝑞 subscript 𝑅 syn 𝑞 subscript 𝑅 col 𝑞 subscript 𝑅 ind 𝑞 2 subscript 𝑅 ind 𝑞 subscript 𝑎 1 (Act independently) 2 subscript 𝑅 ind 𝑞 subscript 𝑅 col 𝑞 subscript 𝑅 ind 𝑞 2 subscript 𝑅 ind 𝑞 2 subscript 𝑅 ind 𝑞 \begin{array}[]{c|cc}&a_{0}\text{ (Collaborate)}&a_{1}\text{ (Act % independently)}\\ \hline\cr a_{0}\text{ (Collaborate)}&(R_{\text{col}}(q)+R_{\text{syn}}(q),R_{% \text{col}}(q)+R_{\text{syn}}(q))&(R_{\text{col}}(q)+R_{\text{ind}}(q),2R_{% \text{ind}}(q))\\ a_{1}\text{ (Act independently)}&(2R_{\text{ind}}(q),R_{\text{col}}(q)+R_{% \text{ind}}(q))&(2R_{\text{ind}}(q),2R_{\text{ind}}(q))\\ \end{array}. start_ARRAY start_ROW start_CELL end_CELL start_CELL italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT (Collaborate) end_CELL start_CELL italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT (Act independently) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT (Collaborate) end_CELL start_CELL ( italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) + italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) , italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) + italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) ) end_CELL start_CELL ( italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) + italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) , 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) ) end_CELL end_ROW start_ROW start_CELL italic_a start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT (Act independently) end_CELL start_CELL ( 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) , italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) + italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) ) end_CELL start_CELL ( 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) , 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) ) end_CELL end_ROW end_ARRAY . If R syn ⁢ ( q ) = 1 > 2 ⁢ R ind ⁢ ( q ) − R col ⁢ ( q ) , subscript 𝑅 syn 𝑞 1 2 subscript 𝑅 ind 𝑞 subscript 𝑅 col 𝑞 R_{\text{syn}}(q)=1>2R_{\text{ind}}(q)-R_{\text{col}}(q), italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) = 1 > 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) - italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) , then this game is a coordination game, and according to Zhang and Hofbauer ( 2016 , Theorem 1) , as τ → 0 → 𝜏 0 \tau\to 0 italic_τ → 0 , the regularized NE converges to the risk-dominant strategy (Harsanyi and Selten, 1988 ) in a 2 × 2 2 2 2\times 2 2 × 2 game. In this setting, by definition, the collaboration strategy ( a 0 , a 0 ) subscript 𝑎 0 subscript 𝑎 0 (a_{0},a_{0}) ( italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) is risk-dominant (Harsanyi and Selten, 1988 ) if: ( R col ⁢ ( q ) + R syn ⁢ ( q ) ) + ( R col ⁢ ( q ) + R ind ⁢ ( q ) ) > ( 2 ⁢ R ind ⁢ ( q ) + 2 ⁢ R ind ⁢ ( q ) ) , subscript 𝑅 col 𝑞 subscript 𝑅 syn 𝑞 subscript 𝑅 col 𝑞 subscript 𝑅 ind 𝑞 2 subscript 𝑅 ind 𝑞 2 subscript 𝑅 ind 𝑞 (R_{\text{col}}(q)+R_{\text{syn}}(q))+(R_{\text{col}}(q)+R_{\text{ind}}(q))>(2% R_{\text{ind}}(q)+2R_{\text{ind}}(q)), ( italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) + italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) ) + ( italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) + italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) ) > ( 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) + 2 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) ) , which is equivalent to R syn ⁢ ( q ) > 3 ⁢ R ind ⁢ ( q ) − 2 ⁢ R col ⁢ ( q ) . subscript 𝑅 syn 𝑞 3 subscript 𝑅 ind 𝑞 2 subscript 𝑅 col 𝑞 R_{\text{syn}}(q)>3R_{\text{ind}}(q)-2R_{\text{col}}(q). italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) > 3 italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) - 2 italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) . Combining the two conditions completes the proof. ∎ Appendix D Deferred Details in Section 2.3 The game is solved using backward induction with the state represented as ( turn , count ) turn count (\text{turn},\texttt{count}) ( turn , count ) , where count denotes the number of times ( a 0 , a 0 ) subscript 𝑎 0 subscript 𝑎 0 (a_{0},a_{0}) ( italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) has occurred in the history of the interactions. Both players choose actions to maximize their expected cumulative utility plus an entropy term times a coefficient τ = 0.1 𝜏 0.1 \tau=0.1 italic_τ = 0.1 . Choices of R col ⁢ ( q ) , R ind ⁢ ( q ) , R syn ⁢ ( q ) , C ⁢ ( q ) subscript 𝑅 col 𝑞 subscript 𝑅 ind 𝑞 subscript 𝑅 syn 𝑞 𝐶 𝑞 R_{\text{col}}(q),R_{\text{ind}}(q),R_{\text{syn}}(q),C(q) italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) , italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) , italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) , italic_C ( italic_q ) . Each instance of a question q 𝑞 q italic_q is associated with parameters drawn as follows: the independent action reward R ind ⁢ ( q ) subscript 𝑅 ind 𝑞 R_{\text{ind}}(q) italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) is sampled from a uniform distribution R ind ⁢ ( q ) ∼ Unif ⁢ ( 0 , 1 ) similar-to subscript 𝑅 ind 𝑞 Unif 0 1 R_{\text{ind}}(q)\sim\text{Unif}(0,1) italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) ∼ Unif ( 0 , 1 ) . The collaborative action reward R col ⁢ ( q ) subscript 𝑅 col 𝑞 R_{\text{col}}(q) italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) is then sampled condition on R ind ⁢ ( q ) subscript 𝑅 ind 𝑞 R_{\text{ind}}(q) italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) , following R col ⁢ ( q ) ∼ Unif ⁢ ( 0 , R ind ⁢ ( q ) ) similar-to subscript 𝑅 col 𝑞 Unif 0 subscript 𝑅 ind 𝑞 R_{\text{col}}(q)\sim\text{Unif}\Big{(}0,R_{\text{ind}}(q)\Big{)} italic_R start_POSTSUBSCRIPT col end_POSTSUBSCRIPT ( italic_q ) ∼ Unif ( 0 , italic_R start_POSTSUBSCRIPT ind end_POSTSUBSCRIPT ( italic_q ) ) . The synergy reward is fixed as R syn ⁢ ( q ) = 1 subscript 𝑅 syn 𝑞 1 R_{\text{syn}}(q)=1 italic_R start_POSTSUBSCRIPT syn end_POSTSUBSCRIPT ( italic_q ) = 1 . Figure 5: Collaboration probability (turn 1) as a function of the threshold C 𝐶 C italic_C , for two different horizons T = 10 𝑇 10 T=10 italic_T = 10 (left) and T = 20 𝑇 20 T=20 italic_T = 20 (right). We set the synergy reward to R syn = 1 subscript 𝑅 syn 1 R_{\mathrm{syn}}=1 italic_R start_POSTSUBSCRIPT roman_syn end_POSTSUBSCRIPT = 1 and vary C 𝐶 C italic_C from T − 1 𝑇 1 T{-}1 italic_T - 1 down to ⌊ ( T − 1 ) / 2 ⌋ 𝑇 1 2 \lfloor(T{-}1)/2\rfloor ⌊ ( italic_T - 1 ) / 2 ⌋ . The red curve (“Multi-Agent”) represents the collaboration probability when both players adaptively learn in a multi-agent setting. The blue curves show the best-response probabilities of Player 1 when facing a fixed opponent with collaboration probabilities π fixed ⁢ ( q ) ∈ { 0.5 , 0.6 , 0.7 } subscript 𝜋 fixed 𝑞 0.5 0.6 0.7 \pi_{\mathrm{fixed}}(q)\in\{0.5,0.6,0.7\} italic_π start_POSTSUBSCRIPT roman_fixed end_POSTSUBSCRIPT ( italic_q ) ∈ { 0.5 , 0.6 , 0.7 } . Each data point represents an average over 5000 random samples of ( R ind , R col ) subscript 𝑅 ind subscript 𝑅 col (R_{\mathrm{ind}},R_{\mathrm{col}}) ( italic_R start_POSTSUBSCRIPT roman_ind end_POSTSUBSCRIPT , italic_R start_POSTSUBSCRIPT roman_col end_POSTSUBSCRIPT ) . Appendix E Deferred Details of the Verifier Models For a reasoning question q 𝑞 q italic_q , the trained verifiers (reward models) assess the correctness of a complete solution path s 𝑠 s italic_s , denoted as p ⁢ ( s ⁢ is correct ∣ q ) 𝑝 conditional 𝑠 is correct 𝑞 p(s\text{ is correct}\mid q) italic_p ( italic_s is correct ∣ italic_q ) (Cobbe et al., 2021 ; Uesato et al., 2022 ; Lightman et al., 2023 ) . These reward models can either focus on the final outcome (outcome reward models) or provide step-by-step evaluations (process reward models). Although the latter generally yields better performance (Lightman et al., 2023 ) , the limited availability of process-level annotated datasets—especially for challenging benchmarks like ANLI (Nie et al., 2020 ) —restricts its applicability. Additionally, while generating detailed trajectories for process supervision (as seen in Wang et al. ( 2024 ) ) can be effective, our primary goal is not to enhance the language model’s domain specificity. Consequently, we chose to adopt a simpler strategy by training a verifier based on a well-tuned output reward model. Verifier Models Structure. We used a quantized version of a language model as the backbone for the verifier. Additionally, we incorporated a linear head layer followed by a softmax layer to ensure that the verifier’s output falls within the range of 0 to 1. The default backbone model is Microsoft Phi-3-mini-128k-instruct (Abdin et al., 2024 ) . In experiments involving different model training setups (see Section 4.6 ), we employed a new verifier with a different base model, specifically the one used in Section 4.6 . In these cases, we utilized Qwen2.5-3B-instruct (Yang et al., 2024 ) and Llama-3-8B-instruct (Dubey et al., 2024 ) as alternative backbone models. E.1 Training Procedure To train the verifier model, we generate tuples ( q i , s i ⁢ j , a i ⁢ j ) subscript 𝑞 𝑖 subscript 𝑠 𝑖 𝑗 subscript 𝑎 𝑖 𝑗 (q_{i},s_{ij},a_{ij}) ( italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_s start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT ) for i ∈ [ Q ] 𝑖 delimited-[] 𝑄 i\in[Q] italic_i ∈ [ italic_Q ] and j ∈ [ S ] 𝑗 delimited-[] 𝑆 j\in[S] italic_j ∈ [ italic_S ] , where q i subscript 𝑞 𝑖 q_{i} italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT is the question, s i ⁢ j subscript 𝑠 𝑖 𝑗 s_{ij} italic_s start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT is one of the S 𝑆 S italic_S generated solutions for question q i subscript 𝑞 𝑖 q_{i} italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT generated by the base model of verifier model, and a i ⁢ j subscript 𝑎 𝑖 𝑗 a_{ij} italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT is the corresponding answer for ( q i , s i ⁢ j ) subscript 𝑞 𝑖 subscript 𝑠 𝑖 𝑗 (q_{i},s_{ij}) ( italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_s start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT ) . We label the token-level subsequences ( q i , s i ⁢ j 1 : x ) subscript 𝑞 𝑖 superscript subscript 𝑠 𝑖 𝑗 : 1 𝑥 (q_{i},s_{ij}^{1:x}) ( italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_s start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) for x ≤ sequence length of ⁢ s i ⁢ j 𝑥 sequence length of subscript 𝑠 𝑖 𝑗 x\leq\texttt{sequence length of}s_{ij} italic_x ≤ sequence length of italic_s start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT as y i ⁢ j = 1 subscript 𝑦 𝑖 𝑗 1 y_{ij}=1 italic_y start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = 1 if a i ⁢ j subscript 𝑎 𝑖 𝑗 a_{ij} italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT is correct, and y i ⁢ j = 0 subscript 𝑦 𝑖 𝑗 0 y_{ij}=0 italic_y start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT = 0 if a i ⁢ j subscript 𝑎 𝑖 𝑗 a_{ij} italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT is incorrect. For the mathematical reasoning task, we utilized the GSM8K dataset (Cobbe et al., 2021 ) , specifically the training set consisting of 7,463 questions, to generate 100 reasoning paths for each questions. For the natural language inference task, we employed the ANLI dataset (Nie et al., 2020 ) , using first 10,000 questions to generate 50 reasoning paths. The trajectories were evaluated based on their outcomes, and we excluded outputs that did not adhere to the required formatting. Specifically, we ensured that the language model first provided reasoning before presenting the final answer in the format \\boxed{}. In our approach, we ensured that each question in the GSM8k dataset had a balanced set of reasoning paths. Specifically, if a question’s 100 reasoning paths contained at least 20 correct and 20 incorrect responses, we randomly selected 20 of each. However, when there were insufficient correct or incorrect paths, we augmented the data by generating additional paths using reference examples. For instance, if no correct reasoning path was available, we provided a correct example from the GSM8k dataset, and if incorrect paths were missing, we guided the language model to produce a response containing a trivial error. Ultimately, each GSM8k question was assigned 20 correct and 20 incorrect reasoning paths. For the ANLI dataset, we applied a similar procedure by starting with 50 reasoning paths per question, from which we randomly sampled 10 correct and 10 incorrect paths, supplementing the data as needed. Throughout this process, we minimized reliance on the original reasoning paths in the dataset since a) to enhance the overall diversity and quality of the generated data and b) to minimize the dependency on the reasoning path in the dataset. Next, we applied binary cross-entropy loss at the token level, aiming to minimize min θ ⁢ ∑ i , j , x ( y i ⁢ j ⁢ log ⁡ Verifier θ ⁢ ( q i , s i ⁢ j 1 : x ) + ( 1 − y i ⁢ j ) ⁢ log ⁡ ( 1 − Verifier θ ⁢ ( q i , s i ⁢ j 1 : x ) ) ) subscript 𝜃 subscript 𝑖 𝑗 𝑥 subscript 𝑦 𝑖 𝑗 subscript Verifier 𝜃 subscript 𝑞 𝑖 superscript subscript 𝑠 𝑖 𝑗 : 1 𝑥 1 subscript 𝑦 𝑖 𝑗 1 subscript Verifier 𝜃 subscript 𝑞 𝑖 superscript subscript 𝑠 𝑖 𝑗 : 1 𝑥 \min_{\theta}\sum_{i,j,x}\left(y_{ij}\log\texttt{Verifier}_{\theta}(q_{i},s_{% ij}^{1:x})+(1-y_{ij})\log(1-\texttt{Verifier}_{\theta}(q_{i},s_{ij}^{1:x}))\right) roman_min start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_i , italic_j , italic_x end_POSTSUBSCRIPT ( italic_y start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT roman_log Verifier start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_s start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) + ( 1 - italic_y start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT ) roman_log ( 1 - Verifier start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_q start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_s start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) ) ) where i 𝑖 i italic_i denotes the question index, j 𝑗 j italic_j represents the generated solution index, and t 𝑡 t italic_t is the token index. By default, we utilized all solution tokens for optimization; however, in practice, focusing on the latter half of the generated solution tokens yielded better results. For model training, we used QLoRA (Dettmers et al., 2024 ) with hyperparameters r = 16 𝑟 16 r=16 italic_r = 16 and α = 32 𝛼 32 \alpha=32 italic_α = 32 . We used a training batch size of 2 and optimized the model using the AdamW (Loshchilov and Hutter, 2019 ) optimizer with β 1 = 0.9 subscript 𝛽 1 0.9 \beta_{1}=0.9 italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = 0.9 , β 2 = 0.95 subscript 𝛽 2 0.95 \beta_{2}=0.95 italic_β start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = 0.95 , and a learning rate of 2 × 10 − 4 2 superscript 10 4 2\times 10^{-4} 2 × 10 start_POSTSUPERSCRIPT - 4 end_POSTSUPERSCRIPT . E.2 Verifier Performance We report the performance of the verifier in Table 6 . GSM8k ANLI Accuracy 0.91 0.92 Table 6: Performance of the verifier on different benchmarks. Accuracy is reported for GSM8K and ANLI. Notably, the verifier demonstrates higher accuracy in evaluating the correctness of answers compared to the accuracy of the LLM in generating correct answers. The verifier is classified as correct if the assigned reward is greater than 0.5 when the LLM-generated solution is correct, or if the reward is less than 0.5 when the LLM-generated solution is incorrect. E.3 Other Observations We experimented with various verifiers built upon different language model bases. Our first observation was that training the model using only the final answer did not perform as well as minimizing the cross-entropy loss over the last half of the generated tokens. Second, the verifier produced interpretable results, aligning with findings from Liu et al. ( 2023 ) . Lastly, when we used training samples from one base model but trained the verifier with a different base model as the backbone, the loss did not decrease, indicating that using the same base model for training is crucial for effective learning. E.4 Proof of Theorem 1 Theorem 1 . Assuming the verifier model is sufficiently expressive, the optimal parameter θ ⋆ superscript 𝜃 ⋆ \theta^{\star} italic_θ start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT that minimizes the expected cross-entropy loss between the true label and the verifier’s output will satisfy Verifier θ ⋆ ⁢ ( q , s 1 : x ) = ℙ ⁢ ( Final answer is correct ∣ q , s 1 : x ) . subscript Verifier superscript 𝜃 ⋆ 𝑞 superscript 𝑠 : 1 𝑥 ℙ conditional Final answer is correct 𝑞 superscript 𝑠 : 1 𝑥 \texttt{Verifier}_{\theta^{\star}}(q,s^{1:x})=\mathbb{P}(\text{Final answer is% correct}\mid q,s^{1:x}). Verifier start_POSTSUBSCRIPT italic_θ start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) = blackboard_P ( Final answer is correct ∣ italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) . Proof. The expected loss can be written as ℒ ⁢ ( θ ) = 𝔼 q , s , x , y ⁢ [ y ⁢ log ⁡ Verifier θ ⁢ ( q , s 1 : x ) + ( 1 − y ) ⁢ log ⁡ ( 1 − Verifier θ ⁢ ( q , s 1 : x ) ) ] . ℒ 𝜃 subscript 𝔼 𝑞 𝑠 𝑥 𝑦 delimited-[] 𝑦 subscript Verifier 𝜃 𝑞 superscript 𝑠 : 1 𝑥 1 𝑦 1 subscript Verifier 𝜃 𝑞 superscript 𝑠 : 1 𝑥 \mathcal{L}(\theta)=\mathbb{E}_{q,s,x,y}\Big{[}y\log\texttt{Verifier}_{\theta}% (q,s^{1:x})+(1-y)\log(1-\texttt{Verifier}_{\theta}(q,s^{1:x}))\Big{]}. caligraphic_L ( italic_θ ) = blackboard_E start_POSTSUBSCRIPT italic_q , italic_s , italic_x , italic_y end_POSTSUBSCRIPT [ italic_y roman_log Verifier start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) + ( 1 - italic_y ) roman_log ( 1 - Verifier start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) ) ] . Defining p θ ⁢ ( q , s 1 : x ) := Verifier θ ⁢ ( q , s 1 : x ) assign subscript 𝑝 𝜃 𝑞 superscript 𝑠 : 1 𝑥 subscript Verifier 𝜃 𝑞 superscript 𝑠 : 1 𝑥 p_{\theta}(q,s^{1:x}):=\texttt{Verifier}_{\theta}(q,s^{1:x}) italic_p start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) := Verifier start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) , we compute the partial derivative with respect to p θ ⁢ ( q ′ , s 1 : x ′ ) subscript 𝑝 𝜃 superscript 𝑞 ′ superscript 𝑠 : 1 superscript 𝑥 ′ p_{\theta}(q^{\prime},s^{1:x^{\prime}}) italic_p start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_q start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT , italic_s start_POSTSUPERSCRIPT 1 : italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ) : 𝔼 q , s , x , y ⁢ [ 𝟏 ⁢ ( q = q ′ , s 1 : x = s 1 : x ′ ) ⁢ ( y p θ ⁢ ( q , s 1 : x ) − 1 − y 1 − p θ ⁢ ( q , s 1 : x ) ) ] , subscript 𝔼 𝑞 𝑠 𝑥 𝑦 delimited-[] 1 formulae-sequence 𝑞 superscript 𝑞 ′ superscript 𝑠 : 1 𝑥 superscript 𝑠 : 1 superscript 𝑥 ′ 𝑦 subscript 𝑝 𝜃 𝑞 superscript 𝑠 : 1 𝑥 1 𝑦 1 subscript 𝑝 𝜃 𝑞 superscript 𝑠 : 1 𝑥 \mathbb{E}_{q,s,x,y}\Bigl{[}\mathbf{1}\bigl{(}q=q^{\prime},s^{1:x}=s^{1:x^{% \prime}}\bigr{)}\Bigl{(}\frac{y}{p_{\theta}(q,s^{1:x})}-\frac{1-y}{1-p_{\theta% }(q,s^{1:x})}\Bigr{)}\Bigr{]}, blackboard_E start_POSTSUBSCRIPT italic_q , italic_s , italic_x , italic_y end_POSTSUBSCRIPT [ bold_1 ( italic_q = italic_q start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT = italic_s start_POSTSUPERSCRIPT 1 : italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ) ( divide start_ARG italic_y end_ARG start_ARG italic_p start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) end_ARG - divide start_ARG 1 - italic_y end_ARG start_ARG 1 - italic_p start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) end_ARG ) ] , so we conclude Verifier θ ⋆ ⁢ ( q , s 1 : x ) = 𝔼 ⁢ [ y ∣ q , s 1 : x ] = ℙ ⁢ ( Final answer is correct ∣ q , s 1 : x ) subscript Verifier superscript 𝜃 ⋆ 𝑞 superscript 𝑠 : 1 𝑥 𝔼 delimited-[] conditional 𝑦 𝑞 superscript 𝑠 : 1 𝑥 ℙ conditional Final answer is correct 𝑞 superscript 𝑠 : 1 𝑥 \texttt{Verifier}_{\theta^{\star}}(q,s^{1:x})=\mathbb{E}[y\mid q,s^{1:x}]=% \mathbb{P}(\text{Final answer is correct }\mid q,s^{1:x}) Verifier start_POSTSUBSCRIPT italic_θ start_POSTSUPERSCRIPT ⋆ end_POSTSUPERSCRIPT end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) = blackboard_E [ italic_y ∣ italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ] = blackboard_P ( Final answer is correct ∣ italic_q , italic_s start_POSTSUPERSCRIPT 1 : italic_x end_POSTSUPERSCRIPT ) since we assumed that the verifier model is sufficiently expressive. ∎ It is worth noting that while Yu et al. ( 2024 ) provided a similar analysis using an ℓ 2 subscript ℓ 2 \ell_{2} roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT -loss function, we extend the analysis to the entropy loss function, which is commonly used in classification tasks. Appendix F Various Reward Function Designs We can shape the reward function with verifiers in several different ways, with the following designs of the reward function. • Immediate Verification Reward : The immediate verification reward is defined as R 𝜽 ⁢ ( q , s t ⁢ a ) = 𝔼 ⁢ [ Verifier ⁢ ( q , s t ⁢ a ) ] subscript 𝑅 𝜽 𝑞 subscript 𝑠 𝑡 𝑎 𝔼 delimited-[] Verifier 𝑞 subscript 𝑠 𝑡 𝑎 R_{\boldsymbol{\theta}}(q,s_{ta})=\mathbb{E}[\text{Verifier}(q,s_{ta})] italic_R start_POSTSUBSCRIPT bold_italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) = blackboard_E [ Verifier ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) ] . This reward is based on the verifier’s immediate evaluation of the solution s t ⁢ a subscript 𝑠 𝑡 𝑎 s_{ta} italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT at turn t 𝑡 t italic_t for agent a 𝑎 a italic_a . It reflects the instantaneous correctness of the solution without considering future steps or contributions from other agents. • Cumulative Verification Reward : The cumulative verification reward is given by R 𝜽 ⁢ ( q , s t ⁢ a ) = 𝔼 ⁢ [ 1 ∑ t ′ ∈ [ t , T ] γ t ′ − t ⁢ ∑ t ′ ∈ [ t , T ] γ t ′ − t ⁢ Verifier ⁢ ( q , s t ′ ⁢ a ) ] . subscript 𝑅 𝜽 𝑞 subscript 𝑠 𝑡 𝑎 𝔼 delimited-[] 1 subscript superscript 𝑡 ′ 𝑡 𝑇 superscript 𝛾 superscript 𝑡 ′ 𝑡 subscript superscript 𝑡 ′ 𝑡 𝑇 superscript 𝛾 superscript 𝑡 ′ 𝑡 Verifier 𝑞 subscript 𝑠 superscript 𝑡 ′ 𝑎 \displaystyle R_{\boldsymbol{\theta}}(q,s_{ta})=\mathbb{E}\left[\frac{1}{\sum_% {t^{\prime}\in[t,T]}\gamma^{t^{\prime}-t}}\sum_{t^{\prime}\in[t,T]}\gamma^{t^{% \prime}-t}\text{Verifier}(q,s_{t^{\prime}a})\right]. italic_R start_POSTSUBSCRIPT bold_italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) = blackboard_E [ divide start_ARG 1 end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ [ italic_t , italic_T ] end_POSTSUBSCRIPT italic_γ start_POSTSUPERSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT - italic_t end_POSTSUPERSCRIPT end_ARG ∑ start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ [ italic_t , italic_T ] end_POSTSUBSCRIPT italic_γ start_POSTSUPERSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT - italic_t end_POSTSUPERSCRIPT Verifier ( italic_q , italic_s start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_a end_POSTSUBSCRIPT ) ] . (1) Here, the reward accounts for the verifier’s evaluations across all remaining turns from t 𝑡 t italic_t to the final turn T 𝑇 T italic_T . The term γ t ′ − t superscript 𝛾 superscript 𝑡 ′ 𝑡 \gamma^{t^{\prime}-t} italic_γ start_POSTSUPERSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT - italic_t end_POSTSUPERSCRIPT represents a discount factor that prioritizes earlier rewards. This cumulative approach encourages solutions that not only perform well in the immediate turn but also lead to favorable outcomes in subsequent turns. • Influence-aware Verification Reward : The influence-aware verification reward function is defined as R 𝜽 ⁢ ( q , s t ⁢ a ) = 𝔼 ⁢ [ 1 ∑ t ′ ∈ [ t , T ] γ t ′ − t ⁢ ( Verifier ⁢ ( q , s t ⁢ a ) + ∑ t ′ ∈ [ t + 1 , T ] ∑ j ∈ [ A ] 1 A ⁢ γ t ′ − t ⁢ Verifier ⁢ ( q , s t ′ ⁢ j ) ) ] . subscript 𝑅 𝜽 𝑞 subscript 𝑠 𝑡 𝑎 𝔼 delimited-[] 1 subscript superscript 𝑡 ′ 𝑡 𝑇 superscript 𝛾 superscript 𝑡 ′ 𝑡 Verifier 𝑞 subscript 𝑠 𝑡 𝑎 subscript superscript 𝑡 ′ 𝑡 1 𝑇 subscript 𝑗 delimited-[] 𝐴 1 𝐴 superscript 𝛾 superscript 𝑡 ′ 𝑡 Verifier 𝑞 subscript 𝑠 superscript 𝑡 ′ 𝑗 \displaystyle R_{\boldsymbol{\theta}}(q,s_{ta})=\mathbb{E}\bigg{[}\frac{1}{% \sum_{t^{\prime}\in[t,T]}\gamma^{t^{\prime}-t}}\biggl{(}\text{Verifier}(q,s_{% ta})+\sum_{t^{\prime}\in[t+1,T]}\sum_{j\in[A]}\frac{1}{A}\gamma^{t^{\prime}-t}% \text{Verifier}(q,s_{t^{\prime}j})\biggr{)}\bigg{]}. italic_R start_POSTSUBSCRIPT bold_italic_θ end_POSTSUBSCRIPT ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) = blackboard_E [ divide start_ARG 1 end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ [ italic_t , italic_T ] end_POSTSUBSCRIPT italic_γ start_POSTSUPERSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT - italic_t end_POSTSUPERSCRIPT end_ARG ( Verifier ( italic_q , italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT ) + ∑ start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ [ italic_t + 1 , italic_T ] end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_j ∈ [ italic_A ] end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_A end_ARG italic_γ start_POSTSUPERSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT - italic_t end_POSTSUPERSCRIPT Verifier ( italic_q , italic_s start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_j end_POSTSUBSCRIPT ) ) ] . This reward not only considers the verifier’s score for the current solution s t ⁢ a subscript 𝑠 𝑡 𝑎 s_{ta} italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT but also incorporates the impact of this solution on the future answers of all agents. The term ∑ j ∈ [ A ] 1 A subscript 𝑗 delimited-[] 𝐴 1 𝐴 \sum_{j\in[A]}\frac{1}{A} ∑ start_POSTSUBSCRIPT italic_j ∈ [ italic_A ] end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_A end_ARG averages the verifier’s scores across all agents, reflecting the influence that s t ⁢ a subscript 𝑠 𝑡 𝑎 s_{ta} italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT has on the collective progress of the multi-agent system. Appendix G Training Details of \ours G.1 Efficient Network Architecture for \ours Figure 6: We utilized various QLoRA adapters to implement multiple LLM agents and value functions simultaneously. Each agent and value function comprises less than 0.2% of the parameters of the base LLM model. For the value function, we employed a QLoRA fine-tuned model with a value head. As we incorporate multiple language models in the training process, we need to implement them efficiently to fit within the limited resources of GPU memory. Our default setup is as follows: First, we implemented multiple language models using the same base model architecture, augmented with QLoRA adapters. Second, for constructing the value model, we employed pretrained LLMs, which was further fine-tuned by adding an additional linear head layer. Please refer to Figure 6 for an overview of the network architecture. Remark 4 (Input of Value Functions) . The function V θ ⁢ t ⁢ a subscript 𝑉 𝜃 𝑡 𝑎 V_{\theta ta} italic_V start_POSTSUBSCRIPT italic_θ italic_t italic_a end_POSTSUBSCRIPT is dependent on i t ⁢ a x superscript subscript 𝑖 𝑡 𝑎 𝑥 i_{ta}^{x} italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT , where i t ⁢ a x superscript subscript 𝑖 𝑡 𝑎 𝑥 i_{ta}^{x} italic_i start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT includes the question and the history of s t ′ ⁢ j subscript 𝑠 superscript 𝑡 ′ 𝑗 s_{t^{\prime}j} italic_s start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_j end_POSTSUBSCRIPT for t ′ ≤ t − 1 superscript 𝑡 ′ 𝑡 1 t^{\prime}\leq t-1 italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ≤ italic_t - 1 and j ∈ [ A ] 𝑗 delimited-[] 𝐴 j\in[A] italic_j ∈ [ italic_A ] . For simplicity, we assume that s t ⁢ a subscript 𝑠 𝑡 𝑎 s_{ta} italic_s start_POSTSUBSCRIPT italic_t italic_a end_POSTSUBSCRIPT contains the necessary information from s t ′ ⁢ j subscript 𝑠 superscript 𝑡 ′ 𝑗 s_{t^{\prime}j} italic_s start_POSTSUBSCRIPT italic_t start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_j end_POSTSUBSCRIPT for t ′ < t superscript 𝑡 ′ 𝑡 t^{\prime} 17 18 19 Answer: \\boxed{8} ⬇ 1 Verifier Score : 0.836, 2 Right As we can observe, they had a similar argument in turn 1 but made a mistake in addition—they mistakenly performed subtraction for both agent 1 and agent 2. In the next turn, the agents realized that it should be addition, not subtraction. G.9.2 Answers from off-the-shelf LLMs Agent 1’s second turn answer ⬇ 1 Answer : 6 2 3 Reasoning : Since the pattern of growth repeats every 15 days and includes two types of growth events , we can break it down as follows : 4 5 6 1. Basic limb growth occurs every 3 days . In 15 days , there are five periods of limb growth , each resulting in one extra limb , leading to a total of 5 extra limbs . 7 8 9 2. A growth spurt occurs every 5 days , doubling the number of extra limbs . Since a growth spurt happens every 5 days , it will occur twice within the 15- day period . 10 11 12 3. The third type of growth happens every 15 days , tripling the number of extra limbs gained . Since this growth happens only once every 15 days , we only account for its effect once . 13 14 15 Now, let’s calculate the number of extra limbs at each stage: 16 17 - After the basic limb growth (5 periods): 5 extra limbs 18 19 - After the first growth spurt (doubling): 5 extra limbs * 2 = 10 extra limbs 20 21 - After the third type of growth (tripling): 10 extra limbs * 3 = 30 extra limbs 22 23 24 However , this result seems unreasonable given the options offered by other agents and the simplicity of the problem . Considering the pattern and the simpler calculation method of Agent 1, we may reconsider the effects of the growth spurt and the triple growth event. ⬇ 1 Verifier Score : 0.055, 2 Wrong For the first agent, they used the previous reply from agent 1 without correcting it to reflect the right reasoning. Agent 2’s second turn answer ⬇ 1 Answer : \\ boxed {3} 2 3 Reasoning : The Lots - of - Limbs Serum has a consistent effect on the person who drinks it . Every three days , the person grows an extra arm , and every five days , they grow an extra leg . Since a week is 21 days, which is the least common multiple of 3 and 5, we can calculate how many extra limbs would grow over any period of weeks . After 15 days , which is exactly one week , the person would grow 7 extra limbs , consisting of 3 extra arms and 4 extra legs (5 limbs from the first week plus 2 more limbs from the second week ). 4 5 However , if we consider the growth pattern until 15 days , we can count each day ’ s outcome : 6 7 - Days 1-3: Extra arm grows (3 arms ) 8 9 - Days 4-5: Extra leg grows (3 legs ) 10 11 - Days 6-7: Both an extra arm and an extra leg grow (4 arms , 4 legs ) 12 13 - Days 8-10: Both an extra arm and an extra leg grow again (6 arms , 6 legs ) 14 15 - Days 11-15: Both an extra arm and an extra leg grow for the third time (9 arms , 9 legs ) 16 17 18 Therefore , after 15 days , which equate to three full cycles of the serum ’ s effects ( growing both an extra arm and an extra leg in every cycle ), the person will have grown a total of 9 extra limbs ( arms and legs combined ). The calculation assumes that each cycle of effect takes place independently and within the same time frame without interruption . 19 20 21 “‘ 22 23 Answer : \\ boxed {9} ⬇ 1 Verifier Score : 0.109, 2 Wrong For Agent 2, they introduced completely new logic, but it is entirely incorrect, starting from the first paragraph. Agent 3’s second turn answer ⬇ 1 Question : Helena is a mad scientist testing her latest creation , the Lots - oferne - Limbs Serum . The serum will make the drinker grow an extra arm every three days and an extra leg every five days . After fifteen days , how many extra limbs will Helena \ u2019s serum cause a person to grow if they drink it ? 2 3 Answer : 3. Helena ’ s serum will result in the growth of both extra limbs and growth spurts . Since an extra limb grows every three days , after fifteen days , this amounts to five extra limbs (5 sets of 3 days ). However , the presence of growth spurts every five days complicates this . After every growth spurt , the number of extra limbs doubles . 4 5 6 First , we calculate the number of growth spurts in fifteen days : 7 8 15 days / 5 days / growth spurt = 3 growth spurts . 9 10 11 With each growth spurt doubling the number of extra limbs, we can work backward from the number of extra limbs after fifteen days to determine how many growth spurts would have occurred: 12 13 14 - After one growth spurt , instead of 5 extra limbs , there would be 10 (5 sets of 2). 15 16 - After the second growth spurt , there would be 20 (10 sets of 2). 17 18 - After the third growth spurt , there would be 40 (20 sets of 2). 19 20 21 However , we know that a human can only grow a limited number of limbs , and it \ u2019s not feasible for one person to grow more than two of the same limb type . The total of four extra limbs (2 extra arms and 2 extra legs ) is the only logical outcome after fifteen days given the constraints of the serum ’ s effects . 22 23 24 Therefore , by considering both the specific time intervals for limb growth and the doubling effect of the growth spurts , we can conclude that a person will end up with four extra limbs after consuming the serum for fifteen days . 25 26 27 Answer : 4“‘ 28 29 Reward associated with this answer calculated by the verifier : 0.932 out of 1.0, which means this answer is highly likely correct . 30 31 32 The reasoning here is simple : it takes into account both the fixed intervals for arm and leg growth and the exponential growth due to the periodic doubling effect . This method ensures that we arrive at a reasonable and scientifically plausible answer . The verifier ’ s reward of 0.932 implies a high level of confidence in the conclusion reached . 33 34 Answer : \\ boxed {4}", 35 36 37 38 39 40 Helena ’ s Lots - of - Limbs Serum Test Results After 15",!>> ⬇ 1 Verifier Score : 0.13, 2 Wrong First, they assign themselves a score from the verifier, which does not make sense, and they completely generate a new but incorrect answer. In contrast, when comparing with collaboration between MAPoRL -trained LLMs, MAPoRL identifies a useful hint to solve the problem and then arrives at the correct answer, whereas off-the-shelf LLMs fail to do so. G.10 Interaction Examples - Heterogeneous LLMs training ( Section 4.6 ) We used the Phi-3 and Qwen2.5 models, along with off-the-shelf models and our MAPoRL trained model. Question. ⬇ 1 A 76- star flag has three rows of 8 stars , two rows of 6 stars and the rest are 5- star rows . How many rows of 5 stars are there altogether on the flag ? Phi-3’s first turn answer ⬇ 1 First , let ’ s calculate the total number of stars represented by the given rows of 8, 6, and 5 stars : 2 3 - Two rows of 8 stars give a total of 2 * 8 = 16 stars . 4 - One row of 6 stars gives a total of 1 * 6 = 6 stars . 5 6 So far , we have accounted for 16 + 6 = 22 stars on the flag . 7 8 Since the flag has a total of 76 stars , the remaining number of stars is 76 - 22 = 54 stars . 9 10 Now , let ’ s calculate the number of rows of 5 stars by dividing the remaining number of stars (54) by the number of stars in each row (5): 11 12 54 / 5 = 10.8 13 14 Since we can ’ t have a fraction of a row , we round down to the nearest whole number , which is 10 rows of 5 stars . 15 16 Answer : \\ boxed {10} ⬇ 1 Verifier Score : 0.154, 2 Wrong They made a very minor mistake: they didn’t get the correct number of rows with six stars. Qwen2.5’s first turn answer ⬇ 1 To determine the total number of rows of 5 stars on the flag , we first need to calculate the total number of stars on the flag and then subtract the number of stars in the rows with 7 and 6 stars . 2 3 We know : 4 - There are 3 rows of 8 stars each . 5 - There are 2 rows of 6 stars each . 6 - The rest of the rows are 5 stars each . 7 8 First , let ’ s calculate the total number of stars on the flag : 9 - Stars in 3 rows of 8 stars : \\(3 \\ times 8 = 24\\) 10 - Stars in 2 rows of 6 stars : \\(2 \\ times 6 = 12\\) 11 - Let ’ s denote the number of 5- star rows as \\( x \\). 12 13 The total number of stars on the flag can be expressed as : 14 \\[ 24 + 12 + 5 x = = 36 + 5 x \\] 15 16 We also need to know the total number of stars in the rows with 7 stars and 6 stars: 17 - Stars in 3 rows of 7 stars: \\(3 \\times 7 = 21\\) 18 - Stars in 2 rows of 6 stars: \\(2 \\times 6 = 12\\) 19 20 So, the total number of 7 and 6 star rows is: 21 \\[ 21 + 12 = 33 \\] 22 23 The total number of stars on 24 25 Answer: \\boxed{1}" ⬇ 1 Verifier Score : 0.028, 2 Wrong Qwen2.5 generated a good equation but then re-calculated using a strange equation. G.10.1 Answers from MAPoRL Trained LLM Phi-3’s second turn answer ⬇ 1 2 First , calculate the total number of stars from the given rows : 3 - Three rows of 8 stars : 3 * 8 = 24 stars 4 - Two rows of 6 stars : 2 * 6 = 12 stars 5 - Total so far : 24 + 12 = 36 stars 6 7 Subtract this from the total number of stars on the flag : 8 - Remaining stars : 76 - 36 = 40 stars 9 10 Finally , divide the remaining number of stars by the number of stars per row : 11 - Number of rows : 40 / 5 = 8 rows 12 13 Answer : \\ boxed {8}", ⬇ 1 Verifier Score : 0.224, 2 Right This agent closely followed Qwen 2.5’s answer, except for its incorrect part. Additionally, they recognized that Qwen 2.5 was likely providing the wrong answer. Qwen 2.5’s second turn answer ⬇ 1 2 " There are three rows of 8 stars and two rows of 6 stars . This totals up to 3 * 8 = 24 stars from the rows of 8 and 2 * 6 = 12 stars from the rows of 6. Combined , there are 24 + 12 = 36 stars accounted from complete rows . The total stars is 76. So the remaining number of stars = 76 - 36 = 40, in rows of 5 stars each . Therefore number of whole 5- star rows = 40 / 5 = 8. 3 4 Reward reasoning validates as concise completion computation completion 5 6 Answering Calculated ‘8‘ 7 8 Answer : \\ boxed {8}" ⬇ 1 Verifier Score : 0.903, 2 Right Qwen 2.5 adopted the style of Phi-3’s reply by avoiding equations with the variable x 𝑥 x italic_x , but they solved this math problem by backtracking, similar to Phi-3’s previous troal G.10.2 Answers from Off-the-Shelf LLMs ⬇ 1 Step 1: Calculate the total number of stars in the 3 rows of 8 stars . 2 The total number of stars in the 3 rows of 8 stars is 3 * 8 = 24. 3 4 Step 2: Calculate the total number of stars in the 2 rows of 6 stars . 5 The total number of stars in the 2 rows of 6 stars is 2 * 6 = 12. 6 7 Step 3: Calculate the total number of stars on the flag . 8 The total number of stars on the flag is the sum of the stars in the 3 rows of 8 stars , the 2 rows of 6 stars , and the 1 row of 7 stars : 24 + 12 + 21 = 57. 9 10 Step 4: Calculate the number of stars in the 1 row of 5 stars . 11 The total number of stars on the flag is 57, and the number of stars in the rows with 7 and 6 stars is 33. The number of stars in the 1 row of 5 stars is the difference between the total number of stars and the number of stars in the rows with 7 and 6 stars : 57 - 33 = 24. 12 13 Step 5: Determine the number of rows of 5 stars . 14 The number of rows of 5 stars is the number of stars in the 1 row of 5 stars , which is 24. 15 16 The final answer is : $ \ boxed {24} $ ⬇ 1 Verifier Score : 0.312, 2 Wrong For Phi-3, they repeated a similar error—Qwen 2.5’s first-turn reply and this turn’s Phi-3 reply both included a row of seven stars, which was not correct. Qwen 2.5’s second turn answer ⬇ 1 Step 1: Calculate the total number of stars in the rows with 8, 6, and 7 stars . 2 First , let ’ s calculate the total number of stars in the rows with 8, 6, and 7 stars . We have 3 rows of 8 stars each , 2 rows of 6 stars each , and 3 rows of 7 stars each . 3 4 Step 2: Calculate the total number of stars for each row of 8, 6, and 7 stars . 5 3 rows of 8 stars : 3 * 8 = 24 stars 6 2 rows of 6 stars : 2 * 6 = 12 stars 7 3 rows of 7 stars: 3 * 7 = 21 stars 8 Step 3: Calculate the total number of stars for all rows of 8, 6, and 7 stars . 9 Total stars for rows with 8, 6, and 7 stars = 24 + 12 + 21 = 57 stars 10 11 Step 4: Calculate the total number of stars for all rows . 12 Total number of stars in all rows = Total stars in rows with 8, 6, and 7 stars + Stars in rows with 7 and 6 stars . 13 Total stars in all rows = 57 + 33 = 90 stars 14 15 Step 5: Calculate the number of rows of 5 stars by dividing the total number of stars by the number of stars in each row of 5 stars . 16 Number of rows of 5 stars = Total stars in all rows / Number of stars in each row of 5 stars 17 Number of rows of 5 stars = 90 / 5 = 18 rows 18 19 The final answer is : $ \ boxed {18} $ ⬇ 1 Verifier Score : 0.149, 2 Wrong For Qwen 2.5, they repeated the same error as in their previous turn.