No single LLM is universally optimal across all tasks. Different models are fine-tuned to specialize in particular domains. The challenge is: how to dynamically coordinate a pool of diverse, specialized LLMs to solve complex problems that exceed any individual model's capabilities?
- Core Problem Definition
- Method
- Conductor Output Format
- Reward Structure
- GRPO Training Objective
- Training Data
- Worker Pool
- Extensions
Learning to Orchestrate Agents with the RL Conductor
arXiv: 2512.04388v4 [cs.LG] | Date: 2026-03-01 Authors: Stefan Nielsen, Edoardo Cetin, Peter Schwendeman, Qi Sun, Jinglue Xu, Yujin Tang (Sakana AI)
1. Core Problem Definition
No single LLM is universally optimal across all tasks. Different models are fine-tuned to specialize in particular domains. The challenge is: how to dynamically coordinate a pool of diverse, specialized LLMs to solve complex problems that exceed any individual model's capabilities?
Prior approaches use manually-designed agentic workflows or learned routers that select from pre-specified topologies. The RL Conductor removes these constraints — it learns to design arbitrary coordination strategies in natural language through end-to-end reinforcement learning.
2. Method
2.1 Conductor Output Format
The Conductor outputs a sequence of workflow steps, each defined by three Python lists:
model_id = [2, 0] # Which worker agent performs each step
subtasks = [ # Natural language instruction for each step
"Develop an efficient algorithm to count complete subarrays",
"Implement the algorithm described by the previous agent in Python"
]
access_list = [[], ["all"]] # Which prior step outputs each worker sees
This formulation enables: - Sequential chains (step N feeds into step N+1) - Parallel branches (independent subtasks executed simultaneously) - Tree-structured workflows (fan-out/fan-in patterns) - Best-of-N (same subtask to multiple agents, select best) - Debate/refinement (iterative improvement with visibility into prior attempts)
2.2 Reward Structure
The Conductor reward rᵢ has two conditions: 1. Format condition: rᵢ = 0 if Python lists cannot be parsed from the response 2. Correctness condition: rᵢ = 1 if the final output matches the solution sᵢ, else rᵢ = 0.5
2.3 GRPO Training Objective
Using Group Relative Policy Optimization (GRPO) with G > 1 grouped completions:
J(θ) = E_{q∼D, {o}∼π(·|q)} [ 1/G Σᵢ ( min(rᵢAᵢ, clip(rᵢ, 1-ε, 1+ε)Aᵢ) - β D_KL(π_θ || π_ref) ) ]
Where the advantage function is:
Aᵢ = (rᵢ - mean({r₁,...,r_G})) / std({r₁,...,r_G})
Training hyperparameters: - 200 GRPO iterations - Batch size: 256 - Optimizer: AdamW - No KL regularization (β = 0) - Base model: Qwen2.5 7B
2.4 Training Data
960 problems from four reasoning domains: - MATH (Hendrycks et al., 2021): Competition mathematics - MMLU (Hendrycks et al., 2020): Multitask language comprehension - RLPR (Yu et al., 2025): Real-world reasoning - LiveCodeBench V1 (Jain et al., 2024): Code generation
2.5 Worker Pool
Proprietary frontier models: Gemini-2.5-Pro, Claude-Sonnet-4, GPT-5 Open-source models: DeepSeek-R1-Distill-Qwen-32B, Gemma3-27B-instruct, Qwen3-32B
3. Extensions
3.1 Adaptive Worker Selection
Finetune a pretrained Conductor with randomized agent pools at each step. For each question, restrict to a random k-model subset from the total pool of n workers. After training, the Conductor generalizes to arbitrary subsets of open/closed-source models.
3.2 Recursive Topologies and Test-Time Scaling
Allow the Conductor to specify itself as a worker. During recursive calls, the Conductor receives its own parent output plus previous agent responses. It can instantiate new workflows or end the coordination loop.
- Maximum recursion depth is a tunable parameter at inference time
- Unlocks a new form of test-time scaling: more recursion calls = more compute = better performance
- Finetuned with same RL algorithm, manually instantiating one recursion call per batch
4. Experimental Results
4.1 Main Results: State-of-the-Art Comparison
Table 1: Comparison with Previous Best "Unconstrained" Results
| Model | M500 | MMLU | RLPR | LCB | AIME25 | BCB | GPQA-D | Avg. |
|---|---|---|---|---|---|---|---|---|
| gemma-3-27b-it | 39.8 | 81.3 | 16.67 | 13.14 | 20.7 | 14.86 | 38.4 | 32.12 |
| Qwen3-32B | 73.5 | 83.5 | 31.00 | 21.21 | 20.0 | 30.41 | 64.1 | 53.81 |
| Qwen3-32B (thinking) | 80.7 | 84.1 | 37.25 | 25.86 | 72.9 | 28.38 | 66.8 | 56.57 |
| R1-Distill-32B | 82.5 | 84.4 | 33.50 | 26.86 | 63.0 | 33.07 | 58.1 | 54.49 |
| Claude Sonnet 4 | 96.0 | 91.4 | 36.70 | 46.54 | 74.3 | 37.16 | 77.7 | 65.69 |
| Gemini 2.5 Pro | 96.0 | 92.4 | 40.55 | 67.24 | 78.3 | 37.51 | 84.8 | 70.97 |
| GPT 5 | 99.0 | 93.5 | 42.20 | 82.90 | 90.8 | 32.75 | 82.3 | 74.78 |
| Conductor (Ours) | 99.4 | 94.1 | 44.75 | 83.93 | 93.3 | 37.86 | 87.5 | 77.27 |
The 7B Conductor surpasses GPT-5 (77.27 vs 74.78 average) and achieves SOTA on GPQA-Diamond and LiveCodeBench.
4.2 Controlled Evaluation vs Multi-Agent Baselines
The Conductor is compared against: - 5-turn self-reflection: Each agent revises up to 5 times - MASRouter (Yue et al., 2025): Learned routing with human-designed topologies - Mixture-of-Agents (MoA) (Wang et al., 2024): Layered agent aggregation - RouterDC (Chen et al., 2024): Router selecting single best-matched agent - Smoothie (Guha et al., 2024): Ensemble routing
Key finding: The Conductor surpasses all baselines while using fewer agent calls (avg 3 steps vs 4-5 for MASRouter). All baselines except RouterDC have strictly higher inference cost.
4.3 Recursive Test-Time Scaling
Table 2: Recursive Conductor on Out-of-Distribution Tasks
| Model | AIME25 | BigCodeBench | GPQA-D | Average |
|---|---|---|---|---|
| gemma-3-27b-it | 6.67 | 10.8 | 33.33 | 16.93 |
| Qwen3-32B | 23.33 | 23.0 | 54.05 | 33.46 |
| Qwen3-32B (thinking) | 23.33 | 20.9 | 59.09 | 34.44 |
| R1-Distill-32B | 30.00 | 24.3 | 51.01 | 35.10 |
| Gemini Pro 2.5 | 46.67 | 35.1 | 75.25 | 52.34 |
| Claude Sonnet 4 | 35.33 | 35.8 | 67.30 | 46.14 |
| GPT 5 | 46.67 | 33.8 | 72.73 | 51.73 |
| Conductor | 66.67 | 37.8 | 81.31 | 61.93 |
| Conductor-Recursive | 66.67 | 40.0 | 82.32 | 63.00 |
Recursion yields additional gains on BigCodeBench (+2.2) and GPQA-Diamond (+1.01) while using < 2× the original agent calls.
4.4 Adaptive Worker Selection Results
When finetuned on randomized model pools: - Open-model-only: Consistently outperforms Claude Sonnet 4 by ~10% - Closed-model-only: Matches pretrained SOTA performance - The Conductor is not exclusively reliant on frontier models
4.5 Conductor Scale Analysis
Comparing 3B vs 7B Conductors: - Both converge to the same worker agent distribution - The 7B variant maintains a clear edge due to superior prompt engineering skills - The performance gap is traced to the larger model's ability to generate better subtask instructions
4.6 Task and Difficulty Adaptivity
The Conductor learns to dynamically allocate compute based on task difficulty: - MMLU (factual): Typically 1-2 agent steps - LiveCodeBench (complex coding): 3-4 agent steps with planning → implementation → verification - The model explicitly reasons about task complexity before designing workflows
5. Training Configuration Summary
| Parameter | Value |
|---|---|
| Base model | Qwen2.5 7B |
| Algorithm | GRPO |
| Training iterations | 200 |
| Batch size | 256 |
| Group size G | > 1 |
| KL coefficient β | 0 |
| Optimizer | AdamW |
| Training problems | 960 |
| Domains | MATH, MMLU, RLPR, LiveCodeBench V1 |
| Max workflow steps | 5 |
| Avg workflow steps (learned) | ~3 |
6. Comparison with Existing Methods
| Method | Avg Score | Cost | Flexibility |
|---|---|---|---|
| Single best model (GPT-5) | 74.78 | 1× | Fixed |
| 5-turn self-reflection | ~55 | 5× | Limited |
| MASRouter | ~60 | 4-5× | Predefined topologies |
| Mixture-of-Agents | ~62 | Multi-layer | Fixed structure |
| RouterDC | ~58 | Variable | Single agent selection |
| RL Conductor | 77.27 | ~3× | Full NL freedom |
7. Critical Analysis
Strengths: - First demonstration that pure end-to-end RL can discover powerful multi-agent coordination strategies - Achieves SOTA with a 7B model coordinating much larger workers (32B+) - Removes manual constraints on subtask specification — natural language as the coordination medium - Two practical extensions (adaptive worker selection, recursive scaling) via short finetuning - Emergent behaviors: verification, debate, planning, iterative refinement - Difficulty-adaptive compute allocation - Open-source code and model weights
Weaknesses: - Worker dependency: Performance relies on having powerful frontier workers; the Conductor cannot create capability that doesn't exist in the pool - Training cost: Requires 960 problems across 4 domains + 200 GRPO iterations with expensive frontier API calls - Small training set: Only 960 problems; unclear how well this scales to broader domains - No theoretical analysis: No formal guarantees about convergence or optimality of discovered strategies - Recursive depth limit: Maximum recursion depth is manually set; no automatic determination of when to stop - Evaluation stochasticity: Results repeated up to 16 times due to stochasticity, suggesting variance concerns
Open Questions: - Can the Conductor discover novel coordination strategies not anticipated by designers? - How does performance scale with worker pool size and diversity? - Can recursive scaling be extended to deeper hierarchies? - What happens when workers have conflicting or adversarial outputs? - Can the Conductor generalize to non-reasoning domains (e.g., creative writing, multimodal tasks)?
Related
- multi-agent-coordination — Multi-agent collaboration frameworks
- grpo — Group Relative Policy Optimization
- test-time-scaling — Test-time compute scaling for reasoning
- agentic-workflows — Manually designed agentic workflow patterns
- mixture-of-agents — MoA ensemble approach
- self-reflection — Self-refinement strategies in LLMs