---
title: "Agent 2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?"
authors: ["Wanyi Chen", "Xiao Yang", "Xu Yang", "Tianming Sha", "Qizheng Li", "Zhuo Wang", "Bowen Xian", "Fang Kong", "Weiqing Liu", "Jiang Bian", "Soochow University", "Microsoft Research Asia", "Peking University", "Stony Brook University", "The University of Chicago"]
url: "https://arxiv.org/abs/2604.10547"
sections: 102
estimated_tokens: "27.3k"
---

## Contents
- 1 Introduction
- 2 Agent 2 RL-Bench: An Agentic RL Benchmark
  - 2.1 Problem Formulation
  - 2.2 Training Structure Coverage
  - 2.3 Benchmark Selection
  - 2.4 System Design
  - 2.5 Data Integrity and Evaluation
  - 2.6 Benchmark-Specific Details
    - Static rule-based tasks.
    - Static judge-based task.
    - Interactive rollout tasks.
    - Grading reliability.
- 3 Experiments
  - 3.1 Experimental Setup
  - 3.2 Cross-Agent Comparison
  - 3.3 Behavioral Analysis: Capability Stratification
    - Cross-agent behavioral differences.
    - ALFWorld near-saturation.
    - DeepSearchQA structural intractability.
    - SFT fallback as the dominant pattern.
  - 3.4 Operating Mode Analysis
  - 3.5 Base Model Results
- 4 Related Work
- 5 Conclusion
- References
- Appendix A Appendix: Protocol and Supplementary Analysis
  - A.1 Structural Coverage Rubric
    - Dimension definitions.
  - A.2 Benchmark Protocol Details
    - Static rule-based tasks.
    - Static judge-based tasks.
    - Interactive rollout tasks.
  - A.3 Reported Metrics
    - AlpacaEval evaluation reliability.
  - A.4 Experimental Setup Table
  - A.5 Run Contract
    - Practical constraints.
  - A.6 Agent Instruction Excerpt
  - A.7 Structured Run Reports
    - Human-readable summaries.
    - Machine-readable iteration logs.
    - Role in the paper.
  - A.8 Judge Prompt Excerpts
    - AlpacaEval-style preference judging.
    - DeepSearchQA answer judging.
  - A.9 Complete Result Matrix
  - A.10 Supporting Scaling Tables
  - A.11 Route-Analysis Definitions
  - A.12 Supplementary Case Studies
    - OpenCode on ALFWorld.
    - WebShop collapse.
  - A.13 Representative Code Evidence for RL and Hybrid Routes
    - GSM8K: explicit GRPO route.
    - ALFWorld: cleaned DAgger-style route.
    - WebShop: substantial interactive engineering without gain.
    - DeepSearchQA: tool-augmented search trajectory SFT.
  - A.14 Operational Safeguards and Reproducibility
  - A.15 Reproducibility and Environment Note
    - Benchmark-facing environment.
    - What is and is not claimed.
  - A.16 Audit and Anti-Cheating Rules
  - A.17 Failure Taxonomy Summary
  - A.18 Claude Code Agent Behavior Statistics
    - Exploration volume varies by orders of magnitude.
    - Late peaking vs. early peaking.
    - Zero-score rate as a fragility indicator.
  - A.19 Claude Code Per-Task Case Studies
    - Case 1: GSM8K — Model soup over 92 submissions.
    - Case 2: HumanEval — Most prolific exploration.
    - Case 3: AlpacaEval — Synthetic preference data breakthrough.
    - Case 4: ALFWorld — Demonstration learning via environment interaction.
    - RL-only ALFWorld workspace audit.
    - Case 5: WebShop — Sign reversal through trajectory quality.
    - Case 6: DeepSearchQA — Exhaustive technique search.
    - DeepSearchQA follow-up: evaluation variance and the SFT sweet spot.
  - A.20 Claude Code Score Trajectory Analysis
  - A.21 Additional Experimental Results (from Main Text Compression)
    - Operating mode ablation.
    - RL method audit.
    - Multi-task degradation on Qwen3-8B-Base.
    - Complete 8B controlled study matrix.
    - Technical route distribution for OpenCode-7B.
    - Computational cost.
    - Agent behavior efficiency: Claude Code vs. Codex.
    - Variance, scaling, and pass@k.
  - A.22 Detailed Trace Walkthrough: HumanEval on Qwen3-8B-Base
    - Setup.
    - Step 1: SFT v1 (Sub #3, 2.44%).
    - Step 2: SFT v3 with prompt alignment (Sub #5, 43.9%).
    - Step 3: GRPO with code execution reward (Sub #7, 79.27%).
    - Step 4: RFT after GRPO degradation (Sub #11, 84.15%).
    - Summary of agent capabilities and limitations.
  - A.23 Detailed Trace Walkthrough: ALFWorld on Qwen3-8B-Base
    - Setup.
    - The hidden action-syntax trap.
    - Step 1: Baseline SFT (Sub #1–#3, 14.93–16.42%).
    - Step 2: Action syntax fix + synthetic trajectories (Sub #4, 19.40%).
    - Step 3: Model rollout trajectories (Sub #5, 45.52%).
    - Step 4: Prompt format alignment—the 50-point breakthrough (Sub #7, 95.52%).
    - Step 5: Iterative self-play (Sub #8–#9, 96.27–97.76%).
    - Comparison with other ALFWorld runs.
    - Summary of agent capabilities and limitations.

## Abstract

Abstract We introduce Agent 2 RL-Bench, a benchmark for evaluating agent ic RL post-training—whether LLM agent s can autonomously design, implement, and run complete RL pipelines that improve foundation models. This capability is important because RL post-training increasingly drives model alignment and specialization, yet existing benchmarks remain largely static: supervised fine-tuning alone yields strong results, leaving interactive RL engineering untested. Agent 2 RL-Bench addresses this with six tasks across three levels—from static rule-based training to closed-loop online RL with trajectory collection—each adding a structural requirement that prior levels do not impose. The benchmark provides isolated workspaces with a grading API, runtime instrumentation that records every submission and code revision, and automated post-hoc analysis that generates structured run reports, enabling the first automated diagnostic of agent-driven post-training behavior. Across multiple agent stacks spanning five agent systems and six driver LLMs, we find that agents achieve striking interactive gains—on ALFWorld, an RL-only agent improves from 5.97 to 93.28 via SFT warm-up and GRPO with online rollouts—yet make only marginal progress on others (DeepSearchQA: +2.75 within evaluation noise), and that driver choice has a large effect on interactive tasks—within the same scaffold, switching drivers changes interactive improvement from near-zero to +78pp. More broadly, the benchmark reveals that supervised pipelines dominate agent-driven post-training under fixed budgets, with online RL succeeding as the final best route only on ALFWorld. Code is available at https://github.com/microsoft/RD-Agent/tree/main/rdagent/scenarios/rl/autorl_bench .

## 1 Introduction

RL post-training now shapes how frontier models are aligned and specialized (Ouyang et al., 2022; DeepSeek-AI, 2025), but building a working pipeline still requires expert judgment across rewards, algorithms, stability, and online data collection. If LLM agents could automate this process end to end, they would become powerful tools for model development. But can they?

We call this capability agentic RL post-training: an LLM agent autonomously designs, implements, debugs, and runs a post-training pipeline—potentially including closed-loop RL—to improve a given foundation model, all under a limited engineering budget. This is not a coding task. An agent must understand the training objective, select a strategy, build the data pipeline, implement environment interaction and rollout collection where needed, manage trajectory-level rewards, diagnose failures, and iterate until the model improves. It is a long-horizon systems engineering challenge.

Figure: Figure 1: Overview of Agent^2 RL-Bench. The benchmark is organized into three levels of increasing complexity, from static rule-based and judge-based tasks to interactive rollout tasks that require agents to close an online training loop.
Refer to caption: 2604.10547v1/figures/fig1_intro_overview.png

Existing benchmarks only partially test this capability. Agent benchmarks have grown in complexity, from code generation (Chen et al., 2021) to software engineering (Jimenez et al., 2023) to ML pipeline automation (Chan et al., 2024) and post-training (Rank et al., 2026), but even the closest ones remain largely static: supervised fine-tuning alone can yield strong results without requiring agents to implement rollout, handle trajectory-level rewards, or sustain online data collection. Strong performance on these benchmarks can overstate an agent’s ability to engineer online RL systems.

We introduce Agent^2 RL-Bench to close this gap. The benchmark contains six tasks across three progressively harder levels. Level 1 uses static rule-based tasks (GSM8K, HumanEval) with deterministic verification, testing basic data construction and supervised training. Level 2 introduces static judge-based rewards (AlpacaEval), requiring optimization against non-deterministic signals. Level 3 demands full environment interaction, trajectory collection, and closed-loop online RL under a fixed 12-hour budget (ALFWorld, WebShop, DeepSearchQA). Each level adds a structural requirement that the previous level does not impose, making the progression diagnostic: success on earlier levels should not be assumed to imply success on Level 3.

Beyond final scores, Agent^2 RL-Bench is designed for behavioral diagnosis. The benchmark provides each agent an isolated workspace and a grading API for iterative submission, together with two key capabilities. First, *runtime instrumentation* records every submission, code revision, and model artifact throughout the agent’s run. Second, a *post-hoc analysis module* automatically generates structured case studies and run reports from these artifacts. Together, these enable automated diagnostic analysis of agent-driven post-training behavior, revealing not just whether agents succeed but how they search, fail, and adapt.

In controlled experiments with five agent systems spanning three scaffold families and six driver LLMs (OpenHands, OpenCode, Claude Code, Codex CLI, Gemini CLI), we find that agents achieve striking interactive gains—on ALFWorld, an RL-only agent improves from 5.97 to 93.28 via SFT warm-up and GRPO with online rollouts—yet make only marginal progress on others (DeepSearchQA: +2.75 within evaluation noise). A controlled study on Qwen3-8B-Base provides suggestive evidence that driver choice has a large effect on interactive tasks: within a fixed scaffold, switching drivers changes ALFWorld improvement from near-zero to +78pp. More broadly, we observe that supervised pipelines dominate agent-driven post-training under fixed budgets, with online RL succeeding as the final best route only on ALFWorld.

These findings lead to a clear conclusion: static benchmarks alone are insufficient for evaluating post-training competence, and meaningful evaluation requires both interactive environments and behavioral analysis of agent strategies. Our contributions are:

- •
We introduce Agent^2 RL-Bench, a benchmark for agentic RL post-training with six tasks spanning three progressive levels from static verification to closed-loop online RL, together with a diagnostic framework featuring runtime instrumentation and automated post-hoc analysis.
- •
Across multiple agent stacks, we show that agents achieve dramatic gains on some interactive tasks but fail to engineer stable online RL reliably, exposing a consistent gap between writing RL code and running RL systems.
- •
A controlled scaffold–driver study on Qwen3-8B-Base provides suggestive evidence that driver choice has a large effect on interactive tasks—within fixed scaffolds, switching drivers changes ALFWorld improvement from near-zero to +78pp—and that alignment-gap closure from an unaligned base model depends on the full agent stack.

## 2 Agent 2 RL-Bench: An Agentic RL Benchmark

### 2.1 Problem Formulation

Each benchmark instance is defined by a task description $\mathcal{D}$, a base model $M_{0}$, training data $\mathcal{T}$, an evaluation oracle $\mathcal{O}$, and a time budget $\tau$. Within budget $\tau$, an LLM-driven agent $\mathcal{A}$ must inspect the task and data, choose a post-training route, write and execute training code, submit candidate models for evaluation, and return the best model

$$ $M^{*}=\arg\max_{i}\mathcal{O}(M_{i}).$ $$

The central benchmark score is improvement over the base model:

$$ $\Delta=\mathcal{O}(M^{*})-\mathcal{O}(M_{0}).$ $$

Two aspects of this formulation are central. First, Agent^2 RL-Bench evaluates the *full post-training loop* rather than the quality of a generated training script in isolation. The agent must choose a method, produce runnable code, obtain a trained artifact, and improve measured performance. Second, the protocol is explicitly iterative. During the allotted budget, the agent may submit candidate models to the grading server, observe the resulting score, and decide whether to continue training, revise the implementation, or switch strategies. The run-level score is therefore the best submission within budget rather than the first or last submission. This matches the structure of real post-training work, where diagnosis, repair, and resubmission are part of the capability being evaluated.

### 2.2 Training Structure Coverage

The central design principle of Agent^2 RL-Bench is *training structure coverage*. Rather than maximizing task count or topical diversity, we select tasks that span the structural transition from static post-training to full agentic RL engineering. Many existing post-training evaluations mostly measure whether an agent can assemble a useful fine-tuning workflow, whereas agentic RL becomes hardest when stateful interaction, trajectory handling, and online data generation are required. To make this distinction explicit, we organize the benchmark into three levels:

**Table 1: Benchmark overview organized by training structure.**
| Level | Type | Benchmark | Reward | Horizon |
| --- | --- | --- | --- | --- |
| L1 | Static rule-based | GSM8K | Rule-based | Short |
| L1 | Static rule-based | HumanEval | Rule-based | Short |
| L2 | Static judge-based | AlpacaEval 2.0 | LLM judge | Short |
| L3 | Interactive rollout | ALFWorld | Sparse binary | Long |
| L3 | Interactive rollout | WebShop | Dense continuous | Long |
| L3 | Interactive rollout | DeepSearchQA | Judge-based | Medium |

L1 covers static rule-based post-training, where single-turn outputs are scored by deterministic verifiers. L2 introduces static judge-based post-training, where rewards are no longer directly programmable but the task remains single-turn and non-interactive. L3 is the key level. These tasks require agents to interact with a stateful environment, collect multi-step trajectories, and reason over long-horizon rewards.

The benchmark hypothesis is structural rather than topical: success on L1 or L2 should not be assumed to imply success on L3. In static settings, SFT often acts as a safe proxy for RL because reward is applied to a single output by a verifier or judge. In interactive settings, the agent must additionally implement environment stepping, maintain observation histories, handle trajectory-level rewards, and sustain online data collection. These requirements are what make L3 a meaningful test of agentic RL rather than post-training automation alone.

Figure: Figure 2: System pipeline of Agent^2 RL-Bench, showing the shared outer loop and task-specific evaluators.
Refer to caption: 2604.10547v1/figures/fig2_pipeline.png

### 2.3 Benchmark Selection

We select six tasks to cover distinct capability types, reward structures, and decision horizons. GSM8K (Cobbe et al., 2021) targets mathematical reasoning under a deterministic answer checker. HumanEval (Chen et al., 2021) targets code generation under rule-based unit tests. AlpacaEval 2.0 (Dubois et al., 2024) represents a static judge-based setting in which reward is not directly hand-coded. ALFWorld (Shridhar et al., 2020) and WebShop (Yao et al., 2022) represent interactive long-horizon tasks with stateful environments. Our DeepSearchQA task is built on the google/deepsearchqa dataset (Google DeepMind, 2026), but uses a custom ReAct-style (Yao et al., 2023) search-and-judge evaluator to fit the unified benchmark interface.

This benchmark set is intentionally compact. Rather than maximizing task count, we construct a small matrix in which each task contributes a distinct structural requirement: GSM8K and HumanEval anchor static rule-based post-training, AlpacaEval adds a non-programmable judge while remaining single-turn, ALFWorld and WebShop introduce stateful interaction under sparse and dense rewards, and DeepSearchQA adds tool use and external search. Together, the suite makes the transition from useful fine-tuning workflows to functioning agentic RL systems explicit.

We additionally summarize benchmark structure using four manually defined dimensions: capability breadth, interaction intensity, reward complexity, and decision horizon. These scores are intended only to summarize benchmark structure and should not be interpreted as precise measurements. A four-dimensional structural coverage visualization, together with the scoring rubric and exact task assignments, is provided in Appendix [A.1](#A1.SS1).

### 2.4 System Design

Each run creates an isolated workspace containing read-only links to the base model and training data, together with writable directories for generated code and model outputs. The benchmark exposes a grading server that evaluates submissions, returns scores, and records the best result within the run. This repeated-submission design is important because agentic RL typically depends on multiple cycles of training, diagnosis, and refinement rather than a single monolithic script.

The benchmark registry currently covers the six tasks described above. Static rule-based tasks are evaluated through OpenCompass-based pipelines. Judge-based tasks use task-specific evaluators. Interactive tasks rely on custom evaluators that support stateful environment interaction and rollout-based evaluation. The agent interface is fixed, while the evaluation backend varies with task structure. Figure [2](#S2.F2) illustrates the full benchmark pipeline.

Operationally, the benchmark enforces a common outer loop while preserving task-specific inner difficulty. Every run must solve the same high-level problem: inspect a workspace, choose a training route, generate code, train a model, and submit to a common grading interface. What varies across tasks is the training structure that must be handled correctly for those submissions to be meaningful.

### 2.5 Data Integrity and Evaluation

Test sets are not mounted into the agent workspace, providing structural protection against leakage. The grading server returns only a scalar score and best-so-far, limiting adaptive overfitting. Our primary metric is improvement over baseline ($\Delta$), with success rate ($\Delta>0$) as a complement. Process metrics (submission count, time to first improvement, valid submission rate) and offline route annotations are recorded for diagnostic analysis. Details on evaluation isolation safeguards are in Appendix [A.16](#A1.SS16).

### 2.6 Benchmark-Specific Details

#### Static rule-based tasks.

For GSM8K and HumanEval, the agent sees training data and task descriptions, while held-out evaluation is handled by the benchmark framework. Intended RL methods include single-turn methods such as GRPO or PPO over sampled completions, but the benchmark explicitly allows SFT as well. These tasks define the lower-structure end of the benchmark and test whether the agent can construct a useful post-training workflow under favorable conditions.

#### Static judge-based task.

AlpacaEval 2.0 differs from rule-based settings because reward is mediated by a judge rather than a deterministic checker. It tests whether agents can optimize against a preference-like signal while remaining in a non-interactive regime, thereby bridging fully programmable rewards and interactive RL. In our implementation, this task preserves the single-turn structure of AlpacaEval while integrating it into the same repeated-submission loop as the other tasks.

#### Interactive rollout tasks.

ALFWorld and WebShop require stateful multi-step interaction, while DeepSearchQA combines multi-step search with judge-based evaluation in a tool-augmented interactive loop. These tasks form the benchmark’s decisive stress test because they require an actual rollout-and-training system rather than a static fine-tuning script. More broadly, the three tasks expose different failure modes: sparse-reward long-horizon interaction in ALFWorld, dense but brittle task-specific reward in WebShop, and tool use plus judge-mediated search in DeepSearchQA.

#### Grading reliability.

Evaluation stochasticity differs across tasks. L1 tasks (GSM8K, HumanEval) use deterministic verifiers, so grading variance is zero for a fixed model checkpoint. AlpacaEval uses an LLM judge with moderate variance ($\pm$0.5–1pp across repeated evaluations). ALFWorld uses deterministic environment episodes (134 fixed episodes), producing stable scores for a given policy. WebShop evaluation over 100 fixed instructions exhibits low variance ($\pm$1pp). DeepSearchQA has the highest grading variance ($\pm$3pp), arising from stochasticity in both the LLM judge and the search-agent interaction; a seed sweep with 20 exact replications confirmed this spread (Section 3). Users of the benchmark should account for this task-dependent evaluation noise when interpreting small score differences, particularly on DeepSearchQA.

## 3 Experiments

### 3.1 Experimental Setup

We organize experiments into two complementary studies. The first is a scaffold comparison on Qwen2.5-7B-Instruct with a uniform 12h budget: OpenHands and OpenCode, both driven by GPT-5.4 (single run per task), and Claude Code driven by Claude Opus 4.6 (single run per task in four operating modes: SFT-only, RL-only, Free, Multi).

The second is a controlled CLI-agent study on Qwen3-8B-Base (unaligned), using a fixed 12h single-run budget: three CLI scaffolds (Claude Code, Codex CLI, Gemini CLI) paired with six driver LLMs (Claude Opus 4.6, Claude Sonnet 4.5, GPT-5.4, GPT-5.2, GPT-4o, gemini-2.5-flash), yielding seven system configurations. This partial factorial design enables within-scaffold driver comparisons that were not possible in the 7B study. Mode constraints in Claude Code are enforced via system-prompt instructions and verified through exhaustive post-hoc workspace audit (grep over all generated code files; results in Appendix [A.19](#A1.SS19.SSS0.Px5)).

Important caveat. The two studies are complementary but not equivalent. The 7B study compares agent systems end-to-end, with scaffold and driver entangled. The 8B study partially disentangles scaffold and driver via within-scaffold driver swaps (Codex CLI with three drivers; Claude Code with two drivers), but the design is not a full factorial—not every scaffold-driver combination is tested, and all 8B results are single runs. Observed effects should therefore be read as suggestive evidence rather than conclusive proofs of isolated scaffold or driver effects. The most causally clean finding remains the *mode comparison* (Section [3.4](#S3.SS4)), where scaffold and driver are held fixed across all conditions.

### 3.2 Cross-Agent Comparison

Table [2](#S3.T2) summarizes the main results.

**Table 2: Agent^2 RL-Bench leaderboard. All runs use a 12h budget. Panel A (7B-Instruct): scaffold comparison. OpenHands/OpenCode: single run (GPT-5.4 driver). Claude Code: best across operating modes (Opus 4.6 driver). Panel B (8B-Base): controlled single-run study on Qwen3-8B-Base across three CLI scaffolds and six driver LLMs. ^† AlpacaEval scores affected by intermittent Azure content-filter failures; use for qualitative trends only.**
|  |  | L1: Static Rule | L2: Judge | L3: Interactive Rollout |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| System | Driver | GSM8K | HumanEval | AlpacaEval^† | ALFWorld | WebShop | DeepSearchQA |
| Panel A: Qwen2.5-7B-Instruct (scaffold comparison, single run per system) |  |  |  |  |  |  |  |
| Baseline | — | 77.70 | 75.61 | 18.90 | 4.85 | 69.00 | 11.75 |
| OpenHands | GPT-5.4 | 81.24 | 78.05 | 18.77 | 9.41 | 40.00 | 7.80 |
| OpenCode | GPT-5.4 | 82.27 | 79.27 | 20.33 | 19.33 | 54.50 | 21.00 |
| Claude Code | Opus 4.6 | 84.00 | 81.71 | 30.76 | 95.52 | 84.00 | 15.00 |
| Panel B: Qwen3-8B-Base, unaligned (controlled 12h CLI study) |  |  |  |  |  |  |  |
| Baseline | — | 83.02 | 62.20 | 14.68 | 8.96 | 0.00 | 12.25 |
| Claude Code | Opus 4.6 | 88.78 | 84.15 | 26.91 | 97.76 | 67.00 | 15.00 |
| Claude Code | Sonnet 4.5 | 88.70 | 65.85 | 19.80 | 15.67 | 0.00 | 18.50 |
| Codex CLI | GPT-5.4 | 84.00 | 0.00 | 22.82 | 14.93 | 57.00 | 12.00 |
| Codex CLI | GPT-5.2 | 89.46 | 69.51 | 18.67 | 87.31 | 75.00 | 12.00 |
| Codex CLI | GPT-4o | 83.32 | 64.63 | 17.19 | 9.70 | 0.00 | 17.00 |
| Gemini CLI | 2.5-flash | 70.13 | 10.98 | 7.14 | 11.19 | 0.00 | 19.00 |

The two panels reveal complementary findings. In the 7B scaffold comparison (Panel A), Claude Code achieves the strongest results on 5 of 6 tasks under a uniform 12h budget, with particularly large gains on interactive tasks: +90.67 on ALFWorld and +15.00 on WebShop versus OpenCode’s +14.48 and $-$14.50, respectively. DeepSearchQA is the one task where OpenCode outperforms (+9.25 vs. +3.25), suggesting that the advantage is environment-dependent.

The controlled 8B study (Panel B) provides suggestive evidence that driver choice has a large effect on interactive tasks. Within Codex CLI, switching from GPT-4o to GPT-5.2 changes ALFWorld improvement from +0.74 to +78.35 using the same scaffold. Within Claude Code, Opus reaches ALFWorld 88.80pp improvement while Sonnet 4.5 reaches only 6.71pp. These are single-run results; the consistent direction across two scaffold families and the large magnitude strengthen the signal, but repeated experiments would be needed to confirm causality. The effect is *task-dependent*: on GSM8K, the six 8B stacks span a 19pp range (70.13–89.46), whereas on ALFWorld they span 89pp (9.70–97.76). Interactive tasks show far larger driver sensitivity than static tasks.

Three cross-cutting observations emerge. First, interactive success is not a single-system phenomenon: both Claude Code + Opus and Codex CLI + GPT-5.2 achieve strong ALFWorld/WebShop gains, while weaker stacks (Gemini CLI, Codex + GPT-4o, Claude Code + Sonnet on interactive tasks) collapse near or below baseline. Second, DeepSearchQA remains structurally hard across all seven 8B systems (best = 19.00, Gemini + 2.5-flash). Third, the weakest stack (Gemini CLI + 2.5-flash) *regresses* below baseline on GSM8K (70.13 vs. 83.02) and HumanEval (10.98 vs. 62.20), demonstrating that the benchmark has genuine discriminative power—poor stacks can actively harm the base model.

### 3.3 Behavioral Analysis: Capability Stratification

Workspace artifact inspection reveals qualitatively distinct agent strategies across the six tasks (detailed per-task analysis in Appendix [A.19](#A1.SS19); score trajectories in Appendix, Figure [5](#A1.F5)). The evidence suggests that the “capability boundary” between static and interactive tasks is better understood as *capability stratification*: different environments are at different stages of tractability, and the boundary shifts with scaffold quality.

#### Cross-agent behavioral differences.

For OpenCode-7B, 40–50% of runs end in copy-only or missing-code behavior, and interactive tasks amplify this weakness (Table [20](#A1.T20) in the Appendix). On WebShop, OpenCode-7B regressed by $-$14.50, while Claude Code achieved +24.00; this sign reversal confirms that trajectory collection quality, not algorithm choice, is the primary bottleneck. Figure [3](#S3.F3) shows that interactive tasks benefit most from extended time budgets, while pass@k analysis (Appendix) confirms that rare runs discover qualitatively better pipelines.

Figure: Figure 3: Scaling analysis across three dimensions for all agent stacks. Top: Time scaling (best-so-far score vs. elapsed hours). Middle: Token scaling (best-so-far score vs. cumulative context tokens processed, in millions). Bottom: Submission scaling (best-so-far score vs. number of submissions). 8B-Base controlled study (solid lines). Interactive tasks (ALFWorld, WebShop) show the sharpest system separation across all three dimensions, while static tasks plateau early.
Refer to caption: 2604.10547v1/x1.png

#### ALFWorld near-saturation.

On ALFWorld, the RL-only mode achieves 93.28 via SFT warm-up followed by GRPO with online rollouts—the cleanest demonstration of online RL succeeding in our benchmark. The Free mode reaches higher scores (95.52 at 12h, 8B-Base) through a composite supervised pipeline combining trajectory collection and training-evaluation format alignment, though this does not constitute online RL engineering in the strict sense (detailed trace in Appendix [A.23](#A1.SS23)).

#### DeepSearchQA structural intractability.

Despite 145 submissions and nearly every known alignment technique (DPO, GRPO, KTO, ORPO, model merging), the best score reached only 15.0 (+2.75 over baseline). A controlled follow-up experiment confirmed this ceiling: both Claude Code (+2.75) and Codex ($-$0.25) converge to $\sim$12–15 on Qwen3-8B-Base. Crucially, a seed sweep (20 exact replications) revealed $\pm$3pp evaluation variance, meaning the reported improvement falls within a single standard deviation of noise and should be treated as inconclusive. For context, even frontier agents with full web access achieve only 66% fully-correct on the official DeepSearchQA leaderboard (Google DeepMind, 2026).

#### SFT fallback as the dominant pattern.

Across all tasks, agents attempted RL methods first but fell back to SFT-based pipelines on 5 of 6 tasks. Static benchmark results are poor predictors of interactive performance: Claude Code’s ALFWorld improvement is 35$\times$ its GSM8K improvement.

### 3.4 Operating Mode Analysis

Since Claude Code is the only scaffold with explicit operating-mode control, we use it to study the effect of constraining the training paradigm. Table [16](#A1.T16) reports per-mode results on both 7B-Instruct (12h) and 8B-Base (12h, Multi vs. Free only).

**Table 3: Claude Code mode ablation. Top: Qwen2.5-7B-Instruct (12h, Opus 4.6). Bottom: Qwen3-8B-Base (12h, Opus 4.6, Free vs. Multi only). Bold = best, underline = worst per task.**
| Mode | GSM8K | HE | AE | ALF | WS | DSQA |
| --- | --- | --- | --- | --- | --- | --- |
| 7B-Instruct |  |  |  |  |  |  |
| SFT-only | 84.00 | 81.71 | 21.32 | 67.91 | 93.00 | 14.00 |
| RL-only | 81.88 | 81.71 | 20.96 | 93.28 | 79.00 | 15.00 |
| Free | 82.26 | 79.27 | 30.76 | 95.52 | 84.00 | 13.00 |
| Multi | 85.06 | 75.61 | 19.54 | 13.43 | 79.00 | 11.50 |
| 8B-Base |  |  |  |  |  |  |
| Free | 74.00 | 84.15 | 26.91 | 97.76 | 67.00 | 15.00 |
| Multi | 80.97 | 60.98 | 10.33 | 85.82 | 13.00 | 7.00 |

The rank ordering of modes *reverses* between L1 and L3 tasks: SFT is near-best for GSM8K but worst for ALFWorld (67.91), while RL-only achieves 93.28 on ALFWorld but is worst on GSM8K. Free mode produces the strongest results on interactive tasks through composite strategies unavailable under single-paradigm constraints. Multi-task joint optimization wins only on GSM8K but severely degrades interactive tasks—a pattern that holds on both 7B-Instruct and 8B-Base. The best-to-worst mode spread exceeds 80 points on ALFWorld, demonstrating that operating mode is a major hidden variable in benchmark reporting.

### 3.5 Base Model Results

Panel B of Table [2](#S3.T2) shows that alignment-gap closure from an unaligned starting point is strongly stack-dependent: the strongest stacks (CC+Opus 4.6, Codex+GPT-5.2) reach near-Instruct or above-Instruct performance on multiple tasks, while the weakest (Gemini+flash) regresses below baseline on GSM8K ($-$12.89) and HumanEval ($-$51.22). Unaligned starting points amplify stack quality differences compared to the 7B-Instruct setting (full analysis in Appendix).

## 4 Related Work

One way to position prior work is as a progression in structural difficulty for agent evaluation. Reasoning and coding benchmarks such as HumanEval (Chen et al., 2021) primarily test localized problem solving. Software engineering benchmarks such as SWE-bench (Jimenez et al., 2023) move to long-context debugging, retrieval, and multi-file coordination. Broader assistant and tool-use benchmarks such as AgentBench (Liu et al., 2023), GAIA (Mialon et al., 2023), and WebArena (Zhou et al., 2023) further expand tool use and interaction. Machine-learning automation benchmarks such as MLAgentBench (Huang et al., 2024), MLE-bench (Chan et al., 2024), and RE-Bench (Wijk et al., 2024) then ask whether agents can execute end-to-end ML workflows.

The closest prior benchmark is PostTrainBench (Rank et al., 2026), which focuses directly on automating LLM post-training. That benchmark establishes an important milestone: agents can sometimes improve base models on bounded post-training tasks. Our work targets the next structural step. Whereas prior post-training benchmarks remain largely static, Agent^2 RL-Bench asks whether an agent can succeed once rollout, trajectory handling, and online data collection become unavoidable. We do not include scripted or expert-designed pipeline baselines in the main study, because the benchmark is designed to measure what agents discover autonomously rather than how close they come to a known-good recipe; this is a deliberate scope choice rather than an oversight. In this sense, our contribution is not a new RL post-training algorithm, but an evaluation setting that isolates the transition from static post-training automation to online, stateful, long-horizon RL engineering.

This work is also motivated by the importance of RL-centered post-training for language models, as exemplified by systems such as InstructGPT (Ouyang et al., 2022) and DeepSeek-R1 (DeepSeek-AI, 2025). On the scaffolding side, our experiments cover three distinct scaffold families: OpenHands-style software agents (Wang et al., 2024), an OpenCode variant, and CLI-native agents (Claude Code, Codex CLI, Gemini CLI). The controlled 8B study pairs these scaffolds with multiple driver LLMs, partially disentangling scaffold and driver effects. Our question is complementary to both lines of work: not how to improve RL post-training algorithms themselves, nor how to build a better general software agent in isolation, but whether an external LLM agent can autonomously engineer such pipelines.

## 5 Conclusion

We introduced Agent^2 RL-Bench, a benchmark for evaluating LLM agents on agentic RL post-training, organized around three levels of training structure coverage.

Our experiments across multiple agent stacks reveal five observations. First, interactive transfer is real but non-uniform: on ALFWorld, an RL-only agent achieves 93.28 via GRPO with online rollouts, while DeepSearchQA remains hard across all systems. Second, operating mode is a major hidden variable: the best-to-worst mode spread exceeds 80 points on ALFWorld, arguing that benchmark reporting must specify the full model-agent-mode stack. Third, the benchmark reveals that agents more consistently engineer supervised pipelines than stable online RL systems: RL succeeded as the final best pipeline only on ALFWorld (93.28 via GRPO). Fourth, driver choice appears to have a large effect on interactive tasks: within fixed scaffolds, switching drivers changes ALFWorld improvement from near-zero to +78pp, while static tasks show modest sensitivity (single-run evidence; repeated experiments would strengthen this conclusion). Fifth, alignment-gap closure from an unaligned base model depends on the full stack: the strongest combinations (Claude Code + Opus 4.6, Codex + GPT-5.2) approach Instruct-level performance while weaker stacks regress below baseline.

Three limitations are especially important: the cross-agent comparison is partially disentangled by the 8B controlled study but remains non-factorial (not every scaffold-driver combination is tested, and all 8B rows are single runs without confidence intervals); AlpacaEval evaluation reliability was affected by intermittent Azure content-filter failures ($<$4% of judge calls); and the best ALFWorld result relies on a supervised composite pipeline rather than pure online RL (RL-only mode achieves 93.28 independently).

We view Agent^2 RL-Bench as an information-efficient probe of an emerging capability frontier. Its value lies in revealing *how* agents approach RL engineering—and where they avoid it.

## References

- J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. Mądry (2024)
MLE-bench: evaluating machine learning agents on machine learning engineering.
arXiv preprint arXiv:2410.07095.
External Links: [Link](https://arxiv.org/abs/2410.07095)
Cited by: [§1](#S1.p3.1),
[§4](#S4.p1.1).
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021)
Evaluating large language models trained on code.
arXiv preprint arXiv:2107.03374.
External Links: [Link](https://arxiv.org/abs/2107.03374)
Cited by: [§1](#S1.p3.1),
[§2.3](#S2.SS3.p1.1),
[§4](#S4.p1.1).
- K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021)
Training verifiers to solve math word problems.
arXiv preprint arXiv:2110.14168.
External Links: [Link](https://arxiv.org/abs/2110.14168)
Cited by: [§2.3](#S2.SS3.p1.1).
- DeepSeek-AI (2025)
DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning.
arXiv preprint arXiv:2501.12948.
External Links: [Link](https://arxiv.org/abs/2501.12948)
Cited by: [§1](#S1.p1.1),
[§4](#S4.p3.1).
- Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto (2024)
Length-controlled alpacaeval: a simple way to debias automatic evaluators.
arXiv preprint arXiv:2404.04475.
External Links: [Link](https://arxiv.org/abs/2404.04475)
Cited by: [§2.3](#S2.SS3.p1.1).
- Google DeepMind (2026)
DeepSearchQA: bridging the comprehensiveness gap for deep research agents.
arXiv preprint arXiv:2601.20975.
External Links: [Link](https://arxiv.org/abs/2601.20975)
Cited by: [§2.3](#S2.SS3.p1.1),
[§3.3](#S3.SS3.SSS0.Px3.p1.3).
- Q. Huang, J. Vora, P. Liang, and J. Leskovec (2024)
MLAgentBench: evaluating language agents on machine learning experimentation.
In International Conference on Machine Learning (ICML),
External Links: [Link](https://openreview.net/forum?id=1Fs1LvjYQW)
Cited by: [§4](#S4.p1.1).
- C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2023)
SWE-bench: can language models resolve real-world GitHub issues?.
arXiv preprint arXiv:2310.06770.
External Links: [Link](https://arxiv.org/abs/2310.06770)
Cited by: [§1](#S1.p3.1),
[§4](#S4.p1.1).
- X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang (2023)
AgentBench: evaluating LLMs as agents.
arXiv preprint arXiv:2308.03688.
External Links: [Link](https://arxiv.org/abs/2308.03688)
Cited by: [§4](#S4.p1.1).
- G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom (2023)
GAIA: a benchmark for general AI assistants.
arXiv preprint arXiv:2311.12983.
External Links: [Link](https://arxiv.org/abs/2311.12983)
Cited by: [§4](#S4.p1.1).
- L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022)
Training language models to follow instructions with human feedback.
arXiv preprint arXiv:2203.02155.
External Links: [Link](https://arxiv.org/abs/2203.02155)
Cited by: [§1](#S1.p1.1),
[§4](#S4.p3.1).
- B. Rank, H. Bhatnagar, A. Prabhu, S. Eisenberg, K. Nguyen, M. Bethge, and M. Andriushchenko (2026)
PostTrainBench: can LLM agents automate LLM post-training?.
arXiv preprint arXiv:2603.08640.
External Links: [Link](https://arxiv.org/abs/2603.08640)
Cited by: [§1](#S1.p3.1),
[§4](#S4.p2.1).
- M. Shridhar, X. Yuan, M. Côté, Y. Bisk, M. Hausknecht, J. Thomason, R. Fergus, J. Pineau, and D. Gordon (2020)
ALFWorld: aligning text and embodied environments for interactive learning.
arXiv preprint arXiv:2010.03768.
External Links: [Link](https://arxiv.org/abs/2010.03768)
Cited by: [§2.3](#S2.SS3.p1.1).
- X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig (2024)
OpenHands: an open platform for AI software developers as generalist agents.
External Links: [Link](https://arxiv.org/abs/2407.16741)
Cited by: [§4](#S4.p3.1).
- H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes (2024)
RE-Bench: evaluating frontier AI R&D capabilities of language model agents against human experts.
arXiv preprint arXiv:2411.15114.
External Links: [Link](https://arxiv.org/abs/2411.15114)
Cited by: [§4](#S4.p1.1).
- S. Yao, H. Chen, J. Yang, and K. Narasimhan (2022)
WebShop: towards scalable real-world web interaction with grounded language agents.
arXiv preprint arXiv:2207.01206.
External Links: [Link](https://arxiv.org/abs/2207.01206)
Cited by: [§2.3](#S2.SS3.p1.1).
- S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023)
ReAct: synergizing reasoning and acting in language models.
In International Conference on Learning Representations (ICLR),
External Links: [Link](https://openreview.net/forum?id=WE_vluYUL-X)
Cited by: [§2.3](#S2.SS3.p1.1).
- S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, Y. Bisk, D. Fried, U. Alon, and G. Neubig (2023)
WebArena: a realistic web environment for building autonomous agents.
arXiv preprint arXiv:2307.13854.
External Links: [Link](https://arxiv.org/abs/2307.13854)
Cited by: [§4](#S4.p1.1).

## Appendix A Appendix: Protocol and Supplementary Analysis

### A.1 Structural Coverage Rubric

Figure [4](#A1.F4) is a benchmark-design visualization rather than an empirical result. Each benchmark is assigned an ordinal score from 1 to 5 on four manually defined structural dimensions. These scores are intended only to summarize structural differences between tasks and to make the benchmark coverage visually explicit.

Figure: Figure 4: Coverage of the six benchmark tasks across four manually defined structural dimensions: capability breadth, interaction intensity, reward complexity, and decision horizon. Scores are ordinal (1–5) and summarize benchmark-level metadata rather than empirical results.
Refer to caption: 2604.10547v1/x2.png

#### Dimension definitions.

- •
Capability breadth: how many qualitatively different abilities must be combined to solve the task. A score of 1 indicates a narrow, mostly single-capability task; a score of 5 indicates a task requiring broad coupling of reasoning, planning, search, tool use, or state tracking.
- •
Interaction intensity: how strongly the task depends on explicit multi-step interaction with a stateful environment or iterative tool loop. A score of 1 indicates a static single-shot task; a score of 5 indicates strong dependence on repeated environment interaction.
- •
Reward complexity: how difficult the reward signal is to specify or optimize against. Rule-based exact-match rewards are low; dense or judge-based rewards are higher.
- •
Decision horizon: the effective length of the action or reasoning sequence needed to complete the task. Single-turn generation is low; long multi-step interaction is high.

**Table 4: Ordinal benchmark-design scores used in Figure [4](#A1.F4).**
| Benchmark | Capability | Interaction | Reward | Horizon |
| --- | --- | --- | --- | --- |
| GSM8K | 2 | 1 | 1 | 1 |
| HumanEval | 2 | 1 | 1 | 1 |
| AlpacaEval | 2 | 1 | 4 | 1 |
| ALFWorld | 4 | 5 | 2 | 5 |
| WebShop | 4 | 5 | 5 | 5 |
| DeepSearchQA | 5 | 4 | 4 | 3 |

### A.2 Benchmark Protocol Details

Table [5](#A1.T5) summarizes the task-level evaluation contracts. The benchmark is unified at the outer loop, but the inner evaluator, reward source, and horizon differ substantially across tasks.

**Table 5: Task protocol summary following the current implementation.**
| Task | Structure | Evaluator Type | Reward Source | Eval Size | Horizon | Primary Difficulty |
| --- | --- | --- | --- | --- | --- | --- |
| GSM8K | Static rule-based | OpenCompass | Exact-match answer check | full split | Short | reasoning-to-training transfer |
| HumanEval | Static rule-based | OpenCompass | unit-test execution | held-out subset | Short | code-generation post-training |
| AlpacaEval 2.0 | Static judge-based | LLM judge | pairwise preference judging | reference split | Short | non-programmable reward |
| ALFWorld | Interactive rollout | environment evaluator | sparse success reward | 134 episodes | Long | rollout and trajectory collection |
| WebShop | Interactive rollout | environment evaluator | task-specific dense reward | 100 instructions | Long | reward shaping and stability |
| DeepSearchQA | Interactive tool-use | custom ReAct evaluator | search-and-judge loop | 200 samples | Medium | tool use, search, and judged QA |

#### Static rule-based tasks.

For GSM8K and HumanEval, the agent receives training data and task descriptions, while held-out evaluation is handled by the benchmark framework. These tasks permit SFT and single-turn RL methods and anchor the low-structure end of the suite.

#### Static judge-based tasks.

AlpacaEval 2.0 occupies a middle point between static verification and interactive RL. The task remains single-turn, but the reward is supplied by an LLM judge rather than a deterministic checker.

#### Interactive rollout tasks.

ALFWorld and WebShop require explicit interaction loops with stateful environments. DeepSearchQA is implemented as a tool-augmented interactive QA task built on the google/deepsearchqa dataset, using a custom ReAct-style search-and-judge evaluator rather than the original paper’s full protocol. Across all three tasks, useful improvement requires a functioning online loop rather than a static fine-tuning script alone.

### A.3 Reported Metrics

We distinguish *benchmark-native outputs* from *offline diagnostics*. This distinction is important for interpreting the main results.

**Table 6: Reported metric types in Agent^2 RL-Bench.**
| Metric Type | Definition |
| --- | --- |
| Benchmark-native outputs | baseline score, per-submission score, best score, improvement over baseline, submission count, runtime metadata |
| Process metrics | valid-submission rate, time to first improvement, time to best, score trajectory, time-used ratio, last-submit gap |
| Offline diagnostics | route labels, coding-success indicators, case-study inspection, trace-based interpretation |

The benchmark-native outputs are produced directly by the grading-server loop and are the basis of the main quantitative claims. Process metrics are derived deterministically from saved run artifacts such as scores.json and run metadata. Offline diagnostics are used only for interpretation and should not be confused with the benchmark’s native scoring interface.

#### AlpacaEval evaluation reliability.

AlpacaEval 2.0 evaluation uses a GPT-4o judge hosted on Azure OpenAI. During agent post-training runs, intermittent Azure content-filter failures caused $<$4% of judge calls to be skipped. Baseline evaluation completed without errors (805/805 judge calls succeeded), but agent-generated outputs occasionally triggered content-filter rejections. Skipped calls receive a dummy “loss” label, which biases win rates slightly downward for affected systems. Because the failure rate is small and approximately uniform across systems, we retain AlpacaEval scores in the results tables for qualitative trend comparison but mark the column with ^† and exclude it from any ranking or average computation. All AlpacaEval columns in this paper should be interpreted with this caveat.

### A.4 Experimental Setup Table

**Table 7: Experimental setup used for the main experiments.**
| Dimension | Configuration |
| --- | --- |
| 7B Repeated-Run Scaffold Comparison |  |
| Agents | OpenHands, OpenCode, Claude Code |
| Driver models | GPT-5.4 (OH/OC), Claude Opus 4.6 (CC) |
| Base model | Qwen2.5-7B-Instruct |
| Time budget | 12h$\times$10 runs (OH/OC); 24h single (CC) |
| CC modes | SFT-only, RL-only, Free, Multi |
| 8B Controlled CLI Study |  |
| Scaffolds | Claude Code, Codex CLI, Gemini CLI |
| Driver models | Opus 4.6, Sonnet 4.5, GPT-5.4, GPT-5.2, GPT-4o, 2.5-flash |
| Base model | Qwen3-8B-Base (unaligned) |
| Time budget | 12h single run per system |
| Shared Evaluation |  |
| ALFWorld eval split | 134 episodes |
| WebShop eval size | 100 instructions |
| DeepSearchQA eval size | 200 held-out samples |
| Judge model | GPT-4o |

### A.5 Run Contract

Each run in Agent^2 RL-Bench is defined by the tuple *(agent scaffold, task, base model, time budget)*. The outer-loop contract is deliberately simple. The runner prepares an isolated workspace, mounts the task-specific data and model artifacts, starts a grading server, computes the baseline score for the provided base model, and then grants the agent autonomous control within the workspace.

At execution time, the agent is given access to a small set of standard environment variables, including the task identifier, the base-model path, the workspace path, the output directory, and the grading-server endpoint GRADING_SERVER_URL. Submissions are made by posting a model path back to the grading server, which evaluates the submission, appends the result to scores.json, and returns the current score together with the best score observed so far.

#### Practical constraints.

The run contract also includes a bounded engineering budget. Agents are not evaluated in an interactive chat setting with human feedback; instead, they are expected to operate autonomously until timeout or natural termination. In the current implementation, some scaffolds are additionally given a lightweight timer.sh utility so they can query the remaining time budget during long runs.

### A.6 Agent Instruction Excerpt

The benchmark also provides an agent-facing instruction file (core/instructions.md) that standardizes what every scaffold is told about the workspace, timing signals, and submission API. We do not reproduce the full file verbatim here; instead, we include the operative excerpt that defines the contract most relevant to reproducibility and auditability.

### A.7 Structured Run Reports

In addition to benchmark-native outputs such as scores.json and run metadata, each run maintains two structured reporting artifacts under reports/. The first is summary.md, a human-readable cumulative report intended for manual inspection. The second is summary.jsonl, a compact machine-readable log with one record per iteration.

#### Human-readable summaries.

The summary.md file is append-only and is designed to make long agent runs auditable without replaying the full terminal trace. Each iteration records the run status, score and improvement, inferred training type, key configuration choices, a short explanation of what was attempted, the rationale for that attempt, the main failure mode or progress signal, and a small code excerpt from the generated training script when available. In practice, this file is the main artifact we consult when reconstructing why a run failed, why a route changed, or why a particular submission improved over baseline.

#### Machine-readable iteration logs.

The companion summary.jsonl file stores a minimal quantitative record for each iteration, including fields such as iteration index, timestamp, duration, status, exit code, score, improvement, training type, and failure type. We use this file for lightweight downstream aggregation, while leaving richer causal interpretation to summary.md and the saved workspace itself.

#### Role in the paper.

These run reports are not part of the benchmark-native scoring interface and are never used to define the main leaderboard-style claims. Instead, they serve as auxiliary instrumentation for the offline analysis reported in Section 3, especially for route analysis, failure taxonomy, and case-study reconstruction. This separation is deliberate: the benchmark score should come only from the evaluator, whereas the run reports help us interpret how the agent reached that score.

### A.8 Judge Prompt Excerpts

Judge-based tasks are important in Agent^2 RL-Bench because they make the reward signal less programmable than exact-match or unit-test metrics. We therefore include representative prompt excerpts here to clarify the form of supervision used in those tasks.

#### AlpacaEval-style preference judging.

When the built-in AlpacaEval pipeline is unavailable or falls back to the project-level API backend, the judge compares two candidate answers to the same instruction and returns one of A, B, or TIE. The prompt is instantiated from the following template:

#### DeepSearchQA answer judging.

DeepSearchQA uses a task-specific answer judge rather than a pairwise preference annotator. The judge receives the answer type, the gold answer, and the predicted answer, and is instructed to return only correct or incorrect. The current implementation uses the following prompt structure:

These excerpts are not the only ingredients in the full evaluation pipeline, but they illustrate the main difference between the two judge-based settings: AlpacaEval relies on preference comparison between two candidate responses, whereas DeepSearchQA relies on answer verification conditioned on a gold target and answer type.

### A.9 Complete Result Matrix

Table [8](#A1.T8) provides the extended result matrix used to support the main text.

**Table 8: Extended result matrix. Mean $\Delta$ denotes improvement over baseline. Improve SR is the fraction of runs with strictly positive improvement; Exec SR is the fraction of runs that complete at least one valid training-evaluation-submission cycle.**
| Agent | Task | Model | Mean $\Delta$ | Std | Max $\Delta$ | Improve SR | Exec SR |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OpenHands | GSM8K | 1.5B | +0.05 | 0.14 | +0.22 | 40.0% | 58.0% |
| OpenHands | HumanEval | 1.5B | +0.80 | 1.10 | +2.10 | 30.0% | 45.0% |
| OpenHands | AlpacaEval | 1.5B | +0.20 | 0.28 | +0.55 | 20.0% | 38.0% |
| OpenHands | ALFWorld | 1.5B | +0.00 | 0.35 | +1.00 | 5.0% | 22.0% |
| OpenHands | WebShop | 1.5B | +0.00 | 0.00 | +0.00 | 0.0% | 12.0% |
| OpenHands | DeepSearchQA | 1.5B | -1.50 | 0.60 | +0.00 | 0.0% | 26.0% |
| OpenHands | GSM8K | 7B | +0.12 | 0.29 | +0.53 | 56.0% | 72.0% |
| OpenHands | HumanEval | 7B | +1.22 | 1.73 | +3.66 | 40.0% | 58.0% |
| OpenHands | AlpacaEval | 7B | +0.30 | 0.42 | +0.95 | 35.0% | 52.0% |
| OpenHands | ALFWorld | 7B | +1.20 | 4.80 | +12.50 | 30.0% | 48.0% |
| OpenHands | WebShop | 7B | -30.00 | 28.00 | +5.00 | 10.0% | 46.0% |
| OpenHands | DeepSearchQA | 7B | -3.70 | 3.40 | +2.70 | 20.0% | 44.0% |
| OpenCode | GSM8K | 1.5B | +0.08 | 0.18 | +0.31 | 45.0% | 62.0% |
| OpenCode | HumanEval | 1.5B | +1.10 | 1.35 | +2.70 | 40.0% | 52.0% |
| OpenCode | AlpacaEval | 1.5B | +0.50 | 0.46 | +1.10 | 50.0% | 64.0% |
| OpenCode | ALFWorld | 1.5B | +0.00 | 1.20 | +4.00 | 20.0% | 40.0% |
| OpenCode | WebShop | 1.5B | +0.00 | 2.10 | +3.50 | 10.0% | 36.0% |
| OpenCode | DeepSearchQA | 1.5B | -2.30 | 2.20 | +1.00 | 15.0% | 34.0% |
| OpenCode | GSM8K | 7B | +1.15 | 0.85 | +2.70 | 60.0% | 78.0% |
| OpenCode | HumanEval | 7B | +2.44 | 1.50 | +4.80 | 75.0% | 84.0% |
| OpenCode | AlpacaEval | 7B | +1.86 | 0.90 | +3.20 | 60.0% | 76.0% |
| OpenCode | ALFWorld | 7B | +11.12 | 16.50 | +38.00 | 55.0% | 70.0% |
| OpenCode | WebShop | 7B | -15.50 | 19.80 | +6.00 | 30.0% | 68.0% |
| OpenCode | DeepSearchQA | 7B | +9.50 | 18.00 | +39.50 | 25.0% | 61.0% |

### A.10 Supporting Scaling Tables

**Table 9: Time-budget scaling. For OpenHands/OpenCode, entries are from the 7B setting. For Claude Code, @12h and @24h compare the same Free-mode run at the 12h mark versus final (24h).**
| Task | Agent | @6h | @12h | @24h |
| --- | --- | --- | --- | --- |
| GSM8K | OpenHands | +0.10 | +0.12 | — |
| GSM8K | OpenCode | +0.95 | +1.15 | — |
| GSM8K | Claude Code | +1.89 | +4.56 | +5.70 |
| HumanEval | OpenHands | +1.05 | +1.22 | — |
| HumanEval | OpenCode | +2.05 | +2.44 | — |
| HumanEval | Claude Code | +3.66 | +3.66 | +10.98 |
| AlpacaEval | OpenHands | +0.22 | +0.30 | — |
| AlpacaEval | OpenCode | +1.55 | +1.86 | — |
| AlpacaEval | Claude Code | +11.05 | +11.86 | +14.29 |
| ALFWorld | OpenHands | +0.60 | +1.20 | — |
| ALFWorld | OpenCode | +7.80 | +11.12 | — |
| ALFWorld | Claude Code | +76.86 | +90.67 | +91.42 |
| WebShop | OpenHands | -35.00 | -30.00 | — |
| WebShop | OpenCode | -21.50 | -15.50 | — |
| WebShop | Claude Code | +4.00 | +15.00 | +19.00 |
| DeepSearchQA | OpenHands | -3.70 | -3.70 | — |
| DeepSearchQA | OpenCode | +6.00 | +9.50 | — |
| DeepSearchQA | Claude Code | +1.50 | +1.25 | +2.75 |

**Table 10: pass@k expected best improvement for representative 7B settings at $k\in\{1,3,5\}$.**
| Task | Agent | $k=1$ | $k=3$ | $k=5$ |
| --- | --- | --- | --- | --- |
| GSM8K | OpenHands | +0.12 | +0.17 | +0.19 |
| GSM8K | OpenCode | +1.15 | +1.40 | +1.50 |
| HumanEval | OpenHands | +1.22 | +1.55 | +1.68 |
| HumanEval | OpenCode | +2.44 | +2.80 | +2.95 |
| AlpacaEval | OpenHands | +0.30 | +0.42 | +0.48 |
| AlpacaEval | OpenCode | +1.86 | +2.10 | +2.20 |
| ALFWorld | OpenHands | +1.20 | +1.85 | +2.10 |
| ALFWorld | OpenCode | +11.12 | +13.80 | +14.90 |
| WebShop | OpenHands | -30.00 | -26.00 | -24.50 |
| WebShop | OpenCode | -15.50 | -13.20 | -12.50 |
| DeepSearchQA | OpenHands | -3.70 | -3.10 | -2.70 |
| DeepSearchQA | OpenCode | +9.50 | +12.00 | +13.10 |

### A.11 Route-Analysis Definitions

The technical-route analysis in the main text is based on the generated train.py script in each saved workspace. Table [11](#A1.T11) summarizes the labels used in that analysis.

**Table 11: Route labels used in the technical-route analysis.**
| Route | Operational definition |
| --- | --- |
| GRPO | train.py contains explicit GRPO-style training components |
| PPO | train.py contains explicit PPO-style training components |
| SFT-only | train.py contains supervised fine-tuning components but no explicit RL trainer |
| SFT+RL | train.py contains both supervised and RL-style training components |
| Copy | the run copies or reuses the base model without substantive training logic |
| Placeholder / Missing | the training script is absent or effectively a placeholder rather than executable training code |
| Coding success | a run that produces substantive training code, operationalized as GRPO, PPO, SFT-only, or SFT+RL |
| Improvement success | a run with at least one submission whose improvement over baseline is strictly positive |

This analysis is intentionally operational rather than semantic. Its purpose is not to recover the full intent of every generated script, but to separate runs that genuinely attempt training from runs that fail earlier and only produce superficial artifacts.

### A.12 Supplementary Case Studies

#### OpenCode on ALFWorld.

A representative positive ALFWorld case follows a three-stage structure: offline expert SFT, online beta-mixed DAgger collection, and aggregated SFT on the resulting data. The run first submits a model below baseline and only later recovers, which makes it a useful example of late route discovery rather than steady incremental improvement. We therefore interpret it primarily as route-discovery evidence. To make the route auditable without relying on the exact historical workspace artifact, we also retain a cleaned reproduction variant of the same route shape and summarize its key components in Appendix [A.13](#A1.SS13).

#### WebShop collapse.

WebShop illustrates a different failure mode: a run may generate substantial code and still fail to produce useful learning signal. In the representative negative case, the training script implements oracle bootstrap trajectories, relaxed DAgger collection, explicit step-level reward shaping, and advantage-weighted behavioral cloning, yet the run still fails to improve over baseline and ultimately terminates with an engine-initialization error on a later submission. This case supports the distinction between “code exists” and “useful RL training happened.” In interactive tasks, reward handling, trajectory quality, and training stability are part of the problem itself rather than secondary implementation details.

### A.13 Representative Code Evidence for RL and Hybrid Routes

The following excerpts are included only as code-level evidence for route shape. They are not benchmark-native metrics, and they should not be interpreted as replacing the quantitative results in the main paper.

#### GSM8K: explicit GRPO route.

One representative GSM8K run uses an explicit GRPO trainer with exact-match reward over extracted final answers:

This excerpt is useful because it shows that at least some static-task runs are not merely SFT pipelines: the agent can instantiate a genuine RL-style route when the reward is cleanly programmable.

#### ALFWorld: cleaned DAgger-style route.

To make the discovered ALFWorld route auditable without depending on a task-specific prompt file, we distilled the route into a cleaned reproduction variant. The core mechanics remain expert imitation, beta-mixed online collection, shaped transition scoring, and replay-based retraining:

These fragments make clear that the route is not plain SFT. It combines expert warm-starting, online collection under a decaying teacher policy, reward-shaped filtering, and replay-weighted retraining.

#### WebShop: substantial interactive engineering without gain.

The representative WebShop failure is also structurally informative. The generated code is not a placeholder; it contains oracle trajectory bootstrap, DAgger-style collection, and weighted behavioral cloning:

This negative case is useful because it shows that interactive failure often occurs after substantial engineering effort. The bottleneck is not simply whether code is produced, but whether the resulting reward, data, and training loop produce a stable learning signal.

#### DeepSearchQA: tool-augmented search trajectory SFT.

A representative DeepSearchQA route is not PPO-style RL, but it is still structurally informative because it constructs explicit search trajectories rather than plain single-turn answers:

This route is useful in the appendix because it shows that some interactive settings are approached through tool-augmented trajectory construction plus SFT, rather than through explicit policy-gradient RL.

### A.14 Operational Safeguards and Reproducibility

The benchmark includes several implementation choices intended to make repeated evaluation stable and auditable. Each run is assigned an isolated workspace; submissions are recorded through a grading server rather than by directly overwriting a leaderboard file; and evaluation is serialized within a run to avoid multiple heavyweight evaluators contending for the same GPU simultaneously. The current implementation also caches successful evaluations by resolved path and latest model-file modification time, so that identical submissions are not re-evaluated unnecessarily.

We also explicitly constrain reported evidence to artifacts that can be reconstructed from saved workspaces. Benchmark-native outputs are derived from the grading-server loop, while supplemental process metrics are computed from stored score trajectories and run metadata. The remaining open implementation details now concern only the final behavior-trace visualization.

### A.15 Reproducibility and Environment Note

The benchmark code, task registry, evaluator implementations, and agent-facing protocol files are maintained in the project repository and are sufficient to reproduce the benchmark logic reported in this paper. The main experimental results additionally depend on infrastructure choices that are not benchmark-specific, including API access to the driver model, local model hosting for the target models, and the cluster environment used to execute long-running agent jobs.

#### Benchmark-facing environment.

The benchmark itself assumes a Linux execution environment with Python, PyTorch/Transformers/TRL dependencies, benchmark-specific task packages, and a grading server reachable through a local HTTP endpoint. The main experiments use a 12-hour time budget per run and repeated submissions within that budget. Hardware details matter mainly for runtime and throughput rather than for the definition of the benchmark contract itself.

#### What is and is not claimed.

We treat the benchmark implementation, saved workspaces, and grading artifacts as the reproducible core. We do not make quantitative token-cost or dollar-cost claims in this version, and we do not claim that cluster orchestration details are necessary to reproduce the benchmark’s qualitative conclusions. Where exact infrastructure scripts are environment-specific, we instead document the benchmark contract, the task protocol, and representative run commands in the released codebase.

### A.16 Audit and Anti-Cheating Rules

Because Agent^2 RL-Bench evaluates autonomous agents with repeated submissions, the integrity rules need to be explicit. Table [12](#A1.T12) summarizes the safeguards that are currently implemented and auditable in the released system.

**Table 12: Implemented audit and anti-cheating rules.**
| Rule category | Implemented behavior |
| --- | --- |
| Test-set isolation | Held-out evaluation data are not mounted into the agent workspace. Agents only receive training-visible data and task descriptions. |
| Submission interface | Scores are produced only through the grading server. Agents cannot directly overwrite leaderboard files or final result tables. |
| Baseline reuse | Trivial submission of the untouched base model is treated as non-improving behavior and maps back to baseline performance. |
| Artifact logging | Each run stores score trajectories, run metadata, and workspace artifacts for later inspection. |
| Evaluator serialization | Heavyweight evaluation is serialized within a run to reduce contention and make score traces easier to audit. |
| Path-based caching | Identical submissions may be cached by resolved path and file-modification state to avoid repeated redundant evaluation, without changing the best-score semantics. |

These safeguards are intended to reduce straightforward leakage or score inflation, not to claim perfect adversarial security. We therefore restrict the main paper’s integrity claims to mechanisms that are already present in the implementation and can be inspected from the released code and artifacts.

### A.17 Failure Taxonomy Summary

The main paper argues that the interactive boundary is exposed not by a single failure mode, but by several recurrent breakdowns in the online loop. Table [13](#A1.T13) summarizes the dominant categories we repeatedly observed when auditing saved workspaces and score traces.

**Table 13: Failure taxonomy used for offline diagnosis of interactive-task runs.**
| Failure type | Typical task | Observable symptom | Main audit artifact |
| --- | --- | --- | --- |
| Rollout construction failure | ALFWorld, WebShop | The run produces training code, but the environment loop is incomplete, brittle, or never yields a usable trajectory set. | train.py, task-specific eval.py, missing or degenerate submissions |
| Reward/data design failure | WebShop, DeepSearchQA | The run implements non-trivial collection and training logic, yet the resulting data or reward signal is too noisy or misaligned to improve performance. | Generated training script, score trajectory, failure notes in summary.md |
| Training collapse / instability | WebShop, ALFWorld | Early submissions exist, but later iterations regress, crash, or fail during model loading, engine startup, or evaluation. | scores.json, run logs, final workspace artifacts |
| Early coding failure | All tasks, especially weaker-model settings | The run terminates before producing substantive training code or a valid submission. | Missing or placeholder train.py, zero-submission runs |
| Static shortcut without transfer | GSM8K, HumanEval versus L3 tasks | The scaffold can assemble a workable static SFT/RL route, but the same competence does not carry over once interactive rollout becomes mandatory. | Cross-task comparison of route labels, submissions, and improvements |

This taxonomy is intentionally diagnostic rather than benchmark-native. It is used to interpret why interactive tasks behave differently from static ones, not to redefine the benchmark score itself.

### A.18 Claude Code Agent Behavior Statistics

Table [14](#A1.T14) summarizes the workspace-level statistics for the best-performing Claude Code workspace per task. These numbers are derived from saved artifacts (scores.json, code/, output/) and are not benchmark-native metrics.

**Table 14: Claude Code workspace statistics per task (best workspace per task). Submissions = total valid grading requests; Code files = distinct .py files written under code/; Model versions = directories under output/; Zero rate = fraction of submissions scoring 0 (training/evaluation crash); Best@Sub = submission number at which the best score was achieved.**
| Task | Mode | Subs | Code | Models | Zero% | Best@Sub | Primary Strategies |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GSM8K | SFT | 92 | 14 | 146 | 20% | #90 | SFT, model soup, seed sweep |
| HumanEval | SFT | 195 | 28 | 336 | 23% | #26 | SFT, model averaging, task arithmetic |
| AlpacaEval | Free | 23 | 11 | 47 | 0% | #23 | SFT $\to$ DPO, synthetic preference gen. |
| ALFWorld | Free | 4 | 4 | 5 | 0% | #4 | Demonstration SFT (env. interaction) |
| WebShop | Free | 16 | 4 | 20 | 19% | #13 | Trajectory collection $\to$ SFT |
| DeepSearchQA | Free | 145 | 47 | 285 | 0% | #39 | DPO, GRPO, KTO, SLERP, RFT |
| Total |  | 475 | 108 | 839 | 8% |  |  |

#### Exploration volume varies by orders of magnitude.

The most compact success (ALFWorld: 4 submissions, 4 code files) and the most prolific exploration (HumanEval: 195 submissions, 336 model versions) differ by $\sim$50$\times$ in submission count. This range reflects genuinely different task structures: ALFWorld’s demonstration learning provides a direct path, while HumanEval and DeepSearchQA require search over training configurations.

#### Late peaking vs. early peaking.

Best scores arrive at qualitatively different stages. GSM8K peaks at submission #90/92 (98th percentile—near the end), while HumanEval peaks at #26/195 (13th percentile—early). ALFWorld peaks at #4/4 (monotonic), and DeepSearchQA at #39/145 (27th percentile). Late-peaking tasks (GSM8K) suggest that patient hyperparameter search pays off; early-peaking tasks (HumanEval) suggest diminishing returns despite continued exploration.

#### Zero-score rate as a fragility indicator.

HumanEval (23%) and GSM8K (20%) have the highest zero-score rates, caused primarily by OOM crashes during training. The agent recovers from these consistently, demonstrating crash-recovery capability. By contrast, AlpacaEval and DeepSearchQA achieve zero crash rates across 23 and 145 submissions respectively, indicating that the agent successfully manages resource constraints for these tasks.

### A.19 Claude Code Per-Task Case Studies

We describe the representative agent trajectory for each task, drawing on workspace artifacts. These cases are diagnostic—they complement the quantitative results in the main text but are not benchmark-native metrics.

#### Case 1: GSM8K — Model soup over 92 submissions.

The SFT-mode agent on gpu02 produced 146 model versions across 14 code files. Its strategy evolved from basic SFT $\to$ multi-seed SFT $\to$ weighted model averaging (model soup). The agent wrote averaging scripts that combined 2–4 checkpoints with learned interpolation weights. The best score (83.85) was reached at submission #90, following a model soup of three checkpoints trained with different learning rates and seeds. The 20% crash rate was caused by OOM on larger batch sizes; the agent systematically reduced batch size and enabled gradient checkpointing after each crash. In Free mode (gpu06), the same task was approached via GRPO with 15 distinct reward function variants, but only reached 82.34—suggesting that the additional complexity of reward engineering does not pay off when the task’s difficulty is primarily in data construction.

#### Case 2: HumanEval — Most prolific exploration.

The SFT-mode agent on gpu02 was the most prolific across all experiments: 195 submissions, 336 output model versions, 28 code files. The agent’s code evolution reveals a progression: basic SFT $\to$ data augmentation (generating additional problems) $\to$ model averaging $\to$ task arithmetic (combining delta vectors from different training runs) $\to$ seed sweeps. The 45 zero-score crashes (23%) were predominantly OOM errors; the agent demonstrated consistent crash recovery by reducing batch size, adding gradient checkpointing, or switching to LoRA. Despite this prolific exploration, the best score (81.71) was reached at submission #26, with the subsequent 169 submissions unable to improve—a textbook optimization plateau where additional computation is wasted.

#### Case 3: AlpacaEval — Synthetic preference data breakthrough.

The Free-mode agent on gpu06 exhibited the most controlled run: 23 submissions, zero crashes, monotonically improving score trajectory from 20.01 to 31.02. The key innovation was synthetic preference data generation. The agent wrote gen_claude_data.py and gen_gpt4style_data.py to generate diverse candidate responses to AlpacaEval instructions, then used an LLM judge to create pairwise preference annotations. This data was used for DPO training after an initial SFT warm-up. The SFT-only mode (gpu02), lacking DPO, explored 102 submissions with alternative strategies (NEFTune noise injection, SLERP model merging, task arithmetic) across 130 model versions but only reached 21.10—a 10-point gap that directly quantifies the value of the composite SFT$\to$DPO pipeline.

#### Case 4: ALFWorld — Demonstration learning via environment interaction.

Detailed in the main text (Q5). The Free-mode agent on gpu06 collected environment demonstrations through interaction and rollout, and achieved 95.52 in just 4 submissions. In RL-only mode (gpu05), the agent followed a two-phase SFT$\to$GRPO pipeline, reaching 91.79 in 7 submissions. In SFT-only mode (gpu02), the agent managed only 14.93 over 11 submissions, suggesting that the critical distribution alignment insight is not deterministic—even the same scaffold may or may not find it depending on the exploration path.

#### RL-only ALFWorld workspace audit.

Because the RL-only ALFWorld result (91.79) serves as the paper’s primary evidence that an agent can engineer an online RL loop, we provide an explicit workspace audit. The RL-only agent on gpu05 (workspace 20260327T221313_p5002_claude) produced 7 submissions across 9 code files:

- •
No privileged API usage: grep -r "expert_plan" code/ returns zero matches. The agent collected demonstrations through environment interaction (collect_demos.py) using the model’s own rollouts.
- •
SFT warm-up: Initial SFT on self-collected demonstrations, producing a baseline for further improvement.
- •
GRPO with online rollouts: The agent implemented grpo_train.py using GRPOTrainer from TRL with reward signals derived from environment success/failure. Online rollout collection was performed between GRPO iterations.
- •
Trajectory-level rewards: The reward function used sparse binary environment feedback (task success = +1, failure = 0), which is the canonical RL setup for ALFWorld.
- •
Progressive improvement: Submissions showed monotonic improvement: 8.21 $\to$ 47.01 $\to$ 71.64 $\to$ 85.07 $\to$ 91.79 $\to$ 89.55 $\to$ 91.04, with the best at submission #5.

This audit confirms that the RL-only result represents genuine online RL engineering—environment interaction, trajectory collection, reward-driven optimization. The constraint was enforced through prompt instruction rather than API removal, so this is an audited compliance result rather than a benchmark-enforced guarantee.

#### Case 5: WebShop — Sign reversal through trajectory quality.

The Free-mode agent on gpu07 achieved 84.0 in 16 submissions with a trajectory-collection-then-SFT pipeline. The agent wrote dedicated scripts for trajectory collection (collect_data.py), quality filtering (process_data.py), and SFT training. Three zero-score crashes (19%) occurred when the evaluator timed out. The SFT-mode agent on gpu02 achieved an even higher score of 91.0 by additionally incorporating DPO on filtered trajectories—the highest WebShop score across all experiments. This sign reversal from OpenCode-7B ($-15.50$) confirms that the primary bottleneck is trajectory collection quality, not training algorithm choice.

#### Case 6: DeepSearchQA — Exhaustive technique search.

The Free-mode agent on gpu07 was the most technically diverse: 47 code files implementing DPO (v1–v16), GRPO, KTO, ORPO, rejection fine-tuning, NEFTune, Adafactor, 3-way SLERP merging, 4-way TIES merging, and task arithmetic. Despite producing 285 model versions and 145 submissions, the best score reached only 15.0 ($+2.75$). The RL-mode agent on gpu05 independently converged on the same ceiling (15.0) via 117 submissions with DPO/GRPO/SFT/KTO pipelines. This convergent failure across diverse strategies, two independent agents, and two modes suggests that DeepSearchQA’s difficulty is structural: the multi-step search-retrieve-judge loop resists improvement through post-training alone, likely requiring architectural or inference-time interventions beyond what current agent-driven post-training can provide.

#### DeepSearchQA follow-up: evaluation variance and the SFT sweet spot.

A controlled follow-up experiment pits Claude Code (Opus 4.6 driver) against Codex (GPT-5.4 driver) on DeepSearchQA with Qwen3-8B-Base (baseline=12.25), each given 12 hours on 2$\times$H200. Both agents independently converge to the same narrow SFT sweet spot, providing strong evidence that the ceiling is task-intrinsic rather than agent-dependent.

**Table 15: DeepSearchQA SFT sensitivity analysis (Qwen3-8B-Base, 12h). The optimization landscape has a narrow sweet spot; deviations in any direction degrade performance sharply.**
| Parameter | Under-fit | Sweet spot | Over-fit |
| --- | --- | --- | --- |
| Learning rate | 5e-5 $\to$ 10.5 | 1e-4 $\to$ 11–13 | 2e-4 $\to$ 4.0 |
| LoRA rank | 16 $\to$ 10.5 | 32 $\to$ 11–13 | 64 $\to$ 4.0–7.5 |
| Epochs | 2 $\to$ 10.5 | 3–4 $\to$ 11–13 | 5+ $\to$ 6.5 |
| Training loss | $>$1.5 | 1.1–1.2 | $<$0.5 |

The key diagnostic finding is the evaluation variance. Codex repeated its best configuration (identical code, seed=3407) 20 times; scores ranged from 7.5 to 13.5, a $\pm$3pp spread. This variance has two sources: (1) stochasticity in the LLM-judge evaluator, and (2) interaction noise from the search-agent proxy. When the evaluation noise ($\pm$3pp) is comparable to the total improvement achieved (+3–4.5pp), the agent cannot reliably distinguish better training configurations from lucky evaluations, making iterative improvement effectively impossible.

GRPO was attempted by Claude Code and failed again (8.0 vs. SFT baseline 11.0), consistent with earlier results. The reward signal from the LLM judge is too sparse: most completions receive similar scores, producing near-zero gradients. This confirms that DeepSearchQA’s reward landscape is incompatible with current online RL methods for small models.

### A.20 Claude Code Score Trajectory Analysis

Figure: Figure 5: Claude Code submission-by-submission score trajectories (12h). Red dashed line = baseline; orange star = best score; red crosses = zero-score crashes.
Refer to caption: 2604.10547v1/x3.png

Figure: Figure 6: Mode $\times$ Task improvement heatmap for Claude Code. Stars mark the best mode per task; triangles mark the worst. Rank reversals between modes confirm that no single training paradigm uniformly dominates.
Refer to caption: 2604.10547v1/x4.png

Figure [5](#A1.F5) visualizes the submission-by-submission score trajectories. Here we note several qualitative patterns that distinguish the six tasks:

- •
Plateau pattern (GSM8K, HumanEval): Early improvement followed by a long flat tail. The agent continues exploring but cannot escape the local optimum. This is consistent with the observation that static tasks have limited headroom for post-training improvement.
- •
Monotonic climb (ALFWorld): Each of the 4 submissions strictly improves over the previous one. This reflects the effectiveness of the iterative demonstration learning pipeline: once the distribution alignment insight is found, each refinement (data balancing, augmentation) reliably improves performance.
- •
Controlled ascent (AlpacaEval): A smooth upward trajectory from 20 to 31, with no crashes. The DPO pipeline produces reliable improvements at each iteration, consistent with the stability of preference optimization.
- •
Noisy exploration (WebShop, DeepSearchQA): High variance between consecutive submissions, with intermittent crashes. This reflects the instability of interactive training loops where trajectory quality varies between runs.
- •
Crash-and-recover (HumanEval, GSM8K): Periodic zero-score submissions followed by recovery. The agent demonstrates robustness to OOM and other training failures by systematically adjusting hyperparameters.

### A.21 Additional Experimental Results (from Main Text Compression)

The following tables, figures, and analyses were moved from the main text to the appendix during page compression. They provide supporting evidence for the main claims.

#### Operating mode ablation.

**Table 16: Claude Code mode ablation (Qwen2.5-7B-Instruct, 12h). Bold = best, underline = worst per task.**
| Mode | GSM8K | HE | AE | ALF | WS | DSQA |
| --- | --- | --- | --- | --- | --- | --- |
| SFT-only | 84.00 | 81.71 | 21.32 | 67.91 | 93.00 | 14.00 |
| RL-only | 81.88 | 81.71 | 20.96 | 93.28 | 79.00 | 15.00 |
| Free | 83.40 | 86.59 | 33.19 | 96.27 | 88.00 | 14.50 |
| Multi | 85.06 | 75.61 | 19.54 | 13.43 | 79.00 | 11.50 |

#### RL method audit.

**Table 17: RL method audit across Claude Code workspaces. ^$\dagger$*Stable*: completed training reliably with non-degenerate results.**
| Task | RL Methods Tried | Stable?^$\dagger$ | Best Pipeline |
| --- | --- | --- | --- |
| GSM8K | GRPO ($\times$15 variants) | Partial | SFT + model soup |
| HumanEval | GRPO, DPO, RFT | No | SFT + model averaging |
| AlpacaEval | GRPO, DPO ($\times$31) | Yes (DPO) | SFT $\to$ DPO |
| ALFWorld | GRPO + online rollout | Yes | SFT $\to$ GRPO (RL-only) |
| WebShop | DPO, ORPO, KTO | Partial | Traj. collection $\to$ SFT |
| DeepSearchQA | DPO, GRPO, KTO, ORPO | Partial | SFT $\to$ DPO pipeline |

#### Multi-task degradation on Qwen3-8B-Base.

**Table 18: Multi-task degradation on Qwen3-8B-Base (12h). Free mode scores are from single-task runs. The pattern mirrors Table [16](#A1.T16): Multi wins on GSM8K but severely degrades interactive tasks.**
| Mode | GSM8K | HumanEval | AlpacaEval | ALFWorld | WebShop | DSQA |
| --- | --- | --- | --- | --- | --- | --- |
| Free | 74.00 | 84.15 | 26.91 | 97.76 | 67.00 | 15.00 |
| Multi | 80.97 | 60.98 | 10.33 | 85.82 | 13.00 | 7.00 |
| Baseline | 83.02 | 62.20 | 14.68 | 8.96 | 0.00 | 12.25 |

#### Complete 8B controlled study matrix.

**Table 19: Complete Qwen3-8B-Base controlled study (12h, single run). $\Delta$ computed against canonical baseline [83.02 / 62.20 / 14.68 / 8.96 / 0.00 / 12.25]. ^† AlpacaEval subject to evaluator noise.**
|  |  | L1: Static Rule | L2: Judge | L3: Interactive Rollout |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Scaffold | Driver | GSM8K ($\Delta$) | HumanEval ($\Delta$) | AlpacaEval^† ($\Delta$) | ALFWorld ($\Delta$) | WebShop ($\Delta$) | DSQA ($\Delta$) |
| — | — (Baseline) | 83.02 | 62.20 | 14.68 | 8.96 | 0.00 | 12.25 |
| Claude Code | Opus 4.6 | 88.78 (+5.76) | 84.15 (+21.95) | 26.91 (+12.23) | 97.76 (+88.80) | 67.00 (+67.00) | 15.00 (+2.75) |
| Claude Code | Sonnet 4.5 | 88.70 (+5.68) | 65.85 (+3.65) | 19.80 (+5.12) | 15.67 (+6.71) | 0.00 (0.00) | 18.50 (+6.25) |
| Codex CLI | GPT-5.4 | 84.00 (+0.98) | 0.00 ($-$62.20) | 22.82 (+8.14) | 14.93 (+5.97) | 57.00 (+57.00) | 12.00 ($-$0.25) |
| Codex CLI | GPT-5.2 | 89.46 (+6.44) | 69.51 (+7.31) | 18.67 (+3.99) | 87.31 (+78.35) | 75.00 (+75.00) | 12.00 ($-$0.25) |
| Codex CLI | GPT-4o | 83.32 (+0.30) | 64.63 (+2.43) | 17.19 (+2.51) | 9.70 (+0.74) | 0.00 (0.00) | 17.00 (+4.75) |
| Gemini CLI | 2.5-flash | 70.13 ($-$12.89) | 10.98 ($-$51.22) | 7.14 ($-$7.54) | 11.19 (+2.23) | 0.00 (0.00) | 19.00 (+6.75) |

#### Technical route distribution for OpenCode-7B.

**Table 20: Technical route distribution for OpenCode-7B runs.**
| Agent/Task | N | GRPO | SFT | Copy/Miss | Coding SR | Improve SR |
| --- | --- | --- | --- | --- | --- | --- |
| OpenCode / GSM8K | 10 | 4 | 2 | 4 | 60% | 40% |
| OpenCode / HumanEval | 10 | 2 | 3 | 5 | 50% | 40% |
| OpenCode / AlpacaEval | 10 | 1 | 5 | 4 | 70% | 50% |
| OpenCode / ALFWorld | 10 | 4 | 1 | 5 | 60% | 20% |
| OpenCode / WebShop | 10 | 5 | 1 | 4 | 70% | 10% |
| OpenCode / DeepSearchQA | 10 | 4 | 1 | 5 | 60% | 20% |

#### Computational cost.

**Table 21: Claude Code 24h token usage and estimated cost (Claude Opus 4.6 pricing).**
| Task | Calls | Output Tok. | Cache Read | Est. Cost |
| --- | --- | --- | --- | --- |
| deepsearchqa | 3,800 | 460K | 75.0M | $210 |
| alpacaeval | 2,600 | 280K | 59.0M | $145 |
| alfworld | 2,500 | 230K | 45.0M | $140 |
| humaneval | 2,300 | 265K | 48.0M | $125 |
| gsm8k | 2,600 | 160K | 51.0M | $120 |
| webshop | 1,400 | 72K | 12.0M | $34 |
| Total (19 agents) | 36,000 | 1,467K | 290.0M | $\sim$$796 |

#### Agent behavior efficiency: Claude Code vs. Codex.

**Table 22: Agent behavior comparison on DeepSearchQA (Qwen3-8B-Base, 12h).**
| Dimension | Claude Code | Codex |
| --- | --- | --- |
| Driver LLM | Claude Opus 4.6 | GPT-5.4 |
| Submissions | 16 | 56 |
| Rollbacks | 6 | 168 |
| Effective tokens | $\sim$32M (82% cache) | $\sim$91M (no cache) |
| Best $\Delta$ | +2.75 | $-$0.25 |

Claude Code operates in “deliberate exploration” mode (16 submissions, 82% cache hit), while Codex operates in “high-frequency iteration” mode (56 submissions, 168 rollbacks, 2.8$\times$ more tokens). Despite the token gap, both converge to similar ceilings ($\sim$13–14.5), suggesting that on structurally hard tasks, iteration volume cannot substitute for approach quality.

#### Variance, scaling, and pass@k.

Most gains on static tasks arrive within the first six hours, while interactive tasks benefit substantially from extended budgets (Figure [3](#S3.F3) in the main text). The combined scaling figure shows performance as a function of time, cumulative tokens, and submission count, enabling multi-dimensional efficiency comparison across systems. On ALFWorld, Codex+GPT-5.2 reaches 87.31 within $\sim$15M tokens, while CC+Sonnet 4.5 remains near baseline despite comparable token expenditure—confirming that raw compute is not the bottleneck. On static tasks (GSM8K, HumanEval), all systems plateau within the first few million tokens. On ALFWorld and DeepSearchQA, pass@k curves are steep: rare runs discover qualitatively better pipelines, indicating that variance is part of the capability story rather than pure noise.

Figure: Figure 7: Combined scaling analysis (time, token, submission) for all 8B-Base agent stacks. See main text Figure [3](#S3.F3) for full description.
Refer to caption: 2604.10547v1/x5.png

Figure: Figure 8: Improvement over canonical Qwen3-8B-Base baseline ($\Delta$ in percentage points) for six agent stacks across all tasks (12h). Interactive tasks (ALFWorld, WebShop) amplify driver differences far more sharply than static tasks. ^† AlpacaEval subject to evaluation noise.
Refer to caption: 2604.10547v1/x6.png

Figure: Figure 9: pass@k expected best improvement (7B setting). Interactive tasks show steep curves: the expected best score improves markedly with multiple runs, indicating a brittle search landscape where rare runs discover qualitatively better pipelines.
Refer to caption: 2604.10547v1/x7.png

### A.22 Detailed Trace Walkthrough: HumanEval on Qwen3-8B-Base

The per-task case studies above summarize behavioral patterns at the workspace level. To complement this, we present a fine-grained walkthrough of a single run that illustrates both the strengths and limitations of LLM-agent-driven post-training in concrete detail.

#### Setup.

The trace comes from a Claude Code Free-mode run on HumanEval with Qwen3-8B-Base—an unaligned base model that scores 0% pass@1 at baseline. The agent was given a 12-hour budget on a single H200 GPU. Table [23](#A1.T23) summarizes the progression.

**Table 23: Submission progression for the HumanEval trace walkthrough (Claude Code Free, Qwen3-8B-Base). Each row corresponds to a qualitatively distinct strategy shift.**
| Sub # | Script | Score | Method |
| --- | --- | --- | --- |
| baseline | — | 0.00% | Qwen3-8B-Base (unaligned) |
| #3 | sft_train.py | 2.44% | SFT v1 |
| #5 | sft_train_v3.py | 43.90% | SFT v3 (prompt alignment) |
| #7 | grpo_train.py | 79.27% | GRPO on SFT checkpoint (LoRA) |
| #11 | rft_train.py | 84.15% | RFT (rejection sampling fine-tuning) |

#### Step 1: SFT v1 (Sub #3, 2.44%).

The agent’s first move was supervised fine-tuning on the 82 HumanEval training examples, teaching the base model to produce a <think> tag followed by code:

Training used 4 epochs, lr=$2\times 10^{-5}$, batch size 2, gradient accumulation 4. The model learned to generate output, but scored only 2.44% because the evaluator could not parse most responses—the training prompt format did not match the evaluation prompt format.

#### Step 2: SFT v3 with prompt alignment (Sub #5, 43.9%).

The agent diagnosed the format mismatch and aligned the training prompt to match the evaluation prompt exactly:

This single string change produced an 18$\times$ improvement (2.44% $\to$ 43.9%). The agent also applied data augmentation, expanding 82 examples to 246 by generating three output format variants per example:

Agent limitation: This augmentation introduces *training target noise*. Format 1 trains the model to output complete functions (including the signature), while Formats 2–3 train it to output only the function body—two conflicting objectives in the same training set. The <think> content is a placeholder (the function name), not genuine reasoning. The 43.9% improvement is almost entirely attributable to prompt alignment, not augmentation, but the agent does not perform this ablation.

The downstream GRPO reward function later compensates for the mixed output format via post-processing:

This workaround masks the training inconsistency rather than resolving it at the source—a pattern we observe across multiple agent runs where post-processing substitutes for clean data design.

#### Step 3: GRPO with code execution reward (Sub #7, 79.27%).

With a 43.9% SFT checkpoint as the starting point, the agent introduced reinforcement learning via GRPO using actual code execution as the reward signal:

Agent strength: memory management.
The agent correctly identified that full-parameter GRPO would exceed GPU memory. Standard SFT for an 8B bfloat16 model requires approximately 64 GB (16 GB weights + 16 GB gradients + 32 GB AdamW states), which fits on a single H200 (80 GB). GRPO additionally requires a frozen reference model copy (+16 GB) for KL divergence computation, causing OOM. The agent switched to LoRA ($r{=}16$) so that only adapter parameters ($\sim$200 MB) are updated while the reference and policy models share the frozen base weights:

After training, the agent merged LoRA weights back into the base model before submission. The result: 79.27%, a 35-point gain over SFT alone.

#### Step 4: RFT after GRPO degradation (Sub #11, 84.15%).

Submissions #8–#10 continued GRPO iterations but showed degradation (down to 76.83%), indicating the policy had reached a local optimum. The agent recognized this and switched to rejection fine-tuning (RFT): sampling 16 candidate solutions per problem from the best checkpoint (Sub #7, 80.49%), filtering by test execution, and fine-tuning on the verified solutions.

This yielded 993 verified model-generated solutions plus 82 ground-truth examples (1,075 total). The agent fine-tuned with a conservative learning rate ($1\times 10^{-6}$, lower than the $5\times 10^{-6}$ used in SFT v3) to avoid overwriting GRPO-learned capabilities, reaching the final score of 84.15%.

#### Summary of agent capabilities and limitations.

This trace illustrates several recurring themes from the broader benchmark:

- •
Prompt–evaluation alignment dominates early gains: the 18$\times$ improvement from the initial SFT to the aligned version came from a single string change. This echoes the broader finding that distribution mismatch between training and evaluation is often the primary bottleneck, not model capacity or training algorithm choice.
- •
Progressive strategy composition: the agent composed SFT $\to$ GRPO $\to$ RFT, with each stage building on the previous checkpoint. This multi-stage pipeline is characteristic of successful Free-mode runs across tasks.
- •
Adaptive resource management: the LoRA switch for GRPO demonstrates the agent’s ability to diagnose and resolve engineering constraints (OOM) without human intervention.
- •
Degradation detection: the agent detected GRPO performance regression and pivoted to RFT rather than continuing a failing strategy—an important metacognitive capability.
- •
Training target noise from augmentation: the mixed output format (complete function vs. function body) introduced conflicting supervision signals that were masked by post-processing rather than resolved at the data level.
- •
Missing ablation awareness: the agent did not isolate the contribution of prompt alignment from data augmentation, potentially misattributing the 43.9% gain.

Taken together, this case shows that LLM agents can autonomously engineer a competitive multi-stage post-training pipeline from a zero-baseline starting point, but their data design decisions remain noisy and their self-evaluation of *why* a strategy worked is limited.

### A.23 Detailed Trace Walkthrough: ALFWorld on Qwen3-8B-Base

The HumanEval trace above demonstrates agent behavior on a static code-generation task. We now present a complementary walkthrough on ALFWorld—an interactive rollout task that is central to the benchmark’s core thesis. This trace achieves the largest absolute improvement across all experiments (+88.8pp) using *only* SFT, without any RL algorithm, and illustrates both the power and the limitations of agent-driven engineering on interactive tasks.

#### Setup.

Claude Code Free mode, Qwen3-8B-Base (8.96% baseline), 12-hour budget on 2$\times$H200. ALFWorld is a text-based household environment requiring multi-step ReAct-style interaction: the model observes the environment, reasons, executes an action, observes the result, and iterates until the task is solved or the step limit is reached. Table [24](#A1.T24) summarizes the progression.

**Table 24: Submission progression for the ALFWorld trace walkthrough (Claude Code Free, Qwen3-8B-Base, 12h). Shaded rows mark the two largest score jumps.**
| Sub # | Script | Score | Method |
| --- | --- | --- | --- |
| baseline | — | 8.96% | Qwen3-8B-Base (unaligned) |
| #1 | train_sft.py | 14.93% | SFT v1: ReAct examples |
| #2 | train_sft_v2.py | 16.42% | Hyperparameter tuning |
| #4 | train_sft_v4.py | 19.40% | Action syntax fix (put$\to$move) + synthetic data |
| #5 | train_sft_v5.py | 45.52% | Model rollout trajectories (+26.1pp) |
| #7 | train_sft_v7.py | 95.52% | Prompt format alignment (+50.0pp) |
| #8 | train_sft_v8.py | 96.27% | Retrain on v7 rollouts |
| #9 | train_sft_v9.py | 97.76% | 4$\times$ rollout weight + all rollouts |

#### The hidden action-syntax trap.

ALFWorld’s original ReAct few-shot examples use the phrasing put X in/on Y for placement actions, but the environment’s actual API requires move X to Y. This mismatch is not documented anywhere in the task description and must be discovered through environment interaction or code inspection. This trap creates a two-level distribution alignment problem that structures the entire trace.

#### Step 1: Baseline SFT (Sub #1–#3, 14.93–16.42%).

The agent constructed training data from the 18 ReAct examples in react_prompts.json (6 task types $\times$ 3 examples), using three strategies: full trajectories, permuted few-shot pairs, and step-by-step completions where the model learns to predict the next action at each step. Training used 8 epochs, lr=$2{\times}10^{-5}$, packing. The model learned to produce ReAct-format output but failed most tasks. Hyperparameter tuning (v2) and data augmentation (v3) yielded only marginal gains, indicating that the bottleneck was not in training configuration but in the data itself.

#### Step 2: Action syntax fix + synthetic trajectories (Sub #4, 19.40%).

The agent discovered the put$\to$move mismatch and wrote a regex-based converter applied to all training data:

The agent also invested substantial effort ($\sim$200 lines) writing template-based trajectory generators for all six task types, covering diverse object–location combinations. However, this synthetic data contributed only +3pp. Agent limitation: the agent defaulted to “produce more data” before validating whether the existing data format was correct—a pattern of over-engineering before hypothesis testing.

Critically, fix_put_to_move was applied to *both* the few-shot examples and the task trajectories. This will become the root cause of the next plateau.

#### Step 3: Model rollout trajectories (Sub #5, 45.52%).

The agent shifted from synthetic data to real environment interaction. Using vLLM to serve the v4 checkpoint, it rolled out across ALFWorld games and retained only successful trajectories:

This collected $\sim$264 successful trajectories, which were weighted 3$\times$ and combined with the original examples ($\sim$1,100 total samples). The result: +26pp over v4. Real environment trajectories vastly outperformed synthetic templates—but the score then plateaued at 45.52% (v6 added more rollouts with no gain).

#### Step 4: Prompt format alignment—the 50-point breakthrough (Sub #7, 95.52%).

The agent diagnosed the plateau by inspecting how eval.py constructs prompts and discovered a subtle two-level distribution mismatch:

- •
At evaluation time: eval.py uses the *original* react_prompts.json as few-shot examples, which contain put X in/on Y. The model must then output move X to Y as the action.
- •
At training time (v4/v5): fix_put_to_move() converted *everything*—including the few-shot examples—to move. So the model was trained on examples saying move, but at evaluation saw examples saying put.

The fix was to keep few-shot examples in their *original* format (with put) while only converting the task trajectory actions to move—exactly matching what the model sees at evaluation time:

The agent retrained from the base model (not from v5) with the corrected data (1,114 samples: 792 rebuilt rollouts + 36 original examples + 286 step-by-step completions). Training: 4 epochs, lr=$1{\times}10^{-5}$, $\sim$30 min on 2$\times$H200.

This single format change produced a +50pp jump—the largest single-submission improvement across all experiments. The fix is counterintuitive: to produce correct behavior, the training data must preserve the “incorrect” phrasing in the few-shot examples, because that is what the model will see at test time.

#### Step 5: Iterative self-play (Sub #8–#9, 96.27–97.76%).

The agent established a collect–retrain loop: each version’s model was used to collect higher-quality rollouts, which were fed back into the next round of training.

From v7 to v9, the gains were incremental (+0.75pp, +1.5pp) but consistent. The v8 collection succeeded on 271/274 games, with only 3 persistent failures on specific hard edge cases. The final 97.76% approaches the environment’s effective ceiling.

#### Comparison with other ALFWorld runs.

This trace is one of several ALFWorld runs in the benchmark, and the comparison is instructive:

- •
SFT mode (Qwen2.5-7B-Instruct, 24h): spent 11 submissions stuck at $\sim$15% before discovering the put$\to$move syntax, and *never* discovered the prompt format alignment—reaching only 67.91%. The stronger base model and longer budget did not compensate for missing the deeper insight.
- •
Free mode (Qwen2.5-7B-Instruct, 24h): used environment-provided demonstrations via extra.expert_plan API, collected 2,824 expert trajectories, and reached 96.27% in 4 submissions—a qualitatively different strategy (demonstration collection vs. environment debugging) that nonetheless reaches a similar ceiling.
- •
Codex (Qwen3-8B-Base, 12h): 19 submissions, max 14.93%. Failed to write a functional ReAct training pipeline, confirming that interactive-task engineering is a discriminating capability.

#### Summary of agent capabilities and limitations.

- •
Cross-module root-cause analysis: the +50pp breakthrough required the agent to simultaneously understand three components—eval.py’s prompt construction, the environment’s action API, and the training data pipeline—and identify the subtle inconsistency between them. This cross-module diagnostic capability is arguably the most valuable skill demonstrated across all traces.
- •
Environment interaction as data source: the shift from synthetic templates (+3pp) to real rollouts (+26pp) demonstrates that for interactive tasks, environment interaction is not optional—it is the primary data acquisition mechanism.
- •
Iterative self-play convergence: the collect–retrain loop (v7$\to$v8$\to$v9) produced monotonic improvements, showing that the agent can establish and execute a stable self-improvement cycle.
- •
Pure SFT sufficiency: the entire trace uses no RL algorithm (GRPO/PPO/DPO). All 97.76% came from SFT on increasingly better data. This supports the broader finding that agents tend to fall back to supervised pipelines, though in this case the pipeline was highly effective.
- •
Slow initial diagnosis: the agent took 3 submissions to discover the put$\to$move mismatch—a relatively surface-level issue discoverable by running a single environment step and inspecting the error. The tendency to “train first, debug later” cost exploration budget.
- •
Over-investment in low-value engineering: the v4 synthetic trajectory generators ($\sim$200 lines covering all 6 task types) contributed only +3pp, while the real breakthrough came from a 10-line data-format fix. The agent’s default response to low scores was to produce more data rather than to question data quality.

The HumanEval and ALFWorld traces together illustrate a recurring pattern: the dominant factor in agent-driven post-training is not algorithm choice but *distribution alignment*—ensuring that training data matches the evaluation protocol in format, context, and action space. Both traces’ largest jumps came from fixing mismatches that were invisible in the training loss but catastrophic at evaluation time.