Title: 2604.10674
ArXiv: 2604.10674

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

Title:

Content selection saved. Describe the issue below:

Description:

License: arXiv.org perpetual non-exclusive license

arXiv:2604.10674v1 [cs.LG] 12 Apr 2026

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

Hao Wang

∗1,5†

, Guozhi Wang

∗5‡

, Han Xiao

∗2

, Yufeng Zhou

5

, Yue Pan

5

,

Jichao Wang

1

, Ke Xu

3

, Yafei Wen

5

, Xiaohu Ruan

5

, Xiaoxin Chen

5

, Honggang Qi

4§

1

Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences

2

The Chinese University of Hong Kong

3

University of Science and Technology of China

4

University of Chinese Academy of Sciences

5

vivo AI Lab

wanghao251@mails.ucas.ac.cn

hgqi@ucas.ac.cn

11085439@vivo.com

∗

Equal contribution

§

Corresponding author

‡

Project lead

†

Intern at vivo

skill-sd.github.io

Abstract

Reinforcement learning (RL) has been widely used to train LLM agents for multi-turn interactive tasks, but its sample efficiency is severely limited by sparse rewards and long horizons. On-policy self-distillation (OPSD) alleviates this by providing dense token-level supervision from a privileged teacher that has access to ground-truth answers. However, such fixed privileged information cannot capture the diverse valid strategies in agent tasks, and naively combining OPSD with RL often leads to training collapse. To address these limitations, we introduce Skill-SD, a framework that turns the agent’s own trajectories into dynamic training-only supervision. Completed trajectories are summarized into compact natural language

skills

that describe successful behaviors, mistakes, and workflows. These skills serve as dynamic privileged information conditioning only the teacher, while the student always acts under the plain task prompt and learns to internalize the guidance through distillation. To stabilize the training, we derive an importance-weighted reverse-KL loss to provide gradient-correct token-level distillation, and dynamically synchronize the teacher with the improving student. Experimental results on agentic benchmarks demonstrate that Skill-SD substantially outperforms the standard RL baseline, improving both vanilla GRPO (+14.0%/+10.9% on AppWorld/Sokoban) and vanilla OPD (+42.1%/+40.6%).

1

Introduction

Reinforcement learning (RL) has become the dominant paradigm for post-training LLM agents on multi-turn interactive tasks. These agents typically act over ReAct-style trajectories that interleave reasoning with external tool calls

(Yao et al.,

2023

)

, operating software through APIs

(Liu et al.,

2025a

; Trivedi et al.,

2024

)

, navigating web interfaces

(Yao et al.,

2022

)

, and solving sequential planning problems. Group Relative Policy Optimization (GRPO)

(Shao et al.,

2024

)

enables effective policy optimization without a separate value network, and recent work shows that GRPO-based training can produce agents surpassing much larger models

(Chen et al.,

2025

; Feng et al.,

2025

; Zhang et al.,

2026a

)

. However, RL inherently suffers from sparse, delayed reward signals and high sample complexity. In long-horizon agentic tasks, the reward is often binary (task completed or not), making it extremely sparse and providing nearly negligible guidance about

which

tokens or actions are actually useful.

On-policy distillation (OPD) addresses this gap by providing dense token-level supervision from a privileged teacher. On-policy self-distillation (OPSD) further removes the need for a seperate teacher model: the same model serves as both teacher (with privileged information) and student (without), enabling efficient single-model training. Self-Distilled Reasoner

(Zhao et al.,

2026

)

matches GRPO performance with 8–12

×

\times

fewer tokens; SDPO

(Hübotter et al.,

2026

)

reaches GRPO-level accuracy 6

×

\times

faster; KDRL

(Xu et al.,

2025

)

further shows that combining distillation with policy gradients outperforms either alone. These successes, however, are largely confined to tasks with unique ground-truth answers such as mathematical proofs and code solutions, where a fixed correct answer naturally serves as the teacher’s privileged information.

Extending self-distillation to multi-turn agent tasks raises two challenges that prior work has not addressed.

(1) What should the teacher know?

Unlike math problems with a single verifiable answer, agent tasks usually admit diverse valid strategies: an AppWorld task can be solved through different API call sequences, and a Sokoban puzzle can be approached from multiple directions. Conditioning the teacher on any single fixed solution constrains the student’s exploration and fails to provide rich guidance.

(2) How to keep training stable?

Naively coupling self-distillation with RL for agents often leads to training collapse. When the teacher generates trajectories (off-policy), its distribution drifts from the student’s as training progresses, destabilizing the importance-weighted objective.

We present Skill-SD, a framework that turns the agent’s own trajectory history into a dynamic, training-only teacher signal. Skill-SD summarizes completed trajectories into compact natural-language

skills

that capture successful behaviors, mistakes, and high-level workflows. These skills provide reusable strategic guidance rather than a single fixed action sequence, allowing the student to explore its own solutions.
During training, these skills condition only the teacher; the student generates on-policy trajectories under the plain task prompt. In this way, the student learns to internalize the skills without introducing retrieval dependence at inference time.
In this self-distillation setting, the teacher and the student share the same parameters but are conditioned on different prompts. This prompt discrepancy introduces a distribution mismatch similar to off-policy learning: the student’s on-policy tokens are scored under the teacher’s skill-augmented distribution.
Recent analyses show that naive

k

3

k_{3}

differentiation can yield gradient-biased updates even in standard on-policy training

(Tang & Munos,

2025

)

. The same issue persists in our cross-prompt setting. To address this, we derive an importance-weighted reverse-KL loss (

ρ

⋅

k

3

\rho\cdot k_{3}

) that corrects this distribution mismatch, ensuring unbiased token-level gradient updates for the self-distillation objective. Moreover, a frozen teacher inevitably falls behind as the student improves, degrading both the skill-conditioned guidance and the distillation signal. To keep the teacher calibrated, Skill-SD dynamically synchronizes the teacher from the latest student checkpoint, so that the privileged signal co-evolves with the student’s improving policy. Our contributions are summarized as follows:

•

Skill as dynamic teacher signal.

We propose using trajectory-derived natural-language skills as dynamic privileged information that conditions only the teacher during self-distillation. This preserves the diversity of valid action paths while allows for the skill internalization for the student model (§

3.2

).

•

Importance-weighted reverse-KL loss.

We derive an importance-weighted reverse-KL loss for the cross-prompt self-distillation, where teacher and student share parameters but differ in prompt conditioning. This loss corrects the per-token gradient bias of the naive

k

3

k_{3}

estimator under distribution mismatch (§

3.3

, Appendix

D

).

•

Necessity of dynamic teacher synchronization.

Through systematic ablation, we show that periodically synchronizing the teacher with the improving student is essential for stable training: off-policy teacher-owned rollouts collapse during mid-training, while frozen teachers converge to lower plateaus. (§

4.3

).

We evaluate Skill-SD on multiple agentic benchmarks. Using Qwen3-4B-Instruct-2507 as the base model, Skill-SD achieves 64.9% accuracy on AppWorld and 62.5% on Sokoban, outperforming vanilla GRPO by +14.0% and +10.9%, respectively, while also surpassing vanilla OPD by large margins. These results demonstrate the effectiveness of skill-conditioned self-distillation for training LLM agents.

2

Related Work

On-policy distillation and self-distillation for LLMs.

Recent self-distillation methods for LLMs

(Furlanello et al.,

2018

; Agarwal et al.,

2024

)

differ along two axes. First, some rely on distillation alone

(Zhao et al.,

2026

; Ye et al.,

2026b

)

, whereas others integrate distillation with RL—by converting environment feedback into a dense self-distillation target

(Hübotter et al.,

2026

)

, unifying KD and RL in a single objective

(Xu et al.,

2025

)

, or mitigating their interference through selective imitation

(Zhang et al.,

2026c

)

. Second, the privileged context is typically

fixed or externally supplied

—ground-truth solutions, offline experience, or frontier-model trajectories as in concurrent work pi-Distill

(Penaloza et al.,

2026

)

. Concurrent OEL

(Ye et al.,

2026a

)

extracts transferable experience from trajectories and consolidates it via context distillation, but uses distillation alone without RL reward signals; OpenClaw-RL

(Wang et al.,

2026

)

combines environment-derived next-state signals with RL for agent training. Skill-SD integrates self-distillation with RL in a joint objective, using analytical skills generated from the agent’s own rollouts, and formalizes the cross-prompt importance-weighted KL estimator that this setting requires (Appendix

D

).

KL-regularized policy optimization and estimator design.

A recent insight in the

k

1

/

k

2

/

k

3

k_{1}/k_{2}/k_{3}

KL estimator family

(Schulman et al.,

2017

; Schulman,

2020

)

is that KL

value

estimation and KL

gradient

optimization are distinct problems: the widely used

k

3

k_{3}

estimator provides unbiased value estimates but biased gradients when differentiated directly as a loss

(Tang & Munos,

2025

; Liu et al.,

2025c

)

. RPG

(Zhang et al.,

2026b

)

further studies the interaction of importance weighting, KL direction, and clipped updates at scale. These analyses consider the standard same-prompt setting; our cross-prompt configuration, where teacher and student share parameters but differ in prompt conditioning, requires the importance weighting derived in Appendix

D

.

Reinforcement learning for multi-turn LLM agents.

GRPO

(Shao et al.,

2024

; DeepSeek-AI,

2025

)

has become the predominant RL algorithm for LLM training due to its memory efficiency. Adapting RL to multi-turn agent settings introduces new challenges: LOOP

(Chen et al.,

2025

)

adapts PPO for long-horizon tasks without a value network, GiGPO

(Feng et al.,

2025

)

proposes two-level advantage estimation for multi-turn GRPO, RAGEN

(Wang et al.,

2025

)

identifies the “echo trap” instability, and AgentEvolver

(Zhai et al.,

2025

)

introduces self-attributing mechanisms for fine-grained credit assignment. Agent-R1

(Cheng et al.,

2025

)

, AgentGym-RL

(Xi et al.,

2026

)

, and AgentRL

(Zhang et al.,

2026a

)

scale multi-turn RL across diverse environments with principled environment design

(Wang & Ammanabrolu,

2025

; Zhao et al.,

2025

; Liu et al.,

2025b

)

. These works improve RL algorithms or infrastructure for agents but do not incorporate self-distillation. Skill-SD complements this line by adding an auxiliary distillation loss that transfers skill-conditioned teacher knowledge alongside RL reward signals.

Experience, reflection, and memory in agents.

Reflexion

(Shinn et al.,

2023

)

stores self-reflections in episodic memory, ExpeL

(Zhao et al.,

2024

)

extracts reusable experience, Agent-R

(Yuan et al.,

2025

)

trains agents to reflect via iterative self-training, ECHO

(Hu et al.,

2025

)

rewrites hindsight trajectories, and EvolveR

(Wu et al.,

2025

)

synthesizes trajectory outcomes into strategic principles. UI-Genie

(Xiao et al.,

2025

)

iteratively boosts agent performance through a self-improving loop between agent model and reward model, while UI-Mem

(Xiao et al.,

2026

; Liu et al.,

2026

)

builds a self-evolving experience memory for online RL.
A common design across these works is to retrieve experience and append it to the agent’s prompt to improve trajectory generation. This creates two related tensions: within training, trajectories are generated under

π

​

(

a

∣

h

,

e

)

\pi(a\mid h,e)

but the policy-gradient update targets

π

​

(

a

∣

h

,

∅

)

\pi(a\mid h,\varnothing)

, inducing an unmodeled behavior–target mismatch in the importance ratio; at inference time, stripping experience causes a performance gap, while retaining it makes the policy dependent on retrieval. Skill-SD avoids both issues by conditioning only the teacher on skills during training, while the student generates on-policy trajectories under the plain task prompt. The auxiliary SDL loss distills the teacher’s token-level guidance into the student, so the student is trained and evaluated under the same plain prompt, without any retrieval dependency.

3

Method

Figure 1:

Skill-SD overview. (1) The student generates on-policy rollouts and receives task-level rewards. (2) Completed trajectories are asynchronously summarized into compact skills and stored in a UCB-indexed buffer. (3) The same token sequence is re-scored under the student prompt and the skill-augmented teacher prompt; GRPO provides task-level credit assignment while the importance-weighted SDL loss transfers token-level teacher knowledge. Token color intensity reflects prediction probability.

3.1

Problem Setup and GRPO Backbone

Figure

1

illustrates the overall Skill-SD pipeline. Each task instance

x

x

defines a multi-turn interaction with horizon

H

H

. A rollout

τ

=

(

y

1

,

…

,

y

T

)

\tau=(y_{1},\dots,y_{T})

denotes the sequence of action tokens emitted by the agent while interacting with the environment, where

T

≤

H

T\leq H

counts all action tokens across turns. We use a

completion-rate

reward

R

​

(

x

,

τ

)

=

1

K

​

∑

k

=

1

K

𝟙

​

[

c

k

​

(

τ

)

​

is satisfied

]

∈

[

0

,

1

]

,

R(x,\tau)=\frac{1}{K}\sum_{k=1}^{K}\mathbbm{1}\!\left[c_{k}(\tau)\ \text{is satisfied}\right]\in[0,1],

(1)

where

{

c

k

}

k

=

1

K

\{c_{k}\}_{k=1}^{K}

are task-specific verification criteria: state-based unit tests in AppWorld

(Trivedi et al.,

2024

)

and boxes-on-target counts in Sokoban (see §

4.1

for details). This fine-grained reward preserves partial progress that would be discarded by a binary success signal.

For each task, we sample a group of

G

G

trajectories from the student policy and normalize rewards within the group:

A

^

i

=

R

i

−

1

G

​

∑

j

=

1

G

R

j

std

​

(

R

1

,

…

,

R

G

)

+

ϵ

,

\hat{A}_{i}=\frac{R_{i}-\frac{1}{G}\sum_{j=1}^{G}R_{j}}{\mathrm{std}(R_{1},\dots,R_{G})+\epsilon},

(2)

where

R

i

=

R

​

(

x

,

τ

i

)

R_{i}=R(x,\tau_{i})

. The GRPO objective is:

ℒ

GRPO

​

(

θ

)

=

−

1

N

​

∑

i

=

1

G

∑

t

=

1

T

i

min

⁡

(

r

i

,

t

​

A

^

i

,

clip

⁡

(

r

i

,

t

,

1

−

ε

l

,

1

+

ε

h

)

​

A

^

i

)

,

\mathcal{L}_{\text{GRPO}}(\theta)=-\frac{1}{N}\sum_{i=1}^{G}\sum_{t=1}^{T_{i}}\min\!\left(r_{i,t}\hat{A}_{i},\;\operatorname{clip}(r_{i,t},1-\varepsilon_{l},1+\varepsilon_{h})\hat{A}_{i}\right),

(3)

with student trust-region ratio

r

i

,

t

=

π

θ

stu

​

(

y

i

,

t

∣

x

,

y

i

,

<

t

)

sg

⁡

(

π

θ

old

stu

​

(

y

i

,

t

∣

x

,

y

i

,

<

t

)

)

,

r_{i,t}=\frac{\pi_{\theta}^{\text{stu}}(y_{i,t}\mid x,y_{i,<t})}{\operatorname{sg}\!\left(\pi_{\theta_{\text{old}}}^{\text{stu}}(y_{i,t}\mid x,y_{i,<t})\right)},

(4)

and

N

=

∑

i

=

1

G

T

i

N=\sum_{i=1}^{G}T_{i}

is the total number of valid action tokens in the group, implementing

token-mean

reduction where both long and short trajectories contribute in proportion to their valid-token counts. Following DAPO

(Yu et al.,

2025

)

, we use asymmetric clipping bounds with

ε

h

>

ε

l

\varepsilon_{h}>\varepsilon_{l}

to prevent premature entropy collapse.

3.2

Skill-Conditioned Teacher and Dynamic Self-Evolution

Unlike math or code generation, multi-turn agent tasks rarely have a unique ground-truth solution: many action sequences can solve the same task. This rules out fixed-reference distillation but leaves room for a softer form of privileged information: structured summaries of what worked and what failed in past attempts. Injecting such summaries into the student would leak privileged information into the evaluation-time policy; conditioning only the teacher preserves a clean student interface while still transferring the knowledge gap.

The teacher in Skill-SD is therefore

skill-conditioned

: it receives the same task prompt augmented with compact natural-language skills extracted from prior trajectories. Each skill summarizes three aspects of an attempt:

e

=

Analyze

​

(

τ

,

x

)

=

(

e

success

,

e

mistake

,

e

workflow

)

,

e=\mathrm{Analyze}(\tau,x)=\bigl(e_{\text{success}},e_{\text{mistake}},e_{\text{workflow}}\bigr),

(5)

capturing what worked, what failed, and what high-level workflow should be followed next time. We do not store full trajectories as skills. Multi-turn tasks admit many valid action sequences, and a single canonical trace would overconstrain exploration.

Given task

x

x

, the student acts on the plain prompt, while the teacher sees the prompt augmented with retrieved skills:

π

θ

stu

(

⋅

∣

x

,

y

<

t

)

\displaystyle\pi_{\theta}^{\text{stu}}(\cdot\mid x,y_{<t})

=

π

θ

(

⋅

∣

x

,

y

<

t

)

,

\displaystyle=\pi_{\theta}(\cdot\mid x,y_{<t}),

(6)

π

θ

¯

tea

(

⋅

∣

x

,

S

(

x

)

,

y

<

t

)

\displaystyle\pi_{\bar{\theta}}^{\text{tea}}(\cdot\mid x,S(x),y_{<t})

=

π

θ

¯

(

⋅

∣

x

⊕

S

(

x

)

,

y

<

t

)

.

\displaystyle=\pi_{\bar{\theta}}(\cdot\mid x\oplus S(x),y_{<t}).

(7)

Here

S

​

(

x

)

S(x)

denotes the retrieved skills and

θ

¯

\bar{\theta}

is the teacher parameter state. In the

dynamic

setting,

θ

¯

\bar{\theta}

is synchronized from the latest student checkpoint at each iteration; in the

frozen

setting,

θ

¯

\bar{\theta}

is fixed throughout training.

Skill retrieval is lightweight. For each task, we select the single highest-scoring skill using a UCB criterion:

score

​

(

e

)

=

r

¯

​

(

e

)

+

c

​

ln

⁡

N

ucb

n

​

(

e

)

,

\mathrm{score}(e)=\bar{r}(e)+c\sqrt{\frac{\ln N_{\text{ucb}}}{n(e)}},

(8)

where

r

¯

​

(

e

)

\bar{r}(e)

is the mean reward of skill

e

e

,

N

ucb

N_{\text{ucb}}

is the total number of retrievals for the same task,

n

​

(

e

)

n(e)

is the number of times

e

e

has been selected for that task, and

c

c

controls the exploration–exploitation tradeoff. Skills with

n

​

(

e

)

=

0

n(e)=0

are selected first, ensuring that newly generated skills are always tried before the UCB score is computed. All statistics are maintained per-task. An auxiliary LLM is used only to summarize trajectories into skills; it does not participate in the optimization objective. This is a task-local form of context distillation

(Snell et al.,

2022

; Hsieh et al.,

2023

)

: the teacher gets richer context during training, but the student does not receive it at test time.

3.3

Importance-weighted reverse-KL loss

The skill-conditioned teacher provides a richer action distribution than the student, but distilling from it is not straightforward. Student and teacher condition on different prompts, so their log-probability ratio is no longer a standard on-policy quantity. Naively differentiating the

k

3

k_{3}

divergence estimator yields biased gradients even in standard on-policy settings

(Tang & Munos,

2025

)

; in our cross-prompt regime the same issue persists, so an explicit importance-correction term is needed to restore per-token unbiasedness.

In Skill-SD, trajectories are sampled from the old student under the plain prompt, and the skill-conditioned teacher re-scores those same tokens under the augmented prompt. Specifically, we sample

τ

i

∼

π

θ

old

stu

(

⋅

∣

x

)

,

\tau_{i}\sim\pi_{\theta_{\text{old}}}^{\text{stu}}(\cdot\mid x),

(9)

while recording the old-student log-probabilities during rollout. We then re-forward the sampled sequence under the current student and under the fixed teacher view

π

θ

¯

tea

\pi_{\bar{\theta}}^{\text{tea}}

to obtain

log

⁡

π

θ

stu

\log\pi_{\theta}^{\text{stu}}

and

log

⁡

π

θ

¯

tea

\log\pi_{\bar{\theta}}^{\text{tea}}

.

For each sampled token, define the current-student to teacher-reference log-ratio

ℓ

i

,

t

=

log

⁡

π

θ

stu

​

(

y

i

,

t

∣

x

,

y

i

,

<

t

)

−

log

⁡

π

θ

¯

tea

​

(

y

i

,

t

∣

x

,

S

​

(

x

)

,

y

i

,

<

t

)

,

\ell_{i,t}=\log\pi_{\theta}^{\text{stu}}(y_{i,t}\mid x,y_{i,<t})-\log\pi_{\bar{\theta}}^{\text{tea}}(y_{i,t}\mid x,S(x),y_{i,<t}),

(10)

and the on-policy importance weight

ρ

i

,

t

on

=

π

θ

stu

​

(

y

i

,

t

∣

x

,

y

i

,

<

t

)

sg

⁡

(

π

θ

old

stu

​

(

y

i

,

t

∣

x

,

y

i

,

<

t

)

)

.

\rho_{i,t}^{\text{on}}=\frac{\pi_{\theta}^{\text{stu}}(y_{i,t}\mid x,y_{i,<t})}{\operatorname{sg}\!\left(\pi_{\theta_{\text{old}}}^{\text{stu}}(y_{i,t}\mid x,y_{i,<t})\right)}.

(11)

We then optimize the auxiliary self-distillation loss

ℒ

SDL

​

(

θ

)

=

1

N

​

∑

i

=

1

G

∑

t

=

1

T

i

ρ

i

,

t

on

​

(

e

−

ℓ

i

,

t

−

1

+

ℓ

i

,

t

)

.

\mathcal{L}_{\text{SDL}}(\theta)=\frac{1}{N}\sum_{i=1}^{G}\sum_{t=1}^{T_{i}}\rho_{i,t}^{\text{on}}\left(e^{-\ell_{i,t}}-1+\ell_{i,t}\right).

(12)

The auxiliary SDL term transfers teacher-side skill knowledge into the student under the correct sampling distribution. The weight

ρ

i

,

t

on

\rho_{i,t}^{\text{on}}

provides the necessary importance correction, since differentiating bare

k

3

k_{3}

directly produces biased gradients

(Tang & Munos,

2025

; Liu et al.,

2025c

)

. The formal gradient identity is given in Appendix

D

.

For theoretical completeness and for the ablation study in §

4.3

, we also derive the importance weight under teacher-owned (off-policy) rollout, where trajectories are sampled from

π

θ

old

tea

(

⋅

∣

x

⊕

S

(

x

)

)

\pi_{\theta_{\text{old}}}^{\text{tea}}(\cdot\mid x\oplus S(x))

. The same construction yields

ρ

i

,

t

off

=

π

θ

stu

​

(

y

i

,

t

∣

x

,

y

i

,

<

t

)

sg

⁡

(

π

θ

old

tea

​

(

y

i

,

t

∣

x

,

S

​

(

x

)

,

y

i

,

<

t

)

)

,

\rho_{i,t}^{\text{off}}=\frac{\pi_{\theta}^{\text{stu}}(y_{i,t}\mid x,y_{i,<t})}{\operatorname{sg}\!\left(\pi_{\theta_{\text{old}}}^{\text{tea}}(y_{i,t}\mid x,S(x),y_{i,<t})\right)},

(13)

preserving the same gradient-correct interpretation under teacher sampling (Appendix

D.3

).

Equation (

12

) is a

sampled-token

reverse-KL objective rather than a full-vocabulary distillation loss, and it uses the same

token-mean

aggregation as the GRPO term. For long-horizon agent traces, sampled-token distillation avoids the memory overhead of storing full-vocabulary logits at every token position and requires only the re-forward pass that is already needed for computing

r

i

,

t

r_{i,t}

.

Visualizing distillation dynamics.

Figure

2

visualizes how the student and teacher token distributions evolve during training on a representative AppWorld task. At the start of training, the student and teacher assign notably different probabilities to the same tokens; by the end, their distributions have largely converged. The teacher’s distribution remains relatively stable across epochs, while the student progressively learns to match it through the SDL loss. The SDL loss curve (right) decreases by 59.3% over training, confirming this convergence.

Figure 2:

Token-level self-distillation dynamics on an AppWorld task. Token color intensity is proportional to prediction probability. At Epoch 0, student and teacher distributions differ substantially; after convergence, they have largely aligned. Right: the SDL loss decreases by 59.3% over training.

3.4

Training Objective and Procedure

GRPO and SDL are complementary: GRPO uses trajectory-level reward to select good rollouts, while SDL uses the skill-conditioned teacher to redistribute probability mass at the token level within those rollouts. Neither signal alone suffices. RL without per-token guidance is sample-inefficient in long-horizon tasks, and distillation without reward grounding can drift toward the teacher even when the teacher is wrong. Combining them lets the reward filter bad trajectories while the teacher accelerates credit assignment on good ones.

The overall training objective combines GRPO reward maximization with the auxiliary SDL term:

ℒ

total

​

(

θ

)

=

ℒ

GRPO

​

(

θ

)

+

λ

​

ℒ

SDL

​

(

θ

)

,

\mathcal{L}_{\text{total}}(\theta)=\mathcal{L}_{\text{GRPO}}(\theta)+\lambda\mathcal{L}_{\text{SDL}}(\theta),

(14)

where

λ

\lambda

controls the distillation strength. The two terms act at different resolutions:

ℒ

GRPO

\mathcal{L}_{\text{GRPO}}

provides task-level credit assignment through group-relative advantages, determining

which

trajectories should be reinforced, while

ℒ

SDL

\mathcal{L}_{\text{SDL}}

supplies dense token-level guidance toward the skill-conditioned teacher, shaping

how

probability mass is redistributed along those trajectories. These signals are often complementary, but they can disagree when teacher-preferred actions are not aligned with the reward-improving direction on a given sample. The coefficient

λ

\lambda

mediates this trade-off: small

λ

\lambda

recovers nearly pure GRPO, while larger

λ

\lambda

more strongly constrains the student toward the teacher distribution. Within a given update, Proposition

2

shows that

∇

θ

(

ρ

⋅

k

3

)

=

ρ

⋅

s

θ

⋅

ℓ

t

\nabla_{\theta}(\rho\cdot k_{3})=\rho\cdot s_{\theta}\cdot\ell_{t}

, so the SDL gradient is exactly zero on sampled tokens where teacher and student agree (

ℓ

t

=

0

\ell_{t}=0

). In the on-policy regime, where

ρ

t

on

\rho_{t}^{\text{on}}

stays near

1

1

under small PPO-style updates, the SDL term is correspondingly self-damping as student–teacher disagreement shrinks; Figure

2

confirms this convergence empirically. This property makes SDL an auxiliary shaping signal rather than the primary learning driver, motivating a small

λ

\lambda

in practice. Note that although

ρ

t

on

\rho_{t}^{\text{on}}

and the GRPO clipping ratio

r

t

r_{t}

coincide numerically in the main branch, they serve distinct roles—importance correction for SDL versus student-centered trust region for GRPO—as detailed in Appendix

D.4

(Table

3

). Algorithm

1

summarizes a single training iteration.

Algorithm 1

Skill-SD training

0:

Student policy

π

θ

\pi_{\theta}

, skill bank

ℬ

\mathcal{B}

, group size

G

G

, SDL coefficient

λ

\lambda

, learning rate

η

\eta

1:

for

iteration

=

1

,

2

,

…

=1,2,\dots

do

2:

θ

old

←

θ

\theta_{\text{old}}\leftarrow\theta

;  sync teacher

θ

¯

←

θ

\bar{\theta}\leftarrow\theta

⊳

\triangleright

dynamic

; or keep

θ

¯

\bar{\theta}

fixed (

frozen

)

3:

— Rollout —

4:

for

each task

x

x

in the batch

do

5:

S

​

(

x

)

←

UCB

​

-

​

retrieve

​

(

ℬ

​

(

x

)

)

S(x)\leftarrow\mathrm{UCB\text{-}retrieve}\!\left(\mathcal{B}(x)\right)

⊳

\triangleright

Eq.

8

;

∅

\emptyset

if no prior skills

6:

Sample

{

τ

i

}

i

=

1

G

∼

π

θ

old

stu

(

⋅

∣

x

)

\{\tau_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}^{\text{stu}}(\cdot\mid x)

;  record

log

⁡

π

θ

old

stu

\log\pi_{\theta_{\text{old}}}^{\text{stu}}

7:

Compute rewards

R

i

=

R

​

(

x

,

τ

i

)

R_{i}=R(x,\tau_{i})

and advantages

A

^

i

\hat{A}_{i}

⊳

\triangleright

Eq.

2

8:

end

for

9:

— Re-score —

10:

for

each

(

x

,

{

τ

i

}

)

(x,\{\tau_{i}\})

do

11:

Compute

log

⁡

π

θ

stu

​

(

y

i

,

t

∣

x

,

y

i

,

<

t

)

\log\pi_{\theta}^{\text{stu}}(y_{i,t}\mid x,\,y_{i,<t})

⊳

\triangleright

current student, plain prompt

12:

Compute

log

⁡

π

θ

¯

tea

​

(

y

i

,

t

∣

x

⊕

S

​

(

x

)

,

y

i

,

<

t

)

\log\pi_{\bar{\theta}}^{\text{tea}}(y_{i,t}\mid x\oplus S(x),\,y_{i,<t})

⊳

\triangleright

teacher, augmented prompt

13:

end

for

14:

— Policy Update —

15:

r

i

,

t

=

ρ

i

,

t

on

←

π

θ

stu

/

sg

⁡

(

π

θ

old

stu

)

r_{i,t}=\rho_{i,t}^{\text{on}}\leftarrow\pi_{\theta}^{\text{stu}}\big/\operatorname{sg}\!\left(\pi_{\theta_{\text{old}}}^{\text{stu}}\right)

⊳

\triangleright

trust-region ratio & IS weight

16:

ℓ

i

,

t

←

log

⁡

π

θ

stu

−

log

⁡

π

θ

¯

tea

\ell_{i,t}\leftarrow\log\pi_{\theta}^{\text{stu}}-\log\pi_{\bar{\theta}}^{\text{tea}}

⊳

\triangleright

student–teacher log-ratio (Eq.

10

)

17:

ℒ

←

ℒ

GRPO

​

(

θ

;

r

,

A

^

)

+

λ

​

ℒ

SDL

​

(

θ

;

ρ

on

,

ℓ

)

\mathcal{L}\leftarrow\mathcal{L}_{\text{GRPO}}(\theta;\,r,\,\hat{A})\ +\ \lambda\,\mathcal{L}_{\text{SDL}}(\theta;\,\rho^{\text{on}},\,\ell)

⊳

\triangleright

Eqs.

3

,

12

18:

θ

←

θ

−

η

​

∇

θ

ℒ

\theta\leftarrow\theta-\eta\,\nabla_{\theta}\mathcal{L}

19:

Async:

summarize new trajectories

→

\to

skills; update

ℬ

\mathcal{B}

⊳

\triangleright

consumed next iter.

20:

end

for

4

Experiments

4.1

Setup

Benchmarks.

We evaluate Skill-SD on two multi-turn agentic environments that test complementary capabilities. AppWorld

(Trivedi et al.,

2024

)

emphasizes real-world API coordination, multi-app state management, and adaptive replanning from environment feedback; errors are generally recoverable through subsequent API calls. Sokoban, by contrast, demands spatial reasoning and long-horizon planning under high irreversibility: a single misstep (e.g., pushing a box into a corner) can render the puzzle unsolvable. Together, the two benchmarks span the spectrum from feedback-driven tool use to deliberative forward planning, and give a more complete picture of agent capabilities.

AppWorld

(Trivedi et al.,

2024

)

is a comprehensive multi-app API benchmark with 9 simulated consumer applications and 457 APIs. Tasks require multi-step reasoning, cross-application coordination, and iterative API interactions. Correctness is verified by state-based unit tests that check whether the correct state changes were made without undesired side effects. The official training set contains 105 tasks, of which 90 are publicly available

(Trivedi et al.,

2024

; Chen et al.,

2025

)

. We train on all 90 available tasks and evaluate on the 57-task development set, with a maximum of

H

=

40

H=40

interaction turns.

Sokoban

is a classic puzzle-planning benchmark requiring strategic reasoning and long-horizon planning. We use

6

×

6

6\times 6

rooms with 2 boxes and a maximum of

H

=

40

H=40

steps. Levels are procedurally generated using the reverse-play method of

gym-sokoban

(Schrader,

2018

)

, where the

search depth

parameter controls the depth of the reverse DFS and thus the minimum solution complexity. Since task diversity is limited, we adopt a difficulty curriculum

(Bengio et al.,

2009

)

to maximize signal extraction from each level: the training set consists of 96 levels where the easiest 30% use search depth 15, the middle 50% use depth 20, and the hardest 20% use depth 25. The test set consists of 64 levels with uniform search depth 25.

Models and metrics.

All experiments use Qwen3-4B-Instruct-2507

(Qwen Team,

2025

)

as the base model (full hyperparameters in Table

5

). We report

accuracy

(pass@1) and

completion rate

on the AppWorld 57-task development set and the Sokoban 64-level test set. The completion rate is the same dense reward used for training (Eq.

1

).

Baselines.

To isolate the contribution of each component, we compare against:

•

Vanilla GRPO

: standard GRPO with completion-rate reward, without skill prompting or self-distillation.

•

Skill-Augmented GRPO

: GRPO with skills prepended to the rollout prompt, but without the SDL loss (

λ

=

0

\lambda=0

). This isolates the effect of skill prompting from gradient-based distillation.

•

Vanilla OPD

: on-policy distillation without RL reward. The student generates rollouts under the plain prompt; the reverse-KL divergence between the skill-conditioned frozen teacher and the student at each token serves as the per-token advantage in a policy gradient objective. No environment reward is used (

ℒ

GRPO

=

0

\mathcal{L}_{\text{GRPO}}=0

), so learning is driven entirely by the distillation signal.

•

Base model

: the Qwen3-4B-Instruct-2507 checkpoint without agent finetuning.

4.2

Main Results

Table 1:

Main results on AppWorld and Sokoban (Qwen3-4B-Instruct-2507). Skill-SD uses on-policy student rollout with dynamic teacher synchronization. Subscripts denote absolute change from the base model.

AppWorld

Sokoban

Avg.

Method

Acc.

Comp.

Acc.

Comp.

Acc.

Comp.

Base Model

8.8

0.0

39.1

0.0

12.5

0.0

32.0

0.0

10.6

0.0

35.6

0.0

Vanilla OPD

22.8

↑

14.0

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!14.0}}

59.7

↑

20.6

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!20.6}}

21.9

↑

9.4

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!9.4}}

37.5

↑

5.5

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!5.5}}

22.4

↑

11.7

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!11.7}}

48.6

↑

13.0

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!13.0}}

Vanilla GRPO

50.9

↑

42.1

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!42.1}}

76.3

↑

37.2

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!37.2}}

51.6

↑

39.1

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!39.1}}

68.8

↑

36.8

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!36.8}}

51.2

↑

40.6

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!40.6}}

72.5

↑

36.9

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!36.9}}

Skill-Augmented GRPO

42.1

↑

33.3

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!33.3}}

76.1

↑

37.0

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!37.0}}

20.3

↑

7.8

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!7.8}}

37.5

↑

5.5

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!5.5}}

31.2

↑

20.6

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!20.6}}

56.8

↑

21.2

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!21.2}}

\rowcolor

bestrowblue

Skill-SD

64.9

↑

56.1

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!56.1}}

84.9

↑

45.8

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!45.8}}

62.5

↑

50.0

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!50.0}}

71.1

↑

39.1

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!39.1}}

63.7

↑

53.1

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!53.1}}

78.0

↑

42.4

{}_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\uparrow\!42.4}}

Table

1

compares Skill-SD against all baselines on both benchmarks. On AppWorld, Skill-SD achieves 64.9% accuracy and 84.9% completion rate, outperforming Skill-Augmented GRPO by +22.8% accuracy and Vanilla GRPO by +14.0%. On Sokoban, Skill-SD reaches 62.5% accuracy and 71.1% completion rate, surpassing Vanilla GRPO by +10.9%. Skill-Augmented GRPO underperforms Vanilla GRPO on both benchmarks (42.1% vs. 50.9% on AppWorld; 20.3% vs. 51.6% on Sokoban), demonstrating that directly injecting skills into the student prompt is counterproductive. Vanilla OPD remains the weakest reward-free method on both benchmarks, confirming that self-distillation alone cannot replace reward-driven optimization.

Training dynamics.

Figure

3

shows the training dynamics of all baselines on both benchmarks. On AppWorld, Skill-Augmented GRPO achieves the highest

training

accuracy but overfits severely: its validation accuracy (42.1%, Table

1

) falls well below Skill-SD (64.9%). Skill-SD improves steadily on both training and validation throughout. On Sokoban, Skill-SD starts slower than Vanilla GRPO but overtakes it after step 60 and continues to improve steadily. In both environments, Vanilla OPD remains at low performance throughout training.

Figure 3:

Training curves for all baselines on AppWorld (left) and Sokoban (right). On AppWorld, Skill-Augmented GRPO achieves the highest

training

accuracy while Skill-SD steadily narrows the gap. On Sokoban, Skill-SD overtakes Vanilla GRPO after step 60 and maintains steady gains.

4.3

Ablation Analysis

We validate Skill-SD’s two core design choices by examining what happens when each is replaced. The first choice is

student-owned rollout

: in Skill-SD, the student generates trajectories under the plain prompt and the teacher re-scores them; the alternative is teacher-owned rollout, where the teacher generates trajectories under the skill-augmented prompt and the student re-forwards them. The second choice is

dynamic teacher synchronization

: in Skill-SD, the teacher is synchronized from the latest student checkpoint at each iteration; the alternative is a frozen teacher that retains its initial parameters throughout training. Table

2

reports the effect of each replacement.

Table 2:

Ablation: effect of replacing Skill-SD’s design choices. Skill-SD uses on-policy student rollout with dynamic teacher synchronization (bolded). Subscripts denote absolute change from Skill-SD.

∗

Training collapsed during mid-training; values reflect the checkpoint before collapse.

AppWorld

Sokoban

Avg.

Rollout

Teacher

Acc.

Comp.

Acc.

Comp.

Acc.

Comp.

On-policy

Frozen

49.1

↓

15.8

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!15.8}}

79.0

↓

5.9

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!5.9}}

50.0

↓

12.5

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!12.5}}

63.3

↓

7.8

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!7.8}}

49.6

↓

14.1

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!14.1}}

71.1

↓

6.9

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!6.9}}

\rowcolor

bestrowblue

On-policy

Dynamic

64.9

0.0

84.9

0.0

62.5

0.0

71.1

0.0

63.7

0.0

78.0

0.0

Off-policy

∗

Frozen

45.6

↓

19.3

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!19.3}}

78.8

↓

6.1

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!6.1}}

12.5

↓

50.0

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!50.0}}

31.3

↓

39.8

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!39.8}}

29.1

↓

34.6

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!34.6}}

55.0

↓

23.0

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!23.0}}

Off-policy

∗

Dynamic

42.1

↓

22.8

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!22.8}}

76.5

↓

8.4

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!8.4}}

10.9

↓

51.6

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!51.6}}

32.0

↓

39.1

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!39.1}}

26.5

↓

37.2

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!37.2}}

54.3

↓

23.7

{}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow\!23.7}}

Figure 4:

Training dynamics of the four teacher–student configurations on AppWorld (left) and Sokoban (right). Skill-SD (a, on-policy + dynamic) trains stably to high performance. On-policy + frozen (b) converges stably but to a lower plateau. Both off-policy variants (c, d) achieve strong early performance but collapse during mid-training. The pattern replicates across both environments.

Student-owned rollout is essential.

Teacher-owned (off-policy) rollout achieves strong early performance but collapses during mid-training in both environments (Figure

4

c,d), regardless of whether the teacher is frozen or dynamically refreshed. As the student policy diverges from the teacher’s rollout distribution, the importance ratios become increasingly unstable, eventually destabilizing training. The collapse is far more severe on Sokoban (12.5% accuracy, matching the base model without agent finetuning) than on AppWorld (45.6%). This asymmetry reflects the environments’ different error tolerances: in AppWorld, a suboptimal action from distribution mismatch can often be corrected in subsequent API calls, so performance degrades gradually; in Sokoban, a single wrong push can render the puzzle unsolvable, so even mild distribution drift causes catastrophic failure. This confirms that sustainable self-distillation requires student-owned rollout, and that the cost of off-policy mismatch scales with task irreversibility.

Dynamic synchronization improves on-policy training.

Within the on-policy regime, dynamic teacher synchronization provides consistent gains over a frozen teacher: +15.8% accuracy on AppWorld and +12.5% on Sokoban. The frozen on-policy configuration remains stable but converges to a lower performance plateau.

Skills should guide the teacher, not the student.

Skill-Augmented GRPO underperforms Vanilla GRPO on both benchmarks (42.1% vs. 50.9% on AppWorld; 20.3% vs. 51.6% on Sokoban), showing that directly injecting skills into the student prompt is counterproductive. This is analogous to the privileged-information framework of

Vapnik & Vashist (

2009

)

: Skill-Augmented GRPO optimizes return under the augmented policy

π

θ

E

​

(

a

|

h

)

:=

π

θ

​

(

a

|

h

,

e

)

\pi^{E}_{\theta}(a|h):=\pi_{\theta}(a|h,e)

conditioned on skills, but at evaluation the agent uses the restricted policy

π

θ

0

​

(

a

|

h

)

:=

π

θ

​

(

a

|

h

,

∅

)

\pi^{0}_{\theta}(a|h):=\pi_{\theta}(a|h,\varnothing)

without skills.

Even with infinite data, maximizing

J

E

​

(

θ

)

=

𝔼

τ

∼

π

θ

E

​

[

R

​

(

τ

)

]

J_{E}(\theta)=\mathbb{E}_{\tau\sim\pi^{E}_{\theta}}[R(\tau)]

does not in general maximize

J

0

​

(

θ

)

=

𝔼

τ

∼

π

θ

0

​

[

R

​

(

τ

)

]

J_{0}(\theta)=\mathbb{E}_{\tau\sim\pi^{0}_{\theta}}[R(\tau)]

: the two objectives correspond to different conditional policies that share parameters

θ

\theta

. A gradient step

θ

+

=

θ

+

α

​

g

E

\theta^{+}=\theta+\alpha\,g_{E}

improves the evaluation objective only to the extent that

g

0

⊤

​

g

E

>

0

g_{0}^{\top}g_{E}>0

, and the alignment decays when skill-conditioned tokens dominate the computation or compete with the representations needed for planning.

Inspecting the training dynamics (Figure

3

) corroborates this: on AppWorld, Skill-Augmented GRPO achieves the highest training score of any method yet collapses on validation, providing direct evidence of overfitting to the skill-conditioned policy; on Sokoban, skills actively disrupt spatial reasoning even during training. Skill-SD avoids this pitfall by confining skills to a training-only teacher branch, analogous to System 2

→

\to

System 1 compilation

(Yu et al.,

2024

)

, so that the student internalizes useful components through gradient-based distillation rather than prompt-level injection.

Pure self-distillation without reward signals fails.

Vanilla OPD achieves only 22.8% accuracy on AppWorld and 21.9% on Sokoban, far below every reward-optimized variant. Privileged-context self-distillation alone cannot replace reward-driven learning in long-horizon interactive tasks.

Figure 5:

Effect of SDL coefficient

λ

\lambda

on AppWorld training (left) and validation (right) completion rate.

λ

=

0.001

\lambda=0.001

achieves the best validation performance;

λ

=

0.01

\lambda=0.01

over-regularizes and suppresses exploration, while

λ

=

0.0005

\lambda=0.0005

provides insufficient teacher guidance.

SDL weight

λ

\lambda

controls the RL–distillation balance.

Figure

5

sweeps

λ

∈

{

0.01

,

0.005

,

0.001

,

0.0005

}

\lambda\in\{0.01,0.005,0.001,0.0005\}

on AppWorld. All four values produce similar training curves in the first 100 steps; after that,

λ

=

0.001

\lambda=0.001

continues to improve and reaches the highest validation completion rate (

∼

\sim

80% with extended training beyond the checkpoint used in Table

1

), while

λ

=

0.01

\lambda=0.01

plateaus and becomes unstable. Reducing

λ

\lambda

below the optimum also hurts:

λ

=

0.0005

\lambda=0.0005

provides insufficient teacher guidance and performs comparably to the larger but noisier

λ

=

0.005

\lambda=0.005

, both settling around 70–75% validation completion rate. The pattern reflects a bias–variance trade-off: larger

λ

\lambda

regularizes toward the teacher but overpowers the RL signal; smaller

λ

\lambda

recovers nearly pure GRPO and loses the dense supervision benefit. At

λ

=

0.001

\lambda=0.001

, SDL acts as a mild shaping term that guides the student without dominating the combined gradient.

4.4

Training Efficiency

The primary computational overhead of Skill-SD relative to vanilla GRPO is the external LLM API calls used to summarize trajectories into skills. In a naïve synchronous implementation, these API requests would block the rollout and parameter-update pipeline, substantially increasing wall-clock time per training step. We adopt a fully asynchronous architecture that decouples skill generation from the main training loop: API requests are dispatched in the background and their results are consumed by subsequent iterations rather than the current one. With this design, the API latency is largely overlapped with rollout and gradient computation, so the additional wall-clock overhead per training step is marginal.

5

Conclusion

We introduced Skill-SD, a framework that combines skill-conditioned self-distillation with GRPO for training multi-turn LLM agents. The teacher is conditioned on analytical skills distilled from trajectories; the student generates on-policy trajectories under the plain task prompt. This separation gives Skill-SD per-token supervision without constraining exploration. Dynamic teacher synchronization keeps the distillation signal calibrated as the student improves, and the importance-weighted SDL loss ensures per-token unbiased gradients for the distillation loss (Proposition

2

). These gains are achieved with only 90 AppWorld tasks and 96 Sokoban levels, showing that Skill-SD remains effective under the small task pools typical of multi-turn interactive benchmarks. On both AppWorld, an API-based multi-app coordination benchmark, and Sokoban, a spatial planning benchmark with irreversible moves, Skill-SD substantially outperforms vanilla GRPO, skill-augmented GRPO, and pure on-policy distillation. The lesson: skills should guide the teacher, not the student. Off-policy variants suffer mid-training collapse despite strong early performance, and frozen teachers converge to lower plateaus, indicating that the co-evolution of teacher and student, rather than any single architectural choice, drives the gains.

Limitations

We discuss several directions for improvement. First, skill retrieval currently uses a lightweight UCB bandit criterion rather than embedding-based semantic retrieval; while this keeps the training pipeline simple and avoids introducing an additional model, a learned retrieval component may improve skill selection as the bank grows. Second, we use sampled-token distillation (evaluating only the generated token) rather than full-vocabulary distillation; although this is far cheaper for long-horizon traces, partial-vocabulary approximations may improve the fidelity of the self-evolution signal. Third, our gradient analysis is token-level and per-update; extending it to trajectory-level KL bounds under evolving teacher references remains an open theoretical question.

References

Agarwal et al. (2024)

Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem.

On-policy distillation of language models: Learning from self-generated mistakes.

In

International Conference on Learning Representations

, 2024.

Auer et al. (2002)

Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer.

Finite-time analysis of the multiarmed bandit problem.

Machine Learning

, 47(2–3):235–256, 2002.

Bengio et al. (2009)

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.

Curriculum learning.

In

Proceedings of the 26th International Conference on Machine Learning (ICML)

, pp.  41–48, 2009.

doi:

10.1145/1553374.1553380

.

Bytedance Seed (2026)

Bytedance Seed.

Seed1.8 model card: Towards generalized real-world agency.

arXiv preprint arXiv:2603.20633

, 2026.

Chen et al. (2025)

Kevin Chen, Marco Cusumano-Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl.

Reinforcement learning for long-horizon interactive LLM agents.

arXiv preprint arXiv:2502.01600

, 2025.

Cheng et al. (2025)

Mingyue Cheng, Jie Ouyang, Shuo Yu, Ruiran Yan, Yucong Luo, Zirui Liu, Daoyu Wang, Qi Liu, and Enhong Chen.

Agent-R1: Training powerful LLM agents with end-to-end reinforcement learning.

arXiv preprint arXiv:2511.14460

, 2025.

DeepSeek-AI (2025)

DeepSeek-AI.

DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.

arXiv preprint arXiv:2501.12948

, 2025.

Feng et al. (2025)

Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An.

Group-in-group policy optimization for LLM agent training.

In

Advances in Neural Information Processing Systems

, 2025.

Furlanello et al. (2018)

Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.

Born again neural networks.

In

Proceedings of the 35th International Conference on Machine Learning (ICML)

, pp.  1607–1616, 2018.

URL

https://proceedings.mlr.press/v80/furlanello18a.html

.

Hsieh et al. (2023)

Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister.

Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.

In

Findings of the Association for Computational Linguistics: ACL 2023

, pp.  8003–8017, 2023.

doi:

10.18653/v1/2023.findings-acl.507

.

URL

https://aclanthology.org/2023.findings-acl.507/

.

Hu et al. (2025)

Michael Y. Hu, Benjamin Van Durme, Jacob Andreas, and Harsh Jhamtani.

Sample-efficient online learning in LM agents via hindsight trajectory rewriting.

arXiv preprint arXiv:2510.10304

, 2025.

Hübotter et al. (2026)

Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause.

Reinforcement learning via self-distillation.

arXiv preprint arXiv:2601.20802

, 2026.

Liu et al. (2025a)

Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Liang Liu, Yaxuan Guo, Han Xiao, Weifeng Lin, Yuxiang Chai, Yue Han, Shuai Ren, et al.

Llm-powered gui agents in phone automation: Surveying progress and prospects.

arXiv preprint arXiv:2504.19838

, 2025a.

Liu et al. (2025b)

Guangyi Liu, Pengxiang Zhao, Liang Liu, Zhiming Chen, Yuxiang Chai, Shuai Ren, Hao Wang, Shibo He, and Wenchao Meng.

Learnact: Few-shot mobile gui agent with a unified demonstration benchmark.

arXiv preprint arXiv:2504.13805

, 2025b.

Liu et al. (2026)

Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Qinyi Luo, Shunye Tang, Yuxiang Chai, Weifeng Lin, Han Xiao, WenHao Wang, Siheng Chen, et al.

Memgui-bench: Benchmarking memory of mobile gui agents in dynamic environments.

arXiv preprint arXiv:2602.06075

, 2026.

Liu et al. (2025c)

Kezhao Liu, Jason Klein Liu, Mingtao Chen, and Yiming Liu.

Rethinking KL regularization in RLHF: From value estimation to gradient optimization.

arXiv preprint arXiv:2510.01555

, 2025c.

Penaloza et al. (2026)

Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia.

Privileged information distillation for language models.

arXiv preprint arXiv:2602.04942

, 2026.

Qwen Team (2025)

Qwen Team.

Qwen3 technical report.

arXiv preprint arXiv:2505.09388

, 2025.

Schrader (2018)

Max-Philipp Schrader.

gym-sokoban: Reinforcement learning environment for the game of sokoban.

https://github.com/mpSchrader/gym-sokoban

, 2018.

Schulman (2020)

John Schulman.

Approximating KL divergence.

http://joschu.net/blog/kl-approx.html

, 2020.

Blog post.

Schulman et al. (2017)

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.

Proximal policy optimization algorithms.

arXiv preprint arXiv:1707.06347

, 2017.

Shao et al. (2024)

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo.

DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.

arXiv preprint arXiv:2402.03300

, 2024.

Shinn et al. (2023)

Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.

Reflexion: Language agents with verbal reinforcement learning.

In

Advances in Neural Information Processing Systems

, 2023.

Snell et al. (2022)

Charlie Snell, Dan Klein, and Ruiqi Zhong.

Learning by distilling context.

arXiv preprint arXiv:2209.15189

, 2022.

doi:

10.48550/ARXIV.2209.15189

.

Tang & Munos (2025)

Yunhao Tang and Rémi Munos.

On a few pitfalls in KL divergence gradient estimation for RL.

arXiv preprint arXiv:2506.09477

, 2025.

Trivedi et al. (2024)

Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian.

AppWorld: A controllable world of apps and people for benchmarking interactive coding agents.

In

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics

, pp.  16022–16076, 2024.

Vapnik & Vashist (2009)

Vladimir Vapnik and Akshay Vashist.

A new learning paradigm: Learning using privileged information.

Neural Networks

, 22(5–6):544–557, 2009.

doi:

10.1016/j.neunet.2009.06.042

.

Wang & Ammanabrolu (2025)

Ruiyi Wang and Prithviraj Ammanabrolu.

A practitioner’s guide to multi-turn agentic reinforcement learning.

arXiv preprint arXiv:2510.01132

, 2025.

Wang et al. (2026)

Yinjie Wang, Xuyang Chen, Xiaolong Jin, Mengdi Wang, and Ling Yang.

OpenClaw-RL: Train any agent simply by talking.

arXiv preprint arXiv:2603.10165

, 2026.

Wang et al. (2025)

Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Fei-Fei Li, Lijuan Wang, Yejin Choi, and Manling Li.

RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning.

arXiv preprint arXiv:2504.20073

, 2025.

Wu et al. (2025)

Rong Wu, Xiaoman Wang, Jianbiao Mei, Pinlong Cai, Daocheng Fu, Cheng Yang, Licheng Wen, Xuemeng Yang, Yufan Shen, Yuxin Wang, and Botian Shi.

EvolveR: Self-evolving LLM agents through an experience-driven lifecycle.

arXiv preprint arXiv:2510.16079

, 2025.

Xi et al. (2026)

Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, Wei He, Yiwen Ding, Guanyu Li, Zehui Chen, Zhengyin Du, Xuesong Yao, Yufei Xu, Jiecao Chen, Tao Gui, Zuxuan Wu, Qi Zhang, Xuanjing Huang, and Yu-Gang Jiang.

AgentGym-RL: Training LLM agents for long-horizon decision making through multi-turn reinforcement learning.

In

International Conference on Learning Representations

, 2026.

Xiao et al. (2025)

Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, et al.

Ui-genie: A self-improving approach for iteratively boosting mllm-based mobile gui agents.

arXiv preprint arXiv:2505.21496

, 2025.

Xiao et al. (2026)

Han Xiao, Guozhi Wang, Hao Wang, Shilong Liu, Yuxiang Chai, Yue Pan, Yufeng Zhou, Xiaoxin Chen, Yafei Wen, and Hongsheng Li.

Ui-mem: Self-evolving experience memory for online reinforcement learning in mobile gui agents.

arXiv preprint arXiv:2602.05832

, 2026.

Xu et al. (2025)

Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, and Fei Mi.

KDRL: Post-training reasoning LLMs via unified knowledge distillation and reinforcement learning.

arXiv preprint arXiv:2506.02208

, 2025.

Yan et al. (2025)

Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang.

Learning to reason under off-policy guidance.

In

Advances in Neural Information Processing Systems

, 2025.

Yao et al. (2022)

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan.

WebShop: Towards scalable real-world web interaction with grounded language agents.

In

Advances in Neural Information Processing Systems

, pp.  20744–20757, 2022.

Yao et al. (2023)

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.

ReAct: Synergizing reasoning and acting in language models.

In

International Conference on Learning Representations

, 2023.

Ye et al. (2026a)

Tianzhu Ye, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei.

Online experiential learning for language models.

arXiv preprint arXiv:2603.16856

, 2026a.

Ye et al. (2026b)

Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei.

On-policy context distillation for language models.

arXiv preprint arXiv:2602.12275

, 2026b.

Yu et al. (2024)

Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov.

Distilling system 2 into system 1.

arXiv preprint arXiv:2407.06023

, 2024.

Yu et al. (2025)

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang.

DAPO: An open-source LLM reinforcement learning system at scale.

In

Advances in Neural Information Processing Systems

, 2025.

Yuan et al. (2025)

Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen.

Agent-R: Training language model agents to reflect via iterative self-training.

arXiv preprint arXiv:2501.11425

, 2025.

Zhai et al. (2025)

Yunpeng Zhai et al.

AgentEvolver: Towards efficient self-evolving agent system.

arXiv preprint arXiv:2511.10395

, 2025.

Zhang et al. (2026a)

Hanchen Zhang et al.

AgentRL: Scaling agentic reinforcement learning with a multi-turn, multi-task framework.

In

International Conference on Learning Representations

, 2026a.

Zhang et al. (2026b)

Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Yang Yuan, Quanquan Gu, and Andrew Chi-Chih Yao.

On the design of KL-regularized policy gradient algorithms for LLM reasoning.

In

International Conference on Learning Representations

, 2026b.

Zhang et al. (2026c)

Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, and Stefano Soatto.

Reinforcement-aware knowledge distillation for LLM reasoning.

arXiv preprint arXiv:2602.22495

, 2026c.

Zhao et al. (2024)

Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang.

ExpeL: LLM agents are experiential learners.

In

Proceedings of the AAAI Conference on Artificial Intelligence

, pp.  19632–19642, 2024.

Zhao et al. (2025)

Pengxiang Zhao, Guangyi Liu, Yaozhen Liang, Weiqing He, Zhengxi Lu, Yuehao Huang, Yaxuan Guo, Kexin Zhang, Hao Wang, Liang Liu, et al.

Mas-bench: A unified benchmark for shortcut-augmented hybrid mobile gui agents.

arXiv preprint arXiv:2509.06477

, 2025.

Zhao et al. (2026)

Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover.

Self-distilled reasoner: On-policy self-distillation for large language models.

arXiv preprint arXiv:2601.18734

, 2026.

Appendix A

Skill Format

Each skill is a structured JSON object with three fields summarizing a completed trajectory. During training, skills are prepended to the teacher prompt to provide task-specific guidance.

Example Skill

success_analysis:

“The strategic intent to use specialized apps (Splitwise for identifying roommates, Venmo for sending payment requests) and prioritize authentication for restricted APIs was sound, as these tools are designed to streamline expense splitting and peer-to-peer payments.”

mistake_analysis:

“Key failures stemmed from unvalidated assumptions (using a hypothetical bill amount instead of retrieving the actual receipt from the file system) and neglecting to verify API parameter requirements upfront, leading to repeated authentication errors.”

golden_workflow:

“1. Access the file system to retrieve the actual internet bill amount from the stored receipt. 2. Authenticate with a contact app using valid credentials to obtain roommate emails. 3. Authenticate with Venmo, calculate each roommate’s equal share, then send payment requests via the correct API.”

Appendix B

Trajectory Example and Skill Generation

We present a complete AppWorld interaction trajectory and the skill generated from it. The task requires exporting all Spotify library data to a CSV file, then terminating the account. This trajectory was collected during training and is shown with minimal editing (long API outputs are truncated with

[...]

).

Task:

Export a unique list of all the songs in my song and album library and all playlists in my Spotify account into “~/backups/spotify.csv”. The file should have headers “Title” and “Artists” (artists separated by “|”). Terminate my account after this backup is complete.

Agent – Step 1

To solve this task, we need to: (1) Access the Spotify account using login. (2) Retrieve the full list of songs, albums, and playlists. (3) Extract song titles and artists. (4) Combine into a unique list. (5) Export to CSV. (6) Terminate the account.

passwords = apis.supervisor.show_account_passwords()

Environment – Step 1

[{"account_name": "spotify", "password": "CgR}U-F"}, {"account_name": "file_system", "password": "+ok}#yD"}, ...]

Agent – Step 2

The Spotify password is

"CgR}U-F"

. I will now log in to Spotify.

login_result = apis.spotify.login(username=’ka_burt@gmail.com’, password=spotify_password)

Environment – Step 2

{"access_token": "eyJhb...", "token_type": "Bearer"}

Agent – Step 3

Login was successful. Now let me check the available Spotify APIs.

print(apis.api_docs.show_api_descriptions(app_name=’spotify’))

Environment – Step 3

[show_song_library, show_album_library, show_playlist_library, show_song, delete_account, ...]

[74 API endpoints; truncated]

Agent – Step 4

The relevant APIs are

show_song_library

,

show_album_library

,

show_playlist_library

, and

show_song

. Let me retrieve the song library first.

song_library = apis.spotify.show_song_library(access_token=spotify_access_token)

Environment – Step 4

Returns 5 songs with titles and artists. E.g.,

{"song_id": 202, "title": "Summer’s End", "artists": [{"name": "Ethan Wallace"}]}

, …

Agent – Step 5

Now let me retrieve the album library and playlist library.

album_library = apis.spotify.show_album_library(access_token=...)

playlists = apis.spotify.show_playlist_library(access_token=...)

Environment – Step 5

Returns 5 albums (16 song IDs total) and 5 playlists (39 song IDs total).

Agent – Step 6

Now I need to fetch details for each song ID from albums and playlists using

show_song

.

song = apis.spotify.show_song(access_token=spotify_access_token, song_id=song_id)

Environment – Step 6:

Error

Exception: Unexpected parameter ’access_token’ passed to the show_song API. Allowed parameters are: [’song_id’]

Agent – Step 7

I made a mistake in the API call. Let me check the API specification for

show_song

.

print(apis.api_docs.show_api_doc(app_name=’spotify’, api_name=’show_song’))

Environment – Step 7

{"parameters": [{"name": "song_id", "type": "integer", "required": true}]}

— confirms

show_song

only takes

song_id

, no

access_token

.

Agent – Step 8

Confirmed. I will fix the code to call

show_song(song_id=...)

without

access_token

, and collect all songs from song library, albums, and playlists into a deduplicated list.

song = apis.spotify.show_song(song_id=song_id)

[loops over all song IDs]

Environment – Step 8

59

— 59 unique songs extracted successfully.

Agent – Step 9

59 unique songs collected. Now I need to write the CSV. Let me check the

file_system

APIs and log in.

file_system_login = apis.file_system.login(password=file_system_password)

Environment – Step 9:

Error

Validation error. Reason: username: Field required

Agent – Step 10

The

file_system.login

requires both

username

and

password

. Let me check the API docs.

print(apis.api_docs.show_api_doc(app_name=’file_system’, api_name=’login’))

Environment – Step 10

Confirms:

username

(string, required) and

password

(string, required).

Agent – Step 11

Now I will log in with both username and password, create the CSV, and write it.

apis.file_system.login(username=’ka_burt@gmail.com’, password=...)

apis.file_system.create_file(file_path=’~/backups/spotify.csv’, content=csv_content, ...)

Environment – Step 11

CSV file created successfully.

Agent – Step 12

Backup complete. Now terminate the Spotify account.

apis.spotify.delete_account(access_token=spotify_access_token)

Environment – Step 12

Spotify account terminated successfully.

Agent – Step 13

All tasks completed. Marking as done.

apis.supervisor.complete_task()

Environment – Step 13

Execution successful.

Generated skill.

The following skill is produced from this trajectory by the auxiliary summarizer:

Generated Skill

success_analysis:

“The strategic approach of systematically retrieving data from song library, album library, and playlists before combining into a deduplicated export was sound, as was the correct sequencing of backup-then-terminate.”

mistake_analysis:

“Two API parameter errors occurred: (1) passing

access_token

to

show_song

, which only accepts

song_id

; (2) omitting

username

when calling

file_system.login

. Both stemmed from not consulting API specifications before the first call.”

golden_workflow:

“1. Retrieve credentials via supervisor. 2. Log in to Spotify and collect song/album/playlist libraries. 3. For each song ID, call

show_song(song_id=...)

without extra parameters. 4. Log in to file system with both username and password, then write deduplicated CSV. 5. Delete Spotify account and mark task complete.”

Appendix C

Prompt Templates

The AppWorld system prompt is adapted from the official AppWorld benchmark

(Trivedi et al.,

2024

)

. The full prompt including all API specifications is available in the benchmark repository.

AppWorld System Prompt (abbreviated)

I am your supervisor and you are a super intelligent AI Assistant whose job is to achieve my day-to-day tasks completely autonomously.

To do this, you will need to interact with app/s (e.g., spotify, venmo etc) using their associated APIs on my behalf. For this you will undertake a

multi-step conversation

using a python REPL environment.

Here are three key APIs that you need to know to get more information

# To get a list of apps that are available to you.

print(apis.api_docs.show_app_descriptions())

# To get the list of apis under any app, e.g. spotify

print(apis.api_docs.show_api_descriptions(app_name=’spotify’))

# To get the specification of a particular api

print(apis.api_docs.show_api_doc(app_name=’spotify’, api_name=’login’))

Each code execution will produce an output that you can use in subsequent calls. Using these APIs, you can now generate code, that I will execute, to solve the task. Let’s start with the task

My name is:

{main_user.first_name}

{main_user.last_name}

. My personal email is

{main_user.email}

and phone number is

{main_user.phone_number}

.

Task: How many playlists do I have in Spotify?

[

A multi-turn few-shot demonstration follows, showing authentication, API discovery, pagination, and calling

apis.supervisor.complete_task()

. See the AppWorld benchmark repository for the full prompt.

]

Key instructions:

(1) Make sure to end code blocks with

```

followed by a newline.

(2) Remember you can use the variables in your code in subsequent code blocks.

(3) Always look at API specifications before calling an API.

(4) Write small chunks of code and only one chunk per step.

(5) Many APIs return items in “pages”. Loop over

page_index

.

(6) Once you have completed the task, call

apis.supervisor.complete_task()

.

My name is:

{first_name}

{last_name}

. My personal email is

{email}

and phone number is

{phone}

.

Task:

{input_str}

Sokoban System Prompt

You are an expert agent operating in the Sokoban environment.

# Symbols and Their Meaning

- Walls (

#

): These block movement. You can’t move through or push anything into walls.

- Floor (

_

): Open spaces where you can walk and move boxes.

- Targets (

O

): The spots where boxes need to go.

- Boxes (

X

): These are what you need to push onto the targets.

- Player (

P

): That’s you! You’ll move around the grid to push boxes.

- Box on Target (

T

): A box successfully placed on a target.

- Player on Target (

S

): You standing on a target.

# Your Goal

Your goal is to push all the boxes (

X

) onto the target spots (

O

). Once all boxes are on the targets, you win!

# Rules

1. You can only move up, down, left, or right. And don’t overthink it.

2. You can only push boxes. You can’t pull them, so plan ahead to avoid getting stuck.

3. You can’t walk through or push boxes into walls (

#

).

4. To avoid traps, do not push boxes into corners or against walls where they can’t be moved again.

You will be provided the current map observation.

After your thought process, you MUST output your final single action enclosed in

``` ```

.

Valid Actions:

up

|

|

down

|

|

left

|

|

right

Skill Generation Prompt

# Trajectory Analysis and Golden Skill Generation

You are an expert trajectory analyst. Your task is to analyze the given trajectory and generate a structured golden skill that captures key learnings.

## Conversation History

{conversation_history}

## Trajectory Outcome

-

Result

:

{result_status}

-

Total Steps

:

{step_count}

-

Completion Rate

:

{completion_rate}

## Output Requirements (JSON)

You need to deeply analyze the Trajectory Outcome and Conversation History I provided, thinking step by step, and then summarize the skill to guide the model in future encounters.

Output a unique JSON object containing exactly these 3 fields:

1.

success_analysis

: A comprehensive paragraph summarizing the

strategic intent

that led (or would lead) to success.

2.

mistake_analysis

: A comprehensive paragraph analyzing the

root causes

of failures or inefficiencies observed in the trajectory.

3.

golden_workflow

: A concise description of the

ideal overall process

to complete this task from start to finish.

## REMEMBER

The analysis must be concise. As an expert, the skill you summarize will guide the model’s subsequent operations. Therefore, it’s crucial to adopt a guiding approach, focusing primarily on the steps that confuse the model.

Your success-to-failure output ratio should follow the Completion Rate (

{completion_rate}

). If the Completion Rate is high, you should focus on summarizing the success analysis; otherwise, you should focus on summarizing the mistake analysis.

Appendix D

Importance-Weighted SDL under Student and Teacher Rollouts

This appendix formalizes the sampled-token self-distillation term used in Eq. (

12

). The analysis follows the modern distinction between

KL value estimation

and

KL gradient optimization

: a bare

k

3

k_{3}

estimator can be a good value estimator while still producing the wrong gradient when differentiated directly

(Schulman,

2020

; Tang & Munos,

2025

; Liu et al.,

2025c

)

. That distinction matters here because the auxiliary loss is interpreted as self-evolution. If its gradient is biased under the actual sampling policy, then the loss may still move the model, but it no longer cleanly implements the intended teacher-to-student transfer direction.

D.1

Estimator family

Let

q

θ

q_{\theta}

denote the current student distribution,

p

p

a fixed teacher reference distribution, and

μ

\mu

the actual sampling distribution at a fixed token position. Define

ℓ

=

log

⁡

q

θ

​

(

y

)

p

​

(

y

)

.

\ell=\log\frac{q_{\theta}(y)}{p(y)}.

(15)

The standard reverse-KL estimator family includes

k

1

​

(

ℓ

)

\displaystyle k_{1}(\ell)

=

ℓ

,

\displaystyle=\ell,

(16)

k

2

​

(

ℓ

)

\displaystyle k_{2}(\ell)

=

1

2

​

ℓ

2

,

\displaystyle=\tfrac{1}{2}\ell^{2},

(17)

k

3

​

(

ℓ

)

\displaystyle k_{3}(\ell)

=

e

−

ℓ

−

1

+

ℓ

.

\displaystyle=e^{-\ell}-1+\ell.

(18)

Among these,

k

3

k_{3}

is attractive because it is non-negative and is widely used as a low-variance reverse-KL estimator

(Schulman,

2020

)

. But recent analyses show that differentiating

k

3

k_{3}

directly does not in general yield the desired reverse-KL gradient

(Tang & Munos,

2025

; Liu et al.,

2025c

)

.

D.2

General sampled-token SDL

Definition 1

(Importance-weighted SDL)

.

For a fixed prefix, let

y

t

∼

μ

(

⋅

∣

y

<

t

)

y_{t}\sim\mu(\cdot\mid y_{<t})

. Define

ℓ

t

\displaystyle\ell_{t}

≜

log

⁡

q

θ

​

(

y

t

∣

y

<

t

)

−

log

⁡

p

​

(

y

t

∣

y

<

t

)

,

\displaystyle\triangleq\log q_{\theta}(y_{t}\mid y_{<t})-\log p(y_{t}\mid y_{<t}),

(19)

ρ

t

μ

\displaystyle\rho_{t}^{\mu}

≜

q

θ

​

(

y

t

∣

y

<

t

)

sg

⁡

(

μ

​

(

y

t

∣

y

<

t

)

)

,

\displaystyle\triangleq\frac{q_{\theta}(y_{t}\mid y_{<t})}{\operatorname{sg}\!\left(\mu(y_{t}\mid y_{<t})\right)},

(20)

where

sg

⁡

(

⋅

)

\operatorname{sg}(\cdot)

denotes stop-gradient.

The choice of

ρ

t

μ

\rho_{t}^{\mu}

is structurally forced. Its denominator must match the actual sampling distribution

μ

\mu

, otherwise the importance correction does not close under the sampling expectation. Its numerator must be the trainable student, otherwise the product rule would mix fixed-reference terms with the wrong score function and the resulting gradient would not reduce to the intended reverse-KL form.

Proposition 2

(Per-token SDL gradient identity)

.

Under Definition

1

,

∇

θ

(

ρ

t

μ

⋅

k

3

​

(

ℓ

t

)

)

=

ρ

t

μ

⋅

s

θ

​

(

y

t

)

⋅

ℓ

t

,

\nabla_{\theta}\!\bigl(\rho_{t}^{\mu}\cdot k_{3}(\ell_{t})\bigr)=\rho_{t}^{\mu}\cdot s_{\theta}(y_{t})\cdot\ell_{t},

(21)

where

s

θ

​

(

y

t

)

=

∇

θ

log

⁡

q

θ

​

(

y

t

∣

y

<

t

)

s_{\theta}(y_{t})=\nabla_{\theta}\log q_{\theta}(y_{t}\mid y_{<t})

. Moreover, for any fixed prefix

y

<

t

y_{<t}

,

𝔼

y

t

∼

μ

(

⋅

∣

y

<

t

)

[

∇

θ

(

ρ

t

μ

⋅

k

3

(

ℓ

t

)

)

]

=

∇

θ

D

KL

(

q

θ

(

⋅

∣

y

<

t

)

∥

p

(

⋅

∣

y

<

t

)

)

.

\mathbb{E}_{y_{t}\sim\mu(\cdot\mid y_{<t})}\!\left[\nabla_{\theta}\!\bigl(\rho_{t}^{\mu}\cdot k_{3}(\ell_{t})\bigr)\right]=\nabla_{\theta}D_{\mathrm{KL}}\!\bigl(q_{\theta}(\cdot\mid y_{<t})\,\|\,p(\cdot\mid y_{<t})\bigr).

(22)

Proof.

We apply the product rule:

∇

θ

(

ρ

t

μ

⋅

k

3

)

=

(

∇

θ

ρ

t

μ

)

⋅

k

3

+

ρ

t

μ

⋅

(

∇

θ

k

3

)

.

\nabla_{\theta}(\rho_{t}^{\mu}\cdot k_{3})=(\nabla_{\theta}\rho_{t}^{\mu})\cdot k_{3}+\rho_{t}^{\mu}\cdot(\nabla_{\theta}k_{3}).

(23)

Because the denominator of

ρ

t

μ

\rho_{t}^{\mu}

is frozen under stop-gradient,

∇

θ

ρ

t

μ

=

ρ

t

μ

⋅

s

θ

​

(

y

t

)

.

\nabla_{\theta}\rho_{t}^{\mu}=\rho_{t}^{\mu}\cdot s_{\theta}(y_{t}).

(24)

Since

log

⁡

p

​

(

y

t

∣

y

<

t

)

\log p(y_{t}\mid y_{<t})

is computed at the pre-update parameters and does not depend on the current

θ

\theta

,

∇

θ

ℓ

t

=

s

θ

​

(

y

t

)

\nabla_{\theta}\ell_{t}=s_{\theta}(y_{t})

. Using

k

3

′

​

(

ℓ

)

=

1

−

e

−

ℓ

k_{3}^{\prime}(\ell)=1-e^{-\ell}

, we obtain

∇

θ

k

3

​

(

ℓ

t

)

=

(

1

−

e

−

ℓ

t

)

⋅

s

θ

​

(

y

t

)

.

\nabla_{\theta}k_{3}(\ell_{t})=(1-e^{-\ell_{t}})\cdot s_{\theta}(y_{t}).

(25)

Substituting both expressions gives

∇

θ

(

ρ

t

μ

⋅

k

3

)

\displaystyle\nabla_{\theta}(\rho_{t}^{\mu}\cdot k_{3})

=

ρ

t

μ

⋅

s

θ

​

(

y

t

)

⋅

k

3

​

(

ℓ

t

)

+

ρ

t

μ

⋅

(

1

−

e

−

ℓ

t

)

⋅

s

θ

​

(

y

t

)

\displaystyle=\rho_{t}^{\mu}\cdot s_{\theta}(y_{t})\cdot k_{3}(\ell_{t})+\rho_{t}^{\mu}\cdot(1-e^{-\ell_{t}})\cdot s_{\theta}(y_{t})

(26)

=

ρ

t

μ

⋅

s

θ

​

(

y

t

)

​

[

k

3

​

(

ℓ

t

)

+

1

−

e

−

ℓ

t

]

.

\displaystyle=\rho_{t}^{\mu}\cdot s_{\theta}(y_{t})\left[k_{3}(\ell_{t})+1-e^{-\ell_{t}}\right].

(27)

Using

k

3

​

(

ℓ

t

)

=

e

−

ℓ

t

−

1

+

ℓ

t

k_{3}(\ell_{t})=e^{-\ell_{t}}-1+\ell_{t}

, the bracket simplifies to

ℓ

t

\ell_{t}

, proving Eq. (

21

). Taking expectation under the sampling distribution yields

𝔼

y

t

∼

μ

(

⋅

∣

y

<

t

)

​

[

∇

θ

(

ρ

t

μ

⋅

k

3

)

]

\displaystyle\mathbb{E}_{y_{t}\sim\mu(\cdot\mid y_{<t})}\!\left[\nabla_{\theta}(\rho_{t}^{\mu}\cdot k_{3})\right]

=

∑

y

t

μ

​

(

y

t

∣

y

<

t

)

​

q

θ

​

(

y

t

∣

y

<

t

)

μ

​

(

y

t

∣

y

<

t

)

​

s

θ

​

(

y

t

)

​

ℓ

t

\displaystyle=\sum_{y_{t}}\mu(y_{t}\mid y_{<t})\frac{q_{\theta}(y_{t}\mid y_{<t})}{\mu(y_{t}\mid y_{<t})}s_{\theta}(y_{t})\ell_{t}

(28)

=

∑

y

t

q

θ

​

(

y

t

∣

y

<

t

)

​

s

θ

​

(

y

t

)

​

ℓ

t

\displaystyle=\sum_{y_{t}}q_{\theta}(y_{t}\mid y_{<t})s_{\theta}(y_{t})\ell_{t}

(29)

=

∇

θ

D

KL

(

q

θ

(

⋅

∣

y

<

t

)

∥

p

(

⋅

∣

y

<

t

)

)

,

\displaystyle=\nabla_{\theta}D_{\mathrm{KL}}\!\bigl(q_{\theta}(\cdot\mid y_{<t})\,\|\,p(\cdot\mid y_{<t})\bigr),

(30)

which proves the conditional unbiasedness claim.
∎

D.3

On-policy and off-policy corollaries

On-policy main method.

For the main Skill-SD update,

μ

=

π

θ

old

stu

(

⋅

∣

x

,

y

<

t

)

\mu=\pi_{\theta_{\text{old}}}^{\text{stu}}(\cdot\mid x,y_{<t})

and

p

=

π

θ

¯

tea

(

⋅

∣

x

,

S

(

x

)

,

y

<

t

)

p=\pi_{\bar{\theta}}^{\text{tea}}(\cdot\mid x,S(x),y_{<t})

. Therefore

ρ

t

on

=

π

θ

stu

​

(

y

t

∣

x

,

y

<

t

)

sg

⁡

(

π

θ

old

stu

​

(

y

t

∣

x

,

y

<

t

)

)

\rho_{t}^{\text{on}}=\frac{\pi_{\theta}^{\text{stu}}(y_{t}\mid x,y_{<t})}{\operatorname{sg}\!\left(\pi_{\theta_{\text{old}}}^{\text{stu}}(y_{t}\mid x,y_{<t})\right)}

(31)

is the importance weight for the main self-evolution signal. Numerically,

ρ

t

on

\rho_{t}^{\text{on}}

equals the GRPO ratio

r

t

r_{t}

, but the role is different:

ρ

t

on

\rho_{t}^{\text{on}}

weights the auxiliary reverse-KL term, whereas

r

t

r_{t}

defines the clipped reinforcement-learning surrogate.

Off-policy comparison branch.

For the teacher-rollout comparison branch,

μ

=

π

θ

old

tea

(

⋅

∣

x

,

S

(

x

)

,

y

<

t

)

\mu=\pi_{\theta_{\text{old}}}^{\text{tea}}(\cdot\mid x,S(x),y_{<t})

, which yields

ρ

t

off

=

π

θ

stu

​

(

y

t

∣

x

,

y

<

t

)

sg

⁡

(

π

θ

old

tea

​

(

y

t

∣

x

,

S

​

(

x

)

,

y

<

t

)

)

.

\rho_{t}^{\text{off}}=\frac{\pi_{\theta}^{\text{stu}}(y_{t}\mid x,y_{<t})}{\operatorname{sg}\!\left(\pi_{\theta_{\text{old}}}^{\text{tea}}(y_{t}\mid x,S(x),y_{<t})\right)}.

(32)

This is the off-policy correction emphasized in RPG-style analyses

(Zhang et al.,

2026b

)

: without it, the differentiated

k

3

k_{3}

term would not follow the intended reverse-KL transfer direction under teacher sampling.

D.4

Why SDL and GRPO need distinct interpretations

Skill-SD uses two different quantities for two different jobs:

•

The main-branch SDL weight

ρ

t

on

=

π

θ

stu

/

π

θ

old

stu

\rho_{t}^{\text{on}}=\pi_{\theta}^{\text{stu}}/\pi_{\theta_{\text{old}}}^{\text{stu}}

must match the

student sampling distribution

.

•

The off-policy SDL weight

ρ

t

off

=

π

θ

stu

/

π

θ

old

tea

\rho_{t}^{\text{off}}=\pi_{\theta}^{\text{stu}}/\pi_{\theta_{\text{old}}}^{\text{tea}}

must match the

teacher sampling distribution

.

•

The GRPO clipping ratio

r

t

=

π

θ

stu

/

π

θ

old

stu

r_{t}=\pi_{\theta}^{\text{stu}}/\pi_{\theta_{\text{old}}}^{\text{stu}}

must stay centered at the

old student policy

, even when we apply DAPO-style clip-higher asymmetric bounds

(Yu et al.,

2025

)

.

If one naively reused the teacher-denominator ratio for clipping, the trust region would no longer be centered at

1

1

because the teacher and student differ by prompt conditioning. This is exactly the type of mis-centering that recent off-policy analyses warn against

(Yan et al.,

2025

; Zhang et al.,

2026b

)

. In the main on-policy branch,

ρ

t

on

\rho_{t}^{\text{on}}

happens to equal

r

t

r_{t}

numerically, but the interpretation is still different: the former defines the self-evolution update, whereas the latter defines the clipped RL surrogate.

Table 3:

Importance-weighted SDL and student-centered GRPO in Skill-SD.

Quantity

Definition

Purpose

Main SDL weight

ρ

t

on

\rho_{t}^{\text{on}}

π

θ

stu

/

sg

⁡

(

π

θ

old

stu

)

\pi_{\theta}^{\text{stu}}/\operatorname{sg}(\pi_{\theta_{\text{old}}}^{\text{stu}})

Match student sampling in the main method

Off-policy SDL weight

ρ

t

off

\rho_{t}^{\text{off}}

π

θ

stu

/

sg

⁡

(

π

θ

old

tea

)

\pi_{\theta}^{\text{stu}}/\operatorname{sg}(\pi_{\theta_{\text{old}}}^{\text{tea}})

Match teacher sampling in the comparison branch

GRPO clipping ratio

r

t

r_{t}

π

θ

stu

/

sg

⁡

(

π

θ

old

stu

)

\pi_{\theta}^{\text{stu}}/\operatorname{sg}(\pi_{\theta_{\text{old}}}^{\text{stu}})

Center trust region at old student

Appendix E

Additional Experimental Details

Variant definition.

Table

4

summarizes the exact axes varied in the current AppWorld study.

Table 4:

Variant definitions for the AppWorld ablation matrix.

Variant

Rollout Source

Teacher Refresh

Distillation

Off-policy + dynamic

Teacher

Dynamic self-evolution

Yes

Off-policy + frozen

Teacher

Frozen

Yes

On-policy + dynamic

Student

Dynamic self-evolution

Yes

On-policy + frozen

Student

Frozen

Yes

Skill-Augmented GRPO

Skill-conditioned student

N/A

No

Vanilla GRPO

Student

N/A

No

Vanilla OPD

Teacher/student re-forward pair

Frozen

Yes, no RL

Baseline

Qwen3-4B-Instruct-2507

N/A

No

AppWorld protocol.

We use the official AppWorld train/dev split

(Trivedi et al.,

2024

)

. Accuracy is pass@1 on the dev set. Completion rate is the fraction of task-specific unit tests satisfied by the final environment state.

Loss reduction and clipping.

Both

ℒ

GRPO

\mathcal{L}_{\text{GRPO}}

and

ℒ

SDL

\mathcal{L}_{\text{SDL}}

use

verl

’s

token-mean

reduction: losses are summed over valid action tokens and divided by the total valid-token count. For GRPO, we use DAPO’s clip-higher setting

(Yu et al.,

2025

)

, i.e., asymmetric clipping with

ε

h

>

ε

l

\varepsilon_{h}>\varepsilon_{l}

.

Appendix F

Training Hyperparameters

We implement Skill-SD using the rLLM framework

1

1

1

https://github.com/rllm-org/rllm

with the verl backend

2

2

2

https://github.com/volcengine/verl

. Trajectory summarization into skills is performed by Seed1.8

(Bytedance Seed,

2026

)

. GRPO follows

Shao et al. (

2024

)

. Multi-turn rollouts use a

Token-In-Token-Out

(TITO) mode, where the model reads and generates raw tokens rather than going through the chat completion API. This avoids the token-ID inconsistencies that arise when applying chat templates to multi-turn message histories—a discrepancy that can cause the training distribution to diverge from the rollout distribution and destabilize advantage estimation.

Table 5:

Training hyperparameters for AppWorld and Sokoban experiments.

Hyperparameter

Value

Group size

G

G

(rollouts per task)

4

Train batch size

16

Optimization epochs per batch

1

Max prompt length

8192 tokens

Max response length

20480 tokens

Dynamic batch sizing

Enabled

Ulysses sequence parallel size

4

Max tokens per GPU

14336

Training temperature

1.0

Evaluation temperature

0.0 (greedy)

GPU memory utilization

0.7

Entropy coefficient

0.0

DAPO clip

ε

l

\varepsilon_{l}

0.2

DAPO clip

ε

h

\varepsilon_{h}

0.28

Advantage normalization

ε

\varepsilon

1

×

10

−

8

1\times 10^{-8}

Learning rate

1

×

10

−

6

1\times 10^{-6}

Weight decay

0.0

UCB exploration coefficient

c

c

2

\sqrt{2}

SDL coefficient

λ

\lambda

0.001

0.001

Optimizer

AdamW

GPU

8

×

8{\times}

NVIDIA L40S

The UCB exploration coefficient

c

=

2

c=\sqrt{2}

follows the UCB1 algorithm of

Auer et al. (

2002

)

, which is theoretically optimal for rewards bounded in

[

0

,

1

]

[0,1]

via Hoeffding’s inequality, guaranteeing logarithmic regret.

BETA