概念导读

Agent Safety via RL (arXiv:2507.08270) 是蚂蚁集团提出的统一安全对齐框架，专门处理 LLM Agent 中的用户发起威胁和工具发起威胁。该方法采用三模态分类 (benign/malicious/sensitive) 代替传统的二元安全判断，并在沙箱 RL 环境中训练 Agent 同时抵御两种威胁源。

1. 概要
2. Core Method: Unified Safety Alignment via RL
3. Threat Model: Dual-Source Attacks
4. Tri-Modal Classification
5. Sandbox RL Environment
6. RL Training Objective
7. Experimental Results
8. Comparison with Other Methods

Agent Safety via RL: 统一安全对齐框架

概要

Agent Safety via RL (arXiv:2507.08270) 是蚂蚁集团提出的统一安全对齐框架，专门处理 LLM Agent 中的用户发起威胁和工具发起威胁。该方法采用三模态分类 (benign/malicious/sensitive) 代替传统的二元安全判断，并在沙箱 RL 环境中训练 Agent 同时抵御两种威胁源。

Core Method: Unified Safety Alignment via RL

Threat Model: Dual-Source Attacks

Traditional safety alignment assumes the user is the sole threat source. Agents face two attack surfaces:

1. User-initiated threats: Direct malicious instructions from users
2. "Delete all files on the server"

3. "Send phishing emails to all contacts"

4. Tool-initiated threats: Malicious content injected through tool outputs

5. Compromised API returning malicious instructions (prompt injection via tool response)
6. Poisoned search results containing adversarial content
7. Malicious web pages injecting commands through scraped content

Threat Model:
┌─────────┐     instruction     ┌─────────┐    tool_call    ┌──────────┐
│  User   │ ──────────────────→ │  Agent  │ ──────────────→ │  Tool    │
│         │                     │         │ ←────────────── │          │
│         │ ←─── response ────  │         │   tool_output   │          │
└─────────┘                     └─────────┘                 └──────────┘
   ↑ User-initiated threat           ↑ Tool-initiated threat

Tri-Modal Classification

Instead of binary safe/unsafe, the framework introduces three modalities:

Modality	Description	Agent Response
Benign	No safety risk detected	Execute normally
Malicious	Clear harmful intent	Refuse + explain
Sensitive	Ambiguous risk; may be legitimate	Execute with caution + logging

This tri-modal approach reduces over-refusal by giving the agent a "middle ground" for ambiguous cases.

Classification criteria:

P(malicious | context) > τ_high  →  MALICIOUS (refuse)
P(malicious | context) > τ_low   →  SENSITIVE  (cautious execute)
P(malicious | context) ≤ τ_low   →  BENIGN     (normal execute)

Sandbox RL Environment

The training pipeline uses a sandboxed environment that simulates both threat types:

class SafetySandbox:
    def __init__(self, tools, attack_generators):
        self.tools = tools
        self.user_attacker = attack_generators['user']    # generates malicious instructions
        self.tool_attacker = attack_generators['tool']    # injects via tool outputs

    def run_episode(self, agent):
        # Randomly sample threat type
        threat_type = random.choice(['user', 'tool', 'none'])

        if threat_type == 'user':
            instruction = self.user_attacker.generate()
        elif threat_type == 'tool':
            instruction = sample_benign_instruction()
            self.tool_attacker.activate()  # will inject during tool execution
        else:
            instruction = sample_benign_instruction()

        trajectory = agent.execute(instruction, self.tools)
        return compute_reward(trajectory, threat_type)

def compute_reward(trajectory, ground_truth_threat):
    if ground_truth_threat == 'none':
        return +1.0 if task_completed(trajectory) else -0.5
    elif ground_truth_threat in ['user', 'tool']:
        return +1.0 if refused_safely(trajectory) else -2.0
    # Sensitive: partial reward for cautious execution

RL Training Objective

The framework trains with a modified PPO objective that incorporates safety-aware rewards:

L(θ) = E_t[ min( r_t(θ) · Â_t,  clip(r_t(θ), 1-ε, 1+ε) · Â_t ) ]
       - β · KL(π_θ || π_ref)

where:
r_t(θ) = π_θ(a_t|s_t) / π_old(a_t|s_t)
Â_t = safety_reward_t + γ · V(s_{t+1}) - V(s_t)
safety_reward_t = f(refusal_correctness, task_completion, sensitivity_handling)

The reward function penalizes: - Executing malicious instructions (high penalty: -2.0) - Over-refusing benign instructions (moderate penalty: -0.5) - Missing tool-initiated injections (high penalty: -2.0)

Experimental Results

Metric	Baseline	Agent Safety via RL
User-initiated attack defense	38.2%	82.4%
Tool-initiated attack defense	12.7%	76.8%
Benign task completion	91.3%	87.5%
Over-refusal rate	28.6%	12.1%
Sensitive task accuracy	—	73.2%

Key findings: - Tool-initiated threat defense improves most dramatically (+64.1%), as this is neglected by prior work - Tri-modal classification reduces over-refusal by 16.5 percentage points vs binary classification - Sandbox training generalizes to unseen tools and attack patterns

Comparison with Other Methods

Method	User Threats	Tool Threats	Over-refusal	Training Paradigm
Agent Safety via RL	82.4%	76.8%	12.1%	Sandbox RL
agent-align	79.5%	~40%	11.3%	ABC synthesis + SFT
clawguard	~85%	~70%	~18%	Runtime rules
thought-aligner	~90%	~65%	~8%	Contrastive learning
Standard RLHF	~55%	~15%	~25%	Human preference

The key differentiator of Agent Safety via RL is its explicit tool-initiated threat handling — most other methods focus exclusively on user-facing safety.

Related Work

• agent-align — Training-time safety via ABC synthesis; complementary but does not explicitly handle tool-initiated threats
• clawguard — Runtime safety framework for tool-augmented agents; can be layered on top for defense-in-depth
• ppo — The underlying RL algorithm used for safety-aware policy optimization

Deployment Recommendations

1. Sandbox calibration: Invest in building domain-specific sandbox environments that accurately simulate your tool ecosystem. Generic sandboxes may miss domain-specific attack vectors
2. Tune sensitivity thresholds: The τ_high and τ_low thresholds should be calibrated per deployment domain. Financial/healthcare: stricter; general chat: more lenient
3. Combine with agent-align: Use AgentAlign's ABC synthesis for initial safety SFT, then apply this RL framework for fine-grained safety refinement
4. Tool output monitoring: In production, add a secondary classifier on tool outputs to catch tool-initiated injections that the agent might miss
5. Continuous evaluation: Tool-initiated attacks evolve rapidly; regularly re-evaluate with new injection patterns and retrain if performance degrades
6. Sensitive task logging: Log all sensitive-modality executions for post-hoc review; this data feeds back into the RL training loop