--- title: "PerlAD: Towards Enhanced Closed-loop End-to-end Autonomous Driving with Pseudo-simulation-based Reinforcement Learning" arxiv_id: "2603.14908" source: "https://arxiv.org/html/2603.14908" authors: "Yinfeng Gao, Qichao Zhang, Deqing Liu, Zhongpu Xia, Guang Li, Kun Ma, Guang Chen, Hangjun Ye, Long Chen, Da-Wei Ding, Dongbin Zhao" published: 2026-03-16 categories: "cs.RO, cs.CV" venue: "IEEE Robotics and Automation Letters (RA-L)" affiliations: "UST Beijing, Xiaomi EV, CASIA, UCAS" --- # PerlAD: Towards Enhanced Closed-loop End-to-end Autonomous Driving with Pseudo-simulation-based Reinforcement Learning ## Abstract End-to-end autonomous driving policies based on Imitation Learning (IL) often struggle in closed-loop execution due to the misalignment between inadequate open-loop training objectives and real driving requirements. While Reinforcement Learning (RL) offers a solution by directly optimizing driving goals via reward signals, the rendering-based training environments introduce the rendering gap and are inefficient due to high computational costs. To overcome these challenges, we present a novel Pseudo-simulation-based RL method for closed-loop end-to-end autonomous driving, PerlAD. Based on offline datasets, PerlAD constructs a pseudo-simulation that operates in vector space, enabling efficient, rendering-free trial-and-error training. To bridge the gap between static datasets and dynamic closed-loop environments, PerlAD introduces a prediction world model that generates reactive agent trajectories conditioned on the ego vehicle's plan. Furthermore, to facilitate efficient planning, PerlAD utilizes a hierarchical decoupled planner that combines IL for lateral path generation and RL for longitudinal speed optimization. Comprehensive experimental results demonstrate that PerlAD achieves state-of-the-art performance on the Bench2Drive benchmark, surpassing the previous E2E RL method by 10.29% in Driving Score without requiring expensive online interactions. Additional evaluations on the DOS benchmark further confirm its reliability in handling safety-critical occlusion scenarios. ## I Introduction End-to-end (E2E) autonomous driving has garnered significant attention. Most mainstream methods rely on Imitation Learning (IL), trained on large datasets of expert demonstrations. However, IL faces fundamental challenges: inadequate training objective (minimizing geometric deviation vs. real driving goals like safety and efficiency), and causal confusion (learning spurious correlations). RL explicitly integrates driving goals through reward modeling, enabling trial-and-error exploration. Existing E2E RL approaches are hindered by: rendering-based simulation domain gap (game engine simulators), prohibitive computational costs (3DGS reconstruction), or static open-loop setting that fails to capture dynamic interactions. PerlAD constructs a pseudo-simulation environment in vector space using real sensor data from offline datasets, eliminating complex rendering. It incorporates a Prediction World Model (PWM) that predicts surrounding agents' trajectories conditioned on the ego plan. A decoupled planner combines IL for smooth lateral path generation and RL for interactive longitudinal speed optimization. ## II Related Works ### II-A IL-based End-to-end Driving - UniAD, VAD, SparseDrive: unified multi-query frameworks for joint perception, prediction, and planning - Diffusion-based models for multi-modal trajectory generation - Latent-space world models for self-supervised learning - Large language models for interpretable reasoning ### II-B RL-based End-to-end Driving - Game-engine simulators (Raw2Drive): domain gap from rendering - 3DGS-based training (RAD): prohibitive computational costs - Open-loop RL fine-tuning (WorldRFT): non-reactive settings fail for closed-loop ## III Problem Definition E2E driving formulated as POMDP. Observations from surround-view cameras. Action space decoupled into lateral (waypoints) and longitudinal (target speed scalar). Reward function incorporates safety and efficiency terms. ## IV Method ### IV-A Pseudo-Simulation Environment Rendering-free, operates in vector space on GPU in parallel. **IV-A1 Ego Simulation**: Bicycle kinematics model + two PID controllers → ego future trajectory. **IV-A2 Agent Simulation**: Top-1 predicted trajectory interpolated to high-frequency simulated motion. **IV-A3 Static Map**: Lane markings as polylines from dataset, assumed unchanged. **IV-A4 Reward Function**: Four components: - r_col: collision reward (-30 for vehicles, -50 for pedestrians, -10 for cones) - r_lk: lane-keeping reward (-30 for double solid, -10 for single solid) - r_prog: progress reward (normalized path completion) - r_dist: distance reward (negative L2 to ground-truth future) Total reward: weighted sum across simulation steps with discount factor γ. ### IV-B End-to-End Autonomous Driving Model **IV-B1 Sparse Perception**: Based on SparseDrive. Agent queries and map queries as learnable parameters. **IV-B2 Unified Transformer Blocks**: Temporal-spatial interaction with cross/self attention. Decoupled ego planning queries for lateral and longitudinal planning. **IV-B3 Decoupled Planner (DeP)**: - Lateral Planning: Multi-modal path planning with path anchors → regression + classification heads - Longitudinal Planning: Discrete action space, classification over target speeds, conditioned on selected lateral path **IV-B4 Prediction World Model (PWM)**: - Ego-Conditional Prediction: GRU-based autoregressive generation with ego displacement embedding - During supervised training: ego trajectory from ground truth - During RL training: ego trajectory from DeP's output via ego simulation - Multi-modal prediction with modality embeddings ### IV-C Training Strategies **Two-stage training**: - Stage 1: Only sparse perception (detection + mapping losses) - Stage 2: Frozen perception, train transformer blocks + DeP + PWM (prediction + planning losses) **IV-C1 IL-based Lateral Planning Training**: Dense supervision via regression + classification loss. **IV-C2 RL-based Longitudinal Planning Training**: REINFORCE with group-standardized advantage estimation (inspired by DeepSeek-R1/GRPO). Sample N speed actions → pseudo-simulation rewards → policy gradient update with entropy bonus. **IV-C3 Lateral-Longitudinal Alignment**: Curriculum strategy. Early: GT path for longitudinal input. Final 1/3: predicted path. Modified lateral path classification with clipped rewards. **IV-C4 Reactive Training Simulation**: Early: GT agent trajectories. Final 1/3: progressively replace with PWM's reactive predictions. ## V Experiments ### Bench2Drive Results (Table I) - **PerlAD: DS=78.70, SR=57.27%** (SoTA) - vs. SparseDrive (IL baseline): DS=44.54 (+76.7% improvement) - vs. Raw2Drive (previous RL SoTA, NeurIPS 25): DS=71.36 (+10.29%) - Multi-ability: Best Merging (40%), Overtaking (75.56%), Emergency Brake (68.33%) ### DOS Benchmark (Table II) - PerlAD Average DS: 86.83 (vs. ReasonPlan 78.02, UniAD 71.09) ### Prediction World Model (Table III-IV) - PWM improves ADE/FDE vs. vanilla prediction - Counterfactual collision: 70/200 (PWM) vs. 147/200 (vanilla) vs. 166/200 (logged replay) ### Ablation Studies - RL-lon vs IL-lon: DS 65.01 vs 32.81 (critical) - +LLA: DS 70.28 - +RTS: DS 74.00 - Full reward: DS 74.00 vs distance-only 53.58 ## VI Conclusion Three core innovations: data-driven pseudo-simulation, Prediction World Model, hierarchical decoupled planner with alignment. Future: latent world model for sensory feature extrapolation, reward modeling from human preference data, extending RL toward coupled planning for diverse driving conditions. ## References 30 references including: UniAD (CVPR 23), VAD (ICCV 23), SparseDrive (ICRA 25), Raw2Drive (NeurIPS 25), DriveTransformer (ICLR 25), DiffAD, Think2Drive, DeepSeek-R1, RAD, WorldRFT, etc.