1|# Proximal Policy Optimization (PPO) 2| 3|Proximal Policy Optimization (PPO) is a policy gradient method for reinforcement learning that improves training stability by limiting the size of policy updates via a clipped surrogate objective. It is the foundational RL algorithm extended by proximal-policy-distillation. 4| 5|## Key Properties 6| 7|- Uses a clipped probability ratio to prevent destructive large policy updates. 8|- Can reuse samples from a rollout buffer over multiple epochs (sample efficiency). 9|- Includes an entropy bonus term to encourage exploration. 10| 11|[src: raw/ingested/2024/07/2407.15134.md]

Related

• proximal-policy-distillation — Distillation method derived from PPO.
• grp-o — Related RL optimization method used in LLM post-training.
• reinforcement-learning-from-human-feedback — Broader RLHF context.