Researchers present Bellman Policy Optimization (BPO), a reinforcement learning approach for training large language models with verifiable rewards that does not require a separate critic network. The method is derived from Policy Mirror Descent and targets autoregressive generation. It aims to improve reasoning performance in LLMs while simplifying the training setup.
A new arXiv paper proposes an approach called on-policy self-distillation that acts as a "policy reheater" for reinforcement learning with verifiable rewards. The authors target entropy collapse, a failure mode where model policies become overly concentrated, cutting rollout diversity and weakening the learning signal. The method is presented as a way to keep exploration alive during RL training of large language models.
A new arXiv paper introduces Hindsight-Anchored Policy Optimization, a method for training large language models with verifiable rewards. It uses hindsight learning combined with a Thompson sampling-inspired adaptive gate to address cold-start problems in sparse-reward, on-policy training. The approach builds on mixed-policy methods that blend off-policy and on-policy data.
A new arXiv paper proposes forking rollout trees at points where the model's beliefs shift, rather than at arbitrary intermediate steps, to assign credit in critic-free reinforcement learning with verifiable rewards. Because each fork adds sampling cost, the authors argue that concentrating branches on belief changes yields step-level value estimates more efficiently. The approach is aimed at improving how tree-structured rollouts trade compute for credit assignment.
An arXiv paper introduces a post-training method that blends supervised fine-tuning with reinforcement learning by averaging the output logits of a frozen reference policy with those of the trainable policy. The technique is integrated into GRPO, letting a model retain its supervised behaviors while still improving on verifiable rewards during reinforcement learning.