Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning
arXiv:2610.21682v1 Announce Type: cross Abstract: Reinforcement learning (RL) can become off-policy due to training-inference mismatch, mini-batch updates, and asynchronous training.The resulting mismatch is especially harmful in Mixture-of-Experts (MoE) models, where rollout--training discrepancies can change routing decisions and lead to router drift. Rollout Routing Replay (R3) mitigates this mismatch by reusing expert selections from the rollout stage during training. Our analysis shows tha…