Asynchrony Is Nearly Free for Evolution Strategies on Long-Horizon LLM Agentic Tasks
arXiv:2610.04196v2 Announce Type: replace Abstract: Synchronous policy updates can limit the efficiency of post-training for long-horizon LLM agents. When trajectories span multiple interaction turns, completion times can vary substantially, and synchronous update barriers leave faster workers idling, wasting compute resources. Asynchronous reinforcement learning addresses this inefficiency but introduces policy lag and off-policy optimization. Evolution strategies (ES) are a backpropagation-fr…