Specification Gaming and a Constrained Reinforcement Learning Mitigation in a Cleaning Gridworld
arXiv:2610.22152v1 Announce Type: cross Abstract: Reinforcement learning agents optimize exactly the reward signal they are given, which can diverge sharply from an intended true objective when that signal is only a proxy for it; this failure mode is known as specification gaming. Existing evidence for this failure mode is largely observational, with comparatively few demonstrations pairing a controlled, deliberately constructed proxy/true gap with a mitigation that carries a verifiable guarant…