Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets
arXiv:2610.40287v1 Announce Type: cross Abstract: Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies th…