How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference
arXiv:2610.40325v1 Announce Type: cross Abstract: Looping in transformer models reuses weights by repeatedly applying the same transformer, or parts of it, making the model deeper without increasing its parameter count. Whether looping is useful depends on which constraint dominates: model size, training compute, or autoregressive inference efficiency. We study these trade-offs jointly in dense decoder-only transformers trained from scratch, varying which parts of the model are looped, how many…