In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model
arXiv:2610.21901v1 Announce Type: cross Abstract: Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks. The prevailing strategy determines how the model behaves beyond its training distribution, controlling potential capabilities and risks. Recent work has characterised many different strategies theoretically, and empirically described the transition between them as the quality and quantity of pre…