Always Learning, Always Mixing: Efficient and Simple Data Mixing All the Time
arXiv:2605.15220v2 Announce Type: replace-cross Abstract: Data mixing decides how to combine different sources or types of data and is a consequential problem throughout language model training. In pretraining, data composition is a key determinant of model quality; in continual learning and adaptation, it governs what is retained and acquired. Yet existing data mixing methods address only one phase of this lifecycle at a time: some require smaller proxy models tied to a single training phase,…