Why Meta trained a small 8B model on 15 trillion tokens, and why that's cheaper than it sounds
There's a rule of thumb from a famous paper: for the best result per dollar of training, feed a model about 20 tokens of text for every parameter it has. An 8-billion-parameter model times 20 is 160 billion tokens. Meta's own blog post puts the "Chinchilla-optimal" amount for an 8B model at about 200 billion. Llama 3 8B was trained on over 15 trillion. That's roughly 75 to 95 times more than the rule says. Either Meta wasted a ridiculous amount of compute, or the rule is answering a different q…