What it actually costs to train an LLM: GPUs, days and dollars, worked out step by step
You've seen the headlines. "Training a big model costs tens of millions of dollars." Okay, but why that number? Where does it come from? I wanted to count it myself, with a formula short enough to fit on a napkin, and then check the answer against something real. Turns out you can get surprisingly close. And the places where you don't get close are the most interesting part. No jargon after this, promise. Parameters: the numbers inside the model that get adjusted during training. "8B" means 8 b…