Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
arXiv:2609.01431v3 Announce Type: replace-cross Abstract: Optimal hyperparameter scaling laws describe how the best hyperparameters for large language model (LLM) training change with model and data scale, enabling practitioners to predict optimal configurations at production scales without expensive large-scale tuning. However, estimating these scaling laws conventionally requires exhaustive grid searches over thousands of training runs, consuming enormous computational resources. We introduce…