Rethinking Bayesian Optimization for Co-Optimizing LLM Training Configurations
arXiv:2602.08351v3 Announce Type: replace-cross Abstract: Fine-tuning an LLM to maximize performance on a downstream task requires finding the optimal data and model configuration. This is a black-box optimization problem that Bayesian optimization (BO) addresses by sequentially evaluating training configurations and using observed performance feedback to adaptively guide the search towards better configurations. However, directly applying BO to the joint data-model space suffers from two funda…