Part 7/14:
- No a priori knowledge: the function's behavior is unknown, making traditional gradient-based methods ineffective.
This setting is classified as a blackbox optimization problem, similar to hyperparameter tuning in machine learning or A/B testing, where only input-output observations are available.
Bayesian Optimization (BO): An Effective Strategy
Bayesian Optimization emerges as a practical solution for such problems. It constructs a probabilistic surrogate model—often a Gaussian Process (GP)—that approximates the true expensive function with a cheap-to-evaluate proxy.