Efficient hyperparameter optimization

Abstract

Lade ̌nı ́ hyperparametru ̊ ma ́ znac ̌ny ́ vliv na vlastnosti vy 'sledne ́ho mo- delu, a proto bylo ve ̌nova ́no vy ́voji robustnı ́ch a efektivnı ́ch algoritmu ̊ pro tuto u ́lohu velke ́ u 'silı ́. Neda ́vno bylo vyvinuto ne ̌kolik novy ́ch algoritmu ̊ vyuz ̌ı ́vajı ́cı ́ch c ̌a 'stec ̌ny ́ch vyhodnocenı ́ optimalizovane ́ funkce. Nicme ́ne ̌ z lite- ratury nenı ́ zr ̌ejme ́, jak si tyto algoritmy vedou na rozmanity ́ch proble ́mech. V te ́to diplomove ́ pra ́ci jsme experimenta ́lne ̌ srovnali aktua ́lnı ́ algoritmy pro lade ̌nı ́hyperparametru ̊v mnoha u ́loha ́ch. Tyto u ́lohy se skla ́daly z tabula ́rnı ́ch benchmarku ̊ a rea ́lny ́ch u ́loh hluboke ́ho uc ̌enı ́, vc ̌etne ̌ datovy ́ch sad z oblasti zdravotnictvı ́. Vy 'sledky ukazujı ́, z ̌e neda ́vne ́ multi-fidelity techniky dosahujı ́ leps ̌ı ́ch vy 'sledku ̊ nez ̌ na ́hodne ́ prohleda ́va ́nı ́. Pr ̌esto vs ̌ak z ̌a ́dny ́ algoritmus nepoda ́val konzistentne ̌ nejleps ̌ı ́ vy ́kon ve vs ̌ech proble ́mech, coz ̌ zdu ̊razn ̌uje potr ̌ebu pru ̊be ̌z ̌ny ́ch srovna ́vacı ́ch studiı ́ v oblasti optimalizace hyperparame- tru ̊.

Hyperparameter optimization significantly impacts model performance and substantial effort went into the development of robust and efficient algo- rithms for this task. Our research found that several new algorithms utilizing partial evaluations have been published recently. However, it is not clear from the literature how the algorithms perform in various scenarios. In this thesis, we compared the leading algorithms through experiments on diverse tasks, including tabular benchmarks and real-world deep-learning problems, with a special focus on healthcare datasets. The results show that the recent multi-fidelity techniques outperform random search. Nevertheless, no single algorithm consistently excelled across all problems, highlighting the need for ongoing comparison studies in hyperparameter optimization.