在本論文中，我們介紹了 PBHS，一種基於預測的超參數調優調度程序。近年來，由於數據可用性和計算能力的快速增長，深度學習技術得到迅速發展。同時，一些實驗表明，模型架構及其超參數是影響最終性能的關鍵因素，因此隨機選擇的超參數配置可能導致糟糕的性能。隨著深度學習技術在日常應用中的不斷普及，開發一種具有良好設計算法的工具變得至關重要，該工具能夠通過自動調整超參數來開發模型，並在時間和資源限制下找到具有相應超參數配置的最佳性能。現有的解決方案僅根據它們的当前的性能終止候選配置，這可能會錯誤地終止一些實際上非常有前途的候選配置。因此，我們設計了一個自動調優調度器，結合數學方法來預測學習曲線的趨勢，使調度器可以向前看。我們用各種神經網絡和數據集測試我們的調度器，證明它能取得比現有解決方案更好的表現。

In this thesis, we present PBHS, a prediction-based hyperparameter tuning scheduler. Recent years have seen the Cambrian explosion of deep learning technology powered by the rapid increment in both data availability and computational ability. Meanwhile, several experiments have shown that model architecture as well as its hyperparameters are the critical factors that affect the final performance, such that random network configuration can lead to catastrophic performance. As deep learning technology continuing to take places in everyday applications, it becomes crucial to develop a tool with well-designed algorithm that is able to develop the model by tuning hyperparameters automatically and find the best performance with corresponding hyperparameter configuration under time and resources limitations. Existing solutions only kill the candidate configurations base on their \emph{current} performance, which may falsely kill some candidates that are actually very promising. Thus, we design an auto-tuning scheduler combining with mathematical method to predict the trend of learning curve, so that the scheduler can \emph{look forward}. We test our scheduler with various neural networks and data set, and prove it effective.

[1]Bergstra, J., and Bengio, Y. Random search for hyper-parameter optimization.
Journal of machine learning research 13, 2 (2012).
[2]Domhan, T., Springenberg, J. T., and Hutter, F. Speeding up automatic hyper- parameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelli- gence (2015).
[3]He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CoRR abs/1512.03385 (2015).
[4]Jamieson, K., and Talwalkar, A. Non-stochastic best arm identification and hyperparameter optimization. In Proceedings of the 19th International Confer- ence on Artificial Intelligence and Statistics (Cadiz, Spain, 09–11 May 2016),
A. Gretton and C. C. Robert, Eds., vol. 51 of Proceedings of Machine Learning Research, PMLR, pp. 240–248.
[5]Klein, A., Falkner, S., Springenberg, J. T., and Hutter, F. Learning curve pre- diction with bayesian neural networks.
[6]Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Hardt, M., Recht, B., and Talwalkar, A. Massively parallel hyperparameter tuning, 2019.
[7]Li, L., Jamieson, K. G., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. Hyperband: Bandit-based configuration evaluation for hyperparameter opti- mization. In ICLR (Poster) (2017).
[8]Liaw, R., Bhardwaj, R., Dunlap, L., Zou, Y., Gonzalez, J., Stoica, I., and Tu- manov, A. Hypersched: Dynamic resource reallocation for model develop- ment on a deadline. CoRR abs/2001.02338 (2020).
[9]Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J. E., and Stoica, I. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118 (2018).
[10]Simonyan, K., and Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv e-prints (Sept. 2014), arXiv:1409.1556.
[11]Sukhbaatar, S., Szlam, A., Weston, J., and Fergus, R. Weakly supervised mem- ory networks. CoRR abs/1503.08895 (2015).
Weston, J., Bordes, A., Chopra, S., Rush, A. M., Van Merriënboer, B., Joulin, A., and Mikolov, T. Towards ai-complete question answering: A set of pre- requisite toy tasks. arXiv preprint arXiv:1502.05698 (2015).