Fine-tuning adaptive stochastic optimizers: determining the optimal hyperparameter ϵ via gradient magnitude histogram analysis

Abstract

Stochastic optimizers play a crucial role in the successful training of deep neural network models. To achieve optimal model performance, designers must carefully select both model and optimizer hyperparameters. However, this process is frequently demand...