HERO: Hessian-Enhanced Robust Optimization for Unifying and Improving Generalization and Quantization Performance
TimeTuesday, July 12th10:30am - 10:52am PDT
Location3000, Level 3
ML Algorithms and Applications
DescriptionImproving the model generalizability on unseen data is desired, as well as making the model robust under fixed-point quantization. Minimizing the training loss, however, provides few guarantees on the generalization and quantization performance. This work fulfills the need of improving generalization and quantization performance simultaneously by theoretically unifying them under the framework of improving the model's robustness against bounded weight perturbation and minimizing the eigenvalues of the Hessian matrix with respect to the weight. We therefore propose HERO, a Hessian-enhanced robust optimization method, to achieve this goal. HERO enables both state-of-the-art generalization and quantization performance for common model architectures.