Research
 

ISAAC Newton: Input-based Approximate Curvature for Newton’s Method

Authors

Authors

  • Felix Petersen
  • Tobias Sutter
  • Christian Borgelt
  • Dongsung Huh
  • Hilde Kuehne
  • Yuekai Sun
  • Oliver Deussen
Cite
Research

ISAAC Newton: Input-based Approximate Curvature for Newton’s Method

ICLR

Cite Paper

Authors

  • Felix Petersen
  • Tobias Sutter
  • Christian Borgelt
  • Dongsung Huh
  • Hilde Kuehne
  • Yuekai Sun
  • Oliver Deussen

Published on

05/05/2023

Categories

ICLR

We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons. We show that it is possible to compute a good conditioner based on only the input to a respective layer without a substantial computational overhead. The proposed method allows effective training even in small-batch stochastic regimes, which makes it competitive to first-order as well as second-order methods.

Please cite our work using the BibTeX below.

@inproceedings{
petersen2023isaac,
title={{ISAAC} Newton: Input-based Approximate Curvature for Newton's Method},
author={Felix Petersen and Tobias Sutter and Christian Borgelt and Dongsung Huh and Hilde Kuehne and Yuekai Sun and Oliver Deussen},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=0paCJSFW7j}
}
Close Modal

Please cite our work using the BibTeX below.

@inproceedings{
petersen2023isaac,
title={{ISAAC} Newton: Input-based Approximate Curvature for Newton's Method},
author={Felix Petersen and Tobias Sutter and Christian Borgelt and Dongsung Huh and Hilde Kuehne and Yuekai Sun and Oliver Deussen},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=0paCJSFW7j}
}