Linear Mode Connectivity and The Lottery Ticket Hypothesis

ICML

Cite Paper

Authors

Jonathan Frankle
Gintare Karolina Dziugaite
Daniel M. Roy
Michael Carbin

Published on

12/11/2019

Categories

ICML Machine Learning Neural and Evolutionary Computing

We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a linearly connected region. We use this technique to study iterative magnitude pruning (IMP), the procedure used by work on the lottery ticket hypothesis to identify subnetworks that could have trained in isolation to full accuracy. We find that these subnetworks only reach full accuracy when they are stable to SGD noise, which either occurs at initialization for small-scale settings (MNIST) or early in training for large-scale settings (ResNet-50 and Inception-v3 on ImageNet).

This paper has been published at ICML 2020

Please cite our work using the BibTeX below.

@misc{frankle2020linear,
      title={Linear Mode Connectivity and the Lottery Ticket Hypothesis}, 
      author={Jonathan Frankle and Gintare Karolina Dziugaite and Daniel M. Roy and Michael Carbin},
      year={2020},
      eprint={1912.05671},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}