Research
 

Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework

Authors

Authors

Cite
Research

Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework

ICML

Cite Paper

Authors

Published on

07/23/2022

Categories

ICML

As a seminal tool in self-supervised representation learning, contrastive learning has gained unprecedented attention in recent years. In essence, contrastive learning aims to leverage pairs of positive and negative samples for representation learning, which relates to exploiting neighborhood information in a feature space. By investigating the connection between contrastive learning and neighborhood component analysis (NCA), we provide a novel stochastic nearest neighbor viewpoint of contrastive learning and subsequently propose a series of contrastive losses that outperform the existing ones. Under our proposed framework, we show a new methodology to design integrated contrastive losses that could simultaneously achieve good accuracy and robustness on downstream tasks. With the integrated framework, we achieve up to 6% improvement on the standard accuracy and 17% improvement on the robust accuracy.

Please cite our work using the BibTeX below.

@InProceedings{pmlr-v162-ko22a,
  title = 	 {Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework},
  author =       {Ko, Ching-Yun and Mohapatra, Jeet and Liu, Sijia and Chen, Pin-Yu and Daniel, Luca and Weng, Lily},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {11387--11412},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/ko22a/ko22a.pdf},
  url = 	 {https://proceedings.mlr.press/v162/ko22a.html},
  abstract = 	 {As a seminal tool in self-supervised representation learning, contrastive learning has gained unprecedented attention in recent years. In essence, contrastive learning aims to leverage pairs of positive and negative samples for representation learning, which relates to exploiting neighborhood information in a feature space. By investigating the connection between contrastive learning and neighborhood component analysis (NCA), we provide a novel stochastic nearest neighbor viewpoint of contrastive learning and subsequently propose a series of contrastive losses that outperform the existing ones. Under our proposed framework, we show a new methodology to design integrated contrastive losses that could simultaneously achieve good accuracy and robustness on downstream tasks. With the integrated framework, we achieve up to 6% improvement on the standard accuracy and 17% improvement on the robust accuracy.}
}
Close Modal

Please cite our work using the BibTeX below.

@InProceedings{pmlr-v162-ko22a,
  title = 	 {Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework},
  author =       {Ko, Ching-Yun and Mohapatra, Jeet and Liu, Sijia and Chen, Pin-Yu and Daniel, Luca and Weng, Lily},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {11387--11412},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/ko22a/ko22a.pdf},
  url = 	 {https://proceedings.mlr.press/v162/ko22a.html},
  abstract = 	 {As a seminal tool in self-supervised representation learning, contrastive learning has gained unprecedented attention in recent years. In essence, contrastive learning aims to leverage pairs of positive and negative samples for representation learning, which relates to exploiting neighborhood information in a feature space. By investigating the connection between contrastive learning and neighborhood component analysis (NCA), we provide a novel stochastic nearest neighbor viewpoint of contrastive learning and subsequently propose a series of contrastive losses that outperform the existing ones. Under our proposed framework, we show a new methodology to design integrated contrastive losses that could simultaneously achieve good accuracy and robustness on downstream tasks. With the integrated framework, we achieve up to 6% improvement on the standard accuracy and 17% improvement on the robust accuracy.}
}