While task-specific finetuning of deep networks pretrained with self-supervision has led to significant empirical advances in NLP, their large size makes the standard finetuning approach difficult to apply to multi-task, memory-constrained settings, as storing the full model parameters for each task become prohibitively expensive. We propose as a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. This approach views finetuning as learning a task-specific diff vector that is applied on top of the pretrained parameter vector, which remains fixed and is shared across different tasks. The diff vector is adaptively pruned during training with a differentiable approximation to the -norm penalty to encourage sparsity. Diff pruning becomes parameter-efficient as the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5 of the pretrained model’s parameters per task.
Please cite our work using the BibTeX below.
@misc{guo2021parameterefficient,
title={Parameter-Efficient Transfer Learning with Diff Pruning},
author={Demi Guo and Alexander M. Rush and Yoon Kim},
year={2021},
eprint={2012.07463},
archivePrefix={arXiv},
primaryClass={cs.CL}
}