Language-Guided Audio-Visual Source Separation via Trimodal Consistency



  • Reuben Tan
  • Arijit Ray
  • Andrea Burns
  • Bryan A. Plummer
  • Justin Salamon
  • Oriol Nieto
  • Bryan Russell
  • Kate Saenko

Published on




We propose a self-supervised approach for learning to perform audio source separation in videos based on natural language queries, using only unlabeled video and audio pairs as training data. A key challenge in this task is learning to associate the linguistic description of a soundemitting object to its visual features and the corresponding components of the audio waveform, all without access to annotations during training. To overcome this challenge, we adapt off-the-shelf vision-language foundation models to provide pseudo-target supervision via two novel loss functions and encourage a stronger alignment between the audio, visual and natural language modalities. During inference, our approach can separate sounds given text, video and audio input, or given text and audio input alone. We demonstrate the effectiveness of our self-supervised approach on three audio-visual separation datasets, including MUSIC, SOLOS and AudioSet, where we outperform state-of-the-art strongly supervised approaches despite not using object detectors or text labels during training. Our project page including publicly available code can be found at

This work was presented at CVPR 2023.

Please cite our work using the BibTeX below.

  author    = {Reuben Tan and Arijit Ray and Andrea Burns and Bryan A. Plummer and Justin Salamon and Oriol Nieto and Bryan Russell and Kate Saenko},
  title     = {Language-Guided Audio-Visual Source Separation via Trimodal Consistency},
  journal   = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2023},
Close Modal