Audio-visual object localization and separation using low-rank and sparsity

Pu, Jie, Panagakis, Yannis ORCID logoORCID:, Petridis, Stavros and Pantic, Maja (2017) Audio-visual object localization and separation using low-rank and sparsity. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 05-09 Mar 2017, New Orleans, LA, USA. e-ISBN 9781509041176, e-ISBN 9781509041169, pbk-ISBN 9781509041183. ISSN 2379-190X [Conference or Workshop Item] (doi:10.1109/ICASSP.2017.7952687)

PDF - Final accepted version (with author's formatting)
Download (1MB) | Preview


The ability to localize visual objects that are associated with an audio source and at the same time seperate the audio signal is a corner stone in several audio-visual signal processing applications. Past efforts usually focused on localizing only the visual objects, without audio separation abilities. Besides, they often rely computational expensive pre-processing steps to segment images pixels into object regions before applying localization approaches. We aim to address the problem of audio-visual source localization and separation in an unsupervised manner. The proposed approach employs low-rank in order to model the background visual and audio information and sparsity in order to extract the sparsely correlated components between the audio and visual modalities. In particular, this model decomposes each dataset into a sum of two terms: the low-rank matrices capturing the background uncorrelated information, while the sparse correlated components modelling the sound source in visual modality and the associated sound in audio modality. To this end a novel optimization problem, involving the minimization of nuclear norms and matrix ℓ1-norms is solved. We evaluated the proposed method in 1) visual localization and audio separation and 2) visual-assisted audio denoising. The experimental results demonstrate the effectiveness of the proposed method.

Item Type: Conference or Workshop Item (Paper)
Research Areas: A. > School of Science and Technology > Computer Science
Item ID: 23780
Notes on copyright: © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works
Useful Links:
Depositing User: Yannis Panagakis
Date Deposited: 06 Mar 2018 18:31
Last Modified: 19 Feb 2023 18:32

Actions (login required)

View Item View Item


Activity Overview
6 month trend
6 month trend

Additional statistics are available via IRStats2.