Audio-visual object localization and separation using low-rank and sparsity

Pu, Jie, Panagakis, Yannis ORCID: https://orcid.org/0000-0003-0153-5210, Petridis, Stavros and Pantic, Maja (2017) Audio-visual object localization and separation using low-rank and sparsity. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 05-09 Mar 2017, New Orleans, LA, USA. e-ISBN 9781509041176, e-ISBN 9781509041169, pbk-ISBN 9781509041183. ISSN 2379-190X [Conference or Workshop Item] (doi:10.1109/ICASSP.2017.7952687)

Preview

PDF - Final accepted version (with author's formatting) Download (1MB) | Preview

Abstract

The ability to localize visual objects that are associated with an audio source and at the same time seperate the audio signal is a corner stone in several audio-visual signal processing applications. Past efforts usually focused on localizing only the visual objects, without audio separation abilities. Besides, they often rely computational expensive pre-processing steps to segment images pixels into object regions before applying localization approaches. We aim to address the problem of audio-visual source localization and separation in an unsupervised manner. The proposed approach employs low-rank in order to model the background visual and audio information and sparsity in order to extract the sparsely correlated components between the audio and visual modalities. In particular, this model decomposes each dataset into a sum of two terms: the low-rank matrices capturing the background uncorrelated information, while the sparse correlated components modelling the sound source in visual modality and the associated sound in audio modality. To this end a novel optimization problem, involving the minimization of nuclear norms and matrix ℓ1-norms is solved. We evaluated the proposed method in 1) visual localization and audio separation and 2) visual-assisted audio denoising. The experimental results demonstrate the effectiveness of the proposed method.

Actions (login required)

View Item

Statistics

DownloadsShow export options

Export as CSVXMLJSON

Activity Overview

6 month trend

386Downloads

6 month trend

288Hits

Additional statistics are available via IRStats2.