The effects of audiovisual inputs on solving the cocktail party problem in the human brain: An fMRI study

Yuanqing Li, Fangyi Wang, Yongbin Chen, Andrzej Cichocki, Terrence Sejnowski

    Research output: Contribution to journalArticlepeer-review

    13 Citations (Scopus)

    Abstract

    At cocktail parties, our brains often simultaneously receive visual and auditory information. Although the cocktail party problem has been widely investigated under auditory-only settings, the effects of audiovisual inputs have not. This study explored the effects of audiovisual inputs in a simulated cocktail party. In our fMRI experiment, each congruent audiovisual stimulus was a synthesis of 2 facial movie clips, each of which could be classified into 1 of 2 emotion categories (crying and laughing). Visual-only (faces) and auditory-only stimuli (voices) were created by extracting the visual and auditory contents from the synthesized audiovisual stimuli. Subjects were instructed to selectively attend to 1 of the 2 objects contained in each stimulus and to judge its emotion category in the visual-only, auditory-only, and audiovisual conditions. The neural representations of the emotion features were assessed by calculating decoding accuracy and brain pattern-related reproducibility index based on the fMRI data. We compared the audiovisual condition with the visual-only and auditory-only conditions and found that audiovisual inputs enhanced the neural representations of emotion features of the attended objects instead of the unattended objects. This enhancement might partially explain the benefits of audiovisual inputs for the brain to solve the cocktail party problem.

    Original languageEnglish
    Pages (from-to)3623-3637
    Number of pages15
    JournalCerebral Cortex
    Volume28
    Issue number10
    DOIs
    Publication statusPublished - 1 Oct 2018

    Keywords

    • audiovisual inputs
    • cocktail party problem
    • decoding
    • neural representation
    • reproducibility

    Fingerprint

    Dive into the research topics of 'The effects of audiovisual inputs on solving the cocktail party problem in the human brain: An fMRI study'. Together they form a unique fingerprint.

    Cite this