Learning to look up: Realtime monocular gaze correction using machine learning

Daniil Kononenko, Victor Lempitsky

    Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

    25 Citations (Scopus)

    Abstract

    We revisit the well-known problem of gaze correction and present a solution based on supervised machine learning. At training time, our system observes pairs of images, where each pair contains the face of the same person with a fixed angular difference in gaze direction. It then learns to synthesize the second image of a pair from the first one. After learning, the system gets the ability to redirect the gaze of a previously unseen person by the same angular difference as in the training set. Unlike many previous solutions to gaze problem in videoconferencing, ours is purely monocular, i.e. it does not require any hardware apart from an in-built web-camera of a laptop. Being based on efficient machine learning predictors such as decision forests, the system is fast (runs in real-time on a single core of a modern laptop). In the paper, we demonstrate results on a variety of videoconferencing frames and evaluate the method quantitatively on the hold-out set of registered images. The supplementary video shows example sessions of our system at work.

    Original languageEnglish
    Title of host publicationIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015
    PublisherIEEE Computer Society
    Pages4667-4675
    Number of pages9
    ISBN (Electronic)9781467369640
    DOIs
    Publication statusPublished - 14 Oct 2015
    EventIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015 - Boston, United States
    Duration: 7 Jun 201512 Jun 2015

    Publication series

    NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
    Volume07-12-June-2015
    ISSN (Print)1063-6919

    Conference

    ConferenceIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015
    Country/TerritoryUnited States
    CityBoston
    Period7/06/1512/06/15

    Fingerprint

    Dive into the research topics of 'Learning to look up: Realtime monocular gaze correction using machine learning'. Together they form a unique fingerprint.

    Cite this