Generalized tensor models for recurrent neural networks

Valentin Khrulkov, Oleksii Hrinchuk, Ivan Oseledets

    Research output: Contribution to conferencePaperpeer-review

    2 Citations (Scopus)

    Abstract

    Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data. However, this observed efficiency is not yet entirely explained by theory. It is known that a certain class of multiplicative RNNs enjoys the property of depth efficiency - a shallow network of exponentially large width is necessary to realize the same score function as computed by such an RNN. Such networks, however, are not very often applied to real life tasks. In this work, we attempt to reduce the gap between theory and practice by extending the theoretical analysis to RNNs which employ various nonlinearities, such as Rectified Linear Unit (ReLU), and show that they also benefit from properties of universality and depth efficiency. Our theoretical results are verified by a series of extensive computational experiments.

    Original languageEnglish
    Publication statusPublished - 2019
    Event7th International Conference on Learning Representations, ICLR 2019 - New Orleans, United States
    Duration: 6 May 20199 May 2019

    Conference

    Conference7th International Conference on Learning Representations, ICLR 2019
    Country/TerritoryUnited States
    CityNew Orleans
    Period6/05/199/05/19

    Fingerprint

    Dive into the research topics of 'Generalized tensor models for recurrent neural networks'. Together they form a unique fingerprint.

    Cite this