AUTOMATIC DIFFERENTIATION FOR RIEMANNIAN OPTIMIZATION ON LOW-RANK MATRIX AND TENSOR-TRAIN MANIFOLDS

Alexander Novikov, Maxim Rakhuba, Ivan Oseledets

Research output: Contribution to journalArticlepeer-review

Abstract

In scientific computing and machine learning applications, matrices and more general multidimensional arrays (tensors) can often be approximated with the help of low-rank decompositions. Since matrices and tensors of fixed rank form smooth Riemannian manifolds, one of the popular tools for finding low-rank approximations is to use Riemannian optimization. Nevertheless, efficient implementation of Riemannian gradients and Hessians, required in Riemannian optimization algorithms, can be a nontrivial task in practice. Moreover, in some cases, analytic formulas are not even available. In this paper, we build upon automatic differentiation and propose a method that, given an implementation of the function to be minimized, efficiently computes Riemannian gradients and matrix-by-vector products between an approximate Riemannian Hessian and a given vector.

Original languageEnglish
Pages (from-to)A843-A869
JournalSIAM Journal on Scientific Computing
Volume44
Issue number2
DOIs
Publication statusPublished - 2022

Keywords

  • automatic differentiation
  • low-rank approximation
  • Riemannian optimization
  • tensor-train decomposition

Fingerprint

Dive into the research topics of 'AUTOMATIC DIFFERENTIATION FOR RIEMANNIAN OPTIMIZATION ON LOW-RANK MATRIX AND TENSOR-TRAIN MANIFOLDS'. Together they form a unique fingerprint.

Cite this