Denoising Score Matching via Random Fourier Features

Olga Tsymboi, Yermek Kapushev, Evgeny Burnaev, Ivan Oseledets

Research output: Contribution to journalArticlepeer-review

Abstract

The density estimation is one of the core problems in statistics. Despite this, existing techniques like maximum likelihood estimation are computationally inefficient in case of complex parametric families due to the intractability of the normalizing constant. For this reason, an interest in score matching has increased, being independent on the normalizing constant. However, such an estimator is consistent only for distributions with the full space support. One of the approaches to make it consistent is to add noise to the input data called Denoising Score Matching. In this work we build computationally efficient algorithm for density estimate using kernel exponential family as a model distribution. The usage of the kernel exponential family is motivated by the richness of this class of densities. To avoid calculating an intractable normalizing constant we use Denoising Score Matching objective. The computational complexity issue is approached by applying Random Fourier Features-based approximation of the kernel function. We derive an exact analytical expression for this case which allows dropping additional regularization terms based on the higher-order derivatives as they are already implicitly included. Moreover, the obtained expression explicitly depends on the noise variance, so that the validation loss can be straightforwardly used to tune the noise level. Along with benchmark experiments, the method was tested on various synthetic distributions to study the behavior of the method in different cases. The empirical study shows comparable quality to the competing approaches, while the proposed method being computationally faster. The latter one enables scaling up to complex high-dimensional data.

Original languageEnglish
JournalIEEE Access
DOIs
Publication statusPublished - 2022

Keywords

  • Computational complexity
  • Computational modeling
  • Convergence
  • denoising score matching
  • density estimation
  • Kernel
  • kernel exponential family
  • Maximum likelihood estimation
  • Noise level
  • Noise reduction
  • random Fourier features
  • score matching

Fingerprint

Dive into the research topics of 'Denoising Score Matching via Random Fourier Features'. Together they form a unique fingerprint.

Cite this