Topics on Generative Models in Machine Learning
Time: Mon 2021-06-14 13.00
Doctoral student: Carl Ringqvist , Matematisk statistik
Opponent: Professor Thomas Schön, nst för informationsteknologi, Uppsala universitet, Uppsala
Supervisor: Professor Henrik Hult, Matematisk statistik
Latent variable models have been extensively studied within the field of machine learning in recent years. Especially in combination with neural networks and training through back propagation, they have proven successful for a variety of tasks; notably sample gener- ation, clustering, disentanglement and interpolation. This thesis con- sists of four papers, treating different subjects in this context. In Paper A, the Infinite Swapping algorithm is applied to the Restricted Boltz- mann Machine model. The Infinite Swapping algorithm is an extension of Parallel Tempering, an algorithm developed for speeding up conver- gence of Markov Chains. Since the Restricted Boltzmann Machine requires sample generation during training, such techniques are rele- vant for application to the model. Previously, Parallel Tempering has been demonstrated to yield superior training results when compared to preceding methods. Paper A continues this effort by adapting the Infinite Swapping algorithm to the setting of Restricted Boltzmann Machines. The remaining three papers treats the Variational Autoen- coder model. In paper B and C, methods for stochastic interpolation is introduced, and mathematically investigated. In this context, inter- polation is taken to mean a semantically meaningful transformation from one observation to another. Within image analysis, this trans- lates to displaying a sensible visual transformation from one object to another. Previously presented interpolation methods within the field have exclusively focused on the concept of deterministic interpolation; essentially aiming to find ’the correct’ or in some sense optimal in- terpolation path. In paper B and C, a different view of interpolation is introduced; where the correct interpolation paths are characterised by a distribution that is possible to sample from. It is proven that the suggested interpolation distribution produces samples that lie in the manifold specified by the Variational Autoencoder prior if hyper parameters are chosen correctly; thus giving some theoretical assur- ance that the interpolation distribution produces sensible samples in general. In paper D, the Variational Autoencoder framework is inves- tigated for clustering. Here, clustering is viewed from a probabilistic perspective. Given a multi-modal distribution, each mode is viewed as a cluster label, and an observation is assigned a label through following its density gradient until a mode is reached. An effective method for estimating the density gradient for Variational Autoencoders is pro- posed, and empirically tested. A method for estimating the inherent number of clusters of data in this context is further introduced, and it is demonstrated that it performs significantly better on data processed with the density gradient, compared to when applied to original data.