Publication:
On the importance of hidden bias and hidden entropy in representational efficiency of the Gaussian-Bipolar Restricted Boltzmann Machines

Thumbnail Image

Organizational Units

Program

KU Authors

Co-Authors

Advisor

Publication Date

2018

Language

English

Type

Journal Article

Journal Title

Journal ISSN

Volume Title

Abstract

In this paper, we analyze the role of hidden bias in representational efficiency of the Gaussian-Bipolar Restricted Boltzmann Machines (GBPRBMs), which are similar to the widely used Gaussian-Bernoulli RBMs. Our experiments show that hidden bias plays an important role in shaping of the probability density function of the visible units. We define hidden entropy and propose it as a measure of representational efficiency of the model. By using this measure, we investigate the effect of hidden bias on the hidden entropy and provide a full analysis of the hidden entropy as function of the hidden bias for small models with up to three hidden units. We also provide an insight into understanding of the representational efficiency of the larger scale models. Furthermore, we introduce Normalized Empirical Hidden Entropy (NEHE) as an alternative to hidden entropy that can be computed for large models. Experiments on the MNIST, CIFAR-10 and Faces data sets show that NEHE can serve as measure of representational efficiency and gives an insight on minimum number of hidden units required to represent the data.

Description

Source:

Neural Networks

Publisher:

Elsevier

Keywords:

Subject

Computer science, Neurosciences and neurology

Citation

Endorsement

Review

Supplemented By

Referenced By

Copy Rights Note

1

Views

2

Downloads

View PlumX Details