Autoencoder

Reducing the Dimensionality of
Data with Neural Networks, Science, 2006

High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors.

Encoder and Decoder

Multilayer “encoder” network to transform the high-dimensional data into a low-dimensional code and a similar “decoder” network to recover the data from the code.

Basic structure

Unfolding and Fine-tuning

Stacked Encoder

A single layer of binary features is not the best way to model the structure … The first layer of feature detectors then become the visible units for learning the next RBM. This layer-by-layer learning can be repeated as many times as desired.

The layer-by-layer learning algorithm is nonetheless a very effective way to pretrain the weights of a deep autoencoder. Each layer of features captures strong, high-order correlations between the activities of units in the layer below. For a wide variety of data sets, this is an efficient way to progressively reveal low-dimensional, nonlinear structure.

Summary

• The autoencoder can work as a tool to reduce the dimensionality of data by learning the low-dimensional codes.
• The pretraining precedure of autoencoder can provide good initial weights for the neural networks.

Example