SWAPNA MEDIKONDA: May 2020

AUTOENCODERS

In this session i am going to discuss about

Who introduced Autoencoders
Why we need Autoencoders
What is an AutoEncoder
How Autoencoders are implemented
Where we use Autoencoders (Applications)
Drawbacks of Autoencoders.

Autoencoders are introduced by Hinton and PDP Group to address the problem of "Backpropagation with out a teacher". Transferring high definition images through the internet is a slow process.

To avoid this problem there is a technique that compress image at source side and transfer to destination through internet and at destination side reconstruct the image.

There are some traditional methods also there like PCA (Principal Component Analysis) Algorithm to compress and reconstruct the image. In PCA noise is high.

To reduce high noise autoencoders are introduced. An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore signal “noise”. Along with the reduction side, a reconstructing side is learnt, where the autoencoder tries to generate from the reduced encoding a representation as close as possible to its original input.

Properties of an Autoencoder:

Unsupervised : An autoencoder is a unsupervised Learning Algorithm, that means data without output lables.
Data Specific: Autoencoders are able to compress the data similar to which they have been trained.
Lossy: Autoencoders are lossy because it cannot predict the image exactly same as input image.

Architecture of Autoencoders:

The simplest form of an autoencoder is a feedforward, non-recurrent neural network similar to single layer perceptrons that participate in multilayer perceptrons (MLP) – having an input layer, an output layer and one or more hidden layers connecting them – where the output layer has the same number of nodes (neurons) as the input layer, and with the purpose of reconstructing its inputs (minimizing the difference between the input and the output) instead of predicting the target value Y given inputs X. Therefore, autoencoders are unsupervised learning models (do not require labeled inputs to enable learning).

Generally, in autoencoder architecture there are 3 components

An encoder that maps into the code.
Code acts as a bottleneck.
The decoder that maps the code to a reconstruction of original data.

Encoder is responsible to map input features ‘ X ’ to latent space representation (Code) ‘ h ’.

X --> h

Decoder is responsible to map coded representation ‘ h ‘ to ‘ X’ ‘

h --> X’

Types of Autoencoders

1. Regularized Autoencoders

-Sparse Autoencoders

Sparse autoencoder may include more (rather than fewer) hidden units than inputs, but only a small number of the hidden units are allowed to be active at once. This sparsity constraint forces the model to respond to the unique statistical features of the input data used for training.

-Denoising Autoencoders

Even though autoencoders takes a noise image as input, these are able to reconstruct the image.

-Contractive Autoencoders

2.Variational Autoencoders

Like other autoencoders, variational autoencoders also consist of an encoder and a decoder. But here, the decoder is the generator model. Variational autoencoders also carry out the construction process from the latent code space. But in VAEs, the latent coding space is continuous.

Applications of Autoencoders:

-Dimensionality Reduction

-Information Retrieval

-Anomaly Detection

-Image Processing

-Drug discovery

-Population synthesis

-Machine Translation

-Image Coloring

-Denoising Image

-Feature Variation

-Watermark Removal

Limitations of Autoencoders:

We have seen how autoencoders can be used for image compression and reconstruction of images. But in reality, they are not very efficient in the process of compressing images. Also, they are only efficient when reconstructing images similar to what they have been trained on.

Implementation of undercomplete Autoencoder:

from keras.layers import Input, Dense

from keras.models import Model

# this is the size of our encoded representations

encoding_dim = 32 #by taking 32 compression factor reduced to 784/32

# this is our input placeholder

input_img = Input(shape=(784,)) # we have 28*28 dimensional image so 28*28=784 pixel values for each image

# "encoded" is the encoded representation of the input

encoded = Dense(encoding_dim, activation='relu')(input_img)# here we are defining hidden layer as dense layer and its previous layer is input_img

# "decoded" is the lossy reconstruction of the input

decoded = Dense(784, activation='sigmoid')(encoded) # here we are defining layer output layer its previous layer is 'encoded'.

# this model maps an input to its reconstruction

autoencoder = Model(input_img, decoded) #Building a model for Autoencoder

# this model maps an input to its encoded representation

encoder = Model(input_img, encoded) #Building a model for encoder

# create a placeholder for an encoded (32-dimensional) input

encoded_input = Input(shape=(encoding_dim,))

# retrieve the last layer of the autoencoder model

decoder_layer = autoencoder.layers[-1]

# create the decoder model

decoder = Model(encoded_input, decoder_layer(encoded_input)) #Building a model for decoder

autoencoder.compile(optimizer = 'adadelta',loss= 'binary_crossentropy')#Compile Autoencoder

#GET DATA

from keras.datasets import mnist

import numpy as np

(x_train, _), (x_test, _) = mnist.load_data() #ignoring output lables

x_train = x_train.astype('float32') / 255. # normalization with Decimal Scaling

x_test = x_test.astype('float32') / 255.

# reshaping from (no.of images * 28 * 28) to (no. of images *784

x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))

x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

print (x_train.shape)

print (x_test.shape)

#train with x_train

autoencoder.fit(x_train, x_train,epochs=50,

batch_size=256,

shuffle=True,

validation_data=(x_test, x_test))

# encode and decode

encoded_imgs = encoder.predict(x_test)

decoded_imgs = decoder.predict(encoded_imgs)

Result after plotting original images with decoded images using matplotlib first row belongs to actual images and second row belongs to predicted images: