What are neural nets?

bottom of page

The learning process

Types of Neural Nets

As mentioned before, several types of neural nets exist.

They can be distinguished by

their type (feedforward or feedback),

their structure

and the learning algorithm they use.

The type of a neural net indicates, if the neurons of one of the net's layers may be connected among each other. Feedforward neural nets allow only neuron connections between two different layers, while nets of the feedback type have also connections between neurons of the same layer.

In this section, a selection of neural nets will be described.

Perceptron

The Perceptron was first introduced by F. Rosenblatt in 1958.
It is a very simple neural net type with two neuron layers that accepts only binary input and output values (0 or 1). The learning process is supervised and the net is able to solve basic logical operations like AND or OR. It is also used for pattern classification purposes.
More complicated logical operations (like the XOR problem) cannot be solved by a Perceptron.

Perceptron characteristics
sample structure
type	feedforward
neuron layers	1 input layer 1 output layer
input value types	binary
activation function	hard limiter
learning method	supervised
learning algorithm	Hebb learning rule
mainly used in	simple logical operations pattern classification

top of page

Multi-Layer-Perceptron

The Multi-Layer-Perceptron was first introduced by M. Minsky and S. Papert in 1969.
It is an extended Perceptron and has one ore more hidden neuron layers between its input and output layers.
Due to its extended structure, a Multi-Layer-Perceptron is able to solve every logical operation, including the XOR problem.

Multi-Layer-Perceptron characteristics
sample structure
type	feedforward
neuron layers	1 input layer 1 or more hidden layers 1 output layer
input value types	binary
activation function	hard limiter / sigmoid
learning method	supervised
learning algorithm	delta learning rule backpropagation (mostly used)
mainly used in	complex logical operations pattern classification

top of page

Backpropagation Net

The Backpropagation Net was first introduced by G.E. Hinton, E. Rumelhart and R.J. Williams in 1986
and is one of the most powerful neural net types.
It has the same structure as the Multi-Layer-Perceptron and uses the backpropagation learning algorithm.

Backpropagation Net characteristics
sample structure
type	feedforward
neuron layers	1 input layer 1 or more hidden layers 1 output layer
input value types	binary
activation function	sigmoid
learning method	supervised
learning algorithm	backpropagation
mainly used in	complex logical operations pattern classification speech analysis

top of page

Hopfield Net

The Hopfield Net was first introduced by physicist J.J. Hopfield in 1982 and belongs to neural net types which are called "thermodynamical models".
It consists of a set of neurons, where each neuron is connected to each other neuron. There is no differentiation between input and output neurons.
The main application of a Hopfield Net is the storage and recognition of patterns, e.g. image files.

Hopfield Net characteristics
sample structure
type	feedback
neuron layers	1 matrix
input value types	binary
activation function	signum / hard limiter
learning method	unsupervised
learning algorithm	delta learning rule simulated annealing (mostly used)
mainly used in	pattern association optimization problems

top of page

Kohonen Feature Map

The Kohonen Feature Map was first introduced by finnish professor Teuvo Kohonen (University of Helsinki) in 1982.
It is probably the most useful neural net type, if the learning process of the human brain shall be simulated. The "heart" of this type is the feature map, a neuron layer where neurons are organizing themselves according to certain input values.
The type of this neural net is both feedforward (input layer to feature map) and feedback (feature map).
(A Kohonen Feature Map is used in the sample applet)

Kohonen Feature Map characteristics
sample structure
type	feedforward / feedback
neuron layers	1 input layer 1 map layer
input value types	binary, real
activation function	sigmoid
learning method	unsupervised
learning algorithm	selforganization
mainly used in	pattern classification optimization problems simulation

What are neural nets?

top of page

The learning process