ANN and back propagation

An Artificial Neural Network (ANN) is a set of connected input/output units (neurons) with weights on the connections, arranged in layers. It learns by adjusting the weights so that the network predicts the correct class label. A neuron computes the net input Iⱼ = Σᵢ wᵢⱼOᵢ + θⱼ and the output Oⱼ = 1/(1 + e^(−Iⱼ)) (sigmoid).

Input layer   Hidden layer   Output layer
  (x1) ---w--> (h1) ---w--->
  (x2) ---w--> (h2) ---w---> (y)
  (x3) ---w--> (h3) ---w--->

Backpropagation algorithm

Backpropagation learns by repeatedly processing the training tuples and comparing the network output with the known target value. The weights are changed to minimise the mean squared error, working backwards from the output layer to the first hidden layer, using gradient descent.

1. Initialise all weights and biases with small random numbers (e.g. −1 to 1), and choose the learning rate l.
2. Forward pass: feed a tuple to the input units (output = input); compute the net input and output Oⱼ of each hidden and output unit.
3. Backward pass (error calculation):
   • Output unit j with target Tⱼ: Errⱼ = Oⱼ(1 − Oⱼ)(Tⱼ − Oⱼ)
   • Hidden unit j: Errⱼ = Oⱼ(1 − Oⱼ) Σₖ Errₖ wⱼₖ
4. Update weights and biases:
   • Δwᵢⱼ = l · Errⱼ · Oᵢ, wᵢⱼ = wᵢⱼ + Δwᵢⱼ
   • Δθⱼ = l · Errⱼ, θⱼ = θⱼ + Δθⱼ
5. Terminate when all Δw in the previous epoch were very small, or the percentage of misclassified tuples is below a threshold, or a maximum number of epochs is reached. Otherwise repeat from step 2 for the next tuple (case updating) or after all tuples (epoch updating).

The learning rate l (between 0 and 1) helps avoid getting stuck at a local minimum and controls the speed of convergence: if it is too small, learning is very slow; if it is too large, the weights oscillate. Input values are normalised to the range 0-1 before training.