Neural network; back-propagation

An artificial neural network is a network of simple, interconnected processing units (neurons) with adjustable weights. Each neuron computes the weighted sum of its inputs and applies an activation function. A network has an input layer, hidden layer(s) and an output layer. It learns from examples by changing weights.

Back-propagation learning

Back-propagation is a supervised learning method for multilayer feed-forward networks. It uses gradient descent to minimise the squared error E = ½ Σ (t − y)² between target and actual output.

 input --> hidden --> output --> error = t - y
   ^                                   |
   +----- weights adjusted <-----------+
          (error propagated backwards)

Steps

1. Initialise weights and biases to small random values; choose learning rate η.
2. Feed-forward: for an input pattern, compute hidden outputs h_j = f(Σ w_ij x_i + b_j), then output y_k = f(Σ v_jk h_j + b_k), with f(z) = 1/(1 + e^(−z)).
3. Output error: δ_k = (t_k − y_k) y_k (1 − y_k).
4. Back-propagate to hidden layer: δ_j = h_j (1 − h_j) Σ_k δ_k v_jk.
5. Update weights: v_jk ← v_jk + η δ_k h_j; w_ij ← w_ij + η δ_j x_i (biases similarly).
6. Repeat steps 2-5 for all patterns; stop when the total error is below a tolerance or after a set number of epochs.

Small example (verified)

2-2-1 network, x = (1, 0), t = 1, η = 0.5, initial weights 0.5, 0.4, 0.3, 0.2 (input to hidden), biases 0.1, hidden-to-output 0.6, 0.7, output bias 0.1. Forward pass gives h = (0.6457, 0.6225), y = 0.7157; δ_output = 0.0579; δ_hidden = (0.0079, 0.0095). Updated output weights are 0.6187 and 0.7180; the new output for the same input is 0.7267, closer to the target 1.

Features: handles non-linear problems such as XOR; drawbacks: local minima, slow training, needs a large labelled data set.