Back propagation algorithm

Back-propagation (BP) is the standard supervised learning algorithm for multilayer feed-forward neural networks. It computes the error at the output, propagates it backwards through the layers, and adjusts each weight by gradient descent to reduce the error (generalised delta rule).

Algorithm (sigmoid units, learning rate η)

1. Initialise all weights and biases with small random values.
2. Forward pass: for each training pair (x, t), compute hidden outputs h_j = f(Σ w_ij x_i + b_j) and output y_k = f(Σ v_jk h_j + b_k), with f(z) = 1/(1 + e^(-z)).
3. Output error term: δ_k = (t_k − y_k) · y_k (1 − y_k)
4. Hidden error term: δ_j = h_j (1 − h_j) · Σ_k δ_k v_jk
5. Weight update: Δv_jk = η δ_k h_j; Δw_ij = η δ_j x_i (biases likewise with input 1).
6. Repeat for all patterns (an epoch) until the total error E = ½ Σ (t − y)² is small enough.

Worked example (verified)

Network 2-2-1, input x = (1, 0), target t = 1, η = 0.5. Weights: x1→h1 = 0.5, x1→h2 = 0.4, x2→h1 = 0.3, x2→h2 = 0.2, hidden biases 0.1, 0.1, h→y weights 0.6, 0.7, output bias 0.1.

h1 = f(0.5+0.1) = 0.6457,  h2 = f(0.4+0.1) = 0.6225
y  = f(0.6*0.6457 + 0.7*0.6225 + 0.1) = 0.7157
δy = (1-0.7157)*0.7157*0.2843 = 0.0579
δh1 = 0.0579*0.6*0.6457*0.3543 = 0.0079
δh2 = 0.0579*0.7*0.6225*0.3775 = 0.0095
v1 = 0.6+0.5*0.0579*0.6457 = 0.6187,  v2 = 0.7180
new w(x1->h1) = 0.5+0.5*0.0079 = 0.5040

After the update, the output for the same input rises from 0.7157 to 0.7267, closer to 1.

How it helps machine learning

• It lets multilayer networks learn non-linear mappings (e.g. XOR) that a single perceptron cannot.
• It trains hidden layers efficiently by reusing computed errors.
• It is the basis of modern deep learning (with variants such as momentum and stochastic gradient descent).

Limitations: may get stuck in local minima, slow convergence, needs many labelled examples.