Neural network, back-propagation, perceptron

Neural network

An artificial neural network is a set of simple processing units (neurons) connected by weighted links, organised in an input layer, hidden layer(s) and an output layer. Each neuron computes y = f(Σ wᵢxᵢ + b). The network learns by adjusting weights from examples.

Back-propagation algorithm

Back-propagation is the supervised gradient-descent method used to train multilayer feed-forward networks.

1. Initialise weights with small random values; choose learning rate η.
2. Forward pass: compute hidden outputs h_j = f(Σ w_ij x_i + b_j) and output y_k = f(Σ v_jk h_j + b_k), f(z) = 1/(1 + e^(−z)).
3. Output error: δ_k = (t_k − y_k) y_k (1 − y_k).
4. Hidden error: δ_j = h_j (1 − h_j) Σ_k δ_k v_jk (error sent backwards).
5. Update: v_jk ← v_jk + η δ_k h_j; w_ij ← w_ij + η δ_j x_i (biases likewise).
6. Repeat for all training patterns (epochs) until the total error ½ Σ (t − y)² is small.

Example (verified): for a 2-2-1 network, input (1,0), target 1, η = 0.5, one update raises the output from 0.7157 to 0.7267.

Perceptron

A perceptron (Rosenblatt) is a single neuron with a step activation: y = 1 if Σ wᵢxᵢ + b > 0, else 0.

 x1 --w1-->\
 x2 --w2--->( Σ + b )--> step --> y

Learning rule: w_i ← w_i + η (t − y) x_i, b ← b + η (t − y). It converges for linearly separable problems (AND, OR) but cannot learn XOR, which needs a hidden layer trained by back-propagation.