Short note: Neural network, perceptron rule

Characteristics of neural networks

• Learn from examples by adjusting the weights of connections; need no explicit rules.
• Can approximate complex non-linear functions; multilayer networks are universal approximators.
• Tolerant to noisy data and able to classify patterns they were not trained on.
• Work in parallel (many simple neurons), and can be fast once trained.
• Inputs must be numeric (categorical data must be encoded and scaled to 0-1).
• Weaknesses: long training time, choice of topology and learning rate by trial, risk of local minima, and low interpretability (black box).

Perceptron and its learning rule

The perceptron is the simplest neural network: a single neuron that computes y = f(Σ wᵢxᵢ + b), with f a step function giving output 1 if the sum > 0, else 0 (or −1). It can classify only linearly separable classes (such as AND, OR, but not XOR).

Perceptron learning rule (error-correction): for each training example (x, target t) with actual output y:

wᵢ(new) = wᵢ(old) + η (t − y) xᵢ

where η is the learning rate (0 < η ≤ 1). The bias is updated as b = b + η(t − y).

• If the output is correct (t = y), weights do not change.
• If wrong, weights move toward (t > y) or away from (t < y) the input.

Training repeats over all examples (epochs) until all are classified correctly. For linearly separable data the rule converges in a finite number of steps (perceptron convergence theorem).