Derive linear least squares

Fitting a straight line y = ax + b to n data points by least squares means choosing a and b so that the total squared vertical error is smallest.

Let (xᵢ, yᵢ), i = 1 … n, be the data and y = ax + b the line to be fitted. The residual (error) at each point is

eᵢ = yᵢ − (axᵢ + b)

The least squares method chooses a and b so that the sum of squares of the residuals is minimum:

S(a, b) = Σ eᵢ² = Σ (yᵢ − axᵢ − b)²

For S to be minimum, its partial derivatives with respect to a and b must be zero.

∂S/∂b = −2 Σ (yᵢ − axᵢ − b) = 0   →   Σyᵢ = a Σxᵢ + n b
∂S/∂a = −2 Σ xᵢ(yᵢ − axᵢ − b) = 0 →   Σxᵢyᵢ = a Σxᵢ² + b Σxᵢ

These are the normal equations:

Σy  = a Σx  + n b                 ... (1)
Σxy = a Σx² + b Σx                ... (2)

Solving (1) and (2) by Cramer's rule:

a = (n Σxy − Σx Σy) / (n Σx² − (Σx)²)
b = (Σy − a Σx) / n  =  (Σx² Σy − Σx Σxy) / (n Σx² − (Σx)²)

The second derivatives are ∂²S/∂a² = 2Σx² > 0 and ∂²S/∂b² = 2n > 0, and the Hessian determinant 4[nΣx² − (Σx)²] > 0 (unless all xᵢ are equal), so this stationary point is a true minimum.

Procedure: make a table of x, y, xy, x²; find Σx, Σy, Σxy, Σx²; substitute in the formulas for a and b; the fitted line is y = ax + b.