Skip to main content

Least squares significance, normal equations

Short answer

Least squares minimizes the sum of squared vertical deviations, giving a best-fit curve that uses all noisy data and avoids sign cancellation. For y = ax + b, setting partial derivatives of S = sum(y_i - a x_i - b)^2 to zero yields sum y = nb + a sum x and sum xy = b sum x + a sum x^2.

What is the practical significance of the least squares method of curve fitting? Derive the normal equations to fit a given set of data to a linear equation (y = ax + b).

Practical significance of the least squares method

Experimental data contain measurement errors, so the points do not lie exactly on any curve and an interpolating polynomial through every point would follow the noise. The least squares method finds the curve of a chosen form (line, parabola, exponential, power, …) that passes as close as possible to all points on the whole, by minimising the sum of the squares of the vertical deviations.

  • It gives the best-fit relation between variables from noisy data, so that values can be predicted (interpolation and extrapolation) and trends studied.
  • It uses all the data points, so one bad reading has less effect than in an exact-fit polynomial.
  • Squaring the errors removes the sign problem (positive and negative errors cannot cancel) and gives smooth equations, the normal equations, which are easy to solve.
  • Non-linear laws (y = axᵇ, y = abˣ, y = aeᵇˣ) can be reduced to a straight line by taking logarithms, so the same method fits them.
  • It is the basis of regression, calibration of instruments, curve fitting in engineering and science, and signal and data analysis.

Normal equations for y = ax + b

Let (xᵢ, yᵢ), i = 1 … n, be the data. The error at each point is eᵢ = yᵢ − axᵢ − b. Minimise

S = Σ (yᵢ − axᵢ − b)²

Setting the partial derivatives to zero:

∂S/∂a = −2 Σ xᵢ(yᵢ − axᵢ − b) = 0
∂S/∂b = −2 Σ (yᵢ − axᵢ − b) = 0

which give the normal equations

Σy  = n b + a Σx
Σxy = b Σx + a Σx²

Solving them:

a = (n Σxy − Σx Σy) / (n Σx² − (Σx)²)
b = (Σy − a Σx) / n

The fitted line is y = ax + b.

✓ Answer checked by Claude Opus 5.5 (high effort)

Related questions

All Numerical Method chapter 4 questions →