Least squares fit for calibration data

The method of least squares chooses $a$ and $b$ so that $\sum (y_i - a - b x_i)^2$ is a minimum. Solving the normal equations gives

$$b = \frac{n\sum xy - \sum x \sum y}{n\sum x^2 - (\sum x)^2}, \qquad a = \frac{\sum y - b\sum x}{n}$$

$n = 6$:

$$b = \frac{6(222.86) - (30)(30.42)}{6(220) - 30^2} = \frac{1337.16 - 912.6}{1320 - 900} = \frac{424.56}{420} = 1.0109$$ $$a = \frac{30.42 - 1.0109(30)}{6} = 0.0157\ \text{mm}$$

Fitted line: $y = 0.0157 + 1.0109\,x$.

The residuals add up to zero (as they must), and $\sum e^2 = 0.0021\ \text{mm}^2$.

Estimate at $x = 5$ mm: $y = 0.0157 + 1.0109(5) = 5.070$ mm.

• Intercept $a = 0.016$ mm is the zero error of the gauge.
• Slope $b = 1.011$ means the gauge reads about 1.1 % high (scale error); the calibration factor is $1/b = 0.989$.

Answer: $y = 0.0157 + 1.0109x$ (mm); reading at 5 mm = 5.070 mm.