--- title: "Least squares fit for calibration data" subject: "Metrology" chapter: 2 key: "semester_3/metrology:least-squares-calibration-line" source: https://www.bctengineer.com/q/semester_3/metrology/least-squares-calibration-line site: BCT Engineering --- # Least squares fit for calibration data Metrology · Chapter 2: Errors in measurement - Asked in: year not given, 6 marks - Video: [What is Simple Linear Regression in Statistics | Linear Regression Using Least Squares Method](https://www.youtube.com/watch?v=JAJ-bIp5Mvw) — Digital E-Learning - Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14 - Page: https://www.bctengineer.com/q/semester_3/metrology/least-squares-calibration-line - Whole chapter: https://www.bctengineer.com/md/semester_3/metrology/chapter-2 ## Question A dial gauge is calibrated against slip gauges. The applied displacement $x$ (mm) and the dial reading $y$ (mm) are: (0, 0.02), (2, 2.05), (4, 4.03), (6, 6.10), (8, 8.08), (10, 10.14). Using the method of least squares fit the line $y = a + bx$, find the residuals and use the line to estimate the reading for a true displacement of 5 mm. State what the slope and intercept mean. ## Short answer Using least squares, y = 0.0157 + 1.0109x (mm). Residuals sum to zero. At x=5 mm, estimated reading = 5.070 mm. Intercept 0.016 mm is zero error; slope 1.011 indicates scale error, gage reads about 1.1% high. ## Answer The method of least squares chooses $a$ and $b$ so that $\sum (y_i - a - b x_i)^2$ is a minimum. Solving the normal equations gives $$ b = \frac{n\sum xy - \sum x \sum y}{n\sum x^2 - (\sum x)^2}, \qquad a = \frac{\sum y - b\sum x}{n} $$ | $x$ | $y$ | $xy$ | $x^2$ | |---|---|---|---| | 0 | 0.02 | 0 | 0 | | 2 | 2.05 | 4.10 | 4 | | 4 | 4.03 | 16.12 | 16 | | 6 | 6.10 | 36.60 | 36 | | 8 | 8.08 | 64.64 | 64 | | 10 | 10.14 | 101.40 | 100 | | **30** | **30.42** | **222.86** | **220** | $n = 6$: $$ b = \frac{6(222.86) - (30)(30.42)}{6(220) - 30^2} = \frac{1337.16 - 912.6}{1320 - 900} = \frac{424.56}{420} = 1.0109 $$ $$ a = \frac{30.42 - 1.0109(30)}{6} = 0.0157\ \text{mm} $$ Fitted line: $y = 0.0157 + 1.0109\,x$. | $x$ | $y$ | fitted | residual | |---|---|---|---| | 0 | 0.02 | 0.0157 | +0.0043 | | 2 | 2.05 | 2.0374 | +0.0126 | | 4 | 4.03 | 4.0591 | -0.0291 | | 6 | 6.10 | 6.0809 | +0.0191 | | 8 | 8.08 | 8.1026 | -0.0226 | | 10 | 10.14 | 10.1243 | +0.0157 | The residuals add up to zero (as they must), and $\sum e^2 = 0.0021\ \text{mm}^2$. Estimate at $x = 5$ mm: $y = 0.0157 + 1.0109(5) = 5.070$ mm. - Intercept $a = 0.016$ mm is the zero error of the gauge. - Slope $b = 1.011$ means the gauge reads about 1.1 % high (scale error); the calibration factor is $1/b = 0.989$. **Answer: $y = 0.0157 + 1.0109x$ (mm); reading at 5 mm = 5.070 mm.** ## Check yourself The answer's key points as multiple-choice questions, with the correct option and why. 1. The method of least squares chooses a and b so that: A. The largest residual is zero B. The sum of the squares of the residuals is a minimum C. The line passes through every point D. The sum of the y values equals the sum of the x values Answer: B. Least squares minimises the sum of (yi - a - b xi)^2, which leads to the normal equations. (Key point: Least squares principle) 2. The slope in the answer is b = [n Σxy - Σx Σy] / [n Σx^2 - (Σx)^2]. With Σx = 30, Σy = 30.42, Σxy = 222.86, Σx^2 = 220 and n = 6, b is about: A. 0.989 B. 1.0109 C. 1.1 D. 0.0157 Answer: B. b = (1337.16 - 912.6)/(1320 - 900) = 424.56/420 = 1.0109. (Key point: Slope formula) 3. What does the intercept a = 0.016 mm represent? A. Scale error of the gauge B. The calibration factor C. The zero error of the gauge D. The standard deviation Answer: C. The reading when the true displacement is zero is the zero error of the gauge. (Key point: Meaning of intercept) 4. The slope b = 1.011 indicates the gauge: A. Reads about 1.1 % high (scale error); calibration factor 1/b = 0.989 B. Reads about 1.1 % low C. Has a zero error of 1.011 mm D. Has no scale error Answer: A. A slope greater than 1 means the dial moves more than the true displacement, so it reads high by about 1.1 %, and the correction factor is 1/b. (Key point: Meaning of slope) 5. What must be true of the residuals of a least squares fit with an intercept? A. They all have equal sign B. They add up to zero C. Their squares add up to zero D. They equal the zero error Answer: B. The residuals sum to zero (as they must); only their squares sum to a minimum value, here 0.0021 mm^2. (Key point: Residual property) ## Related in Metrology - [Slip gauges: wringing and building a stack](https://www.bctengineer.com/question.md?key=semester_3%2Fmetrology%3Aslip-gauge-wringing-and-stack) — Metrology, chapter 4 - [Limits of a 40 H7/g6 hole-shaft fit](https://www.bctengineer.com/question.md?key=semester_3%2Fmetrology%3Afit-40-h7-g6-calculation) — Metrology, chapter 9 - [Constructing and interpreting X-bar and R charts](https://www.bctengineer.com/question.md?key=semester_3%2Fmetrology%3Ax-bar-r-chart-construction) — Metrology, chapter 15 - [Optical lever comparator magnification](https://www.bctengineer.com/question.md?key=semester_3%2Fmetrology%3Aoptical-lever-comparator-numerical) — Metrology, chapter 7 ## Related in other subjects - [Calibration line and nonlinearity of transducer](https://www.bctengineer.com/question.md?key=semester_4%2Finstrumentation_and_measurement%3Acalibration-least-squares-nonlinearity) — Instrumentation and Measurement, chapter 3 - [Quadratic least squares fit](https://www.bctengineer.com/question.md?key=semester_4%2Fnumerical_method%3Aquadratic-fit-least-squares) — Numerical Method, chapter 4 - [Two-compartment vessel gauges: reading of gauge C](https://www.bctengineer.com/question.md?key=semester_2%2Ffundamental_of_thermodynamics_and_heat_transfer%3Atwo-compartment-gauges-a4bar-b2bar) — Fundamental of Thermodynamics & Heat Transfer, chapter 1 - [Least squares: y = ax + b + c/x](https://www.bctengineer.com/question.md?key=semester_4%2Fnumerical_method%3Aleast-squares-ax-b-c-over-x) — Numerical Method, chapter 4