---
title: "Calibration line and nonlinearity of transducer"
subject: "Instrumentation and Measurement"
chapter: 3
key: "semester_4/instrumentation_and_measurement:calibration-least-squares-nonlinearity"
source: https://www.bctengineer.com/q/semester_4/instrumentation_and_measurement/calibration-least-squares-nonlinearity
site: BCT Engineering
---

# Calibration line and nonlinearity of transducer

Instrumentation and Measurement · Chapter 3: Static Characteristics of Measurement System

- Asked in: year not given, 8 marks
- Video: [ENGR 313 - 01.04 Linear Calibration](https://www.youtube.com/watch?v=Rx7umkkZ2LE) — ENGR 313 - Circuits and Instrumentation
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_4/instrumentation_and_measurement/calibration-least-squares-nonlinearity
- Whole chapter: https://www.bctengineer.com/md/semester_4/instrumentation_and_measurement/chapter-3

## Question

A pressure transducer was calibrated with the following data.

| Pressure (bar) | 0 | 2 | 4 | 6 | 8 | 10 |
|---|---|---|---|---|---|---|
| Output (V) | 0.02 | 0.98 | 2.05 | 3.00 | 3.98 | 5.01 |

Fit the least-squares straight line, find the sensitivity and the zero offset, and calculate the maximum nonlinearity as a percentage of full-scale output (FSO).

## Short answer

Fit y = mx + c by least squares; sensitivity (slope) = 0.499 V/bar, zero offset = 0.0138 V. Maximum deviation is 0.0419 V at 4 bar, full-scale output 4.99 V, giving nonlinearity 0.84% of FSO.

## Answer

Method: fit $y = mx + c$ by least squares, then find the largest difference between the data and the line.

#### Least-squares fit
For $n = 6$ points: $\sum x = 30$, $\sum y = 15.04$, $\sum x^2 = 220$, $\sum xy = 110.10$ (units bar, V).

Here $\sum xy = 0 + 1.96 + 8.20 + 18.00 + 31.84 + 50.10 = 110.10$.

$$
m = \frac{n\sum xy - \sum x \sum y}{n\sum x^2 - (\sum x)^2} = \frac{6(110.10) - 30(15.04)}{6(220) - 900} = \frac{660.6 - 451.2}{420} = \frac{209.4}{420} = 0.4986\ \text{V/bar}
$$

$$
c = \frac{\sum y - m\sum x}{n} = \frac{15.04 - 0.4986(30)}{6} = 0.0138\ \text{V}
$$

So the line is $y = 0.4986\,x + 0.0138$.

| x (bar) | y (V) | Fitted (V) | Deviation (V) |
|---|---|---|---|
| 0 | 0.02 | 0.0138 | +0.0062 |
| 2 | 0.98 | 1.0110 | -0.0310 |
| 4 | 2.05 | 2.0081 | +0.0419 |
| 6 | 3.00 | 3.0052 | -0.0052 |
| 8 | 3.98 | 4.0024 | -0.0224 |
| 10 | 5.01 | 4.9995 | +0.0105 |

#### Nonlinearity
Maximum deviation $= 0.0419$ V at 4 bar.

Full-scale output $= 5.01 - 0.02 = 4.99$ V.

$$
\text{Nonlinearity} = \frac{0.0419}{4.99}\times 100 = 0.84\%\ \text{of FSO}
$$

**Answer: sensitivity (slope) $= 0.499$ V/bar, zero offset $= 0.0138$ V, maximum nonlinearity $= 0.84\%$ of FSO.**

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. What is the sensitivity found from the fit?
   A. 0.0138 V/bar
   B. 0.84 V/bar
   C. 0.499 V/bar
   D. 4.99 V/bar
   Answer: C. Sensitivity is the slope m = 0.4986, about 0.499 V/bar. (Key point: Sensitivity)

2. What is the zero offset and how is it found?
   A. c = (Σy - mΣx)/n = 0.0138 V
   B. c = 5.01 V
   C. c = m/n = 0.083 V
   D. c = 0.0419 V
   Answer: A. The intercept c = (15.04 - 0.4986×30)/6 = 0.0138 V. (Key point: Zero offset)

3. How is nonlinearity as % of FSO obtained?
   A. 0.0419/5.01 × 100 using the fitted slope
   B. Maximum deviation (0.0419 V) divided by FSO (4.99 V) × 100 = 0.84%
   C. Average deviation divided by the slope
   D. Slope divided by FSO
   Answer: B. The largest deviation is 0.0419 V at 4 bar and FSO = 5.01 - 0.02 = 4.99 V, giving 0.84%. (Key point: Nonlinearity)

## Related in Instrumentation and Measurement

- [Zero drift and sensitivity drift](https://www.bctengineer.com/question.md?key=semester_4%2Finstrumentation_and_measurement%3Azero-drift-sensitivity-drift) — Instrumentation and Measurement, chapter 3
- [Potentiometer transducer and loading error](https://www.bctengineer.com/question.md?key=semester_4%2Finstrumentation_and_measurement%3Apotentiometer-transducer-loading-error) — Instrumentation and Measurement, chapter 5
- [Second-order transducer frequency and step response](https://www.bctengineer.com/question.md?key=semester_4%2Finstrumentation_and_measurement%3Asecond-order-transducer-numerical) — Instrumentation and Measurement, chapter 4

## Related in other subjects

- [Least squares fit for calibration data](https://www.bctengineer.com/question.md?key=semester_3%2Fmetrology%3Aleast-squares-calibration-line) — Metrology, chapter 2
- [Fit straight line: humidity and moisture](https://www.bctengineer.com/question.md?key=semester_5%2Fprobability_and_statistics%3Ahumidity-moisture-straight-line-fit) — Probability and Statistics, chapter 5
- [Quadratic least squares fit](https://www.bctengineer.com/question.md?key=semester_4%2Fnumerical_method%3Aquadratic-fit-least-squares) — Numerical Method, chapter 4
- [Piezoelectric transducer sensitivity and output voltage](https://www.bctengineer.com/question.md?key=semester_4%2Finstrumentation_i%3Apiezoelectric-transducer-output-voltage) — Instrumentation I, chapter 3