---
title: "Derive linear least squares"
subject: "Numerical Method"
chapter: 4
key: "semester_4/numerical_method:least-squares-linear-derive"
source: https://www.bctengineer.com/q/semester_4/numerical_method/least-squares-linear-derive
site: BCT Engineering
---

# Derive linear least squares

Numerical Method · Chapter 4: Interpolation

- Asked in: 2071 Magh, 8 marks
- Video: [How to Derive the Best Slope and Intercept in a Simple Linear Regression](https://www.youtube.com/watch?v=20tt07vSRV0) — The Standard Deviant
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_4/numerical_method/least-squares-linear-derive
- Whole chapter: https://www.bctengineer.com/md/semester_4/numerical_method/chapter-4

## Question

Derive expression for least square method of fitting a linear curve.

## Short answer

Minimize S=Σ(yᵢ−axᵢ−b)². Normal equations: Σy=aΣx+nb, Σxy=aΣx²+bΣx. Solve: a=(nΣxy−ΣxΣy)/(nΣx²−(Σx)²), b=(Σy−aΣx)/n.

## Answer

Fitting a straight line y = ax + b to n data points by least squares means choosing a and b so that the total squared vertical error is smallest.

Let (xᵢ, yᵢ), i = 1 … n, be the data and y = ax + b the line to be fitted. The residual (error) at each point is

```text
eᵢ = yᵢ − (axᵢ + b)
```

The least squares method chooses a and b so that the sum of squares of the residuals is minimum:

```text
S(a, b) = Σ eᵢ² = Σ (yᵢ − axᵢ − b)²
```

For S to be minimum, its partial derivatives with respect to a and b must be zero.

```text
∂S/∂b = −2 Σ (yᵢ − axᵢ − b) = 0   →   Σyᵢ = a Σxᵢ + n b
∂S/∂a = −2 Σ xᵢ(yᵢ − axᵢ − b) = 0 →   Σxᵢyᵢ = a Σxᵢ² + b Σxᵢ
```

These are the **normal equations**:

```text
Σy  = a Σx  + n b                 ... (1)
Σxy = a Σx² + b Σx                ... (2)
```

Solving (1) and (2) by Cramer's rule:

```text
a = (n Σxy − Σx Σy) / (n Σx² − (Σx)²)
b = (Σy − a Σx) / n  =  (Σx² Σy − Σx Σxy) / (n Σx² − (Σx)²)
```

The second derivatives are ∂²S/∂a² = 2Σx² > 0 and ∂²S/∂b² = 2n > 0, and the Hessian determinant 4[nΣx² − (Σx)²] > 0 (unless all xᵢ are equal), so this stationary point is a true minimum.

**Procedure:** make a table of x, y, xy, x²; find Σx, Σy, Σxy, Σx²; substitute in the formulas for a and b; the fitted line is y = ax + b.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. What is the residual at point i?
   A. yi + a xi + b
   B. a xi + b
   C. yi/xi
   D. yi - (a xi + b)
   Answer: D. Vertical difference between data and line. (Key point: Residual)

2. How are a and b found?
   A. Set ∂S/∂a = 0 and ∂S/∂b = 0
   B. Set S = 0
   C. Set a = b
   D. Differentiate with respect to x
   Answer: A. Partial derivatives of S vanish at the minimum. (Key point: Minimisation)

3. What is the slope a?
   A. (nΣxy - ΣxΣy)/(nΣx² - (Σx)²)
   B. (Σy - Σx)/n
   C. (nΣx² - (Σx)²)/(nΣxy - ΣxΣy)
   D. (Σxy - ΣxΣy)/Σx²
   Answer: A. From Cramer's rule. (Key point: Slope)

4. Why is the stationary point a minimum?
   A. S'' terms 2Σx² and 2n are positive and the Hessian determinant is positive
   B. a is positive
   C. S is negative
   D. n is large
   Answer: A. Positive second derivatives and positive Hessian determinant. (Key point: Minimum check)

## Related in Numerical Method

- [Least squares significance, normal equations](https://www.bctengineer.com/question.md?key=semester_4%2Fnumerical_method%3Aleast-squares-significance-normal) — Numerical Method, chapter 4
- [Quadratic least squares fit](https://www.bctengineer.com/question.md?key=semester_4%2Fnumerical_method%3Aquadratic-fit-least-squares) — Numerical Method, chapter 4
- [Least squares: y = ax + b + c/x](https://www.bctengineer.com/question.md?key=semester_4%2Fnumerical_method%3Aleast-squares-ax-b-c-over-x) — Numerical Method, chapter 4

## Related in other subjects

- [Least squares fit for calibration data](https://www.bctengineer.com/question.md?key=semester_3%2Fmetrology%3Aleast-squares-calibration-line) — Metrology, chapter 2
- [Regular expression and FSA: at least three b's](https://www.bctengineer.com/question.md?key=semester_4%2Fdiscrete_structure%3Afsa-at-least-three-b) — Discrete Structure, chapter 2
- [Derive expression for tile drain spacing](https://www.bctengineer.com/question.md?key=semester_6%2Firrigation_and_drainage_engineering%3Atile-drain-spacing-derivation) — Irrigation and Drainage Engineering, chapter 9, asked 8×
- [Derive development length expression](https://www.bctengineer.com/question.md?key=semester_7%2Fdesign_of_rcc_structure%3Aderive-development-length-expression) — Design of RCC Structure, chapter 6, asked 6×