---
title: "Holdout and cross validation"
subject: "Data Mining"
chapter: 3
key: "semester_7/data_mining:holdout-cross-validation"
source: https://www.bctengineer.com/q/semester_7/data_mining/holdout-cross-validation
site: BCT Engineering
---

# Holdout and cross validation

Data Mining · Chapter 3: Classification

- Asked in: year not given
- Video: [Holdout, Cross validation & Bootstrapping 🔥](https://www.youtube.com/watch?v=B9mcXRgSYDo) — Perfect Computer Engineer
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_7/data_mining/holdout-cross-validation
- Whole chapter: https://www.bctengineer.com/md/semester_7/data_mining/chapter-3

## Question

How can you measure classifier accuracy by Holdout and Cross validation methods?

## Short answer

Holdout splits data into training and test sets; accuracy is measured on the test set. k-fold cross-validation divides data into k folds, using each fold as test once, giving a more reliable accuracy estimate.

## Answer

The **accuracy** of a classifier is the percentage of test tuples correctly classified. It must be measured on records **not used for training**, otherwise the estimate is too optimistic.

#### Holdout method
- The data are randomly divided into two **independent** sets: a **training set** (usually two-thirds, 66-70%) used to build the model, and a **test set** (one-third) used to estimate accuracy.
- Simple and fast, but the estimate depends on the particular split; the training set is smaller than the full data.
- **Random subsampling** repeats holdout k times and averages the accuracies, reducing this variance.

```text
Data --> [ Training 2/3 ] --> build model --+
     --> [ Test 1/3     ] --> accuracy  <---+
```

#### k-fold cross-validation
1. Randomly divide the data into k mutually exclusive subsets (folds) D₁, …, D_k of about equal size (k = 10 is usual).
2. Perform k rounds. In round i, use Dᵢ as the test set and the remaining k − 1 folds as the training set.
3. **Accuracy = total number of correct classifications over the k rounds / total number of tuples**. (Or the average of the fold accuracies.)

Every tuple is used once for testing and k − 1 times for training, so the estimate is less biased and more reliable than holdout.

#### Variations
- **Leave-one-out**: k = number of tuples; each fold has one tuple. Good for small data, costly for large.
- **Stratified cross-validation**: each fold keeps approximately the class distribution of the whole data.
- **Bootstrap**: sample with replacement; about 63.2% of tuples go to training, and the rest to testing.

Cross-validation takes about k times more computation than holdout.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. In holdout, how is data usually split?
   A. Half training, half validation, no test
   B. About two-thirds training and one-third test
   C. 90% training and 10% test
   D. All data used for both
   Answer: B. Independent train and test sets, typically 2/3 and 1/3. (Key point: Holdout split)

2. In k-fold cross-validation, each fold is:
   A. Used once as test and k-1 times for training
   B. Used only for training
   C. Used k times as test
   D. Dropped after one round
   Answer: A. Every tuple is tested once. (Key point: k-fold procedure)

3. What is leave-one-out?
   A. k = 2
   B. Holdout with 1% test
   C. k-fold with k equal to number of tuples
   D. Bootstrap with replacement
   Answer: C. Each fold holds a single tuple. (Key point: Leave-one-out)

4. About what share of tuples goes to the training set in bootstrap?
   A. 33.3%
   B. 50%
   C. 90%
   D. 63.2%
   Answer: D. Sampling with replacement gives about 63.2% distinct tuples for training. (Key point: Bootstrap)

## Related in Data Mining

- [Rule based classifier; coverage and accuracy](https://www.bctengineer.com/question.md?key=semester_7%2Fdata_mining%3Arule-based-classifier-measures) — Data Mining, chapter 3

## Related in other subjects

- [Device to measure 7.5 mW microwave power](https://www.bctengineer.com/question.md?key=semester_7%2Frf_and_microwave_engineering%3Ameasure-7-5mw-power) — RF and Microwave Engineering, chapter 8
- [Characterizing control system by speed, accuracy, stability](https://www.bctengineer.com/question.md?key=semester_3%2Fcontrol_systems%3Aspeed-accuracy-stability) — Control Systems, chapter 1