---
title: "Markov chain with features and applications"
subject: "Simulation and Modeling"
chapter: 5
key: "semester_8/simulation_and_modeling:markov-chain-explain-applications"
source: https://www.bctengineer.com/q/semester_8/simulation_and_modeling/markov-chain-explain-applications
site: BCT Engineering
---

# Markov chain with features and applications

Simulation and Modeling · Chapter 5: Markov chains

- Asked in: 2073 Bhadra, 2+4 marks; 2070 Bhadra, 6 marks; 2069 Bhadra (old course), 10 marks (asked 3 times)
- Video: [Markov Chains Clearly Explained! Part - 1](https://www.youtube.com/watch?v=i3AkTO9HLXo) — Normalized Nerd
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_8/simulation_and_modeling/markov-chain-explain-applications
- Whole chapter: https://www.bctengineer.com/md/semester_8/simulation_and_modeling/chapter-5

## Question

Define / explain Markov chain with an appropriate example. List the key features and applications of Markov chain.

## Short answer

A Markov chain is a memoryless stochastic process where the next state depends only on the current state. Example: weather (Sunny/Rainy) with fixed transition probabilities. Key features: finite states, memoryless property, transition matrix with row sums 1, homogeneity, n-step probabilities via P^n, steady-state convergence. Applications: brand switching, weather forecasting, PageRank, speech recognition, reliability, inventory, MCMC.

## Answer

#### Definition

A **Markov chain** is a stochastic process in which the system moves between a finite set of states, and the probability of the next state depends only on the present state, not on how the system reached it. This is the **Markov (memoryless) property**:

P(X(n+1) = j | X(n) = i, X(n-1), ..., X(0)) = P(X(n+1) = j | X(n) = i) = pij

The probabilities pij are the **transition probabilities**. They are collected in a **transition matrix P**, where each row sums to 1.

#### Example: weather

Two states: Sunny (S) and Rainy (R). If today is sunny, tomorrow is sunny with probability 0.8 and rainy with 0.2. If today is rainy, tomorrow is rainy with 0.6 and sunny with 0.4.

```text
          0.2
     +---------->+
  (S)             (R)   S->S 0.8, R->R 0.6
     +<----------+
          0.4
```

P = [[0.8, 0.2], [0.4, 0.6]] (rows and columns in the order S, R).

If today is sunny, the distribution after 2 days is the first row of P squared: P(S) = 0.8 x 0.8 + 0.2 x 0.4 = 0.72 and P(R) = 0.28. After many days the chain settles to the **steady state** pi, found from pi = pi P with sum of pi = 1: pi(S) = 2/3, pi(R) = 1/3.

#### Key features

1. Finite (or countable) set of states.
2. **Memoryless property**: the next state depends only on the current state.
3. Transition matrix is square, with non-negative entries and each row summing to 1.
4. **Homogeneous**: transition probabilities do not change with time.
5. n-step probabilities are given by the matrix power P^n, and the state distribution after n steps is p(n) = p(0) P^n.
6. For a regular (irreducible, aperiodic) chain, p(n) converges to a unique steady-state vector pi, independent of the starting state, satisfying pi = pi P.
7. States may be transient, recurrent or absorbing.

#### Applications

- Market share and brand switching (Coke/Pepsi, Sony/Samsung).
- Weather prediction and queue length modelling.
- Google PageRank and web page navigation.
- Speech recognition, text generation and bioinformatics (hidden Markov models).
- Reliability and maintenance: machine working or failed states.
- Inventory and stock level changes; credit rating and customer lifetime.
- Computer systems: modelling job states and packet traffic.
- Random number and Monte Carlo methods (MCMC).

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. What is the Markov (memoryless) property?
   A. The next state depends only on the present state, not on how it was reached
   B. The next state depends on all previous states equally
   C. The next state is always the same as the present state
   D. The next state is chosen at random with equal probability
   Answer: A. P(next | present, past) = P(next | present). (Key point: Markov property)

2. Which is true of a transition matrix?
   A. Each column sums to 1 and entries may be negative
   B. It is square with non-negative entries and each row sums to 1
   C. It is rectangular
   D. Its entries sum to 1 overall
   Answer: B. Row sums equal 1 because from each state the system must go somewhere. (Key point: Transition matrix)

3. With P = [[0.8,0.2],[0.4,0.6]] and sunny today, what is P(sunny) after 2 days?
   A. 0.64
   B. 0.80
   C. 0.72
   D. 0.60
   Answer: C. 0.8x0.8 + 0.2x0.4 = 0.72. (Key point: Weather example)

4. How is the steady-state vector found?
   A. Solve pi = P^2 pi with sum 0
   B. Take the average of the rows
   C. Multiply P by its transpose
   D. Solve pi = pi P with sum of pi = 1
   Answer: D. For the weather example it gives pi(S) = 2/3 and pi(R) = 1/3. (Key point: Steady state)

## Related in Simulation and Modeling

- [Honda-Yamaha Markov, key features](https://www.bctengineer.com/question.md?key=semester_8%2Fsimulation_and_modeling%3Amarkov-honda-yamaha) — Simulation and Modeling, chapter 5
- [Markov applications; Sony-Samsung 75/85](https://www.bctengineer.com/question.md?key=semester_8%2Fsimulation_and_modeling%3Amarkov-sony-samsung-75-85) — Simulation and Modeling, chapter 5, asked 2×

## Related in other subjects

- [Key features of infrastructure development policies](https://www.bctengineer.com/question.md?key=semester_7%2Ftechnology_environment_and_society%3Ainfrastructure-development-policies) — Technology, Environment and Society, chapter 2
- [Synchronous motor features, applications and excitation](https://www.bctengineer.com/question.md?key=semester_5%2Felectric_machines_ii%3Amotor-features-applications-excitation) — Electric Machines II, chapter 2