Skip to main content

Markov chain with features and applications

Short answer

A Markov chain is a memoryless stochastic process where the next state depends only on the current state. Example: weather (Sunny/Rainy) with fixed transition probabilities. Key features: finite states, memoryless property, transition matrix with row sums 1, homogeneity, n-step probabilities via P^n, steady-state convergence. Applications: brand switching, weather forecasting, PageRank, speech recognition, reliability, inventory, MCMC.

Define / explain Markov chain with an appropriate example. List the key features and applications of Markov chain.

Definition

A Markov chain is a stochastic process in which the system moves between a finite set of states, and the probability of the next state depends only on the present state, not on how the system reached it. This is the Markov (memoryless) property:

P(X(n+1) = j | X(n) = i, X(n-1), ..., X(0)) = P(X(n+1) = j | X(n) = i) = pij

The probabilities pij are the transition probabilities. They are collected in a transition matrix P, where each row sums to 1.

Example: weather

Two states: Sunny (S) and Rainy (R). If today is sunny, tomorrow is sunny with probability 0.8 and rainy with 0.2. If today is rainy, tomorrow is rainy with 0.6 and sunny with 0.4.

          0.2
     +---------->+
  (S)             (R)   S->S 0.8, R->R 0.6
     +<----------+
          0.4

P = [[0.8, 0.2], [0.4, 0.6]] (rows and columns in the order S, R).

If today is sunny, the distribution after 2 days is the first row of P squared: P(S) = 0.8 x 0.8 + 0.2 x 0.4 = 0.72 and P(R) = 0.28. After many days the chain settles to the steady state pi, found from pi = pi P with sum of pi = 1: pi(S) = 2/3, pi(R) = 1/3.

Key features

  1. Finite (or countable) set of states.
  2. Memoryless property: the next state depends only on the current state.
  3. Transition matrix is square, with non-negative entries and each row summing to 1.
  4. Homogeneous: transition probabilities do not change with time.
  5. n-step probabilities are given by the matrix power P^n, and the state distribution after n steps is p(n) = p(0) P^n.
  6. For a regular (irreducible, aperiodic) chain, p(n) converges to a unique steady-state vector pi, independent of the starting state, satisfying pi = pi P.
  7. States may be transient, recurrent or absorbing.

Applications

  • Market share and brand switching (Coke/Pepsi, Sony/Samsung).
  • Weather prediction and queue length modelling.
  • Google PageRank and web page navigation.
  • Speech recognition, text generation and bioinformatics (hidden Markov models).
  • Reliability and maintenance: machine working or failed states.
  • Inventory and stock level changes; credit rating and customer lifetime.
  • Computer systems: modelling job states and packet traffic.
  • Random number and Monte Carlo methods (MCMC).

✓ Answer checked by Claude Opus 5.5 (high effort)

Related questions

All Simulation and Modeling chapter 5 questions →