MLSE equalization

Maximum Likelihood Sequence Estimation (MLSE) equalization is a non-linear equalizer that decides on the most probable sequence of transmitted symbols, rather than one symbol at a time. It is optimum for channels with ISI and white Gaussian noise.

 r(t) +---------+ y(k) +----------------+  a^(k)
 ---->| Matched |----->| MLSE: Viterbi  |------->
      | filter  |      | trellis search |  data
      +---------+      +----------------+
           ^                   ^
           |   +-----------+   |
           +---| Channel   |---+
               | estimator |  (CIR h0..hL from
               +-----------+   training sequence)

Working

1. The matched filter maximises SNR and its output is sampled once per symbol to give y(k).
2. The channel estimator uses the known training sequence to estimate the discrete channel impulse response h₀ … h_L.
3. Because ISI comes from the last L symbols, the channel behaves like a finite-state machine with Mᴸ states, drawn as a trellis.
4. For every path in the trellis the expected output ŷ(k) = Σ hⱼ·a(k−j) is formed and the metric |y(k) − ŷ(k)|² is accumulated.
5. The Viterbi algorithm keeps only one survivor (lowest metric) per state at each step, so the search is efficient.
6. The surviving path with the least total metric is the maximum-likelihood sequence and is output.

Merits: minimum sequence error probability; handles deep spectral nulls better than linear equalizers. Limitation: complexity grows exponentially with channel memory (Mᴸ states). GSM receivers use a Viterbi (MLSE) equalizer.