ISI, Nyquist zero-ISI conditions and practical constraints

Inter Symbol Interference (ISI)

ISI is the distortion in which the pulse of one symbol spreads into the time slots of neighbouring symbols, so that the sample taken at the decision instant contains contributions from earlier and later symbols. It is caused by the limited bandwidth and dispersion of the channel and filters. The received sample at $t = kT_b$ is

$$y(kT_b) = \mu a_k + \underbrace{\mu\sum_{n\ne k} a_n\, p\big((k-n)T_b\big)}_{\text{ISI}} + n(kT_b)$$

ISI raises the bit error rate even when there is no noise, and it closes the eye in the eye pattern.

Nyquist conditions for zero ISI

Let $p(t)$ be the overall pulse (transmit filter, channel and receive filter) with $p(0)=1$.

• Time domain: $p(nT_b) = \begin{cases} 1, & n = 0 \\ 0, & n \ne 0 \end{cases}$, i.e. the pulse must pass through zero at all other sampling instants.
• Frequency domain: $\displaystyle\sum_{n=-\infty}^{\infty} P\left(f - \frac{n}{T_b}\right) = T_b$ (constant), i.e. the shifted copies of the spectrum must add to a flat value.

The simplest solution is the ideal Nyquist channel $P(f) = T_b\,\text{rect}(fT_b)$, bandwidth $B_0 = \frac{R_b}{2}$, giving $p(t) = \text{sinc}(t/T_b)$.

Physical constraints in implementing the Nyquist criterion

1. Unrealizable filter: the ideal response needs a flat spectrum up to $R_b/2$ and an abrupt fall to zero (brick-wall). Such a filter is non-causal and cannot be built.
2. Infinite, slowly decaying pulse: the sinc pulse lasts from $-\infty$ to $\infty$ and its tails decay only as $1/|t|$. Any truncation brings back ISI.
3. Timing sensitivity: because the tails are large, a small timing error (jitter) in the sampling instant gives large ISI; the sum of ISI terms can even diverge.
4. Channel uncertainty: the criterion applies to the overall response, but the real channel response is not known exactly and may change with time, so fixed filters cannot meet it exactly.
5. Bandwidth penalty in practice: practical solutions such as the raised cosine pulse need extra bandwidth, $B = \frac{R_b}{2}(1+\alpha)$ with roll-off $0 \le \alpha \le 1$, and equalizers or partial response (duobinary) coding are needed to handle the remaining problems.