ISI and Nyquist pulse shaping criterion

Inter symbol interference

Inter symbol interference (ISI) is distortion in which the energy of one symbol spreads into the time slots of nearby symbols. It happens because the channel and filters have limited bandwidth, so pulses get wider in time. At a sampling instant, the received sample then holds tails of other pulses as well as the wanted symbol:

$$y(iT_b) = \mu a_i + \underbrace{\mu \sum_{k \ne i} a_k\, p\big((i-k)T_b\big)}_{\text{ISI}} + n(iT_b)$$

Nyquist criterion for zero ISI

If $p(t)$ is the overall pulse (transmit filter, channel and receive filter), ISI is zero when

$$p(nT_b) = \begin{cases} 1, & n = 0 \\ 0, & n \ne 0 \end{cases}$$

In the frequency domain this is

$$\sum_{k=-\infty}^{\infty} P\!\left(f - \frac{k}{T_b}\right) = T_b \quad \text{(constant)}$$

That is, copies of the pulse spectrum shifted by multiples of $R_b = 1/T_b$ must add up to a flat spectrum.

Nyquist pulse shaping for mitigation of ISI

1. Ideal Nyquist channel (minimum bandwidth). Choose a rectangular spectrum of width $B_0 = R_b/2$:

$$P(f) = \frac{1}{R_b}\,\text{rect}\!\left(\frac{f}{R_b}\right) \;\Rightarrow\; p(t) = \frac{\sin \pi R_b t}{\pi R_b t}$$

The sinc pulse is 1 at $t = 0$ and 0 at every other $nT_b$, so ISI is zero. $B_0 = R_b/2$ is the Nyquist bandwidth, and $R_b = 2B_0$ is the Nyquist rate. In practice it fails because the brick-wall filter cannot be built and the tails decay only as $1/|t|$, so any timing jitter adds up to large ISI.

2. Raised-cosine pulse shaping (practical). The spectrum rolls off smoothly with a cosine shape, with odd symmetry about $B_0$, so the folded spectrum is still flat:

$$p(t) = \text{sinc}(2B_0 t)\,\frac{\cos(2\pi\alpha B_0 t)}{1 - 16\alpha^2B_0^2t^2}$$ $$B_T = B_0(1 + \alpha) = \frac{R_b}{2}(1 + \alpha)$$

 P(f)
  |______
  |      \       alpha = 0   : brick wall, B = Rb/2
  |       \      alpha = 0.5 : B = 0.75 Rb
  |        \__   alpha = 1   : B = Rb
  +--------+--+---> f
          B0  B0(1+a)

The pulse shaping is usually split equally between transmitter and receiver as root-raised-cosine filters, which also gives matched filtering. Any ISI that remains because the channel is not ideal is removed with equalisers.