Skip to main content

Quantization, rounding vs truncation, dead band

Short answer

Quantization maps analog to digital, introduces noise power ~q²/12, limiting SNR to ~6 dB/bit. Rounding is better (zero-mean error −q/2 to q/2, power q²/12) than truncation (bias, error power q²/3). In recursive filters, product quantization may cause zero-input limit cycles—persistent small oscillations. Dead band is the output range [−β, β] where limit cycles are sustained; for a first-order filter |y| ≤ (q/2)/(1−|a|).

What is the importance of quantization in Digital Signal Processing? Which one is better, rounding or truncation? Explain about limit cycles in recursive system. Define dead band.

Importance of quantization

A digital system has finite word length, so the continuous amplitude of a sampled signal must be mapped to a finite set of levels (A/D conversion). Quantization is also applied to filter coefficients and to the results of multiplications. It is what makes DSP implementable on real hardware. Its price is an error (noise) with power of about q²/12 for step size q, so the word length (number of bits) decides the signal-to-noise ratio, about 6 dB per bit.

Rounding or truncation

Rounding is better. Its error lies in (−q/2, +q/2], is symmetric and has zero mean, so its total error power is only q²/12. Truncation error lies in (−q, 0]: it has the same variance (q²/12) but a mean of −q/2 (a DC bias), so it adds a systematic offset and its mean-square error is q²/3, four times larger. Truncation is only simpler to implement (just drop bits).

Limit cycles in a recursive system

In a recursive (IIR) filter, the product quantization is inside the feedback loop. Even with zero (or constant) input, the rounding of the products can make the output oscillate with a small steady amplitude instead of decaying to zero. These sustained oscillations are called limit cycles. There are two kinds:

  • Zero-input limit cycles: small-amplitude oscillation or constant output caused by rounding in the feedback products.
  • Overflow oscillations: large-amplitude oscillation caused by arithmetic overflow (wrap-around) in adders.

Example: y[n] = 0.9y[n−1] + x[n] with rounding to integers. For x[n] = 0 and y[−1] = 5, rounding 0.9×5 = 4.5 gives 5 again, and the output stays at 5 forever, though ideally it should decay to 0.

Dead band

The range of output values [−β, +β] in which a limit cycle can get stuck is the dead band. For a first-order filter y[n] = a·y[n−1] + x[n], the dead band is |y| ≤ (q/2) / (1 − |a|). While the output is inside this band, rounding makes the effective pole sit on the unit circle, so the output does not decay further. Using more bits (smaller q) reduces the dead band.

✓ Answer checked by Claude Opus 5.5 (high effort)

Related questions

All Digital Signal Analysis and Processing chapter 4 questions →