Define TM mode; TM11 cut-off 5×3 cm

Transverse magnetic (TM) mode

A TM mode is a wave mode in which the magnetic field has no component along the direction of propagation (H_z = 0), while the electric field has a longitudinal component E_z ≠ 0. In a rectangular guide

E_z = E₀ sin(mπx/a) sin(nπy/b) e^{−jβz}

with m ≥ 1 and n ≥ 1 (if either is zero all the field components vanish). The lowest TM mode is TM₁₁.

Cut-off of TM₁₁ (a = 5 cm, b = 3 cm, air-filled, perfect walls)

f_c = (c/2) √[(m/a)² + (n/b)²]

(1/a)² = (1/0.05)² = 400
(1/b)² = (1/0.03)² = 1111.1
sum    = 1511.1,   √ = 38.87 m⁻¹

f_c(TM₁₁) = 1.5×10⁸ × 38.87 = 5.83×10⁹ Hz

Answer: f_c(TM₁₁) = 5.83 GHz (λ_c = 5.14 cm). For comparison the dominant TE₁₀ has f_c = 3 GHz.