Line of intersection lies on cone

How I read the question. "The plane through OX and OY" is a misprint for two planes, one through OX and one through OY, that include an angle α. Only one plane (z = 0) contains both axes, so there would be no line of intersection. With this reading the result is exactly the stated cone.

Setting up the planes

• A plane through the x-axis OX has equation y = p z (it contains every point (x, 0, 0)).
• A plane through the y-axis OY has equation x = q z.

Normals: n₁ = (0, 1, −p) and n₂ = (1, 0, −q).

Angle between the planes

cos α = n₁·n₂ / (|n₁||n₂|) = pq / √((1 + p²)(1 + q²))

Line of intersection

Both equations hold on the line of intersection, x = q z, y = p z. For any point (x, y, z) on it, p = y/z and q = x/z. Put these in cos α:

cos α = (xy/z²) / √((1 + y²/z²)(1 + x²/z²))
      = xy / √((y² + z²)(x² + z²))

Eliminate the roots

cos²α = x²y² / [(y² + z²)(x² + z²)]

tan²α = (1 − cos²α)/cos²α
      = [(y² + z²)(x² + z²) − x²y²] / (x²y²)
      = z²(x² + y² + z²) / (x²y²)

since (y² + z²)(x² + z²) − x²y² = z²(x² + y² + z²).

So the line of intersection lies on

z²(x² + y² + z²) = x²y² tan²α. ∎