Polar coordinates: y²√(x²+y²)

Region. x runs from 0 to a and y from 0 to √(a² − x²): this is the quarter of the circle x² + y² = a² lying in the first quadrant. In polar coordinates: 0 ≤ r ≤ a, 0 ≤ θ ≤ π/2.

Change of variables

x = r cos θ, y = r sin θ, √(x² + y²) = r, dy dx = r dr dθ.

y² √(x²+y²) dy dx = (r² sin²θ)(r)(r dr dθ) = r⁴ sin²θ dr dθ

Integrate

I = ∫₀^{π/2} sin²θ dθ · ∫₀^a r⁴ dr
  = ∫₀^{π/2} sin²θ dθ · (a⁵/5)

Now ∫₀^{π/2} sin²θ dθ = ∫₀^{π/2} (1 − cos 2θ)/2 dθ = π/4.

I = (π/4)(a⁵/5) = π a⁵/20

Answer: π a⁵/20.