Double pendulum angular momentum, m1 = 3, m2 = 4 kg

Take the origin at the fixed pivot $O$, $x$ vertically downward, $y$ to the right and $\hat k$ out of the plane ($+\hat k$ = counter-clockwise). Assume both arms lie on the same (right-hand) side of the vertical, and that $\omega_2$ is the absolute angular velocity of the lower arm (as is usual in this problem). Angular momentum of the system about $O$:

$$\vec H_O = m_1(\vec r_1 \times \vec v_1) + m_2(\vec r_2 \times \vec v_2)$$

Position vectors

Upper arm ($a=0.4$ m at $30^\circ$ to the $x$-axis), lower arm ($b=0.5$ m at $60^\circ$ to the vertical):

$$\begin{aligned} \vec r_1 &= a(\cos 30^\circ\,\hat i + \sin 30^\circ\,\hat j) = (0.346\,\hat i + 0.200\,\hat j)\ \text{m}\\ \vec r_{2/1} &= b(\cos 60^\circ\,\hat i + \sin 60^\circ\,\hat j) = (0.250\,\hat i + 0.433\,\hat j)\ \text{m}\\ \vec r_2 &= \vec r_1 + \vec r_{2/1} = (0.596\,\hat i + 0.633\,\hat j)\ \text{m} \end{aligned}$$

Velocities

$$\begin{aligned} \vec v_1 &= \omega_1\hat k \times \vec r_1 = (-0.800\,\hat i + 1.386\,\hat j)\ \text{m/s}\\ \vec v_2 &= \vec v_1 + \omega_2\hat k \times \vec r_{2/1} = (-2.965\,\hat i + 2.636\,\hat j)\ \text{m/s} \end{aligned}$$

Angular momentum

$$\begin{aligned} H_1 &= m_1(x_1 v_{1y} - y_1 v_{1x}) = 3(0.346\times 1.386 - 0.200\times -0.800) = 1.92\ \text{kg m}^2/\text{s}\\ H_2 &= m_2(x_2 v_{2y} - y_2 v_{2x}) = 4(0.596\times 2.636 - 0.633\times -2.965) = 13.80\ \text{kg m}^2/\text{s} \end{aligned}$$ $$\vec H_O = H_1 + H_2 = 15.72\,\hat k\ \text{kg m}^2/\text{s}$$

Answer: $\vec H_O \approx 15.72\,\hat k$ kg m²/s (counter-clockwise, about the axis perpendicular to the plane).