3D rotation about arbitrary axis

Let the axis pass through the points P1(x1, y1, z1) and P2(x2, y2, z2). The idea is to make the axis coincide with a coordinate axis (z), rotate about it, and then undo the first transformations.

Axis vector V = P2 − P1 = (a', b', c'), length L = √(a'² + b'² + c'²). Unit vector u = (a, b, c) = V/L. Let d = √(b² + c²).

Steps

1. Translate the axis so that P1 is at the origin: T(−x1, −y1, −z1).
2. Rotate about the x-axis by angle α to bring the axis into the xz plane, with cosα = c/d and sinα = b/d.

Rx(α) = | 1  0     0    0 |
        | 0  c/d  -b/d  0 |
        | 0  b/d   c/d  0 |
        | 0  0     0    1 |

3. Rotate about the y-axis by angle β to bring the axis onto the z-axis, with cosβ = d and sinβ = a. The axis must turn clockwise (seen from +y) to reach the z-axis, so the matrix has the clockwise form:

Ry(β) = | d  0 -a  0 |
        | 0  1  0  0 |
        | a  0  d  0 |
        | 0  0  0  1 |

4. Rotate about the z-axis by the required angle θ: Rz(θ).
5. Apply the inverse of step 3: Ry(−β).
6. Apply the inverse of step 2: Rx(−α).
7. Translate back: T(x1, y1, z1).

Composite matrix

M = T(P1) · Rx(−α) · Ry(−β) · Rz(θ) · Ry(β) · Rx(α) · T(−P1)

Each point of the object is multiplied by M: P' = M·P. When d = 0 (the axis is along the x-axis), the Rx step is skipped.