Well, you can't "translate line PQ to origin" because the origin is a point not a line! You mean "translate point P (or point Q) to origin". And you can't "rotate Q to x-axis" because the x-axis is a line not a point! Here you mean "rotate PQ to x-axis" (or, perhaps better. "rotate PQ to z-axis" so that your rotation is in the xy-plane). And you seem to have forgotten to actually do the original rotation between the second and third lines.
Let's say point P is \(\displaystyle (x_0, y_0, z_0)\) and Q is \(\displaystyle (x_1, y_1, z_1)\). The "translate P to origin" is \(\displaystyle x-> x- x+x_0\), \(\displaystyle y-> y- y_0\) and \(\displaystyle z-> z- z_0\). Q becomes \(\displaystyle (x_2, y_2, z_2)= (x_1- x_0, y_1- y_0, z_1- z_0)\). To rotate that to the z-axis I would divide it into two rotations, one about the z-axis so that the x coordinate becomes 0 and (to preserve length) the y coordinate becomes \(\displaystyle r= \sqrt{x_2^2+ y_2^2}\).
Since a rotation matrix is anti-symmetric, we want \(\displaystyle \begin{bmatrix} a & b & 0 \\ -b & a & 0 \\ 0 & 0 & 1\end{bmatrix}\begin{bmatrix} x_2 \\ y_2 \\ z_2\end{bmatrix}= \begin{bmatrix}ax_2+ by_2 \\ ay_2-bx_2 \\ z_2\end{bmatrix}= \begin{bmatrix}0 \\ r \\ z_2\end{bmatrix}\).
So we must have \(\displaystyle ax_2+ by_2= 0\) and \(\displaystyle ay_2- bx_2= r\). We have calculated \(\displaystyle x_2\) and \(\displaystyle y_2\) so can solve those equations for a and b.
Now rotate around the x-axis to get \(\displaystyle \begin{bmatrix} 0 \\ 0 \\ \rho\end{bmatrix}\) where \(\displaystyle \rho= \sqrt{r^2+ z_2^2}= \sqrt{x_2^2+ y_2^2+ z_2^2}\). To do that we must have \(\displaystyle \begin{bmatrix}1 & 0 & 0 \\0 & a & b \\ 0 & -b & a \end{bmatrix}\begin{bmatrix} 0 \\ r \\ z_3 \end{bmatrix}= \begin{bmatrix}0 \\ ar+ bz_2 \\ -br+ az_2\end{bmatrix}= \begin{bmatrix}0 \\ 0 \\ \rho\end{bmatrix}\).
So we must have \(\displaystyle ar+ bz_2= 0\) and \(\displaystyle -br+ az_2= \rho\). Again, solve those two equations for a and b.
