Hello, math-a-phobic!
I like your substitutions . . .
\displaystyle \;\; Let
x = sin θ , y = cos θ \displaystyle x\,=\,\sin\theta,\;\;y\,=\,\cos\theta x = sin θ , y = cos θ
and remember that:
x 2 + y 2 = 1 \displaystyle \,x^2\,+\,y^2\:=\:1 x 2 + y 2 = 1
\(\displaystyle \left(\sin\theta\right)^{\frac{1}{2}}\cos\theta \,-\,\left(\sin\theta\right)^{\frac{5}{2}}\cos\theta \;= \;\cos^3\theta\sqrt{\sin\theta\)
We have:
x 1 2 y − x 5 2 y \displaystyle \,x^{\frac{1}{2}}y\,-\,x^{\frac{5}{2}}y x 2 1 y − x 2 5 y
Factor: \(\displaystyle \,x^{\frac{1}{2}}y\,\left(\underbrace{1\,-\,x^2})\)
Then:
x 1 2 y ⋅ y 2 = y 3 x ⇒ cos 3 θ sin θ \displaystyle \;\;\;x^{\frac{1}{2}}y\,\cdot\, y^2\;= \;y^3\sqrt{x}\;\;\Rightarrow\;\;\cos^3\theta\,\sqrt{\sin\theta} x 2 1 y ⋅ y 2 = y 3 x ⇒ cos 3 θ sin θ
\(\displaystyle \cos\theta \,- \,\L\frac{\cos\theta}{1\,-\,\tan\theta} \;= \;\frac{\sin\theta\cos\theta}{\sin\theta\,-\,\cos\theta}\)
We have: \(\displaystyle \,y \,-\,\L\frac{y}{1\,-\,\frac{x}{y}}\)
Factor: \(\displaystyle \L\,y\left(1\,-\,\frac{1}{1\,-\,\frac{x}{y}}\right)\)
Multiply top and bottom of the fraction by \(\displaystyle y:\L\;y\left(1\,-\,\frac{y}{y\,-\,x}\right)\)
Combine: \(\displaystyle \L\,y\left(\frac{y\,-\,x\,-\,y}{y\,-\,x}\right)\;=\;\frac{-xy}{y\,-\,x}\;=\;\frac{xy}{x\,-\,y}\;\;\Rightarrow\;\;\frac{\sin\theta\cos\theta}{\sin\theta\,-\,\cos\theta}{\)
2 + cos 2 θ − 3 cos 4 θ = sin 2 θ ( 2 + 3 cos 2 θ ) \displaystyle 2\,+\,\cos^2\theta\,-\,3\cos^4\theta\;=\;\sin^2\theta(2\,+\,3\cos^2\theta) 2 + cos 2 θ − 3 cos 4 θ = sin 2 θ ( 2 + 3 cos 2 θ )
On the right, we have:
x 2 ( 2 + 3 y 2 ) \displaystyle \,x^2(2\,+\,3y^2) x 2 ( 2 + 3 y 2 ) \displaystyle \;\; Then we have:
( 1 − y 2 ⏞ ) ( 2 + 3 y 2 ) \displaystyle \;\;\,(\overbrace{1\,-\,y^2})(2\,+\,3y^2) ( 1 − y 2 ) ( 2 + 3 y 2 )
Multiply:
2 + y 2 − 3 y 4 ⇒ 2 + cos 2 θ − 3 cos 4 θ \displaystyle \,2\,+\,y^2\,-\,3y^4\;\;\Rightarrow\;\;2\,+\,\cos^2\theta\,-\,3\cos^4\theta 2 + y 2 − 3 y 4 ⇒ 2 + cos 2 θ − 3 cos 4 θ 
Please check to see what I did wrong. 
