It appears in step 5 you tried to use the following false identity:
[MATH]\cos(x+y)=\cos(x)+\cos(y)[/MATH]
If I were to evaluate the given expression, I would begin with:
[MATH]a=\cos\left(\arcsin\left(\frac{2}{3}\right)+2\arcsin\left(-\frac{1}{3}\right)\right)[/MATH]
Apply the identity:
[MATH]\arcsin(-x)=-\arcsin(x)[/MATH]
[MATH]a=\cos\left(\arcsin\left(\frac{2}{3}\right)-2\arcsin\left(\frac{1}{3}\right)\right)[/MATH]
Apply the angle-difference identity for cosine:
[MATH]a=\cos\left(\arcsin\left(\frac{2}{3}\right)\right)\cos(\left(2\arcsin\left(\frac{1}{3}\right)\right)+\sin\left(\arcsin\left(\frac{2}{3}\right)\right)\sin(\left(2\arcsin\left(\frac{1}{3}\right)\right)[/MATH]
Apply double-angle identities for cosine and sine:
[MATH]a=\cos\left(\arcsin\left(\frac{2}{3}\right)\right)\left(1-2\sin^2\left(\arcsin\left(\frac{1}{3}\right)\right)\right)+\sin\left(\arcsin\left(\frac{2}{3}\right)\right)2\sin\left(\arcsin\left(\frac{1}{3}\right)\right)\cos\left(\arcsin\left(\frac{1}{3}\right)\right)[/MATH]
Using [MATH]\cos\left(\arcsin(x)\right)=\sqrt{1-x^2}[/MATH] there results:
[MATH]a=\sqrt{1-\left(\frac{2}{3}\right)^2}\left(1-2\left(\frac{1}{3}\right)^2\right)+\frac{2}{3}\cdot2\cdot\frac{1}{3}\sqrt{1-\left(\frac{1}{3}\right)^2}[/MATH]
[MATH]a=\frac{\sqrt{5}}{3}\cdot\frac{7}{9}+\frac{4}{9}\cdot\frac{2\sqrt{2}}{3}[/MATH]
[MATH]a=\frac{7\sqrt{5}+8\sqrt{2}}{27}[/MATH]
And this is equivalent to the answer given by your book.
