Your problem is \(\displaystyle \sqrt{\frac{x^{2}+1}{(\sqrt{x}+1)^{3}}}\)
Or equivalently \(\displaystyle \frac{\sqrt{x^{2}+1}}{\sqrt{(\sqrt{x}+1)^{3}}}\)
\(\displaystyle =\frac{(x^{2}+1)^{\frac{1}{2}}}{(\sqrt{x}+1)^{\frac{3}{2}}}\)
We can write this as \(\displaystyle \sqrt{x^{2}+1}(\sqrt{x}+1)^{\frac{-3}{2}}\)
Now, instead of the quotient rule, use the product rule.
\(\displaystyle (fg)'=fg'+gf'\)
Let \(\displaystyle f=(x^{2}+1)^{\frac{1}{2}}, \;\ g=(\sqrt{x}+1)^{\frac{-3}{2}}\)
\(\displaystyle f'=\frac{1}{2}2x(x^{2}+1)^{\frac{-1}{2}}, \;\ g'=\frac{-3}{2}(\sqrt{x}+1)^{\frac{-5}{2}}\cdot \frac{1}{2\sqrt{x}}\)
With problems like this, the chain rule is essentially the derivative of the inside times the derivative of the outside.
Take the derivative of \(\displaystyle (x^{2}+1)^{\frac{1}{2}}\) for instance,
The derivative of the outside is \(\displaystyle \frac{1}{2}(x^{2}+1)^{\frac{-1}{2}}\)
The derivative of the inside is the derivative of \(\displaystyle x^{2}+1\), which is \(\displaystyle 2x\).
Now, take the derivative of \(\displaystyle (\sqrt{x}+1)^{\frac{-3}{2}}\)
The derivative of the outside is \(\displaystyle \frac{-3}{2}(\sqrt{x}+1)^{\frac{-5}{2}}\)
The derivative of the inside is the derivative of \(\displaystyle \sqrt{x}+1\), which is \(\displaystyle \frac{1}{2\sqrt{x}}\).
So, we get upon multiplying them:
\(\displaystyle \frac{-3}{2}(\sqrt{x}+1)^{\frac{-5}{2}}\cdot \frac{1}{2}x^{\frac{-1}{2}}\)
Which simplifies to \(\displaystyle \frac{-3}{4}x^{\frac{-1}{2}}(\sqrt{x}+1)^{\frac{-5}{2}}\)
Or you can write it as \(\displaystyle \frac{-3}{4\sqrt{x}(\sqrt{x}+1)^{\frac{5}{2}}}\)
Now, can you put it all together?. It's rather messy.
\(\displaystyle (x^{2}+1)^{\frac{1}{2}}\cdot \frac{-3}{4}x^{\frac{-1}{2}}(\sqrt{x}+1)^{\frac{-5}{2}}+(\sqrt{x}+1)^{\frac{-3}{2}}\cdot \frac{1}{2}2x(x^{2}+1)^{\frac{-1}{2}}\)
Now, simplify this down to some form. Books often times have things simplified to an unrecognizable form.
The algebra is usually the worst part of calculus for students.
But, we can hammer into some shape:
\(\displaystyle \frac{x^{2}+4x^{\frac{3}{2}}-3}{4(\sqrt{x}+1)^{\frac{5}{2}}\sqrt{x(x^{2}+1)}}\)