Is this your problem:
\(\displaystyle \int\frac{1}{\sqrt{u(au-b)^{2}-4u}}du=\int\frac{1}{\sqrt{u[(au-b)^{2}-4]}}du\)
You did not say what you are integrating with respect to. Is it u?. I would presume so,
Now, if we let \(\displaystyle t=au-b, \;\ \frac{dt}{a}=du, \;\ u=\frac{t+b}{a}\), it may at first look promising. But that extra u in there makes it a little rougher.
We get \(\displaystyle \frac{1}{a}\int\frac{1}{\sqrt{\frac{(t+b)(t^{2}-4)}{a}}}dt\)
If that is it, it is rather tough and may involve Elliptic Integrals. Maple gave me a long, drawn out, convoluted solution involving Elliptic integrals.
Therefore, to do it by hand may prove daunting. May I ask what class this is from and at what level you are?.
Some of the other skilled mathematicians may have a good idea. I do not have time right now, but if you do not get a response, I can look later this afternoon.