At first, I thought you were trying to determine f(x + h), but then you subtracted stuff from it and divided it by stuff, so I can't follow all of your equal signs.
This part of your work is correct: f(x + h) = 5/[(x + h)^2 - 9]
This is just one of the terms in the Difference Quotient's numerator; you need to subtract f(x) from it.
[f(x + h) - f(x)]/h
The derivative of f(x) is the limit of this Difference Quotient as h goes to zero.
You should start by writing out the entire Difference Quotient, and then simplify that.
Since f(x) and f(x + h) are rational functions, we can avoid working with a compound ratio by first factoring out 1/h from the Difference Quotient.
(1/h) * [f(x + h)^2 - f(x)]
This is the same Difference Quotient. Okay ?
So, here's the Difference Quotient for function f:
\(\displaystyle \frac{1}{h} \cdot \left ( \frac{5}{(x + h)^2 - 9} - \frac{5}{x^2 - 9} \right )\)
Next, you could factor out a 5 from the numerators:
\(\displaystyle \frac{5}{h} \cdot \left ( \frac{1}{(x + h)^2 - 9} - \frac{1}{x^2 - 9} \right )\)
Can you continue ? Combine the two ratios inside the parentheses into a single ratio.
You don't need to multiply out the common denominator, but you do need to expand and simplify the numerator, in order to factor out an h to cancel with the bottom of 5/h.
After you cancel h's in the resulting factor 5h/h, take the limit as the remaining h's go to zero.
Let us know, if you have any specific questions about this.