I'm really confused. I know the domain of q is [0, 20/p] but without knowing the value for p I don't know how to pick values of q which are definitely within the domain.
We require both rdicands to be non-negative:
[MATH]q\ge0[/MATH]
[MATH]20-pq\ge0\implies q\le\frac{20}{p}[/MATH]
So, the domain is:
[MATH]\left[0,\frac{20}{p}\right][/MATH]
just as it's given in the problem statement.
So, what I would do is use the point halfway between the left end of the domain and the critical value:
[MATH]q=\frac{0+\dfrac{20}{p(p+1)}}{2}=\frac{10}{p(p+1)}[/MATH]
And now we find:
[MATH]U'\left(\frac{10}{p(p+1)}\right)=\frac{1}{2\sqrt{\dfrac{10}{p(p+1)}}}-\frac{p}{2\sqrt{20-p\left(\dfrac{10}{p(p+1)}\right)}}=\frac{\sqrt{p(p+1})}{2\sqrt{10}}\left(1-\sqrt{\frac{p}{2p+1}}\right)[/MATH]
And so we see that in order for this to be positive, we require:
[MATH]\frac{p}{2p+1}<1[/MATH]
[MATH]\frac{p}{2p+1}-1<0[/MATH]
[MATH]\frac{p-(2p+1)}{2p+1}<0[/MATH]
[MATH]-\frac{p+1}{2p+1}<0[/MATH]
[MATH]\frac{p+1}{2p+1}>0[/MATH]
We see this is true for all non-negative values of \(p\). And so we conclude that on the interval:
[MATH]\left(0,\frac{20}{p(p+1)}\right)[/MATH]
the utility function is increasing. In much the same way, can you show the utility function is decreasing on:
[MATH]\left(\frac{20}{p(p+1)},\frac{20}{p}\right)[/MATH] ?
is a another way of checking the maximum to substitute q=0, q=20/p and q=20/p(p+1) into the original utility fuction?
Because \(U'(q)\) is undefined at the endpoints of the domain, it would also be a good idea to check the value of the utility function at those values. Your professor may be satisfied with you showing:
[MATH]U(q^{*})>U(0)[/MATH]
[MATH]U(q^{*})>U\left(\frac{20}{p}\right)[/MATH]
But, I suspect you are expected to either use the first or second derivative tests to confirm you have maximized the utility function.