You fellas approaches are very clever. I'm going to give it a shot. If it ain't right it'll be wrong, I suppose.
The vertical line of length C divides the triangle into 2 smaller triangles which share the same base length C. The heights of the two triangles are x-a and b-x.
So, the area of the big triangle is:
\(\displaystyle Area=(x-a)C/2+(b-x)C/2=(b-a)C/2\)
As you both pointed out, the area is max when C is max.
The slope of the line is\(\displaystyle \frac{(b^{2}-a^{2})}{(b-a)}=(a+b)\)
So it's equation, in point-slope form, is:
\(\displaystyle y-a^{2}=(a+b)(x-a),\)
\(\displaystyle y=(a+b)\cdot(x)-ab\).
\(\displaystyle C=(a+b)\cdot(x)-ab-x^{2},\) if\(\displaystyle a\leq x \leq b\)
\(\displaystyle \frac{dC}{dx}=a+b-2x\)
Setting to 0 and solving for x we get \(\displaystyle x=\frac{a+b}{2}\)
So P should be at: \(\displaystyle x=\frac{(a+b)}{2}\) and \(\displaystyle y=\frac{(a+b)^{2}}{4}\)
Sub x into C and we have: \(\displaystyle C=\frac{(b-a)^{2}}{4}\)
the area of the triangle is greatest when C=\(\displaystyle \frac{(b-a)^{2}}{2}\)
the area is \(\displaystyle \frac{(b-a)^{3}}{8}\)