Factorise to find roots.
Initial guess:
x3 + 2x2 - 40x + 64 = (x + _)(x + _)(x + _)
Another guess:
Substitute x=2 into P(x) and you get zero. Therefore (x-2) is a factor:
x3 + 2x2 - 40x + 64 = (x - 2)(x + _)(x + _)
It will probably be (x - 2)(x - _)(x + _) to get the -40x but also the +64.
_ and _ have to multiply with 2 to give 64.
Options (factors of 64/2=32):
4, -8 or 8, -4
2, -16 or 16, -2
1, 32 or -32, 1
Plugging in x=-8 gives P(x)=0 so we have (x - 2)(x + 8)(x - _). The last factor must be (x-4), checked with P(4)=0.
So the roots (or zeros) are: -8, 2, 4. 4 being the largest.
Once you get comfortable with this it's just a matter of tapping educated guesses into your calculator.
You could use long division after initially finding (x-2) to be a factor.