Thanks a lot HallsofIvy and Dr.Peterson for giving me some of your valuable time by looking at this problem which perhaps is not for mathematicians.
Since the pressure at point [MATH]A[/MATH] and [MATH]B[/MATH] are equal, we can use this formula
[MATH]P_o + \rho_w gh_w = P_o + \rho_m gh_m[/MATH]
where [MATH]P_o[/MATH] is the atmospheric pressure, in this case it is the Air Pressure. It is not a big deal since it will be cancelled.
the equation will become
[MATH]\rho_w h_w = \rho_m h_m[/MATH]
[MATH]\rho_w (h + 0.61 + 0.61) = \rho_m h[/MATH]
All I have to do now is to plug the values of water density, mercury density, and solve for h.
[MATH]1000h + 1220 = 13600h[/MATH]
[MATH]h = 0.0968 \ m[/MATH]
for (b) i am thinking of [MATH]2 \ psi = 13800 \ N/m^2[/MATH]
for (c) I have to repeat part (a) with a new density [MATH] = 0.85 \cdot 1000[/MATH]