\(\displaystyle f(x,y,z)=(x-0)^{2}+(y-0)^{2}+(z-0)^{2}=x^{2}+y^{2}+z^{2}\)
subject to the constraint:
\(\displaystyle 3xy-z^{2}=1\)
\(\displaystyle 2x=3y{\lambda}\)
\(\displaystyle 2y=3x{\lambda}\)
\(\displaystyle 2z=-2z{\lambda}\)
If z does not equal 0, then \(\displaystyle {\lambda}=-1\)
so \(\displaystyle 2x=-3y\) and \(\displaystyle 2y=-3x, x=y=0\)
Sub into \(\displaystyle 3xy-z^{2}=1\) to get:
\(\displaystyle z^{2}=-1\)
which has no real solution.
If \(\displaystyle z=0\) then \(\displaystyle 3xy-(0)=1, y=\frac{1}{3x}\)
and also \(\displaystyle \frac{2y}{3x}=\frac{2x}{3y}\)
\(\displaystyle y^{2}=x^{2}\) so \(\displaystyle \frac{1}{9x^{2}}=x^{2}\)
\(\displaystyle 9x^{4}=1\), \(\displaystyle x=\frac{1}{sqrt{3}}, x=\frac{-1}{sqrt{3}}\)
Test \(\displaystyle (\frac{1}{sqrt{3}}, \frac{1}{sqrt{3}},0)\)
and
\(\displaystyle (\frac{-1}{sqrt{3}},\frac{-1}{sqrt{3}}, 0)\)
to see if they are closest to the origin.
Is this the answer you need?. Check it out. It's easy to make a mistake.