Subhotosh Khan and Mrspi have given you excellent answers. I think what may be making this problem hard for you is that you are overlooking that you are NOT asked to find x and y. As the previous answers have indicated, the simple way to answer the question ACTUALLY ASKED is to recognize that
\(\displaystyle (x - y)^2 = (x^2 + y^2) - 2xy\) and that you are given the values of (x - y) and (x^2 + y^2).
However, you can solve this the way you were thinking, which is to find values for x and y and then compute xy.
\(\displaystyle x - y = 5 \longrightarrow x = y + 5 \longrightarrow (y + 5)^2 + y^2 = 15\)
\(\displaystyle (y + 5)^2 + y^2 = y^2 + 10y + 25 + y^2 = 2y^2 + 10y + 25\)
\(\displaystyle 2y^2 + 10y + 25 = 15 \longrightarrow 2y^2 + 10y + 10 = 0 \longrightarrow\) \(\displaystyle y = \dfrac{-10 \pm \sqrt{10^2 - (4 * 2 * 10}}{4} = \dfrac{-10 + (\pm \sqrt{100 - 80})}{4} = \dfrac{-10 + 2(\pm \sqrt{5})}{4}\)
\(\displaystyle So\ x = 5 + \dfrac{-10 +2(\pm \sqrt{5})}{4} = \dfrac{20 - 10 + 2(\pm \sqrt{5})}{4} = \dfrac{10 + 2(\pm \sqrt{5})}{4}\)
\(\displaystyle So\ xy = \dfrac{10 +2(\pm \sqrt{5})}{4} * \dfrac{-10 +2(\pm \sqrt{5})}{4} = \) \(\displaystyle \dfrac{[10 * (-10)] + [(10 - 10)(2)(\pm \sqrt{5}\ )] + [2(\pm \sqrt{5})]^2}{16} =\)
\(\displaystyle \dfrac{-100 + [(0)(2)(\pm \sqrt{5})] + [2^2 * (\sqrt{5})^2]}{16} = \dfrac{-100 + (4 * 5)}{16} = \dfrac{-80}{16} = - 5\)
I think you will agree that this is an awful way to solve this problem. BUT it does tell you that there are numbers that work.
\(\displaystyle x - y = \dfrac{10 + 2(\pm \sqrt{5})}{4} - \dfrac{-10 + 2(\pm\ sqrt{5})}{4} =\) \(\displaystyle \dfrac{10 - (-10) + 2(\pm \sqrt{5}) - 2(\pm \sqrt{5})}{4} =\dfrac{10+10}{4} = \dfrac{20}{4} = 5\)
\(\displaystyle x^2 = (\dfrac{10 + 2(\pm \sqrt{5})}{4})^2 = \dfrac{100 + 40(\pm \sqrt{5}) + 20}{16}\)
\(\displaystyle y^2 = (\dfrac{-10 + 2(\pm \sqrt{5})}{4})^2 = \dfrac{100 - 40(\pm \sqrt{5}) + 20}{16}\)
\(\displaystyle x^2 + y^2 = \dfrac{100 + 40(\pm \sqrt{5}) + 20}{16} + \dfrac{100 - 40(\pm \sqrt{5}) + 20}{16} =\) \(\displaystyle \dfrac{100+20+100+20}{16} = \dfrac{240}{16} = 15\)
Aren't you glad they did NOT ask you to find x and y. Moral: look for the easy way.