I am assuming you want to solve the system of equations
x^2-3xy+y^2+1 = 0
3x^2-xy+3y^2 = 13
from the second, we have
xy = 3x^2+3y^2-13
using that in the first, we have
x^2-3(3x^2+3y^2-13)+y^2+1 = 0
x^2-9x^2-9y^2+39+y^2+1 = 0
-8x^2 - 8y^2 = -40
x^2 + y^2 = 5
Using that in either equation, we have xy = 2
So, our solutions are (±1,±2)and (±2,±1)