x2-3xy+y2+1=0, 3x2-xy+3y2=13

Answer 1

I am assuming you want to solve the system of equations

x^2-3xy+y^2+1 = 0
3x^2-xy+3y^2 = 13

from the second, we have
xy = 3x^2+3y^2-13
using that in the first, we have
x^2-3(3x^2+3y^2-13)+y^2+1 = 0
x^2-9x^2-9y^2+39+y^2+1 = 0
-8x^2 - 8y^2 = -40
x^2 + y^2 = 5

Using that in either equation, we have xy = 2

So, our solutions are (±1,±2)and (±2,±1)

Answer 2

or, you could just solve the two equations using the quadratic formula. That would yield

x = [3y±√(5y^2-4)]/2
x = [y±√(152-35y^2)]/6
equate those two and you get y=±2