Rewrite 6x^2 - sqrt(3)xy + 5y^2 - 9 = 0 with the xy term eliminated.

Answer 1

the graph is rotated through an angle θ where
cot2θ = (A-C)/B = -1/√3 so θ = -π/6

So, we need new x'-y' axes where

x = (x'(√3/2)-y'(-1/2))
y = (x'(-1/2)+y'(√3/2))

expand the expressions (yes, you need to pay careful attention to the details!) and you end up with

13x'^2 + 9y'^2 = 18