Question

Xsin(theta)-Ysin(theta)=sqrtX^2+Y^2 and cos^2(theta)/a^2+sin^2(theta)/b^2=1/X^2+Y^2 then find correct relation

Answer 1

xsinθ - ysinθ = √(x^2+y^2)
cos^2θ/a^2 + sin^2θ/b^2 = 1/(x^2+y^2)

not sure what you mean by the "correct relation" We can solve for θ or solve for x and y.

(x-y)^2 sin^2θ = x^2+y^2

so, sin^2θ = (x^2+y^2)/(x-y)^2 and we have

(1-((x^2+y^2)/(x-y)^2))/a^2 + ((x^2+y^2)/(x-y)^2)/b^2 = 1/(x^2+y^2)

I don't know how you want to massage that, but it will remain messy, I'm sure.

I suspect a typo, since you have sinθ twice in the first line. I'd expected a cosθ there somewhere.