5x^2-7xy+12y^2

_______________(OVER)
x^2+xy-20y^2

Please if anyone can help in the steps to answer this...i would be greatful.=)

What is a reflex angle

Are you sure you copied the question correctly?
If the top is 5x^2-17xy-12y^2 it works out really easily.
The way it stands, the top does not factor with integer values