a^2 - 10ab + 3b^2

*I need a lot of help with this.How do I do this one.?

Try:

(a + r b)(a + s b)

Then:

r + s = -10

r*s = 3

From the first equation:

s = -(10 + r)
Inserting in second expression:

r*(10 + r) = -3 -->

r^2 + 10 r + 3 = 0 --->

r = 5 +/- 2*sqrt[11]

You can use either solution of r, taking the other just amounts to switching r and s.

Typo:

r = -5 +/- 2*sqrt[11]