the vector a-5b and a-b are perpendiculat. If vector a and b are unit vectors, then determine axb

Answer 1

If this is in just two dimensions:
A = ax i + ay j
B = bx i + by j

C = A-5B = (ax-5bx) i + (ay-5by)j
D = A- B = (ax- bx) i + (ay- by)j

if perpendicular C dot D = 0
(ax-5bx)(ax-bx)+(ay-5by)(ay-by) = 0
ax^2 -6ax bx + 5bx^2 +ay^2 -6ay by +5 by^2 = 0
but unit vectors so ax^2+ay^2 = 1
and
5(bx^2+by^2) = 5
so
6 - 6 ax bx -6 ay by = 0
so
ax bx + ay by = 1

Now we want to know A x B
determinant of
i j k
ax ay 0
bx by 0
= (ax by - ay bx)k
but
ax bx + ay by = 1
ay = (1-ax bx)/bx
so cross product is
(ax by - 1+ax bx)k
= [ax(bx+by)-1] k

Check carefully, I went pretty fast and there may be errors.