The angle between the vectors a=(6,-2,3) and b=(1,p,-2) is cos^-1(-1/14).

Determine the value of p.

|a|=sqrt(6^2+(-2)^2+3^2)=7
|b|=sqrt(1^2+p^2+(-2)^2)=5+p^2
theta=cos(cos^-1(-1/14))=-1/14

a x b = (4-3p,15,6p+2)
a x b = |a||b|cos(theta)
a x b = -sqrt(5/2) + p

I'm not getting anywhere from here.

The answer key says p=1/sqrt(3)
What do I need to do to solve for p.

3 answers

error in finding |b|
|b| = √(5+p^2)

you need the dot product
a dot b = |a| |b| cosØ
( (6)(1) + (-2)(p) + (3)(-2) ) = 7√(5 + p^2)(-15/14)
-2p = (-15/2)√(5+p^2)
4p = 15√(5+p^2)
16p^2 = 225(5+p^2)
16p^2 = 1125 + 225p^2
209p^2 = -1125

No real solution, so no value of p exists
(no wonder you didn't get anywhere with your solution)
Where did you get (-15/14) from? Isn't it (-1/14)?
You are right, at closer look I now see -1/14, with my failing eyesight I saw -1 1/14, which I changed to -15/14
so picking it up from
( (6)(1) + (-2)(p) + (3)(-2) ) = 7√(5 + p^2)(-15/14)
should have been
( (6)(1) + (-2)(p) + (3)(-2) ) = 7√(5 + p^2)(-1/14)
and then
-2p = (-1/2)√(5+p^2)
4p = √(5+p^2)
16p^2 = 5 + p^2
15p^2 = 5
p^2 = 1/3
p = 1/√3

so it did work out rather nicely , I would have expected you to realize that I worked with a typo and made the correction accordingly.