Question

if cos(x) = 5/13 and sin(y)=3/5 where 0<= x, y <= x/2, determine:

a) cos (x+y)
b) sin (x-y)

Answer 1

sin x = sqrt (1-cos^2 x) = sqrt (1-25/169)=sqrt(144/169) = 12/13
cos y = sqrt (1-sin^2 y) = sqrt (1-9/25) = sqrt(16/25) = 4/5
cos(x+y) = cos x cos y - sin x sin y
= 5/13 * 4/5 - 12/13 * 3/5
etc
then
sin(x-y) = sin x cos y - cos x sin y

Answer 2

You should recognize the two infamous right-angled triangles with
side ratios of 3-4-5, and 5-12-13
If you don't, make quick sketches and use Pythagoras to find the
missing sides of each.

If cosx = 5/13, the cosine is + in I or IV
then sinx = 12/13 if x is in quad I
or sinx = -12/13 if x is in quad IV

if siny = 3/5, the sine is + ins I or II
then cosy = 4/5 if y is in quad I
or cosy = -4/5 if y is in quad II

You MUST know the expansions for cos(x+y) and sin(x-y)
I will do the first one:
cos(x+y) = cosxcosy - sinx siny

Case 1: x is in I, y is in I
cos(x+y) = cosxcosy - sinxsiny
= (5/13)(4/5) - (12/13)(3/5) = -16/65

case 2: x is in I, y is in II,
cos(x+y) = (5/13)(-4/5) - (12/13)(3/5)
= -56/65
[ check this: x= 67.4° , y = 143.1°
cos(67.4+143.1) = -.8615 or appr -56/65 ]

case 3: x is in IV , y is in I
cos(x+y) = (5/13)(4/5) - (-12/13)(3/5) = +56/65

case 4, x is in IV, y is in II
cos(x+y) = (5/13)(-4/5) - (-3/5)(12/13) = 16/65

Your conditions of x ≥ 0 was no problem, but
your second condition that y ≤ x/2 makes this more complicated

Look at my 4 cases:
case 1, x = 67.4° and y = 36.9
is 36.9 ≤ 33.7 ? NO, so case 1 has to be rejected

case 2, x is in I, y is in II
x = 67.4, y = 143.1 . Is y ≤ 1/2 x ??? NO, reject case 2

case 3, x is in IV , y is in I
x = 292.6, 36.9 , is y ≤ 1/2 x ? , YES, so case 3 works

case 4, x is in IV, y is in II
x = 292.6 , x = 143.1, is 143.1 ≤ (1/2)292.6 ?? YES, so case 4 works

so we have
case 3: x is in IV , y is in I
cos(x+y) = (5/13)(4/5) - (-12/13)(3/5) = +56/65
or
case 4, x is in IV, y is in II
cos(x+y) = (5/13)(-4/5) - (-3/5)(12/13) = 16/65


wheuwww! There has to be a better way, but that y ≤ x/2 is a weird restriction.