find maximum value of 4(sinx+acosx)(sinx+bcosx) - (a+b)^2

1 answer

If a and b are constants, then the max occurs where the derivative is zero.
That is where tan(2x) = (a+b)/(ab-1)
I guess you can find some values of a and b where that is easy, or express sinx and cosx for such an angle.