If a ship's flag is 36' above sea level, could it be seen by someone in a building 100' high that was 18 miles away? (please show work)

Yes it is visible. The ship and building are separated by a subtended angle of A = 0.2604 degrees. Between sea level points at each location, the spherical surface of the Earth rises
R (1 - cos A/2) = 0.0102 miles = 54 feet. The is not high enough to block the line of sight between points 100 and 36 feet above sea level. (But it comes close.