Rounding errors are an inevitable problem with navigation calculations due to the fact that you often have to subtract large numbers depending on the specific triangle.
For example you might know something is 1000.1 to five significant figures. Then something else is 1000.2 to five figures.
However if you use the difference anywhere in your calculation, you are down to
1000.2 - 1000.1 = 0.1 or one significant figure in the result.
Thanks Damon for your help yesterday. I see now there are close similarities between the 2 formulas I sighted yesterday.
I worked my problem with both formulas and received identical answers of 958.27 NM.
This distance is a mile or so more than your answers and I have studied my workings and conclude your answers are more accurate as I have made rounding errors. This has been a very useful exercise for me.
I also looked at Wikipedia and note the formula I really need to get to grips with is the more complicated, more accurate one at the bottom of the page. I will study this in due course.
Thanks again
Mike
1 answer
0 answers