Introduction

Meters per degree of latitude and longitude at various latitudes

The length between longitude and latitude lines changes with position on the earth, being greatest at the equator, and least near the poles. Here is a table of meters per degree of latitude, and meters per degree of longitude at various degrees of latitude:

Table 1: table of meters per degree of latitude, and meters per degree of longitude at various degrees of latitude

    Distance between latitude lines varies slightly, while distance between longitude varies heavily. From table 1, it is evident that the maximum distance between longitude lines is about the same length as distance between latitude lines at the equator (zero degrees latitude).

Precision of Float and Double

    As with the distance between longitude and latitude, floating point numbers have various precision depending on the magnitude of the number being stored. Floating point numbers have higher precision when storing smaller numbers, which is an inverse relationship to the distance per degree magnitude (meters per degree longitude is smaller when latitude is greater). This relationship will work to our advantage with longitude, but distance between latitude does not really change with degrees latitude, so  the latitude accuracy will be our limiting factor.

The precision of a floating point number is called the Machine epsilon. For a single precision number (24 bits mantissa) the machine precision is: epsilon = 2-23

The following tables were calculated using the machine precision calculation:

Worst Case Scenarios

Longitude can vary from 0 to 180 degrees, and will have the least precision when at the equator and 180 degrees.

Table 2: Single Precision float accuracy worst case scenario (Global)

The following table represents the works case scenarios for the entire United States Territories:

Table 3: United States total worst case scenario

Some other places of interest

Conclusion

It is shown that within the United States (all territories), the wost case latitude error is less than one meter, and the worst case longitude error is less than two meters. These errors are less than the standard GPS accuracy, as well as accuracy when using differential positioning.

References