Precision.Double

Updated on

Precision.Double (0) is an enumeration that specifies the precision of comparison. It is a member of the Precision.Type and represents an optional parameter for the built-in arithmetic operators to specify double precision.

Examples

Power Query’s default way to store numbers is called Double-precision. It uses a special way to write fractional values. 

This system rounds numbers to the nearest value that it can understand. This can cause small rounding errors, even before any calculations happen. For example, it can’t always understand 0.1 as exactly 1/10th of a value.

= List.Sum( { 0.1, 0.1, 0.1 } )                   // Returns 0,30000000000000004
= List.Sum( { 0.1, 0.1, 0.1 }, 0 )                // Returns 0,30000000000000004
= List.Sum( { 0.1, 0.1, 0.1 }, Precision.Double ) // Returns 0,30000000000000004

= Value.Add( 0.2, 0.1 )                   // Returns 0,30000000000000004
= Value.Add( 0.2, 0.1, 0 )                // Returns 0,30000000000000004
= Value.Add( 0.2, 0.1, Precision.Double ) // Returns 0,30000000000000004

To be able to store more detailed numbers you can make use of another Precision Type, namely Precision.Decimal.

List.Sum( { 0.1, 0.1, 0.1 }, 1 )                 // Returns 0,3
List.Sum( { 0.1, 0.1, 0.1 }, Precision.Decimal ) // Returns 0,3

In this case, the 1/10th fraction of a number is accurately displayed.

Learn more about Precision.Double in the following articles:

Other related enumerations are:

Applies to

Here’s a list of functions that work with Precision.Type:

BI Gorilla Youtube Channel

Contribute » | Contributors: Rick de Groot