Precision.Double - Enumeration

Precision.Double (0) is an enumeration that specifies the precision of comparison. It is a member of the Precision.Type and represents an optional parameter for the built-in arithmetic operators to specify double precision.

Examples

Power Query’s default way to store numbers is called Double-precision. It uses a special way to write fractional values.

This system rounds numbers to the nearest value that it can understand. This can cause small rounding errors, even before any calculations happen. For example, it can’t always understand 0.1 as exactly 1/10th of a value.

= List.Sum( { 0.1, 0.1, 0.1 } )                   // Returns 0,30000000000000004
= List.Sum( { 0.1, 0.1, 0.1 }, 0 )                // Returns 0,30000000000000004
= List.Sum( { 0.1, 0.1, 0.1 }, Precision.Double ) // Returns 0,30000000000000004

= Value.Add( 0.2, 0.1 )                   // Returns 0,30000000000000004
= Value.Add( 0.2, 0.1, 0 )                // Returns 0,30000000000000004
= Value.Add( 0.2, 0.1, Precision.Double ) // Returns 0,30000000000000004

To be able to store more detailed numbers you can make use of another Precision Type, namely Precision.Decimal.

List.Sum( { 0.1, 0.1, 0.1 }, 1 )                 // Returns 0,3
List.Sum( { 0.1, 0.1, 0.1 }, Precision.Decimal ) // Returns 0,3

In this case, the 1/10th fraction of a number is accurately displayed.

Learn more about Precision.Double in the following articles:

Power Query Precision: Avoid Rounding Errors
Learn how to handle precision differences in Power Query. With these tips you can return accurate numbers and prevent rounding errors. » Read more

Other related enumerations are:

Precision.Decimal

Applies to

Here’s a list of functions that work with Precision.Type:

Contribute » | Contributors: Rick de Groot

Examples

Related articles

Related enumerations

Applies to