Precision.Double (0) is an enumeration that specifies the precision of comparison. It is a member of the Precision.Type and represents an optional parameter for the built-in arithmetic operators to specify double precision.
Examples
Power Query’s default way to store numbers is called Double-precision. It uses a special way to write fractional values.
This system rounds numbers to the nearest value that it can understand. This can cause small rounding errors, even before any calculations happen. For example, it can’t always understand 0.1 as exactly 1/10th of a value.
= List.Sum( { 0.1, 0.1, 0.1 } ) // Returns 0,30000000000000004
= List.Sum( { 0.1, 0.1, 0.1 }, 0 ) // Returns 0,30000000000000004
= List.Sum( { 0.1, 0.1, 0.1 }, Precision.Double ) // Returns 0,30000000000000004
= Value.Add( 0.2, 0.1 ) // Returns 0,30000000000000004
= Value.Add( 0.2, 0.1, 0 ) // Returns 0,30000000000000004
= Value.Add( 0.2, 0.1, Precision.Double ) // Returns 0,30000000000000004To be able to store more detailled numbers you can make use of another Precision Type, namely Precision.Decimal.
Related articles
Learn more about Precision.Double in the following articles:
- Power Query Precision: Avoid Rounding Errors
Learn how to handle precision differences in Power Query. With these tips you can return accurate numbers and prevent rounding errors. » Read more
Related enumerations
Other related enumerations are:
Applies to
Here’s a list of functions that work with Precision.Type:
- List.Average
- List.Product
- List.Sum
- Number.IntegerDivide
- Number.Mod
- Value.Add
- Value.Compare
- Value.Divide
- Value.Equals
- Value.Multiply
- Value.NullableEquals
- Value.Subtract
