Precision.Decimal

Updated on

Precision.Decimal (1) is an enumeration that specifies the precision of comparison. It is a member of the Precision.Type and represents an optional parameter for the built-in arithmetic operators to specify decimal precision.

Examples

Decimal precision is a 128-bit floating-point number with a higher precision and a smaller range compared to Double Precision. Because this method allows for more precise numbers, it has less rounding errors. For example, whereas precision decimal cannot understand 0.1 as exactly 1/10th of a value, Precision Double can.

If you need a higher precision, you can provide your functions with Precision.Decimal instead. For instance, the List.Sum function has a Precision Type parameter.

List.Sum( { 0.1, 0.1, 0.1 }, 1 )                 // Returns 0,3
List.Sum( { 0.1, 0.1, 0.1 }, Precision.Decimal ) // Returns 0,3

Compare this with the lower precision of Precision.Double:

List.Sum( { 0.1, 0.1, 0.1 } )                    // Returns 0,30000000000000004
List.Sum( { 0.1, 0.1, 0.1 }, Precision.Double ) // Returns 0,30000000000000004

The same happens for functions like Value.Add:

Value.Add( 0.2, 0.1 )                    // Returns 0,30000000000000004
Value.Add( 0.2, 0.1, 1 )                 // Returns 0,3
Value.Add( 0.2, 0.1, Precision.Decimal ) // Returns 0,3

Learn more about Precision.Decimal in the following articles:

Other related enumerations are:

Applies to

Here’s a list of functions that work with Precision.Type:

Contribute » | Contributors: Rick de Groot