Precision bug: xs:untypedAtomic sum causes false BR-FXEXT-CO-10 / CO-13 failures

[philippfischer143]

Short description

Floating-point coercion in XSLT sum($lines) with xs:untypedAtomic introduces precision drift after xs:decimal(...) conversion.
This can trigger false Schematron failures for tolerance-based checks, including BR-FXEXT-CO-10 and BR-FXEXT-CO-13.

The actual value 142.76 is interpreted as 142.75999999999 creating edge cases to fail where exact values are expected.

Detailed description

When values are provided as xs:untypedAtomic, sum($lines) behaves like floating-point arithmetic.
Converting the result afterwards via xs:decimal(sum($lines)) preserves floating-point artifacts instead of exact decimal precision.

Observed example:

• expected logical amount: 142.76
• computed decimal after untyped sum: 142.759999999999990905052982270717620849609375
• delta to threshold boundary (142.77): approximately 0.010000000000009...
• named failure: threshold delta overflow at 0.01 tolerance boundary
  (<= 0.01 becomes false due to representation error, not business value mismatch)

This can cause false negatives in delta-based business rules, especially:

• BR-FXEXT-CO-10
• BR-FXEXT-CO-13

and similarly can occur in other rules using the same arithmetic pattern. I just checked BR-FXEXT-CO-10 and BR-FXEXT-CO-13.

Example XSLT (exact decimal behavior)

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xsl:output method="text" encoding="UTF-8"/>

  <xsl:template match="/">
    <!-- Force xs:decimal to illustrate exact decimal arithmetic -->
    <xsl:variable name="lines" as="xs:decimal*" select="(xs:decimal('142.76'))"/>
    <xsl:text>sum(lines as decimal) = </xsl:text>
    <xsl:value-of select="sum($lines)"/>
    <xsl:text>&#10;</xsl:text>
  </xsl:template>

</xsl:stylesheet>

Example XSLT (precision artifact)

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xsl:output method="text" encoding="UTF-8"/>

  <xsl:template match="/">
    <!-- Deliberately force xs:double to illustrate floating-point artifacts -->
    <xsl:variable name="lines" as="xs:untypedAtomic*" select="(xs:untypedAtomic('142.76'))"/>
    <xsl:text>sum(lines as untypedAtomic) = </xsl:text>
    <xsl:value-of select="sum($lines)"/>
    <xsl:text>&#10;as decimal = </xsl:text>
    <xsl:value-of select="xs:decimal(sum($lines))"/>
    <xsl:text>&#10;</xsl:text>
  </xsl:template>

</xsl:stylesheet>

Output

== sum-decimal.xslt ==
sum(lines as decimal) = 142.76

== sum-double.xslt ==
sum(lines as untypedAtomic) = 142.76
as decimal = 142.759999999999990905052982270717620849609375