validate column: sum, min, max false success for numeric values with precision > 16 #754
Labels
priority: p1
High priority. Fix may be included in the next release.
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Test BigQuery tables:
Note that tab_long_dec2 data differs from tab_long_dec1. The value in col_16, col_27 and col_18 has been incremented by 1.
Therefore sum and max comparison of these columns should fail.
col_16
Correct behaviour, all columns have validation_status="fail"
col_17
Incorrect behaviour,
sum__col_17
andmax__col_17
have validation_status="success". This is wrong.col_18
Incorrect behaviour,
sum__col_18
andmax__col_18
have validation_status="success". This is wrong.I suspect this relates to code in
data_validation/combiner.py
that casts all values tofloat64
. 64 bit float is a lossy data type for large values.I saw the same issue when testing PostgreSQL too, I believe the problem is not engine specific.
The text was updated successfully, but these errors were encountered: