generate-table-partitions between Teradata and BQ does not work with uppercase primary key names #1136
Labels
priority: p0
Highest priority. Critical issue. Will be fixed prior to next release.
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Hi,
It appears that generate-table-partitions does not work with primary key names in Teradata that are in upper case. Teradata's column names are not case sensitive.
The DDL that was used was
When we try the following command, it works fine
data-validation generate-table-partitions -sc teradata -tc teradata -tbls=udfs.issue1136 -pk=N_NATIONKEY -hash '*' -pn 3 -cdir .
The real purpose is to compare two tables from different databases - so we try to compare teradata with BigQuery table defined as follows:
If we use the command
data-validation generate-table-partitions -sc teradata -tc bq -tbls=udfs.issue1136=pso_data_validator.issue1136 -pk=N_NATIONKEY -hash '*' -pn 3 -cdir .
you get the error:However if you invert the case and try
data-validation generate-table-partitions -sc teradata -tc bq -tbls=udfs.issue1136=pso_data_validator.issue1136 -pk=n_nationkey -hash '*' -pn 3 -cdir .
you get the error:In the first case, ibis is not able to execute the count operation since the BQ column names are all in lower case. In the second case, ibis is not able to execute the count operation since the Teradata column names are all in upper case. So either case you use - it does not work. We try to get the count to make sure the table sizes are within 10% of each other - if not the partition split is not going to be very even.
I checked with a table defined with lower case column names in Oracle and it return ibis column names in lower case. A bit more investigation is needed.
Suggestions on how to fix ?
The text was updated successfully, but these errors were encountered: