Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

validate custom-query: Exception when PostgreSQL column data type has length or precision #681

Closed
nj1973 opened this issue Jan 30, 2023 · 0 comments · Fixed by #723
Closed
Assignees
Labels
priority: p0 Highest priority. Critical issue. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@nj1973
Copy link
Contributor

nj1973 commented Jan 30, 2023

When validating by query between Oracle and PostgreSQL we get an exception if any validated column has a length or precision.

Demonstrating below with a PostgreSQL to PostgreSQL comparison to keep things simple.

Test table:

CREATE TABLE dvt_test.qtab (
  id bigint
, col1 numeric(4)
) ;
INSERT INTO dvt_test.qtab VALUES (1, 111);

Query file:

$ echo "select id,col1 from dvt_test.qtab" > /tmp/q.sql

Validation command:

$ data-validation validate custom-query --custom-query-type row \
--source-query-file /tmp/q.sql --target-query-file /tmp/q.sql \
-sc pg_conn -tc pg_conn --hash '*' --primary-keys id
...
  File "/usr/local/google/home/neiljohnson/github/professional-services-data-validator/.venv/lib/python3.10/site-packages/ibis/expr/datatypes.py", line 1573, in default
    raise com.IbisTypeError('Value {!r} is not a valid datatype'.format(value))
ibis.common.exceptions.IbisTypeError: Value None is not a valid datatype

Column data types are decoded from the query in third_party/ibis/ibis_postgres/client.py in _get_schema_using_query().
The data type string, for example 'numeric(4,0)' is not matched in the list of the accepted types because type mappings do not include the precision/length in parentheses.

@nehanene15 nehanene15 added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p0 Highest priority. Critical issue. Will be fixed prior to next release. labels Jan 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p0 Highest priority. Critical issue. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
3 participants