Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Schema validations ignore not null on Teradata and BigQuery #935

Merged
merged 13 commits into from
Aug 14, 2023

Conversation

nj1973
Copy link
Contributor

@nj1973 nj1973 commented Aug 8, 2023

This change adds code to correctly identify nullable True/False on Teradata and BigQuery, bringing them into line with other engines (MySQL, Oracle, PostgreSQL, Snowflake, SQL Server).

It also adds tests for all of these engines to avoid regressions.

It does not add support or tests for these engines:

  • Hive: Raised issue schema validation: Nullable=False not identified on Hive #934
  • Spanner: The Spanner client does not include nullable information, it appears to get data types from a cursor of a SELECT * ... LIMIT 0 query which does not expose schema attributes. I feel this is too far removed from DVT to tackle

We did talk about adding a flag to give users the option to ignore nullable True/False for schema validations, this would be useful for Hive and Spanner. But I decided that it would be a feature request and not a function of this issue.

@nj1973 nj1973 requested a review from a team as a code owner August 8, 2023 12:42
@nj1973
Copy link
Contributor Author

nj1973 commented Aug 9, 2023

Please do not review this yet. I am getting a Python Failed to build thrift error which I do not understand because I haven't added any new packages. What I have done is import bigquery in a different file so perhaps that has had a knock on effect.

@nehanene15
Copy link
Collaborator

Please do not review this yet. I am getting a Python Failed to build thrift error which I do not understand because I haven't added any new packages. What I have done is import bigquery in a different file so perhaps that has had a knock on effect.

You may want to try re-running the build to make sure it's not an ephemeral error

@nj1973
Copy link
Contributor Author

nj1973 commented Aug 11, 2023

/gcbrun

@nj1973
Copy link
Contributor Author

nj1973 commented Aug 11, 2023

You may want to try re-running the build to make sure it's not an ephemeral error

@nehanene15 , you were right, the build failure was transient. Once the test had run they highlighted other issues for me to work on.

This should now be ready for review.

Copy link
Collaborator

@nehanene15 nehanene15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@nj1973 nj1973 merged commit 936744b into develop Aug 14, 2023
3 checks passed
@nj1973 nj1973 deleted the issue-926-schema-validations-ignore-notnull branch August 14, 2023 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Schema Validations between Teradata and BigQuery ignore nullability constraints
2 participants