Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Hive boolean nan to None, Unsupported ibis data types in structs and arrays #444

Merged
merged 4 commits into from
Apr 14, 2022

Conversation

nehanene15
Copy link
Collaborator

Closes Issue #441

  • Converts NULL boolean values in Hive from returning 'nan' to 'None' to be compatible with BigQuery boolean NULLs
  • Adds support for structs with unsupported ibis data types which resolves the '{!r} cannot be parsed as a datatype'.format(value) ibis.common.exceptions.IbisTypeError when running a column aggregation on a table with struct fields (the validation itself should exclude the struct fields)

@nehanene15
Copy link
Collaborator Author

/gcbrun

@nehanene15 nehanene15 requested a review from dmedora April 13, 2022 21:19
@nehanene15 nehanene15 merged commit e94a1da into develop Apr 14, 2022
@nehanene15 nehanene15 deleted the issue441-hive-boolean branch April 14, 2022 13:53
FX-HAO pushed a commit to FX-HAO/professional-services-data-validator that referenced this pull request Apr 15, 2022
… and arrays (GoogleCloudPlatform#444)

* fix: boolean nan issue for Hive, fixes column agg for tables with unsupported ibis data types

* blacken updates after merge

* add varchar and char support
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants