Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support BQ decimal precision and scale for schema validation #960

Merged
merged 2 commits into from
Aug 29, 2023

Conversation

nehanene15
Copy link
Collaborator

…imit Teradata query to one row for custom query schema
@nehanene15 nehanene15 requested a review from a team as a code owner August 28, 2023 20:26
@nehanene15
Copy link
Collaborator Author

nehanene15 commented Aug 28, 2023

/gcbrun

@nehanene15 nehanene15 requested a review from nj1973 August 29, 2023 03:38
Copy link
Contributor

@nj1973 nj1973 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM aside from comment about Hive (and maybe other engines) having the full scan issue fixed here for Teradata.

@@ -141,7 +141,7 @@ def get_schema(self, table_name: str, database: str = None) -> sch.Schema:
return sch.Schema(schema)

def _get_schema_using_query(self, query):
cur = self.raw_sql(query)
cur = self.raw_sql(f"SELECT TOP 1 * FROM ({query}) AS t0")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might also have this problem for other engines, e.g: Hive:

def _get_schema_using_query(self, query):
    # Removing LIMIT 0 around query since it returns no results in Hive
    cur = self.raw_sql(query)
    cur.fetchall()
    ibis_fields = self._adapt_types(cur.description)
    cur.release()

    return sch.Schema(ibis_fields)

We should be able to use fetchone in place of fetchall for this. Or embed in a subquery. I didn't check all engines so Hive may not be the only one.

Happy for this to be spun into another issue if you need to move quickly on this one.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I'll update this for Hive and add a LIMIT 1 similar to the Ibis impala connection. Spanner only materializes one row, and all other DBs are handled by Ibis.

@nehanene15 nehanene15 merged commit b1d4942 into develop Aug 29, 2023
5 checks passed
@nehanene15 nehanene15 deleted the issue883-bq-decimal-precision branch August 29, 2023 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Schema validation: BigQuery Decimal Precision Schema Validation : Bigquery Numeric
2 participants