fix: generate-table-partitions- fixes Issue 945 and Issue 950 #962
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rlier Added functionality to support Kubernetes Indexed jobs - which when provided with a directory will only run the job corresponding to the index. Tested in a non Kubernetes setup
…bis to turn table expressions into SQL statements. This addresses bugs #945 and #950. Unfortunately, we depend on the version of sqlalchemy being 2.0 or later which has fixed a problem with datetime being rendered by compile - see https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-206ec1f2af3a0c93785758c723ba356f
…D and test cases. Need to check that everything works.
…s' of https://github.com/GoogleCloudPlatform/professional-services-data-validator into 950-filters-are-not-applied-before-generating-partitions
…ifferent, though semantically the same Cleaned up tests and partition_builder to pass flake8 and black
|
/gcbrun |
sundar-mudupalli-work
changed the title
sundar-mudupalli-work
changed the title
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
helensilva14
added
the
priority: p0
nehanene15
approved these changes
Aug 31, 2023
| ``` | ||
| The internal select statement adds the partition number to each row in the table and the external select statement gets the value of the primary keys for the first row. | ||
| ### How to generate the where clauses | ||
| Once we have the first row of each partition, we have to generate the where clauses for each partition in the source and target tables. The best way may be to generate the ibis table expression including the provided filter clause and the additional filter clause from the first rows we have calculated. We can then have _ibis_ `to_sql` convert the table expression into plain text, extract the where clause and use that. _ibis_ depends on _sqlalchemy_, which has a bug in that it does not support rendering date and timestamps by `to_sql` for versions of _sqlalchemy_ prior to 2.0. Until. we migrate to using _sqlalchemy_ 2.0, we may not be able to support dates and timestamps as a primary key column. |
There was a problem hiding this comment.
nit: typo here "Until. we migrate to using sqlalchemy 2.0"
sundar-mudupalli-work
deleted the
950-filters-are-not-applied-before-generating-partitions
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fix generate-table-partitions: Fixes Issue 945 and Issue 950
When generating partition filters, fixes string literals that might have
'and includes filters provided in the data-validation command line.