Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat!: Adds custom query row level hash validation feature. #440

Merged
merged 36 commits into from
Apr 28, 2022
Merged

feat!: Adds custom query row level hash validation feature. #440

merged 36 commits into from
Apr 28, 2022

Conversation

Robby29
Copy link
Member

@Robby29 Robby29 commented Apr 12, 2022

This PR adds the custom query row level hash validation feature with current support for Biqquery and Hive data sources.
Close #387

Robby29 and others added 23 commits March 2, 2022 17:16
@Robby29
added custom-query sub-option to validate command
308d5b4
@Robby29
add source and target query option in custom query
614ade6
@Robby29
added min,max,sum aggregates with custom query
72b55f3
@Robby29
fixed hive t0 column name addition issue
1567c09
@Robby29
added empty query file check
f9ab9f2
@Robby29
linting fixes
455eeab
@Robby29
Merge branch 'GoogleCloudPlatform:develop' into develop
f732c41
@Robby29
Merge branch 'GoogleCloudPlatform:develop' into develop
5b2b24b
@Robby29
added unit tests
994064f
@Robby29
Merge branch 'develop' of https://github.com/Robby29/professional-ser…
9b084e4 
…vices-data-validator into develop
@Robby29
incorporated black linting changes
78b8146
@Robby29
incorporated flake linter changes
cd43524
@Robby29
Merge branch 'GoogleCloudPlatform:develop' into develop
a6d8ab2
@Robby29
Merge branch 'GoogleCloudPlatform:develop' into develop
3bec521
@Raniksingh
Fixed result schema status to validation_status to avoid duplicate co…
7546fa9 
…lumn names
@Raniksingh
Fixed linting on tests folder
cb622fd
@nehanene15
BREAKING CHANGE: update BQ results schema column name 'status' to 'va…
dbea3e7 
…lidation_status'
@Raniksingh
Added script to update Bigquery schema
8d29441
@Raniksingh
Moved bq_utils to right folder
e2ed1d3
@Raniksingh
Updated bash script path and formatting
3f26768
@Robby29
Added custom query row validation feature
56287fa
@Robby29
Added custom query row validation feature.
fc5294d
@Robby29
Merge branch 'develop' of https://github.com/GoogleCloudPlatform/prof…
8c0ea59 
…essional-services-data-validator into develop
@Robby29
Copy link
Member Author

Robby29 commented Apr 12, 2022

/gcbrun

@Robby29
Copy link
Member Author

Robby29 commented Apr 12, 2022

/gcbrun

@Robby29
Copy link
Member Author

Robby29 commented Apr 13, 2022

/gcbrun

@Robby29
Copy link
Member Author

Robby29 commented Apr 14, 2022

/gcbrun

dhercher
dhercher previously requested changes Apr 14, 2022
View reviewed changes
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
data_validation/data_validation.py Show resolved Hide resolved
data_validation/cli_tools.py Show resolved Hide resolved
data_validation/query_builder/custom_query_row_builder.py Outdated Show resolved Hide resolved
data_validation/query_builder/custom_query_row_builder.py Outdated Show resolved Hide resolved
@Robby29
Copy link
Member Author

Robby29 commented Apr 14, 2022

/gcbrun

nehanene15
nehanene15 requested changes Apr 15, 2022
View reviewed changes
Copy link
Collaborator

@nehanene15 nehanene15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two architectural suggestions: Adding a custom query row/column in the CLI, and moving custom query column logic to the CustomQueryBuilder() class.

data_validation/combiner.py Show resolved Hide resolved
data_validation/validation_builder.py Outdated Show resolved Hide resolved
@Robby29
Copy link
Member Author

Robby29 commented Apr 23, 2022

/gcbrun

@Robby29 Robby29 requested a review from nehanene15 April 23, 2022 17:58
nehanene15
nehanene15 reviewed Apr 27, 2022
View reviewed changes
README.md Outdated Show resolved Hide resolved
nehanene15
nehanene15 requested changes Apr 27, 2022
View reviewed changes
Copy link
Collaborator

@nehanene15 nehanene15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One change requested re: how we store the custom_query_type property as a string instead of a list

data_validation/cli_tools.py Show resolved Hide resolved
data_validation/combiner.py Show resolved Hide resolved
data_validation/config_manager.py Outdated Show resolved Hide resolved
data_validation/config_manager.py Show resolved Hide resolved
@Robby29
Copy link
Member Author

Robby29 commented Apr 28, 2022

/gcbrun

@Robby29
Copy link
Member Author

Robby29 commented Apr 28, 2022

/gcbrun

nehanene15
nehanene15 approved these changes Apr 28, 2022
View reviewed changes
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@Robby29
Copy link
Member Author

Robby29 commented Apr 28, 2022

/gcbrun

@Robby29 Robby29 enabled auto-merge (squash) April 28, 2022 08:19
@Robby29
Copy link
Member Author

Robby29 commented Apr 28, 2022

/gcbrun

@Robby29 Robby29 dismissed dhercher’s stale review April 28, 2022 08:29

Already resolved all the comments by Dylan. Using this option as a workaround to move forward

@Robby29 Robby29 merged commit f057fe8 into GoogleCloudPlatform:develop Apr 28, 2022
@release-please release-please bot mentioned this pull request Apr 28, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dhercher dhercher dhercher left review comments

@nehanene15 nehanene15 nehanene15 approved these changes

@renzokuken renzokuken Awaiting requested review from renzokuken

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Custom query validation for row level hash
4 participants
@Robby29 @dhercher @nehanene15 @Raniksingh