refactor: custom query code #607

pshrivastava27 · 2022-10-13T13:52:29Z

This PR includes following changes:

Refactoring custom query (row and column validation) code to use ibis framework (fix Refactor custom query validation to use query builder #539 )
Instead of constructing nested queries manually using string concatenation, we now use ibis framework similar to how we construct queries for regular validations.
Removing table list dependency for custom query row validation (fix Deprecate the usage of --tables-list/-tbls argument from DVT command for custom-query validation #575 )
Resolving issues specific to TD custom query row validation (trailing whitespaces after hash value)
Making flags source-query-file and target-query-file mandatory for custom-query validation commands.
Making primary-keysflag mandatory for custom-query row validation.

pshrivastava27 · 2022-10-13T13:52:50Z

/gcbrun

pshrivastava27 · 2022-10-18T11:56:56Z

/gcbrun

Raniksingh · 2022-10-18T12:56:23Z

Can you also update the README with the following:

Remove tbls from custom query syntax(as this PR will remove tbls dependency from custom query
add PK in the syntax for custom query as its a mandatory argument now
add the syntax for concat option for both regular and custom query row validation syntax.

data_validation/__main__.py

nehanene15

Have we tested with saving the custom query validation to YAML and then running it?
Does this support comparison field validations or only hash/concat?

data_validation/cli_tools.py

data_validation/__main__.py

data_validation/validation_builder.py

data_validation/query_builder/query_builder.py

data_validation/config_manager.py

data_validation/query_builder/query_builder.py

…ted testcase

pshrivastava27 · 2022-10-20T14:54:55Z

Have we tested with saving the custom query validation to YAML and then running it? Does this support comparison field validations or only hash/concat?

Yes. We tested generating the yaml config and then running the validation using config file. Works as expected.

pshrivastava27 · 2022-10-20T14:55:59Z

Can you also update the README with the following:

Remove tbls from custom query syntax(as this PR will remove tbls dependency from custom query

add PK in the syntax for custom query as its a mandatory argument now

add the syntax for concat option for both regular and custom query row validation syntax.

Done here

pshrivastava27 · 2022-10-20T14:56:23Z

/gcbrun

pshrivastava27 · 2022-10-20T16:01:24Z

/gcbrun

data_validation/__main__.py

data_validation/validation_builder.py

pshrivastava27 · 2022-10-21T13:00:59Z

/gcbrun

pshrivastava27 · 2022-10-21T14:34:50Z

/gcbrun

data_validation/__main__.py

data_validation/cli_tools.py

data_validation/validation_builder.py

tests/unit/test_validation_builder.py

pshrivastava27 · 2022-10-26T11:04:42Z

/gcbrun

pshrivastava27 · 2022-10-26T12:11:30Z

/gcbrun

pshrivastava27 · 2022-10-26T13:44:40Z

/gcbrun

pshrivastava27 · 2022-10-27T11:16:53Z

/gcbrun

nehanene15

Logic looks good, just suggested some code style changes.

data_validation/__main__.py

pshrivastava27 · 2022-10-28T11:18:30Z

/gcbrun

nehanene15

Approved. Suggested one last comment to delete the query_type in the arg parser since it is not used.

data_validation/cli_tools.py

pshrivastava27 · 2022-10-31T12:03:23Z

/gcbrun

pshrivastava27 added 7 commits October 13, 2022 13:25

remove table-list dependency for custom query validation

e965986

Fix t0 alias issue while building hive table expression

1cea478

Fix trailing whitespace issue with TD hash function

6d37d69

use ibis expression instead of string manipulation for custom-query

c848ab8

Fix TD hash unmatching issue

498fb32

update flags for custom query parser

c00832c

fix linting issues

32b99e9

pshrivastava27 requested review from nehanene15, Raniksingh and sramsrinivasan October 13, 2022 18:24

revert log level related changes

510d651

sramsrinivasan reviewed Oct 18, 2022

View reviewed changes

nehanene15 requested changes Oct 18, 2022

View reviewed changes

pshrivastava27 added 5 commits October 19, 2022 08:25

updated function get_query_from_file() and moved function and associa…

730c8ca

…ted testcase

Added consts for source_query and target_query

78cbe13

minor modifications to improve code readability

0e6ac69

remove custom_query_builder.py and its testcase

f5b17d9

Updated README

4209754

pshrivastava27 self-assigned this Oct 20, 2022

fix linting

2e6b947

sramsrinivasan requested changes Oct 20, 2022

View reviewed changes

data_validation/__main__.py Outdated Show resolved Hide resolved

data_validation/__main__.py Show resolved Hide resolved

data_validation/validation_builder.py Outdated Show resolved Hide resolved

data_validation/validation_builder.py Outdated Show resolved Hide resolved

improve code readability

4f2b8ef

updated README

fd70fcf

nehanene15 reviewed Oct 21, 2022

View reviewed changes

data_validation/__main__.py Show resolved Hide resolved

nehanene15 reviewed Oct 21, 2022

View reviewed changes

code cleanup

f0eb38b

pshrivastava27 and others added 2 commits October 26, 2022 12:05

update testcase

660486c

Merge branch 'develop' into custom-query-refactor-code

c863851

code cleanup

f4f7703

update README, display warning for tbls list in custom-query validations

0ab8891

nehanene15 reviewed Oct 27, 2022

View reviewed changes

data_validation/__main__.py Outdated Show resolved Hide resolved

data_validation/__main__.py Outdated Show resolved Hide resolved

data_validation/__main__.py Outdated Show resolved Hide resolved

data_validation/__main__.py Outdated Show resolved Hide resolved

improved code readability

d39c63d

nehanene15 approved these changes Oct 28, 2022

View reviewed changes

data_validation/cli_tools.py Outdated Show resolved Hide resolved

data_validation/cli_tools.py Outdated Show resolved Hide resolved

removed unused code

7667b5f

pshrivastava27 enabled auto-merge (squash) October 31, 2022 12:39

pshrivastava27 removed the request for review from Raniksingh October 31, 2022 12:41

sramsrinivasan approved these changes Oct 31, 2022

View reviewed changes

pshrivastava27 merged commit a7f800d into develop Oct 31, 2022

pshrivastava27 deleted the custom-query-refactor-code branch October 31, 2022 14:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: custom query code #607

refactor: custom query code #607

pshrivastava27 commented Oct 13, 2022

pshrivastava27 commented Oct 13, 2022

pshrivastava27 commented Oct 18, 2022

Raniksingh commented Oct 18, 2022

nehanene15 left a comment

pshrivastava27 commented Oct 20, 2022

pshrivastava27 commented Oct 20, 2022

pshrivastava27 commented Oct 20, 2022

pshrivastava27 commented Oct 20, 2022

pshrivastava27 commented Oct 21, 2022

pshrivastava27 commented Oct 21, 2022

pshrivastava27 commented Oct 26, 2022

pshrivastava27 commented Oct 26, 2022

pshrivastava27 commented Oct 26, 2022

pshrivastava27 commented Oct 27, 2022

nehanene15 left a comment

pshrivastava27 commented Oct 28, 2022

nehanene15 left a comment

pshrivastava27 commented Oct 31, 2022

refactor: custom query code #607

refactor: custom query code #607

Conversation

pshrivastava27 commented Oct 13, 2022

This PR includes following changes:

pshrivastava27 commented Oct 13, 2022

pshrivastava27 commented Oct 18, 2022

Raniksingh commented Oct 18, 2022

nehanene15 left a comment

Choose a reason for hiding this comment

pshrivastava27 commented Oct 20, 2022

pshrivastava27 commented Oct 20, 2022

pshrivastava27 commented Oct 20, 2022

pshrivastava27 commented Oct 20, 2022

pshrivastava27 commented Oct 21, 2022

pshrivastava27 commented Oct 21, 2022

pshrivastava27 commented Oct 26, 2022

pshrivastava27 commented Oct 26, 2022

pshrivastava27 commented Oct 26, 2022

pshrivastava27 commented Oct 27, 2022

nehanene15 left a comment

Choose a reason for hiding this comment

pshrivastava27 commented Oct 28, 2022

nehanene15 left a comment

Choose a reason for hiding this comment

pshrivastava27 commented Oct 31, 2022