Skip to content

Commit

Permalink
fix: dry-run bug when running configs, added CODEOWNERS, and docs (#865)
Browse files Browse the repository at this point in the history
  • Loading branch information
@nehanene15
nehanene15 committed May 19, 2023
1 parent 1462deb commit 1779772
Show file tree
Hide file tree
Showing 4 changed files with 180 additions and 267 deletions.
13 changes: 13 additions & 0 deletions CODEOWNERS
View file
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Order is important; the last matching pattern takes the most
# precedence.

# The main branch in this repo is protected and requires reviews
# from code owners. This means that any pull request
# which affects code with a code owner must be approved
# by that code owner before the pull request can be merged
# into the protected branch.

# Pull requests which modify any of the following
# file types must be approved by one of the listed owner(s):

*.py @GoogleCloudPlatform/pso-code-review-committee
2 changes: 1 addition & 1 deletion data_validation/__main__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -466,7 +466,7 @@ def run_validations(args, config_managers):
)
except Exception as e:
logging.error(
"Error %s occured while running config file %s. Skipping it for now.",
"Error %s occurred while running config file %s. Skipping it for now.",
str(e),
config_manager.config[consts.CONFIG_FILE],
)
Expand Down
6 changes: 6 additions & 0 deletions data_validation/cli_tools.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,12 @@ def _configure_validation_config_parser(subparsers):
run_parser = configs_subparsers.add_parser(
"run", help="Run your validation configs"
)
run_parser.add_argument(
"--dry-run",
"-dr",
action="store_true",
help="Prints source and target SQL to stdout in lieu of performing a validation.",
)
run_parser.add_argument(
"--config-file",
"-c",
Expand Down
Loading

2 comments on commit 1779772

@tprashanth
Copy link

@tprashanth tprashanth commented on 1779772 May 29, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Neha - I have installed DVT successfully on RHEL 8. I was able to compare 2 tables successfully between Teradata and BigQuery. When I used config file, it is erroring out

$ data-validation validate column -sc /opt/devtools/professional-services-data-validator/conn/dvt-td-edw -tc /opt/devtools/professional-services-data-validator/conn/dvt-bq-dev -tbls edw.xyz
/usr/local/lib/python3.9/site-packages/third_party/ibis/ibis_teradata/client.py:156: FutureWarning: Using short name for 'orient' is deprecated. Only the options: ('dict', list, 'series', 'split', 'records', 'index') will be used in a future version. Use one of the above to silence this warning.
schema_list = schema_df.to_dict(orient="record")
╒═══════════════════╤═══════════════════╤══════════════════════════════════╤══════════════════════╤════════════════════╤════════════════════╤══════════════════╤═════════════════════╤══════════════════════════════════════╕
│ validation_name │ validation_type │ source_table_name │ source_column_name │ source_agg_value │ target_agg_value │ pct_difference │ validation_status │ run_id │
╞═══════════════════╪═══════════════════╪══════════════════════════════════╪══════════════════════╪════════════════════╪════════════════════╪══════════════════╪═════════════════════╪══════════════════════════════════════╡
│ count │ Column │ edw.xyz │ │ 218101026 │ 1227270 │ -99.4373 │ fail │ 864a6f65-6f52-46e6-85c1-5811cdff7ba3 │
╘═══════════════════╧═══════════════════╧══════════════════════════════════╧══════════════════════╧════════════════════╧════════════════════╧══════════════════╧═════════════════════╧══════════════════════════════════════╛

$ data-validation configs run -c /opt/devtools/professional-services-data-validator/td2bq.yaml
05/29/2023 04:12:40 PM-INFO: Currently running the validation for yml file: /opt/devtools/professional-services-data-validator/td2bq.yaml
05/29/2023 04:12:40 PM-ERROR: Error 'Namespace' object has no attribute 'dry_run' occured while running config file /opt/devtools/professional-services-data-validator/td2bq.yaml. Skipping it for now.

Assuming I probably cloned a older one (based on the fix for --dry_run last week), I repeated all steps today 05/29, but ended with the same error. Can you please help me with possible issues or please let me know if you need any further information.

If I add "--dry_run argument to the above cmd at the last, it is erroring out with:
command:
data-validation
data-validation: error: unrecognized arguments: --dry_run

Do you suggest me to replace __main__.py and cli_tools.py files from my installation directory with the ones on this page?

@tprashanth
Copy link

@tprashanth tprashanth commented on 1779772 May 30, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did wc -l __main__.py and wc -l cli_tools.py. The # of lines matched between the result I got Vs what I see in the git. I also looked for the latest changes in the files that matched.

$ wc -l cli_tools.py
1313 cli_tools.py

$ wc -l __main__.py
585 __main__.py

Please sign in to comment.