Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate schema does not work #384

Closed
dolonmukherji opened this issue Mar 7, 2022 · 2 comments
Closed

Validate schema does not work #384

dolonmukherji opened this issue Mar 7, 2022 · 2 comments

Comments

@dolonmukherji
Copy link

I am trying to use the validate schema option with BigQuery, my source and target connections are the same and I am using 1 table, to compare schema with itself. Below is the command and the error message I got.
Please advise what this error means?

(venv) C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\Scripts>python -m data_validation validate schema -sc bq_source_conn -tc bq_source_conn -tbls workday.Employee_data
C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\ibis\backends\postgres\compiler.py:239: UserWarning: locale specific date formats (%c, %x, %X) are not yet implemented for Windows
'for %s' % platform.system()
C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\third_party\ibis\ibis_oracle\compiler.py:215: UserWarning: locale specific date formats (%c, %x, %X) are not yet implemented for Windows
'for %s' % platform.system()
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.2544.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 193, in run_module_as_main
"main", mod_spec)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.2544.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\data_validation_main
.py", line 385, in
main()
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\data_validation_main
.py", line 379, in main
validate(args)
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\data_validation_main_.py", line 358, in validate
run(args)
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\data_validation_main_.py", line 339, in run
run_validations(args, config_managers)
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\data_validation_main_.py", line 315, in run_validations
run_validation(config_manager, verbose=args.verbose)
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\data_validation_main_.py", line 304, in run_validation
validator.execute()
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\data_validation\data_validation.py", line 98, in execute
return self.result_handler.execute(self.config, result_df)
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\data_validation\result_handlers\text.py", line 61, in execute
self.print_formatted_(result_df)
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\data_validation\result_handlers\text.py", line 46, in print_formatted_
result_df.drop(self.cols_filter_list, axis=1).to_markdown(
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\pandas\core\frame.py", line 4315, in drop
errors=errors,
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\pandas\core\generic.py", line 4153, in drop
obj = obj._drop_axis(labels, axis, level=level, errors=errors)
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\pandas\core\generic.py", line 4188, in _drop_axis
new_axis = axis.drop(labels, errors=errors)
File "C:\Users\dmukherji\AppData\Local\Microsoft\WindowsApps\venv\lib\site-packages\pandas\core\indexes\base.py", line 5591, in drop
raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['difference' 'labels' 'pct_threshold'] not found in axis"

@nehanene15
Copy link
Collaborator

This may have been fixed by #371. Can this be confirmed?

@dolonmukherji
Copy link
Author

This works now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants