Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: get_schema with Postgres #411

Merged
merged 2 commits into from
Apr 23, 2022
Merged

Conversation

FX-HAO
Copy link
Contributor

@FX-HAO FX-HAO commented Mar 30, 2022

Copy link
Collaborator

@dhercher dhercher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good. It would be great to add an integration test to either MySQL or PSQL to validate this PR. Also to BigQuery or a different source which should not be affected to ensure no gression

@FX-HAO FX-HAO force-pushed the develop branch 2 times, most recently from 686dd27 to 2b468cf Compare March 31, 2022 11:07
@FX-HAO
Copy link
Contributor Author

FX-HAO commented Mar 31, 2022

@dhercher added tests for MySQL and PG. There are already schema validations for other sources, see

def test_schema_validation():
validator = data_validation.DataValidation(CONFIG_SCHEMA_VALIDATION, verbose=True)
df = validator.execute()
for validation in df.to_dict(orient="records"):
assert validation["status"] == consts.VALIDATION_STATUS_SUCCESS

@nehanene15
Copy link
Collaborator

/gcbrun

@FX-HAO
Copy link
Contributor Author

FX-HAO commented Apr 4, 2022

don't know why the CI failed, both of tests succeeded on my local. and cannot get the error details

@dhercher
Copy link
Collaborator

dhercher commented Apr 4, 2022

don't know why the CI failed, both of tests succeeded on my local. and cannot get the error details

Traceback (most recent call last):
  File "/workspace/.nox/lint/bin/black", line 8, in <module>
    sys.exit(patched_main())
  File "/workspace/.nox/lint/lib/python3.9/site-packages/black.py", line 4134, in patched_main
    patch_click()
  File "/workspace/.nox/lint/lib/python3.9/site-packages/black.py", line 4123, in patch_click
    from click import _unicodefun  # type: ignore
ImportError: cannot import name '_unicodefun' from 'click' (/workspace/.nox/lint/lib/python3.9/site-packages/click/__init__.py)

@nehanene15
Copy link
Collaborator

don't know why the CI failed, both of tests succeeded on my local. and cannot get the error details

This was an issue with a package being used in our Cloud Build tests. Please pull and merge the latest code from develop to resolve this.

@FX-HAO
Copy link
Contributor Author

FX-HAO commented Apr 5, 2022

@nehanene15 updated with the latest code. please rerun the tests

@dhercher
Copy link
Collaborator

We've been cleaning some dependency issues which cropped up. I'm going to try tests but likely you will need to merge develop again

@dhercher
Copy link
Collaborator

/gcbrun

@nehanene15 nehanene15 requested a review from ngdav April 14, 2022 18:12
@FX-HAO FX-HAO changed the title fix get_schema with Postgres fix: get_schema with Postgres Apr 15, 2022
@FX-HAO
Copy link
Contributor Author

FX-HAO commented Apr 15, 2022

updated

@dhercher
Copy link
Collaborator

/gcbrun

@dhercher dhercher merged commit 4093625 into GoogleCloudPlatform:develop Apr 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants