Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add and document FileSystem connection type #250

Closed
nehanene15 opened this issue May 12, 2021 · 0 comments · Fixed by #254
Closed

Add and document FileSystem connection type #250

nehanene15 opened this issue May 12, 2021 · 0 comments · Fixed by #254
Assignees
Labels
priority: p1 High priority. Fix may be included in the next release. type: cleanup An internal cleanup or hygiene concern.

Comments

@nehanene15
Copy link
Collaborator

Currently, a user can connect to a GCS/S3/local file like so:

data-validation connections add -c pandas Raw --json '{"source_type":"Pandas","table_name":"test","file_path":"test_data.json","file_type":"json"}'
data-validation run -t Column -sc pandas -tc pandas -tbls '[{"schema_name":"","table_name":"test"}]'

And create the Pandas connection like so:

pip install fsspec gcsfs
data-validation connections add -c pandas Raw --json '{"source_type":"Pandas","table_name":"test","file_path":"gs://dataflow-datastream-testing/test_data.json","file_type":"json"}'

We should rename the 'Pandas' connection to 'FileSystem' and build it into the CLI so that users do not need to use the 'Raw' JSON. In the case of a 'FileSystem' connection type, a 'schema_name' should not be required in the tables-list.

@nehanene15 nehanene15 added priority: p1 High priority. Fix may be included in the next release. type: cleanup An internal cleanup or hygiene concern. labels May 12, 2021
@nehanene15 nehanene15 self-assigned this May 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p1 High priority. Fix may be included in the next release. type: cleanup An internal cleanup or hygiene concern.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant