Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow DVT to run multiple YAMLs in parallel up to specified concurrency #608

Closed
nehanene15 opened this issue Oct 13, 2022 · 2 comments
Closed

Allow DVT to run multiple YAMLs in parallel up to specified concurrency #608

nehanene15 opened this issue Oct 13, 2022 · 2 comments
Assignees
@dmedora
Labels
priority: p0 Highest priority. Critical issue. Will be fixed prior to next release.

Comments

@nehanene15
Copy link
Collaborator

nehanene15 commented Oct 13, 2022 •
edited
Loading

DVT should allow users to run multiple YAML configs in parallel up to specified concurrency. This will be beneficial to users that want to run validations that cover chunks of the same table (Issue #598).

Example:
data-validation configs run --config-dir /path/to/yamls --concurrency 10

Document that this will create multiple DB connections.
@nehanene15 nehanene15 added the priority: p0 Highest priority. Critical issue. Will be fixed prior to next release. label Oct 14, 2022
@dmedora dmedora self-assigned this Oct 19, 2022
@dmedora
Copy link
Member

dmedora commented Nov 7, 2022

Work in progress:
Implemented --config-dir support. Still working out multiprocessing; stalled due to pickling errors (unpickleable objects).

@nehanene15
Copy link
Collaborator Author

Closed with PR #654

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dmedora dmedora

Labels
priority: p0 Highest priority. Critical issue. Will be fixed prior to next release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dmedora @nehanene15