Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Random row validations fail on Oracle when sampling 1001+ ids #1157

Closed
nj1973 opened this issue Jun 3, 2024 · 2 comments · Fixed by #1158
Closed

Random row validations fail on Oracle when sampling 1001+ ids #1157

nj1973 opened this issue Jun 3, 2024 · 2 comments · Fixed by #1158
Assignees
@nj1973
Labels
priority: p1 High priority. Fix may be included in the next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@nj1973
Copy link
Contributor

nj1973 commented Jun 3, 2024

When validating with --use-random-row --random-row-batch-size=1001

The validation fails with:

sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-01795: maximum number of expressions in a list is 1000

For this specific error we can build upon the recent enhancement from #1146 and add a 1000 IN list limit for Oracle connections.

But we should also think about why this option was added and what sensible limits we should build in.
@nj1973 nj1973 self-assigned this Jun 3, 2024
@nj1973
Copy link
Contributor Author

nj1973 commented Jun 3, 2024

For info I've tested up to --random-row-batch-size=50000 on Oracle, BigQuery and PostgreSQL and these all complete successfully for standard data types.

@nj1973
Copy link
Contributor Author

nj1973 commented Jun 3, 2024

I found and fixed a bug in the previous get_max_in_list_size() code.

@nj1973 nj1973 linked a pull request Jun 3, 2024 that will close this issue
fix: Fixes bug in get_max_in_list_size #1158
Merged
@helensilva14 helensilva14 added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p1 High priority. Fix may be included in the next release. labels Jun 3, 2024
@helensilva14 helensilva14 mentioned this issue Jun 3, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nj1973 nj1973

Labels
priority: p1 High priority. Fix may be included in the next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Fixes bug in get_max_in_list_size GoogleCloudPlatform/professional-services-data-validator
2 participants
@nj1973 @helensilva14