Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate Column documentation implies that count * will only run if no aggregation flag is provided #983

Closed
sundar-mudupalli-work opened this issue Sep 12, 2023 · 0 comments · Fixed by #1082
Labels
priority: p2 Medium priority. Fix may not be included in next release (e.g. minor documentation, cleanup) type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. type: docs

Comments

@sundar-mudupalli-work
Copy link
Contributor

Hi,

When you do a validate column, a count * is always run, whether another aggregation is specified or not. The documentation says The default aggregation type is a 'COUNT *'. If no aggregation flag (i.e count, sum , min, etc.) is provided, the default aggregation will run.. However, the code always includes a count *. See example below. The document should clearly say COUNT * is always run. Other aggregations will be run if requested..

data-validation --verbose validate column -sc=my_postgres -tc=my_postgres -tbls public.group_test_1=public.group_test_1  --sum revenue
09/12/2023 02:16:07 AM-INFO: {'data_client': <ibis.backends.postgres.Backend object at 0x7f2632f10850>, 'schema_name': 'public', 'table_name': 'group_test_1', 'source_query': None}
09/12/2023 02:16:07 AM-INFO: -- ** Source Query ** --
09/12/2023 02:16:07 AM-INFO: SELECT count(*) AS count, sum(t0.revenue) AS sum__revenue 
FROM public.group_test_1 AS t0
09/12/2023 02:16:07 AM-INFO: {'data_client': <ibis.backends.postgres.Backend object at 0x7f2632f11030>, 'schema_name': 'public', 'table_name': 'group_test_1', 'target_query': None}
09/12/2023 02:16:07 AM-INFO: -- ** Target Query ** --
09/12/2023 02:16:07 AM-INFO: SELECT count(*) AS count, sum(t0.revenue) AS sum__revenue 
FROM public.group_test_1 AS t0
╒═══════════════════╤═══════════════════╤═════════════════════╤══════════════════════╤════════════════════╤════════════════════╤══════════════════╤═════════════════════╤══════════════════════════════════════╕
│ validation_name   │ validation_type   │ source_table_name   │ source_column_name   │   source_agg_value │   target_agg_value │   pct_difference │ validation_status   │ run_id                               │
╞═══════════════════╪═══════════════════╪═════════════════════╪══════════════════════╪════════════════════╪════════════════════╪══════════════════╪═════════════════════╪══════════════════════════════════════╡
│ count             │ Column            │ public.group_test_1 │                      │                  8 │                  8 │                0 │ success             │ 28897fef-32b3-40a2-9058-7195e30025f9 │
├───────────────────┼───────────────────┼─────────────────────┼──────────────────────┼────────────────────┼────────────────────┼──────────────────┼─────────────────────┼──────────────────────────────────────┤
│ sum__revenue      │ Column            │ public.group_test_1 │ revenue              │                360 │                360 │                0 │ success             │ 28897fef-32b3-40a2-9058-7195e30025f9 │
╘═══════════════════╧═══════════════════╧═════════════════════╧══════════════════════╧════════════════════╧════════════════════╧══════════════════╧═════════════════════╧══════════════════════════════════════╛
@sundar-mudupalli-work sundar-mudupalli-work added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Medium priority. Fix may not be included in next release (e.g. minor documentation, cleanup) type: docs labels Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 Medium priority. Fix may not be included in next release (e.g. minor documentation, cleanup) type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. type: docs
Projects
None yet
1 participant