Filters are not applied before generating partitions

[florisvink]

In the documentation is states that the generate partitions command works the same way as validate rows. It seems however that custom filters or a custom sql query source/target are not applied first when generating partitions and so partitions will be generated based on the full source- and target table. In our situation we want to validate the last 6 months of data in a multi billion row dataset so we run the partition generation like:

data-validation generate-table-partitions 
    -sc rs -tc bq \
    -tbls huge_source_table=huge_target_table \ 
    --primary-keys source_order_id,platform_id,order_transaction,source_customer_id
    --comparison-fields restaurant,branch,is_chain,chain_name,account_id,... \
    --partition-num 1000 --filter-status fail \
    --bq-result-handler bq_result_table \
    --filter "order_date_time > '2023-02-15' and platform_id=9"
    -cdir /targetdir_for_partitions

we got partitioned configs like:

...
  filters:
  - source: order_date_time > '2023-02-15' and platform_id=9
    target: order_date_time > '2023-02-15' and platform_id=9
    type: custom
  - source: source_order_id < '447191446' OR source_order_id = '447191446' AND (platform_id
      < 9 OR platform_id = 9 AND (order_transaction < 'Transaction' OR order_transaction
      = 'Transaction' AND (source_customer_id < '1772063')))
    target: source_order_id < '447191446' OR source_order_id = '447191446' AND (platform_id
      < 9 OR platform_id = 9 AND (order_transaction < 'Transaction' OR order_transaction
      = 'Transaction' AND (source_customer_id < '1772063')))
    type: custom
...

As you can see there are two filters:

• one for the filtering (order_date_time > '2023-02-15' and platform_id=9)
• one for partition windows (source_order_id < '447191446' OR source_order_id = '447191446' .... )
  effectively resulting in a partition with very few rows (sometimes even none at all).

I think this is caused by the filter not being in place when running the windowing-query for partition generation