Add the ability to define custom variables in YAML files which will be inherited

[0-Sony]

Is your feature request related to a problem? Please describe.
Currently, we split for each entity, module, a specific YAML file such as : magento/module-sales.yaml , magento/module-quote.yaml, magento/module-customer.yaml, adyen/module-payment.yaml

For these 3 entities, let says that I want to define a limit : 30 or change the interval day in a sql request in a where parameter

But as we work on different project with different needs, the limit value could be changed depending on the project.

tables:
   sales_order:
      where: 'updated_at > date_sub(now(), interval 30 day)'
      limit: 30
   quote:
      where: 'updated_at > date_sub(now(), interval 30 day)'
      limit: 30
  customer_entity:
      where: 'updated_at > date_sub(now(), interval 30 day)'
      limit: 30
      converters:
         email:
           converter: 'randomizeEmail'
           cache_key: 'customer_email'
           unique: true

Describe the solution you'd like

It would be nice if we could define custom variables at the top of the config.yaml file, and each yaml file (extends) would inherited
Also, the custom variables could be also defined in each yaml file, not necessarily in the config.yaml

variables:
    @my_variable : 30
    @cache_key_variable : 'customer_email'

tables:
   sales_order:
      where: 'updated_at > date_sub(now(), interval {{@my_variable}} day)'
      limit: @my_variable
   quote:
      where: 'updated_at > date_sub(now(), interval {{@my_variable}} day)'
      limit: @my_variable
  customer_entity:
      where: 'updated_at > date_sub(now(), interval {{@my_variable}} day)'
      limit: @my_variable
      converters:
         email:
            converter: 'randomizeEmail'
            cache_key: @cache_key_variable 
            unique: true

Regards

[guvra]

It's already possible but only for a few parameters.

In converter definition:

• condition

In table definition:

• where
• sort_order (i've never tested it though)
• skip_conversion_if

It doesn't work with limit because gdpr-dump currently requires an integer value for this parameter. And if we allowed string values, it still would not work with some database engines (e.g. mariadb does not allow sql variables in limit clause).

Example:

variables:
    max_days: 'select 60'
    my_var: 'select ...'

tables:
    table1:
        where: 'created_at > date_sub(now(), interval @max_days day)'

    table2:
        converters:
            col1:
                converter: 'randomizeText'
                condition: '{{col2}} = @my_var'