fix(snowflake): opt-in denormalization of column names

[villebro]

SUMMARY

The PR #24471, which meant to harmonize column naming for Oracle-like databases like Snowflake, caused issues for deployments that were relying on the current behavior of normalizing column names for physical datasets. This PR changes adds a field normalize_columns to the Dataset/SQLA Table models. This defaults to False for new datasets, but for old datsets, this is set to True via a db migration to ensure we don't break existing datasets.

For existing datasets, "Normalize columns" is checked:

When checked, the behavior is consistent with how it was previously, i.e. physical datasets on Snowflake have normalized column names:

For new datasets, the checkbox is unchecked:

In this case, a physical dataset on Snowflake will denormalize the columns, usually showing them as UPPERCASE:

This means, that for new datasets, column names are no longer normalized, unless the flag is checked.

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[john-bodley]

What's the difference between NULL and FALSE?

[villebro]

@john-bodley they're essentially the same. Would you prefer I change it to just nullable=True without a default value, or just have the default value?

[john-bodley]

If we only need two states, then I would stick with TRUE and FALSE, i.e., non-nullable, unless there's a performance or storage cost for using FALSE rather than NULL—which likely will be the predominant value.

[michael-s-molina]

+1

[villebro]

It seems SQLAlchemy is slightly flaky when it comes to assigning default values with the NULL constraint in place. I dug around, and found that the is_sqllab_viz flag on the SqlaTable model is expected to work similarly, and there the migration also had to allow for nullable=True. So reverting back to that.

[eschutho]

Can we add some tests that create a dataset with/without this value to check for api backwards compatibility?

[villebro]

Good idea, I'll do that 👍

[michael-s-molina]

+1

[michael-s-molina]

LGTM. Thank you for the fix @villebro!

[eschutho]

Looks great!