feat!: Ibis Upgrade to 5.1.0

[nehanene15]

Closes #317 (Ibis upgrade)
Closes #839 (BQ BIGNUMERIC is now expressed as decimal(76, 38))
Closes #767 (Spanner timestamp exception)
Closes #833 (MSSQL support for sum on datetime columns)
Closes #832 (MSSQL support for aggregating varchar/char columns)
Closes #719 (MSSQL/BQ column validation support for float data types)
Closes #821 (DB2 custom query index out of range bug fix)
Closes #814 (Python 3.11 support)

Breaking Change: Python 3.7 is not supported anymore. This PR also updates most of the required Python packages.

Summary: Updates Ibis to the newest 5.1.0 release. It touches a lot of files, but mainly the backend third_party connection implementations.

Notes:

• Changed type of labels from array of structs to array of array since the Ibis upgrade interprets tuples as arrays. I.e Previously, a label would be [(“name”, “Neha”)] and now it will be [[“name”, “Neha”]] (This doesn’t affect BQ schema or outputs to the BQ result table)
• Due to above, I had to update tests/unit/test_combiner.py to remove labels from test cases since pandas assert frame equal can’t determine equality of an array
• Snowflake support is in Alpha (has not been tested)
• The Ibis upgrade will continue to support MySQL versions < 8.0 as I overrode the new functionality that required CTEs in third_party/ibis/ibis_mysql

User-focused changes:

• For schema validation, previously we expressed non-nullable fields as string[non-nullable]. Post upgrade it will be expressed as !string

[nj1973]

/gcbrun

[nj1973]

LGTM, awesome job.

[piyushsarraf]

Hi
I was testing the ibis-upgrade branch and saw some errors wrt to BQ and RedShift.
I was doing a BQ - BQ row validation and got a error of 2 packages found for backend 'bigquery'

And for RedShift I was doing a row validation and the error seems to be with version of SQL Alchemy.

[nj1973]

@piyushsarraf , I also saw the "2 packages found for backend 'bigquery'" message. It was because my virtual environment was not clean. I fixed it by removing the venv, creating it fresh and then pip installing DVT.

[nehanene15]

@piyushsarraf
As of now the only blocker to merge the PR is the Redshift unrecognized configuration parameter. It could be because I upgraded the psycopg2-binary package from 2.9.3 to 2.9.6, or due to SQLAlchemy conflicts similar to this - https://stackoverflow.com/questions/76532906/unrecognized-configuration-parameter-standard-conforming-strings-while-queryin

[Raniksingh]

@nehanene15 I just verified the below version which works for Redshift.
psycopg2-binary ==2.9.6 and SQLAlchemy ==1.4.39

[sundar-mudupalli-work]

Hey Neha,
I was excited that Teradata query worked with DVT, and looked into getting generate-partitions to work on Teradata. Good news - I have it working and tested on oracle, sqlserver, bigquery, mysql, postgres, hive and teradata. Test functions for these have been written as well. Not so Good news - All this work is on the ibis-upgrade branch - it is all in my local repository. It is only a few files (4-5), so I can hack it into any branch. Or I need someone to tell me how to merge it into the ibis-upgrade branch without messing other things. Now realize that I should have created another branch off ibis-upgrade.

Thanks.

Sundar Mudupalli