Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[schema] Updating the base column schema #5452

Merged

Conversation

john-bodley
Copy link
Member

@john-bodley john-bodley commented Jul 20, 2018

We've noticed a number of anomalies in our database caused by ill-defined forms and/or table schema definitions. This PR resolves a number of issues related to the columns and table_columns table including:

  • Adds the missing legacy constraint to the table_columns table which was added in PR Adding YAML Import-Export for Datasources to CLI #3978 but didn't contain the corresponding migration. Note this requires reducing the size of the column_name column (which has already been resized in the model) for viability purposes.
  • Ensures that the column_name column is non-NULL.
  • Deletes any orphaned records, i.e., those which are not associated with a datasource/table.

The uniqueness constraint was added to the columns table in this migration.

Note this migration will fail if the table_columns.column_name column is NULL. One must manually fix these records as programmatically trying to remedy these invalid records is difficult as the intent is unclear and the columns may function (from a query standpoint) if the underlying column expression is defined. The following query determines which records are problematic:

SELECT 
    *
FROM 
    table_columns
WHERE 
    column_name IS NULL 

Additionally this migration may truncate the table_columns.column_name column from 256 to 255 characters. Note certain legacy installations have the 256 character size whereas newer installations have the correct 255 character size. The following query determines which records are problematic:

SELECT
    *
FROM 
    table_columns
WHERE
    CHAR_LENGTH(column_name) > 255

Finally this migration will fail if the table_columns table is corrupt in terms of the uniqueness constraint. One must manually consolidate duplicate or non-valid records given there's no way of programmatically removing invalid records. The following query determines whether there are duplicates:

SELECT 
    table_id, 
    column_name,
    COUNT(1)
FROM 
    table_columns
GROUP BY 
    1, 2
HAVING 
    COUNT(1) > 1
ORDER BY 
    COUNT(1) DESC

Note this PR is gated by #5445 and #7084 which ensure that empty strings associated with form-data wont persist in the database and is necessary for ensuring that the relevant entries are non-NULL.

to: @fabianmenges @graceguo-supercat @michellethomas @mistercrunch @timifasubaa

@codecov-io
Copy link

codecov-io commented Jul 20, 2018

Codecov Report

Merging #5452 into master will not change coverage.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #5452   +/-   ##
=======================================
  Coverage   64.91%   64.91%           
=======================================
  Files         428      428           
  Lines       20951    20951           
  Branches     2330     2330           
=======================================
  Hits        13601    13601           
  Misses       7226     7226           
  Partials      124      124
Impacted Files Coverage Δ
superset/connectors/druid/models.py 82.49% <100%> (ø) ⬆️
superset/connectors/sqla/models.py 81.86% <100%> (ø) ⬆️
superset/connectors/base/models.py 89.89% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 48b2af4...a6f2222. Read the comment docs.

@john-bodley john-bodley force-pushed the john-bodley-update-base-columns branch 3 times, most recently from 3cf6f28 to aee4e49 Compare July 21, 2018 05:58
Copy link
Contributor

@fabianmenges fabianmenges left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@john-bodley john-bodley force-pushed the john-bodley-update-base-columns branch from aee4e49 to da99b38 Compare August 15, 2018 15:41
@john-bodley john-bodley added the risk:db-migration PRs that require a DB migration label Mar 22, 2019
@john-bodley john-bodley force-pushed the john-bodley-update-base-columns branch 2 times, most recently from 96d8e64 to 1e30075 Compare March 22, 2019 22:25
@john-bodley john-bodley force-pushed the john-bodley-update-base-columns branch 4 times, most recently from e0ba755 to 10cddf7 Compare March 25, 2019 18:05
@john-bodley john-bodley force-pushed the john-bodley-update-base-columns branch 3 times, most recently from 40124f8 to 0c231b0 Compare April 23, 2019 22:18
@john-bodley john-bodley force-pushed the john-bodley-update-base-columns branch 3 times, most recently from 50ad96d to c289002 Compare April 24, 2019 00:19
@john-bodley john-bodley force-pushed the john-bodley-update-base-columns branch from c289002 to a6f2222 Compare April 24, 2019 00:38
@john-bodley john-bodley merged commit 1388452 into apache:master Apr 24, 2019
xtinec pushed a commit that referenced this pull request Apr 29, 2019
* filter out all nan series (#7313)

* improve not rich tooltip (#7345)

* Create issue_label_bot.yaml (#7341)

* fix: do not save colors without a color scheme (#7347)

* [wtforms] Strip leading/trailing whitespace (#7084)

* [schema] Updating the datasources schema (#5451)

* limit tables/views returned if schema is not provided (#7358)

* limit tables/views returned if schema is not provided

* fix typo

* improve code performance

* handle the case when table name or view name does not present a schema

* Add type anno (#7342)

* Updated local dev instructions to include missing step

* First pass at type annotations

* [schema] Updating the base column schema (#5452)

* Update 937d04c16b64_update_datasources.py (#7361)

* Feature flag for client cache (#7348)

* Feature flag for client cache

* Fix integration test

* Revert "Fix integration test"

This reverts commit 58434ab.

* Feature flag for client cache

* Fix integration tests

* Add feature flag to config.py

* Add another feature check

* Fix more integration tests

* Fix raw HTML in SliceAdder (#7338)

* remove backendSync.json (#7331)

* [bubbles] issue when using duplicated metrics (#7087)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359)

* SUPERSET-8: Update text in docs copyright footer (#7360)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04

* SUPERSET-8: Extra text in docs copyright footer

* [schema] Adding commits and removing unnecessary foreign-key definitions (#7371)

*  Store last selected dashboard in sessionStorage (#7181)

* Store last selected dashboard in sessionStorage

* Fix tests

* [schema] Updating the base metric schema (#5453)

* Fix NoneType bug & fill the test recipients with original recipients if empty (#7365)
xtinec pushed a commit that referenced this pull request May 1, 2019
* Merge lastest from master into lyft-release-sp8 (#7405)

* filter out all nan series (#7313)

* improve not rich tooltip (#7345)

* Create issue_label_bot.yaml (#7341)

* fix: do not save colors without a color scheme (#7347)

* [wtforms] Strip leading/trailing whitespace (#7084)

* [schema] Updating the datasources schema (#5451)

* limit tables/views returned if schema is not provided (#7358)

* limit tables/views returned if schema is not provided

* fix typo

* improve code performance

* handle the case when table name or view name does not present a schema

* Add type anno (#7342)

* Updated local dev instructions to include missing step

* First pass at type annotations

* [schema] Updating the base column schema (#5452)

* Update 937d04c16b64_update_datasources.py (#7361)

* Feature flag for client cache (#7348)

* Feature flag for client cache

* Fix integration test

* Revert "Fix integration test"

This reverts commit 58434ab.

* Feature flag for client cache

* Fix integration tests

* Add feature flag to config.py

* Add another feature check

* Fix more integration tests

* Fix raw HTML in SliceAdder (#7338)

* remove backendSync.json (#7331)

* [bubbles] issue when using duplicated metrics (#7087)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359)

* SUPERSET-8: Update text in docs copyright footer (#7360)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04

* SUPERSET-8: Extra text in docs copyright footer

* [schema] Adding commits and removing unnecessary foreign-key definitions (#7371)

*  Store last selected dashboard in sessionStorage (#7181)

* Store last selected dashboard in sessionStorage

* Fix tests

* [schema] Updating the base metric schema (#5453)

* Fix NoneType bug & fill the test recipients with original recipients if empty (#7365)

* feat: see Presto row and array data types (#7391)

* feat: see Presto row and array data types

* fix: address PR comments

* fix: lint and build issues

* fix: add types

* add stronger type hints where possible

* fix: lint issues and add select_star func in Hive

* add missing pkg init

* fix: build issues

* fix: pylint issues

* fix: use logging instead of print
betodealmeida pushed a commit that referenced this pull request May 3, 2019
* Merge lastest from master into lyft-release-sp8 (#7405)

* filter out all nan series (#7313)

* improve not rich tooltip (#7345)

* Create issue_label_bot.yaml (#7341)

* fix: do not save colors without a color scheme (#7347)

* [wtforms] Strip leading/trailing whitespace (#7084)

* [schema] Updating the datasources schema (#5451)

* limit tables/views returned if schema is not provided (#7358)

* limit tables/views returned if schema is not provided

* fix typo

* improve code performance

* handle the case when table name or view name does not present a schema

* Add type anno (#7342)

* Updated local dev instructions to include missing step

* First pass at type annotations

* [schema] Updating the base column schema (#5452)

* Update 937d04c16b64_update_datasources.py (#7361)

* Feature flag for client cache (#7348)

* Feature flag for client cache

* Fix integration test

* Revert "Fix integration test"

This reverts commit 58434ab.

* Feature flag for client cache

* Fix integration tests

* Add feature flag to config.py

* Add another feature check

* Fix more integration tests

* Fix raw HTML in SliceAdder (#7338)

* remove backendSync.json (#7331)

* [bubbles] issue when using duplicated metrics (#7087)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359)

* SUPERSET-8: Update text in docs copyright footer (#7360)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04

* SUPERSET-8: Extra text in docs copyright footer

* [schema] Adding commits and removing unnecessary foreign-key definitions (#7371)

*  Store last selected dashboard in sessionStorage (#7181)

* Store last selected dashboard in sessionStorage

* Fix tests

* [schema] Updating the base metric schema (#5453)

* Fix NoneType bug & fill the test recipients with original recipients if empty (#7365)

* feat: see Presto row and array data types (#7391)

* feat: see Presto row and array data types

* fix: address PR comments

* fix: lint and build issues

* fix: add types

* Incorporate feedback from initial PR (prematurely merged to lyft-release-sp8) (#7415)

* add stronger type hints where possible

* fix: lint issues and add select_star func in Hive

* add missing pkg init

* fix: build issues

* fix: pylint issues

* fix: use logging instead of print

* feat: view presto row objects in data grid

* fix: address feedback

* fix: spacing
betodealmeida pushed a commit that referenced this pull request May 3, 2019
* Merge lastest from master into lyft-release-sp8 (#7405)

* filter out all nan series (#7313)

* improve not rich tooltip (#7345)

* Create issue_label_bot.yaml (#7341)

* fix: do not save colors without a color scheme (#7347)

* [wtforms] Strip leading/trailing whitespace (#7084)

* [schema] Updating the datasources schema (#5451)

* limit tables/views returned if schema is not provided (#7358)

* limit tables/views returned if schema is not provided

* fix typo

* improve code performance

* handle the case when table name or view name does not present a schema

* Add type anno (#7342)

* Updated local dev instructions to include missing step

* First pass at type annotations

* [schema] Updating the base column schema (#5452)

* Update 937d04c16b64_update_datasources.py (#7361)

* Feature flag for client cache (#7348)

* Feature flag for client cache

* Fix integration test

* Revert "Fix integration test"

This reverts commit 58434ab.

* Feature flag for client cache

* Fix integration tests

* Add feature flag to config.py

* Add another feature check

* Fix more integration tests

* Fix raw HTML in SliceAdder (#7338)

* remove backendSync.json (#7331)

* [bubbles] issue when using duplicated metrics (#7087)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359)

* SUPERSET-8: Update text in docs copyright footer (#7360)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04

* SUPERSET-8: Extra text in docs copyright footer

* [schema] Adding commits and removing unnecessary foreign-key definitions (#7371)

*  Store last selected dashboard in sessionStorage (#7181)

* Store last selected dashboard in sessionStorage

* Fix tests

* [schema] Updating the base metric schema (#5453)

* Fix NoneType bug & fill the test recipients with original recipients if empty (#7365)

* feat: see Presto row and array data types (#7391)

* feat: see Presto row and array data types

* fix: address PR comments

* fix: lint and build issues

* fix: add types

* Incorporate feedback from initial PR (prematurely merged to lyft-release-sp8) (#7415)

* add stronger type hints where possible

* fix: lint issues and add select_star func in Hive

* add missing pkg init

* fix: build issues

* fix: pylint issues

* fix: use logging instead of print

* feat: view presto row objects in data grid

* fix: address feedback

* fix: spacing

* Workaround for no results returned (#7442)

* feat: view presto row objects in data grid (#7436)

* feat: view presto row objects in data grid

* fix: address feedback

* fix: spacing

* feat: Scheduling queries from SQL Lab (#7416)

* Lightweight pipelines POC

* Add docs

* Minor fixes

* Remove Lyft URL

* Use enum

* Minor fix

* Fix unit tests

* Mark props as required
xtinec pushed a commit that referenced this pull request May 10, 2019
* filter out all nan series (#7313)

* improve not rich tooltip (#7345)

* Create issue_label_bot.yaml (#7341)

* fix: do not save colors without a color scheme (#7347)

* [wtforms] Strip leading/trailing whitespace (#7084)

* [schema] Updating the datasources schema (#5451)

* limit tables/views returned if schema is not provided (#7358)

* limit tables/views returned if schema is not provided

* fix typo

* improve code performance

* handle the case when table name or view name does not present a schema

* Add type anno (#7342)

* Updated local dev instructions to include missing step

* First pass at type annotations

* [schema] Updating the base column schema (#5452)

* Update 937d04c16b64_update_datasources.py (#7361)

* Feature flag for client cache (#7348)

* Feature flag for client cache

* Fix integration test

* Revert "Fix integration test"

This reverts commit 58434ab.

* Feature flag for client cache

* Fix integration tests

* Add feature flag to config.py

* Add another feature check

* Fix more integration tests

* Fix raw HTML in SliceAdder (#7338)

* remove backendSync.json (#7331)

* [bubbles] issue when using duplicated metrics (#7087)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359)

* SUPERSET-8: Update text in docs copyright footer (#7360)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04

* SUPERSET-8: Extra text in docs copyright footer

* [schema] Adding commits and removing unnecessary foreign-key definitions (#7371)

*  Store last selected dashboard in sessionStorage (#7181)

* Store last selected dashboard in sessionStorage

* Fix tests

* [schema] Updating the base metric schema (#5453)

* Fix NoneType bug & fill the test recipients with original recipients if empty (#7365)

* Added living goods as among the users of Superset (#7407)

* Added living goods as among the users of Superset

Living Goods is a non profit organisation with operation in africa and the middle east. We work in community health use data heavily on day to day. Superset is our platform of choice for dashboards.

* Update README.md

* [dashboard] allow user re-order top-level tabs (#7390)

* [SQL Lab] Increase timeout threshold for offline check (#7411)

* Bump FAB to 2.0.0 (#7323)

* Bump FAB to 2.0.0

* [tests] whitelist SecurityApi login and refresh endpoints

* [style] Fix, C812 missing trailing commas

* [security] Remove SUPERSET_UPDATE_PERMS flag

Registering sources needs to be performed after the views are
initialized on UPDATE_PERMS=False configuration

* [docs] New, FAB_UPDATE_PERMS and flask fab cli

* [docs] Fix, db upgrade needs to come first, create-admin needs a db

* [cli] New, superset init bootstraps all permissions for FAB and Superset

* [style] Fix, flakes

* [annotations] Improves UX on annotation validation, start_dttm, end_dttm (#7326)

* Setting renderTrigger on label_colors (#7410)

* Refactor out controlUtils.js module + unit tests (#7350)

* [WiP]refactor out a controlUtils.js file

* unit tests

* add missing license

* Addressing comments

* feature: see Presto row and array data types (#7413)

* Merge lastest from master into lyft-release-sp8 (#7405)

* filter out all nan series (#7313)

* improve not rich tooltip (#7345)

* Create issue_label_bot.yaml (#7341)

* fix: do not save colors without a color scheme (#7347)

* [wtforms] Strip leading/trailing whitespace (#7084)

* [schema] Updating the datasources schema (#5451)

* limit tables/views returned if schema is not provided (#7358)

* limit tables/views returned if schema is not provided

* fix typo

* improve code performance

* handle the case when table name or view name does not present a schema

* Add type anno (#7342)

* Updated local dev instructions to include missing step

* First pass at type annotations

* [schema] Updating the base column schema (#5452)

* Update 937d04c16b64_update_datasources.py (#7361)

* Feature flag for client cache (#7348)

* Feature flag for client cache

* Fix integration test

* Revert "Fix integration test"

This reverts commit 58434ab.

* Feature flag for client cache

* Fix integration tests

* Add feature flag to config.py

* Add another feature check

* Fix more integration tests

* Fix raw HTML in SliceAdder (#7338)

* remove backendSync.json (#7331)

* [bubbles] issue when using duplicated metrics (#7087)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359)

* SUPERSET-8: Update text in docs copyright footer (#7360)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04

* SUPERSET-8: Extra text in docs copyright footer

* [schema] Adding commits and removing unnecessary foreign-key definitions (#7371)

*  Store last selected dashboard in sessionStorage (#7181)

* Store last selected dashboard in sessionStorage

* Fix tests

* [schema] Updating the base metric schema (#5453)

* Fix NoneType bug & fill the test recipients with original recipients if empty (#7365)

* feat: see Presto row and array data types (#7391)

* feat: see Presto row and array data types

* fix: address PR comments

* fix: lint and build issues

* fix: add types

* add stronger type hints where possible

* fix: lint issues and add select_star func in Hive

* add missing pkg init

* fix: build issues

* fix: pylint issues

* fix: use logging instead of print

* Removed --console-log and superset runserver (#7421)

* Fixes dashboard export button missing download and #7353 (#7427)

* Added additional German translations to string file (#6604)

* Added additional German translations to string file

Updates to German translation files as per directions

* Removed messages.json

* [fix] Fixing SQL parsing issue (#7374)

* add chinese translate (#7402)

* Quick fix to address deadlock issue (#7434)

* feat: view presto row objects in data grid (#7445)

* Merge lastest from master into lyft-release-sp8 (#7405)

* filter out all nan series (#7313)

* improve not rich tooltip (#7345)

* Create issue_label_bot.yaml (#7341)

* fix: do not save colors without a color scheme (#7347)

* [wtforms] Strip leading/trailing whitespace (#7084)

* [schema] Updating the datasources schema (#5451)

* limit tables/views returned if schema is not provided (#7358)

* limit tables/views returned if schema is not provided

* fix typo

* improve code performance

* handle the case when table name or view name does not present a schema

* Add type anno (#7342)

* Updated local dev instructions to include missing step

* First pass at type annotations

* [schema] Updating the base column schema (#5452)

* Update 937d04c16b64_update_datasources.py (#7361)

* Feature flag for client cache (#7348)

* Feature flag for client cache

* Fix integration test

* Revert "Fix integration test"

This reverts commit 58434ab.

* Feature flag for client cache

* Fix integration tests

* Add feature flag to config.py

* Add another feature check

* Fix more integration tests

* Fix raw HTML in SliceAdder (#7338)

* remove backendSync.json (#7331)

* [bubbles] issue when using duplicated metrics (#7087)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359)

* SUPERSET-8: Update text in docs copyright footer (#7360)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04

* SUPERSET-8: Extra text in docs copyright footer

* [schema] Adding commits and removing unnecessary foreign-key definitions (#7371)

*  Store last selected dashboard in sessionStorage (#7181)

* Store last selected dashboard in sessionStorage

* Fix tests

* [schema] Updating the base metric schema (#5453)

* Fix NoneType bug & fill the test recipients with original recipients if empty (#7365)

* feat: see Presto row and array data types (#7391)

* feat: see Presto row and array data types

* fix: address PR comments

* fix: lint and build issues

* fix: add types

* Incorporate feedback from initial PR (prematurely merged to lyft-release-sp8) (#7415)

* add stronger type hints where possible

* fix: lint issues and add select_star func in Hive

* add missing pkg init

* fix: build issues

* fix: pylint issues

* fix: use logging instead of print

* feat: view presto row objects in data grid

* fix: address feedback

* fix: spacing

* feat: Scheduling queries from SQL Lab (#7416) (#7446)

* Merge lastest from master into lyft-release-sp8 (#7405)

* filter out all nan series (#7313)

* improve not rich tooltip (#7345)

* Create issue_label_bot.yaml (#7341)

* fix: do not save colors without a color scheme (#7347)

* [wtforms] Strip leading/trailing whitespace (#7084)

* [schema] Updating the datasources schema (#5451)

* limit tables/views returned if schema is not provided (#7358)

* limit tables/views returned if schema is not provided

* fix typo

* improve code performance

* handle the case when table name or view name does not present a schema

* Add type anno (#7342)

* Updated local dev instructions to include missing step

* First pass at type annotations

* [schema] Updating the base column schema (#5452)

* Update 937d04c16b64_update_datasources.py (#7361)

* Feature flag for client cache (#7348)

* Feature flag for client cache

* Fix integration test

* Revert "Fix integration test"

This reverts commit 58434ab.

* Feature flag for client cache

* Fix integration tests

* Add feature flag to config.py

* Add another feature check

* Fix more integration tests

* Fix raw HTML in SliceAdder (#7338)

* remove backendSync.json (#7331)

* [bubbles] issue when using duplicated metrics (#7087)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359)

* SUPERSET-8: Update text in docs copyright footer (#7360)

* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04

* SUPERSET-8: Extra text in docs copyright footer

* [schema] Adding commits and removing unnecessary foreign-key definitions (#7371)

*  Store last selected dashboard in sessionStorage (#7181)

* Store last selected dashboard in sessionStorage

* Fix tests

* [schema] Updating the base metric schema (#5453)

* Fix NoneType bug & fill the test recipients with original recipients if empty (#7365)

* feat: see Presto row and array data types (#7391)

* feat: see Presto row and array data types

* fix: address PR comments

* fix: lint and build issues

* fix: add types

* Incorporate feedback from initial PR (prematurely merged to lyft-release-sp8) (#7415)

* add stronger type hints where possible

* fix: lint issues and add select_star func in Hive

* add missing pkg init

* fix: build issues

* fix: pylint issues

* fix: use logging instead of print

* feat: view presto row objects in data grid

* fix: address feedback

* fix: spacing

* Workaround for no results returned (#7442)

* feat: view presto row objects in data grid (#7436)

* feat: view presto row objects in data grid

* fix: address feedback

* fix: spacing

* feat: Scheduling queries from SQL Lab (#7416)

* Lightweight pipelines POC

* Add docs

* Minor fixes

* Remove Lyft URL

* Use enum

* Minor fix

* Fix unit tests

* Mark props as required

* feat: Add `validate_sql_json` endpoint for checking that a given sql query is valid for the chosen database (#7422) (#7462)

merge from lyft-release-sp8 to master

* Adds missing metric sum__SP_RUR_TOTL (#7452)

* Late import for optional lib pyhive (#7471)

* Late import for optional lib pyhive

* fix

* fix: calendar heatmap examples (#7375)

Fixing a set of examples that trip on ValueError vs TypeError

* bugfix: Improve support for special characters in schema and table names (#7297)

* Bugfix to SQL Lab to support tables and schemas with characters that require quoting

* Remove debugging prints

* Add uri encoding to secondary tables call

* Quote schema names for presto

* Quote selected_schema on Snowflake, MySQL and Hive

* Remove redundant parens

* Add python unit tests

* Add js unit test

* Fix flake8 linting error

* [dashboard] After update filter, trigger new queries when charts are visible (#7233)

* trigger query when chart is visible

* add integration test

* fix: alter sql columns to long text #7463 (#7476)

Merge lyft-release-sp8@7bfe7bc to master

* Refactor ConsoleLog (#7428)

* Revised Chinese translation (#7464)

* add chinese translate

* edit chinese translation

* druid connector: avoid using 'dimensions' for scan queries (#7377)

After the following PyDruid change (contained in version 0.5.2)
the Superset Histogram charts rendered with Druid data are
broken:

druid-io/pydruid@0a59a70

Bump the pydruid requirements accordingly in setup.py

Issue: #7368
@mrshu
Copy link
Contributor

mrshu commented Oct 28, 2019

I came here to quickly mention that the following sentence added to UPDATING.md has turned out to be a bit of an understatement for us, as we've updated from v29 to v34 over this past weekend:

Depending on the integrity of the data, manual intervention may be required

Only later, when presented with exceptions of the following sort

sqlalchemy.exc.IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'week-21' for key 'uq_table_columns_column_name'")
[SQL: ALTER TABLE table_columns ADD CONSTRAINT uq_table_columns_column_name UNIQUE (column_name, table_id)]
(Background on this error at: http://sqlalche.me/e/gkpj)

did we find out that the actual problem was that we had schema columns and so called calculated columns whose lowercase representation turned out to be the same (i.e. week as a schema column and WEEK as a calculated column). Since the constraint added in this PR only considers their lowercased forms, many queries finished with integrity errors. The fix was quite straightforward (ensure there are no overlapping lowercase names for schema and calculated columns) but was quite hard to find.

I was therefore wondering if something like this would be worth adding to UPDATING.md, so that we may prevent someone else's surprise when their new version of Superset does not work after upgrade.

Thanks!

@john-bodley
Copy link
Member Author

john-bodley commented Oct 29, 2019 via email

@mrshu
Copy link
Contributor

mrshu commented Oct 29, 2019

Thanks a lot @john-bodley -- I really appreciate that!

@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.34.0 labels Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels risk:db-migration PRs that require a DB migration size/L 🚢 0.34.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants