feat: implement create view as functionality in Sqllab #9794

bkyryliuk · 2020-05-13T00:50:14Z

SUMMARY

Introduces create view as button to the sql lab

Main use case - simplify the user data viz flow and keep it more maintainable.

This feature is controlled on the database level in the configurations.

TEST PLAN

[x] unit tests
[x] dropbox staging
[x] dropbox production

ADDITIONAL INFORMATION

Changes UI
Requires DB Migration.
Confirm DB Migration upgrade and downgrade tested.

More screenshots:

codecov-io · 2020-05-13T00:59:36Z

Codecov Report

Merging #9794 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #9794      +/-   ##
==========================================
+ Coverage   70.79%   70.80%   +0.01%     
==========================================
  Files         587      183     -404     
  Lines       30435    17834   -12601     
  Branches     3152        0    -3152     
==========================================
- Hits        21545    12627    -8918     
+ Misses       8776     5207    -3569     
+ Partials      114        0     -114

Flag	Coverage Δ
#cypress	`?`
#javascript	`?`
#python	`70.80% <100.00%> (-0.15%)`	⬇️

Impacted Files	Coverage Δ
superset/sql_lab.py	`78.07% <ø> (ø)`
superset/models/sql_lab.py	`92.17% <100.00%> (+0.20%)`	⬆️
superset/sql_parse.py	`99.38% <100.00%> (+0.01%)`	⬆️
superset/views/core.py	`74.90% <100.00%> (+0.01%)`	⬆️
superset/views/database/mixins.py	`58.92% <0.00%> (-21.43%)`	⬇️
superset/utils/cache.py	`45.83% <0.00%> (-20.84%)`	⬇️
superset/views/database/api.py	`83.90% <0.00%> (-3.45%)`	⬇️
superset/db_engine_specs/postgres.py	`97.29% <0.00%> (-2.71%)`	⬇️
superset/models/core.py	`83.77% <0.00%> (-1.48%)`	⬇️
superset/jinja_context.py	`84.69% <0.00%> (-1.03%)`	⬇️
... and 404 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65d185f...74af9e6. Read the comment docs.

codecov-commenter · 2020-05-19T02:06:20Z

Codecov Report

Merging #9794 into master will decrease coverage by 11.00%.
The diff coverage is 25.00%.

@@             Coverage Diff             @@
##           master    #9794       +/-   ##
===========================================
- Coverage   70.56%   59.55%   -11.01%     
===========================================
  Files         590      400      -190     
  Lines       31150    12951    -18199     
  Branches     3164     3195       +31     
===========================================
- Hits        21981     7713    -14268     
+ Misses       9060     5057     -4003     
- Partials      109      181       +72

Flag	Coverage Δ
#cypress	`?`
#javascript	`59.55% <25.00%> (+0.01%)`	⬆️
#python	`?`

Impacted Files	Coverage Δ
...erset-frontend/src/SqlLab/components/ResultSet.jsx	`69.82% <0.00%> (-1.86%)`	⬇️
...erset-frontend/src/SqlLab/components/SqlEditor.jsx	`50.90% <25.00%> (-4.22%)`	⬇️
superset-frontend/src/SqlLab/actions/sqlLab.js	`60.56% <100.00%> (-6.25%)`	⬇️
superset-frontend/src/SqlLab/App.jsx	`0.00% <0.00%> (-100.00%)`	⬇️
superset-frontend/src/explore/App.jsx	`0.00% <0.00%> (-100.00%)`	⬇️
superset-frontend/src/SqlLab/index.jsx	`0.00% <0.00%> (-100.00%)`	⬇️
superset-frontend/src/dashboard/App.jsx	`0.00% <0.00%> (-100.00%)`	⬇️
superset-frontend/src/explore/index.jsx	`0.00% <0.00%> (-100.00%)`	⬇️
superset-frontend/src/dashboard/index.jsx	`0.00% <0.00%> (-100.00%)`	⬇️
superset-frontend/src/setup/setupColors.js	`0.00% <0.00%> (-100.00%)`	⬇️
... and 331 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 02fee35...5237b8c. Read the comment docs.

john-bodley

The Python LGTM. Note I would suggest adding a line item to UPDATING.md noting that downtime may be required for the migration given that the locking of the query table may be problematic due to the frequency of use.

john-bodley · 2020-06-17T17:24:41Z

tests/celery_tests.py


-    def test_run_sync_query_cta(self):
+    @parameterized.expand([CtaMethod.TABLE, CtaMethod.VIEW])


I like this pattern and frequently use it with pytest via @pytest.mark.parametrize decorator. Unrelated to this change I wonder if we should consider using pytest as opposed to unittest.

+1000
I would love to!
Pytest supports unittests frameworks, so it would be easy to bootstrap the migration. Sounds like a really good topic for the next community meetup.

Noted. I added this to the agenda.

This pattern could really help clean up many of our tests 👍

Steejay · 2020-06-17T21:41:17Z

Here is what i think might be a better experience as a short term solution:
• i think theres an opportunity to consolidate the CTAS and CVAS buttons > can we consider one button "Save Table" ?
• Once "Save Table" is selected a modal will pop up that allows the user to input a table name and select between two options: CVAS or CTAS (radio button)
• (see 'Save Query' modal)

My concern is that there are a lot of save type buttons (save query, CTAS, CVAS) in a zone that has limited real estate especially in smaller windows. Consolidating similar functions has the opportunity to empower users to make decisions quicker by minimizing options.

If strong tech constraints exist that dont allow for the suggestions here. I think that this current layout is still worth considering.

mistercrunch · 2020-06-18T16:40:54Z

Quick note that SIP-34 does not take CTAS into account. It's probably been missed since it's behind a feature flag (a database configuration flag really...).

Personally my incline would be - for now - to expand the current SAVE button with a btn-dropdown that goes Save Query (the default), CREATE TABLE AS, and CREATE VIEW AS. If picking option 2 or 3 it needs to prompt for a name.

@Steejay what do you think? Also it would be good to find the longer term solution and incorporate to the Figma designs

Steejay · 2020-06-18T16:53:55Z

@mistercrunch if im not mistaken SIP-34 does include CTAS and CVAS but different nomenclature is used. Designs use the terms "Dynamic" and "Static" instead. lmk if im not understanding this correctly..

Save dropdown is clicked > dropdown reveals Save Query, Save Dataset, and Add to Dashboard > Save Dataset is selected > reveals this modal

bkyryliuk · 2020-06-18T19:20:26Z

Great feedback, thank you!
@mistercrunch @Steejay I agree that we should not accumulate the design technical debt and be moving towards SIP-34.
We have tried save query flow with the dropboxers and it was not working well as it is extra couple clicks from running a simple query / being able to visualize it.

As a short term solution I've added a database level control button that enables / disables the CVAS not to pollute the space. As a medium / long term solution - we will start looking with design & users on the cleaner solutions on this front.

villebro

LGTM, but a few minor nits. To avoid being blocked by design efforts, perhaps we could put this behind a feature flag for now so as to not put more strain on UX before the design implications have been properly vetted?

superset/sql_parse.py

superset/models/sql_lab.py

villebro · 2020-06-22T09:27:47Z

tests/celery_tests.py


-    def test_run_sync_query_cta(self):
+    @parameterized.expand([CtaMethod.TABLE, CtaMethod.VIEW])


This pattern could really help clean up many of our tests 👍

bkyryliuk · 2020-06-22T18:48:54Z

LGTM, but a few minor nits. To avoid being blocked by design efforts, perhaps we could put this behind a feature flag for now so as to not put more strain on UX before the design implications have been properly vetted?

@villebro done, not a feature flag -> but DB level control that allows to add CVAS button to the view.
From the previous discussions as I understand we agree that feature is useful, the remaining challenge is design & feature representation in UI. That's smth that can be tackled later and I don't think have a good skillset to implement it in this PR

Implement create view as button in sqllab Make CVAS configurable

Implement create view as button in sqllab Make CVAS configurable Co-authored-by: bogdan kyryliuk <[email protected]> (cherry picked from commit 3db76c6)

ktmud · 2020-07-02T23:42:52Z

@bkyryliuk can we add a loading state to the "Explore" button while it's creating the data source?

Otherwise clicking on it multiple times will create multiple data sources:

(The backend should probably also check for duplications, too?)

Implement create view as button in sqllab Make CVAS configurable Co-authored-by: bogdan kyryliuk <[email protected]>

pull-request-size bot added the size/L label May 13, 2020

bkyryliuk marked this pull request as draft May 13, 2020 00:50

bkyryliuk changed the title ~~WIP Implement create view as functionality~~ [SQL Lab] Implement create view as functionality May 13, 2020

bkyryliuk changed the title ~~[SQL Lab] Implement create view as functionality~~ Feature - Implement create view as functionality May 13, 2020

bkyryliuk changed the title ~~Feature - Implement create view as functionality~~ Feature - Implement create view as functionality in Sqlab May 13, 2020

bkyryliuk changed the title ~~Feature - Implement create view as functionality in Sqlab~~ Feature - Implement create view as functionality in Sqllab May 13, 2020

bkyryliuk force-pushed the bogdan/cvas branch from 368d0cb to 74af9e6 Compare May 13, 2020 00:52

bkyryliuk force-pushed the bogdan/cvas branch 4 times, most recently from 230e094 to 0523ac4 Compare May 13, 2020 23:52

bkyryliuk changed the title ~~Feature - Implement create view as functionality in Sqllab~~ feat: implement create view as functionality in Sqllab May 13, 2020

bkyryliuk force-pushed the bogdan/cvas branch from 0523ac4 to f367985 Compare May 14, 2020 00:07

bkyryliuk closed this May 14, 2020

bkyryliuk reopened this May 14, 2020

bkyryliuk marked this pull request as ready for review May 19, 2020 01:57

bkyryliuk force-pushed the bogdan/cvas branch from f367985 to aca60a3 Compare May 19, 2020 01:59

bkyryliuk force-pushed the bogdan/cvas branch 11 times, most recently from c6ac7c1 to 81755a5 Compare May 19, 2020 21:43

bkyryliuk requested review from john-bodley and villebro June 17, 2020 17:09

bkyryliuk force-pushed the bogdan/cvas branch from 6abe93e to 2a34626 Compare June 17, 2020 17:16

john-bodley reviewed Jun 17, 2020

View reviewed changes

bkyryliuk force-pushed the bogdan/cvas branch 2 times, most recently from 3feacf2 to b055f02 Compare June 17, 2020 20:57

bkyryliuk requested a review from mistercrunch June 18, 2020 16:33

bkyryliuk force-pushed the bogdan/cvas branch 2 times, most recently from 5157c53 to afbe8c7 Compare June 18, 2020 19:08

bkyryliuk added the v0.37 label Jun 19, 2020

bkyryliuk force-pushed the bogdan/cvas branch from afbe8c7 to 1caf0e7 Compare June 20, 2020 00:58

villebro requested changes Jun 22, 2020

View reviewed changes

bkyryliuk force-pushed the bogdan/cvas branch from 1caf0e7 to 5237b8c Compare June 22, 2020 18:54

Implement create view as functionality

578399c

Implement create view as button in sqllab Make CVAS configurable

bkyryliuk force-pushed the bogdan/cvas branch from 5237b8c to 578399c Compare June 22, 2020 19:10

villebro approved these changes Jun 24, 2020

View reviewed changes

bkyryliuk merged commit 3db76c6 into apache:master Jun 24, 2020

bkyryliuk deleted the bogdan/cvas branch June 24, 2020 16:50

auxten pushed a commit to auxten/incubator-superset that referenced this pull request Nov 20, 2020

Implement create view as functionality (apache#9794)

fa05fe9

Implement create view as button in sqllab Make CVAS configurable Co-authored-by: bogdan kyryliuk <[email protected]>

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.37.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement create view as functionality in Sqllab #9794

feat: implement create view as functionality in Sqllab #9794

bkyryliuk commented May 13, 2020 •

edited

Loading

codecov-io commented May 13, 2020

codecov-commenter commented May 19, 2020 •

edited

Loading

john-bodley left a comment

john-bodley Jun 17, 2020 •

edited

Loading

bkyryliuk Jun 17, 2020

john-bodley Jun 17, 2020

villebro Jun 22, 2020

Steejay commented Jun 17, 2020 •

edited

Loading

mistercrunch commented Jun 18, 2020

Steejay commented Jun 18, 2020

bkyryliuk commented Jun 18, 2020

villebro left a comment

villebro Jun 22, 2020

bkyryliuk commented Jun 22, 2020 •

edited

Loading

ktmud commented Jul 2, 2020 •

edited

Loading


		def test_run_sync_query_cta(self):
		@parameterized.expand([CtaMethod.TABLE, CtaMethod.VIEW])

feat: implement create view as functionality in Sqllab #9794

feat: implement create view as functionality in Sqllab #9794

Conversation

bkyryliuk commented May 13, 2020 • edited Loading

SUMMARY

TEST PLAN

ADDITIONAL INFORMATION

codecov-io commented May 13, 2020

Codecov Report

codecov-commenter commented May 19, 2020 • edited Loading

Codecov Report

john-bodley left a comment

Choose a reason for hiding this comment

john-bodley Jun 17, 2020 • edited Loading

Choose a reason for hiding this comment

bkyryliuk Jun 17, 2020

Choose a reason for hiding this comment

john-bodley Jun 17, 2020

Choose a reason for hiding this comment

villebro Jun 22, 2020

Choose a reason for hiding this comment

Steejay commented Jun 17, 2020 • edited Loading

mistercrunch commented Jun 18, 2020

Steejay commented Jun 18, 2020

bkyryliuk commented Jun 18, 2020

villebro left a comment

Choose a reason for hiding this comment

villebro Jun 22, 2020

Choose a reason for hiding this comment

bkyryliuk commented Jun 22, 2020 • edited Loading

ktmud commented Jul 2, 2020 • edited Loading

bkyryliuk commented May 13, 2020 •

edited

Loading

codecov-commenter commented May 19, 2020 •

edited

Loading

john-bodley Jun 17, 2020 •

edited

Loading

Steejay commented Jun 17, 2020 •

edited

Loading

bkyryliuk commented Jun 22, 2020 •

edited

Loading

ktmud commented Jul 2, 2020 •

edited

Loading