refactor: [migration] convert iframe chart into dashboard markdown component #10590

graceguo-supercat · 2020-08-12T16:30:39Z

SUMMARY

Dashboard v2 support native markdown component, and user can also create iframe inside markdown. So that it's not necessary to have iframe/markup/separator viz_type in slices (or add them into dashboard).

This PR will do:

Find iframe type chart that used in dashboard.
Convert iframe slices into dashboard Markdown component. Slices in markup and separator viz_type already converted into dashboard component when dashboard v2 rollout.
Delete iframe/markup/separator slices, and remove viz_type support (python and js)

TEST PLAN

Before migration, create a iframe chart and add into dashboard.
After migration, check the dashboard, iframe chart should be converted into Markdown component with a iframe.

ADDITIONAL INFORMATION

@mistercrunch @etr2460 @john-bodley @serenajiang

…mponent

etr2460

a couple initial comments, i need to reread the getInitialState changes again when i get more time though (I think it's mostly indentation changes, but confirmation would be helpful).

Overall, this looks awesome! I'm so happy to get rid of the iframe viz!

etr2460 · 2020-08-13T00:20:12Z

superset/viz.py

@@ -2040,25 +2040,6 @@ def get_data(self, df: pd.DataFrame) -> VizData:
        return d


-class IFrameViz(BaseViz):


Do we need to remove markdown components from viz.py too?

yes i think i need to.
Actually they were removed in another PR: #9997

etr2460 · 2020-08-13T00:20:35Z

superset/migrations/versions/f80a3b88324b_.py

@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one


Looks like this can be removed? I assume it was a placeholder for the migration I just ran

I create this migration yesterday, but today you merge a new migration script. so both of us are based from same root. this file is to resolve the sequence problem for alembic. otherwise i have to re-create the migration script.

I think you could copy paste your migration into a new migration script and then not need this blank file. Sorry for the migration conflict!

use superset db merge heads will generate migration script and fix the conflicts. no need to modify original file. There are many empty migration script in the codebase which was created for this purpose.

etr2460 · 2020-08-13T00:21:40Z

superset/migrations/versions/978245563a02_migrate_iframe_to_dash_markdown.py

+        ).delete(synchronize_session=False)
+
+    except Exception as ex:
+        logging.exception(f"dashboard {dashboard.id} has error: {ex}")


What state would we end in if only part of the iframes got deleted? I assume the charts would break because we're removing the backend api. Maybe it's not an issue though if the chance of an exception here is quite small

etr2460 · 2020-08-13T00:21:58Z

superset/migrations/versions/978245563a02_migrate_iframe_to_dash_markdown.py

+
+
+def downgrade():
+    pass


I agree this is nearly impossible to do a downgrade for. Maybe add a comment explaining why though?

etr2460 · 2020-08-13T00:22:17Z

superset-frontend/src/visualizations/presets/MainPreset.js

@@ -29,9 +29,7 @@ import ForceDirectedChartPlugin from '@superset-ui/legacy-plugin-chart-force-dir
 import HeatmapChartPlugin from '@superset-ui/legacy-plugin-chart-heatmap';
 import HistogramChartPlugin from '@superset-ui/legacy-plugin-chart-histogram';
 import HorizonChartPlugin from '@superset-ui/legacy-plugin-chart-horizon';
-import IframeChartPlugin from '@superset-ui/legacy-plugin-chart-iframe';


oooo, we can delete these packages next! So much red code!

correct. need another PR in superset-ui codebase (after everything looks good in superset).

etr2460 · 2020-08-13T00:25:32Z

superset/migrations/versions/978245563a02_migrate_iframe_to_dash_markdown.py

+            # find iframe chart position in metadata
+            # and replace it with markdown component
+            position_dict = json.loads(dashboard.position_json or "{}")
+            for key, value in position_dict.items():


nit: replace value with chart_position (or perhaps position_value as it's more clear

i like chart_position. fixed.

etr2460 · 2020-08-13T00:26:52Z

superset/migrations/versions/978245563a02_migrate_iframe_to_dash_markdown.py

+        dashboards = (
+            session.query(Dashboard).filter(Dashboard.id.in_(dashboard_ids)).all()
+        )
+        for i, dashboard in enumerate(dashboards):


we should probably add to UPDATING.md that this should be done during off hours as it could break people's charts while they're editing. They might save over the migrated dash and accidentally add an iframe chart back into it

graceguo-supercat · 2020-08-13T00:50:20Z

superset/migrations/versions/978245563a02_migrate_iframe_to_dash_markdown.py

+        ).all()
+        slices_ids = [slc.id for slc in slices_to_remove]
+
+        # remove dependencies first


i tested in dev-box use production data.
It seems I have to remove dependencies first, where those table use slice_id as foreign key.

graceguo-supercat · 2020-08-13T00:51:41Z

superset-frontend/src/dashboard/reducers/getInitialState.js

-        modified: slice.modified,
-        changed_on: new Date(slice.changed_on).getTime(),
-      };
+    const form_data = {


correct. the only change is just removed if viz_type change. I didn't touch any other code.

etr2460

apart from the empty migration that i think you can get rid of, this lgtm!

codecov-commenter · 2020-08-14T21:28:57Z

Codecov Report

Merging #10590 into master will decrease coverage by 4.53%.
The diff coverage is 1.69%.

@@            Coverage Diff             @@
##           master   #10590      +/-   ##
==========================================
- Coverage   64.37%   59.83%   -4.54%     
==========================================
  Files         775      777       +2     
  Lines       36555    36619      +64     
  Branches     3459     3456       -3     
==========================================
- Hits        23531    21910    -1621     
- Misses      12912    14519    +1607     
- Partials      112      190      +78

Flag	Coverage Δ
#cypress	`?`
#javascript	`60.47% <2.77%> (-0.01%)`	⬇️
#python	`59.45% <1.21%> (-0.49%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...frontend/src/dashboard/reducers/getInitialState.js	`0.00% <0.00%> (-68.61%)`	⬇️
...src/explore/components/controls/VizTypeControl.jsx	`76.92% <ø> (-1.93%)`	⬇️
superset-frontend/src/explore/constants.js	`100.00% <ø> (ø)`
...-frontend/src/visualizations/presets/MainPreset.js	`0.00% <ø> (-100.00%)`	⬇️
...ns/978245563a02_migrate_iframe_to_dash_markdown.py	`0.00% <0.00%> (ø)`
superset/migrations/versions/f80a3b88324b_.py	`0.00% <0.00%> (ø)`
superset/viz.py	`57.55% <ø> (-0.10%)`	⬇️
...rset-frontend/src/explore/components/SaveModal.jsx	`90.76% <100.00%> (-1.88%)`	⬇️
superset/tasks/slack_util.py	`89.47% <100.00%> (ø)`
superset-frontend/src/SqlLab/App.jsx	`0.00% <0.00%> (-100.00%)`	⬇️
... and 155 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acb00f5...9de1d87. Read the comment docs.

stevensuting · 2020-09-02T05:37:47Z

@graceguo-supercat
I am attempting to create a KPI card as shown above. In version 0.36 I could do this by using the markdown chart in HTML mode where I would use 3 iFrames. Each iFrame would point to a big_number_total chart

With 0.37 this is no long possible as I understand.

ERROR: Separator

ERROR:superset.views.base:'separator'
Traceback (most recent call last):
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/base.py", line 173, in wraps
    return f(self, *args, **kwargs)
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/utils/decorators.py", line 67, in wrapper
    check_perms(*args, **kwargs)
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/utils.py", line 426, in check_datasource_perms
    force=False,
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/utils.py", line 111, in get_viz
    viz_obj = viz.viz_types[viz_type](datasource, form_data=form_data, force=force)
KeyError: 'separator'

ERROR: markup

KeyError: 'markup'
ERROR:superset.views.base:'markup'
Traceback (most recent call last):
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/base.py", line 173, in wraps
    return f(self, *args, **kwargs)
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/utils/decorators.py", line 67, in wrapper
    check_perms(*args, **kwargs)
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/utils.py", line 426, in check_datasource_perms
    force=False,
  File "/home/ubuntu/demo-superset/lib/python3.6/site-packages/superset/views/utils.py", line 111, in get_viz
    viz_obj = viz.viz_types[viz_type](datasource, form_data=form_data, force=force)

graceguo-supercat · 2020-09-21T16:28:15Z

@stevensuting Superset will not support markup and separator charts any more. you should use divider and Markdown components in Dashboard.

…mponent (apache#10590) * refactor: [migration] convert iframe chart into dashboard markdown component * remove 3 viz_types * fix comments

pull-request-size bot added the size/L label Aug 12, 2020

graceguo-supercat force-pushed the gg-RemoveIframeVizInDash branch from 537b5b3 to baf5a09 Compare August 12, 2020 16:53

refactor: [migration] convert iframe chart into dashboard markdown co…

eb04aa4

…mponent

graceguo-supercat force-pushed the gg-RemoveIframeVizInDash branch 3 times, most recently from ca6b05f to 78a0816 Compare August 12, 2020 23:21

etr2460 reviewed Aug 13, 2020

View reviewed changes

remove 3 viz_types

a9cf02e

graceguo-supercat force-pushed the gg-RemoveIframeVizInDash branch from 78a0816 to a9cf02e Compare August 13, 2020 00:48

graceguo-supercat commented Aug 13, 2020

View reviewed changes

graceguo-supercat force-pushed the gg-RemoveIframeVizInDash branch from 896900b to 47e3cba Compare August 13, 2020 01:29

fix comments

9da43ac

graceguo-supercat force-pushed the gg-RemoveIframeVizInDash branch from 47e3cba to 9da43ac Compare August 13, 2020 01:34

etr2460 approved these changes Aug 14, 2020

View reviewed changes

Merge branch 'master' into gg-RemoveIframeVizInDash

9de1d87

graceguo-supercat force-pushed the gg-RemoveIframeVizInDash branch from bc59ac3 to 9de1d87 Compare August 14, 2020 21:21

graceguo-supercat merged commit ca9ca99 into apache:master Aug 16, 2020

graceguo-supercat mentioned this pull request Aug 20, 2020

chore: remove iframe and markup viz type support apache-superset/superset-ui#746

Merged

villebro mentioned this pull request Sep 2, 2020

feat(viz): add ECharts Timeseries chart #10752

Merged

6 tasks

graceguo-supercat mentioned this pull request Sep 21, 2020

feat: Stop pending queries when user close dashboard #10836

Closed

graceguo-supercat deleted the gg-RemoveIframeVizInDash branch November 11, 2020 23:17

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.38.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: [migration] convert iframe chart into dashboard markdown component #10590

refactor: [migration] convert iframe chart into dashboard markdown component #10590

graceguo-supercat commented Aug 12, 2020 •

edited

Loading

etr2460 left a comment

etr2460 Aug 13, 2020

graceguo-supercat Aug 13, 2020 •

edited

Loading

etr2460 Aug 13, 2020

graceguo-supercat Aug 13, 2020 •

edited

Loading

etr2460 Aug 14, 2020

graceguo-supercat Aug 14, 2020

etr2460 Aug 13, 2020

etr2460 Aug 13, 2020

etr2460 Aug 13, 2020

graceguo-supercat Aug 13, 2020 •

edited

Loading

etr2460 Aug 13, 2020

graceguo-supercat Aug 13, 2020

etr2460 Aug 13, 2020

graceguo-supercat Aug 13, 2020

graceguo-supercat Aug 13, 2020 •

edited

Loading

graceguo-supercat Aug 13, 2020

etr2460 left a comment

codecov-commenter commented Aug 14, 2020 •

edited

Loading

stevensuting commented Sep 2, 2020

graceguo-supercat commented Sep 21, 2020 •

edited

Loading

		@@ -2040,25 +2040,6 @@ def get_data(self, df: pd.DataFrame) -> VizData:
		return d


		class IFrameViz(BaseViz):

		@@ -0,0 +1,38 @@
		# Licensed to the Apache Software Foundation (ASF) under one

refactor: [migration] convert iframe chart into dashboard markdown component #10590

refactor: [migration] convert iframe chart into dashboard markdown component #10590

Conversation

graceguo-supercat commented Aug 12, 2020 • edited Loading

SUMMARY

TEST PLAN

ADDITIONAL INFORMATION

etr2460 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graceguo-supercat Aug 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graceguo-supercat Aug 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graceguo-supercat Aug 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graceguo-supercat Aug 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etr2460 left a comment

Choose a reason for hiding this comment

codecov-commenter commented Aug 14, 2020 • edited Loading

Codecov Report

stevensuting commented Sep 2, 2020

graceguo-supercat commented Sep 21, 2020 • edited Loading

graceguo-supercat commented Aug 12, 2020 •

edited

Loading

graceguo-supercat Aug 13, 2020 •

edited

Loading

graceguo-supercat Aug 13, 2020 •

edited

Loading

graceguo-supercat Aug 13, 2020 •

edited

Loading

graceguo-supercat Aug 13, 2020 •

edited

Loading

codecov-commenter commented Aug 14, 2020 •

edited

Loading

graceguo-supercat commented Sep 21, 2020 •

edited

Loading