Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Config] Cache static resources #8370

Merged
merged 1 commit into from
Oct 17, 2019
Merged

Conversation

etr2460
Copy link
Member

@etr2460 etr2460 commented Oct 10, 2019

CATEGORY

Choose one

  • Bug Fix
  • Enhancement (new features, refinement)
  • Refactor
  • Add tests
  • Build / Development Environment
  • Documentation

SUMMARY

Since static resources have hashed file names, we shouldn't need to even hit the webserver for them. Right now, we're hitting the server and using the etag to cache it, but this should prevent a trip to the server altogether.

It should make superset a bit more stable while deploying because these files won't be found on half the webservers while in the middle of deploying

This change also adds some flexibility to the http header configuration, allowing you to only set headers if they haven't already been set by superset. This means that we can cache static content with the SEND_FILE_MAX_AGE_DEFAULT var, but disallow caching on everything else with DEFAULT_HTTP_HEADERS = {"Cache-Control": "no-cache"}.

TEST PLAN

CI and a test deploy to see new cache control headers set

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Requires DB Migration.
  • Confirm DB Migration upgrade and downgrade tested.
  • Introduces new feature or API
  • Removes existing feature or API

REVIEWERS

@mistercrunch @dpgaspar @john-bodley

@etr2460 etr2460 added .config and removed size/XS labels Oct 10, 2019
@etr2460 etr2460 changed the title [Config] Cache static resources [WIP] Cache static resources Oct 10, 2019
@codecov-io
Copy link

codecov-io commented Oct 10, 2019

Codecov Report

Merging #8370 into master will decrease coverage by 7.09%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #8370      +/-   ##
==========================================
- Coverage   67.65%   60.55%    -7.1%     
==========================================
  Files         448      332     -116     
  Lines       22498    10254   -12244     
  Branches     2364     2364              
==========================================
- Hits        15220     6209    -9011     
+ Misses       7140     3907    -3233     
  Partials      138      138
Impacted Files Coverage Δ
superset/connectors/druid/__init__.py
superset/views/datasource.py
superset/models/sql_lab.py
superset/connectors/druid/models.py
superset/utils/import_datasource.py
superset/views/database/forms.py
superset/tasks/schedules.py
superset/db_engine_specs/bigquery.py
superset/examples/tabbed_dashboard.py
superset/db_engine_specs/athena.py
... and 104 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c422b49...450fde7. Read the comment docs.

@etr2460 etr2460 force-pushed the etr2460--cache-static-resources branch from cc94271 to 681b522 Compare October 11, 2019 18:20
@etr2460 etr2460 changed the title [WIP] Cache static resources [Config] Cache static resources Oct 11, 2019
@etr2460 etr2460 force-pushed the etr2460--cache-static-resources branch from 681b522 to ba4e385 Compare October 11, 2019 18:32
superset/views/core.py Outdated Show resolved Hide resolved
@etr2460 etr2460 force-pushed the etr2460--cache-static-resources branch from ba4e385 to 48c379a Compare October 14, 2019 16:20
# HTTP_HEADERS is deprecated, this provides backwards compatibility
override_http_headers.update(config.get("HTTP_HEADERS", {}))

for k, v in override_http_headers.items():
Copy link
Member

@john-bodley john-bodley Oct 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be defined as

response.headers.update(override_http_headers)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

@@ -3127,8 +3127,17 @@ class CssTemplateAsyncModelView(CssTemplateModelView):
@app.after_request
def apply_http_headers(response):
"""Applies the configuration's http headers to all responses"""
for k, v in config.get("HTTP_HEADERS").items():

override_http_headers = config.get("OVERRIDE_HTTP_HEADERS", {})
Copy link
Member

@john-bodley john-bodley Oct 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that HTTP_HEADERS and OVERRIDE_HTTP_HEADERS are explicitly defined in the config I prefer not using get(...) as this can be implied that the fields are optional. In Python 3 line #3131 and #3133 could be reduced to:

override_http_headers = {**config["OVERRIDE_HTTP_HEADERS"], **config["HTTP_HEADERS"]}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HTTP_HEADERS will no longer be explicitly defined, but i'll update the other one

@mistercrunch
Copy link
Member

Didn't dig deep but this seems deterministic and looks like it can be done in module scope or cached somehow so that for every response we do a simple affectation (?)

@etr2460 etr2460 force-pushed the etr2460--cache-static-resources branch from 48c379a to a06e3fc Compare October 16, 2019 22:15
@etr2460
Copy link
Member Author

etr2460 commented Oct 16, 2019

The joining of the dict probably could be cached, but not the application of default headers since different responses will have different headers applied. I don't think this is worth that optimization since these dicts are going to be quite small

@etr2460 etr2460 force-pushed the etr2460--cache-static-resources branch from a06e3fc to 450fde7 Compare October 16, 2019 22:57
@etr2460 etr2460 merged commit d3406e0 into master Oct 17, 2019
@etr2460 etr2460 deleted the etr2460--cache-static-resources branch October 17, 2019 16:09
graceguo-supercat pushed a commit that referenced this pull request Nov 13, 2019
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.35.0 labels Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/S 🚢 0.35.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants