feat(kernel): experimental runtime package cache #3724
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds an experimental (hence opt-in) feature that caches the contents of loaded libraries in a directory that persists between executions, in order to spare the time it takes to extract the tarballs. When this feature is enabled, packages present in the cache will be used as-is (i.e: they are not checked for tampering) instead of being extracted from the tarball. The cache is keyed on: - The hash of the tarball - The name of the library - The version of the library Objects in the cache will expire if they are not used for 30 days, and are subsequently removed from disk (this avoids a cache growing extremely large over time). In order to enable the feature, the following environment variables are used: - `JSII_RUNTIME_PACKAGE_CACHE` must be set to `enabled` in order for the package cache to be active at all; - `JSII_RUNTIME_PACKAGE_CACHE_ROOT` can be used to change which directory is used as a cache root. It defaults to: * On MacOS: `$HOME/Library/Caches/com.amazonaws.jsii` * On Linux: `$HOME/.cache/aws/jsii/package-cache` * On Windows: `%LOCALAPPDATA%\AWS\jsii\package-cache` * On other platforms: `$TMP/aws-jsii-package-cache` - `JSII_RUNTIME_PACKAGE_CACHE_TTL` can be used to change the default time entries will remain in cache before expiring if they are not used. This defaults to 30 days, and the value is expressed in days. Set to `0` to immediately expire all the cache's content. When troubleshooting load performance, it is possible to obtain timing data for some critical parts of the library load process within the jsii kernel by setting `JSII_DEBUG_TIMING` environment variable. Related to #3389
Naumel
reviewed
Aug 30, 2022
Naumel
reviewed
Aug 30, 2022
Naumel
reviewed
Aug 30, 2022
Naumel
reviewed
Aug 30, 2022
Naumel
reviewed
Aug 30, 2022
|
@Mergifyio update |
✅ Branch has been successfully updated |
Naumel
approved these changes
Aug 30, 2022
A missing condition in the code generator could have resulted in attempting to read a property on undefined, resulting in a crash.
A missing condition in the code generator could have resulted in attempting to read a property on undefined, resulting in a crash.
|
@Mergifyio update |
❌ Base branch update has failedmerge conflict between base and head |
|
Thank you for contributing! ❤️ I will now look into making sure the PR is up-to-date, then proceed to try and merge it! |
|
Merging (with squash)... |
# Conflicts: # yarn.lock
|
Merging (with squash)... |
|
Merging (with squash)... |
RomainMuller
added a commit
that referenced
this pull request
Sep 26, 2022
Windows does not allow renaming files that are currently open, or directories that contain open files. When antivirus software is used, files are open for analysis by the A/V software, making it impossible to rename files too quickly after they've been created. This was already reported in #992 and addressed, however the issue was re-introduced in #3724, in which tarballs were extracted into temporary directories that were then renamed. Changed the code back to a form taht extracts files directly into their final place instead of staging via a temporary space, and added comments warning maintainers about the specific issue being solved here, so that hopefully the problem does not get re-introduced again in the future. Fixes #3751
mergify bot
pushed a commit
that referenced
this pull request
Sep 26, 2022
Windows does not allow renaming files that are currently open, or directories that contain open files. When antivirus software is used, files are open for analysis by the A/V software, making it impossible to rename files too quickly after they've been created. This was already reported in #992 and addressed, however the issue was re-introduced in #3724, in which tarballs were extracted into temporary directories that were then renamed. Changed the code back to a form taht extracts files directly into their final place instead of staging via a temporary space, and added comments warning maintainers about the specific issue being solved here, so that hopefully the problem does not get re-introduced again in the future. Fixes #3751 --- By submitting this pull request, I confirm that my contribution is made under the terms of the [Apache 2.0 license]. [Apache 2.0 license]: https://www.apache.org/licenses/LICENSE-2.0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds an experimental (hence opt-in) feature that caches the contents of
loaded libraries in a directory that persists between executions, in
order to spare the time it takes to extract the tarballs.
When this feature is enabled, packages present in the cache will be used
as-is (i.e: they are not checked for tampering) instead of being
extracted from the tarball. The cache is keyed on:
Objects in the cache will expire if they are not used for 30 days, and
are subsequently removed from disk (this avoids a cache growing
extremely large over time).
In order to enable the feature, the following environment variables are
used:
JSII_RUNTIME_PACKAGE_CACHEmust be set toenabledin order for thepackage cache to be active at all;
JSII_RUNTIME_PACKAGE_CACHE_ROOTcan be used to change whichdirectory is used as a cache root. It defaults to:
$HOME/Library/Caches/com.amazonaws.jsii$HOME/.cache/aws/jsii/package-cache%LOCALAPPDATA%\AWS\jsii\package-cache$TMP/aws-jsii-package-cacheJSII_RUNTIME_PACKAGE_CACHE_TTLcan be used to change the defaulttime entries will remain in cache before expiring if they are not
used. This defaults to 30 days, and the value is expressed in days.
Set to
0to immediately expire all the cache's content.When troubleshooting load performance, it is possible to obtain timing
data for some critical parts of the library load process within the jsii
kernel by setting
JSII_DEBUG_TIMINGenvironment variable.Related to #3389
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.