Improve ondemand data collection for npm and maven #487
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JonoYang
reviewed
Jul 12, 2024
| @@ -44,7 +44,7 @@ | |||
| ) | |||
|
|
|||
|
|
|||
| def add_package_to_scan_queue(package, pipelines=DEFAULT_PIPELINES, reindex_uri=False, priority=0): | |||
| def add_package_to_scan_queue(package, pipelines=DEFAULT_PIPELINES, reindex_uri=False, priority=100): | |||
There was a problem hiding this comment.
@pombredanne When we add something to the scan queue, do we want it to be placed on top of the stack of packages to be scanned, or should that behavior only be done on requests from the collect or similar endpoints?
There was a problem hiding this comment.
yes, top of the stack when asking for an exact version IMHO
There was a problem hiding this comment.
ack, tracking this in #502
pombredanne
changed the title
Signed-off-by: Jono Yang <[email protected]>
Signed-off-by: Jono Yang <[email protected]>
Signed-off-by: Jono Yang <[email protected]>
Reference: #486 Signed-off-by: Philippe Ombredanne <[email protected]>
Reference: #488 Signed-off-by: Philippe Ombredanne <[email protected]>
Signed-off-by: Philippe Ombredanne <[email protected]>
Signed-off-by: Philippe Ombredanne <[email protected]>
Signed-off-by: Jono Yang <[email protected]>
Reference: #490 Signed-off-by: Philippe Ombredanne <[email protected]>
Reference: #490 Signed-off-by: Philippe Ombredanne <[email protected]>
* Always a package_content type, defaulting to BINARY if not provided * Ensure we always return a package even if there was no enhancement Reference: #490 Signed-off-by: Philippe Ombredanne <[email protected]>
Reference: #496 Signed-off-by: Philippe Ombredanne <[email protected]>
Signed-off-by: Philippe Ombredanne <[email protected]>
* Optionally override package data with new, non-empty manifest data * Expose new experimental API endpoint extra to recollect manifest data with a rescan at collect/reindex_metadata with this override capability * Enable this for Maven only for now Reference: #496 Reference: #494 Signed-off-by: Philippe Ombredanne <[email protected]>
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
JonoYang
force-pushed
the
increase-scannableuri-priority
branch
from
12bba60
to
fc4a1cc
Compare
This was referenced Jul 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR provides various improvements wrt. on-demand data collection, and in particular: