feat: Prevent OOM in NoticeContainer and speed up hasValidationErrors #895
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Store up to 10 M validation notices. This is a measure to prevent OOM in the rare case when each row in a large file (such as stop_times.txt or shapes.txt) produces a notice. Since this case is rare, we just introduce a total limit on the amount of notices instead of counting amount of notices of each type. Note that system errors are not limited since we don't expect to have a lot of them. Now we have a boolean field to flag if there are validation errors. This way hasValidationErrors() now takes O(1) instead of O(n).
lionel-nj
approved these changes
May 6, 2021
|
Thanks for review! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Store up to 10 M validation notices. This is a measure to prevent OOM
in the rare case when each row in a large file (such as stop_times.txt
or shapes.txt) produces a notice. Since this case is rare, we just
introduce a total limit on the amount of notices instead of counting
amount of notices of each type.
Note that system errors are not limited since we don't expect to have
a lot of them.
Now we have a boolean field to flag if there are validation errors. This
way hasValidationErrors() now takes O(1) instead of O(n).
We also documented the methods and return unmodifiableList from getters.
We cannot use ImmutableList since the notice lists are always growing.