Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The IC policies spider has been stale since April 6th 2024. When testing this locally I found that the crawler was receiving 403 responses when accessing the pages with all the links. After some experimentation I found that visiting these sites directly resolved the issue.
I also took this chance to refactor this spider to use the new DocItemFields class and also to gather a new document on the website title "IC Legal Reference Book 2020".
Result of Crawler Run on Dev
Example Metadata