Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The army_g1_pubs spider was getting marked as stale by our crawler monitor. It's last successful run was on March 28th 2024. When running this crawler locally I received the following error:
This indicated that the web site had changed so I updated the crawler to scrape the new website as well as start using the DocItemFields class while maintaining most of the logic from the previous iteration. Also confirmed that the hashes match and only new documents are getting pulled by seeing the "In Previous Hashes" value on dev below.
Result of Crawler Run on Dev
Example Metadata