Patch stig spider

[matthew-kersting]

Description

The stig_pubs spider is reported as stale by our monitoring system.

![WARNING] - MONITORING: CRAWLERS OVERDUE
nato_stanag was last run May 27 2022
dha_pubs was last run Apr 20 2024
ic_policies was last run Apr 06 2024
jcs_pubs was last run Apr 19 2024
test_crawler was last run Apr 20 2022
stig_pubs was last run Mar 12 2024

When running locally I was getting the error below:
2024-04-30 13:10:06 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://public.cyber.mil/stigs/downloads/> (failed 3 times): User timeout caused connection failure: Getting https://public.cyber.mil/stigs/downloads/ took longer than 3.5 seconds.

I updated the download timeout and confirmed that all pdfs are scraped both locally and on dev.

Result of Crawler Run on Dev

[STATS] Crawlers run:
 stig_pubs
        Required CAC: 0
        In Previous Hashes: 45
        Item Scraped Count: 47
        Elapsed Time (sec): 6.293812
        Close Reason: finished

Example Metadata

{
    "doc_name": "STIG V0R0 2016-04-21 DoD CIO Memo - Use of Wearable Devices DoD Accredited Spaces with FAQ",
    "doc_title": "2016-04-21 DoD CIO Memo - Use of Wearable Devices DoD Accredited Spaces with FAQ",
    "doc_num": "V0R0",
    "doc_type": "STIG",
    "display_doc_type": "Memo",
    "publication_date": "2018-11-30T00:00:00",
    "cac_login_required": false,
    "crawler_used": "stig_pubs",
    "downloadable_items": [
        {
            "doc_type": "pdf",
            "download_url": "https://dl.dod.cyber.mil/wp-content/uploads/stigs/pdf/2016-04-21_DoD_CIO_Memo-Use_of_Wearable_Devices_DoD_Accredited_Spaces_with_FAQ.pdf",
            "compression_type": null
        }
    ],
    "source_page_url": "https://public.cyber.mil/stigs/downloads/",
    "source_fqdn": "public.cyber.mil",
    "download_url": "https://dl.dod.cyber.mil/wp-content/uploads/stigs/pdf/2016-04-21_DoD_CIO_Memo-Use_of_Wearable_Devices_DoD_Accredited_Spaces_with_FAQ.pdf",
    "version_hash_raw_data": {
        "doc_name": "STIG V0R0 2016-04-21 DoD CIO Memo - Use of Wearable Devices DoD Accredited Spaces with FAQ",
        "doc_num": "V0R0",
        "publication_date": "2018-11-30T00:00:00",
        "download_url": "https://dl.dod.cyber.mil/wp-content/uploads/stigs/pdf/2016-04-21_DoD_CIO_Memo-Use_of_Wearable_Devices_DoD_Accredited_Spaces_with_FAQ.pdf",
        "display_title": "STIG V0R0: 2016-04-21 DoD CIO Memo - Use of Wearable Devices DoD Accredited Spaces with FAQ"
    },
    "version_hash": "5096feacc4b9c77fbe91a08b3de0506490edb0d1e6744c3aa30a807e655d39f6",
    "display_org": "Security Technical Implementation Guides",
    "data_source": "Security Technical Implementation Guides",
    "source_title": "Unlisted Source",
    "display_source": "Security Technical Implementation Guides - Unlisted Source",
    "display_title": "STIG V0R0: 2016-04-21 DoD CIO Memo - Use of Wearable Devices DoD Accredited Spaces with FAQ",
    "file_ext": "pdf",
    "is_revoked": false,
    "access_timestamp": "2024-04-30T13:28:56"
}

[matthew-kersting]

Wanted to confirm that the metadata is the same before and after refactoring. On these two examples you can see that the only difference is the accessed timestamp