Patch IC Spider

[matthew-kersting]

Description

The IC policies spider has been stale since April 6th 2024. When testing this locally I found that the crawler was receiving 403 responses when accessing the pages with all the links. After some experimentation I found that visiting these sites directly resolved the issue.

I also took this chance to refactor this spider to use the new DocItemFields class and also to gather a new document on the website title "IC Legal Reference Book 2020".

Result of Crawler Run on Dev

[STATS] Crawlers run:
 ic_policies
        Required CAC: 0
        In Previous Hashes: 101
        Item Scraped Count: 105
        Elapsed Time (sec): 43.793115
        Close Reason: finished

Example Metadata

{
    "doc_name": "ICD 107",
    "doc_title": "Civil Liberties, Privacy, and Transparency",
    "doc_num": "107",
    "doc_type": "ICD",
    "display_doc_type": "Directive",
    "publication_date": null,
    "cac_login_required": false,
    "crawler_used": "ic_policies",
    "downloadable_items": [
        {
            "doc_type": "pdf",
            "download_url": "https://www.dni.gov/files/documents/ICD/ICD-107.pdf",
            "compression_type": null
        }
    ],
    "source_page_url": "https://www.dni.gov/index.php/what-we-do/ic-related-menus/ic-related-links/intelligence-community-directives",
    "source_fqdn": "www.dni.gov",
    "download_url": "https://www.dni.gov/files/documents/ICD/ICD-107.pdf",
    "version_hash_raw_data": {
        "doc_name": "ICD 107",
        "doc_num": "107",
        "publication_date": null,
        "download_url": "https://www.dni.gov/files/documents/ICD/ICD-107.pdf",
        "display_title": "ICD 107: Civil Liberties, Privacy, and Transparency"
    },
    "version_hash": "0ee9043e47d457973fb1989f03e8a628697beeea92e17b775e94b93782bdd574",
    "display_org": "Intelligence Community",
    "data_source": "Office of Director of National Intelligence",
    "source_title": "Unlisted Source",
    "display_source": "Office of Director of National Intelligence - Unlisted Source",
    "display_title": "ICD 107: Civil Liberties, Privacy, and Transparency",
    "file_ext": "pdf",
    "is_revoked": false,
    "access_timestamp": "2024-05-15T18:41:36"
}

[Antsega]

nice docker and crawler fix!