Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hotfix/samm spider #183

Merged
merged 19 commits into from
Aug 9, 2023
Merged

Hotfix/samm spider #183

merged 19 commits into from
Aug 9, 2023

Conversation

Antsega
Copy link
Contributor

@Antsega Antsega commented Aug 7, 2023

First class in crawler successfully scrapes pdfs from: https://samm.dsca.mil/listing/chapters
Second class in crawler successfully scrapes pdfs from: https://samm.dsca.mil/policy-memoranda/PolicyMemoList-All , with specific exclusions

Tested locally using shell

 $env:PYTHONPATH = "$(Get-Location)"
 $CRAWLER_DATA_ROOT = "./test-samm"
New-Item -ItemType Directory -Force -Path $CRAWLER_DATA_ROOT| Out-Null
$prevManifestFile = Join-Path $CRAWLER_DATA_ROOT "pre-manifest.json"
New-Item -ItemType File -Force -Path $prevManifestFile | Out-Null
$sammSpider = "dataPipelines/gc_scrapy/gc_scrapy/unfinished/samm_spider.py"    
$outputFile = Join-Path $CRAWLER_DATA_ROOT "output.json"
scrapy runspider $sammSpider `
-a download_output_dir=$CRAWLER_DATA_ROOT `
-a previous_manifest_location=$prevManifestFile `
-o $outputFile

Ripped the @static method from stigg_spider; open to refactoring this method to match desired outcome for naming convention

@takao8 takao8 self-requested a review August 9, 2023 16:20
@takao8 takao8 merged commit 3ae68b1 into dev Aug 9, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants