Scrap Job postings from Dice.com for educational purposes. ;)
- chromedriver : Download the driver corresponding to your chrome version.
- Python Module : Selenium
python crawl.py --job 'data scientist' --location 44000
For help,
python crawl.py --h
- Visit Dice.com.
- Input the search query for provided
jobandlocation. - Scrap the results (links to the job postings in the search result) from the first page.
- Iterate over the links, go to each job posting landing page and scrap the job requirement detailed paragraph and save it to the
detailed_htmlfile.