-
Goal of project is to match people up with jobs
-
Deploy project using a live interactive webapp for prospective job seekers to use
-
Jobs were scraped from Linkedin using Selenium and Beautiful Soup
Using selenium and beautiful soup, two popular python packages for web scraping and web automation, a datasets was scraped from Linkedin.com (n = 3150). The search looked at jobs that resulted from looking up 'Data Science'. This search term yielded the largest number of jobs from the search terms tested.
Linkedin job features |
---|
Company name |
Job title |
Location |
Number of applicants |
Full job description |
Six states were selected as states that are hubs for data science jobs.
States |
---|
Colorado |
California |
Florida |
New York |
Utah |
Washington |
Job_Title | Company | Location | Number_of_Applicants | Description | |
---|---|---|---|---|---|
0 | Data Acquisition Developer | ABB | Broomfield, CO, US | Be among the first 25 applicants | Join ABB and work in a team that is dedicated ... |
1 | DATA SCIENTIST MANAGER | Pilgrim's | Greeley, CO, US | Be among the first 25 applicants | Pilgrim’s is the second largest chicken produc... |
2 | Data Scientist – Logistics Optimization Algori... | ShaleApps | Denver, CO, US | Be among the first 25 applicants | Location: Denver, CO – Relocation Offered. Rem... |
To scrape Linkedin, selenium needed to be used. Beautiful soup is not enough for linkedin because one needs to click on the jobs in order to get the job information. Beautiful soup could have been used to scrape basic information such as job title and company name but because the site doesn't redirect users to the company's website, the full job description was available. In order to get this more detailed information, one needs to be able to interact with the site like a human user. Selenium's strong point is just this; that it allows one to write code that will interact with the website like a human user. Two main features of selenium that allow this are the ability to click on buttons and the ability to type in text to text fields.
In order to scrape Linkedin, the script first would navigate to the search result page and clicked on the 'see more jobs' button a given number of times allowing one to have access to more than just one page. The script then clicked on each job and passed the HTML to a beautiful soup object that then was used to parse for the information of interest.
Companies with the largest number of postings for data science jobs on Linkedin.com
-
Using a TFIDF matrix, job descriptions were vectorized and fed into a K-Means algorithm to attempt to group jobs together based on descriptions alone
-
Wanted to group jobs together and recommend using clusters, but selecting the K value using the Silhouette Score elbow plot method was difficult
- For any interpretable number of K values, the SS score was extremely low
- Using three clusters, we can see a very diverse group of jobs
- Although the each observation - and cluster center for that matter - is located within 10,000 dimensional space, the clusters were visualized by transforming the data into two dimensional space using PCA
- Once again, all of these jobs were found by searching for the keyword 'Data Science'
- Interesting that mobile developer jobs were included
- In two dimensional euclidean space, we can see that the descriptions of the mobile dev jobs were quite different than the data science and big data job descriptions
- Jobs were assessed on several skills
python, r, spark, spss, sql, pandas, numpy, cloud, docker, statistic, java, scala, marketing, sas, stata, excel, tableau
-
Vector for each job was created to represent whether or not the job included each of the skills
- Ex: [1, 1, 1, 1, 0, 1, 1, 0 ,0, 0, 0, 0, 0, 0, 0, 0, 1]
-
User vectors were created by asking users which of the skills they had experience with
- User vector was compared to all job vectors using Jaccard Similarity
- Most similar job was returned
- If multiple jobs were equally similar, the first job in the list was returned
- Docker, Cloud skills only --> Senior Manager of Technology
- Python, R only --> Director - Data Science, Modeling
- Python, Marketing only --> Marketing Data Scientist
- Recommender was deployed on a streamlit app with the help of Docker
- Hosted on Amazon AWS EC2 instance
Here is the homepage of the app:
Users would then select the Data Science skills they have like in this example
Once the user completed their selections, they would click the recommendation button to get a job
- Include jobs outside of data science
- Explore other text embedding methods
- Retain semantic representations of descriptions (e.g. word-to-vec or BERT)
- Allow user to request location from recommender
- Develop strategy to isolate 'important' parts of descriptions
- Most descriptions include fluffy details about company
- Develop more exhaustive list of skills