Project 4

Title: Web Scraping Job Postings

Business Case:

You're working as a data scientist for a contracting firm that's rapidly expanding. Now that they have their most valuable employee (you!), they need to leverage data to win more contracts. Your firm offers technology and scientific solutions and wants to be competitive in the hiring market. Your principal has two main objectives:

Determine the industry factors that are most important in predicting the salary amounts for these data.

Determine the factors that distinguish job categories and titles from each other. For example, can required skills accurately predict job title?

To limit the scope, your principal has suggested that you focus on data-related job postings, e.g. data scientist, data analyst, research scientist, business intelligence, and any others you might think of. You may also want to decrease the scope by limiting your search to a single region.

Hint:

Aggregators like www.mycareersfuture.sg regularly pool job postings from a variety of markets and industries.

Goal:

Scrape your own data from a job aggregation tool like Indeed.com in order to collect the data to best answer these two questions.

Directions

In this project you will be leveraging a variety of skills. The first will be to use the web-scraping and/or API techniques you've learned to collect data on data jobs from Indeed.com or another aggregator. Once you have collected and cleaned the data, you will use it to answer the two questions described above.

QUESTION 1: Factors that impact salary

To predict salary you will be building either a classification or regression model, using features like the location, title, and summary of the job. If framing this as a regression problem, you will be estimating the listed salary amounts. You may instead choose to frame this as a classification problem, in which case you will create labels from these salaries (high vs. low salary, for example) according to thresholds (such as median salary).

You have learned a variety of new skills and models that may be useful for this problem:

NLP

Unsupervised learning and dimensionality reduction techniques (PCA, clustering)

Ensemble methods and decision tree models

SVM models

Whatever you decide to use, the most important thing is to justify your choices and interpret your results. Communication of your process is key. Note that most listings DO NOT come with salary information. You'll need to able to extrapolate or predict the expected salaries for these listings.

QUESTION 2: Factors that distinguish job category

Using the job postings you scraped for part 1 (or potentially new job postings from a second round of scraping), identify features in the data related to job postings that can distinguish job titles from each other. There are a variety of interesting ways you can frame the target variable, for example:

What components of a job posting distinguish data scientists from other data jobs?

What features are important for distinguishing junior vs. senior positions?

Do the requirements for titles vary significantly with industry (e.g. healthcare vs. government)?

You may end up making multiple classification models to tackle different questions. Be sure to clearly explain your hypotheses and framing, any feature engineering, and what your target variables are. The type of classification model you choose is up to you. Be sure to interpret your results and evaluate your models' performance.

BONUS PROBLEM

Your boss would rather tell a client incorrectly that they would get a lower salary job than tell a client incorrectly that they would get a high salary job. Adjust one of your models to ease his mind, and explain what it is doing and any tradeoffs. Plot the ROC curve.

Requirements

1. Scrape and prepare your own data.

2. Create and compare at least two models for each section. One of the two models should be a decision tree or ensemble model.

The other can be a classifier or regression of your choosing (e.g. Ridge, logistic regression, KNN, SVM, etc).

- Section 1: Job Salary Trends

- Section 2: Job Category Factors

3. Prepare a polished Jupyter Notebook with your analysis for a peer audience of data scientists.

- Make sure to clearly describe and label each section

- Comment on your code so that others could, in theory, replicate your work.

4. A brief writeup in an executive summary, written for a non-technical audience.

- Writeups should be at least 500-1000 words, defining any technical terms, explaining your approach, as well as any risks and limitations.

BONUS

5. Answer the salary discussion by using your model to explain the tradeoffs between detecting high vs low salary positions.

6. Convert your executive summary into a public blog post of at least 500 words, in which you document your approach in a tutorial for other aspiring data scientists. Link to this in your notebook.

Suggestions for Getting Started

1. Collect data from Indeed.com (or another aggregator) on data-related jobs to use in predicting salary trends for your analysis.

- Select and parse data from at least 1000 postings for jobs, potentially from multiple location searches.

2. Find out what factors most directly impact salaries (e.g. title, location, department, etc).

- Test, validate, and describe your models. What factors predict salary category? How do your models perform?

3. Discover which features have the greatest importance when determining a low vs. high paying job.

- Your Boss is interested in what overall features hold the greatest significance.

- HR is interested in which SKILLS and KEY WORDS hold the greatest significance.

4. Author an executive summary that details the highlights of your analysis for a non-technical audience.

5. If tackling the bonus question, try framing the salary problem as a classification problem detecting low vs. high salary positions.