Arctic Spider

Author: Alex Rosa
Date: 2-5-18

About

This goal of this project is to scrape and store snow reports for a number of resorts located across the Rocky Mountain region. Using a pre-made / scraped list of resorts. crawler.py navigates through the website and obtains snow, temp and dates for each resort.

The crawler utilizes useragent spoofing, rotating proxies, and a random delay between each request to simulate users and minimize impact on the target website.

Features

API - for tying to hardware devices
WebApp - storing data and analyzing data with dash / plotly

requirements

.config file located in the main directory. This file must contain keys for proxy spoofing api and base url for the website to "crawl"