A scraper built using Scrapy+Python that can quickly get a list of most law enforcement agencies in the US using the PoliceOne.com directory.
- First create a virtual environment with either
virtualenv
,virtualenvwrapper
,Anaconda
, etc. - Once you've created your virtual environment, do
pip install -r requirements.txt
to ensure that all required libraries are installed. - To run the scraper, simply do:
scrapy crawl agencies
on terminal. - If you want to save the output to a JSON file, use
scrapy crawl agencies -o agencies.json
instead. - Once you've exported to JSON, you can also use the
translator.py
file to convert your JSON into a CSV for Excel/etc. based manipulation. Simply do:python translator.py
and as long as you didn't use a different name for the.json
file, you should be good!