Autotrader UK scraper utility

A wrapper utility with few additions based off suhailidrees/autotrader_scraper

Installation

Clone the repo and run pip3 install -r requirements.txt

Configuration

Place the configuration file under conf directory.

Sample Config file

Refer to conf/sample.json

{
        "make" : "Audi",
        "model" : "A5",
        "model_variant" : "",
        "manufacturer_approved" : "",
        "maximum-mileage":40000,
        "max_price": 20000,
        "postcode" : "SW1A 0AA",
        "radius" : 1500,
        "min_year" : 2016,
        "max_year" : 2019,
        "include_writeoff" : "exclude",
        "max_attempts_per_page" : 5,
        "verbose" : "False"
}

Details of the configuration file.

Criteria / Argument	Type	Description	Values	Default Value
make	String	Make of the car	Get these from autotrader.co.uk*. Examples are "Audi", "BMW", "Jaguar"	"Audi"
model	String	Model of the car	Get these from autotrader.co.uk*. Examples are "A3", "A4", "A4" for Audi	"A5"
model_variant	String	Model Variant of the car	Get these from autotrader.co.uk*. Examples are "S line", "Black Edition", "A4" for Audi	""
manufacturer_approved	String	Manufacturer Approved	Takes Y or blank	""
maximum-mileage	String	Maximum Milage	Takes integer, eg. 40000	""
postcode	String	Postcode where you are searching	Example: "CB2 1TN", "NW1 2BH"	"SW1A 0AA"
radius	Integer	Radius of your search from the postcode	Can be any positive integer. Use 1500 for nation-wide search	1500 (i.e. nation-wide)
min_year	Integer	Minimum year of the car's manufacture	Can be any positive integer	1995
max_year	Integer	Maximum year of the car's manufacture	Can be any positive integer	1995
include_writeoff	String	Whether or not to include insurance write-off categories (i.e. Cat S/C/D/N)	"include", "exclude", "writeoff-only"	"include"
max_attempts_per_page	Integer	Maximum times to attempt scraping a page. A request may fail due to connectivity issues, server response issues, etc. and so in the event of a failure, the request will be retried a number of times specified by this argument	Can be any positive integer	5
verbose	Boolean	Whether or not to print progress on the console. Good for debugging	True, False	False

Usage

usage: scraper.py [-h] [--type TYPE] [--outfile OUTFILE] [--verbose VERBOSE] [--inputfile INPUTFILE]

optional arguments:
  -h, --help            show this help message and exit
  --type TYPE           output type csv or json
  --outfile OUTFILE     file prefix
  --verbose VERBOSE     verbose mode
  --inputfile INPUTFILE
                        input file in json

Sample Run

./scraper.py --type csv --inputfile conf/bmw_1_series.json --outfile out/bmw_1.csv

or

./scraper.py --type json --inputfile conf/bmw_1_series.json --outfile out/bmw_1.json