Skip to content

Latest commit

 

History

History
54 lines (43 loc) · 1.12 KB

README.md

File metadata and controls

54 lines (43 loc) · 1.12 KB
Raw

KLV Crawler

This crawler can crawl tens of thousands of apps information from the Xiaomi App Store, and save them into the MongoDB. It is based on Python + Scrapy.

Getting Started

Prerequisties:

You'll need to install:

  • Python (2.7.10)
  • scrapy(python lib)
  • pymongo(python lib)
  • MongoDB (3.0.0+)

Installing

Python(Mac OSX)

sudo pip install scrapy
sudo pip install pymongo

MongoDB(CentOS7)

yum -y install mongodb-org mongodb-org-server
systemctl start mongod

Usage

Download

git clone https://github.com/BitTigerInst/KLV_Crawler.git

Run

cd KLV_Crawler
scrapy crawl cnproxy
scrapy crawl vkea_xiaomi

Team Members
Vkea95
Vkea95

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Project Information

  • category: big data
  • team: KLV Team
  • description: Crawling information from Xiaomi APP store.
  • stack: python, scrapy, mongodb