Web Scraping with Sentiment Analysis

The program helps scrape reviews from websites like

• amazon.com
• flipkart.com
• imdb.com identify the aspect terms from the fetch reviews and then classify the reviews w.r.t to aspect terms and their polarity (positive, negative & netural)

Tech

The software uses a number of open source projects to work properly:

• Python3 - Python is an interpreted, high-level and general-purpose programming language.
• Selenium - Selenium automates browsers.
• Natural Language Toolkit (NLTK) - NLTK, is a suite of libraries and programs for symbolic and statistical natural language processing for English
• TextBlob - library for processing textual data
• Flask - Flask is a micro web framework written in Python.
• Chart.js - Simple yet flexible JavaScript charting for designers & developers
• Requests - Requests is an elegant and simple HTTP library for Python, built for human beings.

Installation

It requires Python3 v3+ to run.

$ cd Web-Scraping-With-Sentiment-Analysis
$ pip3 install -r requirements.txt 

You need to install the below softwares / libraries manually. Download the following dependencies

• BeautifulSoup4 - Download
• NLTK with all data Download

Execution

1. Download chromedriver from here according to your Chrome Version & Operating System.
2. Setup the absolute path of the driver in config.ini file using your favourite text editor. Something like this.

   [selenium]
   chromedriver = /home/shivam/Desktop/chromedriver
   

3. From the root directory, execute the following command

   $ python index.py

   Now, Navigate to localhost:5000 to open the GUI.
4. Select the website for which you want to fetch the reviews -[ ] Amazon -[ ] Flipkart -[ ] Imdb
5. Make sure you enter the review page URL for the link.

Images

Demo

Improvements

• Add more websites
• Update Scrapers

License

MIT