Web Scraping and Text Analysis Tool for Forum Posts

Overview

This Python tool, developed as part of a Bachelor's thesis, is designed for scraping forum posts from specified URLs and conducting text analysis on the collected data. It focuses on extracting posts from automotive forums, analyzing the content for specific keywords related to innovation and opinions, and performing sentiment analysis.

Features

• Web Scraping: Extracts forum posts using BeautifulSoup and requests.
• Data Cleaning: Cleans and preprocesses text data.
• Keyword Analysis: Searches for specific keywords within the posts.
• Sentiment Analysis: Utilizes TextBlobDE for sentiment and polarity analysis.
• Data Storage: Saves data in a Pandas DataFrame and exports to Excel.
• Text Tokenization and Lemmatization: Implements NLTK for text processing.
• Bigram Analysis: Generates and counts bigrams in the text.