The Pole Studio Scraper is a comprehensive tool designed to collect and analyze data from pole dancing studios listed on eversports.de. The scraper handles URL reconstruction, validation, data extraction, and storage, making it a robust solution for gathering structured data from multiple sources.
- URL Reconstruction: Extracts and reformats URLs from the main studio page.
- URL Validation: Ensures URLs are reachable before attempting to scrape.
- Data Scraping: Collects detailed information about studios, workshops, and their specifics.
- Data Storage: Organizes and saves the collected data into CSV files for easy access and analysis.
- Error Handling: Continues processing despite encountering errors, ensuring maximum data retrieval.
- Progress Tracking: Provides real-time progress updates using
tqdm
.
-
Clone the Repository:
git clone https://github.com/yourusername/pole-studio-scraper.git cd pole-studio-scraper
-
Install Dependencies:
pip install -r requirements.txt
-
Prepare Initial Data: Ensure the
polestudios_neue_liste.csv
file containing initial URLs is present in the project directory.
-
Run the Main Script:
- The main entry point for the scraper is the
a_MainFrame.ipynb
notebook, which orchestrates the entire scraping process.
import pandas as pd from a_PyCaller import process_urls from tqdm import tqdm def main(): all_uRLs = pd.read_csv("polestudios_neue_liste.csv") initial_urls = all_uRLs["URL_S"].to_list()[:5] all_pole_studio_data = pd.DataFrame() all_workshops_data = pd.DataFrame() all_workshop_details_data = pd.DataFrame() all_urls_data = pd.DataFrame(columns=['URL']) with tqdm(initial_urls, desc="Processing URLs", dynamic_ncols=True) as pbar: for url in pbar: data = process_urls([url]) if data: # Processing logic # Export DataFrames to CSV all_pole_studio_data.to_csv("Pole_Studio_Übersicht_S.csv", index=False) all_workshops_data.to_csv("Workshop_Liste_SW.csv", index=False) all_workshop_details_data.to_csv("Workshop_Übersicht_E.csv", index=False) all_urls_data.to_csv("All_URLs.csv", index=False) if __name__ == "__main__": main()
- The main entry point for the scraper is the
-
Process URLs:
- The
a_PyCaller.py
script handles the core processing, including URL reconstruction, validation, and data scraping.
from a_URLS_Reconstruction import reconstruct_urls_and_extract_buttons from b_URLS_Validation import validate_urls from c_PoleStudio_Overview_S import scrape_pole_studio from d_Workshop_List_SW import scrape_workshops from e_Workshop_Overview_E import scrape_workshop_details import pandas as pd def process_urls(urls): # URL processing logic return results
- The
- Fork the Repository: Click the 'Fork' button at the top right of the repository page.
- Create a New Branch:
git checkout -b feature-branch
- Commit Your Changes:
git commit -m 'Add new feature'
- Push to the Branch:
git push origin feature-branch
- Create a Pull Request: Submit your changes for review.
This project demonstrates a practical approach to web scraping and data collection, showcasing my skills in Python programming, data processing, and error handling. If you have any questions or suggestions, feel free to reach out.