YouTube Live Data Fetcher 📹

This script fetches live YouTube video data from specified channel IDs, uploads the data to Google Cloud Storage (GCS), and loads it into BigQuery for analysis.

Setup ⚙️

1. Install dependencies:

pip install -r requirements.txt

Obtain API key from Google Cloud Console and update <code>config.json</code>.

Set up Google Cloud credentials:
  Download service account credentials JSON file and update <code>config.json</code> with the file path.

Usage ▶️

Run the script main.py to fetch live video data, upload CSV files to GCS, and load data into BigQuery.

python main.py

If running on Google Cloud Functions with Pub/Sub, ensure the appropriate changes are made for the Pub/Sub trigger.

If running locally in Python, remove the Pub/Sub trigger and event handling.

Example Script (main.py)

Notes 📝

• Ensure proper setup of API key, credentials, and configuration variables in config.json.
• Make sure your environment has necessary permissions to access Google Cloud services.
• Make sure your Big Query Schema mathces the one in the code.
• Make sure your PubSub, BigQuery and Cloud Storage are in the same region server.
• In order to fetch data regarding likes, comments, etc. View the Youtube API V3 Documentation

.gitignore 🔒

Ignore credentials JSON file:

your_google_application_credentials.json

Ignore temporary files:

*.csv

Directory Structure 📂

main.py
config.json
requirements.txt
README.md
.gitignore
  

This setup will help you organize your project on GitHub effectively, making it clear and easy for others to understand and use your YouTube live data fetching script with Google Cloud services. Adjust paths and configurations as per your actual project setup before committing to GitHub.

License ⚖️

This project is licensed under the MIT License - see the LICENSE file for details.