Data Engineering Process

Import, clean, and transform CSV, LOG, JSON files, then load it into MySQL database; perform basics analysis using SQL and display the results.

Setup your environment:

1. Download & install Python to run the program.

• https://www.python.org/downloads/
• https://www.journaldev.com/30076/install-python-windows-10
• https://www.howtogeek.com/197947/how-to-install-python-on-windows/

2. Install the required packages

• pip install -U pandas
• pip install -U sqlalchemy
• pip install -U mysql-connector
• pip install PyMySQL
• pip install json
• pip install -U cryptography

3. Download & Install the MySQL Community database software.

• https://dev.mysql.com/downloads/mysql/
• https://www.sqlshack.com/how-to-install-mysql-database-server-8-0-19-on-windows-10/
• https://www.onlinetutorialspoint.com/mysql/install-mysql-on-windows-10-step-by-step.html

4. Place the files the same directory.

5. In the notebook, change the file path in pd.read_excel() | pd.read_csv() your preferred data name.

6. You should be able to work through the entire notebook

If you run into issues or have questions, log an issue and I'll get back to you ASAP.