In the following repo, you will find a simple ETL process, using different kinds of tools, but basically, Python
.
The goal is download yesterday's data from Spotify, check if the validation process is approved and finally, load the information needed into the database.
You will need to create a copy from the .env.example
and call it env
. There you can put your personal information (TOKEN
, USER-ID
,etc.).
Extract, Transform and Load it's the process that allows to move data from multiple sources, clean them and load them into a SQL database, that could be used into a Data warehouse.
Here we will use the Spotify API. You will need to grab you USER-ID
and generate a TOKEN
in order to use it.
With pip, you can follow this steps:
- Clone the repository
- Create a virtual environment called
env
- Activate the environment and install the requirements with
pip install -r requirements.txt
- Copy the
.env.example
and call itenv
- Change the variables for the ones that you need:
DATABASE_LOCATION
, and from Spotify,USER_ID
,TOKEN
. - Then, you can run your code with
python3 main.py
If you want to use poetry instead, you can skip the first 3 steps and run poetry install
. You need to keep in mind that, to run the code, you'll need to execute: poetry run python3 main.py
Python version: 3.9.5