Finance ETL

ETL pipeline to extract data from multiple file formats, transform it into specific datatypes, and load it into a single source for analysis.

Modules

General overview:

US stock market data is extracted from TradingView with BeautifulSoup and stored in .json files.
Stock symbol data is collected from Finnhub Stock API and stored in a .csv file.
Exchange rate data is collected from Exchange Rates API and stored in a .csv file.
All stock market files are transformed and combined in a single file.
ETL job execution is complete once data is loaded into the database and validated.

DAG graph view:

Name		Name	Last commit message	Last commit date
Latest commit History 42 Commits
airflow/dags		airflow/dags
files		files
tests		tests
.gitignore		.gitignore
README.md		README.md
get_exchange_rate.py		get_exchange_rate.py
get_stock_market.py		get_stock_market.py
get_stock_symbol.py		get_stock_symbol.py
insert_table.py		insert_table.py
run_data_validation.py		run_data_validation.py
run_etl.py		run_etl.py