Skip to content

Data Lake project for ingest and transform financial data and dashboard BI proposal

License

Notifications You must be signed in to change notification settings

Sheitak/datalake-jljq

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Data Lake Project - JLJQ

This project is part of a study into the creation of a Data Lake, and focuses on the field of finance. It gathers different data sources to be formatted to obtain the final visualization of a trend in this field.

Maintainers

Starting Project

Clone this repository and place yourself inside it.

You can copy .env.example file to .env and replace the essential information.

Apache-Airflow Docker

Refer to the Airflow With Docker on the official airflow website for a clean installation with Docker.

Refer to the Airflow Package In Docker For creating Docker File with packages.

1. Initializing Environment

Generate Folders And Define Airflow User If Necessary

mkdir -p ./dags ./logs ./plugins ./config
echo -e "AIRFLOW_UID=$(id -u)" > .env

Initialize Database

docker compose up airflow-init

Run Airflow Set-Up

docker compose up -d

2. Accessing the environment

Running the CLI commands

docker compose run airflow-worker airflow info

OR

Open in browser

localhost:8080 or IP:8080

Defaults Information

Airflow Credentials

username: airflow
password: airflow

API Used

Resources

Copyright (C) 2007 Free Software Foundation, Inc. https://fsf.org/

About

Data Lake project for ingest and transform financial data and dashboard BI proposal

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published