Data Engineering Nanodegree

Data Engineering Nanodegree on Udacity. This course focuses on learning data modeling, date warehousing, data lakes, data pipelines, and data ETL scheduling with Airflow

Following are my learning goals for this nanodegree:

1. Data Modeling [DONE]
   1. Learn Data Modeling with Relational DB: Postgres
   2. Learn Data Modeling with NoSQL DB: Apache Cassandra
2. Cloud Data Warehouse [DONE]
   1. Learn about cloud infrastructure for DW (AWS), and Infrastructure-as-code
   2. Implement a star-schema based DW on AWS Redshift
3. Spark and DataLakes [DONE]
   1. Learning to use Spark to work with massive datasets
   2. Implement data-lake on AWS S3 using Spark on AWS EMR
4. Data pipelines [DONE]
   1. Learn about data-pipeline and data-quality
   2. Implement production data-pipeline using Apache Airflow and build DW on AWS Redshift
5. Capstone Project [DONE]
   1. Apply the knowledge learned in above lessons to perform ETL on immigration and demographics data
   2. Create gold dataset for analysts and data scientists to use and analyze immigration patterns in the US

Each of the course modules are separated into their own python module in this repository. They consists of README that one will need to refer to setup their environment (either locally or on AWS).