Code

Datalake V1#905

Code

Draft

ShadeWyrm wants to merge 22 commits intomaincds-snc/forms-terraform:mainfrom feat/datalake-v1cds-snc/forms-terraform:feat/datalake-v1Copy head branch name to clipboard

+964-1

Commits

Commits on Nov 26, 2024

• * Draft datalake

  ShadeWyrm

  committed
  1a077ef

• * Swap depcrated storage object type to new one.

  ShadeWyrm

  committed
  ad99fcf

• * Add glue service to localstack.

  ShadeWyrm

  committed
  3342118

Commits on Nov 27, 2024

• * add glue to missing local-provider.tf

  ShadeWyrm

  committed
  cb93b7a

• * update endpoint for localstack vs aws

  ShadeWyrm

  committed
  3f8cf66

• * pass s3/vs/localstack to rds_etl script.

  ShadeWyrm

  committed
  de7d785

• * Sorta working ETL!~

  ShadeWyrm

  committed
  8e3a920

Commits on Nov 28, 2024

• * working post-crawler

  ShadeWyrm

  committed
  31e50d3

• * Force update if script changed.

  ShadeWyrm

  committed
  37d1fa4

Commits on Dec 3, 2024

• * Add athena.

  ShadeWyrm

  committed
  f7faf1f

• Merge branch 'main' into feat/datalake-v1

  ShadeWyrm

  authored
  b01fc40

• * Update v ref to checksum

  ShadeWyrm

  committed
  52d1d08

• Update ETL script -> Need to use Java timestamp format for dates.

  ShadeWyrm

  committed
  c492d8d

• * add athena

  ShadeWyrm

  committed
  0175b99

Commits on Dec 6, 2024

• * add timestamp for partitioning data.

  ShadeWyrm

  committed
  8050f9c

• Merge branch 'main' into feat/datalake-v1

  ShadeWyrm

  authored
  5d25c44

Commits on Dec 9, 2024

• * Base historical job.

  ShadeWyrm

  committed
  58f36d8

• * add historical crawler

  ShadeWyrm

  committed
  926ef6a

Commits on Dec 11, 2024

• Merge branch 'main' into feat/datalake-v1

  ShadeWyrm

  authored
  86fedf9

Commits on Dec 12, 2024

• * draft of prepared query for metrics

  ShadeWyrm

  committed
  a1363e9

Commits on Dec 16, 2024

• * Update athena IAM

  ShadeWyrm

  committed
  3ee7022

Commits on Dec 19, 2024

• Merge branch 'main' into feat/datalake-v1

  ShadeWyrm

  authored
  d4e63d3