Genie import DAG prototype #64

callachennault · 2024-11-27T21:17:15Z

This PR introduces two DAGs: genie_data_dag.py, and import_genie_dag.py. Scripts for the two DAGs are contained in the scripts/ directory. Also included in this PR is a Dockerfile for the Airflow worker pods, along with an executor config definition and a python requirements file.

Remote scripts accept a scripts directory - workaround for environment variable setting. Airflow environment propagation not passed to actual remote shell

Add arg for passing scripts directory

Update import_genie_dag.py

…olean

callachennault and others added 30 commits September 16, 2024 18:05

Move imports, create params, pass env vars to tasks

886e633

Add executor config, dockerfile, and requirements

2975eab

rename executor config

0ff67e5

fix executor import

9f391f6

fix executor config import again

2bdfb3b

add sys path append before import

7c72872

Make data and import dag, split bash operations into scripts

5729010

Fix params title

244c729

Fix dep chain

07ab75a

Change trigger rule for pull_data_from_synapse

828f58a

comment out git push for now

2d359e2

remove handle_error function

29bc3ab

Fix param name

71d6a84

remove handle_error function again

3310744

Add handle_error back in

f400020

remove double curly brackets

6db3691

add handle_error back to synapse script

0cef4f4

add test ssh task

f9be41c

Add import_genie_dag, rename old dag to genie_ssh_dag

f8095d6

rename tasks

20d104f

add double brackets to params

8175e92

remove param to test

0aaea0d

fix command arg

40515b9

change default value for importer param and fix logic bug

ba119ac

Add arg for passing scripts directory

46046dc

Remote scripts accept a scripts directory - workaround for environment variable setting. Airflow environment propagation not passed to actual remote shell

Update import_genie_dag.py

f61ed73

Merge pull request #1 from averyniceday/patch-1

7cc0321

Add arg for passing scripts directory

Update import_genie_dag.py

57d1bd9

Merge pull request #2 from averyniceday/patch-1

82383aa

Update import_genie_dag.py

Update import_genie_dag.py

62e251f

averyniceday and others added 30 commits October 7, 2024 13:17

Update import_genie_dag.py

5c13bed

Update import_genie_dag.py

1a316f6

Update import_genie_dag.py

5975481

Update import_genie_dag.py

224683f

Update import_genie_dag.py

1dfb454

Update import_genie_dag.py

d621bec

add genie repo to study path

3acace1

move folder contents instead of folder

a674cb4

cp files instead of mv

eebce75

print git error output

b61a52e

print cwd

281e137

print cwd

8051e8a

hardcode path for testing

504f08a

try to run git status

3871153

Make git_push a bash task

4f7d8c4

set up git user after clone

7dfc4ad

move git commit user setup to git push script

47ef2de

remove python git push task, add git fetch, change push_to_repo to bo…

691bd21

…olean

add fetch genie script

719f5f7

use python task decorator, update fetch script and git push

d3efb5f

instantiate decorated tasks

d5e1cd8

add git fetch to task chain

9b14023

Try fixing python script import

aceaa86

Try short circuit operator

97fb5ec

remove native obj flag

3200754

update triggger rules

f2f3334

Clean up dag, add clean_genie_repo task

d35e7ee

Clean up genie import dag

19502eb

Update start_date, add max_active_runs, remove unused default env vars

db751af

remove genie ssh dag

d265ca5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Genie import DAG prototype #64

Genie import DAG prototype #64

callachennault commented Nov 27, 2024 •

edited

Loading

Genie import DAG prototype #64

Are you sure you want to change the base?

Genie import DAG prototype #64

Conversation

callachennault commented Nov 27, 2024 • edited Loading

callachennault commented Nov 27, 2024 •

edited

Loading