Configure Production Image Setup using LocalExecutor

[btylerburton]

User Story

Datagovteam would like to implement a production quality airflow installation using the LocalExecutor. This is necessary in order to establish a baseline for Airflow performance in Cloud.gov, and will allow us to compare performance against other executors more quantitatively.

Related:

• https://airflow.apache.org/docs/apache-airflow/stable/start.html
• https://github.com/apache/airflow/blob/main/BREEZE.rst#airflow-breeze-ci-environment
• https://docs.astronomer.io/learn/airflow-executors-explained#when-you-should-use-it
• https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html
• https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/production-deployment.html
• https://airflow.apache.org/docs/docker-stack/index.html
• https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html#notes-about-minimum-requirements

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN I run cf login -a api.fr.cloud.gov --sso and authenticate
  AND I run cf target -o gsa-datagov -s <development|staging|production>
  WHEN I run cf push airflow --vars-file my_vars_file --strategy rolling
  THEN I expect to see the instance of the airflow admin panel when I visit the expected cloud.gov internal URL
  AND I will see my test DAG loaded here.

Background

Given the overwhelming number of options for configuration, establishing a performance baseline with a LocalExecutor configured for production, with an external RDS DB, will allow us to benchmark other solutions more effectively.

Cloud.gov also supports building applications from images, and given that we work with containers locally and have been traditionally supporting a both an image-to-container solution for local developement and a buildpak-to-container solution for production, then this POC will attempt to unify those two environments.

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• Build production image using Breeze tools
• Configure Airflow to use LocalExecutor instead of SequentialExecutor
• Configure manifest.yml to use that image when deploying separate applications
• Configure a cloud.gov Postgres RDb to work with Airflow
• Deploy the applications onto cloud.gov a test DAG, either by extending the Docker image and COPYing it into to the contianer DAG folder, or by mounting a persistent storage volume.

[btylerburton]

Deployed prod image successfully using airflow standalone

[btylerburton]

Parked at https://github.com/GSA/datagov-harvester/tree/configure-local-executor

[btylerburton]

Airflow is working, and we are seeing logs!

[btylerburton]

Draft PR is parked at GSA/datagov-harvester#2

README and other cleanup TBD.

[btylerburton]

After much struggle with the Docker image, we pivoted to using the Python buildpack and were happily greeted with a healthy scheduler.

[btylerburton]

PR is here: GSA/datagov-harvester#3