[New Feature]: Deploy Airflow to MCP venues with Terraform (no adaptation yet)

[LucaCinquini]

Develop Terraform scripts to deploy the OTF Airflow distribution to an EKS cluster. Use the official Airflow Helm distribution with customizable values.yaml. Do not add any adaptation yet except possibly for KEDA autoscaler.
Can start with deploying to LocalStack if it speeds up implementation and testing.

Acceptance Criteria:
o Demonstrated deployment of Airflow onto mcp-venue-dev and mcp-venue-test
o CI/CD pipeline for automatic Airflow deployment

[mike-gangl]

Can you provide an update on this ticket?

[drewm-swe]

@LucaCinquini @mike-gangl We've deployed Airflow into mcp-venue-dev using the official Helm chart. However, it is not fully functional due to several EKS configuration issues (CSI, IAM, security groups, etc.). These issues are being worked and will likely require upstream changes to CS' EKS Terraform module.

The Airflow image has been extended to include the cwltool Python package.

[ryanghunter]

• Default IAM-role for default node-group needs more permissions
• associate instances from default node group with a security group (probably through launch template)
• last bullet solution would be to create a launch template in our terraform and modify CS EKS module to accept launch template id as variable
• add csi addon to clusters

[ryanghunter]

Scratch the security group part - it's possible to get the security group with an EKS data block, so we can add the security group rule in the SPS deployment without making any changes to how we deploy EKS now. We just need to make the following changes:

• add csi addon's to cluster deployments
• create IAM role for default node group with our permissions

[ryanghunter]

EKS changes are finished and pushed.

[LucaCinquini]

Thanks Ryan. I will test again tomorrow with the latest changes.

[drewm-swe]

@ryanhunter-jpl can we update the TF module for VPC creation to tag the subnets with "Tier" = "Private" or "Tier" = "Public" for private and public subnets, respectively? This will allow us to easily grab the subnet IDs within Terraform using data sources.

[LucaCinquini]

Discusses with Galen, U-CS will provide an SSM parameter that can be queried to retrieve the list of public and private sub-nets for each venue. Drew needs to make this change, then will merge the code and provide some instructions for Luca to follow. At that point the ticket can be closed.

[LucaCinquini]

Drew completed the development and wrote instructions here:
https://unity-sds.gitbook.io/docs/developer-docs/science-processing/docs/admin-guide/sps-airflow-deployment-with-terraform

Luca was able to follow the instructions and successfully deployed the basic Airflow on top of an existing EKS cluster, including running the smoke test and the 2 sample DAGs.