CLASS advanced Fall 2023 - Research Computing IaC Repository (RCIR)

Table of Contents

• Overview
• Technology Stack
• Repository Structure
  • Root
  • GitHub Workflows
  • Documentation
  • Templates
    • AWS
    • GCP
  • Use-Cases
  • Containers
  • Scripts
  • Environments
• Getting Started
• Design Considerations
• Contribution Guidelines
• License

Overview

This repository provides Infrastructure as Code (IaC) templates and use-case implementations for research computing. It's designed for extensibility, cloud-agnosticism, and community contributions. Initially focused on AWS, GCP and Azure but will be extended to cover more providers in the future.

Technology Stack

• Python: Scripting and automation.
• Docker: Containerization.
• Terraform: IaC.
• Ansible: Configuration management.
• Bash: Shell scripting.
• Ubuntu: Base OS.

Repository Structure

Root

• README.md: This document.
• .gitignore: Git ignore rules.

GitHub Workflows

This folder contains GitHub workflows for CI/CD.

Templates

This folder contains IaC templates for research computing, that can be used to build cloud infrastructure (as shown with the use-cases in the use-cases folder)

AWS

• templates/cloud_providers/aws/terraform: Terraform for AWS.
• templates/cloud_providers/aws/ansible: Ansible for AWS.

GCP

• templates/cloud_providers/gcp/terraform: Terraform for GCP.
• templates/cloud_providers/gcp/ansible: Ansible for GCP.

Use-Cases

This folder contains use-case implementations for research computing, that use the templates in the templates folder.

• use-cases/secure-storage: Secure storage solutions.
• use-cases/kubernetes-cluster: Kubernetes setups.
• use-cases/containerized-workflows: Containerized workflows.

Scripts

This foder contains Bash and Python scripts that could be used to stitch together the various components that make up a use case.

• scripts/bash: Bash scripts.
• scripts/python: Python scripts.

Environments

This folder contains environment-specific configurations.

• environments/aws: AWS-specific configs.
• environments/gcp: GCP-specific configs.
• environments/azure: Azure-specific configs.

Getting Started

1. Clone: git clone https://github.com/your-repo.git
2. Navigate: cd your-repo
3. Install: Follow READMEs in each directory.
4. Run: Follow instructions in each README.
5. Deploy: Use Terraform or Ansible.

Design Considerations

• Modularity: Reusable templates.
• Documentation: Well-documented.
• Version Control: Semantic versioning.
• Testing: Automated tests.
• Security: Secure coding.
• Cloud Agnosticism: Cloud-agnostic.
• Code Reviews: PRs and code reviews.

Contribution Guidelines

See Contribution Guidelines.

License

Acknowledgements

This project was developed as part of the Internet2 CLASS program with contributions from participants at the Washington University in St. Louis, NC State University, UC San Diego, CU Boulder, Ohio State University, National Center for Atmospheric Research, Yale, Northwestern University, University of North Carolina at Chapel Hill and UC Riverside.

Contact: class@internet2.edu

---

Working Space

• Some helpful links:

  • RAD Lab

  • HPC Toolkit

  • EKS Cluster

Open Questions

• What's the best way to organize templates across classes (terraform, yml, containers, cfgs etc.) ?

• How should high level variables (environment, path, instance etc) be handled?

• What is the best way to stitch together the various components that make up a use case? (terraform, ansible, bash, python, containers etc.)

• Is there a need to host conatiner dockerfiles in the repo?

• What are the best practices for storing tf state file ?

• How can we handle Ansible inventory files in a scalable way?