Skip to content

DevOps

Jump to bottom
Manish Sihag edited this page Nov 13, 2024 · 7 revisions

DevOps Continuous Integration / Continuous Deployment

First it was Waterfall, next it was Agile, and now it's DevOps. This is how modern developers approach building great products. With the rise of DevOps has come the new methods of Continuous Integration, Continuous Delivery, (CI/CD) and Continuous Deployment. Conventional software development and delivery methods are rapidly becoming obsolete. Historically, in the agile age, most companies would deploy/ship software in monthly, quarterly, bi-annual, or even annual releases. Now we build and deploy multiple times a day.

Continuous integration focuses on blending the work products of individual developers together into a repository. Often, this is done several times each day, and the primary purpose is to enable early detection of integration bugs, which should eventually result in tighter cohesion and more development collaboration. The aim of continuous delivery is to minimize the friction points that are inherent in the deployment or release processes. Typically, the implementation involves automating each of the steps for build deployments such that a safe code release can be done—ideally—at any moment in time. Continuous deployment is a higher degree of automation, in which a build/deployment occurs automatically whenever a major change is made to the code.

PIMS CI/CD

The PIMS project currently uses GitHub Actions and OpenShift to support CI/CD.

The general high-level workflow is as follows;

  1. Fork Repo
  2. Clone Repo
  3. Submit PR to main branch
  4. GitHub Action - Run Unit Tests
  5. GitHub Action - Run Code Coverage (CodeCov)
  6. GitHub Action - Build and Push Image to Imagesteam(Pull Request Number used as the Tag for the Image Build)
  7. Merge PR into main branch
  8. GitHub Action - Run Deployement to DEV environment (deployes the image tag (PR number))
    • GitHub Action - Run Update Configuration in GitOps Repository - Deployment Configuration ( API or APP based on the changes ) this will update the gitops repository.
  9. ArgoCD - Listens to the changes in HELM chart and apply sync on the openshift resources.
    • OpenShift - Orchestrate Pods
    • OpenShift - Orchestrate Containers
    • OpenShift - Orchestrate Storage
    • OpenShift - Orchestrate Routes

Environments

There are four projects within OpenShift that are named, TOOLS, DEV, TEST, PROD. Three of which represent the environments where PIMS instances will reside. The fourth is TOOLS which hosts tooling such as Jenkins, SonarQube and others.

Environments

Name URL Description
DEV pims-dev.apps.silver.devops.gov.bc.ca The development environment for Developers
TEST pims-test.apps.silver.devops.gov.bc.ca The testing environment for QA and UAT
PROD pims.gov.bc.ca The production environment

Branching

Repository branches are used to manage development. There is only one branch main.

Branch Description
main Default branch for all environments
{pims-####} These branches are normally hosted in forked repositories and link to Jira Stories.

It's essential to note that deployments to DEV, TEST, and PROD environments are not dictated by separate branches. Instead, configurations rely on image tags for control and versioning.

DEV Environment

Deploying changes to the DEV environment involves a straightforward automated process. The workflows are named as follows:

  • API Image Build: EXPRESS API: Image Build on PR
  • APP Image Build: REACT APP: Image Build on PR
  • API Deployment: EXPRESS API: Image Deploy
  • APP Deployment: REACT APP: Image Deploy

Developers initiate the deployment by creating a pull request (PR) to merge their changes into the main branch. This action then triggers the GitHub workflow, specifically either the EXPRESS API: Image Build on PR or REACT APP: Image Build on PR GitHub Action workflow, or both, depending on the nature of the file changes.

Once the PR is approved and ready to merge, the developer will merge the PR. The merging process automatically triggers the GitHub Action, further promoting the image tag in the DEV environment. This streamlined process ensures that the changes are promptly deployed to the DEV environment for initial quality assurance (QA) testing.

Note: To ensure that the build image tag consistently reflects the latest code from the main branch along with any new work, it is mandatory to perform an "Update branch" before merging. The "Update branch" process involves merging the main branch into the temporary branch intended for merging (dev), ensuring that images are updated with the most recent code changes. This practice guarantees that the build image accurately represents the current state of the codebase before any merging occurs.

TEST and PROD Environment Deployment

For changes to be promoted to the TEST environment, a manual trigger of the GitHub Action is essential. The workflows are named as follows:

  • API Deployment: EXPRESS API: TEST Deploy
  • APP Deployment: REACT APP: TEST Deploy

Deployment Workflow

The GitHub Action workflow for API and APP deployment requires one inputs:

  1. EXPRESS API: TEST Deploy: This tag specifies the version of the API image to be deployed.
  2. REACT APP: TEST Deploy: This tag is mandatory to initiate the workflow.

CI/CD Pipelines

In the CI/CD Pipelines Overview table below, it present a comprehensive summary of the key pipelines involved in our development lifecycle. These pipelines, designed for the DEV, TEST, and PROD environments, encompass image builds and deployments, both automated and manual.

Name Destination Trigger Description
EXPRESS API: Image Build on PR Openshift Imagestream Auto Builds the API Image upon PR creation, incorporating the PR number as the tag.
REACT APP: Image Build on PR Openshift Imagestream Auto Constructs the APP Image during PR creation, utilizing the PR number as the tag.
EXPRESS API-Image Deploy on PR merge DEV Auto/Manual Automatically deploys the API Image upon PR merge, utilizing the PR number as the tag. Additionally, can be manually triggered for rollback purposes.
REACT APP-Image Deploy on PR merge DEV Auto/Manual Automatically deploys the APP Image upon PR merge, using the PR number as the tag. Also, can be manually triggered for rollback purposes.
EXPRESS API-Image Deploy TEST Manual Manually initiates the pipeline to deploy the API Image, providing the API image tag and Migration Image tag (if needed).
REACT APP-Image Deploy TEST Manual Manually initiates the pipeline to deploy the APP Image, specifying the API image tag.
EXPRESS API-Image Deploy PROD Manual Manually initiates the pipeline to deploy the API Image, providing the API image tag and Migration Image tag (if needed).
REACT APP-Image Deploy PROD Manual Manually initiates the pipeline to deploy the APP Image, specifying the API image tag.

Image Tagging Strategy

In our CI/CD pipeline, image tagging is a crucial aspect that facilitates version tracking, feature identification, and deployment to specific environments within OpenShift. Our tagging strategy revolves around associating images with their corresponding PR numbers, creating a clear linkage between images and the development lifecycle.

  1. Tagging Trigger: Images are built and tagged automatically when a new PR is created, triggered by changes in files. The tagging process differentiates between API and App changes, ensuring precision in versioning.

Benefits:

  • Traceability: Each image is distinctly tied to a PR number, providing a traceable and identifiable link to the corresponding development work.

  • Prevention of Broken Code: Image builds occur exclusively during PR creation, ensuring that only validated code is merged into the Main branch. This safeguards against the introduction of broken code.

  • Rollback Capability: The tagging strategy enables a straightforward rollback to a known and stable image tag. In case of issues or errors, reverting to a previous image tag is easily achievable, minimizing downtime and facilitating quick issue resolution.

  • Enhanced Readability: PR-based tagging enhances the readability of the image tags. Each tag reflects the specific PR number, making it intuitive and straightforward to understand the context and purpose of each image.

Important Consideration:

  • Non-Sequential Tags: It's essential to note that PR tags are not sequential. For instance, Tag 1921 is not a successor of 1920. If 1920 is merged after 1921, Tag 1920 will incorporate the changes for 1921. This emphasizes the mandatory "Update branch" step, ensuring the branch is updated before any merging occurs.

Development Releases

Development Workflow:

  1. Developers initiate the deployment process by creating a pull request (PR) to merge their changes into the main branch.

  2. The CI/CD pipeline triggers specific workflows, such as EXPRESS API: Image Build on PR and REACT APP: Image Build on PR, automatically building and tagging API and APP images using the PR number.

Automated Deployment on PR Merge:

  1. Upon approval and readiness, developers merge the PR into the main branch.

  2. The GitHub Action workflow (EXPRESS API-Image Deploy on PR merge and REACT APP-Image Deploy on PR merge) is automatically triggered, deploying the API and APP images to the DEV environment using the PR number as the tag.

##DATABASE##

PIMS leverages the Crunchy Postgres Operator to manage PostgreSQL deployments on our Kubernetes cluster. The operator simplifies PostgreSQL database management by providing powerful automation features, including high availability, backup, recovery, monitoring, and scaling.

The Crunchy Postgres Operator automates the provisioning, management, and scaling of PostgreSQL clusters on Kubernetes. It provides a declarative approach for deploying PostgreSQL clusters, enabling continuous management of instances and ensuring high availability through features like failover and replication.

We use Pgbackrest for backup and restore management. it allows us to perform disaster recovery with in few simple steps. complete document is availabe here

Production Releases

Every Sprint, a new releasable increment is developed, adhering to our versioning strategy (1.0.0 = [major.minor.patch]).

Once a releasable increment is deemed ready, the changes can be promoted to the TEST environment for thorough testing and quality assurance. To initiate this promotion to TEST, developers can trigger the GitHub Action pipeline, specifying the desired version tag they want to promote. During the pipeline execution, a prompt will request a tag name. For instance, if developers wish to proceed with the changes deployed in the DEV environment, they can use the tags from the Image Tags Wiki page table to guide the promotion further. Below is an example of the input UI from the GitHub Action that represents the input field for the API test deployment workflow:

Similarly, for the production release, the same steps would be followed by providing the tags currently working in the TEST environment.

Upon successful QA and UAT approval in the TEST environment, a manually triggered production pipeline can be initiated. This production pipeline will prompt for the appropriate tag to release. Developers should input the tag that corresponds to the version that has been thoroughly tested and approved in the TEST environment. This ensures a controlled and deliberate promotion of changes from the TEST environment to the production environment.

Rollback

In the event of a failure during the deployment phase, it becomes necessary to perform a rollback of the release images that were deployed and restore the database backup. Developers have the flexibility to rollback to a previous version at any given point by utilizing tags in each environment.

This comprehensive rollback strategy ensures that developers can quickly and manually initiate a rollback to a known and stable version in case of any deployment issues.

Clone this wiki locally