Skip to content

Latest commit

 

History

History
148 lines (106 loc) · 6.69 KB

best-practice.md

File metadata and controls

148 lines (106 loc) · 6.69 KB

Best practice for production deployments

This document guides you through the process of setting up a production-ready Azimuth deployment following recommended best practice.

Repository

Before building your Azimuth configuration, you must first set up your configuration repository, including initialising git-crypt for encrypting secrets.

It is recommended to use a feature branch workflow to make changes to your Azimuth configuration in a controlled way.

Environments

An Azimuth configuration repository contains multiple environments, some of which contain common configuration ("mixin" environments), and some of which represent a deployment ("concrete" environments).

For a production deployment of Azimuth, there should be at least two concrete environments that are deployed from the main branch:

  • production - the Azimuth deployment that is provided to end users
  • staging - used to validate changes before pushing to production

Both of these environments should be highly-available deployments, in order to fully test the upgrade process.

It is recommended to have a single node aio environment that also follows the production environment as closely as possible. This can be used for testing changes to the configuration before they are merged to main.

In order for validation in the aio and staging environments to be meaningful, the configuration of these environments should be as similar to production as possible. To do this, two site-specific mixins should be used:

  • site - contains configuration that is common between aio, staging and production, e.g. enabled features, available platforms
  • site-ha - contains configuration for the HA setup that is common between staging and production

The environments would then be layered as follows:

base --> singlenode --> site --> aio
base --> ha --> site --> site-ha --> staging
base --> ha --> site --> site-ha --> production

with only necessary differences configured in each environment, e.g. the ingress base domain, between staging and production.

OpenStack projects

Azimuth is usually deployed on the cloud that is being targeted for workloads. It is recommended to have three OpenStack projects for a production Azimuth deployment, to contain:

  • A highly-available (HA) production deployment, e.g. azimuth-production
  • A HA staging deployment, e.g. azimuth-staging
  • All-in-one (AIO) deployments for validating changes, e.g. azimuth-cicd

The production and staging projects must have sufficient quota for a HA Azimuth deployment. The required quota in the CI/CD project will depend on the number of proposed changes that are open concurrently.

Users that login to Azimuth will see a list of the OpenStack projects they can access. Any OpenStack project can be used to deploy Azimuth workloads, as long as the networking requirements have been met. Azimuth creates application credentials to be able to create the user workloads using the appropriate automation.

Deployment Prerequisites

Before proceeding with an Azimuth deployment, you should ensure that you can fulfil all the prerequisites.

The prerequisites include verifying that the target cloud has the required features available, generating OpenStack application credentials and ensuring that the required DNS records and TLS certificates are available.

OpenTofu state

Azimuth deployments use OpenTofu, an open-source fork of Terraform, to manage some parts of the infrastructure.

A remote state store must be configured in order to persist the OpenTofu state across playbook executions. If GitLab is being used for the Azimuth configuration repository, it is recommended to use GitLab-managed Terraform state for this. If not, S3 is the preferred approach.

Continuous delivery

A production Azimuth deployment should use continuous delivery, where changes to the configuration are automatically deployed to the aio and staging environments. The ansible-playbook command should never be executed manually.

The recommended approach is to automatically deploy an independent aio environment for each feature branch, also known as per-branch dynamic review environments. This allows changes to be validated before they are merged to main.

Once a change is merged to main, it will be deployed automatically to the staging environment. Merging to main will also create a job to deploy to production, but that job will require a manual trigger. Once the change has been validated in staging, the job to deploy to production can be actioned.

It is also possible to run integration tests against a deployment. It is recommended that this is configured to run automatically for at least aio and staging environments. It should also be safe to run against production as long as the tests are configured to target an isolated project.

A sample GitLab CI/CD configuration is provided that implements this workflow for GitLab-hosted repositories.

Monitoring

Before going into production, be sure you can access the monitoring.

Also ensure you have configured alert manager such that you will notice any alerts on staging and production.

Disaster recovery

Azimuth uses Velero to backup the data that is required to restore an Azimuth instance in the event of a catastrophic failure. This functionality is not enabled by default, as it requires credentials for an S3 bucket in which the backups will be stored.

It is recommended that disaster recovery is enabled for a production deployment.

Configuration

You are now ready to begin adding configuration to your environments. When building an environment for the first time, it is recommended to follow each documentation page in order, beginning with the Deployment method.

!!! tip

Remember to share as much configuration as possible between all your [environments](./environments.md)!