This repository contains these plugins to support running Velero on Microsoft Azure:
-
An object store plugin for persisting and retrieving backups on Azure Blob Storage. Content of backup is kubernetes resources and metadata files for CSI objects, progress of async operations. It is also used to store the result data of backups and restores include log files, warning/error files, etc.
-
A volume snapshotter plugin for creating snapshots from volumes (during a backup) and volumes from snapshots (during a restore) on Azure Managed Disks.
- Since v1.4.0 the snapshotter plugin can handle the volumes provisioned by CSI driver
disk.csi.azure.com
- Since v1.5.0 the snapshotter plugin can handle the zone-redundant storage(ZRS) managed disks which can be used to support backup/restore across different available zones.
- Since v1.4.0 the snapshotter plugin can handle the volumes provisioned by CSI driver
Below is a listing of plugin versions and respective Velero versions that are compatible.
Plugin Version | Velero Version |
---|---|
v1.6.x | v1.10.x |
v1.5.x | v1.9.x |
v1.4.x | v1.8.x |
v1.3.x | v1.7.x |
v1.2.x | v1.6.x |
v1.1.x | v1.5.x |
v1.1.x | v1.4.x |
v1.0.x | v1.3.x |
v1.0.x | v1.2.0 |
If you would like to file a GitHub issue for the plugin, please open the issue on the core Velero repo
Ensure that the VMs for your agent pool allow Managed Disks. If I/O performance is critical, consider using Premium Managed Disks, which are SSD backed.
To set up Velero on Azure, you:
- Create an Azure storage account and blob container
- Get the resource group containing your VMs and disks
- Set permissions for Velero
- Install and start Velero
You can also use this plugin to create an additional Backup Storage Location.
If you do not have the az
Azure CLI 2.0 installed locally, follow the install guide to set it up.
Run:
az login
By default, Velero will store backups in the same Subscription as your VMs and disks and will not allow you to restore backups to a Resource Group in a different Subscription. To enable backups/restore across Subscriptions you will need to specify the Subscription ID to backup to.
Use az
to switch to the Subscription the backups should be created in.
First, find the Subscription ID by name.
AZURE_BACKUP_SUBSCRIPTION_NAME=<NAME_OF_TARGET_SUBSCRIPTION>
AZURE_BACKUP_SUBSCRIPTION_ID=$(az account list --query="[?name=='$AZURE_BACKUP_SUBSCRIPTION_NAME'].id | [0]" -o tsv)
Second, change the Subscription.
az account set -s $AZURE_BACKUP_SUBSCRIPTION_ID
Execute the next step – creating an storage account and blob container – using the active Subscription.
Velero requires a storage account and blob container in which to store backups.
The storage account can be created in the same Resource Group as your Kubernetes cluster or
separated into its own Resource Group. The example below shows the storage account created in a
separate Velero_Backups
Resource Group.
The storage account needs to be created with a globally unique id since this is used for dns. In
the sample script below, we're generating a random name using uuidgen
, but you can come up with
this name however you'd like, following the Azure naming rules for storage accounts. The
storage account is created with encryption at rest capabilities (Microsoft managed keys) and is
configured to only allow access via https.
Create a resource group for the backups storage account. Change the location as needed.
AZURE_BACKUP_RESOURCE_GROUP=Velero_Backups
az group create -n $AZURE_BACKUP_RESOURCE_GROUP --location WestUS
Create the storage account.
AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"
az storage account create \
--name $AZURE_STORAGE_ACCOUNT_ID \
--resource-group $AZURE_BACKUP_RESOURCE_GROUP \
--sku Standard_GRS \
--encryption-services blob \
--https-only true \
--min-tls-version TLS1_2 \
--kind BlobStorage \
--access-tier Hot
Create the blob container named velero
. Feel free to use a different name, preferably unique to a single Kubernetes cluster. See the FAQ for more details.
BLOB_CONTAINER=velero
az storage container create -n $BLOB_CONTAINER --public-access off --account-name $AZURE_STORAGE_ACCOUNT_ID
(Optional) If you decided to backup to a different Subscription, make sure you change back to the Subscription of your cluster's resources before continuing.
-
Set the name of the Resource Group that contains your Kubernetes cluster's virtual machines/disks.
WARNING: If you're using AKS,
AZURE_RESOURCE_GROUP
must be set to the name of the auto-generated resource group that is created when you provision your cluster in Azure, since this is the resource group that contains your cluster's virtual machines/disks.AZURE_RESOURCE_GROUP=<NAME_OF_RESOURCE_GROUP>
If you are unsure of the Resource Group name, run the following command to get a list that you can select from. Then set the
AZURE_RESOURCE_GROUP
environment variable to the appropriate value.az group list --query '[].{ ResourceGroup: name, Location:location }'
Get your cluster's Resource Group name from the
ResourceGroup
value in the response, and use it to set$AZURE_RESOURCE_GROUP
.
There are several ways Velero can authenticate to Azure: (1) by using a Velero-specific service principal; (2) by using AAD Pod Identity; or (3) by using a storage account access key.
If you plan to use Velero to take Azure snapshots of your persistent volume managed disks, you must use the service principal or AAD Pod Identity method.
If you don't plan to take Azure disk snapshots, any method is valid.
Note: This is only required for (1) by using a Velero-specific service principal and (2) by using ADD Pod Identity.
-
Obtain your Azure Account Subscription ID:
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
-
Specify the role
There are two ways to specify the role: use the built-in role or create a custom one.
You can use the Azure built-in roleContributor
:AZURE_ROLE=Contributor
This will have subscription-wide access, so protect the credential generated with this role.
It is always best practice to assign the minimum required permissions necessary for an application to do its work.
Note: With useAAD flag you will need to provide extra permissions
Storage Blob Data Contributor
covered in point 3 of section: Create service principalHere are the minimum required permissions needed by Velero to perform backups, restores, and deletions:
- Storage Account
Back Compatability and Restic
- Microsoft.Storage/storageAccounts/listkeys/action
- Microsoft.Storage/storageAccounts/regeneratekey/action
AAD Based Auth
- Microsoft.Storage/storageAccounts/blobServices/containers/delete
- Microsoft.Storage/storageAccounts/blobServices/containers/read
- Microsoft.Storage/storageAccounts/blobServices/containers/write
- Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action
Data Actions for AAD auth
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action
- Disk Management
- Microsoft.Compute/disks/read
- Microsoft.Compute/disks/write
- Microsoft.Compute/disks/endGetAccess/action
- Microsoft.Compute/disks/beginGetAccess/action
- Snapshot Management
- Microsoft.Compute/snapshots/read
- Microsoft.Compute/snapshots/write
- Microsoft.Compute/snapshots/delete
- Microsoft.Compute/disks/beginGetAccess/action
- Microsoft.Compute/disks/endGetAccess/action
Use the following commands to create a custom role which has the minimum required permissions:
AZURE_ROLE=Velero az role definition create --role-definition '{ "Name": "'$AZURE_ROLE'", "Description": "Velero related permissions to perform backups, restores and deletions", "Actions": [ "Microsoft.Compute/disks/read", "Microsoft.Compute/disks/write", "Microsoft.Compute/disks/endGetAccess/action", "Microsoft.Compute/disks/beginGetAccess/action", "Microsoft.Compute/snapshots/read", "Microsoft.Compute/snapshots/write", "Microsoft.Compute/snapshots/delete", "Microsoft.Storage/storageAccounts/listkeys/action", "Microsoft.Storage/storageAccounts/regeneratekey/action", "Microsoft.Storage/storageAccounts/blobServices/containers/delete", "Microsoft.Storage/storageAccounts/blobServices/containers/read", "Microsoft.Storage/storageAccounts/blobServices/containers/write", "Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action", ], "DataActions" :[ "Microsoft.Storage/storageAccounts/blobServices/containers/blob/delete", "Microsoft.Storage/storageAccounts/blobServices/containers/blob/read", "Microsoft.Storage/storageAccounts/blobServices/containers/blob/write", "Microsoft.Storage/storageAccounts/blobServices/containers/blob/move/action", "Microsoft.Storage/storageAccounts/blobServices/containers/blob/add/action" ] "AssignableScopes": ["/subscriptions/'$AZURE_SUBSCRIPTION_ID'"] }'
(Optional) If you are using a different Subscription for backups and cluster resources, make sure to specify both subscriptions inside
AssignableScopes
. - Storage Account
-
Obtain your Azure Account Tenant ID:
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
-
Create a service principal.
If you'll be using Velero to backup multiple clusters with multiple blob containers, it may be desirable to create a unique username per cluster rather than the default
velero
.Create service principal and let the CLI generate a password for you. Make sure to capture the password.
(Optional) If you are using a different Subscription for backups and cluster resources, make sure to specify both subscriptions in the
az
command using--scopes
.AZURE_CLIENT_SECRET=`az ad sp create-for-rbac --name "velero" --role $AZURE_ROLE --query 'password' -o tsv \ --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID[ /subscriptions/$AZURE_BACKUP_SUBSCRIPTION_ID]`
NOTE: Ensure that value for
--name
does not conflict with other service principals/app registrations.After creating the service principal, obtain the client id.
AZURE_CLIENT_ID=`az ad sp list --display-name "velero" --query '[0].appId' -o tsv`
-
Assign Additional permissions to the service principal (For Latest Azure Plugin >=1.8.0 with useAAD=true)
If you chose the AAD route, this is an additional permissions required for the service principal to be able to access the storage account.
az role assignment create --assignee $AZURE_CLIENT_ID --role "Storage Blob Data Contributor" --scope /subscriptions/$AZURE_SUBSCRIPTION_ID
-
Now you need to create a file that contains all the relevant environment variables. The command looks like the following:
cat << EOF > ./credentials-velero AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID} AZURE_TENANT_ID=${AZURE_TENANT_ID} AZURE_CLIENT_ID=${AZURE_CLIENT_ID} AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET} AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP} AZURE_CLOUD_NAME=AzurePublicCloud EOF
available
AZURE_CLOUD_NAME
values:AzurePublicCloud
,AzureUSGovernmentCloud
,AzureChinaCloud
,AzureGermanCloud
These instructions have been adapted from the aad-pod-identity documentation.
Before proceeding, ensure that you have installed and configured aad-pod-identity for your cluster.
-
Create an identity for Velero:
export IDENTITY_NAME=velero az identity create \ --subscription $AZURE_SUBSCRIPTION_ID \ --resource-group $AZURE_RESOURCE_GROUP \ --name $IDENTITY_NAME export IDENTITY_CLIENT_ID="$(az identity show -g $AZURE_RESOURCE_GROUP -n $IDENTITY_NAME --subscription $AZURE_SUBSCRIPTION_ID --query clientId -otsv)" export IDENTITY_RESOURCE_ID="$(az identity show -g $AZURE_RESOURCE_GROUP -n $IDENTITY_NAME --subscription $AZURE_SUBSCRIPTION_ID --query id -otsv)"
If you'll be using Velero to backup multiple clusters with multiple blob containers, it may be desirable to create a unique identity name per cluster rather than the default
velero
. -
Assign the identity a role:
export IDENTITY_ASSIGNMENT_ID="$(az role assignment create --role $AZURE_ROLE --assignee $IDENTITY_CLIENT_ID --scope /subscriptions/$AZURE_SUBSCRIPTION_ID --query id -otsv)"
-
In the cluster, create an
AzureIdentity
andAzureIdentityBinding
:cat <<EOF | kubectl apply -f - apiVersion: "aadpodidentity.k8s.io/v1" kind: AzureIdentity metadata: name: $IDENTITY_NAME spec: type: 0 resourceID: $IDENTITY_RESOURCE_ID clientID: $IDENTITY_CLIENT_ID EOF cat <<EOF | kubectl apply -f - apiVersion: "aadpodidentity.k8s.io/v1" kind: AzureIdentityBinding metadata: name: $IDENTITY_NAME-binding spec: azureIdentity: $IDENTITY_NAME selector: $IDENTITY_NAME EOF
-
Create a file that contains all the relevant environment variables:
cat << EOF > ./credentials-velero AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID} AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP} AZURE_CLOUD_NAME=AzurePublicCloud EOF
available
AZURE_CLOUD_NAME
values:AzurePublicCloud
,AzureUSGovernmentCloud
,AzureChinaCloud
,AzureGermanCloud
Note: this option is not valid if you are planning to take Azure snapshots of your managed disks with Velero.
-
Obtain your Azure Storage account access key:
AZURE_STORAGE_ACCOUNT_ACCESS_KEY=`az storage account keys list --account-name $AZURE_STORAGE_ACCOUNT_ID --query "[?keyName == 'key1'].value" -o tsv`
-
Now you need to create a file that contains all the relevant environment variables. The command looks like the following:
cat << EOF > ./credentials-velero AZURE_STORAGE_ACCOUNT_ACCESS_KEY=${AZURE_STORAGE_ACCOUNT_ACCESS_KEY} AZURE_CLOUD_NAME=AzurePublicCloud EOF
available
AZURE_CLOUD_NAME
values:AzurePublicCloud
,AzureUSGovernmentCloud
,AzureChinaCloud
,AzureGermanCloud
Download Velero
Install Velero, including all prerequisites, into the cluster and start the deployment. This will create a namespace called velero
, and place a deployment named velero
in it.
In latest plugin, users can chose to use AAD route for velero to access storage account. Earlier this was done using ListKeys on the storage account which is not a recommended practice.
velero install \
--provider azure \
--plugins velero/velero-plugin-for-microsoft-azure:v1.6.0 \
--bucket $BLOB_CONTAINER \
--secret-file ./credentials-velero \
--backup-location-config useAAD="true",resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \
--snapshot-location-config apiTimeout=<YOUR_TIMEOUT>[,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID]
Limitation: The velero identity needs Reader permission alongside the "storage blob data contributor" role on the storage account. This is because the identity needs to be able to read the storage account properties to fetch the storage account's blob endpoint (azure storage accounts are no longer expected to follow the format of blob.core.windows.net with introduction of DNS zone storage accounts.). To circumvent this issue follow the steps below:
For users facing Storage Account Throttling issues You can chose to provide the Storage Account's blob endpoint directly to Velero. This will help Velero to bypass the need to fetch the storage account properties and hence the need for Reader permission on the storage account. This can be done by providing the blob endpoint in the backup-location-config as shown below:
velero install \
--provider azure \
--plugins velero/velero-plugin-for-microsoft-azure:v1.6.0 \
--bucket $BLOB_CONTAINER \
--secret-file ./credentials-velero \
--backup-location-config storageAccountURI="xxxxxx.blob.core.windows.net",useAAD="true",resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \
--snapshot-location-config apiTimeout=<YOUR_TIMEOUT>[,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID]
Note:
- ARM calls such as ListKeys and GetProperties are the most likely to get throttled, so if you are facing throttling issues, you can try to provide the blob endpoint directly to Velero post which velero will directly talk to storage.
- If you have provided the storageAccountUri, providing the resourceGroup and storageAccount fields are optional.
Migrating from ListKeys to AAD route: If you already had a velero setup using azure plugin < 1.8.0 it must be using the ListKeys approach which is not recommended. To migrate to the AAD route follow the steps below:
You need to assign your velero identity the following permissions:
az role assignment create --role "Storage Blob Data Contributor" --assignee $IDENTITY_CLIENT_ID --scope /subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/$AZURE_BACKUP_RESOURCE_GROUP/providers/Microsoft.Storage/storageAccounts/$AZURE_STORAGE_ACCOUNT_ID
After that update your velero BackupStorageLocation with the useAAD flag as shown below:
velero backup-location set default --provider azure --bucket $BLOB_CONTAINER --config useAAD="true",resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID]
Limitation: Listing storage account access key is still needed for Restic/Kopia to work as expected on Azure. Tracking Issue: #5984. The useAAD route won't accrue to them and users using kopia/restic should not remove the ListKeys permission from the velero identity.
velero install \
--provider azure \
--plugins velero/velero-plugin-for-microsoft-azure:v1.6.0 \
--bucket $BLOB_CONTAINER \
--secret-file ./credentials-velero \
--backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \
--snapshot-location-config apiTimeout=<YOUR_TIMEOUT>[,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID]
If you're using AAD Pod Identity, you now need to add the aadpodidbinding=$IDENTITY_NAME
label to the Velero pod(s), preferably through the Deployment's pod template. `
velero install \
--provider azure \
--plugins velero/velero-plugin-for-microsoft-azure:v1.6.0 \
--bucket $BLOB_CONTAINER \
--secret-file ./credentials-velero \
--backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,storageAccountKeyEnvVar=AZURE_STORAGE_ACCOUNT_ACCESS_KEY[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \
--use-volume-snapshots=false
Additionally, you can specify --use-node-agent
to enable node agent support, and --wait
to wait for the deployment to be ready.
- Specify additional configurable parameters for the
--backup-location-config
flag. - Specify additional configurable parameters for the
--snapshot-location-config
flag. - Customize the Velero installation further to meet your needs.
- Velero does not officially support for Windows containers. If your cluster has both Windows and Linux agent pool, add a node selector to the
velero
deployment to run Velero only on the Linux nodes. This can be done using the below command.kubectl patch deploy velero --namespace velero --type merge --patch '{ \"spec\": { \"template\": { \"spec\": { \"nodeSelector\": { \"beta.kubernetes.io/os\": \"linux\"} } } } }'
For more complex installation needs, use either the Helm chart, or add --dry-run -o yaml
options for generating the YAML representation for the installation.
If you are using Velero v1.6.0 or later, you can create additional Azure Backup Storage Locations that use their own credentials. These can also be created alongside Backup Storage Locations that use other providers.
It is not possible to use different credentials for additional Backup Storage Locations if you are pod based authentication such as AAD Pod Identity.
- Velero 1.6.0 or later
- Azure plugin must be installed, either at install time, or by running
velero plugin add velero/velero-plugin-for-microsoft-azure:plugin-version
, replace theplugin-version
with the corresponding value
To configure a new Backup Storage Location with its own credentials, it is necessary to follow the steps above to create the storage account and blob container to use, and generate the credentials file to interact with that blob container. You can either create a service principal or use a storage account access key to create the credentials file. Once you have created the credentials file, create a Kubernetes Secret in the Velero namespace that contains these credentials:
kubectl create secret generic -n velero bsl-credentials --from-file=azure=</path/to/credentialsfile>
This will create a secret named bsl-credentials
with a single key (azure
) which contains the contents of your credentials file.
The name and key of this secret will be given to Velero when creating the Backup Storage Location, so it knows which secret data to use.
Once the bucket and credentials have been configured, these can be used to create the new Backup Storage Location.
If you are using a service principal, create the Backup Storage Location as follows:
velero backup-location create <bsl-name> \
--provider azure \
--bucket $BLOB_CONTAINER \
--config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \
--credential=bsl-credentials=azure
Otherwise, use the following command if you are using a storage account access key:
velero backup-location create <bsl-name> \
--provider azure \
--bucket $BLOB_CONTAINER \
--config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,storageAccountKeyEnvVar=AZURE_STORAGE_ACCOUNT_ACCESS_KEY[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] \
--credential=bsl-credentials=azure
The Backup Storage Location is ready to use when it has the phase Available
.
You can check this with the following command:
velero backup-location get
To use this new Backup Storage Location when performing a backup, use the flag --storage-location <bsl-name>
when running velero backup create
.
To improve security within Azure, it's good practice to disable public traffic to your Azure Storage Account. If your AKS cluster is in the same Azure Region as your storage account, access to your Azure Storage Account should be easily enabled by a Virtual Network endpoint on your VNet.