After update velero-plugin-for-aws - backup status finalizing never completed

[M-A-X-I-M]

We updated:
velero.image.tag: v1.13.0 -> v1.13.1
velero-plugin-for-aws: v1.9.0 -> v1.9.1
I updated helm chart: 6.3.0 -> 6.4.0

What did you expect to happen:
Complete a shedule backup at night or complete a manual triggered backup, but the backups are stuck in status "finalizing"!

apiVersion: velero.io/v1
kind: Backup
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"velero.io/v1","kind":"Schedule","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"velero","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"velero","argocd.argoproj.io/instance":"velero","helm.sh/chart":"velero-6.4.0"},"name":"velero-scheduled-backup","namespace":"velero"},"spec":{"schedule":"0
1 * *
*","template":{"includedNamespaces":["emp","fluentd","kube-prometheus-stack","loki-stack","mail","openldap-alpine","vault"],"ttl":"168h"},"useOwnerReferencesInBackup":true}}
velero.io/resource-timeout: 10m0s
velero.io/source-cluster-k8s-gitversion: v1.25.3-r0-25.2.19
velero.io/source-cluster-k8s-major-version: '1'
velero.io/source-cluster-k8s-minor-version: 25+
creationTimestamp: '2024-05-21T01:00:50Z'
generation: 24
labels:
app.kubernetes.io/instance: velero
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: velero
argocd.argoproj.io/instance: velero
helm.sh/chart: velero-6.4.0
velero.io/schedule-name: velero-scheduled-backup
velero.io/storage-location: default
managedFields:
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:velero.io/resource-timeout: {}
f:velero.io/source-cluster-k8s-gitversion: {}
f:velero.io/source-cluster-k8s-major-version: {}
f:velero.io/source-cluster-k8s-minor-version: {}
f:labels:
.: {}
f:app.kubernetes.io/instance: {}
f:app.kubernetes.io/managed-by: {}
f:app.kubernetes.io/name: {}
f:argocd.argoproj.io/instance: {}
f:helm.sh/chart: {}
f:velero.io/schedule-name: {}
f:velero.io/storage-location: {}
f:ownerReferences:
.: {}
k:{"uid":"d55a28b9-da0d-4b0c-b83e-4d73425a923a"}: {}
f:spec:
.: {}
f:csiSnapshotTimeout: {}
f:defaultVolumesToFsBackup: {}
f:hooks: {}
f:includedNamespaces: {}
f:itemOperationTimeout: {}
f:metadata: {}
f:snapshotMoveData: {}
f:storageLocation: {}
f:ttl: {}
f:status:
.: {}
f:expiration: {}
f:formatVersion: {}
f:hookStatus: {}
f:phase: {}
f:progress:
.: {}
f:itemsBackedUp: {}
f:totalItems: {}
f:startTimestamp: {}
f:version: {}
manager: velero-server
operation: Update
time: '2024-05-21T01:04:27Z'
name: velero-scheduled-backup-20240521010050
namespace: velero
ownerReferences:
- apiVersion: velero.io/v1
controller: true
kind: Schedule
name: velero-scheduled-backup
uid: d55a28b9-da0d-4b0c-b83e-4d73425a923a
resourceVersion: '508126532'
uid: 7e099c1d-bff4-4828-af2c-e250a404ac70
selfLink: >-
/apis/velero.io/v1/namespaces/velero/backups/velero-scheduled-backup-20240521010050
status:
expiration: '2024-05-28T01:00:50Z'
formatVersion: 1.1.0
hookStatus: {}
phase: Finalizing
progress:
itemsBackedUp: 758
totalItems: 758
startTimestamp: '2024-05-21T01:00:50Z'
version: 1
spec:
csiSnapshotTimeout: 10m0s
defaultVolumesToFsBackup: true
hooks: {}
includedNamespaces:
- emp
- fluentd
- kube-prometheus-stack
- loki-stack
- mail
- openldap-alpine
- vault
itemOperationTimeout: 4h0m0s
metadata: {}
snapshotMoveData: false
storageLocation: default
ttl: 168h0m0s

log of the pod:

time="2024-05-21T09:54:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240316010018 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158"
time="2024-05-21T09:54:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240322010024 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
time="2024-05-21T09:54:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240322010024 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158"
time="2024-05-21T09:54:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240319010021 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
time="2024-05-21T09:54:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240319010021 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158"
time="2024-05-21T09:55:09Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:141"
time="2024-05-21T09:55:09Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
time="2024-05-21T09:55:35Z" level=error msg="Error getting backup item operations" backup=velero/velero-scheduled-backup-20240519010048 controller=backup-finalizer error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:412" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="pkg/controller/backup_finalizer_controller.go:143"
time="2024-05-21T09:55:35Z" level=error msg="Reconciler error" Backup="{"name":"velero-scheduled-backup-20240519010048","namespace":"velero"}" controller=backup controllerGroup=velero.io controllerKind=Backup error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:412" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:123" name=velero-scheduled-backup-20240519010048 namespace=velero reconcileID=""341dc73f-337f-45a9-bed9-7965dd1b1ead""
time="2024-05-21T09:55:51Z" level=info msg="Found 7 backups in the backup location that do not exist in the cluster and need to be synced" backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:136"
time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240319010021 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240319010021 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158"
time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240320010022 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240320010022 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158"
time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240316010018 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240316010018 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158"
time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240321010023 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240321010023 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158"
time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240317010019 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240317010019 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158"
time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240322010024 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240322010024 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158"
time="2024-05-21T09:55:51Z" level=info msg="Attempting to sync backup into cluster" backup=velero-scheduled-backup-20240318010020 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
time="2024-05-21T09:55:51Z" level=error msg="Error getting backup metadata from backup store" backup=velero-scheduled-backup-20240318010020 backupLocation=velero/default controller=backup-sync error="yaml: line 3: mapping values are not allowed in this context" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:315" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:158"
time="2024-05-21T09:56:09Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:141"
time="2024-05-21T09:56:09Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
I0521 09:56:16.589529 1 request.go:697] Waited for 1.04589123s due to client-side throttling, not priority and fairness, request: GET:https://10.247.0.1:443/apis/config.k8s.io/v1beta1?timeout=32s
time="2024-05-21T09:56:22Z" level=error msg="Error getting backup item operations" backup=velero/velero-scheduled-backup-20240520010049 controller=backup-finalizer error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:412" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="pkg/controller/backup_finalizer_controller.go:143"
time="2024-05-21T09:56:22Z" level=error msg="Reconciler error" Backup="{"name":"velero-scheduled-backup-20240520010049","namespace":"velero"}" controller=backup controllerGroup=velero.io controllerKind=Backup error="gzip: invalid header" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:412" error.function=github.com/vmware-tanzu/velero/pkg/persistence.decode logSource="/go/pkg/mod/github.com/bombsimon/logrusr/v3@v3.0.0/logrusr.go:123" name=velero-scheduled-backup-20240520010049 namespace=velero reconcileID=""7d04fff7-6b76-476c-af0d-46cbf09d2c5f""

Environment:
k8s

[M-A-X-I-M]

i have triggered a manul backup and got this as failed reason:

status:
expiration: '2024-06-20T10:32:36Z'
failureReason: >-
rpc error: code = Unknown desc = error putting object
backups/velero-manual-backup/velero-manual-backup.tar.gz: upload multipart
failed, upload id: 0000018F9ABA098949D0105FA03DB64A, cause: operation error
S3: UploadPart, https response error StatusCode: 400, RequestID:
0000018F9ABA09D449CAD4C0D18AF90D, HostID:
32AAAQAAEAABAAAQAAEAABAAAQAAEAABCTV75akIR2oKeSSm1d6sVCh1j7MPruom, api error
XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does
not match what was computed.

[M-A-X-I-M]

when i downgrade the plugin-for-aws image to version v1.9.0 the backups are working