Data mover doesn't work if label-selector applied at restore

[rob-1126]

What steps did you take and what happened:
I tried to use -l on my restore to only restore a few labeled resources but it doesn't work.

Restore logs show time="2024-03-15T19:16:51Z" level=error msg="Namespace , resource restore error: error preparing persistentvolumeclaims//data--registry-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: rx43: no DataUpload result cm found with labels velero.io/pvc-namespace-name=.data--registry-0,velero.io/restore-uid=3d270f78-e106-4431-8aa2-2d7b6bc8c418,velero.io/resource-usage=DataUpload" logSource="pkg/controller/restore_controller.go:580" restore=velero/rx43

I re-ran without the -l component=registry on the restore and it works fine.

What did you expect to happen:

For it to work fine with the label-selector too.

The following information will help us better understand what's going on:

Environment:

• Velero version (use velero version): 1.13.0 on both client and server
• Velero features (use velero client config get features): EnableCSI
• Kubernetes version (use kubectl version): Client: 1.29.0, Server 1.26.12
• Kubernetes installer & version: AKS, unknown
• Cloud provider or hardware configuration: AKS + Workload Identity
• OS (e.g. from /etc/os-release): N/A

image: velero/velero-plugin-for-microsoft-azure:v1.9.0
image: velero/velero-plugin-for-csi:v0.7.0
image: docker.io/velero/velero:v1.13.0

Note there are a LOT of backups (roughly 10k) and this is on Azure w/ Workload Identity.

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[rob-1126]

Another user reported a similar issue on 1.12.x on #6709

[Lyndon-Li]

Looks like we need to further take care of MustHave resources during include-exclude during restore, will fix it.

[ywk253100]

Still got the same issue with the latest dev build on the main branch

Here is the workload:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd0-pv-claim-01
  labels:
    a: b
spec:
  storageClassName: "default"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd0-pv-claim-02
spec:
  storageClassName: "default"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd0-pv-claim-03
spec:
  storageClassName: "default"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: etcd0
    etcd_node: etcd0
    a: b
  name: etcd0
  annotations:
    "pre.hook.backup.velero.io/container": "etcd0"
    "pre.hook.backup.velero.io/command": "[\"/usr/local/bin/etcd\", \"-h\"]"
spec:
  volumes:
    - name: etcd0-storage-01
      persistentVolumeClaim:
        claimName: etcd0-pv-claim-01
    - name: etcd0-storage-02
      persistentVolumeClaim:
        claimName: etcd0-pv-claim-02
    - name: etcd0-storage-03
      persistentVolumeClaim:
        claimName: etcd0-pv-claim-03
  containers:
    - command:
        - sleep
        - 1h
      image: quay.io/coreos/etcd:latest
      volumeMounts:
        - mountPath: "/etcd0.etcd.01"
          name: etcd0-storage-01
        - mountPath: "/etcd0.etcd.02"
          name: etcd0-storage-02
        - mountPath: "/etcd0.etcd.03"
          name: etcd0-storage-03
      name: etcd0
      ports:
        - containerPort: 2379
          name: client
          protocol: TCP
        - containerPort: 2380
          name: server
          protocol: TCP
  restartPolicy: Always
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd1-pv-claim-01
spec:
  storageClassName: "default"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd1-pv-claim-02
spec:
  storageClassName: "default"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd1-pv-claim-03
spec:
  storageClassName: "default"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: etcd1
    etcd_node: etcd1
  name: etcd1
  annotations:
    "pre.hook.backup.velero.io/container": "etcd1"
    "pre.hook.backup.velero.io/command": "[\"/usr/local/bin/etcd\", \"-h\"]"
spec:
  volumes:
    - name: etcd1-storage-01
      persistentVolumeClaim:
        claimName: etcd1-pv-claim-01
    - name: etcd1-storage-02
      persistentVolumeClaim:
        claimName: etcd1-pv-claim-02
    - name: etcd1-storage-03
      persistentVolumeClaim:
        claimName: etcd1-pv-claim-03
  containers:
    - command:
        - sleep
        - 1h
      image: quay.io/coreos/etcd:latest
      volumeMounts:
        - mountPath: "/etcd1.etcd.01"
          name: etcd1-storage-01
        - mountPath: "/etcd1.etcd.02"
          name: etcd1-storage-02
        - mountPath: "/etcd1.etcd.03"
          name: etcd1-storage-03
      name: etcd1
      ports:
        - containerPort: 2379
          name: client
          protocol: TCP
        - containerPort: 2380
          name: server
          protocol: TCP
  restartPolicy: Always

And the debug bundle:
bundle-2024-04-28-08-28-34.tar.gz