When gpu lost, scheduler will assign pod to wrong node

[zhiyu0729]

What happened:
After pod scheduled, it create fail.

Status:         Failed
Reason:         UnexpectedAdmissionError
Message:        Pod Allocate failed due to requested number of devices unavailable for nvidia.com/gpu. Requested: 1, Available: 0, which is unexpected

What you expected to happen:
Pod Shouldn't schedule to this Node.

How to reproduce it (as minimally and precisely as possible):

1. Node with 4 GPU

Capacity:
  nvidia.com/gpu:     4
Allocatable:
  nvidia.com/gpu:     4

2. Start Pod With 4 GPU in this Node.
3. One GPU error detect by device-plugin, then.

Capacity:
  nvidia.com/gpu:     4
Allocatable:
  nvidia.com/gpu:     3

4. Start Volcano Scheduler now.
5. Create Pod With 1 GPU.
6. Pod scheduled to this node, but node no idle gpu resource.

Anything else we need to know?:

Environment:

• Volcano Version: e83119
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[shinytang6]

What happened: After pod scheduled, it create fail.

Status:         Failed
Reason:         UnexpectedAdmissionError
Message:        Pod Allocate failed due to requested number of devices unavailable for nvidia.com/gpu. Requested: 1, Available: 0, which is unexpected

What you expected to happen: Pod Shouldn't schedule to this Node.

How to reproduce it (as minimally and precisely as possible):

1. Node with 4 GPU

Capacity:
  nvidia.com/gpu:     4
Allocatable:
  nvidia.com/gpu:     4

1. Start Pod With 4 GPU in this Node.
2. One GPU error detect by device-plugin, then.

Capacity:
  nvidia.com/gpu:     4
Allocatable:
  nvidia.com/gpu:     3

1. Start Volcano Scheduler now.
2. Create Pod With 1 GPU.
3. Pod scheduled to this node, but node node idle gpu resource.

Anything else we need to know?:

Environment:

• Volcano Version: e83119
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

l am a little confused, since the first pod (4 GPU) occupies the node first, why is the pod (1 GPU) can still be scheduled to the node, did l miss sth ?

[zhiyu0729]

What happened: After pod scheduled, it create fail.

Status:         Failed
Reason:         UnexpectedAdmissionError
Message:        Pod Allocate failed due to requested number of devices unavailable for nvidia.com/gpu. Requested: 1, Available: 0, which is unexpected

What you expected to happen: Pod Shouldn't schedule to this Node.
How to reproduce it (as minimally and precisely as possible):

1. Node with 4 GPU

Capacity:
  nvidia.com/gpu:     4
Allocatable:
  nvidia.com/gpu:     4

1. Start Pod With 4 GPU in this Node.
2. One GPU error detect by device-plugin, then.

Capacity:
  nvidia.com/gpu:     4
Allocatable:
  nvidia.com/gpu:     3

1. Start Volcano Scheduler now.
2. Create Pod With 1 GPU.
3. Pod scheduled to this node, but node node idle gpu resource.

Anything else we need to know?:
Environment:

• Volcano Version: e83119
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

l am a little confused, since the first pod (4 GPU) occupies the node first, why is the pod (1 GPU) can still be scheduled to the node, did l miss sth ?

The problem is here:

When schedule start , sync scheduler cache, tigger AddPod handler, when pod bind to node tasks list:

volcano/pkg/scheduler/api/node_info.go

Lines 336 to 343 in b119114

	func (ni *NodeInfo) allocateIdleResource(ti *TaskInfo) error {
	if ti.Resreq.LessEqual(ni.Idle, Zero) {
	ni.Idle.Sub(ti.Resreq)
	return nil
	}
	return fmt.Errorf("selected node NotReady")
	}

first pod Resreq 4 gpu, node has 3 gpu can allocate, it raise error.

volcano/pkg/scheduler/cache/event_handlers.go

Lines 187 to 205 in a7ecd08

	// AddPod add pod to scheduler cache
	func (sc *SchedulerCache) AddPod(obj interface{}) {
	pod, ok := obj.(*v1.Pod)
	if !ok {
	klog.Errorf("Cannot convert to *v1.Pod: %v", obj)
	return
	}
	sc.Mutex.Lock()
	defer sc.Mutex.Unlock()
	err := sc.addPod(pod)
	if err != nil {
	klog.Errorf("Failed to add pod <%s/%s> into cache: %v",
	pod.Namespace, pod.Name, err)
	return
	}
	klog.V(3).Infof("Added pod <%s/%v> into cache.", pod.Namespace, pod.Name)
	}

but in AddPod it just ingore this pod, so node idle resource is 3 gpu.