Don't fail permenantly when nvml isn't installed

[dashpole]

Reverts a small part of #2419, which changed how the nvidia collector is setup. NVIDIA driver and library installation is often done via daemonset. If cadvisor starts before the library is installed, youdon't get any GPU metrics.

This should fix the problem by returning a manager instead of a noop. Setup will be retried when GetCollector is called.

cc @RenaudWasTaken @iwankgb @bobbypage

I'd like to cherrypick this back to v0.37 as well.

[iwankgb]

LGTM

[bobbypage]

LGTM, thanks for the fix. Agree, let's cherrypick this to v0.37

[liggitt]

Please make sure this gets in the v0.38.x stream as well for kubernetes 1.20

[liggitt]

Setup will be retried when GetCollector is called.

Is that true? I see this, which seems to make GetCollector no-op if setup failed:

// GetCollector returns a collector that can fetch NVIDIA gpu metrics for NVIDIA devices
// present in the devices.list file in the given devicesCgroupPath.
func (nm *nvidiaManager) GetCollector(devicesCgroupPath string) (stats.Collector, error) {
	nc := &nvidiaCollector{}

	if !nm.devicesPresent {
		return &stats.NoopCollector{}, nil
	}

[dashpole]

If there are no GPUs on the node, it returns a no-op. We still fail permanently when there are no GPUs on the node. We no longer fail permanently when nvml is missing

[ruiwen-zhao]

nm.devicesPresent will be true in this case because it is set before initializing NVML

cadvisor/accelerators/nvidia.go

Line 74 in 65fa5b4

nm.devicesPresent = true

[liggitt]

Got it, detectDevices was working, initializeNVML was not. Sorry for the noise.