Skip to content

Commit

Permalink
kubelet: fix /stats/summary endpoint on Windows when init-container…
Browse files Browse the repository at this point in the history
…s are present on the node

Following changes in kubernetes#87730, Kubelet is directly hcsshim to gather stats.
However, unlike `docker stats` API that was used before, hcsshim does not
keep information about exited containers.

When the Kubelet lists containers (`docker_container.go:ListContainers()`),
it sets `All: true`, retrieving non-running containers.

When docker stats is called with such container id, it'll return a valid JSON
with all values set to 0. The non-running containers are filtered later on in the process.

When the hcsshim is called with such container id, it'll return an error, effectively
stopping the stats retrieval for all containers.
  • Loading branch information
vboulineau committed May 14, 2020
1 parent 2db6ec1 commit b1fe9ab
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 4 deletions.
5 changes: 3 additions & 2 deletions pkg/kubelet/dockershim/docker_stats.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,9 @@ func (ds *dockerService) ListContainerStats(ctx context.Context, r *runtimeapi.L
if err != nil {
return nil, err
}

stats = append(stats, containerStats)
if containerStats != nil {
stats = append(stats, containerStats)
}
}

return &runtimeapi.ListContainerStatsResponse{Stats: stats}, nil
Expand Down
8 changes: 7 additions & 1 deletion pkg/kubelet/dockershim/docker_stats_windows.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,13 @@ func (ds *dockerService) getContainerStats(containerID string) (*runtimeapi.Cont

hcsshim_container, err := hcsshim.OpenContainer(containerID)
if err != nil {
return nil, err
// As we moved from using Docker stats to hcsshim directly, we may query HCS with already exited container IDs.
// That will typically happen with init-containers in Exited state. Docker still knows about them but the HCS does not.
// As we don't want to block stats retrieval for other containers, we only log errors.
if !hcsshim.IsNotExist(err) && !hcsshim.IsAlreadyStopped(err) {
klog.Errorf("Error opening container (stats will be missing) '%s': %v", containerID, err)
}
return nil, nil
}
defer func() {
closeErr := hcsshim_container.Close()
Expand Down
12 changes: 11 additions & 1 deletion test/e2e/windows/kubelet_stats.go
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,17 @@ func newKubeletStatsTestPods(numPods int, image imageutils.Config, nodeName stri
},
},
},

InitContainers: []v1.Container{
{
Image: image.GetE2EImage(),
Name: podName,
Command: []string{
"powershell.exe",
"-Command",
"sleep -Seconds 1",
},
},
},
NodeName: nodeName,
},
}
Expand Down

0 comments on commit b1fe9ab

Please sign in to comment.