About CRI metrics of virtlet-based pod.

[miaoyq]

Now, k8s mainly uses cAdvisor to collect container performance metrics, and the runtimes that are not included in the cAdvisor, if they have implemented ContainerStats of CRI, also can collect some of metrics.

At present, we need to autoscale Virtlet Pods according to the resource usage of VM. However, virtlet has not yet implemented the ContainerStats of CRI, and there is no Virtlet Pod metrics data collection in cAdvisor also.

Are we going to implement the CRI metrics in the near future?
At present, how can I get the ContainerStats(VM metrics) of Virtlet-Based Pod?

In addition, k8s achieves the Images garbage collection by collecting the disk usage of the container/image. Is there any way to collect the Virtlet image?

Hope to get someone's help, thank you very much!

[guangxuli]

@jellonek Any suggestion?

[jellonek]

We don't have at the moment this topic on our agenda, but contributions are welcomed.

CPU stats could be probably gathered from libvirt in similar manner to https://libvirt.org/docs/libvirt-appdev-guide-python/en-US/html/libvirt_application_development_guide_using_python-Guest_Domains-Monitoring-vCPU.html
Memory is also very similar: https://libvirt.org/docs/libvirt-appdev-guide-python/en-US/html/libvirt_application_development_guide_using_python-Guest_Domains-Monitoring-Memory.html
At the moment I don't know if they have equivalents in used by us https://github.com/libvirt/libvirt-go-xml

As for image, imo we should return only Allocation value of root disk.

[jellonek]

Ok, we have that on our agenda (i missed it) and probably in near future i'll start working on that.

[jellonek]

Filesystem stats were implemented in #735
Runtime part is in PoC form in #743 - and should be finished in more proper way soon.

Can you verify if that works for you, by testing mirantis/virtlet:jell_rtstats image from docker hub? (it's poc because it has a little simplifications if few places and lacks a proper coverage which i'm doing right now, but core functionality should work).

[miaoyq]

@jellonek OK, Thanks for this feature. :-)

[jellonek]

All of that is already merged and included in last release.

[lukaszo]

@MiaoY please note that when Virtlet is used together with docker you still will not be able to get VMs stats using Kubelet Summary API. It's because how kubelet is designed.

If you want to get Virtlet VMs stats using Summary API then you need to switch to containerd. You can also just directly call ContainerStats then it will work everywhere.

[jellonek]

Directly - means on the node using CRI call on virtlet unix socket.