Multisystem testing

[tsturzl]

The integration tests can pass or fail depending on the system it's actually running on, one example of this is the network cgroups. On some systems the network cgroups mount into 2 different directories "net_prio" and "net_cls", on others they mount into one directory "net_cls,net_prio" or "net_prio,net_cls". This means that on some systems integration tests will pass, and others they will fail and Youki will not run properly. I believe this is specific to how distributions are setup. So it might be worth while to setup some kind of testing in 3-4 of the most popular distributions. These tests would be setup locally only, since our current CI cannot handle it at the moment.

My thoughts on which systems to test is:

• Redhat/Fedora
• Ubuntu LTS
• Latest stable Debian
• Alpine Linux, this is a good candidate to also test a system that doesn't use systemd

This test tooling will likely use a VM, something like vagrant might be most useful here. I don't believe testing container runtimes in a container will work all that well. Another choice might be something like Amazon's Firecracker.

In addition to providing multiple test environments this could also allow Windows and Mac users to more easily test changes.

[YJDoc2]

Hey @tsturzl , can you tell me what direction you have in mind for this? I was working on #82 and when trying to run tests, the cgroup memory limit tests failed, as the kernel I was using was not compiled with that option. But aside from just failing the test, I am now not much confident in making changes required for #82, as I cannot test on my local machine, I might introduce changes that might break the youki in a place I didn't expect, and I would have no way of testing that other than pushing to github and running the action on my fork. I tested https://github.com/nektos/act which lets you test github actions locally, but that didn't work out for me either as somehow the configuration selected by it for the run does not match the one used by github and it gives issues for running rust and go commands.

If you have any idea how we can not only extend the testing to multiple systems, but also somehow allow to test integration locally, I would like to check that out and see if I can make any progress, as resolving/improving this issue will give all contributors confidence that the changes they have made does not break the system.

One though I had was to convert the OCI test in rust, and use those as cargo tests, but I think that would be a huge project by itself, and also we will need to keep up with the changes that are made in the OCI test, which will be extra work. If you can suggest some other ways, maybe making container configuration using docker compose etc to test this which can be used locally as well as on multiple systems, please let me know.

Thanks.

[tsturzl]

@YJDoc2 I did get the base systems mostly setup with the exception of alpine Linux on my fork. Unfortunately I haven't had the time to finish this yet. I'm hoping maybe tomorrow evening I'll have some focus time for this.