[Tracking Issue] Implement Integraion Tests

[YJDoc2]

As per #56 , we are porting the OCI validation tests into rust. This is the tracking issue for the same.
The folder /youki_integration_test contains the currently implemented tests which are lifecyle, creation and huge_tlb

Here is the documentation for developing the integration tests:
https://youki-dev.github.io/youki/developer/e2e/rust_oci_test.html

Default

List of all test, as per https://github.com/opencontainers/runtime-tools/tree/master/validation :

• config_updates_without_affect
  • Original test code in Go: https://github.com/opencontainers/runtime-tools/blob/master/validation/config_updates_without_affect/config_updates_without_affect.go
• create(@sivchari )
  • Original test code in Go: https://github.com/opencontainers/runtime-tools/blob/master/validation/create/create.go
• default
  • Original test code in Go: https://github.com/opencontainers/runtime-tools/blob/master/validation/default/default.go
• delete (@sou1118 )
• delete_only_create_resources
• delete_resources
• hooks
• hooks_stdin
• hostname(@chermehdi)
• kill (@YamasouA )
• kill_no_effect
• killsig
• linux_cgroups_blkio(Add Cgroup V1 block IO integration test #537)
• linux_cgroups_cpus(Integration test linux cgroups cpus #462)
• linux_cgroups_devices
• linux_cgroups_hugetlb
• linux_cgroups_memory(cgroups v1 memory integration test #473)
• linux_cgroups_network(Integration test: cgroup v1 network tests, fix to memory tests #516)
• linux_cgroups_pids(Cgroup v1 pid integration tests #391)
• linux_cgroups_relative_blkio(@furon-kuina)
• linux_cgroups_relative_cpus (@posutsai)
• linux_cgroups_relative_devices
• linux_cgroups_relative_hugetlb(@kmpzr)
• linux_cgroups_relative_memory(@omprakaash)
• linux_cgroups_relative_network (@moz-sec )
• linux_cgroups_relative_pids(@kmpzr)
• linux_devices Add linux_devices test #2708 by @omprakaash
• linux_masked_paths @nayuta-ai
• linux_mount_label @Gekko0114
• linux_ns_itype(Add ns_itype test #389)
• linux_ns_nopath
• linux_ns_path
• linux_ns_path_type
• linux_process_apparmor_profile
• linux_readonly_paths (Readonly paths #582 )
• linux_rootfs_propagation
• linux_seccomp test(integration_test): port 'runtime-tools/validation/linux_seccomp' #2531 by @xiaoyang-sde
• linux_sysctl test(integration_test): port 'runtime-tools/validation/linux_sysctl' #2527 by @xiaoyang-sde
• linux_uid_mappings
• misc_props
• mounts
• pidfile(Add pidfile test #379)
• poststart
• poststart_fail
• poststop
• poststop_fail
• prestart
• prestart_fail
• process ( @sat0ken Add process test  #2968 )
• process_capabilities
• process_capabilities_fail
• process_oom_score_adj
• process_rlimits ( @sat0ken Add test process rlimits #2977)
• process_rlimits_fail
• process_user ( @sat0ken Add test process user #2978)
• root_readonly_true ( @sat0ken Add test root readonly #2976)
• start
• state

Extensions

This contains tests are not in https://github.com/opencontainers/runtime-tools/tree/master/validation , or need are not satisfactory enough.

• cgroups v1
• cgroups v2

Out of these, currently only huge_tlb is ported completely. lifecycle and create tests are implemented but need much improvements, and lifecycle tests should be split into individuals as per listed above.
Please read the README of youki_integration_test to get idea for how one should go about implementing the tests, and what utils etc are available.

[YJDoc2]

Waiting for #360 , which moves test_framework into youki_integration_test, but other than that , ready for individual test claiming.

[utam0k]

@YJDoc2 That's great! It would be a good idea to add a link to the sample implementation you implemented to the description of this issue.

[tsturzl]

I'll have some free time soon and I'd love to pitch in on this effort! I think this tooling will be a force multiplier and I think it's really impactful. Are there any pain points that I could tackle that anyone can think of that might not be a good first issue? Otherwise I'm more than happy to just jump in on cgroups controllers where I'm pretty familiar.

[YJDoc2]

Hey @tsturzl , sorry for late reply. Currently there are no obvious pain points that I know, and as you are already familiar with cgroups, it'd be great if you can work on those tests. Also if you take any test from the first list (which oci-tools have) please comment here, so we can keep track.

@utam0k I am a bit busy for few days, but after that I'm planning to work on either pid test or ns_itype test as currently both fail for runc and youki using oci-tests. Assuming that the issue is in test implementation, and not in the runtime, if we could support those tests, we can theoretically be feature equivalent with oci-tests.

[WormOn]

@utam0k i want to implement "config_updates_without_affect"

[utam0k]

@utam0k i want to implement "config_updates_without_affect"

@WormOn
Sure. I assigned you.

[tsturzl]

I'll start off with linux_cgroups_cpus, I think it will be an easy one to start off with.

[Furisto]

I do not want to take too many good first issues away, but I would like to implement at least one test to get familiar with the test framework. So I would take linux_cgroups_pids if that is alright.

[YJDoc2]

Hey @Furisto
Sure, that'd be great!

[WormOn]

@utam0k i want to implement "config_updates_without_affect"

@WormOn Sure. I assigned you.

thx

[tsturzl]

I'm nearly done with the linux_cgroups_cpus integrations tests. I'd like to take on the memory cgroups tasks. I'd really like to fix a problem that currently exists in the current oci validations with these specific tests.

[tsturzl]

Also, maybe we should limit the context of this tracking issue to cgroups v1 and just generally getting this on par with oci runtime-tools? Perhaps then we can open another tracking issue to detail out what cgroups v2 tests should look like and what needs to get implemented.

[YJDoc2]

Hey @tsturzl , sure you can take memory next 😄 , I'll assign it to you then.
It is a good idea that we use this only for tracking migration, and open separate one for cgroup v2, but I feel it'd be good to mention cgroups v2 here, until we have opened that issue, then we can link that here.

[YJDoc2]

Hey @WormOn, how is the implementation going? Let know if you need any help in working on the test implementation 👍

[tsturzl]

I got bored and started working on linux_cgroups_network, so I'll take that one also

[whereistejas]

Hi! I would like to contribute to this issue by adding a few tests, too. I have been working with rust for about 6 months, now. However, I have no exp with container runtimes, thus, would like to start with small and low priority tests.

I though I could start by adding the root_readonly_true test? Any thoughts?

[YJDoc2]

Hey @whereistejas , thanks for your interest in contribution. I would suggest that rather than starting with root_readonly_test , you might start with craete. The reason behind this is that root_readonly uses a function test_inside_runtime a util function, which I am currently implementing for another test, but facing some issues. I hope to get it worked out in some days.

The create test uses the basic functions which are already implemented in the current utils, you can check the lifecycle tests to get an idea how to implement the create tests. If you need any help or are facing some issues, you can ask them here or on the youki discord.

Thanks :)

[whereistejas]

Hi @YJDoc2, if my understanding is correct you are referring to the following create tests. I can see these tests are already implemented as part of lifecycle/container_create.rs. Please, correct me if I'm wrong.

[YJDoc2]

Hey, yes that are the tests, but they were created as part of proof-of-concept of integration tests themselves, and should be replaced in sometime for their own dedicated tests as per the runtime tools.

Or if you are fine with it, you can wait till merging of #582 which implemented the test_inside_container , then you can work on root_readonly as well.

[whereistejas]

and should be replaced in sometime for their own dedicated tests as per the runtime tools.

So, we would like to have the create tests extracted/refactored out of lifecycle into their own module?

[YJDoc2]

Hey @whereistejas , yes I meant that create tests should be refactored in there own module, as you are working on in #583 👍
Thank you for contributing!

[utam0k]

@sivchari Hi 👋 I saw your post in X ;) How about trying to implement this one?

create

This is the same test in Go:
https://github.com/opencontainers/runtime-tools/blob/master/validation/create/create.go

And this code will help you:
https://github.com/containers/youki/tree/main/tests/contest/contest/src/tests/lifecycle

[sivchari]

Hi @utam0k
I'd try to implement it. Please let assign me about it.

[furon-kuina]

Hi! This is my first time with youki.
I'm very interested in this project, and I'd like to contribute to it however I can!
Would you please suggest me with a newbie-friendly test? If there is one, I'd like to take it on!

[utam0k]

@furon-kuina Sure, a good challenge! I appreciate your challenge. I'll assign you to linux_cgroups_relative_blkio. I guess there already exists a similar implemented test case. I suggest referring them.

[furon-kuina]

@utam0k
Thank you so much! I'll take a look at it!

[YJDoc2]

Hey @furon-kuina , check #2722 once, which was opened for the linux_cgroups_relative_blkio test, but the author could not continue. You can use that as a starting point or reference to get an idea for how to implement the test. (that said, the PR has several issues, failing CI etc so you might need to fix / rewrite it)

[utam0k]

Hey @furon-kuina , check #2722 once, which was opened for the linux_cgroups_relative_blkio test, but the author could not continue. You can use that as a starting point or reference to get an idea for how to implement the test. (that said, the PR has several issues, failing CI etc so you might need to fix / rewrite it)

Ops, thanks for your follow-up 🙏

[furon-kuina]

@YJDoc2 @utam0k
Thanks for the help! I really appreciate it.
I'll study the youki User and Developer Documentation and get started with the implementation.

[posutsai]

Hi @utam0k, I would like to take the linux_cgroups_relative_cpus. Please assign me if possible.

[YJDoc2]

@posutsai assigned to you 👍

[posutsai]

Hi @YJDoc2 and @utam0k,
After a few day survey, seems like testing linux_cgroups_relative_cpus relates to the API of cgroup v1, referring the comparison table. Given my Ubuntu is using cgroup v2, Is it possible to make cgroup v1 and v2 coexist? or is there any tool to simulate the behavior? Thank you.

[YJDoc2]

Hey, so as far as I know, recent Ubuntu versions have dropped support of cgroups v1 by default, you need to do certain kernel feature enabling to enable cgroups v1 on them. I think ubuntu 20.04 has cgroups v1 enabled in them, and we use that in our ci -check #2645 . An easier way for your testing can be to create a vagrant box with this version, mount the youki dir in it, compile your things on host systems, so you don't have to install rust and build deps in the box, but run the tests in the box instead of host (so you'll need the runtime deps).

I have also opened #2890 for standardizing the test envs via vagrant boxes.

[nayuta-ai]

Hi! This is my first time with youki. I am interested in contributing to the youki project!
I would like to know if it would be possible to assign me to the linux_masked_paths.

[utam0k]

Hi! This is my first time with youki. I am interested in contributing to the youki project! I would like to know if it would be possible to assign me to the linux_masked_paths.

@nayuta-ai Sure! I've assigned you. Thanks!

[sou1118]

Hi @utam0k and @YJDoc2
I want to work on delete job. Could you assign me? This is my first OSS and Youki contribution.

[YJDoc2]

Hey @sou1118 , I have assigned you the delete test, just check once if it has already been implemented or not (it is possible that we might have missed crossing it off the list). Also welcome to Youki, and OSS as whole! Hope you'll have fun and find the contributing interesting. Ping me if you need any help!

[moz-sec]

Hi @YJDoc2 and @utam0k
I would like to be assigned to linux_cgroups_relative_network. Could you please assign it to me?
I'm looking forward to making my first OSS contribution!

[YamasouA]

@YJDoc2 @utam0k
Hi.
I'm intereseted in this Issue.
Can you assign any of the test cases to me?

[YJDoc2]

@moz-sec , assigned the linux_cgroups_relative_network to you. Looking forward to you PR!

@YamasouA , I have assigned the kill test to you, please take a look. If you want some other let me know and I can re-assign.

In case either of you need any help while working on this, please feel free to tag/ping me here :)