Organize integrationtest-related directories

[utam0k]

The runtime-tools tests, the HCR integration tests that we will be adding, and the youki integration tests are scattered all over the place, making it difficult to understand.
I want to organize these directories to make them easier to understand.
Incidentally, this is because I did not design it properly the first time. Sorry...

[utam0k]

Current directory structure

.
├── build.sh # build script with youki integration tests
├── Cargo.lock
├── Cargo.toml
├── CODE-OF-CONDUCT.md
├── crates
│   ├── integration_test # youki integration test
│   ├── libcgroups
│   ├── libcontainer
│   ├── liboci-cli
│   ├── libseccomp
│   ├── test_framework # a framework for youki integration test
│   └── youki
├── docs
├── hack
├── integration_test # integration test for opencontainers/runtime-tools
│   └── src
├── integration_test.sh # the script for running integration tests of opencontainers/runtime-tools, not running youki integration tests
├── LICENSE
├── README.md
├── runtimetest # this is the bin which runs the youki integration test inside the container process
│   ├── Cargo.lock
│   ├── Cargo.toml
│   ├── README.md
│   ├── src
│   └── target
├── SECURITY.md
├── tutorial
├── Vagrantfile
├── Vagrantfile.root
└── youki_integration_test

[utam0k]

@YJDoc2
I'd like to hear your option.

[YJDoc2]

Hey, I have some ideas, however, I request a couple of days to try it out in a test crate, so I can understand what works and what doesn't better. I will get back to you after I'm done with that.

An option that I'd be focusing on is separating the integration test related crates and dirs from the youki / container related dirs. Currently both of them are in same workspace, and even though that does not seem particularly bad, I don't feel it is particularly good either. I hope I can come up with some better idea to structure them. That way if in future we decide to have a dedicated repo for them like opencontainers/runtime-tools, they would be easier to move out in it.

[YJDoc2]

@utam0k
I think something along this would be a good idea

.
├── CODE-OF-CONDUCT.md
├── youki_crates        # This will contain only the core youki related crates
│   ├── Cargo.lock
│   ├── Cargo.toml      # This will contain other crates as workspace members
│   ├── target/         # common for all 
│   ├── libcgroups
│   ├── libcontainer
│   ├── liboci-cli
│   ├── libseccomp
│   ├── youki
│   └── build.sh             # This will only build the binaries and copy from target to this dir
│
├── youki_integration_test # contains only the rust integration test related crates
│   │
│   │# each of this dir is a dedicated crate, with each having its own Cargo.toml, Cargo.lock
│   │# and target/ etc.
│   │
│   ├── integration_test     # youki integration test
│   ├── test_framework       # a framework for youki integration test
│   ├── runtimetest          # this is the bin which runs the youki integration test inside the container process
│   └── build.sh             # This will only build the binaries and copy from target to this dir
│
├── oci_integration_test # integration test for opencontainers/runtime-tools
│   └── src
│
├── docs
├── hack
├── run_oci_tests.sh         # the script for running integration tests of opencontainers/runtime-tools,
│                           #  not running youki integration tests (Formerly integration_test.sh)
│
├── run_rust_integration_tests.sh      # the script for running youki integration tests
├── LICENSE
├── README.md
├── SECURITY.md
├── tutorial
├── Vagrantfile
└── Vagrantfile.root

This will separate the three different concerns (youki, oci-tests, rust-integration-tests) into their own separate dirs, as well as separate youki_integration_tests crates from the workspace of youki crates.

Note that in this, the build.sh are scoped, i.e. a build script in a directory will be only concerned with building the stuff in that directory, nothing else. For eg. in youki_crate, the build script will build the youki binary and copy it from youki_crates/target/ to youki_crates/ . It does not have any other responsibility.

This will make a difference specially in the youki_integration_tests, where there won't be any script to run tests in youki_integration_tests/ dir, youki_integration_tests/build.sh will only build two binaries and copy them to the youki_integration_tests/ dir. The responsibility of copying those from youki_integration_tests/ , as well as building youki and copying that binary will be of some script ( here run_rust_integration_tests.sh) in the root dir.

I believe scoping like this will simplify the build process, as each script will have a specific responsibility, as well as a specific scope/ things that it will handle.

Overall, this follows scheme of

├── workspace1
├── workspace2
├── standalone_thing 
├── docs
├── some_script.sh       
└── other files

Let me know what you think of this.
Thanks :)

[utam0k]

@YJDoc2
That's a good idea!

I believe scoping like this will simplify the build process, as each script will have a specific responsibility, as well as a specific scope/ things that it will handle.

I wonder if oci-tests and rust-integration-tests are of the same concern. So how about creating intefation_test directory and then creating these two directories inintegration_test directory. This way, I think we can take care of running contained and cri-o integration tests using youki.
#585
#584

This will separate the three different concerns (youki, oci-tests, rust-integration-tests) into their own separate dirs, as well as separate youki_integration_tests crates from the workspace of youki crates.

What do you think? I want to hear your opinion.

[YJDoc2]

Hey @utam0k , can you please once give some link of the containerd tests (not in our repo, but maybe in some other runtime)

That makes sense , as those do fall in the same concern, and as we go on, we might add integration tests for something else as well, which might be useful to determine if youki is compatible with that or not ( eg containerd and cri-o). I also have a bit of worry that it might become more complex to develop the rust integration tests initially, but that might be less of an issue as the time goes on, which is why I'd like to see the containerd tests, if they exist in some other repo.

I wonder if oci-tests and rust-integration-tests are of the same concern. So how about creating intefation_test directory and then creating these two directories inintegration_test directory. This way, I think we can take care of running contained and cri-o integration tests using youki.

Apart from that, do you have any other thoughts on my suggested method? if not, I feel with your suggested changes, this new structure would be a good one.

[utam0k]

@YJDoc2
I am working in progress now, but so you can see my previous challenge here. I guess it will probably same way in this challenge which I'm working.

Hey @utam0k , can you please once give some link of the containerd tests (not in our repo, but maybe in some other runtime)

[utam0k]

It feels fine for me 💯

Apart from that, do you have any other thoughts on my suggested method? if not, I feel with your suggested changes, this new structure would be a good one.

[YJDoc2]

Hey, sorry for the late reply, I was a bit busy.

I think then making a single dir for all integration test related stuff, and then making subdirs for respective tests seem a good idea, we should go ahead with that. 👍 👍

I would like to re-iterate again on the scoping the shell scripts, so even in the integration tests, each dir will have its own script which will have responsibility of building tests and maybe taking path of runtime and running the tests on the path. They should not have responsibility of building youki or copying youki or anything that falls out of their own dir, as then they will depend on the changes that are outside their own dir, which does not seem a good idea. The actual responsibility of calling the build script and then running the tests should be of a script in either a root dir, or maybe we can make a dedicated dir for keeping scirpts like this.

Sorry again for late reply 😓
Also if you want me to help or do anything related to this or make a PR, let me know in this issue, I'd be happy to help 👍