Add Mdbook documentation

.github/workflows/docs.yaml

@@ -0,0 +1,38 @@
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  changes:
+    runs-on: ubuntu-latest
+    outputs:
+      dirs: ${{ steps.filter.outputs.changes }}
+    steps:
+      - uses: actions/checkout@v2
+      - uses: dorny/paths-filter@v2
+        id: filter
+        with:
+          filters: |
+            docs: docs/**
+  deploy:
+    needs: [changes]
+    if: ${{ !contains(needs.changes.outputs.dirs, '[]') }}
+    runs-on: ubuntu-latest
+    concurrency:
+      group: ${{ github.workflow }}-${{ github.ref }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: Setup mdBook
+        uses: peaceiris/actions-mdbook@v1
+        with:
+          mdbook-version: 'latest'
+      - name: Build mdbook
+        working-directory: ./docs
+        run: mdbook build
+      - name: Deploy
+        uses: peaceiris/actions-gh-pages@v3
+        if: ${{ github.ref == 'refs/heads/main' }}
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./docs/book

docs/.gitignore

@@ -0,0 +1 @@
+book

docs/book.toml

@@ -0,0 +1,6 @@
+[book]
+authors = ["Yashodhan Joshi"]
+language = "en"
+multilingual = false
+src = "src"
+title = "Youki User and Developer Documentation"

docs/doc-draft.md

@@ -1,4 +1,4 @@
-_This is a draft for a high level documentation of Youki. After it is finished this is intended to explain how control flow and high level functioning of Youki happens for development purposes.
+\_This is a draft for a high level documentation of Youki. After it is finished this is intended to explain how control flow and high level functioning of Youki happens for development purposes.
 
 ## Some reference links
 
@@ -78,7 +78,6 @@ The main youki process will set up pipes used as message passing and synchroniza
 
 Note: clone(2) offers us the ability to enter into user and pid namespace by creatng only one process. However, clone(2) can only create new pid namespace, but cannot enter into existing pid namespaces. Therefore, to enter into existing pid namespaces, we would need to fork twice. Currently, there is no getting around this limitation.
 
-
 - [fork(2) man page](https://man7.org/linux/man-pages/man2/fork.2.html)
 - [clone(2) man page](https://man7.org/linux/man-pages/man2/clone.2.html)
 - [pid namespace man page](https://man7.org/linux/man-pages/man7/pid_namespaces.7.html)

docs/src/SUMMARY.md

@@ -0,0 +1,30 @@
+# Summary
+
+[Youki](./youki.md)
+
+---
+
+- [User Documentation](./user/introduction.md)
+  - [Basic Setup](./user/basic_setup.md)
+  - [Basic Usage](./user/basic_usage.md)
+  - [Crates provided](./user/crates.md)
+    - [libcgroups](./user/libcgroups.md)
+    - [libcontainer](./user/libcontainer.md)
+    - [liboci-cli](./user/liboci_cli.md)
+    - [libseccomp](./user/libseccomp.md)
+
+---
+
+- [Developer Documentation](./developer/introduction.md)
+  - [Basics](./developer/basics.md)
+  - [Unwritten Rule](./developer/unwritten_rules.md)
+  - [Good places to start](./developer/good_places_to_start.md)
+  - [This Documentation](./developer/documentation_mdbook.md)
+  - [Crate Specific Information](./developer/crate_specific_information.md)
+    - [libcgroups](./developer/libcgroups.md)
+    - [libcontainer](./developer/libcontainer.md)
+    - [liboci-cli](./developer/liboci_cli.md)
+    - [libseccomp](./developer/libseccomp.md)
+    - [youki](./developer/youki.md)
+    - [test_framework](./developer/test_framework.md)
+    - [integration_test](./developer/integration_test.md)

docs/src/developer/basics.md

@@ -0,0 +1,21 @@
+# Basics
+
+This section has the general information and resources needed to work with any part of youki. As youki is written in Rust, you should know some basic Rust before. If you don't yet, some good resources for that can be found on the Rust's [official site](https://www.rust-lang.org/learn).
+
+## Youki
+
+Youki is a low level container runtime, which deals with the creation and management of Linux containers. Some of other such low-level runtimes are [runc](https://github.com/opencontainers/runc) and [crun](https://github.com/containers/crun). These are usually used by a higher-level runtime such as Docker or Podman to actually create and manage containers, where the higher level runtime provides a much easier interface for users.
+
+Before you start working on developing youki, you should go through [the User documentation](../user/introduction) as it specifies the requirements and setup for running youki. For developing youki, you will need to install the dependencies and clone the repo, as specified in the [Basic Setup](../user/basic_setup.md) and [Basic Usage](../user/basic_usage.md) sections.
+
+## Resources
+
+#### OCI
+
+Open containers initiative is project, which provides a standardization and standardized specification for operating-system-level virtualization. That way components that confirm to the specification provided by OCI spec, can interoperate with each other easily, and developing of new applications becomes easier. For example youuki can be used inplace of runc in Docker, as all three : Docker, runc and youki are OCI compliant, and have a standard interface.
+
+Their main GitHub page is at [https://github.com/opencontainers](https://github.com/opencontainers), and more information about the runtime specifications can be found at [https://github.com/opencontainers/runtime-spec/blob/master/runtime.md](https://github.com/opencontainers/runtime-spec/blob/master/runtime.md).
+
+As youki needs to deal with a lot of low level programming interfaces of Linux Kernel, another good place know is the online man pages project, which can be found at [https://man7.org/](https://man7.org/). Man pages provide detailed information about the programming interfaces of various features of Linux Kernel. You can simply search `man <feature-name>` using a search engine, or you can search at the site itself, at [https://man7.org/linux/man-pages/index.html](https://man7.org/linux/man-pages/index.html). These can be very helpful to know about the behavior and usage of and reasoning behind various kernel features used throughout youki.
+
+Happy developing!!!

docs/src/developer/crate_specific_information.md

@@ -0,0 +1,5 @@
+# Crate Specific Information
+
+This section contains sub-sections for each individual crate in the youki workspace. Each of the seb-section will have information and resources on that particular crate.
+
+In case you are working with some specific crate, you can find resources about it in its section. Also make sure you add any resources that you find when working on them as well.

docs/src/developer/documentation_mdbook.md

@@ -0,0 +1,20 @@
+# This Documentation
+
+This documentation is created using mdbook and aims to provide a concise reference for users and developers of youki. For more information on mdbook itself, you can check out the [mdbook documentation](https://rust-lang.github.io/mdBook/).
+
+Please make sure that you update this documentation along with newly added features and resources that you found helpful while developing, so that it will be helpful for newcomers.
+
+TODO, update after deciding the final way to host the mdbook
+To compile and deploy this mdbook follow the instructions
+
+```
+git worktree add /tmp/book -b gh-pages
+mdbook build
+rm -rf /tmp/book/* # this won't delete the .git directory
+cp -rp book/* /tmp/book/
+cd /tmp/book
+git add -A
+git commit 'new book message'
+git push origin gh-pages
+cd -
+```

docs/src/developer/good_places_to_start.md

@@ -0,0 +1,23 @@
+# Good places to start
+
+First of all, welcome to youki! Hope you have fun while developing and contributing :)
+
+This lists some of the known places that are long-running and would be useful for beginners. But as the things under development can change any time, the best place to check are the issues on the [GitHub repo](https://github.com/containers/youki/issues). You can find issues with labels `good fist issue` or `help wanted` and start working on them.
+
+You can also search for `TODO` or `FIXME` comments in the source, and try working on them, but not all of them are easy places to start, can some of them can be particularly tricky to fix.
+
+---
+
+This lists known parts of youki that can be good for beginners at the time of the writing. Please update as things change.
+
+#### Documentation Comments
+
+Currently youki is decently commented, and those explain most of the public facing API and structs. But there are still places which can use more doc comments, and examples showing usage, so people can use the docs generated by `cargo doc` as a guide.
+
+If you don't know much about container runtime or low level system working, then this can be a good place to start. While going through the code an documenting it, you can learn about it. Make sure that you update this documentation with useful links that you found while commenting some code if it has some peculiar behavior, or it is hard to understand without knowing some background.
+
+#### Integration Tests
+
+You can find more detailed information about this in the integration_test crate, but in brief, we currently use [OCI-runtime-tools](https://github.com/opencontainers/runtime-tools) provided integration tests to validate that youki is OCI spec compliant. But those are written in Go, which makes the developer depend on two language env to compile youki and test it. These tests also have some issues which makes them hard to use on some system setups.
+
+Thus we are porting those test to Rust, so that it can be an Rust implementation of OCI-runtime integration tests, as well as be easy to run on local systems for testing. If you know Go and Rust this can be a great place to start. Check out the [tracking issue](https://github.com/containers/youki/issues/361).

docs/src/developer/integration_test.md

@@ -0,0 +1,9 @@
+# integration_test
+
+This crate contains the Rust port of OCI-runtime tools integration tests, which are used to test if the runtime works as per the OCI spec or not. Initially youki used the original implementation of these test provided in the OCI repository [here](https://github.com/opencontainers/runtime-tools/tree/master/validation). But those tests are written in Go, which made the developers depend on two language environments Rust and Go to compile youki and test it. The Validation tests themselves also have an optional dependency on node js to parse their output, which can make it a third language dependency.
+
+Other than that, those tests also showed some issues while running on some local systems, and thus running the tests would be difficult on local system. As the runtime is a complex piece of software, it becomes useful to have a set of tests that can be run with changes in code, so one can verify that change in one part of youki has not accidentally broken some other part of youki.
+
+Thus we decided to port the tests to Rust, and validate them, so that we have a set of unit tests as well of integration tests to validate the working of runtime. These tests are still under development, and you can check the [tracking issue](https://github.com/containers/youki/issues/361) for more details. More details on working of these tests can be found at [https://github.com/containers/youki/tree/main/crates/integration_test](https://github.com/containers/youki/tree/main/crates/integration_test).
+
+As these tests are under development, these are validated on a standard runtime such as runc in the GitHub CI, so validate the tests themselves.

docs/src/developer/introduction.md

@@ -0,0 +1,17 @@
+# Developer Documentation
+
+This section of the documentation is more oriented towards those who wish to contribute to youki, and contains more information and resources about the working and implementation of it. So if you are thinking of helping, this is a great place to start with.
+
+Also, Thank you! If you have any issues or doubts, you can ask them on youki's discord server [here](https://discord.gg/h7R3HgWUct).
+
+This section is split into following parts
+
+- Basics : This contains general resources and information that you wold need to work with any parts of youki.
+
+- Unwritten Rules : This is the part to make them written! This will contain conventions and rules that were discussed and decided in PRs or just commonly followed when developing.
+
+- Good Places to Start : This section will contain some suggestions about the areas that will be a good place to start for new contributors.
+
+- Crate specific Information : This is split into one sections for each crate, and will contains information and resources specific for that crate.
+
+Happy Contributing!

docs/src/developer/libcgroups.md

@@ -0,0 +1,30 @@
+# libcgroups
+
+This crate provides an interface for working with cgroups in Linux. cgroups or control groups is a Linux kernel feature which can be used to fine-control resources and permissions given to a particular process or a group of processes. You can read more about them on the [cgroups man page](https://man7.org/linux/man-pages/man7/cgroups.7.html).
+
+The initial version of cgroups is called the version 1 was implemented in kernel 2.6.24, and later in kernel version 4.5, a new version of cgroups was released, aimed to solve issues with v1, the version v2.
+
+This crates exposes several functions and modules that can be used to work with cgroups :
+
+- common traits and functions which are used by both v1 and v2 such as
+
+  - Trait CgroupManager, this abstracts over the underlying implementation of interacting with specific version of cgroups, and gives functions to add certain process to a certain cgroup, apply resource restrictions, get statistics of a cgroups, freeze a cgroup, remove a cgroup or get list of all processes belonging to a cgroup. v1 and v2 modules both contain a version specific cgroup manager which implements this trait, and thus either can be given to functions or structs which expects a cgroup manager, depending on which cgroups the host system uses.
+  - Apart from the trait, this also contians functions which help with reading cgroups files, and write data to a cgroup file, which are used throughout this crate.
+  - A function to detect which cgroup setup (v1, v2 or hybrid) is on the host system, as well as a function to get the corresponding cgroups manager.
+
+- Functions and structs to get and store the statistics of a cgroups such as
+
+  - CPU stats including usage and throttling
+  - Memory stats including usage of normal and swap memory, usage of kernel memory, page cache in bytes etc
+  - Pid stat including current active pids nd maximum allowed pids
+  - Block IO stats such as number of bytest transferred to/from a device in the cgroup, io operations performed by a device in the cgroup, amount of time cgroup had access to a device etc
+  - Huge TLB stats such as usage and maximum usage etc.
+  - Function to get pid stats
+  - Function to get supported hugepage size
+  - Function to parse flat keyed data and nested keyed data that can be in a cgroups file
+  - Parse a device number
+
+- Cgroups V1 module which deal with implementing a cgroup manager for systems which have cgroups v1 or hybrid cgroups
+- Cgroups V2 module which deal with implementing a cgroup manager for systems which have cgroups v2
+
+As youki currently depends on systemd as an init system, this crate also exposes module systemd, which provides interface for working with systemd related operations. [systemd resource control](https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html) is a good place to read more about systemd and its involvement in resource control.

docs/src/developer/libcontainer.md

@@ -0,0 +1,51 @@
+# libcontainer
+
+THis crate is one of the core crates of the youki workspace, and has functions and structs that deal with the actual craetion and management of the container processes.
+
+Remember, in the end, a container is just another process in Linux, which has control groups, namespaces, pivot_root and other mechanisms applied to it. The program executing has the impression that is is running on a complete system, but from the host system's perspective, it is just another process, and has attributes such as pid, file descriptors, etc. associated with it like any other process.
+
+Along with the container related functions, this crate also provides Youki Config, a subset of the OCI spec config. This config contains only the essential data required for running the containers, and due to its smaller size, parsing it and passing it around is more efficient than the complete OCI spec config struct.
+
+Other than that, this crate also provides a wrapper over basic Linux sockets, which are then used internally as well as by youki to communicate between the main youki process and the forked container process as well as the intermediate process.
+
+This crate also provides an interface for Apparmor which is another Linux Kernel module allowing to apply security profiles on a per-program basis. More information about it can be found at [https://apparmor.net/](https://apparmor.net/).
+
+### Notes
+
+#### Some other modules expose by this crate are
+
+- rootfs, which is a ramfs like simple filesystem used by kernel during initialization
+- hooks, which allow running of specified program at certain points in the container lifecycle, such as before and after creation, start etc.
+- singals, which provide a wrapper to convert to and from signal numbers and text representation of signal names
+- capabilities, which has functions related to set and reset specific capabilities, as well as to drop extra privileges
+  - [Simple explanation of capabilities](https://blog.container-solutions.com/linux-capabilities-in-practice)
+  - [man page for capabilities](https://man7.org/linux/man-pages/man7/capabilities.7.html)
+- tty module which daels with providing terminal interface to the container process
+  - [pseudoterminal man page](https://man7.org/linux/man-pages/man7/pty.7.html) : Information about the pseudoterminal system, useful to understand console_socket parameter in create subcommand
+
+#### Namespaces : namespaces provide isolation of resources such as filesystem, process ids networks etc on kernel level. This module contains structs and functions related to applying or un-applying namespaces to the calling process.
+
+- [pid namespace man page](https://man7.org/linux/man-pages/man7/pid_namespaces.7.html)
+- [CLONE_NEWUSER flag](https://man7.org/linux/man-pages/man2/clone.2.html)
+
+Note: clone(2) offers us the ability to enter into user and pid namespace by creatng only one process. However, clone(2) can only create new pid namespace, but cannot enter into existing pid namespaces. Therefore, to enter into existing pid namespaces, we would need to fork twice. Currently, there is no getting around this limitation.
+
+- [fork(2) man page](https://man7.org/linux/man-pages/man2/fork.2.html)
+- [clone(2) man page](https://man7.org/linux/man-pages/man2/clone.2.html)
+
+#### Pausing and resuming
+
+Pausing a container indicates suspending all processes in it. This can be done with signals SIGSTOP and SIGCONT, but these can be intercepted. Using cgroups to suspend and resume processes without letting tasks know.
+
+- [cgroups man page](https://man7.org/linux/man-pages/man7/cgroups.7.html)
+- [freezer cgroup kernel documentation](https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-subsystem.txt)
+
+#### The following are some resources that can help understand with various Linux features used in the code of this crate.
+
+- [oom-score-adj](https://dev.to/rrampage/surviving-the-linux-oom-killer-2ki9)
+- [unshare man page](https://man7.org/linux/man-pages/man1/unshare.1.html)
+- [user-namespace man page](https://man7.org/linux/man-pages/man7/user_namespaces.7.html)
+- [wait man page](https://man7.org/linux/man-pages/man3/wait.3p.html)
+- [pipe2 man page](https://man7.org/linux/man-pages/man2/pipe.2.html) : Definition and usage of pipe2
+- [Unix Sockets man page](https://man7.org/linux/man-pages/man7/unix.7.html) : Useful to understand sockets
+- [prctl man page](https://man7.org/linux/man-pages/man2/prctl.2.html) : Process control man pages