03-containerd-mode-operation-manual.md

title	expires_at	tags
containerd-mode Operator's Manual	never	garden-runc-release

Terminology

When it is running in containerd mode, Guardian will delegate all container operations to containerd. This means that containerd is doing all the housekeeping and Guardian is just acting as a client. That's why it is important to understand how Guardian is mapping its terminology to the terminology of containerd. Here is a table describing that mapping.

|---------------|-----------------|----------------------------------------------------------|
| Guardian Term | Containerd Term | RunC Representation                                      |
|---------------|-----------------|----------------------------------------------------------|
| OCI Bundle    | Container       | The config.json used by runc                             | 
| Container     | Task            | The running instance of a runc bundle (runc run)         |
| Process       | Process         | A process exec-ed into a running runc bundle (runc exec) |
|---------------|-----------------|----------------------------------------------------------|

Another important concept of containerd is the concept of namespaces. Each client can work in its own namespace, so that different clients do not have to give their containers globally unique names. Guardian is working in a namespace named "garden".

The ctr client

Containerd has a command line client named ctr that you can use in order to list and inspect containers created by Guardian in containerd mode. It is important to use the right containerd socket and namespace. Here is how to list all the containers (a.k.a tasks in containerd terms):

/var/vcap/packages/containerd/bin/ctr -a /var/vcap/sys/run/containerd/containerd.sock -n garden tasks ls

Look at ctr docs for more info.

Processes

For the time being the containerd mode is limited to container operations. So even when containerd mode is on, all process operations are still being handled by "RunDmc" - the runc mode containerizer. Please have a look at the runc mode ops guide for more details.

The Bundle Config

If you need to look at your RunC bundle config it can be found at:

/var/vcap/data/containerd/state/io.containerd.runtime.v1.linux/garden/$containerid/config.json

You can also use the ctr client:

/var/vcap/packages/containerd/bin/ctr -a /var/vcap/sys/run/containerd/containerd.sock -n garden containers info $containerid

Containerizer

RunContainerd is Guardian's containerd mode containerizer. It is a thin wrapper around containerd. In order to create and manage containers it relies on a containerd instance running on a unix socket. The socket path is /var/vcap/sys/run/containerd/containerd.sock

Interacting with a Container

As Guardian uses containerd to create containers you can use the ctr client to interact with them for debug purposes. For example, to execute a process in a container you can execute:

/var/vcap/packages/containerd/bin/ctr -a /var/vcap/sys/run/containerd/containerd.sock -n garden tasks exec --exec-id my-shell --tty $containerid /bin/sh

Further Reading

To enable the usage of containerd:

garden-runc-release/jobs/garden/spec

Lines 212 to 214 in 861e755

	garden.containerd_mode:
	description: "Use containerd for container lifecycle management. NOTE: cannot be used in combination with bpm or rootless"
	default: false

In addition, in order to enable containerd for container process management:

garden-runc-release/jobs/garden/spec

Lines 231 to 233 in 861e755

	garden.experimental_use_containerd_mode_for_processes:
	description: "(Under development) Use containerd for container process management. Must be used with containerd_mode also set to true. NOTE: cannot be used in combination with bpm or rootless"
	default: false

For Containerd's daemon: We set up the config file here:

garden-runc-release/jobs/garden/spec

Line 22 in 861e755

config/containerd.toml.erb: config/containerd.toml

Containerd's config file is built here: https://github.com/cloudfoundry/garden-runc-release/blob/develop/jobs/garden/templates/config/containerd.toml.erb#L29C17-L51

Garden Start starts up the containerd process by calling the method start_containerd:

garden-runc-release/jobs/garden/templates/bin/garden_start.erb

Line 46 in 861e755

start_containerd

Which then kicks off the steps to invoke containerd with the previously generated config file:

garden-runc-release/jobs/garden/templates/bin/containerd_utils.erb

Lines 28 to 68 in 861e755

	start_containerd() {
	log "starting containerd"
	if pidof containerd; then
	echo "containerd already running"
	return
	fi
	# migrate the containerd state folder to /var/vcap/sys/run
	new_state_dir=/var/vcap/sys/run/containerd/state
	old_state_dir=/var/vcap/data/containerd/state
	if [ -d "$old_state_dir" ] && [ ! -d "$new_state_dir" ]; then
	mkdir -p $(dirname "$new_state_dir")
	mv "$old_state_dir" "$new_state_dir"
	fi
	containerd_config_filepath="$GARDEN_CONFIG_DIR/containerd.toml"
	exec_command="exec"
	<% if p("garden.experimental_rootless_mode") -%>
	maximus=$(/var/vcap/packages/garden-idmapper/bin/maximus)
	cp "$GARDEN_CONFIG_DIR/containerd.toml" "$GARDEN_ROOTLESS_CONFIG_DIR"
	chown "$maximus:$maximus" "$GARDEN_ROOTLESS_CONFIG_DIR/containerd.toml"
	containerd_config_filepath="$GARDEN_ROOTLESS_CONFIG_DIR/containerd.toml"
	exec_command="exec execas --uid $maximus --gid $maximus"
	<% end -%>
	log "running containerd"
	$exec_command /var/vcap/packages/containerd/bin/containerd -c "$containerd_config_filepath" \
	1>> "${LOG_DIR}/containerd.stdout.log" \
	2>> "${LOG_DIR}/containerd.stderr.log" \
	&
	echo "$!" > "$CONTAINERD_PIDFILE"
	address=$(grep containerd.sock $containerd_config_filepath | awk '{print $3}' | tr -d '"')
	while ! /var/vcap/packages/containerd/bin/ctr -a $address --connect-timeout 100ms c ls; do
	log "waiting for containerd to become available"
	sleep 0.1
	done
	}

Garden's start had previously been accomplished by the configuration of this BPM configuration file: https://github.com/cloudfoundry/garden-runc-release/blob/861e755aed81d437f84f6af709344537578dd6e9/jobs/garden/templates/config/bpm.yml.erb#L1C1-L8C22