How to run privileged, with capabilities, or writable filesystem?

[lucidBrot]

Hi, I have some challenge where I would like to modify some of the following settings of the spec.spec.securityContext :

• privileged: true
• capabilities other than SYS_ADMIN. For example NET_ADMIN to be able to create veth interfaces in the container
• readOnlyRootFilesystem: false
• no_new_privs (probably unneccessary, idk)

The use case for me is not just for debugging, but I think in general it would be very useful to have the option to specify these settings in the challenge.yaml config for anyone who needs to make an existing challenge work in kctf to pinpoint the problems.

This issue can be considered related to #344 and #387 .

I am considering to patch kctf for my own use-case to support the few configurations I want, so I had a look at the code. As far as I can tell, a deployment with healthcheck and a deployment without healthcheck are created similarly, from a kctfv1.Challenge struct, which contains the ChallengeSpec, which in turn is probably generated directly from the challenge.yaml file?
As far as I can tell, the deployment function uses the spec parameter but has some hardcoded defaults that can not be changed, such as readOnlyRootFilesystem = true . The SecurityContext seems to be modified, not overwritten, but I don't understand how to specify settings for it.

However, although I think patching this "should be easy ™️ ":

• this could be useful for others as well
• I have not yet managed to figure out how to build even the unmodified kctf distribution from source by myself. The documentation is outdated and the bash script in the linked script expects the PROJECT env var to be set but so far I have only been using the kind cluster (and tbh I am overwhelmed with how to start)
• The comment in Spec.PodTemplate sounds like the functionality I wish for might even already exist? Simply adding the securityContext to the challenge.yaml results in a failure when kctf chal start validates the file.

What are your opinions on this? I am hoping for any of

• Information how this is already possible to do
• advice / steps on how to best build it and which files I would need to touch for that
• agreement that it would make sense to feature a way to configure such options

It might be worth mentioning that I am new to kubernetes, so please excuse me if it turns out the problems I am having are completely unrelated to kctf and could be solved using just kubernetes.

[lucidBrot]

With a challenge.yaml like this (inspired from operator.yaml), it makes the impression that it would work, but then chal-sample is stuck at ContainerCreating:

apiVersion: kctf.dev/v1
kind: Challenge
metadata:
  name: chal-sample
spec:
  deployed: true
  powDifficultySeconds: 0
  network:
    public: false
  podTemplate:
    template:
      spec:
        containers:
          - name: challenge
            securityContext:
              capabilities:
                add:
                  - SYS_ADMIN
              readOnlyRootFilesystem: false
  healthcheck:
    # TIP: disable the healthcheck during development
    enabled: false
  image: kind/challenge:73be8f9fcfc09a9e588aafbcc4b6689d18ec6d15a7ab62e6fb4c36e557c06511

Error Messages:

kubectl get events --sort-by=.metadata.creationTimestamp

shows

8m26s       Normal    ScalingReplicaSet        deployment/chal-sample              Scaled up replica set chal-sample-79887df59 to 1
8m26s       Normal    SuccessfulCreate         replicaset/chal-sample-79887df59    Created pod: chal-sample-79887df59-2txxg
8m26s       Normal    Scheduled                pod/chal-sample-79887df59-2txxg     Successfully assigned default/chal-sample-79887df59-2txxg to kctf-cluster-control-plane
4m26s       Warning   FailedCreatePodSandBox   pod/chal-sample-79887df59-2txxg     Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2s          Warning   FailedCreatePodSandBox   pod/chal-sample-79887df59-2txxg     Failed to create pod sandbox: rpc error: code = Unknown desc = failed to reserve sandbox name "chal-sample-79887df59-2txxg_default_1970720f-ede0-40dd-bfb9-571e6e82271b_0": name "chal-sample-79887df59-2txxg_default_1970720f-ede0-40dd-bfb9-571e6e82271b_0" is reserved for "8939e3d72e33fb52fcb9e660ba1bcb2b6a2bd00e3545654f6e460b67ec09db95"

Since the only other thing I have modified is that it launches /bin/bash instead of nsjail, this config must be wrong despite getting validated without error.

[lucidBrot]

I just completely stopped all challenges and my local kind cluster, deleted the ./kctf/config/* and ~/.kube, made sure no docker containers were still running, and then created a new local kind cluster with a different name. The yaml snippet I posted above is now actually showing up in kubectl edit deploy chal-sample and the filesystem is writeable.

The problem with being stuck at ContainerCreation is a different issue and I'll have to do more debugging to reproduce it reliably. My original question is answered now: Yes, there is a way to specify these things in the yaml.

Caveat: when I specify SYS_ADMIN like in the yaml I posted above, it will appear twice in the config yaml. Not sure whether that can hurt.

[sirdarckcat]

thanks, please reopen if there's news