Simple security context options (Kubernetes)

[zc-devs]

Part of #2545

Adds security context options: privileged, runAsUser, runAsGroup, runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation.

---

Pipeline:

steps:
  test:
    image: alpine
    commands:
      - echo Hello from $(whoami)
    backend_options:
      kubernetes:
        securityContext:
          runAsUser: 999

Log:

{"level":"trace","Security context":{"runAsUser":999},"time":"2023-10-08T15:14:39Z","caller":"/src/pipeline/backend/kubernetes/pod.go:200","message":"Security context that will be used for containers"}

Pod:

spec:
  containers:
    - name: wp-01hc7qqcf5q5gqd86yjytb37q3-0-step-0
      image: alpine
      securityContext:
        privileged: false
        runAsUser: 999

Step output:

+ echo Hello from $(whoami)
Hello from
whoami: unknown uid 999

[zc-devs]

@dominic-p, could you test it and maybe write some docs?

[dominic-p]

Thanks for working on this! Yes, I'd be happy to test and write up some docs. Two questions:

1. Could you point me in the right direction for testing? I'm not a go developer, so I'm not sure what the right approach would be. I'm currently running Woodpecker via the Helm chart. If there's an image published for this PR, I should be able to reference it, but it looks like the CI hasn't run yet.

2. I didn't see fsGroup listed in the securityContext struct definitions. I believe that will be needed in addition to runAsGroup to make sure the mounted volume gets appropriate permissions.

[qwerty287]

@dominic-p I approved the pipeline to run now.
You should be able to get a docker image from this pr using the tag pull_2550

[codecov-commenter]

Codecov Report

Attention: 76 lines in your changes are missing coverage. Please review.

Comparison is base (111a0b4) 34.03% compared to head (ff7b45f) 33.88%.
Report is 3 commits behind head on main.

Files	Patch %	Lines
pipeline/backend/kubernetes/pod.go	0.00%	51 Missing ⚠️
pipeline/frontend/yaml/compiler/convert.go	47.05%	16 Missing and 2 partials ⚠️
pipeline/backend/kubernetes/kubernetes.go	0.00%	4 Missing ⚠️
server/forge/gitea/gitea.go	0.00%	1 Missing ⚠️
server/forge/github/github.go	0.00%	1 Missing ⚠️
server/forge/gitlab/gitlab.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2550      +/-   ##
==========================================
- Coverage   34.03%   33.88%   -0.16%     
==========================================
  Files         217      217              
  Lines       13825    13890      +65     
==========================================
+ Hits         4706     4707       +1     
- Misses       8746     8809      +63     
- Partials      373      374       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dominic-p]

@qwerty287 thanks for doing that. I was able to test the PR. Here's what I'm seeing:

1. The securityContext is being set at the container level (i.e. spec.containers[0].securityContext). That may be useful for some circumstances, but I believe a more common approach (and more useful in general) is to set the securityContext at the Pod level (i.e. spec.securityContext). Maybe both could be supported? The YAML could look like this:

    backend_options:
      kubernetes:
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          runAsGroup: 1000
          fsGroup: 1000
        containerSecurityContext:
          runAsUser: 1000

That said, my vote would be to just support setting the Pod level securityContext since these Pods will typically (always?) have just one container.

2. We're still missing the fsGroup setting which makes sense because I don't believe that's supported at the container level. But, if we switch to the Pod level fsGroup should also be added as a supported option.

[pat-s]

I wonder how much we want to bake into the backend_options or actually port these options to the helm chart?
Most of the options in these PR are usually set for all pods of an instance and shouldn't be necessary to define on the step level?

[zc-devs]

@dominic-p, you're right 1, 2. I did it yesterday, but faced issue with stucking pipeline. I'm figuring it out now.

I made one security context:

type SecurityContext struct {
	// Pod Security Context
	RunAsUser          *int64  `yaml:"runAsUser,omitempty"`
	RunAsGroup         *int64  `yaml:"runAsGroup,omitempty"`
	RunAsNonRoot       *bool   `yaml:"runAsNonRoot,omitempty"`
	SupplementalGroups []int64 `yaml:"supplementalGroups,omitempty"`
	FSGroup            *int64  `yaml:"fsGroup,omitempty"`
	// Container Security Context
	Privileged               *bool `yaml:"privileged,omitempty"`
	ReadOnlyRootFilesystem   *bool `yaml:"readOnlyRootFilesystem,omitempty"`
	AllowPrivilegeEscalation *bool `yaml:"allowPrivilegeEscalation,omitempty"`
}

@pat-s, why should it be ported to the helm chart?

Most of the options are usually set for all pods of an instance and shouldn't be necessary to define on the step level?

Do you mean, I should add WOODPECKER_BACKEND_K8S_POD_SECURITY_* options? I agree. Then step level options could be treated as exclusion: for example, we set runAsNonRoot=true instance-wide, but for some step/pod/container we make exclusion in step config.

I wonder how much we want to bake into the backend_options?

¯_(ツ)_/¯ Probably, all Kube spec 🤣 Seriously, at least I plan to add seccomp and AppArmor. @dominic-p, what of simple SC options (above) do you need and what we can get rid of (except privileged)?

[zc-devs]

Added supplementalGroups and fsGroup.

Pipeline:

steps:
  test:
    image: alpine
    commands:
      - echo Hello from $(whoami)
    backend_options:
      kubernetes:
        securityContext:
          runAsUser: 999
          readOnlyRootFilesystem: true

Pod:

kind: Pod
spec:
  containers:
    - name: wp-01hca2t4zh0r9xsqpbze5ryqww-0-step-0
      image: alpine
      securityContext:
        privileged: false
        readOnlyRootFilesystem: true
  securityContext:
    runAsUser: 999

[pat-s]

Do you mean, I should add WOODPECKER_BACKEND_K8S_POD_SECURITY_* options? I agree. Then step level options could be treated as exclusion: for example, we set runAsNonRoot=true instance-wide, but for some step/pod/container we make exclusion in step config.

Yes. I think in 99% of all cases you want to have these options set on the instance level and not on the step level. Individual overrides are always good to have though.

I think we should add all options at the same time, otherwise people will start to use this notation at first as there are no global options available.

¯_(ツ)_/¯ Probably, all Kube spec 🤣 Seriously, at least I plan to add seccomp and AppArmor. @dominic-p, what of simple SC options (above) do you need and what we can get rid of (except privileged)?

Fair enough. Again, having them on the step level is certainly not bad for granular config.
Yet I also wonder how we might deal with overrides that are actually not wanted. E.g. an admin might want to control if users can override security specific settings on the step level.
Maybe we need an extra var to control for this?

[dominic-p]

@zc-devs for my use case I just need runAsNonRoot, runAsUser, runAsGroup, and fsGroup (although runAsUser may imply runAsNonRoot I'm not sure on that).

I would hesitate to combine Pod and container level config in the same object just because as a user I wouldn't be sure which one I was setting. That said, it's probably mostly a matter of style/preference in a single container Pod.

As for global options in the helm chart, I agree with @zc-devs. If I have multiple steps running in different containers, it's very unlikely that they are all going to need to run as the same UID/GID, for instance. I would still need to configure/override some of the settings at the step level.

[zc-devs]

• Deleted unnecessary options
• Added pipeline schema
• Made runAsNonRoot agent-wide
• Don't set securityContext if all fields aren't set

Test with WOODPECKER_BACKEND_K8S_SECCTX_NONROOT: 'true'

1. Pipeline:

skip_clone: true
steps:
  test:
    image: alpine
    commands:
      - echo Hello from $(whoami)

Error:

Error: container has runAsNonRoot and image will run as root (container: wp-01hcd9bzzxydxr5fes0fhq97rz-0-step-0)

2. Pipeline:

skip_clone: true
steps:
  test:
    image: alpine
    commands:
      - echo Hello from $(whoami)
    backend_options:
      kubernetes:
        securityContext:
          runAsUser: 999
          runAsGroup: 999

Pod:

kind: Pod
spec:
  containers:
    - name: wp-01hcd7tjy79ctkkx3g3hg8hae2-0-step-0
      image: alpine
  securityContext:
    runAsUser: 999
    runAsGroup: 999
    runAsNonRoot: true

3. Pipeline:

skip_clone: true
steps:
  test:
    image: alpine
    commands:
      - echo Hello from $(whoami)
    backend_options:
      kubernetes:
        securityContext:
          runAsUser: 999
          runAsGroup: 999
          privileged: true

Pod:

kind: Pod
spec:
  containers:
    - name: wp-01hcd83q7be5ymh89k5accn3k6-0-step-0
      image: alpine
      securityContext:
        privileged: true
  securityContext:
    runAsUser: 999
    runAsGroup: 999
    runAsNonRoot: true

[zc-devs]

I would hesitate to combine Pod and container level config in the same object just because as a user I wouldn't be sure which one I was setting. That said, it's probably mostly a matter of style/preference in a single container Pod.

It's abstraction here which doesn't have to match 1:1 with Kubernetes. Also as a step is a pod with only one container now, there is no reason to make things more complex.

have these options set on the instance level

If I have multiple steps running in different containers, it's very unlikely that they are all going to need to run as the same UID/GID

Agree, it doesn't make sense to have user and group options agent-wide. Consider this, I added only runAsNonRoot in the agent configuration.

admin might want to control if users can override security specific settings on the step level

Then Pod Security Admission will help, I think.

[dominic-p]

@zc-devs that all makes sense to me. If we run the CI again, I can smoke test the revisions on my cluster.

Once everyone's settled on the semantics for the config I can submit a PR with some docs.

[zc-devs]

@pat-s, could you add build_pr_images label here?

[pat-s]

@zc-devs Your PR images are finally ready!

[zc-devs]

Thanks.
@dominic-p, could you test it (woodpeckerci/woodpecker-server:pull_2735, woodpeckerci/woodpecker-agent:pull_2735)?

[dominic-p]

Sorry for the radio silence here. I've been stuck working on a pretty consuming project the last couple weeks. That's wrapping up now, so I should be able to test this in the next day or two.

[dominic-p]

Ok, I was finally able to test this tonight. It looks good to me.

I'm still not able to run my particular pipeline as we're still missing AppArmor functionality, but the security context setting seem to be working as expected.

Thanks so much for your work on this. Did you want me to take a stab at some documentation on this for the website?

[zc-devs]

Did you want me to take a stab at some documentation on this for the website?

Yeah, that would be great!

[woodpecker-bot]

Deployment of preview was successful: https://woodpecker-ci-woodpecker-pr-2550.surge.sh

[zc-devs]

@qwerty287, could you review again?

[qwerty287]

Yes, looks good so far. Didn't test. And as I wrote already on other PR: I can't really say something about kubernetes-specific stuff

[qwerty287]

I'm merging now. If there are still issues we can fix them later