Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: added gitaly cgroups init image to package #246

Merged
merged 19 commits into from
Dec 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions charts/config/templates/gitaly-cgroups-exemption.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Copyright 2024 Defense Unicorns
# SPDX-License-Identifier: AGPL-3.0-or-later OR LicenseRef-Defense-Unicorns-Commercial

{{- if .Values.gitalyCgroupsInit.enabled }}

apiVersion: uds.dev/v1alpha1
kind: Exemption
metadata:
name: gitaly-cgroups-init-container
namespace: uds-policy-exemptions
spec:
exemptions:
- policies:
- RestrictHostPathWrite
- RestrictVolumeTypes
- RequireNonRootUser
- DisallowPrivileged
matcher:
namespace: {{ .Release.Namespace }}
name: "gitlab-gitaly.*"
title: "gitlab gitaly exemptions"
description: "Exemption allows cgroup modification by init container. See https://docs.gitlab.com/ee/administration/gitaly/kubernetes.html#constrain-git-processes-resource-usage"
{{- end }}
2 changes: 1 addition & 1 deletion charts/config/templates/gitlab-object-store-secret.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ apiVersion: v1
kind: Secret
metadata:
name: gitlab-object-store
namespace: gitlab
namespace: {{ .Release.Namespace }}
type: kubernetes.io/opaque
stringData:
{{- $awsAccessKey := "" }}
Expand Down
4 changes: 4 additions & 0 deletions charts/config/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,10 @@ mirroring:
ports:
- 443

# Add the exemption for the gitaly cgroups init container
gitalyCgroupsInit:
enabled: false

# custom:
# # Notice no `remoteGenerated` field here on custom internal rule
# - direction: Ingress
Expand Down
2 changes: 1 addition & 1 deletion charts/settings/templates/settings-secret.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ apiVersion: v1
kind: Secret
metadata:
name: gitlab-settings-secret
namespace: gitlab
namespace: {{ .Release.Namespace }}
type: Opaque
stringData:
application.json: {{ .Values.settingsJob.application | toJson | quote }}
Expand Down
13 changes: 12 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ By default, the application is configured to work with `uds-package-minio-operat
If you are not using in-cluster MinIO, but rather are using an external cloud providers object storage, you have two options. You can either create an object storage secret manually and disable the generation of the secret or have the helm chart generate one for you based on a set of input values.

> [!NOTE]
> If you would like to opt out of the in-chart secret generation process, you may disable it by setting the zarf variable GENERATE_STORAGE_SECRET to false. Then you can provide your own object store secret, named gitlab-object-store, as needed following GitLab's documentation.
> If you would like to opt out of the in-chart secret generation process, you may disable it by setting the zarf variable `GENERATE_STORAGE_SECRET` to false. Then you can provide your own object store secret, named `gitlab-object-store`, as needed following GitLab's documentation.

When configuring the GitLab to connect to S3 storage in AWS, it is assumed IRSA will be used to connect to the buckets. The prerequisites for this are the buckets created with the appropriate iam roles and policies. Once those are created, two values need to be overridden in the config chart for secret generation: `storage.createSecret.provider` needs to be set to `aws` and `storage.createSecret.region` needs to be set to your AWS regions (i.e `us-gov-west-1`). From there, additional overrides are required in the gitlab chart to finish this setup. Specifically, the gitlab service accounts need to be overridden to have the annotations that are required for IRSA. Below is an example of how you would define the variable overrides where you would then pass in the IAM role ARNs on deploy.

Expand Down Expand Up @@ -327,3 +327,14 @@ This will configure a bot account named `renovatebot` and create a PAT with scop

> [!NOTE]
> If the GitLab instance is configured with a license for Premium or Ultimate, [Gitlab Service Accounts](https://docs.gitlab.com/ee/user/profile/service_accounts.html) will be created. Otherwise, standard user accounts will be created.

## Gitaly HA

To use [custom cgroup sizes for Gitaly](https://docs.gitlab.com/ee/administration/gitaly/kubernetes.html#constrain-git-processes-resource-usage):

1. Set `gitlab.gitaly.cgroups.enabled` to `true` in the `gitlab` chart.
2. Set the cgroup permissions under the pod's resource limits as shown in the [GitLab docs](https://docs.gitlab.com/ee/administration/gitaly/kubernetes.html#constrain-git-processes-resource-usage).
3. Set `gitalyCgroupsInit` to `true` in the `uds-gitlab-config` chart. This causes a policy exemption to be created allowing the init container privileged access to the host nodes, required to customize the cgroups.

> [!NOTE]
> Only the `upstream` and `unicorn` flavors include the Gitaly init container required for this configuration. It will not work if using the `registry1` flavor.
9 changes: 9 additions & 0 deletions values/common-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,15 @@ gitlab:
enabled: true
serviceMonitor:
enabled: true
cgroups:
enabled: false
initContainer:
securityContext:
# This multi-line oddity is a hacky way to bypass: https://gitlab.com/gitlab-org/gitlab/-/issues/507883
runAsUser: |
0
privileged: true
runAsGroup: 0

gitlab-shell:
# override to enable ssh
Expand Down
1 change: 1 addition & 0 deletions values/registry1-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ gitlab:
image:
repository: registry1.dso.mil/ironbank/gitlab/gitlab/gitaly
tag: 17.6.1
# Note, the registry1 flavor is missing the cgroups init image because it's not in ironbank
gitlab-exporter:
image:
repository: registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-exporter
Expand Down
5 changes: 5 additions & 0 deletions values/unicorn-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ gitlab:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitaly
tag: v17.6.1
cgroups:
initContainer:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitaly-init-cgroups
tag: v17.6.1
gitlab-exporter:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-exporter
Expand Down
5 changes: 5 additions & 0 deletions values/upstream-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ gitlab:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitaly
tag: v17.6.1
cgroups:
initContainer:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitaly-init-cgroups
tag: v17.6.1
gitlab-exporter:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-exporter
Expand Down
2 changes: 2 additions & 0 deletions zarf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ components:
- "registry.gitlab.com/gitlab-org/build/cng/certificates:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/cfssl-self-sign:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/gitaly:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/gitaly-init-cgroups:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/gitlab-pages:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/gitlab-shell:v17.6.1"
Expand Down Expand Up @@ -118,6 +119,7 @@ components:
- "registry.gitlab.com/gitlab-org/build/cng/certificates:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/cfssl-self-sign:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/gitaly:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/gitaly-init-cgroups:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/gitlab-pages:v17.6.1"
- "registry.gitlab.com/gitlab-org/build/cng/gitlab-shell:v17.6.1"
Expand Down
Loading