Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bundle a buildkit instance within sandbox to facilitate local image spec builds #3860

Merged
merged 3 commits into from
Jul 20, 2023
Merged

Bundle a buildkit instance within sandbox to facilitate local image spec builds #3860

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
396cbe6
Bundle a buildkit instance within sandbox to facilitate local image s…
jeevb Jul 11, 2023
d26abf6
Add port map
jeevb Jul 17, 2023
c7bb4eb
Merge branch 'master' into jeev/sandbox-buildkitd
jeevb Jul 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions charts/flyte-sandbox/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,10 @@ A Helm chart for the Flyte local sandbox
| postgresql.volumePermissions.enabled | bool | `true` | |
| postgresql.volumePermissions.image.pullPolicy | string | `"Never"` | |
| postgresql.volumePermissions.image.tag | string | `"sandbox"` | |
| sandbox.buildkit.enabled | bool | `true` | |
| sandbox.buildkit.image.pullPolicy | string | `"Never"` | |
| sandbox.buildkit.image.repository | string | `"moby/buildkit"` | |
| sandbox.buildkit.image.tag | string | `"sandbox"` | |
| sandbox.dev | bool | `false` | |
| sandbox.proxy.enabled | bool | `true` | |
| sandbox.proxy.image.pullPolicy | string | `"Never"` | |
Expand Down
8 changes: 8 additions & 0 deletions charts/flyte-sandbox/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,14 @@ Name of PersistentVolume and PersistentVolumeClaim for Minio
{{- printf "%s-minio-storage" .Release.Name -}}
{{- end }}

{{/*
Selector labels for Buildkit
*/}}
{{- define "flyte-sandbox.buildkitSelectorLabels" -}}
{{ include "flyte-sandbox.selectorLabels" . }}
app.kubernetes.io/component: buildkit
{{- end }}

{{/*
Selector labels for Envoy proxy
*/}}
Expand Down
50 changes: 50 additions & 0 deletions charts/flyte-sandbox/templates/buildkit/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
{{- if .Values.sandbox.buildkit.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "flyte-sandbox.fullname" . }}-buildkit
namespace: {{ .Release.Namespace | quote }}
labels: {{- include "flyte-sandbox.labels" . | nindent 4 }}
spec:
replicas: 1
selector:
matchLabels: {{- include "flyte-sandbox.buildkitSelectorLabels" . | nindent 6 }}
template:
metadata:
labels: {{- include "flyte-sandbox.buildkitSelectorLabels" . | nindent 8 }}
spec:
hostNetwork: true
containers:
- name: buildkit
{{- with .Values.sandbox.buildkit.image }}
image: {{ printf "%s:%s" .repository .tag | quote }}
imagePullPolicy: {{ .pullPolicy | quote }}
{{- end }}
args:
- --addr
- unix:///run/buildkit/buildkitd.sock
- --addr
- tcp://0.0.0.0:30003
ports:
- name: tcp
containerPort: 30003
protocol: TCP
readinessProbe:
exec:
command:
- buildctl
- debug
- workers
initialDelaySeconds: 5
periodSeconds: 30
livenessProbe:
exec:
command:
- buildctl
- debug
- workers
initialDelaySeconds: 5
periodSeconds: 30
securityContext:
privileged: true
{{- end }}
6 changes: 6 additions & 0 deletions charts/flyte-sandbox/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,12 @@ sandbox:
# dev Routes requests to an instance of Flyte running locally on a developer's
# development environment. This is only usable if the flyte-binary chart is disabled.
dev: false
buildkit:
enabled: true
image:
repository: moby/buildkit
tag: sandbox
pullPolicy: Never
proxy:
enabled: true
image:
Expand Down
8 changes: 8 additions & 0 deletions docker/sandbox-bundled/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,13 @@ build: flyte manifests
docker buildx build --builder flyte-sandbox --allow security.insecure --load \
--tag flyte-sandbox:latest .

# Port map
# 6443 - k8s API server
# 30000 - Docker Registry
# 30001 - DB
# 30002 - Minio
# 30003 - Buildkit
# 30080 - Flyte Proxy
.PHONY: start
start: FLYTE_SANDBOX_IMAGE := flyte-sandbox:latest
start: FLYTE_DEV := False
Expand All @@ -56,6 +63,7 @@ start:
--publish "30000:30000" \
jeevb marked this conversation as resolved.
Show resolved Hide resolved
--publish "30001:30001" \
--publish "30002:30002" \
--publish "30003:30003" \
--publish "30080:30080" \
$(FLYTE_SANDBOX_IMAGE)
export KUBECONFIG=$(PWD)/.kube/kubeconfig
Expand Down
1 change: 1 addition & 0 deletions docker/sandbox-bundled/images/manifest.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ docker.io/bitnami/postgresql:sandbox=bitnami/postgresql:15.1.0-debian-11-r20
docker.io/envoyproxy/envoy:sandbox=envoyproxy/envoy:v1.23-latest
docker.io/kubernetesui/dashboard:sandbox=kubernetesui/dashboard:v2.7.0
docker.io/library/registry:sandbox=registry:2.8.1
docker.io/moby/buildkit:sandbox=moby/buildkit:buildx-stable-1
docker.io/rancher/local-path-provisioner:v0.0.21
docker.io/rancher/mirrored-coredns-coredns:1.9.1
docker.io/rancher/mirrored-library-busybox:1.34.1
Expand Down
62 changes: 60 additions & 2 deletions docker/sandbox-bundled/manifests/complete.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -784,7 +784,7 @@ type: Opaque
---
apiVersion: v1
data:
haSharedSecret: RDFqS1N3Z1RaaWw2UTRveg==
haSharedSecret: cURTeDZkWDhaQkdIbmhtMA==
proxyPassword: ""
proxyUsername: ""
kind: Secret
Expand Down Expand Up @@ -1268,6 +1268,64 @@ spec:
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: flyte-sandbox
app.kubernetes.io/version: 1.16.0
helm.sh/chart: flyte-sandbox-0.1.0
name: flyte-sandbox-buildkit
namespace: flyte
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: buildkit
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/name: flyte-sandbox
template:
metadata:
labels:
app.kubernetes.io/component: buildkit
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/name: flyte-sandbox
spec:
containers:
- args:
- --addr
- unix:///run/buildkit/buildkitd.sock
- --addr
- tcp://0.0.0.0:30003
image: moby/buildkit:sandbox
imagePullPolicy: Never
livenessProbe:
exec:
command:
- buildctl
- debug
- workers
initialDelaySeconds: 5
periodSeconds: 30
name: buildkit
ports:
- containerPort: 30003
name: tcp
protocol: TCP
readinessProbe:
exec:
command:
- buildctl
- debug
- workers
initialDelaySeconds: 5
periodSeconds: 30
securityContext:
privileged: true
hostNetwork: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: docker-registry
Expand All @@ -1287,7 +1345,7 @@ spec:
metadata:
annotations:
checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81
checksum/secret: bd21016a74ab909d2141658a1698d749f0fc635e05c5bf730c8d72d39df26822
checksum/secret: c1b12622e870e1d0998ee4098b92129f20551c79afa1d776395f7a5687993193
labels:
app: docker-registry
release: flyte-sandbox
Expand Down
62 changes: 60 additions & 2 deletions docker/sandbox-bundled/manifests/dev.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -499,7 +499,7 @@ metadata:
---
apiVersion: v1
data:
haSharedSecret: c3JOc21QU2ZnZDdnUjFnQg==
haSharedSecret: eXNWNzBKd1B3SkpWRG9sWQ==
proxyPassword: ""
proxyUsername: ""
kind: Secret
Expand Down Expand Up @@ -856,6 +856,64 @@ spec:
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: flyte-sandbox
app.kubernetes.io/version: 1.16.0
helm.sh/chart: flyte-sandbox-0.1.0
name: flyte-sandbox-buildkit
namespace: flyte
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: buildkit
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/name: flyte-sandbox
template:
metadata:
labels:
app.kubernetes.io/component: buildkit
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/name: flyte-sandbox
spec:
containers:
- args:
- --addr
- unix:///run/buildkit/buildkitd.sock
- --addr
- tcp://0.0.0.0:30003
image: moby/buildkit:sandbox
imagePullPolicy: Never
livenessProbe:
exec:
command:
- buildctl
- debug
- workers
initialDelaySeconds: 5
periodSeconds: 30
name: buildkit
ports:
- containerPort: 30003
name: tcp
protocol: TCP
readinessProbe:
exec:
command:
- buildctl
- debug
- workers
initialDelaySeconds: 5
periodSeconds: 30
securityContext:
privileged: true
hostNetwork: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: docker-registry
Expand All @@ -875,7 +933,7 @@ spec:
metadata:
annotations:
checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81
checksum/secret: b23a2fd488916ff700053c173927ef317a26fea3a991c9f0e5f9ee11f4ebcd77
checksum/secret: b118994763c1f0f1aa7064bd31201c1a73ec4d6f48642a9b55cebd57051591b2
labels:
app: docker-registry
release: flyte-sandbox
Expand Down