Deploying on GKE using the helm chart not working

[AhmedBytesBits]

My actions before raising this issue

• Following the official documentation for using the official Helm Chart documentation
• Deployment of cvat works for all deployments except workers both the low and default
• Checking Deployment logs using GKE console

Multi-Attach error for volume "pvc-856156a9-8f5e-4ac1-9b5e-e39a490f26c1" Volume is already used by pod(s) mc-backend-worker-default-75bdfc7d9f-4l2w2, mcit-backend-utils-845b68fb99-8wls9, mc-backend-server-74fb85b6c7-pk6kq

Expected Behaviour

Deployment of chart should work on GKE
suggesting a change/improvement, tell us how it should work -->

Current Behaviour

Deployment of utils, default and low worker stuck, which one is stuck is based on competition on the created shared volume

Possible Solution

Avoid using shared folder
or, using affinity to ensure deployment on single node

Steps to Reproduce (for bugs)

1. Change configuration following instructions
2. run helm upgrade -n cvat mcit -i --create-namespace ./helm-chart -f ./helm-chart/values.yaml -f ./helm-chart/values.override.yaml

Context

GKE supports Read/Write shared volumes for pods attached to the same node, cross node attachment for Read/Write is not allowed

Your Environment

• GKE cluster version: 1.22.12-gke.2300
• cvat release: 2.2

[nmanovic]

@ahmedrshdy , do you still have the issue in the latest release?

[grzleadams]

I'm still seeing this on chart v0.7.2 (on-prem RKE, not GKE).

[stykm]

I'm stuck having the same issue as described in the original post (I'm trying to deploy helm-chart that is currently available (for CVAT v2.4.1)

[work-temp-dl-stuff]

I still have this exact same issue when trying to deploy in cloud (We dont have ReadWriteMany volumes, i cannot set affinity to make pods run on single node (permissions)

[AhmedBytesBits]

Unfortunately I gave up k8s deployment

[Keramblock]

Sadly cvat right now is designed to use RWX volume, so you could not proceed without one AFAIK.

Also I believe, that GKE supports them:
https://cloud.google.com/filestore/docs/optimize-multishares

[Zanz2]

I have a deployment that uses ReadWriteOnce via helm and it works fine, you do have to use podaffinity rules to schedule all the pods on the same node though. Example for every backend pod:

affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: tier
              operator: In
              values:
              - backend
          topologyKey: "kubernetes.io/hostname"

Thats because multiple pods can actually access the same volume, but they must be on the same node. But this is sort of against kubernetes principles.

[azhavoro]

fixed in #6137