Bug: Installing via Helm in Argo doesn't apply generated templates, such as ClusterRole azureserviceoperator-manager-role

[jtklahti]

Version of Azure Service Operator
Deployment details:

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                              SELECTOR
azureserviceoperator-controller-manager   0/1     1            0           62m   manager      mcr.microsoft.com/k8s/azureserviceoperator:v2.7.0   control-plane=controller-manager

Installing ASO with ArgoCD using tag v2.8.0 (relevant parts):

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    repoURL: https://github.com/Azure/azure-service-operator.git
    targetRevision: v2.8.0
    path: v2/charts/azure-service-operator
    helm:
      releaseName: aso2
      valuesObject:
        installCRDs: true
        crdPattern: 'containerregistry.azure.com/*'

Describe the bug
Installing ASO with ArgoCD using Helm seems to miss generated templates like azureserviceoperator-manager-role and others.

ArgoCD Helm installation doesn't actually use the Helm package, but fetches the templates from the repository and applies them. This seems to be the reason why the generated templates are not applied.

Because ArgoCD Helm installation works like this, generated templates should be included in the Git repository for ArgoCD to apply them. If there is some other way to workaround ArgoCD Helm installation method, describing it in the documentation would also solve the issue.

To Reproduce
Steps to reproduce the behavior:

• Install ASO with ArgoCD using tag v2.8.0 with Argo Application configured as above
• Check if ClusterRole azureserviceoperator-manager-role is missing and pod does not go to ready state

Simulating roughly what ArgoCD does:

git clone --branch=v2.8.0 --depth 1 git@github.com:Azure/azure-service-operator.git
cd azure-service-operator/v2/charts/azure-service-operator
cat <<EOF | helm template -f- . | grep -C 2 "name: azureserviceoperator-manager-role"
installCRDs: true
crdPattern: 'containerregistry.azure.com/*'
EOF

... which output only ClusterRoleBinding azureserviceoperator-manager-rolebinding that references the missing ClusterRole azureserviceoperator-manager-role.

Expected behavior
ClusterRole azureserviceoperator-manager-role and other generated templates should be applied successfully.

Helm template using Git repo should(?) include generated templates. Compared to Helm package output:

curl -LJO https://github.com/Azure/azure-service-operator/raw/main/v2/charts/azure-service-operator-2.8.0.tgz
tar -xzf azure-service-operator-2.8.0.tgz
cd azure-service-operator
cat <<EOF | helm template -f- . | grep -C 2 "name: azureserviceoperator-manager-role"
installCRDs: true
crdPattern: 'containerregistry.azure.com/*'
EOF

Outputs the ClusterRole correctly:

kind: ClusterRole
metadata:
  name: azureserviceoperator-manager-role
...

Additional context
I was first using ArgoCD v2.10.x, but upgraded to newest v2.12.0. This issue was present in both versions.

[matthchr]

Shouldn't argo be running helm template on the actual Helm chart contents (the tgz?) rather than the directory structure on GitHub?

What does it do for charts like those at https://charts.jetstack.io (cert-manager), where there is no git source of chart templates anyway? It's just the Helm chart. Maybe it just doesn't support this at all?

The correct way to helm template ASO today would be:

• helm repo add aso2 https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
• helm template aso2 aso2/azure-service-operator

This correctly renders everything.

In fact if you look closely at the 2.8.0 tag you'll realize that there isn't even a 2.8.0 chart in it today. That's because today our release process goes like this:

1. Tag v2.8.0
2. Tag creation kicks off github automation that runs a build which produces a container image and updates the release on GH to contain the rendered raw YAML (for CRDs + Deployment), as well as asoctl builds.
3. Once above step is done, PR is automatically opened to update Helm to the 2.8.0 version

The above automation ends up resulting in the Helm chart being added the commit after the 2.8.0 tag. We're expecting folks to consume the v2/charts/ folder from main though as a repository of Helm index file (similar to https://charts.jetstack.io really).

Is there a way to instruct Argo to just use the actual Helm chart tgz rather than the repo source?

[jtklahti]

Oh my... Turns out, this was in my end after all! Just changing the configuration like this solved the issue:

  source:
    repoURL: https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
    targetRevision: v2.8.0
    chart: azure-service-operator
    helm:

So that's a good example of how to install ASO with ArgoCD. With this config, internally ArgoCD seems to fetch the tgz package, extract the archive and then does the same helm templating process that is done with git.

Thanks for the help!

[matthchr]

This does seem like it would be an easy mistake to make. Leaving this issue open for a bit to discuss internally if we should mention this in our docs - maybe the FAQ