Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot install istio 1.2.2 from helm chart with helmfile #789

Closed
mojochao opened this issue Aug 5, 2019 · 11 comments
Closed

Cannot install istio 1.2.2 from helm chart with helmfile #789

mojochao opened this issue Aug 5, 2019 · 11 comments
Labels

Comments

@mojochao
Copy link

mojochao commented Aug 5, 2019

I am trying to install the istio-1.2.2 istio helm chart in istio-1.2.2/install/kubernetes/helm/istio. I have copied both it and the istio-init chart to the root directory of my helmfile repo.

The relevant portion of my helmfile.yaml is:

  - chart: ./istio-init
    name: istio-init
    namespace: istio-system
    values:
      - certmanager:
          enabled: false

  - chart: ./istio
    name: istio
    namespace: istio-system
    values:
      - certmanager:
          enabled: false
        grafana:
          enabled: false
        prometheus:
          enabled: false
        tracing:
          enabled: false
        kiali:
          enabled: false

When I run helmfile apply with just the istio-init chart in my helmfile.yaml file, that chart installs succesfully. When I run helmfile apply with the istio chart added back to my helmfile.yaml file, I see the following error:

$ helmfile apply
Building dependency istio-init
No requirements found in istio-init/charts.

Building dependency istio
in ./helmfile.yaml: helm exited with status 1:
  Error: no 'repository' field specified for dependency: "sidecarInjectorWebhook"

I can succesfully install it with helm install, however.

$ helm install ./istio --name istio --namespace istio-system --set certmanager.enabled=false --set grafana.enabled=false --set prometheus.enabled=false --set tracing.enabled=false --set kiali.enabled=false
<lots of output elided>
NOTES:
Thank you for installing istio.

Your release is named istio.

To get started running application with Istio, execute the following steps:
1. Label namespace that application object will be deployed to by the following command (take default namespace as an example)

$ kubectl label namespace default istio-injection=enabled
$ kubectl get namespace -L istio-injection

2. Deploy your applications

$ kubectl apply -f <your-application>.yaml

For more information on running Istio, visit:
https://istio.io/

What is the correct way to install the istio chart with helmfile? Is this a bug or misconfiguration on my part?

I am using the following versions running on macOS Mojave (10.14.6) against an AWS EKS 1.13 cluster:

$ helm version
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
$ helmfile --version
helmfile version v0.80.2

Many thanks in advance!

@mumoshu
Copy link
Collaborator

mumoshu commented Aug 6, 2019

@mojochao Hey!

I have not yet tried 1.2.2 but I do successfully installed 1.1.3 with helmfile.

I'd say that the order of installation is the key here. Could you try moving istio-init to a dedicated helmfile for ordering?

For instance your root helmfiel.yaml woud look like this:

helmfiles:
- istio-init.yaml
- istio.yaml

Where istio-init.yaml is:

repositories:
- name: istio.io
  url: https://storage.googleapis.com/istio-release/releases/1.1.3/charts/

releases:
- name: istio-init
  chart: istio.io/istio-init
  version: 1.1.3

And istio.yaml is:

repositories:
- name: istio.io
  url: https://storage.googleapis.com/istio-release/releases/1.1.3/charts/

releases:
- name: istio
  chart: istio.io/istio
  version: 1.1.3
  namespace: istio-system
  values:
  # snip

This way you can ensure that helmfile installs istio-init and istio in this order.

@naphta
Copy link
Contributor

naphta commented Aug 7, 2019

The only major hurdle you might face (which I work around) is that you need to wait for the Istio init CRD's to finish applying.

I'm doing this with a post sync script at the minute.

The script I have at present is:

#!/usr/bin/env bash
attempts=1
# Wait for the `istio-init` chart to finish and setup at least 3 of the CRD jobs. CHECK ON NEW ISTIO RELEASES.
while (( attempts <= 300 )); do
  result=$(kubectl get pods -n "${1}" --selector job-name -o jsonpath="{range.items[?(@.status.phase == 'Succeeded')]}{.metadata.name}:{end}" | tr ':' "\n" | wc -l | awk '{ print $1 }')
  echo "[A${attempts}] [istio-init] Jobs succeeded: ${result}"
  if [[ ${result} -eq 3 ]]; then
    echo "[A${attempts}] [istio-init] All jobs completed successfully. Exiting."
    exit 0
  fi
  ((++attempts))
  sleep 1
done

echo "[A${attempts}] [istio-init] Timed out waiting for jobs to complete. Exiting."
exit 1
templates:
  istio-init: &istio-init
    name: istio-init
    namespace: istio-system
    installed: true
    chart: istio/istio-init
    version: 1.2.0
    recreatePods: false
    tillerNamespace: {{ requiredEnv "TILLER_NAMESPACE" }}
    values:
      - global:
          imagePullPolicy: Always
          tag: 1.2.0
    hooks:
      - events:
          - postsync
        showlogs: true
        command: "/bin/bash"
        args:
          - "./wait-for-init.sh"
          - "{{`{{.Release.Namespace}}`}}"

releases:
  - <<: *istio-init

then of course I'm following the top level helmfile as @mumoshu suggested.

The only issue I'm having at the moment is that obviously you need kubectl installed where you run it (is there a possibility of bundling some extras into the docker image? jq and kubectl would be most welcome), and at present it doesn't check how many jobs actually exist to wait for; it's just hardcoded to 3.

@mumoshu
Copy link
Collaborator

mumoshu commented Aug 7, 2019

Wow, so does istio-init now contain something other than CRDs? That's annoying for sure 😢

Maybe it worth enhancing helmfile by adding a dedicated hook type for waiting until a job succeeds without needing kubectl and jq. Would you mind writing a feature request for that? I'm imagining something like this:

hooks:
      - events:
          - postsync
        # not sure how showlogs should work for this type of hook...
        #showlogs: true
        type: waiter
        # maybe we can provide a sensible default apiVersion per kind
        #apiVersion: v1/apps
        kind: Job
        # do you wanna use go template in name?
        name: job-name

@mumoshu
Copy link
Collaborator

mumoshu commented Aug 7, 2019

is there a possibility of bundling some extras into the docker image? jq and kubectl would be most welcome

I'm unsure how much effort I can put to maintain those dependencies to be up-to-date, but I'm willing to accept any contributions to include them.

@naphta
Copy link
Contributor

naphta commented Aug 7, 2019

@mumoshu It contains job pods which apply the CRDs, so sort of yes and no!

I did create an issue for adding in jq and kubectl but I'll create a PR with the changes, I'll see if I can pin kubectl to using latest stable on each build (which I guess is what helm is doing)

@mojochao
Copy link
Author

Thanks for all the suggestions, but I'm still having the same problem. I've created two independent helmfiles:

$ cat istio-init.yaml
environments:
  default:
    values:
      - istio_version: 1.2.4

helmDefaults:
  wait: true

templates:
  istio-init: &istio-init
    chart: ./external/istio-{{ .Environment.Values.istio_version }}/install/kubernetes/helm/istio-init
    name: istio-init
    namespace: istio-system
    installed: true
    version: {{ .Environment.Values.istio_version }}
    recreatePods: false
    tillerNamespace: kube-system
    values:
      - global:
          imagePullPolicy: Always
          tag: {{ .Environment.Values.istio_version }}
    hooks:
      - events:
          - postsync
        showlogs: true
        command: "/bin/bash"
        args:
          - "./wait-for-istio-init.sh"
          - "{{`{{.Release.Namespace}}`}}"

releases:
  - <<: *istio-init

and

$ cat istio.yaml
environments:
  default:
    values:
      - istio_version: 1.2.4

helmDefaults:
  wait: true

releases:
  - chart: ./external/istio-{{ .Environment.Values.istio_version }}/install/kubernetes/helm/istio
    version: {{ .Environment.Values.istio_version }}
    name: istio
    namespace: istio-system
    values:
        prometheus:
          enabled: false

Also created the shell script to wait for init completion mentioned above by @naphta

#!/usr/bin/env bash
# Wait for the `istio-init` chart to finish and setup at least 3 of the CRD jobs. CHECK ON NEW ISTIO RELEASES.
attempts=1
while (( attempts <= 300 )); do
  result=$(kubectl get pods -n "${1}" --selector job-name -o jsonpath="{range.items[?(@.status.phase == 'Succeeded')]}{.metadata.name}:{end}" | tr ':' "\n" | wc -l | awk '{ print $1 }')
  echo "[A${attempts}] [istio-init] Jobs succeeded: ${result}"
  if [[ ${result} -eq 3 ]]; then
    echo "[A${attempts}] [istio-init] All jobs completed successfully. Exiting."
    exit 0
  fi
  ((++attempts))
  sleep 1
done

echo "[A${attempts}] [istio-init] Timed out waiting for jobs to complete. Exiting."
exit 1

Modified my root helmfile.yaml with sub-helmfiles.

...
helmfiles:
  - istio-init.yaml
  - istio.yaml
...

Applying changes always results in:

λ helmfile apply
Building dependency external/istio-1.2.4/install/kubernetes/helm/istio-init
No requirements found in external/istio-1.2.4/install/kubernetes/helm/istio-init/charts.

Comparing istio-init external/istio-1.2.4/install/kubernetes/helm/istio-init

No affected releases
Building dependency external/istio-1.2.4/install/kubernetes/helm/istio
in ./helmfile.yaml: in .helmfiles[1]: in ./istio.yaml: helm exited with status 1:
  Error: no 'repository' field specified for dependency: "sidecarInjectorWebhook"

Does anyone have a repo with a working helmfile of istio that I could try out?

Many thanks everyone!

@mojochao
Copy link
Author

As another data point, just using helm install of both istio-init and istio charts, with no pause in between, results in success:

$ helm install $ISTIO_CHARTS_DIR/istio-init --name istio-init --namespace istio-system && helm install $ISTIO_CHARTS_DIR/istio --name istio --namespace istio-system --set prometheus.enabled=false
NAME:   istio-init
LAST DEPLOYED: Wed Aug 14 15:18:19 2019
NAMESPACE: istio-system
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                     AGE
istio-init-istio-system  1s

==> v1/ClusterRoleBinding
NAME                                        AGE
istio-init-admin-role-binding-istio-system  1s

==> v1/ConfigMap
NAME          DATA  AGE
istio-crd-10  1     1s
istio-crd-11  1     1s
istio-crd-12  1     1s

==> v1/Job
NAME               COMPLETIONS  DURATION  AGE
istio-init-crd-10  0/1          1s        1s
istio-init-crd-11  0/1          1s        1s
istio-init-crd-12  0/1          1s        1s

==> v1/Pod(related)
NAME                     READY  STATUS             RESTARTS  AGE
istio-init-crd-10-6nhnr  0/1    ContainerCreating  0         1s
istio-init-crd-11-8zfmp  0/1    ContainerCreating  0         1s
istio-init-crd-12-6txr8  0/1    ContainerCreating  0         1s

==> v1/ServiceAccount
NAME                        SECRETS  AGE
istio-init-service-account  1        1s


NAME:   istio
LAST DEPLOYED: Wed Aug 14 15:18:21 2019
NAMESPACE: istio-system
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                                 AGE
istio-citadel-istio-system           16s
istio-galley-istio-system            16s
istio-mixer-istio-system             16s
istio-pilot-istio-system             16s
istio-reader                         16s
istio-sidecar-injector-istio-system  16s

==> v1/ClusterRoleBinding
NAME                                                    AGE
istio-citadel-istio-system                              16s
istio-galley-admin-role-binding-istio-system            16s
istio-mixer-admin-role-binding-istio-system             16s
istio-multi                                             16s
istio-pilot-istio-system                                16s
istio-sidecar-injector-admin-role-binding-istio-system  16s

==> v1/ConfigMap
NAME                             DATA  AGE
istio                            2     16s
istio-galley-configuration       1     16s
istio-security-custom-resources  2     16s
istio-sidecar-injector           2     16s

==> v1/Deployment
NAME                    READY  UP-TO-DATE  AVAILABLE  AGE
istio-citadel           1/1    1           1          16s
istio-galley            0/1    1           0          16s
istio-ingressgateway    0/1    1           0          16s
istio-pilot             0/1    1           0          16s
istio-policy            1/1    1           1          16s
istio-sidecar-injector  0/1    1           0          16s
istio-telemetry         1/1    1           1          16s

==> v1/Pod(related)
NAME                                     READY  STATUS             RESTARTS  AGE
istio-citadel-657c84d86f-8bmqz           1/1    Running            0         16s
istio-galley-6d4c54fc76-bnbmp            0/1    ContainerCreating  0         16s
istio-ingressgateway-7f768f54c7-f62pz    0/1    Running            0         16s
istio-pilot-6b65d765b5-gx8mw             0/2    Pending            0         16s
istio-policy-5d7d7d557d-c89gb            2/2    Running            0         16s
istio-sidecar-injector-78949dd945-t9tsq  0/1    Running            0         15s
istio-telemetry-77797d4d8-kfxdn          2/2    Running            0         16s

==> v1/Role
NAME                      AGE
istio-ingressgateway-sds  16s

==> v1/RoleBinding
NAME                      AGE
istio-ingressgateway-sds  16s

==> v1/Service
NAME                    TYPE          CLUSTER-IP      EXTERNAL-IP       PORT(S)                                                                                                                                     AGE
istio-citadel           ClusterIP     10.100.54.218   <none>            8060/TCP,15014/TCP                                                                                                                          16s
istio-galley            ClusterIP     10.100.2.32     <none>            443/TCP,15014/TCP,9901/TCP                                                                                                                  16s
istio-ingressgateway    LoadBalancer  10.100.196.148  aa9ddfd6dbed0...  15020:32451/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31270/TCP,15030:30381/TCP,15031:32106/TCP,15032:32187/TCP,15443:31895/TCP  16s
istio-pilot             ClusterIP     10.100.218.234  <none>            15010/TCP,15011/TCP,8080/TCP,15014/TCP                                                                                                      16s
istio-policy            ClusterIP     10.100.8.126    <none>            9091/TCP,15004/TCP,15014/TCP                                                                                                                16s
istio-sidecar-injector  ClusterIP     10.100.164.17   <none>            443/TCP                                                                                                                                     16s
istio-telemetry         ClusterIP     10.100.251.193  <none>            9091/TCP,15004/TCP,15014/TCP,42422/TCP                                                                                                      16s

==> v1/ServiceAccount
NAME                                    SECRETS  AGE
istio-citadel-service-account           1        16s
istio-galley-service-account            1        16s
istio-ingressgateway-service-account    1        16s
istio-mixer-service-account             1        16s
istio-multi                             1        16s
istio-pilot-service-account             1        16s
istio-security-post-install-account     1        16s
istio-sidecar-injector-service-account  1        16s

==> v1alpha2/attributemanifest
NAME        AGE
istioproxy  15s
kubernetes  15s

==> v1alpha2/handler
NAME           AGE
kubernetesenv  15s
prometheus     15s

==> v1alpha2/instance
NAME                  AGE
attributes            15s
requestcount          15s
requestduration       15s
requestsize           15s
responsesize          15s
tcpbytereceived       15s
tcpbytesent           15s
tcpconnectionsclosed  15s
tcpconnectionsopened  15s

==> v1alpha2/rule
NAME                     AGE
kubeattrgenrulerule      15s
promhttp                 15s
promtcp                  15s
promtcpconnectionclosed  15s
promtcpconnectionopen    15s
tcpkubeattrgenrulerule   15s

==> v1alpha3/DestinationRule
NAME             AGE
istio-policy     16s
istio-telemetry  16s

==> v1beta1/ClusterRole
NAME                                      AGE
istio-security-post-install-istio-system  16s

==> v1beta1/ClusterRoleBinding
NAME                                                   AGE
istio-security-post-install-role-binding-istio-system  16s

==> v1beta1/MutatingWebhookConfiguration
NAME                    AGE
istio-sidecar-injector  15s

==> v1beta1/PodDisruptionBudget
NAME                    MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
istio-galley            1              N/A              0                    16s
istio-ingressgateway    1              N/A              0                    16s
istio-pilot             1              N/A              0                    16s
istio-policy            1              N/A              0                    16s
istio-sidecar-injector  1              N/A              0                    16s
istio-telemetry         1              N/A              0                    16s

==> v2beta1/HorizontalPodAutoscaler
NAME                  REFERENCE                        TARGETS        MINPODS  MAXPODS  REPLICAS  AGE
istio-ingressgateway  Deployment/istio-ingressgateway  <unknown>/80%  1        5        1         16s
istio-pilot           Deployment/istio-pilot           <unknown>/80%  1        5        0         15s
istio-policy          Deployment/istio-policy          <unknown>/80%  1        5        0         15s
istio-telemetry       Deployment/istio-telemetry       <unknown>/80%  1        5        0         15s


NOTES:
Thank you for installing istio.

Your release is named istio.

To get started running application with Istio, execute the following steps:
1. Label namespace that application object will be deployed to by the following command (take default namespace as an example)

$ kubectl label namespace default istio-injection=enabled
$ kubectl get namespace -L istio-injection

2. Deploy your applications

$ kubectl apply -f <your-application>.yaml

For more information on running Istio, visit:
https://istio.io/

@mumoshu
Copy link
Collaborator

mumoshu commented Aug 15, 2019

@mojochao From the error message, I think you're encountering helm/helm#5531 and helm/helm#3742.

How is your local istio charts under ./external obtained?

Anyways, try using the official helm charts repository like:

repositories:
- name: istio.io
  url: https://storage.googleapis.com/istio-release/releases/1.1.3/charts/

releases:
- name: istio-init
  chart: istio.io/istio-init
  version: 1.1.3
repositories:
- name: istio.io
  url: https://storage.googleapis.com/istio-release/releases/1.1.3/charts/

releases:
- name: istio
  chart: istio.io/istio
  version: 1.1.3
  namespace: istio-system

You should change 1.1.3 to whatever version you'd like to use. It is just that I tested it with 1.1.3 before :)

@mumoshu
Copy link
Collaborator

mumoshu commented Aug 15, 2019

If you do need to use the istio charts downloaded locally, try this workaround mentioned in the upstream issue

@mojochao
Copy link
Author

mojochao commented Aug 19, 2019

I decided to do as you suggest @mumoshu, and just use the Istio helm charts repo and everything works now with latest Istio v1.2.4. I did not have to use a shell script to wait for completion, and hope that future installs will be reliable without one.

Many thanks guys! So far, I'm finding helmfile a great fit for my needs, and appreciate all who've contributed to its development.

Cheers!

@TarekAS
Copy link

TarekAS commented Oct 23, 2019

Didn't work for me, unless I run helmfile apply twice. Reason is, by the time istio being installed, the istio-init pods may have not been completed, which causes the installation to fail due to missing CRDs.

Is it possible to wait a specific amount of time after a chart is installed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants