ENV variables from "params"

[rosscdh]

Is your feature request related to a problem? Please describe.
Hi there, firstly thanks for a really well composed system (a few rough edges but all together great!)

Describe the solution you'd like
Id like to know the possibility of allowing argocd app set guestbook -p MY_VAR=MY_VALUE to be used as environment vars with kustomize .. as there are certain workflows that would require this.

Have you thought about contributing yourself?

Would love to.. need to powerup the go skills tho

[alexec]

This looks like a neat feature. As I mentioned in Slack:

Anyone wanting to contribute but worried you don't know Go, spend 2h on the tutorial https://tour.golang.org, I read this before lunch and was commit changes before my first afternoon coffee.

This is where you'd need to add this:

• https://github.com/argoproj/argo-cd/blob/master/util/kustomize/kustomize.go#L135

[rosscdh]

Ok so, using Cobera for cli, its then a case of

1. find the Params hash
2. decide on creating a new proj subcommand probably argocd proj upsert-envvar|delete-envvar NAME=VALUE
3. write a generic env method to add/remove env vars to the cmd.Env
4. find the kustomize command caller and use the updated cmd.Env when calling kustomize and use envsubst or similar to pipe to content

question

1. using kustomize build or the new kubctl -k may be worth investigating the new kustomze integration

Sound about right?

[alexec]

@rosscdh you're bang-on right.

A thought I have is that with kubectl, you have --env="DNS_DOMAIN=cluster" (note we can't use --env, but it's already used, but we could use -e. Environment variables be a nice feature for all the tools (Helm, Ksonnet, plain YAML), but just do Kustomize today.

This would be ... more complex:

1. In types.go to ApplicationSource a Envmap[string]string.
2. Generate the code using make codegen.

Optionally, any of the following depending on how you personally want to set/see the env, change the following:

• NewApplicationCreateCommand
• NewApplicationSetCommand
• NewApplicationGetCommand

And maybe display in the UI somewhere.

Don't implement something you won't use mind you!

@jessesuen thoughts?

[rosscdh]

Unfortunately I am on holiday for 2 weeks in FR if i get a chance ill hack otherwise will do when i get back (if someone else hasn't claimed it)

[rosscdh]

on codegen it seems redis-ha has been removed from : https://kubernetes-charts.storage.googleapis.com

+ helm dependency update ./chart --skip-refresh
Error: Can't get a valid version for repositories redis-ha. Try changing the version constraint in requirements.yaml

[rosscdh]

Manually downloaded and now proceeding (strange internet pipes here)

[rosscdh]

#1718 still [WIP]

[alexec]

@rosscdh you mention that there are examples when you'd use this for Kustomize?

[alexec]

I think you could achieve this now using a custom plugin + envsubst.

[rosscdh]

Yes thats what i thought initially too; in fact i have one build that still makes use of it for secrets atm. Coudl oyu point me at the custom plugin please?

…

On Wed, Jul 3, 2019 at 12:09 AM Alex Collins ***@***.***> wrote: I think you could achieve this now using a custom plugin + envsubst. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1705?email_source=notifications&email_token=AADA6MVSAWQVEY537WLBRFLP5PGZTA5CNFSM4HVCU4BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZCWCGY#issuecomment-507863323>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADA6MWP3WD4A3MKBQPIGZDP5PGZTANCNFSM4HVCU4BA> .

[rosscdh]

Ah https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/ yes that would def work, tho would be better integrated with vaule or similar for secrets. https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/ tho hacking things into the pod is a little anti-patterny.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ptux]

I want set env vars into argocd which kustomize could use.
Something like circle ci does.

[rosscdh]

@ptux I had the same thought, however you should not be using dynamic variables in kustomize (there is a specific reason why they opted against it)

[alexec]

#2415 supports some special env vars for Helm/Jsonnet. I don't think we can do this for Kustomize because we invoke kustomize build . and this does not have parameters nor take the env into account. Am I missing something?

[ptux]

$ kustomize version
Version: {KustomizeVersion:3.1.0 GitCommit:95f3303493fdea243ae83b767978092396169baf BuildDate:2019-07-26T19:21:45+01:00 GoOs:darwin GoArch:amd64}

Here is what I can I do on local .

 $ tree
.
├── kustomization.yaml
└── secret.env

0 directories, 2 files

$ cat kustomization.yaml 
namespace: hello-cloudbuild-dev

secretGenerator:
- name: mysecret
  envs:
  - secret.env
  type: Opaque

generatorOptions:
  disableNameSuffixHash: true

$ cat secret.env 
SECRET_USERNAME
SECRET_PASSWORD

$ env | grep SECRET
SECRET_PASSWORD=password2
SECRET_USERNAME=admin2

$ kustomize build .
apiVersion: v1
data:
  SECRET_PASSWORD: cGFzc3dvcmQy
  SECRET_USERNAME: YWRtaW4y
kind: Secret
metadata:
  name: mysecret
  namespace: hello-cloudbuild-dev
type: Opaque

So, I want to add env to argo CD server.
Somthing like this.

argocd app creat appname --env SECRET_USERNAME=admin2 --env SECRET_PASSWORD=password2

[alexec]

So you want Argo CD to create the secret.env file?

[ptux]

Well, technically speaking, I want Argo CD to create values for env vars.

secret.env file just only holds the keys of env vars but not the values.

SECRET_USERNAME
SECRET_PASSWORD

So we could push secret.env to the git repo.

[alexec]

OK. So you need a file containing a list of the available envvars so you can use them in your secrets? Where do the values of those vars come from?

https://github.com/kubernetes-sigs/kustomize/blob/master/examples/secretGeneratorPlugin.md#secret-values-from-local-files

[anarcher]

I would like to set env vars into argocd which kustomize could use.
IMHO, kustomize custom plugin can use the envs

[olegstepura]

I've created a simple docker container image containing envsubst which can be used as config management plugin to post-process manifests replacing ENV vars. Usage example in the readme: olegstepura/glibc-envsubst

Idea is the following:

Now when your app will be synced by ArgoCD it will run kustomize build --enable-helm . | envsubst piping resulting multifile yaml to envsubst which will substitute ENV vars mounted from cluster-global-vars secret (and ARGOCD_* own ones as well) and resulting yaml with vars replaced will be applied to your cluster.

[pearj]

@olegstepura kustomize already supports environment variables, so you don't need envsubst, as long as the environment variables are available when you run kustomize. It is worth noting that environment variables are slightly fiddly to use with kustomize, but it does work, just not out of the box with ArgoCD.

[olegstepura]

Hi, @pearj !

Thanks! You must be talking about this feature, right?

For some reason I could not make it work. In this example:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
- name: metallb
  version: 0.12.1
  repo: https://metallb.github.io/metallb
  releaseName: metallb
  valuesFile: values.yaml

# values.yaml
configInline:
  address-pools:
    - name: default
      protocol: layer2
      addresses:
        - $(CLUSTER_IP_RANGE)

It keeps var intact after ArgoCD syncs it.

While in argocd-repo-server ENV var is defined:

$ kubectl exec --stdin --tty -n argocd argocd-repo-server-69cc9f789f-758cs  -- env | grep CLUSTER_IP_RANGE
Defaulted container "argocd-repo-server" out of: argocd-repo-server, download-tools (init), copyutil (init)
CLUSTER_IP_RANGE=10.10.10.30-10.10.10.44

I would be glad to get rid of a complex plugin setup if it would work. Any idea what am I missing here?

Edit: I tried both ArgoCD own detection of repo type and using a plugin that simply runs kustomize build --enable-helm .. Both do not replace $(CLUSTER_IP_RANGE)

Edit2: Does not work this way either:

# kustomization.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: metallb
spec:
  destination:
    namespace: metallb-system
  source:
    repoURL: https://metallb.github.io/metallb
    chart: metallb
    targetRevision: 0.12.1
    helm:
      values: |
        configInline:
          address-pools:
            - name: default
              protocol: layer2
              addresses:
                - $(CLUSTER_IP_RANGE)

[pearj]

Hi @olegstepura,

It's only a recently documented feature of being able to use environment variables to populate vars in kustomize.

However, the main catch is that ArgoCD doesn’t let you specify environment variables in the application definition. I’m pretty sure that is what this issue is actually about. It doesn’t look like it’d actually be that hard a change in go to implement and I’m pondering doing it myself.

Until argocd supports it you’d still need a plugin setup that calls kustomize but in your use case you might be able to remove the envsubst step, and use kustomize var substitution.

I did notice that the plugin section in argocd applications do support specifying environment variables. So you could pass in the values you want there and then when you call kustomize it should use your environment variables a kustomize vars if you configure it right.

Here are my notes on using environment variables:

Using Environment Variables

Kustomize can use environment variables and substitute those variables.

It needs the following:

• A ConfigMapGenerator
• A var definition
• A var reference in Kustomize configuration
  • This can apply to data parts of a ConfigMap for example
  • To set a specific label, the var needs a specific path, eg spec/template/metadata/labels/aadpodidbinding
• Reference the variable with $(VARIABLE) syntax

config.env:

AADPODIDBINDING

kustomization.yaml:

patchesStrategicMerge:
- |-
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: secret-agent-controller-manager
    namespace: secret-agent-system
  spec:
    template:
      metadata:
        labels:
          aadpodidbinding: $(AADPODIDBINDING)

configMapGenerator:
- name: secret-agent-config
  envs:
    - config.env

# Not strictly necessary, but allows for a config map with a stable name
generatorOptions:
  disableNameSuffixHash: true

vars:
- name: AADPODIDBINDING
  fieldref:
    fieldpath: data.AADPODIDBINDING
  objref:
    apiVersion: v1
    kind: ConfigMap
    name: secret-agent-config

configurations:
  - kustomizeConfig.yaml

kustomizeConfig.yaml:

varReference:
- kind: Deployment
  path: spec/template/metadata/labels/aadpodidbinding

Then if you call kustomize like this:

AADPODIDBINDING=abc1234 kustomize build . 

Reference:

• https://rm3l.org/using-system-envvars-with-kustomize/
• https://github.com/keleustes/kustomize/tree/allinone/examples/issues/issue_1734
• https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/#configmapgenerator

[tornike]

I am using custom plugin just to pass custom environment variables and run kustomize build with them. I don't get it why kustomize plugin does not have env support like CMP. It's like argocd is pushing you towards using helm, but I don't want to use helm.

In addition to all of these, image-updater only supports kustomize and helm plugins, so not having possibility to pass some parameters is a big disadvantage.

[bburky]

A while ago, I put together a repo demonstrating KV → ConfigMap → vars → string replacement in resources.
https://github.com/bburky/kustomize-transformer-environment-variables

The intended use case was Argo CD, but I never deployed it. I understand that Argo CD's native Kustomize support doesn't pass it any environment variables, and a Config Management Plugin (CMP) would be needed just to pick up the environment variables.

It's really not very nice Kustomize code, and notably leaves around a CM in your deployed environment. The vars documentation mentions "WARNING: There are plans to deprecate vars" and recommends using replacements instead (but it's string replacement is less powerful).

[qixiaobo]

Hi, @pearj !

Thanks! You must be talking about this feature, right?

For some reason I could not make it work. In this example:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
- name: metallb
  version: 0.12.1
  repo: https://metallb.github.io/metallb
  releaseName: metallb
  valuesFile: values.yaml

# values.yaml
configInline:
  address-pools:
    - name: default
      protocol: layer2
      addresses:
        - $(CLUSTER_IP_RANGE)

It keeps var intact after ArgoCD syncs it.

While in argocd-repo-server ENV var is defined:

$ kubectl exec --stdin --tty -n argocd argocd-repo-server-69cc9f789f-758cs  -- env | grep CLUSTER_IP_RANGE
Defaulted container "argocd-repo-server" out of: argocd-repo-server, download-tools (init), copyutil (init)
CLUSTER_IP_RANGE=10.10.10.30-10.10.10.44

I would be glad to get rid of a complex plugin setup if it would work. Any idea what am I missing here?

Edit: I tried both ArgoCD own detection of repo type and using a plugin that simply runs kustomize build --enable-helm .. Both do not replace $(CLUSTER_IP_RANGE)

Edit2: Does not work this way either:

# kustomization.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: metallb
spec:
  destination:
    namespace: metallb-system
  source:
    repoURL: https://metallb.github.io/metallb
    chart: metallb
    targetRevision: 0.12.1
    helm:
      values: |
        configInline:
          address-pools:
            - name: default
              protocol: layer2
              addresses:
                - $(CLUSTER_IP_RANGE)

Really appreciate

[zs-dima]

@olegstepura @pearj vars was deprecated in v5.0.0

[joebowbeer]

kustomize removed the apparent misfeature that allowed naked env assignments to inherit from the shell environment.

However, the requested feature does look like a job for a preprocessor running in a configuration management plugin (CMP).

The preprocessor would inject values into a ConfigMap resource.

[zs-dima]

FluxCD has substitution in this way currently:

https://fluxcd.io/flux/components/kustomize/kustomizations/#post-build-variable-substitution

postBuild:
  substituteFrom:
    - kind: ConfigMap
      name: variables-config
patches:
    - target:
        kind: Kustomization
        group: kustomize.toolkit.fluxcd.io
      patch: |-
        - op: add
          path: /spec/postBuild
          value:
            substituteFrom:
              - kind: ConfigMap
                name: variables-config

kubectl create configmap variables-config \
  --from-literal=DOMAIN=$DOMAIN \
  --from-literal=EMAIL=$EMAIL \
  -n flux-system

Then variables can be used in this way:

    external-dns.kubernetes.io/target: app.${DOMAIN}

Maybe Argo CD can it do the same or find some better solution

[zs-dima]

variables specially helpful to specify children apps ingress for the App of Apps pattern, could be nice to substitude them using root app Helm:

Root app:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    repoURL: https://github.com/user/repository
    path: apps
    env:
      - name: DOMAIN
        value: {{ .Values.app.domain }}

Children apps, as you can see helm values have different structure:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      valuesObject:
        ingressRoute:
          dashboard:
            matchRule: Host(`traefik.${DOMAIN}`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
            annotations:
              external-dns.kubernetes.io/target: traefik.${DOMAIN}

. . .

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      valuesObject:
        ingress:
          annotations:
            external-dns.kubernetes.io/target: longhorn.${DOMAIN}
          host: longhorn.${DOMAIN}

The workaround is to place all repository content under the root app templates folder, but this looks weird.

[mrkhachaturov]

variables specially helpful to specify children apps ingress for the App of Apps pattern, could be nice to substitude them using root app Helm:

Root app:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    repoURL: https://github.com/user/repository
    path: apps
    env:
      - name: DOMAIN
        value: {{ .Values.app.domain }}

Children apps, as you can see helm values have different structure:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      valuesObject:
        ingressRoute:
          dashboard:
            matchRule: Host(`traefik.${DOMAIN}`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
            annotations:
              external-dns.kubernetes.io/target: traefik.${DOMAIN}

. . .

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      valuesObject:
        ingress:
          annotations:
            external-dns.kubernetes.io/target: longhorn.${DOMAIN}
          host: longhorn.${DOMAIN}

The workaround is to place all repository content under the root app templates folder, but this looks weird.

Today I installed ArgoCD and trying to use variables for apps ingresses, that was a BIG surprise that ArgoCD dosn`t support such function.

[joebowbeer]

I suggest either closing this issue (as dup of #9165) or removing kustomize from the description.

kustomize does not by design provide a way to use build-time side effects from env vars or CLI args:

https://kubectl.docs.kubernetes.io/faq/kustomize/eschewedfeatures/#build-time-side-effects-from-cli-args-or-env-variables

A misfeature was even removed: kubernetes/website#35669

The devs would like to remove all vars from kustomize once their capability can be achieved through other means: kubernetes-sigs/kustomize#4049 (More discussion: kubernetes-sigs/kustomize#5046)

With respect to ArgoCD, I think a lot of this capabily can be achieved using templated ApplicationSet generators.

[zs-dima]

@joebowbeer the point is that capability can NOT achieved through other means.
therefore FluxCD provides kustomizations variables substitution:
https://fluxcd.io/flux/components/kustomize/kustomizations/#post-build-variable-substitution

[joebowbeer]

@zs-dima The description mentions using env vars with kustomize, which confused me.

What Flux supports is post-build substitution. This is an extra stage after kustomize build. (Not something kustomize build can/will do.)

kustomize build ./apps/ | flux envsubst --strict

Flux also supports post-render processing for helm. (See #3698 for ArgoCD RFE.)

I think a better title for this issue would mention post-build substitution.

What ArgoCD does provide at this time is CMP; the lovely plugin can perform post-build substitution:

https://github.com/crumbhole/argocd-lovely-plugin

[calizarr]

Technically, you could create a transformer that would create an env.properties file from environment variables and then that gets merged in via the kustomization.yaml with a configMapGenerator which would allow a parent Helm chart to provide values like aws region, aws account id, etc. This is possible using combination of config and replacements

Without using a parent Helm chart, adding to the standard build env vars would also work since those would allow a similar transformer to do the same work.

Regardless, it does require the ability to pass env variables to the ArgoCD basic kustomize. Otherwise one would have to create a CMP that is only for the purposes of injecting environment variables

[andrii-korotkov-verkada]

Is the use case still relevant? I think kustomize shouldn't really depend on env variables, since it'd make things not reproducible. But idk if there's a good alternative these days, so more information about workflows requiring it would be helpful. Sorry if it was mentioned already.