how to customize a manifest by params

[hougangliu]

Take the example as here, whether--namespace and --enable-gang-scheduling are appended into args is based on two params.

For kustomize manifest counterpart, 4 overlays is need to cover all the cases of its ksonnet version.

As for more params, kustomize overlays may exponential increase if we try to match ksonnet version kubeflow.

I think we should discuss about how to handle it.

@jlewi @kkasravi @swiftdiaries

[kkasravi]

@hougangliu @jlewi @swiftdiaries these won't be covered by overlays, rather by vars that are inserted into a ConfigMap.

I'll do this component and update the manifests README.md to show how this was done as an example. I'll open the PR this morning or early afternoon.

[kkasravi]

@hougangliu @jlewi @swiftdiaries @ashahba the enableGangScheduling boolean which, if true, updates the ClusterRole rules sections and adds a flag to the command in the deployment podtemplate will likely be done using a device overlay.
I'm thinking of enabling different overlays in the app.yaml under componentParams for example
app.yaml

...
  componentParams:
    tf-job-operator:
    - name: overlays
      value: [device]

Currently if a platform is specified to kfctl (kftcl init --platform gcp) then this overlay is automatically called. For additional overlays I was thinking of the approach above.

The deploymentNamespace is an early attempt to do multi-user namespaces and would likely be done via a profiles overlay which would involve a number of components and some changes to the tf-job-operator.

Feedback is encouraged.

[hougangliu]

@kkasravi good idea!

[jlewi]

Might this (gang scheduling) be a case where users have to define a new overlay combining the different options?

Can you describe the example in more detail? E.g what are the parameters and what are they controlling?

If IUUC there are different roles and bindings and different command line arguments.

What if we provided different overlays/manifests for the roles and bindings and added users to combine them with the correct command line flags (e.g edit the manifest or overlay to set the command line argument)?

[kkasravi]

i'll have a PR this morning for tf-job-operator, with an overview

[kkasravi]

@jlewi

Can you describe the example in more detail? E.g what are the parameters and what are they controlling?

If IUUC there are different roles and bindings and different command line arguments.

What if we provided different overlays/manifests for the roles and bindings and added users to combine them with the correct command line flags (e.g edit the manifest or overlay to set the command line argument)?

This is the approach being taken in #34.

└── tf-job-operator
    ├── base
    │   ├── config-map.yaml
    │   ├── crd.yaml
    │   ├── deployment.yaml
    │   ├── kustomization.yaml
    │   ├── params.env
    │   ├── params.yaml
    │   ├── service-account.yaml
    │   └── service.yaml
    └── overlays
        ├── cluster
        │   ├── cluster-role-binding.yaml
        │   ├── cluster-role.yaml
        │   ├── deployment.yaml
        │   └── kustomization.yaml
        ├── cluster-gangscheduled
        │   ├── cluster-role-patch.yaml
        │   ├── cluster-role.yaml
        │   ├── deployment-patch.yaml
        │   ├── deployment.yaml
        │   ├── kustomization.yaml
        ├── namespaced
        │   ├── deployment.yaml
        │   ├── kustomization.yaml
        │   ├── role-binding.yaml
        │   └── role.yaml
        └── namespaced-gangscheduled
            ├── deployment-patch.yaml
            ├── deployment.yaml
            ├── kustomization.yaml
            ├── role-patch.yaml
            └── role.yaml

The cluster overlay adds cluster-role and cluster-role-binding resources, the namespaced overlay
adds role and role-binding resources. For cluster, both deployments are modified to add KUBEFLOW envs

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: dashboard-deployment
spec:
  template:
    metadata:
    spec:
      containers:
      - name: dashboard-container
        env:
        - name: KUBEFLOW_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: deployment
spec:
  template:
    spec:
      containers:
      - name: container
        env:
        - name: KUBEFLOW_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace

However, I'm not sure I entirely understand how deploymentNamespace parameter should work with Profiles.
My understand was that the overlay for cluster would have

• crd with scope Cluster
• cluster-role, cluster-role-binding
• dashboard deployment, service-account, service in kubeflow namespace
• tf-operator deployment, service-account, service in kubeflow namespace

User submits a TFJob with no namespace specified.
How does the tf-operator know to run the TFJob in the user's Profile namespace?

If we have an overlay for namespaced then

• crd with scope Namespaced
• role, role-binding
• dashboard deployment, service-account, service in kubeflow namespace
• tf-operator deployment, service-account, service in kubeflow namespace

User submits a TFJob with the kubeflow namespace specified.
How does the tf-operator know to run the TFJob in the user's Profile namespace?

[kkasravi]

In both cases above, the 'deploymentNamespace' would only be used to specify a different namespace other that 'kubeflow' which - would we ever use this?. And (unless I'm missing something), the tf-operator is missing information on where to launch the TFJob for Profiles.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/question	0.73

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.