Importing several private projects in a single devfile fails

[rromannissen]

Describe the bug

I have tried importing a devfile with multiple projects stored in private GitHub repositories using the /f?url= API. The result is that Che fails to clone the repositories without even prompting for the credentials. Only the password request for the last repository is prompted (without previously asking for the user):

After entering the right password, that last clone fails as well and the projects list ends up empty:

The devfile used to reproduce this error is the following:

apiVersion: 1.0.0
metadata:
  name: order-management
projects:
- name: gateway
  source:
    type: git
    location: https://github.com/rromannissen/gateway-crw.git
    branch: master
- name: orders
  source:
    type: git
    location: https://github.com/rromannissen/orders-crw.git
    branch: master
- name: inventory
  source:
    type: git
    location: https://github.com/rromannissen/inventory-crw.git
    branch: master
- name: customers
  source:
    type: git
    location: https://github.com/rromannissen/customers-crw.git
    branch: master

Che version

• latest
• nightly
• other: please specify

Che-Theia@5e48d39 using Theia@e2098321a in https://che.openshift.io and Che-Theia@4587314 using Theia@eb1136b11 in CodeReady Workspaces in OCP 4.6.4.

Steps to reproduce

The devfile used to reproduce this error is the following:

apiVersion: 1.0.0
metadata:
  name: order-management
projects:
- name: gateway
  source:
    type: git
    location: https://github.com/rromannissen/gateway-crw.git
    branch: master
- name: orders
  source:
    type: git
    location: https://github.com/rromannissen/orders-crw.git
    branch: master
- name: inventory
  source:
    type: git
    location: https://github.com/rromannissen/inventory-crw.git
    branch: master
- name: customers
  source:
    type: git
    location: https://github.com/rromannissen/customers-crw.git
    branch: master

Just import it using the /f?url= API.

Expected behavior

Che asks for the credentials of each repository and executes the clone correctly for all projects.

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Reproduced in https://che.openshift.io and CodeReady Workspaces 2.5.0 in OCP 4.6.4.

Installation method

• chectl
  • provide a full command that was used to deploy Eclipse Che (including the output)
  • provide an output of chectl version command
• OperatorHub
• I don't know

OperatorHub for CodeReady Workspaces 2.5.0 in OCP 4.6.4.

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Reproduced in https://che.openshift.io and CodeReady Workspaces 2.5.0 in OCP 4.6.4.

[skabashnyuk]

Yes. Looks like UX is not ideal.
From a technical POV, you can mount GitHub personal access token as described here. https://www.eclipse.org/che/docs/che-7/end-user-guide/mounting-a-secret-as-a-file-or-an-environment-variable-into-a-workspace-container/#mounting-a-git-credential-store-into-a-workspace-container_mounting-a-secret-as-a-file-or-an-environment-variable-into-a-workspace-container

[skabashnyuk]

or this way https://www.eclipse.org/che/docs/che-7/end-user-guide/version-control/#accessing-a-git-repository-via-ssh_version-control

[skabashnyuk]

@l0rd @vinokurig @mshaposhnik
please correct me if I'm wrong. In an ideal world, Theia could look in git.config/.ssh folder depending on git configuration and could initial necessary processes or correctly inform the user about expected actions. Or we could move that to the factory resolution step before Thia. I'm not sure what Is best. WDYT

[rromannissen]

Yes. Looks like UX is not ideal.
From a technical POV, you can mount GitHub personal access token as described here. https://www.eclipse.org/che/docs/che-7/end-user-guide/mounting-a-secret-as-a-file-or-an-environment-variable-into-a-workspace-container/#mounting-a-git-credential-store-into-a-workspace-container_mounting-a-secret-as-a-file-or-an-environment-variable-into-a-workspace-container

As you said, that solves the problem from a technical POV, but that wouldn't be an option for developers without knowledge about Kubernetes. We are evaluating Che/CRW as the standard development platform for customers with several hundreds of developers with various levels of knowledge/experience with Kubernetes. In large companies, development flows are usually abstracted from the platform for developers, and since Che would be aimed to replace local development environments, Kubernetes knowledge shouldn't be a requirement to use it.

Having this implemented in the UI would greatly ease adoption for that kind of companies, as private repositories are the usual scenario there.

[ericwill]

This should have been fixed (at least for HTTPS) in #16401

We can investigate next sprint.

[skabashnyuk]

@ericwill wdyt about #18494 (comment)? At which stage, we should validate the existence of ssh keys or git-config?

[l0rd]

@l0rd @vinokurig @mshaposhnik
please correct me if I'm wrong. In an ideal world, Theia could look in git.config/.ssh folder depending on git configuration and could initial necessary processes or correctly inform the user about expected actions. Or we could move that to the factory resolution step before Thia. I'm not sure what Is best. WDYT

@skabashnyuk agreed. In Theia there should be a git authentication helper that looks at git config or ssh keys before the clone starts and makes things work if it find them. Prompt the user otherwise. But for GitHub that flow should already be possible if the admin adds GitHub as an identity provider in Keycloak. We have created an issue to make it straightforward.

@rromannissen a couple of considerations:

1. if you are using GitHub you may want to enable GitHub authentication. That will enable the VS Code GitHub authentication flow. Be aware that we recently fixed a bug (will be included in CRW 2.5.1)

2. in a scenario with hundreds of Che/CRW users, I would want to provision the namespace for each of those users (including RAM/CPU quotas and LimitRanges of that namespace) before the user starts creating workspaces. Such a process will hopefully be automated (using something like this for instance) and secrets containing an automatically generated ssh key pair could be included in that same flow. With that flow, CRW/Che users should only upload the automatically generated public SSH key (sent to them via an email for example) on GitHub.

[ericwill]

So I was under the impression that factories are gone, right? Doesn't that mean that:

I have tried importing a devfile with multiple projects stored in private GitHub repositories using the /f?url= API.

is not supposed to work?

[l0rd]

factories are gone, right

@ericwill factories are not gone, it's just that it's not possible to manage factories from the dashboard anymore. The factory menu item has been removed and the corresponding API.

[ericwill]

Ah, okay. So I investigated and it seems that the factory API portion of the workspace plugin in che-theia (FactoryInitializer) was removed (because there is no factory API anymore), which explains why this is failing. Currently we are checking credentials for cloning git repos, but if it's done via factory this check doesn't happen.

I think this kind of answers your question @skabashnyuk.

I'll have someone from my team investigate further to see what our options are.

[sunix]

I am having a look

[sunix]

The problem is that the two git clone commands are executed in parallel.
And what has been implemented is not supporting that eclipse-che/che-theia#727

I don't think it make sense to have the commands executed in sequence. Will probably ask the second one to wait for the result of the first one.

[sunix]

so finally implemented in eclipse-che/che-theia#959:

• clone is still done in parallel but each prompt to the user is done in secquence.
• Only ask for credentials once if cloning several repos from the same host.