Skip to content
This repository has been archived by the owner on Jul 23, 2024. It is now read-only.

feat: Implement first phase of move2kube workflow #375

Merged

Conversation

eloycoto
Copy link
Collaborator

This commit adds the first phase of the move2kube workflow example. A user give any git repo, and it'll will be transformed using move2kube, two new folders will be commited (deploy and scripts) inside the repository where the transformed application can be used.

This PR introduces a new 3party dependency that it's the openapi client from move2kube.

The workflow is the following one:

sequenceDiagram
    participant User
    participant Parodos
    participant API as Move2Kube API
    participant web as Move2kube Web
    participant k8s as OpenshiftCluster

    rect rgb(221, 255, 221)
    User ->> Parodos: Enter params
    Note over User,Parodos: Parameters needed:<br/>- Git Repo<br/>- Git Credentials<br/>- GitTargetBranch<br/>- Profile as workspaceID?
    end

    rect rgb(221, 255, 221)
    Parodos ->> API: Get Workspace
    API ->> Parodos: return workspace ID
    Parodos ->> API: Post Project (Workflow ID as project ID)
    API ->> Parodos: Return project ID
    end

    Note over Parodos,API: Here Parodos prepare the plan
    rect rgb(221, 255, 221)
    Parodos --> Parodos: Git Clone
    Parodos --> Parodos: Git archive
    end

    rect rgb(221, 255, 221)
    Parodos --> API: Push ZIP file as project input1 POST /workspaces/{workspace-id}/projects/{project-id}/inputs

    Parodos ->> API: POST /workspaces/{workspace-id}/projects/{project-id}/plan
    Parodos ->> API: Get /workspaces/{workspace-id}/projects/{project-id}/plan

    Note over API,User: Here user get notified to start transformation
    Parodos ->> User: Link to project-id move2kube page
    User ->> web: User execute the transformation step in move2kube

    loop Every minute
        Parodos ->> API: Is transformation ready?
    end

    Note over API,User: Trasnformation completed successfully
    Parodos ->> API: Get outputs
    Parodos ->> Parodos: Branch code
    Parodos ->> Parodos: Commit changes

    end

    Parodos ->> Parodos: Push code to the specified branch

    Note over Parodos: Openshift deployment
    Parodos ->> Parodos: Build images for project
    Parodos ->> k8s: Apply all manifests
    Parodos ->> User: Workflow completed successfully with route
Loading

All the green boxes are created, the not-highlight ones are the ones for the next PR.

For move2kube an workspace need to be created, and the config that I used is the following one:

move2kube:
  containerruntime: docker
  minreplicas: "2"
  services:
    demo:
      "8080":
        servicetype: Ingress
        urlpath: /
      enable: true
      port: "8080"
  target:
    default:
      clustertype: Kubernetes
      ingress:
        host: localhost
        ingressclassname: ""
        tls: ""
    imageregistry:
      quay.io:
        logintype: no authentication
      namespace: parodos
      url: quay.com
  transformers:
    types:
      - ArgoCD
      - Buildconfig
      - ClusterSelector
      - ComposeAnalyser
      - ComposeGenerator
      - ContainerImagesPushScriptGenerator
      - DockerfileDetector
      - DockerfileImageBuildScript
      - DockerfileParser
      - DotNetCore-Dockerfile
      - Gradle
      - Jar
      - Jboss
      - Kubernetes
      - KubernetesVersionChanger
      - Liberty
      - Maven
      - Parameterizer
      - ReadMeGenerator
      - Tomcat
      - WarAnalyser
      - WarRouter
      - WinWebApp-Dockerfile
      - ZuulAnalyser
  transformerselector: ""

To run move2kube the easiest way that I find is the following command:

docker run --rm -it -p 8081:8080 quay.io/konveyor/move2kube-ui

Fix: https://issues.redhat.com/browse/FLPATH-356
Partial-Fix: https://issues.redhat.com/browse/FLPATH-355

Demo: https://drive.google.com/file/d/1k9RwRI4PPBwinDWRyARvTsbFGiQD6vjh/view?usp=sharing

@openshift-ci openshift-ci bot requested review from RichardW98 and ydayagi May 25, 2023 11:03
@eloycoto eloycoto force-pushed the Move2KubeExample branch from 83b6325 to 0ccc391 Compare May 25, 2023 12:02
@pkliczewski pkliczewski requested review from rgolangh and anludke May 25, 2023 12:32
@eloycoto eloycoto force-pushed the Move2KubeExample branch from 0ccc391 to ae2e4e7 Compare May 25, 2023 13:46

@Slf4j
@AllArgsConstructor
public class GitCommitTask extends BaseWorkFlowTask {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see opportunity to share the code with GitBranchTask. There is good amount of the same code.

Copy link
Collaborator Author

@eloycoto eloycoto May 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because the getRepo you mean? It's overwritten after in the workflow examples. the other code I cannot see a room to share the code with

workflow-examples/pom.xml Outdated Show resolved Hide resolved
}

private String getMove2KubeAPIEndpoint() {
return "http://localhost:8081/api/v1";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should make it configurable

}

@Bean(name = "transformWorkFlowChecker")
@Checker(cronExpression = "*/5 * * * * ?")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much time transformation is taking? Do you think checking it so often make sense?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a human process, so can take a lot, but for demo purposes is great.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be defined only by code? meaning, changing the corn-expression requires releasing a new version of the workflow?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, is the only way.

planApi.startPlanning(workspaceID, projectID);

for (int i = 1; i <= 10; ++i) {
GetPlan200Response plan = planApi.getPlan(workspaceID, projectID);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we want to wait exponentially here to get the plan? We could try using async instead. Please take a look how @gciavarrini implemented api wait.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thing is that it's not waiting at all. it's returning 404 does not exists.. so we need to wait until the plan is created

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was exactly the reason why Gloria implemented it in this way.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohh, got it. The main problem that I can see, I need to implement the async way for move2kube API for an example workflow, does that make sense?

"You need to complete some information for your transformation in the following url <a href=\"%s\"> %s</a>",
url, url);

// @TODO userID is the ID, but we need the username, so hardcode it here for now.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please create a task to track this change

@eloycoto eloycoto force-pushed the Move2KubeExample branch 6 times, most recently from a4cae90 to 94b1235 Compare May 26, 2023 10:29
@RichardW98
Copy link
Contributor

somehow the workflow-examples' test is still reported in coverage, shall we remove it and only runs it in maven build?

@eloycoto eloycoto force-pushed the Move2KubeExample branch from 94b1235 to 95193fd Compare May 26, 2023 16:30

private static File createSampleZip() throws IOException {
String zipFileName = "/tmp/test%s.zip".formatted(UUID.randomUUID());
String tempDir = "/tmp/%s/".formatted(UUID.randomUUID().toString());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an API for that: java.nio.file.Files.createTempFile

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know, and I do not need to create the file at all, only the path for creating a sample ZIP:

	private static File createSampleZip() throws IOException {
		String zipFileName = "/tmp/test%s.zip".formatted(UUID.randomUUID());
		String tempDir = "/tmp/%s/".formatted(UUID.randomUUID().toString());
		File zipFile = new File(zipFileName);
		String[] fileNames = { "output/project-name/deploy/bar.txt", "output/project-name/deploy/foo.txt",
				"output/project-name/scripts/bar.sh", "output/project-name/scripts/foo.sh" };

		byte[] buffer = new byte[1024];

		FileOutputStream fos = new FileOutputStream(zipFile);
		ZipOutputStream zos = new ZipOutputStream(fos);

		for (String fileName : fileNames) {
			File file = new File("%s%s".formatted(tempDir, fileName));
			if (file.getParentFile() != null && !file.getParentFile().exists()) {
				assertTrue(file.getParentFile().mkdirs());
			}
			assertTrue(file.createNewFile());

			ZipEntry zipEntry = new ZipEntry(fileName);
			zos.putNextEntry(zipEntry);

			FileInputStream fis = new FileInputStream(file);
			int length;
			while ((length = fis.read(buffer)) > 0) {
				zos.write(buffer, 0, length);
			}

			fis.close();
			zos.closeEntry();
		}

		zos.close();
		return zipFile;
	}

WorkReport report = task.execute(workContext);

// then
assertNull(report.getError());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. pls prefer the use of assertThat.
  2. The order of parameters counts as it used to generate the error message: assertXYX(expected, actual)


private static String move2KubeProjectIDCtxKey = "move2KubeProjectID";

PlanApi planApi;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

private?


PlanApi planApi;

ProjectOutputsApi projectOutputsApi;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

private?

@eloycoto eloycoto force-pushed the Move2KubeExample branch from 95193fd to 0c41e89 Compare May 29, 2023 13:26
eloycoto added 2 commits May 29, 2023 15:35
This commit adds the first phase of the move2kube workflow example. A
user give any git repo, and it'll will be transformed using move2kube,
two new folders will be commited (deploy and scripts) inside the
repository where the transformed application can be used.

This PR introduces a new 3party dependency that it's the openapi client
from move2kube.

The workflow is the following one:

```
sequenceDiagram
    participant User
    participant Parodos
    participant API as Move2Kube API
    participant web as Move2kube Web
    participant k8s as OpenshiftCluster

    rect rgb(221, 255, 221)
    User ->> Parodos: Enter params
    Note over User,Parodos: Parameters needed:<br/>- Git Repo<br/>- Git Credentials<br/>- GitTargetBranch<br/>- Profile as workspaceID?
    end

    rect rgb(221, 255, 221)
    Parodos ->> API: Get Workspace
    API ->> Parodos: return workspace ID
    Parodos ->> API: Post Project (Workflow ID as project ID)
    API ->> Parodos: Return project ID
    end

    Note over Parodos,API: Here Parodos prepare the plan
    rect rgb(221, 255, 221)
    Parodos --> Parodos: Git Clone
    Parodos --> Parodos: Git archive
    end

    rect rgb(221, 255, 221)
    Parodos --> API: Push ZIP file as project input1 POST /workspaces/{workspace-id}/projects/{project-id}/inputs

    Parodos ->> API: POST /workspaces/{workspace-id}/projects/{project-id}/plan
    Parodos ->> API: Get /workspaces/{workspace-id}/projects/{project-id}/plan

    Note over API,User: Here user get notified to start transformation
    Parodos ->> User: Link to project-id move2kube page
    User ->> web: User execute the transformation step in move2kube

    loop Every minute
        Parodos ->> API: Is transformation ready?
    end

    Note over API,User: Trasnformation completed successfully
    Parodos ->> API: Get outputs
    Parodos ->> Parodos: Branch code
    Parodos ->> Parodos: Commit changes

    end

    Parodos ->> Parodos: Push code to the specified branch

    Note over Parodos: Openshift deployment
    Parodos ->> Parodos: Build images for project
    Parodos ->> k8s: Apply all manifests
    Parodos ->> User: Workflow completed successfully with route
```

All the green boxes are created, the not-highlight ones are the ones for the
next PR.

For move2kube an workspace need to be created, and the config that I used is
the following one:

```
move2kube:
  containerruntime: docker
  minreplicas: "2"
  services:
    demo:
      "8080":
        servicetype: Ingress
        urlpath: /
      enable: true
      port: "8080"
  target:
    default:
      clustertype: Kubernetes
      ingress:
        host: localhost
        ingressclassname: ""
        tls: ""
    imageregistry:
      quay.io:
        logintype: no authentication
      namespace: parodos
      url: quay.com
  transformers:
    types:
      - ArgoCD
      - Buildconfig
      - ClusterSelector
      - ComposeAnalyser
      - ComposeGenerator
      - ContainerImagesPushScriptGenerator
      - DockerfileDetector
      - DockerfileImageBuildScript
      - DockerfileParser
      - DotNetCore-Dockerfile
      - Gradle
      - Jar
      - Jboss
      - Kubernetes
      - KubernetesVersionChanger
      - Liberty
      - Maven
      - Parameterizer
      - ReadMeGenerator
      - Tomcat
      - WarAnalyser
      - WarRouter
      - WinWebApp-Dockerfile
      - ZuulAnalyser
  transformerselector: ""
```

To run move2kube the easiest way that I find is the following command:
```
docker run --rm -it -p 8081:8080 quay.io/konveyor/move2kube-ui
```

Fix: https://issues.redhat.com/browse/FLPATH-356
Partial-Fix: https://issues.redhat.com/browse/FLPATH-355

Demo: https://drive.google.com/file/d/1k9RwRI4PPBwinDWRyARvTsbFGiQD6vjh/view?usp=sharing

Signed-off-by: Eloy Coto <eloy.coto@acalustra.com>
Signed-off-by: Eloy Coto <eloy.coto@acalustra.com>
@eloycoto eloycoto force-pushed the Move2KubeExample branch 8 times, most recently from 5c6410e to 7a36e95 Compare May 29, 2023 16:11
@eloycoto
Copy link
Collaborator Author

eloycoto commented May 29, 2023

70% of coverage already 🎉

@masayag

#375 (comment)

The only thing missing is the getRepo in the GitTask. I would like to keep like
that for the shake of making it clear, but it's up to you

#375 (comment)

This is something that we need to revisit. Are you happy if I create a new task
for this?

Almost all the assertTrue,assertEquals are now in assertThat; maybe one is missing, but I didn't spot it.

@piotr,

Can I merge this and work on the async of the client on a different PR? That will unlock the git push prebuild-task

Signed-off-by: Eloy Coto <eloy.coto@acalustra.com>
@eloycoto eloycoto force-pushed the Move2KubeExample branch from 7a36e95 to 92630f7 Compare May 29, 2023 18:10
@openshift-ci
Copy link

openshift-ci bot commented May 29, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RichardW98

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@masayag
Copy link
Collaborator

masayag commented May 29, 2023

/lgtm

@openshift-ci openshift-ci bot added the lgtm label May 29, 2023
@openshift-merge-robot openshift-merge-robot merged commit bcb9821 into rhdhorchestrator:main May 29, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants