Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NO-ISSUE: Change the publication method of JIT Executor native binaries from NPM package to OS-specific Maven modules #2041

Merged
merged 9 commits into from
May 9, 2024
Merged

NO-ISSUE: Change the publication method of JIT Executor native binaries from NPM package to OS-specific Maven modules #2041

merged 9 commits into from
May 9, 2024

Conversation

tiagobento
Copy link
Contributor

Due to the fact that for the Apache release we'll need a buildable zip that produces all artifacts necessary for downstream repos, combined with the usage the native binaries of JIT Executor have on kie-tools, the publication method needed to change from an external GitHub Actions workflow publishing to NPM to Maven modules that are part of the build of this repo.

This change introduces a new profile "jitexecutor-native" that will, depending on the OS, include a specific . This is necessary because native compilation does not support cross-compilation, so we need to run three separate jobs, publishing three separate artifacts, each containing the binary targeting an OS.

The configurations were taken from jitexecutor-runner and replicated through the new OS-specific modules. We have tested this on @ljmotta's fork. See https://github.com/ljmotta/incubator-kie-kogito-apps/actions/runs/8993866752.

The release Jenkins job will still be responsible for publishing all the Maven modules in this repository, however, for jitexecutor-native, we need to rely on the GitHub Actions workflow, since we need binaries for macOS and Windows too. Both unsupported on Apache Jenkins' infrastructure.

@rodrigonull We haven't completed the changes to publish-jitexecutor-native.yml, since we don't know how to properly configure the Maven publication, especially with the custom classifier. Can you help us? We believe that the second mvn clean install for each OS should be changed to mvn clean deploy with the proper configuration, but we're not sure.

The new module jitexecutor-native is only included if the profile jitexecutor-native is explicitly set with -P.

@tiagobento
Copy link
Contributor Author

Need to configure an install:install-file execution during the install phase too, as JARs with custom classifiers apparently are not installed by default.

@tiagobento
Copy link
Contributor Author

To test if the install is successful: mvn dependency:unpack -Dartifact=org.kie.kogito:jitexecutor-native-darwin:999-SNAPSHOT:jar:binaries -Dmdep.unpack.includes='jitexecutor-native-*'

@kie-ci3
Copy link
Contributor

kie-ci3 commented May 8, 2024

PR job #2 was: UNSTABLE
Possible explanation: This should be test failures

Reproducer

build-chain build full_downstream -f 'https://raw.githubusercontent.com/${AUTHOR:apache}/incubator-kie-kogito-pipelines/${BRANCH:main}/.ci/buildchain-config-pr-cdb.yaml' -o 'bc' -p apache/incubator-kie-kogito-apps -u #2041 --skipParallelCheckout

NOTE: To install the build-chain tool, please refer to https://github.com/kiegroup/github-action-build-chain#local-execution

Please look here: https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/pullrequest_jobs/job/kogito-apps-pr/job/PR-2041/2/display/redirect

Test results:

  • PASSED: 1563
  • FAILED: 7

Those are the test failures:

org.kie.kogito.index.mongodb.ProcessDataIndexMongoDBHttpIT.testProcessInstanceEvents 1 expectation failed.
JSON path data.UserTaskInstances[0].potentialGroups[0] doesn't match.
Expected: managers
Actual: null
org.kie.kogito.index.mongodb.ProcessDataIndexMongoDBKafkaIT.testProcessInstanceEvents 1 expectation failed.
JSON path data.UserTaskInstances[0].potentialGroups[0] doesn't match.
Expected: managers
Actual: null
org.kie.kogito.index.ProcessDataIndexMongoDBIT.testProcessInstanceEvents 1 expectation failed.
JSON path data.UserTaskInstances[0].potentialGroups[0] doesn't match.
Expected: managers
Actual: null
org.kie.kogito.index.jdbc.mapper.JobEntityMapperIT.testMapToModel java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[error]: Build step io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor#startKeycloakContainer threw an exception: java.lang.RuntimeException: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/keycloak/keycloak:23.0.7
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startKeycloakContainer(KeycloakDevServicesProcessor.java:249)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849)
at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at java.base/java.lang.Thread.run(Thread.java:833)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/keycloak/keycloak:23.0.7
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:359)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:330)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.lambda$startContainer$4(KeycloakDevServicesProcessor.java:376)
at java.base/java.util.Optional.orElseGet(Optional.java:364)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startContainer(KeycloakDevServicesProcessor.java:401)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startKeycloakContainer(KeycloakDevServicesProcessor.java:197)
... 11 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:344)
... 16 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:563)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:354)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 17 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*Keycloak.*started.*'
at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:909)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:500)
... 19 more
org.kie.kogito.index.mongodb.query.DomainQueryIT.test java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[error]: Build step io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor#startKeycloakContainer threw an exception: java.lang.RuntimeException: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/keycloak/keycloak:23.0.7
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startKeycloakContainer(KeycloakDevServicesProcessor.java:249)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849)
at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at java.base/java.lang.Thread.run(Thread.java:833)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/keycloak/keycloak:23.0.7
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:359)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:330)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.lambda$startContainer$4(KeycloakDevServicesProcessor.java:376)
at java.base/java.util.Optional.orElseGet(Optional.java:364)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startContainer(KeycloakDevServicesProcessor.java:401)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startKeycloakContainer(KeycloakDevServicesProcessor.java:197)
... 11 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:344)
... 16 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:563)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:354)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 17 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*Keycloak.*started.*'
at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:909)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:500)
... 19 more
org.kie.kogito.index.postgresql.reporting.ProcessInstanceVariableMappingIT.testProcessInstanceVariableMapping java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[error]: Build step io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor#startKeycloakContainer threw an exception: java.lang.RuntimeException: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/keycloak/keycloak:23.0.7
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startKeycloakContainer(KeycloakDevServicesProcessor.java:249)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849)
at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at java.base/java.lang.Thread.run(Thread.java:833)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/keycloak/keycloak:23.0.7
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:359)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:330)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.lambda$startContainer$4(KeycloakDevServicesProcessor.java:376)
at java.base/java.util.Optional.orElseGet(Optional.java:364)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startContainer(KeycloakDevServicesProcessor.java:401)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startKeycloakContainer(KeycloakDevServicesProcessor.java:197)
... 11 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:344)
... 16 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:563)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:354)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 17 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*Keycloak.*started.*'
at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:909)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:500)
... 19 more
org.kie.kogito.index.postgresql.mapper.JobEntityMapperIT.testMapToModel java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[error]: Build step io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor#startKeycloakContainer threw an exception: java.lang.RuntimeException: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/keycloak/keycloak:23.0.7
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startKeycloakContainer(KeycloakDevServicesProcessor.java:249)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849)
at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at java.base/java.lang.Thread.run(Thread.java:833)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/keycloak/keycloak:23.0.7
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:359)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:330)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.lambda$startContainer$4(KeycloakDevServicesProcessor.java:376)
at java.base/java.util.Optional.orElseGet(Optional.java:364)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startContainer(KeycloakDevServicesProcessor.java:401)
at io.quarkus.oidc.deployment.devservices.keycloak.KeycloakDevServicesProcessor.startKeycloakContainer(KeycloakDevServicesProcessor.java:197)
... 11 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:344)
... 16 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:563)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:354)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 17 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*Keycloak.*started.*'
at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:909)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:500)
... 19 more

@rodrigonull
Copy link
Contributor

@tiagobento I think we need open a JIRA ticket on ASF infra board to create the MAVEN_USERNAME and MAVEN_PASSWORD secrets on this repository and then we can use the actions/setup-java to configure the deployment settings as described here: https://docs.github.com/en/actions/publishing-packages/publishing-java-packages-with-maven.
We have these secrets created on ASF Jenkins but I don't have access to the values, so if you or @porcelli have these credentials we can open the JIRA ticket.

@tiagobento
Copy link
Contributor Author

Ok, so it looks like the install plugin is correctly installing the binaries jar with the correct classifier without any additional configuration. This PR is now only missing the publication strategy on the GitHub Actions workflow.

The build command for kogito-apps, in my understanding, would be mvn clean install -DskipTests -Pjitexecutor-native, unless anyone knows that we need additional things to build this repo...

@rodrigonull rodrigonull self-requested a review May 9, 2024 21:31
@tiagobento tiagobento merged commit a4c0855 into apache:main May 9, 2024
7 of 8 checks passed
@jstastny-cz
Copy link
Contributor

Do we want to skip this completely during the project PR checks and nightly? If it wouldn't be safer to run at least the linux build also there. If that was the case, I'd suggest adding an activation property to the jitexecutor-native profile, so that it is covered by -Dfull as is the case in kogito-runtimes and drools:
https://github.com/apache/incubator-kie-kogito-pipelines/blob/main/.ci/pull-request-config.yaml#L47

The buildchain config would then need to be also adjusted so that kogito-apps runs with -Dfull in current build-command.

rgdoliveira added a commit to rgdoliveira/kogito-apps that referenced this pull request Jun 10, 2024
@rgdoliveira
NO-ISSUE: Change the publication method of JIT Executor native binari…
9dbfa19 
…es from NPM package to OS-specific Maven modules (apache#2041)

* Add jitexecutor-native module

* Revert unnecessary changes

* Adding headers

* Place binary inside JAR with classifier 'binaries'

* Remove new modules from bom

* Reverting some changes

* Prevent jitexecutor-native-linux from building on macOS

* Dry-run for publish-jitexecutor-native.yml

* Oops

---------

Co-authored-by: Luiz João Motta <luizjoaomotta@gmail.com>
# Conflicts:
#	.github/workflows/publish-jitexecutor-native.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rodrigonull rodrigonull rodrigonull approved these changes

@porcelli porcelli porcelli approved these changes

Assignees
No one assigned
Labels
Do not merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@tiagobento @kie-ci3 @rodrigonull @jstastny-cz @porcelli @ljmotta