Feature: Create baton migration for Quarkus Java 17 upgrade

[carter-cundiff]

Description

Follow on to #263, where we migrated our quarkus projects to java 17. This issue will focus on migrating downstream projects to be compatible with the new quarkus dependencies.

Questions:

• Archetype change: <groupId>${groupId}</groupId> -> <groupId>${project.groupId}</groupId> - worth migrating?
  • Wait until maven 4 upgrade
• What if version no longer matches the expected version?
  • Update regardless

DOD

• Migration to update downstream dependencies to include new version - run on every pom.xml:

  • org.scala-lang:scala-reflect:2.12.1 -> ${version.scala}
  • org.apache.commons:commons-math3:3.2 -> ${version.commons.math}
  • io.smallrye.reactive:smallrye-reactive-messaging-kafka -> ${version.smallrye.reactive.messaging}
  • junit:junit:4.13.2 -> ${version.junit}
  • io.smallrye.reactive:smallrye-reactive-messaging-in-memory -> ${version.smallrye.reactive.messaging}
  • org.jboss.resteasy:resteasy-client -> ${version.resteasy}
  • org.jboss.resteasy:resteasy-jackson2-provider -> ${version.resteasy}
  • org.awaitility:awaitility ->${version.awaitility}
  • io.vertx:vertx-core -> ${version.vertx}
  • io.smallrye.config:smallrye-config:2.10.0 -> ${version.smallrye.config}
  • org.slf4j:slf4j-api:2.0.7 -> ${version.slf4j}
  • org.slf4j:slf4j-simple:1.6.4 -> ${version.slf4j}

  Scenario: Migrate all pom dependencies to the new version
  Scenario: Migrate pom dependencies in profiles and dependency management
  

• Migration to update downstream pom to add the following dependencies - run on Spark pipeline pom.xml:

  • javax.servlet:javax.servlet-api:${version.javax.servlet} - run migration only on pipelines directory

  Scenario: Add javax servlet dependency spark pipeline pom 
  Scenario: Skip spark pipeline pom that already has the javax servlet dependency
  Scenario: Skip pom that does not have jar packaging
  

• Migration to update downstream Java classes to the new package - run on every *.java:

  • javax.<PACKAGE> -> jakarta.<PACKAGE> (ref: jakarta package list)
  • io.smallrye.reactive.messaging.providers.connectors.InMemoryConnector -> io.smallrye.reactive.messaging.memory.InMemoryConnector
  • io.smallrye.reactive.messaging.providers.connectors.InMemorySink -> io.smallrye.reactive.messaging.memory.InMemorySink

  Scenario: Migrate Java classes to use updated package
  

• Migration to update quarkus*-bom to aissemble-quarkus-bom - run on every pom.xml

  Scenario Outline: Migrate quarkus boms to aissemble quarkus bom
  Examples:
  | quarkus-bom |
  | quarkus-universe-bom |
  

• Update cucumber.test.cdi.context.java.vm to implement CdiContext and only include InMemoryConnector.class

• Update to release notes:

  • Add javax -> jakarta migration details to the release notes
  • Instruct downstream projects to replace com.boozallen.aissemble:bom-component with com.boozallen.aissemble:aissemble-quarkus-bom prior to running migrations
  • Instruct downstream projects to remove com.boozallen.aissemble:aissemble-quarkus-bom from root pom.xml and add to any quarkus apps following the migrations

Test Strategy/Script

• OTS Only:

  • Within the aiSSEMBLE repo, run the following and verify it builds successfully:

  mvn clean install -pl :foundation-upgrade -am -Dmaven.build.cache.skipCache
  

• Update your machine to build with Java 11. Run mvn -v to verify it's configured properly. The output should be similar to the following:

$ mvn -v
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /path/to/apache-maven-3.9.6-bin/3311e1d4/apache-maven-3.9.6
Java version: 11.0.20.1, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-amd64

• Create a downstream project from 1.9.4:

mvn archetype:generate -U -DarchetypeGroupId=com.boozallen.aissemble \
  -DarchetypeArtifactId=foundation-archetype \
  -DarchetypeVersion=1.9.4 \
  -DgroupId=com.test \
  -DartifactId=test-434 \
  -DprojectGitUrl=test.url \
  -DprojectName=test-434 \
  && cd test-434

• Add the attached PysparkPipeline.json and SparkPipeline.json to the test-434-pipeline-models/src/main/resources/pipelines/ directory
• Add the attached PersonDictionary.json to the test-434-pipeline-models/src/main/resources/dictionaries/ directory
• Add the attached Person.json to the test-434-pipeline-models/src/main/resources/records/ directory
• Run mvn clean install until all the manual actions are complete
• Add the following to test-434-pipelines/spark-pipeline/src/main/java/com/test/TestSyncStep.java:

+import simple.test.record.Person;
+import simple.test.record.PersonSchema;
+import java.util.List;
+import java.util.stream.Stream;
+import java.util.stream.Collectors;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;

...

    @Override
    protected void executeStepImpl() {
+        logger.info("Saving Person");
+        Person person = new Person();
+        person.setName("Test Person");
+        PersonSchema personSchema = new PersonSchema();
+        List<Row> rows = Stream.of(person).map(PersonSchema::asRow).collect(Collectors.toList());
+        Dataset<Row> dataset = sparkSession.createDataFrame(rows, personSchema.getStructType());
+        saveDataset(dataset, "Person");
+        logger.info("Completed saving Person");
    }

• Run mvn clean install -Dmaven.build.cache.skipCache once to get any remaining manual actions
• Update your machine to build with Java 17. Run mvn -v to verify it's configured properly. The output should be similar to the following:

$ mvn -v
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home:/path/to/apache-maven-3.9.6-bin/3311e1d4/apache-maven-3.9.6
Java version: 17.0.8.1, vendor: Private Build, runtime: /usr/lib/jvm/java-17-openjdk-amd64

• Update the build parent in the root pom.xml to use version 1.10.0-SNAPSHOT
• Replace all references to the bom-component with aissemble-quarkus-bom
• Run the following to perform the migrations: ./mvnw org.technologybrewery.baton:baton-maven-plugin:baton-migrate
• Remove the com.boozallen.aissemble:aissemble-quarkus-bom:${version.aissemble} dependency from the root pom.xml and test-434-tests/test-434-tests-java/pom.xml
• Delete the test-434-pipelines/pyspark-pipeline/poetry.lock and test-434-shared/test-434-data-records-python/poetry.lock files
• Add the following to the bottom of test-434-pipelines/pyspark-pipeline/pyproject.toml and test-434-shared/test-434-data-records-python/pyproject.toml so it can access our SNAPSHOT dependencies:

[[tool.poetry.source]]
name = "devpypi"
url = "https://test.pypi.org/simple/"
priority = "supplemental"

• Build the project once with mvn clean install -Dmaven.build.cache.skipCache
• tilt up
• Once all the tilt resources are ready, start the pyspark-pipeline resource and verify it runs to completion with this log output:

INFO ExampleSyncStepBase: COMPLETE: step execution completed in 0.217286ms

• Next run the spark-pipeline resource and verify it runs to completion with this log output:

INFO TestSyncStep: Completed saving Person

• In a separate terminal, make a curl request to the data access endpoint:

curl --location 'http://localhost:8081/graphql' \
--header 'Content-Type: application/json' \
--data '{"query":"{Person(table: \"Person\") {name}}"}'

• Verify it returns the following:

{"data":{"Person":[{"name":"Test Person"}]}}

References/Additional Context

Quarkus Commit 1
Quarkus Commit 2

[carter-cundiff]

DOD with @ewilkins-csi

[carter-cundiff]

OTS with @csun-cpointe

[csun-cpointe]

Test passed!!
Verified pyspark-pipeline

Verified spark-pipeline

Verified data access records