Merge pull request #188 from boozallen/171-docker-mda-fabric8

#171 updating docker templates to fabric8

DRAFT_RELEASE_NOTES.md

@@ -15,7 +15,7 @@ from `foundation/foundation-archetype/src/main/resources/archetype-resources/pom
 # Major Additions
 
 ## Python Code Linter
-We incorperated PyLint as the approach to linting Python modules to detect errors. This will enable developers to identify and resolve errors during a project build. By default, PyLint checkers classfied as errors are flagged and can be configured through Habushu's [configuration](https://github.com/TechnologyBrewery/habushu/blob/dev/README.md).
+We incorporated PyLint as the approach to linting Python modules to detect errors. This will enable developers to identify and resolve errors during a project build. By default, PyLint checkers classfied as errors are flagged and can be configured through Habushu's [configuration](https://github.com/TechnologyBrewery/habushu/blob/dev/README.md).
 
 There is a known [issue](https://github.com/behave/behave/issues/641) with using PyLint and importing modules from the Behave package that will require a modification to Python test scripts. For existing projects, you will need to change the imports by:
 
@@ -24,16 +24,20 @@ There is a known [issue](https://github.com/behave/behave/issues/641) with using
 + from behave import given, when, then  # pylint: disable=no-name-in-module
 ```
 
+## Transition from Orphedomos to Fabric8
+aiSSEMBLE has changed its Docker management plugin from the [orphedomos-maven-plugin](https://github.com/TechnologyBrewery/orphedomos) to Fabric8's [docker-maven-plugin](https://dmp.fabric8.io). This will enable developers to leverage a Docker management tool that is maintained and supported by a broader community.
+
 # Breaking Changes
 _<A short bulleted list of changes that will cause downstream projects to be partially or wholly inoperable without changes. Instructions for those changes should live in the How To Upgrade section>_
-Note instructions for adapting to these changes are outlined in the upgrade instructions below.
+Note: instructions for adapting to these changes are outlined in the upgrade instructions below.
+- Transitioning from `orphedomos` to `fabric8` may result in breaking changes, see the **Upgrade Steps for Projects with Customized Orphedomos Configurations** for further details.
 
 # Known Issues
-There are no known issues with the 1.8.0 release.
+- There is currently a bug with the Sagemaker training Docker image generated by the `aissemble-sagemaker-training-docker` `Fermenter` profile. The installation of the `logistic-training` module's `requirements.txt` fails, due to an unresolvable set of dependencies.
 
 # Known Vulnerabilities
 | Date<br/>identified | Vulnerability | Severity | Package | Affected <br/>versions | CVE | Fixed <br/>in |
-| ------------------- | ------------- | -------- | ------- | ---------------------- | --- | ------------- |
+|---------------------|---------------|----------|---------|------------------------|-----|---------------|
 
 # How to Upgrade
 The following steps will upgrade your project to 1.8. These instructions consist of multiple phases:
@@ -100,11 +104,16 @@ extension-pkg-whitelist = "pydantic"
 ignore-patterns = '.*pb2[\S]*.py'
 ```
 
+### Upgrade Steps for Projects with Customized Orphedomos Configurations
+If any extra configurations were added to the `orphedomos-maven-plugin` in addition to the generated defaults, executing the migration **will result in loss of these extra configurations**. To facilitate the upgrade, the following steps should be taken:
+1. Before executing the baton migration, it is recommended to ensure some form of version control is in place to preserve your existing `orphedomos-maven-plugin` configurations.
+1. After executing the baton migration, please see the [docker-maven-plugin docs](https://dmp.fabric8.io) to add back any extra configurations, via corresponding configuration analogs.
+
 ## Final Steps - Required for All Projects
 ### Finalizing the Upgrade
 1. Run `./mvnw org.technologybrewery.baton:baton-maven-plugin:baton-migrate` to apply the automatic migrations
 1. Run `./mvnw clean install` and resolve any manual actions that are suggested
     - **NOTE:** This will update any aiSSEMBLE dependencies in 'pyproject.toml' files automatically
-2. Repeat the previous step until all manual actions are resolved
+1. Repeat the previous step until all manual actions are resolved
 
 # What's Changed

build-parent/pom.xml

@@ -549,7 +549,7 @@
                     <groupId>io.fabric8</groupId>
                     <artifactId>docker-maven-plugin</artifactId>
                     <version>${version.fabric8.docker.maven.plugin}</version>
-                    <extensions>true</extensions> 
+                    <extensions>true</extensions>
                     <executions>
                         <execution>
                             <id>default-build</id>

docs/modules/ROOT/pages/archetype.adoc

@@ -74,12 +74,11 @@ suggest the use of a dash to separate terms.
 
 | ``dockerProjectRepositoryUrl``
 | The name of the custom docker repository to use for this project.
-| Should be specified and utilized with the https://github.com/TechnologyBrewery/orphedomos[orphedomos-maven-plugin] to
+| Should be specified and utilized with the fabric8 https://dmp.fabric8.io[docker-maven-plugin] to
 build and push docker images.
 
-Please see https://github.com/TechnologyBrewery/orphedomos?tab=readme-ov-file#repository-url[Orphedomos's documentation on
-repository URLs,role=external,window=_blank]. The https://github.com/TechnologyBrewery/orphedomos?tab=readme-ov-file#repository-id[Repository ID]
-should also be set and match a serverId entry in your settings.xml that holds credentials for the docker login.
+Please see https://dmp.fabric8.io/#registry[fabric8's documentation on
+registry URLs,role=external,window=_blank].
 
 If not specified, this will default to `docker-registry-PLACEHOLDER/repository` and will need to be updated in your root
 `pom.xml` file prior to pushing your first docker image.

docs/modules/ROOT/pages/machine-learning-pipeline-details.adoc

@@ -344,21 +344,21 @@ requires the model training image to be pushed to
 https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html[AWS ECR,role=external,window=_blank]. To enable
 this, you must first create a repository within your ECR registry named
 `<project-name>-<sagemaker-training-step-name>-docker` either through the AWS console or by using the AWS CLI. In
-addition, you must create a "server" within your Maven settings file (typically located at
+addition, you must create a "server" (which corresponds to your registry) within your Maven settings file (typically located at
 `${user.home}/.m2/settings.xml`) and provide your ECR credentials in the following format:
 
 [source]
 ----
 <server>
-  <id>REPO_ID</id>
+  <id>ECR_REGISTRY_URL</id>
   <username>AWS</username>
   <password>REPO_PASSWORD</password>
 </server>
 ----
 
 In the above snippet:
 
-* Replace `REPO_ID` with any name, but it must match the `REPO_ID` you use in the `pom.xml` file discussed below
+* Replace `ECR_REGISTRY_URL` with your ECR registry URL, which should match the `<registry></registry>` value you provide in the `pom.xml` file discussed below
 * To obtain `password`:
 ** First https://maven.apache.org/guides/mini/guide-encryption.html#how-to-create-a-master-password[create a master
 Maven password,role=external,window=_blank]
@@ -369,8 +369,8 @@ Maven password,role=external,window=_blank]
 ==== Pushing Model Training Image to ECR
 
 Finally, in your `<project-name>/<project-name>-docker/<project-name>-<sagemaker-training-step-name>-docker/pom.xml`
-file, you must provide your ECR Repo URL, Repo ID (must match the `REPO_ID` provided in your `settings.xml` file), and
-add your ECR Repo URL as an image name prefix. You must also set `skip` to `false` to enable building and pushing the
+file, you must provide your ECR Registry URL (must match the server id value provided in
+your `settings.xml` file). You must also set `skip` to `false` to enable building and pushing the
 image to ECR. There are comments within that file which will point you towards those required changes.
 
 ==== Data Preparation

foundation/foundation-archetype/src/main/resources/archetype-resources/.mvn/maven-build-cache-config.xml

@@ -22,7 +22,7 @@
         Maven build cache configuration for aiSSEMBLE
 
         This feature requires Maven 3.9+, and is enabled by default. To disable it, you must pass the
-        -Dmaven.build.cache.skipCache=trye flag to your build, or set `enabled` to `false` within this file. It is also
+        -Dmaven.build.cache.skipCache=true flag to your build, or set `enabled` to `false` within this file. It is also
         useful to look over the other build options available:
             https://maven.apache.org/extensions/maven-build-cache-extension/parameters.html
      -->

foundation/foundation-archetype/src/main/resources/archetype-resources/__rootArtifactId__-docker/pom.xml

@@ -61,13 +61,21 @@
                 </configuration>
             </plugin>
             <plugin>
-                <groupId>org.technologybrewery.orphedomos</groupId>
-                <artifactId>orphedomos-maven-plugin</artifactId>
-                <configuration>
-                    <skip>true</skip>
-                    <imageVersion>${project.version}</imageVersion>
-                    <imageName>${project.artifactId}</imageName>
-                </configuration>
+                <groupId>io.fabric8</groupId>
+                <artifactId>docker-maven-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>default-build</id>
+                        <configuration>
+                            <skip>true</skip>
+                            <images>
+                                <image>
+                                    <name>${project.artifactId}:${project.version}</name>
+                                </image>
+                            </images>
+                        </configuration>
+                    </execution>
+                </executions>
             </plugin>
         </plugins>
     </build>

foundation/foundation-archetype/src/main/resources/archetype-resources/pom.xml

@@ -65,7 +65,7 @@
     </dependencyManagement>
 
     <dependencies>
-        <!-- START: workaround to get maven build cache invalidation on new SNAPSHOTS of commonly updated plugins:  -->
+        <!-- START: workaround to get maven build cache invalidation on new SNAPSHOTS of commonly updated plugins -->
         <dependency>
             <groupId>com.boozallen.aissemble</groupId>
             <artifactId>artifacts-maven-plugin</artifactId>
@@ -84,7 +84,21 @@
             <version>${archetypeVersion}</version>
             <scope>provided</scope>
         </dependency>
-        <!-- END: workaround to get maven build cache invalidation on new SNAPSHOTS of commonly updated plugins  -->
+        <!-- END: workaround to get maven build cache invalidation on new SNAPSHOTS of commonly updated plugins -->
+        <!-- START: workaround to get maven build cache invalidation on new SNAPSHOTS of plugin dependencies -->
+        <dependency>
+            <groupId>com.boozallen.aissemble</groupId>
+            <artifactId>foundation-mda</artifactId>
+            <version>${archetypeVersion}</version>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.boozallen.aissemble</groupId>
+            <artifactId>foundation-upgrade</artifactId>
+            <version>${archetypeVersion}</version>
+            <scope>provided</scope>
+        </dependency>
+        <!-- END: workaround to get maven build cache invalidation on new SNAPSHOTS of plugin dependencies -->
     </dependencies>
 
     <scm>

foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/ManualActionNotificationService.java

@@ -655,15 +655,13 @@ public void addSagemakerDockerPomMessage(final GenerationContext context, final
             logger.warn("Unable to find Docker module. Will not be able to direct manual updates for the deploy module's POM.xml");
         } else {
             String pomFilePath = pomPath + File.separator + trainingDockerArtifactId;
-            boolean repoUrlExists = existsInFile(pomFilePath,"<repoUrl>" + "ECR_REPO_URL" + "</repoUrl>");
-            boolean imageNameExists = existsInFile(pomFilePath, "<imageName>" + "ECR_REPO_URL" + "/${dockerImageName}</imageName>");
-            if (repoUrlExists || imageNameExists) {
+            boolean registryUrlExists = existsInFile(pomFilePath,"<registry>ECR_REGISTRY_URL</registry>");
+            if (registryUrlExists) {
                 final String key = getMessageKey(pomFilePath, "pom");
                 VelocityNotification notification = new VelocityNotification(key, new HashSet<>(), "templates/notifications/notification.sagemaker.docker.pom.vm");
                 notification.addToVelocityContext("artifactId", artifactId);
                 notification.addToVelocityContext("dockerArtifactId", trainingDockerArtifactId);
-                notification.addToVelocityContext("repoUrlExists", repoUrlExists);
-                notification.addToVelocityContext("imageNameExists", imageNameExists);
+                notification.addToVelocityContext("registryUrlExists", registryUrlExists);
                 addManualAction(pomFilePath, notification);
             }
         }

foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/generator/MachineLearningPipelineGenerator.java

@@ -96,12 +96,17 @@ private boolean enableAutoTrain(MachineLearningPipeline pipeline) {
     private boolean pipelineContainsOnnxPostAction(MachineLearningPipeline pipeline) {
         boolean containsOnnx = false;
 
-        for (PostAction postAction : pipeline.getTrainingStep().getPostActions()) {
-            if (PipelineUtils.forOnnxModelConversion(postAction)) {
-                containsOnnx = true;
-                break;
+        // this surrounding if-check was added to prevent piplines that use sagemaker training from
+        // throwing a null pointer exception when detecting if Onnx conversion code is needed to be generated
+        if (pipeline.getTrainingStep() != null) {
+            for (PostAction postAction : pipeline.getTrainingStep().getPostActions()) {
+                if (PipelineUtils.forOnnxModelConversion(postAction)) {
+                    containsOnnx = true;
+                    break;
+                }
             }
         }
+
         return containsOnnx;
     }
 

foundation/foundation-mda/src/main/resources/templates/general-docker/airflow.docker.pom.xml.vm

@@ -9,7 +9,7 @@
         <version>${version}</version>
     </parent>
 
-    <packaging>orphedomos</packaging>
+    <packaging>docker-build</packaging>
 
     <artifactId>${artifactId}</artifactId>
 

foundation/foundation-mda/src/main/resources/templates/general-docker/custom.lineage.consumer.docker.pom.xml.vm

@@ -8,7 +8,7 @@
         <version>${version}</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
-    <packaging>orphedomos</packaging>
+    <packaging>docker-build</packaging>
 
     <name>${parentDescriptiveName}::Custom Data Lineage Consumer Docker</name>
     <artifactId>${artifactId}</artifactId>

foundation/foundation-mda/src/main/resources/templates/general-docker/data.access.docker.pom.xml.vm

@@ -11,7 +11,7 @@
     </parent>
 
     <artifactId>${moduleArtifactId}</artifactId>
-    <packaging>orphedomos</packaging>
+    <packaging>docker-build</packaging>
 
     <name>${parentDescriptiveName}::Data Access</name>
     <description>Build for a Data Access Docker container</description>

foundation/foundation-mda/src/main/resources/templates/general-docker/inference.docker.pom.xml.vm

@@ -12,7 +12,7 @@
 
 	<artifactId>${artifactId}</artifactId>
 
-	<packaging>orphedomos</packaging>
+	<packaging>docker-build</packaging>
 
 	<name>${parentDescriptiveName}::Inference</name>
 	<description>Build for an Inference Docker container to run a compliant inference service</description>

foundation/foundation-mda/src/main/resources/templates/general-docker/mlflow.docker.pom.xml.vm

@@ -12,7 +12,7 @@
 
     <artifactId>${artifactId}</artifactId>
 
-    <packaging>orphedomos</packaging>
+    <packaging>docker-build</packaging>
 
     <name>${parentDescriptiveName}::MLflow</name>
     <description>${parentDescriptiveName}::Build for a MLflow container</description>

foundation/foundation-mda/src/main/resources/templates/general-docker/policy.decision.point.docker.pom.xml.vm

@@ -12,7 +12,7 @@
 
     <artifactId>${artifactId}</artifactId>
 
-    <packaging>orphedomos</packaging>
+    <packaging>docker-build</packaging>
 
     <name>${parentDescriptiveName}::Policy Decision Point</name>
     <description>${parentDescriptiveName}::Build for a Policy Decision Point service container</description>

foundation/foundation-mda/src/main/resources/templates/general-docker/sagemaker-training.docker.pom.xml.vm

@@ -12,11 +12,17 @@
 
     <artifactId>${artifactId}</artifactId>
 
-    <packaging>orphedomos</packaging>
+    <packaging>docker-build</packaging>
 
     <name>${parentDescriptiveName}::Training</name>
     <description>Build for a SageMaker Training Docker container to run a SageMaker model training job</description>
 
+    <properties>
+        <!--various target archiectures can be specified
+        via this property as a comma-separated list-->
+        <sagemaker.docker.platforms>linux/amd64</sagemaker.docker.platforms>
+    </properties>
+
     <build>
         <plugins>
             <plugin>
@@ -58,36 +64,39 @@
                 </dependencies>
             </plugin>
             <plugin>
-                <groupId>org.technologybrewery.orphedomos</groupId>
-                <artifactId>orphedomos-maven-plugin</artifactId>
-                <version>${version.orphedomos.plugin}</version>
-                <extensions>true</extensions>
+                <groupId>io.fabric8</groupId>
+                <artifactId>docker-maven-plugin</artifactId>
                 <executions>
                     <execution>
                         <id>multiplatform-build-deploy</id>
                         <phase>prepare-package</phase>
-                        <goals>
-                            <goal>multiplatform-build-deploy</goal>
-                        </goals>
                         <configuration>
-                            <!--Replace with your ECR Repo URL here "aws_account_id.dkr.ecr.aws_region.amazonaws.com"-->
-                            <repoUrl>ECR_REPO_URL</repoUrl>
-                            <!--Replace with your Repo ID here here, this must map to the "id" tag within the associated 
-                             "server" in your Maven settings.xml file-->
-                            <repoId>REPO_ID</repoId>
-                            <imageVersion>latest</imageVersion>
-                            <!--Replace with your ECR Repo URL here "aws_account_id.dkr.ecr.aws_region.amazonaws.com"-->
-                            <imageName>ECR_REPO_URL/${dockerImageName}</imageName>
-                            <buildArgs>
-                                <DOCKER_BASELINE_REPO_ID>${docker.baseline.repo.id}</DOCKER_BASELINE_REPO_ID>
-                                <VERSION_AISSEMBLE>${version.aissemble}</VERSION_AISSEMBLE>
-                            </buildArgs>
-                            <!--Replace with "true" if you you want to skip building and pushing SageMaker model
-                            training image to ECR-->
+                            <images>
+                                <image>
+                                    <name>${project.artifactId}:latest</name>
+                                    <!--Replace with your ECR Registry URL here,
+                                        example: "aws_account_id.dkr.ecr.aws_region.amazonaws.com".
+                                        The registry value must map to the "id" value
+                                        within the associated "server" in your Maven settings.xml file-->
+                                    <registry>ECR_REGISTRY_URL</registry>
+                                    <build>
+                                        <buildx>
+                                            <platforms>
+                                                <platform>${sagemaker.docker.platforms}</platform>
+                                            </platforms>
+                                        </buildx>
+                                        <args>
+                                            <DOCKER_BASELINE_REPO_ID>${docker.baseline.repo.id}/</DOCKER_BASELINE_REPO_ID>
+                                            <VERSION_AISSEMBLE>${version.aissemble}</VERSION_AISSEMBLE>
+                                        </args>
+                                        <contextDir>${project.basedir}</contextDir>
+                                        <dockerFile>./src/main/resources/docker/Dockerfile</dockerFile>
+                                    </build>
+                                </image>
+                            </images>
+                            <!--Replace with "true" if you want to skip building and pushing
+                                the SageMaker model training image to ECR.-->
                             <skip>false</skip>
-                            <targetArchitectures>
-                                <targetArchitecture>linux/amd64</targetArchitecture>
-                            </targetArchitectures>
                         </configuration>
                     </execution>
                 </executions>