diff --git a/DRAFT_RELEASE_NOTES.md b/DRAFT_RELEASE_NOTES.md index ac64f7717..fecfc6ca0 100644 --- a/DRAFT_RELEASE_NOTES.md +++ b/DRAFT_RELEASE_NOTES.md @@ -15,7 +15,7 @@ from `foundation/foundation-archetype/src/main/resources/archetype-resources/pom # Major Additions ## Python Code Linter -We incorperated PyLint as the approach to linting Python modules to detect errors. This will enable developers to identify and resolve errors during a project build. By default, PyLint checkers classfied as errors are flagged and can be configured through Habushu's [configuration](https://github.com/TechnologyBrewery/habushu/blob/dev/README.md). +We incorporated PyLint as the approach to linting Python modules to detect errors. This will enable developers to identify and resolve errors during a project build. By default, PyLint checkers classfied as errors are flagged and can be configured through Habushu's [configuration](https://github.com/TechnologyBrewery/habushu/blob/dev/README.md). There is a known [issue](https://github.com/behave/behave/issues/641) with using PyLint and importing modules from the Behave package that will require a modification to Python test scripts. For existing projects, you will need to change the imports by: @@ -24,16 +24,20 @@ There is a known [issue](https://github.com/behave/behave/issues/641) with using + from behave import given, when, then # pylint: disable=no-name-in-module ``` +## Transition from Orphedomos to Fabric8 +aiSSEMBLE has changed its Docker management plugin from the [orphedomos-maven-plugin](https://github.com/TechnologyBrewery/orphedomos) to Fabric8's [docker-maven-plugin](https://dmp.fabric8.io). This will enable developers to leverage a Docker management tool that is maintained and supported by a broader community. + # Breaking Changes __ -Note instructions for adapting to these changes are outlined in the upgrade instructions below. +Note: instructions for adapting to these changes are outlined in the upgrade instructions below. +- Transitioning from `orphedomos` to `fabric8` may result in breaking changes, see the **Upgrade Steps for Projects with Customized Orphedomos Configurations** for further details. # Known Issues -There are no known issues with the 1.8.0 release. +- There is currently a bug with the Sagemaker training Docker image generated by the `aissemble-sagemaker-training-docker` `Fermenter` profile. The installation of the `logistic-training` module's `requirements.txt` fails, due to an unresolvable set of dependencies. # Known Vulnerabilities | Date
identified | Vulnerability | Severity | Package | Affected
versions | CVE | Fixed
in | -| ------------------- | ------------- | -------- | ------- | ---------------------- | --- | ------------- | +|---------------------|---------------|----------|---------|------------------------|-----|---------------| # How to Upgrade The following steps will upgrade your project to 1.8. These instructions consist of multiple phases: @@ -100,11 +104,16 @@ extension-pkg-whitelist = "pydantic" ignore-patterns = '.*pb2[\S]*.py' ``` +### Upgrade Steps for Projects with Customized Orphedomos Configurations +If any extra configurations were added to the `orphedomos-maven-plugin` in addition to the generated defaults, executing the migration **will result in loss of these extra configurations**. To facilitate the upgrade, the following steps should be taken: +1. Before executing the baton migration, it is recommended to ensure some form of version control is in place to preserve your existing `orphedomos-maven-plugin` configurations. +1. After executing the baton migration, please see the [docker-maven-plugin docs](https://dmp.fabric8.io) to add back any extra configurations, via corresponding configuration analogs. + ## Final Steps - Required for All Projects ### Finalizing the Upgrade 1. Run `./mvnw org.technologybrewery.baton:baton-maven-plugin:baton-migrate` to apply the automatic migrations 1. Run `./mvnw clean install` and resolve any manual actions that are suggested - **NOTE:** This will update any aiSSEMBLE dependencies in 'pyproject.toml' files automatically -2. Repeat the previous step until all manual actions are resolved +1. Repeat the previous step until all manual actions are resolved # What's Changed diff --git a/build-parent/pom.xml b/build-parent/pom.xml index 519edbe99..808874e31 100644 --- a/build-parent/pom.xml +++ b/build-parent/pom.xml @@ -549,7 +549,7 @@ io.fabric8 docker-maven-plugin ${version.fabric8.docker.maven.plugin} - true + true default-build diff --git a/docs/modules/ROOT/pages/archetype.adoc b/docs/modules/ROOT/pages/archetype.adoc index 1f525ba48..f4e324aeb 100644 --- a/docs/modules/ROOT/pages/archetype.adoc +++ b/docs/modules/ROOT/pages/archetype.adoc @@ -74,12 +74,11 @@ suggest the use of a dash to separate terms. | ``dockerProjectRepositoryUrl`` | The name of the custom docker repository to use for this project. -| Should be specified and utilized with the https://github.com/TechnologyBrewery/orphedomos[orphedomos-maven-plugin] to +| Should be specified and utilized with the fabric8 https://dmp.fabric8.io[docker-maven-plugin] to build and push docker images. -Please see https://github.com/TechnologyBrewery/orphedomos?tab=readme-ov-file#repository-url[Orphedomos's documentation on -repository URLs,role=external,window=_blank]. The https://github.com/TechnologyBrewery/orphedomos?tab=readme-ov-file#repository-id[Repository ID] -should also be set and match a serverId entry in your settings.xml that holds credentials for the docker login. +Please see https://dmp.fabric8.io/#registry[fabric8's documentation on +registry URLs,role=external,window=_blank]. If not specified, this will default to `docker-registry-PLACEHOLDER/repository` and will need to be updated in your root `pom.xml` file prior to pushing your first docker image. diff --git a/docs/modules/ROOT/pages/machine-learning-pipeline-details.adoc b/docs/modules/ROOT/pages/machine-learning-pipeline-details.adoc index 7104f2d3f..4902af63c 100644 --- a/docs/modules/ROOT/pages/machine-learning-pipeline-details.adoc +++ b/docs/modules/ROOT/pages/machine-learning-pipeline-details.adoc @@ -344,13 +344,13 @@ requires the model training image to be pushed to https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html[AWS ECR,role=external,window=_blank]. To enable this, you must first create a repository within your ECR registry named `--docker` either through the AWS console or by using the AWS CLI. In -addition, you must create a "server" within your Maven settings file (typically located at +addition, you must create a "server" (which corresponds to your registry) within your Maven settings file (typically located at `${user.home}/.m2/settings.xml`) and provide your ECR credentials in the following format: [source] ---- - REPO_ID + ECR_REGISTRY_URL AWS REPO_PASSWORD @@ -358,7 +358,7 @@ addition, you must create a "server" within your Maven settings file (typically In the above snippet: -* Replace `REPO_ID` with any name, but it must match the `REPO_ID` you use in the `pom.xml` file discussed below +* Replace `ECR_REGISTRY_URL` with your ECR registry URL, which should match the `` value you provide in the `pom.xml` file discussed below * To obtain `password`: ** First https://maven.apache.org/guides/mini/guide-encryption.html#how-to-create-a-master-password[create a master Maven password,role=external,window=_blank] @@ -369,8 +369,8 @@ Maven password,role=external,window=_blank] ==== Pushing Model Training Image to ECR Finally, in your `/-docker/--docker/pom.xml` -file, you must provide your ECR Repo URL, Repo ID (must match the `REPO_ID` provided in your `settings.xml` file), and -add your ECR Repo URL as an image name prefix. You must also set `skip` to `false` to enable building and pushing the +file, you must provide your ECR Registry URL (must match the server id value provided in +your `settings.xml` file). You must also set `skip` to `false` to enable building and pushing the image to ECR. There are comments within that file which will point you towards those required changes. ==== Data Preparation diff --git a/foundation/foundation-archetype/src/main/resources/archetype-resources/.mvn/maven-build-cache-config.xml b/foundation/foundation-archetype/src/main/resources/archetype-resources/.mvn/maven-build-cache-config.xml index 158ba2b3c..616bbe7c8 100644 --- a/foundation/foundation-archetype/src/main/resources/archetype-resources/.mvn/maven-build-cache-config.xml +++ b/foundation/foundation-archetype/src/main/resources/archetype-resources/.mvn/maven-build-cache-config.xml @@ -22,7 +22,7 @@ Maven build cache configuration for aiSSEMBLE This feature requires Maven 3.9+, and is enabled by default. To disable it, you must pass the - -Dmaven.build.cache.skipCache=trye flag to your build, or set `enabled` to `false` within this file. It is also + -Dmaven.build.cache.skipCache=true flag to your build, or set `enabled` to `false` within this file. It is also useful to look over the other build options available: https://maven.apache.org/extensions/maven-build-cache-extension/parameters.html --> diff --git a/foundation/foundation-archetype/src/main/resources/archetype-resources/__rootArtifactId__-docker/pom.xml b/foundation/foundation-archetype/src/main/resources/archetype-resources/__rootArtifactId__-docker/pom.xml index ee1095cb1..5c93c3e6b 100644 --- a/foundation/foundation-archetype/src/main/resources/archetype-resources/__rootArtifactId__-docker/pom.xml +++ b/foundation/foundation-archetype/src/main/resources/archetype-resources/__rootArtifactId__-docker/pom.xml @@ -61,13 +61,21 @@ - org.technologybrewery.orphedomos - orphedomos-maven-plugin - - true - ${project.version} - ${project.artifactId} - + io.fabric8 + docker-maven-plugin + + + default-build + + true + + + ${project.artifactId}:${project.version} + + + + + diff --git a/foundation/foundation-archetype/src/main/resources/archetype-resources/pom.xml b/foundation/foundation-archetype/src/main/resources/archetype-resources/pom.xml index 14b0a5e79..04f128df2 100644 --- a/foundation/foundation-archetype/src/main/resources/archetype-resources/pom.xml +++ b/foundation/foundation-archetype/src/main/resources/archetype-resources/pom.xml @@ -65,7 +65,7 @@ - + com.boozallen.aissemble artifacts-maven-plugin @@ -84,7 +84,21 @@ ${archetypeVersion} provided - + + + + com.boozallen.aissemble + foundation-mda + ${archetypeVersion} + provided + + + com.boozallen.aissemble + foundation-upgrade + ${archetypeVersion} + provided + + diff --git a/foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/ManualActionNotificationService.java b/foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/ManualActionNotificationService.java index 404a45f47..024efa631 100644 --- a/foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/ManualActionNotificationService.java +++ b/foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/ManualActionNotificationService.java @@ -655,15 +655,13 @@ public void addSagemakerDockerPomMessage(final GenerationContext context, final logger.warn("Unable to find Docker module. Will not be able to direct manual updates for the deploy module's POM.xml"); } else { String pomFilePath = pomPath + File.separator + trainingDockerArtifactId; - boolean repoUrlExists = existsInFile(pomFilePath,"" + "ECR_REPO_URL" + ""); - boolean imageNameExists = existsInFile(pomFilePath, "" + "ECR_REPO_URL" + "/${dockerImageName}"); - if (repoUrlExists || imageNameExists) { + boolean registryUrlExists = existsInFile(pomFilePath,"ECR_REGISTRY_URL"); + if (registryUrlExists) { final String key = getMessageKey(pomFilePath, "pom"); VelocityNotification notification = new VelocityNotification(key, new HashSet<>(), "templates/notifications/notification.sagemaker.docker.pom.vm"); notification.addToVelocityContext("artifactId", artifactId); notification.addToVelocityContext("dockerArtifactId", trainingDockerArtifactId); - notification.addToVelocityContext("repoUrlExists", repoUrlExists); - notification.addToVelocityContext("imageNameExists", imageNameExists); + notification.addToVelocityContext("registryUrlExists", registryUrlExists); addManualAction(pomFilePath, notification); } } diff --git a/foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/generator/MachineLearningPipelineGenerator.java b/foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/generator/MachineLearningPipelineGenerator.java index ce2262258..3b71e793b 100644 --- a/foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/generator/MachineLearningPipelineGenerator.java +++ b/foundation/foundation-mda/src/main/java/com/boozallen/aiops/mda/generator/MachineLearningPipelineGenerator.java @@ -96,12 +96,17 @@ private boolean enableAutoTrain(MachineLearningPipeline pipeline) { private boolean pipelineContainsOnnxPostAction(MachineLearningPipeline pipeline) { boolean containsOnnx = false; - for (PostAction postAction : pipeline.getTrainingStep().getPostActions()) { - if (PipelineUtils.forOnnxModelConversion(postAction)) { - containsOnnx = true; - break; + // this surrounding if-check was added to prevent piplines that use sagemaker training from + // throwing a null pointer exception when detecting if Onnx conversion code is needed to be generated + if (pipeline.getTrainingStep() != null) { + for (PostAction postAction : pipeline.getTrainingStep().getPostActions()) { + if (PipelineUtils.forOnnxModelConversion(postAction)) { + containsOnnx = true; + break; + } } } + return containsOnnx; } diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/airflow.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/airflow.docker.pom.xml.vm index 47af43b18..f9c6ee10f 100644 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/airflow.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/airflow.docker.pom.xml.vm @@ -9,7 +9,7 @@ ${version} - orphedomos + docker-build ${artifactId} diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/custom.lineage.consumer.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/custom.lineage.consumer.docker.pom.xml.vm index dd59c6a61..700ac59ee 100644 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/custom.lineage.consumer.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/custom.lineage.consumer.docker.pom.xml.vm @@ -8,7 +8,7 @@ ${version} 4.0.0 - orphedomos + docker-build ${parentDescriptiveName}::Custom Data Lineage Consumer Docker ${artifactId} diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/data.access.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/data.access.docker.pom.xml.vm index 7ea9086b1..909052218 100644 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/data.access.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/data.access.docker.pom.xml.vm @@ -11,7 +11,7 @@ ${moduleArtifactId} - orphedomos + docker-build ${parentDescriptiveName}::Data Access Build for a Data Access Docker container diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/inference.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/inference.docker.pom.xml.vm index ae6c7703a..a86e262b2 100644 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/inference.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/inference.docker.pom.xml.vm @@ -12,7 +12,7 @@ ${artifactId} - orphedomos + docker-build ${parentDescriptiveName}::Inference Build for an Inference Docker container to run a compliant inference service diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/mlflow.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/mlflow.docker.pom.xml.vm index 291659649..a0acd884c 100644 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/mlflow.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/mlflow.docker.pom.xml.vm @@ -12,7 +12,7 @@ ${artifactId} - orphedomos + docker-build ${parentDescriptiveName}::MLflow ${parentDescriptiveName}::Build for a MLflow container diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/policy.decision.point.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/policy.decision.point.docker.pom.xml.vm index 974d7fd82..9e446104c 100644 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/policy.decision.point.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/policy.decision.point.docker.pom.xml.vm @@ -12,7 +12,7 @@ ${artifactId} - orphedomos + docker-build ${parentDescriptiveName}::Policy Decision Point ${parentDescriptiveName}::Build for a Policy Decision Point service container diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/sagemaker-training.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/sagemaker-training.docker.pom.xml.vm index 6fb2a3cc1..17a1cedf4 100755 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/sagemaker-training.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/sagemaker-training.docker.pom.xml.vm @@ -12,11 +12,17 @@ ${artifactId} - orphedomos + docker-build ${parentDescriptiveName}::Training Build for a SageMaker Training Docker container to run a SageMaker model training job + + + linux/amd64 + + @@ -58,36 +64,39 @@ - org.technologybrewery.orphedomos - orphedomos-maven-plugin - ${version.orphedomos.plugin} - true + io.fabric8 + docker-maven-plugin multiplatform-build-deploy prepare-package - - multiplatform-build-deploy - - - ECR_REPO_URL - - REPO_ID - latest - - ECR_REPO_URL/${dockerImageName} - - ${docker.baseline.repo.id} - ${version.aissemble} - - + + + ${project.artifactId}:latest + + ECR_REGISTRY_URL + + + + ${sagemaker.docker.platforms} + + + + ${docker.baseline.repo.id}/ + ${version.aissemble} + + ${project.basedir} + ./src/main/resources/docker/Dockerfile + + + + false - - linux/amd64 - diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/spark-worker.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/spark-worker.docker.pom.xml.vm index 1e5c005af..deb8baca8 100644 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/spark-worker.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/spark-worker.docker.pom.xml.vm @@ -11,7 +11,7 @@ ${artifactId} - orphedomos + docker-build ${parentDescriptiveName}::Spark Build for a Spark Docker container which contains all pipelines diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/training.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/training.docker.pom.xml.vm index d89f4e65b..73064d0a8 100755 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/training.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/training.docker.pom.xml.vm @@ -12,7 +12,7 @@ ${artifactId} - orphedomos + docker-build ${parentDescriptiveName}::Training Build for a Training Docker container to run a model training job diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/vault.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/vault.docker.pom.xml.vm index 7c21e09f8..a3970b011 100644 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/vault.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/vault.docker.pom.xml.vm @@ -12,7 +12,7 @@ ${artifactId} - orphedomos + docker-build ${parentDescriptiveName}::Vault Build for a Vault encryption service diff --git a/foundation/foundation-mda/src/main/resources/templates/general-docker/versioning.docker.pom.xml.vm b/foundation/foundation-mda/src/main/resources/templates/general-docker/versioning.docker.pom.xml.vm index ef84aa83b..46af93341 100644 --- a/foundation/foundation-mda/src/main/resources/templates/general-docker/versioning.docker.pom.xml.vm +++ b/foundation/foundation-mda/src/main/resources/templates/general-docker/versioning.docker.pom.xml.vm @@ -14,7 +14,7 @@ ${parentDescriptiveName}::Versioning Build for a Versioning Docker container to run the aiSSEMBLE Versioning service - orphedomos + docker-build diff --git a/foundation/foundation-mda/src/main/resources/templates/notifications/notification.sagemaker.docker.pom.vm b/foundation/foundation-mda/src/main/resources/templates/notifications/notification.sagemaker.docker.pom.vm index e68baa5c6..bcd7f833c 100644 --- a/foundation/foundation-mda/src/main/resources/templates/notifications/notification.sagemaker.docker.pom.vm +++ b/foundation/foundation-mda/src/main/resources/templates/notifications/notification.sagemaker.docker.pom.vm @@ -1,20 +1,11 @@ -You must configure the default values in the Orphedomos plugin configuration within: -${artifactId}/${dockerArtifactId} - Please refer to the documentation at: https://github.com/boozallen/aissemble/blob/dev/docs/modules/ROOT/pages/machine-learning-pipeline-details.adoc#sagemaker-training -#if($repoUrlExists) - -ECR_REPO_URL - -#end -#if($imageNameExists) - -ECR_REPO_URL/${dockerImageName} - +#if($registryUrlExists) + +ECR_REGISTRY_URL #end The build will fail until you do this. To stop the build from failing temporarily, you can set skip to true, -but building and pushing SageMaker model training image to ECR will not work until the fields are filled in appropriately \ No newline at end of file +but building and pushing SageMaker model training image to ECR will not work until the field is filled in appropriately. \ No newline at end of file diff --git a/foundation/foundation-upgrade/src/main/java/com/boozallen/aissemble/upgrade/migration/v1_8_0/OrphedomosToFabric8Migration.java b/foundation/foundation-upgrade/src/main/java/com/boozallen/aissemble/upgrade/migration/v1_8_0/OrphedomosToFabric8Migration.java index 23732ee2e..5fa53faaf 100644 --- a/foundation/foundation-upgrade/src/main/java/com/boozallen/aissemble/upgrade/migration/v1_8_0/OrphedomosToFabric8Migration.java +++ b/foundation/foundation-upgrade/src/main/java/com/boozallen/aissemble/upgrade/migration/v1_8_0/OrphedomosToFabric8Migration.java @@ -242,6 +242,7 @@ private static String getConfigCode(String indent){ StringUtils.repeat(indent, 5) + "\n" + StringUtils.repeat(indent, 6) + "default-build\n" + StringUtils.repeat(indent, 6) + "\n" + + StringUtils.repeat(indent, 7) + "true\n" + StringUtils.repeat(indent, 7) + "\n" + StringUtils.repeat(indent, 8) + "\n" + StringUtils.repeat(indent, 9) + "" + imageName + ":" + imageVersion + "\n" +