Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#171 updating docker templates to fabric8 #188

Merged
merged 1 commit into from
Jul 5, 2024
Merged

#171 updating docker templates to fabric8 #188

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions DRAFT_RELEASE_NOTES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ from `foundation/foundation-archetype/src/main/resources/archetype-resources/pom
# Major Additions

## Python Code Linter
We incorperated PyLint as the approach to linting Python modules to detect errors. This will enable developers to identify and resolve errors during a project build. By default, PyLint checkers classfied as errors are flagged and can be configured through Habushu's [configuration](https://github.com/TechnologyBrewery/habushu/blob/dev/README.md).
We incorporated PyLint as the approach to linting Python modules to detect errors. This will enable developers to identify and resolve errors during a project build. By default, PyLint checkers classfied as errors are flagged and can be configured through Habushu's [configuration](https://github.com/TechnologyBrewery/habushu/blob/dev/README.md).

There is a known [issue](https://github.com/behave/behave/issues/641) with using PyLint and importing modules from the Behave package that will require a modification to Python test scripts. For existing projects, you will need to change the imports by:

Expand All @@ -24,16 +24,20 @@ There is a known [issue](https://github.com/behave/behave/issues/641) with using
+ from behave import given, when, then # pylint: disable=no-name-in-module
```

## Transition from Orphedomos to Fabric8
aiSSEMBLE has changed its Docker management plugin from the [orphedomos-maven-plugin](https://github.com/TechnologyBrewery/orphedomos) to Fabric8's [docker-maven-plugin](https://dmp.fabric8.io). This will enable developers to leverage a Docker management tool that is maintained and supported by a broader community.

# Breaking Changes
_<A short bulleted list of changes that will cause downstream projects to be partially or wholly inoperable without changes. Instructions for those changes should live in the How To Upgrade section>_
Note instructions for adapting to these changes are outlined in the upgrade instructions below.
Note: instructions for adapting to these changes are outlined in the upgrade instructions below.
- Transitioning from `orphedomos` to `fabric8` may result in breaking changes, see the **Upgrade Steps for Projects with Customized Orphedomos Configurations** for further details.

# Known Issues
There are no known issues with the 1.8.0 release.
- There is currently a bug with the Sagemaker training Docker image generated by the `aissemble-sagemaker-training-docker` `Fermenter` profile. The installation of the `logistic-training` module's `requirements.txt` fails, due to an unresolvable set of dependencies.

# Known Vulnerabilities
| Date<br/>identified | Vulnerability | Severity | Package | Affected <br/>versions | CVE | Fixed <br/>in |
| ------------------- | ------------- | -------- | ------- | ---------------------- | --- | ------------- |
|---------------------|---------------|----------|---------|------------------------|-----|---------------|

# How to Upgrade
The following steps will upgrade your project to 1.8. These instructions consist of multiple phases:
Expand Down Expand Up @@ -100,11 +104,16 @@ extension-pkg-whitelist = "pydantic"
ignore-patterns = '.*pb2[\S]*.py'
```

### Upgrade Steps for Projects with Customized Orphedomos Configurations
If any extra configurations were added to the `orphedomos-maven-plugin` in addition to the generated defaults, executing the migration **will result in loss of these extra configurations**. To facilitate the upgrade, the following steps should be taken:
1. Before executing the baton migration, it is recommended to ensure some form of version control is in place to preserve your existing `orphedomos-maven-plugin` configurations.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like these bullets should be 1., 2., 3., instead of all 1's

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is intentional. That way if we insert another point, we don't have to manually renumber. Markdown is smart enough to increment the numbers when the README is rendered

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oo good to know!

1. After executing the baton migration, please see the [docker-maven-plugin docs](https://dmp.fabric8.io) to add back any extra configurations, via corresponding configuration analogs.

## Final Steps - Required for All Projects
### Finalizing the Upgrade
1. Run `./mvnw org.technologybrewery.baton:baton-maven-plugin:baton-migrate` to apply the automatic migrations
1. Run `./mvnw clean install` and resolve any manual actions that are suggested
- **NOTE:** This will update any aiSSEMBLE dependencies in 'pyproject.toml' files automatically
2. Repeat the previous step until all manual actions are resolved
1. Repeat the previous step until all manual actions are resolved

# What's Changed
2 changes: 1 addition & 1 deletion build-parent/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -549,7 +549,7 @@
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>${version.fabric8.docker.maven.plugin}</version>
<extensions>true</extensions>
<extensions>true</extensions>
<executions>
<execution>
<id>default-build</id>
Expand Down
7 changes: 3 additions & 4 deletions docs/modules/ROOT/pages/archetype.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,12 +74,11 @@ suggest the use of a dash to separate terms.

| ``dockerProjectRepositoryUrl``
| The name of the custom docker repository to use for this project.
| Should be specified and utilized with the https://github.com/TechnologyBrewery/orphedomos[orphedomos-maven-plugin] to
| Should be specified and utilized with the fabric8 https://dmp.fabric8.io[docker-maven-plugin] to
build and push docker images.

Please see https://github.com/TechnologyBrewery/orphedomos?tab=readme-ov-file#repository-url[Orphedomos's documentation on
repository URLs,role=external,window=_blank]. The https://github.com/TechnologyBrewery/orphedomos?tab=readme-ov-file#repository-id[Repository ID]
should also be set and match a serverId entry in your settings.xml that holds credentials for the docker login.
Please see https://dmp.fabric8.io/#registry[fabric8's documentation on
registry URLs,role=external,window=_blank].

If not specified, this will default to `docker-registry-PLACEHOLDER/repository` and will need to be updated in your root
`pom.xml` file prior to pushing your first docker image.
Expand Down
10 changes: 5 additions & 5 deletions docs/modules/ROOT/pages/machine-learning-pipeline-details.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -344,21 +344,21 @@ requires the model training image to be pushed to
https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html[AWS ECR,role=external,window=_blank]. To enable
this, you must first create a repository within your ECR registry named
`<project-name>-<sagemaker-training-step-name>-docker` either through the AWS console or by using the AWS CLI. In
addition, you must create a "server" within your Maven settings file (typically located at
addition, you must create a "server" (which corresponds to your registry) within your Maven settings file (typically located at
`${user.home}/.m2/settings.xml`) and provide your ECR credentials in the following format:

[source]
----
<server>
<id>REPO_ID</id>
<id>ECR_REGISTRY_URL</id>
<username>AWS</username>
<password>REPO_PASSWORD</password>
</server>
----

In the above snippet:

* Replace `REPO_ID` with any name, but it must match the `REPO_ID` you use in the `pom.xml` file discussed below
* Replace `ECR_REGISTRY_URL` with your ECR registry URL, which should match the `<registry></registry>` value you provide in the `pom.xml` file discussed below
* To obtain `password`:
** First https://maven.apache.org/guides/mini/guide-encryption.html#how-to-create-a-master-password[create a master
Maven password,role=external,window=_blank]
Expand All @@ -369,8 +369,8 @@ Maven password,role=external,window=_blank]
==== Pushing Model Training Image to ECR

Finally, in your `<project-name>/<project-name>-docker/<project-name>-<sagemaker-training-step-name>-docker/pom.xml`
file, you must provide your ECR Repo URL, Repo ID (must match the `REPO_ID` provided in your `settings.xml` file), and
add your ECR Repo URL as an image name prefix. You must also set `skip` to `false` to enable building and pushing the
file, you must provide your ECR Registry URL (must match the server id value provided in
your `settings.xml` file). You must also set `skip` to `false` to enable building and pushing the
image to ECR. There are comments within that file which will point you towards those required changes.

==== Data Preparation
Expand Down
2 changes: 1 addition & 1 deletion ...dation-archetype/src/main/resources/archetype-resources/.mvn/maven-build-cache-config.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
Maven build cache configuration for aiSSEMBLE

This feature requires Maven 3.9+, and is enabled by default. To disable it, you must pass the
-Dmaven.build.cache.skipCache=trye flag to your build, or set `enabled` to `false` within this file. It is also
-Dmaven.build.cache.skipCache=true flag to your build, or set `enabled` to `false` within this file. It is also
useful to look over the other build options available:
https://maven.apache.org/extensions/maven-build-cache-extension/parameters.html
-->
Expand Down
22 changes: 15 additions & 7 deletions ...dation-archetype/src/main/resources/archetype-resources/__rootArtifactId__-docker/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,21 @@
</configuration>
</plugin>
<plugin>
<groupId>org.technologybrewery.orphedomos</groupId>
<artifactId>orphedomos-maven-plugin</artifactId>
<configuration>
<skip>true</skip>
<imageVersion>${project.version}</imageVersion>
<imageName>${project.artifactId}</imageName>
</configuration>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<executions>
<execution>
<id>default-build</id>
<configuration>
<skip>true</skip>
<images>
<image>
<name>${project.artifactId}:${project.version}</name>
</image>
</images>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
Expand Down
18 changes: 16 additions & 2 deletions foundation/foundation-archetype/src/main/resources/archetype-resources/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
</dependencyManagement>

<dependencies>
<!-- START: workaround to get maven build cache invalidation on new SNAPSHOTS of commonly updated plugins: -->
<!-- START: workaround to get maven build cache invalidation on new SNAPSHOTS of commonly updated plugins -->
<dependency>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>artifacts-maven-plugin</artifactId>
Expand All @@ -84,7 +84,21 @@
<version>${archetypeVersion}</version>
<scope>provided</scope>
</dependency>
<!-- END: workaround to get maven build cache invalidation on new SNAPSHOTS of commonly updated plugins -->
<!-- END: workaround to get maven build cache invalidation on new SNAPSHOTS of commonly updated plugins -->
<!-- START: workaround to get maven build cache invalidation on new SNAPSHOTS of plugin dependencies -->
<dependency>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S: are we sure we need foundation-mda? It's not really a maven plugin, but rather a dependency that is fed to fermenter. I would have though that the normal reactor would pick it up for the cache without explicitly being set.

If it is that the build cache isn't picking up dependencies listed within plugins, I might just note that so it's easier to know what it is here in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it is indeed the latter case, because after adding these two dependencies, it resolved the inconsistent behavior with the archetype tests. I will add a comment to contextualize

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will also make sure the additions are not part of the plugins comment block

<groupId>com.boozallen.aissemble</groupId>
<artifactId>foundation-mda</artifactId>
<version>${archetypeVersion}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.boozallen.aissemble</groupId>
<artifactId>foundation-upgrade</artifactId>
<version>${archetypeVersion}</version>
<scope>provided</scope>
</dependency>
<!-- END: workaround to get maven build cache invalidation on new SNAPSHOTS of plugin dependencies -->
</dependencies>

<scm>
Expand Down
8 changes: 3 additions & 5 deletions ...foundation-mda/src/main/java/com/boozallen/aiops/mda/ManualActionNotificationService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -655,15 +655,13 @@ public void addSagemakerDockerPomMessage(final GenerationContext context, final
logger.warn("Unable to find Docker module. Will not be able to direct manual updates for the deploy module's POM.xml");
} else {
String pomFilePath = pomPath + File.separator + trainingDockerArtifactId;
boolean repoUrlExists = existsInFile(pomFilePath,"<repoUrl>" + "ECR_REPO_URL" + "</repoUrl>");
boolean imageNameExists = existsInFile(pomFilePath, "<imageName>" + "ECR_REPO_URL" + "/${dockerImageName}</imageName>");
if (repoUrlExists || imageNameExists) {
boolean registryUrlExists = existsInFile(pomFilePath,"<registry>ECR_REGISTRY_URL</registry>");
if (registryUrlExists) {
final String key = getMessageKey(pomFilePath, "pom");
VelocityNotification notification = new VelocityNotification(key, new HashSet<>(), "templates/notifications/notification.sagemaker.docker.pom.vm");
notification.addToVelocityContext("artifactId", artifactId);
notification.addToVelocityContext("dockerArtifactId", trainingDockerArtifactId);
notification.addToVelocityContext("repoUrlExists", repoUrlExists);
notification.addToVelocityContext("imageNameExists", imageNameExists);
notification.addToVelocityContext("registryUrlExists", registryUrlExists);
addManualAction(pomFilePath, notification);
}
}
Expand Down
13 changes: 9 additions & 4 deletions ...mda/src/main/java/com/boozallen/aiops/mda/generator/MachineLearningPipelineGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,12 +96,17 @@ private boolean enableAutoTrain(MachineLearningPipeline pipeline) {
private boolean pipelineContainsOnnxPostAction(MachineLearningPipeline pipeline) {
boolean containsOnnx = false;

for (PostAction postAction : pipeline.getTrainingStep().getPostActions()) {
if (PipelineUtils.forOnnxModelConversion(postAction)) {
containsOnnx = true;
break;
// this surrounding if-check was added to prevent piplines that use sagemaker training from
// throwing a null pointer exception when detecting if Onnx conversion code is needed to be generated
if (pipeline.getTrainingStep() != null) {
for (PostAction postAction : pipeline.getTrainingStep().getPostActions()) {
if (PipelineUtils.forOnnxModelConversion(postAction)) {
containsOnnx = true;
break;
}
}
}

return containsOnnx;
}

Expand Down
2 changes: 1 addition & 1 deletion ...tion/foundation-mda/src/main/resources/templates/general-docker/airflow.docker.pom.xml.vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
<version>${version}</version>
</parent>

<packaging>orphedomos</packaging>
<packaging>docker-build</packaging>

<artifactId>${artifactId}</artifactId>

Expand Down
2 changes: 1 addition & 1 deletion ...mda/src/main/resources/templates/general-docker/custom.lineage.consumer.docker.pom.xml.vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
<version>${version}</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<packaging>orphedomos</packaging>
<packaging>docker-build</packaging>

<name>${parentDescriptiveName}::Custom Data Lineage Consumer Docker</name>
<artifactId>${artifactId}</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion .../foundation-mda/src/main/resources/templates/general-docker/data.access.docker.pom.xml.vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
</parent>

<artifactId>${moduleArtifactId}</artifactId>
<packaging>orphedomos</packaging>
<packaging>docker-build</packaging>

<name>${parentDescriptiveName}::Data Access</name>
<description>Build for a Data Access Docker container</description>
Expand Down
2 changes: 1 addition & 1 deletion ...on/foundation-mda/src/main/resources/templates/general-docker/inference.docker.pom.xml.vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

<artifactId>${artifactId}</artifactId>

<packaging>orphedomos</packaging>
<packaging>docker-build</packaging>

<name>${parentDescriptiveName}::Inference</name>
<description>Build for an Inference Docker container to run a compliant inference service</description>
Expand Down
2 changes: 1 addition & 1 deletion ...ation/foundation-mda/src/main/resources/templates/general-docker/mlflow.docker.pom.xml.vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

<artifactId>${artifactId}</artifactId>

<packaging>orphedomos</packaging>
<packaging>docker-build</packaging>

<name>${parentDescriptiveName}::MLflow</name>
<description>${parentDescriptiveName}::Build for a MLflow container</description>
Expand Down
2 changes: 1 addition & 1 deletion ...n-mda/src/main/resources/templates/general-docker/policy.decision.point.docker.pom.xml.vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

<artifactId>${artifactId}</artifactId>

<packaging>orphedomos</packaging>
<packaging>docker-build</packaging>

<name>${parentDescriptiveName}::Policy Decision Point</name>
<description>${parentDescriptiveName}::Build for a Policy Decision Point service container</description>
Expand Down
59 changes: 34 additions & 25 deletions ...tion-mda/src/main/resources/templates/general-docker/sagemaker-training.docker.pom.xml.vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,17 @@

<artifactId>${artifactId}</artifactId>

<packaging>orphedomos</packaging>
<packaging>docker-build</packaging>

<name>${parentDescriptiveName}::Training</name>
<description>Build for a SageMaker Training Docker container to run a SageMaker model training job</description>

<properties>
<!--various target archiectures can be specified
via this property as a comma-separated list-->
<sagemaker.docker.platforms>linux/amd64</sagemaker.docker.platforms>
</properties>

<build>
<plugins>
<plugin>
Expand Down Expand Up @@ -58,36 +64,39 @@
</dependencies>
</plugin>
<plugin>
<groupId>org.technologybrewery.orphedomos</groupId>
<artifactId>orphedomos-maven-plugin</artifactId>
<version>${version.orphedomos.plugin}</version>
<extensions>true</extensions>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<executions>
<execution>
<id>multiplatform-build-deploy</id>
<phase>prepare-package</phase>
<goals>
<goal>multiplatform-build-deploy</goal>
</goals>
<configuration>
<!--Replace with your ECR Repo URL here "aws_account_id.dkr.ecr.aws_region.amazonaws.com"-->
<repoUrl>ECR_REPO_URL</repoUrl>
<!--Replace with your Repo ID here here, this must map to the "id" tag within the associated
"server" in your Maven settings.xml file-->
<repoId>REPO_ID</repoId>
<imageVersion>latest</imageVersion>
<!--Replace with your ECR Repo URL here "aws_account_id.dkr.ecr.aws_region.amazonaws.com"-->
<imageName>ECR_REPO_URL/${dockerImageName}</imageName>
<buildArgs>
<DOCKER_BASELINE_REPO_ID>${docker.baseline.repo.id}</DOCKER_BASELINE_REPO_ID>
<VERSION_AISSEMBLE>${version.aissemble}</VERSION_AISSEMBLE>
</buildArgs>
<!--Replace with "true" if you you want to skip building and pushing SageMaker model
training image to ECR-->
<images>
<image>
<name>${project.artifactId}:latest</name>
<!--Replace with your ECR Registry URL here,
example: "aws_account_id.dkr.ecr.aws_region.amazonaws.com".
The registry value must map to the "id" value
within the associated "server" in your Maven settings.xml file-->
<registry>ECR_REGISTRY_URL</registry>
<build>
<buildx>
<platforms>
<platform>${sagemaker.docker.platforms}</platform>
</platforms>
</buildx>
<args>
<DOCKER_BASELINE_REPO_ID>${docker.baseline.repo.id}/</DOCKER_BASELINE_REPO_ID>
<VERSION_AISSEMBLE>${version.aissemble}</VERSION_AISSEMBLE>
</args>
<contextDir>${project.basedir}</contextDir>
<dockerFile>./src/main/resources/docker/Dockerfile</dockerFile>
</build>
</image>
</images>
<!--Replace with "true" if you want to skip building and pushing
the SageMaker model training image to ECR.-->
<skip>false</skip>
<targetArchitectures>
<targetArchitecture>linux/amd64</targetArchitecture>
</targetArchitectures>
</configuration>
</execution>
</executions>
Expand Down
Loading