-
Notifications
You must be signed in to change notification settings - Fork 483
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Blog Post: OpenSearch plugin zips now in Maven repo #930
Merged
krisfreedain
merged 14 commits into
opensearch-project:main
from
prudhvigodithi:pluginzips
Aug 25, 2022
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
440d45b
Blob Post: OpenSearch plugin zips now in Maven repo
prudhvigodithi a488d6e
Blob Post: OpenSearch plugin zips now in Maven repo
prudhvigodithi 9517a36
Blob Post: OpenSearch plugin zips now in Maven repo
prudhvigodithi def0426
MVN cli command fix
prudhvigodithi 176c6e0
MVN cli command fix
prudhvigodithi f265e40
MVN cli command fix
prudhvigodithi f39b12e
Add Nate's editorial feedback
Naarcha-AWS 468df99
Delete extraneuos file
Naarcha-AWS 4baf4fe
Add final piece of editorial feedback
Naarcha-AWS 75e744b
Remove summary section
Naarcha-AWS f85f993
Final editorial round
Naarcha-AWS 70ccaa4
Changes with mvn cli's and extra space adjustments
prudhvigodithi 3133ffe
Update date
prudhvigodithi 3c77c4e
Revered Gemfile changes
prudhvigodithi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
--- | ||
name: Prudhvi Godithi | ||
short_name: prudhvigodithi | ||
linkedin: 'prudhvi-godithi' | ||
photo: '/assets/media/authors/prudhvigodithi.jpg' | ||
--- | ||
|
||
**Prudhvi Godithi** is an AWS System Engineer working on the OpenSearch project. |
89 changes: 89 additions & 0 deletions
89
_posts/2022-07-24-opensearch-plugin-zips-now-in-maven-repo.markdown
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,89 @@ | ||||||
--- | ||||||
layout: post | ||||||
title: "OpenSearch plugin zips now in Maven repo" | ||||||
authors: | ||||||
- prudhvigodithi | ||||||
date: 2022-08-23 01:01:01 -0700 | ||||||
categories: | ||||||
- technical-post | ||||||
twittercard: | ||||||
description: "Details on how to consume OpenSearch plugin zips from a Maven repo and the process involved in shipping them to a Maven repo." | ||||||
--- | ||||||
|
||||||
Starting with the release of OpenSearch `2.1.0`, OpenSearch plugin zips are now signed and published to a central Apache Maven [repo](https://repo1.maven.org/maven2/org/opensearch/plugin/). Using the [Release zips](https://repo1.maven.org/maven2/org/opensearch/plugin/) and [Snapshot zips](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/) Maven Repo URLs, OpenSearch plugin zips can now be consumed as a dependency to build other plugins or fetched as standalone components for your OpenSearch cluster. | ||||||
|
||||||
## Motivation | ||||||
|
||||||
Before OpenSearch 2.1, plugin zips used as dependencies could not be downloaded dynamically during runtime because plugin zips were not a part of the version-controlled Maven system. The only mechanism for plugin downloads was each plugin's respective Java jars through Maven coordinates. This system forced users who wanted more control over their OpenSearch plugin configuration to use a dependency plugin zip built on a developer desktop instead of a more reliable version-controlled plugin. | ||||||
|
||||||
Furthermore, to facilitate the plugin availability as a dependency, tests executed against the OpenSearch build process from zip were not accurate, as each local build zip had to find the plugin repo in order to ensure that plugin's availability. These restrictions were challenging to our community because using plugin zips as separate isolated components via a cached mechanism proved to be impossible. | ||||||
|
||||||
## Benefits of Maven | ||||||
|
||||||
With Maven, plugin zips can now be retrieved by: | ||||||
|
||||||
- Downloading each plugin directly using their respective Maven coordinates. | ||||||
- Using clickstream from the central Maven repo, which can be cached later to a local Maven repo. | ||||||
- Fetching the development `SNAPSHOT` version with the same Maven `groupID` as [org.opensearch.plugin](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/). | ||||||
|
||||||
Using OpenSearch plugin zips through Maven offers the following benefits: | ||||||
|
||||||
- Plugins zip in the central Maven repo are already signed with `.asc`, `.md5`, `.sha1`, `.sha256`, and `.sha512` extensions. | ||||||
- Users are no longer required to to check in zips to any `src/` files because zips can be fetched with the right `groupID`, `artifactID`, and `version`. | ||||||
- Tests and continuous integration (CI) workflows can directly run against zips from the Maven repo instead of requiring a manual download. | ||||||
|
||||||
## Maven zip publication with Gradle | ||||||
|
||||||
OpenSearch publishes plugin zips using a custom Gradle plugin, `opensearch.pluginzip`. With OpenSearch 2.1, all OpenSearch Gradle-supported plugins create a new task, `publishPluginZipPublicationToZipStagingRepository`. The task performs all the heavy lifting for users by: | ||||||
|
||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
- Identifying the distribution plugin zip. | ||||||
- Setting the Maven coordinates. | ||||||
- Generating the POM file. | ||||||
- Updating with the user-generated POM fields. | ||||||
- Publishing the zip to your Maven repo. | ||||||
|
||||||
You can find more details about the inner workings of OpenSearch plugins in the [opensearch-plugins repo](https://github.com/opensearch-project/opensearch-plugins/blob/main/BUILDING.md#opensearchpluginzip). | ||||||
|
||||||
**Figure 1**: Workflow that ships generated plugin zips to Maven | ||||||
|
||||||
![Figure 1: Workflow that ship zips to maven]({{ site.baseurl }}/assets/media/blog-images/2022-07-24-opensearch-plugin-zips-now-in-maven-repo/figure1.png){: .img-fluid }**Figure 1**: Workflow that ships generated plugin zips to Maven | ||||||
|
||||||
## Consume plugin in zips | ||||||
|
||||||
You can fetch plugin zips in three different ways. | ||||||
|
||||||
**Using the Maven CLI** | ||||||
|
||||||
Consume from the Central Maven repo: | ||||||
|
||||||
``` | ||||||
mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get -DrepoUrl=https://repo1.maven.org/maven2 -Dartifact=org.opensearch.plugin:opensearch-job-scheduler:2.1.0.0:zip | ||||||
``` | ||||||
|
||||||
Consume from the Snapshot Maven repo: | ||||||
|
||||||
``` | ||||||
mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get -DrepoUrl=https://aws.oss.sonatype.org/content/repositories/snapshots -Dartifact=org.opensearch.plugin:opensearch-job-scheduler:2.1.0.0-SNAPSHOT:zip | ||||||
``` | ||||||
|
||||||
**Gradle Project: Using the build.gradle file** | ||||||
|
||||||
``` | ||||||
dependencies { | ||||||
classpath "org.opensearch.plugin:opensearch-job-scheduler:2.1.0.0@zip" | ||||||
classpath "org.opensearch.plugin:opensearch-knn:2.1.0.0@zip" | ||||||
} | ||||||
``` | ||||||
|
||||||
**Maven Project: Using the pom.xml file** | ||||||
|
||||||
``` | ||||||
<dependencies> | ||||||
<dependency> | ||||||
<groupId>org.opensearch.plugin</groupId> | ||||||
<artifactId>opensearch-job-scheduler</artifactId> | ||||||
<version>2.1.0.0</version> | ||||||
<packaging>zip</packaging> | ||||||
</dependency> | ||||||
</dependencies> | ||||||
``` |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+116 KB
...dia/blog-images/2022-07-24-opensearch-plugin-zips-now-in-maven-repo/figure1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I deleted the Summary section and used that description as the Intro.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sentence Reformation:
Starting with the release of OpenSearch
2.1.0
, OpenSearch plugin zips are now signed and published to a central Apache Maven repo. Using the Release zips and Snapshot zips Maven Repo URL’s, OpenSearch plugin zips can now be consumed as a dependency to build other plugins or fetched as standalone components for your OpenSearch cluster.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In "Maven Repo URL's", change to "Maven repo URLs".