Merge branch 'main' into feature/wrr-get-api

.github/workflows/gradle-check.yml

@@ -7,6 +7,10 @@ on:
       - 'dependabot/**'
   pull_request_target:
     types: [opened, synchronize, reopened]
+  workflow_run:
+    workflows: ["Gradle Precommit"]
+    types:
+      - completed
 
 jobs:
   gradle-check:

.github/workflows/precommit.yml

@@ -1,9 +1,12 @@
 name: Gradle Precommit
 on: [pull_request]
-
+ 
 jobs:
   precommit:
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest] 
     steps:
       - uses: actions/checkout@v2
       - name: Set up JDK 11
@@ -13,4 +16,4 @@ jobs:
           distribution: adopt
       - name: Run Gradle
         run: |
-          ./gradlew precommit --parallel
+          ./gradlew javadoc precommit --parallel

CHANGELOG.md

@@ -1,8 +1,11 @@
 # CHANGELOG
+
 Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ## [Unreleased]
+
 ### Added
+
 - Add support for s390x architecture ([#4001](https://github.com/opensearch-project/OpenSearch/pull/4001))
 - Github workflow for changelog verification ([#4085](https://github.com/opensearch-project/OpenSearch/pull/4085))
 - Point in time rest layer changes for create and delete PIT API ([#4064](https://github.com/opensearch-project/OpenSearch/pull/4064))
@@ -21,10 +24,17 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - [Segment Replication] Add snapshot and restore tests for segment replication feature ([#3993](https://github.com/opensearch-project/OpenSearch/pull/3993))
 - Added missing javadocs for `:example-plugins` modules ([#4540](https://github.com/opensearch-project/OpenSearch/pull/4540))
 - Add missing Javadoc tag descriptions in unit tests ([#4629](https://github.com/opensearch-project/OpenSearch/pull/4629))
+- Add getter for path field in NestedQueryBuilder ([#4636](https://github.com/opensearch-project/OpenSearch/pull/4636))
+- Added precommit support for windows ([#4676](https://github.com/opensearch-project/OpenSearch/pull/4676))
+- Added release notes for 1.3.6 ([#4681](https://github.com/opensearch-project/OpenSearch/pull/4681))
+- Added precommit support for MacOS ([#4682](https://github.com/opensearch-project/OpenSearch/pull/4682))
+- Recommission API changes for service layer ([#4320](https://github.com/opensearch-project/OpenSearch/pull/4320))
 ### Dependencies
 - Bumps `log4j-core` from 2.18.0 to 2.19.0
 - Bumps `reactor-netty-http` from 1.0.18 to 1.0.23
 - Bumps `jettison` from 1.5.0 to 1.5.1
+- Bumps `azure-storage-common` from 12.18.0 to 12.18.1
+- Bumps `forbiddenapis` from 3.3 to 3.4
 
 
 ### Dependencies
@@ -39,8 +49,10 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Bumps `reactor-core` from 3.4.18 to 3.4.23 ([#4548](https://github.com/opensearch-project/OpenSearch/pull/4548))
 - Bumps `jempbox` from 1.8.16 to 1.8.17 ([#4550](https://github.com/opensearch-project/OpenSearch/pull/4550))
 - Bumps `hadoop-hdfs` from 3.3.3 to 3.3.4 ([#4644](https://github.com/opensearch-project/OpenSearch/pull/4644))
+- Bumps `jna` from 5.11.0 to 5.12.1 ([#4656](https://github.com/opensearch-project/OpenSearch/pull/4656))
 
 ### Changed
+
 - Dependency updates (httpcore, mockito, slf4j, httpasyncclient, commons-codec) ([#4308](https://github.com/opensearch-project/OpenSearch/pull/4308))
 - Use RemoteSegmentStoreDirectory instead of RemoteDirectory ([#4240](https://github.com/opensearch-project/OpenSearch/pull/4240))
 - Plugin ZIP publication groupId value is configurable ([#4156](https://github.com/opensearch-project/OpenSearch/pull/4156))
@@ -56,7 +68,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Further simplification of the ZIP publication implementation ([#4360](https://github.com/opensearch-project/OpenSearch/pull/4360))
 - Relax visibility of the HTTP_CHANNEL_KEY and HTTP_SERVER_CHANNEL_KEY to make it possible for the plugins to access associated Netty4HttpChannel / Netty4HttpServerChannel instance ([#4638](https://github.com/opensearch-project/OpenSearch/pull/4638))
 - Load the deprecated master role in a dedicated method instead of in setAdditionalRoles() ([#4582](https://github.com/opensearch-project/OpenSearch/pull/4582))
+- Include Windows OS in Bootstrap initializeNatives() check for definitelyRunningAsRoot() ([#4656](https://github.com/opensearch-project/OpenSearch/pull/4656))
 - Add APIs (GET/PUT) to decommission awareness attribute ([#4261](https://github.com/opensearch-project/OpenSearch/pull/4261))
+- Improve Gradle pre-commit checks to pre-empt Jenkins build ([#4660](https://github.com/opensearch-project/OpenSearch/pull/4660))
+- Update to Apache Lucene 9.4.0 ([#4661](https://github.com/opensearch-project/OpenSearch/pull/4661))
+- Backport Apache Lucene version change for 2.4.0 ([#4677](https://github.com/opensearch-project/OpenSearch/pull/4677))
 
 ### Deprecated
 
@@ -65,6 +81,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Unused object and import within TransportClusterAllocationExplainAction ([#4639](https://github.com/opensearch-project/OpenSearch/pull/4639))
 
 ### Fixed
+
 - `opensearch-service.bat start` and `opensearch-service.bat manager` failing to run ([#4289](https://github.com/opensearch-project/OpenSearch/pull/4289))
 - PR reference to checkout code for changelog verifier ([#4296](https://github.com/opensearch-project/OpenSearch/pull/4296))
 - `opensearch.bat` and `opensearch-service.bat install` failing to run, missing logs directory ([#4305](https://github.com/opensearch-project/OpenSearch/pull/4305))
@@ -89,14 +106,27 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Fixed the ignore_malformed setting to also ignore objects ([#4494](https://github.com/opensearch-project/OpenSearch/pull/4494))
 - [Segment Replication] Ignore lock file when testing cleanupAndPreserveLatestCommitPoint ([#4544](https://github.com/opensearch-project/OpenSearch/pull/4544))
 - Updated jackson to 2.13.4 and snakeyml to 1.32 ([#4556](https://github.com/opensearch-project/OpenSearch/pull/4556))
+- Fixing PIT flaky tests ([4632](https://github.com/opensearch-project/OpenSearch/pull/4632))
+- Fixed day of year defaulting for round up parser ([#4627](https://github.com/opensearch-project/OpenSearch/pull/4627))
+- Fixed the SnapshotsInProgress error during index deletion ([#4570](https://github.com/opensearch-project/OpenSearch/pull/4570))
+- [Segment Replication] Adding check to make sure checkpoint is not processed when a shard's shard routing is primary ([#4630](https://github.com/opensearch-project/OpenSearch/pull/4630))
+- [Bug]: Fixed invalid location of JDK dependency for arm64 architecture([#4613](https://github.com/opensearch-project/OpenSearch/pull/4613))
+- [Bug]: Alias filter lost after rollover ([#4499](https://github.com/opensearch-project/OpenSearch/pull/4499))
+- Fixed the SnapshotsInProgress error during index deletion ([#4570](https://github.com/opensearch-project/OpenSearch/pull/4570))
+- [Segment Replication] Adding check to make sure checkpoint is not processed when a shard's shard routing is primary ([#4630](https://github.com/opensearch-project/OpenSearch/pull/4630))
 - [Bug]: Fixed invalid location of JDK dependency for arm64 architecture([#4613](https://github.com/opensearch-project/OpenSearch/pull/4613))
 - [Bug]: Alias filter lost after rollover ([#4499](https://github.com/opensearch-project/OpenSearch/pull/4499))
+- Attempt to fix Github workflow for Gradle Check job ([#4679](https://github.com/opensearch-project/OpenSearch/pull/4679))
+- Fix flaky DecommissionControllerTests.testTimesOut ([4683](https://github.com/opensearch-project/OpenSearch/pull/4683))
 
 ### Security
+
 - CVE-2022-25857 org.yaml:snakeyaml DOS vulnerability ([#4341](https://github.com/opensearch-project/OpenSearch/pull/4341))
 
 ## [2.x]
+
 ### Added
+
 - Github workflow for changelog verification ([#4085](https://github.com/opensearch-project/OpenSearch/pull/4085))
 - Label configuration for dependabot PRs ([#4348](https://github.com/opensearch-project/OpenSearch/pull/4348))
 - Added RestLayer Changes for PIT stats ([#4217](https://github.com/opensearch-project/OpenSearch/pull/4217))
@@ -112,11 +142,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 ### Removed
 
 ### Fixed
+
 - PR reference to checkout code for changelog verifier ([#4296](https://github.com/opensearch-project/OpenSearch/pull/4296))
 - Commit workflow for dependabot changelog helper ([#4331](https://github.com/opensearch-project/OpenSearch/pull/4331))
 
 ### Security
 
-
 [Unreleased]: https://github.com/opensearch-project/OpenSearch/compare/2.2.0...HEAD
 [2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.2.0...2.x

buildSrc/build.gradle

@@ -110,7 +110,7 @@ dependencies {
   api 'com.netflix.nebula:gradle-info-plugin:11.3.3'
   api 'org.apache.rat:apache-rat:0.13'
   api 'commons-io:commons-io:2.7'
-  api "net.java.dev.jna:jna:5.11.0"
+  api "net.java.dev.jna:jna:5.12.1"
   api 'gradle.plugin.com.github.johnrengelman:shadow:7.1.2'
   api 'org.jdom:jdom2:2.0.6.1'
   api 'org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.7.10'

buildSrc/src/testKit/thirdPartyAudit/build.gradle

@@ -40,7 +40,7 @@ repositories {
 }
 
 dependencies {
-  forbiddenApisCliJar 'de.thetaphi:forbiddenapis:3.3'
+  forbiddenApisCliJar 'de.thetaphi:forbiddenapis:3.4'
   jdkJarHell 'org.opensearch:opensearch-core:current'
   compileOnly "org.${project.properties.compileOnlyGroup}:${project.properties.compileOnlyVersion}"
   implementation "org.${project.properties.compileGroup}:${project.properties.compileVersion}"

buildSrc/version.properties

@@ -1,5 +1,5 @@
 opensearch        = 3.0.0
-lucene            = 9.4.0-snapshot-ddf0d0a
+lucene            = 9.4.0
 
 bundled_jdk_vendor = adoptium
 bundled_jdk = 17.0.4+8
@@ -19,7 +19,7 @@ slf4j             = 1.7.36
 asm               = 9.3
 
 # when updating the JNA version, also update the version in buildSrc/build.gradle
-jna               = 5.5.0
+jna               = 5.12.1
 
 netty             = 4.1.79.Final
 joda              = 2.10.13

client/rest-high-level/src/test/java/org/opensearch/client/PitIT.java

@@ -24,6 +24,7 @@
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 
@@ -71,7 +72,7 @@ public void testCreateAndDeletePit() throws IOException {
         assertTrue(deletePitResponse.getDeletePitResults().get(0).getPitId().equals(createPitResponse.getId()));
     }
 
-    public void testDeleteAllAndListAllPits() throws IOException {
+    public void testDeleteAllAndListAllPits() throws IOException, InterruptedException {
         CreatePitRequest pitRequest = new CreatePitRequest(new TimeValue(1, TimeUnit.DAYS), true, "index");
         CreatePitResponse pitResponse = execute(pitRequest, highLevelClient()::createPit, highLevelClient()::createPitAsync);
         CreatePitResponse pitResponse1 = execute(pitRequest, highLevelClient()::createPit, highLevelClient()::createPitAsync);
@@ -90,16 +91,19 @@ public void testDeleteAllAndListAllPits() throws IOException {
         List<String> pits = getAllPitResponse.getPitInfos().stream().map(r -> r.getPitId()).collect(Collectors.toList());
         assertTrue(pits.contains(pitResponse.getId()));
         assertTrue(pits.contains(pitResponse1.getId()));
+        CountDownLatch countDownLatch = new CountDownLatch(1);
         ActionListener<DeletePitResponse> deletePitListener = new ActionListener<>() {
             @Override
             public void onResponse(DeletePitResponse response) {
+                countDownLatch.countDown();
                 for (DeletePitInfo deletePitInfo : response.getDeletePitResults()) {
                     assertTrue(deletePitInfo.isSuccessful());
                 }
             }
 
             @Override
             public void onFailure(Exception e) {
+                countDownLatch.countDown();
                 if (!(e instanceof OpenSearchStatusException)) {
                     throw new AssertionError("Delete all failed");
                 }
@@ -123,6 +127,7 @@ public void onFailure(Exception e) {
         };
         highLevelClient().getAllPitsAsync(RequestOptions.DEFAULT, getPitsListener);
         highLevelClient().deleteAllPitsAsync(RequestOptions.DEFAULT, deletePitListener);
+        assertTrue(countDownLatch.await(10, TimeUnit.SECONDS));
         // validate no pits case
         getAllPitResponse = highLevelClient().getAllPits(RequestOptions.DEFAULT);
         assertTrue(getAllPitResponse.getPitInfos().size() == 0);

modules/lang-expression/licenses/lucene-expressions-9.4.0.jar.sha1

@@ -0,0 +1 @@
+19749e264805171009836cbedecc5494b13cd920

plugins/analysis-icu/licenses/lucene-analysis-icu-9.4.0.jar.sha1

@@ -0,0 +1 @@
+aa0f250558375922f3091820361156e514fe1842

plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.4.0.jar.sha1

@@ -0,0 +1 @@
+32eb1ad367ab1289804aeed95ea7216711a7764d

plugins/analysis-nori/licenses/lucene-analysis-nori-9.4.0.jar.sha1

@@ -0,0 +1 @@
+63661714be65f882a921d281965b0779fd487b90

plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.4.0.jar.sha1

@@ -0,0 +1 @@
+1034d876551fc21f7835b456dab01db21b9a4af6

plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.4.0.jar.sha1

@@ -0,0 +1 @@
+f704ee4b14e2fe2622bb983f04b36a32df8fd4a7

plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.4.0.jar.sha1

@@ -0,0 +1 @@
+a95ff17b51da6b3da641fa4053e5ee9ea2ff5daf

plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.4.0.jar.sha1

@@ -0,0 +1 @@
+13e1ae2c760d8c0d7990ffe3296e46d9d8e6f842

plugins/repository-azure/build.gradle

@@ -45,7 +45,7 @@ opensearchplugin {
 
 dependencies {
   api 'com.azure:azure-core:1.31.0'
-  api 'com.azure:azure-storage-common:12.18.0'
+  api 'com.azure:azure-storage-common:12.18.1'
   api 'com.azure:azure-core-http-netty:1.12.4'
   api "io.netty:netty-codec-dns:${versions.netty}"
   api "io.netty:netty-codec-socks:${versions.netty}"

plugins/repository-azure/licenses/azure-storage-common-12.18.1.jar.sha1

@@ -0,0 +1 @@
+09f5229d0775dff1b21cc3cb2936de751e79b5ac

release-notes/opensearch.release-notes-1.3.6.md

@@ -0,0 +1,10 @@
+## 2022-10-04 Version 1.3.6 Release Notes
+
+### Upgrades
+* Update to Netty 4.1.80.Final ([#4379](https://github.com/opensearch-project/OpenSearch/pull/4379))
+* Revert to Netty 4.1.79.Final ([#4433](https://github.com/opensearch-project/OpenSearch/pull/4433))
+* Bumps jackson and snakeyaml dependencies ([#4600](https://github.com/opensearch-project/OpenSearch/pull/4600))
+* Bump commons-configuration2 from 2.7 to 2.8.0 in /plugins/repository-hdfs ([#4646](https://github.com/opensearch-project/OpenSearch/pull/4646))
+
+### Bug Fixes
+* Set analyzer to regex query string search ([#4220](https://github.com/opensearch-project/OpenSearch/pull/4220))

server/licenses/jna-5.12.1.jar.sha1

@@ -0,0 +1 @@
+b1e93a735caea94f503e95e6fe79bf9cdc1e985d

server/licenses/lucene-analysis-common-9.4.0.jar.sha1

@@ -0,0 +1 @@
+02fbd4e87241411fcf5d34e92a50bee46ab164dc

server/licenses/lucene-backward-codecs-9.4.0.jar.sha1

@@ -0,0 +1 @@
+259863dfd107645de6146b3c87b4ecee66a4d43d

server/licenses/lucene-core-9.4.0.jar.sha1

@@ -0,0 +1 @@
+cca1116f813c0f0c63acfac4c952baf29d46d76b

server/licenses/lucene-grouping-9.4.0.jar.sha1

@@ -0,0 +1 @@
+51bec1d5acc8ecaf9f50e047d3f86d60c7a958f4

server/licenses/lucene-highlighter-9.4.0.jar.sha1

@@ -0,0 +1 @@
+c8cf8c9308d8fb18a927c7ed267a14ace3990a5f

server/licenses/lucene-join-9.4.0.jar.sha1

@@ -0,0 +1 @@
+99b2d3c8e137a6853a2503456897d47d4f18974b

server/licenses/lucene-memory-9.4.0.jar.sha1

@@ -0,0 +1 @@
+881cb214e79da14de35cb0e8e6779d2722828a96

server/licenses/lucene-misc-9.4.0.jar.sha1

@@ -0,0 +1 @@
+a126123e482e6bf2e7aea670d221a2a39d3277dc

server/licenses/lucene-queries-9.4.0.jar.sha1

@@ -0,0 +1 @@
+fe74dbfe9dba9ee9ee2cb80f151fde97fb4efd12

server/licenses/lucene-queryparser-9.4.0.jar.sha1

@@ -0,0 +1 @@
+13f108a8572fcf0670c7df3ba8dbe1076d0e0dbe

server/licenses/lucene-sandbox-9.4.0.jar.sha1

@@ -0,0 +1 @@
+e7a676a12ea50dcbf64564f4e4022f939f0a627d

server/licenses/lucene-spatial-extras-9.4.0.jar.sha1

@@ -0,0 +1 @@
+84d956d1cb1458c51967af1c4acadd2a1f92634d

server/licenses/lucene-spatial3d-9.4.0.jar.sha1

@@ -0,0 +1 @@
+76887ca708f23b13613e45fb9e307c548b22c6da

server/licenses/lucene-suggest-9.4.0.jar.sha1

@@ -0,0 +1 @@
+406c9c539f262449d3b1e57e7bc4302efeecaf6c