Skip to content

Commit

Permalink
Merge branch 'main' into make_remote_snapshot_index_read_only
Browse files Browse the repository at this point in the history
Signed-off-by: Vishal Sarda <vsarda@amazon.com>
  • Loading branch information
Vishalks committed Oct 21, 2022
2 parents aee4e97 + 515f84b commit 5019b33
Show file tree
Hide file tree
Showing 180 changed files with 5,844 additions and 3,065 deletions.
27 changes: 26 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Introduce experimental searchable snapshot API ([#4680](https://github.com/opensearch-project/OpenSearch/pull/4680))
- Recommissioning of zone. REST layer support. ([#4624](https://github.com/opensearch-project/OpenSearch/pull/4604))
- Make the index restored by remote snapshot as read only but allow deletion ([#4764](https://github.com/opensearch-project/OpenSearch/pull/4764))
- Added in-flight cancellation of SearchShardTask based on resource consumption ([#4565](https://github.com/opensearch-project/OpenSearch/pull/4565))
- Apply reproducible builds configuration for OpenSearch plugins through gradle plugin ([#4746](https://github.com/opensearch-project/OpenSearch/pull/4746))
- Add groupId value propagation tests for ZIP publication task ([#4772](https://github.com/opensearch-project/OpenSearch/pull/4772))
- Add support for GeoJson Point type in GeoPoint field ([#4597](https://github.com/opensearch-project/OpenSearch/pull/4597))

### Dependencies
- Bumps `log4j-core` from 2.18.0 to 2.19.0
Expand All @@ -40,7 +44,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Bumps `azure-storage-common` from 12.18.0 to 12.18.1
- Bumps `forbiddenapis` from 3.3 to 3.4
- Bumps `gson` from 2.9.0 to 2.9.1
- Bumps `protobuf-java` from 3.21.2 to 3.21.7
- Bumps `protobuf-java` from 3.21.2 to 3.21.7k
- Bumps `azure-core` from 1.31.0 to 1.33.0
- Bumps `avro` from 1.11.0 to 1.11.1
- Bumps `woodstox-core` from 6.3.0 to 6.3.1
Expand All @@ -56,6 +60,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Bumps `jempbox` from 1.8.16 to 1.8.17 ([#4550](https://github.com/opensearch-project/OpenSearch/pull/4550))
- Bumps `hadoop-hdfs` from 3.3.3 to 3.3.4 ([#4644](https://github.com/opensearch-project/OpenSearch/pull/4644))
- Bumps `jna` from 5.11.0 to 5.12.1 ([#4656](https://github.com/opensearch-project/OpenSearch/pull/4656))
- Update Jackson Databind to 2.13.4.2 (addressing CVE-2022-42003) ([#4779](https://github.com/opensearch-project/OpenSearch/pull/4779))
- Bumps `tika` from 2.4.0 to 2.5.0 ([#4791](https://github.com/opensearch-project/OpenSearch/pull/4791))
- Exclude jettison version brought in with hadoop-minicluster. ([#4787](https://github.com/opensearch-project/OpenSearch/pull/4787))
- Bump protobuf-java to 3.21.7 in repository-gcs and repository-hdfs ([#]())

### Changed
- Dependency updates (httpcore, mockito, slf4j, httpasyncclient, commons-codec) ([#4308](https://github.com/opensearch-project/OpenSearch/pull/4308))
- Use RemoteSegmentStoreDirectory instead of RemoteDirectory ([#4240](https://github.com/opensearch-project/OpenSearch/pull/4240))
Expand All @@ -69,6 +78,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- GET api for weighted shard routing([#4275](https://github.com/opensearch-project/OpenSearch/pull/4275/))
- Unmute test RelocationIT.testRelocationWhileIndexingRandom ([#4580](https://github.com/opensearch-project/OpenSearch/pull/4580))
- Add DecommissionService and helper to execute awareness attribute decommissioning ([#4084](https://github.com/opensearch-project/OpenSearch/pull/4084))
- Delete api for weighted shard routing([#4400](https://github.com/opensearch-project/OpenSearch/pull/4400/))
- Further simplification of the ZIP publication implementation ([#4360](https://github.com/opensearch-project/OpenSearch/pull/4360))
- Relax visibility of the HTTP_CHANNEL_KEY and HTTP_SERVER_CHANNEL_KEY to make it possible for the plugins to access associated Netty4HttpChannel / Netty4HttpServerChannel instance ([#4638](https://github.com/opensearch-project/OpenSearch/pull/4638))
- Load the deprecated master role in a dedicated method instead of in setAdditionalRoles() ([#4582](https://github.com/opensearch-project/OpenSearch/pull/4582))
Expand All @@ -78,9 +88,12 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Update to Apache Lucene 9.4.0 ([#4661](https://github.com/opensearch-project/OpenSearch/pull/4661))
- Controlling discovery for decommissioned nodes ([#4590](https://github.com/opensearch-project/OpenSearch/pull/4590))
- Backport Apache Lucene version change for 2.4.0 ([#4677](https://github.com/opensearch-project/OpenSearch/pull/4677))
- Use ReplicationFailedException instead of OpensearchException in ReplicationTarget ([#4725](https://github.com/opensearch-project/OpenSearch/pull/4725))
- Fix weighted routing metadata deserialization error on process restart ([#4691](https://github.com/opensearch-project/OpenSearch/pull/4691))
- Refactor Base Action class javadocs to OpenSearch.API ([#4732](https://github.com/opensearch-project/OpenSearch/pull/4732))
- Migrate client transports to Apache HttpClient / Core 5.x ([#4459](https://github.com/opensearch-project/OpenSearch/pull/4459))
- Refactored BalancedAllocator.Balancer to LocalShardsBalancer ([#4761](https://github.com/opensearch-project/OpenSearch/pull/4761))
- Fail weight update when decommission ongoing and fail decommission when attribute not weighed away ([#4839](https://github.com/opensearch-project/OpenSearch/pull/4839))
### Deprecated
### Removed
- Remove deprecated code to add node name into log pattern of log4j property file ([#4568](https://github.com/opensearch-project/OpenSearch/pull/4568))
Expand All @@ -89,6 +102,9 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Remove LegacyESVersion.V_7_2_ and V_7_3_ Constants ([#4702](https://github.com/opensearch-project/OpenSearch/pull/4702))
- Always auto release the flood stage block ([#4703](https://github.com/opensearch-project/OpenSearch/pull/4703))
- Remove LegacyESVersion.V_7_4_ and V_7_5_ Constants ([#4704](https://github.com/opensearch-project/OpenSearch/pull/4704))
- Remove Legacy Version support from Snapshot/Restore Service ([#4728](https://github.com/opensearch-project/OpenSearch/pull/4728))
- Remove deprecated serialization logic from pipeline aggs ([#4847](https://github.com/opensearch-project/OpenSearch/pull/4847))

### Fixed
- `opensearch-service.bat start` and `opensearch-service.bat manager` failing to run ([#4289](https://github.com/opensearch-project/OpenSearch/pull/4289))
- PR reference to checkout code for changelog verifier ([#4296](https://github.com/opensearch-project/OpenSearch/pull/4296))
Expand Down Expand Up @@ -130,7 +146,14 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Fix new race condition in DecommissionControllerTests ([4688](https://github.com/opensearch-project/OpenSearch/pull/4688))
- Fix SearchStats (de)serialization (caused by https://github.com/opensearch-project/OpenSearch/pull/4616) ([#4697](https://github.com/opensearch-project/OpenSearch/pull/4697))
- Fixing Gradle warnings associated with publishPluginZipPublicationToXxx tasks ([#4696](https://github.com/opensearch-project/OpenSearch/pull/4696))
- [BUG]: Remove redundant field from GetDecommissionStateResponse ([#4751](https://github.com/opensearch-project/OpenSearch/pull/4751))
- Fixed randomly failing test ([4774](https://github.com/opensearch-project/OpenSearch/pull/4774))
- Update version check after backport ([4786](https://github.com/opensearch-project/OpenSearch/pull/4786))
- Fix decommission status update to non leader nodes ([4800](https://github.com/opensearch-project/OpenSearch/pull/4800))
- Fix recovery path for searchable snapshots ([4813](https://github.com/opensearch-project/OpenSearch/pull/4813))
- Fix bug in AwarenessAttributeDecommissionIT([4822](https://github.com/opensearch-project/OpenSearch/pull/4822))
- Fix 'org.apache.hc.core5.http.ParseException: Invalid protocol version' under JDK 16+ ([#4827](https://github.com/opensearch-project/OpenSearch/pull/4827))

### Security
- CVE-2022-25857 org.yaml:snakeyaml DOS vulnerability ([#4341](https://github.com/opensearch-project/OpenSearch/pull/4341))

Expand All @@ -144,13 +167,15 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Addition of GeoShape ValueSource level code interfaces for accessing the DocValues.
- Addition of Missing Value feature in the GeoShape Aggregations.
- Install and configure Log4j JUL Adapter for Lucene 9.4 ([#4754](https://github.com/opensearch-project/OpenSearch/pull/4754))
- Added feature to ignore indexes starting with dot during shard limit validation.([#4695](https://github.com/opensearch-project/OpenSearch/pull/4695))
### Changed
### Deprecated
### Removed
### Fixed
- PR reference to checkout code for changelog verifier ([#4296](https://github.com/opensearch-project/OpenSearch/pull/4296))
- Commit workflow for dependabot changelog helper ([#4331](https://github.com/opensearch-project/OpenSearch/pull/4331))
- Better plural stemmer than minimal_english ([#4738](https://github.com/opensearch-project/OpenSearch/pull/4738))
- Disable merge on refresh in DiskThresholdDeciderIT ([#4828](https://github.com/opensearch-project/OpenSearch/pull/4828))

### Security
[Unreleased]: https://github.com/opensearch-project/OpenSearch/compare/2.2.0...HEAD
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,13 @@
package org.opensearch.gradle.plugin

import com.github.jengelman.gradle.plugins.shadow.ShadowPlugin
import org.gradle.api.tasks.bundling.AbstractArchiveTask
import org.opensearch.gradle.BuildPlugin
import org.opensearch.gradle.NoticeTask
import org.opensearch.gradle.Version
import org.opensearch.gradle.VersionProperties
import org.opensearch.gradle.dependencies.CompileOnlyResolvePlugin
import org.opensearch.gradle.info.BuildParams
import org.opensearch.gradle.plugin.PluginPropertiesExtension
import org.opensearch.gradle.test.RestTestBasePlugin
import org.opensearch.gradle.testclusters.RunTask
import org.opensearch.gradle.util.Util
Expand Down Expand Up @@ -134,6 +134,12 @@ class PluginBuildPlugin implements Plugin<Project> {
}
project.configurations.getByName('default')
.extendsFrom(project.configurations.getByName('runtimeClasspath'))
project.tasks.withType(AbstractArchiveTask.class).configureEach { task ->
// ignore file timestamps
// be consistent in archive file order
task.preserveFileTimestamps = false
task.reproducibleFileOrder = true
}
// allow running ES with this plugin in the foreground of a build
project.tasks.register('run', RunTask) {
dependsOn(project.tasks.bundlePlugin)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@

package org.opensearch.gradle.plugin;

import org.gradle.api.tasks.bundling.AbstractArchiveTask;
import org.opensearch.gradle.BwcVersions;
import org.opensearch.gradle.test.GradleUnitTestCase;
import org.gradle.api.Project;
Expand Down Expand Up @@ -64,6 +65,10 @@ public void testApply() {
assertNotNull("plugin extensions has the right type", project.getExtensions().findByType(PluginPropertiesExtension.class));

assertNull("plugin should not create the integTest task", project.getTasks().findByName("integTest"));
project.getTasks().withType(AbstractArchiveTask.class).forEach(t -> {
assertFalse(String.format("task '%s' should not preserve timestamps", t.getName()), t.isPreserveFileTimestamps());
assertTrue(String.format("task '%s' should have reproducible file order", t.getName()), t.isReproducibleFileOrder());
});
}

@Ignore("https://github.com/elastic/elasticsearch/issues/47123")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,76 @@ public void useDefaultValues() throws IOException, URISyntaxException, XmlPullPa
assertEquals(model.getUrl(), "https://github.com/doe/sample-plugin");
}

/**
* If the `group` is defined in gradle's allprojects section then it does not have to defined in publications.
*/
@Test
public void allProjectsGroup() throws IOException, URISyntaxException, XmlPullParserException {
GradleRunner runner = prepareGradleRunnerFromTemplate("allProjectsGroup.gradle", "build", ZIP_PUBLISH_TASK);
BuildResult result = runner.build();

/** Check if build and {@value ZIP_PUBLISH_TASK} tasks have run well */
assertEquals(SUCCESS, result.task(":" + "build").getOutcome());
assertEquals(SUCCESS, result.task(":" + ZIP_PUBLISH_TASK).getOutcome());

// Parse the maven file and validate default values
MavenXpp3Reader reader = new MavenXpp3Reader();
Model model = reader.read(
new FileReader(
new File(
projectDir.getRoot(),
String.join(
File.separator,
"build",
"local-staging-repo",
"org",
"opensearch",
PROJECT_NAME,
"2.0.0.0",
PROJECT_NAME + "-2.0.0.0.pom"
)
)
)
);
assertEquals(model.getVersion(), "2.0.0.0");
assertEquals(model.getGroupId(), "org.opensearch");
}

/**
* The groupId value can be defined on several levels. This tests that the most internal level outweighs other levels.
*/
@Test
public void groupPriorityLevel() throws IOException, URISyntaxException, XmlPullParserException {
GradleRunner runner = prepareGradleRunnerFromTemplate("groupPriorityLevel.gradle", "build", ZIP_PUBLISH_TASK);
BuildResult result = runner.build();

/** Check if build and {@value ZIP_PUBLISH_TASK} tasks have run well */
assertEquals(SUCCESS, result.task(":" + "build").getOutcome());
assertEquals(SUCCESS, result.task(":" + ZIP_PUBLISH_TASK).getOutcome());

// Parse the maven file and validate default values
MavenXpp3Reader reader = new MavenXpp3Reader();
Model model = reader.read(
new FileReader(
new File(
projectDir.getRoot(),
String.join(
File.separator,
"build",
"local-staging-repo",
"level",
"3",
PROJECT_NAME,
"2.0.0.0",
PROJECT_NAME + "-2.0.0.0.pom"
)
)
)
);
assertEquals(model.getVersion(), "2.0.0.0");
assertEquals(model.getGroupId(), "level.3");
}

/**
* In this case the Publication entity is completely missing but still the POM file is generated using the default
* values including the groupId and version values obtained from the Gradle project object.
Expand Down
28 changes: 28 additions & 0 deletions buildSrc/src/test/resources/pluginzip/allProjectsGroup.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
plugins {
id 'java-gradle-plugin'
id 'opensearch.pluginzip'
}

version='2.0.0.0'

// A bundlePlugin task mockup
tasks.register('bundlePlugin', Zip.class) {
archiveFileName = "sample-plugin-${version}.zip"
destinationDirectory = layout.buildDirectory.dir('distributions')
from layout.projectDirectory.file('sample-plugin-source.txt')
}

allprojects {
group = 'org.opensearch'
}

publishing {
publications {
pluginZip(MavenPublication) { publication ->
pom {
name = "sample-plugin"
description = "pluginDescription"
}
}
}
}
30 changes: 30 additions & 0 deletions buildSrc/src/test/resources/pluginzip/groupPriorityLevel.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
plugins {
id 'java-gradle-plugin'
id 'opensearch.pluginzip'
}

version='2.0.0.0'

// A bundlePlugin task mockup
tasks.register('bundlePlugin', Zip.class) {
archiveFileName = "sample-plugin-${version}.zip"
destinationDirectory = layout.buildDirectory.dir('distributions')
from layout.projectDirectory.file('sample-plugin-source.txt')
}

allprojects {
group = 'level.1'
}

publishing {
publications {
pluginZip(MavenPublication) { publication ->
groupId = "level.2"
pom {
name = "sample-plugin"
description = "pluginDescription"
groupId = "level.3"
}
}
}
}
3 changes: 2 additions & 1 deletion buildSrc/version.properties
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,14 @@ bundled_jdk = 17.0.4+8
spatial4j = 0.7
jts = 1.15.0
jackson = 2.13.4
jackson_databind = 2.13.4
jackson_databind = 2.13.4.2
snakeyaml = 1.32
icu4j = 70.1
supercsv = 2.4.0
log4j = 2.17.1
slf4j = 1.7.36
asm = 9.3
jettison = 1.5.1

# when updating the JNA version, also update the version in buildSrc/build.gradle
jna = 5.12.1
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -889,6 +889,7 @@ public void testApiNamingConventions() throws Exception {
"remote_store.restore",
"cluster.put_weighted_routing",
"cluster.get_weighted_routing",
"cluster.delete_weighted_routing",
"cluster.put_decommission_awareness",
"cluster.get_decommission_awareness",
"cluster.delete_decommission_awareness", };
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,10 @@

package org.opensearch.client;

import org.apache.hc.core5.function.Factory;
import org.apache.hc.core5.http.Header;
import org.apache.hc.core5.http.nio.ssl.TlsStrategy;
import org.apache.hc.core5.reactor.ssl.TlsDetails;
import org.apache.hc.core5.util.Timeout;
import org.apache.hc.client5.http.async.HttpAsyncClient;
import org.apache.hc.client5.http.auth.CredentialsProvider;
Expand All @@ -48,6 +50,7 @@
import org.apache.hc.client5.http.impl.async.HttpAsyncClientBuilder;

import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLEngine;

import java.security.AccessController;
import java.security.NoSuchAlgorithmException;
Expand Down Expand Up @@ -311,7 +314,16 @@ private CloseableHttpAsyncClient createHttpClient() {
}

try {
final TlsStrategy tlsStrategy = ClientTlsStrategyBuilder.create().setSslContext(SSLContext.getDefault()).build();
final TlsStrategy tlsStrategy = ClientTlsStrategyBuilder.create()
.setSslContext(SSLContext.getDefault())
// See https://issues.apache.org/jira/browse/HTTPCLIENT-2219
.setTlsDetailsFactory(new Factory<SSLEngine, TlsDetails>() {
@Override
public TlsDetails create(final SSLEngine sslEngine) {
return new TlsDetails(sslEngine.getSession(), sslEngine.getApplicationProtocol());
}
})
.build();

final PoolingAsyncClientConnectionManager connectionManager = PoolingAsyncClientConnectionManagerBuilder.create()
.setMaxConnPerRoute(DEFAULT_MAX_CONN_PER_ROUTE)
Expand Down
Loading

0 comments on commit 5019b33

Please sign in to comment.