Add experimental support for zstd compression. (#3577) (#6982)

* Add experimental support for zstd and lz4 (native) compression.

Add experimental support for zstd (with and without dictionary support) and lz4 (native) compressions as discussed in: #3354.

Users would be able to set the index.codec setting with the values "lz4_native" (for lz4 native), "zstd" (for zstd with dictionary), and "zstd_no_dict" (for zstd without a dictionary).



* Fix license issues, add tests for zstd, zstd_no_dict, and lz4_native compressions.



* Fix DCO and and issues with CodecTests.java.



* Fix forbidden api violation error for lz4-java.



* Fix license headers. Remove and fix unnecessary fields.



* Fix magic numbers. Use more restrictive access modifiers.



* Use protected access modifier for Zstd and LZ4 compression mode classes.



* Allow negative compression levels for zstd. Use more restrictive access modifiers.



* Use a more restrictive permission for loading zstd-jni and lz4-java libraries.



* Rename a file (follow a consistent version naming convention).



* Refactor and create a new custom-codecs sandbox module.



* Remove blank lines.



* Restore Lucene92CustomCodec to extend from FilterCodec.



* Make use of the compressionLevel argument.



* Make Lucene92CustomCodec abstract and use a package-private access modifier.



* Fix missing JavaDoc issues. Remove unused field in PerFieldMappingPostingFormatCodec.



* Fix lint errors.



* Fix the description for the custom-codecs plugin.



* Fix wildcard import and improve documentation.



* Access control exception fixed. Removed lz4-java support for now.

- PRs were made to zstd-jni and lz4-java to use AccessController.doPrivileged.
- The zstd-jni PR is merged since version 1.5.4-1.
- The lz4-java support temporarily removed until the PR gets merged.



* Upgrade plugin to use Lucene95Codec. Rename files accordingly.

- Upgrade plugin to use Lucene95Codec. Rename files accordingly.
- Fix lint issue with plugin-security.
- Remove thridPartyAudit that was there for supporting lz4-java.



* Add test cases for compression/decompression. Other minor changes.

  - add test cases for compression/decompression.
  - rename package.
  - add a CHANGELOG entry.
  - add more checks for signed integer arithmetic.



* Remove ES grant in plugin-security.policy. Fix minor javadoc issue.

  - Remove ES grant in plugin-security.policy file.
  - Replace @link and @see to fix javadoc error.



* Upgrade jettison version to 1.5.4.



* Update SHA for jettison 1.5.4.



---------



(cherry picked from commit f071c9b)

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

CHANGELOG.md

@@ -19,6 +19,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Enable sort optimization for all NumericTypes ([#6464](https://github.com/opensearch-project/OpenSearch/pull/6464)
 - Remove 'cluster_manager' role attachment when using 'node.master' deprecated setting ([#6331](https://github.com/opensearch-project/OpenSearch/pull/6331))
 - Add new cluster settings to ignore weighted round-robin routing and fallback to default behaviour. ([#6834](https://github.com/opensearch-project/OpenSearch/pull/6834))
+- Add experimental support for ZSTD compression. ([#3577](https://github.com/opensearch-project/OpenSearch/pull/3577))
 
 ### Dependencies
 - Bump `org.apache.logging.log4j:log4j-core` from 2.18.0 to 2.20.0 ([#6490](https://github.com/opensearch-project/OpenSearch/pull/6490))
@@ -35,6 +36,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `net.minidev:json-smart` from 2.4.7 to 2.4.10
 - Bump `org.apache.maven:maven-model` from 3.6.2 to 3.9.1
 - Bump `org.codehaus.jettison:jettison` from 1.5.3 to 1.5.4 ([#6878](https://github.com/opensearch-project/OpenSearch/pull/6878))
+- Add `com.github.luben:zstd-jni:1.5.4-1` ([#3577](https://github.com/opensearch-project/OpenSearch/pull/3577))
 
 ### Changed
 - Require MediaType in Strings.toString API ([#6009](https://github.com/opensearch-project/OpenSearch/pull/6009))

sandbox/modules/custom-codecs/build.gradle

@@ -0,0 +1,28 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ *
+ * Modifications Copyright OpenSearch Contributors. See
+ * GitHub history for details.
+ */
+
+apply plugin: 'opensearch.opensearchplugin'
+apply plugin: 'opensearch.yaml-rest-test'
+
+opensearchplugin {
+  name 'custom-codecs'
+  description 'A plugin that implements custom compression codecs.'
+  classname 'org.opensearch.index.codec.customcodecs.CustomCodecPlugin'
+  licenseFile rootProject.file('licenses/APACHE-LICENSE-2.0.txt')
+  noticeFile rootProject.file('NOTICE.txt')
+}
+
+dependencies {
+  api "com.github.luben:zstd-jni:1.5.4-1"
+}
+
+yamlRestTest.enabled = false;
+testingConventions.enabled = false;

sandbox/modules/custom-codecs/licenses/zstd-jni-1.5.4-1.jar.sha1

@@ -0,0 +1 @@
+291ccaacc039e41932de877303edb6af98a91c24

sandbox/modules/custom-codecs/licenses/zstd-jni-LICENSE.txt

@@ -0,0 +1,29 @@
+-----------------------------------------------------------------------------
+** Beginning of "BSD License" text. **
+
+Zstd-jni: JNI bindings to Zstd Library
+
+Copyright (c) 2015-present, Luben Karavelov/ All rights reserved.
+
+BSD License
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this
+  list of conditions and the following disclaimer in the documentation and/or
+  other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

sandbox/modules/custom-codecs/licenses/zstd-jni-NOTICE.txt

@@ -0,0 +1 @@
+The code for the JNI bindings to Zstd library was originally authored by Luben Karavelov

sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/CustomCodecPlugin.java

@@ -0,0 +1,40 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.index.codec.customcodecs;
+
+import org.opensearch.plugins.Plugin;
+import org.opensearch.plugins.EnginePlugin;
+import org.opensearch.index.codec.CodecServiceFactory;
+import org.opensearch.index.IndexSettings;
+
+import java.util.Optional;
+
+/**
+ * A plugin that implements custom codecs. Supports these codecs:
+ * <ul>
+ * <li>ZSTD
+ * <li>ZSTDNODICT
+ * </ul>
+ *
+ * @opensearch.internal
+ */
+public final class CustomCodecPlugin extends Plugin implements EnginePlugin {
+
+    /** Creates a new instance */
+    public CustomCodecPlugin() {}
+
+    /**
+     * @param indexSettings is the default indexSettings
+     * @return the engine factory
+     */
+    @Override
+    public Optional<CodecServiceFactory> getCustomCodecServiceFactory(final IndexSettings indexSettings) {
+        return Optional.of(new CustomCodecServiceFactory());
+    }
+}

sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/CustomCodecService.java

@@ -0,0 +1,67 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.index.codec.customcodecs;
+
+import org.apache.logging.log4j.Logger;
+import org.apache.lucene.codecs.Codec;
+import org.opensearch.common.collect.MapBuilder;
+import org.opensearch.index.codec.CodecService;
+import org.opensearch.index.mapper.MapperService;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Map;
+
+/**
+ * CustomCodecService provides ZSTD and ZSTDNODICT compression codecs.
+ */
+public class CustomCodecService extends CodecService {
+    private final Map<String, Codec> codecs;
+
+    /**
+     * Creates a new CustomCodecService.
+     *
+     * @param mapperService A mapper service.
+     * @param logger A logger.
+     */
+    public CustomCodecService(MapperService mapperService, Logger logger) {
+        super(mapperService, logger);
+        final MapBuilder<String, Codec> codecs = MapBuilder.<String, Codec>newMapBuilder();
+        if (mapperService == null) {
+            codecs.put(Lucene95CustomCodec.Mode.ZSTD.name(), new ZstdCodec());
+            codecs.put(Lucene95CustomCodec.Mode.ZSTDNODICT.name(), new ZstdNoDictCodec());
+        } else {
+            codecs.put(
+                Lucene95CustomCodec.Mode.ZSTD.name(),
+                new PerFieldMappingPostingFormatCodec(Lucene95CustomCodec.Mode.ZSTD, mapperService)
+            );
+            codecs.put(
+                Lucene95CustomCodec.Mode.ZSTDNODICT.name(),
+                new PerFieldMappingPostingFormatCodec(Lucene95CustomCodec.Mode.ZSTDNODICT, mapperService)
+            );
+        }
+        this.codecs = codecs.immutableMap();
+    }
+
+    @Override
+    public Codec codec(String name) {
+        Codec codec = codecs.get(name);
+        if (codec == null) {
+            return super.codec(name);
+        }
+        return codec;
+    }
+
+    @Override
+    public String[] availableCodecs() {
+        ArrayList<String> ac = new ArrayList<String>(Arrays.asList(super.availableCodecs()));
+        ac.addAll(codecs.keySet());
+        return ac.toArray(new String[0]);
+    }
+}

sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/CustomCodecServiceFactory.java

@@ -0,0 +1,27 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.index.codec.customcodecs;
+
+import org.opensearch.index.codec.CodecService;
+import org.opensearch.index.codec.CodecServiceConfig;
+import org.opensearch.index.codec.CodecServiceFactory;
+
+/**
+ * A factory for creating new {@link CodecService} instance
+ */
+public class CustomCodecServiceFactory implements CodecServiceFactory {
+
+    /** Creates a new instance. */
+    public CustomCodecServiceFactory() {}
+
+    @Override
+    public CodecService createCodecService(CodecServiceConfig config) {
+        return new CustomCodecService(config.getMapperService(), config.getLogger());
+    }
+}

sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/Lucene95CustomCodec.java

@@ -0,0 +1,45 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.index.codec.customcodecs;
+
+import org.apache.lucene.codecs.StoredFieldsFormat;
+import org.apache.lucene.codecs.FilterCodec;
+import org.apache.lucene.codecs.lucene95.Lucene95Codec;
+
+abstract class Lucene95CustomCodec extends FilterCodec {
+    public static final int DEFAULT_COMPRESSION_LEVEL = 6;
+
+    /** Each mode represents a compression algorithm. */
+    public enum Mode {
+        ZSTD,
+        ZSTDNODICT
+    }
+
+    private final StoredFieldsFormat storedFieldsFormat;
+
+    /** new codec for a given compression algorithm and default compression level */
+    public Lucene95CustomCodec(Mode mode) {
+        this(mode, DEFAULT_COMPRESSION_LEVEL);
+    }
+
+    public Lucene95CustomCodec(Mode mode, int compressionLevel) {
+        super(mode.name(), new Lucene95Codec());
+        this.storedFieldsFormat = new Lucene95CustomStoredFieldsFormat(mode, compressionLevel);
+    }
+
+    @Override
+    public StoredFieldsFormat storedFieldsFormat() {
+        return storedFieldsFormat;
+    }
+
+    @Override
+    public String toString() {
+        return getClass().getSimpleName();
+    }
+}

sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/Lucene95CustomStoredFieldsFormat.java

@@ -0,0 +1,107 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.index.codec.customcodecs;
+
+import java.io.IOException;
+import java.util.Objects;
+import org.apache.lucene.codecs.StoredFieldsFormat;
+import org.apache.lucene.codecs.StoredFieldsReader;
+import org.apache.lucene.codecs.StoredFieldsWriter;
+import org.apache.lucene.codecs.compressing.CompressionMode;
+import org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat;
+import org.apache.lucene.index.FieldInfos;
+import org.apache.lucene.index.SegmentInfo;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.IOContext;
+
+/** Stored field format used by pluggable codec */
+public class Lucene95CustomStoredFieldsFormat extends StoredFieldsFormat {
+
+    /** A key that we use to map to a mode */
+    public static final String MODE_KEY = Lucene95CustomStoredFieldsFormat.class.getSimpleName() + ".mode";
+
+    private static final int ZSTD_BLOCK_LENGTH = 10 * 48 * 1024;
+    private static final int ZSTD_MAX_DOCS_PER_BLOCK = 4096;
+    private static final int ZSTD_BLOCK_SHIFT = 10;
+
+    private final CompressionMode zstdCompressionMode;
+    private final CompressionMode zstdNoDictCompressionMode;
+
+    private final Lucene95CustomCodec.Mode mode;
+
+    /** default constructor */
+    public Lucene95CustomStoredFieldsFormat() {
+        this(Lucene95CustomCodec.Mode.ZSTD, Lucene95CustomCodec.DEFAULT_COMPRESSION_LEVEL);
+    }
+
+    /**
+     * Creates a new instance.
+     *
+     * @param mode The mode represents ZSTD or ZSTDNODICT
+     */
+    public Lucene95CustomStoredFieldsFormat(Lucene95CustomCodec.Mode mode) {
+        this(mode, Lucene95CustomCodec.DEFAULT_COMPRESSION_LEVEL);
+    }
+
+    /**
+     * Creates a new instance with the specified mode and compression level.
+     *
+     * @param mode The mode represents ZSTD or ZSTDNODICT
+     * @param compressionLevel The compression level for the mode.
+     */
+    public Lucene95CustomStoredFieldsFormat(Lucene95CustomCodec.Mode mode, int compressionLevel) {
+        this.mode = Objects.requireNonNull(mode);
+        zstdCompressionMode = new ZstdCompressionMode(compressionLevel);
+        zstdNoDictCompressionMode = new ZstdNoDictCompressionMode(compressionLevel);
+    }
+
+    @Override
+    public StoredFieldsReader fieldsReader(Directory directory, SegmentInfo si, FieldInfos fn, IOContext context) throws IOException {
+        String value = si.getAttribute(MODE_KEY);
+        if (value == null) {
+            throw new IllegalStateException("missing value for " + MODE_KEY + " for segment: " + si.name);
+        }
+        Lucene95CustomCodec.Mode mode = Lucene95CustomCodec.Mode.valueOf(value);
+        return impl(mode).fieldsReader(directory, si, fn, context);
+    }
+
+    @Override
+    public StoredFieldsWriter fieldsWriter(Directory directory, SegmentInfo si, IOContext context) throws IOException {
+        String previous = si.putAttribute(MODE_KEY, mode.name());
+        if (previous != null && previous.equals(mode.name()) == false) {
+            throw new IllegalStateException(
+                "found existing value for " + MODE_KEY + " for segment: " + si.name + " old = " + previous + ", new = " + mode.name()
+            );
+        }
+        return impl(mode).fieldsWriter(directory, si, context);
+    }
+
+    private StoredFieldsFormat impl(Lucene95CustomCodec.Mode mode) {
+        switch (mode) {
+            case ZSTD:
+                return new Lucene90CompressingStoredFieldsFormat(
+                    "CustomStoredFieldsZstd",
+                    zstdCompressionMode,
+                    ZSTD_BLOCK_LENGTH,
+                    ZSTD_MAX_DOCS_PER_BLOCK,
+                    ZSTD_BLOCK_SHIFT
+                );
+            case ZSTDNODICT:
+                return new Lucene90CompressingStoredFieldsFormat(
+                    "CustomStoredFieldsZstdNoDict",
+                    zstdNoDictCompressionMode,
+                    ZSTD_BLOCK_LENGTH,
+                    ZSTD_MAX_DOCS_PER_BLOCK,
+                    ZSTD_BLOCK_SHIFT
+                );
+            default:
+                throw new AssertionError();
+        }
+    }
+}

sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/PerFieldMappingPostingFormatCodec.java

@@ -0,0 +1,25 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.index.codec.customcodecs;
+
+import org.opensearch.index.mapper.MapperService;
+
+/** PerFieldMappingPostingFormatCodec. {@link org.opensearch.index.codec.PerFieldMappingPostingFormatCodec} */
+public class PerFieldMappingPostingFormatCodec extends Lucene95CustomCodec {
+
+    /**
+     * Creates a new instance.
+     *
+     * @param compressionMode The compression mode (ZSTD or ZSTDNODICT).
+     * @param mapperService The mapper service.
+     */
+    public PerFieldMappingPostingFormatCodec(Lucene95CustomCodec.Mode compressionMode, MapperService mapperService) {
+        super(compressionMode);
+    }
+}