Remove mapping types #155 #303

dreamer-89 · 2022-03-02T18:48:28Z

As part of v2.0 release, mapping types are getting removed from OpenSearch engine. Below are the changes in the opensearch-engine.

Removal of type from rest end-points
Removal of include_type_name parameter from API requests.

As part of this issue, please verify if type removal change on OpenSearch engine impacts this repository. If yes, then please remove the type references/usage from this plugin.

Top level changes are captured in gist below:
https://gist.github.com/dreamer-89/d76eaf639171e8ab32fa7f8b9d6c93d3

For more detailed changes, please check meta issue below:
Related: OpenSearch-engine meta issue

The text was updated successfully, but these errors were encountered:

peterzhuamazon · 2022-03-28T17:40:42Z

@martin-gaievski I see these errors in knn build on 2.0.0-alpha1

logs:

> Task :compileJava
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/codec/KNN80Codec/KNN80CompoundFormat.java:8: error: package org.apache.lucene.codecs.lucene50 does not exist
import org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat;
                                        ^
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/codec/KNN80Codec/KNN80DocValuesFormat.java:11: error: package org.apache.lucene.codecs.lucene80 does not exist
import org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat;
                                        ^
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/codec/KNN87Codec/KNN87Codec.java:9: error: package org.apache.lucene.codecs.lucene87 does not exist
import org.apache.lucene.codecs.lucene87.Lucene87Codec;
                                        ^
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/indices/ModelDao.java:203: error: no suitable method found for mapping(String,String,XContentType)
            CreateIndexRequest request = new CreateIndexRequest(MODEL_INDEX_NAME).mapping("_doc", getMapping(), XContentType.JSON)
                                                                                 ^
    method CreateIndexRequest.mapping(String) is not applicable
      (actual and formal argument lists differ in length)
    method CreateIndexRequest.mapping(BytesReference,XContentType) is not applicable
      (actual and formal argument lists differ in length)
    method CreateIndexRequest.mapping(XContentBuilder) is not applicable
      (actual and formal argument lists differ in length)
    method CreateIndexRequest.mapping(Map<String,?>) is not applicable
      (actual and formal argument lists differ in length)
    method CreateIndexRequest.mapping(String,Map<String,?>) is not applicable
      (actual and formal argument lists differ in length)
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/KNNQuery.java:18: error: KNNQuery is not abstract and does not override abstract method visit(QueryVisitor) in Query
public class KNNQuery extends Query {
       ^
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/KNNWeight.java:80: error: method does not override or implement a method from a supertype
    @Override
    ^
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/codec/KNN80Codec/KNN80Codec.java:28: error: KNN80Codec is not abstract and does not override abstract method knnVectorsFormat() in Codec
public final class KNN80Codec extends Codec {
             ^
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/codec/KNN80Codec/KNN80CompoundFormat.java:30: error: cannot find symbol
        this.delegate = new Lucene50CompoundFormat();
                            ^
  symbol:   class Lucene50CompoundFormat
  location: class KNN80CompoundFormat
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/codec/KNN80Codec/KNN80DocValuesFormat.java:24: error: cannot find symbol
        this(new Lucene80DocValuesFormat());
                 ^
  symbol:   class Lucene80DocValuesFormat
  location: class KNN80DocValuesFormat
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/codec/KNN84Codec/KNN84Codec.java:30: error: KNN84Codec is not abstract and does not override abstract method knnVectorsFormat() in Codec
public final class KNN84Codec extends Codec {
             ^
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/codec/KNN86Codec/KNN86Codec.java:30: error: KNN86Codec is not abstract and does not override abstract method knnVectorsFormat() in Codec
public final class KNN86Codec extends Codec {
             ^
/tmp/tmpadcckeau/k-NN/src/main/java/org/opensearch/knn/index/codec/KNN87Codec/KNN87Codec.java:32: error: cannot find symbol
        this(new Lucene87Codec());
                 ^
  symbol:   class Lucene87Codec
  location: class KNN87Codec
12 errors

> Task :compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org/

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 15s
2 actionable tasks: 2 executed
2022-03-27 07:35:20 ERROR    Error building k-NN, retry with: ./build.sh manifests/2.0.0/opensearch-2.0.0.yml --component k-NN --snapshot
Traceback (most recent call last):
  File "./src/run_build.py", line 80, in <module>
    sys.exit(main())
  File "./src/run_build.py", line 68, in main
    builder.build(build_recorder)
  File "/var/jenkins/workspace/distribution-build-opensearch/src/build_workflow/builder_from_source.py", line 55, in build
    self.git_repo.execute(build_command)
  File "/var/jenkins/workspace/distribution-build-opensearch/src/git/git_repository.py", line 83, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /tmp/tmpadcckeau/k-NN/scripts/build.sh -v 2.0.0 -q alpha1 -p linux -a arm64 -s true -o builds' returned non-zero exit status 1.

Are they related to this PR since I also see mapping.

Thanks.

martin-gaievski · 2022-03-28T17:49:51Z

@martin-gaievski I see these errors in knn build on 2.0.0-alpha1

logs:
Are they related to this PR since I also see mapping.

Thanks.

Build on 2.0 alpha1 snapshot was old on Friday, there is a PR specifically aiming integration ##328. Maybe it's something in latest snapshot, I'm restarting build jobs to see what the errors are (if any)

peterzhuamazon · 2022-03-28T17:53:11Z

@martin-gaievski I see these errors in knn build on 2.0.0-alpha1
logs:
Are they related to this PR since I also see mapping.
Thanks.

Build on 2.0 alpha1 snapshot was old on Friday, there is a PR specifically aiming integration ##328. Maybe it's something in latest snapshot, I'm restarting build jobs to see what the errors are (if any)

We built on latest main on Friday to consume the changes from reta on fixing opensearch-project/OpenSearch#2581

martin-gaievski · 2022-03-28T18:24:16Z

@peterzhuamazon as per stacktrace knn is failing due to a missing references to a lucene 8.x classes. This is changed in lucene 9.1, that supposed to be part of 2.0 alpha1 snapshot. What is the knn branch that is failing, could you please share the link to build job?

bbarani · 2022-03-28T18:31:15Z

@peterzhuamazon I have provided the necessary information to @martin-gaievski

peterzhuamazon · 2022-03-28T19:06:51Z

Hi @martin-gaievski , @dreamer-89 has talked to me and I have provided the commitid:
OpenSearch main f1d35d028ffc807303c75ba677f4c2cff9835041

martin-gaievski · 2022-03-28T19:13:18Z

@peterzhuamazon I think with latest PR to knn previous issue has been fixed. I see from the latest build log that problem is with anomaly detection now:

`BUILD SUCCESSFUL in 54s
10 actionable tasks: 10 executed
++ find . -path 'build/distributions/.zip'

zipPath=./build/distributions/opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip
++ dirname ./build/distributions/opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip
distributions=./build/distributions
echo 'COPY ./build/distributions/.zip'
COPY ./build/distributions/.zip
mkdir -p builds/plugins
cp ./build/distributions/opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip ./builds/plugins
2022-03-28 19:00:37 INFO Recording plugins artifact for anomaly-detection: plugins/opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip (from /tmp/tmps75s70s9/anomaly-detection/builds/plugins/opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip)
2022-03-28 19:00:37 ERROR Error building anomaly-detection, retry with: ./build.sh manifests/2.0.0/opensearch-2.0.0.yml --component anomaly-detection --snapshot
Traceback (most recent call last):
File "./src/run_build.py", line 80, in
sys.exit(main())
File "./src/run_build.py", line 69, in main
builder.export_artifacts(build_recorder)
File "/var/jenkins/workspace/distribution-build-opensearch@2/src/build_workflow/builder_from_source.py", line 65, in export_artifacts
build_recorder.record_artifact(self.component.name, artifact_type, relative_path, absolute_path)
File "/var/jenkins/workspace/distribution-build-opensearch@2/src/build_workflow/build_recorder.py", line 40, in record_artifact
BuildArtifactChecks.check(self.target, artifact_type, artifact_file)
File "/var/jenkins/workspace/distribution-build-opensearch@2/src/build_workflow/build_artifact_checks.py", line 42, in check
instance.check(path)
File "/var/jenkins/workspace/distribution-build-opensearch@2/src/build_workflow/opensearch/build_artifact_check_plugin.py", line 20, in check
raise BuildArtifactCheck.BuildArtifactInvalidError(path, f"Expected filename to include one of {self.target.compatible_component_versions}.")
build_workflow.build_artifact_check.BuildArtifactInvalidError: Artifact opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip is invalid. Expected filename to include one of ['2.0.0.0-alpha1-SNAPSHOT'].`

peterzhuamazon · 2022-03-28T19:14:20Z

@peterzhuamazon I think with latest PR to knn previous issue has been fixed. I see from the latest build log that problem is with anomaly detection now:

`BUILD SUCCESSFUL in 54s 10 actionable tasks: 10 executed ++ find . -path 'build/distributions/.zip'

* zipPath=./build/distributions/opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip
  ++ dirname ./build/distributions/opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip

* distributions=./build/distributions

* echo 'COPY ./build/distributions/_.zip'
  COPY ./build/distributions/_.zip

* mkdir -p builds/plugins

* cp ./build/distributions/opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip ./builds/plugins
  2022-03-28 19:00:37 INFO     Recording plugins artifact for anomaly-detection: plugins/opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip (from /tmp/tmps75s70s9/anomaly-detection/builds/plugins/opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip)
  2022-03-28 19:00:37 ERROR    Error building anomaly-detection, retry with: ./build.sh manifests/2.0.0/opensearch-2.0.0.yml --component anomaly-detection --snapshot
  Traceback (most recent call last):
  File "./src/run_build.py", line 80, in 
  sys.exit(main())
  File "./src/run_build.py", line 69, in main
  builder.export_artifacts(build_recorder)
  File "/var/jenkins/workspace/distribution-build-opensearch@2/src/build_workflow/builder_from_source.py", line 65, in export_artifacts
  build_recorder.record_artifact(self.component.name, artifact_type, relative_path, absolute_path)
  File "/var/jenkins/workspace/distribution-build-opensearch@2/src/build_workflow/build_recorder.py", line 40, in record_artifact
  BuildArtifactChecks.check(self.target, artifact_type, artifact_file)
  File "/var/jenkins/workspace/distribution-build-opensearch@2/src/build_workflow/build_artifact_checks.py", line 42, in check
  instance.check(path)
  File "/var/jenkins/workspace/distribution-build-opensearch@2/src/build_workflow/opensearch/build_artifact_check_plugin.py", line 20, in check
  raise BuildArtifactCheck.BuildArtifactInvalidError(path, f"Expected filename to include one of {self.target.compatible_component_versions}.")
  build_workflow.build_artifact_check.BuildArtifactInvalidError: Artifact opensearch-anomaly-detection-2.0.0.0-SNAPSHOT.zip is invalid. Expected filename to include one of ['2.0.0.0-alpha1-SNAPSHOT'].`

Thanks @martin-gaievski will keep this issue open until I confirm the AD issue.
Thanks.

dreamer-89 added enhancement Breaking Changes Changes in one part of a software system that causes other parts to fail v2.0.0 labels Mar 2, 2022

This was referenced Mar 9, 2022

Remove mappings types opensearch-project/opensearch-plugins#130

Closed

[Meta] Remove mapping types opensearch-project/OpenSearch#1940

Closed

vamshin assigned jmazanec15 Mar 11, 2022

jmazanec15 mentioned this issue Mar 21, 2022

Integration with base OpenSearch 2.0 #328

Merged

3 tasks

peterzhuamazon mentioned this issue Mar 28, 2022

[RELEASE] Release version 2.0.0 (RC1) opensearch-project/opensearch-build#1624

Closed

51 tasks

peterzhuamazon closed this as completed Mar 28, 2022

jmazanec15 mentioned this issue Mar 28, 2022

Remove remaining mapping type #335

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove mapping types #155 #303

Remove mapping types #155 #303

dreamer-89 commented Mar 2, 2022 •

edited

Loading

peterzhuamazon commented Mar 28, 2022

martin-gaievski commented Mar 28, 2022

peterzhuamazon commented Mar 28, 2022

martin-gaievski commented Mar 28, 2022

bbarani commented Mar 28, 2022

peterzhuamazon commented Mar 28, 2022 •

edited

Loading

martin-gaievski commented Mar 28, 2022

peterzhuamazon commented Mar 28, 2022

Remove mapping types #155 #303

Remove mapping types #155 #303

Comments

dreamer-89 commented Mar 2, 2022 • edited Loading

peterzhuamazon commented Mar 28, 2022

martin-gaievski commented Mar 28, 2022

peterzhuamazon commented Mar 28, 2022

martin-gaievski commented Mar 28, 2022

bbarani commented Mar 28, 2022

peterzhuamazon commented Mar 28, 2022 • edited Loading

martin-gaievski commented Mar 28, 2022

peterzhuamazon commented Mar 28, 2022

dreamer-89 commented Mar 2, 2022 •

edited

Loading

peterzhuamazon commented Mar 28, 2022 •

edited

Loading