Move mappers to separate files #448

martin-gaievski · 2022-07-14T17:30:06Z

Signed-off-by: Martin Gaievski gaievski@amazon.com

Description

This is in preparation for new LuceneField mapper to keep size of KNNVectorFieldMapper manageable and to structure project better.
Extract mapper classes from file with abstract class KNNVectorFieldMapper to separate files, created new "mapper" package for new classes.

Issues Resolved

building block for #39

Check List

All tests pass
Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Martin Gaievski <gaievski@amazon.com>

martin-gaievski · 2022-07-14T17:32:13Z

src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java

+        }
+    }
+
+    Optional<float[]> getFloatsFromContext(ParseContext context, int dimension) throws IOException {


This part will be common with Lucene field mapper, extract it to a method to avoid code duplication

martin-gaievski · 2022-07-14T17:32:30Z

src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java

-        if (!KNNSettings.isKNNPluginEnabled()) {
-            throw new IllegalStateException("KNN plugin is disabled. To enable " + "update knn.plugin.enabled setting to true");
+        validateIfKNNPluginEnabled();
+        validateIfCircuitBreakerTriggered();


This part will be common with Lucene field mapper, extract it to a method to avoid code duplication

codecov-commenter · 2022-07-14T17:45:49Z

Codecov Report

Merging #448 (efd487e) into main (7499375) will increase coverage by 0.03%.
The diff coverage is 86.90%.

@@             Coverage Diff              @@
##               main     #448      +/-   ##
============================================
+ Coverage     83.82%   83.86%   +0.03%     
- Complexity      933      948      +15     
============================================
  Files           133      136       +3     
  Lines          3828     3842      +14     
  Branches        349      349              
============================================
+ Hits           3209     3222      +13     
- Misses          457      458       +1     
  Partials        162      162

Impacted Files	Coverage Δ
.../main/java/org/opensearch/knn/index/IndexUtil.java	`55.73% <ø> (ø)`
...n/java/org/opensearch/knn/index/KNNIndexShard.java	`93.75% <ø> (ø)`
...java/org/opensearch/knn/index/KNNQueryBuilder.java	`67.96% <ø> (ø)`
...index/codec/KNN80Codec/KNN80DocValuesConsumer.java	`88.67% <ø> (ø)`
...java/org/opensearch/knn/indices/ModelMetadata.java	`97.08% <ø> (ø)`
...main/java/org/opensearch/knn/plugin/KNNPlugin.java	`100.00% <ø> (ø)`
.../opensearch/knn/plugin/script/KNNScoringSpace.java	`91.66% <ø> (ø)`
...nsearch/knn/plugin/script/KNNScoringSpaceUtil.java	`96.55% <ø> (ø)`
.../opensearch/knn/index/mapper/ModelFieldMapper.java	`71.42% <71.42%> (ø)`
...nsearch/knn/index/mapper/KNNVectorFieldMapper.java	`81.81% <79.16%> (ø)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7499375...efd487e. Read the comment docs.

VijayanB · 2022-07-14T17:49:28Z

src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java

-        if (!KNNSettings.isKNNPluginEnabled()) {
-            throw new IllegalStateException("KNN plugin is disabled. To enable " + "update knn.plugin.enabled setting to true");
+        validateIfKNNPluginEnabled();
+        validateIfCircuitBreakerTriggered();


nit: May be validateIfCircuitBreakerIsNotTriggered??

VijayanB · 2022-07-14T17:52:26Z

src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java

@@ -69,7 +61,7 @@
 */
 public abstract class KNNVectorFieldMapper extends ParametrizedFieldMapper {

-    private static Logger logger = LogManager.getLogger(KNNVectorFieldMapper.class);
+    static Logger logger = LogManager.getLogger(KNNVectorFieldMapper.class);


realized that lombok logger is only private. we used that old logger in one of the sub-classes, I guess that's ok to create it's own logger specifically for that class. It will change a log file content a bit, as I think it places class name in the line prefix.

make it protected rather than package private, why not create the logger in the class which is using it rather than creating it in the parent class.

VijayanB · 2022-07-14T17:53:46Z

src/test/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapperTests.java

+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ *
+ * Modifications Copyright OpenSearch Contributors. See
+ * GitHub history for details.


Suggested change

*

* The OpenSearch Contributors require contributions made to

* this file be licensed under the Apache-2.0 license or a

* compatible open source license.

*

* Modifications Copyright OpenSearch Contributors. See

* GitHub history for details.

navneet1v · 2022-07-14T17:52:13Z

src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java

@@ -69,7 +61,7 @@
 */
 public abstract class KNNVectorFieldMapper extends ParametrizedFieldMapper {

-    private static Logger logger = LogManager.getLogger(KNNVectorFieldMapper.class);
+    static Logger logger = LogManager.getLogger(KNNVectorFieldMapper.class);


why not private?

I was thinking or re-using same logger in sub-class to keep content of the log file same, as it puts the class name there.
But I on second though it should be a problem as long as we keep the content same.
Let me update, I also will use lombok for logger initialization

I would suggest use different loggers for different classes specially for parent and child as sometimes it become difficult to find the right code path.

navneet1v · 2022-07-14T17:53:34Z

src/main/java/org/opensearch/knn/index/mapper/ModelFieldMapper.java

+                "Model \""
+                    + modelId
+                    + "\" from "
+                    + context.mapperService().index().getName()
+                    + "'s mapping does not exist. Because the "
+                    + "\""
+                    + MODEL_ID
+                    + "\" parameter is not updateable, this index will need to "
+                    + "be recreated with a valid model."


We can use String.format here, rather than "+" operator.

…license header Signed-off-by: Martin Gaievski <gaievski@amazon.com>

jmazanec15 · 2022-07-14T19:59:11Z

src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java


+    void validateIfKNNPluginEnabled() {
+        if (!KNNSettings.isKNNPluginEnabled()) {
+            throw new IllegalStateException("KNN plugin is disabled. To enable " + "update knn.plugin.enabled setting to true");


nit: Remove + in middle?

jmazanec15 · 2022-07-14T20:03:39Z

src/main/java/org/opensearch/knn/index/mapper/LegacyFieldMapper.java

+import static org.opensearch.knn.common.KNNConstants.SPACE_TYPE;
+
+/**
+ * Field mapper for original implementation


minor: Do you think we should add what the mapping looks like so users have more context?

{ "type": "knn_vector", "dimension": 128 }

Also, add something like "This is the mapper for the original version. It defaults to using nmslib as the engine and retrieves parameters from index settings."

jmazanec15 · 2022-07-14T20:04:45Z

src/main/java/org/opensearch/knn/index/mapper/LegacyFieldMapper.java

+        String spaceType = indexSettings.get(KNNSettings.INDEX_KNN_SPACE_TYPE.getKey());
+        if (spaceType == null) {
+            log.info(
+                "[KNN] The setting \""


Should this use string builder?

yeah, I think build and/or String.format for params, ack

Signed-off-by: Martin Gaievski <gaievski@amazon.com>

* Move mappers to separate files Signed-off-by: Martin Gaievski <gaievski@amazon.com> (cherry picked from commit cfac544)

* Move mappers to separate files Signed-off-by: Martin Gaievski <gaievski@amazon.com>

Move mappers to separate files

b2d4d24

Signed-off-by: Martin Gaievski <gaievski@amazon.com>

martin-gaievski added Refactoring Improve the design, structure, and implementation while preserving its functionality 2.2.0 backport 2.x labels Jul 14, 2022

martin-gaievski requested a review from a team July 14, 2022 17:30

martin-gaievski self-assigned this Jul 14, 2022

martin-gaievski commented Jul 14, 2022

View reviewed changes

VijayanB reviewed Jul 14, 2022

View reviewed changes

navneet1v reviewed Jul 14, 2022

View reviewed changes

Review comments - lombok logger, added string.format, rename method, …

ead2b6d

…license header Signed-off-by: Martin Gaievski <gaievski@amazon.com>

martin-gaievski requested review from navneet1v and VijayanB July 14, 2022 18:35

VijayanB previously approved these changes Jul 14, 2022

View reviewed changes

jmazanec15 reviewed Jul 14, 2022

View reviewed changes

Adding String.format for exception error messages

efd487e

Signed-off-by: Martin Gaievski <gaievski@amazon.com>

martin-gaievski dismissed VijayanB’s stale review via efd487e July 14, 2022 20:43

martin-gaievski requested review from jmazanec15 and VijayanB July 14, 2022 20:53

jmazanec15 approved these changes Jul 14, 2022

View reviewed changes

VijayanB approved these changes Jul 14, 2022

View reviewed changes

martin-gaievski merged commit cfac544 into opensearch-project:main Jul 14, 2022

opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 14, 2022

Move mappers to separate files (#448)

09704f4

* Move mappers to separate files Signed-off-by: Martin Gaievski <gaievski@amazon.com> (cherry picked from commit cfac544)

opensearch-trigger-bot bot mentioned this pull request Jul 14, 2022

[Backport 2.x] Move mappers to separate files #450

Merged

martin-gaievski pushed a commit that referenced this pull request Jul 21, 2022

Move mappers to separate files (#448) (#450)

9b06925

* Move mappers to separate files Signed-off-by: Martin Gaievski <gaievski@amazon.com>

heemin32 added v2.2.0 and removed 2.2.0 labels Nov 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move mappers to separate files #448

Move mappers to separate files #448

martin-gaievski commented Jul 14, 2022

martin-gaievski Jul 14, 2022

martin-gaievski Jul 14, 2022

codecov-commenter commented Jul 14, 2022 •

edited

Loading

VijayanB Jul 14, 2022

martin-gaievski Jul 14, 2022

VijayanB Jul 14, 2022

martin-gaievski Jul 14, 2022

martin-gaievski Jul 14, 2022

navneet1v Jul 14, 2022

VijayanB Jul 14, 2022

martin-gaievski Jul 14, 2022

navneet1v Jul 14, 2022

martin-gaievski Jul 14, 2022 •

edited

Loading

navneet1v Jul 14, 2022

navneet1v Jul 14, 2022

martin-gaievski Jul 14, 2022

jmazanec15 Jul 14, 2022

martin-gaievski Jul 14, 2022

jmazanec15 Jul 14, 2022

martin-gaievski Jul 14, 2022

jmazanec15 Jul 14, 2022

martin-gaievski Jul 14, 2022

Move mappers to separate files #448

Move mappers to separate files #448

Conversation

martin-gaievski commented Jul 14, 2022

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jul 14, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martin-gaievski Jul 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jul 14, 2022 •

edited

Loading

martin-gaievski Jul 14, 2022 •

edited

Loading