Enable multiQuery optimization for PropertyMapStep and ElementMapStep

Adds possibility to fetch properties and labels of vertices using valueMap, elementMap, propertyMap steps.

Adds fetching modes to properties, values, valueMap, elementMap, propertyMap steps to be able to preFetch all properties (single slice query) or only required properties (separate slice query per each requested property).

Fixes JanusGraph#2444

Signed-off-by: Oleksandr Porunov <alexandr.porunov@gmail.com>

docs/configs/janusgraph-cfg.md

@@ -362,7 +362,7 @@ Configuration options to configure batch queries optimization behavior
 | Name | Description | Datatype | Default Value | Mutability |
 | ---- | ---- | ---- | ---- | ---- |
 | query.batch.enabled | Whether traversal queries should be batched when executed against the storage backend. This can lead to significant performance improvement if there is a non-trivial latency to the backend. If `false` then all other configuration options under `query.batch` namespace are ignored. | Boolean | true | MASKABLE |
-| query.batch.has-step-mode | Properties pre-fetching mode for `has` step. Used only when query.batch.enabled is `true`.<br>Supported modes:<br>- `all_properties` Pre-fetch all vertex properties on any property access<br>- `required_properties_only` Pre-fetch necessary vertex properties for the whole chain of foldable `has` steps<br>- `required_and_next_properties` Prefetch the same properties as with `required_properties_only` mode, but also prefetch
+| query.batch.has-step-mode | Properties pre-fetching mode for `has` step. Used only when query.batch.enabled is `true`.<br>Supported modes:<br>- `all_properties` Pre-fetch all vertex properties on any property access (fetches all vertex properties in a single slice query)<br>- `required_properties_only` Pre-fetch necessary vertex properties for the whole chain of foldable `has` steps (uses a separate slice query per each required property)<br>- `required_and_next_properties` Prefetch the same properties as with `required_properties_only` mode, but also prefetch
 properties which may be needed in the next properties access step like `values`, `properties,` `valueMap`, `elementMap`, or `propertyMap`.
 In case the next step is not one of those properties access steps then this mode behaves same as `required_properties_only`.
 In case the next step is one of the properties access steps with limited scope of properties, those properties will be
@@ -372,6 +372,7 @@ behaves same as `all_properties`.<br>- `required_and_next_properties_or_all` Pre
 `values`, `properties,` `valueMap`, `elementMap`, or `propertyMap` then acts like `all_properties`.<br>- `none` Skips `has` step batch properties pre-fetch optimization.<br> | String | required_and_next_properties | MASKABLE |
 | query.batch.limited | Configure a maximum batch size for queries against the storage backend. This can be used to ensure responsiveness if batches tend to grow very large. The used batch size is equivalent to the barrier size of a preceding `barrier()` step. If a step has no preceding `barrier()`, the default barrier of TinkerPop will be inserted. This option only takes effect if `query.batch.enabled` is `true`. | Boolean | true | MASKABLE |
 | query.batch.limited-size | Default batch size (barrier() step size) for queries. This size is applied only for cases where `LazyBarrierStrategy` strategy didn't apply `barrier` step and where user didn't apply barrier step either. This option is used only when `query.batch.limited` is `true`. Notice, value `2147483647` is considered to be unlimited. | Integer | 2500 | MASKABLE |
+| query.batch.properties-mode | Properties pre-fetching mode for `values`, `properties`, `valueMap`, `propertyMap`, `elementMap` steps. Used only when query.batch.enabled is `true`.<br>Supported modes:<br>- `all_properties` Pre-fetch all vertex properties on any property access (fetches all vertex properties in a single slice query)<br>- `required_properties_only` Pre-fetch necessary vertex properties only (uses a separate slice query per each required property)<br>- `none` Skips vertex properties pre-fetching optimization.<br> | String | required_and_next_properties | MASKABLE |
 | query.batch.repeat-step-mode | Batch mode for `repeat` step. Used only when query.batch.enabled is `true`.<br>These modes are controlling how the child steps with batch support are behaving if they placed to the start of the `repeat`, `emit`, or `until` traversals.<br>Supported modes:<br>- `closest_repeat_parent` Child start steps are receiving vertices for batching from the closest `repeat` step parent only.<br>- `all_repeat_parents` Child start steps are receiving vertices for batching from all `repeat` step parents.<br>- `starts_only_of_all_repeat_parents` Child start steps are receiving vertices for batching from the closest `repeat` step parent (both for the parent start and for next iterations) and also from all `repeat` step parents for the parent start. | String | all_repeat_parents | MASKABLE |
 
 ### schema

janusgraph-core/src/main/java/org/janusgraph/core/TransactionBuilder.java

@@ -16,6 +16,7 @@
 
 import org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration;
 import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryHasStepStrategyMode;
+import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryPropertiesStrategyMode;
 
 import java.time.Instant;
 
@@ -68,7 +69,7 @@ public interface TransactionBuilder {
     TransactionBuilder propertyPrefetching(boolean enabled);
 
     /**
-     * Enable or disable multi-query, i.e. query.batch
+     * Enable or disable multi-query, i.e. `query.batch.enabled`
      *
      * @param enabled
      * @return Object containing properties that will enable/disable multi-query
@@ -151,12 +152,21 @@ public interface TransactionBuilder {
     /**
      * Sets `has` step strategy mode.
      * <p>
-     * Doesn't have any effect if multi-query was disabled via config `query.batch`.
+     * Doesn't have any effect if multi-query was disabled via config `query.batch.enabled = false`.
      *
      * @return Object with the set `has` step strategy mode settings
      */
     TransactionBuilder setHasStepStrategyMode(MultiQueryHasStepStrategyMode hasStepStrategyMode);
 
+    /**
+     * Sets properties strategy mode.
+     * <p>
+     * Doesn't have any effect if multi-query was disabled via config `query.batch.enabled = false`.
+     *
+     * @return Object with the set properties strategy mode settings
+     */
+    TransactionBuilder setPropertiesStrategyMode(MultiQueryPropertiesStrategyMode propertiesStrategyMode);
+
     /**
      * Sets the group name for this transaction which provides a way for gathering
      * reporting on multiple transactions into one group.

janusgraph-core/src/main/java/org/janusgraph/graphdb/configuration/GraphDatabaseConfiguration.java

@@ -58,6 +58,7 @@
 import org.janusgraph.graphdb.query.index.BruteForceIndexSelectionStrategy;
 import org.janusgraph.graphdb.query.index.IndexSelectionStrategy;
 import org.janusgraph.graphdb.query.index.ThresholdBasedIndexSelectionStrategy;
+import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryPropertiesStrategyMode;
 import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryStrategyRepeatStepMode;
 import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryHasStepStrategyMode;
 import org.janusgraph.graphdb.transaction.StandardTransactionBuilder;
@@ -331,8 +332,8 @@ public class GraphDatabaseConfiguration {
     public static final ConfigOption<String> HAS_STEP_BATCH_MODE = new ConfigOption<>(QUERY_BATCH_NS,"has-step-mode",
         String.format("Properties pre-fetching mode for `has` step. Used only when "+USE_MULTIQUERY.toStringWithoutRoot()+" is `true`.<br>" +
                 "Supported modes:<br>" +
-                "- `%s` Pre-fetch all vertex properties on any property access<br>" +
-                "- `%s` Pre-fetch necessary vertex properties for the whole chain of foldable `has` steps<br>" +
+                "- `%s` Pre-fetch all vertex properties on any property access (fetches all vertex properties in a single slice query)<br>" +
+                "- `%s` Pre-fetch necessary vertex properties for the whole chain of foldable `has` steps (uses a separate slice query per each required property)<br>" +
                 "- `%s` Prefetch the same properties as with `%s` mode, but also prefetch\n" +
                 "properties which may be needed in the next properties access step like `values`, `properties,` `valueMap`, `elementMap`, or `propertyMap`.\n" +
                 "In case the next step is not one of those properties access steps then this mode behaves same as `%s`.\n" +
@@ -355,6 +356,17 @@ public class GraphDatabaseConfiguration {
             MultiQueryHasStepStrategyMode.NONE.getConfigName()),
         ConfigOption.Type.MASKABLE, MultiQueryHasStepStrategyMode.REQUIRED_AND_NEXT_PROPERTIES.getConfigName());
 
+    public static final ConfigOption<String> PROPERTIES_BATCH_MODE = new ConfigOption<>(QUERY_BATCH_NS,"properties-mode",
+        String.format("Properties pre-fetching mode for `values`, `properties`, `valueMap`, `propertyMap`, `elementMap` steps. Used only when "+USE_MULTIQUERY.toStringWithoutRoot()+" is `true`.<br>" +
+                "Supported modes:<br>" +
+                "- `%s` Pre-fetch all vertex properties on any property access (fetches all vertex properties in a single slice query)<br>" +
+                "- `%s` Pre-fetch necessary vertex properties only (uses a separate slice query per each required property)<br>" +
+                "- `%s` Skips vertex properties pre-fetching optimization.<br>",
+            MultiQueryPropertiesStrategyMode.ALL_PROPERTIES.getConfigName(),
+            MultiQueryPropertiesStrategyMode.REQUIRED_PROPERTIES_ONLY.getConfigName(),
+            MultiQueryPropertiesStrategyMode.NONE.getConfigName()),
+        ConfigOption.Type.MASKABLE, MultiQueryHasStepStrategyMode.REQUIRED_AND_NEXT_PROPERTIES.getConfigName());
+
     // ################ SCHEMA #######################
     // ################################################
 
@@ -1326,6 +1338,7 @@ public boolean apply(@Nullable String s) {
     private String metricsPrefix;
     private String unknownIndexKeyName;
     private MultiQueryHasStepStrategyMode hasStepStrategyMode;
+    private MultiQueryPropertiesStrategyMode propertiesStrategyMode;
 
     private StoreFeatures storeFeatures = null;
 
@@ -1444,6 +1457,10 @@ public MultiQueryHasStepStrategyMode hasStepStrategyMode() {
         return hasStepStrategyMode;
     }
 
+    public MultiQueryPropertiesStrategyMode propertiesStrategyMode() {
+        return propertiesStrategyMode;
+    }
+
     public boolean adjustQueryLimit() {
         return adjustQueryLimit;
     }
@@ -1572,6 +1589,7 @@ private void preLoadConfiguration() {
         limitedBatchSize = configuration.get(LIMITED_BATCH_SIZE);
         repeatStepMode = selectExactConfig(REPEAT_STEP_BATCH_MODE, MultiQueryStrategyRepeatStepMode.values());
         hasStepStrategyMode = selectExactConfig(HAS_STEP_BATCH_MODE, MultiQueryHasStepStrategyMode.values());
+        propertiesStrategyMode = selectExactConfig(PROPERTIES_BATCH_MODE, MultiQueryPropertiesStrategyMode.values());
 
         indexSelectionStrategy = Backend.getImplementationClass(configuration, configuration.get(INDEX_SELECT_STRATEGY),
             REGISTERED_INDEX_SELECTION_STRATEGIES);

janusgraph-core/src/main/java/org/janusgraph/graphdb/database/StandardJanusGraph.java

@@ -65,8 +65,8 @@
 import org.janusgraph.graphdb.database.cache.SchemaCache;
 import org.janusgraph.graphdb.database.idassigner.VertexIDAssigner;
 import org.janusgraph.graphdb.database.idhandling.IDHandler;
+import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphMultiQueriableReplacementStrategy;
 import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphUnusedMultiQueryRemovalStrategy;
-import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphHasStepStrategy;
 import org.janusgraph.util.IDUtils;
 import org.janusgraph.graphdb.database.index.IndexInfoRetriever;
 import org.janusgraph.graphdb.database.index.IndexUpdate;
@@ -90,7 +90,6 @@
 import org.janusgraph.graphdb.tinkerpop.optimize.strategy.AdjacentVertexHasUniquePropertyOptimizerStrategy;
 import org.janusgraph.graphdb.tinkerpop.optimize.strategy.AdjacentVertexIsOptimizerStrategy;
 import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphIoRegistrationStrategy;
-import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphLocalQueryOptimizerStrategy;
 import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphMixedIndexAggStrategy;
 import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphMixedIndexCountStrategy;
 import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphMultiQueryStrategy;
@@ -146,13 +145,12 @@ public class StandardJanusGraph extends JanusGraphBlueprintsGraph {
                                AdjacentVertexHasIdOptimizerStrategy.instance(),
                                AdjacentVertexIsOptimizerStrategy.instance(),
                                AdjacentVertexHasUniquePropertyOptimizerStrategy.instance(),
-                               JanusGraphLocalQueryOptimizerStrategy.instance(),
+                               JanusGraphMultiQueriableReplacementStrategy.instance(),
                                JanusGraphMultiQueryStrategy.instance(),
                                JanusGraphUnusedMultiQueryRemovalStrategy.instance(),
                                JanusGraphMixedIndexAggStrategy.instance(),
                                JanusGraphMixedIndexCountStrategy.instance(),
                                JanusGraphStepStrategy.instance(),
-                               JanusGraphHasStepStrategy.instance(),
                                JanusGraphIoRegistrationStrategy.instance());
 
         //Register with cache

janusgraph-core/src/main/java/org/janusgraph/graphdb/olap/computer/FulgoraUtil.java

@@ -31,7 +31,7 @@
 import org.janusgraph.core.JanusGraphTransaction;
 import org.janusgraph.graphdb.tinkerpop.optimize.JanusGraphTraversalUtil;
 import org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphVertexStep;
-import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphLocalQueryOptimizerStrategy;
+import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphMultiQueriableReplacementStrategy;
 
 import java.util.List;
 import java.util.Optional;
@@ -42,7 +42,7 @@
  */
 public class FulgoraUtil {
 
-    private static final TraversalStrategies FULGORA_STRATEGIES = TraversalStrategies.GlobalCache.getStrategies(Graph.class).clone().addStrategies(JanusGraphLocalQueryOptimizerStrategy.instance());
+    private static final TraversalStrategies FULGORA_STRATEGIES = TraversalStrategies.GlobalCache.getStrategies(Graph.class).clone().addStrategies(JanusGraphMultiQueriableReplacementStrategy.instance());
 
     public static JanusGraphVertexStep<Vertex> getReverseJanusGraphVertexStep(final MessageScope.Local<?> scope,
                                                                        final JanusGraphTransaction graph) {