Skip to content

Commit

Permalink
Enable multiQuery optimization for PropertyMapStep and ElementMapStep
Browse files Browse the repository at this point in the history
Adds possibility to fetch properties and labels of vertices using valueMap, elementMap, propertyMap steps.

Adds fetching modes to properties, values, valueMap, elementMap, propertyMap steps to be able to preFetch all properties (single slice query) or only required properties (separate slice query per each requested property).

Fixes JanusGraph#2444

Signed-off-by: Oleksandr Porunov <alexandr.porunov@gmail.com>
  • Loading branch information
porunov committed Jun 8, 2023
1 parent dac1ea1 commit 0588749
Show file tree
Hide file tree
Showing 17 changed files with 1,159 additions and 72 deletions.
3 changes: 2 additions & 1 deletion docs/configs/janusgraph-cfg.md
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,7 @@ Configuration options to configure batch queries optimization behavior
| Name | Description | Datatype | Default Value | Mutability |
| ---- | ---- | ---- | ---- | ---- |
| query.batch.enabled | Whether traversal queries should be batched when executed against the storage backend. This can lead to significant performance improvement if there is a non-trivial latency to the backend. If `false` then all other configuration options under `query.batch` namespace are ignored. | Boolean | true | MASKABLE |
| query.batch.has-step-mode | Properties pre-fetching mode for `has` step. Used only when query.batch.enabled is `true`.<br>Supported modes:<br>- `all_properties` Pre-fetch all vertex properties on any property access<br>- `required_properties_only` Pre-fetch necessary vertex properties for the whole chain of foldable `has` steps<br>- `required_and_next_properties` Prefetch the same properties as with `required_properties_only` mode, but also prefetch
| query.batch.has-step-mode | Properties pre-fetching mode for `has` step. Used only when query.batch.enabled is `true`.<br>Supported modes:<br>- `all_properties` Pre-fetch all vertex properties on any property access (fetches all vertex properties in a single slice query)<br>- `required_properties_only` Pre-fetch necessary vertex properties for the whole chain of foldable `has` steps (uses a separate slice query per each required property)<br>- `required_and_next_properties` Prefetch the same properties as with `required_properties_only` mode, but also prefetch
properties which may be needed in the next properties access step like `values`, `properties,` `valueMap`, `elementMap`, or `propertyMap`.
In case the next step is not one of those properties access steps then this mode behaves same as `required_properties_only`.
In case the next step is one of the properties access steps with limited scope of properties, those properties will be
Expand All @@ -372,6 +372,7 @@ behaves same as `all_properties`.<br>- `required_and_next_properties_or_all` Pre
`values`, `properties,` `valueMap`, `elementMap`, or `propertyMap` then acts like `all_properties`.<br>- `none` Skips `has` step batch properties pre-fetch optimization.<br> | String | required_and_next_properties | MASKABLE |
| query.batch.limited | Configure a maximum batch size for queries against the storage backend. This can be used to ensure responsiveness if batches tend to grow very large. The used batch size is equivalent to the barrier size of a preceding `barrier()` step. If a step has no preceding `barrier()`, the default barrier of TinkerPop will be inserted. This option only takes effect if `query.batch.enabled` is `true`. | Boolean | true | MASKABLE |
| query.batch.limited-size | Default batch size (barrier() step size) for queries. This size is applied only for cases where `LazyBarrierStrategy` strategy didn't apply `barrier` step and where user didn't apply barrier step either. This option is used only when `query.batch.limited` is `true`. Notice, value `2147483647` is considered to be unlimited. | Integer | 2500 | MASKABLE |
| query.batch.properties-mode | Properties pre-fetching mode for `values`, `properties`, `valueMap`, `propertyMap`, `elementMap` steps. Used only when query.batch.enabled is `true`.<br>Supported modes:<br>- `all_properties` Pre-fetch all vertex properties on any property access (fetches all vertex properties in a single slice query)<br>- `required_properties_only` Pre-fetch necessary vertex properties only (uses a separate slice query per each required property)<br>- `none` Skips vertex properties pre-fetching optimization.<br> | String | required_properties_only | MASKABLE |
| query.batch.repeat-step-mode | Batch mode for `repeat` step. Used only when query.batch.enabled is `true`.<br>These modes are controlling how the child steps with batch support are behaving if they placed to the start of the `repeat`, `emit`, or `until` traversals.<br>Supported modes:<br>- `closest_repeat_parent` Child start steps are receiving vertices for batching from the closest `repeat` step parent only.<br>- `all_repeat_parents` Child start steps are receiving vertices for batching from all `repeat` step parents.<br>- `starts_only_of_all_repeat_parents` Child start steps are receiving vertices for batching from the closest `repeat` step parent (both for the parent start and for next iterations) and also from all `repeat` step parents for the parent start. | String | all_repeat_parents | MASKABLE |

### schema
Expand Down
Loading

1 comment on commit 0588749

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark

Benchmark suite Current: 0588749 Previous: f3cdce1 Ratio
org.janusgraph.JanusGraphSpeedBenchmark.basicAddAndDelete 22487.688453141844 ms/op 14332.161923092626 ms/op 1.57
org.janusgraph.GraphCentricQueryBenchmark.getVertices 2051.8014344493445 ms/op 1321.1248866847918 ms/op 1.55
org.janusgraph.MgmtOlapJobBenchmark.runClearIndex 224.56849588735182 ms/op 219.3187360514493 ms/op 1.02
org.janusgraph.MgmtOlapJobBenchmark.runReindex 622.2488377622221 ms/op 457.21256604333337 ms/op 1.36
org.janusgraph.JanusGraphSpeedBenchmark.basicCount 471.01293136363745 ms/op 339.7662905587447 ms/op 1.39
org.janusgraph.CQLMultiQueryHasStepBenchmark.getVerticesFilteredByHasStepWithNonHasStepAfterOut 201525.72720288744 ms/op 126935.8882336863 ms/op 1.59
org.janusgraph.CQLMultiQueryBenchmark.getElementsWithUsingEmitRepeatSteps 49266.98369803667 ms/op 29790.763599713093 ms/op 1.65
org.janusgraph.CQLMultiQueryBenchmark.getAllElementsTraversedFromOuterVertex 24291.320150468575 ms/op 14584.747217241957 ms/op 1.67
org.janusgraph.CQLMultiQueryBenchmark.getVerticesWithDoubleUnion 897.1818105917603 ms/op 597.5285513438765 ms/op 1.50
org.janusgraph.CQLMultiQueryHasStepBenchmark.getVerticesFilteredByHasStep 198870.81901944667 ms/op 126664.80159906372 ms/op 1.57
org.janusgraph.CQLMultiQueryBenchmark.getNames 24008.61749512 ms/op 14698.622305737883 ms/op 1.63
org.janusgraph.CQLMultiQueryBenchmark.getVerticesFilteredByAndStep 933.2796089347253 ms/op 648.1313542954531 ms/op 1.44
org.janusgraph.CQLMultiQueryBenchmark.getVerticesFromMultiNestedRepeatStepStartingFromSingleVertex 32625.32965501191 ms/op 20241.54119706591 ms/op 1.61
org.janusgraph.CQLMultiQueryBenchmark.getVerticesWithCoalesceUsage 865.8881925524842 ms/op 580.6018760218902 ms/op 1.49
org.janusgraph.CQLMultiQueryHasStepBenchmark.getAllPropertiesOfVerticesFilteredByHasStep 250358.80243016285 ms/op 158222.3357738333 ms/op 1.58
org.janusgraph.CQLMultiQueryHasStepBenchmark.getVerticesFilteredByHasStepInParentStep 206562.74798850415 ms/op 130344.12783409984 ms/op 1.58
org.janusgraph.CQLMultiQueryBenchmark.getIdToOutVerticesProjection 580.3718549374158 ms/op 398.7058041528264 ms/op 1.46
org.janusgraph.CQLMultiQueryBenchmark.getNeighborNames 24401.722849545076 ms/op 14774.73570899399 ms/op 1.65
org.janusgraph.CQLMultiQueryBenchmark.getElementsWithUsingRepeatUntilSteps 24940.12666646135 ms/op 16211.509679201707 ms/op 1.54
org.janusgraph.CQLMultiQueryBenchmark.getAdjacentVerticesLocalCounts 23370.15574008905 ms/op 15057.017418793792 ms/op 1.55
org.janusgraph.CQLMultiQueryHasStepBenchmark.getSpecificPropertiesOfVerticesFilteredByHasStep 319195.1966225562 ms/op 199029.78159943083 ms/op 1.60

This comment was automatically generated by workflow using github-action-benchmark.

Please sign in to comment.