Support to range filter with json expr #23739

yah01 · 2023-04-26T09:46:58Z

/kind feature
related #23389

mergify · 2023-04-26T10:31:47Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-26T10:32:17Z

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

codecov · 2023-04-26T10:49:49Z

Codecov Report

Merging #23739 (ffcb4b4) into master (016311a) will decrease coverage by 0.35%.
The diff coverage is 52.07%.

@@            Coverage Diff             @@
##           master   #23739      +/-   ##
==========================================
- Coverage   82.02%   81.67%   -0.35%     
==========================================
  Files         744      745       +1     
  Lines       97244    99639    +2395     
==========================================
+ Hits        79765    81385    +1620     
- Misses      14514    15289     +775     
  Partials     2965     2965

Impacted Files	Coverage Δ
...nternal/core/src/query/generated/ExecExprVisitor.h	`100.00% <ø> (ø)`
internal/core/src/segcore/ConcurrentVector.h	`84.42% <ø> (+2.32%)`	⬆️
internal/core/src/segcore/SegmentGrowing.h	`100.00% <ø> (ø)`
internal/core/src/query/PlanProto.cpp	`77.43% <13.33%> (-5.90%)`	⬇️
internal/core/src/common/Json.h	`74.28% <36.36%> (+54.28%)`	⬆️
...ternal/core/src/query/visitors/ExecExprVisitor.cpp	`65.37% <51.83%> (-9.15%)`	⬇️
internal/core/src/query/Parser.cpp	`93.01% <71.42%> (+1.97%)`	⬆️
internal/core/src/query/Expr.h	`100.00% <100.00%> (ø)`
internal/core/src/query/ExprImpl.h	`100.00% <100.00%> (ø)`
...core/src/query/visitors/ExtractInfoExprVisitor.cpp	`100.00% <100.00%> (ø)`
... and 2 more

... and 185 files with indirect coverage changes

mergify · 2023-04-26T11:08:44Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-26T11:09:48Z

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

mergify · 2023-04-26T11:24:26Z

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

mergify · 2023-04-26T11:26:00Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-27T03:23:34Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-27T03:59:15Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-27T04:36:02Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-27T10:32:11Z

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

mergify · 2023-04-27T10:35:36Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-27T19:57:41Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-27T21:24:10Z

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

mergify · 2023-04-27T21:29:20Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-28T04:44:09Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

jiaoew1991 · 2023-04-28T08:17:03Z

/lgtm
/approve

mergify · 2023-04-28T08:44:22Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-28T08:49:09Z

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-04-28T08:50:52Z

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

Signed-off-by: yah01 <yang.cen@zilliz.com>

jiaoew1991 · 2023-04-30T12:34:24Z

/lgtm
/approve

sre-ci-robot · 2023-04-30T12:34:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiaoew1991, yah01

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~internal/core/OWNERS~~ [jiaoew1991]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

xiaofan-luan · 2023-04-30T22:27:56Z

internal/core/src/segcore/ConcurrentVector.cpp

@@ -76,10 +76,12 @@ VectorBase::set_data_raw(ssize_t element_offset,
        }
        case DataType::JSON: {
            auto& json_data = FIELD_DATA(data, json);
-            std::vector<Json> data_raw(json_data.size());
+            std::vector<Json> data_raw{};
+            data_raw.reserve(json_data.size());
            for (auto& json_bytes : json_data) {
                data_raw.emplace_back(simdjson::padded_string(json_bytes));


try to use simdjson::minify to save memory?

User may hope to retrieve the original data without any modification, so keep this unchanged here

xiaofan-luan · 2023-04-30T23:05:35Z

internal/core/src/common/Json.h

+    exist(std::vector<std::string> nested_path) const {
+        std::for_each(
+            nested_path.begin(), nested_path.end(), [](std::string& key) {
+                boost::replace_all(key, "~", "~0");


leave the comment here, make sure code reviewer understand it's the requirement of simdJson

xiaofan-luan · 2023-04-30T23:22:05Z

internal/core/src/query/visitors/ExecExprVisitor.cpp

 #include "query/ExprImpl.h"
 #include "query/Relational.h"
 #include "query/Utils.h"
 #include "segcore/SegmentGrowingImpl.h"
+#include "simdjson/error.h"

 namespace milvus::query {


seems we don't support exist for now right

@xiaocai2333 will support exist expression, this pr provides the exist() method in Json class

sre-ci-robot added the kind/feature Issues related to feature request from users label Apr 26, 2023

sre-ci-robot requested review from bigsheeper and czs007 April 26, 2023 09:47

sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Apr 26, 2023

mergify bot added the dco-passed DCO check passed. label Apr 26, 2023

yah01 force-pushed the json-expr branch 2 times, most recently from 90fcc2a to eaabc0d Compare April 26, 2023 10:27

yah01 force-pushed the json-expr branch from eaabc0d to 51014c9 Compare April 26, 2023 11:04

yah01 force-pushed the json-expr branch from 51014c9 to 274b6cd Compare April 26, 2023 11:19

yah01 force-pushed the json-expr branch from 274b6cd to 0311ff5 Compare April 27, 2023 02:49

sre-ci-robot added size/XL Denotes a PR that changes 500-999 lines. and removed size/L Denotes a PR that changes 100-499 lines. labels Apr 27, 2023

yah01 force-pushed the json-expr branch from 0311ff5 to 59b833b Compare April 27, 2023 03:30

yah01 force-pushed the json-expr branch 4 times, most recently from b5b4b9d to 71095cd Compare April 27, 2023 10:14

yah01 force-pushed the json-expr branch from 71095cd to 67483a0 Compare April 27, 2023 10:39

yah01 force-pushed the json-expr branch from bd69d8e to a53e059 Compare April 27, 2023 21:11

yah01 force-pushed the json-expr branch 2 times, most recently from da4cb0e to d6ea092 Compare April 28, 2023 04:31

yah01 force-pushed the json-expr branch from d6ea092 to bf4cb77 Compare April 28, 2023 05:38

sre-ci-robot assigned jiaoew1991 Apr 28, 2023

sre-ci-robot added lgtm approved labels Apr 28, 2023

yah01 force-pushed the json-expr branch from bf4cb77 to b275c59 Compare April 28, 2023 08:38

sre-ci-robot removed the lgtm label Apr 28, 2023

yah01 force-pushed the json-expr branch from b275c59 to 22561a7 Compare April 28, 2023 08:41

yah01 force-pushed the json-expr branch from 22561a7 to 9ef078c Compare April 28, 2023 08:45

Support to filter with json expr

ffcb4b4

Signed-off-by: yah01 <yang.cen@zilliz.com>

yah01 force-pushed the json-expr branch from 9ef078c to ffcb4b4 Compare April 28, 2023 08:51

sre-ci-robot added the lgtm label Apr 30, 2023

jiaoew1991 added ci-passed manual-pass manually set pass before ci-passed labeled labels Apr 30, 2023

sre-ci-robot merged commit 62eea52 into milvus-io:master Apr 30, 2023

xiaofan-luan reviewed Apr 30, 2023

View reviewed changes

This was referenced May 2, 2023

[Cherry-Pick] Support JSON data type #23839

Merged

[Feature]: Support JSON data type #23389

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support to range filter with json expr #23739

Support to range filter with json expr #23739

yah01 commented Apr 26, 2023

mergify bot commented Apr 26, 2023

mergify bot commented Apr 26, 2023

codecov bot commented Apr 26, 2023 •

edited

Loading

mergify bot commented Apr 26, 2023

mergify bot commented Apr 26, 2023

mergify bot commented Apr 26, 2023

mergify bot commented Apr 26, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 28, 2023

jiaoew1991 commented Apr 28, 2023

mergify bot commented Apr 28, 2023

mergify bot commented Apr 28, 2023

mergify bot commented Apr 28, 2023

jiaoew1991 commented Apr 30, 2023

sre-ci-robot commented Apr 30, 2023

xiaofan-luan Apr 30, 2023

yah01 May 5, 2023

xiaofan-luan Apr 30, 2023

xiaofan-luan Apr 30, 2023

yah01 May 5, 2023

Support to range filter with json expr #23739

Support to range filter with json expr #23739

Conversation

yah01 commented Apr 26, 2023

mergify bot commented Apr 26, 2023

mergify bot commented Apr 26, 2023

codecov bot commented Apr 26, 2023 • edited Loading

Codecov Report

mergify bot commented Apr 26, 2023

mergify bot commented Apr 26, 2023

mergify bot commented Apr 26, 2023

mergify bot commented Apr 26, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 27, 2023

mergify bot commented Apr 28, 2023

jiaoew1991 commented Apr 28, 2023

mergify bot commented Apr 28, 2023

mergify bot commented Apr 28, 2023

mergify bot commented Apr 28, 2023

jiaoew1991 commented Apr 30, 2023

sre-ci-robot commented Apr 30, 2023

xiaofan-luan Apr 30, 2023

Choose a reason for hiding this comment

yah01 May 5, 2023

Choose a reason for hiding this comment

xiaofan-luan Apr 30, 2023

Choose a reason for hiding this comment

xiaofan-luan Apr 30, 2023

Choose a reason for hiding this comment

yah01 May 5, 2023

Choose a reason for hiding this comment

codecov bot commented Apr 26, 2023 •

edited

Loading