Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce simdjson #23644

Merged
merged 1 commit into from
Apr 26, 2023
Merged

Introduce simdjson #23644

merged 1 commit into from
Apr 26, 2023

Conversation

yah01
Copy link
Member

@yah01 yah01 commented Apr 24, 2023

Use simdjson to process json

To enable mmap and reach least memory usage, we won't store the parsed json doc in memory.
Each doc will be parsed while querying on it, the cost to construct a parser is heavy, so this uses thread local parsers to avoid creating parser each time.

This requires to not spawn too many threads in segcore, and reuse the threads but always spawn new ones

related #23389

@sre-ci-robot sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Apr 24, 2023
@mergify mergify bot added the dco-passed DCO check passed. label Apr 24, 2023
@yah01 yah01 force-pushed the simdjson branch 2 times, most recently from e62455d to 31f14b7 Compare April 24, 2023 03:28
@mergify
Copy link
Contributor

mergify bot commented Apr 24, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 24, 2023

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 24, 2023

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 24, 2023

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 24, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 24, 2023

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

@yah01 yah01 mentioned this pull request Apr 24, 2023
1 task
@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

1 similar comment
@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

4 similar comments
@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented Apr 25, 2023

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: yah01 <yang.cen@zilliz.com>
@codecov
Copy link

codecov bot commented Apr 25, 2023

Codecov Report

Merging #23644 (3d047c0) into master (4fb8919) will decrease coverage by 0.35%.
The diff coverage is 36.36%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #23644      +/-   ##
==========================================
- Coverage   82.11%   81.77%   -0.35%     
==========================================
  Files         742      744       +2     
  Lines       96994    99055    +2061     
==========================================
+ Hits        79647    80999    +1352     
- Misses      14378    15087     +709     
  Partials     2969     2969              
Impacted Files Coverage Δ
internal/core/src/common/Types.h 100.00% <ø> (ø)
...ternal/core/src/query/visitors/ExecExprVisitor.cpp 72.96% <ø> (-3.89%) ⬇️
internal/core/src/segcore/ConcurrentVector.cpp 35.64% <0.00%> (-0.73%) ⬇️
internal/core/src/segcore/SegmentSealedImpl.h 100.00% <ø> (ø)
internal/core/src/segcore/Utils.cpp 62.01% <0.00%> (-6.49%) ⬇️
internal/core/src/common/Json.h 13.33% <13.33%> (ø)
internal/core/src/common/Column.h 91.07% <100.00%> (+16.07%) ⬆️
internal/core/src/common/Utils.h 74.78% <100.00%> (-7.83%) ⬇️
internal/core/src/segcore/SegmentGrowingImpl.cpp 67.33% <100.00%> (-0.62%) ⬇️
internal/core/src/segcore/SegmentSealedImpl.cpp 77.20% <100.00%> (-4.47%) ⬇️

... and 158 files with indirect coverage changes

@jiaoew1991
Copy link
Contributor

/lgtm
/approve

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiaoew1991, yah01

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jiaoew1991 jiaoew1991 added ci-passed manual-pass manually set pass before ci-passed labeled labels Apr 26, 2023
@sre-ci-robot sre-ci-robot merged commit 60fdd7e into milvus-io:master Apr 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved area/compilation ci-passed dco-passed DCO check passed. lgtm manual-pass manually set pass before ci-passed labeled size/L Denotes a PR that changes 100-499 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants