-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce simdjson #23644
Introduce simdjson #23644
Conversation
e62455d
to
31f14b7
Compare
@yah01 E2e jenkins job failed, comment |
@yah01 ut workflow job failed, comment |
@yah01 ut workflow job failed, comment |
@yah01 ut workflow job failed, comment |
@yah01 E2e jenkins job failed, comment |
@yah01 ut workflow job failed, comment |
@yah01 E2e jenkins job failed, comment |
@yah01 E2e jenkins job failed, comment |
@yah01 ut workflow job failed, comment |
@yah01 E2e jenkins job failed, comment |
@yah01 ut workflow job failed, comment |
@yah01 ut workflow job failed, comment |
@yah01 ut workflow job failed, comment |
1 similar comment
@yah01 ut workflow job failed, comment |
@yah01 E2e jenkins job failed, comment |
@yah01 E2e jenkins job failed, comment |
4 similar comments
@yah01 E2e jenkins job failed, comment |
@yah01 E2e jenkins job failed, comment |
@yah01 E2e jenkins job failed, comment |
@yah01 E2e jenkins job failed, comment |
Signed-off-by: yah01 <yang.cen@zilliz.com>
Codecov Report
@@ Coverage Diff @@
## master #23644 +/- ##
==========================================
- Coverage 82.11% 81.77% -0.35%
==========================================
Files 742 744 +2
Lines 96994 99055 +2061
==========================================
+ Hits 79647 80999 +1352
- Misses 14378 15087 +709
Partials 2969 2969
|
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jiaoew1991, yah01 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Use simdjson to process json
To enable mmap and reach least memory usage, we won't store the parsed json doc in memory.
Each doc will be parsed while querying on it, the cost to construct a parser is heavy, so this uses thread local parsers to avoid creating parser each time.
This requires to not spawn too many threads in segcore, and reuse the threads but always spawn new ones
related #23389