Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make datanode load statslog lazy if SkipBFStatsLog is true #23779

Merged
merged 1 commit into from
May 5, 2023

Conversation

congqixia
Copy link
Contributor

See also #22994
Related to #23477
/kind improvement

@sre-ci-robot sre-ci-robot added kind/improvement Changes related to something improve, likes ut and code refactor approved labels Apr 27, 2023
@sre-ci-robot sre-ci-robot added the size/M Denotes a PR that changes 30-99 lines. label Apr 27, 2023
@mergify mergify bot added the dco-passed DCO check passed. label Apr 27, 2023
@mergify
Copy link
Contributor

mergify bot commented Apr 27, 2023

@congqixia ut workflow job failed, comment rerun ut can trigger the job again.

@codecov
Copy link

codecov bot commented Apr 27, 2023

Codecov Report

Merging #23779 (e5eb246) into 2.2.0 (4d7f3af) will increase coverage by 0.04%.
The diff coverage is 91.01%.

Impacted file tree graph

@@            Coverage Diff             @@
##            2.2.0   #23779      +/-   ##
==========================================
+ Coverage   80.63%   80.68%   +0.04%     
==========================================
  Files         713      713              
  Lines      101662   101825     +163     
==========================================
+ Hits        81979    82153     +174     
+ Misses      16480    16457      -23     
- Partials     3203     3215      +12     
Impacted Files Coverage Δ
internal/datanode/channel_meta.go 93.78% <88.23%> (-0.60%) ⬇️
internal/datanode/data_sync_service.go 81.56% <100.00%> (+0.06%) ⬆️
internal/datanode/errors.go 100.00% <100.00%> (ø)
internal/datanode/segment.go 84.69% <100.00%> (+2.13%) ⬆️
internal/util/errorutil/util.go 88.09% <100.00%> (+1.98%) ⬆️

... and 34 files with indirect coverage changes

@sre-ci-robot sre-ci-robot added size/L Denotes a PR that changes 100-499 lines. and removed size/M Denotes a PR that changes 30-99 lines. labels Apr 28, 2023
@czs007 czs007 changed the title Make datanode laod statslog lazy if SkipBFStatsLog is true Make datanode load statslog lazy if SkipBFStatsLog is true Apr 28, 2023
@xiaofan-luan
Copy link
Collaborator

/lgtm
/approve

if Params.DataNodeCfg.SkipBFStatsLoad {
// mark segment lazy loading
s.setLoadingLazy(true)
go func() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

didn't really make any sense to me
Why we go the function out but mean time we use a pool?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recalled why add a goroutine to submit task into the work pool: if the worker pool is full, the Submit call will block here until there are some available worker.

@mergify
Copy link
Contributor

mergify bot commented May 4, 2023

@congqixia ut workflow job failed, comment rerun ut can trigger the job again.

1 similar comment
@mergify
Copy link
Contributor

mergify bot commented May 4, 2023

@congqixia ut workflow job failed, comment rerun ut can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented May 4, 2023

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify
Copy link
Contributor

mergify bot commented May 4, 2023

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

1 similar comment
@mergify
Copy link
Contributor

mergify bot commented May 4, 2023

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify mergify bot added the ci-passed label May 4, 2023
if err != nil {
// TODO if not retryable, add rebuild statslog logic
log.Warn("failed to lazy load statslog for segment", zap.Error(err))
if c.retryableLoadError(err) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we take of close here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The closed check is at the beginning of this resubmit function

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@xiaofan-luan
Copy link
Collaborator

/lgtm
/approve

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia, xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify mergify bot added the ci-passed label May 5, 2023
@sre-ci-robot sre-ci-robot merged commit c72db04 into milvus-io:2.2.0 May 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved ci-passed dco-passed DCO check passed. kind/improvement Changes related to something improve, likes ut and code refactor lgtm size/L Denotes a PR that changes 100-499 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants