Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lightning: fix "context cancel" overwrites the real error #44734

Merged
merged 2 commits into from
Jun 21, 2023
Merged

lightning: fix "context cancel" overwrites the real error #44734

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
82ca799
lightning: fix "context cancel" overwrites the real error
lance6716 Jun 16, 2023
2d3ca6b
address comment
lance6716 Jun 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions br/pkg/lightning/backend/local/local.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1145,14 +1145,16 @@ func (local *Backend) generateAndSendJob(
for _, jobRange := range jobRanges {
r := jobRange
eg.Go(func() error {
select {
case <-egCtx.Done():
if egCtx.Err() != nil {
return nil
default:
}
Comment on lines -1148 to 1150
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any difference?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

select {} will lock the channel to read the value so performance is better. And code becomes shorter


failpoint.Inject("beforeGenerateJob", nil)
jobs, err := local.generateJobForRange(egCtx, engine, r, regionSplitSize, regionSplitKeys)
if err != nil {
if common.IsContextCanceledError(err) {
return nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we return nil here, caller will continue running

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's expected? because context is egCtx and in line 1148 they will fail fast.

Copy link
Contributor

@D3Hunter D3Hunter Jun 16, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this happen:

  • parent ctx cancelled,generateAndSendJob returns nil
  • all workers returns nil on ctx.Done()
  • then we log that engine import success

maybe add ctx.Err into firstErr before return

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2d3ca6b

}
return err
}
for _, job := range jobs {
Expand Down Expand Up @@ -1189,6 +1191,9 @@ func (local *Backend) generateJobForRange(
regionSplitSize, regionSplitKeys int64,
) ([]*regionJob, error) {
failpoint.Inject("fakeRegionJobs", func() {
if ctx.Err() != nil {
failpoint.Return(nil, ctx.Err())
}
key := [2]string{string(keyRange.start), string(keyRange.end)}
injected := fakeRegionJobs[key]
// overwrite the stage to regionScanned, because some time same keyRange
Expand Down
57 changes: 57 additions & 0 deletions br/pkg/lightning/backend/local/local_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2028,3 +2028,60 @@ func TestRegionJobResetRetryCounter(t *testing.T) {
}
}
}

func TestCtxCancelIsIgnored(t *testing.T) {
backup := maxRetryBackoffSecond
maxRetryBackoffSecond = 1
t.Cleanup(func() {
maxRetryBackoffSecond = backup
})

_ = failpoint.Enable("github.com/pingcap/tidb/br/pkg/lightning/backend/local/skipSplitAndScatter", "return()")
_ = failpoint.Enable("github.com/pingcap/tidb/br/pkg/lightning/backend/local/fakeRegionJobs", "return()")
_ = failpoint.Enable("github.com/pingcap/tidb/br/pkg/lightning/backend/local/beforeGenerateJob", "sleep(1000)")
_ = failpoint.Enable("github.com/pingcap/tidb/br/pkg/lightning/backend/local/WriteToTiKVNotEnoughDiskSpace", "return()")
t.Cleanup(func() {
_ = failpoint.Disable("github.com/pingcap/tidb/br/pkg/lightning/backend/local/skipSplitAndScatter")
_ = failpoint.Disable("github.com/pingcap/tidb/br/pkg/lightning/backend/local/fakeRegionJobs")
_ = failpoint.Disable("github.com/pingcap/tidb/br/pkg/lightning/backend/local/beforeGenerateJob")
_ = failpoint.Disable("github.com/pingcap/tidb/br/pkg/lightning/backend/local/WriteToTiKVNotEnoughDiskSpace")
})

initRanges := []Range{
{start: []byte{'c'}, end: []byte{'d'}},
{start: []byte{'d'}, end: []byte{'e'}},
}
fakeRegionJobs = map[[2]string]struct {
jobs []*regionJob
err error
}{
{"c", "d"}: {
jobs: []*regionJob{
{
keyRange: Range{start: []byte{'c'}, end: []byte{'d'}},
engine: &Engine{},
injected: getSuccessInjectedBehaviour(),
},
},
},
{"d", "e"}: {
jobs: []*regionJob{
{
keyRange: Range{start: []byte{'d'}, end: []byte{'e'}},
engine: &Engine{},
injected: getSuccessInjectedBehaviour(),
},
},
},
}

ctx := context.Background()
l := &Backend{
BackendConfig: BackendConfig{
WorkerConcurrency: 1,
},
}
e := &Engine{}
err := l.doImport(ctx, e, initRanges, int64(config.SplitRegionSize), int64(config.SplitRegionKeys))
require.ErrorContains(t, err, "the remaining storage capacity of TiKV")
}