Br log truncate cmd fails with connection reset by peer

[niezefeng]

We are facing error due toconnection reset by peer with TiDB v6.5.3/BR v6.5.2

The error was resolved with 3rd attempt. we can see number of logs to be deleted were decreasing each time, so assuming there can be some limitation.

[2023/06/22 06:05:34.262 +00:00] [INFO] [info.go:49] ["Welcome to Backup & Restore (BR)"] [release-version=v6.5.2] [git-hash=29116c0256c52b224da2b34d712c1063d171c0ad] [git-branch=heads/refs/tags/v6.5.2] [go-version=go1.19.8] [utc-build-time="2023-04-19 10:44:07"] [race-enabled=false] .............. [2023/06/22 06:06:01.817 +00:00] [ERROR] [stream.go:513] ["failed to stream"] [command="log truncate"] [error="failed to write back file xxxxx/xxxxxx/.meta: failed to remove the empty meta xxxxx/xxxxxx/xxxx.meta: RequestError: send request failed\ncaused by: Delete \"https://bucketxxxxx.meta\": read tcp xxx.xxx.xxx.xxx:36116-> xxx.xxx.xx:443: read: connection reset by peer"] [errorVerbose="RequestError: send request failed\ncaused by: Delete \"https:/xxxxxx.meta\": read tcp xx.xxx.xxx.xxx:36116->xx.xxx.xxx.xxx:443: read: connection reset by peer\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tidb/br/pkg/storage.(*S3Storage).DeleteFile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/storage/s3.go:533\ngithub.com/pingcap/tidb/br/pkg/restore.(*StreamMetadataSet).doWriteBackForFile\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/stream_metas.go:266\ngithub.com/pingcap/tidb/br/pkg/restore.(*StreamMetadataSet).removeDataFilesAndUpdateMetadata\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/stream_metas.go:254\ngithub.com/pingcap/tidb/br/pkg/restore.(*StreamMetadataSet).RemoveDataFilesAndUpdateMetadataInBatch.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/stream_metas.go:184\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nfailed to remove the empty meta v1/backupmeta/442040825730301953-dc9eecc3-b9be-4da0-8501-5f5c688ad6f9.meta\nfailed to write back file v1/backupmeta/442040825730301953-dc9eecc3-b9be-4da0-8501-5f5c688ad6f9.meta"] [stack="github.com/pingcap/tidb/br/pkg/task.RunStreamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/task/stream.go:513\nmain.streamCommand\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:231\nmain.newStreamTruncateCommand.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/stream.go:143\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]

Seems like BR's retry mechanism can't cover this case.

[BornChanger]

This repo is deprecated, and br support is now part of tidb repo in https://github.com/pingcap/tidb/tree/master/br. please open the issue in tidb repo. thx