Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merge-sort step pre-alloc too many memory #48342

Closed
D3Hunter opened this issue Nov 7, 2023 · 1 comment · Fixed by #48344
Closed

merge-sort step pre-alloc too many memory #48342

D3Hunter opened this issue Nov 7, 2023 · 1 comment · Fixed by #48344
Labels
affects-7.1 affects-7.5 component/ddl This issue is related to DDL of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@D3Hunter
Copy link
Contributor

D3Hunter commented Nov 7, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

import 10t data on 5 16c32g nodes, during merge-sort step some node take too many memory

node0/1/3 oom, memory usage of 2/4 is normal

image

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

with this pr #48277

| Release Version: v7.6.0-test
Edition: Community
Git Commit Hash: 73afea0396530cad3dde87cd5ea1be77f461e793
Git Branch: heads/refs/tags/v7.6.0-test
UTC Build Time: 2023-11-06 14:44:17
GoVersion: go1.21.3
Race Enabled: false
Check Table Before Drop: false
Store: tikv |
+-----------------------------------
@D3Hunter D3Hunter added the type/bug The issue is confirmed as a bug. label Nov 7, 2023
@D3Hunter
Copy link
Contributor Author

D3Hunter commented Nov 7, 2023

we're using MemTotal to get total memory, which is init as MemTotalNormal as InContainer only checks a few keywords. so the calculated memory is too large.

if strings.Contains(string(v), "docker") ||
strings.Contains(string(v), "kubepods") ||
strings.Contains(string(v), "containerd") {

here is the result of /proc/self/cgroup

tc-tidb-0
0::/
tc-tidb-1
1:name=systemd:/
0::/
tc-tidb-2
12:pids:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
11:rdma:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
10:hugetlb:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
9:cpuset:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
8:net_cls,net_prio:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
7:blkio:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
6:freezer:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
5:perf_event:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
4:devices:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
3:cpu,cpuacct:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
2:memory:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
1:name=systemd:/kubepods.slice/kubepods-pod8b980b2a_d5be_4388_9b76_597c6bb4d4e2.slice/cri-containerd-0a636d4ed110533bcbbce02c840ad554b57b68a4b299f0bd14e018fef22cb59c.scope
0::/
tc-tidb-3
222:name=systemd:/
0::/
tc-tidb-4
12:devices:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
11:cpuset:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
10:pids:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
9:hugetlb:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
8:perf_event:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
7:rdma:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
6:blkio:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
5:memory:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
4:net_cls,net_prio:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
3:freezer:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
2:cpu,cpuacct:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
1:name=systemd:/kubepods.slice/kubepods-pod46be7f8f_2fbd_4130_bfeb_953636dcd12c.slice/cri-containerd-92f0b5f187e11536fe62816fc14c493200851a6d8910ddacd0c844a660c15e95.scope
0::/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-7.1 affects-7.5 component/ddl This issue is related to DDL of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant