Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[turborepo] some crashes when running turbo prune --docker #5202

Closed
Klaitos opened this issue Jun 5, 2023 · 2 comments · Fixed by #5230
Closed

[turborepo] some crashes when running turbo prune --docker #5202

Klaitos opened this issue Jun 5, 2023 · 2 comments · Fixed by #5230
Assignees
@chris-olszewski
Labels
kind: bug Something isn't working needs: triage New issues get this label. Remove it after triage

Comments

@Klaitos
Copy link

Klaitos commented Jun 5, 2023

What version of Turborepo are you using?

1.10.1

What package manager are you using / does the bug impact?

npm

What operating system are you using?

Linux

Describe the Bug

Hello,

For almost ~2 weeks now, we are experiencing some random crashes when using turbo prune command on our mono repo in a Docker alpine 3.17 container on a ubuntu 22.04 machine with 4 cores.

It is not happening every time, but few times a week. Retrying the exactly same CI step does not reproduce the issue, the command finally succeed.
We always use last version of turbo repo. (crash is present in 1.9 and 1.10 of turbo repo)
It seems happening more or less right after the 14 or 15 or 16 package iteration but its a guess.
We are using turbo repo remote cache (custom endpoint) but it is not used in this command as far as I know.

(package names have been anonymized)

#22 3.126  - Added @choose/package
#22 3.129  - Added @choose/package
#22 3.131  - Added @choose/package
#22 3.133  - Added @choose/package
#22 3.134  - Added @choose/package
#22 3.135  - Added @choose/package
#22 3.137  - Added @choose/package
#22 3.142  - Added @choose/package
#22 3.143  - Added @choose/package
#22 3.144  - Added @choose/package
#22 3.145  - Added @choose/package
#22 3.146  - Added @choose/package
#22 3.147  - Added @choose/package
#22 3.148  - Added @choose/package
#22 3.149  - Added @choose/package
#22 3.150  - Added @choose/package
#22 3.151  - Added @choose/package
#22 3.184 runtime: bad pointer in frame github.com/vercel/turbo/cli/internal/ffi.Unmarshal[...] at 0xc000017678: 0x1
#22 3.184 fatal error: invalid pointer found on stack

I have also observed this slightly different error

 runtime: bad pointer in frame github.com/vercel/turbo/cli/internal/ffi.RecursiveCopy at 0xc0002776f0: 0x1

FULL STACK TRACE

runtime stack:
#21 4.137 runtime.throw({0x368c4e?, 0x11ba380?})
#21 4.137 	runtime/panic.go:992 +0x71 fp=0x7f210d5ac1d8 sp=0x7f210d5ac1a8 pc=0x918491
#21 4.137 runtime.adjustpointers(0x7f210d5ac5c8?, 0x12daa0?, 0x6ec462?, {0x83e710?, 0x11ba380?})
#21 4.138 	runtime/stack.go:628 +0x1d0 fp=0x7f210d5ac240 sp=0x7f210d5ac1d8 pc=0x931150
#21 4.138 runtime.adjustframe(0x7f210d5ac5c8, 0x7f210d5ac6b8)
#21 4.138 	runtime/stack.go:670 +0xc9 fp=0x7f210d5ac2d0 sp=0x7f210d5ac240 pc=0x931269
#21 4.138 runtime.gentraceback(0x64799789?, 0x7f56db4?, 0x64799772?, 0x1e5f2bd7?, 0x0, 0x0, 0x7fffffff, 0x3a05a8, 0x0?, 0x0)
#21 4.138 	runtime/traceback.go:345 +0xc47 fp=0x7f210d5ac630 sp=0x7f210d5ac2d0 pc=0x93db87
#21 4.138 runtime.copystack(0xc000003ba0, 0x0?)
#21 4.138 	runtime/stack.go:930 +0x2f5 fp=0x7f210d5ac7e8 sp=0x7f210d5ac630 pc=0x931a75
#21 4.138 runtime.shrinkstack(0xc000003ba0)
#21 4.138 	runtime/stack.go:1212 +0x126 fp=0x7f210d5ac808 sp=0x7f210d5ac7e8 pc=0x932946
#21 4.138 runtime.newstack()
#21 4.138 	runtime/stack.go:1060 +0x3aa fp=0x7f210d5ac9c0 sp=0x7f210d5ac808 pc=0x931f0a
#21 4.138 runtime.morestack()
#21 4.138 	runtime/asm_amd64.s:547 +0x8b fp=0x7f210d5ac9c8 sp=0x7f210d5ac9c0 pc=0x948fab
#21 4.138 
#21 4.138 goroutine 6 [running]:
#21 4.138 github.com/vercel/turbo/cli/internal/ffi/proto.(*RecursiveCopyResponse).Reset(0xc0009aaa80?)
#21 4.138 	github.com/vercel/turbo/cli/internal/ffi/proto/messages.pb.go:1626 +0x8a fp=0xc000a594f8 sp=0xc000a594f0 pc=0xddddca
#21 4.138 google.golang.org/protobuf/proto.Reset({0x45fea0, 0xc0009aaa80})
#21 4.138 	google.golang.org/protobuf@v1.28.0/proto/reset.go:18 +0x3e fp=0xc000a59530 sp=0xc000a594f8 pc=0xd6525e
#21 4.138 google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x0, 0x0, 0x0, {0x463618, 0xc0001f4450}, 0x2710}, {0x1, 0x0, 0x0}, ...)
#21 4.138 	google.golang.org/protobuf@v1.28.0/proto/decode.go:87 +0xc5 fp=0xc000a595e8 sp=0xc000a59530 pc=0xd5aa45
#21 4.138 google.golang.org/protobuf/proto.Unmarshal({0x1, 0x0, 0x0}, {0x45fea0?, 0xc0009aaa80?})
#21 4.138 	google.golang.org/protobuf@v1.28.0/proto/decode.go:54 +0x65 fp=0xc000a59640 sp=0xc000a595e8 pc=0xd5a6e5
#21 4.138 github.com/vercel/turbo/cli/internal/ffi.Unmarshal[...]({0xae8d300?, 0x1?}, {0x45fea0?, 0xc0009aaa80?})
#21 4.138 	github.com/vercel/turbo/cli/internal/ffi/ffi.go:32 +0x94 fp=0xc000a596b0 sp=0xc000a59640 pc=0xde14d4
#21 4.138 github.com/vercel/turbo/cli/internal/ffi.RecursiveCopy({0xc00046fe18, 0x13}, {0xc00046b960, 0x1c})
#21 4.138 	github.com/vercel/turbo/cli/internal/ffi/ffi.go:285 +0x165 fp=0xc000a59720 sp=0xc000a596b0 pc=0xde1065
#21 4.138 github.com/vercel/turbo/cli/internal/fs.RecursiveCopy(...)
#21 4.138 	github.com/vercel/turbo/cli/internal/fs/recursive_copy.go:10
#21 4.138 github.com/vercel/turbo/cli/internal/prune.(*prune).prune(0xc000165f60, 0xc000254c30, {0xc000298032, 0x5})
#21 4.138 	github.com/vercel/turbo/cli/internal/prune/prune.go:160 +0x250a fp=0xc000a59f08 sp=0xc000a59720 pc=0xe5ae4a
#21 4.138 github.com/vercel/turbo/cli/internal/prune.ExecutePrune(0x0?, 0xc0001fbd40)
#21 4.138 	github.com/vercel/turbo/cli/internal/prune/prune.go:45 +0xf9 fp=0xc000a59f78 sp=0xc000a59f08 pc=0xe585f9
#21 4.138 github.com/vercel/turbo/cli/internal/cmd.RunWithExecutionState.func1()
#21 4.138 	github.com/vercel/turbo/cli/internal/cmd/root.go:67 +0x107 fp=0xc000a59fe0 sp=0xc000a59f78 pc=0xf38467
#21 4.138 runtime.goexit()
#21 4.138 	runtime/asm_amd64.s:1571 +0x1 fp=0xc000a59fe8 sp=0xc000a59fe0 pc=0x94b081
#21 4.138 created by github.com/vercel/turbo/cli/internal/cmd.RunWithExecutionState
#21 4.138 	github.com/vercel/turbo/cli/internal/cmd/root.go:64 +0x31a
#21 4.138 
#21 4.138 goroutine 1 [select]:
#21 4.138 runtime.gopark(0xc000267ec0?, 0x2?, 0x1?, 0x0?, 0xc000267e44?)
#21 4.138 	runtime/proc.go:361 +0xd6 fp=0xc000068cb0 sp=0xc000068c90 pc=0x91af76
#21 4.138 runtime.selectgo(0xc000068ec0, 0xc000267e40, 0xc000267e40?, 0x0, 0x1e5?, 0x1)
#21 4.138 	runtime/select.go:[328](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:332) +0x772 fp=0xc000068df0 sp=0xc000068cb0 pc=0x92a872
#21 4.138 github.com/vercel/turbo/cli/internal/cmd.RunWithExecutionState(0xc0001fbd40, {0x3528f8, 0x5})
#21 4.138 	github.com/vercel/turbo/cli/internal/cmd/root.go:79 +0x36a fp=0xc000068ef8 sp=0xc000068df0 pc=0xf3814a
#21 4.138 main.main()
#21 4.138 	github.com/vercel/turbo/cli/cmd/turbo/main.go:30 +0x1bb fp=0xc000068f80 sp=0xc000068ef8 pc=0xf38d9b
#21 4.138 runtime.main()
#21 4.138 	runtime/proc.go:250 +0x212 fp=0xc000068fe0 sp=0xc000068f80 pc=0x91abb2
#21 4.138 runtime.goexit()
#21 4.138 	runtime/asm_amd64.s:1571 +0x1 fp=0xc000068fe8 sp=0xc000068fe0 pc=0x94b081
#21 4.138 
#21 4.138 goroutine 2 [force gc (idle)]:
#21 4.138 runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
#21 4.138 	runtime/proc.go:[361](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:365) +0xd6 fp=0xc00005afb0 sp=0xc00005af90 pc=0x91af76
#21 4.138 runtime.goparkunlock(...)
#21 4.138 	runtime/proc.go:367
#21 4.138 runtime.forcegchelper()
#21 4.138 	runtime/proc.go:301 +0xad fp=0xc00005afe0 sp=0xc00005afb0 pc=0x91ae0d
#21 4.138 runtime.goexit()
#21 4.138 	runtime/asm_amd64.s:1571 +0x1 fp=0xc00005afe8 sp=0xc00005afe0 pc=0x94b081
#21 4.138 created by runtime.init.6
#21 4.138 	runtime/proc.go:289 +0x25
#21 4.138 
#21 4.138 goroutine 3 [GC sweep wait]:
#21 4.138 runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
#21 4.138 	runtime/proc.go:361 +0xd6 fp=0xc00005b790 sp=0xc00005b770 pc=0x91af76
#21 4.138 runtime.goparkunlock(...)
#21 4.138 	runtime/proc.go:[367](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:371)
#21 4.138 runtime.bgsweep(0x0?)
#21 4.138 	runtime/mgcsweep.go:297 +0xd7 fp=0xc00005b7c8 sp=0xc00005b790 pc=0x906f77
#21 4.138 runtime.gcenable.func1()
#21 4.138 	runtime/mgc.go:177 +0x26 fp=0xc00005b7e0 sp=0xc00005b7c8 pc=0x8fcb26
#21 4.139 runtime.goexit()
#21 4.139 	runtime/asm_amd64.s:1571 +0x1 fp=0xc00005b7e8 sp=0xc00005b7e0 pc=0x94b081
#21 4.139 created by runtime.gcenable
#21 4.139 	runtime/mgc.go:177 +0x6b
#21 4.139 
#21 4.139 goroutine 4 [sleep]:
#21 4.139 runtime.gopark(0xc00008e000?, 0x2cf62fb0cc?, 0x0?, 0x0?, 0x3a05c8?)
#21 4.139 	runtime/proc.go:361 +0xd6 fp=0xc00005bed0 sp=0xc00005beb0 pc=0x91af76
#21 4.139 runtime.goparkunlock(...)
#21 4.139 	runtime/proc.go:367
#21 4.139 runtime.scavengeSleep(0x3b9aca00)
#21 4.139 	runtime/mgcscavenge.go:248 +0x9d fp=0xc00005bf20 sp=0xc00005bed0 pc=0x904a5d
#21 4.139 runtime.bgscavenge(0x0?)
#21 4.139 	runtime/mgcscavenge.go:[394](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:398) +0x1af fp=0xc00005bfc8 sp=0xc00005bf20 pc=0x904c8f
#21 4.139 runtime.gcenable.func2()
#21 4.139 	runtime/mgc.go:178 +0x26 fp=0xc00005bfe0 sp=0xc00005bfc8 pc=0x8fcac6
#21 4.139 runtime.goexit()
#21 4.139 	runtime/asm_amd64.s:1571 +0x1 fp=0xc00005bfe8 sp=0xc00005bfe0 pc=0x94b081
#21 4.139 created by runtime.gcenable
#21 4.139 	runtime/mgc.go:178 +0xaa
#21 4.139 
#21 4.139 goroutine 18 [finalizer wait]:
#21 4.139 runtime.gopark(0xc0001041a0?, 0x0?, 0x70?, 0xa7?, 0x927b31?)
#21 4.139 	runtime/proc.go:361 +0xd6 fp=0xc00005a630 sp=0xc00005a610 pc=0x91af76
#21 4.139 runtime.goparkunlock(...)
#21 4.139 	runtime/proc.go:367
#21 4.139 runtime.runfinq()
#21 4.139 	runtime/mfinal.go:177 +0xb3 fp=0xc00005a7e0 sp=0xc00005a630 pc=0x8fbbd3
#21 4.139 runtime.goexit()
#21 4.139 	runtime/asm_amd64.s:1571 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x94b081
#21 4.139 created by runtime.createfing
#21 4.139 	runtime/mfinal.go:157 +0x45
#21 4.139 
#21 4.139 goroutine 19 [select, locked to thread]:
#21 4.139 runtime.gopark(0xc0000567a8?, 0x2?, 0xf2?, 0xb2?, 0xc0000567a4?)
#21 4.139 	runtime/proc.go:361 +0xd6 fp=0xc000056618 sp=0xc0000565f8 pc=0x91af76
#21 4.139 runtime.selectgo(0xc0000567a8, 0xc0000567a0, 0x0?, 0x0, 0x0?, 0x1)
#21 4.139 	runtime/select.go:328 +0x772 fp=0xc000056758 sp=0xc000056618 pc=0x92a872
#21 4.139 runtime.ensureSigM.func1()
#21 4.139 	runtime/signal_unix.go:973 +0x1bb fp=0xc0000567e0 sp=0xc000056758 pc=0x92ebfb
#21 4.139 runtime.goexit()
#21 4.139 	runtime/asm_amd64.s:1571 +0x1 fp=0xc0000567e8 sp=0xc0000567e0 pc=0x94b081
#21 4.139 created by runtime.ensureSigM
#21 4.139 	runtime/signal_unix.go:956 +0xbd
#21 4.139 
#21 4.139 goroutine 20 [syscall]:
#21 4.139 runtime.notetsleepg(0x0?, 0x0?)
#21 4.139 	runtime/lock_futex.go:235 +0x34 fp=0xc000056fa0 sp=0xc000056f68 pc=0x8ef4f4
#21 4.139 os/signal.signal_recv()
#21 4.139 	runtime/sigqueue.go:151 +0x2f fp=0xc000056fc0 sp=0xc000056fa0 pc=0x9475cf
#21 4.139 os/signal.loop()
#21 4.139 	os/signal/signal_unix.go:23 +0x19 fp=0xc000056fe0 sp=0xc000056fc0 pc=0xba8e19
#21 4.139 runtime.goexit()
#21 4.139 	runtime/asm_amd64.s:1571 +0x1 fp=0xc000056fe8 sp=0xc000056fe0 pc=0x94b081
#21 4.139 created by os/signal.Notify.func1.1
#21 4.139 	os/signal/signal.go:151 +0x2a
#21 4.139 
#21 4.139 goroutine 5 [chan receive]:
#21 4.139 runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
#21 4.139 	runtime/proc.go:361 +0xd6 fp=0xc00005c700 sp=0xc00005c6e0 pc=0x91af76
#21 4.139 runtime.chanrecv(0xc00012ab40, 0x0, 0x1)
#21 4.139 	runtime/chan.go:577 +0x56c fp=0xc00005c790 sp=0xc00005c700 pc=0x8e9f4c
#21 4.139 runtime.chanrecv1(0x0?, 0x0?)
#21 4.139 	runtime/chan.go:[440](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:444) +0x18 fp=0xc00005c7b8 sp=0xc00005c790 pc=0x8e9978
#21 4.139 github.com/vercel/turbo/cli/internal/signals.NewWatcher.func1()
#21 4.139 	github.com/vercel/turbo/cli/internal/signals/signals.go:56 +0x28 fp=0xc00005c7e0 sp=0xc00005c7b8 pc=0xf2d7e8
#21 4.139 runtime.goexit()
#21 4.139 	runtime/asm_amd64.s:1571 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x94b081
#21 4.139 created by github.com/vercel/turbo/cli/internal/signals.NewWatcher
#21 4.139 	github.com/vercel/turbo/cli/internal/signals/signals.go:55 +0x12f
#21 4.139 
#21 4.139 goroutine 96 [GC worker (idle)]:
#21 4.139 runtime.gopark(0x2cba803211?, 0xc0000468f0?, 0xd0?, 0x7?, 0xdea727?)
#21 4.139 	runtime/proc.go:361 +0xd6 fp=0xc0000e0758 sp=0xc0000e0738 pc=0x91af76
#21 4.139 runtime.gcBgMarkWorker()
#21 4.139 	runtime/mgc.go:1207 +0xe5 fp=0xc0000e07e0 sp=0xc0000e0758 pc=0x8fec05
#21 4.139 runtime.goexit()
#21 4.139 	runtime/asm_amd64.s:1571 +0x1 fp=0xc0000e07e8 sp=0xc0000e07e0 pc=0x94b081
#21 4.139 created by runtime.gcBgMarkStartWorkers
#21 4.139 	runtime/mgc.go:1131 +0x25
#21 4.139 
#21 4.139 goroutine 95 [GC worker (idle)]:
#21 4.139 runtime.gopark(0x2cba7908e1?, 0xc0000[468](https://github.com/appchoose/backend/actions/runs/5152880992/jobs/9279522143#step:6:472)f0?, 0xd0?, 0xf?, 0xdea727?)
#21 4.139 	runtime/proc.go:361 +0xd6 fp=0xc0000e0f58 sp=0xc0000e0f38 pc=0x91af76
#21 4.139 runtime.gcBgMarkWorker()
#21 4.139 	runtime/mgc.go:1207 +0xe5 fp=0xc0000e0fe0 sp=0xc0000e0f58 pc=0x8fec05
#21 4.139 runtime.goexit()
#21 4.139 	runtime/asm_amd64.s:1571 +0x1 fp=0xc0000e0fe8 sp=0xc0000e0fe0 pc=0x94b081
#21 4.139 created by runtime.gcBgMarkStartWorkers
#21 4.139 	runtime/mgc.go:1131 +0x25
#21 4.143 npm notice 
#21 4.144 npm notice New minor version of npm available! 9.5.1 -> 9.6.7
#21 4.144 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.6.7>
#21 4.144 npm notice Run `npm install -g npm@9.6.7` to update!
#21 4.144 npm notice 
#21 ERROR: process "/bin/sh -c npx --yes turbo@1.10.1 prune --scope="legacy" --docker" did not complete successfully: exit code: 2

Thanks

Expected Behavior

The turpo repo prune command succeeded with a lighter package.json.

To Reproduce

Not able to reproduce every time, it seems to be very hazardous.

Reproduction Repo

No response
@Klaitos Klaitos added area: turborepo kind: bug Something isn't working needs: triage New issues get this label. Remove it after triage labels Jun 5, 2023
@Klaitos Klaitos changed the title [turborepo] some crashes when running turbo@1.10.1 prune [turborepo] some crashes when running turbo prune --docker Jun 5, 2023
@chris-olszewski chris-olszewski self-assigned this Jun 5, 2023
@chris-olszewski chris-olszewski mentioned this issue Jun 6, 2023
Merged
@chris-olszewski
Copy link
Member

Thanks for this report! This helped us track down a very interesting mistake we made with our FFI calls from Go to Rust. The description in #5230 has more information.

gsoltis pushed a commit that referenced this issue Jun 7, 2023
@chris-olszewski
fix(ffi): fix Rust dangling pointer (#5230)
c3fd591 
### Description

Fixes #5202

The underlying issue is how protobuf encodes a message with a single
optional field. If that single field is missing the message gets encoded
as a zero length string of bytes. This presents a problem with our usage
of `Vec::as_mut_ptr` which: ["Returns a dangling raw pointer valid for
zero sized reads if the vector didn’t
allocate."](https://doc.rust-lang.org/std/vec/struct.Vec.html#method.as_mut_ptr)
This caused us to send a buffer of length 0, but a dangling data pointer
instead of a null pointer. This wasn't a consistent failure as Go only
checks the validity of all the pointer in the stack under certain
conditions. Specifically when growing/shrinking the stack and needing to
[adjust pointers in a
frame](https://go.googlesource.com/go/+/refs/heads/master/src/runtime/stack.go#626).


### Testing Instructions

Added unit tests that deal with zero length buffers:
- Rust side ensure that if a protobuf message is zero length we return a
buffer with length `0` and the data pointer is the null pointer
- Go side the ensures that a buffer with `0` length and a null data
pointer gets converted to an empty byte slice

---------

Co-authored-by: Chris Olszewski <Chris Olszewski>
@Klaitos
Copy link
Author

Klaitos commented Jun 7, 2023

That was fast ! thanks guys

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chris-olszewski chris-olszewski

Labels
kind: bug Something isn't working needs: triage New issues get this label. Remove it after triage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(ffi): fix Rust dangling pointer vercel/turborepo
2 participants
@Klaitos @chris-olszewski