-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Daemon crash #1770
Comments
@r2jitu thanks-- mind trying |
@whyrusleeping is this the go-msg thing? |
This is an interesting error... does your system have much ram? I believe i have seen similar issues on my 256MB ramnode vps. |
I tried 0.3.8-dev and it still crashed: https://gist.github.com/r2jitu/28f0e7391a86f2ed5784 I'm running on a shared machine, but I'm pretty sure my resource limits are quite high. What I did notice is that after a minute the number of ipfs threads goes from 11 to 125 and the virtual memory bloats to 1.8 GB and resident memory to 99 MB. These numbers keep growing until the program crashes. Since the error is with pthread_create, maybe too many native threads are being created and I'm running into a limit with that server? |
|
Here's a profile of what all the threads were doing at the time of crash:
There are 233 threads running syscall.EpollWait (https://github.com/jbenet/go-reuseport/blob/master/poll/poll_linux.go#L43)
According to this, each thread that is blocked on a syscall gets its own system thread, so even though GOMAXPROCS is set to 3 in main, it looks like an excessive number of system threads are created. |
Wow, I just read through the troubles you had to go through to get SO_REUSEPORT. I hope the Go devs update their net library. This issue is probably a duplicate of #1425. |
Yeah, this is definitely a reuseport issue. I'll prioritize the epoll fix for 0.3.9 |
We're using our own reuseport lib now. Closing (thanks @Kubuxu for going through that hell pit of code) |
The daemon repeatedly crashes every couple minutes while I'm running it on Linux x64 with the prebuilt version 0.3.7. I'm streaming a video from a remote server. I've captured the log of the crash here: https://gist.github.com/r2jitu/997311f4121f4af748a3
The text was updated successfully, but these errors were encountered: