RPC threads dying w/ "Too many open files" #9102
Comments
XertroV
changed the title
|
Is it something that you were able to reproduce? Any way to give us more logs ( |
Tbaut
added
F2-bug 🐞
|
@Tbaut - not sure how to reproduce except to leave nodes running long term. Will add As an aside, I've looked before for how to increase the log level and it wasn't easy to find (well, at least when I looked I didn't find anything). Might be worth adding a wiki page on debugging / increased logging. @ascjones I think the above issues were different for a few reasons:
So unless something particularly significant changed between 1.10.6 and 1.10.9 I suspect it's a different issue. |
|
Happened to another node a few hours ago - just checked. A thing I noticed: There are ~5-6 minutes between those two log lines. 🤔 |
What keyword did you use? it's here: https://wiki.parity.io/FAQ.html#how-can-i-make-parity-to-write-logs |
I think I must have just derped pretty hard. Some vague memories: might have been before the current wiki (was it on github before?) and I think I used google more than searching a particular site. In any case thanks for linking me. |
|
@Tbaut - so good news! 2 of my nodes seem to have lost RPC again. I'll try and get those logs up later today (though they might be large) and filter through for anything relevant I can find. |
|
okay so seems like they actually didn't lose RPC - was just that the log files clobbered the little AWS |
|
@XertroV are you using rpc over http or ipc? |
|
@debris HTTP - the nodes are all remote (publicly accessible) and don't run anything else on them besides stats, healthcheck, etc. Didn't check if IPC was affected, and haven't been able to replicate. Actually just disabled debug logging because the log files were about 20GB |
|
RPC in one of our parity nodes stopped working for us a few hours ago. The following log entry lines up perfectly with our software being unable to talk to parity: We had never seen this issue before and not entirely sure that it is related to @XertroV's problems but wanted to post it here. This node had been running for a month: We're currently upgrading to v1.11.7 and will restart our node. |
|
Facing this issue in v1.11.7(latest stable version). As is the case of other users, I too am running parity along with its monitoring via pm2 using eth-net-intelligenceAPI. So they are the onlyRPC connections to my parity node. |
|
@Tbaut we're still experiencing this problem. We're doing stress tests over IPC, but sometimes need to check something over HTTP, and then we discover that RPC won't respond. Our version is v2.0.7-stable-db3a989-20181010/x86_64-linux-gnu/rustc1.29.0 Here's a piece of the log. You can see that there was only one RPC connection just before the server stopped responding. |
|
The PR that fixes the problem (paritytech/jsonrpc#321) has not been used in any Parity-Ethereum release yet. You can see in the cargo.toml that the jsonrpc branch used is still old. You need to wait until this is included before testing. |
|
Yes, the list of commits from jsonrpc v1.11 branch includes the fix, this same branch was used and back ported to stable. |
and others
pretty self explanatory - all RCP threads die simultaneously, but everything else keeps working (including websockets according to netstat; RPC ports disappear but websocket ports don't)
(Most of those requests are 2x eth-stats running via pm2)
Logs:
The text was updated successfully, but these errors were encountered: