Gitea 1.20 leaking resources #26442
Labels
Comments
It's not WebSocket, it's still EventSource 😆. But yes I could see that this has potential to consume many resources, thought I don't recall it having changed any lately. EventSource code should be pretty much unchanged since at least 2-3 minor gitea versions. |
|
Yes, it's a wild guess, hard to tell what's the real source for the problem but that what helped. There was no such problems with 1.19, nothing else has changed - nor user count or nor behavior |
|
Might need to pprof it. Maybe we already expose a endpoint for it? If not, that might be a nice opt-in debug feature, chi has a built-in middleware for it. |
|
I also found a similar problem when closing Gitea |
|
even after disabling |
|
The error about pthread_create failed: Resource temporarily unavailable normally means that you are trying to create too many threads. Take a look at how many thread numbers are allowed for this linux machine cat /proc/sys/kernel/threads-maxAnd change it to a greater number if the memory is enough. You could also have a monitor record for the thread's number. |
|
already checked that, does not seems anything out of ordinary (but we will check this when everything goes to hell again): $ cat /proc/sys/kernel/threads-max
127248
# Gitea process thead count:
$ ps -eLf | egrep ":[0-9][0-9] /usr/local/bin/gitea" | wc -l | sed -e 's/^ *//'
65
# All processes thread count:
$ ps -eo nlwp | tail -n +2 | awk '{ num_threads += $1 } END { print num_threads }'
351 |
|
Maybe you can use Prometheus & Grafana to track the thread number. |
|
will try to enable |
|
Problem happened again sooner than expected. Thread/process count is not the problem, memory usage is Looks like on push on ~600MB repo gitea process memory usage spikes at least +4GB and it does not recover from that and only restart of gitea process helps:
|
|
A pprof snapshot from admin ui should give some info on the memory allocations. |
|
admin ui pprof allows downloading only cpu profile but not memory profile |
|
Not sure what admin pprof does, but this should work: gitea/custom/conf/app.example.ini Lines 289 to 295 in e5ec57c |
|
thing is this that it's not easily reproducible and I don't know what impact enabling this is on production system as it could take days/weeks before problem happens |
|
Not an expert but I think the pprof webserver has no real perf impact until you actually take snapshots. |
|
I couldn't reproduce this on my local machine. I have tested main and release/v1.20 for pushing linux repository via http/ssh. The memory has almost no change when pushing. Do you have any git hooks or web hooks on that repository? |
|
it has only woodpecker and slack/mattermost webhooks, no special githooks used on this instance. full text code search is enabled on the server (elastic) but it has been enabled for at least year already and on 1.19 there was no issues with that |
|
Maybe you can try to disable code indexer temporarily if possible for some days to take a look whether it's the problem. |
|
Feel free to provide more clues by #28596 (and it has been backported to 1.21) |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Description
After upgrade from 1.19 to 1.20 we noticed that at one point (about twice a day) Gitea would become unstable and would not accept git push anymore and would fail with:
We could not pinpoint exact cause of what is reason for this as in one point it was that it can not create process that was failing at linux syscall level.
Our temporary fix to get it stable again was by disabling
/user/eventswebsocket endpoint at reverse proxy level. This does help but of course on degradation of UX.At peak times we have about 50 users using it simultaneously so it's not that high number.
If anyone has any ideas on where to look to pinpoint the source of problem let me know :)
Gitea Version
1.20
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
2.25.1
Operating System
Ubuntu 20.04
How are you running Gitea?
From binary using systemd
Database
PostgreSQL
The text was updated successfully, but these errors were encountered: