Bug: Openvpn 2.6 causes high CPU usage #2313
Comments
|
@qdm12 is more or less the only maintainer of this project and works on it in his free time.
|
|
Same for me on latest Fedora. |
|
|
qdm12
changed the title
|
Hi, Thank you |
Do you also have the problem with v3.38.0? Anyway if Openvpn 2.6 is at fault, there isn't much I can do as far as I know, nothing changed except the openvpn version. Still a strange issue... It might be worth reporting it to the OpenVPN dev team? 🤔
I'm not sure I understand this fully, the CPU spikes to 100% for Gluetun only, or for the entire machine, and for how long? Does it happen only when querying |
v3.38.0 is slightly better than v3.37.0 when compared to memory usage. v3.38.0 was consuming about 56 MB where as v3.37.0 was consuming about 64 MB.
If OpenVPN version is the only problem, then it is supposed to be informed to them to resolve the issue.
I started the container with --cpus=1 options in 4 core machine. The CPU is 100% only for gluetun container displayed by |
|
In v3.37.0, I also observed that the default health checks were consuming more CPU. A 5-second interval ping is too aggressive for health checks. By disabling health checks and removing the exposed ports, CPU utilization has drastically reduced. I was only able to run 100 containers earlier, which reached about 90% CPU utilization. After removing health checks and deleting the ports, 100 containers are now using only about 10% CPU in total. This shows a significant variation in CPU usage with health checks enabled. Memory utilization has also dropped below 20 MB for each container after removing health checks and exposed ports. Thank you |
Do you mean by removing the |
qdm12
added
Priority: 💤 Low priority
Category: Label missing ❗
and removed
Category: OpenVPN 🔐
Priority: 🚨 Urgent
labels
|
So after some investigation on my side: the Docker healthcheck runs Gluetun in "health client mode", and this spikes the CPU usage because it loads the program (not even because of what the program actually does). For me it goes from 0% to 2% usage, so not a big deal. Possibly your CPU is low power hence this "100% usage" spike. If you want you can disable the docker healthcheck as you did. I tried running the healthcheck without any code (just Now, additional questions:
|
|
Thank you for your response. I have downloaded the Dockerfile, removed the health checks and exposed ports, and subsequently rebuilt the Docker image. From my observations, the primary contributor to CPU usage in version 3.37.0 is the health checks, as evidenced by the process statistics obtained from the The CPU that we can use in common to get the similar result is play with docker. It is available for free and I ran the commands on that host. It is possible that users who previously employed gluetun did not use it with a large number of containers, and therefore might not have noticed the increased CPU utilization. Below are the responses to your queries:
You may need to contact OpenVPN support to investigate why there is a fourfold increase in CPU usage compared to the previous version. Thank you |
Yes totally (my case too). I'm writing up a performance document focused on running many instances and how to reduce CPU and memory usage.
It's not the actual internal Gluetun health check and autohealing doing the TCP dialing every 5 seconds. That part uses pretty much zero CPU; it's running the Gluetun binary that uses CPU, I believe because it's a "fat" 23MB binary and that takes some processing to load. I even tried just doing "nothing" when running gluetun in "health client mode", and it would still use CPU every 5 seconds, showing it's not the actual code responsible for it, it's more the system loading the fat binary program to execute it. I might actually replace it with a shell script since it's specific to Docker, and that would remove the CPU usage spikes.
That's odd indeed. Can you try running v3.39.0 with openvpn 2.6 to see if it changes anything? v3.38.0 was running with |
|
Thank you for the response.
Thank you |
|
Oh my god, openvpn what the hell. This is under bandwidth stress right? And if you revert to using openvpn 2.5, it goes back to "normal" right? For the healthcheck, it's pretty much doomed: the CPU usage spikes to at least 0.9% when running any command, even |
qdm12
added
Category: OpenVPN 🔐
Status: 🟡 Nearly resolved
|
Hi, Please note there are existing issues concerning increased CPU consumption by health checks. For more information, please refer to the links below. One proposed solution is to disable health checks and implement retry mechanisms directly within the service.
Thank you |
|
I would like to provide an update regarding my previous comment. I have successfully downgraded the OpenVPN version from 2.6.11 to 2.5 on version 3.39.0, and observed that the CPU usage has returned to normal levels, consistent with those of the previous versions listed in the table. It appears that version 3.39.0 may pose challenges for users, as the default OpenVPN version 2.6.11 is exhibiting unusually high CPU consumption (160%) even under normal conditions without significant bandwidth stress. Please find the updated table below with the latest version.
In Internet Income script, I am utilizing version v3.37.0 and would prefer not to upgrade unless there are clear benefits compared to previous versions. I have provided all the necessary information. Please proceed with the next steps as required. Thank you |
Indeed, that's already done sort of 😉 The vpn gets auto healed within the container, and the internal healthcheck runs every period no matter what is defined in the Docker image.
Why not use v3.39 with
The odd thing is, on my machines (WSL host and arch linux host), running Gluetun v3.39 + Openvpn 2.6 (default) uses between 0% and 0.1% of CPU when idling. I'm starting to think this is related to your OS/kernel 🤔 Have you tried for example https://github.com/kylemanna/docker-openvpn to see if it makes a difference? Since it's using I'm trying to narrow down which part if at fault:
I'm inclined to think it's 1. right now 🤔 |
qdm12
added
the
Category: Documentation ✒️
|
Thank you for the response.
Yes, openvpn 2.5 with v3.39.0 can be added as the CPU Utilization is normal with this combination.
This is OpenVPN server and not client. I don't see options to use this docker image as openVPN client.
As previously noted, I am using play with docker, which is a free version anyone can use by one click sign-in. |
qdm12
added
Category: Performance 🚀
and removed
Category: Label missing ❗
Status: 🗯️ Waiting for feedback
labels
Is this urgent?
None
Host OS
Ubuntu 22
CPU arch
x86_64
VPN service provider
OPENVPN
What are you using to run the container
docker run
What is the version of Gluetun
Running version latest built on 2024-05-18T18:08:57.405Z (commit 4218dba)
What's the problem 🤔
When the docker image qmcgaw/gluetun is used. The CPU utilization of the container goes to about 100% of 1 CORE where as the following image qmcgaw/gluetun:v3.37.0 uses less than 1% of 1 CORE.
Could you please look into the latest version and compare it with v3.37.0.
Share your logs (at least 10 lines)
Share your configuration
-e LOG_LEVEL=debug -e VPN_SERVICE_PROVIDER=custom -e VPN_TYPE=openvpn -e OPENVPN_USER=vpn -e OPENVPN_PASSWORD=vpn -v $volume -e OPENVPN_CUSTOM_CONFIG=/gluetun/custom.conf -v '/dev/net/tun:/dev/net/tun' --cap-add=NET_ADMIN -e DOT=off -e DOT_PROVIDERS=google,cloudflare -e DOT_CACHING=off -e BLOCK_MALICIOUS=off qmcgaw/gluetunThe text was updated successfully, but these errors were encountered: