gincoind - high CPU usage

[bkostjens]

Im having an issue with gincoind (v. 1.1.0), both on linux as well as on macOS. On both platforms CPU usage is abnormal high, compared to other coin daemons. The described issue is happening with both the precompiled as well as self compiled version of the daemon.

I will post all details I got so far, if more information is needed, please let me know.

Environment 1:

VULTR $20 Ubuntu VM - 2 vCores
Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-127-generic x86_64)

Top output:

top - 08:30:27 up 38 days, 33 min,  1 user,  load average: 2.43, 2.16, 2.06
Tasks: 136 total,   1 running, 135 sleeping,   0 stopped,   0 zombie
%Cpu(s): 11.3 us, 12.1 sy, 46.0 ni, 27.0 id,  0.0 wa,  0.0 hi,  3.5 si,  0.2 st
KiB Mem :  4046296 total,   125952 free,  3603124 used,   317220 buff/cache
KiB Swap:  4194300 total,  1887536 free,  2306764 used.   190992 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND    
  553 gin       20   0 2015168 872792   5308 S  99.0 21.6   2250:29 gincoind   
22447 lpc       20   0 1507900 251968  37860 S   5.6  6.2   2586:51 lightpayco+
 1687 cfl       20   0 1282932 146728   5048 S   4.0  3.6   1450:36 cryptoflowd
31528 gic       20   0 1383164 100788   2356 S   4.0  2.5 519:32.40 giantd     
22004 lpc4      20   0 1370364 231500  36516 S   3.7  5.7   1672:35 lightpayco+
 1692 trtt      20   0 1734088 223540   6944 S   3.3  5.5   1320:45 trittiumd  
22489 lpc2      20   0 1509376 204572  17904 S   2.7  5.1   1755:44 lightpayco+
21936 lpc3      20   0 1333024 216620  26116 S   2.3  5.4   1603:39 lightpayco+
 1688 cfl10     20   0 1180400 127692   3044 S   2.0  3.2 694:28.27 cryptoflowd
 1838 cfl7      20   0 1309184 130116   3768 S   2.0  3.2 752:39.95 cryptoflowd
 1738 cfl3      20   0 1277744 128624   4148 S   1.7  3.2 709:46.51 cryptoflowd
 1785 cfl5      20   0 1280580 128996   4308 S   1.7  3.2 734:04.37 cryptoflowd
 1869 cfl8      20   0 1184388 129808   3340 S   1.7  3.2 714:21.44 cryptoflowd
 1962 cfl9      20   0 1190848 129124   4404 S   1.7  3.2 725:40.44 cryptoflowd
 1709 nix       20   0 1412052 195220   1512 S   1.3  4.8 546:41.94 nixd       
 1718 cfl2      20   0 1182092 129860   4268 S   1.3  3.2 688:15.50 cryptoflowd
 1766 cfl4      20   0 1184712 131820   5560 S   1.3  3.3 711:24.81 cryptoflowd
 1812 cfl6      20   0 1209256 128156   3704 S   1.3  3.2 687:02.77 cryptoflowd
31736 owo       20   0 1223452  50816   2092 S   1.0  1.3  76:13.69 oneworldd  
   13 root      20   0       0      0      0 S   0.7  0.0  33:40.94 ksoftirqd/1

CPU Info:

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Virtual CPU 82d9ed4018dd
stepping	: 4
microcode	: 0x1
cpu MHz		: 2593.904
cache size	: 16384 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f clwb avx512cd xsaveopt arat
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 5187.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Virtual CPU 82d9ed4018dd
stepping	: 4
microcode	: 0x1
cpu MHz		: 2593.904
cache size	: 16384 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f clwb avx512cd xsaveopt arat
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 5187.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

strace of gincoind:

strace of nixd (different coin daemon):

Environment 2:

Mac Mini (late 2012)

I'm happy to provide additional information when needed.

[bedeabza]

The new release 1.2.0.0 should solve the performance issues you experienced. Can you update and confirm?

Thank you!

[bkostjens]

I just updated, but the problem remains the same.

gin@MN1:~$ ./gincoin-cli --version
Gincoin Core RPC client version v1.2.0.0-18c9605

top - 19:01:11 up 50 days, 11:04,  1 user,  load average: 2.36, 2.38, 2.07
Tasks: 139 total,   1 running, 138 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.8 us, 10.6 sy, 36.1 ni, 42.3 id,  0.0 wa,  0.0 hi,  3.0 si,  0.2 st
KiB Mem :  4046296 total,   112976 free,  3161912 used,   771408 buff/cache
KiB Swap:  4194300 total,  1374912 free,  2819388 used.   626108 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND    
  790 gin       20   0 1570260 502692  61120 S  76.4 12.4  20:04.25 gincoind   
20882 lpc       20   0 1530008 199324  37960 S   6.0  4.9 671:09.54 lightpayco+
31528 gic       20   0 1378776 112252   5188 S   3.3  2.8   1260:38 giantd     
 1962 cfl9      20   0 1179348 155588  27672 S   3.0  3.8   1011:12 cryptoflowd
21054 lpc2      20   0 1477200 156132  16892 S   3.0  3.9 410:59.50 lightpayco+

[bkostjens]

I just checked on MacOS as well. There the problem is gone. Only on Linux the problem is still there.

[bkostjens]

Guys, I really want to buy some more nodes, but this issue is stopping me from doing so. Has there been any progress on debugging this problem?

[tts1521]

This seems to happen when the daemon has to many connections to the network ( > 100 ), case in which
is not an issue, it's actual working load.

[bkostjens]

Maybe include a lower max connections config setting in future releases? Or adjust the default to 64 instead of 256?

[bedeabza]

Masternodes are meant to have a good connectivity, implicitly a higher number of connection slots available.

If you need to configure a less demanding node simply add the maxconnections=64 to your config and should be all set.