[BUG] [report v2019.2.3, v3000] 25% memory usage on Raspberry Pi (Ubuntu 64bit)

[sjkummer]

Description of Issue

I am running the latest release v2019.2.3 on a Raspberry Pi 3 B and it uses nearly 25% of the memory (250MB of 1GB). There are no running task - the minion is "idle".

Why is so much memory needed?

Setup

Saltstack minion on a Raspberry Pi 3 B V1.2

Versions Report

Salt Version:
Salt: 2019.2.3

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: 0.31.0
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.17 (default, Nov 7 2019, 10:07:09)
python-gnupg: Not Installed
PyYAML: 5.1.2
PyZMQ: 17.1.2
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 5.1.1
ZMQ: 4.3.2

System Versions:
dist: Ubuntu 19.10 eoan
locale: UTF-8
machine: aarch64
release: 5.3.0-1007-raspi2
system: Linux
version: Ubuntu 19.10 eoan

[sjkummer]

[waynew]

That is a surprising amount of memory. You might see if you can update the number of worker processes to just 1.

I've used minions and masters on my pi before - the only real problem that I saw was that occasionally it would get into a runaway loop that would start eating all my memory, but that's a different issue.

[sjkummer]

Hello Wayne, thanks for your hints!

I have tried to disable multithreading as follows, but now effect. I still got 3 minion processes. Maybe, it's a bug in the current release 🤭

multiprocessing: False

process_count_max: 1

reactor_worker_threads: 1

This runaway loop sound bad 😕did you find a solution for this? ulimit?

[sjkummer]

Upgrading to version 3000 did not make a difference.
The multithreading settings are reset after ever service restart, see #56245

[max-arnold]

Try disabling some execution modules and grains using the disable_modules and disable_grains minion settings respectively.

You can find the list of loaded modules using the following command:

salt RASPBERRYPI sys.list_modules

Then add the modules you don't need to the minion config:

disable_modules:
  - consul
  - glassfish
  - grafana4
  - jboss7
  - ...
  - vsphere

As for grains, you can find the full list of grains modules here, and also disable the ones you don't need:

disable_grains:
  - chronos
  - cimc
  - esxi
  - fibre_channel
  - fx2
  - iscsi
  - junos
  - marathon
  - mdadm
  - mdata
  - metadata
  - napalm
  - nvme
  - nxos
  - panos
  - philips_hue
  - rest_sample
  - smartos
  - ssh_sample
  - zfs

This can also considerably speed up the minion response time: #48773 (comment)

[sjkummer]

Thanks for your further hints!

I tried to disable as many modules as possible but it did not make a difference ... now the three salt-minion processes use 27% of the memory 🙈

My config is as follows now:

master: xxx

multiprocessing: False
process_count_max: 1
reactor_worker_threads: 1

disable_modules:
    - ansible
    - artifactory
    - bigip
    - btrfs
    - buildout
    - composer
    - consul
    - cpan
    - cryptdev
    - data
    - defaults
    - devmap
    - dig
    - django
    - dnsmasq
    - dnsutil
    - drbd
    - event
    - extfs
    - gem
    - genesis
    - git
    - glassfish
    - google_chat
    - grafana4
    - grains
    - hashutil
    - highstate_doc
    - hipchat
    - hosts
    - http
    - incron
    - ini
    - inspector
    - introspect
    - iosconfig
    - ip
    - iptables
    - jboss7
    - jboss7_cli
    - k8s
    - kernelpkg
    - key
    - keyboard
    - kmod
    - locale
    - locate
    - log
    - logrotate
    - lowpkg
    - lvm
    - mandrill
    - match
    - mattermost
    - mine
    - minion
    - modjk
    - mount
    - msteams
    - nagios
    - nagios_rpc
    - namecheap_domains
    - namecheap_domains_dns
    - namecheap_domains_ns
    - namecheap_ssl
    - namecheap_users
    - network
    - nexus
    - nginx
    - nova
    - npm
    - nspawn
    - nxos_api
    - openscap
    - openstack_config
    - opsgenie
    - out
    - pagerduty
    - pagerduty_util
    - pam
    - parallels
    - partition
    - peeringdb
    - pillar
    - pip
    - pkg
    - pkg_resource
    - publish
    - pushover
    - pyenv
    - raid
    - random
    - random_org
    - rbenv
    - rest_sample_utils
    - ret
    - rsync
    - rvm
    - s3
    - s6
    - salt_proxy
    - saltcheck
    - saltutil
    - schedule
    - scsi
    - sdb
    - seed
    - serverdensity_device
    - service
    - shadow
    - slack
    - slsutil
    - smbios
    - smtp
    - solrcloud
    - sqlite3
    - ssh
    - state
    - status
    - statuspage
    - supervisord
    - sys
    - sysctl
    - sysfs
    - syslog_ng
    - system
    - telegram
    - telemetry
    - temp
    - test
    - timezone
    - tls
    - travisci
    - udev
    - uptime
    - user
    - vault
    - vbox_guest
    - virtualenv
    - vsphere
    - x509
    - xfs
    - zabbix
    - zenoss

disable_grains:
  - chronos
  - cimc
  - esxi
  - fibre_channel
  - fx2
  - iscsi
  - junos
  - marathon
  - mdadm
  - mdata
  - metadata
  - napalm
  - nvme
  - nxos
  - panos
  - philips_hue
  - rest_sample
  - smartos
  - ssh_sample
  - zfs
  

[4815162342lost]

Sorry, small offtop: is progress bar on your screen broken (for top)?

Why do we see 60.9/908.2 MB?

[sjkummer]

@4815162342lost
Sorry, I just toggled the summary display mode (m) ... making the "used" percentage visible

[max-arnold]

@sjkummer You disabled too many modules. You'll probably want to run things like state.apply, test.ping, or grains.get. This means you need to enable grains, pillar, pkg, saltutil, service, state, sys, system, test, user and probably other modules. Check the minion log for any errors to see what else is required.

As for 27%, I'm quite surprised. The memory usage should go down. I'll try it on my Pi in a few days.

[sjkummer]

Btw. I also switched from python 2.7 to python 3 ... but no change

[max-arnold]

Ok, it looks like I actually have a PI I can access, and below are my results (Raspbian Buster Lite, Salt 3000 Py3):

Default install

Measurements right after the sudo salt raspberrypi sys.list_modules command was finished.

$ sudo top -p $(pgrep salt-minion | paste -s -d, )
MiB Mem :    926.1 total,    422.4 free,     99.5 used,    404.1 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.    760.2 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 5082 root      20   0   38640  23792  10724 S   0.0   2.5   0:01.33 salt-minion
 5087 root      20   0  100420  38156  11640 S   0.0   4.0   0:22.09 salt-minion
 5089 root      20   0   49148  17872   3836 S   0.0   1.9   0:00.00 salt-minion

With disabled modules/grains

/etc/salt/minion.d/disable.conf

disable_grains:
  - chronos
  - cimc
  - esxi
  - fibre_channel
  - fx2
  - iscsi
  - junos
  - marathon
  - mdadm
  - mdata
  - metadata
  - napalm
  - nvme
  - nxos
  - panos
  - philips_hue
  - rest_sample
  - smartos
  - ssh_sample
  - zfs

disable_modules:
    - aliases
    - alternatives
    - ansible
    - artifactory
    - bigip
    - btrfs
    - buildout
    - chroot
    - cloud
    - composer
    - consul
    - container_resource
    - cpan
    - cryptdev
    - data
    - dig
    - django
    - dnsmasq
    - drbd
    - etcd
    - ethtool
    - extfs
    - freezer
    - gem
    - genesis
    - git
    - glassfish
    - google_chat
    - grafana4
    - highstate_doc
    - hipchat
    - incron
    - ini
    - inspector
    - introspect
    - iosconfig
    - iptables
    - jboss7
    - jboss7_cli
    - k8s
    - key
    - keyboard
    - kmod
    - locate
    - logrotate
    - lvm
    - mandrill
    - mattermost
    - modjk
    - msteams
    - nagios
    - nagios_rpc
    - namecheap_domains
    - namecheap_domains_dns
    - namecheap_domains_ns
    - namecheap_ssl
    - namecheap_users
    - nexus
    - nfs3
    - nginx
    - nova
    - npm
    - nspawn
    - nxos_api
    - openscap
    - openstack_config
    - opsgenie
    - pagerduty
    - pagerduty_util
    - parallels
    - peeringdb
    - publish
    - pushover
    - pyenv
    - raid
    - random
    - random_org
    - rbenv
    - rest_sample_utils
    - rsync
    - rvm
    - s3
    - s6
    - salt_proxy
    - saltcheck
    - scsi
    - sdb
    - seed
    - serverdensity_device
    - slack
    - smbios
    - smtp
    - solrcloud
    - sqlite3
    - ssh
    - statuspage
    - supervisord
    - syslog_ng
    - telegram
    - telemetry
    - tls
    - travisci
    - vault
    - vbox_guest
    - virtualenv
    - vsphere
    - x509
    - xml
    - xfs
    - zabbix
    - zenoss

Measurements right after the sudo salt raspberrypi sys.list_modules command was finished.

$ sudo top -p $(pgrep salt-minion | paste -s -d, )
MiB Mem :    926.1 total,    424.9 free,     96.9 used,    404.2 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.    762.9 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 5303 root      20   0   97268  35300  11604 S   0.3   3.7   0:03.46 salt-minion
 5298 root      20   0   38640  23880  10812 S   0.0   2.5   0:01.31 salt-minion
 5305 root      20   0   49148  17900   3840 S   0.0   1.9   0:00.00 salt-minion

So, the RES memory usage is a bit lower:

In [1]: (23792 + 38156 + 17872) - (35300 + 23880 + 17900)
Out[1]: 2740

And the minion response time is significantly reduced.

What I don't understand is why the RES memory consumption is much higher on your screenshots. Maybe Ubuntu 19.10 is less optimized for PI than Raspbian Buster?

[sjkummer]

Hey @max-arnold thanks for your reproduction! That is really interesting. I now disabled exactly the same list, but I still got the same high memory usage.

This really could be related to the Ubuntu 64bit environment.

I also had to install salt using git sudo sh bootstrap-salt.sh -x python3 git v3000 (the python version does not make a difference).

Just one thing: I double checked the modules using sys.list_modules and even tough most modules are correctly disabled, some specific like ansible are still listet. I set that list in both /etc/salt/minion.d/disable.conf and /etc/salt/minion ... is there something wrong with my setup?

[max-arnold]

Just one thing: I double checked the modules using sys.list_modules and even tough most modules are correctly disabled, some specific like ansible are still listet. I set that list in both /etc/salt/minion.d/disable.conf and /etc/salt/minion ... is there something wrong with my setup?

I see the same thing. It happens because sys.list_modules lists virtual module names, and disable_modules expects file names (without the .py extension). So, to really disable the ansible module you need to specify ansiblegate.

/etc/salt/minion.d/disable.conf

disable_grains:
  - chronos
  - cimc
  - esxi
  - fibre_channel
  - fx2
  - iscsi
  - junos
  - marathon
  - mdadm
  - mdata
  - metadata
  - napalm
  - nvme
  - nxos
  - panos
  - philips_hue
  - rest_sample
  - smartos
  - ssh_sample
  - zfs

disable_modules:
    - aliases
    - alternatives
    - ansiblegate
    - artifactory
    - bigip
    - btrfs
    - zcbuildout
    - chroot
    - cloud
    - composer
    - consul
    - container_resource
    - cpan
    - cryptdev
    - data
    - dig
    - djangomod
    - dnsmasq
    - drbd
    - etcd_mod
    - ethtool
    - extfs
    - freezer
    - gem
    - genesis
    - git
    - glassfish
    - google_chat
    - grafana4
    - highstate_doc
    - hipchat
    - incron
    - ini_manage
    - inspector
    - introspect
    - iosconfig
    - iptables
    - iwtools
    - jboss7
    - jboss7_cli
    - k8s
    - key
    - keyboard
    - kmod
    - locate
    - logrotate
    - lvm
    - mandrill
    - mattermost
    - modjk
    - msteams
    - nagios
    - nagios_rpc
    - namecheap_domains
    - namecheap_domains_dns
    - namecheap_domains_ns
    - namecheap_ssl
    - namecheap_users
    - nexus
    - nfs3
    - nginx
    - nova
    - npm
    - nspawn
    - nxos_api
    - openscap
    - openstack_config
    - opsgenie
    - pagerduty
    - pagerduty_util
    - parallels
    - peeringdb
    - publish
    - pushover_notify
    - pyenv
    - raid
    - random
    - random_org
    - rbenv
    - rest_sample_utils
    - rsync
    - rvm
    - s3
    - s6
    - salt_proxy
    - saltcheck
    - scsi
    - sdb
    - seed
    - serverdensity_device
    - slack_notify
    - smbios
    - smtp
    - solrcloud
    - sqlite3
    - ssh
    - statuspage
    - supervisord
    - syslog_ng
    - telegram
    - telemetry
    - tls
    - travisci
    - vault
    - vbox_guest
    - virtualenv_mod
    - vsphere
    - x509
    - xml
    - xfs
    - zabbix
    - zenoss

[max-arnold]

And I guess you are right - the increased memory consumption is caused by 64 bit arch.

salt-call --versions-report

Salt Version:
           Salt: 3000

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Dec 20 2019, 18:57:59)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist: debian 10.3
         locale: UTF-8
        machine: armv7l
        release: 4.19.97-v7+
         system: Linux
        version: debian 10.3

[waynew]

@max-arnold so you're not seeing the same amount of memory usage?

[max-arnold]

@waynew Yes, that is correct. Earlier I posted all the stats for a 32bit architecture.

[waynew]

@sjkummer Running on a pi2 with raspbian 10 only used about 50M with the default 3000 install. It does seem possible that the 64-bit nature is what's causing increased memory consumption. Unfortunately I don't have access to a 64-bit pi right now.

[sagetherage]

trying to move this forward, I will need to follow up tomorrow or Monday.

[sagetherage]

@sjkummer or @max-arnold are either of you able to open a PR for this? This likely isn't something the Open Core Team can focus on right now, but we can keep it open for the community to pick up. I will leave this as-is and follow up again the next business day.

[max-arnold]

I have no idea how to reduce memory usage (other than switching to a 32-bit OS), so I can't provide a PR

[sjkummer]

Neither do I, sorry.

Update
When using the latest salt version v3001 on Ubuntu 20.04 LTS (arm64), the memory usage looks much better. It's about half as much as before (~100MB)

[sagetherage]

putting this into Blocked for now, but have a reminder set for myself to come back to this ticket this week.

[sagetherage]

this needs a 64-bit RaspberryPi to test, I will attempt to assign this out to get it unblocked.

[sagetherage]

We are able to test this now and may be that we have fixed in v3003.1. Putting this back through triage to test.

[garethgreenaway]

The Silicon release of Salt will have official Ubuntu ARM 64bit packages for LTS releases and we will be testing this OS as part of the testing pipeline. Based on the comments that memory usage was significantly reduced in 3001, we'll go ahead and close this one out. Please feel free to open a new issue if the problem persists.