[Linux] performance regression for disk_partitions

[gozdal]

Summary

• OS: Linux
• Architecture: 64bit
• Psutil version: 5.7.4
• Python version: 3.10
• Type: performance

Description

After #1863 disk_partitions is (or can be) much slower. When you strace vesion 5.7.3 you can see that it only reads local files (/proc/filesystems and /etc/mtab):

20583 open("/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
20583 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
20583 ioctl(3, TCGETS, 0x7ffdcc5ae6f0)  = -1 ENOTTY (Inappropriate ioctl for device)
20583 lseek(3, 0, SEEK_CUR)             = 0
20583 read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tr"..., 8192) = 426
20583 read(3, "", 8192)                 = 0
20583 close(3)                          = 0
20583 stat("/etc/mtab", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
20583 lstat("/etc", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
20583 lstat("/etc/mtab", {st_mode=S_IFLNK|0777, st_size=17, ...}) = 0
20583 readlink("/etc/mtab", "/proc/self/mounts", 4096) = 17
20583 lstat("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
20583 lstat("/proc/self", {st_mode=S_IFLNK|0777, st_size=0, ...}) = 0
20583 readlink("/proc/self", "20583", 4096) = 5
20583 lstat("/proc/20583", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
20583 lstat("/proc/20583/mounts", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
20583 open("/proc/20583/mounts", O_RDONLY|O_CLOEXEC) = 3
20583 futex(0x7f4e5e476570, FUTEX_WAKE_PRIVATE, 2147483647) = 0
20583 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
20583 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4e5f7b6000
20583 read(3, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024
20583 read(3, "net_prio,net_cls 0 0\ncgroup /sys"..., 1024) = 1024
20583 read(3, "\n/dev/sda2 /boot ext4 rw,relatim"..., 1024) = 1024
20583 read(3, "563268k,mode=700 0 0\n", 1024) = 21
20583 read(3, "", 1024)                 = 0
20583 close(3)                          = 0
20583 munmap(0x7f4e5f7b6000, 4096)      = 0
20583 write(1, "[sdiskpart(device='sysfs', mount"..., 5005) = 5005
20583 rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK, sa_restorer=0x7f4e5eb92630}, {sa_handler=0x7f4e5f0778fa, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK, sa_restorer=0x7f4e5eb92630}, 8) = 0
20583 munmap(0x7f4e5f63f000, 299008)    = 0
20583 exit_group(0)                     = ?
20583 +++ exited with 0 +++

However, with getting maxfile and maxpath in 5.7.4 calls statfs which may need to reach to e.g. NFS server. This makes the call potentially very long or even hang indefinitely in case of an unresponsive mount:

20194 open("/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
20194 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
20194 ioctl(3, TCGETS, 0x7fffdf04c8b0)  = -1 ENOTTY (Inappropriate ioctl for device)
20194 lseek(3, 0, SEEK_CUR)             = 0
20194 read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tr"..., 8192) = 426
20194 read(3, "", 8192)                 = 0
20194 close(3)                          = 0
20194 stat("/etc/mtab", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
20194 lstat("/etc", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
20194 lstat("/etc/mtab", {st_mode=S_IFLNK|0777, st_size=17, ...}) = 0
20194 readlink("/etc/mtab", "/proc/self/mounts", 4096) = 17
20194 lstat("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
20194 lstat("/proc/self", {st_mode=S_IFLNK|0777, st_size=0, ...}) = 0
20194 readlink("/proc/self", "20194", 4096) = 5
20194 lstat("/proc/20194", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
20194 lstat("/proc/20194/mounts", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
20194 open("/proc/20194/mounts", O_RDONLY|O_CLOEXEC) = 3
20194 futex(0x7eff15652570, FUTEX_WAKE_PRIVATE, 2147483647) = 0
20194 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
20194 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7eff16992000
20194 read(3, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024
20194 read(3, "net_prio,net_cls 0 0\ncgroup /sys"..., 1024) = 1024
20194 read(3, "\n/dev/sda2 /boot ext4 rw,relatim"..., 1024) = 969
20194 read(3, "", 1024)                 = 0
20194 close(3)                          = 0
20194 munmap(0x7eff16992000, 4096)      = 0
20194 statfs("/sys", {f_type=SYSFS_MAGIC, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
20194 statfs("/proc", {f_type=PROC_SUPER_MAGIC, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|S_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME})` = 0
...

[giampaolo]

maxfile and maxpath fields were remove from the namedtuple returned by disk_partitions() in #2405. This will be available in release 6.0.0.