Reading from /proc is slow for very large servers

[wastl]

When using Python's default open() on large servers, reading from e.g. /proc/net/tcp can be very slow. The reason is that Python tries to determine the buffer size based on the file system it is reading from. In case of /proc, this effectively turns off buffering, and as a consequence listing e.g. open connections can take a minute or more on large servers. A solution is to use io.open() and explicitly set a buffer size:

f = io.open(file, buffering=8192)

It might be worthwhile changing the linux implementation accordingly.

[giampaolo]

You can pass a buffering argument to open() as well: https://docs.python.org/2/library/functions.html#open.
Can you try and see what difference does it make?

[giampaolo]

OK I took a look at this and tried some benchmarks.
This is the diff:

diff --git a/psutil/_pslinux.py b/psutil/_pslinux.py
index 956e2be..4abe3f0 100644
--- a/psutil/_pslinux.py
+++ b/psutil/_pslinux.py
@@ -63,6 +63,7 @@ if HAS_PRLIMIT:
 CLOCK_TICKS = os.sysconf("SC_CLK_TCK")
 PAGESIZE = os.sysconf("SC_PAGE_SIZE")
 BOOT_TIME = None  # set later
+OPEN_BUFFERING = 8192
 if PY3:
     FS_ENCODING = sys.getfilesystemencoding()
 if enum is None:
@@ -126,7 +127,7 @@ TimeoutExpired = None
 # --- utils

 def open_binary(fname, **kwargs):
-    return open(fname, "rb", **kwargs)
+    return open(fname, "rb", buffering=OPEN_BUFFERING, **kwargs)


 def open_text(fname, **kwargs):
@@ -135,7 +136,7 @@ def open_text(fname, **kwargs):
     """
     if PY3 and 'encoding' not in kwargs:
         kwargs['encoding'] = FS_ENCODING
-    return open(fname, "rt", **kwargs)
+    return open(fname, "rt", buffering=OPEN_BUFFERING, **kwargs)


 def get_procfs_path():

...and this is the benchmark script I've been using:

import psutil
import time

def timethis(fun):
    t = time.time()
    for x in range(1000):
        fun()
    elapsed = time.time() - t
    try:
        name = fun.__func__.__name__
    except AttributeError:
        name = fun.func_name
    print("%-20s %.3f" % (name, elapsed))

psutil._psplatform.OPEN_BUFFERING = 8192
for x in range(2):
    print("\nwith buffer = %s" % psutil._psplatform.OPEN_BUFFERING)
    print("=" * 40)
    p = psutil.Process()
    started = time.time()
    timethis(p.name)
    timethis(p.cmdline)
    timethis(p.terminal)
    timethis(p.io_counters)
    timethis(p.cpu_times)
    timethis(p.create_time)
    timethis(p.memory_info_ex)
    # timethis(p.memory_maps)
    timethis(p.num_ctx_switches)
    timethis(p.num_threads)
    timethis(p.threads)
    timethis(p.status)
    timethis(p.open_files)
    timethis(p.connections)
    timethis(p.num_fds)
    timethis(p.ppid)
    timethis(p.uids)
    timethis(p.gids)
    timethis(psutil.virtual_memory)
    timethis(psutil.swap_memory)
    timethis(psutil.cpu_times)
    timethis(psutil.boot_time)
    # timethis(psutil.net_connections)
    psutil._psplatform.OPEN_BUFFERING = -1
    print("total time = %.3f" % (time.time() - started))

...and this is the result:

with buffer = 8192
========================================
name                 0.021
cmdline              0.013
terminal             0.030
io_counters          0.022
cpu_times            0.025
create_time          0.000
memory_info_ex       0.016
num_ctx_switches     0.038
num_threads          0.025
threads              0.034
status               0.021
open_files           0.039
connections          0.033
num_fds              0.007
ppid                 0.022
uids                 0.025
gids                 0.025
virtual_memory       0.036
swap_memory          0.052
cpu_times            0.553
boot_time            0.038
total time = 1.076

with buffer = -1
========================================
name                 0.026
cmdline              0.017
terminal             0.037
io_counters          0.024
cpu_times            0.032
create_time          0.000
memory_info_ex       0.022
num_ctx_switches     0.039
num_threads          0.027
threads              0.041
status               0.022
open_files           0.038
connections          0.032
num_fds              0.006
ppid                 0.024
uids                 0.027
gids                 0.027
virtual_memory       0.038
swap_memory          0.053
cpu_times            0.574
boot_time            0.041
total time = 1.149

Summary: on python 2 using buffering=8192 does not make any substantial difference. I've also tried to use io.open instead of open builtin and results are slower. @wastl do you have any comment on why you think setting a file buffering should result into a speedup?

[wastl]

This is specifically a problem of large servers with many open sockets. In my case, the servers that had problems had around 70000 open sockets. In this case, /proc/net/tcp will be several MB large when reading, and reading byte by byte is a significant performance problem.

The change I describe (setting buffer to 8192) solves the problem for our servers.

[giampaolo]

Have you tried my patch?

 def open_text(fname, **kwargs):
@@ -135,7 +136,7 @@ def open_text(fname, **kwargs):
     """
     if PY3 and 'encoding' not in kwargs:
         kwargs['encoding'] = FS_ENCODING
-    return open(fname, "rt", **kwargs)
+    return open(fname, "rt", buffering=OPEN_BUFFERING, **kwargs)

What kind of speedup do you have?

[giampaolo]

OK, I verified that in case of many connections we can have a substantial speedup. Benchmark script:

import psutil
import resource
import time
import socket

def create_sock():
    s = socket.socket()
    s.bind(('127.0.0.1', 0))
    s.listen(5)
    return s

resource.setrlimit(resource.RLIMIT_NOFILE, (65000, 65000))
socks = []
for x in range(20000):
    socks.append(create_sock())
t = time.time()
psutil.Process().connections()
print time.time() - t

Before the patch (a079e90):

~/svn/psutil {master}$ sudo python bench.py
1.58333420753

After the patch (a079e90):

~/svn/psutil {master}$ sudo python bench.py
0.420724868774

[giampaolo]

Note: the speedup only affects Python 2 (not 3).