Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

memory not release #453

Closed
lixianyang opened this issue Aug 1, 2018 · 20 comments
Closed

memory not release #453

lixianyang opened this issue Aug 1, 2018 · 20 comments
Labels
level:Hard

Comments

@lixianyang
Copy link

lixianyang commented Aug 1, 2018 •
edited
Loading

Problem

I use fluentd in docker with memory limit. I test it some times. The process's memory increase until kernel kill the worker0 and restart.

2018-08-01 02:48:04 +0000 [info]: Worker 0 finished unexpectedly with signal SIGKILL

test:
a script write logs to fluentd (4000/s)

Steps to replicate

Dockerfile

FROM xx.xxxxx.cn/library/fluentd:1.2.2-debian
RUN fluent-gem install fluent-plugin-elasticsearch

pmap output

Expected Behavior or What you need to ask

memory decrease when no log write to es

Using Fluentd and ES plugin versions

  • OS version:Linux 3.10.0-514.21.1.el7.x86_64 centos 7.2
  • Fluentd: v1.2.2
  • ES plugin: 2.11.3
  • gem list:

bigdecimal (1.2.8)
cool.io (1.5.3)
did_you_mean (1.0.0)
dig_rb (1.0.1)
elasticsearch (6.1.0)
elasticsearch-api (6.1.0)
elasticsearch-transport (6.1.0)
excon (0.62.0)
faraday (0.15.2)
fluent-plugin-elasticsearch (2.11.3)
fluentd (1.2.2)
http_parser.rb (0.6.0)
io-console (0.4.5)
json (2.1.0, 1.8.3)
minitest (5.9.0)
msgpack (1.2.4)
multi_json (1.13.1)
multipart-post (2.0.0)
net-telnet (0.1.1)
oj (3.3.10)
power_assert (0.2.7)
psych (2.1.0)
rake (10.5.0)
rdoc (4.2.1)
serverengine (2.0.6)
sigdump (0.2.4)
strptime (0.2.3)
test-unit (3.1.7)
thread_safe (0.3.6)
tzinfo (1.2.5)
tzinfo-data (1.2018.5)
yajl-ruby (1.4.0)

  • ES version: 5.6.9
@lixianyang
Copy link
Author

match config

    @type elasticsearch
    hosts "#{ENV['ES_HOSTS']}"
    index_name xxxxx
    type_name xxxxx
    ssl_verify false
    logstash_format true    
    logstash_prefix xxxxx
    logstash_dateformat %Y%m
    utc_index false
    time_key time
    time_key_exclude_timestamp true
    include_tag_key false
    reload_connections false
    reconnect_on_error true
    template_name xxxxxxxx
    template_file /fluentd/etc/xxxxxxxx.template
    slow_flush_log_threshold 60.0
    request_timeout 60s
    
    <secondary>
        @type file
        path /xx/xx/xx
    </secondary>
    
    <buffer>
        @type file
        path /xx/xx/xx-buffer
        chunk_limit_size 64MB
        total_limit_size 64GB
        flush_mode interval
        flush_interval 5s
        flush_thread_count 4
        
        # retry config
        retry_type periodic
        retry_wait 30
        retry_timeout 24h
        retry_secondary_threshold 0.5
    </buffer>

@cosmo0920
Copy link
Collaborator

cosmo0920 commented Aug 1, 2018 •
edited
Loading

Similar issue is reported on Fluentd core upstream:
fluent/fluentd#1941

@repeatedly
Copy link
Contributor

@lixianyang Could you paste your whole configuration?
If you use other output plugin, memory limit issue doesn't happen, right?

@lixianyang
Copy link
Author

lixianyang commented Aug 2, 2018 •
edited
Loading

@repeatedly Yes, when I use out_file, the memory aways around 50M.
out_file config:

<source>
    @type forward
    port 24224
    bind 0.0.0.0
</source>

<filter docker.**>
    @type record_transformer
    enable_ruby true
    <record>
        time    ${time.strftime('%Y-%m-%dT%H:%M:%S')}
        usec    ${(Time.now.to_f * 1000000).to_i}
    </record>
</filter>

<match docker.**>
    @type file
    path /home/fluent/file-test
    <buffer>
        @type file
        path /home/fluent/paaslog-buffer
        chunk_limit_size 64MB
        total_limit_size 64GB
        flush_mode interval
        flush_interval 5s
        flush_thread_count 4

        # retry config
        retry_type periodic
        retry_wait 30
        retry_timeout 24h
        retry_secondary_threshold 0.5
    </buffer>
</match>

out_es config:

<source>
    @type forward
    port 24224
    bind 0.0.0.0
</source>
<filter docker.**>
    @type record_transformer
    enable_ruby  true
    <record>
        time    ${time.strftime('%Y-%m-%dT%H:%M:%S')}
        usec    ${(Time.now.to_f * 1000000).to_i}
    </record>
</filter>
<match docker.**>
    @type elasticsearch
    hosts "#{ENV['ES_HOSTS']}"
    user elastic
    password changeme
    index_name testlog
    type_name testlog
    ssl_verify false
    logstash_format true    
    logstash_prefix testlog
    logstash_dateformat %Y%m
    utc_index false
    time_key time
    time_key_exclude_timestamp true
    include_tag_key false
    reload_connections false
    reconnect_on_error true
    template_name testlog
    template_file /fluentd/etc/testlog.template
    slow_flush_log_threshold 60.0
    request_timeout 60s
    
    <secondary>
        @type file
        path /home/fluent/testlog
    </secondary>
    
    <buffer>
        @type file
        path /home/fluent/testlog-buffer
        chunk_limit_size 64MB
        total_limit_size 64GB
        flush_mode interval
        flush_interval 5s
        flush_thread_count 4
        
        # retry config
        retry_type periodic
        retry_wait 30
        retry_timeout 24h
        retry_secondary_threshold 0.5
    </buffer>
</match>

testlog.template

{
    "template" : "testlog-*",
        "settings": {
            "index.number_of_shards": 18,
            "index.refresh_interval": "3s",
            "index.query.default_field": "querystring"
        },
        "mappings": {
            "testlog": {
                "_source": { "enabled": true },
                "properties" : {
                    "log": { "type": "text"},
                    "appid": { "type": "keyword"},
                    "source": { "type": "keyword" },
                    "container_name": { "type": "keyword"},
                    "container_id": { "type": "keyword" },
                    "time": { "type": "date", "format": "strict_date_optional_time||epoch_millis"},
                    "usec": {"type": "long"},
                    "HOST": { "type": "keyword" },
                    "PORT": { "type": "keyword"}
                }
            }
        }
}

@lixianyang
Copy link
Author

out_file test pmap -d $pid:(write more than es test log)
26007: /usr/bin/ruby2.3 -Eascii-8bit:ascii-8bit /usr/local/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins --under-supervisor
000056009b9a2000 4K r-x-- ruby2.3
000056009bba2000 4K r---- ruby2.3
000056009bba3000 4K rw--- ruby2.3
00007f8e95600000 30720K rw--- [ anon ]
00007f8e975c3000 24K r-x-- pathname.so
00007f8e975c9000 2044K ----- pathname.so
00007f8e977c8000 4K r---- pathname.so
00007f8e977c9000 4K rw--- pathname.so
00007f8e977ca000 4K r-x-- sha1.so
00007f8e977cb000 2044K ----- sha1.so
00007f8e979ca000 4K r---- sha1.so
00007f8e979cb000 4K rw--- sha1.so
00007f8e979cc000 4K r-x-- md5.so
00007f8e979cd000 2044K ----- md5.so
00007f8e97bcc000 4K r---- md5.so
00007f8e97bcd000 4K rw--- md5.so
00007f8e97bce000 4K r-x-- escape.so
00007f8e97bcf000 2048K ----- escape.so
00007f8e97dcf000 4K r---- escape.so
00007f8e97dd0000 4K rw--- escape.so
00007f8e97dd1000 80K r-x-- libresolv-2.24.so
00007f8e97de5000 2044K ----- libresolv-2.24.so
00007f8e97fe4000 4K r---- libresolv-2.24.so
00007f8e97fe5000 4K rw--- libresolv-2.24.so
00007f8e97fe6000 8K rw--- [ anon ]
00007f8e97fe8000 20K r-x-- libnss_dns-2.24.so
00007f8e97fed000 2044K ----- libnss_dns-2.24.so
00007f8e981ec000 4K r---- libnss_dns-2.24.so
00007f8e981ed000 4K rw--- libnss_dns-2.24.so
00007f8e981ee000 40K r-x-- libnss_files-2.24.so
00007f8e981f8000 2048K ----- libnss_files-2.24.so
00007f8e983f8000 4K r---- libnss_files-2.24.so
00007f8e983f9000 4K rw--- libnss_files-2.24.so
00007f8e983fa000 24K rw--- [ anon ]
00007f8e98400000 2048K rw--- [ anon ]
00007f8e987fe000 4K r-x-- fcntl.so
00007f8e987ff000 2044K ----- fcntl.so
00007f8e989fe000 4K r---- fcntl.so
00007f8e989ff000 4K rw--- fcntl.so
00007f8e98a00000 2048K rw--- [ anon ]
00007f8e98dd7000 100K r-x-- libz.so.1.2.8
00007f8e98df0000 2044K ----- libz.so.1.2.8
00007f8e98fef000 4K r---- libz.so.1.2.8
00007f8e98ff0000 4K rw--- libz.so.1.2.8
00007f8e98ff1000 52K r-x-- zlib.so
00007f8e98ffe000 2048K ----- zlib.so
00007f8e991fe000 4K r---- zlib.so
00007f8e991ff000 4K rw--- zlib.so
00007f8e99200000 2048K rw--- [ anon ]
00007f8e994c5000 4K ----- [ anon ]
00007f8e994c6000 1024K rw--- [ anon ]
00007f8e995c6000 24K r-x-- strptime.so
00007f8e995cc000 2048K ----- strptime.so
00007f8e997cc000 4K r---- strptime.so
00007f8e997cd000 4K rw--- strptime.so
00007f8e997ce000 184K r-x-- date_core.so
00007f8e997fc000 2048K ----- date_core.so
00007f8e999fc000 8K r---- date_core.so
00007f8e999fe000 4K rw--- date_core.so
00007f8e999ff000 4K rw--- [ anon ]
00007f8e99a00000 2048K rw--- [ anon ]
00007f8e99cc6000 76K r-x-- msgpack.so
00007f8e99cd9000 2044K ----- msgpack.so
00007f8e99ed8000 4K r---- msgpack.so
00007f8e99ed9000 4K rw--- msgpack.so
00007f8e99eda000 4K r-x-- nonblock.so
00007f8e99edb000 2048K ----- nonblock.so
00007f8e9a0db000 4K r---- nonblock.so
00007f8e9a0dc000 4K rw--- nonblock.so
00007f8e9a0dd000 12K r-x-- digest.so
00007f8e9a0e0000 2044K ----- digest.so
00007f8e9a2df000 4K r---- digest.so
00007f8e9a2e0000 4K rw--- digest.so
00007f8e9a2e1000 2260K r-x-- libcrypto.so.1.0.2
00007f8e9a516000 2048K ----- libcrypto.so.1.0.2
00007f8e9a716000 112K r---- libcrypto.so.1.0.2
00007f8e9a732000 60K rw--- libcrypto.so.1.0.2
00007f8e9a741000 16K rw--- [ anon ]
00007f8e9a745000 380K r-x-- libssl.so.1.0.2
00007f8e9a7a4000 2048K ----- libssl.so.1.0.2
00007f8e9a9a4000 16K r---- libssl.so.1.0.2
00007f8e9a9a8000 24K rw--- libssl.so.1.0.2
00007f8e9a9ae000 312K r-x-- openssl.so
00007f8e9a9fc000 2044K ----- openssl.so
00007f8e9abfb000 16K r---- openssl.so
00007f8e9abff000 4K rw--- openssl.so
00007f8e9ac00000 2048K rw--- [ anon ]
00007f8e9aedc000 4K ----- [ anon ]
00007f8e9aedd000 1024K rw--- [ anon ]
00007f8e9afdd000 28K r-x-- librt-2.24.so
00007f8e9afe4000 2044K ----- librt-2.24.so
00007f8e9b1e3000 4K r---- librt-2.24.so
00007f8e9b1e4000 4K rw--- librt-2.24.so
00007f8e9b1e5000 64K r-x-- cool.io_ext.so
00007f8e9b1f5000 2044K ----- cool.io_ext.so
00007f8e9b3f4000 4K r---- cool.io_ext.so
00007f8e9b3f5000 4K rw--- cool.io_ext.so
00007f8e9b3f6000 12K r-x-- iobuffer_ext.so
00007f8e9b3f9000 2044K ----- iobuffer_ext.so
00007f8e9b5f8000 4K r---- iobuffer_ext.so
00007f8e9b5f9000 4K rw--- iobuffer_ext.so
00007f8e9b5fa000 20K r-x-- strscan.so
00007f8e9b5ff000 2044K ----- strscan.so
00007f8e9b7fe000 4K r---- strscan.so
00007f8e9b7ff000 4K rw--- strscan.so
00007f8e9b800000 2048K rw--- [ anon ]
00007f8e9baae000 4K ----- [ anon ]
00007f8e9baaf000 1024K rw--- [ anon ]
00007f8e9bbaf000 4K r-x-- wait.so
00007f8e9bbb0000 2048K ----- wait.so
00007f8e9bdb0000 4K r---- wait.so
00007f8e9bdb1000 4K rw--- wait.so
00007f8e9bdb2000 164K r-x-- socket.so
00007f8e9bddb000 2048K ----- socket.so
00007f8e9bfdb000 4K r---- socket.so
00007f8e9bfdc000 4K rw--- socket.so
00007f8e9bfdd000 48K r-x-- yajl.so
00007f8e9bfe9000 2044K ----- yajl.so
00007f8e9c1e8000 4K r---- yajl.so
00007f8e9c1e9000 4K rw--- yajl.so
00007f8e9c1ea000 28K r-x-- generator.so
00007f8e9c1f1000 2044K ----- generator.so
00007f8e9c3f0000 4K r---- generator.so
00007f8e9c3f1000 4K rw--- generator.so
00007f8e9c3f2000 20K r-x-- parser.so
00007f8e9c3f7000 2048K ----- parser.so
00007f8e9c5f7000 4K r---- parser.so
00007f8e9c5f8000 4K rw--- parser.so
00007f8e9c5f9000 20K r-x-- etc.so
00007f8e9c5fe000 2048K ----- etc.so
00007f8e9c7fe000 4K r---- etc.so
00007f8e9c7ff000 4K rw--- etc.so
00007f8e9c800000 2048K rw--- [ anon ]
00007f8e9caf7000 4K ----- [ anon ]
00007f8e9caf8000 1024K rw--- [ anon ]
00007f8e9cbf8000 28K r-x-- stringio.so
00007f8e9cbff000 2044K ----- stringio.so
00007f8e9cdfe000 4K r---- stringio.so
00007f8e9cdff000 4K rw--- stringio.so
00007f8e9ce00000 2048K rw--- [ anon ]
00007f8e9d0f8000 4K ----- [ anon ]
00007f8e9d0f9000 1024K rw--- [ anon ]
00007f8e9d1f9000 8K r-x-- transdb.so
00007f8e9d1fb000 2048K ----- transdb.so
00007f8e9d3fb000 4K r---- transdb.so
00007f8e9d3fc000 4K rw--- transdb.so
00007f8e9d3fd000 8K r-x-- encdb.so
00007f8e9d3ff000 2044K ----- encdb.so
00007f8e9d5fe000 4K r---- encdb.so
00007f8e9d5ff000 4K rw--- encdb.so
00007f8e9d600000 6144K rw--- [ anon ]
00007f8e9dc7d000 1620K r-x-- libc-2.24.so
00007f8e9de12000 2048K ----- libc-2.24.so
00007f8e9e012000 16K r---- libc-2.24.so
00007f8e9e016000 8K rw--- libc-2.24.so
00007f8e9e018000 16K rw--- [ anon ]
00007f8e9e01c000 1036K r-x-- libm-2.24.so
00007f8e9e11f000 2044K ----- libm-2.24.so
00007f8e9e31e000 4K r---- libm-2.24.so
00007f8e9e31f000 4K rw--- libm-2.24.so
00007f8e9e320000 32K r-x-- libcrypt-2.24.so
00007f8e9e328000 2048K ----- libcrypt-2.24.so
00007f8e9e528000 4K r---- libcrypt-2.24.so
00007f8e9e529000 4K rw--- libcrypt-2.24.so
00007f8e9e52a000 184K rw--- [ anon ]
00007f8e9e558000 12K r-x-- libdl-2.24.so
00007f8e9e55b000 2044K ----- libdl-2.24.so
00007f8e9e75a000 4K r---- libdl-2.24.so
00007f8e9e75b000 4K rw--- libdl-2.24.so
00007f8e9e75c000 516K r-x-- libgmp.so.10.3.2
00007f8e9e7dd000 2048K ----- libgmp.so.10.3.2
00007f8e9e9dd000 4K r---- libgmp.so.10.3.2
00007f8e9e9de000 4K rw--- libgmp.so.10.3.2
00007f8e9e9df000 96K r-x-- libpthread-2.24.so
00007f8e9e9f7000 2044K ----- libpthread-2.24.so
00007f8e9ebf6000 4K r---- libpthread-2.24.so
00007f8e9ebf7000 4K rw--- libpthread-2.24.so
00007f8e9ebf8000 16K rw--- [ anon ]
00007f8e9ebfc000 2500K r-x-- libruby-2.3.so.2.3.0
00007f8e9ee6d000 2048K ----- libruby-2.3.so.2.3.0
00007f8e9f06d000 32K r---- libruby-2.3.so.2.3.0
00007f8e9f075000 4K rw--- libruby-2.3.so.2.3.0
00007f8e9f076000 64K rw--- [ anon ]
00007f8e9f086000 368K r-x-- libjemalloc.so.2
00007f8e9f0e2000 2044K ----- libjemalloc.so.2
00007f8e9f2e1000 8K r---- libjemalloc.so.2
00007f8e9f2e3000 4K rw--- libjemalloc.so.2
00007f8e9f2e4000 4K rw--- [ anon ]
00007f8e9f2e5000 140K r-x-- ld-2.24.so
00007f8e9f3f9000 4K ----- [ anon ]
00007f8e9f3fa000 1024K rw--- [ anon ]
00007f8e9f4fa000 4K ----- [ anon ]
00007f8e9f4fb000 28K rw--- [ anon ]
00007f8e9f505000 12K rw--- [ anon ]
00007f8e9f508000 4K r---- ld-2.24.so
00007f8e9f509000 4K rw--- ld-2.24.so
00007f8e9f50a000 4K rw--- [ anon ]
00007ffc68f65000 8188K rw--- [ stack ]
00007ffc6979c000 12K r---- [ anon ]
00007ffc6979f000 8K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
total 158744K

out_es pmap -d $pid:(write 1000000 logs)
13216: /usr/bin/ruby2.3 -Eascii-8bit:ascii-8bit /usr/local/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins --under-supervisor
Address Kbytes Mode Offset Device Mapping
000055a1ef940000 4 r-x-- 0000000000000000 000:00039 ruby2.3
000055a1efb40000 4 r---- 0000000000000000 000:00039 ruby2.3
000055a1efb41000 4 rw--- 0000000000001000 000:00039 ruby2.3
00007f6b1e200000 706560 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b49594000 276 r-x-- 0000000000000000 000:00039 oj.so
00007f6b495d9000 2044 ----- 0000000000045000 000:00039 oj.so
00007f6b497d8000 8 r---- 0000000000044000 000:00039 oj.so
00007f6b497da000 12 rw--- 0000000000046000 000:00039 oj.so
00007f6b497dd000 68 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b497ee000 68 r-x-- 0000000000000000 000:00039 bigdecimal.so
00007f6b497ff000 2044 ----- 0000000000011000 000:00039 bigdecimal.so
00007f6b499fe000 4 r---- 0000000000010000 000:00039 bigdecimal.so
00007f6b499ff000 4 rw--- 0000000000011000 000:00039 bigdecimal.so
00007f6b49a00000 4096 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b49fc3000 24 r-x-- 0000000000000000 000:00039 pathname.so
00007f6b49fc9000 2044 ----- 0000000000006000 000:00039 pathname.so
00007f6b4a1c8000 4 r---- 0000000000005000 000:00039 pathname.so
00007f6b4a1c9000 4 rw--- 0000000000006000 000:00039 pathname.so
00007f6b4a1ca000 4 r-x-- 0000000000000000 000:00039 sha1.so
00007f6b4a1cb000 2044 ----- 0000000000001000 000:00039 sha1.so
00007f6b4a3ca000 4 r---- 0000000000000000 000:00039 sha1.so
00007f6b4a3cb000 4 rw--- 0000000000001000 000:00039 sha1.so
00007f6b4a3cc000 4 r-x-- 0000000000000000 000:00039 md5.so
00007f6b4a3cd000 2044 ----- 0000000000001000 000:00039 md5.so
00007f6b4a5cc000 4 r---- 0000000000000000 000:00039 md5.so
00007f6b4a5cd000 4 rw--- 0000000000001000 000:00039 md5.so
00007f6b4a5ce000 4 r-x-- 0000000000000000 000:00039 escape.so
00007f6b4a5cf000 2048 ----- 0000000000001000 000:00039 escape.so
00007f6b4a7cf000 4 r---- 0000000000001000 000:00039 escape.so
00007f6b4a7d0000 4 rw--- 0000000000002000 000:00039 escape.so
00007f6b4a7d1000 80 r-x-- 0000000000000000 000:00039 libresolv-2.24.so
00007f6b4a7e5000 2044 ----- 0000000000014000 000:00039 libresolv-2.24.so
00007f6b4a9e4000 4 r---- 0000000000013000 000:00039 libresolv-2.24.so
00007f6b4a9e5000 4 rw--- 0000000000014000 000:00039 libresolv-2.24.so
00007f6b4a9e6000 8 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4a9e8000 20 r-x-- 0000000000000000 000:00039 libnss_dns-2.24.so
00007f6b4a9ed000 2044 ----- 0000000000005000 000:00039 libnss_dns-2.24.so
00007f6b4abec000 4 r---- 0000000000004000 000:00039 libnss_dns-2.24.so
00007f6b4abed000 4 rw--- 0000000000005000 000:00039 libnss_dns-2.24.so
00007f6b4abee000 40 r-x-- 0000000000000000 000:00039 libnss_files-2.24.so
00007f6b4abf8000 2048 ----- 000000000000a000 000:00039 libnss_files-2.24.so
00007f6b4adf8000 4 r---- 000000000000a000 000:00039 libnss_files-2.24.so
00007f6b4adf9000 4 rw--- 000000000000b000 000:00039 libnss_files-2.24.so
00007f6b4adfa000 24 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4ae00000 2048 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4b1fe000 4 r-x-- 0000000000000000 000:00039 fcntl.so
00007f6b4b1ff000 2044 ----- 0000000000001000 000:00039 fcntl.so
00007f6b4b3fe000 4 r---- 0000000000000000 000:00039 fcntl.so
00007f6b4b3ff000 4 rw--- 0000000000001000 000:00039 fcntl.so
00007f6b4b400000 2048 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4b7d7000 100 r-x-- 0000000000000000 000:00039 libz.so.1.2.8
00007f6b4b7f0000 2044 ----- 0000000000019000 000:00039 libz.so.1.2.8
00007f6b4b9ef000 4 r---- 0000000000018000 000:00039 libz.so.1.2.8
00007f6b4b9f0000 4 rw--- 0000000000019000 000:00039 libz.so.1.2.8
00007f6b4b9f1000 52 r-x-- 0000000000000000 000:00039 zlib.so
00007f6b4b9fe000 2048 ----- 000000000000d000 000:00039 zlib.so
00007f6b4bbfe000 4 r---- 000000000000d000 000:00039 zlib.so
00007f6b4bbff000 4 rw--- 000000000000e000 000:00039 zlib.so
00007f6b4bc00000 2048 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4bfc6000 24 r-x-- 0000000000000000 000:00039 strptime.so
00007f6b4bfcc000 2048 ----- 0000000000006000 000:00039 strptime.so
00007f6b4c1cc000 4 r---- 0000000000006000 000:00039 strptime.so
00007f6b4c1cd000 4 rw--- 0000000000007000 000:00039 strptime.so
00007f6b4c1ce000 184 r-x-- 0000000000000000 000:00039 date_core.so
00007f6b4c1fc000 2048 ----- 000000000002e000 000:00039 date_core.so
00007f6b4c3fc000 8 r---- 000000000002e000 000:00039 date_core.so
00007f6b4c3fe000 4 rw--- 0000000000030000 000:00039 date_core.so
00007f6b4c3ff000 4 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4c400000 2048 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4c6c6000 76 r-x-- 0000000000000000 000:00039 msgpack.so
00007f6b4c6d9000 2044 ----- 0000000000013000 000:00039 msgpack.so
00007f6b4c8d8000 4 r---- 0000000000012000 000:00039 msgpack.so
00007f6b4c8d9000 4 rw--- 0000000000013000 000:00039 msgpack.so
00007f6b4c8da000 4 r-x-- 0000000000000000 000:00039 nonblock.so
00007f6b4c8db000 2048 ----- 0000000000001000 000:00039 nonblock.so
00007f6b4cadb000 4 r---- 0000000000001000 000:00039 nonblock.so
00007f6b4cadc000 4 rw--- 0000000000002000 000:00039 nonblock.so
00007f6b4cadd000 12 r-x-- 0000000000000000 000:00039 digest.so
00007f6b4cae0000 2044 ----- 0000000000003000 000:00039 digest.so
00007f6b4ccdf000 4 r---- 0000000000002000 000:00039 digest.so
00007f6b4cce0000 4 rw--- 0000000000003000 000:00039 digest.so
00007f6b4cce1000 2260 r-x-- 0000000000000000 000:00039 libcrypto.so.1.0.2
00007f6b4cf16000 2048 ----- 0000000000235000 000:00039 libcrypto.so.1.0.2
00007f6b4d116000 112 r---- 0000000000235000 000:00039 libcrypto.so.1.0.2
00007f6b4d132000 60 rw--- 0000000000251000 000:00039 libcrypto.so.1.0.2
00007f6b4d141000 16 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4d145000 380 r-x-- 0000000000000000 000:00039 libssl.so.1.0.2
00007f6b4d1a4000 2048 ----- 000000000005f000 000:00039 libssl.so.1.0.2
00007f6b4d3a4000 16 r---- 000000000005f000 000:00039 libssl.so.1.0.2
00007f6b4d3a8000 24 rw--- 0000000000063000 000:00039 libssl.so.1.0.2
00007f6b4d3ae000 312 r-x-- 0000000000000000 000:00039 openssl.so
00007f6b4d3fc000 2044 ----- 000000000004e000 000:00039 openssl.so
00007f6b4d5fb000 16 r---- 000000000004d000 000:00039 openssl.so
00007f6b4d5ff000 4 rw--- 0000000000051000 000:00039 openssl.so
00007f6b4d600000 2048 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4d8dc000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f6b4d8dd000 1024 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4d9dd000 28 r-x-- 0000000000000000 000:00039 librt-2.24.so
00007f6b4d9e4000 2044 ----- 0000000000007000 000:00039 librt-2.24.so
00007f6b4dbe3000 4 r---- 0000000000006000 000:00039 librt-2.24.so
00007f6b4dbe4000 4 rw--- 0000000000007000 000:00039 librt-2.24.so
00007f6b4dbe5000 64 r-x-- 0000000000000000 000:00039 cool.io_ext.so
00007f6b4dbf5000 2044 ----- 0000000000010000 000:00039 cool.io_ext.so
00007f6b4ddf4000 4 r---- 000000000000f000 000:00039 cool.io_ext.so
00007f6b4ddf5000 4 rw--- 0000000000010000 000:00039 cool.io_ext.so
00007f6b4ddf6000 12 r-x-- 0000000000000000 000:00039 iobuffer_ext.so
00007f6b4ddf9000 2044 ----- 0000000000003000 000:00039 iobuffer_ext.so
00007f6b4dff8000 4 r---- 0000000000002000 000:00039 iobuffer_ext.so
00007f6b4dff9000 4 rw--- 0000000000003000 000:00039 iobuffer_ext.so
00007f6b4dffa000 20 r-x-- 0000000000000000 000:00039 strscan.so
00007f6b4dfff000 2044 ----- 0000000000005000 000:00039 strscan.so
00007f6b4e1fe000 4 r---- 0000000000004000 000:00039 strscan.so
00007f6b4e1ff000 4 rw--- 0000000000005000 000:00039 strscan.so
00007f6b4e200000 2048 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4e4ae000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f6b4e4af000 1024 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4e5af000 4 r-x-- 0000000000000000 000:00039 wait.so
00007f6b4e5b0000 2048 ----- 0000000000001000 000:00039 wait.so
00007f6b4e7b0000 4 r---- 0000000000001000 000:00039 wait.so
00007f6b4e7b1000 4 rw--- 0000000000002000 000:00039 wait.so
00007f6b4e7b2000 164 r-x-- 0000000000000000 000:00039 socket.so
00007f6b4e7db000 2048 ----- 0000000000029000 000:00039 socket.so
00007f6b4e9db000 4 r---- 0000000000029000 000:00039 socket.so
00007f6b4e9dc000 4 rw--- 000000000002a000 000:00039 socket.so
00007f6b4e9dd000 48 r-x-- 0000000000000000 000:00039 yajl.so
00007f6b4e9e9000 2044 ----- 000000000000c000 000:00039 yajl.so
00007f6b4ebe8000 4 r---- 000000000000b000 000:00039 yajl.so
00007f6b4ebe9000 4 rw--- 000000000000c000 000:00039 yajl.so
00007f6b4ebea000 28 r-x-- 0000000000000000 000:00039 generator.so
00007f6b4ebf1000 2044 ----- 0000000000007000 000:00039 generator.so
00007f6b4edf0000 4 r---- 0000000000006000 000:00039 generator.so
00007f6b4edf1000 4 rw--- 0000000000007000 000:00039 generator.so
00007f6b4edf2000 20 r-x-- 0000000000000000 000:00039 parser.so
00007f6b4edf7000 2048 ----- 0000000000005000 000:00039 parser.so
00007f6b4eff7000 4 r---- 0000000000005000 000:00039 parser.so
00007f6b4eff8000 4 rw--- 0000000000006000 000:00039 parser.so
00007f6b4eff9000 20 r-x-- 0000000000000000 000:00039 etc.so
00007f6b4effe000 2048 ----- 0000000000005000 000:00039 etc.so
00007f6b4f1fe000 4 r---- 0000000000005000 000:00039 etc.so
00007f6b4f1ff000 4 rw--- 0000000000006000 000:00039 etc.so
00007f6b4f200000 2048 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4f4f7000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f6b4f4f8000 1024 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4f5f8000 28 r-x-- 0000000000000000 000:00039 stringio.so
00007f6b4f5ff000 2044 ----- 0000000000007000 000:00039 stringio.so
00007f6b4f7fe000 4 r---- 0000000000006000 000:00039 stringio.so
00007f6b4f7ff000 4 rw--- 0000000000007000 000:00039 stringio.so
00007f6b4f800000 2048 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4faf8000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f6b4faf9000 1024 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b4fbf9000 8 r-x-- 0000000000000000 000:00039 transdb.so
00007f6b4fbfb000 2048 ----- 0000000000002000 000:00039 transdb.so
00007f6b4fdfb000 4 r---- 0000000000002000 000:00039 transdb.so
00007f6b4fdfc000 4 rw--- 0000000000003000 000:00039 transdb.so
00007f6b4fdfd000 8 r-x-- 0000000000000000 000:00039 encdb.so
00007f6b4fdff000 2044 ----- 0000000000002000 000:00039 encdb.so
00007f6b4fffe000 4 r---- 0000000000001000 000:00039 encdb.so
00007f6b4ffff000 4 rw--- 0000000000002000 000:00039 encdb.so
00007f6b50000000 6144 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b50635000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f6b50636000 1024 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b50736000 1620 r-x-- 0000000000000000 000:00039 libc-2.24.so
00007f6b508cb000 2048 ----- 0000000000195000 000:00039 libc-2.24.so
00007f6b50acb000 16 r---- 0000000000195000 000:00039 libc-2.24.so
00007f6b50acf000 8 rw--- 0000000000199000 000:00039 libc-2.24.so
00007f6b50ad1000 16 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b50ad5000 1036 r-x-- 0000000000000000 000:00039 libm-2.24.so
00007f6b50bd8000 2044 ----- 0000000000103000 000:00039 libm-2.24.so
00007f6b50dd7000 4 r---- 0000000000102000 000:00039 libm-2.24.so
00007f6b50dd8000 4 rw--- 0000000000103000 000:00039 libm-2.24.so
00007f6b50dd9000 32 r-x-- 0000000000000000 000:00039 libcrypt-2.24.so
00007f6b50de1000 2048 ----- 0000000000008000 000:00039 libcrypt-2.24.so
00007f6b50fe1000 4 r---- 0000000000008000 000:00039 libcrypt-2.24.so
00007f6b50fe2000 4 rw--- 0000000000009000 000:00039 libcrypt-2.24.so
00007f6b50fe3000 184 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b51011000 12 r-x-- 0000000000000000 000:00039 libdl-2.24.so
00007f6b51014000 2044 ----- 0000000000003000 000:00039 libdl-2.24.so
00007f6b51213000 4 r---- 0000000000002000 000:00039 libdl-2.24.so
00007f6b51214000 4 rw--- 0000000000003000 000:00039 libdl-2.24.so
00007f6b51215000 516 r-x-- 0000000000000000 000:00039 libgmp.so.10.3.2
00007f6b51296000 2048 ----- 0000000000081000 000:00039 libgmp.so.10.3.2
00007f6b51496000 4 r---- 0000000000081000 000:00039 libgmp.so.10.3.2
00007f6b51497000 4 rw--- 0000000000082000 000:00039 libgmp.so.10.3.2
00007f6b51498000 96 r-x-- 0000000000000000 000:00039 libpthread-2.24.so
00007f6b514b0000 2044 ----- 0000000000018000 000:00039 libpthread-2.24.so
00007f6b516af000 4 r---- 0000000000017000 000:00039 libpthread-2.24.so
00007f6b516b0000 4 rw--- 0000000000018000 000:00039 libpthread-2.24.so
00007f6b516b1000 16 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b516b5000 2500 r-x-- 0000000000000000 000:00039 libruby-2.3.so.2.3.0
00007f6b51926000 2048 ----- 0000000000271000 000:00039 libruby-2.3.so.2.3.0
00007f6b51b26000 32 r---- 0000000000271000 000:00039 libruby-2.3.so.2.3.0
00007f6b51b2e000 4 rw--- 0000000000279000 000:00039 libruby-2.3.so.2.3.0
00007f6b51b2f000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b51b3f000 368 r-x-- 0000000000000000 000:00039 libjemalloc.so.2
00007f6b51b9b000 2044 ----- 000000000005c000 000:00039 libjemalloc.so.2
00007f6b51d9a000 8 r---- 000000000005b000 000:00039 libjemalloc.so.2
00007f6b51d9c000 4 rw--- 000000000005d000 000:00039 libjemalloc.so.2
00007f6b51d9d000 4 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b51d9e000 140 r-x-- 0000000000000000 000:00039 ld-2.24.so
00007f6b51eb2000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f6b51eb3000 1024 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b51fb3000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f6b51fb4000 28 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b51fbe000 12 rw--- 0000000000000000 000:00000 [ anon ]
00007f6b51fc1000 4 r---- 0000000000023000 000:00039 ld-2.24.so
00007f6b51fc2000 4 rw--- 0000000000024000 000:00039 ld-2.24.so
00007f6b51fc3000 4 rw--- 0000000000000000 000:00000 [ anon ]
00007ffc8d220000 8188 rw--- 0000000000000000 000:00000 [ stack ]
00007ffc8dafe000 12 r---- 0000000000000000 000:00000 [ anon ]
00007ffc8db01000 8 r-x-- 0000000000000000 000:00000 [ anon ]
ffffffffff600000 4 r-x-- 0000000000000000 000:00000 [ anon ]
mapped: 843208K writeable/private: 748224K shared: 0K

@cosmo0920
Copy link
Collaborator

@lixianyang Could you try typhoeus backend with adding http_backend typhoeus in elasticsearch configuration? This backend can handle keepalive HTTP connection and more efficient requests.

@lixianyang
Copy link
Author

@cosmo0920 It doesn't work. I use docker stats command: 651.3MiB / 10.66GiB(MEM USAGE / LIMIT)
gem list

bigdecimal (1.2.8)
cool.io (1.5.3)
did_you_mean (1.0.0)
dig_rb (1.0.1)
elasticsearch (6.1.0)
elasticsearch-api (6.1.0)
elasticsearch-transport (6.1.0)
ethon (0.11.0)
excon (0.62.0)
faraday (0.15.2)
ffi (1.9.25)
fluent-plugin-elasticsearch (2.11.4)
fluentd (1.2.2)
http_parser.rb (0.6.0)
io-console (0.4.5)
json (2.1.0, 1.8.3)
minitest (5.9.0)
msgpack (1.2.4)
multi_json (1.13.1)
multipart-post (2.0.0)
net-telnet (0.1.1)
oj (3.3.10)
power_assert (0.2.7)
psych (2.1.0)
rake (10.5.0)
rdoc (4.2.1)
serverengine (2.0.6)
sigdump (0.2.4)
strptime (0.2.3)
test-unit (3.1.7)
thread_safe (0.3.6)
typhoeus (1.3.0)
tzinfo (1.2.5)
tzinfo-data (1.2018.5)
yajl-ruby (1.4.0)

fluentd.conf

<source>
    @type forward
    port 24224
    bind 0.0.0.0
</source>
<filter docker.**>
    @type record_transformer
    enable_ruby  true
    <record>
        time    ${time.strftime('%Y-%m-%dT%H:%M:%S')}
        usec    ${(Time.now.to_f * 1000000).to_i}
    </record>
</filter>
<match docker.**>
    @type elasticsearch
    hosts "#{ENV['ES_ADDRS']}"
    user elastic
    password changeme
    index_name testlog
    type_name testlog
    ssl_verify false
    logstash_format true    
    logstash_prefix testlog
    logstash_dateformat %Y%m
    utc_index false
    time_key time
    time_key_exclude_timestamp true
    include_tag_key false
    reload_connections false
    reconnect_on_error true
    template_name testlog
    template_file /fluentd/etc/testlog.template
    slow_flush_log_threshold 60.0
    request_timeout 60s
    http_backend typhoeus
    
    <secondary>
        @type file
        path /home/fluent/testlog
    </secondary>
    
    <buffer>
        @type file
        path /home/fluent/testlog-buffer
        chunk_limit_size 64MB
        total_limit_size 64GB
        flush_mode interval
        flush_interval 5s
        flush_thread_count 4
        
        # retry config
        retry_type periodic
        retry_wait 30
        retry_timeout 24h
        retry_secondary_threshold 0.5
    </buffer>
</match>

pmap -p $pid

Address           Kbytes Mode  Offset           Device    Mapping
0000558b82465000       4 r-x-- 0000000000000000 000:00059 /usr/bin/ruby2.3
0000558b82665000       4 r---- 0000000000000000 000:00059 /usr/bin/ruby2.3
0000558b82666000       4 rw--- 0000000000001000 000:00059 /usr/bin/ruby2.3
00007f2a89c00000  743424 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ab73a6000     276 r-x-- 0000000000000000 000:00059 /var/lib/gems/2.3.0/gems/oj-3.3.10/lib/oj/oj.so
00007f2ab73eb000    2044 ----- 0000000000045000 000:00059 /var/lib/gems/2.3.0/gems/oj-3.3.10/lib/oj/oj.so
00007f2ab75ea000       8 r---- 0000000000044000 000:00059 /var/lib/gems/2.3.0/gems/oj-3.3.10/lib/oj/oj.so
00007f2ab75ec000      12 rw--- 0000000000046000 000:00059 /var/lib/gems/2.3.0/gems/oj-3.3.10/lib/oj/oj.so
00007f2ab75ef000      68 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ab7600000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ab79ee000      68 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/bigdecimal.so
00007f2ab79ff000    2044 ----- 0000000000011000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/bigdecimal.so
00007f2ab7bfe000       4 r---- 0000000000010000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/bigdecimal.so
00007f2ab7bff000       4 rw--- 0000000000011000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/bigdecimal.so
00007f2ab7c00000    6144 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ab8205000      28 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
00007f2ab820c000    2048 ----- 0000000000007000 000:00059 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
00007f2ab840c000       4 r---- 0000000000007000 000:00059 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
00007f2ab840d000       4 rw--- 0000000000008000 000:00059 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
00007f2ab840e000      76 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
00007f2ab8421000    2044 ----- 0000000000013000 000:00059 /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
00007f2ab8620000       4 r---- 0000000000012000 000:00059 /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
00007f2ab8621000       4 rw--- 0000000000013000 000:00059 /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
00007f2ab8622000      68 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.3
00007f2ab8633000    2048 ----- 0000000000011000 000:00059 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.3
00007f2ab8833000       4 r---- 0000000000011000 000:00059 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.3
00007f2ab8834000       4 rw--- 0000000000012000 000:00059 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.3
00007f2ab8835000     200 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libidn.so.11.6.16
00007f2ab8867000    2048 ----- 0000000000032000 000:00059 /lib/x86_64-linux-gnu/libidn.so.11.6.16
00007f2ab8a67000       4 r---- 0000000000032000 000:00059 /lib/x86_64-linux-gnu/libidn.so.11.6.16
00007f2ab8a68000       4 rw--- 0000000000033000 000:00059 /lib/x86_64-linux-gnu/libidn.so.11.6.16
00007f2ab8a69000     360 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.2.0
00007f2ab8ac3000    2044 ----- 000000000005a000 000:00059 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.2.0
00007f2ab8cc2000      40 r---- 0000000000059000 000:00059 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.2.0
00007f2ab8ccc000       8 rw--- 0000000000063000 000:00059 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.2.0
00007f2ab8cce000     104 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
00007f2ab8ce8000    2044 ----- 000000000001a000 000:00059 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
00007f2ab8ee7000       4 r---- 0000000000019000 000:00059 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
00007f2ab8ee8000       4 rw--- 000000000001a000 000:00059 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
00007f2ab8ee9000      12 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libkeyutils.so.1.5
00007f2ab8eec000    2044 ----- 0000000000003000 000:00059 /lib/x86_64-linux-gnu/libkeyutils.so.1.5
00007f2ab90eb000       4 r---- 0000000000002000 000:00059 /lib/x86_64-linux-gnu/libkeyutils.so.1.5
00007f2ab90ec000       4 rw--- 0000000000003000 000:00059 /lib/x86_64-linux-gnu/libkeyutils.so.1.5
00007f2ab90ed000      44 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
00007f2ab90f8000    2044 ----- 000000000000b000 000:00059 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
00007f2ab92f7000       4 r---- 000000000000a000 000:00059 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
00007f2ab92f8000       4 rw--- 000000000000b000 000:00059 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
00007f2ab92f9000    1052 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
00007f2ab9400000    2048 ----- 0000000000107000 000:00059 /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
00007f2ab9600000       8 r---- 0000000000107000 000:00059 /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
00007f2ab9602000      28 rw--- 0000000000109000 000:00059 /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
00007f2ab9609000     212 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libnettle.so.6.3
00007f2ab963e000    2044 ----- 0000000000035000 000:00059 /usr/lib/x86_64-linux-gnu/libnettle.so.6.3
00007f2ab983d000       8 r---- 0000000000034000 000:00059 /usr/lib/x86_64-linux-gnu/libnettle.so.6.3
00007f2ab983f000       4 rw--- 0000000000036000 000:00059 /usr/lib/x86_64-linux-gnu/libnettle.so.6.3
00007f2ab9840000     204 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.3
00007f2ab9873000    2048 ----- 0000000000033000 000:00059 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.3
00007f2ab9a73000       4 r---- 0000000000033000 000:00059 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.3
00007f2ab9a74000       4 rw--- 0000000000034000 000:00059 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.3
00007f2ab9a75000    1576 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.13.1
00007f2ab9bff000    2048 ----- 000000000018a000 000:00059 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.13.1
00007f2ab9dff000      52 r---- 000000000018a000 000:00059 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.13.1
00007f2ab9e0c000       4 rw--- 0000000000197000 000:00059 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.13.1
00007f2ab9e0d000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ab9e0e000    1100 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libunistring.so.0.1.2
00007f2ab9f21000    2044 ----- 0000000000113000 000:00059 /usr/lib/x86_64-linux-gnu/libunistring.so.0.1.2
00007f2aba120000      12 r---- 0000000000112000 000:00059 /usr/lib/x86_64-linux-gnu/libunistring.so.0.1.2
00007f2aba123000       4 rw--- 0000000000115000 000:00059 /usr/lib/x86_64-linux-gnu/libunistring.so.0.1.2
00007f2aba124000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007f2aba125000     308 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.7
00007f2aba172000    2044 ----- 000000000004d000 000:00059 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.7
00007f2aba371000       8 r---- 000000000004c000 000:00059 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.7
00007f2aba373000       4 rw--- 000000000004e000 000:00059 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.7
00007f2aba374000       8 rw--- 0000000000000000 000:00000   [ anon ]
00007f2aba376000      52 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.7
00007f2aba383000    2048 ----- 000000000000d000 000:00059 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.7
00007f2aba583000       4 r---- 000000000000d000 000:00059 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.7
00007f2aba584000       4 rw--- 000000000000e000 000:00059 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.7
00007f2aba585000      12 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libcom_err.so.2.1
00007f2aba588000    2044 ----- 0000000000003000 000:00059 /lib/x86_64-linux-gnu/libcom_err.so.2.1
00007f2aba787000       4 r---- 0000000000002000 000:00059 /lib/x86_64-linux-gnu/libcom_err.so.2.1
00007f2aba788000       4 rw--- 0000000000003000 000:00059 /lib/x86_64-linux-gnu/libcom_err.so.2.1
00007f2aba789000     188 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
00007f2aba7b8000    2048 ----- 000000000002f000 000:00059 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
00007f2aba9b8000       8 r---- 000000000002f000 000:00059 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
00007f2aba9ba000       4 rw--- 0000000000031000 000:00059 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
00007f2aba9bb000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007f2aba9bc000     804 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
00007f2abaa85000    2048 ----- 00000000000c9000 000:00059 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
00007f2abac85000      56 r---- 00000000000c9000 000:00059 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
00007f2abac93000      12 rw--- 00000000000d7000 000:00059 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
00007f2abac96000     288 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
00007f2abacde000    2044 ----- 0000000000048000 000:00059 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
00007f2abaedd000       8 r---- 0000000000047000 000:00059 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
00007f2abaedf000       8 rw--- 0000000000049000 000:00059 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
00007f2abaee1000      48 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libpsl.so.5.1.1
00007f2abaeed000    2048 ----- 000000000000c000 000:00059 /usr/lib/x86_64-linux-gnu/libpsl.so.5.1.1
00007f2abb0ed000       4 r---- 000000000000c000 000:00059 /usr/lib/x86_64-linux-gnu/libpsl.so.5.1.1
00007f2abb0ee000       4 rw--- 000000000000d000 000:00059 /usr/lib/x86_64-linux-gnu/libpsl.so.5.1.1
00007f2abb0ef000     172 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
00007f2abb11a000    2044 ----- 000000000002b000 000:00059 /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
00007f2abb319000       4 r---- 000000000002a000 000:00059 /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
00007f2abb31a000       4 rw--- 000000000002b000 000:00059 /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
00007f2abb31b000     108 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/librtmp.so.1
00007f2abb336000    2048 ----- 000000000001b000 000:00059 /usr/lib/x86_64-linux-gnu/librtmp.so.1
00007f2abb536000       4 r---- 000000000001b000 000:00059 /usr/lib/x86_64-linux-gnu/librtmp.so.1
00007f2abb537000       4 rw--- 000000000001c000 000:00059 /usr/lib/x86_64-linux-gnu/librtmp.so.1
00007f2abb538000     132 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libidn2.so.0.1.4
00007f2abb559000    2044 ----- 0000000000021000 000:00059 /usr/lib/x86_64-linux-gnu/libidn2.so.0.1.4
00007f2abb758000       4 r---- 0000000000020000 000:00059 /usr/lib/x86_64-linux-gnu/libidn2.so.0.1.4
00007f2abb759000       4 rw--- 0000000000021000 000:00059 /usr/lib/x86_64-linux-gnu/libidn2.so.0.1.4
00007f2abb75a000     140 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libnghttp2.so.14.12.3
00007f2abb77d000    2044 ----- 0000000000023000 000:00059 /usr/lib/x86_64-linux-gnu/libnghttp2.so.14.12.3
00007f2abb97c000       4 r---- 0000000000022000 000:00059 /usr/lib/x86_64-linux-gnu/libnghttp2.so.14.12.3
00007f2abb97d000      12 rw--- 0000000000023000 000:00059 /usr/lib/x86_64-linux-gnu/libnghttp2.so.14.12.3
00007f2abb980000     500 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libcurl.so.4.4.0
00007f2abb9fd000    2044 ----- 000000000007d000 000:00059 /usr/lib/x86_64-linux-gnu/libcurl.so.4.4.0
00007f2abbbfc000      12 r---- 000000000007c000 000:00059 /usr/lib/x86_64-linux-gnu/libcurl.so.4.4.0
00007f2abbbff000       4 rw--- 000000000007f000 000:00059 /usr/lib/x86_64-linux-gnu/libcurl.so.4.4.0
00007f2abbc00000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abbfd9000     140 r-x-- 0000000000000000 000:00059 /var/lib/gems/2.3.0/gems/ffi-1.9.25/lib/ffi_c.so
00007f2abbffc000    2044 ----- 0000000000023000 000:00059 /var/lib/gems/2.3.0/gems/ffi-1.9.25/lib/ffi_c.so
00007f2abc1fb000       4 r---- 0000000000022000 000:00059 /var/lib/gems/2.3.0/gems/ffi-1.9.25/lib/ffi_c.so
00007f2abc1fc000       4 rw--- 0000000000023000 000:00059 /var/lib/gems/2.3.0/gems/ffi-1.9.25/lib/ffi_c.so
00007f2abc1fd000       4 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/sha2.so
00007f2abc1fe000    2048 ----- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/sha2.so
00007f2abc3fe000       4 r---- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/sha2.so
00007f2abc3ff000       4 rw--- 0000000000002000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/sha2.so
00007f2abc400000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abc7f9000      24 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/pathname.so
00007f2abc7ff000    2044 ----- 0000000000006000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/pathname.so
00007f2abc9fe000       4 r---- 0000000000005000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/pathname.so
00007f2abc9ff000       4 rw--- 0000000000006000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/pathname.so
00007f2abca00000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abcdca000       4 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/sha1.so
00007f2abcdcb000    2044 ----- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/sha1.so
00007f2abcfca000       4 r---- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/sha1.so
00007f2abcfcb000       4 rw--- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/sha1.so
00007f2abcfcc000       4 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/md5.so
00007f2abcfcd000    2044 ----- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/md5.so
00007f2abd1cc000       4 r---- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/md5.so
00007f2abd1cd000       4 rw--- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest/md5.so
00007f2abd1ce000       4 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/cgi/escape.so
00007f2abd1cf000    2048 ----- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/cgi/escape.so
00007f2abd3cf000       4 r---- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/cgi/escape.so
00007f2abd3d0000       4 rw--- 0000000000002000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/cgi/escape.so
00007f2abd3d1000      80 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libresolv-2.24.so
00007f2abd3e5000    2044 ----- 0000000000014000 000:00059 /lib/x86_64-linux-gnu/libresolv-2.24.so
00007f2abd5e4000       4 r---- 0000000000013000 000:00059 /lib/x86_64-linux-gnu/libresolv-2.24.so
00007f2abd5e5000       4 rw--- 0000000000014000 000:00059 /lib/x86_64-linux-gnu/libresolv-2.24.so
00007f2abd5e6000       8 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abd5e8000      20 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
00007f2abd5ed000    2044 ----- 0000000000005000 000:00059 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
00007f2abd7ec000       4 r---- 0000000000004000 000:00059 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
00007f2abd7ed000       4 rw--- 0000000000005000 000:00059 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
00007f2abd7ee000      40 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libnss_files-2.24.so
00007f2abd7f8000    2048 ----- 000000000000a000 000:00059 /lib/x86_64-linux-gnu/libnss_files-2.24.so
00007f2abd9f8000       4 r---- 000000000000a000 000:00059 /lib/x86_64-linux-gnu/libnss_files-2.24.so
00007f2abd9f9000       4 rw--- 000000000000b000 000:00059 /lib/x86_64-linux-gnu/libnss_files-2.24.so
00007f2abd9fa000      24 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abda00000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abddfe000       4 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/fcntl.so
00007f2abddff000    2044 ----- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/fcntl.so
00007f2abdffe000       4 r---- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/fcntl.so
00007f2abdfff000       4 rw--- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/fcntl.so
00007f2abe000000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abe2d6000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f2abe2d7000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abe3d7000     100 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libz.so.1.2.8
00007f2abe3f0000    2044 ----- 0000000000019000 000:00059 /lib/x86_64-linux-gnu/libz.so.1.2.8
00007f2abe5ef000       4 r---- 0000000000018000 000:00059 /lib/x86_64-linux-gnu/libz.so.1.2.8
00007f2abe5f0000       4 rw--- 0000000000019000 000:00059 /lib/x86_64-linux-gnu/libz.so.1.2.8
00007f2abe5f1000      52 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/zlib.so
00007f2abe5fe000    2048 ----- 000000000000d000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/zlib.so
00007f2abe7fe000       4 r---- 000000000000d000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/zlib.so
00007f2abe7ff000       4 rw--- 000000000000e000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/zlib.so
00007f2abe800000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abeac5000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f2abeac6000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abebc6000      24 r-x-- 0000000000000000 000:00059 /var/lib/gems/2.3.0/gems/strptime-0.2.3/lib/strptime/strptime.so
00007f2abebcc000    2048 ----- 0000000000006000 000:00059 /var/lib/gems/2.3.0/gems/strptime-0.2.3/lib/strptime/strptime.so
00007f2abedcc000       4 r---- 0000000000006000 000:00059 /var/lib/gems/2.3.0/gems/strptime-0.2.3/lib/strptime/strptime.so
00007f2abedcd000       4 rw--- 0000000000007000 000:00059 /var/lib/gems/2.3.0/gems/strptime-0.2.3/lib/strptime/strptime.so
00007f2abedce000     184 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/date_core.so
00007f2abedfc000    2048 ----- 000000000002e000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/date_core.so
00007f2abeffc000       8 r---- 000000000002e000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/date_core.so
00007f2abeffe000       4 rw--- 0000000000030000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/date_core.so
00007f2abefff000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abf000000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abf2c6000      76 r-x-- 0000000000000000 000:00059 /var/lib/gems/2.3.0/gems/msgpack-1.2.4/lib/msgpack/msgpack.so
00007f2abf2d9000    2044 ----- 0000000000013000 000:00059 /var/lib/gems/2.3.0/gems/msgpack-1.2.4/lib/msgpack/msgpack.so
00007f2abf4d8000       4 r---- 0000000000012000 000:00059 /var/lib/gems/2.3.0/gems/msgpack-1.2.4/lib/msgpack/msgpack.so
00007f2abf4d9000       4 rw--- 0000000000013000 000:00059 /var/lib/gems/2.3.0/gems/msgpack-1.2.4/lib/msgpack/msgpack.so
00007f2abf4da000       4 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/io/nonblock.so
00007f2abf4db000    2048 ----- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/io/nonblock.so
00007f2abf6db000       4 r---- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/io/nonblock.so
00007f2abf6dc000       4 rw--- 0000000000002000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/io/nonblock.so
00007f2abf6dd000      12 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest.so
00007f2abf6e0000    2044 ----- 0000000000003000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest.so
00007f2abf8df000       4 r---- 0000000000002000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest.so
00007f2abf8e0000       4 rw--- 0000000000003000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/digest.so
00007f2abf8e1000    2260 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
00007f2abfb16000    2048 ----- 0000000000235000 000:00059 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
00007f2abfd16000     112 r---- 0000000000235000 000:00059 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
00007f2abfd32000      60 rw--- 0000000000251000 000:00059 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
00007f2abfd41000      16 rw--- 0000000000000000 000:00000   [ anon ]
00007f2abfd45000     380 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
00007f2abfda4000    2048 ----- 000000000005f000 000:00059 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
00007f2abffa4000      16 r---- 000000000005f000 000:00059 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
00007f2abffa8000      24 rw--- 0000000000063000 000:00059 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
00007f2abffae000     312 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/openssl.so
00007f2abfffc000    2044 ----- 000000000004e000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/openssl.so
00007f2ac01fb000      16 r---- 000000000004d000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/openssl.so
00007f2ac01ff000       4 rw--- 0000000000051000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/openssl.so
00007f2ac0200000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac048b000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f2ac048c000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac058c000      28 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/librt-2.24.so
00007f2ac0593000    2044 ----- 0000000000007000 000:00059 /lib/x86_64-linux-gnu/librt-2.24.so
00007f2ac0792000       4 r---- 0000000000006000 000:00059 /lib/x86_64-linux-gnu/librt-2.24.so
00007f2ac0793000       4 rw--- 0000000000007000 000:00059 /lib/x86_64-linux-gnu/librt-2.24.so
00007f2ac0794000      64 r-x-- 0000000000000000 000:00059 /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io_ext.so
00007f2ac07a4000    2044 ----- 0000000000010000 000:00059 /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io_ext.so
00007f2ac09a3000       4 r---- 000000000000f000 000:00059 /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io_ext.so
00007f2ac09a4000       4 rw--- 0000000000010000 000:00059 /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io_ext.so
00007f2ac09a5000      12 r-x-- 0000000000000000 000:00059 /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/iobuffer_ext.so
00007f2ac09a8000    2044 ----- 0000000000003000 000:00059 /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/iobuffer_ext.so
00007f2ac0ba7000       4 r---- 0000000000002000 000:00059 /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/iobuffer_ext.so
00007f2ac0ba8000       4 rw--- 0000000000003000 000:00059 /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/iobuffer_ext.so
00007f2ac0ba9000      20 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/strscan.so
00007f2ac0bae000    2044 ----- 0000000000005000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/strscan.so
00007f2ac0dad000       4 r---- 0000000000004000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/strscan.so
00007f2ac0dae000       4 rw--- 0000000000005000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/strscan.so
00007f2ac0daf000       4 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/io/wait.so
00007f2ac0db0000    2048 ----- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/io/wait.so
00007f2ac0fb0000       4 r---- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/io/wait.so
00007f2ac0fb1000       4 rw--- 0000000000002000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/io/wait.so
00007f2ac0fb2000     164 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/socket.so
00007f2ac0fdb000    2048 ----- 0000000000029000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/socket.so
00007f2ac11db000       4 r---- 0000000000029000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/socket.so
00007f2ac11dc000       4 rw--- 000000000002a000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/socket.so
00007f2ac11dd000      48 r-x-- 0000000000000000 000:00059 /var/lib/gems/2.3.0/gems/yajl-ruby-1.4.0/lib/yajl/yajl.so
00007f2ac11e9000    2044 ----- 000000000000c000 000:00059 /var/lib/gems/2.3.0/gems/yajl-ruby-1.4.0/lib/yajl/yajl.so
00007f2ac13e8000       4 r---- 000000000000b000 000:00059 /var/lib/gems/2.3.0/gems/yajl-ruby-1.4.0/lib/yajl/yajl.so
00007f2ac13e9000       4 rw--- 000000000000c000 000:00059 /var/lib/gems/2.3.0/gems/yajl-ruby-1.4.0/lib/yajl/yajl.so
00007f2ac13ea000      28 r-x-- 0000000000000000 000:00059 /var/lib/gems/2.3.0/gems/json-2.1.0/lib/json/ext/generator.so
00007f2ac13f1000    2044 ----- 0000000000007000 000:00059 /var/lib/gems/2.3.0/gems/json-2.1.0/lib/json/ext/generator.so
00007f2ac15f0000       4 r---- 0000000000006000 000:00059 /var/lib/gems/2.3.0/gems/json-2.1.0/lib/json/ext/generator.so
00007f2ac15f1000       4 rw--- 0000000000007000 000:00059 /var/lib/gems/2.3.0/gems/json-2.1.0/lib/json/ext/generator.so
00007f2ac15f2000      20 r-x-- 0000000000000000 000:00059 /var/lib/gems/2.3.0/gems/json-2.1.0/lib/json/ext/parser.so
00007f2ac15f7000    2048 ----- 0000000000005000 000:00059 /var/lib/gems/2.3.0/gems/json-2.1.0/lib/json/ext/parser.so
00007f2ac17f7000       4 r---- 0000000000005000 000:00059 /var/lib/gems/2.3.0/gems/json-2.1.0/lib/json/ext/parser.so
00007f2ac17f8000       4 rw--- 0000000000006000 000:00059 /var/lib/gems/2.3.0/gems/json-2.1.0/lib/json/ext/parser.so
00007f2ac17f9000      20 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/etc.so
00007f2ac17fe000    2048 ----- 0000000000005000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/etc.so
00007f2ac19fe000       4 r---- 0000000000005000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/etc.so
00007f2ac19ff000       4 rw--- 0000000000006000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/etc.so
00007f2ac1a00000    4096 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac1ef7000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f2ac1ef8000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac1ff8000      28 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/stringio.so
00007f2ac1fff000    2044 ----- 0000000000007000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/stringio.so
00007f2ac21fe000       4 r---- 0000000000006000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/stringio.so
00007f2ac21ff000       4 rw--- 0000000000007000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/stringio.so
00007f2ac2200000    2048 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac24f8000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f2ac24f9000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac25f9000       8 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/enc/trans/transdb.so
00007f2ac25fb000    2048 ----- 0000000000002000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/enc/trans/transdb.so
00007f2ac27fb000       4 r---- 0000000000002000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/enc/trans/transdb.so
00007f2ac27fc000       4 rw--- 0000000000003000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/enc/trans/transdb.so
00007f2ac27fd000       8 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/enc/encdb.so
00007f2ac27ff000    2044 ----- 0000000000002000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/enc/encdb.so
00007f2ac29fe000       4 r---- 0000000000001000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/enc/encdb.so
00007f2ac29ff000       4 rw--- 0000000000002000 000:00059 /usr/lib/x86_64-linux-gnu/ruby/2.3.0/enc/encdb.so
00007f2ac2a00000    6144 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac3060000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f2ac3061000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac3161000    1620 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libc-2.24.so
00007f2ac32f6000    2048 ----- 0000000000195000 000:00059 /lib/x86_64-linux-gnu/libc-2.24.so
00007f2ac34f6000      16 r---- 0000000000195000 000:00059 /lib/x86_64-linux-gnu/libc-2.24.so
00007f2ac34fa000       8 rw--- 0000000000199000 000:00059 /lib/x86_64-linux-gnu/libc-2.24.so
00007f2ac34fc000      16 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac3500000    1036 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libm-2.24.so
00007f2ac3603000    2044 ----- 0000000000103000 000:00059 /lib/x86_64-linux-gnu/libm-2.24.so
00007f2ac3802000       4 r---- 0000000000102000 000:00059 /lib/x86_64-linux-gnu/libm-2.24.so
00007f2ac3803000       4 rw--- 0000000000103000 000:00059 /lib/x86_64-linux-gnu/libm-2.24.so
00007f2ac3804000      32 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libcrypt-2.24.so
00007f2ac380c000    2048 ----- 0000000000008000 000:00059 /lib/x86_64-linux-gnu/libcrypt-2.24.so
00007f2ac3a0c000       4 r---- 0000000000008000 000:00059 /lib/x86_64-linux-gnu/libcrypt-2.24.so
00007f2ac3a0d000       4 rw--- 0000000000009000 000:00059 /lib/x86_64-linux-gnu/libcrypt-2.24.so
00007f2ac3a0e000     184 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac3a3c000      12 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libdl-2.24.so
00007f2ac3a3f000    2044 ----- 0000000000003000 000:00059 /lib/x86_64-linux-gnu/libdl-2.24.so
00007f2ac3c3e000       4 r---- 0000000000002000 000:00059 /lib/x86_64-linux-gnu/libdl-2.24.so
00007f2ac3c3f000       4 rw--- 0000000000003000 000:00059 /lib/x86_64-linux-gnu/libdl-2.24.so
00007f2ac3c40000     516 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
00007f2ac3cc1000    2048 ----- 0000000000081000 000:00059 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
00007f2ac3ec1000       4 r---- 0000000000081000 000:00059 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
00007f2ac3ec2000       4 rw--- 0000000000082000 000:00059 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.2
00007f2ac3ec3000      96 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/libpthread-2.24.so
00007f2ac3edb000    2044 ----- 0000000000018000 000:00059 /lib/x86_64-linux-gnu/libpthread-2.24.so
00007f2ac40da000       4 r---- 0000000000017000 000:00059 /lib/x86_64-linux-gnu/libpthread-2.24.so
00007f2ac40db000       4 rw--- 0000000000018000 000:00059 /lib/x86_64-linux-gnu/libpthread-2.24.so
00007f2ac40dc000      16 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac40e0000    2500 r-x-- 0000000000000000 000:00059 /usr/lib/x86_64-linux-gnu/libruby-2.3.so.2.3.0
00007f2ac4351000    2048 ----- 0000000000271000 000:00059 /usr/lib/x86_64-linux-gnu/libruby-2.3.so.2.3.0
00007f2ac4551000      32 r---- 0000000000271000 000:00059 /usr/lib/x86_64-linux-gnu/libruby-2.3.so.2.3.0
00007f2ac4559000       4 rw--- 0000000000279000 000:00059 /usr/lib/x86_64-linux-gnu/libruby-2.3.so.2.3.0
00007f2ac455a000      64 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac456a000     368 r-x-- 0000000000000000 000:00059 /usr/lib/libjemalloc.so.2
00007f2ac45c6000    2044 ----- 000000000005c000 000:00059 /usr/lib/libjemalloc.so.2
00007f2ac47c5000       8 r---- 000000000005b000 000:00059 /usr/lib/libjemalloc.so.2
00007f2ac47c7000       4 rw--- 000000000005d000 000:00059 /usr/lib/libjemalloc.so.2
00007f2ac47c8000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac47c9000     140 r-x-- 0000000000000000 000:00059 /lib/x86_64-linux-gnu/ld-2.24.so
00007f2ac48dd000       8 r-x-- 0000000000000000 000:00000   [ anon ]
00007f2ac48df000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f2ac48e0000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac49e0000       4 r-x-- 0000000000000000 000:00000   [ anon ]
00007f2ac49e1000      16 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac49e5000       4 ----- 0000000000000000 000:00000   [ anon ]
00007f2ac49e6000      12 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac49e9000      12 rw--- 0000000000000000 000:00000   [ anon ]
00007f2ac49ec000       4 r---- 0000000000023000 000:00059 /lib/x86_64-linux-gnu/ld-2.24.so
00007f2ac49ed000       4 rw--- 0000000000024000 000:00059 /lib/x86_64-linux-gnu/ld-2.24.so
00007f2ac49ee000       4 rw--- 0000000000000000 000:00000   [ anon ]
00007ffd9b6de000    8188 rw--- 0000000000000000 000:00000   [ stack ]
00007ffd9bfed000      12 r---- 0000000000000000 000:00000   [ anon ]
00007ffd9bff0000       8 r-x-- 0000000000000000 000:00000   [ anon ]
ffffffffff600000       4 r-x-- 0000000000000000 000:00000   [ anon ]
mapped: 954976K    writeable/private: 796528K    shared: 0K

@lixianyang
Copy link
Author

My test config files: fluentd-es-test

@cosmo0920
Copy link
Collaborator

cosmo0920 commented Sep 4, 2018 •
edited
Loading

I've tested with modified ES plugin and @lixianyang 's test script.

I collected memory usage but I cannot find memory leak on ES plugin.

ES plugin patch:

diff --git a/Gemfile b/Gemfile
index ca132d6..ddb7e03 100644
--- a/Gemfile
+++ b/Gemfile
@@ -6,3 +6,4 @@ gemspec
 gem 'simplecov', require: false
 gem 'coveralls', require: false
 gem 'strptime', require: false if RUBY_ENGINE == "ruby" && RUBY_VERSION =~ /^2/
+gem 'memprof2'
diff --git a/lib/fluent/plugin/out_elasticsearch.rb b/lib/fluent/plugin/out_elasticsearch.rb
index e6cd110..6ea1510 100644
--- a/lib/fluent/plugin/out_elasticsearch.rb
+++ b/lib/fluent/plugin/out_elasticsearch.rb
@@ -4,6 +4,7 @@ require 'excon'
 require 'elasticsearch'
 require 'json'
 require 'uri'
+require 'memprof2'
 begin
   require 'strptime'
 rescue LoadError
@@ -126,8 +127,19 @@ EOC
 
     def initialize
       super
+
+      Memprof2.start
     end
 
+    def shutdown
+      puts " !!!!--- report ---!!!!"
+      Memprof2.report
+      Memprof2.stop
+      puts " !!!!--- report end ---!!!"
+      
+      super
+    end
+
     def configure(conf)
       compat_parameters_convert(conf, :buffer)

Here is memory-report.txt.

Sorted result is:

Memory Usage Where
5247536 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin_helper/thread.rb:70:Thread
3145768 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/excon-0.61.0/lib/excon/connection.rb:187:String
1049200 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/root_agent.rb:226:Thread
557303 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/multi_json-1.13.1/lib/multi_json/adapters/yajl.rb:11:String
125568 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/multi_json-1.13.1/lib/multi_json/adapters/yajl.rb:11:Hash
50024 /home/hhatake/.rbenv/versions/2.5.0/lib/ruby/2.5.0/socket.rb:452:String
32809 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/event.rb:323:String
31476 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/buffer/file_chunk.rb:77:Time
14730 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/registry.rb:111:String
8960 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/section.rb:117:Hash
6912 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:259:Hash
5016 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/out_file.rb:27:Class
4324 /media/work/GitHub/fluent-plugin-elasticsearch/lib/fluent/plugin/out_elasticsearch.rb:302:Regexp
3761 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/buf_file.rb:20:String
3680 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:129:Hash
3288 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/in_forward.rb:25:Class
3264 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/buffer.rb:454:Hash
2944 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/in_forward.rb:247:String
2912 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:130:Hash
2776 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/buf_file.rb:25:Class
2736 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/output.rb:1071:Class
2528 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/buffer/file_chunk.rb:24:Class
2408 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin.rb:158:Class
2344 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:66:Hash
2280 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/configurable.rb:97:Class
2280 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:363:Class
2168 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:67:Hash
2024 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/buffer/chunk.rb:29:Class
2010 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/registry.rb:111:Regexp
2000 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:65:Hash
2000 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/configurable.rb:159:Array
1902 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/buf_file.rb:20:Regexp
1856 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:184:Hash
1856 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:183:Hash
1728 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/multi_json-1.13.1/lib/multi_json/options_cache.rb:12:Hash
1600 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/formatter_out_file.rb:23:Class
1560 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin_helper/server.rb:575:Class
1440 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:326:Array
1408 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/config/configure_proxy.rb:203:Hash
1398 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/buffer/file_chunk.rb:262:String
1392 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/buffer/file_chunk.rb:286:File
1392 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/fluentd-1.1.2/lib/fluent/plugin/buffer/file_chunk.rb:276:File
1168 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/multi_json-1.13.1/lib/multi_json/adapter.rb:5:Class
1128 /media/work/GitHub/fluent-plugin-elasticsearch/vendor/bundle/ruby/2.5.0/gems/multi_json-1.13.1/lib/multi_json/adapters/yajl.rb:7:Class

@cosmo0920
Copy link
Collaborator

i didn't find any culprit of high memory usage/memory leak on ES plugin itself.

@cosmo0920
Copy link
Collaborator

I replaced serialize class to use Oj, but this issue still exists.
#464
Using Oj for serialize class, memory consumption is reduced around 100MB.
But memory consumption increases to be proportional to requests.

@cosmo0920
Copy link
Collaborator

cosmo0920 commented Sep 5, 2018 •
edited
Loading

Currently, I investigated this issue and to be revealed that:

  • Hot spot of memory consumption is JSON serialization/deserialization; mainly, Yajl
  • excon connection/response is also consuming large memory but almost fixed size
  • There is no culprit of memory leak on ES plugin code

@lixianyang
Copy link
Author

@cosmo0920
What's the total memory usage in container when you test?
If the memory consumption increases to be proportional to requests because of JSON serialization/deserialization, how can I avoid fluentd to be killed by oom killer. Do you have any Suggestions?
Thanks for your help

@cosmo0920
Copy link
Collaborator

cosmo0920 commented Sep 5, 2018 •
edited
Loading

With Yajl, around 700〜800MB.
With Oj which is #464 patch case, around 600〜700MB.
In Both of cases, I measured them with modified your test script.

how can I avoid fluentd to be killed by oom killer.

JSON encode/decode needs to consume memory to be proportional JSON size.
It needs to set chunk_limit_size and total_limit_size smaller size, I guess.

If you need to handle enormous logs, you need to increase memory limit.

@lixianyang
Copy link
Author

Thanks

@lintingbin
Copy link

I also have this problem, and memory is high.

pmap -x $pid
12587: /usr/local/bin/ruby -Eascii-8bit:ascii-8bit /usr/local/bundle/bin/fluentd -c conf/fluentd.conf -o logs/fluentd.log --log-rotate-size 10000000 --under-supervisor
Address Kbytes RSS Dirty Mode Mapping
00005640207f3000 4 4 0 r-x-- ruby
00005640209f3000 4 4 4 r---- ruby
00005640209f4000 4 4 4 rw--- ruby
00005640226f9000 52184 46732 46732 rw--- [ anon ]
00007f768c000000 17204 17204 17204 rw--- [ anon ]
00007f768d0cd000 48332 0 0 ----- [ anon ]
00007f7694000000 25584 20936 20936 rw--- [ anon ]
00007f76958fc000 39952 0 0 ----- [ anon ]
00007f769c000000 34500 29272 29272 rw--- [ anon ]
00007f769e1b1000 31036 0 0 ----- [ anon ]
00007f76a4000000 2608 2608 2608 rw--- [ anon ]
00007f76a428c000 62928 0 0 ----- [ anon ]
00007f76ac000000 32432 27352 27352 rw--- [ anon ]
00007f76adfac000 33104 0 0 ----- [ anon ]
00007f76b4000000 30196 27572 27572 rw--- [ anon ]
00007f76b5d7d000 35340 0 0 ----- [ anon ]
00007f76bc000000 33540 22348 22348 rw--- [ anon ]
00007f76be0c1000 31996 0 0 ----- [ anon ]
00007f76c4000000 52168 40608 40608 rw--- [ anon ]
00007f76c72f2000 13368 0 0 ----- [ anon ]
00007f76cc000000 19456 19456 19456 rw--- [ anon ]
00007f76cd300000 46080 0 0 ----- [ anon ]
00007f76d22a5000 4 0 0 ----- [ anon ]
00007f76d22a6000 1024 8 8 rw--- [ anon ]
00007f76d23a6000 4 0 0 ----- [ anon ]
00007f76d23a7000 1024 8 8 rw--- [ anon ]
00007f76d24a7000 4 0 0 ----- [ anon ]
00007f76d24a8000 1024 8 8 rw--- [ anon ]
00007f76d25a8000 4 0 0 ----- [ anon ]
00007f76d25a9000 1024 8 8 rw--- [ anon ]
00007f76d26a9000 4 0 0 ----- [ anon ]
00007f76d26aa000 1024 8 8 rw--- [ anon ]
00007f76d27aa000 4 0 0 ----- [ anon ]
00007f76d27ab000 1024 8 8 rw--- [ anon ]
00007f76d28ab000 4 0 0 ----- [ anon ]
00007f76d28ac000 1024 8 8 rw--- [ anon ]
00007f76d29ac000 4 0 0 ----- [ anon ]
00007f76d29ad000 2052 36 36 rw--- [ anon ]
00007f76d2bae000 4 0 0 ----- [ anon ]
00007f76d2baf000 2052 28 28 rw--- [ anon ]
00007f76d2db0000 4 0 0 ----- [ anon ]
00007f76d2db1000 2052 32 32 rw--- [ anon ]
00007f76d2fb2000 4 0 0 ----- [ anon ]
00007f76d2fb3000 2052 32 32 rw--- [ anon ]
00007f76d31b4000 4 0 0 ----- [ anon ]
00007f76d31b5000 2052 32 32 rw--- [ anon ]
00007f76d33b6000 4 0 0 ----- [ anon ]
00007f76d33b7000 2052 32 32 rw--- [ anon ]
00007f76d35b8000 4 0 0 ----- [ anon ]
00007f76d35b9000 2052 28 28 rw--- [ anon ]
00007f76d37ba000 4 0 0 ----- [ anon ]
00007f76d37bb000 2052 32 32 rw--- [ anon ]
00007f76d39bc000 4 0 0 ----- [ anon ]
00007f76d39bd000 2052 32 32 rw--- [ anon ]
00007f76d3bbe000 4 0 0 ----- [ anon ]
00007f76d3bbf000 2052 32 32 rw--- [ anon ]
00007f76d3dc0000 4 0 0 ----- [ anon ]
00007f76d3dc1000 2052 32 32 rw--- [ anon ]
00007f76d3fc2000 4 0 0 ----- [ anon ]
00007f76d3fc3000 2052 28 28 rw--- [ anon ]
00007f76d41c4000 4 0 0 ----- [ anon ]
00007f76d41c5000 2052 32 32 rw--- [ anon ]
00007f76d43c6000 4 0 0 ----- [ anon ]
00007f76d43c7000 2052 32 32 rw--- [ anon ]
00007f76d45c8000 4 0 0 ----- [ anon ]
00007f76d45c9000 2052 32 32 rw--- [ anon ]
00007f76d47ca000 4 0 0 ----- [ anon ]
00007f76d47cb000 2052 32 32 rw--- [ anon ]
00007f76d49cc000 4 0 0 ----- [ anon ]
00007f76d49cd000 2052 24 24 rw--- [ anon ]
00007f76d4bce000 4 0 0 ----- [ anon ]
00007f76d4bcf000 2052 20 20 rw--- [ anon ]
00007f76d4dd0000 4 0 0 ----- [ anon ]
00007f76d4dd1000 2052 20 20 rw--- [ anon ]
00007f76d4fd2000 4 0 0 ----- [ anon ]
00007f76d4fd3000 2052 24 24 rw--- [ anon ]
00007f76d51d4000 4 0 0 ----- [ anon ]
00007f76d51d5000 2052 20 20 rw--- [ anon ]
00007f76d53d6000 4 0 0 ----- [ anon ]
00007f76d53d7000 2052 28 28 rw--- [ anon ]
00007f76d55d8000 4 0 0 ----- [ anon ]
00007f76d55d9000 2052 32 32 rw--- [ anon ]
00007f76d57da000 4 0 0 ----- [ anon ]
00007f76d57db000 2052 32 32 rw--- [ anon ]
00007f76d59dc000 4 0 0 ----- [ anon ]
00007f76d59dd000 2052 32 32 rw--- [ anon ]
00007f76d5bde000 4 0 0 ----- [ anon ]
00007f76d5bdf000 2052 32 32 rw--- [ anon ]
00007f76d5de0000 4 0 0 ----- [ anon ]
00007f76d5de1000 2052 28 28 rw--- [ anon ]
00007f76d5fe2000 4 0 0 ----- [ anon ]
00007f76d5fe3000 2052 32 32 rw--- [ anon ]
00007f76d61e4000 4 0 0 ----- [ anon ]
00007f76d61e5000 2052 32 32 rw--- [ anon ]
00007f76d63e6000 4 0 0 ----- [ anon ]
00007f76d63e7000 2052 32 32 rw--- [ anon ]
00007f76d65e8000 4 0 0 ----- [ anon ]
00007f76d65e9000 2052 32 32 rw--- [ anon ]
00007f76d67ea000 4 0 0 ----- [ anon ]
00007f76d67eb000 2052 28 28 rw--- [ anon ]
00007f76d69ec000 4 0 0 ----- [ anon ]
00007f76d69ed000 2052 32 32 rw--- [ anon ]
00007f76d6bee000 4 0 0 ----- [ anon ]
00007f76d6bef000 2052 32 32 rw--- [ anon ]
00007f76d6df0000 4 0 0 ----- [ anon ]
00007f76d6df1000 2052 32 32 rw--- [ anon ]
00007f76d6ff2000 4 0 0 ----- [ anon ]
00007f76d6ff3000 2052 32 32 rw--- [ anon ]
00007f76d71f4000 4 0 0 ----- [ anon ]
00007f76d71f5000 2052 28 28 rw--- [ anon ]
00007f76d73f6000 4 0 0 ----- [ anon ]
00007f76d73f7000 2052 32 32 rw--- [ anon ]
00007f76d75f8000 4 0 0 ----- [ anon ]
00007f76d75f9000 2052 32 32 rw--- [ anon ]
00007f76d77fa000 4 0 0 ----- [ anon ]
00007f76d77fb000 2052 32 32 rw--- [ anon ]
00007f76d79fc000 4 0 0 ----- [ anon ]
00007f76d79fd000 2052 32 32 rw--- [ anon ]
00007f76d7bfe000 4 0 0 ----- [ anon ]
00007f76d7bff000 2052 28 28 rw--- [ anon ]
00007f76d7e00000 4 0 0 ----- [ anon ]
00007f76d7e01000 2052 32 32 rw--- [ anon ]
00007f76d8002000 4 0 0 ----- [ anon ]
00007f76d8003000 2052 32 32 rw--- [ anon ]
00007f76d8204000 4 0 0 ----- [ anon ]
00007f76d8205000 2052 32 32 rw--- [ anon ]
00007f76d8406000 4 0 0 ----- [ anon ]
00007f76d8407000 2052 32 32 rw--- [ anon ]
00007f76d8608000 4 0 0 ----- [ anon ]
00007f76d8609000 2052 28 28 rw--- [ anon ]
00007f76d880a000 4 0 0 ----- [ anon ]
00007f76d880b000 2052 32 32 rw--- [ anon ]
00007f76d8a0c000 4 0 0 ----- [ anon ]
00007f76d8a0d000 2052 36 36 rw--- [ anon ]
00007f76d8c0e000 4 0 0 ----- [ anon ]
00007f76d8c0f000 2052 32 32 rw--- [ anon ]
00007f76d8e10000 4 0 0 ----- [ anon ]
00007f76d8e11000 2052 32 32 rw--- [ anon ]
00007f76d9012000 4 0 0 ----- [ anon ]
00007f76d9013000 2052 28 28 rw--- [ anon ]
00007f76d9214000 4 0 0 ----- [ anon ]
00007f76d9215000 2052 32 32 rw--- [ anon ]
00007f76d9416000 4 0 0 ----- [ anon ]
00007f76d9417000 2052 32 32 rw--- [ anon ]
00007f76d9618000 4 0 0 ----- [ anon ]
00007f76d9619000 2052 32 32 rw--- [ anon ]
00007f76d981a000 4 0 0 ----- [ anon ]
00007f76d981b000 2052 32 32 rw--- [ anon ]
00007f76d9a1c000 4 0 0 ----- [ anon ]
00007f76d9a1d000 2052 28 28 rw--- [ anon ]
00007f76d9c1e000 4 0 0 ----- [ anon ]
00007f76d9c1f000 2052 32 32 rw--- [ anon ]
00007f76d9e20000 4 0 0 ----- [ anon ]
00007f76d9e21000 2052 32 32 rw--- [ anon ]
00007f76da022000 4 0 0 ----- [ anon ]
00007f76da023000 2052 32 32 rw--- [ anon ]
00007f76da224000 4 0 0 ----- [ anon ]
00007f76da225000 2052 32 32 rw--- [ anon ]
00007f76da426000 4 0 0 ----- [ anon ]
00007f76da427000 2052 28 28 rw--- [ anon ]
00007f76da628000 4 0 0 ----- [ anon ]
00007f76da629000 2052 32 32 rw--- [ anon ]
00007f76da82a000 4 0 0 ----- [ anon ]
00007f76da82b000 2052 32 32 rw--- [ anon ]
00007f76daa2c000 4 0 0 ----- [ anon ]
00007f76daa2d000 2052 32 32 rw--- [ anon ]
00007f76dac2e000 4 0 0 ----- [ anon ]
00007f76dac2f000 2052 32 32 rw--- [ anon ]
00007f76dae30000 4 0 0 ----- [ anon ]
00007f76dae31000 2052 28 28 rw--- [ anon ]
00007f76db032000 4 0 0 ----- [ anon ]
00007f76db033000 2052 32 32 rw--- [ anon ]
00007f76db234000 4 0 0 ----- [ anon ]
00007f76db235000 2052 32 32 rw--- [ anon ]
00007f76db436000 4 0 0 ----- [ anon ]
00007f76db437000 2052 32 32 rw--- [ anon ]
00007f76db638000 4 0 0 ----- [ anon ]
00007f76db639000 2052 32 32 rw--- [ anon ]
00007f76db83a000 4 0 0 ----- [ anon ]
00007f76db83b000 2052 28 28 rw--- [ anon ]
00007f76dba3c000 4 0 0 ----- [ anon ]
00007f76dba3d000 2052 32 32 rw--- [ anon ]
00007f76dbc3e000 4 0 0 ----- [ anon ]
00007f76dbc3f000 2052 32 32 rw--- [ anon ]
00007f76dbe40000 4 0 0 ----- [ anon ]
00007f76dbe41000 2052 32 32 rw--- [ anon ]
00007f76dc042000 4 0 0 ----- [ anon ]
00007f76dc043000 2052 32 32 rw--- [ anon ]
00007f76dc244000 4 0 0 ----- [ anon ]
00007f76dc245000 2052 28 28 rw--- [ anon ]
00007f76dc446000 4 0 0 ----- [ anon ]
00007f76dc447000 2052 32 32 rw--- [ anon ]
00007f76dc648000 4 0 0 ----- [ anon ]
00007f76dc649000 2052 32 32 rw--- [ anon ]
00007f76dc84a000 4 0 0 ----- [ anon ]
00007f76dc84b000 2052 32 32 rw--- [ anon ]
00007f76dca4c000 4 0 0 ----- [ anon ]
00007f76dca4d000 2052 32 32 rw--- [ anon ]
00007f76dcc4e000 4 0 0 ----- [ anon ]
00007f76dcc4f000 2052 28 28 rw--- [ anon ]
00007f76dce50000 4 0 0 ----- [ anon ]
00007f76dce51000 2052 32 32 rw--- [ anon ]
00007f76dd052000 4 0 0 ----- [ anon ]
00007f76dd053000 2052 32 32 rw--- [ anon ]
00007f76dd254000 4 0 0 ----- [ anon ]
00007f76dd255000 2052 32 32 rw--- [ anon ]
00007f76dd456000 4 0 0 ----- [ anon ]
00007f76dd457000 2052 32 32 rw--- [ anon ]
00007f76dd658000 4 0 0 ----- [ anon ]
00007f76dd659000 2052 28 28 rw--- [ anon ]
00007f76dd85a000 4 0 0 ----- [ anon ]
00007f76dd85b000 2052 32 32 rw--- [ anon ]
00007f76dda5c000 4 0 0 ----- [ anon ]
00007f76dda5d000 2052 32 32 rw--- [ anon ]
00007f76ddc5e000 4 0 0 ----- [ anon ]
00007f76ddc5f000 2052 32 32 rw--- [ anon ]
00007f76dde60000 4 0 0 ----- [ anon ]
00007f76dde61000 2052 32 32 rw--- [ anon ]
00007f76de062000 4 0 0 ----- [ anon ]
00007f76de063000 2052 28 28 rw--- [ anon ]
00007f76de264000 4 0 0 ----- [ anon ]
00007f76de265000 2052 32 32 rw--- [ anon ]
00007f76de466000 4 0 0 ----- [ anon ]
00007f76de467000 2052 32 32 rw--- [ anon ]
00007f76de668000 4 0 0 ----- [ anon ]
00007f76de669000 2052 32 32 rw--- [ anon ]
00007f76de86a000 4 0 0 ----- [ anon ]
00007f76de86b000 2052 32 32 rw--- [ anon ]
00007f76dea6c000 4 0 0 ----- [ anon ]
00007f76dea6d000 2052 28 28 rw--- [ anon ]
00007f76dec6e000 4 0 0 ----- [ anon ]
00007f76dec6f000 2052 32 32 rw--- [ anon ]
00007f76dee70000 4 0 0 ----- [ anon ]
00007f76dee71000 2052 32 32 rw--- [ anon ]
00007f76df072000 4 0 0 ----- [ anon ]
00007f76df073000 2052 32 32 rw--- [ anon ]
00007f76df274000 4 0 0 ----- [ anon ]
00007f76df275000 2052 32 32 rw--- [ anon ]
00007f76df476000 4 0 0 ----- [ anon ]
00007f76df477000 2052 28 28 rw--- [ anon ]
00007f76df678000 4 0 0 ----- [ anon ]
00007f76df679000 2052 32 32 rw--- [ anon ]
00007f76df87a000 4 0 0 ----- [ anon ]
00007f76df87b000 2052 32 32 rw--- [ anon ]
00007f76dfa7c000 4 0 0 ----- [ anon ]
00007f76dfa7d000 2052 32 32 rw--- [ anon ]
00007f76dfc7e000 4 0 0 ----- [ anon ]
00007f76dfc7f000 2052 32 32 rw--- [ anon ]
00007f76dfe80000 4 0 0 ----- [ anon ]
00007f76dfe81000 2052 28 28 rw--- [ anon ]
00007f76e0082000 4 0 0 ----- [ anon ]
00007f76e0083000 2052 32 32 rw--- [ anon ]
00007f76e0284000 4 0 0 ----- [ anon ]
00007f76e0285000 2052 32 32 rw--- [ anon ]
00007f76e0486000 4 0 0 ----- [ anon ]
00007f76e0487000 2052 32 32 rw--- [ anon ]
00007f76e0688000 4 0 0 ----- [ anon ]
00007f76e0689000 2052 32 32 rw--- [ anon ]
00007f76e088a000 4 0 0 ----- [ anon ]
00007f76e088b000 2052 28 28 rw--- [ anon ]
00007f76e0a8c000 4 0 0 ----- [ anon ]
00007f76e0a8d000 2052 32 32 rw--- [ anon ]
00007f76e0c8e000 4 0 0 ----- [ anon ]
00007f76e0c8f000 2052 32 32 rw--- [ anon ]
00007f76e0e90000 4 0 0 ----- [ anon ]
00007f76e0e91000 2052 32 32 rw--- [ anon ]
00007f76e1092000 4 0 0 ----- [ anon ]
00007f76e1093000 2052 32 32 rw--- [ anon ]
00007f76e1294000 4 0 0 ----- [ anon ]
00007f76e1295000 2052 28 28 rw--- [ anon ]
00007f76e1496000 4 0 0 ----- [ anon ]
00007f76e1497000 2052 32 32 rw--- [ anon ]
00007f76e1698000 4 0 0 ----- [ anon ]
00007f76e1699000 2052 32 32 rw--- [ anon ]
00007f76e189a000 4 0 0 ----- [ anon ]
00007f76e189b000 2052 32 32 rw--- [ anon ]
00007f76e1a9c000 4 0 0 ----- [ anon ]
00007f76e1a9d000 2052 32 32 rw--- [ anon ]
00007f76e1c9e000 4 0 0 ----- [ anon ]
00007f76e1c9f000 2052 28 28 rw--- [ anon ]
00007f76e1ea0000 4 0 0 ----- [ anon ]
00007f76e1ea1000 2052 32 32 rw--- [ anon ]
00007f76e20a2000 4 0 0 ----- [ anon ]
00007f76e20a3000 2052 32 32 rw--- [ anon ]
00007f76e22a4000 4 0 0 ----- [ anon ]
00007f76e22a5000 2052 32 32 rw--- [ anon ]
00007f76e24a6000 4 0 0 ----- [ anon ]
00007f76e24a7000 2052 32 32 rw--- [ anon ]
00007f76e26a8000 4 0 0 ----- [ anon ]
00007f76e26a9000 2052 28 28 rw--- [ anon ]
00007f76e28aa000 4 0 0 ----- [ anon ]
00007f76e28ab000 2052 32 32 rw--- [ anon ]
00007f76e2aac000 4 0 0 ----- [ anon ]
00007f76e2aad000 2052 32 32 rw--- [ anon ]
00007f76e2cae000 4 0 0 ----- [ anon ]
00007f76e2caf000 2052 32 32 rw--- [ anon ]
00007f76e2eb0000 4 0 0 ----- [ anon ]
00007f76e2eb1000 2052 32 32 rw--- [ anon ]
00007f76e30b2000 4 0 0 ----- [ anon ]
00007f76e30b3000 2052 28 28 rw--- [ anon ]
00007f76e32b4000 4 0 0 ----- [ anon ]
00007f76e32b5000 2052 32 32 rw--- [ anon ]
00007f76e34b6000 4 0 0 ----- [ anon ]
00007f76e34b7000 2052 32 32 rw--- [ anon ]
00007f76e36b8000 4 0 0 ----- [ anon ]
00007f76e36b9000 2052 32 32 rw--- [ anon ]
00007f76e38ba000 4 0 0 ----- [ anon ]
00007f76e38bb000 2052 36 36 rw--- [ anon ]
00007f76e3abc000 4 0 0 ----- [ anon ]
00007f76e3abd000 2052 28 28 rw--- [ anon ]
00007f76e3cbe000 4 0 0 ----- [ anon ]
00007f76e3cbf000 2052 32 32 rw--- [ anon ]
00007f76e3ec0000 4 0 0 ----- [ anon ]
00007f76e3ec1000 2052 32 32 rw--- [ anon ]
00007f76e40c2000 4 0 0 ----- [ anon ]
00007f76e40c3000 2052 32 32 rw--- [ anon ]
00007f76e42c4000 4 0 0 ----- [ anon ]
00007f76e42c5000 2052 32 32 rw--- [ anon ]
00007f76e44c6000 4 0 0 ----- [ anon ]
00007f76e44c7000 2052 28 28 rw--- [ anon ]
00007f76e46c8000 4 0 0 ----- [ anon ]
00007f76e46c9000 2052 32 32 rw--- [ anon ]
00007f76e48ca000 4 0 0 ----- [ anon ]
00007f76e48cb000 2052 32 32 rw--- [ anon ]
00007f76e4acc000 4 0 0 ----- [ anon ]
00007f76e4acd000 2052 32 32 rw--- [ anon ]
00007f76e4cce000 4 0 0 ----- [ anon ]
00007f76e4ccf000 2052 32 32 rw--- [ anon ]
00007f76e4ed0000 4 0 0 ----- [ anon ]
00007f76e4ed1000 2052 28 28 rw--- [ anon ]
00007f76e50d2000 4 0 0 ----- [ anon ]
00007f76e50d3000 2052 32 32 rw--- [ anon ]
00007f76e52d4000 4 0 0 ----- [ anon ]
00007f76e52d5000 2052 32 32 rw--- [ anon ]
00007f76e54d6000 4 0 0 ----- [ anon ]
00007f76e54d7000 2052 32 32 rw--- [ anon ]
00007f76e56d8000 4 0 0 ----- [ anon ]
00007f76e56d9000 2052 32 32 rw--- [ anon ]
00007f76e58da000 4 0 0 ----- [ anon ]
00007f76e58db000 2052 28 28 rw--- [ anon ]
00007f76e5adc000 4 0 0 ----- [ anon ]
00007f76e5add000 2052 32 32 rw--- [ anon ]
00007f76e5cde000 4 0 0 ----- [ anon ]
00007f76e5cdf000 2052 32 32 rw--- [ anon ]
00007f76e5ee0000 4 0 0 ----- [ anon ]
00007f76e5ee1000 2052 32 32 rw--- [ anon ]
00007f76e60e2000 4 0 0 ----- [ anon ]
00007f76e60e3000 2052 32 32 rw--- [ anon ]
00007f76e62e4000 4 0 0 ----- [ anon ]
00007f76e62e5000 2052 28 28 rw--- [ anon ]
00007f76e64e6000 4 0 0 ----- [ anon ]
00007f76e64e7000 2052 36 36 rw--- [ anon ]
00007f76e66e8000 4 0 0 ----- [ anon ]
00007f76e66e9000 2052 32 32 rw--- [ anon ]
00007f76e68ea000 4 0 0 ----- [ anon ]
00007f76e68eb000 2052 32 32 rw--- [ anon ]
00007f76e6aec000 4 0 0 ----- [ anon ]
00007f76e6aed000 2052 32 32 rw--- [ anon ]
00007f76e6cee000 4 0 0 ----- [ anon ]
00007f76e6cef000 2052 28 28 rw--- [ anon ]
00007f76e6ef0000 4 0 0 ----- [ anon ]
00007f76e6ef1000 2052 32 32 rw--- [ anon ]
00007f76e70f2000 4 0 0 ----- [ anon ]
00007f76e70f3000 2052 32 32 rw--- [ anon ]
00007f76e72f4000 4 0 0 ----- [ anon ]
00007f76e72f5000 2052 32 32 rw--- [ anon ]
00007f76e74f6000 4 0 0 ----- [ anon ]
00007f76e74f7000 2052 32 32 rw--- [ anon ]
00007f76e76f8000 4 0 0 ----- [ anon ]
00007f76e76f9000 2052 28 28 rw--- [ anon ]
00007f76e78fa000 4 0 0 ----- [ anon ]
00007f76e78fb000 2052 32 32 rw--- [ anon ]
00007f76e7afc000 4 0 0 ----- [ anon ]
00007f76e7afd000 2052 32 32 rw--- [ anon ]
00007f76e7cfe000 4 0 0 ----- [ anon ]
00007f76e7cff000 2052 32 32 rw--- [ anon ]
00007f76e7f00000 4 0 0 ----- [ anon ]
00007f76e7f01000 2052 32 32 rw--- [ anon ]
00007f76e8102000 4 0 0 ----- [ anon ]
00007f76e8103000 2052 28 28 rw--- [ anon ]
00007f76e8304000 4 0 0 ----- [ anon ]
00007f76e8305000 2052 32 32 rw--- [ anon ]
00007f76e8506000 4 0 0 ----- [ anon ]
00007f76e8507000 2052 32 32 rw--- [ anon ]
00007f76e8708000 4 0 0 ----- [ anon ]
00007f76e8709000 2052 32 32 rw--- [ anon ]
00007f76e890a000 4 0 0 ----- [ anon ]
00007f76e890b000 2052 32 32 rw--- [ anon ]
00007f76e8b0c000 4 0 0 ----- [ anon ]
00007f76e8b0d000 2052 28 28 rw--- [ anon ]
00007f76e8d0e000 4 0 0 ----- [ anon ]
00007f76e8d0f000 2052 32 32 rw--- [ anon ]
00007f76e8f10000 4 0 0 ----- [ anon ]
00007f76e8f11000 2052 32 32 rw--- [ anon ]
00007f76e9112000 4 0 0 ----- [ anon ]
00007f76e9113000 2052 32 32 rw--- [ anon ]
00007f76e9314000 4 0 0 ----- [ anon ]
00007f76e9315000 2052 32 32 rw--- [ anon ]
00007f76e9516000 4 0 0 ----- [ anon ]
00007f76e9517000 2052 24 24 rw--- [ anon ]
00007f76e9718000 4 0 0 ----- [ anon ]
00007f76e9719000 2052 24 24 rw--- [ anon ]
00007f76e991a000 4 0 0 ----- [ anon ]
00007f76e991b000 2052 28 28 rw--- [ anon ]
00007f76e9b1c000 4 0 0 ----- [ anon ]
00007f76e9b1d000 2052 36 36 rw--- [ anon ]
00007f76e9d1e000 4 0 0 ----- [ anon ]
00007f76e9d1f000 2052 28 28 rw--- [ anon ]
00007f76e9f20000 4 0 0 ----- [ anon ]
00007f76e9f21000 2052 36 36 rw--- [ anon ]
00007f76ea122000 4 0 0 ----- [ anon ]
00007f76ea123000 2052 28 28 rw--- [ anon ]
00007f76ea324000 4 0 0 ----- [ anon ]
00007f76ea325000 2052 36 36 rw--- [ anon ]
00007f76ea526000 4 0 0 ----- [ anon ]
00007f76ea527000 2052 28 28 rw--- [ anon ]
00007f76ea728000 4 0 0 ----- [ anon ]
00007f76ea729000 2052 36 36 rw--- [ anon ]
00007f76ea92a000 4 0 0 ----- [ anon ]
00007f76ea92b000 2052 28 28 rw--- [ anon ]
00007f76eab2c000 4 0 0 ----- [ anon ]
00007f76eab2d000 2052 36 36 rw--- [ anon ]
00007f76ead2e000 4 0 0 ----- [ anon ]
00007f76ead2f000 2052 28 28 rw--- [ anon ]
00007f76eaf30000 4 0 0 ----- [ anon ]
00007f76eaf31000 2052 36 36 rw--- [ anon ]
00007f76eb132000 4 0 0 ----- [ anon ]
00007f76eb133000 2052 28 28 rw--- [ anon ]
00007f76eb334000 4 0 0 ----- [ anon ]
00007f76eb335000 2052 36 36 rw--- [ anon ]
00007f76eb536000 4 0 0 ----- [ anon ]
00007f76eb537000 2052 28 28 rw--- [ anon ]
00007f76eb738000 4 0 0 ----- [ anon ]
00007f76eb739000 2052 36 36 rw--- [ anon ]
00007f76eb93a000 4 0 0 ----- [ anon ]
00007f76eb93b000 2052 28 28 rw--- [ anon ]
00007f76ebb3c000 4 0 0 ----- [ anon ]
00007f76ebb3d000 2052 36 36 rw--- [ anon ]
00007f76ebd3e000 4 0 0 ----- [ anon ]
00007f76ebd3f000 2052 24 24 rw--- [ anon ]
00007f76ebf40000 4 0 0 ----- [ anon ]
00007f76ebf41000 2052 20 20 rw--- [ anon ]
00007f76ec142000 4 0 0 ----- [ anon ]
00007f76ec143000 2052 24 24 rw--- [ anon ]
00007f76ec344000 4 0 0 ----- [ anon ]
00007f76ec345000 2052 20 20 rw--- [ anon ]
00007f76ec546000 4 0 0 ----- [ anon ]
00007f76ec547000 2052 24 24 rw--- [ anon ]
00007f76ec748000 4 0 0 ----- [ anon ]
00007f76ec749000 2052 20 20 rw--- [ anon ]
00007f76ec94a000 4 0 0 ----- [ anon ]
00007f76ec94b000 2052 28 28 rw--- [ anon ]
00007f76ecb4c000 4 0 0 ----- [ anon ]
00007f76ecb4d000 2052 36 36 rw--- [ anon ]
00007f76ecd4e000 4 0 0 ----- [ anon ]
00007f76ecd4f000 2052 28 28 rw--- [ anon ]
00007f76ecf50000 4 0 0 ----- [ anon ]
00007f76ecf51000 2052 36 36 rw--- [ anon ]
00007f76ed152000 4 0 0 ----- [ anon ]
00007f76ed153000 2052 28 28 rw--- [ anon ]
00007f76ed354000 4 0 0 ----- [ anon ]
00007f76ed355000 2052 36 36 rw--- [ anon ]
00007f76ed556000 4 0 0 ----- [ anon ]
00007f76ed557000 2052 24 24 rw--- [ anon ]
00007f76ed758000 4 0 0 ----- [ anon ]
00007f76ed759000 2052 20 20 rw--- [ anon ]
00007f76ed95a000 4 0 0 ----- [ anon ]
00007f76ed95b000 2052 28 28 rw--- [ anon ]
00007f76edb5c000 4 0 0 ----- [ anon ]
00007f76edb5d000 2052 36 36 rw--- [ anon ]
00007f76edd5e000 4 0 0 ----- [ anon ]
00007f76edd5f000 2052 28 28 rw--- [ anon ]
00007f76edf60000 4 0 0 ----- [ anon ]
00007f76edf61000 2052 36 36 rw--- [ anon ]
00007f76ee162000 4 0 0 ----- [ anon ]
00007f76ee163000 2052 28 28 rw--- [ anon ]
00007f76ee364000 4 0 0 ----- [ anon ]
00007f76ee365000 2052 36 36 rw--- [ anon ]
00007f76ee566000 4 0 0 ----- [ anon ]
00007f76ee567000 2052 28 28 rw--- [ anon ]
00007f76ee768000 4 0 0 ----- [ anon ]
00007f76ee769000 2052 36 36 rw--- [ anon ]
00007f76ee96a000 4 0 0 ----- [ anon ]
00007f76ee96b000 2052 28 28 rw--- [ anon ]
00007f76eeb6c000 4 0 0 ----- [ anon ]
00007f76eeb6d000 2052 36 36 rw--- [ anon ]
00007f76eed6e000 4 0 0 ----- [ anon ]
00007f76eed6f000 2052 28 28 rw--- [ anon ]
00007f76eef70000 4 0 0 ----- [ anon ]
00007f76eef71000 2052 36 36 rw--- [ anon ]
00007f76ef172000 4 0 0 ----- [ anon ]
00007f76ef173000 2052 28 28 rw--- [ anon ]
00007f76ef374000 4 0 0 ----- [ anon ]
00007f76ef375000 2052 36 36 rw--- [ anon ]
00007f76ef576000 4 0 0 ----- [ anon ]
00007f76ef577000 2052 24 24 rw--- [ anon ]
00007f76ef778000 4 0 0 ----- [ anon ]
00007f76ef779000 2052 20 20 rw--- [ anon ]
00007f76ef97a000 4 0 0 ----- [ anon ]
00007f76ef97b000 2052 28 28 rw--- [ anon ]
00007f76efb7c000 4 0 0 ----- [ anon ]
00007f76efb7d000 2052 36 36 rw--- [ anon ]
00007f76efd7e000 4 0 0 ----- [ anon ]
00007f76efd7f000 2052 28 28 rw--- [ anon ]
00007f76eff80000 4 0 0 ----- [ anon ]
00007f76eff81000 2052 36 36 rw--- [ anon ]
00007f76f0182000 4 0 0 ----- [ anon ]
00007f76f0183000 2052 28 28 rw--- [ anon ]
00007f76f0384000 4 0 0 ----- [ anon ]
00007f76f0385000 2052 36 36 rw--- [ anon ]
00007f76f0586000 4 0 0 ----- [ anon ]
00007f76f0587000 2052 28 28 rw--- [ anon ]
00007f76f0788000 4 0 0 ----- [ anon ]
00007f76f0789000 2052 36 36 rw--- [ anon ]
00007f76f098a000 4 0 0 ----- [ anon ]
00007f76f098b000 2052 28 28 rw--- [ anon ]
00007f76f0b8c000 4 0 0 ----- [ anon ]
00007f76f0b8d000 2052 36 36 rw--- [ anon ]
00007f76f0d8e000 4 0 0 ----- [ anon ]
00007f76f0d8f000 2052 28 28 rw--- [ anon ]
00007f76f0f90000 4 0 0 ----- [ anon ]
00007f76f0f91000 2052 36 36 rw--- [ anon ]
00007f76f1192000 4 0 0 ----- [ anon ]
00007f76f1193000 2052 28 28 rw--- [ anon ]
00007f76f1394000 4 0 0 ----- [ anon ]
00007f76f1395000 2052 36 36 rw--- [ anon ]
00007f76f1596000 4 0 0 ----- [ anon ]
00007f76f1597000 2052 28 28 rw--- [ anon ]
00007f76f1798000 4 0 0 ----- [ anon ]
00007f76f1799000 2052 36 36 rw--- [ anon ]
00007f76f199a000 4 0 0 ----- [ anon ]
00007f76f199b000 2052 28 28 rw--- [ anon ]
00007f76f1b9c000 4 0 0 ----- [ anon ]
00007f76f1b9d000 2052 36 36 rw--- [ anon ]
00007f76f1d9e000 4 0 0 ----- [ anon ]
00007f76f1d9f000 2052 24 24 rw--- [ anon ]
00007f76f1fa0000 4 0 0 ----- [ anon ]
00007f76f1fa1000 2052 20 20 rw--- [ anon ]
00007f76f21a2000 4 0 0 ----- [ anon ]
00007f76f21a3000 2052 28 28 rw--- [ anon ]
00007f76f23a4000 4 0 0 ----- [ anon ]
00007f76f23a5000 2052 36 36 rw--- [ anon ]
00007f76f25a6000 4 0 0 ----- [ anon ]
00007f76f25a7000 2052 28 28 rw--- [ anon ]
00007f76f27a8000 4 0 0 ----- [ anon ]
00007f76f27a9000 2052 36 36 rw--- [ anon ]
00007f76f29aa000 4 0 0 ----- [ anon ]
00007f76f29ab000 2052 28 28 rw--- [ anon ]
00007f76f2bac000 4 0 0 ----- [ anon ]
00007f76f2bad000 2052 36 36 rw--- [ anon ]
00007f76f2dae000 4 0 0 ----- [ anon ]
00007f76f2daf000 2052 28 28 rw--- [ anon ]
00007f76f2fb0000 4 0 0 ----- [ anon ]
00007f76f2fb1000 2052 36 36 rw--- [ anon ]
00007f76f31b2000 4 0 0 ----- [ anon ]
00007f76f31b3000 2052 28 28 rw--- [ anon ]
00007f76f33b4000 4 0 0 ----- [ anon ]
00007f76f33b5000 2052 36 36 rw--- [ anon ]
00007f76f35b6000 4 0 0 ----- [ anon ]
00007f76f35b7000 2052 28 28 rw--- [ anon ]
00007f76f37b8000 4 0 0 ----- [ anon ]
00007f76f37b9000 2052 36 36 rw--- [ anon ]
00007f76f39ba000 4 0 0 ----- [ anon ]
00007f76f39bb000 2052 28 28 rw--- [ anon ]
00007f76f3bbc000 4 0 0 ----- [ anon ]
00007f76f3bbd000 2052 36 36 rw--- [ anon ]
00007f76f3dbe000 4 0 0 ----- [ anon ]
00007f76f3dbf000 2052 24 24 rw--- [ anon ]
00007f76f3fc0000 4 0 0 ----- [ anon ]
00007f76f3fc1000 2052 20 20 rw--- [ anon ]
00007f76f41c2000 4 0 0 ----- [ anon ]
00007f76f41c3000 2052 28 28 rw--- [ anon ]
00007f76f43c4000 4 0 0 ----- [ anon ]
00007f76f43c5000 2052 36 36 rw--- [ anon ]
00007f76f45c6000 4 0 0 ----- [ anon ]
00007f76f45c7000 2052 24 24 rw--- [ anon ]
00007f76f47c8000 4 0 0 ----- [ anon ]
00007f76f47c9000 2052 20 20 rw--- [ anon ]
00007f76f49ca000 4 0 0 ----- [ anon ]
00007f76f49cb000 2052 28 28 rw--- [ anon ]
00007f76f4bcc000 4 0 0 ----- [ anon ]
00007f76f4bcd000 2052 36 36 rw--- [ anon ]
00007f76f4dce000 4 0 0 ----- [ anon ]
00007f76f4dcf000 2052 24 24 rw--- [ anon ]
00007f76f4fd0000 4 0 0 ----- [ anon ]
00007f76f4fd1000 2052 20 20 rw--- [ anon ]
00007f76f51d2000 4 0 0 ----- [ anon ]
00007f76f51d3000 2052 28 28 rw--- [ anon ]
00007f76f53d4000 4 0 0 ----- [ anon ]
00007f76f53d5000 2052 36 36 rw--- [ anon ]
00007f76f55d6000 4 0 0 ----- [ anon ]
00007f76f55d7000 2052 28 28 rw--- [ anon ]
00007f76f57d8000 4 0 0 ----- [ anon ]
00007f76f57d9000 2052 36 36 rw--- [ anon ]
00007f76f59da000 4 0 0 ----- [ anon ]
00007f76f59db000 2052 28 28 rw--- [ anon ]
00007f76f5bdc000 4 0 0 ----- [ anon ]
00007f76f5bdd000 2052 36 36 rw--- [ anon ]
00007f76f5dde000 4 0 0 ----- [ anon ]
00007f76f5ddf000 2052 28 28 rw--- [ anon ]
00007f76f5fe0000 4 0 0 ----- [ anon ]
00007f76f5fe1000 2052 36 36 rw--- [ anon ]
00007f76f61e2000 4 0 0 ----- [ anon ]
00007f76f61e3000 2052 28 28 rw--- [ anon ]
00007f76f63e4000 4 0 0 ----- [ anon ]
00007f76f63e5000 2052 36 36 rw--- [ anon ]
00007f76f65e6000 4 0 0 ----- [ anon ]
00007f76f65e7000 2052 28 28 rw--- [ anon ]
00007f76f67e8000 4 0 0 ----- [ anon ]
00007f76f67e9000 2052 36 36 rw--- [ anon ]
00007f76f69ea000 4 0 0 ----- [ anon ]
00007f76f69eb000 2052 28 28 rw--- [ anon ]
00007f76f6bec000 4 0 0 ----- [ anon ]
00007f76f6bed000 2052 36 36 rw--- [ anon ]
00007f76f6dee000 4 0 0 ----- [ anon ]
00007f76f6def000 2052 24 24 rw--- [ anon ]
00007f76f6ff0000 4 0 0 ----- [ anon ]
00007f76f6ff1000 2052 24 24 rw--- [ anon ]
00007f76f71f2000 4 0 0 ----- [ anon ]
00007f76f71f3000 2052 28 28 rw--- [ anon ]
00007f76f73f4000 4 0 0 ----- [ anon ]
00007f76f73f5000 2052 36 36 rw--- [ anon ]
00007f76f75f6000 4 0 0 ----- [ anon ]
00007f76f75f7000 2052 24 24 rw--- [ anon ]
00007f76f77f8000 4 0 0 ----- [ anon ]
00007f76f77f9000 2052 20 20 rw--- [ anon ]
00007f76f79fa000 4 0 0 ----- [ anon ]
00007f76f79fb000 2052 28 28 rw--- [ anon ]
00007f76f7bfc000 4 0 0 ----- [ anon ]
00007f76f7bfd000 2052 36 36 rw--- [ anon ]
00007f76f7dfe000 4 0 0 ----- [ anon ]
00007f76f7dff000 2052 28 28 rw--- [ anon ]
00007f76f8000000 65536 64504 64504 rw--- [ anon ]
00007f76fc000000 65536 65488 65488 rw--- [ anon ]
00007f7700000000 65536 65488 65488 rw--- [ anon ]
00007f7704000000 65536 65488 65488 rw--- [ anon ]
00007f7708000000 65536 18176 18176 rw--- [ anon ]
00007f770c000000 63396 26512 26512 rw--- [ anon ]
00007f770fde9000 2140 0 0 ----- [ anon ]
00007f7710000000 64496 52596 52596 rw--- [ anon ]
00007f7713efc000 1040 0 0 ----- [ anon ]
00007f7714000000 10496 10268 10268 rw--- [ anon ]
00007f7714a40000 55040 0 0 ----- [ anon ]
00007f7718000000 65536 33028 33028 rw--- [ anon ]
00007f771c000000 65536 65488 65488 rw--- [ anon ]
00007f7720000000 65536 65488 65488 rw--- [ anon ]
00007f7724000000 65536 65488 65488 rw--- [ anon ]
00007f7728000000 65536 12912 12912 rw--- [ anon ]
00007f772c000000 65536 37416 37416 rw--- [ anon ]
00007f7730000000 65536 65488 65488 rw--- [ anon ]
00007f7734000000 65536 65488 65488 rw--- [ anon ]
00007f7738000000 57812 7120 7120 rw--- [ anon ]
00007f773b875000 7724 0 0 ----- [ anon ]
00007f773c000000 9120 9072 9072 rw--- [ anon ]
00007f773c8e8000 56416 0 0 ----- [ anon ]
00007f7740000000 62920 62268 62268 rw--- [ anon ]
00007f7743d72000 2616 0 0 ----- [ anon ]
00007f7744000000 53200 31920 31920 rw--- [ anon ]
00007f77473f4000 12336 0 0 ----- [ anon ]
00007f7748000000 63504 49960 49960 rw--- [ anon ]
00007f774be04000 2032 0 0 ----- [ anon ]
00007f774c000000 65408 64720 64720 rw--- [ anon ]
00007f774ffe0000 128 0 0 ----- [ anon ]
00007f7750000000 9120 9072 9072 rw--- [ anon ]
00007f77508e8000 56416 0 0 ----- [ anon ]
00007f7754000000 64472 35648 35648 rw--- [ anon ]
00007f7757ef6000 1064 0 0 ----- [ anon ]
00007f7758000000 65536 65488 65488 rw--- [ anon ]
00007f775c000000 65536 65488 65488 rw--- [ anon ]
00007f7760000000 65536 65488 65488 rw--- [ anon ]
00007f7764000000 65536 65488 65488 rw--- [ anon ]
00007f7768000000 65536 60228 60228 rw--- [ anon ]
00007f776c000000 63352 6076 6076 rw--- [ anon ]
00007f776fdde000 2184 0 0 ----- [ anon ]
00007f7770000000 65536 24616 24616 rw--- [ anon ]
00007f7774000000 61320 18212 18212 rw--- [ anon ]
00007f7777be2000 4216 0 0 ----- [ anon ]
00007f77780c1000 4 0 0 ----- [ anon ]
00007f77780c2000 2052 36 36 rw--- [ anon ]
00007f77782c3000 4 0 0 ----- [ anon ]
00007f77782c4000 2052 28 28 rw--- [ anon ]
00007f77784c5000 4 0 0 ----- [ anon ]
00007f77784c6000 2052 36 36 rw--- [ anon ]
00007f77786c7000 4 0 0 ----- [ anon ]
00007f77786c8000 2052 28 28 rw--- [ anon ]
00007f77788c9000 4 0 0 ----- [ anon ]
00007f77788ca000 2052 36 36 rw--- [ anon ]
00007f7778acb000 4 0 0 ----- [ anon ]
00007f7778acc000 2052 28 28 rw--- [ anon ]
00007f7778ccd000 4 0 0 ----- [ anon ]
00007f7778cce000 2052 36 36 rw--- [ anon ]
00007f7778ecf000 4 0 0 ----- [ anon ]
00007f7778ed0000 2052 28 28 rw--- [ anon ]
00007f77790d1000 4 0 0 ----- [ anon ]
00007f77790d2000 2052 36 36 rw--- [ anon ]
00007f77792d3000 4 0 0 ----- [ anon ]
00007f77792d4000 2052 28 28 rw--- [ anon ]
00007f77794d5000 4 0 0 ----- [ anon ]
00007f77794d6000 2052 36 36 rw--- [ anon ]
00007f77796d7000 4 0 0 ----- [ anon ]
00007f77796d8000 2052 24 24 rw--- [ anon ]
00007f77798d9000 4 0 0 ----- [ anon ]
00007f77798da000 2052 20 20 rw--- [ anon ]
00007f7779adb000 4 0 0 ----- [ anon ]
00007f7779adc000 2052 28 28 rw--- [ anon ]
00007f7779cdd000 4 0 0 ----- [ anon ]
00007f7779cde000 2052 36 36 rw--- [ anon ]
00007f7779edf000 4 0 0 ----- [ anon ]
00007f7779ee0000 2052 28 28 rw--- [ anon ]
00007f777a0e1000 4 0 0 ----- [ anon ]
00007f777a0e2000 2052 36 36 rw--- [ anon ]
00007f777a2e3000 4 0 0 ----- [ anon ]
00007f777a2e4000 2052 28 28 rw--- [ anon ]
00007f777a4e5000 4 0 0 ----- [ anon ]
00007f777a4e6000 2052 36 36 rw--- [ anon ]
00007f777a6e7000 4 0 0 ----- [ anon ]
00007f777a6e8000 2052 28 28 rw--- [ anon ]
00007f777a8e9000 4 0 0 ----- [ anon ]
00007f777a8ea000 2052 36 36 rw--- [ anon ]
00007f777aaeb000 4 0 0 ----- [ anon ]
00007f777aaec000 2052 28 28 rw--- [ anon ]
00007f777aced000 4 0 0 ----- [ anon ]
00007f777acee000 2052 36 36 rw--- [ anon ]
00007f777aeef000 4 0 0 ----- [ anon ]
00007f777aef0000 2052 28 28 rw--- [ anon ]
00007f777b0f1000 4 0 0 ----- [ anon ]
00007f777b0f2000 2052 36 36 rw--- [ anon ]
00007f777b2f3000 4 0 0 ----- [ anon ]
00007f777b2f4000 2052 28 28 rw--- [ anon ]
00007f777b4f5000 4 0 0 ----- [ anon ]
00007f777b4f6000 2052 36 36 rw--- [ anon ]
00007f777b6f7000 4 0 0 ----- [ anon ]
00007f777b6f8000 2052 28 28 rw--- [ anon ]
00007f777b8f9000 4 0 0 ----- [ anon ]
00007f777b8fa000 2052 36 36 rw--- [ anon ]
00007f777bafb000 4 0 0 ----- [ anon ]
00007f777bafc000 2052 28 28 rw--- [ anon ]
00007f777bcfd000 4 0 0 ----- [ anon ]
00007f777bcfe000 2052 36 36 rw--- [ anon ]
00007f777beff000 4 0 0 ----- [ anon ]
00007f777bf00000 1024 16 16 rw--- [ anon ]
00007f777c000000 65536 16572 16572 rw--- [ anon ]
00007f7780000000 63780 63732 63732 rw--- [ anon ]
00007f7783e49000 1756 0 0 ----- [ anon ]
00007f7784000000 65536 65488 65488 rw--- [ anon ]
00007f7788000000 65536 65488 65488 rw--- [ anon ]
00007f778c000000 65536 65488 65488 rw--- [ anon ]
00007f7790000000 65536 65488 65488 rw--- [ anon ]
00007f7794000000 65536 29740 29740 rw--- [ anon ]
00007f7798000000 65536 27416 27416 rw--- [ anon ]
00007f779c000000 9244 9196 9196 rw--- [ anon ]
00007f779c907000 56292 0 0 ----- [ anon ]
00007f77a0000000 65536 28648 28648 rw--- [ anon ]
00007f77a4000000 264 216 216 rw--- [ anon ]
00007f77a4042000 65272 0 0 ----- [ anon ]
00007f77a8000000 65536 65488 65488 rw--- [ anon ]
00007f77ac000000 59800 21472 21472 rw--- [ anon ]
00007f77afa66000 5736 0 0 ----- [ anon ]
00007f77b0000000 51532 27956 27956 rw--- [ anon ]
00007f77b3253000 14004 0 0 ----- [ anon ]
00007f77b4000000 57096 18168 18168 rw--- [ anon ]
00007f77b77c2000 8440 0 0 ----- [ anon ]
00007f77b8000000 48824 29284 29284 rw--- [ anon ]
00007f77bafae000 16712 0 0 ----- [ anon ]
00007f77bc000000 65536 42276 42276 rw--- [ anon ]
00007f77c0000000 9052 9016 9016 rw--- [ anon ]
00007f77c08d7000 56484 0 0 ----- [ anon ]
00007f77c4000000 65536 65500 65500 rw--- [ anon ]
00007f77c8000000 65536 65500 65500 rw--- [ anon ]
00007f77cc000000 9256 9160 9160 rw--- [ anon ]
00007f77cc90a000 56280 0 0 ----- [ anon ]
00007f77d0000000 65536 65500 65500 rw--- [ anon ]
00007f77d4000000 64332 64296 64296 rw--- [ anon ]
00007f77d7ed3000 1204 0 0 ----- [ anon ]
00007f77d8000000 65164 62912 62912 rw--- [ anon ]
00007f77dbfa3000 372 0 0 ----- [ anon ]
00007f77dc000000 65520 41012 41012 rw--- [ anon ]
00007f77dfffc000 16 0 0 ----- [ anon ]
00007f77e0000000 65536 65500 65500 rw--- [ anon ]
00007f77e4000000 3016 2980 2980 rw--- [ anon ]
00007f77e42f2000 62520 0 0 ----- [ anon ]
00007f77e8000000 2072 2036 2036 rw--- [ anon ]
00007f77e8206000 63464 0 0 ----- [ anon ]
00007f77ec000000 2396 2360 2360 rw--- [ anon ]
00007f77ec257000 63140 0 0 ----- [ anon ]
00007f77f0000000 27428 1860 1860 rw--- [ anon ]
00007f77f1ac9000 38108 0 0 ----- [ anon ]
00007f77f4000000 7884 7848 7848 rw--- [ anon ]
00007f77f47b3000 57652 0 0 ----- [ anon ]
00007f77f8000000 11524 11340 11340 rw--- [ anon ]
00007f77f8b41000 54012 0 0 ----- [ anon ]
00007f77fc000000 65536 62592 62592 rw--- [ anon ]
00007f7800000000 65536 26360 26360 rw--- [ anon ]
00007f7804000000 65536 65500 65500 rw--- [ anon ]
00007f7808000000 63344 63308 63308 rw--- [ anon ]
00007f780bddc000 2192 0 0 ----- [ anon ]
00007f780c000000 65536 9840 9840 rw--- [ anon ]
00007f7810000000 8716 8680 8680 rw--- [ anon ]
00007f7810883000 56820 0 0 ----- [ anon ]
00007f7814000000 65536 28712 28712 rw--- [ anon ]
00007f78180c1000 1028 12 12 rw--- [ anon ]
00007f78181c2000 4 0 0 ----- [ anon ]
00007f78181c3000 2052 36 36 rw--- [ anon ]
00007f78183c4000 4 0 0 ----- [ anon ]
00007f78183c5000 2052 28 28 rw--- [ anon ]
00007f78185c6000 4 0 0 ----- [ anon ]
00007f78185c7000 2052 36 36 rw--- [ anon ]
00007f78187c8000 4 0 0 ----- [ anon ]
00007f78187c9000 2052 28 28 rw--- [ anon ]
00007f78189ca000 4 0 0 ----- [ anon ]
00007f78189cb000 2052 36 36 rw--- [ anon ]
00007f7818bcc000 4 0 0 ----- [ anon ]
00007f7818bcd000 2052 28 28 rw--- [ anon ]
00007f7818dce000 4 0 0 ----- [ anon ]
00007f7818dcf000 2052 36 36 rw--- [ anon ]
00007f7818fd0000 4 0 0 ----- [ anon ]
00007f7818fd1000 2052 24 24 rw--- [ anon ]
00007f78191d2000 4 0 0 ----- [ anon ]
00007f78191d3000 2052 24 24 rw--- [ anon ]
00007f78193d4000 4 0 0 ----- [ anon ]
00007f78193d5000 2052 28 28 rw--- [ anon ]
00007f78195d6000 4 0 0 ----- [ anon ]
00007f78195d7000 2052 36 36 rw--- [ anon ]
00007f78197d8000 4 0 0 ----- [ anon ]
00007f78197d9000 2052 28 28 rw--- [ anon ]
00007f78199da000 4 0 0 ----- [ anon ]
00007f78199db000 2052 36 36 rw--- [ anon ]
00007f7819bdc000 4 0 0 ----- [ anon ]
00007f7819bdd000 2052 28 28 rw--- [ anon ]
00007f7819dde000 4 0 0 ----- [ anon ]
00007f7819ddf000 2052 36 36 rw--- [ anon ]
00007f7819fe0000 4 0 0 ----- [ anon ]
00007f7819fe1000 2052 28 28 rw--- [ anon ]
00007f781a1e2000 4 0 0 ----- [ anon ]
00007f781a1e3000 2052 36 36 rw--- [ anon ]
00007f781a3e4000 4 0 0 ----- [ anon ]
00007f781a3e5000 2052 28 28 rw--- [ anon ]
00007f781a5e6000 4 0 0 ----- [ anon ]
00007f781a5e7000 2052 36 36 rw--- [ anon ]
00007f781a7e8000 4 0 0 ----- [ anon ]
00007f781a7e9000 2052 28 28 rw--- [ anon ]
00007f781a9ea000 4 0 0 ----- [ anon ]
00007f781a9eb000 2052 36 36 rw--- [ anon ]
00007f781abec000 4 0 0 ----- [ anon ]
00007f781abed000 2052 28 28 rw--- [ anon ]
00007f781adee000 4 0 0 ----- [ anon ]
00007f781adef000 2052 36 36 rw--- [ anon ]
00007f781aff0000 4 0 0 ----- [ anon ]
00007f781aff1000 2052 28 28 rw--- [ anon ]
00007f781b1f2000 4 0 0 ----- [ anon ]
00007f781b1f3000 2052 36 36 rw--- [ anon ]
00007f781b3f4000 4 0 0 ----- [ anon ]
00007f781b3f5000 2052 28 28 rw--- [ anon ]
00007f781b5f6000 4 0 0 ----- [ anon ]
00007f781b5f7000 2052 36 36 rw--- [ anon ]
00007f781b7f8000 4 0 0 ----- [ anon ]
00007f781b7f9000 2052 28 28 rw--- [ anon ]
00007f781b9fa000 4 0 0 ----- [ anon ]
00007f781b9fb000 2052 36 36 rw--- [ anon ]
00007f781bbfc000 4 0 0 ----- [ anon ]
00007f781bbfd000 2052 28 28 rw--- [ anon ]
00007f781bdfe000 4 0 0 ----- [ anon ]
00007f781bdff000 2052 36 36 rw--- [ anon ]
00007f781c000000 65536 17920 17920 rw--- [ anon ]
00007f7820000000 65536 65500 65500 rw--- [ anon ]
00007f7824000000 65536 65500 65500 rw--- [ anon ]
00007f7828000000 10144 9096 9096 rw--- [ anon ]
00007f78289e8000 55392 0 0 ----- [ anon ]
00007f782c000000 65536 65500 65500 rw--- [ anon ]
00007f7830000000 65536 65500 65500 rw--- [ anon ]
00007f7834000000 9112 9076 9076 rw--- [ anon ]
00007f78348e6000 56424 0 0 ----- [ anon ]
00007f7838000000 65536 65500 65500 rw--- [ anon ]
00007f783c000000 65536 65500 65500 rw--- [ anon ]
00007f7840000000 65536 65500 65500 rw--- [ anon ]
00007f7844000000 65536 50620 50620 rw--- [ anon ]
00007f7848000000 65536 34504 34504 rw--- [ anon ]
00007f784c000000 65376 19144 19144 rw--- [ anon ]
00007f784ffd8000 160 0 0 ----- [ anon ]
00007f7850000000 8788 8752 8752 rw--- [ anon ]
00007f7850895000 56748 0 0 ----- [ anon ]
00007f7854000000 65536 31520 31520 rw--- [ anon ]
00007f7858000000 65276 300 300 rw--- [ anon ]
00007f785bfbf000 260 0 0 ----- [ anon ]
00007f785c000000 9144 9108 9108 rw--- [ anon ]
00007f785c8ee000 56392 0 0 ----- [ anon ]
00007f7860000000 65536 40116 40116 rw--- [ anon ]
00007f7864000000 24696 23664 23664 rw--- [ anon ]
00007f786581e000 40840 0 0 ----- [ anon ]
00007f7868000000 4284 4248 4248 rw--- [ anon ]
00007f786842f000 61252 0 0 ----- [ anon ]
00007f786c000000 25256 25000 25000 rw--- [ anon ]
00007f786d8aa000 40280 0 0 ----- [ anon ]
00007f7870000000 9112 9076 9076 rw--- [ anon ]
00007f78708e6000 56424 0 0 ----- [ anon ]
00007f7874000000 7844 7808 7808 rw--- [ anon ]
00007f78747a9000 57692 0 0 ----- [ anon ]
00007f7878000000 26660 26460 26460 rw--- [ anon ]
00007f7879a09000 38876 0 0 ----- [ anon ]
00007f787c0c1000 4 0 0 ----- [ anon ]
00007f787c0c2000 2052 24 24 rw--- [ anon ]
00007f787c2c3000 4 0 0 ----- [ anon ]
00007f787c2c4000 2052 24 24 rw--- [ anon ]
00007f787c4c5000 4 0 0 ----- [ anon ]
00007f787c4c6000 2052 24 24 rw--- [ anon ]
00007f787c6c7000 4 0 0 ----- [ anon ]
00007f787c6c8000 2052 20 20 rw--- [ anon ]
00007f787c8c9000 4 0 0 ----- [ anon ]
00007f787c8ca000 2052 24 24 rw--- [ anon ]
00007f787cacb000 4 0 0 ----- [ anon ]
00007f787cacc000 2052 20 20 rw--- [ anon ]
00007f787cccd000 4 0 0 ----- [ anon ]
00007f787ccce000 2052 28 28 rw--- [ anon ]
00007f787cecf000 4 0 0 ----- [ anon ]
00007f787ced0000 2052 36 36 rw--- [ anon ]
00007f787d0d1000 4 0 0 ----- [ anon ]
00007f787d0d2000 2052 24 24 rw--- [ anon ]
00007f787d2d3000 4 0 0 ----- [ anon ]
00007f787d2d4000 2052 24 24 rw--- [ anon ]
00007f787d4d5000 4 0 0 ----- [ anon ]
00007f787d4d6000 2052 28 28 rw--- [ anon ]
00007f787d6d7000 4 0 0 ----- [ anon ]
00007f787d6d8000 2052 36 36 rw--- [ anon ]
00007f787d8d9000 4 0 0 ----- [ anon ]
00007f787d8da000 2052 24 24 rw--- [ anon ]
00007f787dadb000 4 0 0 ----- [ anon ]
00007f787dadc000 2052 24 24 rw--- [ anon ]
00007f787dcdd000 4 0 0 ----- [ anon ]
00007f787dcde000 2052 28 28 rw--- [ anon ]
00007f787dedf000 4 0 0 ----- [ anon ]
00007f787dee0000 2052 36 36 rw--- [ anon ]
00007f787e0e1000 4 0 0 ----- [ anon ]
00007f787e0e2000 2052 28 28 rw--- [ anon ]
00007f787e2e3000 4 0 0 ----- [ anon ]
00007f787e2e4000 2052 36 36 rw--- [ anon ]
00007f787e4e5000 4 0 0 ----- [ anon ]
00007f787e4e6000 2052 28 28 rw--- [ anon ]
00007f787e6e7000 4 0 0 ----- [ anon ]
00007f787e6e8000 2052 36 36 rw--- [ anon ]
00007f787e8e9000 4 0 0 ----- [ anon ]
00007f787e8ea000 2052 24 24 rw--- [ anon ]
00007f787eaeb000 4 0 0 ----- [ anon ]
00007f787eaec000 2052 20 20 rw--- [ anon ]
00007f787eced000 4 0 0 ----- [ anon ]
00007f787ecee000 2052 28 28 rw--- [ anon ]
00007f787eeef000 4 0 0 ----- [ anon ]
00007f787eef0000 2052 36 36 rw--- [ anon ]
00007f787f0f1000 4 0 0 ----- [ anon ]
00007f787f0f2000 2052 24 24 rw--- [ anon ]
00007f787f2f3000 4 0 0 ----- [ anon ]
00007f787f2f4000 2052 20 20 rw--- [ anon ]
00007f787f4f5000 4 0 0 ----- [ anon ]
00007f787f4f6000 2052 24 24 rw--- [ anon ]
00007f787f6f7000 4 0 0 ----- [ anon ]
00007f787f6f8000 2052 20 20 rw--- [ anon ]
00007f787f8f9000 4 0 0 ----- [ anon ]
00007f787f8fa000 2052 24 24 rw--- [ anon ]
00007f787fafb000 4 0 0 ----- [ anon ]
00007f787fafc000 2052 20 20 rw--- [ anon ]
00007f787fcfd000 4 0 0 ----- [ anon ]
00007f787fcfe000 2052 28 28 rw--- [ anon ]
00007f787feff000 4 0 0 ----- [ anon ]
00007f787ff00000 1024 24 24 rw--- [ anon ]
00007f7880000000 54128 13708 13708 rw--- [ anon ]
00007f78834dc000 11408 0 0 ----- [ anon ]
00007f7884000000 59384 11716 11716 rw--- [ anon ]
00007f78879fe000 6152 0 0 ----- [ anon ]
00007f7888000000 9068 9032 9032 rw--- [ anon ]
00007f78888db000 56468 0 0 ----- [ anon ]
00007f788c000000 31520 20952 20952 rw--- [ anon ]
00007f788dec8000 34016 0 0 ----- [ anon ]
00007f7890000000 14532 14440 14440 rw--- [ anon ]
00007f7890e31000 51004 0 0 ----- [ anon ]
00007f7894000000 15228 15192 15192 rw--- [ anon ]
00007f7894edf000 50308 0 0 ----- [ anon ]
00007f7898000000 13424 13388 13388 rw--- [ anon ]
00007f7898d1c000 52112 0 0 ----- [ anon ]
00007f789c000000 23264 23228 23228 rw--- [ anon ]
00007f789d6b8000 42272 0 0 ----- [ anon ]
00007f78a0000000 65536 24872 24872 rw--- [ anon ]
00007f78a4000000 7884 7848 7848 rw--- [ anon ]
00007f78a47b3000 57652 0 0 ----- [ anon ]
00007f78a8000000 65536 65500 65500 rw--- [ anon ]
00007f78ac000000 65536 65500 65500 rw--- [ anon ]
00007f78b0000000 9104 9068 9068 rw--- [ anon ]
00007f78b08e4000 56432 0 0 ----- [ anon ]
00007f78b4000000 65536 65500 65500 rw--- [ anon ]
00007f78b8000000 65536 29264 29264 rw--- [ anon ]
00007f78bc000000 65536 26168 26168 rw--- [ anon ]
00007f78c0000000 65536 65500 65500 rw--- [ anon ]
00007f78c4000000 65536 18688 18688 rw--- [ anon ]
00007f78c8000000 65536 9952 9952 rw--- [ anon ]
00007f78cc000000 9096 9060 9060 rw--- [ anon ]
00007f78cc8e2000 56440 0 0 ----- [ anon ]
00007f78d0000000 65536 24632 24632 rw--- [ anon ]
00007f78d4000000 65536 29652 29652 rw--- [ anon ]
00007f78d8000000 4868 4804 4804 rw--- [ anon ]
00007f78d84c1000 60668 0 0 ----- [ anon ]
00007f78dc000000 65536 39248 39248 rw--- [ anon ]
00007f78e0000000 65536 65500 65500 rw--- [ anon ]
00007f78e4000000 65536 65500 65500 rw--- [ anon ]
00007f78e8000000 65536 65500 65500 rw--- [ anon ]
00007f78ec000000 65536 65500 65500 rw--- [ anon ]
00007f78f0000000 53004 21236 21236 rw--- [ anon ]
00007f78f33c3000 12532 0 0 ----- [ anon ]
00007f78f4000000 47048 29536 29536 rw--- [ anon ]
00007f78f6df2000 18488 0 0 ----- [ anon ]
00007f78f8000000 48072 42640 42640 rw--- [ anon ]
00007f78faef2000 17464 0 0 ----- [ anon ]
00007f78fc0ad000 1028 12 12 rw--- [ anon ]
00007f78fc1ae000 4 0 0 ----- [ anon ]
00007f78fc1af000 2052 28 28 rw--- [ anon ]
00007f78fc3b0000 4 0 0 ----- [ anon ]
00007f78fc3b1000 2052 36 36 rw--- [ anon ]
00007f78fc5b2000 4 0 0 ----- [ anon ]
00007f78fc5b3000 2052 28 28 rw--- [ anon ]
00007f78fc7b4000 4 0 0 ----- [ anon ]
00007f78fc7b5000 2052 36 36 rw--- [ anon ]
00007f78fc9b6000 4 0 0 ----- [ anon ]
00007f78fc9b7000 2052 24 24 rw--- [ anon ]
00007f78fcbb8000 4 0 0 ----- [ anon ]
00007f78fcbb9000 2052 20 20 rw--- [ anon ]
00007f78fcdba000 4 0 0 ----- [ anon ]
00007f78fcdbb000 2052 28 28 rw--- [ anon ]
00007f78fcfbc000 4 0 0 ----- [ anon ]
00007f78fcfbd000 2052 36 36 rw--- [ anon ]
00007f78fd1be000 4 0 0 ----- [ anon ]
00007f78fd1bf000 2052 24 24 rw--- [ anon ]
00007f78fd3c0000 4 0 0 ----- [ anon ]
00007f78fd3c1000 2052 24 24 rw--- [ anon ]
00007f78fd5c2000 4 0 0 ----- [ anon ]
00007f78fd5c3000 2052 28 28 rw--- [ anon ]
00007f78fd7c4000 4 0 0 ----- [ anon ]
00007f78fd7c5000 2052 36 36 rw--- [ anon ]
00007f78fd9c6000 4 0 0 ----- [ anon ]
00007f78fd9c7000 2052 28 28 rw--- [ anon ]
00007f78fdbc8000 4 0 0 ----- [ anon ]
00007f78fdbc9000 2052 36 36 rw--- [ anon ]
00007f78fddca000 4 0 0 ----- [ anon ]
00007f78fddcb000 2052 28 28 rw--- [ anon ]
00007f78fdfcc000 4 0 0 ----- [ anon ]
00007f78fdfcd000 2052 36 36 rw--- [ anon ]
00007f78fe1ce000 4 0 0 ----- [ anon ]
00007f78fe1cf000 2052 24 24 rw--- [ anon ]
00007f78fe3d0000 4 0 0 ----- [ anon ]
00007f78fe3d1000 2052 20 20 rw--- [ anon ]
00007f78fe5d2000 4 0 0 ----- [ anon ]
00007f78fe5d3000 2052 24 24 rw--- [ anon ]
00007f78fe7d4000 4 0 0 ----- [ anon ]
00007f78fe7d5000 2052 20 20 rw--- [ anon ]
00007f78fe9d6000 4 0 0 ----- [ anon ]
00007f78fe9d7000 2052 24 24 rw--- [ anon ]
00007f78febd8000 4 0 0 ----- [ anon ]
00007f78febd9000 2052 20 20 rw--- [ anon ]
00007f78fedda000 4 0 0 ----- [ anon ]
00007f78feddb000 2052 24 24 rw--- [ anon ]
00007f78fefdc000 4 0 0 ----- [ anon ]
00007f78fefdd000 2052 24 24 rw--- [ anon ]
00007f78ff1de000 4 0 0 ----- [ anon ]
00007f78ff1df000 2052 28 28 rw--- [ anon ]
00007f78ff3e0000 4 0 0 ----- [ anon ]
00007f78ff3e1000 2052 20 20 rw--- [ anon ]
00007f78ff5e2000 4 0 0 ----- [ anon ]
00007f78ff5e3000 2052 24 24 rw--- [ anon ]
00007f78ff7e4000 4 0 0 ----- [ anon ]
00007f78ff7e5000 2052 24 24 rw--- [ anon ]
00007f78ff9e6000 4 0 0 ----- [ anon ]
00007f78ff9e7000 2052 28 28 rw--- [ anon ]
00007f78ffbe8000 20 12 0 r-x-- libnss_dns-2.24.so
00007f78ffbed000 2044 0 0 ----- libnss_dns-2.24.so
00007f78ffdec000 4 4 4 r---- libnss_dns-2.24.so
00007f78ffded000 4 4 4 rw--- libnss_dns-2.24.so
00007f78ffdee000 40 20 0 r-x-- libnss_files-2.24.so
00007f78ffdf8000 2048 0 0 ----- libnss_files-2.24.so
00007f78ffff8000 4 4 4 r---- libnss_files-2.24.so
00007f78ffff9000 4 4 4 rw--- libnss_files-2.24.so
00007f78ffffa000 24 0 0 rw--- [ anon ]
00007f7900000000 58096 24856 24856 rw--- [ anon ]
00007f79038bc000 7440 0 0 ----- [ anon ]
00007f7904007000 4 0 0 ----- [ anon ]
00007f7904008000 2052 36 36 rw--- [ anon ]
00007f7904209000 4 0 0 ----- [ anon ]
00007f790420a000 2052 28 28 rw--- [ anon ]
00007f790440b000 4 0 0 ----- [ anon ]
00007f790440c000 2052 36 36 rw--- [ anon ]
00007f790460d000 4 0 0 ----- [ anon ]
00007f790460e000 3080 1052 1052 rw--- [ anon ]
00007f7904910000 76 0 0 r-x-- libgpg-error.so.0.21.0
00007f7904923000 2044 0 0 ----- libgpg-error.so.0.21.0
00007f7904b22000 4 4 4 r---- libgpg-error.so.0.21.0
00007f7904b23000 4 4 4 rw--- libgpg-error.so.0.21.0
00007f7904b24000 68 0 0 r-x-- libtasn1.so.6.5.3
00007f7904b35000 2048 0 0 ----- libtasn1.so.6.5.3
00007f7904d35000 4 4 4 r---- libtasn1.so.6.5.3
00007f7904d36000 4 4 4 rw--- libtasn1.so.6.5.3
00007f7904d37000 200 0 0 r-x-- libidn.so.11.6.16
00007f7904d69000 2048 0 0 ----- libidn.so.11.6.16
00007f7904f69000 4 4 4 r---- libidn.so.11.6.16
00007f7904f6a000 4 4 4 rw--- libidn.so.11.6.16
00007f7904f6b000 360 0 0 r-x-- libp11-kit.so.0.2.0
00007f7904fc5000 2044 0 0 ----- libp11-kit.so.0.2.0
00007f79051c4000 40 40 40 r---- libp11-kit.so.0.2.0
00007f79051ce000 8 8 8 rw--- libp11-kit.so.0.2.0
00007f79051d0000 104 0 0 r-x-- libsasl2.so.2.0.25
00007f79051ea000 2044 0 0 ----- libsasl2.so.2.0.25
00007f79053e9000 4 4 4 r---- libsasl2.so.2.0.25
00007f79053ea000 4 4 4 rw--- libsasl2.so.2.0.25
00007f79053eb000 80 40 0 r-x-- libresolv-2.24.so
00007f79053ff000 2044 0 0 ----- libresolv-2.24.so
00007f79055fe000 4 4 4 r---- libresolv-2.24.so
00007f79055ff000 4 4 4 rw--- libresolv-2.24.so
00007f7905600000 8 0 0 rw--- [ anon ]
00007f7905602000 12 0 0 r-x-- libkeyutils.so.1.5
00007f7905605000 2044 0 0 ----- libkeyutils.so.1.5
00007f7905804000 4 4 4 r---- libkeyutils.so.1.5
00007f7905805000 4 4 4 rw--- libkeyutils.so.1.5
00007f7905806000 44 0 0 r-x-- libkrb5support.so.0.1
00007f7905811000 2044 0 0 ----- libkrb5support.so.0.1
00007f7905a10000 4 4 4 r---- libkrb5support.so.0.1
00007f7905a11000 4 4 4 rw--- libkrb5support.so.0.1
00007f7905a12000 1052 0 0 r-x-- libgcrypt.so.20.1.6
00007f7905b19000 2048 0 0 ----- libgcrypt.so.20.1.6
00007f7905d19000 8 8 8 r---- libgcrypt.so.20.1.6
00007f7905d1b000 28 28 28 rw--- libgcrypt.so.20.1.6
00007f7905d22000 516 0 0 r-x-- libgmp.so.10.3.2
00007f7905da3000 2048 0 0 ----- libgmp.so.10.3.2
00007f7905fa3000 4 4 4 r---- libgmp.so.10.3.2
00007f7905fa4000 4 4 4 rw--- libgmp.so.10.3.2
00007f7905fa5000 212 0 0 r-x-- libnettle.so.6.3
00007f7905fda000 2044 0 0 ----- libnettle.so.6.3
00007f79061d9000 8 8 8 r---- libnettle.so.6.3
00007f79061db000 4 4 4 rw--- libnettle.so.6.3
00007f79061dc000 204 0 0 r-x-- libhogweed.so.4.3
00007f790620f000 2048 0 0 ----- libhogweed.so.4.3
00007f790640f000 4 4 4 r---- libhogweed.so.4.3
00007f7906410000 4 4 4 rw--- libhogweed.so.4.3
00007f7906411000 1576 0 0 r-x-- libgnutls.so.30.13.1
00007f790659b000 2048 0 0 ----- libgnutls.so.30.13.1
00007f790679b000 52 52 52 r---- libgnutls.so.30.13.1
00007f79067a8000 4 4 4 rw--- libgnutls.so.30.13.1
00007f79067a9000 4 4 4 rw--- [ anon ]
00007f79067aa000 1100 0 0 r-x-- libunistring.so.0.1.2
00007f79068bd000 2044 0 0 ----- libunistring.so.0.1.2
00007f7906abc000 12 12 12 r---- libunistring.so.0.1.2
00007f7906abf000 4 4 4 rw--- libunistring.so.0.1.2
00007f7906ac0000 4 0 0 rw--- [ anon ]
00007f7906ac1000 308 0 0 r-x-- libldap_r-2.4.so.2.10.7
00007f7906b0e000 2044 0 0 ----- libldap_r-2.4.so.2.10.7
00007f7906d0d000 8 8 8 r---- libldap_r-2.4.so.2.10.7
00007f7906d0f000 4 4 4 rw--- libldap_r-2.4.so.2.10.7
00007f7906d10000 8 0 0 rw--- [ anon ]
00007f7906d12000 52 0 0 r-x-- liblber-2.4.so.2.10.7
00007f7906d1f000 2048 0 0 ----- liblber-2.4.so.2.10.7
00007f7906f1f000 4 4 4 r---- liblber-2.4.so.2.10.7
00007f7906f20000 4 4 4 rw--- liblber-2.4.so.2.10.7
00007f7906f21000 12 0 0 r-x-- libcom_err.so.2.1
00007f7906f24000 2044 0 0 ----- libcom_err.so.2.1
00007f7907123000 4 4 4 r---- libcom_err.so.2.1
00007f7907124000 4 4 4 rw--- libcom_err.so.2.1
00007f7907125000 188 0 0 r-x-- libk5crypto.so.3.1
00007f7907154000 2048 0 0 ----- libk5crypto.so.3.1
00007f7907354000 8 8 8 r---- libk5crypto.so.3.1
00007f7907356000 4 4 4 rw--- libk5crypto.so.3.1
00007f7907357000 4 0 0 rw--- [ anon ]
00007f7907358000 804 0 0 r-x-- libkrb5.so.3.3
00007f7907421000 2048 0 0 ----- libkrb5.so.3.3
00007f7907621000 56 56 56 r---- libkrb5.so.3.3
00007f790762f000 12 12 12 rw--- libkrb5.so.3.3
00007f7907632000 288 0 0 r-x-- libgssapi_krb5.so.2.2
00007f790767a000 2044 0 0 ----- libgssapi_krb5.so.2.2
00007f7907879000 8 8 8 r---- libgssapi_krb5.so.2.2
00007f790787b000 8 8 8 rw--- libgssapi_krb5.so.2.2
00007f790787d000 2268 0 0 r-x-- libcrypto.so.1.0.2
00007f7907ab4000 2048 0 0 ----- libcrypto.so.1.0.2
00007f7907cb4000 112 112 112 r---- libcrypto.so.1.0.2
00007f7907cd0000 60 60 60 rw--- libcrypto.so.1.0.2
00007f7907cdf000 16 16 16 rw--- [ anon ]
00007f7907ce3000 380 0 0 r-x-- libssl.so.1.0.2
00007f7907d42000 2048 0 0 ----- libssl.so.1.0.2
00007f7907f42000 16 16 16 r---- libssl.so.1.0.2
00007f7907f46000 24 24 24 rw--- libssl.so.1.0.2
00007f7907f4c000 48 0 0 r-x-- libpsl.so.5.1.1
00007f7907f58000 2048 0 0 ----- libpsl.so.5.1.1
00007f7908158000 4 4 4 r---- libpsl.so.5.1.1
00007f7908159000 4 4 4 rw--- libpsl.so.5.1.1
00007f790815a000 172 0 0 r-x-- libssh2.so.1.0.1
00007f7908185000 2044 0 0 ----- libssh2.so.1.0.1
00007f7908384000 4 4 4 r---- libssh2.so.1.0.1
00007f7908385000 4 4 4 rw--- libssh2.so.1.0.1
00007f7908386000 108 0 0 r-x-- librtmp.so.1
00007f79083a1000 2048 0 0 ----- librtmp.so.1
00007f79085a1000 4 4 4 r---- librtmp.so.1
00007f79085a2000 4 4 4 rw--- librtmp.so.1
00007f79085a3000 132 0 0 r-x-- libidn2.so.0.1.4
00007f79085c4000 2044 0 0 ----- libidn2.so.0.1.4
00007f79087c3000 4 4 4 r---- libidn2.so.0.1.4
00007f79087c4000 4 4 4 rw--- libidn2.so.0.1.4
00007f79087c5000 140 0 0 r-x-- libnghttp2.so.14.12.3
00007f79087e8000 2044 0 0 ----- libnghttp2.so.14.12.3
00007f79089e7000 4 4 4 r---- libnghttp2.so.14.12.3
00007f79089e8000 12 12 12 rw--- libnghttp2.so.14.12.3
00007f79089eb000 500 272 0 r-x-- libcurl.so.4.4.0
00007f7908a68000 2044 0 0 ----- libcurl.so.4.4.0
00007f7908c67000 12 12 12 r---- libcurl.so.4.4.0
00007f7908c6a000 4 4 4 rw--- libcurl.so.4.4.0
00007f7908c6b000 28 12 0 r-x-- libffi.so.6.0.4
00007f7908c72000 2048 0 0 ----- libffi.so.6.0.4
00007f7908e72000 4 4 4 r---- libffi.so.6.0.4
00007f7908e73000 4 4 4 rw--- libffi.so.6.0.4
00007f7908e74000 136 84 0 r-x-- ffi_c.so
00007f7908e96000 2048 0 0 ----- ffi_c.so
00007f7909096000 4 4 4 r---- ffi_c.so
00007f7909097000 4 4 4 rw--- ffi_c.so
00007f7909098000 4 0 0 r-x-- sha2.so
00007f7909099000 2048 0 0 ----- sha2.so
00007f7909299000 4 4 4 r---- sha2.so
00007f790929a000 4 4 4 rw--- sha2.so
00007f790929b000 12 0 0 r-x-- windows_31j.so
00007f790929e000 2044 0 0 ----- windows_31j.so
00007f790949d000 4 4 4 r---- windows_31j.so
00007f790949e000 4 4 4 rw--- windows_31j.so
00007f790949f000 4 0 0 r-x-- sha1.so
00007f79094a0000 2044 0 0 ----- sha1.so
00007f790969f000 4 4 4 r---- sha1.so
00007f79096a0000 4 4 4 rw--- sha1.so
00007f79096a1000 4 0 0 r-x-- md5.so
00007f79096a2000 2044 0 0 ----- md5.so
00007f79098a1000 4 4 4 r---- md5.so
00007f79098a2000 4 4 4 rw--- md5.so
00007f79098a3000 12 8 0 r-x-- escape.so
00007f79098a6000 2044 0 0 ----- escape.so
00007f7909aa5000 4 4 4 r---- escape.so
00007f7909aa6000 4 4 4 rw--- escape.so
00007f7909aa7000 4 0 0 r-x-- fcntl.so
00007f7909aa8000 2044 0 0 ----- fcntl.so
00007f7909ca7000 4 4 4 r---- fcntl.so
00007f7909ca8000 4 4 4 rw--- fcntl.so
00007f7909ca9000 4 0 0 r-x-- nonblock.so
00007f7909caa000 2044 0 0 ----- nonblock.so
00007f7909ea9000 4 4 4 r---- nonblock.so
00007f7909eaa000 4 4 4 rw--- nonblock.so
00007f7909f6c000 12 0 0 r-x-- digest.so
00007f7909f6f000 2048 0 0 ----- digest.so
00007f790a16f000 4 4 4 r---- digest.so
00007f790a170000 4 4 4 rw--- digest.so
00007f790a171000 2472 0 0 r-x-- libcrypto.so.1.1
00007f790a3db000 2048 0 0 ----- libcrypto.so.1.1
00007f790a5db000 120 120 120 r---- libcrypto.so.1.1
00007f790a5f9000 56 56 56 rw--- libcrypto.so.1.1
00007f790a607000 12 12 12 rw--- [ anon ]
00007f790a60a000 396 0 0 r-x-- libssl.so.1.1
00007f790a66d000 2044 0 0 ----- libssl.so.1.1
00007f790a86c000 16 16 16 r---- libssl.so.1.1
00007f790a870000 24 24 24 rw--- libssl.so.1.1
00007f790a876000 336 0 0 r-x-- openssl.so
00007f790a8ca000 2048 0 0 ----- openssl.so
00007f790aaca000 8 8 8 r---- openssl.so
00007f790aacc000 8 8 8 rw--- openssl.so
00007f790aace000 4 4 4 rw--- [ anon ]
00007f790aacf000 72 0 0 r-x-- zlib.so
00007f790aae1000 2044 0 0 ----- zlib.so
00007f790ace0000 4 4 4 r---- zlib.so
00007f790ace1000 4 4 4 rw--- zlib.so
00007f790ace2000 24 12 0 r-x-- strptime.so
00007f790ace8000 2048 0 0 ----- strptime.so
00007f790aee8000 4 4 4 r---- strptime.so
00007f790aee9000 4 4 4 rw--- strptime.so
00007f790aeea000 204 0 0 r-x-- date_core.so
00007f790af1d000 2044 0 0 ----- date_core.so
00007f790b11c000 4 4 4 r---- date_core.so
00007f790b11d000 4 4 4 rw--- date_core.so
00007f790b11e000 4 4 4 rw--- [ anon ]
00007f790b11f000 80 52 0 r-x-- msgpack.so
00007f790b133000 2044 0 0 ----- msgpack.so
00007f790b332000 4 4 4 r---- msgpack.so
00007f790b333000 4 4 4 rw--- msgpack.so
00007f790b334000 64 40 0 r-x-- cool.io_ext.so
00007f790b344000 2044 0 0 ----- cool.io_ext.so
00007f790b543000 4 4 4 r---- cool.io_ext.so
00007f790b544000 4 4 4 rw--- cool.io_ext.so
00007f790b545000 4 4 4 rw--- [ anon ]
00007f790b546000 12 8 0 r-x-- iobuffer_ext.so
00007f790b549000 2044 0 0 ----- iobuffer_ext.so
00007f790b748000 4 4 4 r---- iobuffer_ext.so
00007f790b749000 4 4 4 rw--- iobuffer_ext.so
00007f790b74a000 28 0 0 r-x-- strscan.so
00007f790b751000 2044 0 0 ----- strscan.so
00007f790b950000 4 4 4 r---- strscan.so
00007f790b951000 4 4 4 rw--- strscan.so
00007f790b952000 8 4 0 r-x-- wait.so
00007f790b954000 2044 0 0 ----- wait.so
00007f790bb53000 4 4 4 r---- wait.so
00007f790bb54000 4 4 4 rw--- wait.so
00007f790bb55000 176 84 0 r-x-- socket.so
00007f790bb81000 2044 0 0 ----- socket.so
00007f790bd80000 4 4 4 r---- socket.so
00007f790bd81000 4 4 4 rw--- socket.so
00007f790bd82000 56 52 0 r-x-- yajl.so
00007f790bd90000 2044 0 0 ----- yajl.so
00007f790bf8f000 4 4 4 r---- yajl.so
00007f790bf90000 4 4 4 rw--- yajl.so
00007f790bf91000 36 4 0 r-x-- generator.so
00007f790bf9a000 2044 0 0 ----- generator.so
00007f790c199000 4 4 4 r---- generator.so
00007f790c19a000 4 4 4 rw--- generator.so
00007f790c19b000 20 8 0 r-x-- parser.so
00007f790c1a0000 2048 0 0 ----- parser.so
00007f790c3a0000 4 4 4 r---- parser.so
00007f790c3a1000 4 4 4 rw--- parser.so
00007f790c3a2000 24 4 0 r-x-- etc.so
00007f790c3a8000 2044 0 0 ----- etc.so
00007f790c5a7000 4 4 4 r---- etc.so
00007f790c5a8000 4 4 4 rw--- etc.so
00007f790c5a9000 36 20 0 r-x-- stringio.so
00007f790c5b2000 2044 0 0 ----- stringio.so
00007f790c7b1000 4 4 4 r---- stringio.so
00007f790c7b2000 4 4 4 rw--- stringio.so
00007f790c7b3000 8 4 0 r-x-- transdb.so
00007f790c7b5000 2048 0 0 ----- transdb.so
00007f790c9b5000 4 4 4 r---- transdb.so
00007f790c9b6000 4 4 4 rw--- transdb.so
00007f790c9b7000 8 4 0 r-x-- encdb.so
00007f790c9b9000 2044 0 0 ----- encdb.so
00007f790cbb8000 4 4 4 r---- encdb.so
00007f790cbb9000 4 4 4 rw--- encdb.so
00007f790cbba000 32804 32776 32776 rw--- [ anon ]
00007f790ebc3000 1620 480 0 r-x-- libc-2.24.so
00007f790ed58000 2048 0 0 ----- libc-2.24.so
00007f790ef58000 16 16 16 r---- libc-2.24.so
00007f790ef5c000 8 8 8 rw--- libc-2.24.so
00007f790ef5e000 16 16 16 rw--- [ anon ]
00007f790ef62000 1036 28 0 r-x-- libm-2.24.so
00007f790f065000 2044 0 0 ----- libm-2.24.so
00007f790f264000 4 4 4 r---- libm-2.24.so
00007f790f265000 4 4 4 rw--- libm-2.24.so
00007f790f266000 32 4 0 r-x-- libcrypt-2.24.so
00007f790f26e000 2048 0 0 ----- libcrypt-2.24.so
00007f790f46e000 4 4 4 r---- libcrypt-2.24.so
00007f790f46f000 4 4 4 rw--- libcrypt-2.24.so
00007f790f470000 184 0 0 rw--- [ anon ]
00007f790f49e000 12 4 0 r-x-- libdl-2.24.so
00007f790f4a1000 2044 0 0 ----- libdl-2.24.so
00007f790f6a0000 4 4 4 r---- libdl-2.24.so
00007f790f6a1000 4 4 4 rw--- libdl-2.24.so
00007f790f6a2000 28 4 0 r-x-- librt-2.24.so
00007f790f6a9000 2044 0 0 ----- librt-2.24.so
00007f790f8a8000 4 4 4 r---- librt-2.24.so
00007f790f8a9000 4 4 4 rw--- librt-2.24.so
00007f790f8aa000 96 56 0 r-x-- libpthread-2.24.so
00007f790f8c2000 2044 0 0 ----- libpthread-2.24.so
00007f790fac1000 4 4 4 r---- libpthread-2.24.so
00007f790fac2000 4 4 4 rw--- libpthread-2.24.so
00007f790fac3000 16 4 4 rw--- [ anon ]
00007f790fac7000 100 4 0 r-x-- libz.so.1.2.8
00007f790fae0000 2044 0 0 ----- libz.so.1.2.8
00007f790fcdf000 4 4 4 r---- libz.so.1.2.8
00007f790fce0000 4 4 4 rw--- libz.so.1.2.8
00007f790fce1000 3428 1604 0 r-x-- libruby.so.2.6.1
00007f791003a000 2044 0 0 ----- libruby.so.2.6.1
00007f7910239000 24 24 24 r---- libruby.so.2.6.1
00007f791023f000 12 12 12 rw--- libruby.so.2.6.1
00007f7910242000 68 48 48 rw--- [ anon ]
00007f7910253000 140 44 0 r-x-- ld-2.24.so
00007f7910365000 1044 28 28 rw--- [ anon ]
00007f7910470000 12 12 12 r-x-- [ anon ]
00007f7910473000 12 12 12 rw--- [ anon ]
00007f7910476000 4 4 4 r---- ld-2.24.so
00007f7910477000 4 4 4 rw--- ld-2.24.so
00007f7910478000 4 4 4 rw--- [ anon ]
00007ffc9f082000 8188 56 56 rw--- [ stack ]
00007ffc9f9e9000 8 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]

total kB 10051720 4891440 4888460

@cosmo0920
Copy link
Collaborator

cosmo0920 commented Mar 11, 2019 •
edited
Loading

JSON encode/decode needs to consume memory to be proportional JSON size.
It needs to set chunk_limit_size and total_limit_size smaller size, I guess.

If you need to handle enormous logs, you need to increase memory limit.

Could you check your log flow rate?
If it is high, memory comsumption is increasing by flow rate due to JSON encoding with Yajl.

@lintingbin
Copy link

Log flow rate is high, but how can I find the specific flow rate?
I have set:
prefer_oj_serializer true
http_backend typhoeus
chunk_limit_size 5m
to reduce the memory usage.

@lintingbin
Copy link

I have solved this high memory usage problem. It is the gc problem of ruby. We can solve this problem by ruby-jemalloc.
image
The memory usage can be reduced by about ten times after use jemalloc.

@kvokka
Copy link

kvokka commented Feb 21, 2022 •
edited
Loading

The issue is still in place, the workaround

prefer_oj_serializer true
http_backend typhoeus

is still working with the plugin version 5.1.5 and es version 7.17.0

@kvokka kvokka mentioned this issue Jul 28, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
level:Hard
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@repeatedly @cosmo0920 @lintingbin @lixianyang @kvokka