Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High memory usage #6079

Closed
allenhaozi opened this issue Sep 21, 2022 · 9 comments
Closed

High memory usage #6079

allenhaozi opened this issue Sep 21, 2022 · 9 comments
Labels
status: waiting-for-triage

Comments

@allenhaozi
Copy link

allenhaozi commented Sep 21, 2022 •
edited
Loading

Bug Report

Describe the bug

Very few log files, but really high memory usage

fluent-bit: fluent-bit:1.9.0

NAME                                                     CPU(cores)   MEMORY(bytes)   
alertmanager-prometheus-kube-prometheus-alertmanager-0   2m           47Mi            
fluent-bit-6rglj                                         37m          1102Mi          
fluent-bit-ckmqw                                         2m           9Mi             
fluent-bit-hswrv                                         2m           11Mi            
fluent-bit-kfp7d                                         9m           81Mi            
fluent-bit-mvqq5                                         2m           9Mi             
fluent-bit-ptvhr                                         90m          2457Mi          
fluent-bit-rb2jj                                         58m          1258Mi          
fluent-bit-sf8n2                                         24m          111Mi           
fluent-bit-sk7q5                                         55m          2283Mi         

[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/apisix-etcd-2_ingress-apisix_etcd-4f703a889ed78911e4d1ebfc7fa853fd8178272082a09a22a44113f8be54f623.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/apisix-etcd-2_ingress-apisix_etcd-6801c3613526d5877576908b7ef3c13e029407dbe8820f620b038ca911ad059c.log, inode 13238347
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_calico-node-d616da859c552d8f277514d30842d18ff6163a753b28606730bcb2b7c351e85a.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_calico-node-dbff93e5664fac14dfce6bc0c11d3b309d0da5964a6cd0ad219fc5e44d10b102.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_flexvol-driver-98ad28b4f48e62957fda5542b585cd0a1e8229e7438c66a7126cd51e1c880ca4.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_install-cni-126861a53de82eb2d82bd4e04aea408419be4929d346c4da4b60443ad7cb6bbd.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_kube-flannel-0dfa1b98332458518f032d693b727dc505e693922540cf122643dcd2863cbc4b.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_kube-flannel-6ee75bd916181d22e06dc108b1780e39f0f58331521ed0e462b658af1755ec87.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/cattle-node-agent-9lr9k_cattle-system_agent-3736ccef4a78bf19ee94392fc4facc0e9f2818971774f327b8fd1998f0e0492b.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/cattle-node-agent-9lr9k_cattle-system_agent-f7c334919e4b0574761ee587eb960ff789e096a945c61e4c8131cf5792592ffe.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-qx6ns_consul_client-acl-init-f613e962f1e7d6b620afe2e6618b593a4464c92ad992fcb0d848498eae343e5e.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-qx6ns_consul_consul-933b73dd543a261b48655e5cf9c82452ada82ff2b43f566d993d01638ec59e5b.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/consul-consul-qx6ns_consul_consul-f8a71c8b685b9ae6beafd9980c84ba6d024abfd7708484270d28a9a562522398.log, inode 13238319
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-qx6ns_consul_linkerd-init-ecf90ca5e179038aeee43f280be6c66a116eef0e8f3e5a99b56ce6852cc71a18.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-qx6ns_consul_linkerd-proxy-32a76c1dd299b7e38b442198e758c56ae33fe6627eac8c9cd8b7441362687bc8.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/consul-consul-qx6ns_consul_linkerd-proxy-afe9a713ed29b5c580ed2ef36520c503e99a96d744ad746f55062cafae649960.log, inode 13238325
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-server-1_consul_consul-06b028e5d670acc12c9168ac5b1f50bbb68c0bae234ddad2f6f06bc0524846f5.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/consul-consul-server-1_consul_consul-407efd1759329967ade623d7ed9b352b21c330832b9806f94a6c390eb3fb2ef5.log, inode 22675569
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-server-1_consul_linkerd-init-8f7162f2731db471a4a91eda12ead3c4493b0f5282a3716e439db2e89e2d1f65.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-server-1_consul_linkerd-proxy-480bfa2638208a07a9e20232a9ab4f54f760d7eee5d07074266331bfd9133ec1.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/consul-consul-server-1_consul_linkerd-proxy-495fa5a19e35cc8dede94c85122d64217afa6b671fb4d9da87b87b762244f391.log, inode 13238312
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/csi-nodeplugin-fluid-fsb8z_fluid-system_node-driver-registrar-47868b7c08f2f3f54c6066fcedeee2d5a1de64244c00ba3eed1bdc8d55d6e1a8.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/csi-nodeplugin-fluid-fsb8z_fluid-system_plugins-e1db623a637ccf54ba103f1f36cb932b48d72e3849de49c4e5d6ae1793d19120.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/fluent-bit-ptvhr_logging_fluent-bit-34c01c77f93933596adcc4d857b8eed0eb15dbb749ca72a640fc5d33c0e0abce.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/grafana-7c4bc49848-9d8ng_linkerd-viz_grafana-3729b494dc39cf6bd342a872db7bb2974d7af30b998c3628a6adc8ee8afccd84.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/grafana-7c4bc49848-9d8ng_linkerd-viz_grafana-d58c2d63034f076a6d834753fa08a7871f396094619d4190b718bdf7f27ab6ab.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/grafana-7c4bc49848-9d8ng_linkerd-viz_linkerd-init-c890bc831d786b156e70618b64afe637c0455d04d35a1f072f04b76d1408c7ab.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/grafana-7c4bc49848-9d8ng_linkerd-viz_linkerd-proxy-543022e08d8947314a3f7b29fd99d8f2c5ad15fb282c7a3dde09e3ed02952931.log, inode 13238287
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/grafana-7c4bc49848-9d8ng_linkerd-viz_linkerd-proxy-846b4b566a08a19ee121df6e982a4ea1a4dee4b34d45c7d62f1cb76f454a2788.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/openaios-airbyte-bootloader_airbyte_airbyte-bootloader-container-eb9129259f2c44445d89b83fa048eab5c4ee25726917d2af68fba2180b024c3f.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/openaios-iam-5f66bcf696-6drc2_nlp-iam_openaios-iam-2ecefb36794c4644bb054abe91d0bbe35764fc4c4c32fbc0e4167d8a2107a9bb.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/openaios-iam-7757897c5b-fxx4t_openaios-iam_openaios-iam-1c08115d3da4185bd74faf4d0f099e423d41b84d50c0b7afcaeba581d3143b24.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/openaios-iam-7757897c5b-fxx4t_openaios-iam_openaios-iam-d9b241c75b49f46f51d70870c9c870853184b9eb56c10c857c7dd8218c7c1c30.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/prometheus-node-exporter-q9zf2_logging_node-exporter-835a7511a37f26456c89820135b49c333a42e2ba0f3d4312458507681fcc4c01.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/tap-8589bb5899-72btb_linkerd-viz_linkerd-init-38f0478d3995764e08fba1e47335faf47ad119f4fa62f76c643f556dc7ba835d.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/tap-8589bb5899-72btb_linkerd-viz_linkerd-proxy-69f1b93859330a30decd017c21ee97c414fa419f1fec3abcf0f9f22d504d0d3b.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/tap-8589bb5899-72btb_linkerd-viz_linkerd-proxy-fbb33fead9dbcd15d1de5049dbd44a4492fbeb171fb8cf9b953a0b29a1696170.log, inode 13238278
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/tap-8589bb5899-72btb_linkerd-viz_tap-e5637643561fb641be3dfe8b278f25944f569e0366fae6ae1ac68f0a86bf57db.log, inode 22807104
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/tap-injector-65d747bd67-lbl72_linkerd-viz_tap-injector-14ace39bf19b1d576987692c23460f3efe94352e4ed4b0d210a05081248674a5.log, inode 13238432
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/tap-injector-65d747bd67-lbl72_linkerd-viz_tap-injector-9e2eba69e03c424c6c93a1e731520e162fbd87a179e5913bc4f043bb9c444a24.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/vote-bot-6cd4c65ccd-hnzsn_emojivoto_vote-bot-1b5b9172813a7c311c4cdfff7f0c684a61c34bd7845e63be3dae8b7b63a2fcf9.log, inode 13238285
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/vote-bot-6cd4c65ccd-hnzsn_emojivoto_vote-bot-6092ae56444d9869f80cb9965f1f55dd6fe2b49eeb82d431d274d712d0c0666a.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] inode=13238312 events: IN_MODIFY 
[2022/09/21 17:37:40] [debug] [input chunk] update output instances with new chunk size diff=1489
[2022/09/21 17:37:40] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:40] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:40] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:40] [debug] [upstream] KA connection #34 to loki:3100 has been assigned (recycled)
[2022/09/21 17:37:40] [debug] [http_client] not using http_proxy for header
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] inode=13238278 events: IN_MODIFY 
[2022/09/21 17:37:40] [debug] [input chunk] update output instances with new chunk size diff=1394
[2022/09/21 17:37:40] [debug] [output:loki:loki.0] loki:3100, HTTP status=204
[2022/09/21 17:37:40] [debug] [upstream] KA connection #34 to loki:3100 is now available
[2022/09/21 17:37:40] [debug] [out flush] cb_destroy coro_id=2618
[2022/09/21 17:37:40] [debug] [task] destroy task=0x7fcb8fc54cb0 (task_id=516)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] inode=13238278 events: IN_MODIFY 
[2022/09/21 17:37:40] [debug] [input chunk] update output instances with new chunk size diff=1307
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] inode=13238325 events: IN_MODIFY 
[2022/09/21 17:37:40] [debug] [input chunk] update output instances with new chunk size diff=1451
[2022/09/21 17:37:41] [debug] [input:tail:tail.0] inode=13238285 events: IN_MODIFY 
[2022/09/21 17:37:41] [debug] [input chunk] update output instances with new chunk size diff=749
[2022/09/21 17:37:41] [debug] [task] created task=0x7fcb8fc54cb0 id=516 OK
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [upstream] KA connection #34 to loki:3100 has been assigned (recycled)
[2022/09/21 17:37:41] [debug] [http_client] not using http_proxy for header
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] loki:3100, HTTP status=204
[2022/09/21 17:37:41] [debug] [upstream] KA connection #34 to loki:3100 is now available
[2022/09/21 17:37:41] [debug] [out flush] cb_destroy coro_id=2619
[2022/09/21 17:37:41] [debug] [task] destroy task=0x7fcb92c43ce0 (task_id=50)
[2022/09/21 17:37:41] [debug] [input:tail:tail.0] inode=13238325 events: IN_MODIFY 
[2022/09/21 17:37:41] [debug] [input chunk] update output instances with new chunk size diff=1275
[2022/09/21 17:37:41] [debug] [input chunk] update output instances with new chunk size diff=1262
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238278 events: IN_MODIFY 
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=1307
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238285 events: IN_MODIFY 
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=748
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238347 events: IN_MODIFY 
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=904
[2022/09/21 17:37:42] [debug] [task] created task=0x7fcb92c43ce0 id=50 OK
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238278 events: IN_MODIFY 
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=1394
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [upstream] KA connection #34 to loki:3100 has been assigned (recycled)
[2022/09/21 17:37:42] [debug] [http_client] not using http_proxy for header
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] loki:3100, HTTP status=204
[2022/09/21 17:37:42] [debug] [upstream] KA connection #34 to loki:3100 is now available
[2022/09/21 17:37:42] [debug] [out flush] cb_destroy coro_id=2620
[2022/09/21 17:37:42] [debug] [task] destroy task=0x7fcb8fc54c40 (task_id=517)
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238319 events: IN_MODIFY 
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=1180
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=1250
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238347 events: IN_MODIFY 
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=903
[2022/09/21 17:37:43] [debug] [input:tail:tail.0] inode=13238285 events: IN_MODIFY 
[2022/09/21 17:37:43] [debug] [input chunk] update output instances with new chunk size diff=748
[2022/09/21 17:37:43] [debug] [task] created task=0x7fcb8fc54c40 id=517 OK
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [upstream] KA connection #34 to loki:3100 has been assigned (recycled)
[2022/09/21 17:37:43] [debug] [http_client] not using http_proxy for header
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] loki:3100, HTTP status=204
[2022/09/21 17:37:43] [debug] [upstream] KA connection #34 to loki:3100 is now available
[2022/09/21 17:37:43] [debug] [out flush] cb_destroy coro_id=2621
[2022/09/21 17:37:43] [debug] [task] destroy task=0x7fcb8fc54bd0 (task_id=518)
[2022/09/21 17:37:43] [debug] [input:tail:tail.0] inode=13238278 events: IN_MODIFY 
[2022/09/21 17:37:43] [debug] [input chunk] update output instances with new chunk size diff=1307
@allenhaozi
Copy link
Author

#894

@patrick-stephens
Copy link
Contributor

What's your config? Please follow the issue template as it really helps to capture all the relevant information.

Is this with fresh pods, i.e. no persisted data that has not been sent?
It looks like some failures in the Loki output, are logs being sent for this?

@patrick-stephens patrick-stephens added the waiting-for-user Waiting for more information, tests or requested changes label Sep 21, 2022
@allenhaozi
Copy link
Author

thank you @patrick-stephens

my fluent-bit config:

apiVersion: v1
data:
  custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level debug
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Ignore_Older 1h
        Exclude_Path *_kube-system_*.log,*_logging_*.log
        Refresh_Interval 5
        Skip_Long_Lines On
        DB /var/log/flb_kube.db
        Mem_Buf_Limit 1024MB

    [FILTER]
        Name kubernetes
        Match kube.*
        Buffer_Size 512KB
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        Name loki
        Match kube.*
        Host loki
        Port 3100
        tenant_id ""
        Labels agent=fluent-bit,pod=$kubernetes['pod_name'],namespace=$kubernetes['namespace_name'],name=$kubernetes['labels']['app.kubernetes.io/name'],version=$kubernetes['labels']['app.kubernetes.io/version'],node=$kubernetes['host'],container=$kubernetes['container_name'],instance=$kubernetes['labels']['app.kubernetes.io/instance']
        Label_Keys $stream
        auto_kubernetes_labels off
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: fluent-bit
    meta.helm.sh/release-namespace: logging
  creationTimestamp: "2022-09-19T02:22:59Z"
  labels:
    app.kubernetes.io/instance: fluent-bit
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: fluent-bit
    app.kubernetes.io/version: 1.9.8
    helm.sh/chart: fluent-bit-0.20.8
  name: fluent-bit
  namespace: logging
  resourceVersion: "126984510"
  uid: 1c987657-cf5b-4e27-9031-7000e3a07d2a

loki log info:

level=info ts=2022-09-21T13:08:29.89651657Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:08:29.896537122Z caller=table.go:335 msg="finished handing over table index_19256"
level=info ts=2022-09-21T13:09:29.801949192Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:09:29.802035566Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:09:29.802070393Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:09:29.802097406Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:09:29.80212348Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:09:29.802145177Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:09:29.802174013Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:09:29.896280837Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:09:29.896416004Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:09:29.896450777Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:09:29.896536488Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:09:29.896559498Z caller=table.go:335 msg="finished handing over table index_19256"
level=info ts=2022-09-21T13:09:45.000821372Z caller=checkpoint.go:502 msg="atomic checkpoint finished" old=/data/loki/wal/checkpoint.000124.tmp new=/data/loki/wal/checkpoint.000124
level=info ts=2022-09-21T13:09:45.011993148Z caller=checkpoint.go:573 msg="checkpoint done" time=4m15.014466686s
level=info ts=2022-09-21T13:10:29.80203894Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:10:29.802118817Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:10:29.802139647Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:10:29.802156695Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:10:29.802175513Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:10:29.802187786Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:10:29.802201199Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:10:29.89680063Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:10:29.896832154Z caller=table_manager.go:213 msg="syncing tables"
ts=2022-09-21T13:10:29.896958618Z caller=spanlogger.go:80 level=info msg="building index list cache"
level=info ts=2022-09-21T13:10:29.896972165Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:10:29.897000801Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:10:29.897094435Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:10:29.897134342Z caller=table.go:335 msg="finished handing over table index_19256"
ts=2022-09-21T13:10:29.897255149Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=251.191µs
level=info ts=2022-09-21T13:10:29.897342722Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.37µs distinct_users_len=0
level=info ts=2022-09-21T13:10:29.99703226Z caller=checkpoint.go:615 msg="starting checkpoint"
level=info ts=2022-09-21T13:10:29.997334651Z caller=checkpoint.go:340 msg="attempting checkpoint for" dir=/data/loki/wal/checkpoint.000125
ts=2022-09-21T13:10:36.099466264Z caller=spanlogger.go:80 level=info msg="building index list cache"
ts=2022-09-21T13:10:36.099744711Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=172.947µs
level=info ts=2022-09-21T13:10:36.099819808Z caller=compactor.go:552 msg="compacting table" table-name=index_19255
level=info ts=2022-09-21T13:10:36.099955089Z caller=table.go:138 table-name=index_19255 msg="listed files" count=1
level=info ts=2022-09-21T13:10:36.100043789Z caller=compactor.go:557 msg="finished compacting table" table-name=index_19255
level=info ts=2022-09-21T13:10:36.100070887Z caller=compactor.go:552 msg="compacting table" table-name=index_19256
level=info ts=2022-09-21T13:10:36.100145732Z caller=table.go:138 table-name=index_19256 msg="listed files" count=2
level=info ts=2022-09-21T13:10:36.100167532Z caller=table.go:297 table-name=index_19256 msg="starting compaction of dbs"
level=info ts=2022-09-21T13:10:36.100210147Z caller=table.go:307 table-name=index_19256 msg="using compactor-1663764636.gz as seed file"
level=info ts=2022-09-21T13:10:36.109680003Z caller=util.go:116 table-name=index_19256 file-name=compactor-1663764636.gz msg="downloaded file" total_time=9.45011ms
level=info ts=2022-09-21T13:10:36.110553237Z caller=util.go:116 table-name=index_19256 file-name=loki-0-1663728029801110696-1663764300.gz msg="downloaded file" total_time=672.745µs
level=info ts=2022-09-21T13:10:36.124081021Z caller=util.go:136 msg="compressing the file" src=/data/loki/boltdb-shipper-compactor/index_19256/compactor-1663764636.gz dest=/data/loki/boltdb-shipper-compactor/index_19256/compactor-1663764636.gz.gz
level=info ts=2022-09-21T13:10:36.296783281Z caller=index_set.go:281 table-name=index_19256 msg="removing source db files from storage" count=2
level=info ts=2022-09-21T13:10:36.297714292Z caller=compactor.go:557 msg="finished compacting table" table-name=index_19256
level=info ts=2022-09-21T13:11:29.801821335Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:11:29.801895922Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:11:29.801918072Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:11:29.801939308Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:11:29.801956512Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:11:29.801970722Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:11:29.801986375Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:11:29.896069463Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:11:29.896192401Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:11:29.896216892Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:11:29.896290039Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:11:29.896311959Z caller=table.go:335 msg="finished handing over table index_19256"
level=info ts=2022-09-21T13:12:29.801669404Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:12:29.801773208Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:12:29.801794888Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:12:29.801812861Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:12:29.801830821Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:12:29.801845261Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:12:29.801862381Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:12:29.896512019Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:12:29.896631606Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:12:29.89666187Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:12:29.89674563Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:12:29.896766387Z caller=table.go:335 msg="finished handing over table index_19256"
level=info ts=2022-09-21T13:13:29.801551793Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:13:29.801632587Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:13:29.801654037Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:13:29.801671857Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:13:29.801689057Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:13:29.801732051Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:13:29.801748684Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:13:29.896828604Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:13:29.896976898Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:13:29.897002893Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:13:29.897065941Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:13:29.897085933Z caller=table.go:335 msg="finished handing over table index_19256"
level=info ts=2022-09-21T13:14:29.801861489Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:14:29.801927982Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:14:29.8019487Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:14:29.801964681Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:14:29.801981835Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:14:29.801994078Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:14:29.802007436Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:14:29.89681816Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:14:29.896928704Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:14:29.896952804Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:14:29.897007928Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:14:29.897030271Z caller=table.go:335 msg="finished handing over table index_19256"

@patrick-stephens patrick-stephens removed the waiting-for-user Waiting for more information, tests or requested changes label Sep 21, 2022
@wangyuan0916
Copy link

wangyuan0916 commented Sep 22, 2022 •
edited
Loading

Hi @patrick-stephens I have the same issue, I'm using Fluent-bit 1.9.0 and it is deployed in kubernetes cluster as deployment. There's no data flowing through fluent-bit at this time when I check its dump metrics, and the memory size in input buffer and tasks is low:

[2022/09/21 15:05:53] Fluent Bit Dump

===== Input =====
tail.0 (tail)
│
├─ status
│ └─ overlimit : no
│ ├─ mem size : 22.5K (23003 bytes)
│ └─ mem limit : 4.8M (5000000 bytes)
│
├─ tasks
│ ├─ total tasks : 0
│ ├─ new : 0
│ ├─ running : 0
│ └─ size : 0b (0 bytes)
│
└─ chunks
└─ total chunks : 2
├─ up chunks : 2
├─ down chunks: 0
└─ busy chunks: 0
├─ size : 0b (0 bytes)
└─ size err: 0

forward.1 (forward)
│
├─ status
│ └─ overlimit : no
│ ├─ mem size : 0b (0 bytes)
│ └─ mem limit : 0b (0 bytes)
│
├─ tasks
│ ├─ total tasks : 0
│ ├─ new : 0
│ ├─ running : 0
│ └─ size : 0b (0 bytes)
│
└─ chunks
└─ total chunks : 0
├─ up chunks : 0
├─ down chunks: 0
└─ busy chunks: 0
├─ size : 0b (0 bytes)
└─ size err: 0

===== Storage Layer =====
total chunks : 2
├─ mem chunks : 2
└─ fs chunks : 0
├─ up : 0
└─ down : 0

But its memory usage checking by 'kubectl top pod' is 62Mi. I wonder why the gap is so large...

$ kubectl top pod -A | grep fluent-bit
pks-system fluent-bit-gbcfm 7m 62Mi

The fluent-bit configuration :
fluent-bit.conf:

[SERVICE]
Flush 1
Log_Level warning
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020

input-kubernetes.conf:

[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
multiline.parser docker, cri`

outputs.conf:

[OUTPUT]
Name splunk
Match default
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match k8s-certmanager
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match k8s-dashboard
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match k8s-falco
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match k8s-gatekeeper
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match k8s-monitoring
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match k8s-trident
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match k8s-x509-cert-exporter
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match kube-node-lease
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match kube-public
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match kube-system
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match nsx-system
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name forward
Match pca-uat1-accessdecision
Host ***
Port 24224

[OUTPUT]
Name forward
Match pca-uat2-accessdecision
Host fluentbit-fluent-bit.pch-uat2-shared
Port 24224

[OUTPUT]
Name forward
Match pca-uat3-accessdecision
Host fluentbit-fluent-bit.pch-uat3-shared
Port 24224

[OUTPUT]
Name forward
Match pcc-uat1-contactnote
Host fluentbit-fluent-bit.pch-uat1-shared
Port 24224

[OUTPUT]
Name forward
Match pcc-uat2-contactnote
Host fluentbit-fluent-bit.pch-uat2-shared
Port 24224

[OUTPUT]
Name forward
Match pcc-uat3-contactnote
Host fluentbit-fluent-bit.pch-uat3-shared
Port 24224

[OUTPUT]
Name forward
Match pcd-uat1-pcmdocmgmt
Host fluentbit-fluent-bit.pch-uat1-shared
Port 24224

[OUTPUT]
Name forward
Match pcm-uat1-pcmcore
Host fluentbit-fluent-bit.pch-uat1-shared
Port 24224

[OUTPUT]
Name forward
Match pcm-uat2-pcmcore
Host fluentbit-fluent-bit.pch-uat2-shared
Port 24224

[OUTPUT]
Name forward
Match pcm-uat3-pcmcore
Host fluentbit-fluent-bit.pch-uat3-shared
Port 24224

[OUTPUT]
Name splunk
Match pks-system
Host srp15913lx.juliusbaer.com
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name splunk
Match pks-system-host-monitoring
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

[OUTPUT]
Name forward
Match pmk-uat1-pcmtask
Host fluentbit-fluent-bit.pch-uat1-shared
Port 24224

[OUTPUT]
Name forward
Match pmk-uat2-pcmtask
Host fluentbit-fluent-bit.pch-uat2-shared
Port 24224

[OUTPUT]
Name forward
Match pmk-uat3-pcmtask
Host fluentbit-fluent-bit.pch-uat3-shared
Port 24224

[OUTPUT]
Name splunk
Match vmware-system-csi
Host ***
Port 8088
Splunk_Token ***
tls true
tls.verify false

@mcauto
Copy link

mcauto commented Sep 28, 2022 •
edited
Loading

openssl issue?
confluentinc/librdkafka#3930

@patrick-stephens
Copy link
Contributor

I would also try the latest version 1.9.9 too to confirm it is still present.

@allenhaozi
Copy link
Author

This problem was solved by following @sflanker strategy

#5868 (comment)

@wangyuan0916
Copy link

Hi @allenhaozi We have the same issue with splunk output plugin, can you confirm if it's the same issue?

@allenhaozi
Copy link
Author

hi @wangyuan0916
I followed this plan to upgrade the image and configuration, observed the whole afternoon, so far it has been solved
#5868 (comment)

$k top pods
NAME               CPU(cores)   MEMORY(bytes)   
fluent-bit-2s4vf   4m           29Mi            
fluent-bit-6ksnk   6m           26Mi            
fluent-bit-7cf6q   3m           19Mi            
fluent-bit-7p8b7   20m          85Mi            
fluent-bit-fm5fk   13m          84Mi            
fluent-bit-ggssv   5m           75Mi            
fluent-bit-mfz47   7m           90Mi            
fluent-bit-pxsgv   7m           127Mi           
fluent-bit-tcfbb   4m           28Mi            
loki-0             13m          289Mi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status: waiting-for-triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@allenhaozi @patrick-stephens @mcauto @wangyuan0916