{"action":"closed","issue":{"url":"https://api.github.com/repos/fluent/fluent-bit/issues/693","repository_url":"https://api.github.com/repos/fluent/fluent-bit","labels_url":"https://api.github.com/repos/fluent/fluent-bit/issues/693/labels{/name}","comments_url":"https://api.github.com/repos/fluent/fluent-bit/issues/693/comments","events_url":"https://api.github.com/repos/fluent/fluent-bit/issues/693/events","html_url":"https://github.com/fluent/fluent-bit/issues/693","id":344437553,"node_id":"MDU6SXNzdWUzNDQ0Mzc1NTM=","number":693,"title":"in_systemd memory leak when dynamic tag and filter are enabled","user":{"login":"kabakaev","id":6826176,"node_id":"MDQ6VXNlcjY4MjYxNzY=","avatar_url":"https://avatars1.githubusercontent.com/u/6826176?v=4","gravatar_id":"","url":"https://api.github.com/users/kabakaev","html_url":"https://github.com/kabakaev","followers_url":"https://api.github.com/users/kabakaev/followers","following_url":"https://api.github.com/users/kabakaev/following{/other_user}","gists_url":"https://api.github.com/users/kabakaev/gists{/gist_id}","starred_url":"https://api.github.com/users/kabakaev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kabakaev/subscriptions","organizations_url":"https://api.github.com/users/kabakaev/orgs","repos_url":"https://api.github.com/users/kabakaev/repos","events_url":"https://api.github.com/users/kabakaev/events{/privacy}","received_events_url":"https://api.github.com/users/kabakaev/received_events","type":"User","site_admin":false},"labels":[{"id":171021296,"node_id":"MDU6TGFiZWwxNzEwMjEyOTY=","url":"https://api.github.com/repos/fluent/fluent-bit/labels/bug","name":"bug","color":"fc2929","default":true},{"id":644189272,"node_id":"MDU6TGFiZWw2NDQxODkyNzI=","url":"https://api.github.com/repos/fluent/fluent-bit/labels/fixed","name":"fixed","color":"1d76db","default":false}],"state":"closed","locked":false,"assignee":{"login":"edsiper","id":369718,"node_id":"MDQ6VXNlcjM2OTcxOA==","avatar_url":"https://avatars3.githubusercontent.com/u/369718?v=4","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false},"assignees":[{"login":"edsiper","id":369718,"node_id":"MDQ6VXNlcjM2OTcxOA==","avatar_url":"https://avatars3.githubusercontent.com/u/369718?v=4","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false}],"milestone":null,"comments":9,"created_at":"2018-07-25T13:12:32Z","updated_at":"2018-10-31T03:13:22Z","closed_at":"2018-10-31T03:13:21Z","author_association":"NONE","body":"## Bug Report\r\n`fluent-bit` is consuming RAM till it gets killed by kernel OOM killer at 400MB.\r\n\r\nI tried to find which part is using so much memory. Valgrind suggests that it is some part of `in_systemd_collect()` in `systemd.c`.\r\n\r\nThe issue occurs only when a bunch of filters are enabled between input and oputput.\r\n\r\n**Steps to reproduce**\r\n\r\n```\r\n$ git log -1\r\ncommit adec302571268125aca5f2edd986a3ca715f18d1 (HEAD -> master, origin/master, origin/HEAD)\r\n$ cd /tmp/src/build/\r\n$ sudo apt-get install valgrind\r\n$ cmake   -DFLB_DEBUG=On \\\r\n          -DFLB_TRACE=On \\\r\n          -DFLB_MTRACE=On \\\r\n          -DFLB_VALGRIND=On \\\r\n          -DFLB_JEMALLOC=On \\\r\n          -DFLB_BUFFERING=On \\\r\n          -DFLB_TLS=On \\\r\n          -DFLB_WITHOUT_SHARED_LIB=On \\\r\n          -DFLB_WITHOUT_EXAMPLES=On \\\r\n          -DFLB_HTTP_SERVER=On \\\r\n          -DFLB_OUT_KAFKA=Off ..\r\n$ make -j 8\r\n# create docker container, put custom config into `/fluent-bit/etc/fluent-bit.conf`, as given below\r\n$ /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit --sosreport\r\nFluent-Bit v0.14.0\r\nCopyright (C) Treasure Data\r\n\r\n[2018/07/25 10:31:12] [error] [parser:crio] Invalid format Regex\r\n[2018/07/25 10:31:12] [ info] [engine] started (pid=186)\r\n[2018/07/25 10:31:12] [debug] [in_systemd] add filter: _SYSTEMD_UNIT=docker.service (or)\r\n[2018/07/25 10:31:12] [debug] [in_systemd] add filter: _SYSTEMD_UNIT=docker.service (or)\r\n[2018/07/25 10:31:12] [debug] [in_tail] inotify watch fd=64\r\n[2018/07/25 10:31:12] [debug] [in_tail] scanning path /var/log/k8s/kube-apiserver/audit.log\r\n[2018/07/25 10:31:12] [debug] [in_tail] Cannot read info from: /var/log/k8s/kube-apiserver/audit.log\r\n[2018/07/25 10:31:12] [debug] [filter_modify] Initialized modify filter with 0 conditions and 16 rules\r\n[2018/07/25 10:31:12] [ warn] [filter_kube] merge_json_log is deprecated, enabling 'merge_log' option instead\r\n[2018/07/25 10:31:12] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443\r\n[2018/07/25 10:31:12] [ info] [filter_kube] local POD info OK\r\n[2018/07/25 10:31:12] [ info] [filter_kube] testing connectivity with API server...\r\n[2018/07/25 10:31:12] [debug] [filter_kube] API Server (ns=kube-system, pod=fluent-bit-hwrqx) http_do=0, HTTP Status: 200\r\n[2018/07/25 10:31:12] [ info] [filter_kube] API server connectivity OK\r\n[2018/07/25 10:31:12] [debug] [router] input=systemd.0 'DYNAMIC TAG'\r\n[2018/07/25 10:31:12] [debug] [router] input=systemd.1 'DYNAMIC TAG'\r\n[2018/07/25 10:31:12] [debug] [router] input=systemd.2 'DYNAMIC TAG'\r\n[2018/07/25 10:31:12] [debug] [router] input=tail.0 'DYNAMIC TAG'\r\n\r\nFluent Bit Enterprise - SOS Report\r\n==================================\r\nThe following report aims to be used by Fluent Bit and Fluentd Enterprise\r\nCustomers of Treasure Data. For more details visit:\r\n\r\n    https://fluentd.treasuredata.com\r\n\r\n[Fluent Bit]\r\n    Edition             Community Edition\r\n    Version             0.14.0\r\n    Built Flags          JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_SQLDB FLB_HAVE_BUFFERING FLB_HAVE_TRACE FLB_HAVE_METRICS FLB_HAVE_HTTP_SERVER FLB_HAVE_FLUSH_LIBCO FLB_HAVE_SYSTEMD FLB_HAVE_VALGRIND FLB_HAVE_FORK FLB_HAVE_MTRACE FLB_HAVE_TIMESPEC_GET FLB_HAVE_PROXY_GO FLB\r\n_HAVE_JEMALLOC JEMALLOC_MANGLE FLB_HAVE_LIBBACKTRACE FLB_HAVE_REGEX FLB_HAVE_LUAJIT FLB_HAVE_C_TLS FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY\r\n\r\n[Operating System]\r\n    Name                Linux\r\n    Release             3.10.0-693.21.1.el7.x86_64\r\n    Version             #1 SMP Wed Mar 7 19:03:37 UTC 2018\r\n\r\n[Hardware]\r\n    Architecture        x86_64\r\n    Processors          2\r\n\r\n[Built Plugins]\r\n    Inputs              cpu mem kmsg tail proc disk systemd netif dummy head exec health serial stdin tcp mqtt lib forward random syslog\r\n    Filters             grep stdout throttle kubernetes parser lua record_modifier nest modify\r\n    Outputs             azure counter es exit file forward http influxdb kafka-rest nats null plot splunk stackdriver stdout td lib flowcounter\r\n\r\n[SERVER] Runtime configuration\r\n    Flush               1\r\n    Daemon              Off\r\n    Log_Level           Debug\r\n\r\n[INPUT] Instance\r\n    Name                systemd.0 (systemd, id=0)\r\n    Flags               DYN_TAG\r\n    Threaded            No\r\n    Tag                 sys_systemd_.*\r\n    Mem_Buf_Limit       976.6K\r\n    Path                /var/log/journal\r\n    Max_Entries         5000\r\n    Read_From_Tail      true\r\n\r\n[INPUT] Instance\r\n    Name                systemd.1 (systemd, id=1)\r\n    Flags               DYN_TAG\r\n    Threaded            No\r\n    Tag                 sys_k8s_.*\r\n    Mem_Buf_Limit       976.6K\r\n    Path                /var/log/journal\r\n    Max_Entries         5000\r\n    Systemd_Filter      _SYSTEMD_UNIT=docker.service\r\n    Read_From_Tail      true\r\n\r\n[INPUT] Instance\r\n    Name                systemd.2 (systemd, id=2)\r\n    Flags               DYN_TAG\r\n    Threaded            No\r\n    Tag                 user_k8s_.*\r\n    Mem_Buf_Limit       976.6K\r\n    Path                /var/log/journal\r\n    Max_Entries         5000\r\n    Systemd_Filter      _SYSTEMD_UNIT=docker.service\r\n    Read_From_Tail      true\r\n\r\n[INPUT] Instance\r\n    Name                tail.0 (tail, id=0)\r\n    Flags               DYN_TAG\r\n    Threaded            No\r\n    Tag                 sys_k8saudit\r\n    Mem_Buf_Limit       976.6K\r\n    Path                /var/log/k8s/kube-apiserver/audit.log\r\n    Parser              k8saudit_parser\r\n    Refresh_Interval    61\r\n    Skip_Long_Lines     Off\r\n    Read_From_Tail      true\r\n\r\n[FILTER] Instance\r\n    Name                grep.0 (grep, id=0)\r\n    Match               *_k8s_*\r\n    Regex               CONTAINER_NAME k8s_\r\n    Exclude             CONTAINER_NAME k8s_fluent-bit\r\n[FILTER] Instance\r\n    Name                record_modifie (record_modifier, id=0)\r\n    Match               *_k8s_*\r\n    Remove_key          _EXE\r\n    Remove_key          EXE\r\n    Remove_key          _CAP_EFFECTIVE\r\n    Remove_key          CAP_EFFECTIVE\r\n    Remove_key          _CMDLINE\r\n    Remove_key          CMDLINE\r\n    Remove_key          _COMM\r\n    Remove_key          COMM\r\n    Remove_key          _GID\r\n    Remove_key          GID\r\n    Remove_key          _PID\r\n    Remove_key          PID\r\n    Remove_key          _SOURCE_REALTIME_TIMESTAMP\r\n    Remove_key          SOURCE_REALTIME_TIMESTAMP\r\n    Remove_key          _SYSTEMD_CGROUP\r\n    Remove_key          SYSTEMD_CGROUP\r\n    Remove_key          _SYSTEMD_SLICE\r\n    Remove_key          SYSTEMD_SLICE\r\n    Remove_key          _SYSTEMD_UNIT\r\n    Remove_key          SYSTEMD_UNIT\r\n    Remove_key          _TRANSPORT\r\n    Remove_key          TRANSPORT\r\n    Remove_key          _UID\r\n    Remove_key          UID\r\n    Remove_key          PRIORITY\r\n    Remove_key          SYSLOG_IDENTIFIER\r\n[FILTER] Instance\r\n    Name                modify.0 (modify, id=0)\r\n    Match               *\r\n    Rename              _BOOT_ID BOOT_ID\r\n    Rename              _CAP_EFFECTIVE CAP_EFFECTIVE\r\n    Rename              _CMDLINE CMDLINE\r\n    Rename              _COMM COMM\r\n    Rename              _EXE EXE\r\n    Rename              _GID GID\r\n    Rename              _HOSTNAME HOSTNAME\r\n    Rename              _MACHINE_ID MACHINE_ID\r\n    Rename              _PID PID\r\n    Rename              _SOURCE_REALTIME_TIMESTAMP SOURCE_REALTIME_TIMESTAMP\r\n    Rename              _SYSTEMD_CGROUP SYSTEMD_CGROUP\r\n    Rename              _SYSTEMD_SLICE SYSTEMD_SLICE\r\n    Rename              _SYSTEMD_UNIT SYSTEMD_UNIT\r\n    Rename              _TRANSPORT TRANSPORT\r\n    Rename              _UID UID\r\n    Rename              MESSAGE log\r\n[FILTER] Instance\r\n    Name                grep.1 (grep, id=1)\r\n    Match               user_k8s_*\r\n    Exclude             CONTAINER_NAME k8s_.*_kube-system_\r\n[FILTER] Instance\r\n    Name                grep.2 (grep, id=2)\r\n    Match               sys_k8s_*\r\n    Regex               CONTAINER_NAME k8s_.*_kube-system_\r\n    Regex               CONTAINER_NAME k8s_.*_kube-public_\r\n    Regex               CONTAINER_NAME k8s_.*_tc-system_\r\n[FILTER] Instance\r\n    Name                grep.3 (grep, id=3)\r\n    Match               sys_systemd_.*\r\n    Exclude             CONTAINER_NAME k8s_\r\n[FILTER] Instance\r\n    Name                kubernetes.0 (kubernetes, id=0)\r\n    Match               *_k8s_*\r\n    Use_Journal         true\r\n    Tag_Key             flb-tag\r\n    Kube_URL            https://kubernetes.default.svc:443\r\n    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\r\n    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token\r\n    Merge_JSON_Log      On\r\n    K8S-Logging.Parser  On\r\n    K8S-Logging.Exclude On\r\n\r\n[OUTPUT] Instance\r\n    Name                file.0 (file, mask_id=1)\r\n    Match               *\r\n    TLS Active          No\r\n    Retry Limit         1\r\n    Path                /dev/null\r\n\r\n$ while true ; do logger asdf ; echo ok ; done &\r\n# collect memory usage of a running fluent-bit\r\n$ /usr/bin/valgrind.bin --leak-check=yes /fluent-bit/bin/fluent-bit -c /tmp/v2/fluent-bit.conf &>/tmp/flb.out &\r\n$ sleep 1800\r\n$ gdb\r\n(gdb) set non-stop off\r\n(gdb) target remote | vgdb\r\n(gdb) monitor xtmemory\r\n==162== xtree memory report: /tmp/v2/xtmemory.kcg.162.2\r\n(gdb) quit\r\n```\r\n\r\n[xtmemory.kcg.162.2.gz](https://github.com/fluent/fluent-bit/files/2227976/xtmemory.kcg.162.2.gz)\r\n\r\nUse `kcachegrind` to render the /tmp/v2/xtmemory.kcg.162.2:\r\n![flb-with-filters](https://user-images.githubusercontent.com/6826176/43202527-0d45f3e6-901c-11e8-8453-7cebc5b111d7.png)\r\n\r\n\r\nNow, remove all `[FILTER]` sections from the config and retry.\r\n\r\n```\r\n$ /usr/bin/valgrind.bin --leak-check=yes /fluent-bit/bin/fluent-bit -c /tmp/v2/fluent-bit.conf-512kb-no-filter &> flb-nofilter.out &\r\n$ sleep 1800\r\n$ gdb\r\n(gdb) set non-stop off\r\n(gdb) target remote | vgdb\r\n(gdb) monitor xtmemory\r\n==241== xtree memory report: /tmp/v2/xtmemory.kcg.241.4\r\n(gdb) quit\r\n```\r\n[xtmemory.kcg.241.4.gz](https://github.com/fluent/fluent-bit/files/2227979/xtmemory.kcg.241.4.gz)\r\n\r\nMemory profile without filters looks different:\r\n![flb-no-filters](https://user-images.githubusercontent.com/6826176/43202579-2d41852a-901c-11e8-8f20-3ca19caf9859.png)\r\n\r\nThe `fluent-bit` RAM usage without filters does not increase too much over time.\r\n\r\n**Expected behavior**\r\n`in_systemd` with `[FILTER]`s should not consume too much RAM.\r\n\r\n\r\n**Environment**\r\n* Version used: master rev. adec3025.\r\n* Configuration, filters and plugins: see \"SOSreport\" above.\r\n* Environment name and version: Kubernetes-v1.9.9\r\n* Server type and version: standard docker image from fluent-bit-v0.13.2. Profiling is done on the image built with `cmake` parameters given above.\r\n* Operating System and version: CentOS-7, vanilla docker-v18.03.\r\n\r\n**Additional context**\r\nI aim to collect docker logs from systemd, enrich them with pod info from kubernetes api, and send separately to various stores:\r\n* all logs from user containers should be sent to a user-managed `fluent-bit` instance using `out_forward`;\r\n* all other systemd logs should go into an `elasticsearch` instance.\r\n"}}
{"actor":{"display_login":"edsiper"}}