{"action":"created","issue":{"url":"https://api.github.com/repos/balabit/syslog-ng/issues/1134","repository_url":"https://api.github.com/repos/balabit/syslog-ng","labels_url":"https://api.github.com/repos/balabit/syslog-ng/issues/1134/labels{/name}","comments_url":"https://api.github.com/repos/balabit/syslog-ng/issues/1134/comments","events_url":"https://api.github.com/repos/balabit/syslog-ng/issues/1134/events","html_url":"https://github.com/balabit/syslog-ng/issues/1134","id":167788532,"number":1134,"title":"How to send multiple log files to a syslog-ng server over one TCP connection","user":{"login":"yusukeminowa","id":12887788,"avatar_url":"https://avatars.githubusercontent.com/u/12887788?v=3","gravatar_id":"","url":"https://api.github.com/users/yusukeminowa","html_url":"https://github.com/yusukeminowa","followers_url":"https://api.github.com/users/yusukeminowa/followers","following_url":"https://api.github.com/users/yusukeminowa/following{/other_user}","gists_url":"https://api.github.com/users/yusukeminowa/gists{/gist_id}","starred_url":"https://api.github.com/users/yusukeminowa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yusukeminowa/subscriptions","organizations_url":"https://api.github.com/users/yusukeminowa/orgs","repos_url":"https://api.github.com/users/yusukeminowa/repos","events_url":"https://api.github.com/users/yusukeminowa/events{/privacy}","received_events_url":"https://api.github.com/users/yusukeminowa/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-07-27T07:26:02Z","updated_at":"2016-07-27T13:03:26Z","closed_at":null,"body":"Hello,\r\n\r\nI would like to follow up in regard to below case:\r\n\r\nhttp://www.syslog.org/forum/syslog-ng/how-to-send-multiple-log-files-to-a-syslog-ng-server-over-one-tcp-connection/\r\n\r\nWe are trying to reduce the number of network connections used between syslog-ng clients and syslog-ng server.\r\n\r\nWe use tcp as protocol and the syslog-ng client is opening a connection for each file it is sending.\r\nThis is resulting in 7+ connections from each client. We want to reduce this ideally to a single connection for each client.\r\n\r\n$ netstat -taup |grep 244\r\ntcp        0      0 10.128.xxx.16:33709         10.128.xxx.244:shell        ESTABLISHED 4121/syslog-ng\r\ntcp        0      0 10.128.xxx.16:33024         10.128.xxx.244:shell        ESTABLISHED 4121/syslog-ng\r\ntcp        0      0 10.128.xxx.16:45326         10.128.xxx.244:shell        ESTABLISHED 4121/syslog-ng\r\ntcp        0      0 10.128.xxx.16:42195         10.128.xxx.244:shell        ESTABLISHED 4121/syslog-ng\r\ntcp        0      0 10.128.xxx.16:40593         10.128.xxx.244:shell        ESTABLISHED 4121/syslog-ng\r\ntcp        0      0 10.128.xxx.16:40340         10.128.xxx.244:shell        ESTABLISHED 4121/syslog-ng\r\ntcp        0      0 10.128.xxx.16:42507         10.128.xxx.244:shell        ESTABLISHED 4121/syslog-ng\r\n\r\nOS: Amazon Linux AMI release 2016.03\r\nSyslog version 3.2.5-4\r\n\r\n\r\nWhat we tried:\r\nkeep-alive(yes/no)\r\nso-keepalive(yes/no)\r\n\r\n\r\nConfiguration examples:\r\n\r\n--- syslog-NG-client.conf ---\r\n`@version:3.2\r\n\r\n\\# syslog-ng configuration file.\r\n\\#\r\n\\# This should behave pretty much like the original syslog on RedHat. But\r\n\\# it could be configured a lot smarter.\r\n\\#\r\n\\# See syslog-ng(8) and syslog-ng.conf(5) for more information.\r\n\\#\r\n\r\noptions {\r\n        flush_lines (0);\r\n        time_reopen (10);\r\n        log_fifo_size (1000);\r\n        long_hostnames (off);\r\n        use_dns (no);\r\n        use_fqdn (no);\r\n        create_dirs (no);\r\n        keep_hostname (yes);\r\n};\r\n\r\nsource s_sys {\r\n        file (\"/proc/kmsg\" program_override(\"kernel: \"));\r\n        unix-stream (\"/dev/log\");\r\n        internal();\r\n        # udp(ip(0.0.0.0) port(514));\r\n};\r\n\r\ndestination d_cons { file(\"/dev/console\"); };\r\n######\r\n@define App \"Linux\"\r\n@define InstanceID \"xxxxx\"\r\n@define SyslogServer \"syslogserver\"\r\n@define Port \"514\"\r\n######\r\n@define SubApp \"Messages\"\r\ntemplate `App``SubApp`_template { template(\"`App`/`SubApp`/`InstanceID` ${MSG}\"); template_escape(no); };\r\ndestination d_mesg { tcp(`SyslogServer` port(`Port`) template(`App``SubApp`_template));file(\"/var/log/messages\"); };\r\n@define SubApp \"Secure\"\r\ntemplate `App``SubApp`_template { template(\"`App`/`SubApp`/`InstanceID` ${MSG}\"); template_escape(no); };\r\ndestination d_auth { tcp(`SyslogServer` port(`Port`) template(`App``SubApp`_template));file(\"/var/log/secure\"); };\r\n@define SubApp \"Mail\"\r\ntemplate `App``SubApp`_template { template(\"`App`/`SubApp`/`InstanceID` ${MSG}\"); template_escape(no); };\r\ndestination d_mail { tcp(`SyslogServer` port(`Port`) template(`App``SubApp`_template));file(\"/var/log/maillog\" flush_lines(10)); };\r\n@define SubApp \"Spooler\"\r\ntemplate `App``SubApp`_template { template(\"`App`/`SubApp`/`InstanceID` ${MSG}\"); template_escape(no); };\r\ndestination d_spol { tcp(`SyslogServer` port(`Port`) template(`App``SubApp`_template));file(\"/var/log/spooler\"); };\r\n@define SubApp \"Boot\"\r\ntemplate `App``SubApp`_template { template(\"`App`/`SubApp`/`InstanceID` ${MSG}\"); template_escape(no); };\r\ndestination d_boot { tcp(`SyslogServer` port(`Port`) template(`App``SubApp`_template));file(\"/var/log/boot.log\"); };\r\n@define SubApp \"Cron\"\r\ntemplate `App``SubApp`_template { template(\"`App`/`SubApp`/`InstanceID` ${MSG}\"); template_escape(no); };\r\ndestination d_cron { tcp(`SyslogServer` port(`Port`) template(`App``SubApp`_template));file(\"/var/log/cron\"); };\r\n@define SubApp \"Kernel\"\r\ntemplate `App``SubApp`_template { template(\"`App`/`SubApp`/`InstanceID` ${MSG}\"); template_escape(no); };\r\ndestination d_kern { tcp(`SyslogServer` port(`Port`) template(`App``SubApp`_template));file(\"/var/log/kern\"); };\r\ndestination d_mlal { usertty(\"*\"); };\r\n\r\nfilter f_kernel     { facility(kern); };\r\nfilter f_default    { level(info..emerg) and\r\n                        not (facility(mail)\r\n                        or facility(authpriv)\r\n                        or facility(cron)); };\r\nfilter f_auth       { facility(authpriv); };\r\nfilter f_mail       { facility(mail); };\r\nfilter f_emergency  { level(emerg); };\r\nfilter f_news       { facility(uucp) or\r\n                        (facility(news)\r\n                        and level(crit..emerg)); };\r\nfilter f_boot   { facility(local7); };\r\nfilter f_cron   { facility(cron); };\r\n\r\n#log { source(s_sys); filter(f_kernel); destination(d_cons); };\r\nlog { source(s_sys); filter(f_kernel); destination(d_kern); };\r\nlog { source(s_sys); filter(f_default); destination(d_mesg); };\r\nlog { source(s_sys); filter(f_auth); destination(d_auth); };\r\nlog { source(s_sys); filter(f_mail); destination(d_mail); };\r\nlog { source(s_sys); filter(f_emergency); destination(d_mlal); };\r\nlog { source(s_sys); filter(f_news); destination(d_spol); };\r\nlog { source(s_sys); filter(f_boot); destination(d_boot); };\r\nlog { source(s_sys); filter(f_cron); destination(d_cron); };\r\n`\r\n\r\n--- syslog-NG-server.conf ---\r\n`@version:3.2\r\n\r\n\\# syslog-ng configuration file.\r\n\\#\r\n\\# This should behave pretty much like the original syslog on RedHat. But\r\n\\# it could be configured a lot smarter.\r\n\\#\r\n\\# See syslog-ng(8) and syslog-ng.conf(5) for more information.\r\n\\#\r\n\r\noptions {\r\n        flush_lines (0);\r\n        time_reopen (10);\r\n        log_fifo_size (1000);\r\n        long_hostnames (off);\r\n        use_dns (no);\r\n        use_fqdn (no);\r\n        create_dirs (no);\r\n        keep_hostname (yes);\r\n        log_msg_size(16384);\r\n};\r\n\r\nsource s_sys {\r\n        file (\"/proc/kmsg\" program_override(\"kernel: \"));\r\n        unix-stream (\"/dev/log\");\r\n        internal();\r\n        # udp(ip(0.0.0.0) port(514));\r\n};\r\n\r\ndestination d_cons { file(\"/dev/console\"); };\r\ndestination d_mesg { file(\"/var/log/messages\"); };\r\ndestination d_auth { file(\"/var/log/secure\"); };\r\ndestination d_mail { file(\"/var/log/maillog\" flush_lines(10)); };\r\ndestination d_spol { file(\"/var/log/spooler\"); };\r\ndestination d_boot { file(\"/var/log/boot.log\"); };\r\ndestination d_cron { file(\"/var/log/cron\"); };\r\ndestination d_kern { file(\"/var/log/kern\"); };\r\ndestination d_mlal { usertty(\"*\"); };\r\n\r\nfilter f_kernel     { facility(kern); };\r\nfilter f_default    { level(info..emerg) and\r\n                        not (facility(mail)\r\n                        or facility(authpriv)\r\n                        or facility(cron)); };\r\nfilter f_auth       { facility(authpriv); };\r\nfilter f_mail       { facility(mail); };\r\nfilter f_emergency  { level(emerg); };\r\nfilter f_news       { facility(uucp) or\r\n                        (facility(news)\r\n                        and level(crit..emerg)); };\r\nfilter f_boot   { facility(local7); };\r\nfilter f_cron   { facility(cron); };\r\n\r\n#log { source(s_sys); filter(f_kernel); destination(d_cons); };\r\nlog { source(s_sys); filter(f_kernel); destination(d_kern); };\r\nlog { source(s_sys); filter(f_default); destination(d_mesg); };\r\nlog { source(s_sys); filter(f_auth); destination(d_auth); };\r\nlog { source(s_sys); filter(f_mail); destination(d_mail); };\r\nlog { source(s_sys); filter(f_emergency); destination(d_mlal); };\r\nlog { source(s_sys); filter(f_news); destination(d_spol); };\r\nlog { source(s_sys); filter(f_boot); destination(d_boot); };\r\nlog { source(s_sys); filter(f_cron); destination(d_cron); };\r\n\r\n\\# vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:\r\n####\r\n\r\nparser p_myparser {\r\n csv-parser(\r\n columns(\"APP\", \"SUBAPP\", \"INSTANCE_ID\")\r\n delimiters(\"/: \")\r\n flags(escape-none)\r\n template(\"${PROGRAM}\")\r\n );\r\n};\r\n\r\nsource s_remotemsg { tcp(ip(0.0.0.0) port(514) max-connections(2000)); udp(ip(0.0.0.0) port(514)); };\r\n\r\ndestination d_remotemsg {\r\n file(\"/logs/${APP:-Unknown}/${SUBAPP:-Unknown}/${INSTANCE_ID:-Unknown}_${YEAR}_${MONTH}_${DAY}.log\" create_dirs(yes)\r\n dir_owner(\"root\") dir_group(\"root\") dir_perm(0655)\r\n owner(\"root\") group(\"root\") perm(0655));\r\n};\r\n\r\nlog { source(s_remotemsg); parser(p_myparser); destination(d_remotemsg); };\r\n\r\n####\r\n`\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/balabit/syslog-ng/issues/comments/235578085","html_url":"https://github.com/balabit/syslog-ng/issues/1134#issuecomment-235578085","issue_url":"https://api.github.com/repos/balabit/syslog-ng/issues/1134","id":235578085,"user":{"login":"bazsi","id":714102,"avatar_url":"https://avatars.githubusercontent.com/u/714102?v=3","gravatar_id":"","url":"https://api.github.com/users/bazsi","html_url":"https://github.com/bazsi","followers_url":"https://api.github.com/users/bazsi/followers","following_url":"https://api.github.com/users/bazsi/following{/other_user}","gists_url":"https://api.github.com/users/bazsi/gists{/gist_id}","starred_url":"https://api.github.com/users/bazsi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bazsi/subscriptions","organizations_url":"https://api.github.com/users/bazsi/orgs","repos_url":"https://api.github.com/users/bazsi/repos","events_url":"https://api.github.com/users/bazsi/events{/privacy}","received_events_url":"https://api.github.com/users/bazsi/received_events","type":"User","site_admin":false},"created_at":"2016-07-27T13:03:26Z","updated_at":"2016-07-27T13:03:26Z","body":"Well, syslog-ng opens one TCP connection per TCP destination and not more.\nSo as long as you only have one TCP destination, it should open one\nconnection.\n\nIf it does open more, it might be an indication of a bug, and 3.2.x is\nancient. Any chance of upgrading it?\n\n\n-- \nBazsi\n\nOn Wed, Jul 27, 2016 at 9:26 AM, yusukeminowa <notifications@github.com>\nwrote:\n\n> Hello,\n>\n> I would like to follow up in regard to below case:\n>\n>\n> http://www.syslog.org/forum/syslog-ng/how-to-send-multiple-log-files-to-a-syslog-ng-server-over-one-tcp-connection/\n>\n> We are trying to reduce the number of network connections used between\n> syslog-ng clients and syslog-ng server.\n>\n> We use tcp as protocol and the syslog-ng client is opening a connection\n> for each file it is sending.\n> This is resulting in 7+ connections from each client. We want to reduce\n> this ideally to a single connection for each client.\n>\n> $ netstat -taup |grep 244\n> tcp 0 0 10.128.xxx.16:33709 10.128.xxx.244:shell ESTABLISHED 4121/syslog-ng\n> tcp 0 0 10.128.xxx.16:33024 10.128.xxx.244:shell ESTABLISHED 4121/syslog-ng\n> tcp 0 0 10.128.xxx.16:45326 10.128.xxx.244:shell ESTABLISHED 4121/syslog-ng\n> tcp 0 0 10.128.xxx.16:42195 10.128.xxx.244:shell ESTABLISHED 4121/syslog-ng\n> tcp 0 0 10.128.xxx.16:40593 10.128.xxx.244:shell ESTABLISHED 4121/syslog-ng\n> tcp 0 0 10.128.xxx.16:40340 10.128.xxx.244:shell ESTABLISHED 4121/syslog-ng\n> tcp 0 0 10.128.xxx.16:42507 10.128.xxx.244:shell ESTABLISHED 4121/syslog-ng\n>\n> OS: Amazon Linux AMI release 2016.03\n> Syslog version 3.2.5-4\n>\n> What we tried:\n> keep-alive(yes/no)\n> so-keepalive(yes/no)\n>\n> Configuration examples:\n>\n> --- syslog-NG-client.conf ---\n> cat /etc/syslog-ng/syslog-ng.conf\n> @version <https://github.com/version>:3.2\n> syslog-ng configuration file.\n>\n> #\n> This should behave pretty much like the original syslog on RedHat. But it\n> could be configured a lot smarter.\n>\n> #\n> See syslog-ng(8) and syslog-ng.conf(5) for more information.\n>\n> #\n>\n> options {\n> flush_lines (0);\n> time_reopen (10);\n> log_fifo_size (1000);\n> long_hostnames (off);\n> use_dns (no);\n> use_fqdn (no);\n> create_dirs (no);\n> keep_hostname (yes);\n> };\n>\n> source s_sys {\n> file (\"/proc/kmsg\" program_override(\"kernel: \"));\n> unix-stream (\"/dev/log\");\n> internal();\n> # udp(ip(0.0.0.0) port(514));\n> };\n>\n> destination d_cons { file(\"/dev/console\"); };\n> ######\n> @define <https://github.com/define> App \"Linux\"\n> @define <https://github.com/define> InstanceID \"xxxxx\"\n> @define <https://github.com/define> SyslogServer \"syslogserver\"\n> @define <https://github.com/define> Port \"514\"\n> ######\n> @define <https://github.com/define> SubApp \"Messages\"\n> template AppSubApp_template { template(\"App/SubApp/InstanceID ${MSG}\");\n> template_escape(no); };\n> destination d_mesg { tcp(SyslogServer port(Port) template(AppSubApp_template));file(\"/var/log/messages\");\n> };\n> @define <https://github.com/define> SubApp \"Secure\"\n> template AppSubApp_template { template(\"App/SubApp/InstanceID ${MSG}\");\n> template_escape(no); };\n> destination d_auth { tcp(SyslogServer port(Port) template(AppSubApp_template));file(\"/var/log/secure\");\n> };\n> @define <https://github.com/define> SubApp \"Mail\"\n> template AppSubApp_template { template(\"App/SubApp/InstanceID ${MSG}\");\n> template_escape(no); };\n> destination d_mail { tcp(SyslogServer port(Port) template(AppSubApp_template));file(\"/var/log/maillog\"\n> flush_lines(10)); };\n> @define <https://github.com/define> SubApp \"Spooler\"\n> template AppSubApp_template { template(\"App/SubApp/InstanceID ${MSG}\");\n> template_escape(no); };\n> destination d_spol { tcp(SyslogServer port(Port) template(AppSubApp_template));file(\"/var/log/spooler\");\n> };\n> @define <https://github.com/define> SubApp \"Boot\"\n> template AppSubApp_template { template(\"App/SubApp/InstanceID ${MSG}\");\n> template_escape(no); };\n> destination d_boot { tcp(SyslogServer port(Port) template(AppSubApp_template));file(\"/var/log/boot.log\");\n> };\n> @define <https://github.com/define> SubApp \"Cron\"\n> template AppSubApp_template { template(\"App/SubApp/InstanceID ${MSG}\");\n> template_escape(no); };\n> destination d_cron { tcp(SyslogServer port(Port) template(AppSubApp_template));file(\"/var/log/cron\");\n> };\n> @define <https://github.com/define> SubApp \"Kernel\"\n> template AppSubApp_template { template(\"App/SubApp/InstanceID ${MSG}\");\n> template_escape(no); };\n> destination d_kern { tcp(SyslogServer port(Port) template(AppSubApp_template));file(\"/var/log/kern\");\n> };\n> destination d_mlal { usertty(\"*\"); };\n>\n> filter f_kernel { facility(kern); };\n> filter f_default { level(info..emerg) and\n> not (facility(mail)\n> or facility(authpriv)\n> or facility(cron)); };\n> filter f_auth { facility(authpriv); };\n> filter f_mail { facility(mail); };\n> filter f_emergency { level(emerg); };\n> filter f_news { facility(uucp) or\n> (facility(news)\n> and level(crit..emerg)); };\n> filter f_boot { facility(local7); };\n> filter f_cron { facility(cron); };\n>\n> #log { source(s_sys); filter(f_kernel); destination(d_cons); };\n> log { source(s_sys); filter(f_kernel); destination(d_kern); };\n> log { source(s_sys); filter(f_default); destination(d_mesg); };\n> log { source(s_sys); filter(f_auth); destination(d_auth); };\n> log { source(s_sys); filter(f_mail); destination(d_mail); };\n> log { source(s_sys); filter(f_emergency); destination(d_mlal); };\n> log { source(s_sys); filter(f_news); destination(d_spol); };\n> log { source(s_sys); filter(f_boot); destination(d_boot); };\n> log { source(s_sys); filter(f_cron); destination(d_cron); };\n>\n> --- syslog-NG-server.conf ---\n>\n> [root@jpndevsyslog syslog-ng]# cat syslog-ng.conf\n> @version <https://github.com/version>:3.2\n> syslog-ng configuration file.\n>\n> #\n> This should behave pretty much like the original syslog on RedHat. But it\n> could be configured a lot smarter.\n>\n> #\n> See syslog-ng(8) and syslog-ng.conf(5) for more information.\n>\n> #\n>\n> options {\n> flush_lines (0);\n> time_reopen (10);\n> log_fifo_size (1000);\n> long_hostnames (off);\n> use_dns (no);\n> use_fqdn (no);\n> create_dirs (no);\n> keep_hostname (yes);\n> log_msg_size(16384);\n> };\n>\n> source s_sys {\n> file (\"/proc/kmsg\" program_override(\"kernel: \"));\n> unix-stream (\"/dev/log\");\n> internal();\n> # udp(ip(0.0.0.0) port(514));\n> };\n>\n> destination d_cons { file(\"/dev/console\"); };\n> destination d_mesg { file(\"/var/log/messages\"); };\n> destination d_auth { file(\"/var/log/secure\"); };\n> destination d_mail { file(\"/var/log/maillog\" flush_lines(10)); };\n> destination d_spol { file(\"/var/log/spooler\"); };\n> destination d_boot { file(\"/var/log/boot.log\"); };\n> destination d_cron { file(\"/var/log/cron\"); };\n> destination d_kern { file(\"/var/log/kern\"); };\n> destination d_mlal { usertty(\"*\"); };\n>\n> filter f_kernel { facility(kern); };\n> filter f_default { level(info..emerg) and\n> not (facility(mail)\n> or facility(authpriv)\n> or facility(cron)); };\n> filter f_auth { facility(authpriv); };\n> filter f_mail { facility(mail); };\n> filter f_emergency { level(emerg); };\n> filter f_news { facility(uucp) or\n> (facility(news)\n> and level(crit..emerg)); };\n> filter f_boot { facility(local7); };\n> filter f_cron { facility(cron); };\n>\n> #log { source(s_sys); filter(f_kernel); destination(d_cons); };\n> log { source(s_sys); filter(f_kernel); destination(d_kern); };\n> log { source(s_sys); filter(f_default); destination(d_mesg); };\n> log { source(s_sys); filter(f_auth); destination(d_auth); };\n> log { source(s_sys); filter(f_mail); destination(d_mail); };\n> log { source(s_sys); filter(f_emergency); destination(d_mlal); };\n> log { source(s_sys); filter(f_news); destination(d_spol); };\n> log { source(s_sys); filter(f_boot); destination(d_boot); };\n> log { source(s_sys); filter(f_cron); destination(d_cron); };\n> vim:ft=syslog-ng:ai:si:ts=4:sw=4:et:\n>\n> ####\n>\n> parser p_myparser {\n> csv-parser(\n> columns(\"APP\", \"SUBAPP\", \"INSTANCE_ID\")\n> delimiters(\"/: \")\n> flags(escape-none)\n> template(\"${PROGRAM}\")\n> );\n> };\n>\n> source s_remotemsg { tcp(ip(0.0.0.0) port(514) max-connections(2000));\n> udp(ip(0.0.0.0) port(514)); };\n>\n> destination d_remotemsg {\n> file(\"/logs/${APP:-Unknown}/${SUBAPP:-Unknown}/${INSTANCE_ID:-Unknown}\n> *${YEAR}*${MONTH}_${DAY}.log\" create_dirs(yes)\n> dir_owner(\"root\") dir_group(\"root\") dir_perm(0655)\n> owner(\"root\") group(\"root\") perm(0655));\n> };\n>\n> log { source(s_remotemsg); parser(p_myparser); destination(d_remotemsg); };\n>\n> ####\n>\n> —\n> You are receiving this because you are subscribed to this thread.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/balabit/syslog-ng/issues/1134>, or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AArldnZ4hqZ4OKXLMi0CDy3DIYCyxC2Wks5qZwgLgaJpZM4JV6MD>\n> .\n>\n"}}
{"actor":{"display_login":"bazsi"}}