{"action":"closed","issue":{"url":"https://api.github.com/repos/fluent/fluentd/issues/777","repository_url":"https://api.github.com/repos/fluent/fluentd","labels_url":"https://api.github.com/repos/fluent/fluentd/issues/777/labels{/name}","comments_url":"https://api.github.com/repos/fluent/fluentd/issues/777/comments","events_url":"https://api.github.com/repos/fluent/fluentd/issues/777/events","html_url":"https://github.com/fluent/fluentd/issues/777","id":126881088,"number":777,"title":"multiprocess for input plugin \"forward\"","user":{"login":"mperriere","id":16205859,"avatar_url":"https://avatars.githubusercontent.com/u/16205859?v=3","gravatar_id":"","url":"https://api.github.com/users/mperriere","html_url":"https://github.com/mperriere","followers_url":"https://api.github.com/users/mperriere/followers","following_url":"https://api.github.com/users/mperriere/following{/other_user}","gists_url":"https://api.github.com/users/mperriere/gists{/gist_id}","starred_url":"https://api.github.com/users/mperriere/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mperriere/subscriptions","organizations_url":"https://api.github.com/users/mperriere/orgs","repos_url":"https://api.github.com/users/mperriere/repos","events_url":"https://api.github.com/users/mperriere/events{/privacy}","received_events_url":"https://api.github.com/users/mperriere/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/fluent/fluentd/labels/feature%20request","name":"feature request","color":"0052cc"},{"url":"https://api.github.com/repos/fluent/fluentd/labels/v0.14","name":"v0.14","color":"009800"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-01-15T14:18:11Z","updated_at":"2016-03-15T04:19:14Z","closed_at":"2016-03-15T04:19:14Z","body":"Hello,\r\nfirst a quick overview of our Fluentd utilization:\r\nwe have 40 Apache front server producing 1tb log per day. Those logs are send to 2 fluentd relay then send to an Amazon Kinesis repo.\r\nWe noticed a performance bottleneck with a 5Mb/s. We our 40 HTTP nodes, we should be able to handle at least 40Mb/s.\r\n\r\nWe managed to reach good performances by using @type multiprocess with 20 nodes:\r\ncat td-agent.conf\r\n\r\n```\r\n<source>\r\n  @type multiprocess\r\n  <process>\r\n    cmdline -c /etc/td-agent/td-agent-child1.conf --log /var/log/td-agent/td-agentd_child1.log\r\n    sleep_before_start 1s\r\n    sleep_before_shutdown 5s\r\n    log_level debug\r\n  </process>\r\n  <process>\r\n    cmdline -c /etc/td-agent/td-agent-child2.conf --log /var/log/td-agent/td-agentd_child2.log\r\n    sleep_before_start 2s\r\n    sleep_before_shutdown 5s\r\n    log_level debug\r\n  </process>\r\n  <process>\r\n    cmdline -c /etc/td-agent/td-agent-child3.conf --log /var/log/td-agent/td-agentd_child3.log\r\n    sleep_before_start 3s\r\n    sleep_before_shutdown 5s\r\n    log_level debug\r\n  </process>\r\n  <process>\r\n    cmdline -c /etc/td-agent/td-agent-child4.conf --log /var/log/td-agent/td-agentd_child4.log\r\n    sleep_before_start 4s\r\n    sleep_before_shutdown 5s\r\n    log_level debug\r\n  </process>\r\n  <process>\r\n    cmdline -c /etc/td-agent/td-agent-child5.conf --log /var/log/td-agent/td-agentd_child5.log\r\n    sleep_before_start 5s\r\n    sleep_before_shutdown 5s\r\n    log_level debug\r\n  </process>\r\n</source>\r\n```\r\n\r\nEach of the 5 childs handle 4 HTTP nodes.\r\nBut the configuration becomes quite messy.\r\nThe best solution for us to keep a maintanable configuration set would be an integrated input forward multi process.\r\nWithout this feature, Fluentd is not considered safe for my company to process our HTTP log stream. Kafka or a similar tool will be used instead.\r\n\r\nIs multiprocess for forward plugin planned for the next release?\r\nThank you\r\nMathieu Perriere."}}