{"action":"created","issue":{"url":"https://api.github.com/repos/fluent/fluent-bit/issues/127","repository_url":"https://api.github.com/repos/fluent/fluent-bit","labels_url":"https://api.github.com/repos/fluent/fluent-bit/issues/127/labels{/name}","comments_url":"https://api.github.com/repos/fluent/fluent-bit/issues/127/comments","events_url":"https://api.github.com/repos/fluent/fluent-bit/issues/127/events","html_url":"https://github.com/fluent/fluent-bit/issues/127","id":186858098,"number":127,"title":"Scheduler: improve retries handling","user":{"login":"edsiper","id":369718,"avatar_url":"https://avatars.githubusercontent.com/u/369718?v=3","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false},"labels":[{"id":171021298,"url":"https://api.github.com/repos/fluent/fluent-bit/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":{"login":"edsiper","id":369718,"avatar_url":"https://avatars.githubusercontent.com/u/369718?v=3","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false},"assignees":[{"login":"edsiper","id":369718,"avatar_url":"https://avatars.githubusercontent.com/u/369718?v=3","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2016-11-02T16:54:13Z","updated_at":"2016-12-05T21:56:32Z","closed_at":null,"body":"On a high load test, if many retries are issued, the Scheduler creates a timer file descriptor for each one, but easily these retries could not be scheduled at some point if we run out of file descriptors.\r\n\r\n```\r\n[2016/11/02 10:41:03] [  Error] timerfd: Too many open files, errno=24 at /home/edsiper/coding/fluent-bit/lib/monkey/mk_core/mk_event_epoll.c:175\r\n```\r\n\r\nPossible workarounds:\r\n\r\n1. Let the user increase their file descriptor limits. \r\n2. Do not use a file descriptor for each retry, instead lookup a list of retries every second using just one timer.\r\n\r\nThe approach 1 will let the user same the issue before take some action, note that using timerfd() is good as we just get notifications upon a timeout, we don't query a list.\r\n\r\nApproach 2 add some overhead as we need to iterate a list every second.\r\n\r\nThis needs more thinking..."},"comment":{"url":"https://api.github.com/repos/fluent/fluent-bit/issues/comments/264990595","html_url":"https://github.com/fluent/fluent-bit/issues/127#issuecomment-264990595","issue_url":"https://api.github.com/repos/fluent/fluent-bit/issues/127","id":264990595,"user":{"login":"edsiper","id":369718,"avatar_url":"https://avatars.githubusercontent.com/u/369718?v=3","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false},"created_at":"2016-12-05T21:56:32Z","updated_at":"2016-12-05T21:56:32Z","body":"@nokute78 \r\n\r\nthe latest fixes in master should address the issue, after the changes I cannot longer reproduce the problem.\r\n\r\nNote: keeping this ticket open until the new strategy for schedule retries is implemented.\r\n"}}
{"actor":{"display_login":"edsiper"}}