{"action":"created","issue":{"url":"https://api.github.com/repos/DataDog/datadog-agent/issues/2504","repository_url":"https://api.github.com/repos/DataDog/datadog-agent","labels_url":"https://api.github.com/repos/DataDog/datadog-agent/issues/2504/labels{/name}","comments_url":"https://api.github.com/repos/DataDog/datadog-agent/issues/2504/comments","events_url":"https://api.github.com/repos/DataDog/datadog-agent/issues/2504/events","html_url":"https://github.com/DataDog/datadog-agent/pull/2504","id":372981230,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI1MDIzOTA4","number":2504,"title":"[LOGS]: Made pipeline configurable and added new metrics to the logs-agent","user":{"login":"ajacquemot","id":5777308,"node_id":"MDQ6VXNlcjU3NzczMDg=","avatar_url":"https://avatars0.githubusercontent.com/u/5777308?v=4","gravatar_id":"","url":"https://api.github.com/users/ajacquemot","html_url":"https://github.com/ajacquemot","followers_url":"https://api.github.com/users/ajacquemot/followers","following_url":"https://api.github.com/users/ajacquemot/following{/other_user}","gists_url":"https://api.github.com/users/ajacquemot/gists{/gist_id}","starred_url":"https://api.github.com/users/ajacquemot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajacquemot/subscriptions","organizations_url":"https://api.github.com/users/ajacquemot/orgs","repos_url":"https://api.github.com/users/ajacquemot/repos","events_url":"https://api.github.com/users/ajacquemot/events{/privacy}","received_events_url":"https://api.github.com/users/ajacquemot/received_events","type":"User","site_admin":false},"labels":[{"id":776720186,"node_id":"MDU6TGFiZWw3NzY3MjAxODY=","url":"https://api.github.com/repos/DataDog/datadog-agent/labels/component/logs","name":"component/logs","color":"0052cc","default":false},{"id":796118273,"node_id":"MDU6TGFiZWw3OTYxMTgyNzM=","url":"https://api.github.com/repos/DataDog/datadog-agent/labels/noreno","name":"noreno","color":"edd682","default":false},{"id":695416522,"node_id":"MDU6TGFiZWw2OTU0MTY1MjI=","url":"https://api.github.com/repos/DataDog/datadog-agent/labels/team/ramen","name":"team/ramen","color":"d4c5f9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/DataDog/datadog-agent/milestones/39","html_url":"https://github.com/DataDog/datadog-agent/milestone/39","labels_url":"https://api.github.com/repos/DataDog/datadog-agent/milestones/39/labels","id":3727975,"node_id":"MDk6TWlsZXN0b25lMzcyNzk3NQ==","number":39,"title":"6.8.0","description":"","creator":{"login":"hush-hush","id":311563,"node_id":"MDQ6VXNlcjMxMTU2Mw==","avatar_url":"https://avatars3.githubusercontent.com/u/311563?v=4","gravatar_id":"","url":"https://api.github.com/users/hush-hush","html_url":"https://github.com/hush-hush","followers_url":"https://api.github.com/users/hush-hush/followers","following_url":"https://api.github.com/users/hush-hush/following{/other_user}","gists_url":"https://api.github.com/users/hush-hush/gists{/gist_id}","starred_url":"https://api.github.com/users/hush-hush/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hush-hush/subscriptions","organizations_url":"https://api.github.com/users/hush-hush/orgs","repos_url":"https://api.github.com/users/hush-hush/repos","events_url":"https://api.github.com/users/hush-hush/events{/privacy}","received_events_url":"https://api.github.com/users/hush-hush/received_events","type":"User","site_admin":false},"open_issues":44,"closed_issues":40,"state":"open","created_at":"2018-10-10T15:40:05Z","updated_at":"2018-11-15T05:55:04Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2018-10-23T12:46:03Z","updated_at":"2018-11-15T12:43:09Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/DataDog/datadog-agent/pulls/2504","html_url":"https://github.com/DataDog/datadog-agent/pull/2504","diff_url":"https://github.com/DataDog/datadog-agent/pull/2504.diff","patch_url":"https://github.com/DataDog/datadog-agent/pull/2504.patch"},"body":"### What does this PR do?\r\n\r\n* Moved pipeline parameters out of `constants.go` and made them configurable\r\n* Added metrics for logs collected and logs commited\r\n* Removed metric for logs decoded as it seems to be redundant with logs collected\r\n* Added a bunch of comments to explain how the logs-agent backoff work\r\n* Fixed a bug in the sender when stoping the agent, one log would be committed while it should not\r\n\r\n### Motivation\r\n\r\n* Make clear how the logs-agent work when having to backoff\r\n* Make parameters configurable from `datadog.yaml`\r\n* Have more metrics\r\n\r\n### Additional Notes\r\n\r\nWould be nice to refactor the code to have only one \"abstract\" tailer that could be reused for any kind of input implementation.\r\n\r\nChanged few logic that @iksaif worked on several days ago, I'd be happy to discuss to share the same vision.\r\n\r\n### Open Questions\r\n\r\nFor now several stage of the pipeline can buffer messages, I think we should limit this behaviour only to senders as they are the only blocking stage of the pipeline, this would enable us to reduce the number of inflight data when having networking issues from:\r\n```\r\npipeline.count * ( 2 * pipeline.buffer_size ) + pipeline.buffer_size\r\n```\r\nto:\r\n```\r\npipeline.count * pipeline.buffer_size\r\n```\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/DataDog/datadog-agent/issues/comments/439027567","html_url":"https://github.com/DataDog/datadog-agent/pull/2504#issuecomment-439027567","issue_url":"https://api.github.com/repos/DataDog/datadog-agent/issues/2504","id":439027567,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTAyNzU2Nw==","user":{"login":"iksaif","id":1032963,"node_id":"MDQ6VXNlcjEwMzI5NjM=","avatar_url":"https://avatars1.githubusercontent.com/u/1032963?v=4","gravatar_id":"","url":"https://api.github.com/users/iksaif","html_url":"https://github.com/iksaif","followers_url":"https://api.github.com/users/iksaif/followers","following_url":"https://api.github.com/users/iksaif/following{/other_user}","gists_url":"https://api.github.com/users/iksaif/gists{/gist_id}","starred_url":"https://api.github.com/users/iksaif/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iksaif/subscriptions","organizations_url":"https://api.github.com/users/iksaif/orgs","repos_url":"https://api.github.com/users/iksaif/repos","events_url":"https://api.github.com/users/iksaif/events{/privacy}","received_events_url":"https://api.github.com/users/iksaif/received_events","type":"User","site_admin":false},"created_at":"2018-11-15T12:43:09Z","updated_at":"2018-11-15T12:43:09Z","author_association":"CONTRIBUTOR","body":"Do we want to merge this ?"}}
{"actor":{"display_login":"iksaif"}}