{"action":"created","comment":{"url":"https://api.github.com/repos/moby/moby/pulls/comments/184532374","pull_request_review_id":115750619,"id":184532374,"diff_hunk":"@@ -140,19 +143,48 @@ func New(info logger.Info) (logger.Logger, error) {\n \tif err != nil {\n \t\treturn nil, err\n \t}\n+\n \tcontainerStream := &logStream{\n \t\tlogStreamName:    logStreamName,\n \t\tlogGroupName:     logGroupName,\n \t\tlogCreateGroup:   logCreateGroup,\n+\t\tlogNonBlocking:   logNonBlocking,\n \t\tmultilinePattern: multilinePattern,\n \t\tclient:           client,\n \t\tmessages:         make(chan *logger.Message, 4096),\n \t}\n-\terr = containerStream.create()\n-\tif err != nil {\n-\t\treturn nil, err\n+\n+\tcreationDone := make(chan bool)\n+\tif logNonBlocking {\n+\t\tgo func() {\n+\t\t\tbackoff := 1\n+\t\t\tmaxBackoff := 32\n+\t\t\tfor {\n+\t\t\t\t// If logger is closed we are done\n+\t\t\t\tif containerStream.closed {","path":"daemon/logger/awslogs/cloudwatchlogs.go","position":42,"original_position":42,"commit_id":"830da1de0f64886da06766e40d09b78a2b3be579","original_commit_id":"830da1de0f64886da06766e40d09b78a2b3be579","user":{"login":"cpuguy83","id":799078,"avatar_url":"https://avatars3.githubusercontent.com/u/799078?v=4","gravatar_id":"","url":"https://api.github.com/users/cpuguy83","html_url":"https://github.com/cpuguy83","followers_url":"https://api.github.com/users/cpuguy83/followers","following_url":"https://api.github.com/users/cpuguy83/following{/other_user}","gists_url":"https://api.github.com/users/cpuguy83/gists{/gist_id}","starred_url":"https://api.github.com/users/cpuguy83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpuguy83/subscriptions","organizations_url":"https://api.github.com/users/cpuguy83/orgs","repos_url":"https://api.github.com/users/cpuguy83/repos","events_url":"https://api.github.com/users/cpuguy83/events{/privacy}","received_events_url":"https://api.github.com/users/cpuguy83/received_events","type":"User","site_admin":false},"body":"If we do this, then we need some synchronization.","created_at":"2018-04-26T21:12:25Z","updated_at":"2018-04-26T21:12:25Z","html_url":"https://github.com/moby/moby/pull/36522#discussion_r184532374","pull_request_url":"https://api.github.com/repos/moby/moby/pulls/36522","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/moby/moby/pulls/comments/184532374"},"html":{"href":"https://github.com/moby/moby/pull/36522#discussion_r184532374"},"pull_request":{"href":"https://api.github.com/repos/moby/moby/pulls/36522"}}},"pull_request":{"url":"https://api.github.com/repos/moby/moby/pulls/36522","id":173601340,"html_url":"https://github.com/moby/moby/pull/36522","diff_url":"https://github.com/moby/moby/pull/36522.diff","patch_url":"https://github.com/moby/moby/pull/36522.patch","issue_url":"https://api.github.com/repos/moby/moby/issues/36522","number":36522,"state":"open","locked":false,"title":"Allow awslogs to use non-blocking mode","user":{"login":"IRCody","id":4969568,"avatar_url":"https://avatars1.githubusercontent.com/u/4969568?v=4","gravatar_id":"","url":"https://api.github.com/users/IRCody","html_url":"https://github.com/IRCody","followers_url":"https://api.github.com/users/IRCody/followers","following_url":"https://api.github.com/users/IRCody/following{/other_user}","gists_url":"https://api.github.com/users/IRCody/gists{/gist_id}","starred_url":"https://api.github.com/users/IRCody/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IRCody/subscriptions","organizations_url":"https://api.github.com/users/IRCody/orgs","repos_url":"https://api.github.com/users/IRCody/repos","events_url":"https://api.github.com/users/IRCody/events{/privacy}","received_events_url":"https://api.github.com/users/IRCody/received_events","type":"User","site_admin":false},"body":"```\r\nAllow awslogs to use non-blocking mode\r\n\r\nWhen then non-blocking mode is specified, awslogs will:\r\n\r\n- No longer potentially block calls to logstream.Log(), instead will\r\n  return an error if the awslogs buffer is full. This has the effect of\r\n  dropping log messages sent to awslogs.Log() that are made while the\r\n  buffer is full.\r\n- Wait to initialize the log stream until the first Log() call instead of in\r\n  New(). This has the effect of allowing the container to start in\r\n  the case where Cloudwatch Logs is unreachable.\r\n\r\nBoth of these changes require the --log-opt mode=non-blocking to be\r\nexplicitly set and do not modify the default behavior.\r\n\r\nSigned-off-by: Cody Roseborough <crrosebo@amazon.com>\r\n```\r\nFixes #33803\r\n\r\nSince this has some changes in behavior I wanted to call out some specific scenarios here. All scenarios are talking about behavior when `logopt mode=non-blocking` is set. Default behavior should remain unchanged.\r\n\r\n**Scenario 1:** Unable to connect to CloudWatch Logs on startup\r\n\r\n_Current behavior:_\r\n\r\nFails to start container after the timeout is reached during the createLogStream call.\r\n\r\n_After this PR:_\r\n\r\nContainer will start and initialize the log stream on the first call to Log() (and all subsequent calls until the log stream is created successfully). Errors here are not fatal to the container.\r\n\r\n**Scenario 2:** CloudWatch Logs becomes unavailable and the container is stopped.\r\n\r\n_Current behavior:_\r\n\r\nThe RingLogger [Close()](https://github.com/moby/moby/blob/master/daemon/logger/ring.go\\#L82-L104) will call awslogs.Log() on all remaining messages in the buffer. Each call is potentially blocking on a channel write that ultimately depends on a call to CloudWatch Logs. These will complete but very slowly as it times out multiple calls to CloudWatch Logs.\r\n\r\n_After this PR:_\r\n\r\nawslogs.Log() is no longer blocking. If it's unable to add an item to its internal buffer it returns an error. This has the effect of ignoring log messages that come after the awslogs internal buffer is full.","created_at":"2018-03-07T23:15:04Z","updated_at":"2018-04-26T21:12:25Z","closed_at":null,"merged_at":null,"merge_commit_sha":"4338a98b460fe003281640b95e3b151bf48525fd","assignee":{"login":"dnephin","id":442180,"avatar_url":"https://avatars1.githubusercontent.com/u/442180?v=4","gravatar_id":"","url":"https://api.github.com/users/dnephin","html_url":"https://github.com/dnephin","followers_url":"https://api.github.com/users/dnephin/followers","following_url":"https://api.github.com/users/dnephin/following{/other_user}","gists_url":"https://api.github.com/users/dnephin/gists{/gist_id}","starred_url":"https://api.github.com/users/dnephin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dnephin/subscriptions","organizations_url":"https://api.github.com/users/dnephin/orgs","repos_url":"https://api.github.com/users/dnephin/repos","events_url":"https://api.github.com/users/dnephin/events{/privacy}","received_events_url":"https://api.github.com/users/dnephin/received_events","type":"User","site_admin":false},"assignees":[{"login":"dnephin","id":442180,"avatar_url":"https://avatars1.githubusercontent.com/u/442180?v=4","gravatar_id":"","url":"https://api.github.com/users/dnephin","html_url":"https://github.com/dnephin","followers_url":"https://api.github.com/users/dnephin/followers","following_url":"https://api.github.com/users/dnephin/following{/other_user}","gists_url":"https://api.github.com/users/dnephin/gists{/gist_id}","starred_url":"https://api.github.com/users/dnephin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dnephin/subscriptions","organizations_url":"https://api.github.com/users/dnephin/orgs","repos_url":"https://api.github.com/users/dnephin/repos","events_url":"https://api.github.com/users/dnephin/events{/privacy}","received_events_url":"https://api.github.com/users/dnephin/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":70404987,"url":"https://api.github.com/repos/moby/moby/labels/area/logging","name":"area/logging","color":"fbca04","default":false},{"id":308856530,"url":"https://api.github.com/repos/moby/moby/labels/docs/revisit","name":"docs/revisit","color":"000000","default":false},{"id":237472270,"url":"https://api.github.com/repos/moby/moby/labels/impact/changelog","name":"impact/changelog","color":"bfe5bf","default":false},{"id":175375407,"url":"https://api.github.com/repos/moby/moby/labels/status/2-code-review","name":"status/2-code-review","color":"fad8c7","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/moby/moby/pulls/36522/commits","review_comments_url":"https://api.github.com/repos/moby/moby/pulls/36522/comments","review_comment_url":"https://api.github.com/repos/moby/moby/pulls/comments{/number}","comments_url":"https://api.github.com/repos/moby/moby/issues/36522/comments","statuses_url":"https://api.github.com/repos/moby/moby/statuses/830da1de0f64886da06766e40d09b78a2b3be579","head":{"label":"IRCody:awslogs-non-blocking","ref":"awslogs-non-blocking","sha":"830da1de0f64886da06766e40d09b78a2b3be579","user":{"login":"IRCody","id":4969568,"avatar_url":"https://avatars1.githubusercontent.com/u/4969568?v=4","gravatar_id":"","url":"https://api.github.com/users/IRCody","html_url":"https://github.com/IRCody","followers_url":"https://api.github.com/users/IRCody/followers","following_url":"https://api.github.com/users/IRCody/following{/other_user}","gists_url":"https://api.github.com/users/IRCody/gists{/gist_id}","starred_url":"https://api.github.com/users/IRCody/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IRCody/subscriptions","organizations_url":"https://api.github.com/users/IRCody/orgs","repos_url":"https://api.github.com/users/IRCody/repos","events_url":"https://api.github.com/users/IRCody/events{/privacy}","received_events_url":"https://api.github.com/users/IRCody/received_events","type":"User","site_admin":false},"repo":{"id":121542645,"name":"moby","full_name":"IRCody/moby","owner":{"login":"IRCody","id":4969568,"avatar_url":"https://avatars1.githubusercontent.com/u/4969568?v=4","gravatar_id":"","url":"https://api.github.com/users/IRCody","html_url":"https://github.com/IRCody","followers_url":"https://api.github.com/users/IRCody/followers","following_url":"https://api.github.com/users/IRCody/following{/other_user}","gists_url":"https://api.github.com/users/IRCody/gists{/gist_id}","starred_url":"https://api.github.com/users/IRCody/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IRCody/subscriptions","organizations_url":"https://api.github.com/users/IRCody/orgs","repos_url":"https://api.github.com/users/IRCody/repos","events_url":"https://api.github.com/users/IRCody/events{/privacy}","received_events_url":"https://api.github.com/users/IRCody/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/IRCody/moby","description":"Moby Project - a collaborative project for the container ecosystem to assemble container-based systems","fork":true,"url":"https://api.github.com/repos/IRCody/moby","forks_url":"https://api.github.com/repos/IRCody/moby/forks","keys_url":"https://api.github.com/repos/IRCody/moby/keys{/key_id}","collaborators_url":"https://api.github.com/repos/IRCody/moby/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/IRCody/moby/teams","hooks_url":"https://api.github.com/repos/IRCody/moby/hooks","issue_events_url":"https://api.github.com/repos/IRCody/moby/issues/events{/number}","events_url":"https://api.github.com/repos/IRCody/moby/events","assignees_url":"https://api.github.com/repos/IRCody/moby/assignees{/user}","branches_url":"https://api.github.com/repos/IRCody/moby/branches{/branch}","tags_url":"https://api.github.com/repos/IRCody/moby/tags","blobs_url":"https://api.github.com/repos/IRCody/moby/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/IRCody/moby/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/IRCody/moby/git/refs{/sha}","trees_url":"https://api.github.com/repos/IRCody/moby/git/trees{/sha}","statuses_url":"https://api.github.com/repos/IRCody/moby/statuses/{sha}","languages_url":"https://api.github.com/repos/IRCody/moby/languages","stargazers_url":"https://api.github.com/repos/IRCody/moby/stargazers","contributors_url":"https://api.github.com/repos/IRCody/moby/contributors","subscribers_url":"https://api.github.com/repos/IRCody/moby/subscribers","subscription_url":"https://api.github.com/repos/IRCody/moby/subscription","commits_url":"https://api.github.com/repos/IRCody/moby/commits{/sha}","git_commits_url":"https://api.github.com/repos/IRCody/moby/git/commits{/sha}","comments_url":"https://api.github.com/repos/IRCody/moby/comments{/number}","issue_comment_url":"https://api.github.com/repos/IRCody/moby/issues/comments{/number}","contents_url":"https://api.github.com/repos/IRCody/moby/contents/{+path}","compare_url":"https://api.github.com/repos/IRCody/moby/compare/{base}...{head}","merges_url":"https://api.github.com/repos/IRCody/moby/merges","archive_url":"https://api.github.com/repos/IRCody/moby/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/IRCody/moby/downloads","issues_url":"https://api.github.com/repos/IRCody/moby/issues{/number}","pulls_url":"https://api.github.com/repos/IRCody/moby/pulls{/number}","milestones_url":"https://api.github.com/repos/IRCody/moby/milestones{/number}","notifications_url":"https://api.github.com/repos/IRCody/moby/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/IRCody/moby/labels{/name}","releases_url":"https://api.github.com/repos/IRCody/moby/releases{/id}","deployments_url":"https://api.github.com/repos/IRCody/moby/deployments","created_at":"2018-02-14T17:56:59Z","updated_at":"2018-02-14T17:57:13Z","pushed_at":"2018-04-26T21:01:13Z","git_url":"git://github.com/IRCody/moby.git","ssh_url":"git@github.com:IRCody/moby.git","clone_url":"https://github.com/IRCody/moby.git","svn_url":"https://github.com/IRCody/moby","homepage":"https://mobyproject.org/","size":131922,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":null,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"moby:master","ref":"master","sha":"fef1a5ea809976d0d3d2160f3e0a4459e6482763","user":{"login":"moby","id":27259197,"avatar_url":"https://avatars1.githubusercontent.com/u/27259197?v=4","gravatar_id":"","url":"https://api.github.com/users/moby","html_url":"https://github.com/moby","followers_url":"https://api.github.com/users/moby/followers","following_url":"https://api.github.com/users/moby/following{/other_user}","gists_url":"https://api.github.com/users/moby/gists{/gist_id}","starred_url":"https://api.github.com/users/moby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/moby/subscriptions","organizations_url":"https://api.github.com/users/moby/orgs","repos_url":"https://api.github.com/users/moby/repos","events_url":"https://api.github.com/users/moby/events{/privacy}","received_events_url":"https://api.github.com/users/moby/received_events","type":"Organization","site_admin":false},"repo":{"id":7691631,"name":"moby","full_name":"moby/moby","owner":{"login":"moby","id":27259197,"avatar_url":"https://avatars1.githubusercontent.com/u/27259197?v=4","gravatar_id":"","url":"https://api.github.com/users/moby","html_url":"https://github.com/moby","followers_url":"https://api.github.com/users/moby/followers","following_url":"https://api.github.com/users/moby/following{/other_user}","gists_url":"https://api.github.com/users/moby/gists{/gist_id}","starred_url":"https://api.github.com/users/moby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/moby/subscriptions","organizations_url":"https://api.github.com/users/moby/orgs","repos_url":"https://api.github.com/users/moby/repos","events_url":"https://api.github.com/users/moby/events{/privacy}","received_events_url":"https://api.github.com/users/moby/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/moby/moby","description":"Moby Project - a collaborative project for the container ecosystem to assemble container-based systems","fork":false,"url":"https://api.github.com/repos/moby/moby","forks_url":"https://api.github.com/repos/moby/moby/forks","keys_url":"https://api.github.com/repos/moby/moby/keys{/key_id}","collaborators_url":"https://api.github.com/repos/moby/moby/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/moby/moby/teams","hooks_url":"https://api.github.com/repos/moby/moby/hooks","issue_events_url":"https://api.github.com/repos/moby/moby/issues/events{/number}","events_url":"https://api.github.com/repos/moby/moby/events","assignees_url":"https://api.github.com/repos/moby/moby/assignees{/user}","branches_url":"https://api.github.com/repos/moby/moby/branches{/branch}","tags_url":"https://api.github.com/repos/moby/moby/tags","blobs_url":"https://api.github.com/repos/moby/moby/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/moby/moby/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/moby/moby/git/refs{/sha}","trees_url":"https://api.github.com/repos/moby/moby/git/trees{/sha}","statuses_url":"https://api.github.com/repos/moby/moby/statuses/{sha}","languages_url":"https://api.github.com/repos/moby/moby/languages","stargazers_url":"https://api.github.com/repos/moby/moby/stargazers","contributors_url":"https://api.github.com/repos/moby/moby/contributors","subscribers_url":"https://api.github.com/repos/moby/moby/subscribers","subscription_url":"https://api.github.com/repos/moby/moby/subscription","commits_url":"https://api.github.com/repos/moby/moby/commits{/sha}","git_commits_url":"https://api.github.com/repos/moby/moby/git/commits{/sha}","comments_url":"https://api.github.com/repos/moby/moby/comments{/number}","issue_comment_url":"https://api.github.com/repos/moby/moby/issues/comments{/number}","contents_url":"https://api.github.com/repos/moby/moby/contents/{+path}","compare_url":"https://api.github.com/repos/moby/moby/compare/{base}...{head}","merges_url":"https://api.github.com/repos/moby/moby/merges","archive_url":"https://api.github.com/repos/moby/moby/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/moby/moby/downloads","issues_url":"https://api.github.com/repos/moby/moby/issues{/number}","pulls_url":"https://api.github.com/repos/moby/moby/pulls{/number}","milestones_url":"https://api.github.com/repos/moby/moby/milestones{/number}","notifications_url":"https://api.github.com/repos/moby/moby/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/moby/moby/labels{/name}","releases_url":"https://api.github.com/repos/moby/moby/releases{/id}","deployments_url":"https://api.github.com/repos/moby/moby/deployments","created_at":"2013-01-18T18:10:57Z","updated_at":"2018-04-26T18:34:37Z","pushed_at":"2018-04-26T21:01:16Z","git_url":"git://github.com/moby/moby.git","ssh_url":"git@github.com:moby/moby.git","clone_url":"https://github.com/moby/moby.git","svn_url":"https://github.com/moby/moby","homepage":"https://mobyproject.org/","size":132975,"stargazers_count":48628,"watchers_count":48628,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":14283,"mirror_url":null,"archived":false,"open_issues_count":3166,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":14283,"open_issues":3166,"watchers":48628,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/moby/moby/pulls/36522"},"html":{"href":"https://github.com/moby/moby/pull/36522"},"issue":{"href":"https://api.github.com/repos/moby/moby/issues/36522"},"comments":{"href":"https://api.github.com/repos/moby/moby/issues/36522/comments"},"review_comments":{"href":"https://api.github.com/repos/moby/moby/pulls/36522/comments"},"review_comment":{"href":"https://api.github.com/repos/moby/moby/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/moby/moby/pulls/36522/commits"},"statuses":{"href":"https://api.github.com/repos/moby/moby/statuses/830da1de0f64886da06766e40d09b78a2b3be579"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"cpuguy83"}}