{"action":"created","comment":{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/comments/78476577","id":78476577,"diff_hunk":"@@ -726,7 +726,7 @@ func (dg *dockerGoClient) Stats(id string, ctx context.Context) (<-chan *docker.\n \tgo func() {\n \t\tstatsErr := client.Stats(options)\n \t\tif statsErr != nil {\n-\t\t\tseelog.Warnf(\"Error retrieving stats for container %s: %v\", id, statsErr)\n+\t\t\tseelog.Debugf(\"Error retrieving stats for container %s: %v\", id, statsErr)","path":"agent/engine/docker_container_engine.go","position":5,"original_position":5,"commit_id":"4088cc22745747f460253d146244deab1e907fde","original_commit_id":"4088cc22745747f460253d146244deab1e907fde","user":{"login":"samuelkarp","id":737750,"avatar_url":"https://avatars.githubusercontent.com/u/737750?v=3","gravatar_id":"","url":"https://api.github.com/users/samuelkarp","html_url":"https://github.com/samuelkarp","followers_url":"https://api.github.com/users/samuelkarp/followers","following_url":"https://api.github.com/users/samuelkarp/following{/other_user}","gists_url":"https://api.github.com/users/samuelkarp/gists{/gist_id}","starred_url":"https://api.github.com/users/samuelkarp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samuelkarp/subscriptions","organizations_url":"https://api.github.com/users/samuelkarp/orgs","repos_url":"https://api.github.com/users/samuelkarp/repos","events_url":"https://api.github.com/users/samuelkarp/events{/privacy}","received_events_url":"https://api.github.com/users/samuelkarp/received_events","type":"User","site_admin":false},"body":"Why did you move this down to debug level?","created_at":"2016-09-13T00:00:15Z","updated_at":"2016-09-13T00:00:15Z","html_url":"https://github.com/aws/amazon-ecs-agent/pull/525#discussion_r78476577","pull_request_url":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/525","_links":{"self":{"href":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/comments/78476577"},"html":{"href":"https://github.com/aws/amazon-ecs-agent/pull/525#discussion_r78476577"},"pull_request":{"href":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/525"}}},"pull_request":{"url":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/525","id":84952332,"html_url":"https://github.com/aws/amazon-ecs-agent/pull/525","diff_url":"https://github.com/aws/amazon-ecs-agent/pull/525.diff","patch_url":"https://github.com/aws/amazon-ecs-agent/pull/525.patch","issue_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/525","number":525,"state":"open","locked":false,"title":"Bugfix: Stats engine reconciliation for missed remove events.","user":{"login":"aaithal","id":1183359,"avatar_url":"https://avatars.githubusercontent.com/u/1183359?v=3","gravatar_id":"","url":"https://api.github.com/users/aaithal","html_url":"https://github.com/aaithal","followers_url":"https://api.github.com/users/aaithal/followers","following_url":"https://api.github.com/users/aaithal/following{/other_user}","gists_url":"https://api.github.com/users/aaithal/gists{/gist_id}","starred_url":"https://api.github.com/users/aaithal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaithal/subscriptions","organizations_url":"https://api.github.com/users/aaithal/orgs","repos_url":"https://api.github.com/users/aaithal/repos","events_url":"https://api.github.com/users/aaithal/events{/privacy}","received_events_url":"https://api.github.com/users/aaithal/received_events","type":"User","site_admin":false},"body":"<!--\r\nPlease make sure you've read and understood our contributing guidelines;\r\nhttps://github.com/aws/amazon-ecs-agent/blob/master/CONTRIBUTING.md\r\n\r\nPlease provide the following information:\r\n-->\r\n\r\n### Summary\r\n<!-- What does this pull request do? -->\r\nThis is a quick fix for cases where stats engine might miss\r\n'container removed' event from the docker task engine\r\nevent stream.\r\n\r\nWithout this fix, the stats engine will spin on\r\n'docker stats' api until the container is removed.\r\n\r\nThis is mentioned as a quick fix because this is not addressing the\r\nroot-cause of why an event is sometimes being missed by the\r\nstats engine.\r\n\r\nThere's also the case of 'container added' event being missed\r\nwhich could cause metrics to be misrepresented.\r\n\r\nThis is also proposed as a fix for the issue #515.\r\n\r\nThere's a minor refactor of the task engine code where locks have been\r\nadded to protect api.Container status'es\r\n\r\n### Implementation details\r\n<!-- How are the changes implemented? -->\r\n\r\n* `engine.GetInstanceMetrics()` is used to trigger a state reconciliation in stats engine\r\n* `container.collect()` is used to check if the container is terminal when disconnected from `docker stats`\r\n\r\n### Testing\r\n<!-- How was this tested? -->\r\n<!--\r\nNote for external contributors:\r\n`make short-test` and `make test` can run anywhere in a development environment\r\nlike your laptop.  Please ensure both of these pass before opening the pull\r\nrequest.  `make run-functional-tests` must be run on an EC2 instance with an\r\ninstance profile allowing it access to AWS resources.  Running\r\n`make run-functional-tests` may incur charges to your AWS account; if you're\r\nunable or unwilling to run these tests in your own account, we can run the tests\r\nand provide test results.\r\n-->\r\n- [X] Builds (`make release`)\r\n- [X] Unit tests (`make short-test`) pass\r\n- [X] Integration tests (`make test`) pass\r\n- [X] Functional tests (`make run-functional-tests`) pass\r\n- Manual tests\r\n - [X] Built an agent, where 'stats engine' would deliberately ignore 'container removed' events and verified that it's still reconcile and not spin as it used to before this fix.\r\n\r\nNew tests cover the changes: Yes\r\n\r\n### Description for the changelog\r\n<!--\r\nWrite a short (one line) summary that describes the changes in this\r\npull request for inclusion in the changelog.\r\nYou can see our changelog entry style here:\r\nhttps://github.com/aws/amazon-ecs-agent/commit/c9aefebc2b3007f09468f651f6308136bd7b384f\r\n-->\r\nFixes an issue where stats engine could miss 'remove' events and indefinitely query stats for stopped containers\r\n### Licensing\r\n<!--\r\nPlease confirm that this contribution is under the terms of the Apache 2.0\r\nLicense.\r\n-->\r\nThis contribution is under the terms of the Apache 2.0 License: Yes (AWS employee)\r\n\r\n\r\n","created_at":"2016-09-12T21:57:15Z","updated_at":"2016-09-13T00:00:15Z","closed_at":null,"merged_at":null,"merge_commit_sha":"714db3ba70162a2ca6cd16839ed50f8488f377c8","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/525/commits","review_comments_url":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/525/comments","review_comment_url":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/comments{/number}","comments_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/525/comments","statuses_url":"https://api.github.com/repos/aws/amazon-ecs-agent/statuses/4088cc22745747f460253d146244deab1e907fde","head":{"label":"aaithal:statsFillingUpDisk","ref":"statsFillingUpDisk","sha":"4088cc22745747f460253d146244deab1e907fde","user":{"login":"aaithal","id":1183359,"avatar_url":"https://avatars.githubusercontent.com/u/1183359?v=3","gravatar_id":"","url":"https://api.github.com/users/aaithal","html_url":"https://github.com/aaithal","followers_url":"https://api.github.com/users/aaithal/followers","following_url":"https://api.github.com/users/aaithal/following{/other_user}","gists_url":"https://api.github.com/users/aaithal/gists{/gist_id}","starred_url":"https://api.github.com/users/aaithal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaithal/subscriptions","organizations_url":"https://api.github.com/users/aaithal/orgs","repos_url":"https://api.github.com/users/aaithal/repos","events_url":"https://api.github.com/users/aaithal/events{/privacy}","received_events_url":"https://api.github.com/users/aaithal/received_events","type":"User","site_admin":false},"repo":{"id":30386191,"name":"amazon-ecs-agent","full_name":"aaithal/amazon-ecs-agent","owner":{"login":"aaithal","id":1183359,"avatar_url":"https://avatars.githubusercontent.com/u/1183359?v=3","gravatar_id":"","url":"https://api.github.com/users/aaithal","html_url":"https://github.com/aaithal","followers_url":"https://api.github.com/users/aaithal/followers","following_url":"https://api.github.com/users/aaithal/following{/other_user}","gists_url":"https://api.github.com/users/aaithal/gists{/gist_id}","starred_url":"https://api.github.com/users/aaithal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaithal/subscriptions","organizations_url":"https://api.github.com/users/aaithal/orgs","repos_url":"https://api.github.com/users/aaithal/repos","events_url":"https://api.github.com/users/aaithal/events{/privacy}","received_events_url":"https://api.github.com/users/aaithal/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/aaithal/amazon-ecs-agent","description":"Amazon EC2 Container Service Agent","fork":true,"url":"https://api.github.com/repos/aaithal/amazon-ecs-agent","forks_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/forks","keys_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/keys{/key_id}","collaborators_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/teams","hooks_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/hooks","issue_events_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/issues/events{/number}","events_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/events","assignees_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/assignees{/user}","branches_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/branches{/branch}","tags_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/tags","blobs_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/git/refs{/sha}","trees_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/git/trees{/sha}","statuses_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/statuses/{sha}","languages_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/languages","stargazers_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/stargazers","contributors_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/contributors","subscribers_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/subscribers","subscription_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/subscription","commits_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/commits{/sha}","git_commits_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/git/commits{/sha}","comments_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/comments{/number}","issue_comment_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/issues/comments{/number}","contents_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/contents/{+path}","compare_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/compare/{base}...{head}","merges_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/merges","archive_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/downloads","issues_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/issues{/number}","pulls_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/pulls{/number}","milestones_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/milestones{/number}","notifications_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/labels{/name}","releases_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/releases{/id}","deployments_url":"https://api.github.com/repos/aaithal/amazon-ecs-agent/deployments","created_at":"2015-02-06T00:24:31Z","updated_at":"2015-04-27T22:13:41Z","pushed_at":"2016-09-12T21:49:29Z","git_url":"git://github.com/aaithal/amazon-ecs-agent.git","ssh_url":"git@github.com:aaithal/amazon-ecs-agent.git","clone_url":"https://github.com/aaithal/amazon-ecs-agent.git","svn_url":"https://github.com/aaithal/amazon-ecs-agent","homepage":"http://aws.amazon.com/ecs/","size":4240,"stargazers_count":2,"watchers_count":2,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":2,"default_branch":"master"}},"base":{"label":"aws:dev","ref":"dev","sha":"e56f9df72f7dd6049bfca27733fa93218d3d0722","user":{"login":"aws","id":2232217,"avatar_url":"https://avatars.githubusercontent.com/u/2232217?v=3","gravatar_id":"","url":"https://api.github.com/users/aws","html_url":"https://github.com/aws","followers_url":"https://api.github.com/users/aws/followers","following_url":"https://api.github.com/users/aws/following{/other_user}","gists_url":"https://api.github.com/users/aws/gists{/gist_id}","starred_url":"https://api.github.com/users/aws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aws/subscriptions","organizations_url":"https://api.github.com/users/aws/orgs","repos_url":"https://api.github.com/users/aws/repos","events_url":"https://api.github.com/users/aws/events{/privacy}","received_events_url":"https://api.github.com/users/aws/received_events","type":"Organization","site_admin":false},"repo":{"id":27551774,"name":"amazon-ecs-agent","full_name":"aws/amazon-ecs-agent","owner":{"login":"aws","id":2232217,"avatar_url":"https://avatars.githubusercontent.com/u/2232217?v=3","gravatar_id":"","url":"https://api.github.com/users/aws","html_url":"https://github.com/aws","followers_url":"https://api.github.com/users/aws/followers","following_url":"https://api.github.com/users/aws/following{/other_user}","gists_url":"https://api.github.com/users/aws/gists{/gist_id}","starred_url":"https://api.github.com/users/aws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aws/subscriptions","organizations_url":"https://api.github.com/users/aws/orgs","repos_url":"https://api.github.com/users/aws/repos","events_url":"https://api.github.com/users/aws/events{/privacy}","received_events_url":"https://api.github.com/users/aws/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/aws/amazon-ecs-agent","description":"Amazon EC2 Container Service Agent","fork":false,"url":"https://api.github.com/repos/aws/amazon-ecs-agent","forks_url":"https://api.github.com/repos/aws/amazon-ecs-agent/forks","keys_url":"https://api.github.com/repos/aws/amazon-ecs-agent/keys{/key_id}","collaborators_url":"https://api.github.com/repos/aws/amazon-ecs-agent/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/aws/amazon-ecs-agent/teams","hooks_url":"https://api.github.com/repos/aws/amazon-ecs-agent/hooks","issue_events_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/events{/number}","events_url":"https://api.github.com/repos/aws/amazon-ecs-agent/events","assignees_url":"https://api.github.com/repos/aws/amazon-ecs-agent/assignees{/user}","branches_url":"https://api.github.com/repos/aws/amazon-ecs-agent/branches{/branch}","tags_url":"https://api.github.com/repos/aws/amazon-ecs-agent/tags","blobs_url":"https://api.github.com/repos/aws/amazon-ecs-agent/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/aws/amazon-ecs-agent/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/aws/amazon-ecs-agent/git/refs{/sha}","trees_url":"https://api.github.com/repos/aws/amazon-ecs-agent/git/trees{/sha}","statuses_url":"https://api.github.com/repos/aws/amazon-ecs-agent/statuses/{sha}","languages_url":"https://api.github.com/repos/aws/amazon-ecs-agent/languages","stargazers_url":"https://api.github.com/repos/aws/amazon-ecs-agent/stargazers","contributors_url":"https://api.github.com/repos/aws/amazon-ecs-agent/contributors","subscribers_url":"https://api.github.com/repos/aws/amazon-ecs-agent/subscribers","subscription_url":"https://api.github.com/repos/aws/amazon-ecs-agent/subscription","commits_url":"https://api.github.com/repos/aws/amazon-ecs-agent/commits{/sha}","git_commits_url":"https://api.github.com/repos/aws/amazon-ecs-agent/git/commits{/sha}","comments_url":"https://api.github.com/repos/aws/amazon-ecs-agent/comments{/number}","issue_comment_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/comments{/number}","contents_url":"https://api.github.com/repos/aws/amazon-ecs-agent/contents/{+path}","compare_url":"https://api.github.com/repos/aws/amazon-ecs-agent/compare/{base}...{head}","merges_url":"https://api.github.com/repos/aws/amazon-ecs-agent/merges","archive_url":"https://api.github.com/repos/aws/amazon-ecs-agent/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/aws/amazon-ecs-agent/downloads","issues_url":"https://api.github.com/repos/aws/amazon-ecs-agent/issues{/number}","pulls_url":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls{/number}","milestones_url":"https://api.github.com/repos/aws/amazon-ecs-agent/milestones{/number}","notifications_url":"https://api.github.com/repos/aws/amazon-ecs-agent/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/aws/amazon-ecs-agent/labels{/name}","releases_url":"https://api.github.com/repos/aws/amazon-ecs-agent/releases{/id}","deployments_url":"https://api.github.com/repos/aws/amazon-ecs-agent/deployments","created_at":"2014-12-04T17:31:49Z","updated_at":"2016-09-12T20:24:40Z","pushed_at":"2016-09-12T21:57:16Z","git_url":"git://github.com/aws/amazon-ecs-agent.git","ssh_url":"git@github.com:aws/amazon-ecs-agent.git","clone_url":"https://github.com/aws/amazon-ecs-agent.git","svn_url":"https://github.com/aws/amazon-ecs-agent","homepage":"http://aws.amazon.com/ecs/","size":7371,"stargazers_count":653,"watchers_count":653,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":175,"mirror_url":null,"open_issues_count":71,"forks":175,"open_issues":71,"watchers":653,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/525"},"html":{"href":"https://github.com/aws/amazon-ecs-agent/pull/525"},"issue":{"href":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/525"},"comments":{"href":"https://api.github.com/repos/aws/amazon-ecs-agent/issues/525/comments"},"review_comments":{"href":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/525/comments"},"review_comment":{"href":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/aws/amazon-ecs-agent/pulls/525/commits"},"statuses":{"href":"https://api.github.com/repos/aws/amazon-ecs-agent/statuses/4088cc22745747f460253d146244deab1e907fde"}}}}
{"actor":{"display_login":"samuelkarp"}}