{"action":"created","comment":{"url":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/comments/65596726","id":65596726,"diff_hunk":"@@ -16,7 +16,10 @@ ExecStartPre=/usr/bin/systemctl is-active zk-health.service\n ExecStartPre=/usr/bin/systemctl is-active mesos-master@*\n ExecStartPre=/usr/bin/systemctl is-active marathon@*\n ExecStartPre=/usr/bin/sh -c \"docker pull $($IMAGE)\"\n-ExecStartPre=/usr/bin/sh -c \"/usr/bin/docker run --rm\\\n+ExecStartPre=/usr/bin/sh -c \"/usr/bin/docker run \\\n+--log-opt max-size=10m \\","path":"v3/fleet-local/control/cfn-signal@.service","position":6,"original_position":6,"commit_id":"d3c08195e4a2f02d06d767edad117ed391649787","original_commit_id":"d3c08195e4a2f02d06d767edad117ed391649787","user":{"login":"bossjones","id":709872,"avatar_url":"https://avatars.githubusercontent.com/u/709872?v=3","gravatar_id":"","url":"https://api.github.com/users/bossjones","html_url":"https://github.com/bossjones","followers_url":"https://api.github.com/users/bossjones/followers","following_url":"https://api.github.com/users/bossjones/following{/other_user}","gists_url":"https://api.github.com/users/bossjones/gists{/gist_id}","starred_url":"https://api.github.com/users/bossjones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bossjones/subscriptions","organizations_url":"https://api.github.com/users/bossjones/orgs","repos_url":"https://api.github.com/users/bossjones/repos","events_url":"https://api.github.com/users/bossjones/events{/privacy}","received_events_url":"https://api.github.com/users/bossjones/received_events","type":"User","site_admin":false},"body":"should this be ` --log-opt max-size=`etcdctl get /docker/config/logs-max-size` ?","created_at":"2016-06-02T18:49:04Z","updated_at":"2016-06-02T18:49:04Z","html_url":"https://github.com/adobe-platform/mesos-systemd/pull/230#discussion_r65596726","pull_request_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/230","_links":{"self":{"href":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/comments/65596726"},"html":{"href":"https://github.com/adobe-platform/mesos-systemd/pull/230#discussion_r65596726"},"pull_request":{"href":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/230"}}},"pull_request":{"url":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/230","id":72293828,"html_url":"https://github.com/adobe-platform/mesos-systemd/pull/230","diff_url":"https://github.com/adobe-platform/mesos-systemd/pull/230.diff","patch_url":"https://github.com/adobe-platform/mesos-systemd/pull/230.patch","issue_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/issues/230","number":230,"state":"open","locked":false,"title":"docker initiated logrotation","user":{"login":"f4tq","id":826130,"avatar_url":"https://avatars.githubusercontent.com/u/826130?v=3","gravatar_id":"","url":"https://api.github.com/users/f4tq","html_url":"https://github.com/f4tq","followers_url":"https://api.github.com/users/f4tq/followers","following_url":"https://api.github.com/users/f4tq/following{/other_user}","gists_url":"https://api.github.com/users/f4tq/gists{/gist_id}","starred_url":"https://api.github.com/users/f4tq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f4tq/subscriptions","organizations_url":"https://api.github.com/users/f4tq/orgs","repos_url":"https://api.github.com/users/f4tq/repos","events_url":"https://api.github.com/users/f4tq/events{/privacy}","received_events_url":"https://api.github.com/users/f4tq/received_events","type":"User","site_admin":false},"body":"Docker logs need to be rotated by the container instances themselves when using the default json logger because the container maintains an open file handle to the backing log file. From Coreos, this file can be found in /var/lib/docker/containers/*/*.log. To remove them from Cores though causes overlay and devicemapper corruption (unable to remove layers, images and or instances).\r\nIn summary,\r\n\r\nAll docker run commands have been provisioned with max-file & max-size. The values are pulled from etcd.\r\nA default etcd value is set in init if the user doesn't set one via secrets.json\r\nThe external logrotate unit is removed.\r\nThe docker cleanup script is changed to use the docker client to remove dead containers and untagged layers i.e. those that show as '' when docker images is run.\r\nSetting secrets.json type=etcd and key of /docker/config/logs-max-size and /docker/config/logs-max-file override the default of max-size=10m and max-file=10 i.e. no more than 10 rotated logs and of no more than 10MB in size.\r\nFinally, the related ticket is adobe-community/issues#9054\r\n\r\nTesting with ansible after convergence:\r\n* Setup the [inventory](https://git.corp.adobe.com/fortescu/Mesos4Dexi/tree/master/inventory-ethos-f4tq)\r\n* Checking the journals to make sure all `docker run` commands have targeted options\r\n```\r\nansible coreos_control:coreos_workers -m raw -a 'bash -c \"journalctl --no-pager | grep \\\"docker run\\\" | grep  -v max-size\"' -i ~/Mesos4Dexi/inventory-ethos-f4tq/ -s -f 10\r\n```\r\n* Make sure systemctl ok\r\n```\r\nansible coreos_control:coreos_workers:coreos_proxy -i ~/Mesos4Dexi/inventory-ethos-f4tq/  -m raw -a 'bash -c \"systemctl list-units | grep -v active\"'\r\n```\r\nwhich shouldn't list anything.  \r\n> Note: remove `-v` to see all the appropriately decorated docker commands\r\n\r\n* Test marathon access from each discrete node:\r\n```\r\nansible coreos_control -m raw -a 'curl -L -v -u admin:password \"http://{{ ec2_private_ip_address }}:8080\"' -s -i ~/Mesos4Dexi/inventory-ethos-f4tq/ \r\n```\r\n> Note: change the port/username/password for other well known processes \r\n\r\n* Check that all images are pulled on workers\r\n```\r\nansible coreos_workers -m raw -a \"docker images \" -i ~/Mesos4Dexi/inventory-ethos-f4tq/ -s -f 10 -vvv \r\n```\r\n* Check that all images are running on workers. \r\n```\r\nansible coreos_workers -m raw -a \"docker ps \" -i ~/Mesos4Dexi/inventory-ethos-f4tq/ -s -f 10 -vvv \r\n```\r\n* Inspect last 20 lines of mesos master journal across your control nodes\r\n```\r\nansible coreos_control -m raw -a 'bash -c \"systemctl --no-pager list-units|grep master|grep service|awk \\\"{print \\\\\\$1}\\\"| xargs -n 1 -IXX journalctl --no-pager -u XX | tail -20 \"' -i ~/Mesos4Dexi/inventory-ethos-f4tq/ -s -f 10\r\n```\r\n* Inspect last 20 lines of mesos workers journal across your worker nodes\r\n```\r\nansible coreos_workers -m raw -a 'bash -c \"systemctl --no-pager list-units|grep slave|grep service|awk \\\"{print \\\\\\$1}\\\"| xargs -n 1 -IXX journalctl --no-pager -u XX | tail -20 \"' -i ~/Mesos4Dexi/inventory-ethos-f4tq/ -s -f 10\r\n```\r\nAnyhow, all useful for testing this PR.\r\n\r\n","created_at":"2016-06-02T06:11:16Z","updated_at":"2016-06-02T18:49:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ccb8d76b55f7c06e5fdcbf77e07547c5070cd33a","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/230/commits","review_comments_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/230/comments","review_comment_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/comments{/number}","comments_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/issues/230/comments","statuses_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/statuses/d3c08195e4a2f02d06d767edad117ed391649787","head":{"label":"f4tq:master","ref":"master","sha":"d3c08195e4a2f02d06d767edad117ed391649787","user":{"login":"f4tq","id":826130,"avatar_url":"https://avatars.githubusercontent.com/u/826130?v=3","gravatar_id":"","url":"https://api.github.com/users/f4tq","html_url":"https://github.com/f4tq","followers_url":"https://api.github.com/users/f4tq/followers","following_url":"https://api.github.com/users/f4tq/following{/other_user}","gists_url":"https://api.github.com/users/f4tq/gists{/gist_id}","starred_url":"https://api.github.com/users/f4tq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f4tq/subscriptions","organizations_url":"https://api.github.com/users/f4tq/orgs","repos_url":"https://api.github.com/users/f4tq/repos","events_url":"https://api.github.com/users/f4tq/events{/privacy}","received_events_url":"https://api.github.com/users/f4tq/received_events","type":"User","site_admin":false},"repo":{"id":57340115,"name":"mesos-systemd","full_name":"f4tq/mesos-systemd","owner":{"login":"f4tq","id":826130,"avatar_url":"https://avatars.githubusercontent.com/u/826130?v=3","gravatar_id":"","url":"https://api.github.com/users/f4tq","html_url":"https://github.com/f4tq","followers_url":"https://api.github.com/users/f4tq/followers","following_url":"https://api.github.com/users/f4tq/following{/other_user}","gists_url":"https://api.github.com/users/f4tq/gists{/gist_id}","starred_url":"https://api.github.com/users/f4tq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f4tq/subscriptions","organizations_url":"https://api.github.com/users/f4tq/orgs","repos_url":"https://api.github.com/users/f4tq/repos","events_url":"https://api.github.com/users/f4tq/events{/privacy}","received_events_url":"https://api.github.com/users/f4tq/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/f4tq/mesos-systemd","description":"Systemd scripts to run Mesos on CoreOS","fork":true,"url":"https://api.github.com/repos/f4tq/mesos-systemd","forks_url":"https://api.github.com/repos/f4tq/mesos-systemd/forks","keys_url":"https://api.github.com/repos/f4tq/mesos-systemd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/f4tq/mesos-systemd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/f4tq/mesos-systemd/teams","hooks_url":"https://api.github.com/repos/f4tq/mesos-systemd/hooks","issue_events_url":"https://api.github.com/repos/f4tq/mesos-systemd/issues/events{/number}","events_url":"https://api.github.com/repos/f4tq/mesos-systemd/events","assignees_url":"https://api.github.com/repos/f4tq/mesos-systemd/assignees{/user}","branches_url":"https://api.github.com/repos/f4tq/mesos-systemd/branches{/branch}","tags_url":"https://api.github.com/repos/f4tq/mesos-systemd/tags","blobs_url":"https://api.github.com/repos/f4tq/mesos-systemd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/f4tq/mesos-systemd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/f4tq/mesos-systemd/git/refs{/sha}","trees_url":"https://api.github.com/repos/f4tq/mesos-systemd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/f4tq/mesos-systemd/statuses/{sha}","languages_url":"https://api.github.com/repos/f4tq/mesos-systemd/languages","stargazers_url":"https://api.github.com/repos/f4tq/mesos-systemd/stargazers","contributors_url":"https://api.github.com/repos/f4tq/mesos-systemd/contributors","subscribers_url":"https://api.github.com/repos/f4tq/mesos-systemd/subscribers","subscription_url":"https://api.github.com/repos/f4tq/mesos-systemd/subscription","commits_url":"https://api.github.com/repos/f4tq/mesos-systemd/commits{/sha}","git_commits_url":"https://api.github.com/repos/f4tq/mesos-systemd/git/commits{/sha}","comments_url":"https://api.github.com/repos/f4tq/mesos-systemd/comments{/number}","issue_comment_url":"https://api.github.com/repos/f4tq/mesos-systemd/issues/comments{/number}","contents_url":"https://api.github.com/repos/f4tq/mesos-systemd/contents/{+path}","compare_url":"https://api.github.com/repos/f4tq/mesos-systemd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/f4tq/mesos-systemd/merges","archive_url":"https://api.github.com/repos/f4tq/mesos-systemd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/f4tq/mesos-systemd/downloads","issues_url":"https://api.github.com/repos/f4tq/mesos-systemd/issues{/number}","pulls_url":"https://api.github.com/repos/f4tq/mesos-systemd/pulls{/number}","milestones_url":"https://api.github.com/repos/f4tq/mesos-systemd/milestones{/number}","notifications_url":"https://api.github.com/repos/f4tq/mesos-systemd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/f4tq/mesos-systemd/labels{/name}","releases_url":"https://api.github.com/repos/f4tq/mesos-systemd/releases{/id}","deployments_url":"https://api.github.com/repos/f4tq/mesos-systemd/deployments","created_at":"2016-04-28T23:25:30Z","updated_at":"2016-04-28T23:25:32Z","pushed_at":"2016-06-02T03:41:30Z","git_url":"git://github.com/f4tq/mesos-systemd.git","ssh_url":"git@github.com:f4tq/mesos-systemd.git","clone_url":"https://github.com/f4tq/mesos-systemd.git","svn_url":"https://github.com/f4tq/mesos-systemd","homepage":"","size":485,"stargazers_count":0,"watchers_count":0,"language":"Shell","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"adobe-platform:master","ref":"master","sha":"c7abba9478ae6d12158f84a76b019ece2ee98cb5","user":{"login":"adobe-platform","id":16107061,"avatar_url":"https://avatars.githubusercontent.com/u/16107061?v=3","gravatar_id":"","url":"https://api.github.com/users/adobe-platform","html_url":"https://github.com/adobe-platform","followers_url":"https://api.github.com/users/adobe-platform/followers","following_url":"https://api.github.com/users/adobe-platform/following{/other_user}","gists_url":"https://api.github.com/users/adobe-platform/gists{/gist_id}","starred_url":"https://api.github.com/users/adobe-platform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adobe-platform/subscriptions","organizations_url":"https://api.github.com/users/adobe-platform/orgs","repos_url":"https://api.github.com/users/adobe-platform/repos","events_url":"https://api.github.com/users/adobe-platform/events{/privacy}","received_events_url":"https://api.github.com/users/adobe-platform/received_events","type":"Organization","site_admin":false},"repo":{"id":35570054,"name":"mesos-systemd","full_name":"adobe-platform/mesos-systemd","owner":{"login":"adobe-platform","id":16107061,"avatar_url":"https://avatars.githubusercontent.com/u/16107061?v=3","gravatar_id":"","url":"https://api.github.com/users/adobe-platform","html_url":"https://github.com/adobe-platform","followers_url":"https://api.github.com/users/adobe-platform/followers","following_url":"https://api.github.com/users/adobe-platform/following{/other_user}","gists_url":"https://api.github.com/users/adobe-platform/gists{/gist_id}","starred_url":"https://api.github.com/users/adobe-platform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adobe-platform/subscriptions","organizations_url":"https://api.github.com/users/adobe-platform/orgs","repos_url":"https://api.github.com/users/adobe-platform/repos","events_url":"https://api.github.com/users/adobe-platform/events{/privacy}","received_events_url":"https://api.github.com/users/adobe-platform/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/adobe-platform/mesos-systemd","description":"Systemd scripts to run Mesos on CoreOS","fork":false,"url":"https://api.github.com/repos/adobe-platform/mesos-systemd","forks_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/forks","keys_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/teams","hooks_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/hooks","issue_events_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/issues/events{/number}","events_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/events","assignees_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/assignees{/user}","branches_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/branches{/branch}","tags_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/tags","blobs_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/git/refs{/sha}","trees_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/statuses/{sha}","languages_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/languages","stargazers_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/stargazers","contributors_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/contributors","subscribers_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/subscribers","subscription_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/subscription","commits_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/commits{/sha}","git_commits_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/git/commits{/sha}","comments_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/comments{/number}","issue_comment_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/issues/comments{/number}","contents_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/contents/{+path}","compare_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/merges","archive_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/downloads","issues_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/issues{/number}","pulls_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls{/number}","milestones_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/milestones{/number}","notifications_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/labels{/name}","releases_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/releases{/id}","deployments_url":"https://api.github.com/repos/adobe-platform/mesos-systemd/deployments","created_at":"2015-05-13T19:31:57Z","updated_at":"2016-01-15T19:51:46Z","pushed_at":"2016-06-02T06:11:16Z","git_url":"git://github.com/adobe-platform/mesos-systemd.git","ssh_url":"git@github.com:adobe-platform/mesos-systemd.git","clone_url":"https://github.com/adobe-platform/mesos-systemd.git","svn_url":"https://github.com/adobe-platform/mesos-systemd","homepage":"","size":594,"stargazers_count":1,"watchers_count":1,"language":"Shell","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":19,"mirror_url":null,"open_issues_count":10,"forks":19,"open_issues":10,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/230"},"html":{"href":"https://github.com/adobe-platform/mesos-systemd/pull/230"},"issue":{"href":"https://api.github.com/repos/adobe-platform/mesos-systemd/issues/230"},"comments":{"href":"https://api.github.com/repos/adobe-platform/mesos-systemd/issues/230/comments"},"review_comments":{"href":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/230/comments"},"review_comment":{"href":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/adobe-platform/mesos-systemd/pulls/230/commits"},"statuses":{"href":"https://api.github.com/repos/adobe-platform/mesos-systemd/statuses/d3c08195e4a2f02d06d767edad117ed391649787"}}}}
{"actor":{"display_login":"bossjones"}}