{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/419","repository_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws","labels_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/419/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/419/comments","events_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/419/events","html_url":"https://github.com/kubernetes-incubator/kube-aws/issues/419","id":214962536,"number":419,"title":"\"write /dev/stdout: no space left on device\" errors in kube-apiserver when audit logging is enabled","user":{"login":"mumoshu","id":22009,"avatar_url":"https://avatars0.githubusercontent.com/u/22009?v=4","gravatar_id":"","url":"https://api.github.com/users/mumoshu","html_url":"https://github.com/mumoshu","followers_url":"https://api.github.com/users/mumoshu/followers","following_url":"https://api.github.com/users/mumoshu/following{/other_user}","gists_url":"https://api.github.com/users/mumoshu/gists{/gist_id}","starred_url":"https://api.github.com/users/mumoshu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumoshu/subscriptions","organizations_url":"https://api.github.com/users/mumoshu/orgs","repos_url":"https://api.github.com/users/mumoshu/repos","events_url":"https://api.github.com/users/mumoshu/events{/privacy}","received_events_url":"https://api.github.com/users/mumoshu/received_events","type":"User","site_admin":false},"labels":[{"id":461214840,"url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/milestones/40","html_url":"https://github.com/kubernetes-incubator/kube-aws/milestone/40","labels_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/milestones/40/labels","id":2471447,"number":40,"title":"v0.9.6-rc.5","description":"","creator":{"login":"mumoshu","id":22009,"avatar_url":"https://avatars0.githubusercontent.com/u/22009?v=4","gravatar_id":"","url":"https://api.github.com/users/mumoshu","html_url":"https://github.com/mumoshu","followers_url":"https://api.github.com/users/mumoshu/followers","following_url":"https://api.github.com/users/mumoshu/following{/other_user}","gists_url":"https://api.github.com/users/mumoshu/gists{/gist_id}","starred_url":"https://api.github.com/users/mumoshu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumoshu/subscriptions","organizations_url":"https://api.github.com/users/mumoshu/orgs","repos_url":"https://api.github.com/users/mumoshu/repos","events_url":"https://api.github.com/users/mumoshu/events{/privacy}","received_events_url":"https://api.github.com/users/mumoshu/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":14,"state":"closed","created_at":"2017-04-21T07:06:44Z","updated_at":"2017-04-26T12:48:07Z","due_on":"2017-04-26T07:00:00Z","closed_at":"2017-04-26T12:48:07Z"},"comments":5,"created_at":"2017-03-17T10:22:48Z","updated_at":"2018-02-11T05:51:44Z","closed_at":"2017-04-26T03:15:34Z","author_association":"COLLABORATOR","body":"`/dev/stdout` seems to be rotated when the experimental \"auditLog\" is enabled for kube-apiserver and the tmpfs mounted at `/dev` in the apiserver pod eventually get filled up.\r\n\r\n---\r\n\r\nMy kube-apiserver pods started spamming error messages like:\r\n\r\n```\r\n$ docker logs -f 0d9cfab181d7\r\n*snip*\r\nE0317 09:15:55.998046       1 audit.go:126] Unable to write audit log: 2017-03-17T09:15:55.998018096Z AUDIT: id=\"ccac3ba1-4433-4fad-abbe-00fb118bce8d\" ip=\"127.0.0.1\" method=\"GET\" user=\"<none>\" as=\"<self>\" asgroups=\"<lookup>\" namespace=\"kube-system\" uri=\"/api/v1/namespaces/kube-system/endpoints/kube-scheduler\"\r\n, the error is: write /dev/stdout: no space left on device\r\n```\r\n\r\nHow `/dev/stdout` can be filled up like this?\r\nInvestigating into the primary container of one of the pods revealed that the original `/dev/stdout`(=`/proc/self/fd/1`) had been rotated over and over the \"current\" `/dev/stdout` is now a regular file 😆 \r\n\r\n```\r\n$ docker exec -it 0d9cfab181d7 sh\r\n# df -h\r\nFilesystem      Size  Used Avail Use% Mounted on\r\noverlay          27G  4.4G   22G  18% /\r\ntmpfs           1.9G  1.9G     0 100% /dev\r\ntmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup\r\n/dev/xvda9       27G  4.4G   22G  18% /etc/hosts\r\n/dev/xvda3      985M  588M  347M  63% /etc/ssl/certs\r\nshm              64M     0   64M   0% /dev/shm\r\n```\r\n\r\n```\r\n# ls -lah /dev/stdout\r\n-rw-------. 1 root root 84M Mar 17 09:28 /dev/stdout\r\n# ls -lah /dev\r\ntotal 1.9G\r\ndrwxr-xr-x. 5 root root     760 Jan 12 12:36 .\r\ndrwxr-xr-x. 1 root root    4.0K Dec 27 06:36 ..\r\nlrwxrwxrwx. 1 root root      11 Dec 27 06:36 core -> /proc/kcore\r\nlrwxrwxrwx. 1 root root      13 Dec 27 06:36 fd -> /proc/self/fd\r\ncrw-rw-rw-. 1 root root  1,   7 Dec 27 06:36 full\r\ncrw-rw-rw-. 1 root root 10, 229 Dec 27 06:36 fuse\r\ndrwxrwxrwt. 2 root root      40 Dec 27 06:35 mqueue\r\ncrw-rw-rw-. 1 root root  1,   3 Dec 27 06:36 null\r\nlrwxrwxrwx. 1 root root       8 Dec 27 06:36 ptmx -> pts/ptmx\r\ndrwxr-xr-x. 2 root root       0 Dec 27 06:36 pts\r\ncrw-rw-rw-. 1 root root  1,   8 Dec 27 06:36 random\r\ndrwxrwxrwt. 2 root root      40 Dec 27 06:35 shm\r\nlrwxrwxrwx. 1 root root      15 Dec 27 06:36 stderr -> /proc/self/fd/2\r\nlrwxrwxrwx. 1 root root      15 Dec 27 06:36 stdin -> /proc/self/fd/0\r\n-rw-------. 1 root root     84M Mar 17 09:28 stdout\r\nlrwxrwxrwx. 1 root root      15 Dec 27 06:36 stdout-2016-12-28T06-08-02.236 -> /proc/self/fd/1\r\n-rw-------. 1 root root    100M Dec 29 02:58 stdout-2016-12-29T02-58-48.742\r\n-rw-------. 1 root root    100M Dec 29 23:06 stdout-2016-12-29T23-06-50.389\r\n-rw-------. 1 root root    100M Dec 30 19:23 stdout-2016-12-30T19-23-51.692\r\n-rw-------. 1 root root    100M Dec 31 15:07 stdout-2016-12-31T15-07-42.495\r\n-rw-------. 1 root root    100M Jan  1 11:58 stdout-2017-01-01T11-58-20.435\r\n-rw-------. 1 root root    100M Jan  2 09:45 stdout-2017-01-02T09-45-38.179\r\n-rw-------. 1 root root    100M Jan  3 07:29 stdout-2017-01-03T07-29-39.171\r\n-rw-------. 1 root root    100M Jan  4 04:39 stdout-2017-01-04T04-39-38.577\r\n-rw-------. 1 root root    100M Jan  5 02:16 stdout-2017-01-05T02-16-11.852\r\n-rw-------. 1 root root    100M Jan  6 00:15 stdout-2017-01-06T00-15-05.934\r\n-rw-------. 1 root root    100M Jan  6 21:00 stdout-2017-01-06T21-00-54.748\r\n-rw-------. 1 root root    100M Jan  7 16:02 stdout-2017-01-07T16-02-17.117\r\n-rw-------. 1 root root    100M Jan  8 11:14 stdout-2017-01-08T11-14-23.535\r\n-rw-------. 1 root root    100M Jan  9 06:24 stdout-2017-01-09T06-24-58.923\r\n-rw-------. 1 root root    100M Jan 10 01:14 stdout-2017-01-10T01-14-28.712\r\n-rw-------. 1 root root    100M Jan 10 19:02 stdout-2017-01-10T19-02-30.597\r\n-rw-------. 1 root root    100M Jan 11 15:21 stdout-2017-01-11T15-21-11.240\r\n-rw-------. 1 root root    100M Jan 12 12:36 stdout-2017-01-12T12-36-28.992\r\n-rw-r--r--. 1 root root       0 Dec 27 06:36 termination-log\r\ncrw-rw-rw-. 1 root root  5,   0 Dec 27 06:36 tty\r\ncrw-rw-rw-. 1 root root  1,   9 Dec 27 06:36 urandom\r\ncrw-rw-rw-. 1 root root  1,   5 Dec 27 06:36 zero\r\n```\r\n\r\nWe'd better stop rotating audit log files at least, or even forbid setting `--audit-log-file` to `/dev/stdout`.\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/comments/364724161","html_url":"https://github.com/kubernetes-incubator/kube-aws/issues/419#issuecomment-364724161","issue_url":"https://api.github.com/repos/kubernetes-incubator/kube-aws/issues/419","id":364724161,"user":{"login":"mumoshu","id":22009,"avatar_url":"https://avatars0.githubusercontent.com/u/22009?v=4","gravatar_id":"","url":"https://api.github.com/users/mumoshu","html_url":"https://github.com/mumoshu","followers_url":"https://api.github.com/users/mumoshu/followers","following_url":"https://api.github.com/users/mumoshu/following{/other_user}","gists_url":"https://api.github.com/users/mumoshu/gists{/gist_id}","starred_url":"https://api.github.com/users/mumoshu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumoshu/subscriptions","organizations_url":"https://api.github.com/users/mumoshu/orgs","repos_url":"https://api.github.com/users/mumoshu/repos","events_url":"https://api.github.com/users/mumoshu/events{/privacy}","received_events_url":"https://api.github.com/users/mumoshu/received_events","type":"User","site_admin":false},"created_at":"2018-02-11T05:51:44Z","updated_at":"2018-02-11T05:51:44Z","author_association":"COLLABORATOR","body":"Ah, good catch!"}}
{"actor":{"display_login":"mumoshu"}}