{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24295","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24295/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24295/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24295/events","html_url":"https://github.com/kubernetes/kubernetes/issues/24295","id":148510173,"number":24295,"title":"1000 node cluster, 1 kube-proxy comes up with \"open /sys/module/nf_conntrack/parameters/hashsize: read-only file system\"","user":{"login":"zmerlynn","id":4942464,"avatar_url":"https://avatars.githubusercontent.com/u/4942464?v=3","gravatar_id":"","url":"https://api.github.com/users/zmerlynn","html_url":"https://github.com/zmerlynn","followers_url":"https://api.github.com/users/zmerlynn/followers","following_url":"https://api.github.com/users/zmerlynn/following{/other_user}","gists_url":"https://api.github.com/users/zmerlynn/gists{/gist_id}","starred_url":"https://api.github.com/users/zmerlynn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zmerlynn/subscriptions","organizations_url":"https://api.github.com/users/zmerlynn/orgs","repos_url":"https://api.github.com/users/zmerlynn/repos","events_url":"https://api.github.com/users/zmerlynn/events{/privacy}","received_events_url":"https://api.github.com/users/zmerlynn/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/networking","name":"area/networking","color":"006b75"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2016-04-14T23:20:32Z","updated_at":"2016-04-15T16:51:14Z","closed_at":null,"body":"I just brought up a 1000 node cluster, and had one node with `kube-proxy` flapping in the following manner:\r\n\r\n```\r\nFlag --resource-container has been deprecated, This feature will be removed in a later release.\r\nI0414 21:09:01.727378       5 iptables.go:177] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory\r\nI0414 21:09:01.727453       5 server.go:154] setting OOM scores is unsupported in this build\r\nI0414 21:09:01.778461       5 server.go:200] Using iptables Proxier.\r\nI0414 21:09:01.778514       5 proxier.go:205] missing br-netfilter module or unset br-nf-call-iptables; proxy may not work as intended\r\nI0414 21:09:01.778537       5 server.go:213] Tearing down userspace rules.\r\nI0414 21:09:01.789946       5 conntrack.go:36] Setting nf_conntrack_max to 262144\r\nI0414 21:09:01.789985       5 conntrack.go:41] Setting conntrack hashsize to 65536\r\nopen /sys/module/nf_conntrack/parameters/hashsize: read-only file system\r\nFlag --resource-container has been deprecated, This feature will be removed in a later release.\r\nI0414 21:09:02.770482       5 iptables.go:177] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory\r\nI0414 21:09:02.770529       5 server.go:154] setting OOM scores is unsupported in this build\r\nI0414 21:09:02.802022       5 server.go:200] Using iptables Proxier.\r\nI0414 21:09:02.802068       5 proxier.go:205] missing br-netfilter module or unset br-nf-call-iptables; proxy may not work as intended\r\nI0414 21:09:02.802089       5 server.go:213] Tearing down userspace rules.\r\nI0414 21:09:02.812183       5 conntrack.go:36] Setting nf_conntrack_max to 262144\r\nI0414 21:09:02.812243       5 conntrack.go:41] Setting conntrack hashsize to 65536\r\nopen /sys/module/nf_conntrack/parameters/hashsize: read-only file system\r\nFlag --resource-container has been deprecated, This feature will be removed in a later release.\r\nI0414 21:09:16.992945       6 iptables.go:177] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory\r\nI0414 21:09:16.993003       6 server.go:154] setting OOM scores is unsupported in this build\r\nI0414 21:09:17.024283       6 server.go:200] Using iptables Proxier.\r\nI0414 21:09:17.024313       6 proxier.go:205] missing br-netfilter module or unset br-nf-call-iptables; proxy may not work as intended\r\nI0414 21:09:17.024327       6 server.go:213] Tearing down userspace rules.\r\nI0414 21:09:17.032319       6 conntrack.go:36] Setting nf_conntrack_max to 262144\r\nI0414 21:09:17.032350       6 conntrack.go:41] Setting conntrack hashsize to 65536\r\nopen /sys/module/nf_conntrack/parameters/hashsize: read-only file system\r\n```\r\n\r\nI compared this to a working `kube-proxy`, and it's clear that the `open /sys/module/nf_conntrack/parameters/hashsize: read-only file system` is obviously failing. `dmesg` is showing:\r\n\r\n```\r\n[ 6882.619004] aufs au_opts_verify:1570:docker[6990]: dirperm1 breaks the protection by the permission bits on the lower branch\r\n```\r\n\r\nCould possibly be related to https://github.com/openshift/origin/issues/7977\r\n\r\nKeeping it around for posterity, but this is kind of an expensive cluster."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/210543795","html_url":"https://github.com/kubernetes/kubernetes/issues/24295#issuecomment-210543795","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24295","id":210543795,"user":{"login":"Random-Liu","id":5821883,"avatar_url":"https://avatars.githubusercontent.com/u/5821883?v=3","gravatar_id":"","url":"https://api.github.com/users/Random-Liu","html_url":"https://github.com/Random-Liu","followers_url":"https://api.github.com/users/Random-Liu/followers","following_url":"https://api.github.com/users/Random-Liu/following{/other_user}","gists_url":"https://api.github.com/users/Random-Liu/gists{/gist_id}","starred_url":"https://api.github.com/users/Random-Liu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Random-Liu/subscriptions","organizations_url":"https://api.github.com/users/Random-Liu/orgs","repos_url":"https://api.github.com/users/Random-Liu/repos","events_url":"https://api.github.com/users/Random-Liu/events{/privacy}","received_events_url":"https://api.github.com/users/Random-Liu/received_events","type":"User","site_admin":false},"created_at":"2016-04-15T16:51:14Z","updated_at":"2016-04-15T16:51:14Z","body":"Maybe misclick... Sorry about that... :(\r\nReopen it."}}