{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71071","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71071/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71071/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71071/events","html_url":"https://github.com/kubernetes/kubernetes/issues/71071","id":381133698,"node_id":"MDU6SXNzdWUzODExMzM2OTg=","number":71071,"title":"kube-proxy ipvs mode hangs after a few hours, needs manual restart","user":{"login":"gjcarneiro","id":2197096,"node_id":"MDQ6VXNlcjIxOTcwOTY=","avatar_url":"https://avatars2.githubusercontent.com/u/2197096?v=4","gravatar_id":"","url":"https://api.github.com/users/gjcarneiro","html_url":"https://github.com/gjcarneiro","followers_url":"https://api.github.com/users/gjcarneiro/followers","following_url":"https://api.github.com/users/gjcarneiro/following{/other_user}","gists_url":"https://api.github.com/users/gjcarneiro/gists{/gist_id}","starred_url":"https://api.github.com/users/gjcarneiro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gjcarneiro/subscriptions","organizations_url":"https://api.github.com/users/gjcarneiro/orgs","repos_url":"https://api.github.com/users/gjcarneiro/repos","events_url":"https://api.github.com/users/gjcarneiro/events{/privacy}","received_events_url":"https://api.github.com/users/gjcarneiro/received_events","type":"User","site_admin":false},"labels":[{"id":755527763,"node_id":"MDU6TGFiZWw3NTU1Mjc3NjM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/ipvs","name":"area/ipvs","color":"0052cc","default":false},{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":116712108,"node_id":"MDU6TGFiZWwxMTY3MTIxMDg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/network","name":"sig/network","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-11-15T12:11:10Z","updated_at":"2018-11-27T06:07:16Z","closed_at":null,"author_association":"NONE","body":"<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->\r\n\r\n\r\n**What happened**: upgraded from v1.11.0 to v1.12.2, now kube-proxy hangs, usually in less than a day\r\n\r\n**What you expected to happen**: it should continue working\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:  just have a normal kube-proxy running in ipvs mode.  It gets stuck usually in less than a day.\r\n\r\n**Anything else we need to know?**:\r\n\r\nI've sent SIGABRT to the kube-proxy process to get a stack trace, and the only thing interesting is that when the bug happens goroutine 1 is reading from netlink, whereas when I send SIGABRT and the process is not stuck it is not reading from netlink.\r\n\r\n```\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: goroutine 1 [syscall]:\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.Syscall6(0x2d, 0x3, 0xc42081c000, 0x1000, 0x0, 0xc42083deb0, 0xc42083dea4, 0x40fb86, 0x7f2ab9788aa8, 0x0)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.recvfrom(0x3, 0xc42081c000, 0x1000, 0x1000, 0x0, 0xc42083deb0, 0xc42083dea4, 0x101ffffffffffff, 0x0, 0x1000)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1665 +0xa6\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.Recvfrom(0x3, 0xc42081c000, 0x1000, 0x1000, 0x0, 0x1000, 0x0, 0x0, 0x16ad4c0, 0x1f4d508)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/syscall_unix.go:252 +0xaf\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xc4204d3440, 0x0, 0x0, 0x0, 0x16ad4c0, 0x1f4d508)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:613 +0x9b\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.execute(0xc4204d3440, 0xc42083e150, 0x0, 0x1, 0x2, 0xc420423740, 0x1, 0x2)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:219 +0xd3\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).doCmdwithResponse(0xc420573990, 0xc420990750, 0x0, 0xc42003d501, 0xa, 0x10, 0xc420990701, 0xc42083e230, 0x118dd6b)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:140 +0x20b\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).doCmd(0xc420573990, 0xc420990750, 0x0, 0x1, 0x60000020fc6e0, 0xc420990750)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:149 +0x48\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).NewService(0xc420573990, 0xc420990750, 0x0, 0x0)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/ipvs.go:117 +0x43\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/ipvs.(*runner).AddVirtualServer(0xc42052a340, 0xc420b0e550, 0x1e, 0xc42083e3b8)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/ipvs/ipvs_linux.go:61 +0x60\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).syncService(0xc4201b3e00, 0xc420d53620, 0x21, 0xc420b0e550, 0xc420357901, 0x5, 0xc420d53620)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:1454 +0x635\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).syncProxyRules(0xc4201b3e00)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:803 +0x1081\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).(k8s.io/kubernetes/pkg/proxy/ipvs.syncProxyRules)-fm()\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:395 +0x2a\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/async.(*BoundedFrequencyRunner).tryRun(0xc42042e3f0)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/async/bounded_frequency_runner.go:217 +0xb6\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/async.(*BoundedFrequencyRunner).Loop(0xc42042e3f0, 0xc4200b0120)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/async/bounded_frequency_runner.go:179 +0x211\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).SyncLoop(0xc4201b3e00)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:589 +0x4e\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.(*ProxyServer).Run(0xc4207f2000, 0xc4207f2000, 0x0)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:568 +0x3dd\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.(*Options).Run(0xc4201082c0, 0xc42063e230, 0x0)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:238 +0x69\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.NewProxyCommand.func1(0xc4200f2500, 0xc42063e230, 0x0, 0x7)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:360 +0x156\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200f2500, 0xc4200c4010, 0x7, 0x7, 0xc4200f2500, 0xc4200c4010)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200f2500, 0xc4205dc090, 0x16076a0, 0xc4207a7ee8)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200f2500, 0x1608b88, 0x20dcf40)\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r\nNov 15 11:42:05 hex-48b-pm kube-proxy[5389]: main.main()\r\n```\r\n\r\nThe last lines that it logged were:\r\n\r\n```\r\nNov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392942    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.241:4059/TCP/10.125.2.26:4059\r\nNov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392952    5389 graceful_termination.go:160] Trying to delete rs: 10.128.32.132:5432/TCP/10.125.2.13:6432\r\nNov 15 07:48:52 hex-48b-pm kube-proxy[5389]: E1115 07:48:52.392978    5389 graceful_termination.go:89] Try delete rs \"10.128.32.132:5432/TCP/10.125.2.13:6432\" err: Failed to delete rs \"10.128.32.132:5432/TCP/10.125.2.13:6432\", can't \r\nfind the real server\r\nNov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392985    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.32.132:5432/TCP/10.125.2.13:6432\r\nNov 15 07:48:52 hex-48b-pm kube-proxy[5389]: E1115 07:48:52.392992    5389 graceful_termination.go:183] Try flush graceful termination list err\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393147    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.156:8125/TCP/10.125.2.21:8125\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393260    5389 graceful_termination.go:89] Try delete rs \"10.128.33.156:8125/TCP/10.125.2.21:8125\" err: Failed to delete rs \"10.128.33.156:8125/TCP/10.125.2.21:8125\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393272    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.156:8125/TCP/10.125.2.21:8125\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393281    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.142:6379/TCP/10.125.6.13:6379\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393310    5389 graceful_termination.go:89] Try delete rs \"10.128.33.142:6379/TCP/10.125.6.13:6379\" err: Failed to delete rs \"10.128.33.142:6379/TCP/10.125.6.13:6379\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393317    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.142:6379/TCP/10.125.6.13:6379\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393327    5389 graceful_termination.go:160] Trying to delete rs: 10.128.34.237:6379/TCP/10.125.6.11:6379\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393355    5389 graceful_termination.go:89] Try delete rs \"10.128.34.237:6379/TCP/10.125.6.11:6379\" err: Failed to delete rs \"10.128.34.237:6379/TCP/10.125.6.11:6379\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393362    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.34.237:6379/TCP/10.125.6.11:6379\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393371    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.200:5432/TCP/10.125.6.27:6432\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393397    5389 graceful_termination.go:89] Try delete rs \"10.128.33.200:5432/TCP/10.125.6.27:6432\" err: Failed to delete rs \"10.128.33.200:5432/TCP/10.125.6.27:6432\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393404    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.200:5432/TCP/10.125.6.27:6432\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393412    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.228:6379/TCP/10.125.2.63:6379\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393440    5389 graceful_termination.go:89] Try delete rs \"10.128.33.228:6379/TCP/10.125.2.63:6379\" err: Failed to delete rs \"10.128.33.228:6379/TCP/10.125.2.63:6379\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393447    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.228:6379/TCP/10.125.2.63:6379\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393455    5389 graceful_termination.go:160] Trying to delete rs: 10.128.34.247:5672/TCP/10.125.2.6:5672\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393482    5389 graceful_termination.go:89] Try delete rs \"10.128.34.247:5672/TCP/10.125.2.6:5672\" err: Failed to delete rs \"10.128.34.247:5672/TCP/10.125.2.6:5672\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393489    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.34.247:5672/TCP/10.125.2.6:5672\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393497    5389 graceful_termination.go:160] Trying to delete rs: 10.128.35.47:4007/TCP/10.125.2.15:4007\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393523    5389 graceful_termination.go:89] Try delete rs \"10.128.35.47:4007/TCP/10.125.2.15:4007\" err: Failed to delete rs \"10.128.35.47:4007/TCP/10.125.2.15:4007\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393530    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.35.47:4007/TCP/10.125.2.15:4007\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393538    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.241:4059/TCP/10.125.2.26:4059\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393563    5389 graceful_termination.go:89] Try delete rs \"10.128.33.241:4059/TCP/10.125.2.26:4059\" err: Failed to delete rs \"10.128.33.241:4059/TCP/10.125.2.26:4059\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393571    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.241:4059/TCP/10.125.2.26:4059\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393578    5389 graceful_termination.go:160] Trying to delete rs: 10.128.32.132:5432/TCP/10.125.2.13:6432\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393605    5389 graceful_termination.go:89] Try delete rs \"10.128.32.132:5432/TCP/10.125.2.13:6432\" err: Failed to delete rs \"10.128.32.132:5432/TCP/10.125.2.13:6432\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393613    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.32.132:5432/TCP/10.125.2.13:6432\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393621    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.252:6379/TCP/10.125.6.26:6379\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393648    5389 graceful_termination.go:89] Try delete rs \"10.128.33.252:6379/TCP/10.125.6.26:6379\" err: Failed to delete rs \"10.128.33.252:6379/TCP/10.125.6.26:6379\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393656    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.252:6379/TCP/10.125.6.26:6379\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393664    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.237:6379/TCP/10.125.6.34:6379\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393690    5389 graceful_termination.go:89] Try delete rs \"10.128.33.237:6379/TCP/10.125.6.34:6379\" err: Failed to delete rs \"10.128.33.237:6379/TCP/10.125.6.34:6379\", can't find the real server\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393697    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.237:6379/TCP/10.125.6.34:6379\r\nNov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393704    5389 graceful_termination.go:183] Try flush graceful termination list err\r\nNov 15 07:50:52 hex-48b-pm kube-proxy[5389]: I1115 07:50:52.393838    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.156:8125/TCP/10.125.2.21:8125\r\nNov 15 07:50:52 hex-48b-pm kube-proxy[5389]: E1115 07:50:52.393940    5389 graceful_termination.go:89] Try delete rs \"10.128.33.156:8125/TCP/10.125.2.21:8125\" err: device or resource busy\r\nNov 15 07:50:52 hex-48b-pm kube-proxy[5389]: I1115 07:50:52.393957    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.142:6379/TCP/10.125.6.13:6379\r\nNov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420734    5389 proxier.go:1496] Failed to list IPVS destinations, error: invalid argument\r\nNov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420763    5389 proxier.go:809] Failed to sync endpoint for service: 10.128.33.156:8125/TCP, err: invalid argument\r\nNov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420889    5389 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0\r\nNov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420903    5389 proxier.go:809] Failed to sync endpoint for service: 10.128.35.150:6379/TCP, err: Expected only one service obtained=0\r\nNov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420950    5389 proxier.go:1455] Failed to add IPVS service \"monitoring/extradata-inserter:\": file exists\r\nNov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420961    5389 proxier.go:812] Failed to sync service: 10.128.34.54:80/TCP, err: file exists\r\nNov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.421886    5389 proxier.go:1544] Failed to add destination: 10.125.27.66:80, error: file exists\r\n```\r\n\r\nIf I restart kube-proxy (manually, ugh), it all recovers and works fine.  But it will get stuck again, eventually.\r\n\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`): v1.12.2\r\n- Cloud provider or hardware configuration: bare metal, 64 bit linux\r\n- OS (e.g. from /etc/os-release): Ubuntu 16.04.5 LTS\r\n- Kernel (e.g. `uname -a`): 4.15.0-39-generic (but get it on the original 4.4 kernel as well)\r\n- Install tools: binary from https://storage.googleapis.com/kubernetes-release/release/v1.12.2/kubernetes-server-linux-amd64.tar.gz\r\n- Others:\r\n\r\n<!-- DO NOT EDIT BELOW THIS LINE -->\r\n/kind bug"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/441936696","html_url":"https://github.com/kubernetes/kubernetes/issues/71071#issuecomment-441936696","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71071","id":441936696,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTkzNjY5Ng==","user":{"login":"berlinsaint","id":10629930,"node_id":"MDQ6VXNlcjEwNjI5OTMw","avatar_url":"https://avatars0.githubusercontent.com/u/10629930?v=4","gravatar_id":"","url":"https://api.github.com/users/berlinsaint","html_url":"https://github.com/berlinsaint","followers_url":"https://api.github.com/users/berlinsaint/followers","following_url":"https://api.github.com/users/berlinsaint/following{/other_user}","gists_url":"https://api.github.com/users/berlinsaint/gists{/gist_id}","starred_url":"https://api.github.com/users/berlinsaint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/berlinsaint/subscriptions","organizations_url":"https://api.github.com/users/berlinsaint/orgs","repos_url":"https://api.github.com/users/berlinsaint/repos","events_url":"https://api.github.com/users/berlinsaint/events{/privacy}","received_events_url":"https://api.github.com/users/berlinsaint/received_events","type":"User","site_admin":false},"created_at":"2018-11-27T06:07:16Z","updated_at":"2018-11-27T06:07:16Z","author_association":"NONE","body":"@gjcarneiro do you mean that in kernel 4.15 , it has the same problem?"}}
{"actor":{"display_login":"berlinsaint"}}