{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66509","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66509/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66509/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66509/events","html_url":"https://github.com/kubernetes/kubernetes/issues/66509","id":343731220,"node_id":"MDU6SXNzdWUzNDM3MzEyMjA=","number":66509,"title":"IPVS Proxier: fixed subset of nodes (proxier iptables or ipvs) lose connectivity to remote pods on nodes with ipvs","user":{"login":"matt-duch","id":1303097,"node_id":"MDQ6VXNlcjEzMDMwOTc=","avatar_url":"https://avatars3.githubusercontent.com/u/1303097?v=4","gravatar_id":"","url":"https://api.github.com/users/matt-duch","html_url":"https://github.com/matt-duch","followers_url":"https://api.github.com/users/matt-duch/followers","following_url":"https://api.github.com/users/matt-duch/following{/other_user}","gists_url":"https://api.github.com/users/matt-duch/gists{/gist_id}","starred_url":"https://api.github.com/users/matt-duch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matt-duch/subscriptions","organizations_url":"https://api.github.com/users/matt-duch/orgs","repos_url":"https://api.github.com/users/matt-duch/repos","events_url":"https://api.github.com/users/matt-duch/events{/privacy}","received_events_url":"https://api.github.com/users/matt-duch/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":617149945,"node_id":"MDU6TGFiZWw2MTcxNDk5NDU=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/needs-sig","name":"needs-sig","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-23T17:47:57Z","updated_at":"2018-07-23T17:47:58Z","closed_at":null,"author_association":"NONE","body":"**Is this a BUG REPORT or FEATURE REQUEST?**:\r\n\r\n> Uncomment only one, leave it on its own line:\r\n>\r\n/kind bug\r\n\r\n**What happened**:\r\nWe are attempting to transition our clusters from the iptables to the ipvs proxier.  \r\n\r\nUnder most conditions, this works well. However, for the specific case of:\r\nWhen moving node \"A\" over to ipvs, a fixed subset of other \"client\" nodes can no longer reach pods on node A (directly via pod ip, not via service).\r\nWhether or not the \"client\" node is ipvs or iptables doesn't seem to matter. Switching node A back to iptables solves the problem.\r\nThe only difference we can find between \"client\" nodes that lose the ability to connect to node A pods and those that continue to work, is that the \"client\" node has a pod with a host port. This may not have anything to do with the issue, but we've been unable to find any other differences.\r\n\r\n**What you expected to happen**:\r\nNo interruption in node (or pod) -> pod connectivity.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\nThis appears to be related to how iptables rules (under certain conditions) on client nodes interacts with ipvs proxiers on remote nodes.\r\n\r\n**Anything else we need to know?**:\r\nSample kube-proxy configuration:\r\n```\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  name: kube-proxy\r\n  namespace: kube-system\r\nspec:\r\n  volumes:\r\n  - name: ssl-certs-kubernetes\r\n    hostPath:\r\n      path: /etc/kubernetes/ssl\r\n  - name: ssl-certs-host\r\n    hostPath:\r\n      path: /etc/ssl/certs\r\n  - name: kubeconfig\r\n    hostPath:\r\n      path: /etc/kubernetes/kubeconfig-pod.yaml\r\n  - name: modules\r\n    hostPath:\r\n      path: /lib/modules\r\n  containers:\r\n  - name: kube-proxy\r\n    image: gcr.io/google_containers/kube-proxy:v1.11.1\r\n    command:\r\n    - /usr/local/bin/kube-proxy\r\n    - --master=https://10.0.1.17:443\r\n    - --kubeconfig=/etc/kubernetes/kubeconfig.yaml\r\n    - --proxy-mode=ipvs\r\n    - --hostname-override=10.0.1.82\r\n    - --cluster-cidr=10.240.0.0/12\r\n    - --v=5\r\n    volumeMounts:\r\n    - name: ssl-certs-kubernetes\r\n      readOnly: true\r\n      mountPath: /etc/kubernetes/ssl\r\n    - name: ssl-certs-host\r\n      readOnly: true\r\n      mountPath: /etc/ssl/certs\r\n    - name: kubeconfig\r\n      readOnly: true\r\n      mountPath: /etc/kubernetes/kubeconfig.yaml\r\n    - name: modules\r\n      readOnly: true\r\n      mountPath: /lib/modules\r\n    securityContext:\r\n      privileged: true\r\n  hostNetwork: true\r\n```\r\nWe've tried various combinations of options (no hostname-override, cluster-cidr, etc) without any luck. The iptables configuration is the same, without the proxy-mode flag, and similar combinations of hostname-override and cluster-cidr were explored.\r\n\r\n```\r\nW0723 17:14:34.766268       1 server.go:195] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.\r\nI0723 17:14:34.769562       1 feature_gate.go:230] feature gates: &{map[]}\r\nI0723 17:14:34.828036       1 iptables.go:603] couldn't get iptables-restore version; assuming it doesn't support --wait\r\nI0723 17:14:34.835212       1 iptables.go:200] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory\r\nI0723 17:14:34.940248       1 server_others.go:183] Using ipvs Proxier.\r\nI0723 17:14:34.998197       1 proxier.go:346] nodeIP: 10.0.2.61, isIPv6: false\r\nW0723 17:14:34.998321       1 proxier.go:355] IPVS scheduler not specified, use rr by default\r\nI0723 17:14:34.998379       1 proxier.go:403] minSyncPeriod: 0s, syncPeriod: 30s, burstSyncs: 2\r\nI0723 17:14:34.998475       1 server_others.go:210] Tearing down inactive rules.\r\nI0723 17:14:34.998517       1 iptables.go:425] running iptables -C [OUTPUT -t nat -m comment --comment handle ClusterIPs; NOTE: this must be before the NodePort rules -j KUBE-PORTALS-HOST]\r\nI0723 17:14:35.003635       1 iptables.go:425] running iptables -C [PREROUTING -t nat -m comment --comment handle ClusterIPs; NOTE: this must be before the NodePort rules -j KUBE-PORTALS-CONTAINER]\r\nI0723 17:14:35.004737       1 iptables.go:425] running iptables -C [OUTPUT -t nat -m addrtype --dst-type LOCAL -m comment --comment handle service NodePorts; NOTE: this must be the last rule in the chain -j KUBE-NODEPORT-HOST]\r\nI0723 17:14:35.006570       1 iptables.go:425] running iptables -C [PREROUTING -t nat -m addrtype --dst-type LOCAL -m comment --comment handle service NodePorts; NOTE: this must be the last rule in the chain -j KUBE-NODEPORT-CONTAINER]\r\nI0723 17:14:35.007650       1 iptables.go:425] running iptables -C [INPUT -t filter -m comment --comment Ensure that non-local NodePort traffic can flow -j KUBE-NODEPORT-NON-LOCAL]\r\nI0723 17:14:35.008645       1 iptables.go:425] running iptables -F [KUBE-PORTALS-CONTAINER -t nat]\r\nI0723 17:14:35.009554       1 iptables.go:425] running iptables -F [KUBE-PORTALS-HOST -t nat]\r\nI0723 17:14:35.010430       1 iptables.go:425] running iptables -F [KUBE-NODEPORT-HOST -t nat]\r\nI0723 17:14:35.011321       1 iptables.go:425] running iptables -F [KUBE-NODEPORT-CONTAINER -t nat]\r\nI0723 17:14:35.012224       1 iptables.go:425] running iptables -F [KUBE-NODEPORT-NON-LOCAL -t filter]\r\nI0723 17:14:35.013129       1 iptables.go:425] running iptables -C [INPUT -t filter -m conntrack --ctstate NEW -m comment --comment kubernetes externally-visible service portals -j KUBE-EXTERNAL-SERVICES]\r\nI0723 17:14:35.015808       1 iptables.go:425] running iptables -C [OUTPUT -t filter -m conntrack --ctstate NEW -m comment --comment kubernetes service portals -j KUBE-SERVICES]\r\nI0723 17:14:35.016898       1 iptables.go:425] running iptables -C [OUTPUT -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]\r\nI0723 17:14:35.017887       1 iptables.go:425] running iptables -C [PREROUTING -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]\r\nI0723 17:14:35.018896       1 iptables.go:425] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes postrouting rules -j KUBE-POSTROUTING]\r\nI0723 17:14:35.020534       1 iptables.go:425] running iptables -D [POSTROUTING -t nat -m comment --comment kubernetes postrouting rules -j KUBE-POSTROUTING]\r\nI0723 17:14:35.021587       1 iptables.go:425] running iptables -C [FORWARD -t filter -m comment --comment kubernetes forwarding rules -j KUBE-FORWARD]\r\nI0723 17:14:35.022659       1 iptables.go:425] running iptables -C [INPUT -t filter -m comment --comment kubernetes service portals -j KUBE-SERVICES]\r\nI0723 17:14:35.023612       1 iptables.go:425] running iptables -C [OUTPUT -t filter -m comment --comment kubernetes service portals -j KUBE-SERVICES]\r\nI0723 17:14:35.024563       1 iptables.go:323] running iptables-save [-t nat]\r\nI0723 17:14:35.028040       1 iptables.go:383] running iptables-restore [-T nat --noflush --counters]\r\nI0723 17:14:35.047635       1 iptables.go:323] running iptables-save [-t filter]\r\nI0723 17:14:35.048939       1 iptables.go:383] running iptables-restore [-T filter --noflush --counters]\r\nI0723 17:14:35.049954       1 server.go:448] Version: v1.11.1\r\nI0723 17:14:35.050009       1 oom_linux.go:65] attempting to set \"/proc/self/oom_score_adj\" to \"-999\"\r\nI0723 17:14:35.057410       1 server.go:474] Running in resource-only container \"/kube-proxy\"\r\nI0723 17:14:35.062137       1 healthcheck.go:309] Starting goroutine for healthz on 0.0.0.0:10256\r\nI0723 17:14:35.062430       1 healthz.go:79] No default health checks specified. Installing the ping handler.\r\nI0723 17:14:35.062464       1 healthz.go:83] Installing healthz checkers:\"ping\"\r\nI0723 17:14:35.063144       1 server.go:595] getConntrackMax: using conntrack-min\r\nI0723 17:14:35.063195       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r\nI0723 17:14:35.063292       1 conntrack.go:52] Setting nf_conntrack_max to 131072\r\nI0723 17:14:35.064525       1 mount_linux.go:199] Detected OS without systemd\r\nI0723 17:14:35.064829       1 conntrack.go:83] Setting conntrack hashsize to 32768\r\nI0723 17:14:35.065051       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r\nI0723 17:14:35.065124       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r\nI0723 17:14:35.067382       1 bounded_frequency_runner.go:170] sync-runner Loop running\r\nI0723 17:14:35.077587       1 config.go:202] Starting service config controller\r\nI0723 17:14:35.077637       1 controller_utils.go:1025] Waiting for caches to sync for service config controller\r\nI0723 17:14:35.077682       1 config.go:102] Starting endpoints config controller\r\nI0723 17:14:35.077736       1 controller_utils.go:1025] Waiting for caches to sync for endpoints config controller\r\nI0723 17:14:35.080115       1 reflector.go:202] Starting reflector *core.Endpoints (15m0s) from k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129\r\nI0723 17:14:35.080177       1 reflector.go:240] Listing and watching *core.Endpoints from k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129\r\nI0723 17:14:35.085991       1 reflector.go:202] Starting reflector *core.Service (15m0s) from k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129\r\nI0723 17:14:35.086046       1 reflector.go:240] Listing and watching *core.Service from k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129\r\n```\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\nClient Version: version.Info{Major:\"1\", Minor:\"11\", GitVersion:\"v1.11.0\", GitCommit:\"91e7b4fd31fcd3d5f436da26c980becec37ceefe\", GitTreeState:\"clean\", BuildDate:\"2018-06-27T20:17:28Z\", GoVersion:\"go1.10.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"11\", GitVersion:\"v1.11.1\", GitCommit:\"b1b29978270dc22fecc592ac55d903350454310a\", GitTreeState:\"clean\", BuildDate:\"2018-07-17T18:43:26Z\", GoVersion:\"go1.10.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n\r\n- Cloud provider or hardware configuration: AWS\r\n- OS (e.g. from /etc/os-release): \r\nNAME=\"Container Linux by CoreOS\"\r\nID=coreos\r\nVERSION=1745.7.0\r\nVERSION_ID=1745.7.0\r\nBUILD_ID=2018-06-14-0909\r\nPRETTY_NAME=\"Container Linux by CoreOS 1745.7.0 (Rhyolite)\"\r\nANSI_COLOR=\"38;5;75\"\r\nHOME_URL=\"https://coreos.com/\"\r\nBUG_REPORT_URL=\"https://issues.coreos.com\"\r\nCOREOS_BOARD=\"amd64-usr\"\r\n- Kernel (e.g. `uname -a`):\r\nLinux ip-10-0-1-17 4.14.48-coreos-r2 #1 SMP Thu Jun 14 08:23:03 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux\r\n- Install tools: Scripts\r\n- Others:\r\niptables: v1.4.21\r\nflannel: quay.io/coreos/flannel:v0.10.0\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/407143280","html_url":"https://github.com/kubernetes/kubernetes/issues/66509#issuecomment-407143280","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66509","id":407143280,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNzE0MzI4MA==","user":{"login":"k8s-ci-robot","id":20407524,"node_id":"MDQ6VXNlcjIwNDA3NTI0","avatar_url":"https://avatars0.githubusercontent.com/u/20407524?v=4","gravatar_id":"","url":"https://api.github.com/users/k8s-ci-robot","html_url":"https://github.com/k8s-ci-robot","followers_url":"https://api.github.com/users/k8s-ci-robot/followers","following_url":"https://api.github.com/users/k8s-ci-robot/following{/other_user}","gists_url":"https://api.github.com/users/k8s-ci-robot/gists{/gist_id}","starred_url":"https://api.github.com/users/k8s-ci-robot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k8s-ci-robot/subscriptions","organizations_url":"https://api.github.com/users/k8s-ci-robot/orgs","repos_url":"https://api.github.com/users/k8s-ci-robot/repos","events_url":"https://api.github.com/users/k8s-ci-robot/events{/privacy}","received_events_url":"https://api.github.com/users/k8s-ci-robot/received_events","type":"User","site_admin":false},"created_at":"2018-07-23T17:47:58Z","updated_at":"2018-07-23T17:47:58Z","author_association":"COLLABORATOR","body":"@matt-duch: There are no sig labels on this issue. Please add a sig label.\n\n<details>\n\nA sig label can be added by either:\n\n1. mentioning a sig: `@kubernetes/sig-<group-name>-<group-suffix>`\n    e.g., `@kubernetes/sig-contributor-experience-<group-suffix>` to notify the contributor experience sig, OR\n\n2. specifying the label manually: `/sig <group-name>`\n    e.g., `/sig scalability` to apply the `sig/scalability` label\n\nNote: Method 1 will trigger an email to the group. See the [group list](https://git.k8s.io/community/sig-list.md).\nThe `<group-suffix>` in method 1 has to be replaced with one of these: _**bugs, feature-requests, pr-reviews, test-failures, proposals**_\n\nInstructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.\n</details>"}}
{"actor":{"display_login":"k8s-ci-robot"}}