{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66103","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66103/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66103/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66103/events","html_url":"https://github.com/kubernetes/kubernetes/issues/66103","id":340505793,"node_id":"MDU6SXNzdWUzNDA1MDU3OTM=","number":66103,"title":"IPVS is not working with hostport ","user":{"login":"liqlin2015","id":10703772,"node_id":"MDQ6VXNlcjEwNzAzNzcy","avatar_url":"https://avatars3.githubusercontent.com/u/10703772?v=4","gravatar_id":"","url":"https://api.github.com/users/liqlin2015","html_url":"https://github.com/liqlin2015","followers_url":"https://api.github.com/users/liqlin2015/followers","following_url":"https://api.github.com/users/liqlin2015/following{/other_user}","gists_url":"https://api.github.com/users/liqlin2015/gists{/gist_id}","starred_url":"https://api.github.com/users/liqlin2015/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liqlin2015/subscriptions","organizations_url":"https://api.github.com/users/liqlin2015/orgs","repos_url":"https://api.github.com/users/liqlin2015/repos","events_url":"https://api.github.com/users/liqlin2015/events{/privacy}","received_events_url":"https://api.github.com/users/liqlin2015/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":116712108,"node_id":"MDU6TGFiZWwxMTY3MTIxMDg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/network","name":"sig/network","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-07-12T06:32:41Z","updated_at":"2018-07-17T06:33:29Z","closed_at":null,"author_association":"NONE","body":"## Is this a BUG REPORT or FEATURE REQUEST?\r\n\r\nChoose one: BUG REPORT\r\n\r\n\r\n## Versions\r\n\r\n**kubeadm version** (use `kubeadm version`): \r\n\r\n**Environment**:\r\n- **Kubernetes version** (use `kubectl version`): 1.11.0\r\n- **Cloud provider or hardware configuration**: None\r\n- **OS** (e.g. from /etc/os-release): Ubuntu 16.04\r\n- **Kernel** (e.g. `uname -a`): \r\n- **Others**:\r\n\r\n\r\n## What happened?\r\n\r\nI'm trying to set up a kubernetes (v1.11.0) cluster. Here is the setting of the cluster:\r\n1. Enable IPVS in kube-proxy\r\n2. Use hostport for nginx ingress controller. Deloy nginx ingress controller on the same node as kubenetes core services.\r\n\r\nHost port 443 and 80 of nginx ingress controller is conflict with kubernetes (kube-apiserver) service's port 443 and 80. Nginx ingress controller keeps crashloopback because failed to connect to kube-apiserver. I even cannot run ` telnet 10.0.0.1 443`.\r\n\r\nTry to find the root cause. Here is the iptable rules generated\r\n```\r\nChain PREROUTING (policy ACCEPT)\r\ntarget     prot opt source               destination\r\ncali-PREROUTING  all  --  anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */\r\nCNI-HOSTPORT-DNAT  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL\r\nKUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */\r\nDOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL\r\n\r\nChain INPUT (policy ACCEPT)\r\ntarget     prot opt source               destination\r\n\r\nChain OUTPUT (policy ACCEPT)\r\ntarget     prot opt source               destination\r\ncali-OUTPUT  all  --  anywhere             anywhere             /* cali:tVnHkvAo15HuiPy0 */\r\nCNI-HOSTPORT-DNAT  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL\r\nKUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */\r\nDOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL\r\n\r\nChain POSTROUTING (policy ACCEPT)\r\ntarget     prot opt source               destination\r\ncali-POSTROUTING  all  --  anywhere             anywhere             /* cali:O3lYWMrLQYEMJtB5 */\r\nCNI-HOSTPORT-SNAT  all  --  localhost           !localhost\r\nKUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */\r\nMASQUERADE  all  --  172.17.0.0/16        anywhere\r\n```\r\n\r\n The `CNI-HOSTPORT` related iptable rules are on top of `KUBE-SERVICES` rule.  As you seen the pre-routing rule\r\n\r\n```\r\nChain PREROUTING (policy ACCEPT)\r\ntarget     prot opt source               destination\r\ncali-PREROUTING  all  --  anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */\r\nCNI-HOSTPORT-DNAT  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL\r\nKUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */\r\nDOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL\r\n```\r\n\r\nEvery time I call `https://10.0.0.1:443` it will match `CNI-HOSTPORT-DNAT` rule to redirect to nginx ingress service. That's why I got failure when calling kubernetes api with service ip.\r\n\r\n## What you expected to happen?\r\nShould always take `KUBE-SERVICES` rule first rather than `CNI-HOSTPORT-DNAT`. So the pre-routing rule should be like\r\n\r\n```\r\nChain PREROUTING (policy ACCEPT)\r\ntarget     prot opt source               destination\r\ncali-PREROUTING  all  --  anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */\r\nKUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */\r\nCNI-HOSTPORT-DNAT  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL\r\nDOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL\r\n```\r\n\r\n## How to reproduce it (as minimally and precisely as possible)?\r\nSee above append.\r\n\r\n## Anything else we need to know?\r\n\r\nWhen I restarted kube-proxy, the iptable rules were generated correctly. Nginx ingress controller now started successfully.\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/405475015","html_url":"https://github.com/kubernetes/kubernetes/issues/66103#issuecomment-405475015","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66103","id":405475015,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTQ3NTAxNQ==","user":{"login":"Lion-Wei","id":28696384,"node_id":"MDQ6VXNlcjI4Njk2Mzg0","avatar_url":"https://avatars1.githubusercontent.com/u/28696384?v=4","gravatar_id":"","url":"https://api.github.com/users/Lion-Wei","html_url":"https://github.com/Lion-Wei","followers_url":"https://api.github.com/users/Lion-Wei/followers","following_url":"https://api.github.com/users/Lion-Wei/following{/other_user}","gists_url":"https://api.github.com/users/Lion-Wei/gists{/gist_id}","starred_url":"https://api.github.com/users/Lion-Wei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lion-Wei/subscriptions","organizations_url":"https://api.github.com/users/Lion-Wei/orgs","repos_url":"https://api.github.com/users/Lion-Wei/repos","events_url":"https://api.github.com/users/Lion-Wei/events{/privacy}","received_events_url":"https://api.github.com/users/Lion-Wei/received_events","type":"User","site_admin":false},"created_at":"2018-07-17T06:33:29Z","updated_at":"2018-07-17T06:33:29Z","author_association":"CONTRIBUTOR","body":"That's right. Since `KUBE-SERVICES` and `CNI-HOSTPORT-DNAT` is generated by different process, we can't guarantee the order, even using `iptables -I`.\r\nOn kube-proxy side, clusterIP addresses have to be treated as local, so ipvs rules can work. On cni plugin side, match `dst-type=local` to do DNT for hostPort pod also very reasonable.\r\nSo, for now, the only way I have to solve this problem is make sure `KUBE-SERVICES` is in the top of `PREROUTING` chain. . . "}}
{"actor":{"display_login":"Lion-Wei"}}