{"action":"opened","issue":{"url":"https://api.github.com/repos/coreos/flannel/issues/959","repository_url":"https://api.github.com/repos/coreos/flannel","labels_url":"https://api.github.com/repos/coreos/flannel/issues/959/labels{/name}","comments_url":"https://api.github.com/repos/coreos/flannel/issues/959/comments","events_url":"https://api.github.com/repos/coreos/flannel/issues/959/events","html_url":"https://github.com/coreos/flannel/issues/959","id":303269241,"number":959,"title":"vxlan_network.go:158] failed to add vxlanRoute (10.244.0.0/24 -> 10.244.0.0): invalid argument","user":{"login":"slecrenski","id":1291050,"avatar_url":"https://avatars0.githubusercontent.com/u/1291050?v=4","gravatar_id":"","url":"https://api.github.com/users/slecrenski","html_url":"https://github.com/slecrenski","followers_url":"https://api.github.com/users/slecrenski/followers","following_url":"https://api.github.com/users/slecrenski/following{/other_user}","gists_url":"https://api.github.com/users/slecrenski/gists{/gist_id}","starred_url":"https://api.github.com/users/slecrenski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slecrenski/subscriptions","organizations_url":"https://api.github.com/users/slecrenski/orgs","repos_url":"https://api.github.com/users/slecrenski/repos","events_url":"https://api.github.com/users/slecrenski/events{/privacy}","received_events_url":"https://api.github.com/users/slecrenski/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-07T21:38:46Z","updated_at":"2018-03-07T21:38:46Z","closed_at":null,"author_association":"NONE","body":"Docker: 1.12.6\r\nRHEL: 7.3\r\nLinux k8s-master 3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\nKubernetes 1.9.3\r\nquay.io/calico/node:v2.6.2\r\nquay.io/calico/cni:v1.11.0\r\nquay.io/coreos/flannel:v0.9.1\r\n\r\nAzure Cloud with vnet address space: 10.244.0.0/16\r\n\r\n```json\r\nnet-conf.json: |\r\n    {\r\n      \"Network\": \"10.244.0.0/16\",\r\n      \"Backend\": {\r\n        \"Type\": \"vxlan\"\r\n      }\r\n    }\r\n```\r\n\r\nCluster was initialized with \r\n```bash\r\nkubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=cri\r\n#applied canal networking which runs flannel\r\nkubectl scale deployment kube-dns -n kube-system --replicas=2\r\n#this attempted to launch kube-dns on the agent node.\r\n#the kube-dns container on the agent node never runs because it is unable to communicate to the master node to determine the dns configuration.\r\n```\r\n```bash\r\n[root@k8s-master v2]# kubectl logs kube-dns-6f4fd4bdf-dsjtq -n kube-system -c kubedns\r\nI0307 21:34:17.804073       1 dns.go:48] version: 1.14.6-3-gc36cb11\r\nI0307 21:34:17.805197       1 server.go:69] Using configuration read from directory: /kube-dns-config with period 10s\r\nI0307 21:34:17.805254       1 server.go:112] FLAG: --alsologtostderr=\"false\"\r\nI0307 21:34:17.805264       1 server.go:112] FLAG: --config-dir=\"/kube-dns-config\"\r\nI0307 21:34:17.805271       1 server.go:112] FLAG: --config-map=\"\"\r\nI0307 21:34:17.805277       1 server.go:112] FLAG: --config-map-namespace=\"kube-system\"\r\nI0307 21:34:17.805283       1 server.go:112] FLAG: --config-period=\"10s\"\r\nI0307 21:34:17.805290       1 server.go:112] FLAG: --dns-bind-address=\"0.0.0.0\"\r\nI0307 21:34:17.805296       1 server.go:112] FLAG: --dns-port=\"10053\"\r\nI0307 21:34:17.805303       1 server.go:112] FLAG: --domain=\"cluster.local.\"\r\nI0307 21:34:17.805311       1 server.go:112] FLAG: --federations=\"\"\r\nI0307 21:34:17.805318       1 server.go:112] FLAG: --healthz-port=\"8081\"\r\nI0307 21:34:17.805324       1 server.go:112] FLAG: --initial-sync-timeout=\"1m0s\"\r\nI0307 21:34:17.805330       1 server.go:112] FLAG: --kube-master-url=\"\"\r\nI0307 21:34:17.805336       1 server.go:112] FLAG: --kubecfg-file=\"\"\r\nI0307 21:34:17.805342       1 server.go:112] FLAG: --log-backtrace-at=\":0\"\r\nI0307 21:34:17.805350       1 server.go:112] FLAG: --log-dir=\"\"\r\nI0307 21:34:17.805356       1 server.go:112] FLAG: --log-flush-frequency=\"5s\"\r\nI0307 21:34:17.805362       1 server.go:112] FLAG: --logtostderr=\"true\"\r\nI0307 21:34:17.805368       1 server.go:112] FLAG: --nameservers=\"\"\r\nI0307 21:34:17.805374       1 server.go:112] FLAG: --stderrthreshold=\"2\"\r\nI0307 21:34:17.805390       1 server.go:112] FLAG: --v=\"2\"\r\nI0307 21:34:17.805396       1 server.go:112] FLAG: --version=\"false\"\r\nI0307 21:34:17.805415       1 server.go:112] FLAG: --vmodule=\"\"\r\nI0307 21:34:17.805466       1 server.go:194] Starting SkyDNS server (0.0.0.0:10053)\r\nI0307 21:34:17.805656       1 server.go:213] Skydns metrics enabled (/metrics:10055)\r\nI0307 21:34:17.805677       1 dns.go:146] Starting endpointsController\r\nI0307 21:34:17.805683       1 dns.go:149] Starting serviceController\r\nI0307 21:34:17.805805       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]\r\nI0307 21:34:17.805826       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]\r\nI0307 21:34:18.306107       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:18.806137       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:19.305925       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:19.805901       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:20.305909       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:20.805936       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:21.305954       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:21.805893       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:22.305926       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:22.806025       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:23.305962       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:23.805877       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:24.305931       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:24.805906       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:25.305905       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:25.806023       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:26.305906       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:26.806023       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:27.305930       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:27.805968       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:28.305904       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:28.805886       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:29.305877       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:29.805878       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:30.305896       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:30.805966       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:31.305877       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:31.805931       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:32.305950       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:32.805986       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:33.305935       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:33.805899       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:34.305962       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:34.806082       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:35.305918       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:35.805870       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:36.305998       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:36.805920       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:37.305936       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:37.805872       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:38.306083       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:38.806080       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:39.305912       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:39.805896       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:40.306024       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:40.805991       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:41.305912       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:41.805891       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:42.305905       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:42.805873       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:43.305893       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:43.805927       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:44.305913       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:44.806054       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:45.306072       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:45.805924       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:46.305902       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:46.805889       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:47.305910       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nI0307 21:34:47.806071       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...\r\nE0307 21:34:47.806907       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout\r\nE0307 21:34:47.807363       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout\r\n\r\n```\r\n\r\nCluster is running in Azure with the same virtual network as the pod cidr.\r\n\r\n```bash\r\n[root@k8s-master v2]# kubectl get pods -n kube-system -o wide\r\nNAME                                 READY     STATUS             RESTARTS   AGE       IP             NODE\r\ncanal-9sfh5                          3/3       Running            0          1h        10.244.0.4     k8s-agent1\r\ncanal-jmgzn                          3/3       Running            0          1h        10.244.0.100   k8s-master\r\netcd-k8s-master                      1/1       Running            0          2h        10.244.0.100   k8s-master\r\nkube-apiserver-k8s-master            1/1       Running            0          2h        10.244.0.100   k8s-master\r\nkube-controller-manager-k8s-master   1/1       Running            0          2h        10.244.0.100   k8s-master\r\nkube-dns-6f4fd4bdf-ch98b             3/3       Running            0          2h        10.244.0.24    k8s-master\r\nkube-dns-6f4fd4bdf-dsjtq             1/3       CrashLoopBackOff   48         1h        10.244.3.2     k8s-agent1\r\nkube-proxy-x6j8p                     1/1       Running            0          2h        10.244.0.100   k8s-master\r\nkube-proxy-z5bbd                     1/1       Running            0          1h        10.244.0.4     k8s-agent1\r\nkube-scheduler-k8s-master            1/1       Running            0          2h        10.244.0.100   k8s-master\r\n```\r\n```bash\r\n[root@k8s-master v2]# kubectl describe node k8s-master\r\nName:               k8s-master\r\nRoles:              master\r\nLabels:             beta.kubernetes.io/arch=amd64\r\n                    beta.kubernetes.io/instance-type=Standard_DS2_v2\r\n                    beta.kubernetes.io/os=linux\r\n                    failure-domain.beta.kubernetes.io/region=usgovvirginia\r\n                    failure-domain.beta.kubernetes.io/zone=1\r\n                    kubernetes.io/hostname=k8s-master\r\n                    node-role.kubernetes.io/master=\r\nAnnotations:        flannel.alpha.coreos.com/backend-data={\"VtepMAC\":\"92:b2:1f:03:ff:99\"}\r\n                    flannel.alpha.coreos.com/backend-type=vxlan\r\n                    flannel.alpha.coreos.com/kube-subnet-manager=true\r\n                    flannel.alpha.coreos.com/public-ip=10.244.0.100\r\n                    node.alpha.kubernetes.io/ttl=0\r\n                    volumes.kubernetes.io/controller-managed-attach-detach=true\r\nTaints:             node-role.kubernetes.io/master:NoSchedule\r\nCreationTimestamp:  Wed, 07 Mar 2018 19:25:44 +0000\r\nConditions:\r\n  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message\r\n  ----             ------  -----------------                 ------------------                ------                       -------\r\n  OutOfDisk        False   Wed, 07 Mar 2018 21:31:45 +0000   Wed, 07 Mar 2018 19:25:39 +0000   KubeletHasSufficientDisk     kubelet has sufficient disk space available\r\n  MemoryPressure   False   Wed, 07 Mar 2018 21:31:45 +0000   Wed, 07 Mar 2018 19:25:39 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available\r\n  DiskPressure     False   Wed, 07 Mar 2018 21:31:45 +0000   Wed, 07 Mar 2018 19:25:39 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure\r\n  Ready            True    Wed, 07 Mar 2018 21:31:45 +0000   Wed, 07 Mar 2018 19:26:55 +0000   KubeletReady                 kubelet is posting ready status\r\nAddresses:\r\n  InternalIP:  10.244.0.100\r\n  Hostname:    k8s-master\r\nCapacity:\r\n cpu:     2\r\n memory:  7125792Ki\r\n pods:    110\r\nAllocatable:\r\n cpu:     2\r\n memory:  7023392Ki\r\n pods:    110\r\nSystem Info:\r\n Machine ID:                 aa4f0681ccb6435784669b356fa73d9c\r\n System UUID:                2E21AA4F-77BB-F640-990D-12267E1262C0\r\n Boot ID:                    12d0a064-2b59-411d-8d64-d9c2a61472f0\r\n Kernel Version:             3.10.0-693.21.1.el7.x86_64\r\n OS Image:                   Red Hat Enterprise Linux Server 7.4 (Maipo)\r\n Operating System:           linux\r\n Architecture:               amd64\r\n Container Runtime Version:  docker://1.12.6\r\n Kubelet Version:            v1.9.3\r\n Kube-Proxy Version:         v1.9.3\r\nPodCIDR:                     10.244.0.0/24\r\nExternalID:                  /subscriptions/28865b6d-f25c-4bba-a4f1-a16bfa782571/resourceGroups/kubernetes/providers/Microsoft.Compute/virtualMachines/k8s-master\r\nNon-terminated Pods:         (7 in total)\r\n  Namespace                  Name                                  CPU Requests  CPU Limits  Memory Requests  Memory Limits\r\n  ---------                  ----                                  ------------  ----------  ---------------  -------------\r\n  kube-system                canal-jmgzn                           250m (12%)    0 (0%)      0 (0%)           0 (0%)\r\n  kube-system                etcd-k8s-master                       0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  kube-system                kube-apiserver-k8s-master             250m (12%)    0 (0%)      0 (0%)           0 (0%)\r\n  kube-system                kube-controller-manager-k8s-master    200m (10%)    0 (0%)      0 (0%)           0 (0%)\r\n  kube-system                kube-dns-6f4fd4bdf-ch98b              260m (13%)    0 (0%)      110Mi (1%)       170Mi (2%)\r\n  kube-system                kube-proxy-x6j8p                      0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  kube-system                kube-scheduler-k8s-master             100m (5%)     0 (0%)      0 (0%)           0 (0%)\r\nAllocated resources:\r\n  (Total limits may be over 100 percent, i.e., overcommitted.)\r\n  CPU Requests  CPU Limits  Memory Requests  Memory Limits\r\n  ------------  ----------  ---------------  -------------\r\n  1060m (53%)   0 (0%)      110Mi (1%)       170Mi (2%)\r\nEvents:         <none>\r\n```\r\n\r\n```bash\r\n[root@k8s-master v2]# kubectl describe node k8s-agent1\r\nName:               k8s-agent1\r\nRoles:              <none>\r\nLabels:             beta.kubernetes.io/arch=amd64\r\n                    beta.kubernetes.io/instance-type=Standard_DS2_v2\r\n                    beta.kubernetes.io/os=linux\r\n                    failure-domain.beta.kubernetes.io/region=usgovvirginia\r\n                    failure-domain.beta.kubernetes.io/zone=0\r\n                    kubernetes.io/hostname=k8s-agent1\r\nAnnotations:        flannel.alpha.coreos.com/backend-data={\"VtepMAC\":\"8a:50:80:d4:48:ec\"}\r\n                    flannel.alpha.coreos.com/backend-type=vxlan\r\n                    flannel.alpha.coreos.com/kube-subnet-manager=true\r\n                    flannel.alpha.coreos.com/public-ip=10.244.0.4\r\n                    node.alpha.kubernetes.io/ttl=0\r\n                    volumes.kubernetes.io/controller-managed-attach-detach=true\r\nTaints:             <none>\r\nCreationTimestamp:  Wed, 07 Mar 2018 19:37:25 +0000\r\nConditions:\r\n  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message\r\n  ----             ------  -----------------                 ------------------                ------                       -------\r\n  OutOfDisk        False   Wed, 07 Mar 2018 21:30:51 +0000   Wed, 07 Mar 2018 19:37:25 +0000   KubeletHasSufficientDisk     kubelet has sufficient disk space available\r\n  MemoryPressure   False   Wed, 07 Mar 2018 21:30:51 +0000   Wed, 07 Mar 2018 19:37:25 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available\r\n  DiskPressure     False   Wed, 07 Mar 2018 21:30:51 +0000   Wed, 07 Mar 2018 19:37:25 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure\r\n  Ready            True    Wed, 07 Mar 2018 21:30:51 +0000   Wed, 07 Mar 2018 19:40:54 +0000   KubeletReady                 kubelet is posting ready status\r\nAddresses:\r\n  InternalIP:  10.244.0.4\r\n  Hostname:    k8s-agent1\r\nCapacity:\r\n cpu:     2\r\n memory:  7125792Ki\r\n pods:    110\r\nAllocatable:\r\n cpu:     2\r\n memory:  7023392Ki\r\n pods:    110\r\nSystem Info:\r\n Machine ID:                 aa4f0681ccb6435784669b356fa73d9c\r\n System UUID:                BEF56729-F758-5345-BBA8-536DF72C8981\r\n Boot ID:                    f0e31820-375e-40a0-81d4-f45720bc8222\r\n Kernel Version:             3.10.0-693.21.1.el7.x86_64\r\n OS Image:                   Red Hat Enterprise Linux Server 7.4 (Maipo)\r\n Operating System:           linux\r\n Architecture:               amd64\r\n Container Runtime Version:  docker://1.12.6\r\n Kubelet Version:            v1.9.3\r\n Kube-Proxy Version:         v1.9.3\r\nPodCIDR:                     10.244.3.0/24\r\nExternalID:                  /subscriptions/28865b6d-f25c-4bba-a4f1-a16bfa782571/resourceGroups/kubernetes/providers/Microsoft.Compute/virtualMachines/k8s-agent1\r\nNon-terminated Pods:         (3 in total)\r\n  Namespace                  Name                        CPU Requests  CPU Limits  Memory Requests  Memory Limits\r\n  ---------                  ----                        ------------  ----------  ---------------  -------------\r\n  kube-system                canal-9sfh5                 250m (12%)    0 (0%)      0 (0%)           0 (0%)\r\n  kube-system                kube-dns-6f4fd4bdf-dsjtq    260m (13%)    0 (0%)      110Mi (1%)       170Mi (2%)\r\n  kube-system                kube-proxy-z5bbd            0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\nAllocated resources:\r\n  (Total limits may be over 100 percent, i.e., overcommitted.)\r\n  CPU Requests  CPU Limits  Memory Requests  Memory Limits\r\n  ------------  ----------  ---------------  -------------\r\n  510m (25%)    0 (0%)      110Mi (1%)       170Mi (2%)\r\nEvents:         <none>\r\n```\r\n\r\nI have a very basic configuration.  1 Master Node and 1 Agent Node.  DNS queries are not working on the agent node.  the kube-dns is running on the master node.  Master Node IP 10.244.0.100 and Agent Node IP: 10.244.0.4.\r\n\r\nI am trying to figure out why it is that I cannot communicate with 10.96.0.10 (kube-dns) which is supposed to be routed to the master node (where kube-dns is running).\r\n\r\nI've been looking at log files and enabling level 10 verbosity for the past several hours.  What does this error message mean?\r\n\r\nvxlan_network.go:158] failed to add vxlanRoute (10.244.0.0/24 -> 10.244.0.0): invalid argument\r\n\r\nI am unable to get pods that require kube-dns to run. They just fail with a dns error trying to perform a lookup to kubernetes.default.svc.cluster.local. If i try to scale kube-dns to launch on the non-master node the kube-dns fails to start on that node due to an issue with dns lookup.\r\n\r\nI am unable to get pod-to-kube-dns and node->kube-dns communication working. How can I debug what the issue is?\r\n\r\nThese are RHEL 7.3 nodes with:\r\n```bash\r\nswapoff -a\r\nsetenforce 0\r\nip forwarding enabled\r\nip forwarding is enabled at the nic level in azure as well for both virtual machines.\r\nsystemctl stop firewalld\r\nsystemctl disable firewalld\r\n```\r\n\r\n```bash\r\nkubectl get nodes -o yaml |grep flannel.alpha\r\nflannel.alpha.coreos.com/backend-data: '{\"VtepMAC\":\"4e:6d:96:f9:df:0b\"}'\r\nflannel.alpha.coreos.com/backend-type: vxlan\r\nflannel.alpha.coreos.com/kube-subnet-manager: \"true\"\r\nflannel.alpha.coreos.com/public-ip: 10.244.0.4\r\nflannel.alpha.coreos.com/backend-data: '{\"VtepMAC\":\"22:88:e0:e0:ee:9c\"}'\r\nflannel.alpha.coreos.com/backend-type: vxlan\r\nflannel.alpha.coreos.com/kube-subnet-manager: \"true\"\r\nflannel.alpha.coreos.com/public-ip: 10.244.0.100\r\n```\r\nMaster Node:\r\n```bash\r\nnslookup kubernetes.default.svc.cluster.local 10.96.0.10\r\nServer:\t10.96.0.10\r\nAddress:\t10.96.0.10#53\r\n\r\nName:\tkubernetes.default.svc.cluster.local\r\nAddress: 10.96.0.1\r\n```\r\nSlave Node:\r\n```bash\r\nnslookup kubernetes.default.svc.cluster.local 10.96.0.10\r\n;; connection timed out; trying next origin\r\n;; connection timed out; no servers could be reached\r\n```\r\n\r\nInterestingly enough I can do things like this\r\n```bash\r\n[root@k8s-agent1 ~]# wget https://10.96.0.1:443/api/v1/nodes?resourceVersion=0\r\n--2018-03-07 21:22:34--  https://10.96.0.1/api/v1/nodes?resourceVersion=0\r\nConnecting to 10.96.0.1:443... connected.\r\nERROR: cannot verify 10.96.0.1's certificate, issued by ‘/CN=kubernetes’:\r\n  Unable to locally verify the issuer's authority.\r\nTo connect to 10.96.0.1 insecurely, use `--no-check-certificate'.\r\n```\r\nSlave Node Iptables:\r\n```bash\r\niptables-save | grep kube-dns\r\n-A KUBE-SEP-LGXZUSYJZFXP55VS -s 10.244.0.20/32 -m comment --comment \"kube-system/kube-dns:dns-tcp\" -j KUBE-MARK-MASQ\r\n-A KUBE-SEP-LGXZUSYJZFXP55VS -p tcp -m comment --comment \"kube-system/kube-dns:dns-tcp\" -m tcp -j DNAT --to-destination 10.244.0.20:53\r\n-A KUBE-SEP-WHU5MQLF6I7CQ4PO -s 10.244.0.20/32 -m comment --comment \"kube-system/kube-dns:dns\" -j KUBE-MARK-MASQ\r\n-A KUBE-SEP-WHU5MQLF6I7CQ4PO -p udp -m comment --comment \"kube-system/kube-dns:dns\" -m udp -j DNAT --to-destination 10.244.0.20:53\r\n-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment \"kube-system/kube-dns:dns cluster IP\" -m udp --dport 53 -j KUBE-MARK-MASQ\r\n-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment \"kube-system/kube-dns:dns cluster IP\" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU\r\n-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment \"kube-system/kube-dns:dns-tcp cluster IP\" -m tcp --dport 53 -j KUBE-MARK-MASQ\r\n-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment \"kube-system/kube-dns:dns-tcp cluster IP\" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4\r\n-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment \"kube-system/kube-dns:dns-tcp\" -j KUBE-SEP-LGXZUSYJZFXP55VS\r\n-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment \"kube-system/kube-dns:dns\" -j KUBE-SEP-WHU5MQLF6I7CQ4PO\r\n```\r\nAnyway to use tcpdump to figure out this issue?\r\n\r\nWhat does this error mean? vxlan_network.go:158] failed to add vxlanRoute (10.244.0.0/24 -> 10.244.0.0): invalid argument\r\n\r\nMaster is running at 10.244.0.100 and agent node is running at 10.244.0.4.\r\n\r\n--master\r\n```bash\r\nip route\r\ndefault via 10.244.0.1 dev eth0 proto static metric 100\r\n10.244.0.0/16 dev eth0 proto kernel scope link src 10.244.0.100 metric 100\r\n10.244.0.20 dev cali48fa6642c60 scope link\r\n10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink\r\n168.63.129.16 via 10.244.0.1 dev eth0 proto dhcp metric 100\r\n169.254.169.254 via 10.244.0.1 dev eth0 proto dhcp metric 100\r\n172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1\r\n```\r\n--agent\r\n```bash\r\ndefault via 10.244.0.1 dev eth0 proto static metric 100\r\n10.244.0.0/16 dev eth0 proto kernel scope link src 10.244.0.4 metric 100\r\n10.244.1.20 dev cali66d03ab9413 scope link\r\n168.63.129.16 via 10.244.0.1 dev eth0 proto dhcp metric 100\r\n169.254.169.254 via 10.244.0.1 dev eth0 proto dhcp metric 100\r\n172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1\r\n```\r\n\r\n```bash\r\nkubectl logs canal-6dw5b -n kube-system -c kube-flannel\r\nI0307 17:17:25.701615 1 main.go:474] Determining IP address of default interface\r\nI0307 17:17:25.702468 1 main.go:487] Using interface with name eth0 and address 10.244.0.4\r\nI0307 17:17:25.702485 1 main.go:504] Defaulting external address to interface address (10.244.0.4)\r\nI0307 17:17:25.716693 1 kube.go:130] Waiting 10m0s for node controller to sync\r\nI0307 17:17:25.716731 1 kube.go:283] Starting kube subnet manager\r\nI0307 17:17:26.716969 1 kube.go:137] Node controller sync successful\r\nI0307 17:17:26.716990 1 main.go:234] Created subnet manager: Kubernetes Subnet Manager - k8s-agent1\r\nI0307 17:17:26.716997 1 main.go:237] Installing signal handlers\r\nI0307 17:17:26.717072 1 main.go:352] Found network config - Backend type: vxlan\r\nI0307 17:17:26.717123 1 vxlan.go:119] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false\r\nI0307 17:17:26.741083 1 main.go:299] Wrote subnet file to /run/flannel/subnet.env\r\nI0307 17:17:26.741102 1 main.go:303] Running backend.\r\nI0307 17:17:26.741112 1 main.go:321] Waiting for all goroutines to exit\r\nI0307 17:17:26.741126 1 vxlan_network.go:56] watching for new subnet leases\r\nE0307 17:17:26.742469 1 vxlan_network.go:158] failed to add vxlanRoute (10.244.0.0/24 -> 10.244.0.0): invalid argument\r\nI0307 17:17:26.753958 1 iptables.go:114] Some iptables rules are missing; deleting and recreating rules\r\nI0307 17:17:26.753975 1 iptables.go:136] Deleting iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN\r\nI0307 17:17:26.754326 1 iptables.go:114] Some iptables rules are missing; deleting and recreating rules\r\nI0307 17:17:26.754340 1 iptables.go:136] Deleting iptables rule: -s 10.244.0.0/16 -j ACCEPT\r\nI0307 17:17:26.755962 1 iptables.go:136] Deleting iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE\r\nI0307 17:17:26.756540 1 iptables.go:136] Deleting iptables rule: -d 10.244.0.0/16 -j ACCEPT\r\nI0307 17:17:26.758839 1 iptables.go:124] Adding iptables rule: -s 10.244.0.0/16 -j ACCEPT\r\nI0307 17:17:26.759144 1 iptables.go:136] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.1.0/24 -j RETURN\r\nI0307 17:17:26.762439 1 iptables.go:136] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE\r\nI0307 17:17:26.762971 1 iptables.go:124] Adding iptables rule: -d 10.244.0.0/16 -j ACCEPT\r\nI0307 17:17:26.765618 1 iptables.go:124] Adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN\r\nI0307 17:17:26.771098 1 iptables.go:124] Adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE\r\nI0307 17:17:26.774826 1 iptables.go:124] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.1.0/24 -j RETURN\r\nI0307 17:17:26.778155 1 iptables.go:124] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE\r\n\r\n```"}}
{"actor":{"display_login":"slecrenski"}}