{"action":"created","issue":{"url":"https://api.github.com/repos/openshift/origin/issues/19276","repository_url":"https://api.github.com/repos/openshift/origin","labels_url":"https://api.github.com/repos/openshift/origin/issues/19276/labels{/name}","comments_url":"https://api.github.com/repos/openshift/origin/issues/19276/comments","events_url":"https://api.github.com/repos/openshift/origin/issues/19276/events","html_url":"https://github.com/openshift/origin/issues/19276","id":312579413,"number":19276,"title":"EgressNetworkPolicy rewrite drop traffic for too long","user":{"login":"mjudeikis","id":3225744,"avatar_url":"https://avatars1.githubusercontent.com/u/3225744?v=4","gravatar_id":"","url":"https://api.github.com/users/mjudeikis","html_url":"https://github.com/mjudeikis","followers_url":"https://api.github.com/users/mjudeikis/followers","following_url":"https://api.github.com/users/mjudeikis/following{/other_user}","gists_url":"https://api.github.com/users/mjudeikis/gists{/gist_id}","starred_url":"https://api.github.com/users/mjudeikis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mjudeikis/subscriptions","organizations_url":"https://api.github.com/users/mjudeikis/orgs","repos_url":"https://api.github.com/users/mjudeikis/repos","events_url":"https://api.github.com/users/mjudeikis/events{/privacy}","received_events_url":"https://api.github.com/users/mjudeikis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2018-04-09T15:33:01Z","updated_at":"2018-04-13T12:39:36Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I have multiple EgressNetworkPolicies in my cluster.\r\n\r\nPolicy:\r\n```\r\n{\r\n    \"kind\": \"EgressNetworkPolicy\",\r\n    \"apiVersion\": \"v1\",\r\n    \"metadata\": {\r\n        \"name\": \"default\"\r\n    },\r\n    \"spec\": {\r\n        \"egress\": [\r\n            {\r\n                \"type\": \"Allow\",\r\n                \"to\": {\r\n                    \"cidrSelector\": \"1.2.3.0/24\"\r\n                }\r\n            },\r\n            {\r\n                \"type\": \"Allow\",\r\n                \"to\": {\r\n                    \"dnsName\": \"www.foo.com\"\r\n                }\r\n            },\r\n            {\r\n                \"type\": \"Deny\",\r\n                \"to\": {\r\n                    \"cidrSelector\": \"0.0.0.0/0\"\r\n                }\r\n            }\r\n        ]\r\n    }\r\n}\r\n```\r\n\r\nnamespace:\r\n```\r\n[root@console-REPL ~]# oc describe netnamespace labs-dev                                                 \r\nName:           labs-dev                            \r\nCreated:        About an hour ago                   \r\nLabels:         <none>                              \r\nAnnotations:    <none>                              \r\nName:           labs-dev                            \r\nID:             1674326                             \r\nEgress IPs:     <none>                              \r\n[root@console-REPL ~]# oc describe netnamespace labs-dev \r\n```\r\n\r\nSo I have rules created:\r\n```\r\n[root@node1 ~]# ovs-ofctl -O OpenFlow13 dump-flows br0 | grep 198c56                                                                                                                                               \r\n cookie=0x0, duration=3105.808s, table=60, n_packets=0, n_bytes=0, priority=100,ip,nw_dst=172.30.26.73,nw_frag=later actions=load:0x198c56->NXM_NX_REG1[],load:0x2->NXM_NX_REG2[],goto_table:80                    \r\n cookie=0x0, duration=3105.801s, table=60, n_packets=0, n_bytes=0, priority=100,tcp,nw_dst=172.30.26.73,tp_dst=8080 actions=load:0x198c56->NXM_NX_REG1[],load:0x2->NXM_NX_REG2[],goto_table:80                     \r\n cookie=0x0, duration=3140.219s, table=80, n_packets=0, n_bytes=0, priority=100,reg0=0x198c56,reg1=0x198c56 actions=output:NXM_NX_REG2[]                                                                           \r\n cookie=0x0, duration=253.253s, table=101, n_packets=0, n_bytes=0, priority=3,ip,reg0=0x198c56,nw_dst=1.2.3.0/24 actions=output:2                                                                                  \r\n cookie=0x0, duration=253.247s, table=101, n_packets=0, n_bytes=0, priority=2,ip,reg0=0x198c56,nw_dst=107.23.81.155 actions=output:2                                                                               \r\n cookie=0x0, duration=253.242s, table=101, n_packets=0, n_bytes=0, priority=2,ip,reg0=0x198c56,nw_dst=107.23.205.115 actions=output:2                                                                              \r\n cookie=0x0, duration=253.233s, table=101, n_packets=0, n_bytes=0, priority=1,ip,reg0=0x198c56 actions=drop                                                                                                        \r\n[root@node1 ~]#\r\n```\r\n\r\nAll good and nice. But there is this piece of code:\r\nhttps://github.com/openshift/origin/blob/master/pkg/network/node/ovscontroller.go#L426\r\nwhich runs each ~30 min.\r\n\r\nNow my `www.foo.com` is behind internal dns and its slower than usual.\r\n\r\n*Spolier. From this point I do some assumption without knowing code base*\r\nThis part of code get the long execution and hangs (?)\r\nhttps://github.com/openshift/origin/blob/master/pkg/network/node/ovscontroller.go#L447-L487\r\n\r\nDNS should be taken from the cache, but by any change, it can be blocked by this:\r\nhttps://github.com/openshift/origin/blob/master/pkg/network/node/ovscontroller.go#L490\r\n\r\nLong story short, time between:\r\n`otx.AddFlow(\"table=101, reg0=%d, cookie=1, priority=65535, actions=drop\", vnid)`\r\nand\r\n`otx.DeleteFlows(\"table=101, reg0=%d, cookie=1/1\", vnid)`\r\n in the code causes downtime.\r\n\r\nstill trying to debug onsite with the cluster. will update as I find something. But any comments are welcome.\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/openshift/origin/issues/comments/381122503","html_url":"https://github.com/openshift/origin/issues/19276#issuecomment-381122503","issue_url":"https://api.github.com/repos/openshift/origin/issues/19276","id":381122503,"user":{"login":"danwinship","id":96040,"avatar_url":"https://avatars0.githubusercontent.com/u/96040?v=4","gravatar_id":"","url":"https://api.github.com/users/danwinship","html_url":"https://github.com/danwinship","followers_url":"https://api.github.com/users/danwinship/followers","following_url":"https://api.github.com/users/danwinship/following{/other_user}","gists_url":"https://api.github.com/users/danwinship/gists{/gist_id}","starred_url":"https://api.github.com/users/danwinship/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danwinship/subscriptions","organizations_url":"https://api.github.com/users/danwinship/orgs","repos_url":"https://api.github.com/users/danwinship/repos","events_url":"https://api.github.com/users/danwinship/events{/privacy}","received_events_url":"https://api.github.com/users/danwinship/received_events","type":"User","site_admin":false},"created_at":"2018-04-13T12:39:36Z","updated_at":"2018-04-13T12:39:36Z","author_association":"CONTRIBUTOR","body":"@mojsha most users have not noticed any disruption. It may be that the disruption only occurs if, eg, you have a DNS-based rule and the hostname it refers to has slow/flaky DNS servers.\r\n\r\nTraffic is silently dropped during the interruption; it is not actively rejected. Assuming the interruption is brief, then the packets will get retried and you won't even notice."}}
{"actor":{"display_login":"danwinship"}}