{"action":"created","issue":{"url":"https://api.github.com/repos/cilium/cilium/issues/3472","repository_url":"https://api.github.com/repos/cilium/cilium","labels_url":"https://api.github.com/repos/cilium/cilium/issues/3472/labels{/name}","comments_url":"https://api.github.com/repos/cilium/cilium/issues/3472/comments","events_url":"https://api.github.com/repos/cilium/cilium/issues/3472/events","html_url":"https://github.com/cilium/cilium/issues/3472","id":310614981,"number":3472,"title":"All Kafka CI tests failing due to proxymap error \"Unable to lookup original destination\"","user":{"login":"manalibhutiyani","id":31253464,"avatar_url":"https://avatars3.githubusercontent.com/u/31253464?v=4","gravatar_id":"","url":"https://api.github.com/users/manalibhutiyani","html_url":"https://github.com/manalibhutiyani","followers_url":"https://api.github.com/users/manalibhutiyani/followers","following_url":"https://api.github.com/users/manalibhutiyani/following{/other_user}","gists_url":"https://api.github.com/users/manalibhutiyani/gists{/gist_id}","starred_url":"https://api.github.com/users/manalibhutiyani/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manalibhutiyani/subscriptions","organizations_url":"https://api.github.com/users/manalibhutiyani/orgs","repos_url":"https://api.github.com/users/manalibhutiyani/repos","events_url":"https://api.github.com/users/manalibhutiyani/events{/privacy}","received_events_url":"https://api.github.com/users/manalibhutiyani/received_events","type":"User","site_admin":false},"labels":[{"id":809259668,"url":"https://api.github.com/repos/cilium/cilium/labels/priority/1.0-blocker","name":"priority/1.0-blocker","color":"b60205","default":false},{"id":813047287,"url":"https://api.github.com/repos/cilium/cilium/labels/priority/insane","name":"priority/insane","color":"b60205","default":false}],"state":"open","locked":false,"assignee":{"login":"rlenglet","id":126329,"avatar_url":"https://avatars3.githubusercontent.com/u/126329?v=4","gravatar_id":"","url":"https://api.github.com/users/rlenglet","html_url":"https://github.com/rlenglet","followers_url":"https://api.github.com/users/rlenglet/followers","following_url":"https://api.github.com/users/rlenglet/following{/other_user}","gists_url":"https://api.github.com/users/rlenglet/gists{/gist_id}","starred_url":"https://api.github.com/users/rlenglet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rlenglet/subscriptions","organizations_url":"https://api.github.com/users/rlenglet/orgs","repos_url":"https://api.github.com/users/rlenglet/repos","events_url":"https://api.github.com/users/rlenglet/events{/privacy}","received_events_url":"https://api.github.com/users/rlenglet/received_events","type":"User","site_admin":false},"assignees":[{"login":"rlenglet","id":126329,"avatar_url":"https://avatars3.githubusercontent.com/u/126329?v=4","gravatar_id":"","url":"https://api.github.com/users/rlenglet","html_url":"https://github.com/rlenglet","followers_url":"https://api.github.com/users/rlenglet/followers","following_url":"https://api.github.com/users/rlenglet/following{/other_user}","gists_url":"https://api.github.com/users/rlenglet/gists{/gist_id}","starred_url":"https://api.github.com/users/rlenglet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rlenglet/subscriptions","organizations_url":"https://api.github.com/users/rlenglet/orgs","repos_url":"https://api.github.com/users/rlenglet/repos","events_url":"https://api.github.com/users/rlenglet/events{/privacy}","received_events_url":"https://api.github.com/users/rlenglet/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-04-02T21:18:38Z","updated_at":"2018-04-03T00:21:27Z","closed_at":null,"author_association":"OWNER","body":"Note that this is causing basic Kafka to fail. Hence marked as a 1.0 blocker.\r\n \r\n```\r\nroot@k8s1:~# cilium bpf proxy list | grep 10.10.0.214 | grep 14107 | wc -l\r\n810\r\nroot@k8s1:~# cilium bpf proxy list | grep 10.10.0.214 | grep 14107 | head \r\n10.10.0.214:57898 (6) => 14107   10.10.0.66:9092 identity 32541 lifetime 2569 \r\n10.10.0.214:59186 (6) => 14107   10.10.0.66:9092 identity 32541 lifetime 2611 \r\n10.10.0.214:57956 (6) => 14107   10.10.0.66:9092 identity 32541 lifetime 2571 \r\n10.10.0.214:58256 (6) => 14107   10.10.0.66:9092 identity 32541 lifetime 2581 \r\n10.10.0.214:59180 (6) => 14107   10.10.0.66:9092 identity 32541 lifetime 2611 \r\n10.10.0.214:57972 (6) => 14107   10.10.0.66:9092 identity 32541 lifetime 2572 \r\n10.10.0.214:59170 (6) => 14107   10.10.0.66:9092 identity 32541 lifetime 2610 \r\n10.10.0.214:58590 (6) => 14107   10.10.0.66:9092 identity 32541 lifetime 2592 \r\n10.10.0.214:58604 (6) => 14107   10.10.0.66:9092 identity 32541 lifetime 2592 \r\n10.10.0.214:58122 (6) => 14107   10.10.0.66:9092 identity 32541 lifetime 2576 \r\n```\r\n\r\nIf you see the output of the entries, when a new produce message happens on L7 the number increase in k8s1\r\n```---------------------------------------------------\r\nBeforeAnyTraffic: Pod \"cilium-2gnpq\" entries in the proxy 1\r\nBeforeAnyTraffic: Pod \"cilium-dcb6x\" entries in the proxy 1\r\n---------------------------------------------------\r\n---------------------------------------------------\r\nTrafficWithoutL7: Pod \"cilium-2gnpq\" entries in the proxy 1\r\nTrafficWithoutL7: Pod \"cilium-dcb6x\" entries in the proxy 1\r\n---------------------------------------------------\r\nAfterAppliedL7: Pod \"cilium-2gnpq\" entries in the proxy 1\r\nAfterAppliedL7: Pod \"cilium-dcb6x\" entries in the proxy 1\r\n---------------------------------------------------\r\n---------------------------------------------------\r\nAfterL7Traffic: Pod \"cilium-2gnpq\" entries in the proxy 789\r\nAfterL7Traffic: Pod \"cilium-dcb6x\" entries in the proxy 1\r\n```\r\n\r\nhttps://gist.github.com/eloycoto/4f234ae22876870e33151e30ab536b2e\r\n\r\nAlso Eloy tested this on Kafka tests(runtime as well as k8s) and it was failing.\r\n\r\nEnvoy is working fine."},"comment":{"url":"https://api.github.com/repos/cilium/cilium/issues/comments/378088719","html_url":"https://github.com/cilium/cilium/issues/3472#issuecomment-378088719","issue_url":"https://api.github.com/repos/cilium/cilium/issues/3472","id":378088719,"user":{"login":"rlenglet","id":126329,"avatar_url":"https://avatars3.githubusercontent.com/u/126329?v=4","gravatar_id":"","url":"https://api.github.com/users/rlenglet","html_url":"https://github.com/rlenglet","followers_url":"https://api.github.com/users/rlenglet/followers","following_url":"https://api.github.com/users/rlenglet/following{/other_user}","gists_url":"https://api.github.com/users/rlenglet/gists{/gist_id}","starred_url":"https://api.github.com/users/rlenglet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rlenglet/subscriptions","organizations_url":"https://api.github.com/users/rlenglet/orgs","repos_url":"https://api.github.com/users/rlenglet/repos","events_url":"https://api.github.com/users/rlenglet/events{/privacy}","received_events_url":"https://api.github.com/users/rlenglet/received_events","type":"User","site_admin":false},"created_at":"2018-04-03T00:21:27Z","updated_at":"2018-04-03T00:21:27Z","author_association":"CONTRIBUTOR","body":"The proxy map size was different between BPF (524288) and userspace (:\r\n```\r\n$ grep -r 'define.*PROXY_MAP_SIZE' bpf\r\nbpf/node_config.h:#define PROXY_MAP_SIZE 524288\r\n```\r\n```\r\nvar (\r\n\t// Proxy4Map represents the BPF map for IPv4 proxy\r\n\tProxy4Map = bpf.NewMap(Proxy4MapName,\r\n\t\t...,\r\n\t\t...,\r\n\t\t...,\r\n\t\t8192,\r\n\t\t...,\r\n\t\t...).WithNonPersistent()\r\n)\r\n```\r\nAfter setting the same value in userspace, the failure doesn't occur anymore."}}
{"actor":{"display_login":"rlenglet"}}