{"action":"opened","issue":{"url":"https://api.github.com/repos/weaveworks/flux/issues/1181","repository_url":"https://api.github.com/repos/weaveworks/flux","labels_url":"https://api.github.com/repos/weaveworks/flux/issues/1181/labels{/name}","comments_url":"https://api.github.com/repos/weaveworks/flux/issues/1181/comments","events_url":"https://api.github.com/repos/weaveworks/flux/issues/1181/events","html_url":"https://github.com/weaveworks/flux/issues/1181","id":336881193,"node_id":"MDU6SXNzdWUzMzY4ODExOTM=","number":1181,"title":"Flux does not work properly for cluster with large number of namespaces","user":{"login":"mwhittington21","id":29389868,"node_id":"MDQ6VXNlcjI5Mzg5ODY4","avatar_url":"https://avatars1.githubusercontent.com/u/29389868?v=4","gravatar_id":"","url":"https://api.github.com/users/mwhittington21","html_url":"https://github.com/mwhittington21","followers_url":"https://api.github.com/users/mwhittington21/followers","following_url":"https://api.github.com/users/mwhittington21/following{/other_user}","gists_url":"https://api.github.com/users/mwhittington21/gists{/gist_id}","starred_url":"https://api.github.com/users/mwhittington21/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwhittington21/subscriptions","organizations_url":"https://api.github.com/users/mwhittington21/orgs","repos_url":"https://api.github.com/users/mwhittington21/repos","events_url":"https://api.github.com/users/mwhittington21/events{/privacy}","received_events_url":"https://api.github.com/users/mwhittington21/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-29T06:40:55Z","updated_at":"2018-06-29T06:40:55Z","closed_at":null,"author_association":"NONE","body":"# The problem\r\n\r\nWorks great on all the clusters I manage until I get to a snowflake that has 30k+ namespaces. When that happens the daemon won't output any logs and will max out on it's limit of 50 requests per second to the API server.\r\n\r\nI assume it is because it is trying to list any deployments happening in those namespaces.\r\n\r\n# The solution\r\n\r\nAs clusters with this many namespaces are quite rare I would like to propose the following solution.\r\n\r\nI think that being able to whitelist some namespaces that you care about for Flux management would be a good way to solve this. If no namespaces are whitelisted then it would default to its current behaviour. If any namesapces are provided then it will only look for objects within the specified namespaces. This means that current clusters are unaffected while large clusters can pick and choose.\r\n\r\n# Symptoms and debugging\r\n\r\nExample logs from cluster that has 30k+ namespaces:\r\n\r\n```\r\nus-east-1_sync --docker-config=/docker-creds/config.json --registry-trace=true\r\nts=2018-06-29T05:28:24.9841875Z caller=main.go:138 version=1.4.1\r\nts=2018-06-29T05:28:25.006700363Z caller=main.go:227 component=cluster identity=/etc/fluxd/ssh/identity\r\nts=2018-06-29T05:28:25.006744553Z caller=main.go:228 component=cluster identity.pub=\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCZ4gypLgAlw/Ja5SWTpXP26HoAvaswz7S+0550I8AdZx5bwGw23StBgFZ4KrfqNlpbtNPx9dwRLu4SK9uFaP5ocJ/9ghNfv6VoBtvdvOGmVZxljMrFAQza3adZef67WkOaV23FwPJ0EYfEKRK5EHR2Ju1IJWwaqsLqjmEoRYQGVUaNgcA76UQfFmNE8yQmvXcyVoBZc2Go3ILMjnk4RD3CK63I+EUhOiYTuyAuWY05Dc9XFctalNcpHzli/Eas/79qRyAeg1zyaAI/a7trULE9SPXlKXwfch7hZQvYRtfLqPKu5B8XjvfRga3sFHDETgLxtjmyALaHJFYe7AKLTiQP\"\r\nts=2018-06-29T05:28:25.006774386Z caller=main.go:229 component=cluster host=https://10.155.0.1:443 version=kubernetes-v1.10.3\r\nts=2018-06-29T05:28:25.006843526Z caller=main.go:241 component=cluster kubectl=/usr/local/bin/kubectl\r\nts=2018-06-29T05:28:25.007820146Z caller=main.go:249 component=cluster ping=true\r\nts=2018-06-29T05:28:25.00939803Z caller=main.go:368 url=ssh://git@stash.atlassian.com:7997/kube/goliath-flux user=\"Weave Flux\" email=support@weave.works sync-tag=bbci/prod/us-east-1_sync notes-ref=flux set-author=false\r\nts=2018-06-29T05:28:25.009441423Z caller=main.go:423 upstream=\"no upstream URL given\"\r\nts=2018-06-29T05:28:25.009678217Z caller=loop.go:89 component=sync-loop err=\"git repo not ready\"\r\nts=2018-06-29T05:28:25.009843024Z caller=images.go:15 component=sync-loop msg=\"polling images\"\r\nts=2018-06-29T05:28:25.010617777Z caller=images.go:21 component=sync-loop error=\"getting unlocked automated services: git repo not ready\"\r\nts=2018-06-29T05:28:25.010434008Z caller=main.go:440 addr=:3030\r\nts=2018-06-29T05:28:25.307131203Z caller=checkpoint.go:24 component=checkpoint msg=\"up to date\" latest=1.4.1\r\nts=2018-06-29T05:28:30.360228703Z caller=loop.go:102 component=sync-loop event=refreshed url=ssh://git@stash.atlassian.com:7997/kube/goliath-flux branch=bbci/prod/us-east-1 HEAD=7bb72f2c1dff479e90fcec2e8a96f09aba65d6a3\r\n```\r\n\r\nGraphs showing Flux smashing the API server with requests.\r\n\r\n![screen shot 2018-06-29 at 4 27 23 pm](https://user-images.githubusercontent.com/29389868/42077115-261f5e36-7bbb-11e8-9699-939f5d48957d.png)\r\n"}}
{"actor":{"display_login":"mwhittington21"}}