{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27667","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27667/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27667/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27667/events","html_url":"https://github.com/kubernetes/kubernetes/issues/27667","id":161035740,"number":27667,"title":"Kubernetes api, controller, and client performance work","user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/performance","name":"area/performance","color":"d4c5f9"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/CSI-API%20Machinery%20SIG","name":"team/CSI-API Machinery SIG","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2016-06-18T17:50:41Z","updated_at":"2016-06-19T09:50:48Z","closed_at":null,"body":"This is a roundup issue for Kubernetes 1.4 and a tracker to orient folks on the ongoing work.  It is focused on all components that touch the API, but not necessarily the total system.  It also serves to identify areas where folks can get involved.\r\n\r\nTrack issues using [labels performance and CSI-API-Machinery](https://github.com/kubernetes/kubernetes/issues?utf8=✓&q=label%3Aarea%2Fperformance+label%3A%22team%2FCSI-API+Machinery+SIG%22+)\r\n\r\n### Ongoing efforts\r\n\r\nMajor focus: We want to get to our target cluster size metrics (5k nodes and 300k pods) in 1.4 or 1.5, and we are within an order of magnitude of that with 1.3 in many dimensions.\r\n\r\nAreas of attack: We are mostly CPU bound on the controllers and API servers today, and most of that is overhead due to garbage collection.  We also have had a fair number of contention areas that were algorithmic or lock related in the core components.  We've got a good feedback loop for identifying critical bottlenecks, but we could probably stand to get more profiles and metrics gathered from the high scale systems.\r\n\r\n* Reduce the CPU and garbage created by clients and the API server when handling API requests\r\n  * Introduce protobuf\r\n    * [complete #8132] Used for most API objects when talking between client and server\r\n    * Use protobuf for storage format\r\n    * Handle mixed content type storage (objects that embed other objects like Templates or third party resources)\r\n    * [inprogress #27258] Handle optional slices and maps correctly in protobuf\r\n    * [inprogress](https://github.com/gogo/protobuf/issues/174) Investigate use of string arena unmarshalling in protobuf to avoid []byte -> string allocations in decode\r\n  * Optimize the conversion, defaulting, and decoding code paths\r\n     * [complete](https://github.com/kubernetes/kubernetes/issues/25019) Implement more efficient Quantity, using int64 and better serialization\r\n     * [complete #24966] Remove the use of reflection in conversion and defaulting\r\n       * Auto generate fast path conversion functions (AddGenericConversionFunc) for each API type\r\n     * [complete #24727] Make conversions of identical types (builtins, maps, slices, pointers) an assignment, not a copy\r\n     * [inprogress #27659] Further optimization of identical type conversion using unsafe.Pointer\r\n     * Split defaulting from conversion into its own function, possibly by generating an entire \"Default_Pod\" function from component parts, which allows us to avoid having to walk deep objects\r\n  * Use the versioned client where possible to avoid conversion altogether\r\n    * [inprogress](???) Versioned client in controllers depends on the shared cache work (all controllers sharing a cache have to use the same object type)\r\n  * Ensure that the Update and Create code paths are efficient (guaranteed update loop generates a lot of objects)\r\n  * SelfLink and URL.String() are fairly expensive in the encode/decode path\r\n  * For any JSON gaps (non-proto clients) we could look at easyjson, which appears to do more than ugorgji to reduce memory allocation during decode.\r\n* Reduce the amount of duplicate objects being used by controller and admission caches and the amount of total memory used by caches\r\n  * [complete] Separate re-listing and re-syncing (former talks to api server, latter traverses cache) to reduce the amount of duplicate calls to the server and wasted conversion\r\n  * [complete](https://github.com/kubernetes/kubernetes/issues/14978) Add a SharedInformer and use it for Pod caches (all 8 of them)\r\n  * [inprogress] Use shared Namespace caches in admission controllers\r\n  * Investigate and identify other opportunities for shared caches\r\n  * [inprogress] Investigate size of objects kept in shared and unshared caches for duplication and waste\r\n    * Pods tend to have larger objects https://github.com/coreos/kscale/pull/37/files#diff-a423e914cbd27c95d2c0b950e2e4ee5bR14 than necessary - maps and quantity too large\r\n    * Investigate using slices instead of maps for our many small maps, or having an `ArrayMap` equivalent to Java\r\n    * Investigate flyweight cache patterns for certain controllers\r\n* Identify unnecessary blocking / stalls in clients, watches, caches, and controllers\r\n   * [complete] Several fixes to 1.3 scheduler and caches\r\n   * [complete] Avoid flushing aggressively in watches to reduces syscalls\r\n   * [inprogress](@deads2k) Shared caches currently have coarse locks, investigate striping and reducing the duration of the locks\r\n   * Investigate whether we have any slow cache reader problems with our shared caches\r\n   * Audit the watch cache for locking issues\r\n   * Better tooling and documentation to allow people to perform this sort of debugging\r\n   * ???\r\n* Profile and optimize specific components and workloads \r\n  * Collect profile traces from kubemark and high scale clusters automatically\r\n  * [Scheduler](https://github.com/kubernetes/kubernetes/issues/18418) (lots of eyes on this, but also lots of algorithmic improvements possible)\r\n  * Kubelet\r\n  * [inprogress] Replication controller\r\n  * Node controller (known issues with Timer, other garbage creation)\r\n  * Quota \r\n     * [inprogress] Significant improvements including contention striping and batch assignment\r\n     * Optimize and audit use of resource.Quantity in these codepaths\r\n  * Authentication / Authorization / RBAC (has been a significant source of perf bottlenecks in OpenShift in the more complex scenarios)\r\n  * ???\r\n* Reduce traffic between components\r\n  * Investigation colocation of certain processes if necessary\r\n    * OpenShift offers all-in-one processes that bundle etcd, controller, apiserver, as does a few others, and there is some value in compact deployments having an efficient master (downsides are increased memory use and contention)\r\n    * [inprogress](https://github.com/openshift/origin/pull/8082) Investigate loopback connections when in the same process (at the net.Chan level)\r\n    * [mentioned by justinsb] Directly embed etcd and make calls to the Go API\r\n  * Reduce network bandwidth by sending less objects (not yet a blocker in any particular area)\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/226988667","html_url":"https://github.com/kubernetes/kubernetes/issues/27667#issuecomment-226988667","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27667","id":226988667,"user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"created_at":"2016-06-19T09:50:47Z","updated_at":"2016-06-19T09:50:47Z","body":"@smarterclayton - thanks for putting this together. I have few other things on my mind, but let me get back to it on Monday.\r\n\r\nOne thing that gives significant improvements on scale is avoid a lot of unnecessary computations in cacher - in large clusters we know that only O(1) watchers are interested in all pods/nodes and most of them are iterested in only one node/pods from that node.\r\n\r\nI have already a PR in-flight https://github.com/kubernetes/kubernetes/pull/27277 that seems to save ~75% of cpu usage in apiserver in empty 2k-node cluster.  This may be enough or we may consider doing it more sophisticated. But for 2k-node clusters this seem to give huge improvements.\r\n\r\nI have some other thought, but let me write them on Monday."}}
{"actor":{"display_login":"wojtek-t"}}