{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/7751","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/7751/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/7751/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/7751/events","html_url":"https://github.com/rancher/rancher/issues/7751","id":206418341,"node_id":"MDU6SXNzdWUyMDY0MTgzNDE=","number":7751,"title":"Slow upgrade / any action and high CPU utilization","user":{"login":"tholcman","id":791679,"node_id":"MDQ6VXNlcjc5MTY3OQ==","avatar_url":"https://avatars0.githubusercontent.com/u/791679?v=4","gravatar_id":"","url":"https://api.github.com/users/tholcman","html_url":"https://github.com/tholcman","followers_url":"https://api.github.com/users/tholcman/followers","following_url":"https://api.github.com/users/tholcman/following{/other_user}","gists_url":"https://api.github.com/users/tholcman/gists{/gist_id}","starred_url":"https://api.github.com/users/tholcman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tholcman/subscriptions","organizations_url":"https://api.github.com/users/tholcman/orgs","repos_url":"https://api.github.com/users/tholcman/repos","events_url":"https://api.github.com/users/tholcman/events{/privacy}","received_events_url":"https://api.github.com/users/tholcman/received_events","type":"User","site_admin":false},"labels":[{"id":145695744,"node_id":"MDU6TGFiZWwxNDU2OTU3NDQ=","url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":28,"created_at":"2017-02-09T06:35:36Z","updated_at":"2018-08-28T10:43:13Z","closed_at":"2018-08-28T10:43:13Z","author_association":"NONE","body":"**Rancher Versions:**\r\nServer: 1.3.4\r\n*Cattle: 0.175.10*\r\nhealthcheck: 0.2.0\r\nipsec: 0.0.2\r\nnetwork-services: 0.0.8\r\nscheduler: 0.3.0\r\nkubernetes (if applicable):\r\n\r\n**Docker Version:** 1.13.0\r\n\r\n**OS and where are the hosts located? (cloud, bare metal, etc):** ~50 hosts Ubuntu16, Bare Metal ~250 containers in 5 services.\r\n\r\n**Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)** External DB r3.large, 2 Rancher hosts c4.xlarge (4 cores)\r\n\r\n**Environment Type: (Cattle/Kubernetes/Swarm/Mesos)** Cattle v0.175.10\r\n\r\n**Steps to Reproduce:**\r\nRun rolling upgrade/finish/rollback on any service or perform almost ANY action on containers\r\n\r\n**Results:** \r\nIt takes hours to rotate ~50 containers\r\nboth Rancher \"masters\" have CPU Utilization 100%\r\nWhen running basic java prfiler:\r\n`./jvmtop.sh 9 --profile -n 50`\r\nwe got:\r\n```\r\n JvmTop 0.8.0 alpha - 06:15:02,  amd64,  4 cpus, Linux 4.4.0-59-, load avg 12.90\r\n http://code.google.com/p/jvmtop\r\n Profiling PID 9:         io.cattle.platform.launcher.Main \r\n  23.02% (     2.13s) org.yaml.snakeyaml.resolver.Resolver.resolve()\r\n  15.21% (     1.63s) org.yaml.snakeyaml.emitter.Emitter.writePlain()\r\n  14.23% (     1.52s) ....yaml.snakeyaml.representer.SafeRepresenter$Represent()\r\n   5.87% (     0.63s) ....yaml.snakeyaml.representer.Representer.representJava()\r\n   5.78% (     0.62s) org.mariadb.jdbc.internal.util.buffer.ReadUtil.readFully()\r\n   4.10% (     0.44s) org.yaml.snakeyaml.emitter.Emitter.writeIndent()\r\n   3.97% (     0.43s) org.yaml.snakeyaml.emitter.Emitter.writeIndicator()\r\n   3.41% (     0.37s) org.yaml.snakeyaml.emitter.Emitter.analyzeScalar()\r\n...\r\n```\r\n\r\nIt spends more 70% of CPU time encoding and decoding YAMLs!!!!!!"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/416537553","html_url":"https://github.com/rancher/rancher/issues/7751#issuecomment-416537553","issue_url":"https://api.github.com/repos/rancher/rancher/issues/7751","id":416537553,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjUzNzU1Mw==","user":{"login":"superseb","id":2620179,"node_id":"MDQ6VXNlcjI2MjAxNzk=","avatar_url":"https://avatars2.githubusercontent.com/u/2620179?v=4","gravatar_id":"","url":"https://api.github.com/users/superseb","html_url":"https://github.com/superseb","followers_url":"https://api.github.com/users/superseb/followers","following_url":"https://api.github.com/users/superseb/following{/other_user}","gists_url":"https://api.github.com/users/superseb/gists{/gist_id}","starred_url":"https://api.github.com/users/superseb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/superseb/subscriptions","organizations_url":"https://api.github.com/users/superseb/orgs","repos_url":"https://api.github.com/users/superseb/repos","events_url":"https://api.github.com/users/superseb/events{/privacy}","received_events_url":"https://api.github.com/users/superseb/received_events","type":"User","site_admin":false},"created_at":"2018-08-28T10:43:13Z","updated_at":"2018-08-28T10:43:13Z","author_association":"MEMBER","body":"Closing as stale, if you encounter issues with performance, please file a new issue with details so we can investigate."}}
{"actor":{"display_login":"superseb"}}