{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/fleet/issues/1418","repository_url":"https://api.github.com/repos/coreos/fleet","labels_url":"https://api.github.com/repos/coreos/fleet/issues/1418/labels{/name}","comments_url":"https://api.github.com/repos/coreos/fleet/issues/1418/comments","events_url":"https://api.github.com/repos/coreos/fleet/issues/1418/events","html_url":"https://github.com/coreos/fleet/pull/1418","id":130621277,"number":1418,"title":"Add unit state cache to mitigate the iteration between fleet and systemd - dbus","user":{"login":"hectorj2f","id":3602792,"avatar_url":"https://avatars.githubusercontent.com/u/3602792?v=3","gravatar_id":"","url":"https://api.github.com/users/hectorj2f","html_url":"https://github.com/hectorj2f","followers_url":"https://api.github.com/users/hectorj2f/followers","following_url":"https://api.github.com/users/hectorj2f/following{/other_user}","gists_url":"https://api.github.com/users/hectorj2f/gists{/gist_id}","starred_url":"https://api.github.com/users/hectorj2f/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hectorj2f/subscriptions","organizations_url":"https://api.github.com/users/hectorj2f/orgs","repos_url":"https://api.github.com/users/hectorj2f/repos","events_url":"https://api.github.com/users/hectorj2f/events{/privacy}","received_events_url":"https://api.github.com/users/hectorj2f/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/fleet/labels/area/performance","name":"area/performance","color":"bfe5bf"},{"url":"https://api.github.com/repos/coreos/fleet/labels/priority/P2","name":"priority/P2","color":"fbca04"},{"url":"https://api.github.com/repos/coreos/fleet/labels/reviewed/needs%20rework","name":"reviewed/needs rework","color":"bfdadc"},{"url":"https://api.github.com/repos/coreos/fleet/labels/technology/systemd","name":"technology/systemd","color":"d4c5f9"}],"state":"open","locked":false,"assignee":{"login":"tixxdz","id":1549291,"avatar_url":"https://avatars.githubusercontent.com/u/1549291?v=3","gravatar_id":"","url":"https://api.github.com/users/tixxdz","html_url":"https://github.com/tixxdz","followers_url":"https://api.github.com/users/tixxdz/followers","following_url":"https://api.github.com/users/tixxdz/following{/other_user}","gists_url":"https://api.github.com/users/tixxdz/gists{/gist_id}","starred_url":"https://api.github.com/users/tixxdz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tixxdz/subscriptions","organizations_url":"https://api.github.com/users/tixxdz/orgs","repos_url":"https://api.github.com/users/tixxdz/repos","events_url":"https://api.github.com/users/tixxdz/events{/privacy}","received_events_url":"https://api.github.com/users/tixxdz/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/coreos/fleet/milestones/29","html_url":"https://github.com/coreos/fleet/milestones/v0.13.0","labels_url":"https://api.github.com/repos/coreos/fleet/milestones/29/labels","id":1515019,"number":29,"title":"v0.13.0","description":"","creator":{"login":"jonboulle","id":2601015,"avatar_url":"https://avatars.githubusercontent.com/u/2601015?v=3","gravatar_id":"","url":"https://api.github.com/users/jonboulle","html_url":"https://github.com/jonboulle","followers_url":"https://api.github.com/users/jonboulle/followers","following_url":"https://api.github.com/users/jonboulle/following{/other_user}","gists_url":"https://api.github.com/users/jonboulle/gists{/gist_id}","starred_url":"https://api.github.com/users/jonboulle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonboulle/subscriptions","organizations_url":"https://api.github.com/users/jonboulle/orgs","repos_url":"https://api.github.com/users/jonboulle/repos","events_url":"https://api.github.com/users/jonboulle/events{/privacy}","received_events_url":"https://api.github.com/users/jonboulle/received_events","type":"User","site_admin":false},"open_issues":14,"closed_issues":26,"state":"open","created_at":"2016-01-19T11:57:47Z","updated_at":"2016-04-21T14:55:40Z","due_on":"2016-04-11T22:00:00Z","closed_at":null},"comments":21,"created_at":"2016-02-02T09:12:11Z","updated_at":"2016-04-21T15:25:20Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/coreos/fleet/pulls/1418","html_url":"https://github.com/coreos/fleet/pull/1418","diff_url":"https://github.com/coreos/fleet/pull/1418.diff","patch_url":"https://github.com/coreos/fleet/pull/1418.patch"},"body":"Unit state cache: \r\n\r\n**Motivation:** Inactive units induce an overload to systemd-dbus communication which linearly increases proportionally to the amount of existing inactive units. Therefore, we decided to research about solutions to minimize the workload of fleet caused by this issue. We decided to build a cache to reduce the amount of calls from systemd to dbus. Fleet periodically performs operations to gather the state of the units in the cluster, so for each inactive unit located in a host, a read operation is triggered to obtain the state from dbus. Therefore, you could have an impression of how this operation could use more resources than the desired.\r\n\r\nNevertheless, we thought about another optimal solution during the development of this cache. This solution requires to modify the `go-dbus` library implementation to provide an event-driven mechanism. This event-driven mechanism would notify every time an operation affects the state of a unit. These notifications would enable to know when the state of a unit change without having to constantly query systemd to get its state.  Obviously we declined this solution as it required the modification of third party software used by fleet.\r\n\r\n## Implementation\r\nWe added a new flag `enable_unitstate_cache` that will activate the usage of the unit state cache and therefore to minimize the usage of the system resources. The unit state cache is used to give the state of inactive units whenever no operation has been recently triggered over them. Otherwise the system will query to dbus in order to get its final state. Consequently, we register operations that are being triggered over the units in order to decide whether or not to use its cached state.\r\n\r\n### Test scenario:\r\n- 3 node cluster\r\n- fleet version 0.11.3\r\n- CoreOS stable (835.9.0)\r\n- etcd version 2.2.1\r\n\r\n## Metrics analyzed:\r\n- CPU usage\r\n- Memory usage\r\n- FS bytes usage\r\n- NET bytes IN\r\n- NET bytes OUT\r\n- NET bytes TOTAL \r\n- Capacity used\r\n\r\n### Savings deducted from these experiments:\r\n\r\nA brief and quick summary:\r\n- There is a clear CPU reduction using the unit state cache. In the following experiments, this benefit can be identified as approx. 10% CPU usage reduction. However this amount will increase/decrease based on the amount of inactive units.\r\n- There is a clear Memory usage reduction using the unit state cache.  In the following experiments, this benefit is up to approx. 100Mb Memory usage reduction. However this amount will increase/decrease based on the amount of inactive units.\r\n- FS bytes used during the experiments slightly INCREASED with the unit state implementation.\r\n- NET bytes IN/OUT used during the experiments slightly INCREASED in the implementation WITHOUT unit cache.\r\n- Capacity used obviously decreased when using the unit state cache.\r\n\r\n\r\n### Experiment 1:\r\nWithout any iteration in the cluster, we evaluated how 3 inactive units could use the system resources of a host.\r\n\r\n#### CPU usage of our implementation using an UNIT STATE CACHE:\r\n<img width=\"1139\" alt=\"screen shot 2016-01-21 at 6 06 52 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488013/e27eac54-c069-11e5-8270-90e8e0268def.png\">\r\n\r\n#### Memory usage of our implementation using an UNIT STATE CACHE:\r\n<img width=\"1133\" alt=\"screen shot 2016-01-21 at 6 07 11 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488014/e2848f2a-c069-11e5-8f3f-4335946689c5.png\">\r\n\r\n#### CPU usage WITHOUT unit state cache:\r\n<img width=\"1134\" alt=\"screen shot 2016-01-21 at 6 04 31 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488084/28bcae3c-c06a-11e5-9bcc-5f106455dfe6.png\">\r\n\r\n#### Memory usage WITHOUT unit state cache:\r\n<img width=\"1139\" alt=\"screen shot 2016-01-21 at 6 04 59 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488085/28bdcd6c-c06a-11e5-92c2-26b49f5fd1fa.png\">\r\n\r\nAs shown above, there is a slight difference caused by these periodic queries to dbus in order to get the state of inactive units.\r\n\r\n### Experiment 2:\r\nWe used a benchmarking tool that starts 900 units in the cluster, and consequently stop+destroy all after a timeout.\r\n\r\n#### CPU usage of our implementation using an UNIT STATE CACHE:\r\n<img width=\"1127\" alt=\"screen shot 2016-01-21 at 5 39 26 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12487096/00660e50-c066-11e5-8e5d-477e1a3587b2.png\">\r\n\r\n#### CPU usage WITHOUT unit state cache:\r\n<img width=\"1130\" alt=\"screen shot 2016-01-21 at 6 15 52 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488263/0df22ad6-c06b-11e5-9982-4b4ad56675cb.png\">\r\n--------------------\r\n### Memory usage of our implementation using an UNIT STATE CACHE:\r\n<img width=\"1077\" alt=\"screen shot 2016-01-21 at 3 59 26 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12487117/199194c6-c066-11e5-8c26-81850174d56e.png\">\r\n\r\n### Memory usage WITHOUT unit state cache:\r\n<img width=\"1132\" alt=\"screen shot 2016-01-21 at 6 15 38 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488262/0de70872-c06b-11e5-8986-cdcbc7f12350.png\">\r\n--------------------\r\n### FS bytes used of our implementation using an UNIT STATE CACHE:\r\n<img width=\"875\" alt=\"screen shot 2016-01-21 at 3 59 37 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12487119/19951f88-c066-11e5-956c-898712b940ab.png\">\r\n### FS bytes used WITHOUT unit state cache:\r\n<img width=\"876\" alt=\"screen shot 2016-01-21 at 3 57 13 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488707/3e9df2f8-c06d-11e5-9673-7b6d558371b6.png\">\r\n--------------------\r\n### Capacity usage of our implementation using an UNIT STATE CACHE:\r\n<img width=\"878\" alt=\"screen shot 2016-01-21 at 3 59 44 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12487118/1993d8a8-c066-11e5-9fd4-2952173705de.png\">\r\n### Capacity used WITHOUT unit state cache:\r\n<img width=\"875\" alt=\"screen shot 2016-01-21 at 3 57 21 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488709/3ea51d26-c06d-11e5-81d2-3b8435e7ac8f.png\">\r\n--------------------\r\n### Net bytes IN of our implementation using an UNIT STATE CACHE:\r\n<img width=\"878\" alt=\"screen shot 2016-01-21 at 3 59 54 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12487116/1990d072-c066-11e5-993c-f06b3b95d7d0.png\">\r\n\r\n### Net bytes IN WITHOUT unit state cache:\r\n<img width=\"881\" alt=\"screen shot 2016-01-21 at 3 56 42 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488706/3e9c0ee8-c06d-11e5-8f6e-138ff6a37809.png\">\r\n----------------------\r\n### Net bytes OUT of our implementation using an UNIT STATE CACHE:\r\n<img width=\"873\" alt=\"screen shot 2016-01-21 at 4 00 01 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12487121/19aae1a6-c066-11e5-9927-98af0d682a23.png\">\r\n\r\n### Net bytes OUT WITHOUT unit state cache:\r\n<img width=\"871\" alt=\"screen shot 2016-01-21 at 3 56 50 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488708/3ea1047a-c06d-11e5-9df1-69a46a78d27a.png\">\r\n\r\n--------------------\r\n### Net bytes total of our implementation using an UNIT STATE CACHE:\r\n<img width=\"886\" alt=\"screen shot 2016-01-21 at 4 00 10 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12487120/19a9d4f0-c066-11e5-9c84-77ab220e1c27.png\">\r\n### Net bytes total WITHOUT unit state cache:\r\n<img width=\"1138\" alt=\"screen shot 2016-01-21 at 6 38 58 pm\" src=\"https://cloud.githubusercontent.com/assets/3602792/12488966/55b57cd0-c06e-11e5-959e-cf59a4ec509a.png\">\r\n\r\n\r\nNOTE: As shown in above Figures (CPU usage, Mem usage, ...) there are two spikes that represent the start operation and the stop/destroy operation which is performed after a fixed timeout."},"comment":{"url":"https://api.github.com/repos/coreos/fleet/issues/comments/212971564","html_url":"https://github.com/coreos/fleet/pull/1418#issuecomment-212971564","issue_url":"https://api.github.com/repos/coreos/fleet/issues/1418","id":212971564,"user":{"login":"kayrus","id":4047483,"avatar_url":"https://avatars.githubusercontent.com/u/4047483?v=3","gravatar_id":"","url":"https://api.github.com/users/kayrus","html_url":"https://github.com/kayrus","followers_url":"https://api.github.com/users/kayrus/followers","following_url":"https://api.github.com/users/kayrus/following{/other_user}","gists_url":"https://api.github.com/users/kayrus/gists{/gist_id}","starred_url":"https://api.github.com/users/kayrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kayrus/subscriptions","organizations_url":"https://api.github.com/users/kayrus/orgs","repos_url":"https://api.github.com/users/kayrus/repos","events_url":"https://api.github.com/users/kayrus/events{/privacy}","received_events_url":"https://api.github.com/users/kayrus/received_events","type":"User","site_admin":false},"created_at":"2016-04-21T15:25:20Z","updated_at":"2016-04-21T15:25:20Z","body":"@hectorj2f I'm not quite sure, but I suppose this improvement should be done in low-level side. I've started to implement systemd-dbus optimizations withing this PR: https://github.com/coreos/fleet/pull/1564\r\n"}}