{"action":"opened","issue":{"url":"https://api.github.com/repos/coreos/fleet/issues/1455","repository_url":"https://api.github.com/repos/coreos/fleet","labels_url":"https://api.github.com/repos/coreos/fleet/issues/1455/labels{/name}","comments_url":"https://api.github.com/repos/coreos/fleet/issues/1455/comments","events_url":"https://api.github.com/repos/coreos/fleet/issues/1455/events","html_url":"https://github.com/coreos/fleet/issues/1455","id":137889120,"number":1455,"title":"usability: significant performance/usability issues with fleetctl batch unit processing and slow etcd storage","user":{"login":"kayrus","id":4047483,"avatar_url":"https://avatars.githubusercontent.com/u/4047483?v=3","gravatar_id":"","url":"https://api.github.com/users/kayrus","html_url":"https://github.com/kayrus","followers_url":"https://api.github.com/users/kayrus/followers","following_url":"https://api.github.com/users/kayrus/following{/other_user}","gists_url":"https://api.github.com/users/kayrus/gists{/gist_id}","starred_url":"https://api.github.com/users/kayrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kayrus/subscriptions","organizations_url":"https://api.github.com/users/kayrus/orgs","repos_url":"https://api.github.com/users/kayrus/repos","events_url":"https://api.github.com/users/kayrus/events{/privacy}","received_events_url":"https://api.github.com/users/kayrus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-03-02T14:32:48Z","updated_at":"2016-03-02T14:32:48Z","closed_at":null,"body":"probably relates to [this](https://github.com/coreos/fleet/issues/1425) issue and #1344, #1433 PRs.\r\n\r\nWhen etcd datadir is located on slow storage (i.e. hdd), fleetctl acts really slow. when you move etcd storage into tmpfs - it works really fast.\r\n\r\nCoreOS: 955.0.0\r\neach CoreOS node:\r\netcd2: 2.2.5\r\nfleet: latest master\r\n512 Mb RAM\r\n1 CPU\r\n\r\nCluster of 3 etcd member nodes and three fleet machines:\r\n\r\n```sh\r\n$ fleetctl list-machines\r\nMACHINE         IP              METADATA\r\n0903835b...     192.168.122.166 hostname=coreos1\r\n78b98c5d...     192.168.122.188 hostname=coreos3\r\nf8f4ec94...     192.168.122.135 hostname=coreos2\r\n```\r\n\r\n```sh\r\n$ etcdctl member list\r\n4c32b6c5bcf8cfd4: name=78b98c5dfe468973b93e098a5a5829ea peerURLs=http://192.168.122.188:2380 clientURLs=http://192.168.122.188:2379\r\n92b755ba67be7f8b: name=0903835ba13faf7f62cf5d8ddc7812de peerURLs=http://192.168.122.166:2380 clientURLs=http://192.168.122.166:2379\r\ncafc731b5b471889: name=f8f4ec94d4193d2f160e8bd22ed360be peerURLs=http://192.168.122.135:2380 clientURLs=http://192.168.122.135:2379\r\n```\r\n\r\nping time between etcd members: `~0.469 ms`\r\n\r\nfleetd configuration:\r\n\r\n```ini\r\n# /etc/systemd/system/fleet.service.d/10-extra-options.conf\r\n[Service]\r\nEnvironment=FLEET_ETCD_REQUEST_TIMEOUT=2 # I used that to avoid etcd registry timeouts.\r\n# /etc/systemd/system/fleet.service.d/10-opt-bin.conf\r\n[Service]\r\nExecStart=\r\nExecStart=/opt/bin/fleetd\r\n# /run/systemd/system/fleet.service.d/20-cloudinit.conf\r\n[Service]\r\nEnvironment=\"FLEET_METADATA=hostname=coreos1\"\r\nEnvironment=\"FLEET_PUBLIC_IP=coreos1\"\r\n```\r\n\r\nTemplate unit file `hello@.service`:\r\n\r\n```ini\r\n[Unit]\r\nDescription=My Service\r\n\r\n[Service]\r\nTimeoutStartSec=0\r\nExecStart=/bin/sh -c \"trap 'exit 0' INT TERM; while true; do echo Hello World %i; sleep 10; done\"\r\n```\r\n\r\nSteps to reproduce:\r\n\r\n```sh\r\n$ fleetctl submit hello@.service\r\nUnit hello@.service inactive\r\n$ time fleetctl start hello@{1..10}.service\r\nUnit hello@1.service inactive\r\nUnit hello@2.service inactive\r\nUnit hello@3.service inactive on 0903835b.../192.168.122.166\r\nUnit hello@4.service inactive\r\nUnit hello@5.service inactive on 0903835b.../192.168.122.166\r\nUnit hello@6.service inactive\r\nUnit hello@7.service inactive\r\nUnit hello@8.service inactive\r\nUnit hello@9.service inactive\r\nUnit hello@10.service inactive on 0903835b.../192.168.122.166\r\nUnit hello@1.service launched on 0903835b.../192.168.122.166\r\nUnit hello@3.service launched on f8f4ec94.../192.168.122.135\r\nUnit hello@2.service launched on 78b98c5d.../192.168.122.188\r\nUnit hello@4.service launched on 0903835b.../192.168.122.166\r\nUnit hello@6.service launched on 78b98c5d.../192.168.122.188\r\nUnit hello@7.service launched on 78b98c5d.../192.168.122.188\r\nUnit hello@5.service launched on f8f4ec94.../192.168.122.135\r\nUnit hello@8.service launched on f8f4ec94.../192.168.122.135\r\nUnit hello@9.service launched on 0903835b.../192.168.122.166\r\nUnit hello@10.service launched on 0903835b.../192.168.122.166\r\n\r\nreal    0m26.688s\r\nuser    0m0.057s\r\nsys     0m0.007s\r\n```\r\n\r\n26 seconds to schedule 10 units? Is it by design: at first submit all units, then start them? I thought that fleetctl should just submit unit with desired state, but not submit unit, then change desired state.\r\n\r\nThen... Why for some units we can see additional info like `Unit hello@3.service inactive on 0903835b.../192.168.122.166`?\r\n\r\nAnother example without template:\r\n\r\n```sh\r\n$ for i in {1..10}; do sed \"s/%i/$i/g\" hello@.service > hello$i.service; done\r\n$ time fleetctl start hello{1..10}.service\r\nUnit hello1.service inactive\r\nUnit hello2.service inactive\r\nUnit hello3.service inactive\r\nUnit hello4.service inactive\r\nUnit hello5.service inactive\r\nUnit hello6.service inactive\r\nUnit hello7.service inactive\r\nUnit hello8.service inactive\r\nUnit hello9.service inactive\r\nUnit hello10.service inactive\r\nUnit hello2.service launched\r\nUnit hello3.service launched\r\nUnit hello4.service launched on 78b98c5d.../192.168.122.188\r\nUnit hello5.service launched on 0903835b.../192.168.122.166\r\nUnit hello1.service launched on 0903835b.../192.168.122.166\r\nUnit hello7.service launched on f8f4ec94.../192.168.122.135\r\nUnit hello10.service launched on 78b98c5d.../192.168.122.188\r\nUnit hello6.service launched on 0903835b.../192.168.122.166\r\nUnit hello8.service launched on 0903835b.../192.168.122.166\r\nUnit hello9.service launched on f8f4ec94.../192.168.122.135\r\n\r\nreal    0m29.444s\r\nuser    0m0.050s\r\nsys     0m0.011s\r\n```\r\n\r\nWe can see that there is no difference.\r\n\r\nHere is the [script](https://gist.github.com/kayrus/a7cceefef4cd973dd1ff) which submits units directly into etcd registry with desired state and it doesn't wait for etcd response:\r\n\r\n```sh\r\n$ time ./units_submiter.sh\r\n... ... ...\r\n... ... ...\r\n... ... ...\r\nreal    0m11.895s\r\nuser    0m0.279s\r\nsys     0m0.151s\r\n```\r\n\r\nThis example shows how we can improve fleet performance by three times (in case when we have slow etcd storage).\r\n\r\nI suggest to use something like \"parallel connections\" limit. We can create N parallel tasks to submit units. Go channels should help. And in addition we have to make sure that fleetctl uses single TCP connection with etcd for all queries.\r\n\r\nP.S. Just in case, command to remove all unit files: `fleetctl list-unit-files --no-legend | awk '{print \"/usr/bin/fleetctl destroy \"$1}' | sh -s`\r\n\r\n/cc @jonboulle @tixxdz "}}