{"action":"created","issue":{"url":"https://api.github.com/repos/concourse/concourse/issues/1966","repository_url":"https://api.github.com/repos/concourse/concourse","labels_url":"https://api.github.com/repos/concourse/concourse/issues/1966/labels{/name}","comments_url":"https://api.github.com/repos/concourse/concourse/issues/1966/comments","events_url":"https://api.github.com/repos/concourse/concourse/issues/1966/events","html_url":"https://github.com/concourse/concourse/issues/1966","id":289329290,"number":1966,"title":"Docker image jobs take a long time to schedule","user":{"login":"avanier","id":4009658,"avatar_url":"https://avatars2.githubusercontent.com/u/4009658?v=4","gravatar_id":"","url":"https://api.github.com/users/avanier","html_url":"https://github.com/avanier","followers_url":"https://api.github.com/users/avanier/followers","following_url":"https://api.github.com/users/avanier/following{/other_user}","gists_url":"https://api.github.com/users/avanier/gists{/gist_id}","starred_url":"https://api.github.com/users/avanier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/avanier/subscriptions","organizations_url":"https://api.github.com/users/avanier/orgs","repos_url":"https://api.github.com/users/avanier/repos","events_url":"https://api.github.com/users/avanier/events{/privacy}","received_events_url":"https://api.github.com/users/avanier/received_events","type":"User","site_admin":false},"labels":[{"id":764529113,"url":"https://api.github.com/repos/concourse/concourse/labels/accepted","name":"accepted","color":"28a745","default":false},{"id":93695076,"url":"https://api.github.com/repos/concourse/concourse/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/concourse/concourse/milestones/24","html_url":"https://github.com/concourse/concourse/milestone/24","labels_url":"https://api.github.com/repos/concourse/concourse/milestones/24/labels","id":3113531,"number":24,"title":"v3.9.0","description":null,"creator":{"login":"vito","id":1880,"avatar_url":"https://avatars1.githubusercontent.com/u/1880?v=4","gravatar_id":"","url":"https://api.github.com/users/vito","html_url":"https://github.com/vito","followers_url":"https://api.github.com/users/vito/followers","following_url":"https://api.github.com/users/vito/following{/other_user}","gists_url":"https://api.github.com/users/vito/gists{/gist_id}","starred_url":"https://api.github.com/users/vito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vito/subscriptions","organizations_url":"https://api.github.com/users/vito/orgs","repos_url":"https://api.github.com/users/vito/repos","events_url":"https://api.github.com/users/vito/events{/privacy}","received_events_url":"https://api.github.com/users/vito/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":24,"state":"closed","created_at":"2018-02-13T15:09:05Z","updated_at":"2018-02-26T16:41:35Z","due_on":null,"closed_at":"2018-02-26T16:41:35Z"},"comments":26,"created_at":"2018-01-17T16:29:50Z","updated_at":"2018-03-27T09:36:17Z","closed_at":"2018-01-29T19:40:02Z","author_association":"NONE","body":"* Concourse version: `3.8.0`\r\n* Deployment type (BOSH/Docker/binary): Binary\r\n* Infrastructure/IaaS: GCE\r\n\r\nOur setup is composed of a minimum of 4 nodes in an autoscaling group. Nodes are configured as `1-standard-4` _(4vcpu, 15GiB ram)_ with attached NVMe of 375GiB. The os is Ubuntu 16.04 LTS.\r\n\r\nThe following settings are of interest :\r\n- `CONCOURSE_WORK_DIR` is redirected to NVMe storage\r\n- `CONCOURSE_BAGGAGECLAIM_DRIVER` is set to `overlay`\r\n- `CONCOURSE_BAGGAGECLAIM_VOLUMES` is redirected to NVMe storage\r\n- `CONCOURSE_BAGGAGECLAIM_OVERLAYS_DIR` is redirected to NVMe storage\r\n- `CONCOURSE_GARDEN_DESTROY_CONTAINERS_ON_STARTUP=true`\r\n- `CONCOURSE_GARDEN_DEPOT` is redirected to NVMe storage\r\n- `CONCOURSE_GARDEN_PROPERTIES_PATH` is redirected to NVMe storage\r\n- `CONCOURSE_GARDEN_ASSETS_DIR` is redirected to NVMe storage\r\n- The NVMe is formatted with the following settings `mkfs.xfs -f -n ftype=1 /dev/nvme0n1p1` and is mounted with `noatime` which is aligned with the Google storage recommendations for GCE\r\n- The NVMe partition is labeled GPT and is block aligned\r\n\r\nWe tested the NVMe on those instances and get performance in the ballpark of 330 MiB/s and 18k random IOPS on those kind of instances. We're hoping this should be sufficient to pull large images.\r\n\r\nWith the settings above, we are seeing certain tasks relying the Docker Image Resource where we know the images contain a lot of small files taking an average of 15 minutes before scheduling. The behaviour seems the consistent with what was observed in #1404 where a build will _\"hang\"_ for a while in the loading state before announcing it's running.\r\n\r\n![unit-assets-web__4_-_concourse](https://user-images.githubusercontent.com/4009658/35053055-1faef652-fb77-11e7-9595-e0d52609919a.png)\r\n\r\nWe tried reverting to `btrfs` only to enjoy stalled workers and garden variety mayhem. (...see what I did there? 😛 )\r\n\r\nThis issue seems to be following in the wake of #1230, for which we had a meeting with @vito @jama-pivotal, and @topherbullock . _(Thanks again for that guys! It's much appreciated!)_ I'm not hoping for a quick fix to this, but I'm hoping to outline that this situation is a quantifiable problem for the organizations where we're running Concourse. I am also volunteering whatever resources you believe might help us alleviate the problem.\r\n\r\nAs one would say... _Halp!_\r\n\r\ncc @typositoire @fkoclas @baptiste-bonnaudet"},"comment":{"url":"https://api.github.com/repos/concourse/concourse/issues/comments/376460541","html_url":"https://github.com/concourse/concourse/issues/1966#issuecomment-376460541","issue_url":"https://api.github.com/repos/concourse/concourse/issues/1966","id":376460541,"user":{"login":"leshik","id":443678,"avatar_url":"https://avatars1.githubusercontent.com/u/443678?v=4","gravatar_id":"","url":"https://api.github.com/users/leshik","html_url":"https://github.com/leshik","followers_url":"https://api.github.com/users/leshik/followers","following_url":"https://api.github.com/users/leshik/following{/other_user}","gists_url":"https://api.github.com/users/leshik/gists{/gist_id}","starred_url":"https://api.github.com/users/leshik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leshik/subscriptions","organizations_url":"https://api.github.com/users/leshik/orgs","repos_url":"https://api.github.com/users/leshik/repos","events_url":"https://api.github.com/users/leshik/events{/privacy}","received_events_url":"https://api.github.com/users/leshik/received_events","type":"User","site_admin":false},"created_at":"2018-03-27T09:36:17Z","updated_at":"2018-03-27T09:36:17Z","author_association":"NONE","body":"It didn't help, unfortunately. And `overlay2` is painfully slow."}}
{"actor":{"display_login":"leshik"}}