{"action":"created","issue":{"url":"https://api.github.com/repos/NixOS/nix/issues/1128","repository_url":"https://api.github.com/repos/NixOS/nix","labels_url":"https://api.github.com/repos/NixOS/nix/issues/1128/labels{/name}","comments_url":"https://api.github.com/repos/NixOS/nix/issues/1128/comments","events_url":"https://api.github.com/repos/NixOS/nix/issues/1128/events","html_url":"https://github.com/NixOS/nix/issues/1128","id":188911828,"number":1128,"title":"nix-daemon is slow to download files from the cache","user":{"login":"stapelberg","id":55506,"avatar_url":"https://avatars.githubusercontent.com/u/55506?v=3","gravatar_id":"","url":"https://api.github.com/users/stapelberg","html_url":"https://github.com/stapelberg","followers_url":"https://api.github.com/users/stapelberg/followers","following_url":"https://api.github.com/users/stapelberg/following{/other_user}","gists_url":"https://api.github.com/users/stapelberg/gists{/gist_id}","starred_url":"https://api.github.com/users/stapelberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stapelberg/subscriptions","organizations_url":"https://api.github.com/users/stapelberg/orgs","repos_url":"https://api.github.com/users/stapelberg/repos","events_url":"https://api.github.com/users/stapelberg/events{/privacy}","received_events_url":"https://api.github.com/users/stapelberg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-11-12T13:30:56Z","updated_at":"2016-11-23T01:11:39Z","closed_at":null,"body":"I’m currently rebuilding my system to enable debug symbols for all packages.\r\n\r\nA long time (many tens of minutes) is spent in just downloading files from the cache. Take this representative example, sampled from hundreds of others which show the same behavior:\r\n\r\n```\r\n*** Downloading ‘https://cache.nixos.org/nar/1wyksagbra4m12z5qka02jnaylc9w93szg36yxm9h4c93bszr0ci.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/6ida1j75qnn82q3i0z7hz1x03w0kg4hs-ghostscript-9.20.tar.xz’...\r\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n                                 Dload  Upload   Total   Spent    Left  Speed\r\n100 24.7M  100 24.7M    0     0  1220k      0  0:00:20  0:00:20 --:--:-- 1330k\r\n```\r\n\r\nThe reported speed (1.3 MB/s, 20s for 24MB) strikes me as unusually slow:\r\n* The reported load average during the rebuild is `0,08, 0,07, 0,02`, so the rebuild is not CPU bound.\r\n* I’m using an SSD, so I don’t assume the rebuild is I/O bound either.\r\n* I’m on a 1 Gbps fiber ethernet connection and my provider has a direct network interconnect with amazon. I frequently reach download speeds of ≥ 100 MB/s. In fact, when repeating the same download just a couple of seconds later, I see:\r\n\r\n```\r\n$ wget -O/dev/null https://cache.nixos.org/nar/1wyksagbra4m12z5qka02jnaylc9w93szg36yxm9h4c93bszr0ci.nar.xz\r\n--2016-11-12 14:20:35--  https://cache.nixos.org/nar/1wyksagbra4m12z5qka02jnaylc9w93szg36yxm9h4c93bszr0ci.nar.xz\r\nResolving cache.nixos.org (cache.nixos.org)... 2600:9000:200c:ec00:10:dfa4:b0c0:93a1, 2600:9000:200c:4000:10:dfa4:b0c0:93a1, 2600:9000:200c:5c00:10:dfa4:b0c0:93a1, ...\r\nConnecting to cache.nixos.org (cache.nixos.org)|2600:9000:200c:ec00:10:dfa4:b0c0:93a1|:443... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 25973788 (25M) [binary/octet-stream]\r\nSaving to: ‘/dev/null’\r\n\r\n/dev/null                                        100%[=======================================================================================================>]  24,77M  19,9MB/s    in 1,2s    \r\n\r\n2016-11-12 14:20:36 (19,9 MB/s) - ‘/dev/null’ saved [25973788/25973788]\r\n```\r\n\r\nI’m wondering whether this is because of the way the curl/xz/nix-store processes are wired up?\r\n\r\n```\r\nroot     30420  0.0  0.0  50288  5480 ?        Ss   14:17   0:00 nix-daemon --daemon\r\nroot     30437  0.9  0.2  70920 17536 ?        Ssl  14:17   0:02  \\_ nix-daemon 30436\r\nroot     30440  0.0  0.2 207556 21832 ?        S    14:17   0:00      \\_ /nix/store/r9ww7pwkmphr1zbwb1sfnjw9d2y90js2-perl-5.22.2/bin/perl -w -I/nix/store/vfjzlc2si28i2yslaay7413iqvv84ixp-nix-1.11.4 /nix/store/vfjzlc2si28i2yslaay7413iqvv84ixp-nix-1.11.4/libexec/nix/substituters/download-from-binary-cache.pl --query\r\nroot     30757  0.0  0.2 206228 20448 ?        Ss   14:21   0:00      \\_ /nix/store/r9ww7pwkmphr1zbwb1sfnjw9d2y90js2-perl-5.22.2/bin/perl -w -I/nix/store/vfjzlc2si28i2yslaay7413iqvv84ixp-nix-1.11.4 /nix/store/vfjzlc2si28i2yslaay7413iqvv84ixp-nix-1.11.4/libexec/nix/substituters/download-from-binary-cache.pl --substitute /nix/store/2nmvd0m2iwhqcj64012yfj5i49f3hiyj-gnome-bluetooth-3.20.0.tar.xz /nix/store/2nmvd0m2iwhqcj64012yfj5i49f3hiyj-gnome-bluetooth-3.20.0.tar.xz\r\nroot     30758  0.0  0.0 125136  3100 ?        S    14:21   0:00          \\_ sh -c /nix/store/zs006n5dlp80fy5g54rdcfipwf7kca8b-curl-7.50.3-bin/bin/curl --fail --location --connect-timeout 0 -A 'Nix/1.11.4' 'https://cache.nixos.org/nar/1nsbw12hd9d3c0rjmixfr5i8ddvmm0prsh21hxr79g0wd6b909k1.nar.xz' | /nix/store/zn2a3m1if5vsz7h9gnijywkj44va1hws-xz-5.2.2-bin/bin/xz -d | /nix/store/vfjzlc2si28i2yslaay7413iqvv84ixp-nix-1.11.4/bin/nix-store --restore /nix/store/2nmvd0m2iwhqcj64012yfj5i49f3hiyj-gnome-bluetooth-3.20.0.tar.xz\r\nroot     30759  0.0  0.0 134712  6340 ?        S    14:21   0:00              \\_ /nix/store/zs006n5dlp80fy5g54rdcfipwf7kca8b-curl-7.50.3-bin/bin/curl --fail --location --connect-timeout 0 -A Nix/1.11.4 https://cache.nixos.org/nar/1nsbw12hd9d3c0rjmixfr5i8ddvmm0prsh21hxr79g0wd6b909k1.nar.xz\r\nroot     30760  0.0  0.0 119304   400 ?        S    14:21   0:00              \\_ /nix/store/zn2a3m1if5vsz7h9gnijywkj44va1hws-xz-5.2.2-bin/bin/xz -d\r\nroot     30761  0.0  0.0  52376  4432 ?        S    14:21   0:00              \\_ /nix/store/vfjzlc2si28i2yslaay7413iqvv84ixp-nix-1.11.4/bin/nix-store --restore /nix/store/2nmvd0m2iwhqcj64012yfj5i49f3hiyj-gnome-bluetooth-3.20.0.tar.xz\r\n```\r\n\r\nHence, I have the following questions:\r\n\r\n1. Can we make the nix-daemon use libcurl instead of an out-of-process curl, so that we get pipelining, keepalive, etc.? Downloading many small files is _way_ faster when re-using the same TCP connection (which keeps the window size high).\r\n2. Would it be beneficial to decouple the steps (curl, xz, nix-store) into distinct phases?\r\n\r\nThanks!"},"comment":{"url":"https://api.github.com/repos/NixOS/nix/issues/comments/262412841","html_url":"https://github.com/NixOS/nix/issues/1128#issuecomment-262412841","issue_url":"https://api.github.com/repos/NixOS/nix/issues/1128","id":262412841,"user":{"login":"fpletz","id":114159,"avatar_url":"https://avatars.githubusercontent.com/u/114159?v=3","gravatar_id":"","url":"https://api.github.com/users/fpletz","html_url":"https://github.com/fpletz","followers_url":"https://api.github.com/users/fpletz/followers","following_url":"https://api.github.com/users/fpletz/following{/other_user}","gists_url":"https://api.github.com/users/fpletz/gists{/gist_id}","starred_url":"https://api.github.com/users/fpletz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fpletz/subscriptions","organizations_url":"https://api.github.com/users/fpletz/orgs","repos_url":"https://api.github.com/users/fpletz/repos","events_url":"https://api.github.com/users/fpletz/events{/privacy}","received_events_url":"https://api.github.com/users/fpletz/received_events","type":"User","site_admin":false},"created_at":"2016-11-23T01:11:39Z","updated_at":"2016-11-23T01:11:39Z","body":"This is due to nghttp2 which curl uses for HTTP/2 support. See https://github.com/NixOS/nixpkgs/pull/20639."}}
{"actor":{"display_login":"fpletz"}}