{"action":"created","issue":{"url":"https://api.github.com/repos/rust-lang/cargo/issues/2526","repository_url":"https://api.github.com/repos/rust-lang/cargo","labels_url":"https://api.github.com/repos/rust-lang/cargo/issues/2526/labels{/name}","comments_url":"https://api.github.com/repos/rust-lang/cargo/issues/2526/comments","events_url":"https://api.github.com/repos/rust-lang/cargo/issues/2526/events","html_url":"https://github.com/rust-lang/cargo/issues/2526","id":144165509,"number":2526,"title":"Migrate to xz compression for crates.io crates","user":{"login":"alexcrichton","id":64996,"avatar_url":"https://avatars.githubusercontent.com/u/64996?v=3","gravatar_id":"","url":"https://api.github.com/users/alexcrichton","html_url":"https://github.com/alexcrichton","followers_url":"https://api.github.com/users/alexcrichton/followers","following_url":"https://api.github.com/users/alexcrichton/following{/other_user}","gists_url":"https://api.github.com/users/alexcrichton/gists{/gist_id}","starred_url":"https://api.github.com/users/alexcrichton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexcrichton/subscriptions","organizations_url":"https://api.github.com/users/alexcrichton/orgs","repos_url":"https://api.github.com/users/alexcrichton/repos","events_url":"https://api.github.com/users/alexcrichton/events{/privacy}","received_events_url":"https://api.github.com/users/alexcrichton/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rust-lang/cargo/labels/E-hard","name":"E-hard","color":"02e10c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-03-29T06:04:37Z","updated_at":"2016-03-29T18:28:18Z","closed_at":null,"body":"## Why?\r\n\r\n**tl;dr;** unless you have a 52MB/s internet connection, xz is probably faster, but my math should be checked!\r\n\r\nThe statistics here are operating over the entirety of crates.io at this time, including all crates and all versions ever published. Currently we use `flate2` which is backed by miniz using the [best compression](https://github.com/rust-lang/cargo/blob/25e1301a4b72a31c0798aee4a8e8335d12d2a8ee/src/cargo/ops/cargo_package.rs#L119) for tarballs corresponding to [level 9](https://github.com/alexcrichton/flate2-rs/blob/522256b653087b615b5244fd601ab599026d2430/src/lib.rs#L110). The \"zlib\" numbers here are generated from compiling `flate2` against zlib instead of miniz. The xz numbers are generated with the [xz2 crate](https://github.com/alexcrichton/xz2-rs) also using compression level 9.\r\n\r\nFirst up, let's take a look at what we're just storing on S3. This is just the size of all published crates.\r\n\r\n| stat | val | % smaller |\r\n|-----|-------|-------|\r\n| miniz | 3776697673 | 0.0 |\r\n| zlib | 3776147960 | 0.01 |\r\n| xz | 2411082764 | 36.16 |\r\n\r\nNext, let's multiply each version's size by how many times it's been downloaded. This in theory the number of bytes that have been transferred out of S3\r\n\r\n| stat | val | % smaller |\r\n|-----|-------|-------|\r\n| miniz | 3502228200434 | 0.0 |\r\n| zlib | 3501544526571 | 0.02 |\r\n| xz | 2373770137784 | 32.22 |\r\n\r\nNext up is how long it took (in nanoseconds) in total to decompress all crates on my local computer.\r\n\r\n| stat | val | ns per byte | % slower |\r\n|-----|-------|-------|------------|\r\n| miniz  | 118891793644 | 31.484 | 0.0 |\r\n| zlib | 118952441288 | 31.501 | 0.05 |\r\n| xz | 144860343353 | 60.081 | 90.83 |\r\n\r\nOk, so the [claims of xz](http://tukaani.org/xz/) are correct in that it's about 30% smaller than gzip, but the decompression time is much larger! If we assume that these numbers are true in the average, however, let's do some math to figure out how fast your bandwidth needs to be to break even.\r\n\r\nFirst up we've got:\r\n\r\n```\r\ntime = bytes / BW + bytes * time_per_byte\r\n```\r\n\r\nSo if we assume that xz crates are on average 36.16% smaller and use the timings we found above for decompressing, we have:\r\n\r\n```\r\nbytes / BW + bytes * 31.484 = bytes * (1 - .3616) / BW + bytes * (1 - .3616) * 60.081\r\n            1 / BW + 31.484 = (1 - .3616) / BW + (1 - .3616) * 60.081                \r\n            1 + BW * 31.484 = (1 - .3616) + (1 - .3616) * 60.081 * BW                \r\n            1 + BW * 31.484 = .6384 + 38.36 * BW                                     \r\n                      .3616 = 6.876 * BW                                             \r\n                    0.05258 = BW                                                     \r\n```\r\n\r\nNow that's 0.05258 bytes per nanosecond, which translates to 52,580,000 bytes per second which is 52.58 MB per second. \r\n\r\nSo... if my math is right, xz is faster for download + decompression unless you have a 52MB/s uplink to crates.io. I kinda doubt *anyone* really has that kind of bandwidth to our bucket, so that should mean that on average the smaller download size of xz-compressed crates will more than make up for the longer decompression times.\r\n\r\nNote that I didn't bother getting statistics about compression times. xz is *massively* slower than gzip, but I don't think it matters much for most as `cargo publish` is super rare.\r\n\r\n## How?\r\n\r\nOk, so I haven't actually thought much about this. I was basically curious recently and just wanted to make sure that these numbers and statistics didn't disappear. For backwards compatibility we need to continue to publish gzip crates for quite awhile (maybe forever). Our S3 bill can take the hit though, that's fine. This means that `cargo publish` will create both tarballs and upload them.\r\n\r\nThe index would grow another field to store the sha256 hash of the xz crate, and new publishes would fill that in.\r\n\r\nNext, when downloading a crate Cargo could start saying \"I'm xz compatible\" at which point crates.io would redirect to the xz url (currently it redirects to a gz url) for any crate which has an xz-compressed tarball available (crates.io would grow a flag for this for all published versions).\r\n\r\nI... think that would cover our bases? Certainly a lot of work so I probably won't be able to get around to this any time soon, but wanted to put this out there in case others were interested :)"},"comment":{"url":"https://api.github.com/repos/rust-lang/cargo/issues/comments/203039165","html_url":"https://github.com/rust-lang/cargo/issues/2526#issuecomment-203039165","issue_url":"https://api.github.com/repos/rust-lang/cargo/issues/2526","id":203039165,"user":{"login":"larsbergstrom","id":475847,"avatar_url":"https://avatars.githubusercontent.com/u/475847?v=3","gravatar_id":"","url":"https://api.github.com/users/larsbergstrom","html_url":"https://github.com/larsbergstrom","followers_url":"https://api.github.com/users/larsbergstrom/followers","following_url":"https://api.github.com/users/larsbergstrom/following{/other_user}","gists_url":"https://api.github.com/users/larsbergstrom/gists{/gist_id}","starred_url":"https://api.github.com/users/larsbergstrom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/larsbergstrom/subscriptions","organizations_url":"https://api.github.com/users/larsbergstrom/orgs","repos_url":"https://api.github.com/users/larsbergstrom/repos","events_url":"https://api.github.com/users/larsbergstrom/events{/privacy}","received_events_url":"https://api.github.com/users/larsbergstrom/received_events","type":"User","site_admin":false},"created_at":"2016-03-29T18:28:18Z","updated_at":"2016-03-29T18:28:18Z","body":"The one place where you might see that sort of bandwidth is on EC2 machines. In my experience, their connection to S3 buckets is pretty unreal.\r\n\r\nI like the strategy of having both formats and letting the cargo client choose (with a reasonable default) given that the S3 cost is negligible. I'm not worried about *our* platforms, but for people trying to stand up rust+cargo on some wonky new embedded platform with a strange linux variant, every new package can be a nightmare --- I'm thinking of my experiences with these \"custom\" ARM linux builds for PandaBoard, etc. that I've had to play with for Servo. Though, admittedly, one could argue that you should always be cross-compiling to target them instead.\r\n\r\n"}}