{"action":"created","issue":{"url":"https://api.github.com/repos/rust-lang/rust/issues/33454","repository_url":"https://api.github.com/repos/rust-lang/rust","labels_url":"https://api.github.com/repos/rust-lang/rust/issues/33454/labels{/name}","comments_url":"https://api.github.com/repos/rust-lang/rust/issues/33454/comments","events_url":"https://api.github.com/repos/rust-lang/rust/issues/33454/events","html_url":"https://github.com/rust-lang/rust/issues/33454","id":153403972,"number":33454,"title":"Fuzzing incremental compilation against crates.io packages","user":{"login":"nikomatsakis","id":155238,"avatar_url":"https://avatars.githubusercontent.com/u/155238?v=3","gravatar_id":"","url":"https://api.github.com/users/nikomatsakis","html_url":"https://github.com/nikomatsakis","followers_url":"https://api.github.com/users/nikomatsakis/followers","following_url":"https://api.github.com/users/nikomatsakis/following{/other_user}","gists_url":"https://api.github.com/users/nikomatsakis/gists{/gist_id}","starred_url":"https://api.github.com/users/nikomatsakis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nikomatsakis/subscriptions","organizations_url":"https://api.github.com/users/nikomatsakis/orgs","repos_url":"https://api.github.com/users/nikomatsakis/repos","events_url":"https://api.github.com/users/nikomatsakis/events{/privacy}","received_events_url":"https://api.github.com/users/nikomatsakis/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rust-lang/rust/labels/A-incr-comp","name":"A-incr-comp","color":"d7e012"},{"url":"https://api.github.com/repos/rust-lang/rust/labels/E-mentor","name":"E-mentor","color":"02E10C"},{"url":"https://api.github.com/repos/rust-lang/rust/labels/P-medium","name":"P-medium","color":"eb6420"},{"url":"https://api.github.com/repos/rust-lang/rust/labels/T-compiler","name":"T-compiler","color":"bfd4f2"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2016-05-06T08:33:22Z","updated_at":"2016-05-24T20:27:00Z","closed_at":null,"body":"To really have confidence in incremental compilation, we need to do large-scale testing. One specific way to do this is to fuzz against different versions of crates from crates.io. This corresponds to a very particular, and common, workflow:\r\n\r\n- `cargo update` to get new packages\r\n- `cargo build` to build with them\r\n\r\nThe idea here would be to write a tool which iterates over the last N published crates (that is, the last N versions that have been published to crates.io). For each version V that has been published, we would:\r\n\r\n- Create a fresh directory\r\n- Load version V-1 (that is, the previous version available on crates.io)\r\n  - if there is no previous version, skip this crate\r\n- Compile version V-1 using `-Z incremental=some-temporary-directory`\r\n- Load version V in place (overwriting the old files)\r\n- Compile version V using `-Z incremental=some-temporary-directory`\r\n- Create another directory\r\n- Compile version V from scratch\r\n- Compare the results of the two compilations of version V:\r\n  - they should produce the same errors/warnings\r\n  - if successful, they should produce the same output file\r\n    - or at least it should contain the same symbols etc\r\n    - we'll have to work this out a bit =)\r\n\r\nAs of this writing, we're not yet at the point where incremental compilation actually *works*, but we are already tracking dependencies throughout the front-end; just tracking those dependencies sometimes causes ICEs, which means this tool would be of immediate use. Plus, we expect to be tracking dependencies soon. \r\n\r\nI imagine that this tool would be written in Rust. Rather than placing it in `src/test/tools`, it might be better if it lived as its own crate (maybe in `rust-lang`?), so that it can draw on the full crates.io ecosystem for utility functions and so forth. We would eventually just run it on a regular and continous basis and open issues for any problems that we find.\r\n\r\nSince it has relatively few prerequisites, this bug seems like a good \"entry vector\" for contributing to Rust. I would love to \"mentor\" someone to work on it."},"comment":{"url":"https://api.github.com/repos/rust-lang/rust/issues/comments/221391258","html_url":"https://github.com/rust-lang/rust/issues/33454#issuecomment-221391258","issue_url":"https://api.github.com/repos/rust-lang/rust/issues/33454","id":221391258,"user":{"login":"nikomatsakis","id":155238,"avatar_url":"https://avatars.githubusercontent.com/u/155238?v=3","gravatar_id":"","url":"https://api.github.com/users/nikomatsakis","html_url":"https://github.com/nikomatsakis","followers_url":"https://api.github.com/users/nikomatsakis/followers","following_url":"https://api.github.com/users/nikomatsakis/following{/other_user}","gists_url":"https://api.github.com/users/nikomatsakis/gists{/gist_id}","starred_url":"https://api.github.com/users/nikomatsakis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nikomatsakis/subscriptions","organizations_url":"https://api.github.com/users/nikomatsakis/orgs","repos_url":"https://api.github.com/users/nikomatsakis/repos","events_url":"https://api.github.com/users/nikomatsakis/events{/privacy}","received_events_url":"https://api.github.com/users/nikomatsakis/received_events","type":"User","site_admin":false},"created_at":"2016-05-24T20:27:00Z","updated_at":"2016-05-24T20:27:00Z","body":"@mrmiywj I am sorry it took me so long to get back to you! I just did a detailed read through your crate and ran some experiments. It does indeed seem quite close.\r\n\r\nA few things:\r\n\r\n1. The current version seems to simply test that compilation succeeds. That's a good metric, but we could do better -- for example, I think that the output (in terms of errors, warnings etc) ought to be the same when you are building incrementally as when you are building from scratch. But maybe we can add this later. I'd also like to test that the generated binary is \"the same\" but I'm not sure how to do this in any kind of reliable fashion. (Hmm, maybe we can instrument the compiler to dump the MIR or something internal that we have more control over.)\r\n2. We should (optionally) run the tests too, I think.\r\n3. We probably want either a mode of this tool or some other tool that can automatically generate the list of packages and revisions to test. In other words, I'd like to be able to generate a list of the last 5 versions of each package in crates.io or something and run against all of those, logging any failures. (Perhaps with a blacklist as well.)\r\n"}}
{"actor":{"display_login":"nikomatsakis"}}