{"action":"created","issue":{"url":"https://api.github.com/repos/grosser/parallel_tests/issues/514","repository_url":"https://api.github.com/repos/grosser/parallel_tests","labels_url":"https://api.github.com/repos/grosser/parallel_tests/issues/514/labels{/name}","comments_url":"https://api.github.com/repos/grosser/parallel_tests/issues/514/comments","events_url":"https://api.github.com/repos/grosser/parallel_tests/issues/514/events","html_url":"https://github.com/grosser/parallel_tests/issues/514","id":168366384,"number":514,"title":"If rspec randomizes seeds in subprocesses, how do you recreate with parallel_rspec with N different seeds?","user":{"login":"jrafanie","id":19339,"avatar_url":"https://avatars.githubusercontent.com/u/19339?v=3","gravatar_id":"","url":"https://api.github.com/users/jrafanie","html_url":"https://github.com/jrafanie","followers_url":"https://api.github.com/users/jrafanie/followers","following_url":"https://api.github.com/users/jrafanie/following{/other_user}","gists_url":"https://api.github.com/users/jrafanie/gists{/gist_id}","starred_url":"https://api.github.com/users/jrafanie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrafanie/subscriptions","organizations_url":"https://api.github.com/users/jrafanie/orgs","repos_url":"https://api.github.com/users/jrafanie/repos","events_url":"https://api.github.com/users/jrafanie/events{/privacy}","received_events_url":"https://api.github.com/users/jrafanie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-07-29T16:44:22Z","updated_at":"2016-07-29T17:04:18Z","closed_at":null,"body":"We use `parallel_tests` on travis which gives us 2 cores to run tests in parallel.  Whenever it's run, `parallel_tests` splits up the specs into two groups and forks processes to run the two test groups in two(N cores) unique seed values.  See below.\r\n\r\nMy example travis output:\r\n\r\n```\r\n2 processes for 928 specs, ~ 464 specs per process\r\n\r\nRandomized with seed 10860\r\n.........\r\nRandomized with seed 52073\r\n........\r\n<lots of dots>\r\n\r\n\r\nFinished in 19 minutes 36 seconds (files took 46.47 seconds to load)\r\n6847 examples, 0 failures, 2 pending\r\n\r\nRandomized with seed 10860\r\n\r\n........\r\n<a few more dots>\r\n\r\nFinished in 21 minutes 47 seconds (files took 55.43 seconds to load)\r\n6137 examples, 1 failure, 2 pending\r\n\r\nFailed examples:\r\nrspec ./spec/models/user_spec.rb:37 # Blah\r\n\r\nRandomized with seed 52073\r\n\r\n```\r\n\r\n\r\n\r\n\r\nI don't know how to tell `parallel_tests` to dispatch with the two different seeds you see above.  If it could, I'm not sure how to tell `parallel_tests` which spec group gets which seed.  That becomes even more complicated with an 8 core machine since you have 8 test groups and 8 different seeds.\r\n\r\nInstead, if the underlying test library supports seeds, perhaps `parallel_tests` should generate a unique seed each run and tell each process to use that seed for this run?  That way, each test run would run with exactly one seed and I could easily recreate a failure like this:\r\n\r\n```\r\nSPEC_OPTS=\"--fail-fast --seed 2710\" parallel_rspec -n 2 spec/models\r\n```\r\n\r\nDoes this make sense?  I can do a pull request if this seems reasonable."},"comment":{"url":"https://api.github.com/repos/grosser/parallel_tests/issues/comments/236236183","html_url":"https://github.com/grosser/parallel_tests/issues/514#issuecomment-236236183","issue_url":"https://api.github.com/repos/grosser/parallel_tests/issues/514","id":236236183,"user":{"login":"jrafanie","id":19339,"avatar_url":"https://avatars.githubusercontent.com/u/19339?v=3","gravatar_id":"","url":"https://api.github.com/users/jrafanie","html_url":"https://github.com/jrafanie","followers_url":"https://api.github.com/users/jrafanie/followers","following_url":"https://api.github.com/users/jrafanie/following{/other_user}","gists_url":"https://api.github.com/users/jrafanie/gists{/gist_id}","starred_url":"https://api.github.com/users/jrafanie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrafanie/subscriptions","organizations_url":"https://api.github.com/users/jrafanie/orgs","repos_url":"https://api.github.com/users/jrafanie/repos","events_url":"https://api.github.com/users/jrafanie/events{/privacy}","received_events_url":"https://api.github.com/users/jrafanie/received_events","type":"User","site_admin":false},"created_at":"2016-07-29T17:04:18Z","updated_at":"2016-07-29T17:04:18Z","body":"> all seeds are dumped, so you could try them all :D\r\n\r\nYes, I tried that and it wasn't reliable since I used `SPEC_OPTS=\"--fail-fast --seed 2710\"` which makes both test groups use seed 2710.  It's possible that the sporadic failure only happens when test group 1 is seed 2710 and test group 2 is seed 1234, so it may not recreate when both test groups are using 2710 or 1234.  \r\n\r\n> either way there might be some runtime ordering with parallel processes\r\n> that you will not be able to reproduce ...\r\n\r\nRight, but does the idea of having a single random seed for this run make sense?\r\nOf course, that would require parallel_tests to know how to generate a unique seed for each library but the alternative is possibly never being able to recreate the sporadic failure.\r\n\r\n> ideally do not pollute the filesystem -> use Dir.mktmpdir / FakeFS\r\n\r\nYes, if only it was that easy to detect.  That's the idea behind running with random seeds so these types of sporadic failures become \"possible\" so we can squash and fix.  The concern is that random seeds on each subgroup of tests make it really hard to recreate.\r\n\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"jrafanie"}}