{"action":"created","issue":{"url":"https://api.github.com/repos/hammerlab/magic-rdds/issues/18","repository_url":"https://api.github.com/repos/hammerlab/magic-rdds","labels_url":"https://api.github.com/repos/hammerlab/magic-rdds/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/hammerlab/magic-rdds/issues/18/comments","events_url":"https://api.github.com/repos/hammerlab/magic-rdds/issues/18/events","html_url":"https://github.com/hammerlab/magic-rdds/issues/18","id":172344124,"number":18,"title":"TestSparkListener race condition","user":{"login":"ryan-williams","id":465045,"avatar_url":"https://avatars.githubusercontent.com/u/465045?v=3","gravatar_id":"","url":"https://api.github.com/users/ryan-williams","html_url":"https://github.com/ryan-williams","followers_url":"https://api.github.com/users/ryan-williams/followers","following_url":"https://api.github.com/users/ryan-williams/following{/other_user}","gists_url":"https://api.github.com/users/ryan-williams/gists{/gist_id}","starred_url":"https://api.github.com/users/ryan-williams/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryan-williams/subscriptions","organizations_url":"https://api.github.com/users/ryan-williams/orgs","repos_url":"https://api.github.com/users/ryan-williams/repos","events_url":"https://api.github.com/users/ryan-williams/events{/privacy}","received_events_url":"https://api.github.com/users/ryan-williams/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hammerlab/magic-rdds/labels/bug","name":"bug","color":"ee0701"},{"url":"https://api.github.com/repos/hammerlab/magic-rdds/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"closed","locked":false,"assignee":{"login":"ryan-williams","id":465045,"avatar_url":"https://avatars.githubusercontent.com/u/465045?v=3","gravatar_id":"","url":"https://api.github.com/users/ryan-williams","html_url":"https://github.com/ryan-williams","followers_url":"https://api.github.com/users/ryan-williams/followers","following_url":"https://api.github.com/users/ryan-williams/following{/other_user}","gists_url":"https://api.github.com/users/ryan-williams/gists{/gist_id}","starred_url":"https://api.github.com/users/ryan-williams/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryan-williams/subscriptions","organizations_url":"https://api.github.com/users/ryan-williams/orgs","repos_url":"https://api.github.com/users/ryan-williams/repos","events_url":"https://api.github.com/users/ryan-williams/events{/privacy}","received_events_url":"https://api.github.com/users/ryan-williams/received_events","type":"User","site_admin":false},"assignees":[{"login":"ryan-williams","id":465045,"avatar_url":"https://avatars.githubusercontent.com/u/465045?v=3","gravatar_id":"","url":"https://api.github.com/users/ryan-williams","html_url":"https://github.com/ryan-williams","followers_url":"https://api.github.com/users/ryan-williams/followers","following_url":"https://api.github.com/users/ryan-williams/following{/other_user}","gists_url":"https://api.github.com/users/ryan-williams/gists{/gist_id}","starred_url":"https://api.github.com/users/ryan-williams/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryan-williams/subscriptions","organizations_url":"https://api.github.com/users/ryan-williams/orgs","repos_url":"https://api.github.com/users/ryan-williams/repos","events_url":"https://api.github.com/users/ryan-williams/events{/privacy}","received_events_url":"https://api.github.com/users/ryan-williams/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2016-08-21T22:05:23Z","updated_at":"2016-08-21T22:44:41Z","closed_at":"2016-08-21T22:44:40Z","body":"After much back and forth on #13 and #9, we merged in tests that verify the number of Spark stages that have been run using `TestSparkListener`.\r\n\r\nUnfortunately, using the `SparkListener` API for this is just not thread-safe, and I was able to somewhat consistently reproduce `LazyZippedWithIndexTest` failing as a result in one specific context locally (running all tests from in IntelliJ; running just `LazyZippedWithIndexTest` in IntelliJ worked, as did both approaches from the command-line).\r\n\r\nI've thought about ways to make our tests robusts to the asynchronous nature of the SparkListenerEvents working their way through the SparkLIstenerBus to the TestSparkListener, but don't see any obvious/satisfying solutions.\r\n\r\nA better approach would be to reach in to the SparkContext / DAGScheduler and pull this information out directly; I will send a PR that does this shortly, and moves extant tests over."},"comment":{"url":"https://api.github.com/repos/hammerlab/magic-rdds/issues/comments/241287116","html_url":"https://github.com/hammerlab/magic-rdds/issues/18#issuecomment-241287116","issue_url":"https://api.github.com/repos/hammerlab/magic-rdds/issues/18","id":241287116,"user":{"login":"ryan-williams","id":465045,"avatar_url":"https://avatars.githubusercontent.com/u/465045?v=3","gravatar_id":"","url":"https://api.github.com/users/ryan-williams","html_url":"https://github.com/ryan-williams","followers_url":"https://api.github.com/users/ryan-williams/followers","following_url":"https://api.github.com/users/ryan-williams/following{/other_user}","gists_url":"https://api.github.com/users/ryan-williams/gists{/gist_id}","starred_url":"https://api.github.com/users/ryan-williams/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryan-williams/subscriptions","organizations_url":"https://api.github.com/users/ryan-williams/orgs","repos_url":"https://api.github.com/users/ryan-williams/repos","events_url":"https://api.github.com/users/ryan-williams/events{/privacy}","received_events_url":"https://api.github.com/users/ryan-williams/received_events","type":"User","site_admin":false},"created_at":"2016-08-21T22:44:40Z","updated_at":"2016-08-21T22:44:40Z","body":"Worked around and documented by #19, not really interested in fixing this directly atm so going to close."}}
{"actor":{"display_login":"ryan-williams"}}