{"action":"created","issue":{"url":"https://api.github.com/repos/apache/storm/issues/1528","repository_url":"https://api.github.com/repos/apache/storm","labels_url":"https://api.github.com/repos/apache/storm/issues/1528/labels{/name}","comments_url":"https://api.github.com/repos/apache/storm/issues/1528/comments","events_url":"https://api.github.com/repos/apache/storm/issues/1528/events","html_url":"https://github.com/apache/storm/pull/1528","id":162863717,"number":1528,"title":"STORM-1934 Fix race condition between sync-supervisor and sync-processes (1.x)","user":{"login":"HeartSaVioR","id":1317309,"avatar_url":"https://avatars.githubusercontent.com/u/1317309?v=3","gravatar_id":"","url":"https://api.github.com/users/HeartSaVioR","html_url":"https://github.com/HeartSaVioR","followers_url":"https://api.github.com/users/HeartSaVioR/followers","following_url":"https://api.github.com/users/HeartSaVioR/following{/other_user}","gists_url":"https://api.github.com/users/HeartSaVioR/gists{/gist_id}","starred_url":"https://api.github.com/users/HeartSaVioR/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HeartSaVioR/subscriptions","organizations_url":"https://api.github.com/users/HeartSaVioR/orgs","repos_url":"https://api.github.com/users/HeartSaVioR/repos","events_url":"https://api.github.com/users/HeartSaVioR/events{/privacy}","received_events_url":"https://api.github.com/users/HeartSaVioR/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2016-06-29T08:01:28Z","updated_at":"2016-07-04T06:29:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/storm/pulls/1528","html_url":"https://github.com/apache/storm/pull/1528","diff_url":"https://github.com/apache/storm/pull/1528.diff","patch_url":"https://github.com/apache/storm/pull/1528.patch"},"body":"Please refer [STORM-1933](https://issues.apache.org/jira/browse/STORM-1933) and [STORM-1934](https://issues.apache.org/jira/browse/STORM-1934) to see reason of this issue and concept to fix.\r\n\r\n* sync-supervisor just downloads new topology code and writes new local assignment\r\n  * shutting down workers and removing topology code is moved to sync-processes\r\n* sync-processes does all of jobs based on local assignment and allocated workers\r\n* remove unused / unneeded codes\r\n\r\nPR for master: #1529 \r\n\r\nHere's my test result for this patch:\r\n\r\n* `mvn clean install` 5 times: not met supervisor intermittent failure (STORM-1933)\r\n  * will try more times\r\n* kill worker via `kill`, `kill -9`, `restart worker` from UI: no issue on restarting worker\r\n* rebalance topology to change workers (3 -> 2): to test that new assignment has same worker port but different executors compared to assigned workers\r\n  * worker is recognized as :disallowed, and killed & relaunched\r\n\r\nRebalance test in details:\r\n\r\n- Writing new assignment\r\n```\r\n6701 {:storm-id \"test-topology2-4-1467185073\", :executors ([7 7] [5 5] [3 3] [1 1]), :resources [0.0 0.0 0.0]}, \r\n6702 {:storm-id \"test-topology2-4-1467185073\", :executors ([6 6] [4 4] [2 2]), :resources [0.0 0.0 0.0]}\r\n```\r\n\r\n- Assigned executors:\r\n```\r\n6701 {:storm-id \"test-topology2-4-1467185073\", :executors [[7 7] [5 5] [3 3] [1 1]], :resources #object[org.apache.storm.generated.WorkerResources 0x40c4d31c \"WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0)\"]}, \r\n6702 {:storm-id \"test-topology2-4-1467185073\", :executors [[6 6] [4 4] [2 2]], :resources #object[org.apache.storm.generated.WorkerResources 0x4ba861f4 \"WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0)\"]}}\r\n```\r\n\r\n- Allocated:\r\n```\r\n\"2e9bea10-02b7-4e55-88e7-b194b9917a63\" [:disallowed {:time-secs 1467185407, :storm-id \"test-topology2-4-1467185073\", :executors [[3 3] [6 6] [-1 -1]], :port 6703}], \r\n\"4630c4bf-9786-47ff-9f3b-6b42d9781b9d\" [:disallowed {:time-secs 1467185407, :storm-id \"test-topology2-4-1467185073\", :executors [[7 7] [1 1] [-1 -1] [4 4]], :port 6701}], \r\n\"b9a622d2-5e5b-4311-999c-8c8dd92da6b6\" [:disallowed {:time-secs 1467185406, :storm-id \"test-topology2-4-1467185073\", :executors [[2 2] [-1 -1] [5 5]], :port 6702}]}\r\n```\r\n\r\nNOTE: Due to forward reference, I have to move `sync-processes` to just before `mk-synchronize-supervisor`. Major changes are done in sync-processes so reviewers need to compare before & after manually. Sorry about that.\r\n\r\nSince supervisor.clj is already ported to Java in master branch, I should have time to read ported code, and modify to be in sync.\r\n\r\nPlease review and comment while I'm working against master branch. Thanks!"},"comment":{"url":"https://api.github.com/repos/apache/storm/issues/comments/230214690","html_url":"https://github.com/apache/storm/pull/1528#issuecomment-230214690","issue_url":"https://api.github.com/repos/apache/storm/issues/1528","id":230214690,"user":{"login":"HeartSaVioR","id":1317309,"avatar_url":"https://avatars.githubusercontent.com/u/1317309?v=3","gravatar_id":"","url":"https://api.github.com/users/HeartSaVioR","html_url":"https://github.com/HeartSaVioR","followers_url":"https://api.github.com/users/HeartSaVioR/followers","following_url":"https://api.github.com/users/HeartSaVioR/following{/other_user}","gists_url":"https://api.github.com/users/HeartSaVioR/gists{/gist_id}","starred_url":"https://api.github.com/users/HeartSaVioR/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HeartSaVioR/subscriptions","organizations_url":"https://api.github.com/users/HeartSaVioR/orgs","repos_url":"https://api.github.com/users/HeartSaVioR/repos","events_url":"https://api.github.com/users/HeartSaVioR/events{/privacy}","received_events_url":"https://api.github.com/users/HeartSaVioR/received_events","type":"User","site_admin":false},"created_at":"2016-07-04T06:29:19Z","updated_at":"2016-07-04T06:29:19Z","body":"@harshach \r\nAnd sync-processes (`get-valid-new-worker-ids`) checks topology files before launching workers.\r\nIf there's an issue which corrupts topology code or not having topology code, sync-processes doesn't launch worker, and next runs of sync-supervisor will try to download topology code again."}}
{"actor":{"display_login":"HeartSaVioR"}}