{"action":"created","issue":{"url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/189","repository_url":"https://api.github.com/repos/yahoo/CaffeOnSpark","labels_url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/189/labels{/name}","comments_url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/189/comments","events_url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/189/events","html_url":"https://github.com/yahoo/CaffeOnSpark/issues/189","id":189957259,"number":189,"title":"Job stuck in reduce phase","user":{"login":"mouendless","id":10915169,"avatar_url":"https://avatars.githubusercontent.com/u/10915169?v=3","gravatar_id":"","url":"https://api.github.com/users/mouendless","html_url":"https://github.com/mouendless","followers_url":"https://api.github.com/users/mouendless/followers","following_url":"https://api.github.com/users/mouendless/following{/other_user}","gists_url":"https://api.github.com/users/mouendless/gists{/gist_id}","starred_url":"https://api.github.com/users/mouendless/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mouendless/subscriptions","organizations_url":"https://api.github.com/users/mouendless/orgs","repos_url":"https://api.github.com/users/mouendless/repos","events_url":"https://api.github.com/users/mouendless/events{/privacy}","received_events_url":"https://api.github.com/users/mouendless/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"mriduljain","id":2134869,"avatar_url":"https://avatars.githubusercontent.com/u/2134869?v=3","gravatar_id":"","url":"https://api.github.com/users/mriduljain","html_url":"https://github.com/mriduljain","followers_url":"https://api.github.com/users/mriduljain/followers","following_url":"https://api.github.com/users/mriduljain/following{/other_user}","gists_url":"https://api.github.com/users/mriduljain/gists{/gist_id}","starred_url":"https://api.github.com/users/mriduljain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mriduljain/subscriptions","organizations_url":"https://api.github.com/users/mriduljain/orgs","repos_url":"https://api.github.com/users/mriduljain/repos","events_url":"https://api.github.com/users/mriduljain/events{/privacy}","received_events_url":"https://api.github.com/users/mriduljain/received_events","type":"User","site_admin":false},"assignees":[{"login":"mriduljain","id":2134869,"avatar_url":"https://avatars.githubusercontent.com/u/2134869?v=3","gravatar_id":"","url":"https://api.github.com/users/mriduljain","html_url":"https://github.com/mriduljain","followers_url":"https://api.github.com/users/mriduljain/followers","following_url":"https://api.github.com/users/mriduljain/following{/other_user}","gists_url":"https://api.github.com/users/mriduljain/gists{/gist_id}","starred_url":"https://api.github.com/users/mriduljain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mriduljain/subscriptions","organizations_url":"https://api.github.com/users/mriduljain/orgs","repos_url":"https://api.github.com/users/mriduljain/repos","events_url":"https://api.github.com/users/mriduljain/events{/privacy}","received_events_url":"https://api.github.com/users/mriduljain/received_events","type":"User","site_admin":false}],"milestone":null,"comments":15,"created_at":"2016-11-17T06:58:39Z","updated_at":"2016-11-30T06:58:29Z","closed_at":null,"body":"I'm running mnist data but get some problem.\r\nMy cluster has one master node and 6 data nodes, each data node with 128GB memory, and caffeOnSpark is running under CPU mode.\r\nThe job runs well for executor = 1, but it would get stuck in the reduce phase if I increase executor number, as shown below\r\n![stuck_reduce](https://cloud.githubusercontent.com/assets/10915169/20379022/cf63bd50-acd4-11e6-8a19-73784edfa7ad.jpg)\r\n\r\nFor executor = 1 the UI turn out like this:\r\n![complete_reduce](https://cloud.githubusercontent.com/assets/10915169/20379930/bf73ad9c-acd9-11e6-83ea-c5deff382ef3.jpg)\r\n\r\nIt finished success and show me some results, but the reduces are skipped. \r\n\r\nThe log where the program got stuck was like below\r\n```\r\nI1116 14:51:57.595015 12024 layer_factory.hpp:77] Creating layer data\r\nI1116 14:51:57.595031 12024 net.cpp:100] Creating Layer data\r\nI1116 14:51:57.595053 12024 net.cpp:408] data -> data\r\nI1116 14:51:57.595067 12024 net.cpp:408] data -> label\r\nI1116 14:51:57.595149 12024 net.cpp:150] Setting up data\r\nI1116 14:51:57.595165 12024 net.cpp:157] Top shape: 100 1 28 28 (78400)\r\nI1116 14:51:57.595181 12024 net.cpp:157] Top shape: 100 (100)\r\nI1116 14:51:57.595190 12024 net.cpp:165] Memory required for data: 314000\r\nI1116 14:51:57.595197 12024 layer_factory.hpp:77] Creating layer label_data_1_split\r\nI1116 14:51:57.595208 12024 net.cpp:100] Creating Layer label_data_1_split\r\nI1116 14:51:57.595216 12024 net.cpp:434] label_data_1_split <- label\r\nI1116 14:51:57.595227 12024 net.cpp:408] label_data_1_split -> label_data_1_split_0\r\nI1116 14:51:57.595239 12024 net.cpp:408] label_data_1_split -> label_data_1_split_1\r\nI1116 14:51:57.595252 12024 net.cpp:150] Setting up label_data_1_split\r\nI1116 14:51:57.595263 12024 net.cpp:157] Top shape: 100 (100)\r\nI1116 14:51:57.595283 12024 net.cpp:157] Top shape: 100 (100)\r\nI1116 14:51:57.595291 12024 net.cpp:165] Memory required for data: 314800\r\nI1116 14:51:57.595299 12024 layer_factory.hpp:77] Creating layer conv1\r\nI1116 14:51:57.595315 12024 net.cpp:100] Creating Layer conv1\r\nI1116 14:51:57.595324 12024 net.cpp:434] conv1 <- data\r\nI1116 14:51:57.595351 12024 net.cpp:408] conv1 -> conv1\r\nI1116 14:51:57.595386 12024 net.cpp:150] Setting up conv1\r\nI1116 14:51:57.595397 12024 net.cpp:157] Top shape: 100 20 24 24 (1152000)\r\nI1116 14:51:57.595404 12024 net.cpp:165] Memory required for data: 4922800\r\nI1116 14:51:57.595418 12024 layer_factory.hpp:77] Creating layer pool1\r\nI1116 14:51:57.595429 12024 net.cpp:100] Creating Layer pool1\r\nI1116 14:51:57.595437 12024 net.cpp:434] pool1 <- conv1\r\nI1116 14:51:57.595446 12024 net.cpp:408] pool1 -> pool1\r\nI1116 14:51:57.595460 12024 net.cpp:150] Setting up pool1\r\nI1116 14:51:57.595470 12024 net.cpp:157] Top shape: 100 20 12 12 (288000)\r\nI1116 14:51:57.595477 12024 net.cpp:165] Memory required for data: 6074800\r\nI1116 14:51:57.595484 12024 layer_factory.hpp:77] Creating layer conv2\r\nI1116 14:51:57.595497 12024 net.cpp:100] Creating Layer conv2\r\nI1116 14:51:57.595505 12024 net.cpp:434] conv2 <- pool1\r\nI1116 14:51:57.595515 12024 net.cpp:408] conv2 -> conv2\r\nI1116 14:51:57.595755 12024 net.cpp:150] Setting up conv2\r\nI1116 14:51:57.595768 12024 net.cpp:157] Top shape: 100 50 8 8 (320000)\r\nI1116 14:51:57.595777 12024 net.cpp:165] Memory required for data: 7354800\r\nI1116 14:51:57.595789 12024 layer_factory.hpp:77] Creating layer pool2\r\nI1116 14:51:57.595799 12024 net.cpp:100] Creating Layer pool2\r\nI1116 14:51:57.595808 12024 net.cpp:434] pool2 <- conv2\r\nI1116 14:51:57.595816 12024 net.cpp:408] pool2 -> pool2\r\nI1116 14:51:57.595830 12024 net.cpp:150] Setting up pool2\r\nI1116 14:51:57.595840 12024 net.cpp:157] Top shape: 100 50 4 4 (80000)\r\nI1116 14:51:57.595849 12024 net.cpp:165] Memory required for data: 7674800\r\nI1116 14:51:57.595855 12024 layer_factory.hpp:77] Creating layer ip1\r\nI1116 14:51:57.595866 12024 net.cpp:100] Creating Layer ip1\r\nI1116 14:51:57.595875 12024 net.cpp:434] ip1 <- pool2\r\nI1116 14:51:57.595887 12024 net.cpp:408] ip1 -> ip1\r\nI1116 14:51:57.600900 12024 net.cpp:150] Setting up ip1\r\nI1116 14:51:57.600922 12024 net.cpp:157] Top shape: 100 500 (50000)\r\nI1116 14:51:57.600930 12024 net.cpp:165] Memory required for data: 7874800\r\nI1116 14:51:57.600945 12024 layer_factory.hpp:77] Creating layer relu1\r\nI1116 14:51:57.600957 12024 net.cpp:100] Creating Layer relu1\r\nI1116 14:51:57.600966 12024 net.cpp:434] relu1 <- ip1\r\nI1116 14:51:57.600976 12024 net.cpp:395] relu1 -> ip1 (in-place)\r\nI1116 14:51:57.600988 12024 net.cpp:150] Setting up relu1\r\nI1116 14:51:57.600997 12024 net.cpp:157] Top shape: 100 500 (50000)\r\nI1116 14:51:57.601004 12024 net.cpp:165] Memory required for data: 8074800\r\nI1116 14:51:57.601012 12024 layer_factory.hpp:77] Creating layer ip2\r\nI1116 14:51:57.601024 12024 net.cpp:100] Creating Layer ip2\r\nI1116 14:51:57.601032 12024 net.cpp:434] ip2 <- ip1\r\nI1116 14:51:57.601043 12024 net.cpp:408] ip2 -> ip2\r\nI1116 14:51:57.601111 12024 net.cpp:150] Setting up ip2\r\nI1116 14:51:57.601122 12024 net.cpp:157] Top shape: 100 10 (1000)\r\nI1116 14:51:57.601130 12024 net.cpp:165] Memory required for data: 8078800\r\nI1116 14:51:57.601140 12024 layer_factory.hpp:77] Creating layer ip2_ip2_0_split\r\nI1116 14:51:57.601151 12024 net.cpp:100] Creating Layer ip2_ip2_0_split\r\nI1116 14:51:57.601166 12024 net.cpp:434] ip2_ip2_0_split <- ip2\r\nI1116 14:51:57.601177 12024 net.cpp:408] ip2_ip2_0_split -> ip2_ip2_0_split_0\r\nI1116 14:51:57.601189 12024 net.cpp:408] ip2_ip2_0_split -> ip2_ip2_0_split_1\r\nI1116 14:51:57.601202 12024 net.cpp:150] Setting up ip2_ip2_0_split\r\nI1116 14:51:57.601212 12024 net.cpp:157] Top shape: 100 10 (1000)\r\nI1116 14:51:57.601222 12024 net.cpp:157] Top shape: 100 10 (1000)\r\nI1116 14:51:57.601228 12024 net.cpp:165] Memory required for data: 8086800\r\nI1116 14:51:57.601235 12024 layer_factory.hpp:77] Creating layer accuracy\r\nI1116 14:51:57.601246 12024 net.cpp:100] Creating Layer accuracy\r\nI1116 14:51:57.601254 12024 net.cpp:434] accuracy <- ip2_ip2_0_split_0\r\nI1116 14:51:57.601263 12024 net.cpp:434] accuracy <- label_data_1_split_0\r\nI1116 14:51:57.601274 12024 net.cpp:408] accuracy -> accuracy\r\nI1116 14:51:57.601286 12024 net.cpp:150] Setting up accuracy\r\nI1116 14:51:57.601296 12024 net.cpp:157] Top shape: (1)\r\nI1116 14:51:57.601303 12024 net.cpp:165] Memory required for data: 8086804\r\nI1116 14:51:57.601311 12024 layer_factory.hpp:77] Creating layer loss\r\nI1116 14:51:57.601321 12024 net.cpp:100] Creating Layer loss\r\nI1116 14:51:57.601330 12024 net.cpp:434] loss <- ip2_ip2_0_split_1\r\nI1116 14:51:57.601337 12024 net.cpp:434] loss <- label_data_1_split_1\r\nI1116 14:51:57.601347 12024 net.cpp:408] loss -> loss\r\nI1116 14:51:57.601359 12024 layer_factory.hpp:77] Creating layer loss\r\nI1116 14:51:57.601379 12024 net.cpp:150] Setting up loss\r\nI1116 14:51:57.601389 12024 net.cpp:157] Top shape: (1)\r\nI1116 14:51:57.601397 12024 net.cpp:160]     with loss weight 1\r\nI1116 14:51:57.601408 12024 net.cpp:165] Memory required for data: 8086808\r\nI1116 14:51:57.601415 12024 net.cpp:226] loss needs backward computation.\r\nI1116 14:51:57.601424 12024 net.cpp:228] accuracy does not need backward computation.\r\nI1116 14:51:57.601433 12024 net.cpp:226] ip2_ip2_0_split needs backward computation.\r\nI1116 14:51:57.601439 12024 net.cpp:226] ip2 needs backward computation.\r\nI1116 14:51:57.601447 12024 net.cpp:226] relu1 needs backward computation.\r\nI1116 14:51:57.601454 12024 net.cpp:226] ip1 needs backward computation.\r\nI1116 14:51:57.601461 12024 net.cpp:226] pool2 needs backward computation.\r\nI1116 14:51:57.601469 12024 net.cpp:226] conv2 needs backward computation.\r\nI1116 14:51:57.601475 12024 net.cpp:226] pool1 needs backward computation.\r\nI1116 14:51:57.601483 12024 net.cpp:226] conv1 needs backward computation.\r\nI1116 14:51:57.601491 12024 net.cpp:228] label_data_1_split does not need backward computation.\r\nI1116 14:51:57.601500 12024 net.cpp:228] data does not need backward computation.\r\nI1116 14:51:57.601506 12024 net.cpp:270] This network produces output accuracy\r\nI1116 14:51:57.601514 12024 net.cpp:270] This network produces output loss\r\nI1116 14:51:57.601531 12024 net.cpp:283] Network initialization done.\r\nI1116 14:51:57.601585 12024 solver.cpp:60] Solver scaffolding done.\r\nI1116 14:51:57.601702 12024 socket.cpp:221] Waiting for valid port [0]\r\nI1116 14:51:57.601812 12041 socket.cpp:162] Assigned socket server port [50184]\r\nI1116 14:51:57.602773 12041 socket.cpp:175] Socket Server ready [0.0.0.0]\r\nI1116 14:51:57.611819 12024 socket.cpp:221] Waiting for valid port [50184]\r\nI1116 14:51:57.611841 12024 socket.cpp:229] Valid port found [50184]\r\nI1116 14:51:57.611863 12024 CaffeNet.cpp:262] Socket adapter: longzhou-hdp1.lz.dscc:50184\r\nI1116 14:51:57.612177 12024 CaffeNet.cpp:402] 0-th Socket addr: \r\nI1116 14:51:57.612201 12024 CaffeNet.cpp:402] 1-th Socket addr: longzhou-hdp1.lz.dscc:50184\r\nI1116 14:51:57.612211 12024 CaffeNet.cpp:402] 2-th Socket addr: longzhou-hdp1.lz.dscc:50184\r\nI1116 14:51:57.612221 12024 CaffeNet.cpp:402] 3-th Socket addr: longzhou-hdp1.lz.dscc:50184\r\nI1116 14:51:57.612236 12024 JniCaffeNet.cpp:145] 0-th local addr: \r\nI1116 14:51:57.612246 12024 JniCaffeNet.cpp:145] 1-th local addr: longzhou-hdp1.lz.dscc:50184\r\nI1116 14:51:57.612253 12024 JniCaffeNet.cpp:145] 2-th local addr: longzhou-hdp1.lz.dscc:50184\r\nI1116 14:51:57.612262 12024 JniCaffeNet.cpp:145] 3-th local addr: longzhou-hdp1.lz.dscc:50184\r\n16/11/16 14:51:57 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1018 bytes result sent to driver\r\n16/11/16 14:51:58 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 14\r\n16/11/16 14:51:58 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14)\r\n16/11/16 14:51:58 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 4\r\n16/11/16 14:51:58 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1584.0 B, free 18.7 KB)\r\n16/11/16 14:51:58 INFO broadcast.TorrentBroadcast: Reading broadcast variable 4 took 12 ms\r\n16/11/16 14:51:58 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 2.6 KB, free 21.3 KB)\r\n16/11/16 14:51:58 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 3\r\n16/11/16 14:51:58 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 170.0 B, free 21.4 KB)\r\n16/11/16 14:51:58 INFO broadcast.TorrentBroadcast: Reading broadcast variable 3 took 7 ms\r\n16/11/16 14:51:58 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 1536.0 B, free 22.9 KB)\r\nI1116 14:51:58.208997 12024 common.cpp:61] 0-th string is NULL\r\nI1116 14:51:58.209087 12024 socket.cpp:257] Trying to connect with ...[longzhou-hdp3.lz.dscc:56137]\r\nI1116 14:51:58.209530 12024 socket.cpp:321] Connected to server [longzhou-hdp3.lz.dscc:56137] with client_fd [374]\r\nI1116 14:51:58.228672 12041 socket.cpp:188] Accepted the connection from client [longzhou-hdp2.lz.dscc]\r\nI1116 14:51:58.230128 12041 socket.cpp:188] Accepted the connection from client [longzhou-hdp4.lz.dscc]\r\nI1116 14:51:58.276432 12041 socket.cpp:188] Accepted the connection from client [longzhou-hdp3.lz.dscc]\r\nI1116 14:51:59.209713 12024 socket.cpp:257] Trying to connect with ...[longzhou-hdp2.lz.dscc:51049]\r\nI1116 14:51:59.210247 12024 socket.cpp:321] Connected to server [longzhou-hdp2.lz.dscc:51049] with client_fd [378]\r\nI1116 14:52:00.210403 12024 socket.cpp:257] Trying to connect with ...[longzhou-hdp4.lz.dscc:52620]\r\nI1116 14:52:00.210934 12024 socket.cpp:321] Connected to server [longzhou-hdp4.lz.dscc:52620] with client_fd [294]\r\n16/11/16 14:52:01 INFO caffe.CaffeProcessor: Interleave enabled\r\nI1116 14:52:01.218786 12048 CaffeNet.cpp:633] Interleaved\r\nI1116 14:52:01.218842 12048 MemoryInputAdapter.cpp:15] MemoryInputAdapter is used\r\nI1116 14:52:01.218858 12048 MemoryInputAdapter.cpp:15] MemoryInputAdapter is used\r\n16/11/16 14:52:01 INFO caffe.CaffeProcessor: Start transformer for train in CaffeProcessor StartThreads\r\n16/11/16 14:52:01 INFO caffe.CaffeProcessor: Start transformer for validation in CaffeProcessor StartThreads\r\n16/11/16 14:52:01 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 908 bytes result sent to driver\r\n16/11/16 14:52:01 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 16\r\n16/11/16 14:52:01 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 16)\r\n16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 5\r\n16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.1 KB, free 25.0 KB)\r\n16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Reading broadcast variable 5 took 10 ms\r\n16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.3 KB, free 28.3 KB)\r\n16/11/16 14:52:01 INFO storage.BlockManager: Found block rdd_1_0 locally\r\n16/11/16 14:52:01 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 16). 2830 bytes result sent to driver\r\n16/11/16 14:52:01 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 17\r\n16/11/16 14:52:01 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 17)\r\n16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 6\r\n16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2.0 KB, free 30.3 KB)\r\n16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Reading broadcast variable 6 took 10 ms\r\n16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.1 KB, free 33.5 KB)\r\n16/11/16 14:52:01 INFO storage.BlockManager: Found block rdd_1_0 locally\r\n16/11/16 14:52:01 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 17). 2016 bytes result sent to driver\r\n16/11/16 14:52:01 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 21\r\n16/11/16 14:52:01 INFO executor.Executor: Running task 1.0 in stage 6.0 (TID 21)\r\n16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 7\r\n16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.0 KB, free 35.5 KB)\r\n16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Reading broadcast variable 7 took 10 ms\r\n16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 3.1 KB, free 38.6 KB)\r\n16/11/16 14:52:01 INFO storage.BlockManager: Found block rdd_3_1 locally\r\n16/11/16 14:52:01 INFO executor.Executor: Finished task 1.0 in stage 6.0 (TID 21). 2015 bytes result sent to driver\r\n16/11/16 14:52:01 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 25\r\n16/11/16 14:52:01 INFO executor.Executor: Running task 2.0 in stage 7.0 (TID 25)\r\n16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 8\r\n16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 1309.0 B, free 39.9 KB)\r\n16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Reading broadcast variable 8 took 9 ms\r\n16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 2.0 KB, free 42.0 KB)\r\n16/11/16 14:52:01 INFO executor.Executor: Finished task 2.0 in stage 7.0 (TID 25). 938 bytes result sent to driver\r\n16/11/16 14:52:02 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 29\r\n16/11/16 14:52:02 INFO executor.Executor: Running task 0.0 in stage 8.0 (TID 29)\r\n16/11/16 14:52:02 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 9\r\n16/11/16 14:52:02 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 2.5 KB, free 44.5 KB)\r\n16/11/16 14:52:02 INFO broadcast.TorrentBroadcast: Reading broadcast variable 9 took 10 ms\r\n16/11/16 14:52:02 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 4.1 KB, free 48.6 KB)\r\n16/11/16 14:52:02 INFO storage.BlockManager: Found block rdd_1_0 locally\r\n16/11/16 14:52:02 INFO executor.Executor: Finished task 0.0 in stage 8.0 (TID 29). 2211 bytes result sent to driver\r\n16/11/16 14:52:02 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 31\r\n16/11/16 14:52:02 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 31)\r\n16/11/16 14:52:02 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 33\r\n16/11/16 14:52:02 INFO executor.Executor: Running task 1.0 in stage 9.0 (TID 33)\r\n16/11/16 14:52:02 INFO spark.MapOutputTrackerWorker: Updating epoch to 1 and clearing cache\r\n16/11/16 14:52:02 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 10\r\n16/11/16 14:52:02 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.6 KB, free 51.2 KB)\r\n16/11/16 14:52:02 INFO broadcast.TorrentBroadcast: Reading broadcast variable 10 took 8 ms\r\n16/11/16 14:52:02 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.4 KB, free 55.5 KB)\r\n16/11/16 14:52:02 INFO spark.MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them\r\n16/11/16 14:52:02 INFO spark.MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them\r\n16/11/16 14:52:02 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@192.168.19.110:2671)\r\n16/11/16 14:52:02 INFO spark.MapOutputTrackerWorker: Got the output locations\r\n16/11/16 14:52:02 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks\r\n16/11/16 14:52:02 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks\r\n16/11/16 14:52:02 INFO storage.ShuffleBlockFetcherIterator: Started 3 remote fetches in 15 ms\r\n16/11/16 14:52:02 INFO storage.ShuffleBlockFetcherIterator: Started 3 remote fetches in 16 ms\r\n```\r\n\r\nAnd no more log for this container after it said \"Started remote fetches\"\r\n\r\nStill my cluster runs well with other spark programs with reduce phase (not skipped)."},"comment":{"url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/comments/263798458","html_url":"https://github.com/yahoo/CaffeOnSpark/issues/189#issuecomment-263798458","issue_url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/189","id":263798458,"user":{"login":"mriduljain","id":2134869,"avatar_url":"https://avatars.githubusercontent.com/u/2134869?v=3","gravatar_id":"","url":"https://api.github.com/users/mriduljain","html_url":"https://github.com/mriduljain","followers_url":"https://api.github.com/users/mriduljain/followers","following_url":"https://api.github.com/users/mriduljain/following{/other_user}","gists_url":"https://api.github.com/users/mriduljain/gists{/gist_id}","starred_url":"https://api.github.com/users/mriduljain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mriduljain/subscriptions","organizations_url":"https://api.github.com/users/mriduljain/orgs","repos_url":"https://api.github.com/users/mriduljain/repos","events_url":"https://api.github.com/users/mriduljain/events{/privacy}","received_events_url":"https://api.github.com/users/mriduljain/received_events","type":"User","site_admin":false},"created_at":"2016-11-30T06:58:29Z","updated_at":"2016-11-30T06:58:29Z","body":"During any kind of training we need to make sure that every executor has only one task. executor-core takes care if this as it is essentially the max number of tasks per executor. If we have more than one task per executor, then we get into a situation where partition allocation happens based on whichever tasks are active at a particular time. So suppose spark needs to schedule 4 partitions and we have executor-core as 4; it can schedule all 4 partitions in 4 tasks(if they are ready/active) for an executor , leaving the other executors starving for partitions. As per the design of COS, all executors need to process their partitions in parallel and in sync with others. More tasks per executor creates an imbalance."}}
{"actor":{"display_login":"mriduljain"}}