{"action":"closed","issue":{"url":"https://api.github.com/repos/tensorpack/tensorpack/issues/864","repository_url":"https://api.github.com/repos/tensorpack/tensorpack","labels_url":"https://api.github.com/repos/tensorpack/tensorpack/issues/864/labels{/name}","comments_url":"https://api.github.com/repos/tensorpack/tensorpack/issues/864/comments","events_url":"https://api.github.com/repos/tensorpack/tensorpack/issues/864/events","html_url":"https://github.com/tensorpack/tensorpack/issues/864","id":350658494,"node_id":"MDU6SXNzdWUzNTA2NTg0OTQ=","number":864,"title":"[Faster RCNN]SyncMultiGPUTrainerReplicated:8 GPU training much faster than 1 GPU?","user":{"login":"meteortwinkle","id":6299886,"node_id":"MDQ6VXNlcjYyOTk4ODY=","avatar_url":"https://avatars3.githubusercontent.com/u/6299886?v=4","gravatar_id":"","url":"https://api.github.com/users/meteortwinkle","html_url":"https://github.com/meteortwinkle","followers_url":"https://api.github.com/users/meteortwinkle/followers","following_url":"https://api.github.com/users/meteortwinkle/following{/other_user}","gists_url":"https://api.github.com/users/meteortwinkle/gists{/gist_id}","starred_url":"https://api.github.com/users/meteortwinkle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/meteortwinkle/subscriptions","organizations_url":"https://api.github.com/users/meteortwinkle/orgs","repos_url":"https://api.github.com/users/meteortwinkle/repos","events_url":"https://api.github.com/users/meteortwinkle/events{/privacy}","received_events_url":"https://api.github.com/users/meteortwinkle/received_events","type":"User","site_admin":false},"labels":[{"id":676134026,"node_id":"MDU6TGFiZWw2NzYxMzQwMjY=","url":"https://api.github.com/repos/tensorpack/tensorpack/labels/examples","name":"examples","color":"e99695","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-08-15T01:51:51Z","updated_at":"2018-08-22T18:48:16Z","closed_at":"2018-08-22T18:48:16Z","author_association":"NONE","body":"1. What did:\r\nTraining command:\r\nCUDA_VISIBLE_DEVICES='0' python /mydata/frcnn_train/tensorpack/examples/FasterRCNN/train.py --load /data_one/data/FasterRCNN/COCO-R50C4-MaskRCNN-Standard.npz --config MODE_MASK=True MODE_FPN=False  DATA.BASEDIR=/data_one/data/COCO/2017/ DATA.VAL='val2017' \r\n\r\nCUDA_VISIBLE_DEVICES='0,1,2,3,4,5,6,7' python /mydata/frcnn_train/tensorpack/examples/FasterRCNN/train.py --load /data_one/data/FasterRCNN/COCO-R50C4-MaskRCNN-Standard.npz --config MODE_MASK=True MODE_FPN=False  DATA.BASEDIR=/data_one/data/COCO/2017/ DATA.VAL='val2017' \r\n\r\n2、What observed：\r\nepoch | step | horovod 1-1| multi-gpu 1-1| multi-gpu 1-8 | horovod 1-8\r\n0 | 0 | 5 minutes 53 seconds | 6 minutes 5 seconds | 6 minutes 2 seconds | 14 minutes 38 seconds\r\n1 | 500 | 4 minutes 32 seconds | 4 minutes 20 seconds | 2 minutes 42 seconds | 11 minutes 26 seconds\r\n2 | 1000 | 4 minutes 26 seconds | 4 minutes 18 seconds | 2 minutes 19 seconds | 9 minutes 46 seconds\r\n3 | 1500 | 4 minutes 4 seconds | 4 minutes 14 seconds | 2 minutes 3 seconds | 8 minutes 39 seconds\r\n4 | 2000 | 3 minutes 57 seconds | 3 minutes 53 seconds | 1 minute 48 seconds | 7 minutes 40 seconds\r\n5 | 2500 | 3 minutes 38 seconds | 3 minutes 46 seconds | 1 minute 49 seconds | 6 minutes 28 seconds\r\n6 | 3000 | 3 minutes 42 seconds | 3 minutes 41 seconds | 1 minute 36 seconds | 6 minutes 15 seconds\r\n7 | 3500 | 3 minutes 38 seconds | 3 minutes 32 seconds | 1 minute 38 seconds | 5 minutes 39 seconds\r\n8 | 4000 | 3 minutes 32 seconds | 3 minutes 30 seconds | 1 minute 35 seconds | 5 minutes 12 seconds\r\n9 | 4500 | 3 minutes 33 seconds | 3 minutes 36 seconds | 1 minute 29 seconds | 5 minutes 16 seconds\r\n10 | 5000 | 3 minutes 28 seconds | 3 minutes 26 seconds | 1 minute 29 seconds | 5 minutes 19 seconds\r\n11 | 5500 | 3 minutes 32 seconds | 3 minutes 22 seconds | 1 minute 30 seconds | 4 minutes 45 seconds\r\n12 | 6000 | 3 minutes 29 seconds | 3 minutes 21 seconds | 1 minute 29 seconds | 4 minutes 46 seconds\r\n13 | 6500 | 3 minutes 24 seconds | 3 minutes 25 seconds | 1 minute 26 seconds | 4 minutes 39 seconds\r\n14 | 7000 | 3 minutes 25 seconds | 3 minutes 26 seconds | 1 minute 27 seconds | 4 minutes 40 seconds\r\n15 | 7500 | 3 minutes 31 seconds | 3 minutes 22 seconds | 1 minute 28 seconds | 4 minutes 39 seconds\r\n16 | 8000 | 3 minutes 22 seconds | 3 minutes 25 seconds | 1 minute 26 seconds | 4 minutes 35 seconds\r\n17 | 8500 | 3 minutes 24 seconds | 3 minutes 25 seconds | 1 minute 28 seconds | 4 minutes 23 seconds\r\n18 | 9000 | 3 minutes 21 seconds | 3 minutes 25 seconds | 1 minute 26 seconds | 4 minutes 27 seconds\r\n19 | 9500 | 3 minutes 21 seconds | 3 minutes 25 seconds | 1 minute 27 seconds | 4 minutes 34 seconds\r\n20 | 10000 | 3 minutes 25 seconds | 3 minutes 24 seconds | 1 minute 27 seconds | 4 minutes 24 seconds\r\n21 | 10500 | 3 minutes 20 seconds | 3 minutes 25 seconds | 1 minute 24 seconds | 4 minutes 25 seconds\r\n22 | 11000 | 3 minutes 22 seconds | 3 minutes 20 seconds | 1 minute 26 seconds | 4 minutes 15 seconds\r\n23 | 11500 | 3 minutes 21 seconds | 3 minutes 20 seconds | 1 minute 26 seconds | 4 minutes 20 seconds\r\n24 | 12000 | 3 minutes 20 seconds | 3 minutes 19 seconds | 1 minute 26 seconds | 4 minutes 16 seconds\r\n25 | 12500 | 3 minutes 23 seconds | 3 minutes 26 seconds | 1 minute 26 seconds | 4 minutes 18 seconds\r\n26 | 13000 | 3 minutes 21 seconds | 3 minutes 19 seconds | 1 minute 25 seconds | 4 minutes 19 seconds\r\n27 | 13500 | 3 minutes 19 seconds | 3 minutes 22 seconds | 1 minute 23 seconds | 4 minutes 14 seconds\r\n28 | 14000 | 3 minutes 17 seconds | 3 minutes 23 seconds | 1 minute 23 seconds | 4 minutes 16 seconds\r\n29 | 14500 | 3 minutes 18 seconds | 3 minutes 23 seconds | 1 minute 24 seconds | 4 minutes 15 seconds\r\n30 | 15000 | 3 minutes 16 seconds | 3 minutes 22 seconds | 1 minute 25 seconds | 4 minutes 10 seconds\r\n31 | 15500 | 3 minutes 21 seconds | 3 minutes 23 seconds | 1 minute 22 seconds | 4 minutes 11 seconds\r\n32 | 16000 | 3 minutes 16 seconds | 3 minutes 22 seconds | 1 minute 22 seconds | 4 minutes 14 seconds\r\n33 | 16500 | 3 minutes 16 seconds | 3 minutes 18 seconds | 1 minute 23 seconds | 4 minutes 10 seconds\r\n34 | 17000 | 3 minutes 19 seconds | 3 minutes 20 seconds | 1 minute 22 seconds | 4 minutes 11 seconds\r\n35 | 17500 | 3 minutes 19 seconds | 3 minutes 18 seconds | 1 minute 23 seconds | 4 minutes 6 seconds\r\n36 | 18000 | 3 minutes 20 seconds | 3 minutes 17 seconds | 1 minute 24 seconds | 4 minutes 8 seconds\r\n37 | 18500 | 3 minutes 20 seconds | 3 minutes 21 seconds | 1 minute 23 seconds | 4 minutes 12 seconds\r\n38 | 19000 | 3 minutes 21 seconds | 3 minutes 24 seconds | 1 minute 22 seconds | 4 minutes 13 seconds\r\n39 | 19500 | 3 minutes 18 seconds | 3 minutes 18 seconds | 1 minute 22 seconds | 4 minutes 9 seconds\r\n40 | 20000 | 3 minutes 18 seconds | 3 minutes 21 seconds | 1 minute 22 seconds | 4 minutes 13 seconds\r\n41 | 20500 | 3 minutes 21 seconds | 3 minutes 18 seconds | 1 minute 22 seconds | 4 minutes 9 seconds\r\n42 | 21000 | 3 minutes 19 seconds | 3 minutes 18 seconds | 1 minute 23 seconds | 4 minutes 12 seconds\r\n43 | 21500 | 3 minutes 20 seconds | 3 minutes 21 seconds | 1 minute 23 seconds | 4 minutes 15 seconds\r\n44 | 22000 | 3 minutes 20 seconds | 3 minutes 18 seconds | 1 minute 22 seconds | 4 minutes 12 seconds\r\n45 | 22500 | 3 minutes 20 seconds | 3 minutes 19 seconds | 1 minute 23 seconds | 4 minutes 9 seconds\r\n46 | 23000 | 3 minutes 22 seconds | 3 minutes 19 seconds | 1 minute 23 seconds | 4 minutes 12 seconds\r\n47 | 23500 | 3 minutes 17 seconds | 3 minutes 19 seconds | 1 minute 22 seconds | 4 minutes 11 seconds\r\n48 | 24000 | 3 minutes 19 seconds | 3 minutes 19 seconds | 1 minute 22 seconds | 4 minutes 8 seconds\r\n49 | 24500 | 3 minutes 20 seconds | 3 minutes 19 seconds | 1 minute 24 seconds | 4 minutes 7 seconds\r\n50 | 25000 | 3 minutes 15 seconds | 3 minutes 21 seconds | 1 minute 23 seconds | 4 minutes 9 seconds\r\n51 | 25500 | 3 minutes 18 seconds | 3 minutes 20 seconds | 1 minute 22 seconds | 4 minutes 7 seconds\r\n52 | 26000 | 3 minutes 20 seconds | 3 minutes 19 seconds | 1 minute 23 seconds | 4 minutes 9 seconds\r\n53 | 26500 | 3 minutes 19 seconds | 3 minutes 19 seconds | 1 minute 23 seconds | 4 minutes 20 seconds\r\n54 | 27000 | 3 minutes 19 seconds | 3 minutes 22 seconds | 1 minute 24 seconds | 4 minutes 11 seconds\r\n55 | 27500 | 3 minutes 17 seconds | 3 minutes 20 seconds | 1 minute 22 seconds | 4 minutes 9 seconds\r\n56 | 28000 | 3 minutes 17 seconds | 3 minutes 16 seconds | 1 minute 23 seconds | 4 minutes 9 seconds\r\n57 | 28500 | 3 minutes 19 seconds | 3 minutes 18 seconds | 1 minute 21 seconds | 4 minutes 9 seconds\r\n58 | 29000 | 3 minutes 21 seconds | 3 minutes 19 seconds | 1 minute 23 seconds | 4 minutes 10 seconds\r\n59 | 29500 | 3 minutes 20 seconds | 3 minutes 22 seconds | 1 minute 22 seconds | 4 minutes 9 seconds\r\n60 | 30000 | 3 minutes 23 seconds | 3 minutes 20 seconds | 1 minute 23 seconds | 4 minutes 7 seconds\r\n61 | 30500 | 3 minutes 16 seconds | 3 minutes 18 seconds | 1 minute 22 seconds | 4 minutes 11 seconds\r\n62 | 31000 | 3 minutes 17 seconds | 3 minutes 19 seconds | 1 minute 22 seconds | 4 minutes 6 seconds\r\n63 | 31500 | 3 minutes 22 seconds | 3 minutes 19 seconds | 1 minute 23 seconds | 4 minutes 9 seconds\r\n64 | 32000 | 3 minutes 16 seconds | 3 minutes 22 seconds | 1 minute 23 seconds | 4 minutes 7 seconds\r\n65 | 32500 | 3 minutes 17 seconds | 3 minutes 20 seconds | 1 minute 23 seconds | 4 minutes 9 seconds\r\n66 | 33000 | 3 minutes 17 seconds | 3 minutes 21 seconds | 1 minute 22 seconds | 4 minutes 7 seconds\r\n67 | 33500 | 3 minutes 16 seconds | 3 minutes 22 seconds | 1 minute 21 seconds | 4 minutes 8 seconds\r\n68 | 34000 | 3 minutes 18 seconds | 3 minutes 20 seconds | 1 minute 23 seconds | 4 minutes 7 seconds\r\n69 | 34500 | 3 minutes 18 seconds | 3 minutes 18 seconds | 1 minute 23 seconds | 4 minutes 5 seconds\r\n70 | 35000 | 3 minutes 17 seconds | 3 minutes 18 seconds | 1 minute 22 seconds | 4 minutes 6 seconds\r\n71 | 35500 | 3 minutes 19 seconds | 3 minutes 18 seconds | 1 minute 22 seconds | 4 minutes 8 seconds\r\n72 | 36000 | 3 minutes 20 seconds | 3 minutes 17 seconds | 1 minute 21 seconds | 4 minutes 7 seconds\r\n73 | 36500 | 3 minutes 17 seconds | 3 minutes 18 seconds | 1 minute 21 seconds | 4 minutes 12 seconds\r\n74 | 37000 | 3 minutes 21 seconds | 3 minutes 20 seconds | 1 minute 21 seconds | 4 minutes 7 seconds\r\n\r\n3. What  expected：\r\nWhy  was the training  so fast on 8 GPUs，even much faster than on 1 GPU？\r\nAnd, I have also tried the other backbone（ex. default）, it showed that the model trained faster on  2 GPUs(or more)  than on 1 GPU.\r\n\r\n4、Training Enviroment:\r\nV100*8\r\ntf:1.8\r\npython:3.5\r\ntensorpack:0.8.6"}}
{"actor":{"display_login":"ppwwyyxx"}}