{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/wdt/issues/138","repository_url":"https://api.github.com/repos/facebook/wdt","labels_url":"https://api.github.com/repos/facebook/wdt/issues/138/labels{/name}","comments_url":"https://api.github.com/repos/facebook/wdt/issues/138/comments","events_url":"https://api.github.com/repos/facebook/wdt/issues/138/events","html_url":"https://github.com/facebook/wdt/issues/138","id":194018760,"number":138,"title":"Always fails to transfer the whole files","user":{"login":"nxtreaming","id":3108107,"avatar_url":"https://avatars.githubusercontent.com/u/3108107?v=3","gravatar_id":"","url":"https://api.github.com/users/nxtreaming","html_url":"https://github.com/nxtreaming","followers_url":"https://api.github.com/users/nxtreaming/followers","following_url":"https://api.github.com/users/nxtreaming/following{/other_user}","gists_url":"https://api.github.com/users/nxtreaming/gists{/gist_id}","starred_url":"https://api.github.com/users/nxtreaming/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nxtreaming/subscriptions","organizations_url":"https://api.github.com/users/nxtreaming/orgs","repos_url":"https://api.github.com/users/nxtreaming/repos","events_url":"https://api.github.com/users/nxtreaming/events{/privacy}","received_events_url":"https://api.github.com/users/nxtreaming/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-12-07T10:58:44Z","updated_at":"2016-12-07T20:01:51Z","closed_at":null,"body":"I get the following log:\r\nroot@DQ5606-C12-2:/home/wdt_test# wdt --directory /home/wdt_test | ssh cn.wdthost.com wdt --directory /data/dev/wdt_test - \r\nI1207 02:50:52.842162 21692 WdtFlags.cpp:63] wdt>       Running WDT 1.27.1612021 p 27\r\nI1207 02:50:52.842989 21692 EncryptionUtils.cpp:83] wdt>        Openssl library initialized\r\nI1207 02:50:52.843101 21692 WdtFlags.cpp:63] wdt>       Running WDT 1.27.1612021 p 27\r\nI1207 02:50:52.843622 21692 WdtResourceController.cpp:29] wdt>  Updated max number of senders for _root controller_ to 0\r\nI1207 02:50:52.843740 21692 WdtResourceController.cpp:22] wdt>  Updated max number of receivers for _root controller_ to 0\r\nI1207 02:50:52.843777 21692 Throttler.cpp:73] wdt>      No average rate specified\r\nI1207 02:50:52.843806 21692 Throttler.cpp:80] wdt>      No peak rate specified\r\nI1207 02:50:52.843847 21692 Wdt.cpp:24] wdt>    One time initialization of WDT for wdt\r\nI1207 02:50:52.843894 21692 Throttler.cpp:98] wdt>      Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0\r\nI1207 02:50:52.843991 21692 Receiver.cpp:46] wdt>       WDT Receiver 1.27.1612021 p 27\r\nI1207 02:50:52.844125 21692 WdtBase.cpp:82] wdt>        Setting transfer id 540054088\r\nI1207 02:50:52.844192 21692 WdtBase.cpp:94] wdt>        using wdt protocol version 27\r\nI1207 02:50:52.844244 21692 FileCreator.cpp:319] wdt>   dir already exists /\r\nI1207 02:50:52.844298 21692 FileCreator.cpp:319] wdt>   dir already exists /home/\r\nI1207 02:50:52.844334 21692 FileCreator.cpp:319] wdt>   dir already exists /home/wdt_test/\r\nI1207 02:50:52.844372 21692 Receiver.cpp:170] wdt>      aes128gcm encryption is enabled for this transfer \r\nI1207 02:50:52.844410 21692 Receiver.cpp:173] wdt>      Receiver generating encryption key for type aes128gcm\r\nI1207 02:50:52.844635 21692 EncryptionUtils.cpp:115] wdt>       New encryption params 0x7ffc1d2991b0 2:...11920925460756788275...\r\nI1207 02:50:52.845219 21692 ReceiverThread.cpp:987] wdt>        Thread[0, port: 47908]  Listening on port 47908\r\nI1207 02:50:52.845518 21692 ReceiverThread.cpp:987] wdt>        Thread[1, port: 49590]  Listening on port 49590\r\nI1207 02:50:52.845783 21692 ReceiverThread.cpp:987] wdt>        Thread[2, port: 45411]  Listening on port 45411\r\nI1207 02:50:52.846050 21692 ReceiverThread.cpp:987] wdt>        Thread[3, port: 35384]  Listening on port 35384\r\nI1207 02:50:52.846314 21692 ReceiverThread.cpp:987] wdt>        Thread[4, port: 58549]  Listening on port 58549\r\nI1207 02:50:52.846580 21692 ReceiverThread.cpp:987] wdt>        Thread[5, port: 49513]  Listening on port 49513\r\nI1207 02:50:52.846845 21692 ReceiverThread.cpp:987] wdt>        Thread[6, port: 54616]  Listening on port 54616\r\nI1207 02:50:52.847108 21692 ReceiverThread.cpp:987] wdt>        Thread[7, port: 47603]  Listening on port 47603\r\nI1207 02:50:52.847178 21692 Receiver.cpp:212] wdt>      Registered 8 successful sockets\r\nI1207 02:50:52.847273 21692 wdtCmdLine.cpp:321] wdt>    Starting receiver with connection url wdt://DQ5606-C12-2?dir=%2fhome%2fwdt_test&enc=2:...11920925460756788275...&id=540054088&ports=47908,49590,45411,35384,58549,49513,54616,47603&recpv=27\r\nI1207 02:50:52.847404 21692 Receiver.cpp:455] wdt>      Starting (receiving) server on ports [ 47908 49590 45411 35384 58549 49513 54616 47603 ] Target dir : /home/wdt_test\r\nI1207 02:50:52.847502 21692 Throttler.cpp:73] wdt>      No average rate specified\r\nI1207 02:50:52.847534 21692 Throttler.cpp:80] wdt>      No peak rate specified\r\nI1207 02:50:52.847564 21692 WdtBase.cpp:183] wdt>       Enabling throttling avgRate: -1 Mbytes/sec, peakRate: 0 Mbytes/sec, bucketLimit: 0 Mbytes, throttlerLogTimeMillis: 0\r\nI1207 02:50:52.847981 21702 Receiver.cpp:396] wdt>      Progress reporter updating every 200 ms\r\nI1207 02:50:53.048244 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:53.248508 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:53.448701 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:53.648888 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:53.849079 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:54.049265 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:54.249464 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:54.449650 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:54.649834 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:54.850025 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:55.050211 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:55.250371 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:55.450556 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:55.650743 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:55.850953 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:56.051161 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:44.761514 13436 WdtFlags.cpp:63] wdt>       Running WDT 1.27.1612021 p 27\r\nI1207 18:50:44.762029 13436 EncryptionUtils.cpp:83] wdt>        Openssl library initialized\r\nI1207 18:50:44.762055 13436 WdtFlags.cpp:63] wdt>       Running WDT 1.27.1612021 p 27\r\nI1207 18:50:44.762279 13436 WdtResourceController.cpp:29] wdt>  Updated max number of senders for _root controller_ to 0\r\nI1207 18:50:44.762300 13436 WdtResourceController.cpp:22] wdt>  Updated max number of receivers for _root controller_ to 0\r\nI1207 18:50:44.762308 13436 Throttler.cpp:73] wdt>      No average rate specified\r\nI1207 18:50:44.762310 13436 Throttler.cpp:80] wdt>      No peak rate specified\r\nI1207 18:50:44.762320 13436 Wdt.cpp:24] wdt>    One time initialization of WDT for wdt\r\nI1207 18:50:44.762336 13436 Throttler.cpp:98] wdt>      Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0\r\nI1207 18:50:44.762446 13436 wdtCmdLine.cpp:282] wdt>    Parsed url as wdt://DQ5606-C12-2?dir=%2fdata%2fdev%2fwdt_test&enc=2:...11920925460756788275...&id=540054088&ports=47908,49590,45411,35384,58549,49513,54616,47603&recpv=27\r\nI1207 18:50:44.762462 13436 wdtCmdLine.cpp:365] wdt>    Making Sender with encryption set = 1\r\nI1207 18:50:44.762475 13436 WdtResourceController.cpp:357] wdt> First time (default) is seen, creating.\r\nI1207 18:50:44.762483 13436 WdtResourceController.cpp:29] wdt>  Updated max number of senders for  to 0\r\nI1207 18:50:44.762488 13436 WdtResourceController.cpp:22] wdt>  Updated max number of receivers for  to 1\r\nI1207 18:50:44.762498 13436 Throttler.cpp:98] wdt>      Updating the rates avgRateBytesPerSec : -1.04858e+06 bucketRateBytesPerSec : 0 bytesTokenBucketLimit : 0\r\nI1207 02:50:56.251382 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:50:56.269986 21699 Receiver.cpp:93] wdt>       Starting new transfer,  peerIp 120.132.7.182 , transfer id 540054088\r\nI1207 02:50:56.270177 21699 WdtSocket.cpp:67] wdt>      Successfully read encryption settings 49513 aes128gcm\r\nI1207 02:50:56.274632 21694 WdtSocket.cpp:67] wdt>      Successfully read encryption settings 47908 aes128gcm\r\nI1207 18:50:44.762518 13436 Sender.cpp:47] wdt> WDT Sender 1.27.1612021 p 27\r\nI1207 18:50:44.762532 13436 WdtResourceController.cpp:379] wdt> Successfully added a sender for  identifier DQ5606-C12-2\r\nI1207 18:50:44.762543 13436 WdtBase.cpp:94] wdt>        using wdt protocol version 27\r\nI1207 18:50:44.762549 13436 Sender.cpp:84] wdt> Encryption is enabled for this transfer\r\nI1207 18:50:44.762572 13436 DirectorySourceQueue.cpp:127] wdt>  Root dir now /data/dev/wdt_test/\r\nI1207 18:50:44.762593 13436 Sender.cpp:311] wdt>        Client (sending) to DQ5606-C12-2, Using ports [ 47908 49590 45411 35384 58549 49513 54616 47603 ]\r\nI1207 18:50:44.762599 13436 Sender.cpp:325] wdt>        Skipping throttler setup. External throttler set.Throttler details : avgRate: -1 Mbytes/sec, peakRate: 0 Mbytes/sec, bucketLimit: 0 Mbytes, throttlerLogTimeMillis: 0\r\nI1207 18:50:44.763731 13439 Sender.cpp:41] wdt> Starting a new transfer 540054088 to DQ5606-C12-2\r\nI1207 18:50:44.763741 13437 DirectorySourceQueue.cpp:240] wdt>  Exploring root dir /data/dev/wdt_test/ include_pattern :  exclude_pattern :  prune_dir_pattern : \r\nI1207 18:50:44.764354 13437 DirectorySourceQueue.cpp:389] wdt>  Number of files explored: 1 opened 0 with direct 0 errors false\r\nI1207 18:50:44.764436 13446 Sender.cpp:418] wdt>        Progress reporter tracking every 200 ms\r\nI1207 18:50:44.964593 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 18:50:44.973379 13443 SenderThread.cpp:85] wdt>   Thread[5, port: 49513]  Connection took 1 attempt(s) and 0.209555 seconds. port 49513\r\nI1207 18:50:44.976073 13438 SenderThread.cpp:85] wdt>   Thread[0, port: 47908]  Connection took 1 attempt(s) and 0.212263 seconds. port 47908\r\nI1207 18:50:44.984274 13444 SenderThread.cpp:85] wdt>   Thread[6, port: 54616]  Connection took 1 attempt(s) and 0.220449 seconds. port 54616\r\nI1207 18:50:44.986655 13439 SenderThread.cpp:85] wdt>   Thread[1, port: 49590]  Connection took 1 attempt(s) and 0.222851 seconds. port 49590\r\nI1207 02:50:56.283303 21695 WdtSocket.cpp:67] wdt>      Successfully read encryption settings 49590 aes128gcm\r\nI1207 18:50:44.988518 13442 SenderThread.cpp:85] wdt>   Thread[4, port: 58549]  Connection took 1 attempt(s) and 0.224649 seconds. port 58549\r\nI1207 02:50:56.287132 21698 WdtSocket.cpp:67] wdt>      Successfully read encryption settings 58549 aes128gcm\r\nI1207 02:50:56.291884 21700 WdtSocket.cpp:67] wdt>      Successfully read encryption settings 54616 aes128gcm\r\nI1207 18:50:45.000504 13440 SenderThread.cpp:85] wdt>   Thread[2, port: 45411]  Connection took 1 attempt(s) and 0.236658 seconds. port 45411\r\nI1207 02:50:56.311111 21696 WdtSocket.cpp:67] wdt>      Successfully read encryption settings 45411 aes128gcm\r\nI1207 02:50:56.354485 21701 ReceiverThread.cpp:195] wdt>        Thread[7, port: 47603]  entered ACCEPT_WITH_TIMEOUT state\r\nI1207 02:50:56.354607 21697 ReceiverThread.cpp:195] wdt>        Thread[3, port: 35384]  entered ACCEPT_WITH_TIMEOUT state\r\nI1207 02:50:56.451607 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.00102392 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:45.164758 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:56.651818 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000487037 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:45.364905 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:56.852028 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000319506 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:45.565044 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:57.052247 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000237731 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:45.765179 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:57.252461 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000189284 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:45.965308 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:57.452673 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000157241 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:46.165446 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:57.652895 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000134475 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:46.365588 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:57.853124 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000117467 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:46.565747 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:58.053350 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0.000104279 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:46.765897 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:58.253563 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 9.37532e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:46.966025 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:58.453788 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 8.51572e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:47.166177 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:58.654011 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 7.80053e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:47.366334 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:58.854233 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 7.19616e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:47.566481 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:59.054456 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 6.67871e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:47.766621 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:59.254686 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 6.23066e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:47.966785 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:59.454839 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 5.8391e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:48.166957 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:59.655055 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 5.49373e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:48.367097 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:50:59.855269 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 5.18694e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:48.567258 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:00.055480 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.91261e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:48.767405 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:00.255697 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.66583e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:48.967564 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:00.455852 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.44272e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:49.167713 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:00.656059 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.23993e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:49.367856 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:00.856274 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 4.05484e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:49.567999 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:01.056495 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.88522e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:49.768143 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:01.256732 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.72922e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:49.968300 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:01.456986 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.58525e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:50.168464 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:01.657230 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.45198e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:50.368628 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:01.857601 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.3282e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:50.568780 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:02.057842 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.21306e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:50.765522 13441 WdtSocket.cpp:282] wdt>     socket io timed out after 5000 ms, retries 8754349 fd 6 doneBytes 0\r\nI1207 18:50:50.765550 13445 WdtSocket.cpp:282] wdt>     socket io timed out after 5000 ms, retries 7521722 fd 8 doneBytes 0\r\nE1207 18:50:50.765683 13441 WdtSocket.cpp:324] wdt>     Unable to read tag at end of stream got -1 needed 16 0x00000000000000000000000000000000\r\nE1207 18:50:50.765702 13445 WdtSocket.cpp:324] wdt>     Unable to read tag at end of stream got -1 needed 16 0x00000000000000000000000000000000\r\nE1207 02:51:02.064534 21697 WdtSocket.cpp:29] wdt>      Failed to read encryption settings 0 35384\r\nE1207 02:51:02.064535 21701 WdtSocket.cpp:29] wdt>      Failed to read encryption settings 0 47603\r\nE1207 02:51:02.064581 21697 ReceiverThread.cpp:44] Read error on 35384 after 0: Resource temporarily unavailable [11]\r\nE1207 02:51:02.064587 21701 ReceiverThread.cpp:44] Read error on 47603 after 0: Resource temporarily unavailable [11]\r\nE1207 02:51:02.064631 21697 ReceiverThread.cpp:265] wdt>        Thread[3, port: 35384]  socket read failure 256 -1\r\nE1207 02:51:02.064643 21701 ReceiverThread.cpp:265] wdt>        Thread[7, port: 47603]  socket read failure 256 -1\r\nI1207 02:51:02.064666 21701 ReceiverThread.cpp:195] wdt>        Thread[7, port: 47603]  entered ACCEPT_WITH_TIMEOUT state\r\nI1207 02:51:02.064647 21697 ReceiverThread.cpp:195] wdt>        Thread[3, port: 35384]  entered ACCEPT_WITH_TIMEOUT state\r\nI1207 18:50:50.768913 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\nI1207 02:51:02.258024 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.10564e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 18:50:50.969048 13446 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 0 Mbytes/s, Recent throughput 0 Mbytes/s 1 Files discovered (complete)\r\n*** Error in `wdt': free(): invalid pointer: 0x00007fd614000930 ***\r\nI1207 02:51:02.333204 21697 ReceiverThread.cpp:237] wdt>        Thread[3, port: 35384]  entered SEND_LOCAL_CHECKPOINT state checkpoint-port: 35384 num-blocks: 0 seq-id: 0 block-offset: 0 received-bytes: 0\r\nI1207 02:51:02.333727 21701 ReceiverThread.cpp:237] wdt>        Thread[7, port: 47603]  entered SEND_LOCAL_CHECKPOINT state checkpoint-port: 47603 num-blocks: 0 seq-id: 0 block-offset: 0 received-bytes: 0\r\nI1207 02:51:02.458220 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 3.00516e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nE1207 02:51:02.570698 21697 WdtSocket.cpp:256] non-retryable error encountered during socket io 31 0 1: Connection reset by peer [104]\r\nE1207 02:51:02.570750 21697 WdtSocket.cpp:29] wdt>      Failed to read encryption settings -1 35384\r\nE1207 02:51:02.570780 21697 ReceiverThread.cpp:44] Read error on 35384 after 0: Connection reset by peer [104]\r\nE1207 02:51:02.570806 21697 ReceiverThread.cpp:265] wdt>        Thread[3, port: 35384]  socket read failure 256 -1\r\nI1207 02:51:02.570835 21697 ReceiverThread.cpp:195] wdt>        Thread[3, port: 35384]  entered ACCEPT_WITH_TIMEOUT state\r\nI1207 02:51:02.570884 21697 EncryptionUtils.cpp:366] wdt>       Encryption finish tag = 0x5ca08bb044c63b00118e0fde4f84db1b\r\nE1207 02:51:02.571048 21701 WdtSocket.cpp:256] non-retryable error encountered during socket io 32 0 1: Connection reset by peer [104]\r\nE1207 02:51:02.571104 21701 WdtSocket.cpp:29] wdt>      Failed to read encryption settings -1 47603\r\nE1207 02:51:02.571120 21701 ReceiverThread.cpp:44] Read error on 47603 after 0: Connection reset by peer [104]\r\nE1207 02:51:02.571135 21701 ReceiverThread.cpp:265] wdt>        Thread[7, port: 47603]  socket read failure 256 -1\r\nI1207 02:51:02.571148 21701 ReceiverThread.cpp:195] wdt>        Thread[7, port: 47603]  entered ACCEPT_WITH_TIMEOUT state\r\nI1207 02:51:02.571177 21701 EncryptionUtils.cpp:366] wdt>       Encryption finish tag = 0xea11379ebf080cedf46543d84682b1fc\r\nI1207 02:51:02.658404 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.91099e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nW1207 02:51:02.667558 21700 ReceiverThread.cpp:75] wdt> Eof on 27\r\nE1207 02:51:02.667672 21700 ReceiverThread.cpp:537] wdt>        Thread[6, port: 54616]  could not read entire content for ffmpeg.tar.bz2 port 54616\r\nI1207 02:51:02.667717 21700 ReceiverThread.cpp:195] wdt>        Thread[6, port: 54616]  entered ACCEPT_WITH_TIMEOUT state\r\nE1207 02:51:02.667752 21700 EncryptionUtils.cpp:473] wdt>       Need tag for gcm mode \r\nW1207 02:51:02.719172 21695 ReceiverThread.cpp:75] wdt> Eof on 23\r\nE1207 02:51:02.719282 21695 ReceiverThread.cpp:537] wdt>        Thread[1, port: 49590]  could not read entire content for ffmpeg.tar.bz2 port 49590\r\nI1207 02:51:02.719326 21695 ReceiverThread.cpp:195] wdt>        Thread[1, port: 49590]  entered ACCEPT_WITH_TIMEOUT state\r\nE1207 02:51:02.719358 21695 EncryptionUtils.cpp:473] wdt>       Need tag for gcm mode \r\nE1207 02:51:02.757189 21699 ReceiverThread.cpp:572] wdt>        Thread[5, port: 49513]  socket read failure 256 0\r\nI1207 02:51:02.757382 21699 ReceiverThread.cpp:195] wdt>        Thread[5, port: 49513]  entered ACCEPT_WITH_TIMEOUT state\r\nE1207 02:51:02.757423 21699 EncryptionUtils.cpp:473] wdt>       Need tag for gcm mode \r\nW1207 02:51:02.780213 21698 ReceiverThread.cpp:75] wdt> Eof on 25\r\nE1207 02:51:02.780335 21698 ReceiverThread.cpp:537] wdt>        Thread[4, port: 58549]  could not read entire content for ffmpeg.tar.bz2 port 58549\r\nI1207 02:51:02.780380 21698 ReceiverThread.cpp:195] wdt>        Thread[4, port: 58549]  entered ACCEPT_WITH_TIMEOUT state\r\nE1207 02:51:02.780412 21698 EncryptionUtils.cpp:473] wdt>       Need tag for gcm mode \r\nI1207 02:51:02.858681 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.82251e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nW1207 02:51:02.864729 21694 ReceiverThread.cpp:75] wdt> Eof on 21\r\nE1207 02:51:02.864847 21694 ReceiverThread.cpp:537] wdt>        Thread[0, port: 47908]  could not read entire content for ffmpeg.tar.bz2 port 47908\r\nI1207 02:51:02.864890 21694 ReceiverThread.cpp:195] wdt>        Thread[0, port: 47908]  entered ACCEPT_WITH_TIMEOUT state\r\nE1207 02:51:02.864923 21694 EncryptionUtils.cpp:473] wdt>       Need tag for gcm mode \r\nI1207 02:51:03.058962 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.73924e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:03.259237 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.66075e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:03.459511 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.58663e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:03.659790 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.51653e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:03.860057 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.45013e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:04.060333 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.38714e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:04.260581 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.32732e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nW1207 02:51:04.331140 21696 ReceiverThread.cpp:75] wdt> Eof on 29\r\nE1207 02:51:04.331253 21696 ReceiverThread.cpp:537] wdt>        Thread[2, port: 45411]  could not read entire content for ffmpeg.tar.bz2 port 45411\r\nI1207 02:51:04.331310 21696 ReceiverThread.cpp:195] wdt>        Thread[2, port: 45411]  entered ACCEPT_WITH_TIMEOUT state\r\nE1207 02:51:04.331352 21696 EncryptionUtils.cpp:473] wdt>       Need tag for gcm mode \r\nI1207 02:51:04.460783 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.27043e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nE1207 02:51:04.571967 21697 ReceiverThread.cpp:218] wdt>        Thread[3, port: 35384]  accept() failed with error CONN_ERROR timeout 2000\r\nI1207 02:51:04.572010 21697 ReceiverThread.cpp:866] wdt>        Thread[3, port: 35384]  entered FINISH_WITH_ERROR state\r\nI1207 02:51:04.572070 21697 Receiver.cpp:30] wdt>       Adding global checkpoint 35384 0 0\r\nE1207 02:51:04.572098 21701 ReceiverThread.cpp:218] wdt>        Thread[7, port: 47603]  accept() failed with error CONN_ERROR timeout 2000\r\nI1207 02:51:04.572106 21697 ReceiverThread.cpp:964] wdt>        Thread[3, port: 35384]  Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0585938 (100% overhead). Total bytes = 60. Wasted bytes due to failure = 0 (100% overhead). Encryption type = aes128gcm.\r\nI1207 02:51:04.572131 21701 ReceiverThread.cpp:866] wdt>        Thread[7, port: 47603]  entered FINISH_WITH_ERROR state\r\nI1207 02:51:04.572213 21701 Receiver.cpp:30] wdt>       Adding global checkpoint 47603 0 0\r\nI1207 02:51:04.572237 21701 ReceiverThread.cpp:964] wdt>        Thread[7, port: 47603]  Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0585938 (100% overhead). Total bytes = 60. Wasted bytes due to failure = 0 (100% overhead). Encryption type = aes128gcm.\r\nI1207 02:51:04.661051 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.21624e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:04.861266 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.1646e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:05.061517 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.11529e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:05.261729 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.06819e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:05.461952 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 2.02314e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:05.662168 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.98001e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:05.862378 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.93869e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:06.062590 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.89905e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:06.262804 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.861e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:06.463014 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.82445e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:06.663224 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.7893e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:06.863433 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.75549e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:07.063647 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.72292e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:07.263859 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.69155e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:07.464063 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.66129e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:07.664271 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.6321e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:07.864480 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.60392e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:08.064704 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.57669e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:08.264933 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.55037e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:08.465144 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.52492e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:08.665357 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.50029e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nE1207 02:51:08.668330 21700 ReceiverThread.cpp:218] wdt>        Thread[6, port: 54616]  accept() failed with error CONN_ERROR timeout 6000\r\nI1207 02:51:08.668367 21700 ReceiverThread.cpp:866] wdt>        Thread[6, port: 54616]  entered FINISH_WITH_ERROR state\r\nI1207 02:51:08.668419 21700 Receiver.cpp:30] wdt>       Adding global checkpoint 54616 0 0\r\nI1207 02:51:08.668440 21700 ReceiverThread.cpp:964] wdt>        Thread[6, port: 54616]  Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 7222578. Wasted bytes due to failure = 7222545 (100% overhead). Encryption type = aes128gcm.\r\nE1207 02:51:08.719959 21695 ReceiverThread.cpp:218] wdt>        Thread[1, port: 49590]  accept() failed with error CONN_ERROR timeout 6000\r\nI1207 02:51:08.719993 21695 ReceiverThread.cpp:866] wdt>        Thread[1, port: 49590]  entered FINISH_WITH_ERROR state\r\nI1207 02:51:08.720039 21695 Receiver.cpp:30] wdt>       Adding global checkpoint 49590 0 0\r\nI1207 02:51:08.720057 21695 ReceiverThread.cpp:964] wdt>        Thread[1, port: 49590]  Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 8882978. Wasted bytes due to failure = 8882945 (100% overhead). Encryption type = aes128gcm.\r\nE1207 02:51:08.758036 21699 ReceiverThread.cpp:218] wdt>        Thread[5, port: 49513]  accept() failed with error CONN_ERROR timeout 6000\r\nI1207 02:51:08.758093 21699 ReceiverThread.cpp:866] wdt>        Thread[5, port: 49513]  entered FINISH_WITH_ERROR state\r\nI1207 02:51:08.758157 21699 Receiver.cpp:30] wdt>       Adding global checkpoint 49513 0 0\r\nI1207 02:51:08.758193 21699 ReceiverThread.cpp:964] wdt>        Thread[5, port: 49513]  Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0292969 (100% overhead). Total bytes = 16777246. Wasted bytes due to failure = 16777216 (100% overhead). Encryption type = aes128gcm.\r\nE1207 02:51:08.781049 21698 ReceiverThread.cpp:218] wdt>        Thread[4, port: 58549]  accept() failed with error CONN_ERROR timeout 6000\r\nI1207 02:51:08.781083 21698 ReceiverThread.cpp:866] wdt>        Thread[4, port: 58549]  entered FINISH_WITH_ERROR state\r\nI1207 02:51:08.781129 21698 Receiver.cpp:30] wdt>       Adding global checkpoint 58549 0 0\r\nI1207 02:51:08.781147 21698 ReceiverThread.cpp:964] wdt>        Thread[4, port: 58549]  Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 10327465. Wasted bytes due to failure = 10327432 (100% overhead). Encryption type = aes128gcm.\r\nI1207 02:51:08.865571 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.47644e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nE1207 02:51:08.865777 21694 ReceiverThread.cpp:218] wdt>        Thread[0, port: 47908]  accept() failed with error CONN_ERROR timeout 6000\r\nI1207 02:51:08.865810 21694 ReceiverThread.cpp:866] wdt>        Thread[0, port: 47908]  entered FINISH_WITH_ERROR state\r\nI1207 02:51:08.865855 21694 Receiver.cpp:30] wdt>       Adding global checkpoint 47908 0 0\r\nI1207 02:51:08.865874 21694 ReceiverThread.cpp:964] wdt>        Thread[0, port: 47908]  Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 1342578. Wasted bytes due to failure = 1342545 (100% overhead). Encryption type = aes128gcm.\r\nI1207 02:51:09.065784 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.45334e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:09.266005 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.43095e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:09.466218 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.40924e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:09.666429 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.38818e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:09.866642 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.36774e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:10.066854 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.34789e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nI1207 02:51:10.267066 21702 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.32861e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (complete)\r\nE1207 02:51:10.331954 21696 ReceiverThread.cpp:218] wdt>        Thread[2, port: 45411]  accept() failed with error CONN_ERROR timeout 6000\r\nI1207 02:51:10.331990 21696 ReceiverThread.cpp:866] wdt>        Thread[2, port: 45411]  entered FINISH_WITH_ERROR state\r\nI1207 02:51:10.332042 21696 Receiver.cpp:30] wdt>       Adding global checkpoint 45411 0 0\r\nI1207 02:51:10.332068 21696 Receiver.cpp:107] wdt>      Ending the transfer 540054088\r\nI1207 02:51:10.332085 21696 ReceiverThread.cpp:964] wdt>        Thread[2, port: 45411]  Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.0322266 (100% overhead). Total bytes = 2661065. Wasted bytes due to failure = 2661032 (100% overhead). Encryption type = aes128gcm.\r\nE1207 02:51:10.332516 21692 Reporting.cpp:200] wdt>     Did not receive all the blocks sent by the sender -1 0\r\nI1207 02:51:10.332576 21692 Reporting.cpp:339] wdt>     wdt transfer progress 0 Mbytes, completed 0%, Average throughput 1.32242e-05 Mbytes/s, Recent throughput 0 Mbytes/s 0 Files discovered (incomplete)\r\nW1207 02:51:10.332618 21692 Receiver.cpp:331] wdt>      WDT receiver's transfer has been finished\r\nI1207 02:51:10.332646 21692 Receiver.cpp:332] wdt>      Transfer status (local) = CONN_ERROR, (remote) = OK. Number of blocks transferred = 0. Data Mbytes = 0. Header Kbytes = 0.307617 (100% overhead). Total bytes = 47214030. Wasted bytes due to failure = 47213715 (100% overhead). Encryption type = none. Previously sent bytes : 0.\r\nW1207 02:51:10.332726 21692 Receiver.cpp:300] wdt>      Threads have already been joined. Returning the transfer report\r\nE1207 02:51:10.332757 21692 Reporting.cpp:200] wdt>     Did not receive all the blocks sent by the sender -1 0\r\nE1207 02:51:10.333192 21692 wdtCmdLine.cpp:374] wdt>    Returning with code 3 CONN_ERROR\r\nI1207 02:51:10.333254 21692 WdtResourceController.cpp:294] wdt> Shutting down the controller (0 senders 0 receivers)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/facebook/wdt/issues/comments/265557080","html_url":"https://github.com/facebook/wdt/issues/138#issuecomment-265557080","issue_url":"https://api.github.com/repos/facebook/wdt/issues/138","id":265557080,"user":{"login":"ldemailly","id":3664595,"avatar_url":"https://avatars.githubusercontent.com/u/3664595?v=3","gravatar_id":"","url":"https://api.github.com/users/ldemailly","html_url":"https://github.com/ldemailly","followers_url":"https://api.github.com/users/ldemailly/followers","following_url":"https://api.github.com/users/ldemailly/following{/other_user}","gists_url":"https://api.github.com/users/ldemailly/gists{/gist_id}","starred_url":"https://api.github.com/users/ldemailly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ldemailly/subscriptions","organizations_url":"https://api.github.com/users/ldemailly/orgs","repos_url":"https://api.github.com/users/ldemailly/repos","events_url":"https://api.github.com/users/ldemailly/events{/privacy}","received_events_url":"https://api.github.com/users/ldemailly/received_events","type":"User","site_admin":false},"created_at":"2016-12-07T20:01:51Z","updated_at":"2016-12-07T20:01:51Z","body":"It looks like your sender failed to connect to 2 of the 8 ports\r\n\r\ncan you try a few things:\r\n\r\na) reduce the number of ports  ( -num_ports=4  for instance)\r\nb) use gdb as mentioned above\r\nc) run with -v=1 and maybe a small file see if the error/crash happens also with small and verbose\r\nd) you may have some ipv4/ipv6 issue - try passing -ipv4 or -ipv6 depending on which you have\r\ne) maybe some ports are blocked, if you know ports that are not blocked use those, for instance wdt  -start_port 12345\r\nf) which OS is this on ? (uname -a)\r\ng) to rule out openssl version issues, try -encryption_type none\r\n\r\nplease let me know more, in particular the crash, that shouldn't happen even if caused by port blocking etc\r\n\r\nthanks!"}}
{"actor":{"display_login":"ldemailly"}}