{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/azure-storage-fuse/issues/79","repository_url":"https://api.github.com/repos/Azure/azure-storage-fuse","labels_url":"https://api.github.com/repos/Azure/azure-storage-fuse/issues/79/labels{/name}","comments_url":"https://api.github.com/repos/Azure/azure-storage-fuse/issues/79/comments","events_url":"https://api.github.com/repos/Azure/azure-storage-fuse/issues/79/events","html_url":"https://github.com/Azure/azure-storage-fuse/issues/79","id":295110219,"number":79,"title":"INFO: task cp:48332 blocked for more than 120 seconds.","user":{"login":"mkiernan","id":24193619,"avatar_url":"https://avatars1.githubusercontent.com/u/24193619?v=4","gravatar_id":"","url":"https://api.github.com/users/mkiernan","html_url":"https://github.com/mkiernan","followers_url":"https://api.github.com/users/mkiernan/followers","following_url":"https://api.github.com/users/mkiernan/following{/other_user}","gists_url":"https://api.github.com/users/mkiernan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkiernan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkiernan/subscriptions","organizations_url":"https://api.github.com/users/mkiernan/orgs","repos_url":"https://api.github.com/users/mkiernan/repos","events_url":"https://api.github.com/users/mkiernan/events{/privacy}","received_events_url":"https://api.github.com/users/mkiernan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-02-07T11:55:11Z","updated_at":"2018-02-07T18:54:10Z","closed_at":null,"author_association":"NONE","body":"I'm attempting to copy a 270GB file from an azure storage blob container onto a Linux 7.1 machine. \r\n\r\nI see the file appear in the temp directory, but it never makes it to the target, and the command itself bombs out with the below error. \r\n\r\nIs there any way to increase the timeouts here to avoid this error ?\r\n\r\ncp /mnt/hpcpoc/bcbio-data.tar.gz .\r\ncp: cannot open ‘/mnt/hpcpoc/bcbio-data.tar.gz’ for reading: No such file or directory\r\n\r\nThe system log shows : \r\n\r\n[ 2112.479803] fuse init (API version 7.22)\r\n[ 2112.484293] SELinux: initialized (dev fuse, type fuse), uses genfs_contexts\r\n[ 2112.495617] SELinux: initialized (dev fusectl, type fusectl), uses genfs_contexts\r\n[ 2252.473409] SELinux: initialized (dev fuse, type fuse), uses genfs_contexts\r\n[ 2410.892831] SELinux: initialized (dev fuse, type fuse), uses genfs_contexts\r\n[ 2522.446944] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs\r\n[ 2761.277854] INFO: task cp:48332 blocked for more than 120 seconds.\r\n[ 2761.282241] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[ 2761.287717] cp              D ffff881cc6e33680     0 48332  30695 0x00000084\r\n[ 2761.292772]  ffff881c8493bb60 0000000000000086 ffff881c80e6e660 ffff881c8493bfd8\r\n[ 2761.297958]  ffff881c8493bfd8 ffff881c8493bfd8 ffff881c80e6e660 ffff881c8431c640\r\n[ 2761.304768]  ffff881cb0088000 ffff881c8493bb90 ffff881c8431c720 000000000000000e\r\n[ 2761.311518] Call Trace:\r\n[ 2761.313653]  [<ffffffff816098d9>] schedule+0x29/0x70\r\n[ 2761.316914]  [<ffffffffa038853d>] __fuse_request_send+0x13d/0x2c0 [fuse]\r\n[ 2761.323007]  [<ffffffff81098240>] ? wake_up_bit+0x30/0x30\r\n[ 2761.328180]  [<ffffffffa03886d2>] fuse_request_send+0x12/0x20 [fuse]\r\n[ 2761.332465]  [<ffffffffa0391189>] fuse_do_open+0xd9/0x190 [fuse]\r\n[ 2761.336597]  [<ffffffffa03913b6>] fuse_open_common+0x56/0x80 [fuse]\r\n[ 2761.341828]  [<ffffffffa03913f0>] fuse_open+0x10/0x20 [fuse]\r\n[ 2761.347202]  [<ffffffff811c46af>] do_dentry_open+0x18f/0x2b0\r\n[ 2761.351338]  [<ffffffffa03913e0>] ? fuse_open_common+0x80/0x80 [fuse]\r\n[ 2761.356953]  [<ffffffff811c4849>] vfs_open+0x39/0x70\r\n[ 2761.362651]  [<ffffffff811d56fd>] do_last+0x1ed/0x1270\r\n[ 2761.368872]  [<ffffffff8117e981>] ? __do_fault+0x401/0x510\r\n[ 2761.374420]  [<ffffffff811abc7e>] ? kmem_cache_alloc_trace+0x1ce/0x1f0\r\n[ 2761.382905]  [<ffffffff811d6842>] path_openat+0xc2/0x490\r\n[ 2761.386539]  [<ffffffff811d7f12>] ? user_path_at_empty+0x72/0xc0\r\n[ 2761.392951]  [<ffffffff811d800b>] do_filp_open+0x4b/0xb0\r\n[ 2761.396709]  [<ffffffff811e4a87>] ? __alloc_fd+0xa7/0x130\r\n[ 2761.406552]  [<ffffffff811c5d03>] do_sys_open+0xf3/0x1f0\r\n[ 2761.411674]  [<ffffffff811c5e1e>] SyS_open+0x1e/0x20\r\n[ 2761.415425]  [<ffffffff81614409>] system_call_fastpath+0x16/0x1b\r\n[ 2881.541377] INFO: task cp:48332 blocked for more than 120 seconds.\r\n[ 2881.545869] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[ 2881.551002] cp              D ffff881cc6e33680     0 48332  30695 0x00000084\r\n[ 2881.556126]  ffff881c8493bb60 0000000000000086 ffff881c80e6e660 ffff881c8493bfd8\r\n[ 2881.562713]  ffff881c8493bfd8 ffff881c8493bfd8 ffff881c80e6e660 ffff881c8431c640\r\n[ 2881.567291]  ffff881cb0088000 ffff881c8493bb90 ffff881c8431c720 000000000000000e\r\n[ 2881.572445] Call Trace:\r\n[ 2881.574689]  [<ffffffff816098d9>] schedule+0x29/0x70\r\n[ 2881.578096]  [<ffffffffa038853d>] __fuse_request_send+0x13d/0x2c0 [fuse]\r\n[ 2881.582243]  [<ffffffff81098240>] ? wake_up_bit+0x30/0x30\r\n[ 2881.586438]  [<ffffffffa03886d2>] fuse_request_send+0x12/0x20 [fuse]\r\n[ 2881.591777]  [<ffffffffa0391189>] fuse_do_open+0xd9/0x190 [fuse]\r\n[ 2881.596142]  [<ffffffffa03913b6>] fuse_open_common+0x56/0x80 [fuse]\r\n[ 2881.600479]  [<ffffffffa03913f0>] fuse_open+0x10/0x20 [fuse]\r\n[ 2881.604594]  [<ffffffff811c46af>] do_dentry_open+0x18f/0x2b0\r\n[ 2881.608699]  [<ffffffffa03913e0>] ? fuse_open_common+0x80/0x80 [fuse]\r\n[ 2881.613091]  [<ffffffff811c4849>] vfs_open+0x39/0x70\r\n[ 2881.626316]  [<ffffffff811d56fd>] do_last+0x1ed/0x1270\r\n[ 2881.633230]  [<ffffffff8117e981>] ? __do_fault+0x401/0x510\r\n[ 2881.637221]  [<ffffffff811abc7e>] ? kmem_cache_alloc_trace+0x1ce/0x1f0\r\n[ 2881.641507]  [<ffffffff811d6842>] path_openat+0xc2/0x490\r\n[ 2881.645256]  [<ffffffff811d7f12>] ? user_path_at_empty+0x72/0xc0\r\n[ 2881.649572]  [<ffffffff811d800b>] do_filp_open+0x4b/0xb0\r\n[ 2881.654705]  [<ffffffff811e4a87>] ? __alloc_fd+0xa7/0x130\r\n[ 2881.659826]  [<ffffffff811c5d03>] do_sys_open+0xf3/0x1f0\r\n[ 2881.664622]  [<ffffffff811c5e1e>] SyS_open+0x1e/0x20\r\n[ 2881.668002]  [<ffffffff81614409>] system_call_fastpath+0x16/0x1b\r\n[ 3001.781327] INFO: task cp:48332 blocked for more than 120 seconds.\r\n[ 3001.784806] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[ 3001.790138] cp              D ffff881cc6e33680     0 48332      1 0x00000084\r\n[ 3001.796839]  ffff881c8493bb60 0000000000000086 ffff881c80e6e660 ffff881c8493bfd8\r\n[ 3001.808432]  ffff881c8493bfd8 ffff881c8493bfd8 ffff881c80e6e660 ffff881c8431c640\r\n[ 3001.813958]  ffff881cb0088000 ffff881c8493bb90 ffff881c8431c720 000000000000000e\r\n[ 3001.819736] Call Trace:\r\n[ 3001.821800]  [<ffffffff816098d9>] schedule+0x29/0x70\r\n[ 3001.825217]  [<ffffffffa038853d>] __fuse_request_send+0x13d/0x2c0 [fuse]\r\n[ 3001.829662]  [<ffffffff81098240>] ? wake_up_bit+0x30/0x30\r\n[ 3001.834705]  [<ffffffffa03886d2>] fuse_request_send+0x12/0x20 [fuse]\r\n[ 3001.839300]  [<ffffffffa0391189>] fuse_do_open+0xd9/0x190 [fuse]\r\n[ 3001.844352]  [<ffffffffa03913b6>] fuse_open_common+0x56/0x80 [fuse]\r\n[ 3001.848903]  [<ffffffffa03913f0>] fuse_open+0x10/0x20 [fuse]\r\n[ 3001.854519]  [<ffffffff811c46af>] do_dentry_open+0x18f/0x2b0\r\n[ 3001.859771]  [<ffffffffa03913e0>] ? fuse_open_common+0x80/0x80 [fuse]\r\n[ 3001.864392]  [<ffffffff811c4849>] vfs_open+0x39/0x70\r\n[ 3001.867908]  [<ffffffff811d56fd>] do_last+0x1ed/0x1270\r\n[ 3001.871605]  [<ffffffff8117e981>] ? __do_fault+0x401/0x510\r\n[ 3001.875500]  [<ffffffff811abc7e>] ? kmem_cache_alloc_trace+0x1ce/0x1f0\r\n[ 3001.879784]  [<ffffffff811d6842>] path_openat+0xc2/0x490\r\n[ 3001.883592]  [<ffffffff811d7f12>] ? user_path_at_empty+0x72/0xc0\r\n[ 3001.887850]  [<ffffffff811d800b>] do_filp_open+0x4b/0xb0\r\n[ 3001.892056]  [<ffffffff811e4a87>] ? __alloc_fd+0xa7/0x130\r\n[ 3001.895856]  [<ffffffff811c5d03>] do_sys_open+0xf3/0x1f0\r\n[ 3001.899774]  [<ffffffff811c5e1e>] SyS_open+0x1e/0x20\r\n[ 3001.904157]  [<ffffffff81614409>] system_call_fastpath+0x16/0x1b\r\n[ 3121.951954] INFO: task cp:48332 blocked for more than 120 seconds.\r\n[ 3121.959194] \"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\" disables this message.\r\n[ 3121.964258] cp              D ffff881cc6e33680     0 48332      1 0x00000084\r\n[ 3121.968868]  ffff881c8493bb60 0000000000000086 ffff881c80e6e660 ffff881c8493bfd8\r\n[ 3121.973588]  ffff881c8493bfd8 ffff881c8493bfd8 ffff881c80e6e660 ffff881c8431c640\r\n[ 3121.978599]  ffff881cb0088000 ffff881c8493bb90 ffff881c8431c720 000000000000000e\r\n[ 3121.983644] Call Trace:\r\n[ 3121.985749]  [<ffffffff816098d9>] schedule+0x29/0x70\r\n[ 3121.989248]  [<ffffffffa038853d>] __fuse_request_send+0x13d/0x2c0 [fuse]\r\n[ 3121.993793]  [<ffffffff81098240>] ? wake_up_bit+0x30/0x30\r\n[ 3121.997540]  [<ffffffffa03886d2>] fuse_request_send+0x12/0x20 [fuse]\r\n[ 3122.002439]  [<ffffffffa0391189>] fuse_do_open+0xd9/0x190 [fuse]\r\n[ 3122.006829]  [<ffffffffa03913b6>] fuse_open_common+0x56/0x80 [fuse]\r\n[ 3122.010776]  [<ffffffffa03913f0>] fuse_open+0x10/0x20 [fuse]\r\n[ 3122.014478]  [<ffffffff811c46af>] do_dentry_open+0x18f/0x2b0\r\n[ 3122.018232]  [<ffffffffa03913e0>] ? fuse_open_common+0x80/0x80 [fuse]\r\n[ 3122.022315]  [<ffffffff811c4849>] vfs_open+0x39/0x70\r\n[ 3122.025675]  [<ffffffff811d56fd>] do_last+0x1ed/0x1270\r\n[ 3122.029083]  [<ffffffff8117e981>] ? __do_fault+0x401/0x510\r\n[ 3122.032872]  [<ffffffff811abc7e>] ? kmem_cache_alloc_trace+0x1ce/0x1f0\r\n[ 3122.036930]  [<ffffffff811d6842>] path_openat+0xc2/0x490\r\n[ 3122.040539]  [<ffffffff811d7f12>] ? user_path_at_empty+0x72/0xc0\r\n[ 3122.044565]  [<ffffffff811d800b>] do_filp_open+0x4b/0xb0\r\n[ 3122.048168]  [<ffffffff811e4a87>] ? __alloc_fd+0xa7/0x130\r\n[ 3122.052617]  [<ffffffff811c5d03>] do_sys_open+0xf3/0x1f0\r\n[ 3122.056207]  [<ffffffff811c5e1e>] SyS_open+0x1e/0x20\r\n[ 3122.059732]  [<ffffffff81614409>] system_call_fastpath+0x16/0x1b\r\n[ 3724.178760] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs\r\n[ 4919.284778] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs\r\n[cyclecloud@ip-0A630404 BCBIO]$"},"comment":{"url":"https://api.github.com/repos/Azure/azure-storage-fuse/issues/comments/363871974","html_url":"https://github.com/Azure/azure-storage-fuse/issues/79#issuecomment-363871974","issue_url":"https://api.github.com/repos/Azure/azure-storage-fuse/issues/79","id":363871974,"user":{"login":"seguler","id":22362788,"avatar_url":"https://avatars0.githubusercontent.com/u/22362788?v=4","gravatar_id":"","url":"https://api.github.com/users/seguler","html_url":"https://github.com/seguler","followers_url":"https://api.github.com/users/seguler/followers","following_url":"https://api.github.com/users/seguler/following{/other_user}","gists_url":"https://api.github.com/users/seguler/gists{/gist_id}","starred_url":"https://api.github.com/users/seguler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/seguler/subscriptions","organizations_url":"https://api.github.com/users/seguler/orgs","repos_url":"https://api.github.com/users/seguler/repos","events_url":"https://api.github.com/users/seguler/events{/privacy}","received_events_url":"https://api.github.com/users/seguler/received_events","type":"User","site_admin":false},"created_at":"2018-02-07T18:54:10Z","updated_at":"2018-02-07T18:54:10Z","author_association":"MEMBER","body":"We are using it as a buffer to boost performance. You may be reading this sequentially, but we do not know whether you will do random IO, or seek to beginning at any time. So instead of going to block blob all the time and take a huge latency hit, we read it from the buffer cache on disk.\r\n\r\nSo if you are working with 270GB file on your /mnt folder. Then you need at least 540GB space on /mnt assuming you also have the cache directory on /mnt/blobfusetmp"}}
{"actor":{"display_login":"seguler"}}