{"action":"opened","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/13524","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/13524/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/13524/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/13524/events","html_url":"https://github.com/pytorch/pytorch/issues/13524","id":376987541,"node_id":"MDU6SXNzdWUzNzY5ODc1NDE=","number":13524,"title":"dead loop when iterate through dataloader using customized datasetfolder","user":{"login":"nimning","id":7147016,"node_id":"MDQ6VXNlcjcxNDcwMTY=","avatar_url":"https://avatars2.githubusercontent.com/u/7147016?v=4","gravatar_id":"","url":"https://api.github.com/users/nimning","html_url":"https://github.com/nimning","followers_url":"https://api.github.com/users/nimning/followers","following_url":"https://api.github.com/users/nimning/following{/other_user}","gists_url":"https://api.github.com/users/nimning/gists{/gist_id}","starred_url":"https://api.github.com/users/nimning/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nimning/subscriptions","organizations_url":"https://api.github.com/users/nimning/orgs","repos_url":"https://api.github.com/users/nimning/repos","events_url":"https://api.github.com/users/nimning/events{/privacy}","received_events_url":"https://api.github.com/users/nimning/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-02T22:30:41Z","updated_at":"2018-11-02T22:30:41Z","closed_at":null,"author_association":"NONE","body":"## 🐛 Bug\r\n\r\n<!-- Dataloader runs into dead loop for a customized datasetfolder -->\r\n## Description\r\nI have a lot of images with .gif and .oct-stream extension. Since the ImageFolder will ignore those files, I use the DatasetFolder and provide my img_extension and loader as suggested by other forks on this forum. I create a dataloader and try to iterate through it. Unfortunately, it got stuck somewhere forever. The deadloop occurs when the dataloader iterator calls __next()__.\r\n\r\n## To Reproduce\r\nThe folder structure is the following. I have attached all images except '01.octet-stream' as this forum does not support it.\r\n\r\n```\r\n# debug/0/01.octet-stream\r\n# debug/0/02.jpeg\r\n# debug/0/03.gif\r\n\r\n# debug/1/01.octet-stream\r\n# debug/1/02.jpeg\r\n# debug/1/03.gif\r\n```\r\nHere is my code, you can run them directly on notebook.\r\n\r\n\r\n```\r\nfrom __future__ import print_function, division\r\n\r\nimport torch\r\nimport torch.nn as nn\r\nimport torch.optim as optim\r\nfrom torch.optim import lr_scheduler\r\nimport numpy as np\r\nimport torchvision\r\nfrom torchvision import datasets, models, transforms\r\nfrom sklearn.utils.class_weight import compute_class_weight\r\nimport matplotlib.pyplot as plt\r\nimport torch.nn.functional as F\r\nimport time\r\nimport os\r\nimport copy\r\nfrom torchvision.datasets import ImageFolder\r\n\r\nplt.ion()\r\n```\r\n\r\n\r\n```\r\ndata_transforms = transforms.Compose([transforms.Resize(300),\r\n        transforms.CenterCrop(299),\r\n        transforms.RandomHorizontalFlip(),\r\n        transforms.ToTensor(),\r\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])\r\n\r\nimg_extensions = ['.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif', '.gif', '.octet-stream']\r\n```\r\n\r\n\r\n```\r\ndef my_loader(path):\r\n    from torchvision import get_image_backend\r\n    from PIL import Image\r\n    \r\n                \r\n    def my_pil_loader(path):\r\n        print (\"loading {}\".format(path))\r\n        with open(path, 'rb') as f:\r\n            img = Image.open(f)\r\n            return img.convert('RGB')\r\n\r\n    if get_image_backend() == 'accimage':\r\n        print('{} uses accimage'.format(path))\r\n        try:\r\n            return accimage_loader(path)\r\n        except IOError:\r\n            print('{} accimage loading fail, using PIL'.format(path))\r\n            return my_pil_loader(path)\r\n    else:\r\n        print('{} uses PIL'.format(path))\r\n        return my_pil_loader(path)\r\n```\r\n\r\n\r\n\r\n\r\n```\r\nmy_loader('./debug/0/03.gif')\r\n\r\ndata_dir = './debug/'\r\nbatch_size = 32\r\n\r\nimage_datasets = datasets.DatasetFolder(data_dir, my_loader, img_extensions,\r\n                                          data_transforms)\r\ndataloaders = torch.utils.data.DataLoader(image_datasets, batch_size=batch_size,\r\n                                             shuffle=True, num_workers=4)\r\n\r\ndataset_sizes = len(image_datasets)\r\nprint(dataset_sizes)\r\n```\r\n\r\n\r\nEverything works fine so far. However, when I try to iterate through the dataloader and run the following code, the program got stuck forever! It seems the code runs into dead loop somewhere even before loading images as I do not see any print information during loading images. What's wrong with implementation?\r\n\r\nIf I replace the DatasetFolder with 'ImageFolder' and get rid of the customized loader and extension, everything works fine. Very wired....\r\n\r\n```\r\nindex = 0\r\nfor inputs, labels in dataloaders:\r\n    print(index)\r\n    print('inputs')\r\n    print(inputs.size())\r\n    print('labels')\r\n    print(labels.size())\r\n```\r\n\r\n![02|113x100](upload://ouh9gvsiwVomCZAQKDgVvVW3D51.jpeg) ![03|499x374](upload://9FqhLJWfeGluvmSlQHfcuJ25PgI.gif)\r\n\r\n<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->\r\n\r\n## Expected behavior\r\nProgram got stuck when running the following code snippet \r\n```\r\nindex = 0\r\nfor inputs, labels in dataloaders:\r\n    print(index)\r\n    print('inputs')\r\n    print(inputs.size())\r\n    print('labels')\r\n    print(labels.size())\r\n```\r\nHowever, the datasetsfolder object works \r\n`image_datasets[0]`\r\n```\r\n./debug/0\\01.octet-stream uses PIL\r\nloading ./debug/0\\01.octet-stream\r\n(tensor([[[ 0.2967,  0.0912,  0.1254,  ..., -0.3369, -0.2513, -0.2856],\r\n          [ 0.2111,  0.2282,  0.1768,  ..., -0.4054, -0.3369, -0.3198],\r\n          [ 0.1939,  0.2796,  0.2282,  ..., -0.5596, -0.4568, -0.2684],\r\n          ...,\r\n          [-1.5870, -1.5014, -1.5699,  ..., -1.6555, -1.8097, -1.8439],\r\n          [-1.6727, -1.5699, -1.5699,  ..., -1.9809, -1.9295, -1.7754],\r\n          [-1.6384, -1.5185, -1.5528,  ..., -1.7069, -1.6898, -1.6042]],\r\n \r\n         [[ 0.5728,  0.6604,  0.7304,  ..., -0.0924, -0.1099, -0.0749],\r\n          [ 0.7304,  0.6254,  0.6954,  ..., -0.1975, -0.1450, -0.1099],\r\n          [ 0.7304,  0.7129,  0.7479,  ..., -0.3025, -0.1800, -0.0924],\r\n          ...,\r\n          [-1.1253, -1.1429, -1.1429,  ..., -1.6331, -1.5630, -1.5980],\r\n          [-1.0728, -1.1779, -1.2654,  ..., -1.4755, -1.5455, -1.5980],\r\n          [-1.1604, -1.1429, -1.1604,  ..., -1.5980, -1.6155, -1.6155]],\r\n \r\n         [[ 0.7054,  0.6356,  0.6531,  ...,  0.0605,  0.0431,  0.0256],\r\n          [ 0.7576,  0.7402,  0.7228,  ...,  0.0605,  0.0431, -0.0092],\r\n          [ 0.7576,  0.7925,  0.7925,  ..., -0.0267, -0.0615, -0.0441],\r\n          ...,\r\n          [-0.7064, -0.6715, -0.6541,  ..., -1.2990, -1.2467, -1.2467],\r\n          [-0.7936, -0.6541, -0.6193,  ..., -1.3513, -1.3164, -1.1073],\r\n          [-0.8284, -0.7064, -0.7064,  ..., -1.2816, -1.2990, -1.2119]]]), 0)\r\n```\r\n\r\nAfter some debugging, it seems the dead loop occurs when the following is called\r\n```\r\ndataloader_iter = dataloaders.__iter__()\r\ndataloader_iter._get_batch()\r\n```\r\n\r\nIt seems the dead loop starts when the following function is called\r\n`dataloader_iter.data_queue.get()`\r\n\r\n<!-- How can I get the dataloader work? -->\r\n\r\n## Environment\r\n\r\nPlease copy and paste the output from our\r\n[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)\r\n(or fill out the checklist below manually).\r\n\r\nYou can get the script and run it with:\r\n```\r\nwget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py\r\n# For security purposes, please check the contents of collect_env.py before running it.\r\npython collect_env.py\r\n```\r\n\r\n - PyTorch Version:0.4.1\r\n - OS :Windows\r\n - How you installed PyTorch:`conda\r\n - Build command you used (if compiling from source):\r\n - Python version: 3.6.5\r\n - CUDA/cuDNN version: NA\r\n - GPU models and configuration:NA\r\n - Any other relevant information:\r\n\r\n## Additional context\r\n\r\n<!-- Add any other context about the problem here. -->\r\n"}}
{"actor":{"display_login":"nimning"}}