{"action":"created","issue":{"url":"https://api.github.com/repos/PkuRainBow/OCNet/issues/12","repository_url":"https://api.github.com/repos/PkuRainBow/OCNet","labels_url":"https://api.github.com/repos/PkuRainBow/OCNet/issues/12/labels{/name}","comments_url":"https://api.github.com/repos/PkuRainBow/OCNet/issues/12/comments","events_url":"https://api.github.com/repos/PkuRainBow/OCNet/issues/12/events","html_url":"https://github.com/PkuRainBow/OCNet/issues/12","id":362990316,"node_id":"MDU6SXNzdWUzNjI5OTAzMTY=","number":12,"title":"Segmentation fault","user":{"login":"Spandan-Madan","id":21960611,"node_id":"MDQ6VXNlcjIxOTYwNjEx","avatar_url":"https://avatars0.githubusercontent.com/u/21960611?v=4","gravatar_id":"","url":"https://api.github.com/users/Spandan-Madan","html_url":"https://github.com/Spandan-Madan","followers_url":"https://api.github.com/users/Spandan-Madan/followers","following_url":"https://api.github.com/users/Spandan-Madan/following{/other_user}","gists_url":"https://api.github.com/users/Spandan-Madan/gists{/gist_id}","starred_url":"https://api.github.com/users/Spandan-Madan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Spandan-Madan/subscriptions","organizations_url":"https://api.github.com/users/Spandan-Madan/orgs","repos_url":"https://api.github.com/users/Spandan-Madan/repos","events_url":"https://api.github.com/users/Spandan-Madan/events{/privacy}","received_events_url":"https://api.github.com/users/Spandan-Madan/received_events","type":"User","site_admin":false},"labels":[{"id":1026769599,"node_id":"MDU6TGFiZWwxMDI2NzY5NTk5","url":"https://api.github.com/repos/PkuRainBow/OCNet/labels/good%20first%20issue","name":"good first issue","color":"7057ff","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-09-24T00:14:00Z","updated_at":"2018-09-26T06:49:26Z","closed_at":null,"author_association":"NONE","body":"The model fails to do a forward pass in the train step. The error reported is just \"Segmentation fault\" :- \r\n\r\n```\r\ndataset          cityscapes_train\r\nbatch_size       1\r\ndata_dir         ./dataset/cityscapes\r\ndata_list        ./dataset/list/cityscapes/train.lst\r\nignore_label     255\r\ninput_size       769,769\r\nis_training      False\r\nlearning_rate    0.01\r\nmomentum         0.9\r\nnot_restore_last False\r\nnum_classes      19\r\nstart_iters      0\r\nnum_steps        40000\r\npower            0.9\r\nrandom_mirror    True\r\nrandom_scale     True\r\nrandom_seed      304\r\nrestore_from     ./pretrained_model/resnet101-imagenet.pth\r\nsave_num_images  2\r\nsave_pred_every  5000\r\nsnapshot_dir     checkpoint/snapshots_resnet101_asp_oc_dsn_1e-2_5e-4_8_40000/\r\nweight_decay     0.0005\r\ngpu              0,3,4\r\nohem_thres       0.7\r\nohem_thres1      0.8\r\nohem_thres2      0.5\r\nuse_weight       True\r\nuse_val          False\r\nuse_extra        False\r\nohem             False\r\nohem_keep        0\r\nnetwork          resnet101\r\nmethod           asp_oc_dsn\r\nreduce           True\r\nohem_single      False\r\nuse_parallel     False\r\ndsn_weight       0.4\r\npair_weight      1\r\nseed             304\r\noutput_path      ./seg_output_eval_set\r\nstore_output     False\r\nuse_flip         False\r\nuse_ms           False\r\npredict_choice   whole\r\nwhole_scale      1\r\nstart_epochs     0\r\nend_epochs       120\r\nsave_epoch       20\r\ncriterion        ce\r\neval             False\r\nfix_lr           False\r\nlog_file         \r\nuse_normalize_transform False\r\n/data/graphics/toyota-pytorch/OCNet/network/../oc_module/base_oc_block.py:69: UserWarning: nn.init.constant is now deprecated in favor of nn.init.constant_.\r\n  nn.init.constant(self.W.weight, 0)\r\n/data/graphics/toyota-pytorch/OCNet/network/../oc_module/base_oc_block.py:70: UserWarning: nn.init.constant is now deprecated in favor of nn.init.constant_.\r\n  nn.init.constant(self.W.bias, 0)\r\n/afs/csail.mit.edu/u/s/smadan/miniconda3/envs/py_36_tens_gpu/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py:24: UserWarning: \r\n    There is an imbalance between your GPUs. You may want to exclude GPU 3 which\r\n    has less than 75% of the memory or cores of GPU 1. You can do so by setting\r\n    the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES\r\n    environment variable.\r\n  warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))\r\nw/ class balance\r\n41650 images are loaded!\r\nlearning_rate: 0.01\r\ntorch.Size([1, 3, 769, 769])\r\nSegmentation fault\r\n```\r\n\r\nI added a bunch of print statements and saw that the error is happening in the step \r\n\r\n```\r\npreds = model(images)\r\n```\r\n\r\nI checked the GPU usage, there was over 11GB of GPU memory free when the error occured, so it's not a memory issue. Also, when I ran the .sh file initially, it was reporting errors because the directories for log/log_train and log_test were not created. I created them manually, and that error was resolved. But not, forward pass fails in the first iteration itself. Any leads?"},"comment":{"url":"https://api.github.com/repos/PkuRainBow/OCNet/issues/comments/424604592","html_url":"https://github.com/PkuRainBow/OCNet/issues/12#issuecomment-424604592","issue_url":"https://api.github.com/repos/PkuRainBow/OCNet/issues/12","id":424604592,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNDYwNDU5Mg==","user":{"login":"fanyix","id":6924959,"node_id":"MDQ6VXNlcjY5MjQ5NTk=","avatar_url":"https://avatars3.githubusercontent.com/u/6924959?v=4","gravatar_id":"","url":"https://api.github.com/users/fanyix","html_url":"https://github.com/fanyix","followers_url":"https://api.github.com/users/fanyix/followers","following_url":"https://api.github.com/users/fanyix/following{/other_user}","gists_url":"https://api.github.com/users/fanyix/gists{/gist_id}","starred_url":"https://api.github.com/users/fanyix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fanyix/subscriptions","organizations_url":"https://api.github.com/users/fanyix/orgs","repos_url":"https://api.github.com/users/fanyix/repos","events_url":"https://api.github.com/users/fanyix/events{/privacy}","received_events_url":"https://api.github.com/users/fanyix/received_events","type":"User","site_admin":false},"created_at":"2018-09-26T06:49:26Z","updated_at":"2018-09-26T06:49:26Z","author_association":"NONE","body":"@Spandan-Madan Have you tried delete /tmp/torch_extensions/inplace_abn and re-run? it might cache binary you compiled before"}}
{"actor":{"display_login":"fanyix"}}