{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/5219","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/5219/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/5219/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/5219/events","html_url":"https://github.com/pytorch/pytorch/issues/5219","id":296852766,"node_id":"MDU6SXNzdWUyOTY4NTI3NjY=","number":5219,"title":"Segmentation fault when calling autograd.grad() in autograd.Function.backward","user":{"login":"csrhddlam","id":16692643,"node_id":"MDQ6VXNlcjE2NjkyNjQz","avatar_url":"https://avatars1.githubusercontent.com/u/16692643?v=4","gravatar_id":"","url":"https://api.github.com/users/csrhddlam","html_url":"https://github.com/csrhddlam","followers_url":"https://api.github.com/users/csrhddlam/followers","following_url":"https://api.github.com/users/csrhddlam/following{/other_user}","gists_url":"https://api.github.com/users/csrhddlam/gists{/gist_id}","starred_url":"https://api.github.com/users/csrhddlam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/csrhddlam/subscriptions","organizations_url":"https://api.github.com/users/csrhddlam/orgs","repos_url":"https://api.github.com/users/csrhddlam/repos","events_url":"https://api.github.com/users/csrhddlam/events{/privacy}","received_events_url":"https://api.github.com/users/csrhddlam/received_events","type":"User","site_admin":false},"labels":[{"id":443484135,"node_id":"MDU6TGFiZWw0NDM0ODQxMzU=","url":"https://api.github.com/repos/pytorch/pytorch/labels/high%20priority","name":"high priority","color":"F22613","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-02-13T19:13:00Z","updated_at":"2018-06-11T18:47:34Z","closed_at":null,"author_association":"NONE","body":"Hi everyone,\r\n\r\nI was trying to implement a memory saving version of [GUNN](https://arxiv.org/abs/1711.09280) by reconstructing input during backward as [tbung](https://github.com/tbung/pytorch-revnet) did, but my training on cifar10 occasionally crashes with segmentation fault. It can seldom get through 50 epochs.\r\n\r\n**Error message:**\r\n```\r\nEpoch: [268][670/782]   Time 0.354 (0.331)      Data 0.001 (0.001)      Loss 0.0003 (0.0014)    Prec@1 100.000 (99.984) Prec@5 100.000 (100.000)\r\n\r\nThread 6 \"python\" received signal SIGSEGV, Segmentation fault.\r\n[Switching to Thread 0x7fff22ba1700 (LWP 23233)]\r\nstd::__push_heap<__gnu_cxx::__normal_iterator<torch::autograd::FunctionTask*, std::vector<torch::autograd::FunctionTask, std::allocator<torch::autograd::FunctionTask> > >, long, torch::autograd::FunctionTask, torch::autograd::CompareFunctionTaskTime> (__first=..., __holeIndex=5, __topIndex=__topIndex@entry=0, __value=..., __comp=__comp@entry=...)\r\n    at /home/why/anaconda3/gcc/include/c++/bits/stl_heap.h:182\r\n182           while (__holeIndex > __topIndex\r\n(gdb) bt\r\n#0  std::__push_heap<__gnu_cxx::__normal_iterator<torch::autograd::FunctionTask*, std::vector<torch::autograd::FunctionTask, std::allocator<torch::autograd::FunctionTask> > >, long, torch::autograd::FunctionTask, torch::autograd::CompareFunctionTaskTime> (__first=..., \r\n    __holeIndex=5, __topIndex=__topIndex@entry=0, __value=..., __comp=__comp@entry=...) at /home/why/anaconda3/gcc/include/c++/bits/stl_heap.h:182\r\n#1  0x00007fffecd33480 in std::push_heap<__gnu_cxx::__normal_iterator<torch::autograd::FunctionTask*, std::vector<torch::autograd::FunctionTask> >, torch::autograd::CompareFunctionTaskTime> (__comp=..., __last=..., __first=...)\r\n    at /home/why/anaconda3/gcc/include/c++/bits/stl_heap.h:221\r\n#2  std::priority_queue<torch::autograd::FunctionTask, std::vector<torch::autograd::FunctionTask, std::allocator<torch::autograd::FunctionTask> >, torch::autograd::CompareFunctionTaskTime>::push(torch::autograd::FunctionTask&&) (\r\n    __x=<unknown type in /home/why/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so, CU 0x2314a43, DIE 0x23cd8e1>, this=0x7fff2be84fe0) at /home/why/anaconda3/gcc/include/c++/bits/stl_queue.h:507\r\n#3  torch::autograd::ReadyQueue::push (this=0x7fff2be84fe0, item=...) at torch/csrc/autograd/engine.cpp:128\r\n#4  0x00007fffecd364ed in torch::autograd::Engine::thread_main (this=0x7fffee562680 <engine>, graph_task=0x0) at torch/csrc/autograd/engine.cpp:199\r\n#5  0x00007fffecd329d4 in torch::autograd::Engine::thread_init (this=this@entry=0x7fffee562680 <engine>, device=device@entry=-1) at torch/csrc/autograd/engine.cpp:150\r\n#6  0x00007fffecd5bf0a in torch::autograd::python::PythonEngine::thread_init (this=0x7fffee562680 <engine>, device=-1) at torch/csrc/autograd/python_engine.cpp:34\r\n#7  0x00007ffff7b0dc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6\r\n#8  0x00007ffff73376ba in start_thread (arg=0x7fff22ba1700) at pthread_create.c:333\r\n#9  0x00007ffff67553dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109\r\n```\r\nUnfortunately, I cannot reproduce this error in several lines, so I'm attaching the full model:\r\n```\r\nfrom __future__ import absolute_import\r\n\r\nimport torch\r\nimport torch.nn as nn\r\nimport math\r\nfrom torch.autograd import Function, Variable, grad\r\nimport numpy as np\r\nfrom itertools import chain\r\n\r\n\r\n__all__ = ['gunn']\r\n\r\n\r\nclass Gunn_function(Function):\r\n    @staticmethod\r\n    def forward(ctx, x, list_of_modules, *parameters_of_list_of_modules):\r\n        # ctx.mark_dirty(x)  # 0.3\r\n        # x = Variable(x, volatile=True)  # 0.3\r\n\r\n        ctx.gather, ctx.updater, ctx.scatter = list_of_modules\r\n\r\n        var_temp = ctx.updater(ctx.gather(x))\r\n        var_dx = ctx.scatter(var_temp)\r\n\r\n        x.data.add_(var_dx.data)\r\n\r\n        ctx.x = x.data\r\n        ctx.temp = var_temp.data\r\n\r\n        # x = x.data  # 0.3\r\n        return x\r\n\r\n    @staticmethod\r\n    def backward(ctx, gradient):\r\n        with torch.enable_grad():\r\n            var_temp = Variable(ctx.temp, requires_grad=True)\r\n            var_dx = ctx.scatter(var_temp)\r\n\r\n            ctx.x.add_(-var_dx.data)  # change x back to input\r\n\r\n            var_x = Variable(ctx.x, requires_grad=True)\r\n            var_temp2 = ctx.updater(ctx.gather(var_x))\r\n\r\n        parameters_tuple1 = tuple(filter(lambda x: x.requires_grad, ctx.scatter.parameters()))\r\n        parameters_tuple2 = tuple(filter(lambda x: x.requires_grad, chain(ctx.gather.parameters(), ctx.updater.parameters())))\r\n        temp_grad, *parameters_grads1 = torch.autograd.grad(var_dx, (var_temp,) + parameters_tuple1, gradient)\r\n        x_grad, *parameters_grads2 = torch.autograd.grad(var_temp2, (var_x,) + parameters_tuple2, temp_grad)\r\n\r\n        return (x_grad + gradient, None, ) + tuple(parameters_grads2 + parameters_grads1)\r\n\r\n\r\nclass Update(nn.Module):\r\n    def __init__(self, in_channels, out_channels, K):\r\n        super(Update, self).__init__()\r\n        self.conv1 = nn.Conv2d(in_channels, out_channels * K, kernel_size=1, bias=False)\r\n        self.bn1 = nn.BatchNorm2d(out_channels * K)\r\n        self.conv2 = nn.Conv2d(out_channels * K, out_channels * K, kernel_size=3, padding=1, bias=False)\r\n        self.bn2 = nn.BatchNorm2d(out_channels * K)\r\n        self.conv3 = nn.Conv2d(out_channels * K, out_channels, kernel_size=1, bias=False)\r\n        self.bn3 = nn.BatchNorm2d(out_channels)\r\n        self.conv4 = nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False)\r\n        self.bn4 = nn.BatchNorm2d(out_channels)\r\n        self.relu = nn.ReLU()\r\n\r\n    def forward(self, x):\r\n        y = self.conv4.forward(x)\r\n        y = self.bn4.forward(y)\r\n\r\n        x = self.conv1.forward(x)\r\n        x = self.bn1.forward(x)\r\n        x = self.relu.forward(x)\r\n\r\n        x = self.conv2.forward(x)\r\n        x = self.bn2.forward(x)\r\n        x = self.relu.forward(x)\r\n\r\n        x = self.conv3.forward(x)\r\n        x = self.bn3.forward(x)\r\n        return x + y\r\n\r\n\r\ndef get_tensor(L, P, p, type='gunn'):\r\n    if type == 'identity':\r\n        return torch.eye(L * P)\r\n    elif type == 'gunn':\r\n        return torch.cat([torch.zeros(L * p, L), torch.eye(L * (P - p), L)], dim=0)\r\n    raise NotImplementedError\r\n\r\n\r\nclass Gunn_layer(nn.Module):\r\n    def __init__(self, N, P, K):\r\n        super(Gunn_layer, self).__init__()\r\n        self.P = P\r\n        L = N // P\r\n\r\n        for p in range(P):\r\n            # gather = nn.Conv2d(N, L, kernel_size=1, bias=False)\r\n            # gather.weight.data = get_tensor(L, P, p, 'identity').t().unsqueeze(2).unsqueeze(3)\r\n            # gather.weight.requires_grad = False\r\n            # gather.inited = True\r\n            gather = nn.Sequential()\r\n            updater = Update(N, L, K)\r\n\r\n            scatter = nn.Conv2d(L, N, kernel_size=1, bias=False)\r\n            scatter.weight.data = get_tensor(L, P, p, 'gunn').unsqueeze(2).unsqueeze(3)\r\n            scatter.weight.requires_grad = False\r\n            scatter.inited = True\r\n\r\n            self.add_module('gather' + str(p), gather)\r\n            self.add_module('updater' + str(p), updater)\r\n            self.add_module('scatter' + str(p), scatter)\r\n\r\n    def forward(self, x):\r\n        for p in range(self.P):\r\n            parameters = list(filter(lambda param: param.requires_grad, chain(self._modules['gather' + str(p)].parameters(), self._modules['updater' + str(p)].parameters(), self._modules['scatter' + str(p)].parameters())))\r\n            modules = (self._modules['gather' + str(p)], self._modules['updater' + str(p)], self._modules['scatter' + str(p)])\r\n            x = Gunn_function.apply(x, modules, *parameters)\r\n        return x\r\n\r\n\r\nclass Gunn(nn.Module):\r\n    def __init__(self, num_classes=10):\r\n        super(Gunn, self).__init__()\r\n        self.num_classes = num_classes\r\n        N1 = 240\r\n        N2 = 300\r\n        N3 = 360\r\n\r\n        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False)\r\n        self.bn1 = nn.BatchNorm2d(64)\r\n        self.relu = nn.ReLU(inplace=True)\r\n\r\n        self.trans0_conv = nn.Conv2d(64, N1, kernel_size=1, bias=False)\r\n        self.trans0_bn = nn.BatchNorm2d(N1)\r\n\r\n        self.layer1 = Gunn_layer(N1, 20, 2)\r\n\r\n        self.trans1_conv = nn.Conv2d(N1, N2, kernel_size=1, bias=False)\r\n        self.trans1_bn = nn.BatchNorm2d(N2)\r\n\r\n        self.layer2 = Gunn_layer(N2, 25, 2)\r\n\r\n        self.trans2_conv = nn.Conv2d(N2, N3, kernel_size=1, bias=False)\r\n        self.trans2_bn = nn.BatchNorm2d(N3)\r\n\r\n        self.layer3 = Gunn_layer(N3, 30, 2)\r\n\r\n        self.trans3_conv = nn.Conv2d(N3, N3, kernel_size=1, bias=False)\r\n        self.trans3_bn = nn.BatchNorm2d(N3)\r\n\r\n        self.avgpool = nn.AvgPool2d(2, 2)\r\n        self.GAP = nn.AvgPool2d(8)\r\n        self.fc = nn.Linear(N3, num_classes)\r\n\r\n        for m in self.modules():\r\n            if hasattr(m, 'inited'):\r\n                continue\r\n            if isinstance(m, nn.Conv2d):\r\n                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels\r\n                m.weight.data.normal_(0, math.sqrt(2. / n))\r\n            elif isinstance(m, nn.BatchNorm2d):\r\n                m.weight.data.fill_(1)\r\n                m.bias.data.zero_()\r\n\r\n    def forward(self, x):\r\n        x = self.conv1(x)\r\n        x = self.bn1(x)\r\n        x = self.relu(x)  # 32x32\r\n\r\n        x = self.trans0_conv(x)\r\n        x = self.trans0_bn(x)\r\n        x = self.relu(x)\r\n\r\n        x = self.layer1(x)  # 32x32\r\n\r\n        x = self.trans1_conv(x)\r\n        x = self.trans1_bn(x)\r\n        x = self.relu(x)\r\n        x = self.avgpool(x)  # 16x16\r\n\r\n        x = self.layer2(x)\r\n\r\n        x = self.trans2_conv(x)\r\n        x = self.trans2_bn(x)\r\n        x = self.relu(x)\r\n        x = self.avgpool(x)  # 8x8\r\n\r\n        x = self.layer3(x)  # 8x8\r\n\r\n        x = self.trans3_conv(x)\r\n        x = self.trans3_bn(x)\r\n        x = self.relu(x)\r\n        x = self.GAP(x)\r\n        x = x.view(x.size(0), -1)\r\n        output = self.fc(x)\r\n\r\n        return output\r\n\r\n\r\ndef gunn(**kwargs):\r\n    return Gunn(**kwargs)\r\n\r\n```\r\nTraining this `gunn()` on cifar10 using standard classification script should reproduce the segmentation fault, but it may take several hours. Training four models separately on four GPUs can accelerate reproducing.\r\n\r\nI'm using pytorch master cloned yesterday, at commit 5eefe87d4eab296c0eb28394f9daf4659d03d890, Ubuntu 16.04, Python 3.6, Cuda9.0, Cudnn7.0.5, gcc 4.8.5, Titan X pascal."},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/396346231","html_url":"https://github.com/pytorch/pytorch/issues/5219#issuecomment-396346231","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/5219","id":396346231,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NjM0NjIzMQ==","user":{"login":"ivankreso","id":2056432,"node_id":"MDQ6VXNlcjIwNTY0MzI=","avatar_url":"https://avatars3.githubusercontent.com/u/2056432?v=4","gravatar_id":"","url":"https://api.github.com/users/ivankreso","html_url":"https://github.com/ivankreso","followers_url":"https://api.github.com/users/ivankreso/followers","following_url":"https://api.github.com/users/ivankreso/following{/other_user}","gists_url":"https://api.github.com/users/ivankreso/gists{/gist_id}","starred_url":"https://api.github.com/users/ivankreso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ivankreso/subscriptions","organizations_url":"https://api.github.com/users/ivankreso/orgs","repos_url":"https://api.github.com/users/ivankreso/repos","events_url":"https://api.github.com/users/ivankreso/events{/privacy}","received_events_url":"https://api.github.com/users/ivankreso/received_events","type":"User","site_admin":false},"created_at":"2018-06-11T18:47:34Z","updated_at":"2018-06-11T18:47:34Z","author_association":"NONE","body":"I have fixed this by creating and passing the DummyFunction below instead of nullptr here:\r\nhttps://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/engine.cpp#L262\r\n\r\n```\r\nstruct DummyFunction : public Function {\r\n  virtual variable_list apply(const variable_list& inputs) {\r\n    return outputs;\r\n  };\r\n\r\n  variable_list outputs;\r\n};\r\n```\r\nI forgot to mention that this segfault appears only when I use torch.utils.checkpoint feature."}}
{"actor":{"display_login":"ivankreso"}}