<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Hello,</p>
<p dir="auto">I'm having a problem of loading a serialized tensor from a file.<br>
My tensor shape is [309000001, 2, 5] the dtype is torch.int8<br>
When I deserialize the tensor using torch.load(), it yell "invalid memory size".<br>
The line that it complain is:<br>
</p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/pytorch/pytorch/blob/b740b92f3600840e09d4c93f3138333838d1e474/torch/serialization.py#L528-L529">pytorch/torch/serialization.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 528 to 529
      in
      <a data-pjax="true" class="commit-tease-sha" href="/pytorch/pytorch/commit/b740b92f3600840e09d4c93f3138333838d1e474">b740b92</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L528" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="528"></td>
          <td id="LC528" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">deserialized_objects</span>[<span class="pl-s1">root_key</span>] <span class="pl-c1">=</span> <span class="pl-en">restore_location</span>( </td>
        </tr>

        <tr class="border-0">
          <td id="L529" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="529"></td>
          <td id="LC529" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-en">data_type</span>(<span class="pl-s1">size</span>), <span class="pl-s1">location</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<br>
It seem like the variable it used is destructured from here:<br>
<div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/pytorch/pytorch/blob/b740b92f3600840e09d4c93f3138333838d1e474/torch/serialization.py#L525">pytorch/torch/serialization.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 525
      in
      <a data-pjax="true" class="commit-tease-sha" href="/pytorch/pytorch/commit/b740b92f3600840e09d4c93f3138333838d1e474">b740b92</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L525" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="525"></td>
          <td id="LC525" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">data_type</span>, <span class="pl-s1">root_key</span>, <span class="pl-s1">location</span>, <span class="pl-s1">size</span>, <span class="pl-s1">view_metadata</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<br>
Which also came from here:<br>
<div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/pytorch/pytorch/blob/b740b92f3600840e09d4c93f3138333838d1e474/torch/serialization.py#L517">pytorch/torch/serialization.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 517
      in
      <a data-pjax="true" class="commit-tease-sha" href="/pytorch/pytorch/commit/b740b92f3600840e09d4c93f3138333838d1e474">b740b92</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L517" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="517"></td>
          <td id="LC517" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">saved_id</span>[<span class="pl-c1">1</span>:] </td>
        </tr>
    </tbody></table>
  </div>
</div>
<br>
So I insert <code class="notranslate">print(saved_id)</code> where it yell:<br>
Now, it print:<br>
<code class="notranslate">('storage', &lt;class 'torch.CharStorage'&gt;, '2149339138576', 'cpu', -1204967286, None)</code><p></p>
<p dir="auto">It seem that size is negative.</p>
<h2 dir="auto">To Reproduce</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch

torch.save(torch.zeros([309237982, 2, 5], dtype=torch.int8), './sample.pt')
torch.load('./sample.pt')"><pre class="notranslate"><code class="notranslate">import torch

torch.save(torch.zeros([309237982, 2, 5], dtype=torch.int8), './sample.pt')
torch.load('./sample.pt')
</code></pre></div>
<p dir="auto">It can only be reproduce in MS.Windows environment.<br>
My Windows version is:<br>
10.0.17134.472<br>
My Python version is:<br>
3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 13:35:33) [MSC v.1900 64 bit (AMD64)]<br>
Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Run above code in MS. Windows</li>
</ol>

<hr>
<p dir="auto">RuntimeError                              Traceback (most recent call last)<br>
 in <br>
2<br>
3 torch.save(torch.zeros([309237982, 2, 5], dtype=torch.int8), './sample.pt')<br>
----&gt; 4 torch.load('./sample.pt')</p>
<p dir="auto">c:\users\nattapong\workspace\lstm_nlp\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module)<br>
365         f = open(f, 'rb')<br>
366     try:<br>
--&gt; 367         return _load(f, map_location, pickle_module)<br>
368     finally:<br>
369         if new_fd:</p>
<p dir="auto">c:\users\nattapong\workspace\lstm_nlp\lib\site-packages\torch\serialization.py in _load(f, map_location, pickle_module)<br>
538     unpickler = pickle_module.Unpickler(f)<br>
539     unpickler.persistent_load = persistent_load<br>
--&gt; 540     result = unpickler.load()<br>
541<br>
542     deserialized_storage_keys = pickle_module.load(f)</p>
<p dir="auto">c:\users\nattapong\workspace\lstm_nlp\lib\site-packages\torch\serialization.py in persistent_load(saved_id)<br>
504                 print(data_type, size)<br>
505                 deserialized_objects[root_key] = restore_location(<br>
--&gt; 506                     data_type(size), location)<br>
507             storage = deserialized_objects[root_key]<br>
508             if view_metadata is not None:</p>
<p dir="auto">RuntimeError: $ Torch: invalid memory size -- maybe an overflow? at ..\aten\src\TH\THGeneral.cpp:188</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Tensor is loaded without error.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">Please copy and paste the output from our<br>
<a href="https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py" rel="nofollow">environment collection script</a><br>
(or fill out the checklist below manually).</p>
<p dir="auto">You can get the script and run it with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py"><pre class="notranslate"><code class="notranslate">wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
</code></pre></div>
<ul dir="auto">
<li>PyTorch Version (e.g., 1.0): 1.0</li>
<li>OS (e.g., Linux): Windows 10</li>
<li>How you installed PyTorch (<code class="notranslate">conda</code>, <code class="notranslate">pip</code>, source): pip</li>
<li>Build command you used (if compiling from source):</li>
<li>Python version: 3.6.7</li>
<li>CUDA/cuDNN version: N/A</li>
<li>GPU models and configuration: N/A</li>
<li>Any other relevant information: No problem on Linux</li>
</ul>
<h2 dir="auto">Additional context</h2>

<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/peterjc123/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/peterjc123">@peterjc123</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mszhanyi/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mszhanyi">@mszhanyi</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/skyline75489/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/skyline75489">@skyline75489</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nbcsm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nbcsm">@nbcsm</a></p>
<h2 dir="auto">Issue description</h2>
<p dir="auto">From this <a href="https://discuss.pytorch.org/t/pytorch-windows-eoferror-ran-out-of-input-when-num-workers-0/25918/2" rel="nofollow">post</a>, the following error will be thrown when the input size exceeds 4GB.</p>
<div class="highlight highlight-text-python-traceback notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;train.py&quot;, line 159, in &lt;module&gt;
    main()
  File &quot;train.py&quot;, line 111, in main
    for i_batch, sample_batched in enumerate(Dataloader_Train):
  File &quot;C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\site-packages\torch\utils\data\dataloader.py&quot;, line 451, in __iter__
    return _DataLoaderIter(self)
  File &quot;C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\site-packages\torch\utils\data\dataloader.py&quot;, line 239, in __init__
    w.start()
  File &quot;C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\process.py&quot;, line 105, in start
    self._popen = self._Popen(self)
  File &quot;C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\context.py&quot;, line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File &quot;C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\context.py&quot;, line 313, in _Popen
    return Popen(process_obj)
  File &quot;C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\popen_spawn_win32.py&quot;, line 66, in __init__
    reduction.dump(process_obj, to_child)
  File &quot;C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\reduction.py&quot;, line 59, in dump
    ForkingPickler(file, protocol).dump(obj)
OverflowError: cannot serialize a bytes object larger than 4 GiB

(py3_pt3_orig) C:\Users\TestUser\Documents\eliaseulig\DeepDSA&gt;Traceback (most recent call last):
  File &quot;&lt;string&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\spawn.py&quot;, line 106, in spawn_main
    exitcode = _main(fd)
  File &quot;C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\spawn.py&quot;, line 116, in _main
    self = pickle.load(from_parent)
EOFError: Ran out of input"><pre class="notranslate">Traceback (most recent call last):
  File <span class="pl-s">"train.py"</span>, line <span class="pl-c1">159</span>, in <span class="pl-en">&lt;module&gt;</span>
    main()
  File <span class="pl-s">"train.py"</span>, line <span class="pl-c1">111</span>, in <span class="pl-en">main</span>
    <span class="pl-k">for</span> i_batch, sample_batched <span class="pl-k">in</span> <span class="pl-c1">enumerate</span>(Dataloader_Train):
  File <span class="pl-s">"C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\site-packages\torch\utils\data\dataloader.py"</span>, line <span class="pl-c1">451</span>, in <span class="pl-en">__iter__</span>
    <span class="pl-k">return</span> _DataLoaderIter(<span class="pl-c1">self</span>)
  File <span class="pl-s">"C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\site-packages\torch\utils\data\dataloader.py"</span>, line <span class="pl-c1">239</span>, in <span class="pl-en">__init__</span>
    w.start()
  File <span class="pl-s">"C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\process.py"</span>, line <span class="pl-c1">105</span>, in <span class="pl-en">start</span>
    <span class="pl-c1">self</span>._popen <span class="pl-k">=</span> <span class="pl-c1">self</span>._Popen(<span class="pl-c1">self</span>)
  File <span class="pl-s">"C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\context.py"</span>, line <span class="pl-c1">212</span>, in <span class="pl-en">_Popen</span>
    <span class="pl-k">return</span> _default_context.get_context().Process._Popen(process_obj)
  File <span class="pl-s">"C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\context.py"</span>, line <span class="pl-c1">313</span>, in <span class="pl-en">_Popen</span>
    <span class="pl-k">return</span> Popen(process_obj)
  File <span class="pl-s">"C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\popen_spawn_win32.py"</span>, line <span class="pl-c1">66</span>, in <span class="pl-en">__init__</span>
    reduction.dump(process_obj, to_child)
  File <span class="pl-s">"C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\reduction.py"</span>, line <span class="pl-c1">59</span>, in <span class="pl-en">dump</span>
    ForkingPickler(<span class="pl-v">file</span>, protocol).dump(obj)
<span class="pl-en">OverflowError</span>: <span class="pl-s">cannot serialize a bytes object larger than 4 GiB</span>

(py3_pt3_orig) C:\Users\TestUser\Documents\eliaseulig\DeepDSA&gt;Traceback (most recent call last):
  File <span class="pl-s">"&lt;string&gt;"</span>, line <span class="pl-c1">1</span>, in <span class="pl-en">&lt;module&gt;</span>
  File <span class="pl-s">"C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\spawn.py"</span>, line <span class="pl-c1">106</span>, in <span class="pl-en">spawn_main</span>
    exitcode <span class="pl-k">=</span> _main(fd)
  File <span class="pl-s">"C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\spawn.py"</span>, line <span class="pl-c1">116</span>, in <span class="pl-en">_main</span>
    <span class="pl-c1">self</span> <span class="pl-k">=</span> pickle.load(from_parent)
<span class="pl-en">EOFError</span>: <span class="pl-s">Ran out of input</span></pre></div>
<h2 dir="auto">Code example</h2>
<p dir="auto">Please try to provide a minimal example to repro the bug.<br>
Error messages and stack traces are also helpful.</p>