text: user9: I ran into the following error when using cooler to load a contact pairs file:
  ubuntu@ip-172-31-8-42:/data/hg-tmp$ FILE=GSE109344_monkey_fibro_allValidPairs.txt.gz; sudo docker run -v $(pwd):/data $TAG cooler cload pairs -c1 2 -p1 3 -c2 5 -p2 6 /data/chromInfo.rheMac2.
  simple.txt:${BINSIZE} /data/$FILE /data/${FILE}.cool;
  Traceback (most recent call last):
    File "/usr/local/bin/cooler", line 11, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python2.7/site-packages/cooler/cli/cload.py", line 477, in pairs
      map(pipeline, reader),
    File "cytoolz/functoolz.pyx", line 493, in cytoolz.functoolz.Compose.__call__
    File "/usr/local/lib/python2.7/site-packages/cooler/create/_ingest.py", line 425, in _aggregate_records
      .aggregate(agg)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/generic.py", line 1315, in aggregate
      return super(DataFrameGroupBy, self).aggregate(arg, *args, **kwargs)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/generic.py", line 186, in aggregate
      result, how = self._aggregate(arg, _level=_level, *args, **kwargs)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/base.py", line 498, in _aggregate
      result = _agg(arg, _agg_1dim)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/base.py", line 449, in _agg
      result[fname] = func(fname, agg_how)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/base.py", line 432, in _agg_1dim
      return colg.aggregate(how, _level=(_level or 0) + 1)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/generic.py", line 760, in aggregate
      return getattr(self, func_or_funcs)(*args, **kwargs)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/groupby.py", line 1227, in size
      result = self.grouper.size()
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/ops.py", line 225, in size
      ids, _, ngroup = self.group_info
    File "pandas/_libs/properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/ops.py", line 252, in group_info
      comp_ids, obs_group_ids = self._get_compressed_labels()
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/ops.py", line 272, in _get_compressed_labels
      return compress_group_index(group_index, sort=self.sort)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/sorting.py", line 379, in compress_group_index
      comp_ids, obs_group_ids = table.get_labels_groupby(group_index)
    File "pandas/_libs/hashtable_class_helper.pxi", line 1255, in pandas._libs.hashtable.Int64HashTable.get_labels_groupby
    File "pandas/_libs/hashtable_class_helper.pxi", line 199, in pandas._libs.hashtable.Int64Vector.resize
  MemoryError: cannot allocate memory for array
  
  The machine has 8Gb of RAM. Any ideas?
  
  user4497: It seems to be some kind of a memory leak in pandas's read_csv, maybe due to it doing gzip-decompression.
  Doing the decompression in a separate process and piping the result into cooler cload pairs should do the trick.
  gzip -dc GSE109344_monkey_fibro_allValidPairs.txt.gz | cooler cload pairs -c1 2 -p1 3 -c2 5 -p2 6 /data/chromInfo.rheMac2.
  simple.txt:${BINSIZE} - /data/${FILE}.cool
  
  
original_text: 9: <p>I ran into the following error when using <code>cooler</code> to load a contact pairs file:</p>
  
  <pre><code>ubuntu@ip-172-31-8-42:/data/hg-tmp$ FILE=GSE109344_monkey_fibro_allValidPairs.txt.gz; sudo docker run -v $(pwd):/data $TAG cooler cload pairs -c1 2 -p1 3 -c2 5 -p2 6 /data/chromInfo.rheMac2.
  simple.txt:${BINSIZE} /data/$FILE /data/${FILE}.cool;
  Traceback (most recent call last):
    File "/usr/local/bin/cooler", line 11, in &lt;module&gt;
      sys.exit(cli())
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python2.7/site-packages/cooler/cli/cload.py", line 477, in pairs
      map(pipeline, reader),
    File "cytoolz/functoolz.pyx", line 493, in cytoolz.functoolz.Compose.__call__
    File "/usr/local/lib/python2.7/site-packages/cooler/create/_ingest.py", line 425, in _aggregate_records
      .aggregate(agg)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/generic.py", line 1315, in aggregate
      return super(DataFrameGroupBy, self).aggregate(arg, *args, **kwargs)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/generic.py", line 186, in aggregate
      result, how = self._aggregate(arg, _level=_level, *args, **kwargs)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/base.py", line 498, in _aggregate
      result = _agg(arg, _agg_1dim)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/base.py", line 449, in _agg
      result[fname] = func(fname, agg_how)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/base.py", line 432, in _agg_1dim
      return colg.aggregate(how, _level=(_level or 0) + 1)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/generic.py", line 760, in aggregate
      return getattr(self, func_or_funcs)(*args, **kwargs)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/groupby.py", line 1227, in size
      result = self.grouper.size()
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/ops.py", line 225, in size
      ids, _, ngroup = self.group_info
    File "pandas/_libs/properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/ops.py", line 252, in group_info
      comp_ids, obs_group_ids = self._get_compressed_labels()
    File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/ops.py", line 272, in _get_compressed_labels
      return compress_group_index(group_index, sort=self.sort)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/sorting.py", line 379, in compress_group_index
      comp_ids, obs_group_ids = table.get_labels_groupby(group_index)
    File "pandas/_libs/hashtable_class_helper.pxi", line 1255, in pandas._libs.hashtable.Int64HashTable.get_labels_groupby
    File "pandas/_libs/hashtable_class_helper.pxi", line 199, in pandas._libs.hashtable.Int64Vector.resize
  MemoryError: cannot allocate memory for array
  </code></pre>
  
  <p>The machine has 8Gb of RAM. Any ideas?</p>
   4497: <p>It seems to be some kind of a memory leak in pandas's <code>read_csv</code>, maybe due to it doing gzip-decompression.</p>
  
  <p>Doing the decompression in a separate process and piping the result into <code>cooler cload pairs</code> should do the trick.</p>
  
  <pre><code>gzip -dc GSE109344_monkey_fibro_allValidPairs.txt.gz | cooler cload pairs -c1 2 -p1 3 -c2 5 -p2 6 /data/chromInfo.rheMac2.
  simple.txt:${BINSIZE} - /data/${FILE}.cool
  </code></pre>
  