{"action":"created","issue":{"url":"https://api.github.com/repos/PyTables/PyTables/issues/531","repository_url":"https://api.github.com/repos/PyTables/PyTables","labels_url":"https://api.github.com/repos/PyTables/PyTables/issues/531/labels{/name}","comments_url":"https://api.github.com/repos/PyTables/PyTables/issues/531/comments","events_url":"https://api.github.com/repos/PyTables/PyTables/issues/531/events","html_url":"https://github.com/PyTables/PyTables/issues/531","id":140939885,"number":531,"title":"OverflowError: writing pandas data frame to hdf5","user":{"login":"DavidTolan","id":4189503,"avatar_url":"https://avatars.githubusercontent.com/u/4189503?v=3","gravatar_id":"","url":"https://api.github.com/users/DavidTolan","html_url":"https://github.com/DavidTolan","followers_url":"https://api.github.com/users/DavidTolan/followers","following_url":"https://api.github.com/users/DavidTolan/following{/other_user}","gists_url":"https://api.github.com/users/DavidTolan/gists{/gist_id}","starred_url":"https://api.github.com/users/DavidTolan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DavidTolan/subscriptions","organizations_url":"https://api.github.com/users/DavidTolan/orgs","repos_url":"https://api.github.com/users/DavidTolan/repos","events_url":"https://api.github.com/users/DavidTolan/events{/privacy}","received_events_url":"https://api.github.com/users/DavidTolan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-03-15T11:07:20Z","updated_at":"2016-03-23T03:49:56Z","closed_at":null,"body":"sorry I can't give you more details, but much of the data is confidential\r\n\r\n**Versions of code**\r\npandas                    0.17.0              np110py27_0\r\npytables                  3.2.2               np110py27_1\r\n\r\nshape of the dataframe (it's quite big)\r\n(57163046, 24)\r\n\r\nI have printed out the min/max of all int64s in the frame just before storing, eg\r\nprint frame[frame.dtypes[frame.dtypes == \"int64\"].index.tolist()].max()\r\nnone of the max are close to 2^63, nor are there any large negative numbers.\r\n\r\nsome of the data frame fields are strings, perhaps 80 chars or longer.\r\n\r\n**original call to pandas**\r\nframe.to_hdf(hdf5_file_name, self.key_name(name_tuple), **complevel=9, complib='blosc')**\r\n\r\ntrace back\r\n  File \"C:\\Anaconda2\\lib\\site-packages\\log_file_parser-0.1.10-py2.7.egg\\log_file_parser\\src\\hdf5.py\", line 39, in write_frames\r\n    frame.to_hdf(hdf5_file_name, self.key_name(name_tuple), complevel=9, complib='blosc')\r\n  File \"C:\\Anaconda2\\lib\\site-packages\\pandas\\core\\generic.py\", line 938, in to_hdf\r\n    return pytables.to_hdf(path_or_buf, key, self, **kwargs)\r\n  File \"C:\\Anaconda2\\lib\\site-packages\\pandas\\io\\pytables.py\", line 270, in to_hdf\r\n    f(store)\r\n  File \"C:\\Anaconda2\\lib\\site-packages\\pandas\\io\\pytables.py\", line 265, in <lambda>\r\n    f = lambda store: store.put(key, value, **kwargs)\r\n  File \"C:\\Anaconda2\\lib\\site-packages\\pandas\\io\\pytables.py\", line 827, in put\r\n    self._write_to_group(key, value, append=append, **kwargs)\r\n  File \"C:\\Anaconda2\\lib\\site-packages\\pandas\\io\\pytables.py\", line 1265, in _write_to_group\r\n    s.write(obj=value, append=append, complib=complib, **kwargs)\r\n  File \"C:\\Anaconda2\\lib\\site-packages\\pandas\\io\\pytables.py\", line 2805, in write\r\n    self.write_array('block%d_values' % i, blk.values, items=blk_items)\r\n  File \"C:\\Anaconda2\\lib\\site-packages\\pandas\\io\\pytables.py\", line 2572, in write_array\r\n    vlarr.append(value)\r\n  File \"C:\\Anaconda2\\lib\\site-packages\\tables\\vlarray.py\", line 537, in append\r\n    self._append(nparr, nobjects)\r\n  File \"tables\\hdf5extension.pyx\", line 1929, in tables.hdf5extension.VLArray._append (tables\\hdf5extension.c:20849)\r\nOverflowError: Python int too large to convert to C long\r\n\r\nI have a work around by discarding some less important data, and ending up with a much smaller dataframe. It looks like the problem is with the size of the frame, but it could also be in the discarded data.\r\n\r\nIf you have any specific questions, on reproducing this, I will help as best I can.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/PyTables/PyTables/issues/comments/200155154","html_url":"https://github.com/PyTables/PyTables/issues/531#issuecomment-200155154","issue_url":"https://api.github.com/repos/PyTables/PyTables/issues/531","id":200155154,"user":{"login":"andreabedini","id":69135,"avatar_url":"https://avatars.githubusercontent.com/u/69135?v=3","gravatar_id":"","url":"https://api.github.com/users/andreabedini","html_url":"https://github.com/andreabedini","followers_url":"https://api.github.com/users/andreabedini/followers","following_url":"https://api.github.com/users/andreabedini/following{/other_user}","gists_url":"https://api.github.com/users/andreabedini/gists{/gist_id}","starred_url":"https://api.github.com/users/andreabedini/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreabedini/subscriptions","organizations_url":"https://api.github.com/users/andreabedini/orgs","repos_url":"https://api.github.com/users/andreabedini/repos","events_url":"https://api.github.com/users/andreabedini/events{/privacy}","received_events_url":"https://api.github.com/users/andreabedini/received_events","type":"User","site_admin":false},"created_at":"2016-03-23T03:49:56Z","updated_at":"2016-03-23T03:49:56Z","body":"[This](https://github.com/PyTables/PyTables/blob/v.3.2.2/tables/hdf5extension.pyx#L1929) is the incriminated line. `_append` is declared `def _append(self, ndarray nparr, int nobjects)`, and this breaks if nobjects doesn't fit into an C sized int.\r\n"}}