{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/9590","repository_url":"https://api.github.com/repos/pydata/pandas","labels_url":"https://api.github.com/repos/pydata/pandas/issues/9590/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/9590/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/9590/events","html_url":"https://github.com/pydata/pandas/issues/9590","id":59860276,"number":9590,"title":"Support for tables with unlimited columns in HDF","user":{"login":"Chris7","id":963909,"avatar_url":"https://avatars.githubusercontent.com/u/963909?v=3","gravatar_id":"","url":"https://api.github.com/users/Chris7","html_url":"https://github.com/Chris7","followers_url":"https://api.github.com/users/Chris7/followers","following_url":"https://api.github.com/users/Chris7/following{/other_user}","gists_url":"https://api.github.com/users/Chris7/gists{/gist_id}","starred_url":"https://api.github.com/users/Chris7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Chris7/subscriptions","organizations_url":"https://api.github.com/users/Chris7/orgs","repos_url":"https://api.github.com/users/Chris7/repos","events_url":"https://api.github.com/users/Chris7/events{/privacy}","received_events_url":"https://api.github.com/users/Chris7/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pydata/pandas/labels/Difficulty%20Advanced","name":"Difficulty Advanced","color":"fbca04"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Effort%20High","name":"Effort High","color":"006b75"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Enhancement","name":"Enhancement","color":"4E9A06"},{"url":"https://api.github.com/repos/pydata/pandas/labels/IO%20HDF5","name":"IO HDF5","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/pydata/pandas/milestones/32","html_url":"https://github.com/pydata/pandas/milestones/Next%20Major%20Release","labels_url":"https://api.github.com/repos/pydata/pandas/milestones/32/labels","id":933188,"number":32,"title":"Next Major Release","description":"after 0.18.0 of course","creator":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"open_issues":995,"closed_issues":140,"state":"open","created_at":"2015-01-13T10:53:19Z","updated_at":"2016-08-10T22:23:10Z","due_on":"2017-08-31T04:00:00Z","closed_at":null},"comments":10,"created_at":"2015-03-04T20:55:36Z","updated_at":"2016-08-10T22:30:58Z","closed_at":null,"body":"I wrote a small class that will allow users to utilize HDF tables with significantly more columns than the ~2000 limit currently imposed. I work with mass spec data, and have matrices of ~20,000 x 20,000 x sample_number to deal with so I need things to be written on disk, and compressed.\r\n\r\nI'm asking if this is something that would be appreciated to be cleaned up and put in as a PR. It's currently just addressing my needs as they arise, but I could extend and optimize it if there's interest. Alternatively, if I'm reinventing a wheel that already exists in pandas, I'd love to know that as well.\r\n\r\nHere's the code for the interested\r\n\r\n```python\r\nclass WideTable(object):\r\n    \"\"\"\r\n    Breaks up a massive table into individual chunks and stores these chunks as separate tables.\r\n    Extends the select and append method to be pandas-like and handle this indexing to mimic a single\r\n    DataFrame\r\n    \"\"\"\r\n    def __init__(self, store, chunk_size=None, min_itemsize=255):\r\n        self.store = store\r\n        self.index_name = '_store_index'\r\n        self.chunk_size_name = '_chunk_size'\r\n        self.index_itemsize = min_itemsize\r\n        try:\r\n            self.store_index = self.store.select(self.index_name)\r\n        except KeyError:\r\n            self.store_index = pd.DataFrame(columns=['chunk'])\r\n            self.store.put(self.index_name, self.store_index)\r\n        finally:\r\n            self.tables = [i[0] for i in self.store.iteritems()]\r\n        try:\r\n            # stored as a series\r\n            self.chunk_size = self.store.select(self.chunk_size_name)\r\n        except KeyError:\r\n            self.chunk_size = pd.Series(chunk_size if chunk_size is not None else 100)\r\n            self.store.append(self.chunk_size_name, self.chunk_size)\r\n        self.chunk_size = self.chunk_size[0]\r\n        \r\n    def get_chunk_name(self, name, chunk):\r\n        return '_{0}_{1}'.format(name, chunk)\r\n        \r\n    def append(self, name, data, column=0):\r\n        # get the existing chunks from the data and insert them to those blocks\r\n        chunks = self.store_index['chunk']\r\n        existing_map = chunks[data.columns]\r\n        new = existing_map[existing_map.isnull()] # Series of new columns\r\n        existing = existing_map.dropna().to_dict() # dict of {'column': 'chunk'}, we want {'chunk': columns set}\r\n        existing_rev = {}\r\n        for key, value in existing.iteritems():\r\n            try:\r\n                existing_rev[value].append(key)\r\n            except KeyError:\r\n                existing_rev[value] = [key]\r\n        \r\n        # update existing entries\r\n        columns_added = []\r\n        for chunk_index, chunk_columns in existing_rev.iteritems():\r\n            print self.get_chunk_name(name, chunk_index), chunk_columns\r\n            self.store.append(self.get_chunk_name(name, chunk_index), data.loc[:,chunk_columns])\r\n            columns_added.extend(chunk_columns)\r\n        \r\n        new_cols = list(set(data.columns)-set(columns_added))\r\n        if new_cols:\r\n            data2 = data[new_cols]\r\n            current_chunk = chunks.astype('int').max()\r\n            if pd.isnull(current_chunk):\r\n                current_chunk = 0\r\n            space_taken = len(chunks.where(chunks == str(current_chunk)).dropna())\r\n            if space_taken >= self.chunk_size:\r\n                current_chunk += 1\r\n                end = self.chunk_size\r\n            else:\r\n                end = self.chunk_size-space_taken\r\n            if end > len(new_cols):\r\n                end = len(new_cols)\r\n            start = 0\r\n            while start < len(new_cols):\r\n                chunk_name = self.get_chunk_name(name, current_chunk)\r\n                td = data2.iloc[:,start:end]\r\n                self.store.append(chunk_name, td)\r\n                for col_name in td.columns:\r\n                    self.store_index.loc[col_name,'chunk'] = str(current_chunk)\r\n                self.tables.append(chunk_name)\r\n                start = end\r\n                end += self.chunk_size\r\n                if end > len(new_cols):\r\n                    end = len(new_cols)\r\n                current_chunk += 1\r\n            self.store.put(self.index_name, self.store_index)\r\n        \r\n    def get_matching_tables(self, name):\r\n        return [i for i in self.tables if i.startswith('/_{0}_'.format(name))]\r\n        \r\n    def select(self, name, **kwargs):\r\n        t = pd.DataFrame()\r\n        for i in self.get_matching_tables(name):\r\n            values = self.store.select(i,**kwargs)\r\n            if values.empty:\r\n                continue\r\n            t = t.join(values, how='outer')\r\n        return t\r\n```\r\n\r\nBasic usage:\r\n\r\n```python\r\nimport pandas as pd\r\nimport numpy as np\r\n# chunk size is how many columns you want per sub-table and is only required for new tables\r\nwt = WideTable(pd.HDFStore('wide_table.h5', complevel=9, complib='blosc'), chunk_size=3)\r\ndf = pd.DataFrame(np.random.randn(8, 10), index=range(8), columns=[chr(i) for i in xrange(70,70+10)])\r\nwt.append('test_table', df)\r\n\r\n# show that chunk size is optional\r\nwt = WideTable(pd.HDFStore('wide_table.h5', complevel=9, complib='blosc'))\r\nprint wt.chunk_size\r\n# select data\r\nprint wt.select('test_table', where='columns==J')\r\n# across tables\r\nprint wt.select('test_table', where='columns==[N,I,H]')\r\nprint wt.select('test_table', where='columns==[F,J,O]')\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/pydata/pandas/issues/comments/239024901","html_url":"https://github.com/pydata/pandas/issues/9590#issuecomment-239024901","issue_url":"https://api.github.com/repos/pydata/pandas/issues/9590","id":239024901,"user":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"created_at":"2016-08-10T22:30:58Z","updated_at":"2016-08-10T22:30:58Z","body":"@hnykda this is just a proposal. would take a pull-request to complete this."}}
{"actor":{"display_login":"jreback"}}