{"action":"closed","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/13288","repository_url":"https://api.github.com/repos/pydata/pandas","labels_url":"https://api.github.com/repos/pydata/pandas/issues/13288/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/13288/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/13288/events","html_url":"https://github.com/pydata/pandas/issues/13288","id":156863011,"number":13288,"title":"BUG: DataFrame.describe() breaks with a column index of object type and numeric entries","user":{"login":"pijucha","id":9507544,"avatar_url":"https://avatars.githubusercontent.com/u/9507544?v=3","gravatar_id":"","url":"https://api.github.com/users/pijucha","html_url":"https://github.com/pijucha","followers_url":"https://api.github.com/users/pijucha/followers","following_url":"https://api.github.com/users/pijucha/following{/other_user}","gists_url":"https://api.github.com/users/pijucha/gists{/gist_id}","starred_url":"https://api.github.com/users/pijucha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pijucha/subscriptions","organizations_url":"https://api.github.com/users/pijucha/orgs","repos_url":"https://api.github.com/users/pijucha/repos","events_url":"https://api.github.com/users/pijucha/events{/privacy}","received_events_url":"https://api.github.com/users/pijucha/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pydata/pandas/labels/Bug","name":"Bug","color":"e10c02"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Difficulty%20Novice","name":"Difficulty Novice","color":"fbca04"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Effort%20Low","name":"Effort Low","color":"006b75"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/pydata/pandas/milestones/40","html_url":"https://github.com/pydata/pandas/milestones/0.18.2","labels_url":"https://api.github.com/repos/pydata/pandas/milestones/40/labels","id":1639795,"number":40,"title":"0.18.2","description":"","creator":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"open_issues":183,"closed_issues":137,"state":"open","created_at":"2016-03-11T21:24:45Z","updated_at":"2016-05-31T14:12:52Z","due_on":"2016-06-14T04:00:00Z","closed_at":null},"comments":7,"created_at":"2016-05-25T22:17:16Z","updated_at":"2016-05-31T14:12:51Z","closed_at":"2016-05-31T14:12:51Z","body":"Preparing a commit for another issue in `.describe()`, I encountered this puzzling bug, surprisingly easy to trigger.\r\n\r\n#### Symptoms\r\n\r\n```python\r\ndf = pd.DataFrame({'A': list(\"BCDE\"), 0: [1,2,3,4]})\r\ndf.describe()\r\n# Long traceback listing formatting and internal functions...\r\nValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'\r\n```\r\n\r\nHowever:\r\n\r\n```python\r\ndf.describe(include='all')\r\n               0    A\r\ncount   4.000000    4\r\nunique       NaN    4\r\ntop          NaN    D\r\nfreq         NaN    1\r\nmean    2.500000  NaN\r\nstd     1.290994  NaN\r\nmin     1.000000  NaN\r\n25%     1.750000  NaN\r\n50%     2.500000  NaN\r\n75%     3.250000  NaN\r\nmax     4.000000  NaN\r\n\r\n# It's OK if we don't print on screen:\r\nx = df.describe()\r\nx.columns\r\nOut[8]: Index([0], dtype='int64')\r\n\r\n# Fixing this suspicious index (int works too):\r\nx.columns = x.columns.astype(object)\r\nx\r\nOut[10]: \r\n              0\r\ncount  4.000000\r\nmean   2.500000\r\nstd    1.290994\r\nmin    1.000000\r\n25%    1.750000\r\n50%    2.500000\r\n75%    3.250000\r\nmax    4.000000\r\n```\r\n\r\nSame issue happens with a simpler data frame:\r\n```python\r\ndf0 = pd.DataFrame([1,2,3,4])\r\n# It's  OK now\r\ndf0.describe()\r\nOut[28]: \r\n              0\r\ncount  4.000000\r\nmean   2.500000\r\nstd    1.290994\r\nmin    1.000000\r\n25%    1.750000\r\n50%    2.500000\r\n75%    3.250000\r\nmax    4.000000\r\n\r\n# Modify column index:\r\ndf0.columns = pd.Index([0], dtype=object)\r\ndf0.describe()\r\n# ...\r\nValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'\r\n```\r\nCurrent version (but the bug is also present in pandas release 0.18.1):\r\n```\r\npd.show_versions()\r\n\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.5.1.final.0\r\npython-bits: 64\r\nOS: Linux\r\nOS-release: 4.1.20-1\r\nmachine: x86_64\r\nprocessor: Intel(R)_Core(TM)_i5-2520M_CPU_@_2.50GHz\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: en_US.UTF-8\r\n\r\npandas: 0.18.1+64.g7ed22fe.dirty\r\nnose: 1.3.7\r\npip: 8.1.2\r\nsetuptools: 21.0.0\r\nCython: 0.24\r\nnumpy: 1.11.0\r\nscipy: 0.17.0.dev0+3f3c371\r\nIPython: 4.0.1\r\n...\r\n```\r\n\r\n#### Reason\r\nSome internal function gets confused by dtypes of a column index, I guess. But the faulty index is created in `.describe()`.\r\n```python\r\n# Output from %debug df.describe()\r\n# NDFrame.describe() in pandas/core/generic.py:\r\n#\r\n   4943             data = self\r\n   4944         else:\r\n   4945             data = self.select_dtypes(include=include, exclude=exclude)\r\n   4946 \r\n   4947         ldesc = [describe_1d(s, percentiles) for _, s in data.iteritems()]\r\n   4948         # set a convenient order for rows\r\n   4949         names = []\r\n   4950         ldesc_indexes = sorted([x.index for x in ldesc], key=len)\r\n   4951         for idxnames in ldesc_indexes:\r\n   4952             for name in idxnames:\r\n   4953                 if name not in names:\r\n   4954                     names.append(name)\r\n   4955 \r\n   4956         d = pd.concat(ldesc, join_axes=pd.Index([names]), axis=1)\r\n1> 4957         d.columns = self.columns._shallow_copy(values=d.columns.values)\r\n   4958         d.columns.names = data.columns.names\r\n   4959         return d\r\n```\r\n`_shallow_copy()` in the marked line changes `d.columns`:\r\n```python\r\nipdb> p d.columns\r\nInt64Index([0], dtype='int64')\r\nipdb> n\r\n> /home/users/piotr/workspace/pandas-pijucha/pandas/core/generic.py(4958)describe()\r\n1  4957         d.columns = self.columns._shallow_copy(values=d.columns.values)\r\n-> 4958         d.columns.names = data.columns.names\r\n   4959         return d\r\nipdb> p d.columns\r\nIndex([0], dtype='int64')\r\n```\r\n#### Possible solutions\r\n\r\nLines 4957-4958 are actually used to fix issues that `pd.concat` brings about. They try to pass the column structure from `self` to `d`.\r\nI think a simpler solution is replacing these lines with:\r\n```python\r\n d = pd.concat(ldesc, join_axes=pd.Index([names]), axis=1)\r\n d.columns = data.columns\r\n return d\r\n```\r\nor\r\n```python\r\nd = pd.DataFrame(pd.concat(ldesc, axis=1), index = pd.Index(names), columns = data.columns)\r\nreturn d\r\n```\r\n`data` is a subframe of `self` and retains the same column structure.\r\n\r\n`pd.concat` has some parameters that help pass a hierarchical index but can't do anything on its own with a categorical one.\r\n\r\nI'm going to submit a pull request with this fix together with some others related with `describe()`. I hope I haven't overlooked anything obvious. But if so, any comments are very welcome.\r\n"}}
{"actor":{"display_login":"jreback"}}