{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/13335","repository_url":"https://api.github.com/repos/pydata/pandas","labels_url":"https://api.github.com/repos/pydata/pandas/issues/13335/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/13335/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/13335/events","html_url":"https://github.com/pydata/pandas/issues/13335","id":157717126,"number":13335,"title":"Pandas groupby extremely slow in python3 for certain sets of single precision floating point data","user":{"login":"RogerThomas","id":3211163,"avatar_url":"https://avatars.githubusercontent.com/u/3211163?v=3","gravatar_id":"","url":"https://api.github.com/users/RogerThomas","html_url":"https://github.com/RogerThomas","followers_url":"https://api.github.com/users/RogerThomas/followers","following_url":"https://api.github.com/users/RogerThomas/following{/other_user}","gists_url":"https://api.github.com/users/RogerThomas/gists{/gist_id}","starred_url":"https://api.github.com/users/RogerThomas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RogerThomas/subscriptions","organizations_url":"https://api.github.com/users/RogerThomas/orgs","repos_url":"https://api.github.com/users/RogerThomas/repos","events_url":"https://api.github.com/users/RogerThomas/events{/privacy}","received_events_url":"https://api.github.com/users/RogerThomas/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pydata/pandas/labels/Dtypes","name":"Dtypes","color":"e102d8"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Numeric","name":"Numeric","color":"006b75"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Performance","name":"Performance","color":"a10c02"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/pydata/pandas/milestones/40","html_url":"https://github.com/pydata/pandas/milestones/0.18.2","labels_url":"https://api.github.com/repos/pydata/pandas/milestones/40/labels","id":1639795,"number":40,"title":"0.18.2","description":"","creator":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"open_issues":185,"closed_issues":147,"state":"open","created_at":"2016-03-11T21:24:45Z","updated_at":"2016-06-01T14:43:22Z","due_on":"2016-06-14T04:00:00Z","closed_at":null},"comments":4,"created_at":"2016-05-31T16:41:01Z","updated_at":"2016-06-01T17:35:16Z","closed_at":null,"body":"In python3 with certain sets of single precision floating point data pandas groupby is up to ~150 slower than the same data in python2\r\n#### Code Sample, a copy-pastable example if possible\r\n```python\r\nimport numpy as np\r\nimport pandas as pd\r\nfrom numpy.random import random\r\nfrom time import time\r\n\r\n\r\ndef do_groupby(df):\r\n    df.groupby(['a'])['b'].sum()\r\n\r\n\r\ndef main():\r\n    tmp1 = (random(10000) * 0.1).astype(np.float32)\r\n    tmp2 = (random(10000) * 10.0).astype(np.float32)\r\n    tmp = np.concatenate((tmp1, tmp2))\r\n    arr = np.repeat(tmp, 100)\r\n    df = pd.DataFrame(dict(a=arr, b=arr))\r\n    t1 = time()\r\n    do_groupby(df)\r\n    print(\"Took: %s\" % (time() - t1,))\r\n\r\nmain()\r\n```\r\nOn my machine\r\npython2 this_file.py\r\nThe groupby takes around 0.1s\r\n\r\nWhere as python3 this_file.py\r\nThe groupby takes around 11s\r\n\r\nWith some investigation the discrepancy in run time between python versions varies hugely between the actual data but seems to have the biggest difference when half the data is roughly 100 times smaller than the other half.\r\n\r\nHaving profiled this, it seems this function is taking almost all the 11s in the python3 version in this method\r\nhttps://github.com/pydata/pandas/blob/master/pandas/hashtable.pyx#L538\r\n\r\nHowever I have no idea what is causing the run time discrepancy.\r\n#### output of ``pd.show_versions()``\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.5.1.final.0\r\npython-bits: 64\r\nOS: Linux\r\nOS-release: 4.4.0-22-generic\r\nmachine: x86_64\r\nprocessor: x86_64\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: en_IE.UTF-8\r\n\r\npandas: 0.18.1\r\nnose: 1.3.7\r\npip: 8.1.2\r\nsetuptools: 21.2.2\r\nCython: 0.23.4\r\nnumpy: 1.11.0\r\nscipy: 0.16.1\r\nstatsmodels: 0.6.1\r\nxarray: None\r\nIPython: 4.2.0\r\nsphinx: 1.3.1\r\npatsy: 0.4.1\r\ndateutil: 2.5.3\r\npytz: 2016.4\r\nblosc: None\r\nbottleneck: None\r\ntables: 3.2.0\r\nnumexpr: 2.5.2\r\nmatplotlib: 1.5.0\r\nopenpyxl: None\r\nxlrd: None\r\nxlwt: None\r\nxlsxwriter: None\r\nlxml: None\r\nbs4: None\r\nhtml5lib: None\r\nhttplib2: 0.9.2\r\napiclient: None\r\nsqlalchemy: 1.0.9\r\npymysql: 0.7.4.None\r\npsycopg2: None\r\njinja2: 2.8\r\nboto: 2.38.0\r\npandas_datareader: None\r\n"},"comment":{"url":"https://api.github.com/repos/pydata/pandas/issues/comments/223067721","html_url":"https://github.com/pydata/pandas/issues/13335#issuecomment-223067721","issue_url":"https://api.github.com/repos/pydata/pandas/issues/13335","id":223067721,"user":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"created_at":"2016-06-01T17:35:16Z","updated_at":"2016-06-01T17:35:16Z","body":"see the issue I referenced. I think you can fix it by changing the hashing code as indicated.\r\n\r\nI think that python 3 is taking great care with the hashing, somewhat to the detriment of perf. (which is what the ``Float64Hashtable`` ultimately does via ``kh_*`` functions). The adjustment makes it much faster (and for all intents and purposes works the same)."}}
{"actor":{"display_login":"jreback"}}