{"action":"closed","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/13281","repository_url":"https://api.github.com/repos/pydata/pandas","labels_url":"https://api.github.com/repos/pydata/pandas/issues/13281/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/13281/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/13281/events","html_url":"https://github.com/pydata/pandas/issues/13281","id":156708037,"number":13281,"title":"Wrong behaviour of groupby with diff and merge","user":{"login":"rmoutie","id":19568452,"avatar_url":"https://avatars.githubusercontent.com/u/19568452?v=3","gravatar_id":"","url":"https://api.github.com/users/rmoutie","html_url":"https://github.com/rmoutie","followers_url":"https://api.github.com/users/rmoutie/followers","following_url":"https://api.github.com/users/rmoutie/following{/other_user}","gists_url":"https://api.github.com/users/rmoutie/gists{/gist_id}","starred_url":"https://api.github.com/users/rmoutie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rmoutie/subscriptions","organizations_url":"https://api.github.com/users/rmoutie/orgs","repos_url":"https://api.github.com/users/rmoutie/repos","events_url":"https://api.github.com/users/rmoutie/events{/privacy}","received_events_url":"https://api.github.com/users/rmoutie/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-25T09:49:49Z","updated_at":"2016-05-25T12:00:32Z","closed_at":"2016-05-25T12:00:32Z","body":"While writing a piece of code similar to the example below, I stumbled on a problematic interaction between groupby, diff and merge.\r\n\r\nMy debugging efforts showed that this problem is likely related to the \"fast_apply\" optimisation Pandas uses when using apply().\r\n\r\nAs shown below, when using string values as grouping keys, the first value encountered overrides all the others. Removing either the diff call, the merge, or replacing \"a\" and \"b\" by numerical values all solve the issue, and give a correct (albeit obviously incomplete for my purpose) result.\r\n\r\n#### Code\r\n```\r\nimport pandas as pd\r\nimport numpy as np\r\n\r\nbar = np.random.randint(0, 10, 4)\r\nfoo = ['a'] * 2 + ['b'] * 2\r\nvalues = zip(foo, bar)\r\ndf = pd.DataFrame(values, columns=['foo', 'bar'])\r\nprint df\r\n\r\n\r\ndef try_diff_merge(df):\r\n    df['inf_bar'] = df.bar < 6\r\n    df['inf_bar_diff'] = df['inf_bar'].diff()\r\n    df_merge = df.groupby(['inf_bar']).count()\r\n    df_merge.drop(['inf_bar_diff'], axis=1, inplace=True)\r\n    df_merge = pd.merge(df, df_merge, how='left', right_index=True,\r\n                        left_on=['inf_bar'])\r\n    return df_merge\r\n\r\ndf_group = df.groupby(['foo']).apply(try_diff_merge)\r\nprint df_group\r\n```\r\n#### Output\r\n```\r\n  foo  bar\r\n0   a    2\r\n1   a    0\r\n2   b    6\r\n3   b    8\r\n  foo_x  bar_x inf_bar inf_bar_diff  foo_y  bar_y\r\n0     a      2    True          NaN      2      2\r\n1     a      0    True        False      2      2\r\n2     a      6   False          NaN      2      2\r\n3     a      8   False        False      2      2\r\n```\r\n#### Expected Output\r\n```\r\n  foo  bar\r\n0   a    2\r\n1   a    0\r\n2   b    6\r\n3   b    8\r\n  foo_x  bar_x inf_bar inf_bar_diff  foo_y  bar_y\r\n0     a      2    True          NaN      2      2\r\n1     a      0    True        False      2      2\r\n2     b      6   False          NaN      2      2\r\n3     b      8   False        False      2      2\r\n```\r\n\r\n#### output of ``pd.show_versions()``\r\n```\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 2.7.11.final.0\r\npython-bits: 64\r\nOS: Windows\r\nOS-release: 7\r\nmachine: AMD64\r\nprocessor: Intel64 Family 6 Model 61 Stepping 4, GenuineIntel\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: None\r\n\r\npandas: 0.18.0\r\nnose: 1.3.7\r\npip: 8.1.1\r\nsetuptools: 20.3\r\nCython: 0.23.4\r\nnumpy: 1.10.4\r\nscipy: 0.17.0\r\nstatsmodels: 0.6.1\r\nxarray: None\r\nIPython: 4.1.2\r\nsphinx: 1.3.5\r\npatsy: 0.4.0\r\ndateutil: 2.5.1\r\npytz: 2016.2\r\nblosc: None\r\nbottleneck: 1.0.0\r\ntables: 3.2.2\r\nnumexpr: 2.5\r\nmatplotlib: 1.5.1\r\nopenpyxl: 2.3.2\r\nxlrd: 0.9.4\r\nxlwt: 1.0.0\r\nxlsxwriter: 0.8.4\r\nlxml: 3.6.0\r\nbs4: 4.4.1\r\nhtml5lib: None\r\nhttplib2: None\r\napiclient: None\r\nsqlalchemy: 1.0.12\r\npymysql: None\r\npsycopg2: 2.6.1 (dt dec pq3 ext lo64)\r\njinja2: 2.8\r\nboto: 2.39.0\r\nNone\r\n```\r\n"}}
{"actor":{"display_login":"jreback"}}