{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/3227","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/3227/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/3227/comments","events_url":"https://api.github.com/repos/dask/dask/issues/3227/events","html_url":"https://github.com/dask/dask/issues/3227","id":301091949,"number":3227,"title":"Error using df.index in and condition","user":{"login":"postelrich","id":3619876,"avatar_url":"https://avatars2.githubusercontent.com/u/3619876?v=4","gravatar_id":"","url":"https://api.github.com/users/postelrich","html_url":"https://github.com/postelrich","followers_url":"https://api.github.com/users/postelrich/followers","following_url":"https://api.github.com/users/postelrich/following{/other_user}","gists_url":"https://api.github.com/users/postelrich/gists{/gist_id}","starred_url":"https://api.github.com/users/postelrich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/postelrich/subscriptions","organizations_url":"https://api.github.com/users/postelrich/orgs","repos_url":"https://api.github.com/users/postelrich/repos","events_url":"https://api.github.com/users/postelrich/events{/privacy}","received_events_url":"https://api.github.com/users/postelrich/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-02-28T16:29:29Z","updated_at":"2018-02-28T19:34:23Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"When trying to filter on the index and another condition I get a `ValueError: 'dtype' inference failed in 'elemwise'` error. The identical condition works in pandas. Issue seems related to the fact that `df.index` returns a dask array. On dask 0.17.1.\r\n\r\nExample\r\n```\r\nimport pandas as pd\r\nimport dask.dataframe as dd\r\n\r\ndf = pd.DataFrame({'time': pd.date_range('2018-01-01', periods=10),\r\n                   'mycat': list('ABCABCABCA')})\r\ndf = df.set_index('time')\r\ndf['mycat'] = df.mycat.astype('category')\r\n(df.index > '2018-01-01') & (df.mycat == 'A')\r\n\r\nddf = dd.from_pandas(df, npartitions=2)\r\n(ddf.index > '2018-01-01') & (ddf.mycat == 'A')\r\n```\r\n\r\nStacktrace\r\n```\r\n---------------------------------------------------------------------------\r\nValueError                                Traceback (most recent call last)\r\n<ipython-input-7-11ce8cc93904> in <module>()\r\n      8 (df.index > '2018-01-01') & (df.mycat == 'A')\r\n      9 ddf = dd.from_pandas(df, npartitions=2)\r\n---> 10 (ddf.index > '2018-01-01') & (ddf.mycat == 'A')\r\n     11 # (ddf.mycat == 'A') & (ddf.index > '2018-01-01')\r\n\r\n~/anaconda3/envs/cme/lib/python3.6/site-packages/dask/array/core.py in __and__(self, other)\r\n   1455 \r\n   1456     def __and__(self, other):\r\n-> 1457         return elemwise(operator.and_, self, other)\r\n   1458 \r\n   1459     def __rand__(self, other):\r\n\r\n~/anaconda3/envs/cme/lib/python3.6/site-packages/dask/array/core.py in elemwise(op, *args, **kwargs)\r\n   2935                 if not is_scalar_for_elemwise(a) else a\r\n   2936                 for a in args]\r\n-> 2937         dt = apply_infer_dtype(op, vals, {}, 'elemwise', suggest_dtype=False)\r\n   2938         need_enforce_dtype = any(not is_scalar_for_elemwise(a) and a.ndim == 0 for a in args)\r\n   2939 \r\n\r\n~/anaconda3/envs/cme/lib/python3.6/site-packages/dask/array/core.py in apply_infer_dtype(func, args, kwargs, funcname, suggest_dtype)\r\n    544         msg = None\r\n    545     if msg is not None:\r\n--> 546         raise ValueError(msg)\r\n    547     return o.dtype\r\n    548 \r\n\r\nValueError: `dtype` inference failed in `elemwise`.\r\n\r\nOriginal error is below:\r\n------------------------\r\nAttributeError(\"'numpy.ndarray' object has no attribute 'index'\",)\r\n\r\nTraceback:\r\n---------\r\n  File \"/Users/rpostelnik/anaconda3/envs/cme/lib/python3.6/site-packages/dask/array/core.py\", line 529, in apply_infer_dtype\r\n    o = func(*args, **kwargs)\r\n  File \"/Users/rpostelnik/anaconda3/envs/cme/lib/python3.6/site-packages/dask/dataframe/core.py\", line 1706, in __array_wrap__\r\n    index = context[1][0].index\r\n```"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/369356383","html_url":"https://github.com/dask/dask/issues/3227#issuecomment-369356383","issue_url":"https://api.github.com/repos/dask/dask/issues/3227","id":369356383,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2018-02-28T19:34:23Z","updated_at":"2018-02-28T19:34:23Z","author_association":"OWNER","body":"Ah, @jcrist 's response narrowly beat mine.  I'm happy to take a backseat on this decision if preferred.  I suspect that both of you likely have more experience and thoughts on this than I do."}}
{"actor":{"display_login":"mrocklin"}}