{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/13040","repository_url":"https://api.github.com/repos/pydata/pandas","labels_url":"https://api.github.com/repos/pydata/pandas/issues/13040/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/13040/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/13040/events","html_url":"https://github.com/pydata/pandas/issues/13040","id":152013181,"number":13040,"title":"API: use argument-free lambdas for injecting DataFrames columns as variables?","user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pydata/pandas/labels/API%20Design","name":"API Design","color":"AD7FA8"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Needs%20Discussion","name":"Needs Discussion","color":"207de5"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/pydata/pandas/milestones/39","html_url":"https://github.com/pydata/pandas/milestones/0.19.0","labels_url":"https://api.github.com/repos/pydata/pandas/milestones/39/labels","id":1570595,"number":39,"title":"0.19.0","description":"after 0.18.x series","creator":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"open_issues":29,"closed_issues":0,"state":"open","created_at":"2016-02-08T15:30:21Z","updated_at":"2016-05-05T05:14:26Z","due_on":"2016-08-31T04:00:00Z","closed_at":null},"comments":12,"created_at":"2016-04-30T06:20:39Z","updated_at":"2016-05-05T22:08:34Z","closed_at":null,"body":"With [a little bit of magic](https://github.com/shoyer/pandas-magic/blob/master/pandas_magic/__init__.py#L13-L33), we could make the following syntax work:\r\n```python\r\n(df[lambda: sepal_length > 3]\r\n .groupby(lambda: pd.cut(sepal_width, 5))\r\n .apply(lambda: petal_width.mean()))\r\n```\r\n\r\nSyntax like `df[lambda: sepal_length > 3]` is an alternative to more verbose alternatives like the recently added `df[lambda x: x.sepal_length > 3]` (https://github.com/pydata/pandas/issues/11485). Here we use `lambda` essentially in place of a macro that would allow for delayed evaluation (which of course are not supported by Python syntax).\r\n\r\nMy proposal is to add support for such \"thunks\" to every pandas method that accepts a callable function that currently must take a single argument work on DataFrames.\r\n\r\nUnder the covers, this works by (1) copying the `globals()` dictionary at evaluation time and (2) injecting the current DataFrame into it. We would further ensure that this only works on lambda functions, by checking `f.func_name == '<lambda>'`.\r\n\r\nThe main gotcha is that [it isn't possible to dynamically override ~~local~~ non-global variables](http://stackoverflow.com/questions/8028708/dynamically-set-local-variable-in-python) without [some true dark magic](https://gist.github.com/njsmith/2347382#file-inject-py-L40). This means that code like the following is going to behave contrary to expectations:\r\n```python\r\ndef x_plus_one(df):\r\n    x = 0\r\n    # uses x = 0 instead of df.x\r\n    return df.pipe(lambda: x + 1)\r\n\r\ndf = pd.DataFrame({'x': np.arange(100)})\r\nresult = x_plus_one(df)  # all 1s, not range(1, 101)\r\n```\r\nIs this so bad? Shadowing variables in an outer scope is already poor design, but this is a pretty serious departure from expectations.\r\n\r\nThe other danger is that this could mask bugs, e.g., if a user mistakenly types `df.pipe(lambda: x)` instead of `df.pipe(lambda x: x)`. This is an unavoidable danger of spelling two APIs with similar syntax.\r\n\r\nOn the plus side, this proposal is safer than @njsmith's \"true dark magic\" context manager (see above) for injecting DataFrame columns, because there's no possibility of variable assignment inside a `lambda`.\r\n\r\nWould this be a good idea for pandas?"},"comment":{"url":"https://api.github.com/repos/pydata/pandas/issues/comments/217294080","html_url":"https://github.com/pydata/pandas/issues/13040#issuecomment-217294080","issue_url":"https://api.github.com/repos/pydata/pandas/issues/13040","id":217294080,"user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"created_at":"2016-05-05T22:08:34Z","updated_at":"2016-05-05T22:08:34Z","body":"@rsdenijs I wrote this some months ago, and it worked on an earlier version of pandas with Python 2.7. It is quite likely that something has broken my hack -- you are welcome to look into fixing it."}}