{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14579","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14579/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14579/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14579/events","html_url":"https://github.com/apache/spark/pull/14579","id":170364337,"number":14579,"title":"[SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers","user":{"login":"MLnick","id":1036807,"avatar_url":"https://avatars.githubusercontent.com/u/1036807?v=3","gravatar_id":"","url":"https://api.github.com/users/MLnick","html_url":"https://github.com/MLnick","followers_url":"https://api.github.com/users/MLnick/followers","following_url":"https://api.github.com/users/MLnick/following{/other_user}","gists_url":"https://api.github.com/users/MLnick/gists{/gist_id}","starred_url":"https://api.github.com/users/MLnick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MLnick/subscriptions","organizations_url":"https://api.github.com/users/MLnick/orgs","repos_url":"https://api.github.com/users/MLnick/repos","events_url":"https://api.github.com/users/MLnick/events{/privacy}","received_events_url":"https://api.github.com/users/MLnick/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-08-10T08:48:38Z","updated_at":"2016-08-10T18:58:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14579","html_url":"https://github.com/apache/spark/pull/14579","diff_url":"https://github.com/apache/spark/pull/14579.diff","patch_url":"https://github.com/apache/spark/pull/14579.patch"},"body":"JIRA: https://issues.apache.org/jira/browse/SPARK-16921\r\n\r\nContext managers are a natural way to capture closely related setup and teardown code in Python. It can be useful to apply this pattern to persisting/unpersisting RDDs and DataFrames.\r\n\r\nThis PR makes RDDs and DataFrames implement the context manager `__enter__` and `__exit__`  functions, allowing code such as:\r\n\r\n```python\r\nwith labeled_data.persist():\r\n    model = pipeline.fit(labeled_data)\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nNew doc tests."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/238968415","html_url":"https://github.com/apache/spark/pull/14579#issuecomment-238968415","issue_url":"https://api.github.com/repos/apache/spark/issues/14579","id":238968415,"user":{"login":"MLnick","id":1036807,"avatar_url":"https://avatars.githubusercontent.com/u/1036807?v=3","gravatar_id":"","url":"https://api.github.com/users/MLnick","html_url":"https://github.com/MLnick","followers_url":"https://api.github.com/users/MLnick/followers","following_url":"https://api.github.com/users/MLnick/following{/other_user}","gists_url":"https://api.github.com/users/MLnick/gists{/gist_id}","starred_url":"https://api.github.com/users/MLnick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MLnick/subscriptions","organizations_url":"https://api.github.com/users/MLnick/orgs","repos_url":"https://api.github.com/users/MLnick/repos","events_url":"https://api.github.com/users/MLnick/events{/privacy}","received_events_url":"https://api.github.com/users/MLnick/received_events","type":"User","site_admin":false},"created_at":"2016-08-10T18:58:47Z","updated_at":"2016-08-10T18:58:47Z","body":"yeah it would break pipelining - I don't think it will necessarily throw an error though.\r\n\r\ne.g.\r\n\r\n```\r\nIn [22]: rdd = sc.parallelize([\"b\", \"a\", \"c\"])\r\n\r\nIn [23]: type(rdd)\r\nOut[23]: pyspark.rdd.RDD\r\n\r\nIn [24]: mapped = rdd.map(lambda x: x)\r\n\r\nIn [25]: type(mapped)\r\nOut[25]: pyspark.rdd.PipelinedRDD\r\n\r\nIn [26]: mapped._is_pipelinable()\r\nOut[26]: True\r\n\r\nIn [27]: p = mapped.cache()\r\n\r\nIn [28]: type(p)\r\nOut[28]: pyspark.rdd.PersistedRDD\r\n\r\nIn [29]: p._is_pipelinable()\r\n---------------------------------------------------------------------------\r\nAttributeError                            Traceback (most recent call last)\r\n<ipython-input-29-02496125eccd> in <module>()\r\n----> 1 p._is_pipelinable()\r\n\r\nAttributeError: 'PersistedRDD' object has no attribute '_is_pipelinable'\r\n\r\nIn [30]: mapped2 = p.map(lambda x: x)\r\n\r\nIn [31]: type(mapped2)\r\nOut[31]: pyspark.rdd.PipelinedRDD\r\n```\r\n\r\nSo I think chaining will work, but the pipelined RDD thinks mapped2 is the 1st transformation, while it is actually the 2nd. I _think_ this will just be an efficiency issue rather than a correctness issue however. \r\n\r\nWe could possibly work around it with some type checking etc but it then starts to feel like adding more complexity than the feature is worth..."}}
{"actor":{"display_login":"MLnick"}}