{"push_id":1433892180,"size":2,"distinct_size":2,"ref":"refs/heads/branch-2.0","head":"0758df6d5a172fe3d559ea702a05135d42f1fda2","before":"729cadb6bf897a24e1ac1d26d8bc6f59d88cc289","commits":[{"sha":"254e33f4b2db14fe9438b67023a0b721f9f61a3f","author":{"name":"Sandeep Singh","email":"69043e619d79bcf4ae068e5b131d66c33e73727d@techaddict.me"},"message":"[SPARK-18274][ML][PYSPARK] Memory leak in PySpark JavaWrapper\n\n## What changes were proposed in this pull request?\nIn`JavaWrapper `'s destructor make Java Gateway dereference object in destructor, using `SparkContext._active_spark_context._gateway.detach`\nFixing the copying parameter bug, by moving the `copy` method from `JavaModel` to `JavaParams`\n\n## How was this patch tested?\n```scala\nimport random, string\nfrom pyspark.ml.feature import StringIndexer\n\nl = [(''.join(random.choice(string.ascii_uppercase) for _ in range(10)), ) for _ in range(int(7e5))]  # 700000 random strings of 10 characters\ndf = spark.createDataFrame(l, ['string'])\n\nfor i in range(50):\n    indexer = StringIndexer(inputCol='string', outputCol='index')\n    indexer.fit(df)\n```\n* Before: would keep StringIndexer strong reference, causing GC issues and is halted midway\nAfter: garbage collection works as the object is dereferenced, and computation completes\n* Mem footprint tested using profiler\n* Added a parameter copy related test which was failing before.\n\nAuthor: Sandeep Singh <sandeep@techaddict.me>\nAuthor: jkbradley <joseph.kurata.bradley@gmail.com>\n\nCloses #15843 from techaddict/SPARK-18274.\n\n(cherry picked from commit 78bb7f8071379114314c394e0167c4c5fd8545c5)\nSigned-off-by: Joseph K. Bradley <joseph@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/254e33f4b2db14fe9438b67023a0b721f9f61a3f"},{"sha":"0758df6d5a172fe3d559ea702a05135d42f1fda2","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18617][SPARK-18560][TESTS] Fix flaky test: StreamingContextSuite. Receiver data should be deserialized properly\n\n## What changes were proposed in this pull request?\n\nAvoid to create multiple threads to stop StreamingContext. Otherwise, the latch added in #16091 can be passed too early.\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #16105 from zsxwing/SPARK-18617-2.\n\n(cherry picked from commit 086b0c8f6788b205bc630d5ccf078f77b9751af3)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/0758df6d5a172fe3d559ea702a05135d42f1fda2"}]}
{"actor":{"display_login":"ueshin"}}