{"push_id":2853468434,"size":3,"distinct_size":3,"ref":"refs/heads/master","head":"925449283dcaef80e0f77e60aea6ef988bd697b4","before":"8440e3072898e545f0aadbbf123df3450f222a67","commits":[{"sha":"39a02d8f75def7191c66d388729ba1721c92188d","author":{"name":"ankurgupta","email":"5321457e76eb7f871d898a6af269bd30caa76633@cloudera.com"},"message":"[SPARK-24415][CORE] Fixed the aggregated stage metrics by retaining stage objects in liveStages until all tasks are complete\n\nThe problem occurs because stage object is removed from liveStages in\nAppStatusListener onStageCompletion. Because of this any onTaskEnd event\nreceived after onStageCompletion event do not update stage metrics.\n\nThe fix is to retain stage objects in liveStages until all tasks are complete.\n\n1. Fixed the reproducible example posted in the JIRA\n2. Added unit test\n\nCloses #22209 from ankuriitg/ankurgupta/SPARK-24415.\n\nAuthored-by: ankurgupta <ankur.gupta@cloudera.com>\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/39a02d8f75def7191c66d388729ba1721c92188d"},{"sha":"c66eef84409b103accfcaa5073d50426e70b7870","author":{"name":"Dongjoon Hyun","email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org"},"message":"[SPARK-25306][SQL][FOLLOWUP] Change `test` to `ignore` in FilterPushdownBenchmark\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up of #22313 and aim to ignore the micro benchmark test which takes over 2 minutes in Jenkins.\n- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/4939/consoleFull\n\n## How was this patch tested?\n\nThe test case should be ignored in Jenkins.\n```\n[info] FilterPushdownBenchmark:\n...\n[info] - Pushdown benchmark with many filters !!! IGNORED !!!\n```\n\nCloses #22336 from dongjoon-hyun/SPARK-25306-2.\n\nAuthored-by: Dongjoon Hyun <dongjoon@apache.org>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/c66eef84409b103accfcaa5073d50426e70b7870"},{"sha":"925449283dcaef80e0f77e60aea6ef988bd697b4","author":{"name":"WeichenXu","email":"b702da6e50975f71fc6618ef763b578cd392d17f@databricks.com"},"message":"[SPARK-22666][ML][SQL] Spark datasource for image format\n\n## What changes were proposed in this pull request?\n\nImplement an image schema datasource.\n\nThis image datasource support:\n  - partition discovery (loading partitioned images)\n  - dropImageFailures (the same behavior with `ImageSchema.readImage`)\n  - path wildcard matching (the same behavior with `ImageSchema.readImage`)\n  - loading recursively from directory (different from `ImageSchema.readImage`, but use such path: `/path/to/dir/**`)\n\nThis datasource **NOT** support:\n  - specify `numPartitions` (it will be determined by datasource automatically)\n  - sampling (you can use `df.sample` later but the sampling operator won't be pushdown to datasource)\n\n## How was this patch tested?\nUnit tests.\n\n## Benchmark\nI benchmark and compare the cost time between old `ImageSchema.read` API and my image datasource.\n\n**cluster**: 4 nodes, each with 64GB memory, 8 cores CPU\n**test dataset**: Flickr8k_Dataset (about 8091 images)\n\n**time cost**:\n- My image datasource time (automatically generate 258 partitions):  38.04s\n- `ImageSchema.read` time (set 16 partitions): 68.4s\n- `ImageSchema.read` time (set 258 partitions):  90.6s\n\n**time cost when increase image number by double (clone Flickr8k_Dataset and loads double number images)**:\n- My image datasource time (automatically generate 515 partitions):  95.4s\n- `ImageSchema.read` (set 32 partitions): 109s\n- `ImageSchema.read` (set 515 partitions):  105s\n\nSo we can see that my image datasource implementation (this PR) bring some performance improvement compared against old`ImageSchema.read` API.\n\nCloses #22328 from WeichenXu123/image_datasource.\n\nAuthored-by: WeichenXu <weichen.xu@databricks.com>\nSigned-off-by: Xiangrui Meng <meng@databricks.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/925449283dcaef80e0f77e60aea6ef988bd697b4"}]}
{"actor":{"display_login":"ueshin"}}