{"push_id":2815043051,"size":4,"distinct_size":4,"ref":"refs/heads/master","head":"42035a4fec6eb216427486b5067a45fceb65cc2d","before":"99d2e4e00711cffbfaee8cb3da9b6b3feab8ff18","commits":[{"sha":"72ecfd095062ad61c073f9b97bf3c47644575d60","author":{"name":"Bago Amirbekian","email":"f9ab3b835b5fadeacae8b72242f24755535a123f@databricks.com"},"message":"[SPARK-25149][GRAPHX] Update Parallel Personalized Page Rank to test with large vertexIds\n\n## What changes were proposed in this pull request?\n\nrunParallelPersonalizedPageRank in graphx checks that `sources` are <= Int.MaxValue.toLong, but this is not actually required. This check seems to have been added because we use sparse vectors in the implementation and sparse vectors cannot be indexed by values > MAX_INT. However we do not ever index the sparse vector by the source vertexIds so this isn't an issue. I've added a test with large vertexIds to confirm this works as expected.\n\n## How was this patch tested?\n\nUnit tests.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nCloses #22139 from MrBago/remove-veretexId-check-pppr.\n\nAuthored-by: Bago Amirbekian <bago@databricks.com>\nSigned-off-by: Joseph K. Bradley <joseph@databricks.com>","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/72ecfd095062ad61c073f9b97bf3c47644575d60"},{"sha":"6c5cb85856235efd464b109558896f81ae2c4c75","author":{"name":"Jungtaek Lim","email":"71c146bf4d269d28958f88a73164915cec8cbc61@gmail.com"},"message":"[SPARK-24763][SS] Remove redundant key data from value in streaming aggregation\n\n## What changes were proposed in this pull request?\n\nThis patch proposes a new flag option for stateful aggregation: remove redundant key data from value.\nEnabling new option runs similar with current, and uses less memory for state according to key/value fields of state operator.\n\nPlease refer below link to see detailed perf. test result:\nhttps://issues.apache.org/jira/browse/SPARK-24763?focusedCommentId=16536539&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16536539\n\nSince the state between enabling the option and disabling the option is not compatible, the option is set to 'disable' by default (to ensure backward compatibility), and OffsetSeqMetadata would prevent modifying the option after executing query.\n\n## How was this patch tested?\n\nModify unit tests to cover both disabling option and enabling option.\nAlso did manual tests to see whether propose patch improves state memory usage.\n\nCloses #21733 from HeartSaVioR/SPARK-24763.\n\nAuthored-by: Jungtaek Lim <kabhwan@gmail.com>\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/6c5cb85856235efd464b109558896f81ae2c4c75"},{"sha":"ac0174e55af2e935d41545721e9f430c942b3a0c","author":{"name":"Gengliang Wang","email":"bfe1a7b36425bd89859a70a36586361d28ec35eb@databricks.com"},"message":"[SPARK-25129][SQL] Make the mapping of com.databricks.spark.avro to built-in module configurable\n\n## What changes were proposed in this pull request?\n\nIn https://issues.apache.org/jira/browse/SPARK-24924, the data source provider com.databricks.spark.avro is mapped to the new package org.apache.spark.sql.avro .\n\nAs per the discussion in the [Jira](https://issues.apache.org/jira/browse/SPARK-24924) and PR #22119, we should make the mapping configurable.\n\nThis PR also improve the error message when data source of Avro/Kafka is not found.\n\n## How was this patch tested?\n\nUnit test\n\nCloses #22133 from gengliangwang/configurable_avro_mapping.\n\nAuthored-by: Gengliang Wang <gengliang.wang@databricks.com>\nSigned-off-by: Xiao Li <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/ac0174e55af2e935d41545721e9f430c942b3a0c"},{"sha":"42035a4fec6eb216427486b5067a45fceb65cc2d","author":{"name":"Jungtaek Lim","email":"71c146bf4d269d28958f88a73164915cec8cbc61@gmail.com"},"message":"[SPARK-24441][SS] Expose total estimated size of states in HDFSBackedStateStoreProvider\n\n## What changes were proposed in this pull request?\n\nThis patch exposes the estimation of size of cache (loadedMaps) in HDFSBackedStateStoreProvider as a custom metric of StateStore.\n\nThe rationalize of the patch is that state backed by HDFSBackedStateStoreProvider will consume more memory than the number what we can get from query status due to caching multiple versions of states. The memory footprint to be much larger than query status reports in situations where the state store is getting a lot of updates: while shallow-copying map incurs additional small memory usages due to the size of map entities and references, but row objects will still be shared across the versions. If there're lots of updates between batches, less row objects will be shared and more row objects will exist in memory consuming much memory then what we expect.\n\nWhile HDFSBackedStateStore refers loadedMaps in HDFSBackedStateStoreProvider directly, there would be only one `StateStoreWriter` which refers a StateStoreProvider, so the value is not exposed as well as being aggregated multiple times. Current state metrics are safe to aggregate for the same reason.\n\n## How was this patch tested?\n\nTested manually. Below is the snapshot of UI page which is reflected by the patch:\n\n<img width=\"601\" alt=\"screen shot 2018-06-05 at 10 16 16 pm\" src=\"https://user-images.githubusercontent.com/1317309/40978481-b46ad324-690e-11e8-9b0f-e80528612a62.png\">\n\nPlease refer \"estimated size of states cache in provider total\" as well as \"count of versions in state cache in provider\".\n\nCloses #21469 from HeartSaVioR/SPARK-24441.\n\nAuthored-by: Jungtaek Lim <kabhwan@gmail.com>\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/42035a4fec6eb216427486b5067a45fceb65cc2d"}]}
{"actor":{"display_login":"dongjoon-hyun"}}