{"push_id":2551264275,"size":5,"distinct_size":5,"ref":"refs/heads/master","head":"6ea582e36ab0a2e4e01340f6fc8cfb8d493d567d","before":"487faf17ab96c8edb729501dfb1ff82f7b2c6031","commits":[{"sha":"e3de6ab30d52890eb08578e55eb4a5d2b4e7aa35","author":{"name":"Maxim Gekk","email":"05b3389ec46e7c78a10351def567499e4feebe3f@databricks.com"},"message":"[SPARK-24068] Propagating DataFrameReader's options to Text datasource on schema inferring\n\n## What changes were proposed in this pull request?\n\nWhile reading CSV or JSON files, DataFrameReader's options are converted to Hadoop's parameters, for example there:\nhttps://github.com/apache/spark/blob/branch-2.3/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L302\n\nbut the options are not propagated to Text datasource on schema inferring, for instance:\nhttps://github.com/apache/spark/blob/branch-2.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala#L184-L188\n\nThe PR proposes propagation of user's options to Text datasource on scheme inferring in similar way as user's options are converted to Hadoop parameters if schema is specified.\n\n## How was this patch tested?\nThe changes were tested manually by using https://github.com/twitter/hadoop-lzo:\n\n```\nhadoop-lzo> mvn clean package\nhadoop-lzo> ln -s ./target/hadoop-lzo-0.4.21-SNAPSHOT.jar ./hadoop-lzo.jar\n```\nCreate 2 test files in JSON and CSV format and compress them:\n```shell\n$ cat test.csv\ncol1|col2\na|1\n$ lzop test.csv\n$ cat test.json\n{\"col1\":\"a\",\"col2\":1}\n$ lzop test.json\n```\nRun `spark-shell` with hadoop-lzo:\n```\nbin/spark-shell --jars ~/hadoop-lzo/hadoop-lzo.jar\n```\nreading compressed CSV and JSON without schema:\n```scala\nspark.read.option(\"io.compression.codecs\", \"com.hadoop.compression.lzo.LzopCodec\").option(\"inferSchema\",true).option(\"header\",true).option(\"sep\",\"|\").csv(\"test.csv.lzo\").show()\n+----+----+\n|col1|col2|\n+----+----+\n|   a|   1|\n+----+----+\n```\n```scala\nspark.read.option(\"io.compression.codecs\", \"com.hadoop.compression.lzo.LzopCodec\").option(\"multiLine\", true).json(\"test.json.lzo\").printSchema\nroot\n |-- col1: string (nullable = true)\n |-- col2: long (nullable = true)\n```\n\nAuthor: Maxim Gekk <maxim.gekk@databricks.com>\nAuthor: Maxim Gekk <max.gekk@gmail.com>\n\nCloses #21182 from MaxGekk/text-options.","distinct":true,"url":"https://api.github.com/repos/htynkn/spark/commits/e3de6ab30d52890eb08578e55eb4a5d2b4e7aa35"},{"sha":"9498e528d21e286e496da6ea9bf9c7ad73a7b5bd","author":{"name":"Dongjoon Hyun","email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org"},"message":"[SPARK-23355][SQL][DOC][FOLLOWUP] Add migration doc for TBLPROPERTIES\n\n## What changes were proposed in this pull request?\n\nIn Apache Spark 2.4, [SPARK-23355](https://issues.apache.org/jira/browse/SPARK-23355) fixes a bug which ignores table properties during convertMetastore for tables created by STORED AS ORC/PARQUET.\n\nFor some Parquet tables having table properties like TBLPROPERTIES (parquet.compression 'NONE'), it was ignored by default before Apache Spark 2.4. After upgrading cluster, Spark will write uncompressed file which is different from Apache Spark 2.3 and old.\n\nThis PR adds a migration note for that.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #21269 from dongjoon-hyun/SPARK-23355-DOC.","distinct":true,"url":"https://api.github.com/repos/htynkn/spark/commits/9498e528d21e286e496da6ea9bf9c7ad73a7b5bd"},{"sha":"7e7350285dc22764f599671d874617c0eea093e5","author":{"name":"Lu WANG","email":"f5d8070e57b18bcecf5fec8e7ae5ed1bab3fbdf8@databricks.com"},"message":"[SPARK-24132][ML] Instrumentation improvement for classification\n\n## What changes were proposed in this pull request?\n\n- Add OptionalInstrumentation as argument for getNumClasses in ml.classification.Classifier\n\n- Change the function call for getNumClasses in train() in ml.classification.DecisionTreeClassifier, ml.classification.RandomForestClassifier, and ml.classification.NaiveBayes\n\n- Modify the instrumentation creation in ml.classification.LinearSVC\n\n- Change the log call in ml.classification.OneVsRest and ml.classification.LinearSVC\n\n## How was this patch tested?\n\nManual.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Lu WANG <lu.wang@databricks.com>\n\nCloses #21204 from ludatabricks/SPARK-23686.","distinct":true,"url":"https://api.github.com/repos/htynkn/spark/commits/7e7350285dc22764f599671d874617c0eea093e5"},{"sha":"cac9b1dea1bb44fa42abf77829c05bf93f70cf20","author":{"name":"Ryan Blue","email":"4c9a82ce72ca2519f38d0af0abbb4cecb9fceca9@apache.org"},"message":"[SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0.\n\n## What changes were proposed in this pull request?\n\nThis updates Parquet to 1.10.0 and updates the vectorized path for buffer management changes. Parquet 1.10.0 uses ByteBufferInputStream instead of byte arrays in encoders. This allows Parquet to break allocations into smaller chunks that are better for garbage collection.\n\n## How was this patch tested?\n\nExisting Parquet tests. Running in production at Netflix for about 3 months.\n\nAuthor: Ryan Blue <blue@apache.org>\n\nCloses #21070 from rdblue/SPARK-23972-update-parquet-to-1.10.0.","distinct":true,"url":"https://api.github.com/repos/htynkn/spark/commits/cac9b1dea1bb44fa42abf77829c05bf93f70cf20"},{"sha":"6ea582e36ab0a2e4e01340f6fc8cfb8d493d567d","author":{"name":"DB Tsai","email":"0430fe8116355e2ecfa7f2b79341c5ad5a243c66@apple.com"},"message":"[SPARK-24181][SQL] Better error message for writing sorted data\n\n## What changes were proposed in this pull request?\n\nThe exception message should clearly distinguish sorting and bucketing in `save` and `jdbc` write.\n\nWhen a user tries to write a sorted data using save or insertInto, it will throw an exception with message that `s\"'$operation' does not support bucketing right now\"\"`.\n\nWe should throw `s\"'$operation' does not support sortBy right now\"\"` instead.\n\n## How was this patch tested?\n\nMore tests in `DataFrameReaderWriterSuite.scala`\n\nAuthor: DB Tsai <d_tsai@apple.com>\n\nCloses #21235 from dbtsai/fixException.","distinct":true,"url":"https://api.github.com/repos/htynkn/spark/commits/6ea582e36ab0a2e4e01340f6fc8cfb8d493d567d"}]}
{"actor":{"display_login":"htynkn"}}