{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15813","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15813/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15813/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15813/events","html_url":"https://github.com/apache/spark/pull/15813","id":188088890,"number":15813,"title":"[SPARK-18362][SQL] Use TextFileFormat in JsonFileFormat and CSVFileFormat","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-11-08T20:00:04Z","updated_at":"2016-11-08T21:57:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15813","html_url":"https://github.com/apache/spark/pull/15813","diff_url":"https://github.com/apache/spark/pull/15813.diff","patch_url":"https://github.com/apache/spark/pull/15813.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis patch significantly improves the IO / file listing performance of schema inference in Spark's built-in CSV and JSON data sources.\r\n\r\nPreviously, these data sources used the legacy `SparkContext.hadoopFile` and `SparkContext.hadoopRDD` methods to read files during their schema inference steps, causing huge file-listing bottlenecks on the driver.\r\n\r\nThis patch refactors this logic to use Spark SQL's `text` data source to read files during this step. The text data source still performs some unnecessary file listing (since in theory we already have resolved the table prior to schema inference and therefore should be able to scan without performing _any_ extra listing), but that listing is much faster and takes place in parallel. In one production workload operating over tens of thousands of JSON files, this change managed to reduce schema inference time from 7 minutes to 2 minutes.\r\n\r\n## How was this patch tested?\r\n\r\nExisting unit tests, plus manual benchmarking on a production workload."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/259272368","html_url":"https://github.com/apache/spark/pull/15813#issuecomment-259272368","issue_url":"https://api.github.com/repos/apache/spark/issues/15813","id":259272368,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-11-08T21:57:17Z","updated_at":"2016-11-08T21:57:17Z","body":"**[Test build #68359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68359/consoleFull)** for PR 15813 at commit [`acce60d`](https://github.com/apache/spark/commit/acce60d4f584f549a44f6dd90e3db5e6694a61f0).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}