{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14638","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14638/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14638/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14638/events","html_url":"https://github.com/apache/spark/pull/14638","id":171080035,"number":14638,"title":"[SPARK-11374][SQL] Support `skip.header.line.count` option in Hive External Table","user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":27,"created_at":"2016-08-14T21:31:30Z","updated_at":"2016-11-14T18:20:57Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14638","html_url":"https://github.com/apache/spark/pull/14638","diff_url":"https://github.com/apache/spark/pull/14638.diff","patch_url":"https://github.com/apache/spark/pull/14638.patch"},"body":"## What changes were proposed in this pull request?\n\n`CREATE EXTERNAL TABLE` or `CREATE TABLE ... LOCATION` uses external files as a data source for the tables. External files like CSV frequently contains one or more header lines as their own metadata. So, **skip.header.line.count** table property is used in order to exclude those metadata lines from data lines.\n\n[SPARK-11374](https://issues.apache.org/jira/browse/SPARK-11374) is a long lasting issue since 1.5.1, and all current Spark versions have the same situation. It focused on EXTERNAL TABLE on CSV, but it's a general need for external tables. This PR implements the logic to handle `skip.header.line.count` for EXTERNAL TABLEs. The following is an example for SPARK-11374.\n\n**CSV Data**\n\n``` bash\n$ cat /data/test.csv \nc1,c2\n1,a\n2,b\n```\n\n**Scala**\n\n``` scala\nscala> spark.read.option(\"header\",\"true\").csv(\"/data\").show\n+---+---+\n| c1| c2|\n+---+---+\n|  1|  a|\n|  2|  b|\n+---+---+\n```\n\n**SQL**\n\n``` scala\nscala> sql(\"CREATE TABLE t1 (id INT, b VARCHAR(10)) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/data' TBLPROPERTIES('skip.header.line.count'='1')\")\nscala> sql(\"SELECT * FROM t1\").show\n+---+---+\n| id|  b|\n+---+---+\n|  1|  a|\n|  2|  b|\n+---+---+\n```\n## How was this patch tested?\n\nPass the Jenkins test with a new test case.\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/260416797","html_url":"https://github.com/apache/spark/pull/14638#issuecomment-260416797","issue_url":"https://api.github.com/repos/apache/spark/issues/14638","id":260416797,"user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"created_at":"2016-11-14T18:20:57Z","updated_at":"2016-11-14T18:20:57Z","body":"Retest this please."}}
{"actor":{"display_login":"dongjoon-hyun"}}