{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21618","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21618/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21618/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21618/events","html_url":"https://github.com/apache/spark/pull/21618","id":335070885,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk2ODgzMzA2","number":21618,"title":"[SPARK-20408][SQL] Get the glob path in parallel to reduce resolve relation time","user":{"login":"xuanyuanking","id":4833765,"node_id":"MDQ6VXNlcjQ4MzM3NjU=","avatar_url":"https://avatars0.githubusercontent.com/u/4833765?v=4","gravatar_id":"","url":"https://api.github.com/users/xuanyuanking","html_url":"https://github.com/xuanyuanking","followers_url":"https://api.github.com/users/xuanyuanking/followers","following_url":"https://api.github.com/users/xuanyuanking/following{/other_user}","gists_url":"https://api.github.com/users/xuanyuanking/gists{/gist_id}","starred_url":"https://api.github.com/users/xuanyuanking/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xuanyuanking/subscriptions","organizations_url":"https://api.github.com/users/xuanyuanking/orgs","repos_url":"https://api.github.com/users/xuanyuanking/repos","events_url":"https://api.github.com/users/xuanyuanking/events{/privacy}","received_events_url":"https://api.github.com/users/xuanyuanking/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-06-23T05:33:14Z","updated_at":"2018-06-23T07:05:10Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21618","html_url":"https://github.com/apache/spark/pull/21618","diff_url":"https://github.com/apache/spark/pull/21618.diff","patch_url":"https://github.com/apache/spark/pull/21618.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR change the work of getting glob path in parallel, which can make complex wildcard path more quickly, the mainly changes in details:\r\n1.Add new function getGlobbedPaths in DataSource, return all paths represented by the wildcard pattern, use a local thread pool to do this while the paths number expanded from patten lager than `spark.sql.sources.parallelGetGlobbedPath.threshold`. The local thread pool size controlled by `spark.sql.sources.parallelGetGlobbedPath.numThreads`\r\n2.Add new function expandGlobPath in SparkHadoopUtil, to expand the dir represented by the patten, here we mainly reuse the logic in org.apache.hadoop.fs.Globber.glob().\r\n\r\n## How was this patch tested?\r\n\r\nAdd UT in SparkHadoopUtilSuite.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/399643708","html_url":"https://github.com/apache/spark/pull/21618#issuecomment-399643708","issue_url":"https://api.github.com/repos/apache/spark/issues/21618","id":399643708,"node_id":"MDEyOklzc3VlQ29tbWVudDM5OTY0MzcwOA==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-06-23T07:05:10Z","updated_at":"2018-06-23T07:05:10Z","author_association":"NONE","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}