{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15581","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15581/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15581/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15581/events","html_url":"https://github.com/apache/spark/pull/15581","id":184429201,"number":15581,"title":"[SPARK-18044][STREAMING] FileStreamSource should not infer partitions in every batch","user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars.githubusercontent.com/u/3182036?v=3","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-21T08:36:13Z","updated_at":"2016-10-21T08:39:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15581","html_url":"https://github.com/apache/spark/pull/15581","diff_url":"https://github.com/apache/spark/pull/15581.diff","patch_url":"https://github.com/apache/spark/pull/15581.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIn `FileStreamSource.getBatch`, we will create a `DataSource` with specified schema, to avoid inferring the schema again and again. However, we don't pass the partition columns, and will infer the partition again and again.\r\n\r\nThis PR fixes it by keeping the partition columns in `FileStreamSource`, like schema.\r\n\r\n## How was this patch tested?\r\n\r\nN/A\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/255326230","html_url":"https://github.com/apache/spark/pull/15581#issuecomment-255326230","issue_url":"https://api.github.com/repos/apache/spark/issues/15581","id":255326230,"user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars.githubusercontent.com/u/3182036?v=3","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"created_at":"2016-10-21T08:39:30Z","updated_at":"2016-10-21T08:39:30Z","body":"CC @zsxwing @brkyvz @yhuai  "}}
{"actor":{"display_login":"cloud-fan"}}