{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11034","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11034/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11034/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11034/events","html_url":"https://github.com/apache/spark/pull/11034","id":130844260,"number":11034,"title":"[SPARK-13149][SQL]Add FileStreamSource","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":25,"created_at":"2016-02-02T22:45:36Z","updated_at":"2016-02-09T22:15:03Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11034","html_url":"https://github.com/apache/spark/pull/11034","diff_url":"https://github.com/apache/spark/pull/11034.diff","patch_url":"https://github.com/apache/spark/pull/11034.patch"},"body":"`FileStreamSource` is an implementation of `org.apache.spark.sql.execution.streaming.Source`. It takes advantage of the existing `HadoopFsRelationProvider` to support various file formats. It remembers files in each batch and stores it into the metadata files so as to recover them when restarting. The metadata files are stored in the file system. There will be a further PR to clean up the metadata files periodically. \r\n\r\nThis is based on the initial work from @marmbrus."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/182103180","html_url":"https://github.com/apache/spark/pull/11034#issuecomment-182103180","issue_url":"https://api.github.com/repos/apache/spark/issues/11034","id":182103180,"user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"created_at":"2016-02-09T22:15:03Z","updated_at":"2016-02-09T22:15:03Z","body":"Offline discussion: \r\nIt will be good to have a unit test that documents/tests the behavior of what file source gets generated when format and/or schema is provided/not provided and existing files present / not present, what is the schema generated in the source. So that we have clear understanding of what the behavior is."}}